Inconsistent SSLEngine behavior for closing outbound while in handshake in 11ea22

[Xuelei Fan]

Hi Tim,

Would you mind look at the code I posted in the following thread:
http://mail.openjdk.java.net/pipermail/security-dev/2018-July/017708.html

In the update, we are trying make the synchronization more simple and 
robust.  I appreciate if you could comment by the end of this week.

Note that with this update, a complete TLS connection should close both 
inbound and outbound explicitly.  However, existing applications may not 
did this way because TLS 1.2 and prior version can work around it.  But 
for TLS 1.3, it is possible to hang the application if the connection is 
not closed.  If the source code update is not available, please consider 
to use the "jdk.tls.acknowledgeCloseNotify" System Property as a workaround.

Thanks,
Xuelei

On 7/18/2018 11:51 AM, Tim Brooks wrote:
> Yes. I can test once there is a patch. My inquiry was motivated by some 
> work on Elasticsearch fyi. I can test a patch against that work.
> 
> https://github.com/elastic/elasticsearch/issues/32144
> 
> - Tim
> 
>> On Jul 17, 2018, at 8:40 PM, Xuelei Fan <xuelei.fan at oracle.com 
>> <mailto:xuelei.fan at oracle.com>> wrote:
>>
>> Hi,
>>
>> We are working on the JDK 11 close issue.
>> https://bugs.openjdk.java.net/browse/JDK-8207009
>>
>> I appreciate if you can help test if we have a patch.
>>
>> Thanks,
>> Xuelei
>>
>> On 7/17/2018 4:26 PM, Tim Brooks wrote:
>>> My understanding is that when you are interested in closing the 
>>> underlying socket when using the SSLEngine, you must call 
>>> closeOutbound() and WRAP and UNWRAP until both isInboundDone() and 
>>> isOutboundDone() return true.
>>> One edge case of this is if you are interested in closing the socket 
>>> prior to the completion of a handshake. In JDK 10.0.1 (and I believe 
>>> prior JDKs) this was the behavior for one way in which this arises:
>>> 1. Initiate handshake
>>> 2. UNWRAP data from client
>>> 3. WRAP data to send to client. Handshake status is "NEED_UNWRAP"
>>> 4. Call closeOutbound() (perhaps the server is shutting down and you 
>>> want to close the connection).
>>> 5. Handshake status now returns "NEED_WRAP"
>>> JDK10:
>>> isInboundDone() - returns false
>>> isOutboundDone() - returns false
>>> A call to wrap() produces 7 bytes and status = CLOSED. Handshake 
>>> status is now NEED_UNWRAP.
>>> isInboundDone() - returns false
>>> isOutboundDone() - returns true
>>> JDK11:
>>> isInboundDone() - returns true
>>> isOutboundDone() - returns false
>>> A call to wrap() throws the following exception:
>>> javax.net.ssl.SSLException: Cannot kickstart, the connection is 
>>> broken or closed
>>> at 
>>> java.base/sun.security.ssl.TransportContext.kickstart(TransportContext.java:205)
>>> at 
>>> java.base/sun.security.ssl.SSLEngineImpl.writeRecord(SSLEngineImpl.java:167)
>>> at java.base/sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:138)
>>> at java.base/sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:116)
>>> at java.base/javax.net.ssl.SSLEngine.wrap(SSLEngine.java:471)
>>> I’m not sure what the procedure for closing a connection prior to 
>>> handshake completion is for TLS. But obviously this is a scenario 
>>> that can arise. It seems wrong to me that the state transitions for 
>>> the SSLEngine do not handle this. The fact that “isOutboundDone()” 
>>> returns false, but I cannot WRAP seems to be an issue.
>