Inconsistent SSLEngine behavior for closing outbound while in handshake in 11ea22

[Xuelei Fan]

Hi,

We are working on the JDK 11 close issue.
    https://bugs.openjdk.java.net/browse/JDK-8207009

I appreciate if you can help test if we have a patch.

Thanks,
Xuelei

On 7/17/2018 4:26 PM, Tim Brooks wrote:
> My understanding is that when you are interested in closing the underlying socket when using the SSLEngine, you must call closeOutbound() and WRAP and UNWRAP until both isInboundDone() and isOutboundDone() return true.
> 
> One edge case of this is if you are interested in closing the socket prior to the completion of a handshake. In JDK 10.0.1 (and I believe prior JDKs) this was the behavior for one way in which this arises:
> 
> 1. Initiate handshake
> 2. UNWRAP data from client
> 3. WRAP data to send to client. Handshake status is "NEED_UNWRAP"
> 4. Call closeOutbound() (perhaps the server is shutting down and you want to close the connection).
> 5. Handshake status now returns "NEED_WRAP"
> 
> JDK10:
> 
> isInboundDone() - returns false
> isOutboundDone() - returns false
> 
> A call to wrap() produces 7 bytes and status = CLOSED. Handshake status is now NEED_UNWRAP.
> 
> isInboundDone() - returns false
> isOutboundDone() - returns true
> 
> JDK11:
> 
> isInboundDone() - returns true
> isOutboundDone() - returns false
> 
> A call to wrap() throws the following exception:
> 
> javax.net.ssl.SSLException: Cannot kickstart, the connection is broken or closed
> 	at java.base/sun.security.ssl.TransportContext.kickstart(TransportContext.java:205)
> 	at java.base/sun.security.ssl.SSLEngineImpl.writeRecord(SSLEngineImpl.java:167)
> 	at java.base/sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:138)
> 	at java.base/sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:116)
> 	at java.base/javax.net.ssl.SSLEngine.wrap(SSLEngine.java:471)
> 
> I’m not sure what the procedure for closing a connection prior to handshake completion is for TLS. But obviously this is a scenario that can arise. It seems wrong to me that the state transitions for the SSLEngine do not handle this. The fact that “isOutboundDone()” returns false, but I cannot WRAP seems to be an issue.
>