Inconsistent SSLEngine behavior for closing outbound while in handshake in 11ea22

[Tim Brooks]

My understanding is that when you are interested in closing the underlying socket when using the SSLEngine, you must call closeOutbound() and WRAP and UNWRAP until both isInboundDone() and isOutboundDone() return true.

One edge case of this is if you are interested in closing the socket prior to the completion of a handshake. In JDK 10.0.1 (and I believe prior JDKs) this was the behavior for one way in which this arises:

1. Initiate handshake
2. UNWRAP data from client
3. WRAP data to send to client. Handshake status is "NEED_UNWRAP"
4. Call closeOutbound() (perhaps the server is shutting down and you want to close the connection).
5. Handshake status now returns "NEED_WRAP"

JDK10:

isInboundDone() - returns false
isOutboundDone() - returns false

A call to wrap() produces 7 bytes and status = CLOSED. Handshake status is now NEED_UNWRAP.

isInboundDone() - returns false
isOutboundDone() - returns true

JDK11:

isInboundDone() - returns true
isOutboundDone() - returns false

A call to wrap() throws the following exception:

javax.net.ssl.SSLException: Cannot kickstart, the connection is broken or closed
	at java.base/sun.security.ssl.TransportContext.kickstart(TransportContext.java:205)
	at java.base/sun.security.ssl.SSLEngineImpl.writeRecord(SSLEngineImpl.java:167)
	at java.base/sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:138)
	at java.base/sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:116)
	at java.base/javax.net.ssl.SSLEngine.wrap(SSLEngine.java:471)

I’m not sure what the procedure for closing a connection prior to handshake completion is for TLS. But obviously this is a scenario that can arise. It seems wrong to me that the state transitions for the SSLEngine do not handle this. The fact that “isOutboundDone()” returns false, but I cannot WRAP seems to be an issue.