Change in closeOutbound() behavior for unused SSLEngine

[Xuelei Fan]

The TLS protocol was changed to use half-close policy since TLS 1.3.  As 
means that sslEngine.closeOutbound() will close the outbound and keep 
the inbound open.  "NEED_UNWRAP" is used to indicate that the engine can 
still be used to decode input message.

For the specific case bellow, it is reasonable to expect 
"NOT_HANDSHAKING" as the handshaking has not been started.  On the other 
side, as only the inbound open, it is also reasonable to me to use 
"NEED_UNWRAP" although there is nothing to unwrap.  I think, using 
""NOT_HANDSHAKING" may lead to confusing about what the next operation, 
wrap() or unwrap(), could be in practice.  CLOSED is not an option to me 
as the inbound is still open.

I understand there might be some compatibility issues for the use of 
half-close policy.  I may close both inbound and outbound of an engine 
in the application code if the connection is not used.

Is there a known compatibility impact on you applications?

Thanks & Regards,
Xuelei

On 10/16/2019 2:53 AM, Simone Bordet wrote:
> Hi,
> 
> SSLContext sslContext = SSLContext.getDefault();
> SSLEngine sslEngine = sslContext.createSSLEngine();
> sslEngine.closeOutbound();
> SSLEngineResult.HandshakeStatus hsStatus = sslEngine.getHandshakeStatus();
> System.err.println("hsStatus = " + hsStatus);
> 
> This prints "NOT_HANDSHAKING" in Java 8 and "NEED_UNWRAP" in JDK 11+.
> 
> In both cases, trying to wrap() consumes and produces 0 bytes (so the
> close_notify is not generated, which I think is fine given that the
> SSLEngine was never used) and produces a CLOSED result.
> 
> This case is common for connections that are established but never
> used (not even a TLS byte was exchanged).
> 
> Is this change in behavior expected?
> 
> I find strange that calling closeOutbound() results in a NEED_UNWRAP
> (as there is nothing to read).
> 
> Thanks!
>