RFR: 8263779: SSLEngine reports NEED_WRAP continuously without producing any further output

[djelinski]

On Wed, 31 Mar 2021 20:52:57 GMT, Xue-Lei Andrew Fan <xuelei at openjdk.org> wrote:

> As described in the bug, by connecting the SSLEngine with a misbehaving peer SSL implementation, it can get into a state where it calling `wrap` reports getStatus == OK, getHandshakeStatus === NEED_WRAP but still doesn't produce any further output.   It happens when the output bound is not empty.
> 
> The handshake status could have more precise status if the out bound.  The patch was confirmed by the bug submitter.

Would it be possible to add a test for that?

src/java.base/share/classes/sun/security/ssl/TransportContext.java line 590:

> 588:     HandshakeStatus getHandshakeStatus() {
> 589:         if (!outputRecord.isEmpty()) {
> 590:             // If no handshaking, special case to wrap alters or

Suggestion:

            // If not handshaking, special case to wrap alerts or

src/java.base/share/classes/sun/security/ssl/TransportContext.java line 592:

> 590:             // If no handshaking, special case to wrap alters or
> 591:             // post-handshake messages.
> 592:             if (!isOutboundClosed()) {

If I'm reading the [TransportContect#closeNotify](https://github.com/openjdk/jdk/blob/627ad9fe22a153410c14d0b2061bb7dee2c300af/src/java.base/share/classes/sun/security/ssl/TransportContext.java#L275) and [TransportContext#passiveInboundClose](https://github.com/openjdk/jdk/blob/627ad9fe22a153410c14d0b2061bb7dee2c300af/src/java.base/share/classes/sun/security/ssl/TransportContext.java#L524) correctly, non-empty output record with both inbound and outbound closed happens when we reply with our close_notify to peer's. Now we will return NOT_HANDSHAKING which appears to be wrong.

-------------

PR: https://git.openjdk.java.net/jdk/pull/3292