RFR: 8263779: SSLEngine reports NEED_WRAP continuously without producing any further output

[Xue-Lei Andrew Fan]

On Mon, 12 Apr 2021 06:30:43 GMT, djelinski <github.com+30433125+djelinski at openjdk.org> wrote:

>> As described in the bug, by connecting the SSLEngine with a misbehaving peer SSL implementation, it can get into a state where it calling `wrap` reports getStatus == OK, getHandshakeStatus === NEED_WRAP but still doesn't produce any further output.   It happens when the output bound is not empty.
>> 
>> The handshake status could have more precise status if the out bound.  The patch was confirmed by the bug submitter.
>
> src/java.base/share/classes/sun/security/ssl/TransportContext.java line 590:
> 
>> 588:     HandshakeStatus getHandshakeStatus() {
>> 589:         if (!outputRecord.isEmpty()) {
>> 590:             // If no handshaking, special case to wrap alters or
> 
> Suggestion:
> 
>             // If not handshaking, special case to wrap alerts or

Thank you for the correction.

> src/java.base/share/classes/sun/security/ssl/TransportContext.java line 592:
> 
>> 590:             // If no handshaking, special case to wrap alters or
>> 591:             // post-handshake messages.
>> 592:             if (!isOutboundClosed()) {
> 
> If I'm reading the [TransportContect#closeNotify](https://github.com/openjdk/jdk/blob/627ad9fe22a153410c14d0b2061bb7dee2c300af/src/java.base/share/classes/sun/security/ssl/TransportContext.java#L275) and [TransportContext#passiveInboundClose](https://github.com/openjdk/jdk/blob/627ad9fe22a153410c14d0b2061bb7dee2c300af/src/java.base/share/classes/sun/security/ssl/TransportContext.java#L524) correctly, non-empty output record with both inbound and outbound closed happens when we reply with our close_notify to peer's. Now we will return NOT_HANDSHAKING which appears to be wrong.

Good catch!  I will have an update.  Thank you for the code review.

-------------

PR: https://git.openjdk.java.net/jdk/pull/3292