SSLEngine.wrap(...) returns NOT_HANDSHAKING even when the alert was not consumed yet in latest JDK12 release (possible regression).

[Xuelei Fan]

Once TLS handshaking completed, the status would be NOT_HANDSHAKING.  It 
is unlikely to know the content type of the unconsumed data.  When the 
data get consumed, the status may be adjusted accordingly.

Xuelei

On 3/13/2019 7:17 AM, Norman Maurer wrote:
> I assumed that it should return NEED_UNWRAP if there is something left that needs to be consumed (that’s also what happened in JDK11 and before).
> 
> How should we know then that there is an alert to consume ?
> > Bye
> Norman
> 
>> Am 13.03.2019 um 15:11 schrieb Xuelei Fan <xuelei.fan at oracle.com>:
>>
>> Hi Norman,
>>
>> What's the problem in the application?  It is possible that status is NOT_HANDSHAKING even there are bytes left that should be consumed, for example alert or application data.  NOT_HANDSHAKING means that the handshaking is not in progress.
>>
>> Thanks,
>> Xuelei
>>
>>> On 2/28/2019 12:24 AM, Norman Maurer wrote:
>>> Hi all,
>>> I think I found a possible regression / bug in the latest JDK12 release when trying to upgrade the Netty CI server to test with the latest JDK12 release. The problem is that SSLEngine.wrap(…) returns NOT_HANDSHAKING even when there are bytes left that should be consumed (the alert itself). My understanding is that it should only return “NOT_HANDSHAKING” once we also consumed the alert. Please correct me if I wrong tho.
>>> I pushed a reproducer for this here:
>>> https://github.com/normanmaurer/jdk12_ssl_engine_unwrap_bug
>>> When running this on the latest JDK12 release (and later JDK versions) it will fail with an AssertionError, while everything works as expected when using earlier Java versions.
>>> Here is the Java version I used to reproduce:
>>> # java -version
>>> openjdk version "12" 2019-03-19
>>> OpenJDK Runtime Environment (build 12+33)
>>> OpenJDK 64-Bit Server VM (build 12+33, mixed mode, sharing)
>>> It seems like this was not always the case for Java12 tho, as I can not reproduce it with this version:
>>> #java -version
>>> openjdk version "12-ea" 2019-03-19
>>> OpenJDK Runtime Environment (build 12-ea+27)
>>> OpenJDK 64-Bit Server VM (build 12-ea+27, mixed mode, sharing)
>>> I don't have all the “in between” releases on my machine atm so I can not tell exactly on which release this “broke” :/
>>> Thanks
>>> Norman