Reproducer for JDK-8221218 again

[Domien Nowicki]

Hi everyone,


Recently we've started to use SSLEngine to create our own TLS server. 
However we sometimes got the following exception "SSLHandshakeException: 
Insufficient buffer remaining for AEAD cipher fragment (2)." when 
dealing with client disconnections at server side during the TLS 
handshake. This has been tested with latest JDK21.


We started to investigate and traced it down to the following scenario:

- Setup: Standard SSLSocket as client (all defaults) connects to our TLS 
server using Java SSLEngine as internal engine. Both client and server 
have support for TLSv1.3


1. Client sends hello

       "client version"      : "TLSv1.2",

         "supported_versions (43)": {
           "versions": [TLSv1.3, TLSv1.2]
         },

2. Server processes the client hello, and attempts to send server hello:

       "server version"      : "TLSv1.2",

         "supported_versions (43)": {
           "selected version": [TLSv1.3]
         },

3. From this point, server has switched to TLSv1.3 and is using a 
GcmReadCipher (from SSLCipher class)

4. However, in this scenario, due to network congestion or server 
application being slow, the server hello response is delayed and did not 
reach the client yet

5. The client, still assuming it is operating under TLSv1.2, gets a 
timeout and decides to close the socket. This produces TLSv1.2 alerts 
(user canceled and close notification) and reaches the server

6. Server tries to decrypt the TLSv1.2 alerts, but since TLSv1.2 alerts 
are not encrypted, gets the "SSLHandshakeException: Insufficient buffer 
remaining for AEAD cipher fragment (2)." exception.


If the client did receive the server hello response, it would switch to 
TLSv1.3 properly and would encrypt TLSv1.3 alerts from then on as expected.

Additionally, if we use the same scenario but force TLSv1.2 on the 
server side, then the server does not use the GcmReadCipher at this 
point, and it can process the TLSv1.2 alerts properly.


So it seems the JDK SSLEngine did not take the above scenario into 
account, or are we missing something?


Best regards,

Domien

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/security-dev/attachments/20240503/a58f7d71/attachment.htm>