RFR: 8268965: TCP Connection Reset when connecting simple socket to SSL server

[Alexey Bakhtin]

On Thu, 17 Jun 2021 13:20:54 GMT, Alexey Bakhtin <abakhtin at openjdk.org> wrote:

> Please review the fix for JDK-8268965.
> 
> The new jtreg test is added for the described issue.
> sun/security/ssl and javax/net/ssl tests are passed

Hi Xuelei,
Thank you for your interest in this issue.
In this particular test case, a simple socket connects to the SSL server with a dummy message and waiting for the server's response.
SSL server fails to verify the message header and aborts the connection. The stack trace to socket close on the server side is:
java.lang.Throwable
	at java.base/sun.security.ssl.SSLSocketImpl.closeSocket(SSLSocketImpl.java:1785)
	at java.base/sun.security.ssl.SSLSocketImpl.shutdown(SSLSocketImpl.java:1756)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:431)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:309)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:145)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1500)
	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1415)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:450)
	at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:915)
	at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1006)
	at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:966)
	at java.base/java.io.DataInputStream.readUnsignedShort(DataInputStream.java:343)
	at java.base/java.io.DataInputStream.readUTF(DataInputStream.java:595)
	at java.base/java.io.DataInputStream.readUTF(DataInputStream.java:570)
	at SSLSocketReset$ServerThread.run(SSLSocketReset.java:97)
On the client side application waiting for response from the server but fails due to socket close by the server.
Current JDK code causes the client fails with:
java.net.SocketException: Connection reset
	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:183)
	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:137)
	at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966)
	at java.base/java.io.DataInputStream.readFully(DataInputStream.java:201)
	at java.base/java.io.DataInputStream.readUTF(DataInputStream.java:614)
	at java.base/java.io.DataInputStream.readUTF(DataInputStream.java:570)
	at SSLSocketReset.main(SSLSocketReset.java:60)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
	at java.base/java.lang.Thread.run(Thread.java:833)

In case of proposed patch applied, the client fails with 
java.io.EOFException
	at java.base/java.io.DataInputStream.readFully(DataInputStream.java:203)
	at java.base/java.io.DataInputStream.readUTF(DataInputStream.java:614)
	at java.base/java.io.DataInputStream.readUTF(DataInputStream.java:570)
	at SSLSocketReset.main(SSLSocketReset.java:60)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
	at java.base/java.lang.Thread.run(Thread.java:833)

SSL server closes the socket during the handshake, so no changes if we try to close the socket from the application

-------------

PR: https://git.openjdk.java.net/jdk/pull/4520