JDK 11+21 SSLSocket.close() deadlock?

[Simone Bordet]

Hi,

Please look at the stack traces below.

The server code accept() a SSLSocket, then calls startHandshake() in a
different thread.
The client code sends the TLS handshake bytes very slowly.
The server code waits for a bit for the handshake to finish, then
attempts to close the socket, but it cannot due to the deadlock below.
The client stops sending the TLS handshake bytes and now the server is
stuck forever.

I think invoking close() should wake up the read() with an exception,
rather than trying to grab the same lock held by the read, as
asynchronous closes from other threads happen all the times.

The issue does not happen with JDK 9, 10, 11-ea+18, but does happen
with 11-ea+21.

Thanks!

---

"main at 1" prio=5 tid=0x1 nid=NA waiting for monitor entry
  java.lang.Thread.State: BLOCKED
waiting for pool-1-thread-1 at 2224 to release lock on <0xac7> (a
sun.security.ssl.SSLSocketImpl)
  at sun.security.ssl.SSLSocketImpl.close(SSLSocketImpl.java:447)

"pool-1-thread-1 at 2224" prio=5 tid=0x19 nid=NA runnable
  java.lang.Thread.State: RUNNABLE
blocks main at 1
  at java.net.SocketInputStream.socketRead0(SocketInputStream.java:-1)
  at java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
  at java.net.SocketInputStream.read(SocketInputStream.java:168)
  at java.net.SocketInputStream.read(SocketInputStream.java:140)
  at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:464)
  at sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:167)
  at sun.security.ssl.SSLTransport.decode(SSLTransport.java:108)
  at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:877)
  at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:810)
  - locked <0xac7> (a sun.security.ssl.SSLSocketImpl)
  at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:383)

-- 
Simone Bordet
---
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless.   Victoria Livschitz