JDK 11+21 SSLSocket.close() deadlock?

[Simone Bordet]

Hi,

On Wed, Jul 11, 2018 at 2:25 AM Xuelei Fan <xuelei.fan at oracle.com> wrote:
>
> Hi Simone,
>
> Thank you for reporting this issue.  Now it is tracked in JBS:
>     https://bugs.openjdk.java.net/browse/JDK-8207004
>
> In the following stacks, only one lock (on 0xac) can be observed.  Can I
> understand that the read() is blocked, and then the close() is blocked
> as well?  Did you have a test that we can use to reproduce this issue?

Yes, it's pretty crude but reduces at minimum the dependencies.
You have to provide a valid sslContext variable, but that should be
done for your environment.

Looking at sun.security.ssl.SSLSocketImpl.close(), I see that it tries
to grab the lock for read and then for write.
While the test case below shows a problem for reads, I believe the
same problem can be shown when the SSLSocket is blocked in writes due
to TCP congestion, and some other thread tries to asynchronously
close() it.

Thanks!

public void testDeadlock() throws Exception
{
    int port = 8899;
    // TODO: provide a valid sslContext here.
    SSLServerSocket acceptor =
(SSLServerSocket)sslContext.getServerSocketFactory().createServerSocket(port);

    Socket client = new Socket("localhost", port);

    SSLSocket sslServer = (SSLSocket)acceptor.accept();

    CompletableFuture.runAsync(() ->
    {
        try
        {
            sslServer.startHandshake();
        }
        catch (Throwable x)
        {
            x.printStackTrace();
        }
    });

    // Wait for sslServer to start reading from the client.
    Thread.sleep(1000);

    // Hangs here.
    sslServer.close();

    client.close();
}



-- 
Simone Bordet
---
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless.   Victoria Livschitz