RFR 8182757: JDWP: Socket Transport handshake hangs on Solaris

[Gerald Thornbrugh]

Hi,

Please review this JDK-10 fix for JDK-8182757. 

The bug: https://bugs.openjdk.java.net/browse/JDK-8182757 <https://bugs.openjdk.java.net/browse/JDK-8182757>

The webrev: http://cr.openjdk.java.net/~gthornbr/8182757/webrev.00 <http://cr.openjdk.java.net/~gthornbr/8182757/webrev.00>

If a socket is being setup without a fixed port using the SO_REUSEADDR flag it can lead to other
processes interfering with the poll/receive process of a debugger/debuggee configuring a socket
for communication. When SO_REUSEADDR is used other processes can attempt a listen() on
the same port and receive a connect from the debuggee. This causes the debugger to stay in
poll() waiting for a connect and the debuggee stays in recv() waiting to receive data from the
"rogue" process that will never send it.

This can also lead to connections being terminated early on the debuggee side when the “rogue”
process terminates the connection because it does not receive what it expected from the client
process (i.e. the debuggee).

The fix is to not use the SO_REUSEADDR flag for non-fixed port sockets. This keeps “rogue”
processes from reusing the port address and from stealing the connects sent by the debuggee.

The changes to src/jdk.jdi/share/classes/com/sun/tools/jdi/SocketTransportService.java addresses
when JDI (the debugger side) is acting in “server mode”.

The changes to src/jdk.jdwp.agent/share/native/libdt_socket/socketTransport.c addresses when
JDWP (the debuggee side) is acting in “server mode”.

I have run the JDK JDI tests and the internal Oracle VM/NSK JPDA tests on all platforms without errors.
We were able to reproduce the failures with a specific load on a test machine while running JDI
related tests.  After we applied the fix to this system we were not able to reproduce the failures.

Thanks,

Gerald
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/serviceability-dev/attachments/20170802/a3a2c5dd/attachment.html>