RFR 8066708: JMXStartStopTest fails to connect to port 38112

[Stuart Marks]

On 12/8/14 12:35 PM, Jaroslav Bachorik wrote:
> Please, review the following test change
>
> Issue : https://bugs.openjdk.java.net/browse/JDK-8066708
> Webrev: http://cr.openjdk.java.net/~jbachorik/8066708/webrev.00
>
> The test fails very intermittently when RMI registry is trying to bind to a port
> previously used in the test (via ServerSocket).
>
> This seems to be caused by the sockets created via `new ServerSocket(0)` and
> being in reusable mode. The fix attempts to prevent this by explicitly
> forbidding the reusable mode.

Hi Jaroslav,

I happened to see this fly by, and there are (I think) some similar issues going 
on in the RMI tests.

But first I'll note that I don't think setReuseAddress() will have the effect 
that you want. Typically it's set to true before binding a socket, so that a 
subsequent bind operation will succeed even if the address/port is already in 
use. ServerSockets created with new ServerSocket(0) are already bound, and I'm 
not sure what calling setReuseAddress(false) will do on such sockets. The spec 
says behavior is undefined, but my bet is that it does nothing.

I guess it doesn't hurt to try this out to see if it makes a difference, but I 
don't have much confidence it will help.

The potential similarity to the RMI tests is exemplified by JDK-8049202 (sorry, 
this bug report isn't open) but briefly this tests the RMI registry as follows:

1. Opens port 1099 using new ServerSocket(1099) [1099 is the default
    RMI registry port] in order to ensure that 1099 isn't in use by
    something else already;

2. If this succeeds, it immediately closes the ServerSocket.

3. Then it creates a new RMI registry on port 1099.

In principle, this should succeed, yet it fails around 10% of the time on some 
systems. The error is "port already in use". My best theory is that even though 
the socket has just been closed by a user program, the kernel has to run the 
socket through some of the socket states such as FIN_WAIT_1, FIN_WAIT_2, or 
CLOSING before the socket is actually closed and is available for reuse. If a 
program -- even the same one -- attempts to open a socket on the same port 
before the socket has reached its final state, it will get an "already in use 
error".

If this is true I don't believe that setting SO_REUSEADDR will work if the 
socket is in one of these final states. (I remember reading this somewhere but 
I'm not sure where at the moment. I can try to dig it up if there is interest.)

I admit this is just a theory and I'm open to alternatives, and I'm also open to 
hearing about ways to deal with this problem.

Could something similar be going on with this JMX test?

s'marks