RFR(S): 8199811: com/sun/jdi/ProcessAttachTest.java fails intermittently: Remote thread failed for unknown reason

Mon Aug 6 04:28:33 UTC 2018

Hi David,

On 8/5/18 5:33 PM, David Holmes wrote:
> Hi Chris,
>
> On 4/08/2018 4:23 AM, Chris Plummer wrote:
>> Hello,
>>
>> Please review the following fix for JDK12:
>>
>> http://cr.openjdk.java.net/~cjplummer/8199811/webrev.00
>>
>> https://bugs.openjdk.java.net/browse/JDK-8199811
>>
>> The root of the problem is that there is no code in the solaris or 
>> windows AttachListener support that ensures that the listener is done 
>> being initialized before attaching and  attempting to enqueue the 
>> first command. The enqueue operation fails when is sees that the 
>> listener is not attached yet.
>>
>> I was able to force this failure to happen every time by adding a 10 
>> second sleep in attach_listener_thread_entry() just before the call 
>> to AttachListener::set_initialized(). This did not cause macosx or 
>> linux to fail, but did make solaris fail (failures had not been noted 
>> previously) and windows to fail (failures previously had been 
>> observed, but very rarely).
>
> I'm having trouble seeing the complete code paths here to understand 
> the control flow for initialization and subsequent use. How do we get 
> to the enqueue logic (that fails) if the initialization logic has not 
> yet completed? Is the init logic asynchronous? (If so I would expect 
> many more failures of this nature.)
It's seems to be very convoluted on Windows, and somewhat on Solaris 
too. Took me a while to figure it out enough to fix this issue. Yes, it 
appears the init logic is async w.r.t. the enqueue logic. On Windows the 
init logic is always triggered during VM startup. I had previously 
inserted some pns2() calls to see how it gets initialized. Here's what 
it produced:

V  [jvm.dll+0x30b7fd]  AttachListener::init+0x2dd  (attachlistener.cpp:466)
V  [jvm.dll+0xd59513]  Threads::create_vm+0x873  (thread.cpp:3817)
V  [jvm.dll+0x862e71]  JNI_CreateJavaVM_inner+0xd1  (jni.cpp:3945)
V  [jvm.dll+0x8667af]  JNI_CreateJavaVM+0x1f  (jni.cpp:4036)
C  [java.exe+0x36df]
C  [java.exe+0x19fcc]
C  [KERNEL32.DLL+0x13d2]
C  [ntdll.dll+0x154f4]

V  [jvm.dll+0x30c473]  AttachListener::pd_init+0x23  (attachlistener_windows.cpp:394)
V  [jvm.dll+0x30a6fc]  attach_listener_thread_entry+0x17c  (attachlistener.cpp:352)
V  [jvm.dll+0xd622ac]  JavaThread::run+0x3ec  (thread.cpp:1732)
V  [jvm.dll+0xba191e]  thread_native_entry+0x11e  (os_windows.cpp:456)
C  [ucrtbase.DLL+0x1d885]
C  [KERNEL32.DLL+0x13d2]
C  [ntdll.dll+0x154f4]

So AttachListener::init() starts a new thread with 
attach_listener_thread_entry() as the entrypoint. That's also the method 
that eventually calls  AttachListener::set_initialized(), and also where 
I stuck the 10 second delay before the call to 
AttachListener::set_initialized() to make it fail every time on Windows 
and Solaris. AttachListener::init() only starts the listener thread. It 
does not wait for it to become initialized.

The enqeue() is done in a very strange way on Windows that I don't fully 
understand. I could never get a native stack trace since it's done from 
a thread that is not attached to the JVM.  From the attaching side  you 
get the following exception when the enqueue() fails:

java.lang.InternalError: Remote thread failed for unknown reason (100)
     at jdk.attach/sun.tools.attach.VirtualMachineImpl.enqueue(Native 
Method)
     at 
jdk.attach/sun.tools.attach.VirtualMachineImpl.execute(VirtualMachineImpl.java:106)
     at 
jdk.attach/sun.tools.attach.HotSpotVirtualMachine.executeCommand(HotSpotVirtualMachine.java:309)
     at 
jdk.attach/sun.tools.attach.HotSpotVirtualMachine.getAgentProperties(HotSpotVirtualMachine.java:202)
     at 
jdk.jdi/com.sun.tools.jdi.ProcessAttachingConnector.attach(ProcessAttachingConnector.java:103)
     at ProcessAttachTest.tryDebug(ProcessAttachTest.java:107)
     at ProcessAttachTest.runTest(ProcessAttachTest.java:87)
     at ProcessAttachTest.main(ProcessAttachTest.java:66)
     at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
     at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
     at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at java.base/java.lang.reflect.Method.invoke(Method.java:566)
     at 
com.sun.javatest.regtest.agent.MainActionHelper$SameVMRunnable.run(MainActionHelper.java:229)
     at java.base/java.lang.Thread.run(Thread.java:834)

If you look at the windows 
Java_sun_tools_attach_VirtualMachineImpl_enqueue() method, you'll see 
how it creates a process in the target VM, and has it execute 
JVM_EnqueueOperation(), which calls Win32AttachListener::enqueue(), 
which is where the return code of 100 (ATTACH_ERROR_DISABLED) is 
produced, and also where I put my sleep code to give initialization a 
chance to complete.
>
>> The proposed fix is to have the enqueue code sleep for up to 20 
>> seconds (in 1 second intervals) waiting for initialization to be 
>> complete. I found this fixed the problem, even with the 10 second 
>> sleep in attach_listener_thread_entry() still in place. A shorter 
>> sleep is probably fine. I'm open to suggestions. Since this timing 
>> issue was so rare, my guess is that a single 1 second sleep is likely 
>> to always fix it, but since it is so hard to reproduce (without the 
>> 10 second sleep in place), I can't say for sure.
>
> That seems reasonable.
>
> Not sure what the interruption issue is that you and Gary discussed. 
> The os-level sleep function can only be interrupted by signals, and 
> this thread shouldn't be receiving any signals in general.So it's not 
> something I would be concerned about.
Ok.
>
>> Another approach to fixing this would be to use some sort of 
>> synchronization between the init and enqueue code, like a condition 
>> variable. I think I know how to do this with pthread_cond_wait() and 
>> pthread_cond_signal(), although it gets to be a bit tricky since I'd 
>> probably have to make the enqueue code create the condvar if 
>> initialization is not yet complete, and then have the initialization 
>> code check for the existence of the condvar when initialization is 
>> complete, and signal on it if it exists. I'm pretty sure there's a 
>> potential for race condition in there. I haven't thought it through 
>> enough to say for sure. I also looked a bit at condition variable 
>> support on windows, and it looks like I could do something similar 
>> there too. However, I think the sleep approach I have implement is 
>> far more straight forward and less error prone, so I'd prefer to 
>> stick with it if others approve.
>
> Can't comment on this without understanding exactly where the race is.
Ok. Let me know if you need any more details.

thanks,

Chris
>
> Thanks,
> David
>
>>
>> thanks,
>>
>> Chris
>>
>>