RFR: 8281615: Deadlock caused by jdwp agent [v2]

[Chris Plummer]

On Wed, 16 Feb 2022 01:17:12 GMT, Zhengyu Gu <zgu at openjdk.org> wrote:

>>> I am not sure if it is possible, but checking bagSize(deletedSignatures) == 0 seems to race against classTrack_reset() where it does not take handlerLock lock.
>> 
>> I had thought of that too, but I think the way `classTrack_reset()` is called, it is likely not possible for there to be a `classTrack_processUnloads()` also coming in because everything is shut down:
>> 
>> ``` 
>>     threadControl_onDisconnect();
>>     standardHandlers_onDisconnect();
>> 
>>     /*
>>      * Cut off the transport immediately. This has the effect of
>>      * cutting off any events that the eventHelper thread might
>>      * be trying to send.
>>      */
>>     transport_close();
>>     debugMonitorDestroy(cmdQueueLock);
>> 
>>     /* Reset for a new connection to this VM if it's still alive */
>>     if ( ! gdata->vmDead ) {
>>         debugInit_reset(getEnv());    <--- calls classTrack_reset()
>>     }
>
> But shutting down debug loop does not seem to have effect on ongoing jvmti callback, e.g. thread 4 in the bug report.

The thread 4 callback is triggered due to the debugger having requested `CLASS_UNLOAD` events. This is shutdown by `eventHandler_reset()`, which is called by `debugInit_reset()` before it calls `classTrack_reset()`. So that means by the time we get to `classTrack_reset()`, the thread 4 callback is no longer possible.

-------------

PR: https://git.openjdk.java.net/jdk/pull/7461