RFR: 8281615: Deadlock caused by jdwp agent [v2]

[Zhengyu Gu]

On Tue, 15 Feb 2022 20:08:19 GMT, Chris Plummer <cjplummer at openjdk.org> wrote:

>> Zhengyu Gu has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   David and Chris' comments
>
> src/jdk.jdwp.agent/share/native/libjdwp/classTrack.c line 112:
> 
>> 110:       debugMonitorExit(classTrackLock);
>> 111:     } while (retry == JNI_TRUE);
>> 112:     bagDestroyBag(new_bag);
> 
> One other comment about `classTrack_processUnloads()` in general that also applies to the original version. It seems pretty inefficient. Every time the debug agent gets an event it is called. On almost ever call it is returning and empty back and allocating a new one. It seems a check for `bagSize(deletedSignatures) == 0` and returning NULL if true would help performance. I also believe this can be done outside of the lock (would like David's opinion on this).

I am not sure if it is possible, but checking `bagSize(deletedSignatures) == 0` seems to race against `classTrack_reset()` where it does not take `handlerLock` lock.

-------------

PR: https://git.openjdk.java.net/jdk/pull/7461