RFR: 8281615: Deadlock caused by jdwp agent [v2]

[Zhengyu Gu]

On Tue, 15 Feb 2022 20:02:50 GMT, Chris Plummer <cjplummer at openjdk.org> wrote:

>> Zhengyu Gu has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   David and Chris' comments
>
> src/jdk.jdwp.agent/share/native/libjdwp/classTrack.c line 111:
> 
>> 109:       }
>> 110:       debugMonitorExit(classTrackLock);
>> 111:     } while (retry == JNI_TRUE);
> 
> I don't think the retry is necessary. Nor is the check for `deletedSignatures != NULL`. I assume you are trying to cover the case where initially deletedSignatures was NULL (so no bag was allocated), but is not NULL by the time you grab the lock. If you return when NULL like I suggested above, you won't have this issue. Note that if not NULL, there is no way another thread can get into this same code in clear it. This is because all callers first grab the handlerLock. Grabbing of the classTrackLock here is only done to synchronize with cbTrackingObjectFree(), not with other callers of classTrack_processUnloads().

I was worried that `classTrack_activate()` races against 'classTrack_processUnloads()` to set `deletedSignatures`, and you are right, it is also protected under ` handlerLock` lock. It simplifies a code a lot.

-------------

PR: https://git.openjdk.java.net/jdk/pull/7461