RFR: 8352773: JVMTI should disable events during java upcalls [v4]

[Chris Plummer]

On Thu, 10 Apr 2025 17:59:15 GMT, Serguei Spitsyn <sspitsyn at openjdk.org> wrote:

>> As noted in [JDK-8352088](https://bugs.openjdk.org/browse/JDK-8352088), JVMTI `GetThreadGroupChildren` does an upcall to java. This results in  a`ClassPrepare` event the first time it does this, and these events can cause problems (deadlocks) for the debugger or debug agent. The  [JDK-8352088](https://bugs.openjdk.org/browse/JDK-8352088) was fixed to get rid of class loading during Java upcall from `GetThreadGroupChildren`. However, some other events can be generated as well. It is more safe to disable all JVMTI events during debugger-related upcalls originated by JVMTI.
>> The `ClassPrepare` events are important for the debug agent. So, an assert was added into `ClassPrepare` event generation to make sure there are no attempts to post this event during upcalls.
>> Some specific implementation details can be added to the first PR comment.
>> 
>> Testing:
>>  - Verified with the test `jdk/com/sun/jdi/EarlyThreadGroupChildrenTest.java` that was added with the fix of [JDK-8352088](https://bugs.openjdk.org/browse/JDK-8352088):
>>    - the assert described above is fired if the fix of JDK-8352088 is removed
>>    - the test is passed without if the fix of JDK-8352088 is removed and the assert is removed
>>  - Ran mach5 tiers 1-6
>
> Serguei Spitsyn has updated the pull request incrementally with one additional commit since the last revision:
> 
>   fix a build time error for minimal VM config

Marked as reviewed by cjplummer (Reviewer).

-------------

PR Review: https://git.openjdk.org/jdk/pull/24539#pullrequestreview-2758302537