RFR: 8352773: JVMTI should disable events during java upcalls

[Coleen Phillimore]

On Wed, 9 Apr 2025 08:14:04 GMT, Serguei Spitsyn <sspitsyn at openjdk.org> wrote:

> As noted in [JDK-8352088](https://bugs.openjdk.org/browse/JDK-8352088), JVMTI `GetThreadGroupChildren` does an upcall to java. This results in  a`ClassPrepare` event the first time it does this, and these events can cause problems (deadlocks) for the debugger or debug agent. The  [JDK-8352088](https://bugs.openjdk.org/browse/JDK-8352088) was fixed to get rid of class loading during Java upcall from `GetThreadGroupChildren`. However, some other events can be generated as well. It is more safe to disable all JVMTI events during debugger-related upcalls originated by JVMTI.
> The `ClassPrepare` events are important for the debug agent. So, an assert was added into `ClassPrepare` event generation to make sure there are no attempts to post this event during upcalls.
> Some specific implementation details can be added to the first PR comment.
> 
> Testing:
>  - Verified with the test `jdk/com/sun/jdi/EarlyThreadGroupChildrenTest.java` that was added with the fix of [JDK-8352088](https://bugs.openjdk.org/browse/JDK-8352088):
>    - the assert described above is fired if the fix of JDK-8352088 is removed
>    - the test is passed without if the fix of JDK-8352088 is removed and the assert is removed
>  - Ran mach5 tiers 1-6

So this is another case where you have to ignore JVMTI event like in VTMS transitions?  It looks like a good way to fix this in general.

-------------

Marked as reviewed by coleenp (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/24539#pullrequestreview-2754548627