Integrated: 8352773: JVMTI should disable events during java upcalls

[Serguei Spitsyn]

On Wed, 9 Apr 2025 08:14:04 GMT, Serguei Spitsyn <sspitsyn at openjdk.org> wrote:

> As noted in [JDK-8352088](https://bugs.openjdk.org/browse/JDK-8352088), JVMTI `GetThreadGroupChildren` does an upcall to java. This results in  a`ClassPrepare` event the first time it does this, and these events can cause problems (deadlocks) for the debugger or debug agent. The  [JDK-8352088](https://bugs.openjdk.org/browse/JDK-8352088) was fixed to get rid of class loading during Java upcall from `GetThreadGroupChildren`. However, some other events can be generated as well. It is more safe to disable all JVMTI events during debugger-related upcalls originated by JVMTI.
> The `ClassPrepare` events are important for the debug agent. So, an assert was added into `ClassPrepare` event generation to make sure there are no attempts to post this event during upcalls.
> Some specific implementation details can be added to the first PR comment.
> 
> Testing:
>  - Verified with the test `jdk/com/sun/jdi/EarlyThreadGroupChildrenTest.java` that was added with the fix of [JDK-8352088](https://bugs.openjdk.org/browse/JDK-8352088):
>    - the assert described above is fired if the fix of JDK-8352088 is removed
>    - the test is passed without if the fix of JDK-8352088 is removed and the assert is removed
>  - Ran mach5 tiers 1-6

This pull request has now been integrated.

Changeset: 1c34f3cd
Author:    Serguei Spitsyn <sspitsyn at openjdk.org>
URL:       https://git.openjdk.org/jdk/commit/1c34f3cdb1df1b9bd01c6795e19a78753e3b555a
Stats:     39 lines in 6 files changed: 37 ins; 0 del; 2 mod

8352773: JVMTI should disable events during java upcalls

Reviewed-by: lmesnik, dholmes, cjplummer, coleenp

-------------

PR: https://git.openjdk.org/jdk/pull/24539