RFR: 8352773: JVMTI should disable events during java upcalls

[Serguei Spitsyn]

On Wed, 9 Apr 2025 20:38:08 GMT, Chris Plummer <cjplummer at openjdk.org> wrote:

>> As noted in [JDK-8352088](https://bugs.openjdk.org/browse/JDK-8352088), JVMTI `GetThreadGroupChildren` does an upcall to java. This results in  a`ClassPrepare` event the first time it does this, and these events can cause problems (deadlocks) for the debugger or debug agent. The  [JDK-8352088](https://bugs.openjdk.org/browse/JDK-8352088) was fixed to get rid of class loading during Java upcall from `GetThreadGroupChildren`. However, some other events can be generated as well. It is more safe to disable all JVMTI events during debugger-related upcalls originated by JVMTI.
>> The `ClassPrepare` events are important for the debug agent. So, an assert was added into `ClassPrepare` event generation to make sure there are no attempts to post this event during upcalls.
>> Some specific implementation details can be added to the first PR comment.
>> 
>> Testing:
>>  - Verified with the test `jdk/com/sun/jdi/EarlyThreadGroupChildrenTest.java` that was added with the fix of [JDK-8352088](https://bugs.openjdk.org/browse/JDK-8352088):
>>    - the assert described above is fired if the fix of JDK-8352088 is removed
>>    - the test is passed without if the fix of JDK-8352088 is removed and the assert is removed
>>  - Ran mach5 tiers 1-6
>
> src/hotspot/share/prims/jvmtiEnvBase.cpp line 867:
> 
>> 865:   // This call collects the strong and weak groups
>> 866:   JavaThread* THREAD = current_thread;
>> 867:   JvmtiJavaUpcallMark jjum(current_thread);
> 
> Add comment like you did above for the JvmtiEnv::InterruptThread case.

Okay, thanks! Added now.

> src/hotspot/share/prims/jvmtiExport.cpp line 1419:
> 
>> 1417:     // ClassPrepare events are important for JDWP agent but not expected during such upcalls.
>> 1418:     // Catch if this invariant is not broken.
>> 1419:     assert(!thread->is_in_java_upcall(), "unexpected ClassPrepare event during JVMTI upcall");
> 
> I think we should do this for ClassLoad also.

Okay, thanks! Updated now.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/24539#discussion_r2036227345
PR Review Comment: https://git.openjdk.org/jdk/pull/24539#discussion_r2036227287