RFR: 8341273: JVMTI is not properly hiding some continuation related methods [v3]

[Alex Menkov]

On Wed, 9 Oct 2024 22:58:33 GMT, Serguei Spitsyn <sspitsyn at openjdk.org> wrote:

>> This fixes a problem in the VTMS (Virtual Thread Mount State) transition frames hiding mechanism.
>> Please, see a fix description in the first comment.
>> 
>> Testing:
>>  - Verified with new test `vthread/CheckHiddenFrames`
>>  - Mach5 tiers 1-6 are passed
>
> Serguei Spitsyn has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Disallow NotifyFramePop for enter/enter0/VirtualThread.run/VThreadContinuation.run

src/hotspot/share/prims/jvmtiEnvBase.cpp line 588:

> 586:   // There should not be any VTMS transition here. This is for safety.
> 587:   if (java_thread->is_in_VTMS_transition()) {
> 588:     jvf = JvmtiEnvBase::check_and_skip_hidden_frames(java_thread, jvf);

The code now checks `java_thread->is_in_VTMS_transition()`, so it may be simplified to
Suggestion:

    jvf = JvmtiEnvBase::check_and_skip_hidden_frames(true, jvf);

src/hotspot/share/prims/jvmtiEnvBase.cpp line 691:

> 689: 
> 690:   if (jt->is_in_VTMS_transition()) {
> 691:     jvf = check_and_skip_hidden_frames(jt->is_in_VTMS_transition(), jvf);

Suggestion:

    jvf = check_and_skip_hidden_frames(true, jvf);

src/hotspot/share/prims/jvmtiEnvBase.cpp line 753:

> 751:     // Skip hidden frames only for carrier threads
> 752:     // which are in non-temporary VTMS transition.
> 753:     jvf = check_and_skip_hidden_frames(jt, jvf);

Can be
Suggestion:

    jvf = check_and_skip_hidden_frames(true, jvf);

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/21397#discussion_r1802038615
PR Review Comment: https://git.openjdk.org/jdk/pull/21397#discussion_r1802039013
PR Review Comment: https://git.openjdk.org/jdk/pull/21397#discussion_r1802039786