RFR: 8341273: JVMTI is not properly hiding some continuation related methods [v3]

[Serguei Spitsyn]

On Tue, 15 Oct 2024 21:59:41 GMT, Serguei Spitsyn <sspitsyn at openjdk.org> wrote:

>> src/hotspot/share/prims/jvmtiEnvBase.cpp line 692:
>> 
>>> 690:   if (jt->is_in_VTMS_transition()) {
>>> 691:     jvf = check_and_skip_hidden_frames(jt->is_in_VTMS_transition(), jvf);
>>> 692:   } else if (is_virtual) { // filter out pure continuations
>> 
>> Not sure I follow the logic here.
>> As far as I understand yield/yield need to be filtered out only for unmounted virtual threads. But `is_virtual` here means we have mounted VT?
>> Looks like almost all callers call this method only if `jt->is_in_JTMS_transilition()` is true
>> Exception is a call from `get_vthread_jvf()` when vthread is mounted.
>
> This seems to be a good catch. The call to `skip_top_jvmti_annotated_frames()` should not be needed.
> It does not harm either, so I've added it for safety.
> Will remove it and rerun mach5 tiers to make sure there is nothing unexpected.

Please, hold on. My last comment was not fully incorrect.
Not only `yield()/yield0()` need to be filtered out for virtual threads.
There are more cases for a `JavaThread` which is not in a VTMS transition executing a virtual thread. All the calls below represents inconsistency and must be hidden:

 - `notifyJvmtiStart()`: This function is called in a VTMS MOUNT transition which it ends. So its return happens with the thread identity `VIRTUAL`.
 - `notifyJvmtiEnd()`: This function is called with the thread identity `VIRTUAL`. It is starting a VTMS UNMOUNT transition, so its return happens in a VTMS transition.
 - `notifyJvmtiMount(false)`:  This call happens in a VTMS transition. It finishes the VTMS transition and its return happen with the thread identity 'VIRTUAL'.
 - `notifyJvmtiUnmount(true)`:  This call happens with the thread identity 'VIRTUAL'. It starts a VTMS UNMOUNT transition, so its return happens in a VTMS transition.

I've just discovered that we have two more cases to hide frames for carrier threads that are NOT in a VTMS transitions:
- `notifyJvmtiMount(true)`: This call happens with the thread identity `CARRIER`. It starts a VTMS MOUNT transition, so its return happens in a VTMS transition.
 - `notifyJvmtiUnmount(false)`:  This call happens in a VTMS UNMOUNT transition. It finishes transition, so its return happens with the thread identity `CARRIER`.

An update is needed for these two cases.
All the cases above can be potentially observed at a safepoint/handshake in a non-transitioned state.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/21397#discussion_r1803874831