RFR: 8341273: JVMTI is not properly hiding some continuation related methods

[Serguei Spitsyn]

On Mon, 7 Oct 2024 22:03:36 GMT, Serguei Spitsyn <sspitsyn at openjdk.org> wrote:

> This fixes a problem in the VTMS (Virtual Thread Mount State) transition frames hiding mechanism.
> Please, see a fix description in the first comment.
> 
> Testing:
>  - Verified with new test `vthread/CheckHiddenFrames`
>  - Mach5 tiers 1-6 are passed

The frames which are in VTMS transition should not be visible to the JVMTI agents including debug agent because the thread identity can be incorrect. The JVMTI events are not posted when `java_thread->is_in_VTMS_transition() == true`.
All the JVMTI functions returning stack related info do skip frames that are in transition. The hiding mechanism is using the annotation `@JvmtiMountTransition` to mark the `notifyJvmti*` methods and the bit `java_thread->is_in_VTMS_transition()`.
It occurred that the methods `yield()` and `yield0()` can be present in stack trace of an unmounted virtual threads.
The bit `java_thread->is_in_VTMS_transition() ` is not set in such a case.
The fix is to add the annotation `@JvmtiMountTransition` to the methods `yield()` and `yield0()` and correct the frames skipping mechanism to account for such frames as well.
The update also includes:
 - fix in one of the `JvmtiHandshake::execute()` functions which is needed for better stability and safety
 - tweak in the test which expects frames with `yield()` and `yield0()` methods to be present at the top

-------------

PR Comment: https://git.openjdk.org/jdk/pull/21397#issuecomment-2398025342