RFR: 8341273: JVMTI is not properly hiding some continuation related methods [v3]

[Serguei Spitsyn]

On Wed, 9 Oct 2024 22:58:33 GMT, Serguei Spitsyn <sspitsyn at openjdk.org> wrote:

>> This fixes a problem in the VTMS (Virtual Thread Mount State) transition frames hiding mechanism.
>> Please, see a fix description in the first comment.
>> 
>> Testing:
>>  - Verified with new test `vthread/CheckHiddenFrames`
>>  - Mach5 tiers 1-6 are passed
>
> Serguei Spitsyn has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Disallow NotifyFramePop for enter/enter0/VirtualThread.run/VThreadContinuation.run

I've pushed two updates.
The first one addresses the suggestions from Leonid:
 - minor tweaks in new test
 - moved skipping hidden methods `yield()` and `yield0()` from `check_and_skip_hidden_frames()` to a separate function
 - explained in a comment of `JvmtiHandshake::execute()` why all VTMS transitions are disabled for platform thread
 - added extra safety/stability checks and a couple of asserts around calls to `check_and_skip_hidden_frames()`
 
 Second update fixes one more `NotifyFramePop` issue with the frames at the virtual thread start:
  - `enter()`, `enter0`, `VThreadContinuation$1.run()` and `VirtualThread.run()`
  
 This the bottom of stack trace from debugger:

  0: java/lang/VirtualThread: run(Ljava/lang/Runnable;)V
  1: java/lang/VirtualThread$VThreadContinuation$1: run()V
  2: jdk/internal/vm/Continuation: enter0()V
  3: jdk/internal/vm/Continuation: enter(Ljdk/internal/vm/Continuation;Z)V

The fix includes three parts:
- the `notifyJvmtiStart()`/`notifyJvmtiEnd()` notification calls are moved from `VirtualThread.run()` to the `VThreadContinuation$1.run()`
- the annotation `@JvmtiMountTransition` has been added to the `VThreadContinuation$1.run()`
- the `NotifyFramePop` is changed to return `JVMTI_ERRO_OPAQUE_FRAME` for frames with `enter()` and methods annotated with `@JvmtiMountTransition`

-------------

PR Comment: https://git.openjdk.org/jdk/pull/21397#issuecomment-2404706773