RFR: 8341273: JVMTI is not properly hiding some continuation related methods [v11]

[Serguei Spitsyn]

On Thu, 24 Oct 2024 22:58:01 GMT, Alex Menkov <amenkov at openjdk.org> wrote:

>> Serguei Spitsyn has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   review: removed asserts from continuationFreezeThaw.cpp again
>
> src/hotspot/share/prims/jvmtiEnvBase.cpp line 667:
> 
>> 665: 
>> 666: javaVFrame*
>> 667: JvmtiEnvBase::check_and_skip_hidden_frames(bool is_in_VTMS_transition, javaVFrame* jvf) {
> 
> reworked function looks much better! Now it's clear what the function does and I have a question what it should do.
> The function checks `@JvmtiMountTransition` annotation first even if the thread is in transition.
> If `@JvmtiMountTransition` is present, the code doesn't care about `@ChangesCurrentThread` (and doesn't case about `@JvmtiMountTransition` if in_transition + `@ChangesCurrentThread`).
> But we have 2 methods in VirtualThread class (`switchToCarrierThread()` and `switchToVirtualThread()`) with both annotations.
> So if the function is called when `switchToCarrierThread` is on top, the function skips this frame, but if the thread calls some other function without `@JvmtiMountTransition` annotation from `switchToCarrierThread`, the function returns `switchToCarrierThread`.

Good catch, thanks.
These two functions are impacted by temporary VTMS transitions. It seems we fail to hide frames in these transitions while we skip the JVMTI events in their context. I'll try to fix this issue.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/21397#discussion_r1817699939