RFR: 8319244: implement JVMTI handshakes support for virtual threads [v4]

[Patricio Chilano Mateo]

On Thu, 16 Nov 2023 07:11:03 GMT, Serguei Spitsyn <sspitsyn at openjdk.org> wrote:

>> Thank you for the suggestion. Will check it.
>
> I've added the check for `!jdk_internal_vm_Continuation::done(cont)` into ` JvmtiEnvBase::is_vthread_alive(oop vt)` but then decided to remove it again. This is racy for `JvmtiHandshake` execution. As you correctly stated, the change of state to `TERMINATED` happens after `notifyJvmtiUnmount()`. The target virtual thread will be blocked in the `notifyJvmtiUnmount()` because the `JvmtiVTMSTransitionDisabler` is set. This gives us a guaranty that the target virtual thread won't change its state to `TERMINATED` while a handshake is executed. But it becomes not true if we add the `!jdk_internal_vm_Continuation::done(cont)` check.
> Form the other hand, absence of this check allows for target virtual thread stack to become empty (with no frames). This is a known problem but I'd prefer to attack it separately.

So the problematic case I'm thinking is when the JvmtiVTMSTransitionDisabler starts after the vthread executed notifyJvmtiUnmount(), i.e the vthread is already outside the transition, but before changing the state to TERMINATED. JvmtiVTMSTransitionDisabler will proceed, and since the carrierThread field has already been cleared we will treat it as an unmounted vthread. Then we can see first that is alive in JvmtiHandshake::execute() but then we could hit the assert that is already TERMINATED in JvmtiEnvBase::get_vthread_jvf().

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/16460#discussion_r1395892738