RFR: 8319244: implement JVMTI handshakes support for virtual threads [v4]

[Serguei Spitsyn]

On Thu, 16 Nov 2023 15:20:31 GMT, Patricio Chilano Mateo <pchilanomate at openjdk.org> wrote:

>> I've added the check for `!jdk_internal_vm_Continuation::done(cont)` into ` JvmtiEnvBase::is_vthread_alive(oop vt)` but then decided to remove it again. This is racy for `JvmtiHandshake` execution. As you correctly stated, the change of state to `TERMINATED` happens after `notifyJvmtiUnmount()`. The target virtual thread will be blocked in the `notifyJvmtiUnmount()` because the `JvmtiVTMSTransitionDisabler` is set. This gives us a guaranty that the target virtual thread won't change its state to `TERMINATED` while a handshake is executed. But it becomes not true if we add the `!jdk_internal_vm_Continuation::done(cont)` check.
>> Form the other hand, absence of this check allows for target virtual thread stack to become empty (with no frames). This is a known problem but I'd prefer to attack it separately.
>
> So the problematic case I'm thinking is when the JvmtiVTMSTransitionDisabler starts after the vthread executed notifyJvmtiUnmount(), i.e the vthread is already outside the transition, but before changing the state to TERMINATED. JvmtiVTMSTransitionDisabler will proceed, and since the carrierThread field has already been cleared we will treat it as an unmounted vthread. Then we can see first that is alive in JvmtiHandshake::execute() but then we could hit the assert that is already TERMINATED in JvmtiEnvBase::get_vthread_jvf().

Thanks! This is a valid concern. Will try to fix this.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/16460#discussion_r1396000487