RFR: 8319244: implement JVMTI handshakes support for virtual threads [v4]

[Serguei Spitsyn]

On Thu, 16 Nov 2023 16:27:23 GMT, Serguei Spitsyn <sspitsyn at openjdk.org> wrote:

>> So the problematic case I'm thinking is when the JvmtiVTMSTransitionDisabler starts after the vthread executed notifyJvmtiUnmount(), i.e the vthread is already outside the transition, but before changing the state to TERMINATED. JvmtiVTMSTransitionDisabler will proceed, and since the carrierThread field has already been cleared we will treat it as an unmounted vthread. Then we can see first that is alive in JvmtiHandshake::execute() but then we could hit the assert that is already TERMINATED in JvmtiEnvBase::get_vthread_jvf().
>
> Thanks! This is a valid concern. Will try to fix this.

I'm suggesting to fix it this way for the unmounted case only:

@@ -1976,6 +1976,13 @@ JvmtiHandshake::execute(JvmtiUnitedHandshakeClosure* hs_cl, ThreadsListHandle* t
       return;
     }
     if (target_jt == nullptr) {    // unmounted virtual thread
+      // JvmtiVTMSTransitionDisabler can start after the vthread executed notifyJvmtiUnmount(), i.e.
+      // the vthread is already outside the transition, but before changing the state to TERMINATED.
+      // Changing the state to TERMINATED is racy, so we check if the continuation is done in advance.
+      oop cont = java_lang_VirtualThread::continuation(target_h());
+      if (jdk_internal_vm_Continuation::done(cont)) {
+        return;
+      }
       hs_cl->do_vthread(target_h); // execute handshake closure callback on current thread directly
     }
   }

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/16460#discussion_r1396084391