RFR: 8337331: crash: pinned virtual thread will lead to jvm crash when running with the javaagent option [v2]

[Jiawei Tang]

On Mon, 29 Jul 2024 09:53:40 GMT, Alan Bateman <alanb at openjdk.org> wrote:

>> Jiawei Tang has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision:
>> 
>>   8337331: crash: pinned virtual thread will lead to jvm crash when running with the javaagent option
>
> test/hotspot/jtreg/serviceability/jvmti/vthread/VThreadTraceWithAgent/TestPinCaseWithTrace.java line 2:
> 
>> 1: /*
>> 2:  * Copyright (c) 2024, 2024, Oracle and/or its affiliates. All rights reserved.
> 
> I assume you didn't mean to include a date range on a new test.

Change it.

> test/hotspot/jtreg/serviceability/jvmti/vthread/VThreadTraceWithAgent/TestPinCaseWithTrace.java line 66:
> 
>> 64:             System.out.println("call native: " + nativeFuncPin(1));
>> 65:         });
>> 66:     }
> 
> Does this really need to use a custom scheduler? If not,  running the test with -Djdk.virtualThreadScheduler.maxPoolSize=1 would be simpler. If you really need a custom scheduler, the test can use jdk.test.lib.thread.VThreadScheduler.  Also to create a pinning scenario it can use jdk.test.lib.thread.VThreadPinner to avoid needing to add JNI code. You'll see examples of both in other tests.

Thanks, now I use `-Djdk.virtualThreadScheduler.maxPoolSize=1` instead.

> test/hotspot/jtreg/serviceability/jvmti/vthread/VThreadTraceWithAgent/TestPinCaseWithTrace.java line 70:
> 
>> 68:     static int native2Java(int b) {
>> 69:         try {
>> 70:             Thread.sleep(500); // try yield, will pin, javaagent+tracePinnedThreads will crash here (because of the class `PinnedThreadPrinter`)
> 
> As noted in the JBS issue, -Djdk.tracePinnedThreads has been very problematic and has been removed in the loom repo as part of the object monitor changes.

I have read the code in loom and this issue can be resolved by using JFR event instead. But I hope this could be fixed since using javaagent is very common in java application. The root cause is that no new class should be use after the vthread is pinned, since a agent can change the class bytecode and need to use `JvmtiVTMSTransitionDisabler` when transforming class. However, this vthread is in VTMS, it cannot jump out the loop.

Using `-Djdk.tracePinnedThreads=full` will use the class `PinnedThreadPrinter` so we end in a deadlock.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/20373#discussion_r1695049376
PR Review Comment: https://git.openjdk.org/jdk/pull/20373#discussion_r1695050299
PR Review Comment: https://git.openjdk.org/jdk/pull/20373#discussion_r1695050607