RFR: 8369515: Deadlock between JVMTI and JNI ReleasePrimitiveArrayCritical [v2]

[Patricio Chilano Mateo]

On Thu, 18 Dec 2025 01:21:17 GMT, David Holmes <dholmes at openjdk.org> wrote:

>> As discussed in JBS the deadlock occurs when the call to `ReleasePrimitiveArrayCritical` performs the transition from native to VM, and in the process checks for special runtime exit conditions - which includes the `obj_deopt_suspend` request. The simple solution is to define a custom JNI ENTRY with custom `ThreadInVMfromNative` that elides the exit check.
>> 
>> The change is limited to `ReleasePrimitiveArrayCritical` and `ReleaseStringCritical`.
>> 
>> UPDATE: we are now employing a much simpler solution.
>> 
>> There is no regression test as this has only been seen in long running stress tests.
>> 
>> Testing:
>>    -tiers 1-6
>
> David Holmes has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since the last revision:
> 
>  - Greatly simplifed fix to just defer object_deopt whilst in JNI critical region
>  - Merge branch 'master' into 8369515-jni-critical
>  - Revert "8369515"
>    
>    This reverts commit 3beb23ccbf5adb98d8c6ad404d40c603bbf499dc.
>  - 8369515

Looks good to me.

FTR we are not processing this special exit condition when transitioning from `_thread_blocked` to `_thread_in_vm` already, so I would assume it should be also okay for any native->vm transition.

src/hotspot/share/runtime/javaThread.cpp line 1057:

> 1055:   // We mustn't block for object deopt if the thread is
> 1056:   // currently executing in a JNI critical region, as that
> 1057:   // can cause deadlock because the GCLocker is held.

I think the last part of the comment might be confusing because it's the deopt suspender that helds the lock waiting for this thread to exit the critical region [[1]](https://github.com/openjdk/jdk/blob/0b2712400b55d4a512db225d090c2f06f01f7f1f/src/hotspot/share/gc/shared/gcLocker.cpp#L108). Maybe "... cause deadlock because the suspender may allocate and block waiting for this thread to exit the critical region"?

-------------

Marked as reviewed by pchilanomate (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/28779#pullrequestreview-3594584056
PR Comment: https://git.openjdk.org/jdk/pull/28779#issuecomment-3671778255
PR Review Comment: https://git.openjdk.org/jdk/pull/28779#discussion_r2632288550