RFR: 8369515: Deadlock between JVMTI and JNI ReleasePrimitiveArrayCritical

[Richard Reingruber]

On Fri, 12 Dec 2025 04:20:11 GMT, David Holmes <dholmes at openjdk.org> wrote:

> As discussed in JBS the deadlock occurs when the call to `ReleasePrimitiveArrayCritical` performs the transition from native to VM, and in the process checks for special runtime exit conditions - which includes the `obj_deopt_suspend` request. The simple solution is to define a custom JNI ENTRY with custom `ThreadInVMfromNative` that elides the exit check.
> 
> The change is limited to `ReleasePrimitiveArrayCritical` and `ReleaseStringCritical`.
> 
> There is no regression test as this has only been seen in long running stress tests.
> 
> Testing:
>    -tiers 1-6

@dholmes-ora I do think that also _entering_ a critical region is problematic if it is nested.
I'm currently testing with https://github.com/reinrich/jdk/commit/d7ce2ccb2150c92929b8d9140b4709833d188474 where a thread doesn't suspend for EscapeBarriers while in a critical region.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/28779#issuecomment-3656372702