RFR: 8375209: Strengthen Xcheck:jni for JNI critical regions [v3]

[Jorn Vernee]

On Wed, 14 Jan 2026 10:22:35 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

>> [JDK-8375188](https://bugs.openjdk.org/browse/JDK-8375188) shows that misuse for JNI critical is able to deadlock the GC. Therefore, I think we really want to have `-Xcheck:jni` to preempt these cases and allow us to diagnose the issues in the field.
>> 
>> G1 and Shenandoah are not exactly affected by this deadlock. ZGC never had this check, AFAICS, and is affected by the deadlock. We used to have some capability in Serial/Parallel prior to [JDK-8192647](https://bugs.openjdk.org/browse/JDK-8192647), AFAICS: https://github.com/openjdk/jdk/commit/a9c9f7f0cbb2f2395fef08348bf867ffa8875d73#diff-d27fc793db1bf9314b322d494cd1c3269629fe27a605b4441de08d543d020fc3L341-L344
>> 
>> Additional testing:
>>  - [x] New test, 100x repetitions
>>  - [ ] Linux x86_64 server fastdebug, `all` with `-XX:+UseParallelGC -Xcheck:jni`
>
> Aleksey Shipilev has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Capture bad state at the end of JNI transition, check criticality on JNI function enters
>   Move the check to native->Java transition

Thanks for taking a stab at this. Yes, unfortunately not everything uses the nice helper functions to do thread state transitions. Particularly, Java -> native calls do this in platform specific assembly code.

One place you've missed is in `TemplateInterpreterGenerator::generate_native_entry`. The code is similar to the compiled native wrapper, with an `if (CheckJNICalls) { ... }`, so I think you can just add the same change there. There are also Java -> native -> Java transitions in the downcall stub we generate for FFM calls, but we don't expect native code at the other end of a downcall to interact with JNI, so it's not really important to check there (we also don't check for pending exceptions).

AFAIK we need to do a safepoint poll any time we go from a safe state (native or blocked) to an unsafe sate (java or vm), since a safepoint might be in progress during the transition. The fact that `Get*Critical` and `Release*Critical` transition to the vm state seems like an issue to me, since that transition also includes a safepoint poll.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/29206#issuecomment-3749714795