RFR: 8318737: Fallback linker passes bad JNI handle

[Aleksey Shipilev]

On Tue, 24 Oct 2023 17:01:35 GMT, Jorn Vernee <jvernee at openjdk.org> wrote:

> The result of `FindClass` is a local JNI handle (in `find_class_from_class_loader`, called from `jni_FindClass` [1]). As such, we need to wrap the return value of `FindClass` in a global reference when storing it inside fallbackLinker.c.
> 
> While investigating this, I also noticed an existing bug in `JNIHandles::handle_type`. This method is used from the implementation of `GetObjectRefType` ([2]), and from the implementation of `-Xcheck:jni` code. The former specifies that `JNIInvalidRefType` is a valid return value, and the latter compares the result against `JNIInvalidRefType`. However, if the handle is not any valid type, the implementation bottoms out in a `ShouldNotReachHere()`, meaning `JNIHandles::handle_type` can never return `JNIInvalidRefType`. I've fixed this by letting the enclosing if/else chain fall through to just returning the default result, which is `JNIInvalidRefType`. In that case, I observe the expected stack trace when running with `-Xcheck:jni`. For example:
> 
> 
> FATAL ERROR in native method: Bad global or local ref passed to JNI
>         at jdk.internal.foreign.abi.fallback.LibFallback.doDowncall(java.base at 22-internal/Native Method)
>         at jdk.internal.foreign.abi.fallback.LibFallback.doDowncall(java.base at 22-internal/LibFallback.java:94)
>         at jdk.internal.foreign.abi.fallback.FallbackLinker.doDowncall(java.base at 22-internal/FallbackLinker.java:197)
>         at java.lang.invoke.LambdaForm$DMH/0x000001b585008000.invokeStaticInit(java.base at 22-internal/LambdaForm$DMH)
>         at java.lang.invoke.LambdaForm$MH/0x000001b585029400.invoke(java.base at 22-internal/LambdaForm$MH)
>         at java.lang.invoke.LambdaForm$MH/0x000001b58502d000.invokeExact_MT(java.base at 22-internal/LambdaForm$MH)
>         at TestUpcallDeopt.payload(TestUpcallDeopt.java:93)
>         at TestUpcallDeopt.main(TestUpcallDeopt.java:84)
>         at java.lang.invoke.LambdaForm$DMH/0x000001b585006800.invokeStatic(java.base at 22-internal/LambdaForm$DMH)
>         at java.lang.invoke.LambdaForm$MH/0x000001b58502a800.invoke(java.base at 22-internal/LambdaForm$MH)
>         at java.lang.invoke.Invokers$Holder.invokeExact_MT(java.base at 22-internal/Invokers$Holder)
>         at jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(java.base at 22-internal/DirectMethodHandleAccessor.java:154)
>         at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(java.base at 22-internal/DirectMethodHandleAccessor.java:103)
>         at java.lang.reflect.Method.invoke(java.base at 22-internal/Method.java:580...

A few comments:

src/hotspot/share/runtime/jniHandles.cpp line 202:

> 200:       ShouldNotReachHere();
> 201:     }
> 202:   } else if (is_local_handle(thread, handle) || is_frame_handle(thread, handle)) {

Should we still add `ShouldNotReachHere()` at global `else` branch? This would make the if-else chain exhaustive with the early warning if some handle type is not used. The prior code did this already.

src/java.base/share/native/libfallbackLinker/fallbackLinker.c line 52:

> 50:     return; // let caller deal with exception
> 51:   }
> 52:   LibFallback_class = (*env)->NewGlobalRef(env, LibFallback_class_local);

I think `NewGlobalRef` can return `nullptr` on OOME: https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/functions.html#NewGlobalRef

We should check for this too?

-------------

PR Review: https://git.openjdk.org/jdk/pull/16349#pullrequestreview-1696818430
PR Review Comment: https://git.openjdk.org/jdk/pull/16349#discussion_r1371441723
PR Review Comment: https://git.openjdk.org/jdk/pull/16349#discussion_r1371445731