[jdk23u] RFR: 8337753: Target class of upcall stub may be unloaded

[pabloko]

On Mon, 14 Oct 2024 12:34:18 GMT, Jorn Vernee <jvernee at openjdk.org> wrote:

> Backport of: https://github.com/openjdk/jdk/pull/20479
> 
> There were some trivial conflicts in `stubGenerator_aarch64.cpp` and `stubGenerator_riscv.cpp`. Some code still there in jdk23, already removed in later versions.

Hello, we're experiencing frequent crashes on JDK (windows+zgc) wich this pull seems to fix. Our issue does not seem related to class unloading, but gc related, sadly we could not reproduce the issue in a isolated tester since the logic is deep behind propietary framework that we cannot disclose, but [here's one](https://gist.github.com/pabloko/3464d93f7731bc85a6964ccf4d8c3fdc) of the hundreds of similar crash dumps we have collected. In this case the crash is triggered by calling `System.gc()` a few times.

#  Internal Error (sharedRuntime.cpp:1548), pid=860, tid=11468
#  guarantee(callee != nullptr && callee->is_method()) failed: bad handshake

We initially observed this happening from seconds to hours or even days of running, but definitely is more frequent when an JVMTI agent (debugger) is attached. 

We been digging into this issue for some time and came to the conclusion that the `vmtarget` method of the `LambdaForm` cached inside the upcall stub isn't reachable anymore, while the MethodHandle and associated classes are still live. I've tracked this PR originally on the jdk24 mainline, and observed the new path wich caches the MethodHandle instead, and then resolve the target on each call using a leaf method, that seems to be more in line with the [design document](https://wiki.openjdk.org/display/HotSpot/Method+handles+and+invokedynamic) of Method Handles (Lambda form optimization).

I've compiled both JDK24 mainline and JDK23u with this pull applied and i can confirm that now jvm works as expected with no noticeable performance regression, the indirection of fetching the compiled code is marginal compared with other parts of the `on_entry` upcall stub like java thread stack frame setup or backup/restore every register in the cpu.

Given the critical nature of this issue, we strongly advise reconsidering the inclusion of this fix in JDK23u. This problem has become a significant reliability concern for the us, rendering FFM upcalls untrustworthy and constitute a silent killer of the JVM.

-------------

PR Comment: https://git.openjdk.org/jdk23u/pull/163#issuecomment-2438415048