RFR: 8313899: JVMCI exception Translation can fail in TranslatedException.<clinit>

[Doug Simon]

On Tue, 8 Aug 2023 20:52:29 GMT, Doug Simon <dnsimon at openjdk.org> wrote:

> In a test that stresses metaspace (such as `vmTestbase/vm/mlvm/hiddenloader/stress/oome/metaspace/Test.java`) that also uses `-Xcomp -XX:-TieredCompilation`, we've seen a failure in `TranslatedException.<clinit>` due to exhausted metaspace:
> 
> java.lang.OutOfMemoryError: Metaspace
>         at jdk.internal.vm.TranslatedException.encodeThrowable(java.base at 21/TranslatedException.java:176)
>         at jdk.internal.vm.TranslatedException.<clinit>(java.base at 21/TranslatedException.java:61)
>         at jdk.internal.vm.VMSupport.encodeThrowable(java.base at 21/VMSupport.java:171)
> 
> This PR pushes a fix such that this exception is properly handled in the VM (i.e. causing a compilation bailout) instead of leading to a VM crash.
> 
> The PR includes 2 bits of debug code guarded by system properties that enable the handling to be tested in libgraal. The test itself is not included as libgraal is not part of OpenJDK.

src/hotspot/share/jvmci/jvmciEnv.cpp line 472:

> 470:                             vmSymbols::encodeThrowable_name(),
> 471:                             vmSymbols::encodeThrowable_signature(), &jargs, THREAD);
> 472:     if (handle_pending_exception(THREAD, buffer, buffer_size)) {

This is the actual bug fix: handle any exception occurring in the Java upcall.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/15198#discussion_r1290014253