RFR: 8335553: [Graal] Compiler thread calls into jdk.internal.vm.VMSupport.decodeAndThrowThrowable and crashes in OOM situation [v2]

[Yudi Zheng]

On Tue, 9 Jul 2024 13:46:46 GMT, Doug Simon <dnsimon at openjdk.org> wrote:

>> This PR addresses intermittent failures in jtreg GC stress tests. The failures occur under these conditions:
>> 1. Using a libgraal build with assertions enabled as the top tier JIT compiler. Such a libgraal build will cause a VM exit if an assertion or GraalError occurs in a compiler thread (as this catches more errors in testing).
>> 2. A libgraal compiler thread makes a call into the VM (via `CompilerToVM`) to a routine that performs a HotSpot heap allocation that fails.
>> 3. The resulting OOME is wrapped in a GraalError, causing the VM to exit as described in 1.
>> 
>> An OOME thrown in these specific conditions should not exit the VM as it not related to an OOME in the app or test. Instead, the failure should be treated as a bailout and the libgraal compiler should continue.
>> 
>> To accomplish this, libgraal needs to be able to distinguish a GraalError caused by an OOME. This PR modifies the exception translation code to make this possible.
>
> Doug Simon has updated the pull request incrementally with one additional commit since the last revision:
> 
>   fixed TestTranslatedException

src/hotspot/share/jvmci/jvmciCompilerToVM.cpp line 782:

> 780:       while (true) {
> 781:         // Trigger an OutOfMemoryError
> 782:         objArrayOop next = oopFactory::new_objectArray(0x7FFFFFFF, CHECK_NULL);

Shall we check for pending exception and break here?

test/jdk/jdk/internal/vm/TestTranslatedException.java line 167:

> 165:     private static void assertThrowableEquals(Throwable originalIn, Throwable decodedIn) {
> 166:         Throwable original = originalIn;
> 167:         Throwable decoded = decodedIn;

What is the purpose of this renaming?

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/20083#discussion_r1670646934
PR Review Comment: https://git.openjdk.org/jdk/pull/20083#discussion_r1670607742