RFR: 8335553: [Graal] Compiler thread calls into jdk.internal.vm.VMSupport.decodeAndThrowThrowable and crashes in OOM situation [v2]

[Tom Rodriguez]

On Tue, 9 Jul 2024 13:46:46 GMT, Doug Simon <dnsimon at openjdk.org> wrote:

>> This PR addresses intermittent failures in jtreg GC stress tests. The failures occur under these conditions:
>> 1. Using a libgraal build with assertions enabled as the top tier JIT compiler. Such a libgraal build will cause a VM exit if an assertion or GraalError occurs in a compiler thread (as this catches more errors in testing).
>> 2. A libgraal compiler thread makes a call into the VM (via `CompilerToVM`) to a routine that performs a HotSpot heap allocation that fails.
>> 3. The resulting OOME is wrapped in a GraalError, causing the VM to exit as described in 1.
>> 
>> An OOME thrown in these specific conditions should not exit the VM as it not related to an OOME in the app or test. Instead, the failure should be treated as a bailout and the libgraal compiler should continue.
>> 
>> To accomplish this, libgraal needs to be able to distinguish a GraalError caused by an OOME. This PR modifies the exception translation code to make this possible.
>
> Doug Simon has updated the pull request incrementally with one additional commit since the last revision:
> 
>   fixed TestTranslatedException

looks good.

-------------

Marked as reviewed by never (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/20083#pullrequestreview-2169495478