RFR: 8335553: [Graal] Compiler thread calls into jdk.internal.vm.VMSupport.decodeAndThrowThrowable and crashes in OOM situation

[Doug Simon]

On Mon, 8 Jul 2024 19:01:05 GMT, Doug Simon <dnsimon at openjdk.org> wrote:

> This PR addresses intermittent failures in jtreg GC stress tests. The failures occur under these conditions:
> 1. Using a libgraal build with assertions enabled as the top tier JIT compiler. Such a libgraal build will cause a VM exit if an assertion or GraalError occurs in a compiler thread (as this catches more errors in testing).
> 2. A libgraal compiler thread makes a call into the VM (via `CompilerToVM`) to a routine that performs a HotSpot heap allocation that fails.
> 3. The resulting OOME is wrapped in a GraalError, causing the VM to exit as described in 1.
> 
> An OOME thrown in these specific conditions should not exit the VM as it not related to an OOME in the app or test. Instead, the failure should be treated as a bailout and the libgraal compiler should continue.
> 
> To accomplish this, libgraal needs to be able to distinguish a GraalError caused by an OOME. This PR modifies the exception translation code to make this possible.

src/hotspot/share/utilities/exceptions.cpp line 114:

> 112: #endif // ASSERT
> 113: 
> 114:   if (h_exception.is_null() && !thread->can_call_java()) {

There is no reason to replace an existing exception object with a dummy exception object in the case where the current thread cannot call into Java. Since the exception object already exists, no Java call is necessary.

This change is necessary to allow the libgraal exception translation mechanism to know that an OOME is being translated.

src/hotspot/share/utilities/exceptions.cpp line 208:

> 206:                               Handle h_loader, Handle h_protection_domain) {
> 207:   // Check for special boot-strapping/compiler-thread handling
> 208:   if (special_exception(thread, file, line, h_cause)) return;

This fixes a long standing bug where `special_exception` is being queried with the *cause* of the exception being thrown instead of the *name* of the exception being thrown.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/20083#discussion_r1669153819
PR Review Comment: https://git.openjdk.org/jdk/pull/20083#discussion_r1669148553