RFR: JDK-8292769: [JVMCI] OutOfMemoryError thrown when attaching the libgraal isolate causes HotSpot to crash.

[Tom Rodriguez]

On Tue, 23 Aug 2022 12:03:02 GMT, Tomáš Zezula <duke at openjdk.org> wrote:

> Fixes a virtual machine crash when the libgraal isolate does not have enough heap memory available and its memory is exhausted. The [crash](https://bugs.openjdk.org/secure/attachment/100520/hs_err_pid679001.log) is caused by an unhandled `OutOfMemoryError` thrown in the isolate's `AttachCurrentThread`.
> The isolate's `AttachCurrentThread` is [fixed](https://github.com/oracle/graal/commit/62ede96fd3ddc9e5ddb4804ed369f61cfa0c0a21#diff-9eedff7eb9fceec5f37c48cf089ff8e24103279e1384a2c0fa1fc9446b5dc438)  to report an `OutOfMemoryError` using a `JNI_ENOMEM` result. This pull request aborts a compilation when `AttachCurrentThread` fails with `JNI_ENOMEM`.

src/hotspot/share/jvmci/jvmciEnv.cpp line 239:

> 237: JVMCIEnv::JVMCIEnv(JavaThread* thread, JVMCICompileState* compile_state, const char* file, int line):
> 238:     _throw_to_caller(false), _file(file), _line(line), _attach_threw_OOME(false), _compile_state(compile_state) {
> 239:   init_env_mode_runtime(thread, NULL, false);

why is it only non-fatal for this single path?  Some comments explaining the reason for it being fatal in some paths and not in others would help.

src/hotspot/share/jvmci/jvmciEnv.cpp line 241:

> 239:   init_env_mode_runtime(thread, NULL, false);
> 240:   if (_attach_threw_OOME) {
> 241:     compile_state->set_failure(true, "Out of memory");

Could this be more explicit like `JVMCI compiler attach returned out of memory`?  cc @dougxc

-------------

PR: https://git.openjdk.org/jdk/pull/9978