RFR: JDK-8292769: [JVMCI] OutOfMemoryError thrown when attaching the libgraal isolate causes HotSpot to crash.

[Doug Simon]

On Thu, 25 Aug 2022 18:51:30 GMT, Tom Rodriguez <never at openjdk.org> wrote:

>> Fixes a virtual machine crash when the libgraal isolate does not have enough heap memory available and its memory is exhausted. The [crash](https://bugs.openjdk.org/secure/attachment/100520/hs_err_pid679001.log) is caused by an unhandled `OutOfMemoryError` thrown in the isolate's `AttachCurrentThread`.
>> The isolate's `AttachCurrentThread` is [fixed](https://github.com/oracle/graal/commit/62ede96fd3ddc9e5ddb4804ed369f61cfa0c0a21#diff-9eedff7eb9fceec5f37c48cf089ff8e24103279e1384a2c0fa1fc9446b5dc438)  to report an `OutOfMemoryError` using a `JNI_ENOMEM` result. This pull request aborts a compilation when `AttachCurrentThread` fails with `JNI_ENOMEM`.
>
> src/hotspot/share/jvmci/jvmciEnv.cpp line 239:
> 
>> 237: JVMCIEnv::JVMCIEnv(JavaThread* thread, JVMCICompileState* compile_state, const char* file, int line):
>> 238:     _throw_to_caller(false), _file(file), _line(line), _attach_threw_OOME(false), _compile_state(compile_state) {
>> 239:   init_env_mode_runtime(thread, NULL, false);
> 
> why is it only non-fatal for this single path?  Some comments explaining the reason for it being fatal in some paths and not in others would help.

I would say something like:

// In case of OOME, there's a good chance a subsequent attempt to attach might succeed.
// Other errors most likely indicate a non-recoverable error in the JVMCI runtime.

Note also that we will soon implement a change that fatal errors on libjvmci compiler threads will simply discard the thread.

-------------

PR: https://git.openjdk.org/jdk/pull/9978