RFR: JDK-8292769: [JVMCI] OutOfMemoryError thrown when attaching the libgraal isolate causes HotSpot to crash. [v3]

[Tomáš Zezula]

On Thu, 25 Aug 2022 19:14:54 GMT, Doug Simon <dnsimon at openjdk.org> wrote:

>> src/hotspot/share/jvmci/jvmciEnv.cpp line 241:
>> 
>>> 239:   init_env_mode_runtime(thread, NULL, false);
>>> 240:   if (_attach_threw_OOME) {
>>> 241:     compile_state->set_failure(true, "Out of memory");
>> 
>> why is it only non-fatal for this single path?  Some comments explaining the reason for it being fatal in some paths and not in others would help.
>
> I would say something like:
> 
> // In case of OOME, there's a good chance a subsequent attempt to attach might succeed.
> // Other errors most likely indicate a non-recoverable error in the JVMCI runtime.
> 
> Note also that we will soon implement a change that fatal errors on libjvmci compiler threads will simply discard the thread.

I've added the comment.
The compile broker compilation can be safely restarted. The affected method will be compiled later when the compiler isolate has enough memory. The other code paths don't have this property. One possible place where we can handle the isolate OOME in a non-fatal way is the `CompilerToVM#attachCurrentThread`. In this place, we can use JNI to throw OOME to the caller.

-------------

PR: https://git.openjdk.org/jdk/pull/9978