RFR: JDK-8155004: CrashOnOutOfMemoryError doesn't work for OOM caused by inability to create threads

[David Phillips]

On Tue, 20 Apr 2021 10:52:26 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

> Greetings,
> 
> this is an old issue and I'd like to fix it. If we fail to create a java thread due to platform limitations we throw an OOM. But we fail to honor the various xxxOnOutOfMemoryError switches.
> 
> The fix is very straightforward. 
> 
> If fixes 
> - CrashOnOutOfMemoryError
> - ExitOnOutOfMemoryError
> - HeapDumpOnOutOfMemoryError
> - and, in theory "OnOutOfMemoryError=<user comand>".
> 
> the latter only in theory because most of the times whatever prevented the thread to start up will also prevent the fork needed to get the user command running.
> 
> One remaining question, maybe for a future RFE, is how we want to handle native threads creation error. AFAICS currently, failing to create a native thread may or may not result in a fatal shutdown, a log output, or just be ignored, depending on the thread. 
> 
> If `...OnOutOfMemoryError` is specified, should native thread creation failure be handled the same way as a java thread?
> 
> - No if I take the option name literally, since there is no OOM involved
> - Yes if I take into account what these switches are actually used for - analysis or quick shutdown of a JVM inside a container in case of an OOM. Since it is completely random which thread is hit by the limit.
> 
> Thanks, Thomas

Until the JVM has an option to handle this, you can use the simple [jvmkill](https://github.com/airlift/jvmkill) JVMTI agent which does a `SIGKILL` to the current process on the `JVMTI_EVENT_RESOURCE_EXHAUSTED` event.

-------------

PR: https://git.openjdk.org/jdk/pull/3586