RFC: call report_java_out_of_memory_error() for -XX:AbortVMOnException=java.lang.OutOfMemoryError

[Liu, Xin]

Hi,

Recently I revisit JDK-8155004/JDK-8257790 because a new team trip over.
-XX:AbortVMOnException=java.lang.OutOfMemoryError works. I wonder
whether it is a good idea to call report_java_out_of_memory_error() when
OOME is trapped. In this way, HotSpot will trigger OnOutOfMemoryError
callbacks.

I understand JDK-8257790 is not a bug. I don't want to overthrow that
conclusion. I just wonder if we can handle it better in the presence of
-XX:AbortVMOnException=java.lang.OutOfMemoryError.

For Java webservers, OOME may lead to a zombie process. We may have a
bug in code or indeed run out of memory. OOME is suppressed or terminate
the thread but don't terminate the java process. eg.

public class Main {
    volatile static boolean done = false;

    public static void main(String[] args) {
        String msg = "a long long message.";
        // write your code here
        Runnable runnable = () -> {
            int cnt = Integer.MAX_VALUE / msg.length() + 1;
            //it will throw a OutOfMemoryError.
            msg.repeat(cnt);
            done = true;
        };

        Thread thread = new Thread(runnable);
        thread.start();
        while(!done) {
        } // this simulates the main loop of event handling
    }
}

Java developers can use
-XX:AbortVMOnException=java.lang.OutOfMemoryError to exercise fail-fast
principle. Java web application which handle traffics are usually
distributed in a cluster. A failure of a single host usually is not a
big deal. As long as java exits, it's easy to restart and backfill it.

My proposing change is very simple. Just call
report_java_out_of_memory() if value_string is OOME. It's no-op if users
never specify anything. If they do specify flags like
Crash/ExitOnOutOfMemory,  OnOutOfMemoryError or
HeapDumpOnOutOfMemoryError, HotSpot will let report_java_out_of_memory
does the cleanup job. fatal() works but too brutal. I think we should
let java exits with error code.


diff --git a/src/hotspot/share/utilities/exceptions.cpp
b/src/hotspot/share/utilities/exceptions.cpp
index bd95b8306be..fd8a83deaf3 100644
--- a/src/hotspot/share/utilities/exceptions.cpp
+++ b/src/hotspot/share/utilities/exceptions.cpp
@@ -538,6 +538,9 @@ void Exceptions::debug_check_abort(const char
*value_string, const char* message
       strstr(value_string, AbortVMOnException)) {
     if (AbortVMOnExceptionMessage == NULL || (message != NULL &&
         strstr(message, AbortVMOnExceptionMessage))) {
+      if(!strcmp(value_string, "java.lang.OutOfMemoryError")) {
+        report_java_out_of_memory(message);
+      }
       fatal("Saw %s, aborting", value_string);
     }
   }


thanks,
--lx