RFR: 8273608: Deadlock when jcmd of OnError attaches to itself [v4]

[Xin Liu]

On Tue, 5 Oct 2021 18:27:45 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> Xin Liu has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Cleanup: remove a statment of debugging.
>
> test/hotspot/jtreg/runtime/ErrorHandling/TestOutOfMemoryErrorFromNIO.java line 25:
> 
>> 23: 
>> 24: /*
>> 25:  * @test TestOutOfMemoryErrorFromNIO
> 
> I would rename the test. The point is not to test OOM from NIO, but to test XX:OnError with jcmd which attaches to the parent. 
> 
> You could do the same with any OOM, right? E.g. just start the VM with a very small MaxMetaspaceSize. You would not even need a main function for that, since the VM would not come up but throw a OOM right away,
> 
> Proposal (but feel free to come up with something better): "TestOnErrorWithSelfAttachingJCmd"

I found HotSpot is very subtle when it comes to `MaxMetaspaceSize` and `-XX:AbortVMOnException=java.lang.OutOfMemoryError`

On my host, the watershed is 3M. If I give it 2M, it will ignore `AbortVMOnException`. 


java -XX:MaxMetaspaceSize=2M -XX:AbortVMOnException=java.lang.OutOfMemoryError 
Error occurred during initialization of VM
OutOfMemoryError: Metaspace


Flip code blocks here can solve this problem. is it intentional? 


diff --git a/src/hotspot/share/memory/metaspace.cpp b/src/hotspot/share/memory/metaspace.cpp
index 74b5d1a0b30..efe430f7d3c 100644
--- a/src/hotspot/share/memory/metaspace.cpp
+++ b/src/hotspot/share/memory/metaspace.cpp
@@ -974,15 +974,16 @@ void Metaspace::report_metadata_oome(ClassLoaderData* loader_data, size_t word_s
         space_string);
   }

-  if (!is_init_completed()) {
-    vm_exit_during_initialization("OutOfMemoryError", space_string);
-  }
-
   if (out_of_compressed_class_space) {
     THROW_OOP(Universe::out_of_memory_error_class_metaspace());
   } else {
     THROW_OOP(Universe::out_of_memory_error_metaspace());
   }
+
+  if (!is_init_completed()) {
+    vm_exit_during_initialization("OutOfMemoryError", space_string);
+  }
+
 }

-------------

PR: https://git.openjdk.java.net/jdk/pull/5590