RFR: 8371990: Remove two second delayed OOME after GC shutdown [v3]

[Kim Barrett]

On Tue, 18 Nov 2025 12:46:28 GMT, Stefan Karlsson <stefank at openjdk.org> wrote:

>> In [JDK-8366865](https://bugs.openjdk.org/browse/JDK-8366865) the shutdown code was tweaked so that allocating code would try to block for two seconds and if the JVM didn't shut down within that time, an OOME was thrown from the allocating thread.
>> 
>> One of the reason why this code was introduced was to deal with shutdown problem where the thread that were shutting down the JVM would first initiate the shutdown of the GC and *after* that the thread would call the JVMTI shutdown events and callbacks. The JVMTI callbacks could call arbitrary Java code that could try to allocate memory, and if the heap was filled up, it would have to wait for a GC to do its thing and hand back memory. But the GC had initiated its termination protocol and could be unresponsive to that request, which in term would lead to hanging JVM process.
>> 
>> The problem described above was finally fixed with [JDK-8367902](https://bugs.openjdk.org/browse/JDK-8367902).
>> 
>> So, I propose that we get rid of the workaround put into place with [JDK-8366865](https://bugs.openjdk.org/browse/JDK-8366865).
>> 
>> The proposed patch restructures the GC shutdown a little bit. The idea is all threads that want to schedule a GC VM Operation already take the Heap_lock, and while holding that lock they check the `_is_shutting_down` variable. If the the JVM indeed is shutting down, the threads refuse to schedule the GC operation.
>> 
>> Depending on the type of thread that is trying to schedule the GC operation we do one out of two things:
>> 
>> 1) If it is a Java thread, we simply block the thread from running. The thread is either a daemon thread and the blocking of the thread will not hinder the shutdown. Or, the thread is a non-daemon thread but the Java code called System.halt, which doesn't wait for non-daemon threads.
>> 
>> 2) If it is a Concurrent GC thread, then we let the thread proceed but with the order to skip the GC operation. This is done because the current shutdown code calls "stop" on the Concurrent GC threads and then wait for them to signal back when they have stopped running their code. So, we need to let them run to completion.
>> 
>> There are some G1 specific details to look at:
>> 
>> 1) I've reverted the G1 `concurrent_mark_is_terminating` checks.
>> 
>> 2) `try_collect_concurrently` queries the `_is_shutting_down` while holding the lock, and then uses that queried value after the lock is released.
>> 
>> 3) I've left some breadcrumbs in `should_clear_region`. Any suggestions on what to d...
>
> Stefan Karlsson has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Add comment to block_if_java_thread

A few remaining comment nits.

Marked as reviewed by kbarrett (Reviewer).

src/hotspot/share/gc/shared/gcVMOperations.cpp line 132:

> 130:   if (_is_shutting_down) {
> 131:     // Block forever if a Java thread is triggering a GC after
> 132:     // the GC have started to shut down.

s/have/has/

src/hotspot/share/gc/shared/gcVMOperations.hpp line 143:

> 141: 
> 142:   // This function returns the value of CollectedHeap::is_shutting_down() that
> 143:   // was recorded in the prologue. Unlike, CollectedHeap::is_shutting_down(),

Remove comma after "Unlike".

src/hotspot/share/gc/shared/gcVMOperations.hpp line 146:

> 144:   // this function can be called without acquiring the Heap_lock.
> 145:   //
> 146:   // This funciton exists so that code that tries to schedule a GC operation

s/funciton/function/

-------------

PR Review: https://git.openjdk.org/jdk/pull/28349#pullrequestreview-3479724039
PR Review: https://git.openjdk.org/jdk/pull/28349#pullrequestreview-3479729718
PR Review Comment: https://git.openjdk.org/jdk/pull/28349#discussion_r2539666824
PR Review Comment: https://git.openjdk.org/jdk/pull/28349#discussion_r2539665681
PR Review Comment: https://git.openjdk.org/jdk/pull/28349#discussion_r2539666064