RFR: 8366865: G1: Allocation GC Pauses Triggered after JVM has started shutdown

[David Holmes]

On Wed, 10 Sep 2025 09:57:18 GMT, Ivan Walulya <iwalulya at openjdk.org> wrote:

> Please review this patch to skip VM_GC_Collect_Operations  if initiated after the VM shutdown process has begun. We add a _is_shutting_down flag to CollectedHeap, which is set while holding the Heap_lock. This ensures mutual exclusion with VM_GC_Collect_Operations, which also require the Heap_lock.
> 
> Skipping VM_GC_Collect_Operation would otherwise cause allocation requests to fail (resulting in OutOfMemoryError) if requesting daemon threads were allowed to continue, we instead block these threads on a monitor. They remain stalled until they are terminated as part of the VM shutdown sequence.
> 
> Testing: Tier 1-7

src/hotspot/share/gc/shared/collectedHeap.cpp line 616:

> 614:   // If the VM is shutting down, we may have skipped VM_CollectForAllocation.
> 615:   // To avoid returning nullptr (which could cause premature OOME), we stall
> 616:   // allocation requests here until the VM shutdown is complete.

If I understand the shutdown sequence correctly, I don't think you can do this without risking a hang. The thread doing the shutdown can potentially execute code that requires allocation after `is_shutting_down()` returns true. This is due to the JVMTI events posted from `before_exit`:

if (JvmtiExport::should_post_thread_life()) {
    JvmtiExport::post_thread_end(thread);
  }

  // Always call even when there are not JVMTI environments yet, since environments
  // may be attached late and JVMTI must track phases of VM execution
  JvmtiExport::post_vm_death();

I think if you can't GC during shutdown then you have to simply let the allocation fail.

src/hotspot/share/runtime/mutexLocker.cpp line 85:

> 83: Monitor* InitCompleted_lock           = nullptr;
> 84: Monitor* BeforeExit_lock              = nullptr;
> 85: Monitor* VMExit_lock                  = nullptr;

It is unfortunate that we need yet-another exit related mutex.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/27190#discussion_r2338299593
PR Review Comment: https://git.openjdk.org/jdk/pull/27190#discussion_r2338286600