RFR: 8346920: Serial: Support allocation in old generation when heap is almost full [v2]

[Axel Boldt-Christmas]

On Fri, 24 Jan 2025 11:14:01 GMT, Albert Mingkun Yang <ayang at openjdk.org> wrote:

>> This PR restores the support for allocation in old-gen when heap is almost full, detected as non-empty young-gen after full gc, which was removed by mistake in [JDK-8333786](https://bugs.openjdk.org/browse/JDK-8333786). With this change, the benchmark attached to the ticket now completes in ~13 GC.
>> 
>> Test: tier1-5
>
> Albert Mingkun Yang has updated the pull request incrementally with one additional commit since the last revision:
> 
>   review

Parallel has the same problem right?

It feels like we are throwing OOM to late. When we have a promotion failure and the following full fails to evacuate eden, maybe we should just bail rather than allocating in old. 

I played around with something like this. Obviously not OK as it does not clear soft refs etc. But I think it is worth discussing if we should be so free with allocating new objects directly in the old generation.

diff --git a/src/hotspot/share/gc/parallel/parallelScavengeHeap.cpp b/src/hotspot/share/gc/parallel/parallelScavengeHeap.cpp
index 6b1b25dd6aa..d0152aa1a99 100644
--- a/src/hotspot/share/gc/parallel/parallelScavengeHeap.cpp
+++ b/src/hotspot/share/gc/parallel/parallelScavengeHeap.cpp
@@ -452,10 +452,15 @@ HeapWord* ParallelScavengeHeap::satisfy_failed_allocation(size_t size, bool is_t
     return expand_heap_and_allocate(size, is_tlab);
   }
 
+  const unsigned tfc = total_full_collections();
   // If young-gen can handle this allocation, attempt young-gc firstly.
   bool should_run_young_gc = is_tlab || should_alloc_in_eden(size);
   collect_at_safepoint(!should_run_young_gc);
 
+  if (UseNewCode && should_run_young_gc && tfc != total_full_collections() && !young_gen()->eden_space()->is_empty()) {
+    return nullptr;
+  }
+
   result = expand_heap_and_allocate(size, is_tlab);
   if (result != nullptr) {
     return result;
diff --git a/src/hotspot/share/gc/serial/serialHeap.cpp b/src/hotspot/share/gc/serial/serialHeap.cpp
index 23f1df6fc66..62f9d877404 100644
--- a/src/hotspot/share/gc/serial/serialHeap.cpp
+++ b/src/hotspot/share/gc/serial/serialHeap.cpp
@@ -529,10 +529,16 @@ HeapWord* SerialHeap::satisfy_failed_allocation(size_t size, bool is_tlab) {
     return result;   // Could be null if we are out of space.
   }
 
+  const unsigned tfc = total_full_collections();
+
   // If young-gen can handle this allocation, attempt young-gc firstly.
   bool should_run_young_gc = _young_gen->should_allocate(size, is_tlab);
   collect_at_safepoint(!should_run_young_gc);
 
+  if (UseNewCode && should_run_young_gc && tfc != total_full_collections() && !young_gen()->eden()->is_empty()) {
+    return nullptr;
+  }
+
   result = attempt_allocation(size, is_tlab, false /*first_only*/);
   if (result != nullptr) {
     return result;

-------------

PR Review: https://git.openjdk.org/jdk/pull/23270#pullrequestreview-2575676583