RFR: 8321401: Shenandoah: Each mutator must see FullGC before throwing OOM

[William Kemper]

On Tue, 5 Dec 2023 20:30:54 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

> Require each thread to observe unproductive Full GC before it throws OOM exception.

src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 885:

> 883:     // This will notify the collector to start a cycle, but will raise
> 884:     // an OOME to the mutator if the last Full GCs have not made progress.
> 885:     if (result == nullptr && !req.is_lab_alloc() && get_gc_no_progress_count() > ShenandoahNoProgressThreshold) {

We need to be careful here. This essentially backs out the fix for:
* https://bugs.openjdk.org/browse/JDK-8316632
* https://bugs.openjdk.org/browse/JDK-8314935

For the original change, I ran tier1, tier2 and tier3 tests with Shenandoah enabled on x86 and aarch64, per @shipilev 's request (https://github.com/openjdk/jdk/pull/15852#issuecomment-1729099298).

make test TEST="tier1 tier2 tier3" TEST_VM_OPTS="-XX:+UseShenandoahGC" CONF=fastdebug

These tests, specifically,  will likely timeout without this check:

make test TEST="runtime/reflect/ReflectOutOfMemoryError.java gc/InfiniteList.java runtime/ClassInitErrors/TestOutOfMemoryDuringInit.java compiler/uncommontrap/TestDeoptOOM.java" TEST_VM_OPTS="-Xlog:gc*=info:file=/tmp/oome.log -XX:+UseShenandoahGC" JTREG="REPEAT_COUNT=50" CONF=fastdebug

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/16985#discussion_r1416404947