RFR: 8321401: Shenandoah: Each mutator must see FullGC before throwing OOM

[Kelvin Nilsen]

On Tue, 5 Dec 2023 23:38:14 GMT, William Kemper <wkemper at openjdk.org> wrote:

>> Require each thread to observe unproductive Full GC before it throws OOM exception.
>
> src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 885:
> 
>> 883:     // This will notify the collector to start a cycle, but will raise
>> 884:     // an OOME to the mutator if the last Full GCs have not made progress.
>> 885:     if (result == nullptr && !req.is_lab_alloc() && get_gc_no_progress_count() > ShenandoahNoProgressThreshold) {
> 
> We need to be careful here. This essentially backs out the fix for:
> * https://bugs.openjdk.org/browse/JDK-8316632
> * https://bugs.openjdk.org/browse/JDK-8314935
> 
> For the original change, I ran tier1, tier2 and tier3 tests with Shenandoah enabled on x86 and aarch64, per @shipilev 's request (https://github.com/openjdk/jdk/pull/15852#issuecomment-1729099298).
> 
> make test TEST="tier1 tier2 tier3" TEST_VM_OPTS="-XX:+UseShenandoahGC" CONF=fastdebug
> 
> These tests, specifically,  will likely timeout without this check:
> 
> make test TEST="runtime/reflect/ReflectOutOfMemoryError.java gc/InfiniteList.java runtime/ClassInitErrors/TestOutOfMemoryDuringInit.java compiler/uncommontrap/TestDeoptOOM.java" TEST_VM_OPTS="-Xlog:gc*=info:file=/tmp/oome.log -XX:+UseShenandoahGC" JTREG="REPEAT_COUNT=50" CONF=fastdebug

I'll run these tests and report the results.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/16985#discussion_r1417525665