RFR: 8321401: Shenandoah: Each mutator must see FullGC before throwing OOM

[Kelvin Nilsen]

On Wed, 6 Dec 2023 15:36:58 GMT, Kelvin Nilsen <kdnilsen at openjdk.org> wrote:

>> src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp line 885:
>> 
>>> 883:     // This will notify the collector to start a cycle, but will raise
>>> 884:     // an OOME to the mutator if the last Full GCs have not made progress.
>>> 885:     if (result == nullptr && !req.is_lab_alloc() && get_gc_no_progress_count() > ShenandoahNoProgressThreshold) {
>> 
>> We need to be careful here. This essentially backs out the fix for:
>> * https://bugs.openjdk.org/browse/JDK-8316632
>> * https://bugs.openjdk.org/browse/JDK-8314935
>> 
>> For the original change, I ran tier1, tier2 and tier3 tests with Shenandoah enabled on x86 and aarch64, per @shipilev 's request (https://github.com/openjdk/jdk/pull/15852#issuecomment-1729099298).
>> 
>> make test TEST="tier1 tier2 tier3" TEST_VM_OPTS="-XX:+UseShenandoahGC" CONF=fastdebug
>> 
>> These tests, specifically,  will likely timeout without this check:
>> 
>> make test TEST="runtime/reflect/ReflectOutOfMemoryError.java gc/InfiniteList.java runtime/ClassInitErrors/TestOutOfMemoryDuringInit.java compiler/uncommontrap/TestDeoptOOM.java" TEST_VM_OPTS="-Xlog:gc*=info:file=/tmp/oome.log -XX:+UseShenandoahGC" JTREG="REPEAT_COUNT=50" CONF=fastdebug
>
> I'll run these tests and report the results.

For the four explicit tests, 50 runs each, all passed.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/16985#discussion_r1417622364