RFR (S) 8231249: Shenandoah: GC retries are too aggressive for tests that expect OOME

[Roman Kennke]

Ok. Thanks!

Roman


> Bug:
>    https://bugs.openjdk.java.net/browse/JDK-8231249
> 
> This readily reproduces with vmTestbase_nsk_jvmti. Some tests (notably
> vmTestbase/nsk/jvmti/GarbageCollectionStart/gcstart001) are filling up the memory until OOME
> happens, and even count the OOMEs happened. Unfortunately, that clashes with Shenandoah's quite
> aggressive retry policy. At some point, test is able to sneak past allocation failure and make
> progress, and less aggressive policy helps that.
> 
> Fix:
>    http://cr.openjdk.java.net/~shade/8231249/webrev.01/
> 
> Since Degenerated GC would dive into Full GC on low progress, we can just trust the flag for
> retries. We also need to poll the flag after at least one attempt at handling the alloc failure
> (thus entering degen-full sequence). This makes retry logic less aggressive, which saves us from
> test timeouts.
> 
> Testing: vmTestbase_nsk_jvmti; hotspot_gc_shenandoah
>