RFR (S) 8231249: Shenandoah: GC retries are too aggressive for tests that expect OOME

[Aleksey Shipilev]

Bug:
  https://bugs.openjdk.java.net/browse/JDK-8231249

This readily reproduces with vmTestbase_nsk_jvmti. Some tests (notably
vmTestbase/nsk/jvmti/GarbageCollectionStart/gcstart001) are filling up the memory until OOME
happens, and even count the OOMEs happened. Unfortunately, that clashes with Shenandoah's quite
aggressive retry policy. At some point, test is able to sneak past allocation failure and make
progress, and less aggressive policy helps that.

Fix:
  http://cr.openjdk.java.net/~shade/8231249/webrev.01/

Since Degenerated GC would dive into Full GC on low progress, we can just trust the flag for
retries. We also need to poll the flag after at least one attempt at handling the alloc failure
(thus entering degen-full sequence). This makes retry logic less aggressive, which saves us from
test timeouts.

Testing: vmTestbase_nsk_jvmti; hotspot_gc_shenandoah

-- 
Thanks,
-Aleksey