RFR: 8253857: Shenandoah: Bugs in ShenandoahEvacOOMHandler related code

[Roman Kennke]

On Wed, 30 Sep 2020 18:57:43 GMT, Zhengyu Gu <zgu at openjdk.org> wrote:

> There are a couple of bugs in ShenandoahEvacOOMHandler related code.
> 
> 1) ShenandoahEvacOOMHandler::register_thread()
> It does not do OOM check before blindly cmpxchg in threads_in_evac. The cmpxchg can success even OOM is in progress,
> that results the thread to continue evacuate objects.
> 2) When a nested OOM scope encounters OOM, it does not decrease threads_in_evac count before entering
> wait_for_no_evac_threads() loop, it deadlocks itself.
> Test:
> - [x] hotspot_gc_shenandoah on Linux x86_64

src/hotspot/share/gc/shenandoah/shenandoahEvacOOMHandler.cpp line 58:

> 56:       return;
> 57:     } else {
> 58:       // Failure:

Don't we still need this block?

-------------

PR: https://git.openjdk.java.net/jdk/pull/439