RFR: 8253857: Shenandoah: Bugs in ShenandoahEvacOOMHandler related code

[Roman Kennke]

On Wed, 30 Sep 2020 18:57:43 GMT, Zhengyu Gu <zgu at openjdk.org> wrote:

> There are a couple of bugs in ShenandoahEvacOOMHandler related code.
> 
> 1) ShenandoahEvacOOMHandler::register_thread()
> It does not do OOM check before blindly cmpxchg in threads_in_evac. The cmpxchg can success even OOM is in progress,
> that results the thread to continue evacuate objects.
> 2) When a nested OOM scope encounters OOM, it does not decrease threads_in_evac count before entering
> wait_for_no_evac_threads() loop, it deadlocks itself.
> Test:
> - [x] hotspot_gc_shenandoah on Linux x86_64

Looks good! Thanks!

-------------

Marked as reviewed by rkennke (Reviewer).

PR: https://git.openjdk.java.net/jdk/pull/439