RFR: 8253857: Shenandoah: Bugs in ShenandoahEvacOOMHandler related code

[Zhengyu Gu]

There are a couple of bugs in ShenandoahEvacOOMHandler related code.

1) ShenandoahEvacOOMHandler::register_thread()
It does not do OOM check before blindly cmpxchg in threads_in_evac. The cmpxchg can success even OOM is in progress,
that results the thread to continue evacuate objects.

2) When a nested OOM scope encounters OOM, it does not decrease threads_in_evac count before entering
wait_for_no_evac_threads() loop, it deadlocks itself.

-------------

Commit messages:
 - 8253857: Shenandoah: Bugs in ShenandoahEvacOOMHandler related code

Changes: https://git.openjdk.java.net/jdk/pull/439/files
 Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=439&range=00
  Issue: https://bugs.openjdk.java.net/browse/JDK-8253857
  Stats: 26 lines in 2 files changed: 8 ins; 7 del; 11 mod
  Patch: https://git.openjdk.java.net/jdk/pull/439.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/439/head:pull/439

PR: https://git.openjdk.java.net/jdk/pull/439