RFR: 8253857: Shenandoah: Bugs in ShenandoahEvacOOMHandler related code

[Zhengyu Gu]

On Tue, 6 Oct 2020 12:55:09 GMT, Roman Kennke <rkennke at openjdk.org> wrote:

>> There are a couple of bugs in ShenandoahEvacOOMHandler related code.
>> 
>> 1) ShenandoahEvacOOMHandler::register_thread()
>> It does not do OOM check before blindly cmpxchg in threads_in_evac. The cmpxchg can success even OOM is in progress,
>> that results the thread to continue evacuate objects.
>> 2) When a nested OOM scope encounters OOM, it does not decrease threads_in_evac count before entering
>> wait_for_no_evac_threads() loop, it deadlocks itself.
>> Test:
>> - [x] hotspot_gc_shenandoah on Linux x86_64
>
> src/hotspot/share/gc/shenandoah/shenandoahEvacOOMHandler.cpp line 58:
> 
>> 56:       return;
>> 57:     } else {
>> 58:       // Failure:
> 
> Don't we still need this block?

Yes, we do need to update threads_in_evac value for retry.

-------------

PR: https://git.openjdk.java.net/jdk/pull/439