Shenandoah: nmethod barrier needs to abandon per-nmethod lock before entering OOM protocol

[Roman Kennke]

So maybe the deadlock can be resolved somehow else? E.g. by entering the
scopes/locks always in the same order? E.g.:

nmethod-barrier does:
 1. per-nmethod-lock 2. OOM scope

concurrent nmethod processing does:
 1. OOM scope 2. per-nmethod-lock

Can this be changed that concurrent nmethod processing does grab the OOM
scope *after* taking the per-nmethod-lock, just like the barrier does?

Roman

> There can be deadlock when nmethod barrier fails to evacuate oops and
> enter OOM protocol while holding per-nmethod lock, cause concurrent
> nmethod processing may try to obtain the specific lock while under OOM
> protocol.
> 
> The solution is to have nmethod barrier temporary abandon the lock
> before it enters OOM protocol, let concurrent thread to process the
> nmethod. Upon completion of OOM, nmethod barrier can continue to
> evacuate/disarm the nmethod, since the operations are idempotent.
> 
> Webrev:
> http://cr.openjdk.java.net/~zgu/shenandoah/nmeth_barrier_deadlock/webrev.00/
> 
> 
> 
> Test:
>   hotspot_gc_shenandoah (fastdebug and release)
> 
>   specjvm (fastdebug and release) with options:
> ${JAVA_HOME}/bin/java -jar jmh-specjvm2016.jar -foe true -f 1 -wi 5 -i 5
> -t 1 -w 1s -r 1s --jvmArgs "-Xmx1g -Xms1g
> -XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGC
> -XX:+UnlockDiagnosticVMOptions -XX:ShenandoahGCHeuristics=aggressive
> -XX:+ShenandoahOOMDuringEvacALot"
> 
> Thanks,
> 
> -Zhengyu