Shenandoah: nmethod barrier needs to abandon per-nmethod lock before entering OOM protocol

[Zhengyu Gu]

There can be deadlock when nmethod barrier fails to evacuate oops and 
enter OOM protocol while holding per-nmethod lock, cause concurrent 
nmethod processing may try to obtain the specific lock while under OOM 
protocol.

The solution is to have nmethod barrier temporary abandon the lock 
before it enters OOM protocol, let concurrent thread to process the 
nmethod. Upon completion of OOM, nmethod barrier can continue to 
evacuate/disarm the nmethod, since the operations are idempotent.

Webrev: 
http://cr.openjdk.java.net/~zgu/shenandoah/nmeth_barrier_deadlock/webrev.00/


Test:
   hotspot_gc_shenandoah (fastdebug and release)

   specjvm (fastdebug and release) with options:
${JAVA_HOME}/bin/java -jar jmh-specjvm2016.jar -foe true -f 1 -wi 5 -i 5 
-t 1 -w 1s -r 1s --jvmArgs "-Xmx1g -Xms1g 
-XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGC 
-XX:+UnlockDiagnosticVMOptions -XX:ShenandoahGCHeuristics=aggressive 
-XX:+ShenandoahOOMDuringEvacALot"

Thanks,

-Zhengyu