RFR 8247670: Shenandoah: deadlock during class unloading OOME

[Zhengyu Gu]

The deadlock is caused by one thread holding per-nmethod lock, then 
encountering evac-oom. At the same time, another thread entering 
evac-oom scope, then acquiring the same per-nmethod lock.

The first thread expects the second thread to see evac-oom and exit the 
scope, but the second thread is blocked on acquiring per-nmethod lock.

The solution is to introduce an abortable locker on per-nmethod lock. If 
the second thread can not acquire the lock, but see evac-oom, it simply 
aborts, so it can exit evac-oom scope.

The solution does come with penalties:

If the second thread is a Java thread (via nmethod entry barrier), the 
nmethod will be deopt.

If the second thread is worker, it causes current code root processing 
to abort, then restart.


Bug: https://bugs.openjdk.java.net/browse/JDK-8247670
Webrev: http://cr.openjdk.java.net/~zgu/JDK-8247670/webrev.00/

Test:
   hotspot_gc_shenandoah (x86_64 and aarch64)

Thanks,

-Zhengyu