RFR 8247670: Shenandoah: deadlock during class unloading OOME

[Zhengyu Gu]

The deadlock is caused by JDK-8245288. After further investigation, 
JDK-8245288 change does not seem to provide improvement that reported in 
original bug. Let's just backout JDK-8245288.

Webrev: http://cr.openjdk.java.net/~zgu/JDK-8247670/webrev.01/

Test:
   hotspot_gc_shenandoah

Thanks,

-Zhengyu


On 6/16/20 3:48 PM, Zhengyu Gu wrote:
> The deadlock is caused by one thread holding per-nmethod lock, then 
> encountering evac-oom. At the same time, another thread entering 
> evac-oom scope, then acquiring the same per-nmethod lock.
> 
> The first thread expects the second thread to see evac-oom and exit the 
> scope, but the second thread is blocked on acquiring per-nmethod lock.
> 
> The solution is to introduce an abortable locker on per-nmethod lock. If 
> the second thread can not acquire the lock, but see evac-oom, it simply 
> aborts, so it can exit evac-oom scope.
> 
> The solution does come with penalties:
> 
> If the second thread is a Java thread (via nmethod entry barrier), the 
> nmethod will be deopt.
> 
> If the second thread is worker, it causes current code root processing 
> to abort, then restart.
> 
> 
> Bug: https://bugs.openjdk.java.net/browse/JDK-8247670
> Webrev: http://cr.openjdk.java.net/~zgu/JDK-8247670/webrev.00/
> 
> Test:
>    hotspot_gc_shenandoah (x86_64 and aarch64)
> 
> Thanks,
> 
> -Zhengyu