Shenandoah: nmethod barrier needs to abandon per-nmethod lock before entering OOM protocol

[Zhengyu Gu]

On 9/10/19 2:11 PM, Roman Kennke wrote:
> So maybe the deadlock can be resolved somehow else? E.g. by entering the
> scopes/locks always in the same order? E.g.:
> 
> nmethod-barrier does:
>   1. per-nmethod-lock 2. OOM scope
> 
> concurrent nmethod processing does:
>   1. OOM scope 2. per-nmethod-lock
> 
> Can this be changed that concurrent nmethod processing does grab the OOM
> scope *after* taking the per-nmethod-lock, just like the barrier does?

It is possible to have concurrent thread does:
   1. per-nmethod-lock, 2. OOM scope

but it is quite expensive, it requires setting up/tearing down OOM scope 
for each nmethod.


-Zhengyu

> 
> Roman
> 
>> There can be deadlock when nmethod barrier fails to evacuate oops and
>> enter OOM protocol while holding per-nmethod lock, cause concurrent
>> nmethod processing may try to obtain the specific lock while under OOM
>> protocol.
>>
>> The solution is to have nmethod barrier temporary abandon the lock
>> before it enters OOM protocol, let concurrent thread to process the
>> nmethod. Upon completion of OOM, nmethod barrier can continue to
>> evacuate/disarm the nmethod, since the operations are idempotent.
>>
>> Webrev:
>> http://cr.openjdk.java.net/~zgu/shenandoah/nmeth_barrier_deadlock/webrev.00/
>>
>>
>>
>> Test:
>>    hotspot_gc_shenandoah (fastdebug and release)
>>
>>    specjvm (fastdebug and release) with options:
>> ${JAVA_HOME}/bin/java -jar jmh-specjvm2016.jar -foe true -f 1 -wi 5 -i 5
>> -t 1 -w 1s -r 1s --jvmArgs "-Xmx1g -Xms1g
>> -XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGC
>> -XX:+UnlockDiagnosticVMOptions -XX:ShenandoahGCHeuristics=aggressive
>> -XX:+ShenandoahOOMDuringEvacALot"
>>
>> Thanks,
>>
>> -Zhengyu
>