Shenandoah: nmethod barrier needs to abandon per-nmethod lock before entering OOM protocol

[Roman Kennke]

>> So maybe the deadlock can be resolved somehow else? E.g. by entering the
>> scopes/locks always in the same order? E.g.:
>>
>> nmethod-barrier does:
>>   1. per-nmethod-lock 2. OOM scope
>>
>> concurrent nmethod processing does:
>>   1. OOM scope 2. per-nmethod-lock
>>
>> Can this be changed that concurrent nmethod processing does grab the OOM
>> scope *after* taking the per-nmethod-lock, just like the barrier does?
> 
> It is possible to have concurrent thread does:
>   1. per-nmethod-lock, 2. OOM scope
> 
> but it is quite expensive, it requires setting up/tearing down OOM scope
> for each nmethod.

Is that so bad? barriers do that per-oop.

*or* change the nmethod barrier to do:

 1. OOM scope 2. per-nmethod-lock

instead?

Roman



> -Zhengyu
> 
>>
>> Roman
>>
>>> There can be deadlock when nmethod barrier fails to evacuate oops and
>>> enter OOM protocol while holding per-nmethod lock, cause concurrent
>>> nmethod processing may try to obtain the specific lock while under OOM
>>> protocol.
>>>
>>> The solution is to have nmethod barrier temporary abandon the lock
>>> before it enters OOM protocol, let concurrent thread to process the
>>> nmethod. Upon completion of OOM, nmethod barrier can continue to
>>> evacuate/disarm the nmethod, since the operations are idempotent.
>>>
>>> Webrev:
>>> http://cr.openjdk.java.net/~zgu/shenandoah/nmeth_barrier_deadlock/webrev.00/
>>>
>>>
>>>
>>>
>>> Test:
>>>    hotspot_gc_shenandoah (fastdebug and release)
>>>
>>>    specjvm (fastdebug and release) with options:
>>> ${JAVA_HOME}/bin/java -jar jmh-specjvm2016.jar -foe true -f 1 -wi 5 -i 5
>>> -t 1 -w 1s -r 1s --jvmArgs "-Xmx1g -Xms1g
>>> -XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGC
>>> -XX:+UnlockDiagnosticVMOptions -XX:ShenandoahGCHeuristics=aggressive
>>> -XX:+ShenandoahOOMDuringEvacALot"
>>>
>>> Thanks,
>>>
>>> -Zhengyu
>>