Shenandoah: nmethod barrier needs to abandon per-nmethod lock before entering OOM protocol

[Zhengyu Gu]

On 9/10/19 2:19 PM, Roman Kennke wrote:
>>> So maybe the deadlock can be resolved somehow else? E.g. by entering the
>>> scopes/locks always in the same order? E.g.:
>>>
>>> nmethod-barrier does:
>>>    1. per-nmethod-lock 2. OOM scope
>>>
>>> concurrent nmethod processing does:
>>>    1. OOM scope 2. per-nmethod-lock
>>>
>>> Can this be changed that concurrent nmethod processing does grab the OOM
>>> scope *after* taking the per-nmethod-lock, just like the barrier does?
>>
>> It is possible to have concurrent thread does:
>>    1. per-nmethod-lock, 2. OOM scope
>>
>> but it is quite expensive, it requires setting up/tearing down OOM scope
>> for each nmethod.
> 
> Is that so bad? barriers do that per-oop.
Donn't know, but could have thousands nmethods.


> 
> *or* change the nmethod barrier to do:
> 
>   1. OOM scope 2. per-nmethod-lock

This won't work.

The  thread, that causeed OOM, will wait other threads get out of OOM 
scope while holding the lock, and another thread can stick waiting for 
the lock inside OOM scope.

-Zhengyu

> 
> instead?
> 
> Roman
> 
> 
> 
>> -Zhengyu
>>
>>>
>>> Roman
>>>
>>>> There can be deadlock when nmethod barrier fails to evacuate oops and
>>>> enter OOM protocol while holding per-nmethod lock, cause concurrent
>>>> nmethod processing may try to obtain the specific lock while under OOM
>>>> protocol.
>>>>
>>>> The solution is to have nmethod barrier temporary abandon the lock
>>>> before it enters OOM protocol, let concurrent thread to process the
>>>> nmethod. Upon completion of OOM, nmethod barrier can continue to
>>>> evacuate/disarm the nmethod, since the operations are idempotent.
>>>>
>>>> Webrev:
>>>> http://cr.openjdk.java.net/~zgu/shenandoah/nmeth_barrier_deadlock/webrev.00/
>>>>
>>>>
>>>>
>>>>
>>>> Test:
>>>>     hotspot_gc_shenandoah (fastdebug and release)
>>>>
>>>>     specjvm (fastdebug and release) with options:
>>>> ${JAVA_HOME}/bin/java -jar jmh-specjvm2016.jar -foe true -f 1 -wi 5 -i 5
>>>> -t 1 -w 1s -r 1s --jvmArgs "-Xmx1g -Xms1g
>>>> -XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGC
>>>> -XX:+UnlockDiagnosticVMOptions -XX:ShenandoahGCHeuristics=aggressive
>>>> -XX:+ShenandoahOOMDuringEvacALot"
>>>>
>>>> Thanks,
>>>>
>>>> -Zhengyu
>>>
>