Shenandoah: nmethod barrier needs to abandon per-nmethod lock before entering OOM protocol

[Roman Kennke]

>>>> So maybe the deadlock can be resolved somehow else? E.g. by entering
>>>> the
>>>> scopes/locks always in the same order? E.g.:
>>>>
>>>> nmethod-barrier does:
>>>>    1. per-nmethod-lock 2. OOM scope
>>>>
>>>> concurrent nmethod processing does:
>>>>    1. OOM scope 2. per-nmethod-lock
>>>>
>>>> Can this be changed that concurrent nmethod processing does grab the
>>>> OOM
>>>> scope *after* taking the per-nmethod-lock, just like the barrier does?
>>>
>>> It is possible to have concurrent thread does:
>>>    1. per-nmethod-lock, 2. OOM scope
>>>
>>> but it is quite expensive, it requires setting up/tearing down OOM scope
>>> for each nmethod.
>>
>> Is that so bad? barriers do that per-oop.
> Donn't know, but could have thousands nmethods.

I think it should be ok to do that. Setting up / leaving OOM scope
shouldn't be more expensive than acquiring/releasing a lock. It would
keep the change much simpler/cleaner, wouldn't it?

Roman


>>
>> *or* change the nmethod barrier to do:
>>
>>   1. OOM scope 2. per-nmethod-lock
> 
> This won't work.
> 
> The  thread, that causeed OOM, will wait other threads get out of OOM
> scope while holding the lock, and another thread can stick waiting for
> the lock inside OOM scope.
> 
> -Zhengyu
> 
>>
>> instead?
>>
>> Roman
>>
>>
>>
>>> -Zhengyu
>>>
>>>>
>>>> Roman
>>>>
>>>>> There can be deadlock when nmethod barrier fails to evacuate oops and
>>>>> enter OOM protocol while holding per-nmethod lock, cause concurrent
>>>>> nmethod processing may try to obtain the specific lock while under OOM
>>>>> protocol.
>>>>>
>>>>> The solution is to have nmethod barrier temporary abandon the lock
>>>>> before it enters OOM protocol, let concurrent thread to process the
>>>>> nmethod. Upon completion of OOM, nmethod barrier can continue to
>>>>> evacuate/disarm the nmethod, since the operations are idempotent.
>>>>>
>>>>> Webrev:
>>>>> http://cr.openjdk.java.net/~zgu/shenandoah/nmeth_barrier_deadlock/webrev.00/
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Test:
>>>>>     hotspot_gc_shenandoah (fastdebug and release)
>>>>>
>>>>>     specjvm (fastdebug and release) with options:
>>>>> ${JAVA_HOME}/bin/java -jar jmh-specjvm2016.jar -foe true -f 1 -wi 5
>>>>> -i 5
>>>>> -t 1 -w 1s -r 1s --jvmArgs "-Xmx1g -Xms1g
>>>>> -XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGC
>>>>> -XX:+UnlockDiagnosticVMOptions -XX:ShenandoahGCHeuristics=aggressive
>>>>> -XX:+ShenandoahOOMDuringEvacALot"
>>>>>
>>>>> Thanks,
>>>>>
>>>>> -Zhengyu
>>>>
>>
>