Shenandoah: nmethod barrier needs to abandon per-nmethod lock before entering OOM protocol

[Zhengyu Gu]

On 9/10/19 4:36 PM, Roman Kennke wrote:
> Looks good.
> 
> If it really turns out to be a performance problem, then we can still do
> what you originally proposed.
> 
> BTW, we do skip nmethods that we know have no oops at all, do we?

No, because we still have to disarm them and following 
NMethod::unload_nmethod_caches() may potential invoke native barrier, so 
we still need to setup OOM scope.

-Zhengyu


> 
> Roman
> 
>>> I think it should be ok to do that. Setting up / leaving OOM scope
>>> shouldn't be more expensive than acquiring/releasing a lock. It would
>>> keep the change much simpler/cleaner, wouldn't it?
>>
>> In that case, of course, it is much simpler ..
>>
>> http://cr.openjdk.java.net/~zgu/shenandoah/nmeth_barrier_deadlock/webrev.01/
>>
>>
>> passed specjvm tests, other tests in progress.
>>
>> Thanks,
>>
>> -Zhengyu
>>
>>
>>>
>>> Roman
>>>
>>>
>>>>>
>>>>> *or* change the nmethod barrier to do:
>>>>>
>>>>>     1. OOM scope 2. per-nmethod-lock
>>>>
>>>> This won't work.
>>>>
>>>> The  thread, that causeed OOM, will wait other threads get out of OOM
>>>> scope while holding the lock, and another thread can stick waiting for
>>>> the lock inside OOM scope.
>>>>
>>>> -Zhengyu
>>>>
>>>>>
>>>>> instead?
>>>>>
>>>>> Roman
>>>>>
>>>>>
>>>>>
>>>>>> -Zhengyu
>>>>>>
>>>>>>>
>>>>>>> Roman
>>>>>>>
>>>>>>>> There can be deadlock when nmethod barrier fails to evacuate oops
>>>>>>>> and
>>>>>>>> enter OOM protocol while holding per-nmethod lock, cause concurrent
>>>>>>>> nmethod processing may try to obtain the specific lock while
>>>>>>>> under OOM
>>>>>>>> protocol.
>>>>>>>>
>>>>>>>> The solution is to have nmethod barrier temporary abandon the lock
>>>>>>>> before it enters OOM protocol, let concurrent thread to process the
>>>>>>>> nmethod. Upon completion of OOM, nmethod barrier can continue to
>>>>>>>> evacuate/disarm the nmethod, since the operations are idempotent.
>>>>>>>>
>>>>>>>> Webrev:
>>>>>>>> http://cr.openjdk.java.net/~zgu/shenandoah/nmeth_barrier_deadlock/webrev.00/
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Test:
>>>>>>>>       hotspot_gc_shenandoah (fastdebug and release)
>>>>>>>>
>>>>>>>>       specjvm (fastdebug and release) with options:
>>>>>>>> ${JAVA_HOME}/bin/java -jar jmh-specjvm2016.jar -foe true -f 1 -wi 5
>>>>>>>> -i 5
>>>>>>>> -t 1 -w 1s -r 1s --jvmArgs "-Xmx1g -Xms1g
>>>>>>>> -XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGC
>>>>>>>> -XX:+UnlockDiagnosticVMOptions -XX:ShenandoahGCHeuristics=aggressive
>>>>>>>> -XX:+ShenandoahOOMDuringEvacALot"
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> -Zhengyu
>>>>>>>
>>>>>
>>>>
>