Shenandoah: nmethod barrier needs to abandon per-nmethod lock before entering OOM protocol

[Roman Kennke]

> On 9/10/19 4:36 PM, Roman Kennke wrote:
>> Looks good.
>>
>> If it really turns out to be a performance problem, then we can still do
>> what you originally proposed.
>>
>> BTW, we do skip nmethods that we know have no oops at all, do we?
> 
> No, because we still have to disarm them and following
> NMethod::unload_nmethod_caches() may potential invoke native barrier, so
> we still need to setup OOM scope.

Hmm. Ok then.

Thanks,
Roman

> -Zhengyu
> 
> 
>>
>> Roman
>>
>>>> I think it should be ok to do that. Setting up / leaving OOM scope
>>>> shouldn't be more expensive than acquiring/releasing a lock. It would
>>>> keep the change much simpler/cleaner, wouldn't it?
>>>
>>> In that case, of course, it is much simpler ..
>>>
>>> http://cr.openjdk.java.net/~zgu/shenandoah/nmeth_barrier_deadlock/webrev.01/
>>>
>>>
>>>
>>> passed specjvm tests, other tests in progress.
>>>
>>> Thanks,
>>>
>>> -Zhengyu
>>>
>>>
>>>>
>>>> Roman
>>>>
>>>>
>>>>>>
>>>>>> *or* change the nmethod barrier to do:
>>>>>>
>>>>>>     1. OOM scope 2. per-nmethod-lock
>>>>>
>>>>> This won't work.
>>>>>
>>>>> The  thread, that causeed OOM, will wait other threads get out of OOM
>>>>> scope while holding the lock, and another thread can stick waiting for
>>>>> the lock inside OOM scope.
>>>>>
>>>>> -Zhengyu
>>>>>
>>>>>>
>>>>>> instead?
>>>>>>
>>>>>> Roman
>>>>>>
>>>>>>
>>>>>>
>>>>>>> -Zhengyu
>>>>>>>
>>>>>>>>
>>>>>>>> Roman
>>>>>>>>
>>>>>>>>> There can be deadlock when nmethod barrier fails to evacuate oops
>>>>>>>>> and
>>>>>>>>> enter OOM protocol while holding per-nmethod lock, cause
>>>>>>>>> concurrent
>>>>>>>>> nmethod processing may try to obtain the specific lock while
>>>>>>>>> under OOM
>>>>>>>>> protocol.
>>>>>>>>>
>>>>>>>>> The solution is to have nmethod barrier temporary abandon the lock
>>>>>>>>> before it enters OOM protocol, let concurrent thread to process
>>>>>>>>> the
>>>>>>>>> nmethod. Upon completion of OOM, nmethod barrier can continue to
>>>>>>>>> evacuate/disarm the nmethod, since the operations are idempotent.
>>>>>>>>>
>>>>>>>>> Webrev:
>>>>>>>>> http://cr.openjdk.java.net/~zgu/shenandoah/nmeth_barrier_deadlock/webrev.00/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Test:
>>>>>>>>>       hotspot_gc_shenandoah (fastdebug and release)
>>>>>>>>>
>>>>>>>>>       specjvm (fastdebug and release) with options:
>>>>>>>>> ${JAVA_HOME}/bin/java -jar jmh-specjvm2016.jar -foe true -f 1
>>>>>>>>> -wi 5
>>>>>>>>> -i 5
>>>>>>>>> -t 1 -w 1s -r 1s --jvmArgs "-Xmx1g -Xms1g
>>>>>>>>> -XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGC
>>>>>>>>> -XX:+UnlockDiagnosticVMOptions
>>>>>>>>> -XX:ShenandoahGCHeuristics=aggressive
>>>>>>>>> -XX:+ShenandoahOOMDuringEvacALot"
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> -Zhengyu
>>>>>>>>
>>>>>>
>>>>>
>>