Shenandoah: nmethod barrier needs to abandon per-nmethod lock before entering OOM protocol

[Zhengyu Gu]

> I think it should be ok to do that. Setting up / leaving OOM scope
> shouldn't be more expensive than acquiring/releasing a lock. It would
> keep the change much simpler/cleaner, wouldn't it?

In that case, of course, it is much simpler ..

http://cr.openjdk.java.net/~zgu/shenandoah/nmeth_barrier_deadlock/webrev.01/

passed specjvm tests, other tests in progress.

Thanks,

-Zhengyu


> 
> Roman
> 
> 
>>>
>>> *or* change the nmethod barrier to do:
>>>
>>>    1. OOM scope 2. per-nmethod-lock
>>
>> This won't work.
>>
>> The  thread, that causeed OOM, will wait other threads get out of OOM
>> scope while holding the lock, and another thread can stick waiting for
>> the lock inside OOM scope.
>>
>> -Zhengyu
>>
>>>
>>> instead?
>>>
>>> Roman
>>>
>>>
>>>
>>>> -Zhengyu
>>>>
>>>>>
>>>>> Roman
>>>>>
>>>>>> There can be deadlock when nmethod barrier fails to evacuate oops and
>>>>>> enter OOM protocol while holding per-nmethod lock, cause concurrent
>>>>>> nmethod processing may try to obtain the specific lock while under OOM
>>>>>> protocol.
>>>>>>
>>>>>> The solution is to have nmethod barrier temporary abandon the lock
>>>>>> before it enters OOM protocol, let concurrent thread to process the
>>>>>> nmethod. Upon completion of OOM, nmethod barrier can continue to
>>>>>> evacuate/disarm the nmethod, since the operations are idempotent.
>>>>>>
>>>>>> Webrev:
>>>>>> http://cr.openjdk.java.net/~zgu/shenandoah/nmeth_barrier_deadlock/webrev.00/
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Test:
>>>>>>      hotspot_gc_shenandoah (fastdebug and release)
>>>>>>
>>>>>>      specjvm (fastdebug and release) with options:
>>>>>> ${JAVA_HOME}/bin/java -jar jmh-specjvm2016.jar -foe true -f 1 -wi 5
>>>>>> -i 5
>>>>>> -t 1 -w 1s -r 1s --jvmArgs "-Xmx1g -Xms1g
>>>>>> -XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGC
>>>>>> -XX:+UnlockDiagnosticVMOptions -XX:ShenandoahGCHeuristics=aggressive
>>>>>> -XX:+ShenandoahOOMDuringEvacALot"
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> -Zhengyu
>>>>>
>>>
>>