Shenandoah: nmethod barrier needs to abandon per-nmethod lock before entering OOM protocol

[Zhengyu Gu]

On 9/10/19 2:04 PM, Roman Kennke wrote:
>> On 9/10/19 1:28 PM, Roman Kennke wrote:
>>> Do you need to store the lock in the thread-local data? Can't you simply
>>> keep it in the evac-scope-object?
>>
>> Is there a way to get to evac-scope from OOM handler?
> 
> Hmm, right.
> 
> So we have a whole bunch of stuff already in TLD that is there for
> dealing with OOM-during-evac. You are about to add another one.
> 
> Maybe it makes sense to put the OOM scope object in TLD instead, and let
> that keep the fields? It doesn't have to be a pointer, it can be flat,
> and effectively have the same layout as it is now, but it would be clearer.
> 
> Also, I am not convinced that this is safe. Would the code that held the
> lock still be good after it comes back from OOM handler?

I think it is safe. As I mentioned, oop evacuation and nmethod disarming 
are idempotent.

So, nmethod barrier -> [1] evacuate oops -> [2] OOM -> [3] unlock -> [4] 
lock -> [5] continue evacuating oops -> [6] disarm nmethod.

[5][6] are idempotent.


-Zhengyu

> 
> Roman
> 
> 
>>
>> -Zhengyu
>>
>>>
>>> Roman
>>>
>>>> There can be deadlock when nmethod barrier fails to evacuate oops and
>>>> enter OOM protocol while holding per-nmethod lock, cause concurrent
>>>> nmethod processing may try to obtain the specific lock while under OOM
>>>> protocol.
>>>>
>>>> The solution is to have nmethod barrier temporary abandon the lock
>>>> before it enters OOM protocol, let concurrent thread to process the
>>>> nmethod. Upon completion of OOM, nmethod barrier can continue to
>>>> evacuate/disarm the nmethod, since the operations are idempotent.
>>>>
>>>> Webrev:
>>>> http://cr.openjdk.java.net/~zgu/shenandoah/nmeth_barrier_deadlock/webrev.00/
>>>>
>>>>
>>>>
>>>>
>>>> Test:
>>>>     hotspot_gc_shenandoah (fastdebug and release)
>>>>
>>>>     specjvm (fastdebug and release) with options:
>>>> ${JAVA_HOME}/bin/java -jar jmh-specjvm2016.jar -foe true -f 1 -wi 5 -i 5
>>>> -t 1 -w 1s -r 1s --jvmArgs "-Xmx1g -Xms1g
>>>> -XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGC
>>>> -XX:+UnlockDiagnosticVMOptions -XX:ShenandoahGCHeuristics=aggressive
>>>> -XX:+ShenandoahOOMDuringEvacALot"
>>>>
>>>> Thanks,
>>>>
>>>> -Zhengyu
>>>
>