Shenandoah: nmethod barrier needs to abandon per-nmethod lock before entering OOM protocol

[Roman Kennke]

> On 9/10/19 1:28 PM, Roman Kennke wrote:
>> Do you need to store the lock in the thread-local data? Can't you simply
>> keep it in the evac-scope-object?
> 
> Is there a way to get to evac-scope from OOM handler?

Hmm, right.

So we have a whole bunch of stuff already in TLD that is there for
dealing with OOM-during-evac. You are about to add another one.

Maybe it makes sense to put the OOM scope object in TLD instead, and let
that keep the fields? It doesn't have to be a pointer, it can be flat,
and effectively have the same layout as it is now, but it would be clearer.

Also, I am not convinced that this is safe. Would the code that held the
lock still be good after it comes back from OOM handler?

Roman


> 
> -Zhengyu
> 
>>
>> Roman
>>
>>> There can be deadlock when nmethod barrier fails to evacuate oops and
>>> enter OOM protocol while holding per-nmethod lock, cause concurrent
>>> nmethod processing may try to obtain the specific lock while under OOM
>>> protocol.
>>>
>>> The solution is to have nmethod barrier temporary abandon the lock
>>> before it enters OOM protocol, let concurrent thread to process the
>>> nmethod. Upon completion of OOM, nmethod barrier can continue to
>>> evacuate/disarm the nmethod, since the operations are idempotent.
>>>
>>> Webrev:
>>> http://cr.openjdk.java.net/~zgu/shenandoah/nmeth_barrier_deadlock/webrev.00/
>>>
>>>
>>>
>>>
>>> Test:
>>>    hotspot_gc_shenandoah (fastdebug and release)
>>>
>>>    specjvm (fastdebug and release) with options:
>>> ${JAVA_HOME}/bin/java -jar jmh-specjvm2016.jar -foe true -f 1 -wi 5 -i 5
>>> -t 1 -w 1s -r 1s --jvmArgs "-Xmx1g -Xms1g
>>> -XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGC
>>> -XX:+UnlockDiagnosticVMOptions -XX:ShenandoahGCHeuristics=aggressive
>>> -XX:+ShenandoahOOMDuringEvacALot"
>>>
>>> Thanks,
>>>
>>> -Zhengyu
>>