Shenandoah: nmethod barrier needs to abandon per-nmethod lock before entering OOM protocol

[Roman Kennke]

>>>> Do you need to store the lock in the thread-local data? Can't you
>>>> simply
>>>> keep it in the evac-scope-object?
>>>
>>> Is there a way to get to evac-scope from OOM handler?
>>
>> Hmm, right.
>>
>> So we have a whole bunch of stuff already in TLD that is there for
>> dealing with OOM-during-evac. You are about to add another one.
>>
>> Maybe it makes sense to put the OOM scope object in TLD instead, and let
>> that keep the fields? It doesn't have to be a pointer, it can be flat,
>> and effectively have the same layout as it is now, but it would be
>> clearer.
> Make sense. Can we do it in separate CR?

Yes, ok.

> Thanks,
> 
> -Zhengyu
> 
> 
>>
>> Also, I am not convinced that this is safe. Would the code that held the
>> lock still be good after it comes back from OOM handler?
>>
>> Roman
>>
>>
>>>
>>> -Zhengyu
>>>
>>>>
>>>> Roman
>>>>
>>>>> There can be deadlock when nmethod barrier fails to evacuate oops and
>>>>> enter OOM protocol while holding per-nmethod lock, cause concurrent
>>>>> nmethod processing may try to obtain the specific lock while under OOM
>>>>> protocol.
>>>>>
>>>>> The solution is to have nmethod barrier temporary abandon the lock
>>>>> before it enters OOM protocol, let concurrent thread to process the
>>>>> nmethod. Upon completion of OOM, nmethod barrier can continue to
>>>>> evacuate/disarm the nmethod, since the operations are idempotent.
>>>>>
>>>>> Webrev:
>>>>> http://cr.openjdk.java.net/~zgu/shenandoah/nmeth_barrier_deadlock/webrev.00/
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Test:
>>>>>     hotspot_gc_shenandoah (fastdebug and release)
>>>>>
>>>>>     specjvm (fastdebug and release) with options:
>>>>> ${JAVA_HOME}/bin/java -jar jmh-specjvm2016.jar -foe true -f 1 -wi 5
>>>>> -i 5
>>>>> -t 1 -w 1s -r 1s --jvmArgs "-Xmx1g -Xms1g
>>>>> -XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGC
>>>>> -XX:+UnlockDiagnosticVMOptions -XX:ShenandoahGCHeuristics=aggressive
>>>>> -XX:+ShenandoahOOMDuringEvacALot"
>>>>>
>>>>> Thanks,
>>>>>
>>>>> -Zhengyu
>>>>
>>
>