Shenandoah: nmethod barrier needs to abandon per-nmethod lock before entering OOM protocol

[Roman Kennke]

Looks good.

If it really turns out to be a performance problem, then we can still do
what you originally proposed.

BTW, we do skip nmethods that we know have no oops at all, do we?

Roman

>> I think it should be ok to do that. Setting up / leaving OOM scope
>> shouldn't be more expensive than acquiring/releasing a lock. It would
>> keep the change much simpler/cleaner, wouldn't it?
> 
> In that case, of course, it is much simpler ..
> 
> http://cr.openjdk.java.net/~zgu/shenandoah/nmeth_barrier_deadlock/webrev.01/
> 
> 
> passed specjvm tests, other tests in progress.
> 
> Thanks,
> 
> -Zhengyu
> 
> 
>>
>> Roman
>>
>>
>>>>
>>>> *or* change the nmethod barrier to do:
>>>>
>>>>    1. OOM scope 2. per-nmethod-lock
>>>
>>> This won't work.
>>>
>>> The  thread, that causeed OOM, will wait other threads get out of OOM
>>> scope while holding the lock, and another thread can stick waiting for
>>> the lock inside OOM scope.
>>>
>>> -Zhengyu
>>>
>>>>
>>>> instead?
>>>>
>>>> Roman
>>>>
>>>>
>>>>
>>>>> -Zhengyu
>>>>>
>>>>>>
>>>>>> Roman
>>>>>>
>>>>>>> There can be deadlock when nmethod barrier fails to evacuate oops
>>>>>>> and
>>>>>>> enter OOM protocol while holding per-nmethod lock, cause concurrent
>>>>>>> nmethod processing may try to obtain the specific lock while
>>>>>>> under OOM
>>>>>>> protocol.
>>>>>>>
>>>>>>> The solution is to have nmethod barrier temporary abandon the lock
>>>>>>> before it enters OOM protocol, let concurrent thread to process the
>>>>>>> nmethod. Upon completion of OOM, nmethod barrier can continue to
>>>>>>> evacuate/disarm the nmethod, since the operations are idempotent.
>>>>>>>
>>>>>>> Webrev:
>>>>>>> http://cr.openjdk.java.net/~zgu/shenandoah/nmeth_barrier_deadlock/webrev.00/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Test:
>>>>>>>      hotspot_gc_shenandoah (fastdebug and release)
>>>>>>>
>>>>>>>      specjvm (fastdebug and release) with options:
>>>>>>> ${JAVA_HOME}/bin/java -jar jmh-specjvm2016.jar -foe true -f 1 -wi 5
>>>>>>> -i 5
>>>>>>> -t 1 -w 1s -r 1s --jvmArgs "-Xmx1g -Xms1g
>>>>>>> -XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGC
>>>>>>> -XX:+UnlockDiagnosticVMOptions -XX:ShenandoahGCHeuristics=aggressive
>>>>>>> -XX:+ShenandoahOOMDuringEvacALot"
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> -Zhengyu
>>>>>>
>>>>
>>>