RFR: Implement protocol for safe OOM during evacuation handling

[Roman Kennke]

Am 05.03.2018 um 17:20 schrieb Zhengyu Gu:
>>>> I don't think so. After the protocol is done, it will keep up the
>>>> OOM_MARKER_MASK until it's cleared during safepoint. This is checked
>>>> upon entry of the protocol, and does a fast-path-return after
>>>> setting up
>>>> the flags to return with RB.
>>>>
>>>> If it happens to enter repeatedly while the protocol is in progress, it
>>>> will participate in it as normal, e.g. will be blocked to enter until
>>>> the protocol is done, and then observe the OOM_MARKER_MASK and return
>>>> with RB as above.
>>>>
>>>> It must be ok to enter into the protocol repeatedly without race. If
>>>> you
>>>> find a race please describe it to me.
>>>>
>>> What I described above, is an example, no?
>>>
>>> L#491 drops counter to 0, so WB exit.
>>> L#492 fails to terminate, re-enters protocol. However, there is no check
>>> if OOM protocol is in progress, then goes to L#484, and find some
>>> thing ...
>>
>> There is a check upon (re-)entry:
>>
>> https://paste.fedoraproject.org/paste/dVol6OSw8S37dH5RBNztMQ
>>
>> Which means it raises the TL-flag again in wait_for_no_evac_flag() and
>> thus cause any encapsulated call to evac_obj() to return with RB. The
>> actual wait loop will not be entered again if the protocol was already
>> done.
> 
> Okay, I missed flag checking in evac_obj.
> 

So, there turned out to be one problem in the protocol: the part that
checks if we already done it was inside #ifdef ASSERT, and thus it was
deadlocking in release because evacuate_object() was entered multiply,
and thus the counter went negative. Fixed by moving the check out of the
#ifdef.

Differential:
http://cr.openjdk.java.net/~rkennke/safe-oom-during-evac/webrev.09.diff
Full:
http://cr.openjdk.java.net/~rkennke/safe-oom-during-evac/webrev.09

I would also like to draw the discussion on thread.hpp/thread.cpp
changes: it seems like we may be able to forgo those and fold the
oom_during_evac flag into the existing gc_state. I don't care very much
about evac_allowed flags, we could drop that. WDYT?

Roman