RFR: Implement protocol for safe OOM during evacuation handling

[Roman Kennke]

Am 02.03.2018 um 13:16 schrieb Roman Kennke:
> This implements a protocol for safe OOM during evacuation handling.
> 
> It's been explained and discussed here:
> http://mail.openjdk.java.net/pipermail/shenandoah-dev/2018-February/005185.html
> 
> .. and in code.
> 
> Passes hotspot_gc_shenandoah and manual testing with
> specjvm/-XX:+ShenandoahOOMDuringEvacALot modulo existing bug in
> full-gc with pinned-region
> 
> Ok?
> 
> http://cr.openjdk.java.net/~rkennke/safe-oom-during-evac/webrev.07/
> 
> Roman
> 

So here's a slight modification of it (based on the latest
eliminate-wb-stub patch), and maybe some more explanations too:

- any evacuation code (e.g. write-barrier, GC workers) are protected by
the ShenandoahOOMDuringEvacScope. This takes care of all the necessary
protocol, and if successful, sets a thread-local variable to indicate
that it's either safe to evac or not. The SH::evacuate_object() will
then read that TL field and either return with a simple and then-safe
RB, or do the evacuation as normal. The same TL field is currently also
used to verify that any call into evacuate_object() is scoped in
ShenandoahOOMDuringEvacScope
- The protocol has been designed to allow repeated calls into
evac_object even if OOM-during-evac is active:
  - workers may work in strides. OOMing at one object doesn't mean it's
not attempted again for the next
  - write-barriers may return, and go into another write-barrier before
reaching a safepoint

... it is ok to do that, the protocol will lead to simple+safe RB when
calling into evac_object() again.

- There are situations when we need to *leave* the scope. Most
importantly, workers (in partial and traversal) need to signal the
terminator that they are ready, which will cause them to wait other
workers to finish, and in which they will *not* be able to give up the
OOM-counter. We must leave the scope before signalling the terminator,
and we have ShenandoahOOMDuringEvacScopeLeaver for that. There are a few
other situations where we need to leave the scope to avoid nested
scoping. Leaving the scope like this is ok because of the above
mentioned design to allow repeated calls into the protocol.


http://cr.openjdk.java.net/~rkennke/safe-oom-during-evac/webrev.08/

Passes: all hotspot_gc_shenandoah fastdebug/release and specjvm
fastdebug/release

ok to push (after the eliminate wb-stub went in) ?

Roman