RFC: Safe OOM during evac

[Roman Kennke]

Am 19.10.2017 um 12:27 schrieb Roman Kennke:
> Hi all,
>
> I want to outline the problem that we have with OOM during evacuation, 
> and summarize what we have so far in order to handle OOM during evac 
> correctly, and describe one way that I have in mind how to do it.
>
> The problem appears when a Java threads gets into a write-barrier and 
> fails to evacuate the object because it's run out of memory (e.g. both 
> GCLAB and shared evac exhausted). In this case we still need to ensure 
> to return a singular object, even if it's in from-space, otherwise we 
> risk inconsistency (subsequent write may end up in wrong object copy). 
> However, there might still be another Java thread which succeeds to 
> evacuate that same object at the ~ same time because it still has 
> GCLAB left.
>
> Here's what we came up with so far in IRC discussions:
>
> - Throw OOME. This is the absolute minimum solution, and we should 
> probably just do that right now until we implemented a better one (and 
> we might even use it as fallback for solutions that are not 100% 
> proof). This is better IMO than to pretend we're ok and risk heap 
> inconsistencies.
>
> - Make write-barrier slow-path/runtime-calls non-leaf calls. Then we 
> could just safepoint and do a full-GC *while we're in the barrier*. 
> This would be to most correct solution. Unfortunately it means it 
> would make it very hard to optimize the write-barriers in C2, and the 
> performance impact is likely not acceptable. We may try to do a 
> prototype (again) and see how far Roland can take it though. The 
> problem here is that we need debug info at the call sites, and C2 
> maintains debug info only at certain points in the ideal graph. 
> Consequently, we can move write-barriers only to such points and not 
> as freely as we can do now.
>
> - Keep an evacuation reserve that we use only for evacuations or maybe 
> even only for write-barriers or maybe even only as fallback for 
> write-barriers that OOM'ed. This does very likely solve it for 
> 99.999.. % of the cases, but discussions on IRC have shown that it is 
> very hard to come up with a 100% safe upper bound for this reserve 
> size, that allows us to theoretically prove that OOM during evac 
> cannot ever happen. We might combine this with solution #1 though: 
> i.e. make it safe in all but the most extreme pathologic cases, and 
> throw OOME if we hit a wall. I am still not very happy with the 
> prospect to fail in extreme rare cases, possibly in production 
> environments under high pressure.
>
> - Extend the brooks pointer protocol to prevent concurrent evacs. Let 
> me outline my idea here:
>
> If a write barrier runs OOM, we need to prevent other threads from 
> successfully evacuating 'our' object. We can do so by CASing an 
> 'impossible' value into its brooks pointers: this guarantees that 
> other threads fail to successfully install a brooks ptr *OR* give us 
> the other thread's copy (which would be fine too). Problem: we need to 
> deal with that special value everywhere else, most importantly in 
> read-barriers. The best thing I could come up with so far is to use 
> $OBJECT_ADDR | 1 as blocker value, i.e. CAS-set the lowest bit in the 
> self-pointing brooks ptr. This can easily be decoded in read-barriers 
> (and all other relevant code) by masking out that lowest bit using AND 
> ~1. Full-GC would fix the brooks ptr to normal value. I don't have a 
> good feeling what the performance impact would be. Something similar 
> happens for decoding compressed oops, and that is commonly accepted 
> (but is less frequent). The actual brooks-ptr-load probably dominates 
> the masking and we wouldn't even really notice? On the upside, this 
> makes the oom_during_evac() path truly non-blocking: we don't need to 
> wait for GC workers and not for other Java threads and not for 
> evacuation to be turned off or any such thing. (which also means, it 
> truly complies with being non-blocking for leaf-calls). I believe it's 
> a correct solution too: no from-space copy can slip through it. I can 
> imagine to come up with a prototype for this and make it optional (by 
> a flag) so that we can measure its impact or even give the option to 
> combine it with any of the other options we have (e.g. evac-reserve).
>
>
So, I made a prototype for this and SPECjvm with and without it.

First with a clean checkout build:

https://paste.fedoraproject.org/paste/8W8tKz5WGlvaT5iR5cUbFA

And this with additional masking in the read barrier:

https://paste.fedoraproject.org/paste/WT4TRm25gAbdsYSZfJqt4g

First some things to notice:

- compiler regularily crashes with an NPE. -ShenandoahOptimizeFinals 
plus Roland's recent patch for this seems to make it go away. I ran all 
benchmarks with that patch and flag applied.
- serial's performance pattern is totally erratic with huge variance 
between 3K and 12K. We can disregard this number and need to look into it
- XML crashes hard inside a C2 compiled method

other than that, I see no significant impact of the masking read barrier.

We also might want to run some memory-reading gcbench tests. In case 
anybody wants to try that, here is the patch:

http://cr.openjdk.java.net/~rkennke/safe-oom-during-evac/webrev.00/ 
<http://cr.openjdk.java.net/%7Erkennke/safe-oom-during-evac/webrev.00/>

If nobody screams stop, I'm going to add some test machinery 
(+ShenandoahOOMDuringEvacALot) and additional testcases (probably hook 
up to gcold and gcbasher), and then RFR/RFC the patch.

Thoughts?

Roman