RFR (S): Avoid evacuation if concurrent GC was cancelled

[Roman Kennke]

Am Montag, den 05.12.2016, 19:09 +0100 schrieb Aleksey Shipilev:
> Okay! How about this then?
>  http://cr.openjdk.java.net/~shade/shenandoah/cancel-no-
> evac/webrev.02/

Hmm, you still don't check for cancelled gc after final-mark pause.
Notice how initial-evacuation can, in theory, fail and cause full-gc. 

(Infact, if that happens, there'd be no need to exit the final-mark
safepoint: we could jump right into full-gc. However, that sounds a bit
tricky: would need to ensure that shenandoahConcurrentThread doesn't
start evacuation or another full-gc after that 'embedded' full-gc.)

I like the comments though!

Not your fault, but I find the use of both heap->cancelled_gc() and
should_terminate() confusing. Not sure if it can be consolidated
somehow? Not necessarily in this patch though.

Another crazy pants idea to consider: if GC gets cancelled during
marking, we could short-cut the full-gc: instead of throwing away the
half-completed mark-bitmap, we could have full-gc pick up both the
half-completed mark bitmap *and* the current taskqueues from concurrent
marking, and complete that, and then do the full-compact with it. The
idea here is that if we fail during marking, in all likelyhood we're
*almost* done with marking and don't necessarily need to make
everything again. Downside would be that the mark bitmap is slightly
pessimistic because of SATB.

Roman