RFR (S): Avoid evacuation if concurrent GC was cancelled

[Roman Kennke]

Some comments:

- GC can be cancelled during final-mark-pause. Might be worth to keep
the check for cancelled-gc after init-mark-pause. Same after
evacuation: if evacuation gets cancelled, we don't need to reset the
bitmaps because now it's done at start of full-gc. I think.

- This here looks wrong:

+  // b. Cancel evacuation, if in progress
+  if (_heap->is_evacuation_in_progress()) {
+    MutexLocker mu(Threads_lock);
+    _heap->set_evacuation_in_progress(false);
+  }

This happens during safepoint. The VMThread would hold the Threads_lock
and the above would deadlock.

We need to acquire the Threads_lock only when turning off evacuation
outside of a safepoint.

Roman

Am Montag, den 05.12.2016, 17:00 +0100 schrieb Aleksey Shipilev:
> Hi,
> 
> Currently, when concurrent GC is canceled, we still enter the VM
> operation for
> concurrent evacuation, only to exit it quickly and slide into the
> full GC. This
> causes *two* back-to-back safepoints: one short from evac, and
> another large for
> full GC. While short one is normally short, it can hit the unlucky
> scheduling
> outlier and drag the pause time up.
> 
> This change avoids going to evac if conc GC was canceled:
>   http://cr.openjdk.java.net/~shade/shenandoah/cancel-no-evac/webrev.
> 01/
> 
> Additionally, it resets the mark bitmaps before full GC with parallel
> workers,
> not concurrent ones, which would be important once Zhengyu trims down
> the number
> of concurrent workers.
> 
> Testing: hotspot_gc_shenandoah, jcstress (tests-all/quick)
> 
> Thanks,
> -Aleksey
>