RFR: Fix double-marking

[Roman Kennke]

Aleksey observed last week that Shenandoah has a tendency to get into
an equilibrium where two GC cycles would go back-to-back. It goes like
this:

GC cycle #1 marks through heap and finds X garbage and evacuates that,
but cannot reclaim it right away, because it can only be reclaimed when
all references have been update, which happens during next cycle. This
means that memory is still low after evacuation, and hence start
another cycle right away. That cycle updates all refs, and then
reclaims memory from cycle #1, but since it's so close to the previous
cycle, it doesn't find that much garbage, and leaves a larger gap until
the next cycle. This would usually start out with a small oscillation
but tends to amplify itself after some cycles.

Another way to put it is that GC cycle #2 is degenerated and basically
only serves as an update-refs and reclamation cycle for #1.

The root cause is the dependency between cycles that lies in the
floating 'dead' garbage that the previous cycle generated.

This patch fixes the problem by accounting for that floating garbage
when starting the next cycle: when a lot of memory is about to get
reclaimed, we can start the cycle later, if only little memory is
reclaimed, we need to start earlier. This counteracts the dynamics that
leads to the observed equilibrium and now we get nice evenly spaced out
GC cycles. This also leads to improved performance:

On moderate GC load, we I measured a baseline of:

[728,745s][info][gc,stats] Concurrent Marking          =    84,93 s (a
=   719732 us) (n =   118) (lvls, us
=   427734,   644531,   712891,   787109,  1350562)

With the patch, this goes down to:

[724,359s][info][gc,stats] Concurrent Marking          =    67,38 s (a
=   701885 us) (n =    96) (lvls, us
=   394531,   625000,   681641,   742188,  1311134)

With adaptive heuristics it looks even (slightly) better:

[722,760s][info ][gc,stats] Concurrent Marking          =    64,67 s (a
=   726670 us) (n =    89) (lvls, us
=   380859,   662109,   718750,   783203,  1056381)

Notice that with adaptive heuristics, we'd get occasional full-GCs
before that patch, which doesn't happen anymore too. Under heavy load,
the numbers look like this:

Baseline:
[868,046s][info][gc,stats] Concurrent Marking          =   333,44 s (a
=  1096839 us) (n =   304) (lvls, us
=   349609,  1015625,  1093750,  1171875,  1382350)
Dynamic-patched:
[834,421s][info][gc,stats] Concurrent Marking          =   273,95 s (a
=  1070101 us) (n =   256) (lvls, us
=   541016,  1015625,  1074219,  1113281,  1465707)
Adaptive-patched:
[825,412s][info ][gc,stats] Concurrent Marking          =   257,65 s (a
=  1073544 us) (n =   240) (lvls, us
=   365234,  1015625,  1074219,  1132812,  1398973)

Again, no more full-GCs with adaptive.

http://cr.openjdk.java.net/~rkennke/fixdoublemarks/webrev.00/

Ok to push?

Roman