RFR: 8253971: ZGC: Flush mark stacks after processing concurrent roots

[Per Liden]

On Mon, 5 Oct 2020 05:55:18 GMT, Erik Österlund <eosterlund at openjdk.org> wrote:

>> I looked into why almost all mark cycles see non-zero "mark completions". In other words, we almost always have some
>> amount of mark work left to handle in the mark end pause. It turns out that worker threads don't flush their mark
>> stacks in ZMarkConcurrentRootsTask::work(), which means they can hide work (in their thread local mark stacks) until
>> those stacks are finally flushed out in ZMark::try_end(). The reason work can be hidden is that the set of worker
>> threads executing ZMarkConcurrentRootsTask is not necessarily the same set of worker threads executing ZMarkTask. As a
>> result, the mark end pause often becomes longer than it otherwise would have.  After fixing this, I did some tests with
>> Dacapo, which shows the following improvement:  Before: Mark End Pause (avg/max): 0.391 / 1.142 ms After: Mark End
>> Pause (avg/max): 0.130 / 0.294 ms
>
> Looks good. We might also want to poke out mark stacks on the VM thread when we handshake threads, using a concurrent
> VM operation. But maybe that is a problem for another day.

> I am curious if it's worth the trouble to make the flush conditional.

I don't think we need to special-case the flushing. This condition is almost always true, and flushing out local mark
stacks is inexpensive. I also think it's a somewhat nice/simplifying invariant, that all stacks are published on the
stripes when worker threads start working.

-------------

PR: https://git.openjdk.java.net/jdk/pull/495