RFR: 8253971: ZGC: Flush mark stacks after processing concurrent roots

[Per Liden]

On Sat, 3 Oct 2020 14:56:13 GMT, Albert Mingkun Yang <ayang at openjdk.org> wrote:

>> I looked into why almost all mark cycles see non-zero "mark completions". In other words, we almost always have some
>> amount of mark work left to handle in the mark end pause. It turns out that worker threads don't flush their mark
>> stacks in ZMarkConcurrentRootsTask::work(), which means they can hide work (in their thread local mark stacks) until
>> those stacks are finally flushed out in ZMark::try_end(). The reason work can be hidden is that the set of worker
>> threads executing ZMarkConcurrentRootsTask is not necessarily the same set of worker threads executing ZMarkTask. As a
>> result, the mark end pause often becomes longer than it otherwise would have.  After fixing this, I did some tests with
>> Dacapo, which shows the following improvement:  Before: Mark End Pause (avg/max): 0.391 / 1.142 ms After: Mark End
>> Pause (avg/max): 0.130 / 0.294 ms
>
> I wonder if it makes sense to unify this PR with the same logic in `ZMarkRootsTask` to a function, sth like
> `PostRootsScan`. Additionally, such flushing is only needed if `ParallelGCThreads != ConcGCThreads`, right?

@albertnetymk Yes, the problem stems from ZWorkers::nconcurrent() potentially being less than ZWorkers::nworkers().

-------------

PR: https://git.openjdk.java.net/jdk/pull/495