Workers count boosting after Mark Start

[Per Liden]

Hi,

On 10/16/19 3:49 PM, Pierre Mevel wrote:
> Good morning,
> 
> Apologies to bother you once again. As i explained in an earlier email (
> https://mail.openjdk.java.net/pipermail/zgc-dev/2019-September/000736.html),
> I'm seeing a lot of Allocation Stalls using ZGC, the issue being that my
> application's allocation rate is quite high.
> 
> I have GC cycles running back to back, whatever the value of ConcGcThreads.
> (The amount of application threads being superior to the amount of vCPUs, I
> even tried things like 40 ConcGcThreads on a 20 vCPUs machine, but at some
> point I don't want context switching to be too big of a factor).
> 
> However I never see, in debug mode, the workers count boosting log lines.
> Looking at the code (correct me if I'm wrong), I'm seeing that only at the
> beginning of a cycle can the workers count be boosted to
> Max(ParallelGcThreads, ConcGcThreads), as it happens in the do_operation
> method of the VM_ZMarkStart class.
> 
> For the allocation threads to be stalled at Mark Start, it should mean that
> Heap Memory is full at the start of the cycle. Since the cycles start
> preemptively with timeUntilGC, etc..., it only happens if the cycles run
> back to back, and heap memory is full at the start of the cycle, meaning
> that allocation rate was higher than the "freeing rate" during concurrent
> relocation.

That's correct. We only boost the number of worker threads if we've in 
deep trouble. If Java threads are stalled at mark start, we can just as 
well give more CPU to the GC to resolve the situation faster. But we 
also want to be careful to avoid prematurely boosting workers, thereby 
stealing CPU time from the application.

> 
> In my case, allocation stalls appear during concurrent mark.
> 
> Would it be possible, in theory, for the `should_boost_worker_threads`
> method from zDriver to be called within `concurrent_mark_continue`, change
> the amount of workers and continue the ZMarkTask with more workers?

Changing the number of worker threads in places other than mark start is 
technically possible, but I'm not sure it would help a lot. 
concurrent_mark_continue is kind of special, and isn't called for most 
workloads. It's basically only there to deal with the case where the 
application is continuously resurrecting soft/weak references and the 1 
ms marking done in mark end wasn't able to reach the end of the graph.

If you have back-to-back GCs and increasing ConcGCThreads doesn't help, 
then your only real option is to increase the max heap size.

cheers,
Per

> 
> Best regards and thanks in advance for your answer,
> 
> Pierre Mével
> pierre.mevel at activeviam.com
> ActiveViam
> 46 rue de l'arbre sec, 75001 Paris
>