Tuning ShenandoahGC with 420 GB heaps

[Aleksey Shipilev]

On 3/5/19 2:17 PM, Jeroen Borgers wrote:
> Actual pacing delays histogram:
> 
>       From -         To         Count         Sum
>       1 ms -       2 ms:      1391036      695518 ms
>       2 ms -       4 ms:      1005742     1005742 ms
>       4 ms -       8 ms:       821336     1642672 ms
>       8 ms -      16 ms:       702322     2809288 ms
>      16 ms -      32 ms:       617988     4943904 ms
>      32 ms -      64 ms:     11181201   178899216 ms
>      64 ms -     128 ms:         3687      117984 ms
>     128 ms -     256 ms:           12         768 ms
>     256 ms -     512 ms:            0           0 ms
>     512 ms -    1024 ms:            0           0 ms
>    1024 ms -    2048 ms:           24       12288 ms
>    2048 ms -    4096 ms:           18       18432 ms
>    4096 ms -    8192 ms:          195      399360 ms
>    8192 ms -   16384 ms:          478     1957888 ms
>                   Total:     15724039   192503060 ms
> 
> There are many > 64 ms., while we specify 50 ms as max? I don't understand. And the total time
> individual threads are stopped is huge: 192_503 s. Much more than the STW time:
> 
> Total Pauses (G)            =  4133.93 s (a =    95487 us) (n = 43293) (lvls, us =      525,   
>  2969,    10938,    18750, 34999971)
> 
> If I multiply the latter by 48 (assuming every hardware thread is utilized by 1 app thread; all are
> affected during STW) I get: 198_384 s. Similar to the total pacing time. 
> 
> Does this mean that our app is stopped by pacing in the same order as by STW pauses?

Actual pacing delay is calculated as the difference between thread entering pacing path and exiting
from it. So, if Java thread got to be paced, and then STW pause hits while Java thread is there, the
length of that STW pause would be counted into pacing delay. Which is corroborated by your data?

I'd say the first mode 1..128 ms is the pacing mostly without STW stops, and the other mode
1024..16384 ms is Degen/Full STWs that you experience.

-Aleksey