Troubles with Shenandoah

[Aleksey Shipilev]

On 4/8/19 9:46 PM, Simone Bordet wrote:
> Holy cow!

Yes, and also consider there are lots of threads that already bash kernel with scheduling requests.
On my machine, there is ~18% sys time and 500K ctxsw/sec. Kernel probably has not time to deal with
committing stuff on top of that storm.

>> Yes. Just don't be surprised when concurrent collectors do not give the latency boost there :)
> 
> I'm not expecting a median latency boost here.
> I actually had no expectations and was just curious as to what could
> have happened when running the CometD benchmark with ZGC and
> Shenandoah.

You're not likely to get the max latency improvements either, as generational STW GCs would be able
to hit the similar 1..3ms pauses consistently.

> But we have people interested in reducing max latency as much as
> possible, even sacrificing some throughput and median latency, so ZGC
> and Shenandoah are worth exploring.
> We (as the Jetty team) take the hit of trying these new GCs early so
> at least we have some knowledge when people ask us about them :)

Sure, appreciated. Looking forward for results with always-fully-committed memory!

I have just tried with the latest jdk12 release binary with -Xmx10g -Xms10g -XX:+AlwaysPreTouch, and
there are no apparent troubles on server side:

Average CPU Load: 672.1718/1600
========================================
                   @  _  502 µs (8410972, 99.96%) ^50% ^85% ^95% ^99% ^99.9%
@                     _  1,004 µs (2223, 0.03%)
@                     _  1,506 µs (504, 0.01%)
@                     _  2,008 µs (160, 0.00%)
@                     _  2,510 µs (93, 0.00%)
@                     _  3,012 µs (71, 0.00%)
@                     _  3,514 µs (51, 0.00%)
@                     _  4,016 µs (52, 0.00%)
@                     _  4,518 µs (27, 0.00%)
@                     _  5,020 µs (23, 0.00%)
@                     _  5,522 µs (11, 0.00%)
@                     _  6,024 µs (9, 0.00%)
@                     _  6,526 µs (11, 0.00%)
@                     _  7,028 µs (1, 0.00%)
@                     _  7,530 µs (1, 0.00%)
@                     _  8,033 µs (0, 0.00%)
@                     _  8,535 µs (0, 0.00%)
@                     _  9,037 µs (4, 0.00%)
@                     _  9,539 µs (1, 0.00%)
@                     _  10,041 µs (2, 0.00%)
Requests - Latency: 8414216 samples | min/avg/50th%/99th%/max = 2/8/6/23/10,043 µs
Requests times (total/avg/max - stddev): 77669/0/10 ms - 0
Requests (total/failed/max - rate): 8414218/0/39 - 81772 requests/s
========================================
                   @  _  2,276 µs (4842264, 48.40%)
           @          _  4,553 µs (2615535, 26.14%) ^50%
       @              _  6,830 µs (1591673, 15.91%) ^85%
   @                  _  9,106 µs (725789, 7.25%) ^95%
 @                    _  11,383 µs (159531, 1.59%) ^99%
@                     _  13,660 µs (49446, 0.49%)
@                     _  15,936 µs (12426, 0.12%) ^99.9%
@                     _  18,213 µs (3629, 0.04%)
@                     _  20,490 µs (1939, 0.02%)
@                     _  22,767 µs (985, 0.01%)
@                     _  25,043 µs (665, 0.01%)
@                     _  27,320 µs (493, 0.00%)
@                     _  29,597 µs (251, 0.00%)
@                     _  31,873 µs (78, 0.00%)
@                     _  34,150 µs (39, 0.00%)
@                     _  36,427 µs (10, 0.00%)
@                     _  38,704 µs (44, 0.00%)
@                     _  40,980 µs (4, 0.00%)
@                     _  43,257 µs (0, 0.00%)
@                     _  45,534 µs (1, 0.00%)
Messages - Processing: 10004802 samples | min/avg/50th%/99th%/max = 13/3,072/2,383/10,698/45,547 µs
========================================
Jetty Thread Pool - Tasks = 4640206 | Concurrent Threads max = 113 | Queue Size max = 684 | Queue
Latency avg/max = 1/23 ms | Task Latency avg/max = 0/1097 ms

...but I don't know how to properly interpret the results.

The whole exercise was the source of interesting bugs, we already fixed a few, and other fixes are
in pipeline.

Cheers,
-Aleksey