Troubles with Shenandoah

[Simone Bordet]

Hi,

On Mon, Apr 8, 2019 at 10:01 PM Aleksey Shipilev <shade at redhat.com> wrote:
> You're not likely to get the max latency improvements either, as generational STW GCs would be able
> to hit the similar 1..3ms pauses consistently.

Well, G1 is in the 15-25 ms range, while ZGC is < 2 ms.

> Sure, appreciated. Looking forward for results with always-fully-committed memory!
>
> I have just tried with the latest jdk12 release binary with -Xmx10g -Xms10g -XX:+AlwaysPreTouch, and
> there are no apparent troubles on server side:
>
> Average CPU Load: 672.1718/1600
> ========================================
>                    @  _  502 µs (8410972, 99.96%) ^50% ^85% ^95% ^99% ^99.9%
> @                     _  1,004 µs (2223, 0.03%)
> @                     _  1,506 µs (504, 0.01%)
> @                     _  2,008 µs (160, 0.00%)
> @                     _  2,510 µs (93, 0.00%)
> @                     _  3,012 µs (71, 0.00%)
> @                     _  3,514 µs (51, 0.00%)
> @                     _  4,016 µs (52, 0.00%)
> @                     _  4,518 µs (27, 0.00%)
> @                     _  5,020 µs (23, 0.00%)
> @                     _  5,522 µs (11, 0.00%)
> @                     _  6,024 µs (9, 0.00%)
> @                     _  6,526 µs (11, 0.00%)
> @                     _  7,028 µs (1, 0.00%)
> @                     _  7,530 µs (1, 0.00%)
> @                     _  8,033 µs (0, 0.00%)
> @                     _  8,535 µs (0, 0.00%)
> @                     _  9,037 µs (4, 0.00%)
> @                     _  9,539 µs (1, 0.00%)
> @                     _  10,041 µs (2, 0.00%)
> Requests - Latency: 8414216 samples | min/avg/50th%/99th%/max = 2/8/6/23/10,043 µs
> Requests times (total/avg/max - stddev): 77669/0/10 ms - 0
> Requests (total/failed/max - rate): 8414218/0/39 - 81772 requests/s
> ========================================
>                    @  _  2,276 µs (4842264, 48.40%)
>            @          _  4,553 µs (2615535, 26.14%) ^50%
>        @              _  6,830 µs (1591673, 15.91%) ^85%
>    @                  _  9,106 µs (725789, 7.25%) ^95%
>  @                    _  11,383 µs (159531, 1.59%) ^99%
> @                     _  13,660 µs (49446, 0.49%)
> @                     _  15,936 µs (12426, 0.12%) ^99.9%
> @                     _  18,213 µs (3629, 0.04%)
> @                     _  20,490 µs (1939, 0.02%)
> @                     _  22,767 µs (985, 0.01%)
> @                     _  25,043 µs (665, 0.01%)
> @                     _  27,320 µs (493, 0.00%)
> @                     _  29,597 µs (251, 0.00%)
> @                     _  31,873 µs (78, 0.00%)
> @                     _  34,150 µs (39, 0.00%)
> @                     _  36,427 µs (10, 0.00%)
> @                     _  38,704 µs (44, 0.00%)
> @                     _  40,980 µs (4, 0.00%)
> @                     _  43,257 µs (0, 0.00%)
> @                     _  45,534 µs (1, 0.00%)
> Messages - Processing: 10004802 samples | min/avg/50th%/99th%/max = 13/3,072/2,383/10,698/45,547 µs
> ========================================
> Jetty Thread Pool - Tasks = 4640206 | Concurrent Threads max = 113 | Queue Size max = 684 | Queue
> Latency avg/max = 1/23 ms | Task Latency avg/max = 0/1097 ms

Did you run with the default benchmark configuration?
If so, can you try to increase the "batch count" to 10_000 or more?
Shenandoah was ok for me for short runs, but not for longer runs.

> ...but I don't know how to properly interpret the results.

The first graph is the request _dispatch_ time - but because requests
are asynchronous the major part of the work is done outside the
request dispatching (that's why the times are smaller than the other
graph).
The second graph is the actual application (CometD) processing time.
We have another option where the requests are not asynchronous so the
first graph would always show Jetty processing time + CometD
processing time, so we can derive the Jetty overhead.

> The whole exercise was the source of interesting bugs, we already fixed a few, and other fixes are
> in pipeline.

So what exact version can I use to run this benchmark with Shenandoah?
The latest 13? Should I wait for the fixes in the pipeline?
I can build Shenandoah from the sources.

Thanks!

-- 
Simone Bordet
---
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless.   Victoria Livschitz