<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Also, for comparison purposes, using platform threads over virtual threads with default parallelism shows much better avg/tail latency and a slightly less throughput.<div class=""><br class=""></div><div class="">Using platform thread per connection:</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">robertengels@macmini go-wrk % wrk -H 'Host: imac' -H 'Accept: text/plain,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 20 -c 1000 --timeout 8 -t 2 <a href="http://imac:8080/plaintext" class="">http://imac:8080/plaintext</a></span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Running 20s test @ <a href="http://imac:8080/plaintext" class="">http://imac:8080/plaintext</a></span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 2 threads and 1000 connections</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Thread Stats Avg Stdev Max +/- Stdev</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Latency 28.09ms 70.26ms 818.88ms 92.55%</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Req/Sec 53.96k 4.02k 63.58k 71.25%</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Latency Distribution</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 50% 8.51ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 75% 9.44ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 90% 41.65ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 99% 378.31ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 2148390 requests in 20.03s, 290.94MB read</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Requests/sec: 107280.14</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Transfer/sec: 14.53MB</span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span></div><div>Using virtual threads with default parallelism:</div><div><br class=""></div><div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">robertengels@macmini go-wrk % wrk -H 'Host: imac' -H 'Accept: text/plain,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 20 -c 1000 --timeout 8 -t 2 <a href="http://imac:8080/plaintext" class="">http://imac:8080/plaintext</a></span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Running 20s test @ <a href="http://imac:8080/plaintext" class="">http://imac:8080/plaintext</a></span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 2 threads and 1000 connections</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Thread Stats Avg Stdev Max +/- Stdev</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Latency 67.63ms 146.27ms 2.42s 87.57%</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Req/Sec 55.05k 4.93k 70.66k 73.50%</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Latency Distribution</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 50% 5.63ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 75% 48.40ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 90% 262.75ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 99% 674.56ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 2192116 requests in 20.03s, 296.86MB read</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Requests/sec: 109430.53</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Transfer/sec: 14.82MB</span></div></div><div><br class=""></div><div><br class=""><blockquote type="cite" class=""><div class="">On Aug 14, 2024, at 10:47 AM, Robert Engels <<a href="mailto:robaho@icloud.com" class="">robaho@icloud.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html; charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">I was able to correct the errors. The initial accepts were failing. I adjusted <i class="">kern.ipc.somaxconn </i>appropriately and it was resolved. So back to the problem. I have a new theory that maybe someone can validate.<div class=""><br class=""></div><div class="">I theorize that without reducing the parallelism all cores are in use, which means that any GC activity has no cores to run on and competes with the carrier thread - introducing latency - and since this is a ping/pong RTT type test, the delay is enough to make the processors go into a park/unpark cycle.</div><div class=""><br class=""></div><div class="">Which leads me to the question does Java need a better (or dynamic) way of determining the optimum parallelism based on load? It seems the only way to set the parallelism is at startup, and it is constant.</div><div class=""><br class=""></div><div class="">With default parallelism:</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 13px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">robertengels@macmini go-wrk % wrk -H 'Host: imac' -H 'Accept: text/plain,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 20 -c 1000 --timeout 8 -t 2 <a href="http://imac:8080/plaintext" class="">http://imac:8080/plaintext</a></span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 13px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Running 20s test @ <a href="http://imac:8080/plaintext" class="">http://imac:8080/plaintext</a></span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 13px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 2 threads and 1000 connections</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 13px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Thread Stats Avg Stdev Max +/- Stdev</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 13px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Latency 83.98ms 199.72ms 4.22s 89.05%</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 13px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Req/Sec 51.75k 5.30k 61.84k 75.75%</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 13px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Latency Distribution</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 13px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 50% 5.29ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 13px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 75% 65.42ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 13px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 90% 301.31ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 13px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 99% 788.27ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 13px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 2059873 requests in 20.03s, 278.95MB read</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 13px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Requests/sec: 102854.15</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 13px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Transfer/sec: 13.93MB</span></div></div><div class=""><br class=""></div><div class=""><div class="">and with reduced parallelism (3):</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">robertengels@macmini go-wrk % wrk -H 'Host: imac' -H 'Accept: text/plain,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 20 -c 1000 --timeout 8 -t 2 <a href="http://imac:8080/plaintext" class="">http://imac:8080/plaintext</a></span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Running 20s test @ <a href="http://imac:8080/plaintext" class="">http://imac:8080/plaintext</a></span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 2 threads and 1000 connections</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Thread Stats Avg Stdev Max +/- Stdev</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Latency 10.08ms 13.10ms 408.99ms 96.65%</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Req/Sec 56.10k 5.95k 65.77k 74.25%</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Latency Distribution</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 50% 8.55ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 75% 9.86ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 90% 13.21ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 99% 45.54ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 2232236 requests in 20.02s, 302.29MB read</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Requests/sec: 111475.38</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; font-size: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Transfer/sec: 15.10MB</span></div></div><div class=""><br class=""></div><div class="">notice the huge difference in average, tail and variance of latency. The more than order of magnitude in max latency is very concerning.</div><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Aug 14, 2024, at 8:44 AM, robert engels <<a href="mailto:robaho@icloud.com" class="">robaho@icloud.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Hi Alan. I agree and I’ll try to resolve this today. But what I am struggling with is why the parallelism is the only thing that matters. The poller mode or number of pollers make no difference. <br class=""><br class="">If I lower the parallelism I get better performance (matches the NIO - all systems have roughly the same error rate regardless of throughput). <br class=""><br class=""><blockquote type="cite" class="">On Aug 14, 2024, at 1:15 AM, Alan Bateman <<a href="mailto:Alan.Bateman@oracle.com" class="">Alan.Bateman@oracle.com</a>> wrote:<br class=""><br class=""><br class=""><br class="">On 13/08/2024 16:34, robert engels wrote:<br class=""><blockquote type="cite" class="">I did. It didn’t make any difference. I checked the thread dump as well and the extras were created. <br class=""><br class="">Surprised that lowering the priority didn’t help - so now I need to think about other options. It feels like something when the carriers can use all the cores that the poller is prevented from running - like some sort of lock being held by the carrier/vt and do it thrashes around until it eventually gets a chance. <br class=""><br class=""></blockquote>With pollerMode=2 then the the work is done on virtual threads so I assume you don't see the master poller stealing CPU cycles.<br class=""><br class="">In any case, what you describe sounds a bit like bursty usage where all FJP workers are scanning for work before parking, and/or something else that may be macOS specific when running out of some resource. I suspect tuning the networking params to remove the errors/timeouts might make it a bit easier to study.<br class=""><br class="">-Alan<br class=""><br class=""><br class=""></blockquote></div></div></blockquote></div><br class=""></div></div></div></blockquote></div><br class=""></div></body></html>