<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><meta http-equiv="content-type" content="text/html; charset=utf-8" class=""><div dir="auto" class="">Hi. <div dir="ltr" class=""></div><div class=""><br class=""></div><div class="">I believe I have discovered what is essentially a priority inversion problem in the Loom network stack.</div><div class=""><br class=""></div><div class="">I have been comparing a NIO based http server framework with one using VT.</div><div class=""><br class=""></div><div class="">The VT framework when using the Loom defaults consistently uses significantly more CPU and achieves less throughput and overall performance.</div><div class=""><br class=""></div><div class="">By lowering the parallelism, the VT framework exceeds the NIO framework in throughput and overall performance.</div><div class=""><br class=""></div><div class="">My research and hypothesis is that if you create as many carrier threads as CPUs, then they compete withe the network poller threads. By reducing the number of carrier threads, the poller can run. If the poller can’t run, then many/all of the carrier threads will eventually park waiting for a runnable task, but there won’t be one until the poller runs - which is why the poller must have priority over the carrier threads.</div><div class=""><br class=""></div><div class=""><b class="">I believe the best solution would be to lower the native priority of carrier threads as explicitly configuring the number of carrier threads accurately will be nearly impossible for varied workloads. I suspect this would also help GC be more deterministic as well.</b></div><div class=""><br class=""></div><div class="">Using default parallelism:</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">robertengels@macmini go-wrk % ./go-wrk -c 1000 -d 30 -T 10000 <a href="http://imac:8080/plaintext" class="">http://imac:8080/plaintext</a></span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Running 30s test @ <a href="http://imac:8080/plaintext" class="">http://imac:8080/plaintext</a></span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 1000 goroutine(s) running concurrently</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">3557027 requests in 29.525311463s, 424.03MB read</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Requests/sec:<span class="Apple-tab-span" style="white-space:pre"> </span>120473.82</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Transfer/sec:<span class="Apple-tab-span" style="white-space:pre"> </span>14.36MB</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Overall Requests/sec:<span class="Apple-tab-span" style="white-space:pre"> </span>116021.08</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Overall Transfer/sec:<span class="Apple-tab-span" style="white-space:pre"> </span>13.83MB</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Fastest Request:<span class="Apple-tab-span" style="white-space:pre"> </span>112µs</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Avg Req Time:<span class="Apple-tab-span" style="white-space:pre"> </span>8.3ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Slowest Request:<span class="Apple-tab-span" style="white-space:pre"> </span>839.967ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Number of Errors:<span class="Apple-tab-span" style="white-space:pre"> </span>2465</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Error Counts:<span class="Apple-tab-span" style="white-space:pre"> </span>broken pipe=2,connection reset by peer=2451,net/http: timeout awaiting response headers=12</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">10%:<span class="Apple-tab-span" style="white-space:pre"> </span>2.102ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">50%:<span class="Apple-tab-span" style="white-space:pre"> </span>2.958ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">75%:<span class="Apple-tab-span" style="white-space:pre"> </span>3.108ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">99%:<span class="Apple-tab-span" style="white-space:pre"> </span>3.198ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">99.9%:<span class="Apple-tab-span" style="white-space:pre"> </span>3.201ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">99.9999%:<span class="Apple-tab-span" style="white-space:pre"> </span>3.201ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">99.99999%:<span class="Apple-tab-span" style="white-space:pre"> </span>3.201ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">stddev:<span class="Apple-tab-span" style="white-space:pre"> </span>32.52ms</span></div></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span></div><div class="">and using reduced parallelism of 3:</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">robertengels@macmini go-wrk % ./go-wrk -c 1000 -d 30 -T 10000 <a href="http://imac:8080/plaintext" class="">http://imac:8080/plaintext</a></span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Running 30s test @ <a href="http://imac:8080/plaintext" class="">http://imac:8080/plaintext</a></span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 1000 goroutine(s) running concurrently</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">4059418 requests in 29.092649689s, 483.92MB read</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Requests/sec:<span class="Apple-tab-span" style="white-space:pre"> </span>139534.14</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Transfer/sec:<span class="Apple-tab-span" style="white-space:pre"> </span>16.63MB</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Overall Requests/sec:<span class="Apple-tab-span" style="white-space:pre"> </span>132608.44</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Overall Transfer/sec:<span class="Apple-tab-span" style="white-space:pre"> </span>15.81MB</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Fastest Request:<span class="Apple-tab-span" style="white-space:pre"> </span>115µs</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Avg Req Time:<span class="Apple-tab-span" style="white-space:pre"> </span>7.166ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Slowest Request:<span class="Apple-tab-span" style="white-space:pre"> </span>811.999ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Number of Errors:<span class="Apple-tab-span" style="white-space:pre"> </span>2361</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Error Counts:<span class="Apple-tab-span" style="white-space:pre"> </span>net/http: timeout awaiting response headers=51,connection reset by peer=2310</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">10%:<span class="Apple-tab-span" style="white-space:pre"> </span>1.899ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">50%:<span class="Apple-tab-span" style="white-space:pre"> </span>2.383ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">75%:<span class="Apple-tab-span" style="white-space:pre"> </span>2.478ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">99%:<span class="Apple-tab-span" style="white-space:pre"> </span>2.541ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">99.9%:<span class="Apple-tab-span" style="white-space:pre"> </span>2.543ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">99.9999%:<span class="Apple-tab-span" style="white-space:pre"> </span>2.544ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">99.99999%:<span class="Apple-tab-span" style="white-space:pre"> </span>2.544ms</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); font-size: 10px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">stddev:<span class="Apple-tab-span" style="white-space:pre"> </span>32.88ms</span></div></div><div class=""><br class=""></div><div class="">More importantly, the reduced parallelism has a cpu idle percentage of 30% (which matches the NIO framework) whereas the default parallelism has an idle of near 0 % (due to scheduler thrashing).</div><div class=""><br class=""></div><div class="">The attached JFR screenshot (I have also attached the JFR captures) tells the story. #2 is the VT with default parallelism. #3 is the NIO based framework, and #4 is VT with reduced parallelism. #2 clearly shows the thrashing that is occurring with threads parking and unparking and the scheduler waiting for work.</div><div class=""><br class=""></div><div class=""></div></div></div></div></div></body></html>