<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">I ran the test multiple times - each had failures. As soon as a failure (SocketException) was reported I ran the netstat.<div class=""><br class=""></div><div class="">The reports of ‘memory delayed’ seems to align with the number of reported failures - which would mean that the OS is returning ENOBUFS as a temporary condition.<br class=""><div class=""><br class=""></div><div class="">Here are the results:</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">iMac:helidon_test robertengels$ netstat -nm</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">9138/15954 mbufs in use:</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>8231 mbufs allocated to data</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>1 mbufs allocated to ancillary data</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>665 mbufs allocated to packet headers</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>1 mbufs allocated to socket names and addresses</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>240 mbufs allocated to packet tags</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>6816 mbufs allocated to caches</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">941/2730 mbuf 2KB clusters in use</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">3/50678 mbuf 4KB clusters in use</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">4100/10922 mbuf 16KB clusters in use</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">398566 KB allocated to network (18.2% in use)</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">0 KB returned to the system</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">0 requests for memory denied</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">10163 requests for memory delayed</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">131 calls to drain routines</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">iMac:helidon_test robertengels$ netstat -nm</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">29711/29910 mbufs in use:</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>27064 mbufs allocated to data</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>2407 mbufs allocated to packet headers</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>240 mbufs allocated to packet tags</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>199 mbufs allocated to caches</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">3136/3238 mbuf 2KB clusters in use</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">3121/51132 mbuf 4KB clusters in use</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">10728/10922 mbuf 16KB clusters in use</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">398566 KB allocated to network (50.3% in use)</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">0 KB returned to the system</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">0 requests for memory denied</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">10342 requests for memory delayed</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">136 calls to drain routines</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">iMac:helidon_test robertengels$ netstat -nm</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">61785/76680 mbufs in use:</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>54018 mbufs allocated to data</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>7527 mbufs allocated to packet headers</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>240 mbufs allocated to packet tags</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>14895 mbufs allocated to caches</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">7991/10098 mbuf 2KB clusters in use</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">20468/44823 mbuf 4KB clusters in use</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">10909/10922 mbuf 16KB clusters in use</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">398566 KB allocated to network (72.9% in use)</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">0 KB returned to the system</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">0 requests for memory denied</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">10407 requests for memory delayed</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">136 calls to drain routines</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">iMac:helidon_test robertengels$ netstat -nm</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">18438/23213 mbufs in use:</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>14924 mbufs allocated to data</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>3274 mbufs allocated to packet headers</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>240 mbufs allocated to packet tags</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>4775 mbufs allocated to caches</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">4168/4346 mbuf 2KB clusters in use</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">1245/49996 mbuf 4KB clusters in use</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">6159/10922 mbuf 16KB clusters in use</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">398566 KB allocated to network (29.9% in use)</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">0 KB returned to the system</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">0 requests for memory denied</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">10668 requests for memory delayed</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">141 calls to drain routines</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">iMac:helidon_test robertengels$ netstat -nm</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">19765/22908 mbufs in use:</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>15860 mbufs allocated to data</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>3665 mbufs allocated to packet headers</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>240 mbufs allocated to packet tags</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>3143 mbufs allocated to caches</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">3975/4130 mbuf 2KB clusters in use</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">10/50751 mbuf 4KB clusters in use</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">7783/10922 mbuf 16KB clusters in use</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">398566 KB allocated to network (35.2% in use)</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">0 KB returned to the system</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">0 requests for memory denied</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">11091 requests for memory delayed</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">146 calls to drain routines</span></div><div style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">iMac:helidon_test robertengels$ </span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class="">I then killed the server process and ran netstat again, and it reported:</span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">iMac:~ robertengels$ netstat -nm</span></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">596/11888 mbufs in use:</span></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>348 mbufs allocated to data</span></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>8 mbufs allocated to packet headers</span></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>240 mbufs allocated to packet tags</span></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>11292 mbufs allocated to caches</span></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">280/2730 mbuf 2KB clusters in use</span></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">0/51669 mbuf 4KB clusters in use</span></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">0/10922 mbuf 16KB clusters in use</span></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">398566 KB allocated to network (0.9% in use)</span></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">0 KB returned to the system</span></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">0 requests for memory denied</span></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">11091 requests for memory delayed</span></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">146 calls to drain routines</span></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Monaco; color: rgb(16, 16, 16); background-color: rgba(255, 255, 255, 0.85);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">iMac:~ robertengels$ </span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class="">It could be that the server process is leaking sockets - but I don’t think so. I am going to perform some heap analysis to make sure.</span></div></span></div><div><br class=""><blockquote type="cite" class=""><div class="">On Nov 7, 2023, at 10:59 AM, Robert Engels <<a href="mailto:rengels@ix.netcom.com" class="">rengels@ix.netcom.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Will do. It does seem temporary. For instance with h2load testing, with 100k requests it will fail 10k of them - but this may be overstated by 10x since I have it set to pipeline 10 requests per connection. <br class=""><br class="">If I let the system sit for a bit - probably cleaning up the TIMEDWAIT connections - it will often run without failure. But rapid invocations of the test lead to the cited exception. <br class=""><br class=""><blockquote type="cite" class="">On Nov 7, 2023, at 10:33 AM, Alan Bateman <<a href="mailto:Alan.Bateman@oracle.com" class="">Alan.Bateman@oracle.com</a>> wrote:<br class=""><br class="">On 07/11/2023 13:28, Robert Engels wrote:<br class=""><blockquote type="cite" class="">I suspect that what is happening is that the JVM gets the epoll notification that the socket is writable but by the time the writer runs the network buffers have been exhausted.<br class=""><br class="">This should be an expected condition and seems like a JDK bug introduced with VT support. I checked earlier JDK releases (jdk15) and the sync -> non blocking code was not present in SocketOutputStream.java<br class=""></blockquote>I don't think this is specific to virtual threads as we're seeing it elsewhere too. If there is no space in the socket write buffer then EAGAIN/EWOULDBLOCK is handled. The intermittent ENOBUFS, which we're only seen on macos-aarch64 when under load, is hard to explain. We've had also some intermittent ENOMEM when joining multicast groups, also not specific to virtual threads.<br class=""><br class="">Your mail said you can duplicate this readily. It would be useful if could capture the output of `netstat -nm` at around the time that this happens so see if this is a real resource exhaustion issue or not.<br class=""><br class="">-Alan<br class=""><br class=""><br class=""></blockquote></div></div></blockquote></div><br class=""></div></div></body></html>