<div dir="ltr">I will rewrite everything again as it lacks some context and information.<br><br><br>Hey team<br><br>We are experiencing some weird halting issues when scheduling tasks with ScheduledExecutorService and VT.<br><br>We are observing some <b>random</b> near deadlock issues / halting of executors which we are not able to reproduce when using platform threads.<br>It happens after a random amount of time.<br><br>This is one example of a near deadlock situation, where the scheduling platform thread successfully acquires a semaphore permit, assumingly starting a virtual thread, but the debug log "Scheduling..." is not being printed at all, preventing the finally block or the catch Throwable block from releasing the acquired semaphore permit, and reaching to drained semaphore.<br><br>Few notes:<div>- We are testing on EA build
23-loom+4-102<br><div>- The scheduler is running on a platform thread<br><b>- Flight recorder does not indicate any pinned VT<br></b>- This code is running within a Docker container based on ubuntu:20.04 image<br>- Using G1 GC<br><br>Has anyone experienced this issue before? I hope we are doing something wrong 😅<br><div><br><div style="padding:0px 2px"><div style="color:rgb(0,0,0);font-family:Consolas;font-size:10pt"><p style="margin:0px"> ```ScheduledFuture<?> <span style="color:rgb(0,0,192)">deviceFutureTask</span> = <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,0,128)">scheduler</span>.scheduleAtFixedRate(() -> {</p><p style="margin:0px"> <span style="color:rgb(127,0,85);font-weight:bold">try</span> {</p><p style="margin:0px"> logger.debug(<span style="color:rgb(42,0,255)">"[{}] Trying to acquire permit to schedule task [{}] for device. Number of available permits: [{}]"</span>,</p><p style="margin:0px"> device, task, semaphore.availablePermits());</p><p style="margin:0px"> <span style="color:rgb(127,0,85);font-weight:bold">if</span> (semaphore.tryAcquire(waitingTimeout, TimeUnit.MILLISECONDS)) {</p><p style="margin:0px"> logger.debug(<span style="color:rgb(42,0,255)">"[{}] Acquired permit to schedule [{}] task for device"</span>, device, task);</p><p style="margin:0px"> Thread.ofVirtual().start(() -> {</p><p style="margin:0px"> <span style="color:rgb(127,0,85);font-weight:bold">try</span> {</p><p style="margin:0px"> logger.debug(<span style="color:rgb(42,0,255)">"[{}] Scheduling [{}] task for device"</span>, device, task);</p><p style="margin:0px"> <span style="color:rgb(63,127,95)">// some I/O intensive work</span></p><p style="margin:0px"> logger.debug(<span style="color:rgb(42,0,255)">"[{}] Finished processing [{}] task for device"</span>, device, task);</p><p style="margin:0px"> } <span style="color:rgb(127,0,85);font-weight:bold">catch</span> (Exception e) {</p><p style="margin:0px"> logger.error(<span style="color:rgb(42,0,255)">"[{}] Failed to process [{}] task for device"</span>, device, task, e);</p><p style="margin:0px"> } <span style="color:rgb(127,0,85);font-weight:bold">finally</span> {</p><p style="margin:0px"> semaphore.release();</p><p style="margin:0px"> }</p><p style="margin:0px"> });</p><p style="margin:0px"> } <span style="color:rgb(127,0,85);font-weight:bold">else</span> {</p><p style="margin:0px"> logger.error(<span style="color:rgb(42,0,255)">"Timed out while waiting for permit to schedule task [{}] for device [{}]"</span>, task, device);</p><p style="margin:0px"> }</p><p style="margin:0px"> } <span style="color:rgb(127,0,85);font-weight:bold">catch</span> (Throwable t) {</p><p style="margin:0px"> logger.error(<span style="color:rgb(42,0,255)">"Failed to execute task [{}] for device [{}]"</span>, task, deviceId, t);</p><p style="margin:0px"> semaphore.release();</p><p style="margin:0px"> }</p><p style="margin:0px"> }, (<span style="color:rgb(127,0,85);font-weight:bold">long</span>) (<span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,0,128)">entropy</span> * <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,0,128)">schedulingInterval</span>), <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,0,128)">schedulingInterval</span>, TimeUnit.<span style="color:rgb(0,0,192);font-style:italic;font-weight:bold">MILLISECONDS</span>);```</p></div></div></div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, 21 Jul 2024 at 10:16, Yuval Lombard <<a href="mailto:yuval.l@securithings.com">yuval.l@securithings.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Did not mention by I based the code on version 23-loom+4-102<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, 21 Jul 2024 at 10:05, Yuval Lombard <<a href="mailto:yuval.l@securithings.com" target="_blank">yuval.l@securithings.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hey team<br><br>We are experiencing some weird halting issues when scheduling tasks with ScheduledExecutorService and VT.<br><br>We are observing some near deadlock issues / halting of executors which we are not able to reproduce when using platform threads<br><br>This is one example of a near deadlock situation, where for some reason virtual threads are starting to execute a task, and not reaching the finally block nor the catch Throwable block to release the acquired semaphore permit, reaching to drained semaphore.<br><br>Are you familiar with this behavior? I hope we are doing something wrong 😅<br><br>Note - The scheduler is running on a platform thread<br><div><br><div style="padding:0px 2px"><div style="color:rgb(0,0,0);font-family:Consolas;font-size:10pt;white-space:pre-wrap"><p style="margin:0px"> ```ScheduledFuture<?> <span style="color:rgb(0,0,192)">deviceFutureTask</span> = <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,0,128)">scheduler</span>.scheduleAtFixedRate(() -> {</p><p style="margin:0px"> <span style="color:rgb(127,0,85);font-weight:bold">try</span> {</p><p style="margin:0px"> logger.debug(<span style="color:rgb(42,0,255)">"[{}] Trying to acquire permit to schedule task [{}] for device. Number of available permits: [{}]"</span>,</p><p style="margin:0px"> device, task, semaphore.availablePermits());</p><p style="margin:0px"> <span style="color:rgb(127,0,85);font-weight:bold">if</span> (semaphore.tryAcquire(waitingTimeout, TimeUnit.MILLISECONDS)) {</p><p style="margin:0px"> logger.debug(<span style="color:rgb(42,0,255)">"[{}] Acquired permit to schedule [{}] task for device"</span>, device, task);</p><p style="margin:0px"> Thread.ofVirtual().start(() -> {</p><p style="margin:0px"> <span style="color:rgb(127,0,85);font-weight:bold">try</span> {</p><p style="margin:0px"> logger.debug(<span style="color:rgb(42,0,255)">"[{}] Scheduling [{}] task for device"</span>, device, task);</p><p style="margin:0px"> <span style="color:rgb(63,127,95)">// some I/O intensive work</span></p><p style="margin:0px"> logger.debug(<span style="color:rgb(42,0,255)">"[{}] Finished processing [{}] task for device"</span>, device, task);</p><p style="margin:0px"> } <span style="color:rgb(127,0,85);font-weight:bold">catch</span> (Exception e) {</p><p style="margin:0px"> logger.error(<span style="color:rgb(42,0,255)">"[{}] Failed to process [{}] task for device"</span>, device, task, e);</p><p style="margin:0px"> } <span style="color:rgb(127,0,85);font-weight:bold">finally</span> {</p><p style="margin:0px"> semaphore.release();</p><p style="margin:0px"> }</p><p style="margin:0px"> });</p><p style="margin:0px"> } <span style="color:rgb(127,0,85);font-weight:bold">else</span> {</p><p style="margin:0px"> logger.error(<span style="color:rgb(42,0,255)">"Timed out while waiting for permit to schedule task [{}] for device [{}]"</span>, task, device);</p><p style="margin:0px"> }</p><p style="margin:0px"> } <span style="color:rgb(127,0,85);font-weight:bold">catch</span> (Throwable t) {</p><p style="margin:0px"> logger.error(<span style="color:rgb(42,0,255)">"Failed to execute task [{}] for device [{}]"</span>, task, deviceId, t);</p><p style="margin:0px"> semaphore.release();</p><p style="margin:0px"> }</p><p style="margin:0px"> }, (<span style="color:rgb(127,0,85);font-weight:bold">long</span>) (<span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,0,128)">entropy</span> * <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,0,128)">schedulingInterval</span>), <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,0,128)">schedulingInterval</span>, TimeUnit.<span style="color:rgb(0,0,192);font-style:italic;font-weight:bold">MILLISECONDS</span>);```</p><p style="margin:0px"><br></p></div></div><div>We are </div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div dir="ltr"><div style="font-size:12.8px"><p class="MsoNormal" style="color:rgb(34,34,34);font-size:small;background-image:initial;background-position:initial;background-repeat:initial">Kind regards,<b style="font-size:13.3333px;color:rgb(0,176,240);font-family:Ebrima"><br><br>Yuval Lombard</b><br></p><p class="MsoNormal" style="color:rgb(34,34,34);font-size:small;background-image:initial;background-position:initial;background-repeat:initial"><font color="#3b3838" face="Ebrima"><span style="font-size:12px"><b>Lead Software Engineer</b></span></font></p><p class="MsoNormal" style="color:rgb(34,34,34);font-size:small;background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:9pt;font-family:Ebrima;color:rgb(59,56,56)">+972.50.548.0111</span><span style="font-size:9pt;font-family:Ebrima;color:rgb(64,64,64)"><u></u><u></u></span></p><p class="MsoNormal" style="color:rgb(34,34,34);font-size:small;background-image:initial;background-position:initial;background-repeat:initial"><font color="#0000ff" face="Ebrima"><span style="font-size:12px"><a href="mailto:yuval.l@securithings.com" style="color:rgb(17,85,204)" target="_blank">yuval.l@securithings.com</a></span></font></p><p class="MsoNormal" style="color:rgb(34,34,34);font-size:small;background-image:initial;background-position:initial;background-repeat:initial"><img src="cid:ii_l13cyhga0" alt="logo_black.png" width="200" height="47"></p></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div dir="ltr"><div style="font-size:12.8px"><p class="MsoNormal" style="color:rgb(34,34,34);font-size:small;background-image:initial;background-position:initial;background-repeat:initial">Kind regards,<b style="font-size:13.3333px;color:rgb(0,176,240);font-family:Ebrima"><br><br>Yuval Lombard</b><br></p><p class="MsoNormal" style="color:rgb(34,34,34);font-size:small;background-image:initial;background-position:initial;background-repeat:initial"><font color="#3b3838" face="Ebrima"><span style="font-size:12px"><b>Lead Software Engineer</b></span></font></p><p class="MsoNormal" style="color:rgb(34,34,34);font-size:small;background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:9pt;font-family:Ebrima;color:rgb(59,56,56)">+972.50.548.0111</span><span style="font-size:9pt;font-family:Ebrima;color:rgb(64,64,64)"><u></u><u></u></span></p><p class="MsoNormal" style="color:rgb(34,34,34);font-size:small;background-image:initial;background-position:initial;background-repeat:initial"><font color="#0000ff" face="Ebrima"><span style="font-size:12px"><a href="mailto:yuval.l@securithings.com" style="color:rgb(17,85,204)" target="_blank">yuval.l@securithings.com</a></span></font></p><p class="MsoNormal" style="color:rgb(34,34,34);font-size:small;background-image:initial;background-position:initial;background-repeat:initial"><img src="cid:ii_l13cyhga0" alt="logo_black.png" width="200" height="47"></p></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div dir="ltr"><div style="font-size:12.8px"><p class="MsoNormal" style="color:rgb(34,34,34);font-size:small;background-image:initial;background-position:initial;background-repeat:initial">Kind regards,<b style="font-size:13.3333px;color:rgb(0,176,240);font-family:Ebrima"><br><br>Yuval Lombard</b><br></p><p class="MsoNormal" style="color:rgb(34,34,34);font-size:small;background-image:initial;background-position:initial;background-repeat:initial"><font color="#3b3838" face="Ebrima"><span style="font-size:12px"><b>Lead Software Engineer</b></span></font></p><p class="MsoNormal" style="color:rgb(34,34,34);font-size:small;background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:9pt;font-family:Ebrima;color:rgb(59,56,56)">+972.50.548.0111</span><span style="font-size:9pt;font-family:Ebrima;color:rgb(64,64,64)"><u></u><u></u></span></p><p class="MsoNormal" style="color:rgb(34,34,34);font-size:small;background-image:initial;background-position:initial;background-repeat:initial"><font color="#0000ff" face="Ebrima"><span style="font-size:12px"><a href="mailto:yuval.l@securithings.com" style="color:rgb(17,85,204)" target="_blank">yuval.l@securithings.com</a></span></font></p><p class="MsoNormal" style="color:rgb(34,34,34);font-size:small;background-image:initial;background-position:initial;background-repeat:initial"><img src="cid:ii_l13cyhga0" alt="logo_black.png" width="200" height="47"></p></div></div></div></div>