<div class="__aliyun_email_body_block"><div style="clear:both;"><div style="margin:.0px;padding:.0px;border:.0px;outline:.0px;">> I am surprised that anything works at all. You may have success if you make your executor create instances of CarrierThread - which is public.</div><div style="margin:.0px;padding:.0px;border:.0px;outline:.0px;">Thanks, but the `<span >CarrierThread` needs a ForkJoinPool, but I'm using TheadPoolExecutor, have to adapt that.</span></div></div><div style="clear:both;"><span style="font-family:Tahoma,Arial,STHeiti,SimSun;"><br ></span></div><div style="clear:both;"><br ></div><blockquote style="margin-right:.0px;margin-top:.0px;margin-bottom:.0px;font-family:Tahoma,Arial,STHeiti,SimSun;font-size:14.0px;color:#000000;"><div style="clear:both;">------------------------------------------------------------------</div><div style="clear:both;">发件人：何品(虎鸣) <hepin.p@alibaba-inc.com></div><div style="clear:both;">发送时间：2024年6月5日(星期三) 17:37</div><div style="clear:both;">收件人：Alan Bateman<Alan.Bateman@oracle.com>; "何品(虎鸣)"<hepin.p@alibaba-inc.com>; "loom-dev"<loom-dev@openjdk.org></div><div style="clear:both;">主 题：回复：Virtual thread hang and all threads stop running on JDK21</div><div style="clear:both;"><br ></div><div style="clear:both;"><span style="font-family:Tahoma,Arial,STHeiti,SimSun;">Thanks, </span></div><div style="clear:both;"><span style="font-family:Tahoma,Arial,STHeiti,SimSun;">1. it hung when we are sharing the common usage of the Virtualthread (the default scheduler) cross two modules.</span></div><div style="clear:both;"><span style="font-family:Tahoma,Arial,STHeiti,SimSun;">2. after that, I try to hack the virtual thread builder with separated ThreadPoolExecutor.</span></div><div style="clear:both;"><span style="font-family:Tahoma,Arial,STHeiti,SimSun;">3. but still , it hung.</span></div><div style="clear:both;"><span style="font-family:Tahoma,Arial,STHeiti,SimSun;"><br ></span></div><div style="clear:both;"><span style="font-family:Tahoma,Arial,STHeiti,SimSun;">when it hung, `<span class=" __aliyun_node_has_color" style="color:#000000;font-family:Helvetica,Tahoma,Arial;font-size:14.0px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:.0px;text-transform:none;word-spacing:.0px;white-space:normal;text-decoration-thickness:initial;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;">jcmd Thread.print` prints nothing. and dump programly print nothing too.</span></span></div><div style="clear:both;"><span class=" __aliyun_node_has_color" style="font-family:Helvetica,Tahoma,Arial;color:#000000;font-size:14.0px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:.0px;text-transform:none;word-spacing:.0px;white-space:normal;text-decoration-thickness:initial;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;"><br ></span></div><div style="clear:both;"><span class=" __aliyun_node_has_color" style="font-family:Helvetica,Tahoma,Arial;color:#000000;font-size:14.0px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:.0px;text-transform:none;word-spacing:.0px;white-space:normal;text-decoration-thickness:initial;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;">After switch back to normal thread, it never hang.</span></div><div style="clear:both;"><span style="font-family:Helvetica,Tahoma,Arial;">Some information:<br >module A is using `Object.notifyAll, Object.wait` and module B is using `CompletableFuture.get` (may > 100 times in one rune)</span></div><div style="clear:both;"><span style="font-family:Helvetica,Tahoma,Arial;">I was thinking if that could be a problem of notification missing, where in Module A, the concurrency is 3000， protected by a <span >semaphore, but the underling Carrier Thread is only 128, and 3000 > 128. </span></span></div><div style="clear:both;"><span class=" __aliyun_node_has_color" style="font-family:Helvetica,Tahoma,Arial;color:#000000;font-size:14.0px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:.0px;text-transform:none;word-spacing:.0px;white-space:normal;text-decoration-thickness:initial;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;"><br ></span></div><div style="clear:both;"><div class=" __aliyun_node_has_color __aliyun_node_has_bgcolor" style="background-color:#1e1f22;color:#bcbec4;"><pre style="font-family:'Source Code Pro',monospace;font-size:20.0px;"><span class=" __aliyun_node_has_color" style="color:#cf8e6d;">public void </span><span class=" __aliyun_node_has_color" style="color:#56a8f5;">init</span><span class=" __aliyun_node_has_color" style="color:#e8ba36;">() </span><span class=" __aliyun_node_has_color" style="color:#54a857;">{<br ></span><span class=" __aliyun_node_has_color" style="color:#54a857;"> </span><span class=" __aliyun_node_has_color" style="color:#7a7e85;">// </span><span class=" __aliyun_node_has_color" style="color:#7a7e85;font-family:'Courier New',monospace;">监控线程<br ></span><span class=" __aliyun_node_has_color" style="color:#7a7e85;font-family:'Courier New',monospace;"> </span><span class=" __aliyun_node_has_color" style="color:#c77dbb;">VTExecutor</span>.execute<span class=" __aliyun_node_has_color" style="color:#e8ba36;">(</span><span class=" __aliyun_node_has_color" style="color:#54a857;">() </span>-> <span class=" __aliyun_node_has_color" style="color:#359ff4;">{<br ></span><span class=" __aliyun_node_has_color" style="color:#359ff4;"> </span><span class=" __aliyun_node_has_color" style="color:#cf8e6d;">for </span><span class=" __aliyun_node_has_color" style="color:#54a857;">(</span>; ; <span class=" __aliyun_node_has_color" style="color:#54a857;">) </span><span class=" __aliyun_node_has_color" style="color:#6e7ed9;">{<br ></span><span class=" __aliyun_node_has_color" style="color:#6e7ed9;"> </span><span class=" __aliyun_node_has_color" style="color:#cf8e6d;">try </span><span class=" __aliyun_node_has_color" style="color:#179387;">{<br ></span><span class=" __aliyun_node_has_color" style="color:#179387;"> </span>Thread.<span style="font-style:italic;">sleep</span><span class=" __aliyun_node_has_color" style="color:#359ff4;">(</span><span class=" __aliyun_node_has_color" style="color:#c77dbb;font-style:italic;">IDLE_SLEEP_TIME</span><span class=" __aliyun_node_has_color" style="color:#359ff4;">)</span>;<br > <span class=" __aliyun_node_has_color" style="color:#179387;">} </span><span class=" __aliyun_node_has_color" style="color:#cf8e6d;">catch </span><span class=" __aliyun_node_has_color" style="color:#359ff4;">(</span>InterruptedException e<span class=" __aliyun_node_has_color" style="color:#359ff4;">) </span><span class=" __aliyun_node_has_color" style="color:#179387;">{<br ></span><span class=" __aliyun_node_has_color" style="color:#179387;"> </span><span class=" __aliyun_node_has_color" style="color:#7a7e85;">// ignore<br ></span><span class=" __aliyun_node_has_color" style="color:#7a7e85;"> </span><span class=" __aliyun_node_has_color" style="color:#179387;">}<br ></span><span class=" __aliyun_node_has_color" style="color:#179387;"> </span><span class=" __aliyun_node_has_color" style="color:#c77dbb;font-style:italic;">log</span>.info<span class=" __aliyun_node_has_color" style="color:#359ff4;">(<br ></span><span class=" __aliyun_node_has_color" style="color:#359ff4;"> </span><span class=" __aliyun_node_has_color" style="color:#6aab73;">"BlockableVirtualThreadExecutor completed tasks:[{}] inflight tasks:[{}] semaphore available permits: [{}]"</span>,<br > <span class=" __aliyun_node_has_color" style="color:#c77dbb;">completedTaskCounter</span>.get<span class=" __aliyun_node_has_color" style="color:#6e7ed9;">()</span>,<br > <span class=" __aliyun_node_has_color" style="color:#c77dbb;">inflightTaskCounter</span>.get<span class=" __aliyun_node_has_color" style="color:#6e7ed9;">()</span>,<br > <span class=" __aliyun_node_has_color" style="color:#c77dbb;">semaphore</span>.availablePermits<span class=" __aliyun_node_has_color" style="color:#6e7ed9;">()</span><span class=" __aliyun_node_has_color" style="color:#359ff4;">)</span>;<br > <span class=" __aliyun_node_has_color" style="color:#6e7ed9;">}<br ></span><span class=" __aliyun_node_has_color" style="color:#6e7ed9;"> </span><span class=" __aliyun_node_has_color" style="color:#359ff4;">}</span><span class=" __aliyun_node_has_color" style="color:#e8ba36;">)</span>;<br ><span class=" __aliyun_node_has_color" style="color:#54a857;">}</span></pre></div><span class=" __aliyun_node_has_color" style="font-family:Helvetica,Tahoma,Arial;color:#000000;font-size:14.0px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:.0px;text-transform:none;word-spacing:.0px;white-space:normal;text-decoration-thickness:initial;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;"><br ></span></div><div style="clear:both;"><span style="font-family:Helvetica,Tahoma,Arial;">When it hung, this code will never prints any more.</span></div><div style="clear:both;"><div class=" __aliyun_node_has_color __aliyun_node_has_bgcolor" style="background-color:#1e1f22;color:#bcbec4;"><pre style="font-family:'Source Code Pro',monospace;font-size:20.0px;"><span class=" __aliyun_node_has_color" style="color:#cf8e6d;">public static void </span><span class=" __aliyun_node_has_color" style="color:#56a8f5;">jstack</span><span class=" __aliyun_node_has_color" style="color:#e8ba36;">(</span>OutputStream stream<span class=" __aliyun_node_has_color" style="color:#e8ba36;">) </span><span class=" __aliyun_node_has_color" style="color:#cf8e6d;">throws </span>Exception <span class=" __aliyun_node_has_color" style="color:#54a857;">{<br ></span><span class=" __aliyun_node_has_color" style="color:#54a857;"> </span>ThreadMXBean threadMXBean = ManagementFactory.<span style="font-style:italic;">getThreadMXBean</span><span class=" __aliyun_node_has_color" style="color:#e8ba36;">()</span>;<br > <span class=" __aliyun_node_has_color" style="color:#cf8e6d;">long</span><span class=" __aliyun_node_has_color" style="color:#e8ba36;">[] </span>allThreadIds = threadMXBean.getAllThreadIds<span class=" __aliyun_node_has_color" style="color:#e8ba36;">()</span>;<br > ThreadInfo<span class=" __aliyun_node_has_color" style="color:#e8ba36;">[] </span>threadInfos = threadMXBean.getThreadInfo<span class=" __aliyun_node_has_color" style="color:#e8ba36;">(</span>allThreadIds, <span class=" __aliyun_node_has_color" style="color:#cf8e6d;">true</span>, <span class=" __aliyun_node_has_color" style="color:#cf8e6d;">true</span><span class=" __aliyun_node_has_color" style="color:#e8ba36;">)</span>;<br > <span class=" __aliyun_node_has_color" style="color:#cf8e6d;">for </span><span class=" __aliyun_node_has_color" style="color:#e8ba36;">(</span>ThreadInfo threadInfo : threadInfos<span class=" __aliyun_node_has_color" style="color:#e8ba36;">) </span><span class=" __aliyun_node_has_color" style="color:#359ff4;">{<br ></span><span class=" __aliyun_node_has_color" style="color:#359ff4;"> </span><span class=" __aliyun_node_has_color" style="color:#cf8e6d;">if </span><span class=" __aliyun_node_has_color" style="color:#54a857;">(</span>threadInfo != <span class=" __aliyun_node_has_color" style="color:#cf8e6d;">null</span><span class=" __aliyun_node_has_color" style="color:#54a857;">) </span><span class=" __aliyun_node_has_color" style="color:#6e7ed9;">{<br ></span><span class=" __aliyun_node_has_color" style="color:#6e7ed9;"> </span>stream.write<span class=" __aliyun_node_has_color" style="color:#359ff4;">(</span><span style="font-style:italic;">printThreadInfoDepth</span><span class=" __aliyun_node_has_color" style="color:#6e7ed9;">(</span>threadInfo, <span class=" __aliyun_node_has_color" style="color:#2aacb8;">1000</span><span class=" __aliyun_node_has_color" style="color:#6e7ed9;">)</span>.getBytes<span class=" __aliyun_node_has_color" style="color:#6e7ed9;">()</span><span class=" __aliyun_node_has_color" style="color:#359ff4;">)</span>;<br > <span class=" __aliyun_node_has_color" style="color:#6e7ed9;">}<br ></span><span class=" __aliyun_node_has_color" style="color:#6e7ed9;"> </span><span class=" __aliyun_node_has_color" style="color:#359ff4;">}<br ></span><span class=" __aliyun_node_has_color" style="color:#54a857;">}</span></pre></div><span class=" __aliyun_node_has_color" style="font-family:Helvetica,Tahoma,Arial;color:#000000;font-size:14.0px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:.0px;text-transform:none;word-spacing:.0px;white-space:normal;text-decoration-thickness:initial;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;"><br ></span></div><div style="clear:both;"><span style="font-family:Tahoma,Arial,STHeiti,SimSun;"><br ></span></div><div style="clear:both;"><span style="font-family:Tahoma,Arial,STHeiti,SimSun;"><br ></span></div><div style="clear:both;"><span style="font-family:Tahoma,Arial,STHeiti,SimSun;"><br ></span></div><div ><div style="clear:both;">------------------------------------------------------------------</div><div style="clear:both;">发件人：Alan Bateman <Alan.Bateman@oracle.com></div><div style="clear:both;">发送时间：2024年6月4日(星期二) 21:45</div><div style="clear:both;">收件人："何品(虎鸣)"<hepin.p@alibaba-inc.com>; "loom-dev"<loom-dev@openjdk.org></div><div style="clear:both;">主 题：Re: Virtual thread hang and all threads stop running on JDK21</div><div style="clear:both;"><br ></div>
On 04/06/2024 14:29, 何品(虎鸣) wrote:<br >
<div >
<div class=" __aliyun_node_has_color" style="line-height:1.7;font-family:Tahoma,Arial,STHeiti,SimSun;font-size:14.0px;color:#000000;">
<div style="clear:both;"><br >
</div>
<div style="clear:both;">Hi, we are using Virtual threads on
JDK21.0.2, and find it can stop running, all threads
stopped/hung, and we can't even call thread dump.</div>
<div style="clear:both;">We try to separate the Virtual
Thread's scheduler with reflections. but that does not work
too.</div>
<div style="clear:both;"><br >
</div>
<div style="clear:both;">Our userbase are pretty simple:</div>
<div style="clear:both;">A: VT Executor backing by 128-sized
ThreadPoolExecutor for running java code with
`CompletableFuture.get`</div>
<div style="clear:both;">B: VT Executor backing by 64-sized
ThreadPoolExecutor for retrieving data with HTTP client,
with max concurrency to 3000 (virtual threads)</div>
<div style="clear:both;"><br >
</div>
<div style="clear:both;">Even if A and B are using Different
backing ThreadPool, they all stop responding。</div>
<br >
</div>
</div>
There isn't much to go on here. There isn't any support at this time
for custom schedulers so I can't tell if the hang is when you use
the default scheduler or only when you use reflection to hack in and
set your own?<br >
<br >
Also when you say "thread dump", do you mean the HotSpot VM thread
dump (ctrl-\, jstack, jcmd Thread.print) or do you mean the thread
dump that prints all virtual threads (jcmd Thread.dump_to_file)?<br >
<br >
-Alan<br >
<br >
<br >
</div></blockquote></div>