Experiencing an issue with ScheduledExecutorService alongside VT

[robert engels]

If you look at the thread dump, you will see:

"MQTT Snd: test2:con08:13" #891110 [2129] daemon prio=5 os_prio=0 cpu=2772.51ms elapsed=4692.79s tid=0x00007f8a2c028dc0 nid=2129 in Object.wait()  [0x00007f89fe0df000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait0(java.base at 23-loom/Native Method)
        - waiting on <no object reference available>
        at java.lang.Object.wait(java.base at 23-loom/Object.java:389)
        at java.lang.Object.wait(java.base at 23-loom/Object.java:347)
        at org.eclipse.paho.client.mqttv3.internal.ClientState.get(ClientState.java:833)
        - locked <0x000000070d880b08> (a java.lang.Object)
        at org.eclipse.paho.client.mqttv3.internal.CommsSender.run(CommsSender.java:129)
        at java.lang.Thread.runWith(java.base at 23-loom/Thread.java:1595)
        at java.lang.Thread.run(java.base at 23-loom/Thread.java:1582)

I am guessing you have library code using monitors. I am concerned about the “no object reference available” above.

I also suggest testing with a lighter weight VT native http server like github.com/robaho/httpserver <http://github.com/robaho/httpserver> to avoid so many framework threads - it should make things easier to diagnose.

Most importantly, use

jcmd <PID> Thread.dump_to_file -format=text <file>

so that the thread dumps contain virtual thread information.

> On Jul 21, 2024, at 9:10 AM, robert engels <rengels at ix.netcom.com> wrote:
> 
> Hi. The code looks incorrect (although it is not complete so hard to tell).
> 
> The outer exception catch is releasing the semaphore even though it may have never acquired it.
> 
> Since the code is not complete, there is no way to validate what you are seeing.
> 
> I suggest trying to provide a standalone reproducible test case.
> 
>> On Jul 21, 2024, at 8:11 AM, Yuval Lombard <yuval.l at securithings.com <mailto:yuval.l at securithings.com>> wrote:
>> 
>> Hi Alan,
>> 
>> Attaching the thread dump when the issue occurs (as a txt file)
>> 
>> It might be useful
>> 
>> On Sun, 21 Jul 2024 at 12:41, Yuval Lombard <yuval.l at securithings.com <mailto:yuval.l at securithings.com>> wrote:
>> Hi Alan
>> 
>> Thanks for replying!
>> 
>> When the issue occurs the Log you mentioned is not being printed - it is as if (or maybe it is even what is really happening) the virtual thread is not being scheduled (by its internal scheduler)
>> I to a thread dump by executing "kill -3 <pid>" and did not see the VT, will  `java <pid> Thread.dump_to_file <file>` produce a more detailed view of the living threads?
>> 
>> I will try to run the later EA build
>> 
>> On Sun, 21 Jul 2024 at 11:14, Alan Bateman <Alan.Bateman at oracle.com <mailto:Alan.Bateman at oracle.com>> wrote:
>> On 21/07/2024 08:05, Yuval Lombard wrote:
>> > Hey team
>> >
>> > We are experiencing some weird halting issues when scheduling tasks 
>> > with ScheduledExecutorService and VT.
>> >
>> > We are observing some near deadlock issues / halting of executors 
>> > which we are not able to reproduce when using platform threads
>> >
>> > This is one example of a near deadlock situation, where for some 
>> > reason virtual threads are starting to execute a task, and not 
>> > reaching the finally block nor the catch Throwable block to release 
>> > the acquired semaphore permit, reaching to drained semaphore.
>> 
>> 
>> Is "Scheduling .. task for device" logged? Did you take thread dump with 
>> `java <pid> Thread.dump_to_file <file>` and try to identify the virtual 
>> thread?
>> 
>> One of the later mails mentions 23-loom+4-102 and not seeing JFR events. 
>> Can you re-run with the latest EA build (currently 24-loom+2-24) as that 
>> has changes for the JFR events that may provide some insight here.
>> 
>> -Alan
>> 
>> 
>> -- 
>> Kind regards,
>> 
>> Yuval Lombard
>> 
>> Lead Software Engineer
>> 
>> +972.50.548.0111
>> 
>> yuval.l at securithings.com <mailto:yuval.l at securithings.com>
>> <logo_black.png>
>> 
>> 
>> 
>> -- 
>> Kind regards,
>> 
>> Yuval Lombard
>> 
>> Lead Software Engineer
>> 
>> +972.50.548.0111
>> 
>> yuval.l at securithings.com <mailto:yuval.l at securithings.com>
>> <logo_black.png>
>> 
>> <thread_dump.txt>
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/loom-dev/attachments/20240721/1f6f83b1/attachment-0001.htm>