Monitoring virtual threads via thread dumps

[Stig Rohde Døssing]

Hi,

I've been playing around with virtual threads a bit, and had some questions
regarding their behavior in thread dumps. Our application dumps all threads
periodically to our logging tool, which we use as an after-the-fact way to
gain insights into what the application was doing when it misbehaved. We
also use them as a low resolution profiler option, since hotspots will
often be visible in these dumps, and we attach metadata like request-level
information to them to include in dumps so we can tell which user is
responsible if the system is under load.

I set up a small example application (on Java 20) that starts a virtual
thread which just spins while the main thread dumps the threads via jstack,
Thread.getAllStackTraces and the new HotSpotDiagnosticMXBean.dumpThreads.

My first few questions relate to the ability to tell what carrier threads
are spending their time on, based on thread dumps.

For the profiling use case we have, we basically would like to see a
snapshot of what code the application is executing at a given moment.
Thread dumps for carrier threads don't include the stack of the virtual
thread being executed. Would it be possible to include these stacks, or is
there a reason it is preferable not to list them?

When looking at the output of jstack, I noticed that the carrier thread
says which virtual thread it's carrying:

"ForkJoinPool-1-worker-1" #20 [5113] daemon prio=5 os_prio=0 cpu=9823.42ms
elapsed=9.87s tid=0x00007f8e9c1860b0  [0x00007f8e4fbfb000]
   Carrying virtual thread #19
        at jdk.internal.vm.Continuation.run(java.base at 20/Continuation.java:257)


The output from getAllStackTraces and the MXBean don't include this tidbit,
which I found a little odd: Jstack doesn't include virtual threads in the
dump, so I can't look up thread #19, and the MXBean which does list thread
19 doesn't say which virtual thread the carrier is executing. Should the
MXBean output be enhanced with this information? It seems a little
unfortunate to have to do two dumps with separate tools to get both pieces
of information. Having both pieces of information would let us "link
together" the carrier thread and virtual thread, solving the issue of not
being able to tell which virtual threads are being executed.

Unrelated to the question of using dumps for profiling, I noticed that when
using the HotSpotDiagnosticMXBean to dump threads, virtual threads created
directly via Thread.ofVirtual are not listed, you have to use
Executors.newVirtualThreadPerTaskExecutor. Should this be called out more
strongly in the documentation? I don't see this mentioned outside the JEP,
and it caught me by surprise, which isn't ideal since I probably wouldn't
have thought to check this kind of thing in a real application, and it
would make debugging such an application harder.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/loom-dev/attachments/20230328/4f54189f/attachment.htm>