[External] : Re: Project Loom VirtualThreads hang

[Ron Pressler]

On 3 Jan 2023, at 14:03, Robert Engels <rengels at ix.netcom.com<mailto:rengels at ix.netcom.com>> wrote:

Hi Ron.

I think there is another traditional use case of “virtual threads” which is to increase efficiency for short cycle tasks.

You can write a simple blocking queue and a message passing system - and with as little as 32 threads you will see > 50% system cpu time due to trashing in the scheduler.

A virtual thread should be far more efficient in regards to user space locking/blocking/wake-up - my tests show only a few % system cpu time. To do this well though the vthread scheduler needs to be super efficient and the locking constructs rewritten knowing they’re on green threads - because the spin time/block decisions are much different (eg upwards of 16 microseconds to perform an os switch to another thread on the cpu vs a few hundred nanoseconds for a green thread switch).

Quite possibly, but that use-case is not a primary focus for us at the moment because there’s more efficiency to be gained for the simple servers that use hundreds of thousands of virtual threads. Not only is that use-case more common, but the efficiency can increase by hundreds of percent, while reducing context-switching costs has an impact in the single digits for I/O workloads (see https://inside.java/2020/08/07/loom-performance/).

That’s not to say we’re not interested in scheduling improvements — far from it, and we’ll definitely do them with time — but in terms of prioritising effort, we’ll probably first invest more where there’s more to be gained.

— Ron


On Jan 3, 2023, at 3:51 AM, Ron Pressler <ron.pressler at oracle.com<mailto:ron.pressler at oracle.com>> wrote:

 Hi.

Let me first address some of the topics that have come up in this thread before I get to the main point.

The meaning of Thread.yield is “inform the runtime that this thread doesn’t have anything useful to do at the moment, but it might again in the future”. What the runtime does with that information is up to the runtime. As Alan said, in JDK 20 we’ve changed the implementation to have the scheduler spend more time to more aggressively look for other work to do, but Thread.yield shouldn’t be used for scheduling control. It should be used (and very sparingly) to say “I temporarily have nothing useful to do.”

Virtual threads are scheduled preemptively, not cooperatively. This means that the runtime makes the decision when to deschedule (preempt) one thread and schedule another without cooperation from user code. However, the virtual thread scheduler currently does not employ time-sharing, i.e. it does not decide to preempt a thread based on it exceeding some allotted time-slice quota on the CPU. The reason for that is we haven’t yet identified a use-case where time-sharing can help for the use-cases virtual threads address (although we’re very interested to hear about such use-cases if anyone comes across one). Virtual threads are mostly intended to write servers, non-realtime kernels primarily employ time-sharing when the CPU is at 100%, servers don’t usually run at 100% CPU and when they do people aren’t generally happy with the result. So servers don’t rely on time-sharing even without virtual threads (at least not the kind that requires special support), but when we identify a use case where time-sharing could help server workloads we can consider adding it to the preemption considerations.

Now for the central point. Virtual threads have scalability benefits for a single reason: their high number. This is due to Little’s law. Replacing a set of platform threads with virtual threads — as the JEP explains — should not yield any significant benefits. The benefits come when employing a very high number of virtual threads. For example, the Helidon framework creates 3 million new virtual threads every seconds under high load. As a rule of thumb, if your application doesn’t create a few thousand or so brand-new virtual threads every second, then you’re not using them in the manner for which they were intended and will not see substantial benefits. Virtual threads replace short individual *tasks* in your application, not platform threads. They are best thought of as a business logic entity representing a task rather than an “execution resource.” For those who are used to managing threads as resources, this is a big change in how they think about threads, and will take some getting used to.

— Ron
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/loom-dev/attachments/20230103/cb622d50/attachment-0001.htm>