Jetty and Loom

[Alan Bateman]

On 30/12/2020 08:14, Greg Wilkins wrote:
> Hi,
>
> The Jetty team have blogged about our initial experiments with Loom virtual
> threads:
>
>     - https://webtide.com/do-looms-claims-stack-up-part-1/
>     - https://webtide.com/do-looms-claims-stack-up-part-2/
>
> It hasn't all been plain sailing and I'm sure/hope that we'll be corrected
> on some aspects etc.
>
> However, out of that work we have produced a branch of jetty (
> https://github.com/eclipse/jetty.project/tree/jetty-10.0.x-loom) that we
> think has a good way to integrate Loom.     We've not just replaced the
> Thread Pool with some kind of virtual thread factory, as that would still
> leave jetty doing lots of work internally thinking some threads can't block
> and avoiding head-of-line blocking for flow control etc. etc.
Thanks for the links to the experiments.

The experiment with the CometD chat is interesting. Would it be possible 
for upcoming part 3 to include the code, or links, for the both the 
async and simpler synchronous examples? Loom is all about code that is 
simpler to maintain, debug, and profile and will be interesting to see 
if that is so without sacrificing scalability compared to the async code 
in this example. If I read the output correctly in part 2 then the 
duration of 1000 client run is about 10s (do I have that right?) and you 
might need to run for longer to ensure that everything is warmed up.

The experiment with call stacks with 1000+ frames does demonstrate that 
virtual threads can run existing code but it does beg the question as to 
whether some of these bloated libraries are important when working in 
the small. The same goes for thread locals that are caching large graphs 
of objects (I assume the "latency statistics" TL that you mention must 
be large). There is exploratory work under way to provide alternative 
solutions for some of today's uses of TLs and the _nonBlocking TL in the 
linked code is an example of an idiom that the scope variables will help 
with. TLs are a very general mechanism and there will be cases where the 
usages needs to be re-examined by the maintainer of the code (Loom will 
have diagnostics options to help identify TL usages in virtual threads).

Using thread pools to limit concurrency is a discussion in itself. It 
works for the JDBC example because it is limited to 100 connections in 
the test but it may not be the right level of granularity for other 
usages, esp. fan-out to other services where the number of network 
connections can significantly out number any limit you put on the number 
of threads. Also thread pools are not without their issues. The cost of 
starting threads that you discuss is one, but there is also the issue of 
TLs outliving tasks, and subtle issue of trying to cancel/interrupt at 
around the time that a task completes and a thread is borrowed for 
another task.

Can you expand a bit more on the "eat what you kill scheduling" issue? I 
can't quite tell if you ran into an issue or not. For now at least, if a 
virtual thread invokes a selection operation then it will pin the 
carrier thread (this is due to the way that selection operations are 
specified rather than anything else) but it does use the FJP 
managedBlocker mechanism to increase parallelism during the operation. I 
can't quite tell if the "postponed indefinitely" is a reference to the 
first thread that invokes the selection operation (before consuming one 
selection key) or followers that consume the selected keys not handled 
by the first thread.

The approach to off-load from the Jetty core threads to virtual threads 
that execute the user's code seems reasonable, at least from a distance. 
It will at least be expedient as you already have an async core that you 
want to keep. Helidon MP have something similar where it can run in a 
mode that creates a virtual thread to execute the user/application code. 
The important thing is that the application isn't forced into using 
callbacks or split up code into stages in order to avoid blocking 
operations.

-Alan.