Virtual vs platform performance at 10K HTTP requests

[Alan Bateman]

On 27/11/2025 16:38, Cay Horstmann wrote:
> I did both with David's benchmark. Of course, the semaphore is the way 
> to go. With default Tomcat settings, 200 or 2000 permits work fine, 
> and then virtual threads give significantly better throughput than a 
> small number of platform threads. As one would expect.
>
> But back to those 10000 simultaneous connections. As David observed, 
> left unthrottled, virtual threads do really poorly. In my experiment, 
> *much* more poorly than platform threads, when using a 
> Executors.newCachedThreadPool().

10k or 50k+ concurrent connections shouldn't be an issue but it may 
involve tweaking a number of sysctl.conf/equivalent settings to get 
there. Elliot Barlas published a fun project at one point where he had 
several million TCP connections with 2 virtual threads per connection.

I think the issue we are discussing here is partly about storming the 
server with SYN packets to establish the TCP connections. Servers 
typically have rate limiting and other mitigations to avoid SYN flood 
attacks. This is why we were asking about the connection backlog and 
other configuration. It's possible the thread pool bench isn't going to 
see this because it's using synchronous/blocking APIs to establish the 
TCP connection and so the number of outstanding connects in progress is 
limited by the size of the thread pool.

The other part that seems relevant is that the protocol in the 
benchmarks is HTTP/1.1 which supports persistent connections, 
essentially re-use of a TCP connection if possible to avoid creating a 
new TCP connection for each HTTP server. There's another layer of 
configuration here. I don't know Tomcat's defaults but it is likely that 
it keep 100 or 200 connections alive for future HTTP requests. Once the 
concurrency exceeds this then it's likely there is a new TCP connection 
established for each request. The thread pool bench is 4 threads and 
likely well below the maximum number of persistent connections.

So yes, moving from thread pools to virtual threads may shine light on 
the "next" concurrency bottleneck, the bottle hidden by limiting the 
number of threads. The guidance that we've given at conferences and in 
the JEPs is to use a semaphore/equivalent to limit concurrent at the 
right level of granularity, be that number of concurrent database 
connections or whatever it might be.

>
> I am wondering why that would be, and whether it is something that is 
> worth addressing, because it seems like something that people could 
> run into in practice. FWIW, I stumbled upon 
> https://bugs.openjdk.org/browse/JDK-8360046 which addresses a somewhat 
> similar scenario.

That one is an over subscription issue that arises with producer 
starting a lot of virtual threads that "do nothing". It helps us to 
reduce the gap between ForkJoinTasks that "do nothing" with virtual 
threads that "do nothing".

-Alan