Reporting issues with HTTP and virtual threads

[Daniel Fuchs]

Hi Philip,

Thanks for your interest in the HttpClient!

Nice article. Some comments:

Problem #2

 > The obvious fix to this would be to establish multiple HTTP/2 
connections each allowing 100 concurrent streams.

 From RFC 9113:

"Clients SHOULD NOT open more than one HTTP/2 connection to a
  given host and port pair, where the host is derived from a URI,
  a selected alternative service [ALT-SVC], or a configured proxy."

That said, I recognize that immediately throwing an exception is not
the best user friendly behavior. We are aware of the issue and
exploring alternatives.

In the mean time, the work around I would recommend is to
control (throttle) the number of concurrent requests
made to the same host in the client application. This can
be done with a simple Semaphore initialized with 100 permits.
As long as the server allows 100 concurrent streams (which is
the default recommended by the specification), this would work.
Acquire the semaphore before sending a request, and release it
upon request completion (whether successful or unsuccessful).


 >  Multiple HttpClient instances appear to reuse the same connection 
for a given scheme:host:port and will never establish a second, third, 
etc. connection.

Each client instance has its own connection pool, by design.
There's no global connection pool. There is no sharing of connection
between client instances.
I don't know what is causing the behavior you observe, but it can't
be connection sharing between different HttpClient instances:
this simply cannot happen. There is no global pool.

Problem #3

 > I played with the number of concurrent virtual threads and also with 
platform threads pools and quickly found that Java’s HttpClient creates 
one additional platform thread for every concurrent HTTP request coming 
from a unique thread be it platform or virtual.

The client does not create one thread per request.
What you are observing here is the behavior of the default
client executor service: if no executor is provided
at construction time, the HttpClient will simply uses an executor
provided by `Executors.newCachedThreadPool(ThreadFactory)`.
So what you are observing is just the default behavior of a
cached thread pool executor when a new Runnable is submitted.
Tasks submitted by the HttpClient should be mostly short lived
and are non blocking (except for short synchronization / locks),
but some may be longer lived - like for instance tasks that need
to be executed on behalf of the SSLEngine. My understanding is
that the number of cached threads will shrink back if they remain
idle for too long (see [1])

[1] 
https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/util/concurrent/Executors.html#newCachedThreadPool()

As you have discovered, it is possible to configure a HttpClient
(through HttpClient.Builder) with the executor of your choice.
You can pass a FixedThreadPoolExecutor, or even a
newVirtualThreadPerTask executor. You can even share the supplied
executor trough several HttpClient instances if such is your wish.
Though task submitted by the HttpClient should not block,
be careful that custom code may execute in that executor,
for instance, if you supply your own BodyHandler / BodySubscriber.
So if you block there, it will block a thread, which may lead
to thread starvation if enough threads are blocked.

Problem #4

This is more nasty, and a well known issue.
It is not specific to the HttpClient.

The issue here is when there is contention on
a j.u.c.Lock accessed both from outside a synchronized
block and from within a synchronized block, causing
the carrier thread to be pinned.

We have reviewed the HttpClient code for JDK 21, and
changed it to use ReentrantLock instead of synchronized wherever
needed to make sure it remains virtual thread friendly.
Some bugs were still present however and were fixed after GA
(see for instance https://bugs.openjdk.org/browse/JDK-8316031
and https://bugs.openjdk.org/browse/JDK-8317736).
But I don't believe the issue you are observing is caused by
any of those.

The issue you are observing with the HttpResponseInputStream
is just another occurrence of the thread pining limitation.

The HttpResponseInputStream uses an internal j.u.c.Lock / condition
to wait for data to become available. The code that submits
data to this queue will use that lock from outside any synchronized
block, but if the code that read the data calls the
HttpResponseInputStream from within a synchronized block, then
that lock will be accessed both from within and from outside
of a synchronized block, which will cause pinning of your
virtual thread carrier if data is not readily available.
This is the behavior you are observing. Nothing can be
done within the HttpClient itself, since the synchronized
block is not there.

Work is being done in the Loom project to try and solve the
issue with thread pinning when using monitors. Hopefully
this limitation will be lifted in a future release of the
JDK.

The only way to solve that issue at the moment is to change
the application and avoid reading from a synchronized block.

I hope this can shed some light on the behaviors you have
been observing. Obviously these issues will simply go away
when the underlying thread pinning issue is fixed.

best regards,

-- daniel

On 27/11/2023 21:57, Philip Boutros wrote:
> Hi
> 
> I recently ran into a number of HTTP related issues with virtual threads 
> as documented in this article.
> 
> https://medium.com/@phil_3582/java-virtual-threads-some-early-gotchas-to-look-out-for-f65df1bad0db?source=friends_link&sk=f1f6ed425c4d17cdc1188f85a0d13d4d <https://medium.com/@phil_3582/java-virtual-threads-some-early-gotchas-to-look-out-for-f65df1bad0db?source=friends_link&sk=f1f6ed425c4d17cdc1188f85a0d13d4d> (Friend link, no paywall)
> 
> Very briefly:
> 1. HTTP/2 concurrent connection limitation (not related to virtual threads)
> 2. HttpClient worker thread count explosion when used with virtual threads
> 3. ReadableByteChannel/HttpResponseInputStream deadlock when used with 
> virtual threads
> 
> Advise on the best way to report these? I don't have Author status to 
> report them using https://bugs.openjdk.org/ <https://bugs.openjdk.org/>
> 
> Thanks
> -P