AssertionError in ResponseSubscribers.HttpResponseInputStream.read() in J11 only

[Robert Stupp]

Hi,

Recently I hit a situation where HTTP requests with the “new” Java HttpClient ran into TCP-connection-resets, but sadly only on GitHub hosted runners and so far I had no luck to reproduce locally. This is why I wanted to setup a reproducer for that.

While trying to build that reproducer, I ran into another unrelated, but locally reproducible j.l.AssertionError in recent versions of Java 11 (Java 17 and newer are fine). Reproducing the AE is pretty easy (although the reproducer is a bit big): clone https://github.com/snazy/http-client-tcp-reset-repro.git <https://github.com/snazy/http-client-tcp-reset-repro.git> + `./gradlew test` using Java 11. I’ve opened https://bugs.openjdk.org/browse/JDK-8294773 <https://bugs.openjdk.org/browse/JDK-8294773> for this.

The original issue, for which I’m trying to build a reproducer for, is strange and hard to narrow down, because it only happens on GH hosted runners with the “new” Java HttpClient.

Background (for the TCP-conn-reset): We are using the plain old HttpURLConnection approach, and wanted to use the new HttpClient when it’s available (with Java 11+).

Requests with small payloads (in request and response) are fine - but bigger-ish (couple kb and more) payloads somewhat consistently run into TCP-connection-resets, which do not happen with HttpURLConnection and the Apache HTTP client. “Somewhat consistently” means: it happens often, but not always (kinda flaky).

The reproducer I was trying to build for that uses Jetty 9, Jetty 11 + the HTTP server in the JDK on the server side + HttpURLConnection, Apache HTTP client and the new Java HttpClient on the client side. The matrix of http server/client implementations is that big, because I wanted to narrow down whether there’s a specific combination that causes the TCP connection reset.

Some things I tried without success:
* Enable debug logging for the new HttpClient (no luck, debug logging changes the timing and the TCP conn reset doesn’t happen)
* Delay the request processing with a “Thread.sleep(10)” lets the issue disappear as well
* Using Java 17 + 18
* Using a different OpenJDK distribution (was Azul Zulu, tried Temurin)
* Using a local Docker container with CPU + memory somewhat similar to the GH hosted runner
* Using very similar IP settings (sysctl stuff for net.code + net.ipv4) locally

What I could figure out so far is that the new HttpClient creates a connection for mostly every HTTP request, it reuses existing connections locally. This happens even if there are HTTP request that were completely fine. (<— only on GH hosted runners).

I’ll continue to try to get a reproducer for the TCP-connection-reset issue.

Robert

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/net-dev/attachments/20221004/4b9a9549/attachment-0001.htm>