AssertionError in ResponseSubscribers.HttpResponseInputStream.read() in J11 only

Tue Oct 4 15:18:01 UTC 2022

Yea, it’s like a dup of that, which hasn’t been back ported to 11.

The issue that drives me crazy is the TCP connection reset one. Do you have any idea what might cause it? It doesn’t happen with other HTTP clients (Apache or HttpURLConnection). What I see for the TCP connection resets is always something like this (stack trace from Java 11). With the reproducer I’m trying to build I want to get Jackson and other stuff out of the game.

Caused by: java.io.IOException: closed
	at java.net.http/jdk.internal.net.http.ResponseSubscribers$HttpResponseInputStream.current(ResponseSubscribers.java:368)
	at java.net.http/jdk.internal.net.http.ResponseSubscribers$HttpResponseInputStream.read(ResponseSubscribers.java:403)
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._loadMore(UTF8StreamJsonParser.java:220)
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._loadMoreGuaranteed(UTF8StreamJsonParser.java:2401)
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2486)
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2466)
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:297)
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextTextValue(UTF8StreamJsonParser.java:1281)
	at com.fasterxml.jackson.databind.deser.std.StringCollectionDeserializer.deserialize(StringCollectionDeserializer.java:192)
	... 100 more
Caused by: java.io.IOException: chunked transfer encoding, state: READING_DATA
	at java.net.http/jdk.internal.net.http.common.Utils.wrapWithExtraDetail(Utils.java:330)
	at java.net.http/jdk.internal.net.http.Http1Response$BodyReader.onReadError(Http1Response.java:758)
	at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.checkForErrors(Http1AsyncReceiver.java:297)
	at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.flush(Http1AsyncReceiver.java:263)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: Connection reset by peer
	at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
	at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:245)
	at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:223)
	at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:356)
	at java.net.http/jdk.internal.net.http.SocketTube.readAvailable(SocketTube.java:1153)
	at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.read(SocketTube.java:821)
	at java.net.http/jdk.internal.net.http.SocketTube$SocketFlowTask.run(SocketTube.java:175)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:271)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:224)
	at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.signalReadable(SocketTube.java:763)
	at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$ReadEvent.signalEvent(SocketTube.java:941)
	at java.net.http/jdk.internal.net.http.SocketTube$SocketFlowEvent.handle(SocketTube.java:245)
	at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.handleEvent(HttpClientImpl.java:957)
	at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.lambda$run$3(HttpClientImpl.java:912)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:912)

> On 4. Oct 2022, at 17:03, Daniel Fuchs <daniel.fuchs at oracle.com> wrote:
> 
> Hi Robert,
> 
> Could this be a duplicate of JDK-8228970?
> I would suggest applying the fix and see if the problem
> persists.
> 
> best regards,
> 
> -- daniel
> 
> On 04/10/2022 15:54, Robert Stupp wrote:
>> Hi,
>> Recently I hit a situation where HTTP requests with the “new” Java HttpClient ran into TCP-connection-resets, but sadly only on GitHub hosted runners and so far I had no luck to reproduce locally. This is why I wanted to setup a reproducer for that.
>> While trying to build that reproducer, I ran into another unrelated, but locally reproducible j.l.AssertionError in recent versions of Java 11 (Java 17 and newer are fine). Reproducing the AE is pretty easy (although the reproducer is a bit big): clone https://github.com/snazy/http-client-tcp-reset-repro.git <https://github.com/snazy/http-client-tcp-reset-repro.git> + `./gradlew test` using Java 11. I’ve opened https://bugs.openjdk.org/browse/JDK-8294773 <https://bugs.openjdk.org/browse/JDK-8294773> for this.
>> The original issue, for which I’m trying to build a reproducer for, is strange and hard to narrow down, because it only happens on GH hosted runners with the “new” Java HttpClient.
>> Background (for the TCP-conn-reset): We are using the plain old HttpURLConnection approach, and wanted to use the new HttpClient when it’s available (with Java 11+).
>> Requests with small payloads (in request and response) are fine - but bigger-ish (couple kb and more) payloads somewhat consistently run into TCP-connection-resets, which do not happen with HttpURLConnection and the Apache HTTP client. “Somewhat consistently” means: it happens often, but not always (kinda flaky).
>> The reproducer I was trying to build for that uses Jetty 9, Jetty 11 + the HTTP server in the JDK on the server side + HttpURLConnection, Apache HTTP client and the new Java HttpClient on the client side. The matrix of http server/client implementations is that big, because I wanted to narrow down whether there’s a specific combination that causes the TCP connection reset.
>> Some things I tried without success:
>> * Enable debug logging for the new HttpClient (no luck, debug logging changes the timing and the TCP conn reset doesn’t happen)
>> * Delay the request processing with a “Thread.sleep(10)” lets the issue disappear as well
>> * Using Java 17 + 18
>> * Using a different OpenJDK distribution (was Azul Zulu, tried Temurin)
>> * Using a local Docker container with CPU + memory somewhat similar to the GH hosted runner
>> * Using very similar IP settings (sysctl stuff for net.code + net.ipv4) locally
>> What I could figure out so far is that the new HttpClient creates a connection for mostly every HTTP request, it reuses existing connections locally. This happens even if there are HTTP request that were completely fine. (<— only on GH hosted runners).
>> I’ll continue to try to get a reproducer for the TCP-connection-reset issue.
>> Robert
> 