<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hello Nicolas,</p>
<p>It's good to hear that the workaround is functional. I expect
this to be fixed in JDK 22 and likely be backported in one of the
21 update releases.<br>
</p>
<p>-Jaikiran<br>
</p>
<div class="moz-cite-prefix">On 22/07/23 5:35 pm, Nicolas Henneaux
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CALcOUbjEHJkbb-u8=ESsdOnbP-=P2ufWpUfpZ10XDLKXbAmM0A@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>Hi Jaikiran,</div>
<div><br>
</div>
<div>Great news you nailed down the issue and already working on
a fix!</div>
<div><br>
</div>
<div>I have set the idle timeout to 60s and there was not a
single occurrence of those errors for 2 days. This is
effectively the trigger of those errors. I would prefer to
avoid using a non-default idle timeout but I can live with it
for some time. Once you'll manage to get a fix on that, could
you indicate the schedule delivery for it?</div>
<div><br>
</div>
<div>Thanks a lot for the help,</div>
<div><br>
</div>
<div>Best regards,</div>
<div><br>
</div>
<div>Nicolas<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, 21 Jul 2023 at 12:35,
Jaikiran Pai <<a href="mailto:jai.forums2013@gmail.com"
moz-do-not-send="true" class="moz-txt-link-freetext">jai.forums2013@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello Nicolas,<br>
</p>
<p>I have been able to reproduce this issue in a test case
and I realize what's going on. I'll be working on a fix.</p>
<p>I'm just curious if you were able to workaround this
issue with the use of a non-default idle timeout using the
system properties I noted.<br>
</p>
<p>-Jaikiran<br>
</p>
<div>On 20/07/23 1:01 pm, Jaikiran Pai wrote:<br>
</div>
<blockquote type="cite"> On 20/07/23 12:19 pm, Nicolas
Henneaux wrote:<br>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>Hi Jaikiran!<br>
<br>
</div>
Thanks for your quick feedback!<br>
<br>
</div>
It happens on a regular and consistent basis
mostly when execution health checks between
microservices. On about 100 microservices deployed
on 100 VM, it happens ~400 times every hours.<br>
<br>
</div>
I haven't find a consistent pattern for now. I
cannot attach the whole stacktrace as it is a
closed source code. I can provide some details. A
ScheduleExecutor is launching a check on healthiness
HTTP call every 29s (was 30s but I desync it to
avoid idle timeout of 30s alignment with
HttpClient). This is the biggest source of those
errors but not the only one, there are regular HTTP
calls received on a Jetty server and calling another
service that are failing.<br>
<br>
</div>
<div>I have already tried to enable full debug without
much help for now. Is there something in particular
you are willing to check?<br>
</div>
</div>
</blockquote>
<p>Thank you for these additional details. I don't have
any specific log message in mind for now. I'll look into
the code and try to reproduce this in a test case.<br>
</p>
<p>From the stacktrace it appears that this affects HTTP2
connection in your application. One workaround would be
to increase the idle connection timeout for these
connections, to reduce the chances of this issue. There
are system properties that control this value. If you
want to configure this only for HTTP2 connections, then
jdk.httpclient.keepalive.timeout.h2 is the system
property to set, the value represents number of seconds
(and defaults to 30). More details about this and other
system properties applicable for HttpClient are
available here <a
href="https://docs.oracle.com/en/java/javase/20/docs/api/java.net.http/module-summary.html"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://docs.oracle.com/en/java/javase/20/docs/api/java.net.http/module-summary.html</a></p>
<p>-Jaikiran<br>
</p>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>Best regards,</div>
<div><br>
</div>
<div>Nicolas<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, 20 Jul 2023
at 08:11, Jaikiran Pai <<a
href="mailto:jai.forums2013@gmail.com"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">jai.forums2013@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello Nicolas,</p>
<p>This looks like it will need to be
investigated. I've opened <a
href="https://bugs.openjdk.org/browse/JDK-8312433"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://bugs.openjdk.org/browse/JDK-8312433</a>
to tracking this. It would help further if you
could attach the complete exception logs along
with timestamps, if any.<br>
</p>
<p>Is this consistently reproducible? Perhaps you
could launch your application passing it the
-Djdk.internal.httpclient.debug=true system
property? That will generate very verbose logs,
so you might want to use that sparingly and
remove that system property when you have
collected the logs for this issue. <br>
</p>
<p>-Jaikiran<br>
</p>
<div>On 20/07/23 11:00 am, Nicolas Henneaux wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>Hi all,<br>
<br>
</div>
Since I updated Java from 19.0.2 to 20.0.1,
I got some errors with connection idle
shutdown or closed stream usage when trying
to send HTTP requests with
java.net.http.HttpClient.<br>
<br>
</div>
<div>The two following errors are happening. I
suspect those are linked even if they don't
trigger together. <br>
</div>
<div>1. <font size="1">java.net.http.HttpConnectTimeoutException:
HTTP connection idle, no active streams.
Shutting down.<br>
at
jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:890)
~[java.net.http:?]</font></div>
<div>2. <font size="1"> java.io.IOException:
Stream 9 cancelled<br>
at
jdk.internal.net.http.Stream.cancel(Stream.java:1229)
~[java.net.http:?]</font></div>
<div>The detailed stacks are copied below.</div>
<div><br>
</div>
<div>Error 1. is caused by an idle connection
trying to be used to send the request from
what I understood from the code. Error 2.
despite saying it is a cancelled streamed is
actually happening when the stream is closed
if I haven't missed something in the code.</div>
<div>I suspect either the idle connections are
not always properly discarded to create a
new one or there is a race condition when
discarding the idle connections. </div>
<div><br>
</div>
<div>I was not able so far to find where this
could happen and I hope you could help me to
find where the problem lies.</div>
<div><br>
</div>
<div>Thank you in advance for your help,</div>
<div><br>
</div>
<div>Best regards,</div>
<div><br>
</div>
<div>Nicolas Henneaux<br>
</div>
<div><font size="1"><br>
</font></div>
<div><font size="2">Detailed stacks<br>
</font></div>
<div><font size="1">1. Caused by:
java.net.http.HttpConnectTimeoutException:
HTTP connection idle, no active streams.
Shutting down.<br>
at
jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:890)
~[java.net.http:?]<br>
at
jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:133)
~[java.net.http:?]<br>
(...)<br>
Caused by:
java.net.http.HttpConnectTimeoutException:
HTTP connection idle, no active streams.
Shutting down.<br>
at
jdk.internal.net.http.Http2Connection$IdleConnectionTimeoutEvent.handle(Http2Connection.java:204)
~[java.net.http:?]<br>
at
jdk.internal.net.http.HttpClientImpl.purgeTimeoutsAndReturnNextDeadline(HttpClientImpl.java:1689)
~[java.net.http:?]<br>
at
jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:1299)
~[java.net.http:?]<br>
<br>
<br>
2. Caused by: java.io.IOException: Stream
9 cancelled<br>
at
jdk.internal.net.http.Stream.cancel(Stream.java:1229)
~[java.net.http:?]<br>
at
jdk.internal.net.http.Http2Connection.putStream(Http2Connection.java:1237)
~[java.net.http:?]<br>
at
jdk.internal.net.http.Stream.registerStream(Stream.java:851)
~[java.net.http:?]<br>
at
jdk.internal.net.http.Http2Connection.registerNewStream(Http2Connection.java:1337)
~[java.net.http:?]<br>
at
jdk.internal.net.http.Http2Connection.sendFrame(Http2Connection.java:1361)
~[java.net.http:?]<br>
at
jdk.internal.net.http.Stream.sendHeadersAsync(Stream.java:823)
~[java.net.http:?]<br>
at
jdk.internal.net.http.Exchange.lambda$responseAsyncImpl0$8(Exchange.java:550)
~[java.net.http:?]<br>
at
jdk.internal.net.http.Exchange.checkFor407(Exchange.java:430)
~[java.net.http:?]<br>
at
jdk.internal.net.http.Exchange.lambda$responseAsyncImpl0$9(Exchange.java:554)
~[java.net.http:?]<br>
at
java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
~[?:?]<br>
at
java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:950)
~[?:?]<br>
at
java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2372)
~[?:?]<br>
at
jdk.internal.net.http.Exchange.responseAsyncImpl0(Exchange.java:554)
~[java.net.http:?]<br>
at
jdk.internal.net.http.Exchange.responseAsyncImpl(Exchange.java:406)
~[java.net.http:?]<br>
at
jdk.internal.net.http.Exchange.responseAsync(Exchange.java:398)
~[java.net.http:?]<br>
at
jdk.internal.net.http.MultiExchange.responseAsyncImpl(MultiExchange.java:409)
~[java.net.http:?]<br>
at
jdk.internal.net.http.MultiExchange.lambda$responseAsync0$2(MultiExchange.java:342)
~[java.net.http:?]<br>
at
java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
~[?:?]<br>
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
~[?:?]<br>
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773)
~[?:?]<br>
at
jdk.internal.net.http.HttpClientImpl$DelegatingExecutor.execute(HttpClientImpl.java:177)
~[java.net.http:?]<br>
at
java.util.concurrent.CompletableFuture.completeAsync(CompletableFuture.java:2719)
~[?:?]<br>
at
jdk.internal.net.http.MultiExchange.responseAsync(MultiExchange.java:295)
~[java.net.http:?]<br>
at
jdk.internal.net.http.HttpClientImpl.sendAsync(HttpClientImpl.java:983)
~[java.net.http:?]<br>
at
jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:875)
~[java.net.http:?]<br>
at
jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:133)
~[java.net.http:?]</font></div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</body>
</html>