<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
On 20/07/23 12:19 pm, Nicolas Henneaux wrote:<br>
<blockquote type="cite"
cite="mid:CALcOUbh9Ffnuuv9CZD+BAnjtcNPjX0Nchx3qyhNGMUai-qj2xg@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>
<div>
<div>
<div>Hi Jaikiran!<br>
<br>
</div>
Thanks for your quick feedback!<br>
<br>
</div>
It happens on a regular and consistent basis mostly when
execution health checks between microservices. On about 100
microservices deployed on 100 VM, it happens ~400 times
every hours.<br>
<br>
</div>
I haven't find a consistent pattern for now. I cannot attach
the whole stacktrace as it is a closed source code. I can
provide some details. A ScheduleExecutor is launching a check
on healthiness HTTP call every 29s (was 30s but I desync it to
avoid idle timeout of 30s alignment with HttpClient). This is
the biggest source of those errors but not the only one, there
are regular HTTP calls received on a Jetty server and calling
another service that are failing.<br>
<br>
</div>
<div>I have already tried to enable full debug without much help
for now. Is there something in particular you are willing to
check?<br>
</div>
</div>
</blockquote>
<p>Thank you for these additional details. I don't have any specific
log message in mind for now. I'll look into the code and try to
reproduce this in a test case.<br>
</p>
<p>From the stacktrace it appears that this affects HTTP2 connection
in your application. One workaround would be to increase the idle
connection timeout for these connections, to reduce the chances of
this issue. There are system properties that control this value.
If you want to configure this only for HTTP2 connections, then
jdk.httpclient.keepalive.timeout.h2 is the system property to set,
the value represents number of seconds (and defaults to 30). More
details about this and other system properties applicable for
HttpClient are available here
<a class="moz-txt-link-freetext" href="https://docs.oracle.com/en/java/javase/20/docs/api/java.net.http/module-summary.html">https://docs.oracle.com/en/java/javase/20/docs/api/java.net.http/module-summary.html</a></p>
<p>-Jaikiran<br>
</p>
<blockquote type="cite"
cite="mid:CALcOUbh9Ffnuuv9CZD+BAnjtcNPjX0Nchx3qyhNGMUai-qj2xg@mail.gmail.com">
<div dir="ltr">
<div><br>
</div>
<div>Best regards,</div>
<div><br>
</div>
<div>Nicolas<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, 20 Jul 2023 at 08:11,
Jaikiran Pai <<a href="mailto:jai.forums2013@gmail.com"
moz-do-not-send="true" class="moz-txt-link-freetext">jai.forums2013@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello Nicolas,</p>
<p>This looks like it will need to be investigated. I've
opened <a
href="https://bugs.openjdk.org/browse/JDK-8312433"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://bugs.openjdk.org/browse/JDK-8312433</a>
to tracking this. It would help further if you could
attach the complete exception logs along with timestamps,
if any.<br>
</p>
<p>Is this consistently reproducible? Perhaps you could
launch your application passing it the
-Djdk.internal.httpclient.debug=true system property? That
will generate very verbose logs, so you might want to use
that sparingly and remove that system property when you
have collected the logs for this issue. <br>
</p>
<p>-Jaikiran<br>
</p>
<div>On 20/07/23 11:00 am, Nicolas Henneaux wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>Hi all,<br>
<br>
</div>
Since I updated Java from 19.0.2 to 20.0.1, I got some
errors with connection idle shutdown or closed stream
usage when trying to send HTTP requests with
java.net.http.HttpClient.<br>
<br>
</div>
<div>The two following errors are happening. I suspect
those are linked even if they don't trigger together.
<br>
</div>
<div>1. <font size="1">java.net.http.HttpConnectTimeoutException:
HTTP connection idle, no active streams. Shutting
down.<br>
at
jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:890)
~[java.net.http:?]</font></div>
<div>2. <font size="1"> java.io.IOException: Stream 9
cancelled<br>
at
jdk.internal.net.http.Stream.cancel(Stream.java:1229)
~[java.net.http:?]</font></div>
<div>The detailed stacks are copied below.</div>
<div><br>
</div>
<div>Error 1. is caused by an idle connection trying to
be used to send the request from what I understood
from the code. Error 2. despite saying it is a
cancelled streamed is actually happening when the
stream is closed if I haven't missed something in the
code.</div>
<div>I suspect either the idle connections are not
always properly discarded to create a new one or there
is a race condition when discarding the idle
connections. </div>
<div><br>
</div>
<div>I was not able so far to find where this could
happen and I hope you could help me to find where the
problem lies.</div>
<div><br>
</div>
<div>Thank you in advance for your help,</div>
<div><br>
</div>
<div>Best regards,</div>
<div><br>
</div>
<div>Nicolas Henneaux<br>
</div>
<div><font size="1"><br>
</font></div>
<div><font size="2">Detailed stacks<br>
</font></div>
<div><font size="1">1. Caused by:
java.net.http.HttpConnectTimeoutException: HTTP
connection idle, no active streams. Shutting down.<br>
at
jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:890)
~[java.net.http:?]<br>
at
jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:133)
~[java.net.http:?]<br>
(...)<br>
Caused by:
java.net.http.HttpConnectTimeoutException: HTTP
connection idle, no active streams. Shutting down.<br>
at
jdk.internal.net.http.Http2Connection$IdleConnectionTimeoutEvent.handle(Http2Connection.java:204)
~[java.net.http:?]<br>
at
jdk.internal.net.http.HttpClientImpl.purgeTimeoutsAndReturnNextDeadline(HttpClientImpl.java:1689)
~[java.net.http:?]<br>
at
jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:1299)
~[java.net.http:?]<br>
<br>
<br>
2. Caused by: java.io.IOException: Stream 9
cancelled<br>
at
jdk.internal.net.http.Stream.cancel(Stream.java:1229)
~[java.net.http:?]<br>
at
jdk.internal.net.http.Http2Connection.putStream(Http2Connection.java:1237)
~[java.net.http:?]<br>
at
jdk.internal.net.http.Stream.registerStream(Stream.java:851)
~[java.net.http:?]<br>
at
jdk.internal.net.http.Http2Connection.registerNewStream(Http2Connection.java:1337)
~[java.net.http:?]<br>
at
jdk.internal.net.http.Http2Connection.sendFrame(Http2Connection.java:1361)
~[java.net.http:?]<br>
at
jdk.internal.net.http.Stream.sendHeadersAsync(Stream.java:823)
~[java.net.http:?]<br>
at
jdk.internal.net.http.Exchange.lambda$responseAsyncImpl0$8(Exchange.java:550)
~[java.net.http:?]<br>
at
jdk.internal.net.http.Exchange.checkFor407(Exchange.java:430)
~[java.net.http:?]<br>
at
jdk.internal.net.http.Exchange.lambda$responseAsyncImpl0$9(Exchange.java:554)
~[java.net.http:?]<br>
at
java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
~[?:?]<br>
at
java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:950)
~[?:?]<br>
at
java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2372)
~[?:?]<br>
at
jdk.internal.net.http.Exchange.responseAsyncImpl0(Exchange.java:554)
~[java.net.http:?]<br>
at
jdk.internal.net.http.Exchange.responseAsyncImpl(Exchange.java:406)
~[java.net.http:?]<br>
at
jdk.internal.net.http.Exchange.responseAsync(Exchange.java:398)
~[java.net.http:?]<br>
at
jdk.internal.net.http.MultiExchange.responseAsyncImpl(MultiExchange.java:409)
~[java.net.http:?]<br>
at
jdk.internal.net.http.MultiExchange.lambda$responseAsync0$2(MultiExchange.java:342)
~[java.net.http:?]<br>
at
java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
~[?:?]<br>
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
~[?:?]<br>
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773)
~[?:?]<br>
at
jdk.internal.net.http.HttpClientImpl$DelegatingExecutor.execute(HttpClientImpl.java:177)
~[java.net.http:?]<br>
at
java.util.concurrent.CompletableFuture.completeAsync(CompletableFuture.java:2719)
~[?:?]<br>
at
jdk.internal.net.http.MultiExchange.responseAsync(MultiExchange.java:295)
~[java.net.http:?]<br>
at
jdk.internal.net.http.HttpClientImpl.sendAsync(HttpClientImpl.java:983)
~[java.net.http:?]<br>
at
jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:875)
~[java.net.http:?]<br>
at
jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:133)
~[java.net.http:?]</font></div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</body>
</html>