<div dir="ltr"><div><div><div><div>Hi Jaikiran!<br><br></div>Thanks for your quick feedback!<br><br></div>It happens on a regular and consistent basis mostly when execution health checks between microservices. On about 100 microservices deployed on 100 VM, it happens ~400 times every hours.<br><br></div>I haven't find a consistent pattern for now. I cannot attach  the whole stacktrace as it is a closed source code. I can provide some details. A ScheduleExecutor is launching a check on healthiness HTTP call every 29s (was 30s but I desync it to avoid idle timeout of 30s alignment with HttpClient). This is the biggest source of those errors but not the only one, there are regular HTTP calls received on a Jetty server and calling another service that are failing.<br><br></div><div>I have already tried to enable full debug without much help for now. Is there something in particular you are willing to check?<br></div><div><br></div><div>Best regards,</div><div><br></div><div>Nicolas<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 20 Jul 2023 at 08:11, Jaikiran Pai <<a href="mailto:jai.forums2013@gmail.com">jai.forums2013@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
  
    
  
  <div>
    <p>Hello Nicolas,</p>
    <p>This looks like it will need to be investigated. I've opened
      <a href="https://bugs.openjdk.org/browse/JDK-8312433" target="_blank">https://bugs.openjdk.org/browse/JDK-8312433</a> to tracking this. It
      would help further if you could attach the complete exception logs
      along with timestamps, if any.<br>
    </p>
    <p>Is this consistently reproducible? Perhaps you could launch your
      application passing it the -Djdk.internal.httpclient.debug=true
      system property? That will generate very verbose logs, so you
      might want to use that sparingly and remove that system property
      when you have collected the logs for this issue. <br>
    </p>
    <p>-Jaikiran<br>
    </p>
    <div>On 20/07/23 11:00 am, Nicolas Henneaux
      wrote:<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr">
        <div>
          <div>Hi all,<br>
            <br>
          </div>
          Since I updated Java from 19.0.2 to 20.0.1, I got some errors
          with connection idle shutdown or closed stream usage when
          trying to send HTTP requests with java.net.http.HttpClient.<br>
          <br>
        </div>
        <div>The two following errors are happening. I suspect those are
          linked even if they don't trigger together. <br>
        </div>
        <div>1. <font size="1">java.net.http.HttpConnectTimeoutException:
            HTTP connection idle, no active streams. Shutting down.<br>
            at
            jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:890)
            ~[java.net.http:?]</font></div>
        <div>2. <font size="1"> java.io.IOException: Stream 9 cancelled<br>
            at jdk.internal.net.http.Stream.cancel(Stream.java:1229)
            ~[java.net.http:?]</font></div>
        <div>The detailed stacks are copied below.</div>
        <div><br>
        </div>
        <div>Error 1. is caused by an idle connection trying to be used
          to send the request from what I understood from the code.
          Error 2. despite saying it is a cancelled streamed is actually
          happening when the stream is closed if I haven't missed
          something in the code.</div>
        <div>I suspect either the idle connections are not always
          properly discarded to create a new one or there is a race
          condition when discarding the idle connections. </div>
        <div><br>
        </div>
        <div>I was not able so far to find where this could happen and I
          hope you could help me to find where the problem lies.</div>
        <div><br>
        </div>
        <div>Thank you in advance for your help,</div>
        <div><br>
        </div>
        <div>Best regards,</div>
        <div><br>
        </div>
        <div>Nicolas Henneaux<br>
        </div>
        <div><font size="1"><br>
          </font></div>
        <div><font size="2">Detailed stacks<br>
          </font></div>
        <div><font size="1">1. Caused by:
            java.net.http.HttpConnectTimeoutException: HTTP connection
            idle, no active streams. Shutting down.<br>
            at
            jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:890)
            ~[java.net.http:?]<br>
            at
            jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:133)
            ~[java.net.http:?]<br>
            (...)<br>
            Caused by: java.net.http.HttpConnectTimeoutException: HTTP
            connection idle, no active streams. Shutting down.<br>
            at
jdk.internal.net.http.Http2Connection$IdleConnectionTimeoutEvent.handle(Http2Connection.java:204)
            ~[java.net.http:?]<br>
            at
jdk.internal.net.http.HttpClientImpl.purgeTimeoutsAndReturnNextDeadline(HttpClientImpl.java:1689)
            ~[java.net.http:?]<br>
            at
jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:1299)
            ~[java.net.http:?]<br>
            <br>
            <br>
            2. Caused by: java.io.IOException: Stream 9 cancelled<br>
            at jdk.internal.net.http.Stream.cancel(Stream.java:1229)
            ~[java.net.http:?]<br>
            at
jdk.internal.net.http.Http2Connection.putStream(Http2Connection.java:1237)
            ~[java.net.http:?]<br>
            at
            jdk.internal.net.http.Stream.registerStream(Stream.java:851)
            ~[java.net.http:?]<br>
            at
jdk.internal.net.http.Http2Connection.registerNewStream(Http2Connection.java:1337)
            ~[java.net.http:?]<br>
            at
jdk.internal.net.http.Http2Connection.sendFrame(Http2Connection.java:1361)
            ~[java.net.http:?]<br>
            at
            jdk.internal.net.http.Stream.sendHeadersAsync(Stream.java:823)
            ~[java.net.http:?]<br>
            at
jdk.internal.net.http.Exchange.lambda$responseAsyncImpl0$8(Exchange.java:550)
            ~[java.net.http:?]<br>
            at
            jdk.internal.net.http.Exchange.checkFor407(Exchange.java:430)
            ~[java.net.http:?]<br>
            at
jdk.internal.net.http.Exchange.lambda$responseAsyncImpl0$9(Exchange.java:554)
            ~[java.net.http:?]<br>
            at
java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
            ~[?:?]<br>
            at
java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:950)
            ~[?:?]<br>
            at
java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2372)
            ~[?:?]<br>
            at
            jdk.internal.net.http.Exchange.responseAsyncImpl0(Exchange.java:554)
            ~[java.net.http:?]<br>
            at
            jdk.internal.net.http.Exchange.responseAsyncImpl(Exchange.java:406)
            ~[java.net.http:?]<br>
            at
            jdk.internal.net.http.Exchange.responseAsync(Exchange.java:398)
            ~[java.net.http:?]<br>
            at
jdk.internal.net.http.MultiExchange.responseAsyncImpl(MultiExchange.java:409)
            ~[java.net.http:?]<br>
            at
jdk.internal.net.http.MultiExchange.lambda$responseAsync0$2(MultiExchange.java:342)
            ~[java.net.http:?]<br>
            at
java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
            ~[?:?]<br>
            at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
            ~[?:?]<br>
            at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773)
            ~[?:?]<br>
            at
jdk.internal.net.http.HttpClientImpl$DelegatingExecutor.execute(HttpClientImpl.java:177)
            ~[java.net.http:?]<br>
            at
java.util.concurrent.CompletableFuture.completeAsync(CompletableFuture.java:2719)
            ~[?:?]<br>
            at
jdk.internal.net.http.MultiExchange.responseAsync(MultiExchange.java:295)
            ~[java.net.http:?]<br>
            at
            jdk.internal.net.http.HttpClientImpl.sendAsync(HttpClientImpl.java:983)
            ~[java.net.http:?]<br>
            at
            jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:875)
            ~[java.net.http:?]<br>
            at
            jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:133)
            ~[java.net.http:?]</font></div>
      </div>
    </blockquote>
  </div>

</blockquote></div>