[httpclient] HTTP2: Memory Leak with Proxy

[Chris Hegarty]

Albert,

I haven’t yet looked at what happens in JDK 10, but just to say, since things have moved on a lot in JDK 11 EA, that the same test runs with a reasonable amount of memory and CPU with JDK 11 EA [1]. I updated the test a little, since some the names have been changed with the standardisation ( see javadoc for more details [2] ).

-Chris.

[1] http://jdk.java.net/11/
[2] https://download.java.net/java/early_access/jdk11/docs/api/java.net.http/java/net/http/package-summary.html

---

$ cat BadProxyLeak.java 

import java.net.InetSocketAddress;
import java.net.ProxySelector;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.util.concurrent.TimeUnit;

public class BadProxyLeak {
    public static void main(String[] args) throws InterruptedException {
        // 1st heap dump
//        Thread.sleep(10000);

        {
            HttpClient client = HttpClient.newBuilder()
                    // malicious proxy?
                    .proxy(ProxySelector.of(new InetSocketAddress("165.165.248.90", 8080)))
                    .build();

            HttpRequest req = HttpRequest
                    // target is not relevant
                    .newBuilder(URI.create("https://www.google.de"))
                    // leak occurs only with HTTP_2
                    .version(HttpClient.Version.HTTP_2)
//                    .version(HttpClient.Version.HTTP_1_1)
                    .GET()
                    .build();


            // use same client for every request
            // most likely happens on the first retry, if not try again or increase i
            for (int i = 0; i < 10; i++) {
                System.out.println("Request " + i);
                // body handler is not relevant
                HttpResponse.BodyHandler<?> t = HttpResponse.BodyHandlers.ofString();

                try {
                    // happens with both async and sync send
                    client.sendAsync(req, t).get(15, TimeUnit.SECONDS);
//                client.send(req, handler);
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
        }

        // 3rd heap dump
        System.out.println("Generating heap dump manually");
        // space is not released
        Thread.sleep(60000);
    }
}


> On 7 Aug 2018, at 17:05, Chris Hegarty <chris.hegarty at oracle.com> wrote:
> 
> 
>> On 7 Aug 2018, at 16:55, Albert Schimpf <albi646 at gmx.de> wrote:
>> 
>> Hi,
>> 
>> by bad I mean that this proxy is the only one out of ~1000 proxies which causes this behavior. It's also the only one which causes SSLExceptions (General SSL engine problem) and EOFExceptions. I don't think that particular proxy is a valid proxy. Using curl indicates that this is a SSL handshake problem (https):
>> 
>> * Rebuilt URL to: www.google.de/
>> *   Trying 165.165.248.90…
> 
> Oh, you mean the proxy at that actual IP address. Ok got it.
> 
> 
>> * TCP_NODELAY set
>> * Connected to 165.165.248.90 (165.165.248.90) port 8080 (#0)
>> * successfully set certificate verify locations:
>> *   CAfile: /etc/ssl/certs/ca-certificates.crt
>>  CApath: /etc/ssl/certs
>> * TLSv1.2 (OUT), TLS handshake, Client hello (1):
>> * OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 165.165.248.90:8080
>> * Closing connection 0
>> curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 165.165.248.90:8080
>> 
>> I don't know that much about the protocol itself,  so I don't think I can help very much.
>> 
>> Thank you for looking into this!
> 
> 
> -Chris.
> 
> 
>> Best,
>> Albert
>> 
>> 
>> On 07.08.2018 12:05, Chris Hegarty wrote:
>>> Hi Albert,
>>> 
>>> Very strange indeed. Thanks for reporting it, I’ll investigate.
>>> 
>>> What do your mean by “bad proxy”. What is bad about it, and how does it behave?
>>> 
>>> -Chris.
>>> 
>>> 
>>>> On 7 Aug 2018, at 10:25, Albert Schimpf <albi646 at gmx.de> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> I stumbled upon some strange behavior when using the new Java httpclient.
>>>> 
>>>> The issue is very simple to reproduce. Send a GET request via a known bad proxy:
>>>> 
>>>> HttpClient client = HttpClient.newBuilder()
>>>>                    .proxy(ProxySelector.of(BAD_PROXY))
>>>>                    .build();
>>>> 
>>>> HttpRequest req = HttpRequest
>>>>                    // target is not relevant
>>>>                    .newBuilder(...)
>>>>                    .GET()
>>>>                    .build();
>>>> 
>>>> // body handler is not relevant
>>>> HttpResponse.BodyHandler<?> t = HttpResponse.BodyHandler.asString();
>>>> 
>>>> // happens with both async and sync send
>>>> client.sendAsync(req, t).get(30, TimeUnit.SECONDS);
>>>> 
>>>> The result is that the heap size increases dramatically (to about 1.5GB) and resources are not released. CPU consumption increases by a constant factor, too. I have tried many variations of the above code, and the only thing which seems to work (i.e. heap size does not explode) is to set the HTTP version to 1.1.
>>>> 
>>>> In my main application this leads to both memory and CPU starvation (4GB memory limit, 100% CPU usage). It usually uses only 5% CPU and 200MB memory at worst.
>>>> 
>>>> I have attached a working example code with a bad proxy. I uploaded the generated garbage collection log and three heap dumps (before, during, and after the request) to dropbox:
>>>> 
>>>> https://www.dropbox.com/s/ulqnmrmgr58rrul/debug.zip
>>>> 
>>>> I tried the 10.0.0-openjdk and 10.0.1-zulu version. I can reproduce the issue 100% of times.
>>>> 
>>>> Am I doing something wrong? Is this to be expected if one somehow happens to use a bad proxy? If this is to be expected, how can I protect my application against such behavior?
>>>> 
>>>> 
>>>> Best,
>>>> Albert
>>>> <mon.png><Main.java>
>> 
>