Very long response headers and java.net.http.HttpClient?

Wed Jul 24 20:13:27 UTC 2024

Hello,

I'm documenting some guidelines for using java.net.http.HttpClient
defensively for my team. For example: "Always set a request timeout",
"Don't assume HTTP response entities are small and/or will fit in memory",
etc.

One guideline I'd like to document is "Set a maximum for HTTP response
header size." However, I can't seem to find a way to set that limit, either
in documentation or in OpenJDK code.

I tried my best to search the archives for this mailing list for any
mentions, but came up empty.

To make sure my head is on straight and there isn't an undocumented limit
set by default, I wrote the attached (very quick and dirty) client and
server programs. LongResponseHeaderDemoServer opens a raw server socket and
reads (what it assumes is) a well-formed HTTP request, and then prints an
HTTP response which includes a response header of infinite length.
LongResponseHeaderDemoHttpClient uses java.net.http.HttpClient to make a
request and print the response body.

When I run LongResponseHeaderDemoServer in one terminal and make a curl
request to the server in another terminal, this is what curl spits out:

$ curl -vvv -D - http://localhost:3000
* Host localhost:3000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:3000...
* Connected to localhost (::1) port 3000
> GET / HTTP/1.1
> Host: localhost:3000
> User-Agent: curl/8.6.0
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Content-Type: text/plain
Content-Type: text/plain
< Connection: close
Connection: close
< Content-Length: 3
Content-Length: 3
* Closing connection
curl: (100) A value or data field grew larger than allowed

So curl detects the long response header and bails out. Safe and sane.

However, when I run LongResponseHeaderDemoServer in one terminal and
run LongResponseHeaderDemoHttpClient in another terminal, this is what
happens:

$ java LongResponseHeaderDemoHttpClient
Exception in thread "main" java.io.IOException: Requested array size
exceeds VM limit
at
java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:966)
at
java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:133)
at LongResponseHeaderDemoHttpClient.main(LongResponseHeaderDemoHttpClient
.java:13)
Caused by: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at java.base/java.util.Arrays.copyOf(Arrays.java:3541)
at
java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:242)
at
java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:806)
at java.base/java.lang.StringBuilder.append(StringBuilder.java:246)
at
java.net.http/jdk.internal.net.http.Http1HeaderParser.readResumeHeader(Http1HeaderParser.java:250)
at
java.net.http/jdk.internal.net.http.Http1HeaderParser.parse(Http1HeaderParser.java:124)
at
java.net.http/jdk.internal.net.http.Http1Response$HeadersReader.handle(Http1Response.java:605)
at
java.net.http/jdk.internal.net.http.Http1Response$HeadersReader.handle(Http1Response.java:536)
at
java.net.http/jdk.internal.net.http.Http1Response$Receiver.accept(Http1Response.java:527)
at
java.net.http/jdk.internal.net.http.Http1Response$HeadersReader.tryAsyncReceive(Http1Response.java:583)
at
java.net.http/jdk.internal.net.http.Http1AsyncReceiver.flush(Http1AsyncReceiver.java:233)
at
java.net.http/jdk.internal.net.http.Http1AsyncReceiver$$Lambda/0x00000008010dbd50.run(Unknown
Source)
at
java.net.http/jdk.internal.net.http.common.SequentialScheduler$LockingRestartableTask.run(SequentialScheduler.java:182)
at
java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:149)
at
java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:207)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.runWith(Thread.java:1596)
at java.base/java.lang.Thread.run(Thread.java:1583)

Ostensibly, HttpClient just keeps on reading the never-ending header until
it OOMs. This seems to confirm that there is no default limit to header
size. It also seems like A Very Bad Thing to me. This suggests that any
time a program makes an HTTP request to an untrusted source using
HttpClient, for example when crawling the web, they are at risk of an OOM.

For grins, I also wrote an
application LongResponseHeaderDemoHttpURLConnection that does the same
thing as LongResponseHeaderDemoHttpClient, just using HttpURLConnection
instead of HttpClient. When I run LongResponseHeaderDemoServer in one
terminal and LongResponseHeaderDemoHttpURLConnection in another terminal,
this is what happens:

$ java LongResponseHeaderDemoHttpURLConnection
Exception in thread "main" java.lang.NegativeArraySizeException: -1610612736
at java.base/sun.net.www.MessageHeader.mergeHeader(MessageHeader.java:526)
at java.base/sun.net.www.MessageHeader.parseHeader(MessageHeader.java:481)
at
java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:804)
at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:726)
at
java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1688)
at
java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589)
at java.base/java.net.URL.openStream(URL.java:1161)
at LongResponseHeaderDemoHttpURLConnection.main(
LongResponseHeaderDemoHttpURLConnection.java:12)

So HttpURLConnection doesn't handle things gracefully either, but at least
it doesn't OOM. That seems like a bug, too, but perhaps less severe.

For reference, here's my java version:

$ java -version
openjdk version "21.0.2" 2024-01-16 LTS
OpenJDK Runtime Environment Corretto-21.0.2.13.1 (build 21.0.2+13-LTS)
OpenJDK 64-Bit Server VM Corretto-21.0.2.13.1 (build 21.0.2+13-LTS, mixed
mode, sharing)

Can anyone check my work, and maybe reproduce? And ideally, can someone
with more knowledge than me about java.net.http.HttpClient and/or
java.net.HttpURLConnection please comment? Is this real, or have I made a
mistake somewhere along the way? If it's real, what's next? A bug report?

Andy Boothe
*Email*: andy.boothe at gmail.com
*Mobile*: (979) 574-1089
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jdk-dev/attachments/20240724/eadf1a2f/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: LongResponseHeaderDemoHttpClient.java
Type: application/octet-stream
Size: 587 bytes
Desc: not available
URL: <https://mail.openjdk.org/pipermail/jdk-dev/attachments/20240724/eadf1a2f/LongResponseHeaderDemoHttpClient-0001.java>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: LongResponseHeaderDemoHttpURLConnection.java
Type: application/octet-stream
Size: 508 bytes
Desc: not available
URL: <https://mail.openjdk.org/pipermail/jdk-dev/attachments/20240724/eadf1a2f/LongResponseHeaderDemoHttpURLConnection-0001.java>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: LongResponseHeaderDemoServer.java
Type: application/octet-stream
Size: 1687 bytes
Desc: not available
URL: <https://mail.openjdk.org/pipermail/jdk-dev/attachments/20240724/eadf1a2f/LongResponseHeaderDemoServer-0001.java>