sun.nio.ch.Util: Don't cache an unlimited amount of memory

Tue Dec 29 17:47:43 UTC 2015

Hi,

There is more stuff affected by this: Also the „good old” RandomAccessFile or FileInput/OutputStream. Although those don’t use DirectBuffers internally, those allocate a native buffer with libc’s malloc() if the byte[] is larger than 8192 bytes. If you transfer a large byte[] using those old APIs it allocates (only for short time, so its better) a malloced buffer. But this buffer is freed after transfer, but its also an overhead as it fragments libc’s heap. If you buffer is < 8192 bytes it uses a buffer on stack, so when using those old APIs prefer to chunk at 8192 bytes.

We were affected by both problems in Lucene since the early days. E.g, we sometimes load norms for millions of documents from disk, which can be a Gigabyte of stuff. In early days this was loaded in a single read (using above old APIs, now with NIO.2 or mmap) – causing OOM because the malloc failed. But also with NIO this is a desaster, because we partly use heap buffers (unless mmap is used for Lucene indexes, which is preferred nowadays on 64 bits).

To work around those problems, Lucene’s IO API (wrapped around mmap, nio.2 file or channels) uses chunks automatically. E.g. when writing to an OutputStream, we automatically chunk in 8192 bytes (to not cause any mallocs in non-NIO code or any additional direct buffers). This is done using a wrapper stream (extends FilterOutputStream), e.g., see https://goo.gl/TijUFu for details. Very simple, very effective (if you use streams).

Uwe

-----

Uwe Schindler

uschindler at apache.org 

ASF Member, Apache Lucene PMC / Committer

Bremen, Germany

http://lucene.apache.org/

From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of Ariel Weisberg
Sent: Tuesday, December 29, 2015 5:57 PM
To: nio-dev at openjdk.java.net
Subject: Re: sun.nio.ch.Util: Don't cache an unlimited amount of memory

Hi,

Channel.new[Input | Output]Stream can trigger this as well. It's an odd thing to put in the javadoc there since this is an artifact of a specific Channel implementation and not Channels in general.

One solution that would be nice is to break down large writes into several large smaller writes so the per thread buffer size is at least bounded and not set to the size of the largest seen IO.

For sockets this is usually not a problem although I can imagine there is some implementation somewhere that will care. For files this is a bit more problematic especially if O_SYNC or O_DIRECT are in use. Filesystems on down also trigger different behaviors based off of read/write size so hiding the fact that something is one big IO can have an impact. My intuition is that something based on breaking down IOs is not going o pass muster, but it exists.

Could this be handled better with a different pooling approach that is not per thread over some threshold? There are a bunch of permutations that could trade off allocation/deallocation for concurrency vs. blocking threads waiting for operations to complete freeing up globally pooled buffers to bound peak footprint.

Regards,

Ariel

On Tue, Dec 29, 2015, at 10:59 AM, Evan Jones wrote:

No, we don't do scatter/gather I/O. Most Twitter services (Finagle which uses Netty 3) call NIO with variable sized heap ByteBuffers. The "leak" is caused by the fact that each thread caches a single direct ByteBuffer of the maximum size it has ever seen. Hence, if 0.01% of requests eventually cause a ~100 MB chunk to be sent, then each thread ends up with slowly growing native memory usage that never decreases, even if the "typical" size is ~1 MB. I still think it is very surprising and wrong that the JDK caches such enormous amounts of memory in this scenario. I would much rather have a "performance problem" that I can fix by managing my own buffers, than a mysterious native memory leak that is difficult to track down.

My (simple) proposal will have a performance impact for applications that do large I/O with heap ByteBuffers. However, I would argue those apps are already slow because of the copy, and probably won't notice :).

Would you be interested in a more sophisticated solution that would allow huge cached buffers to eventually expire? For example, if the "recent" usage of the cache has been much smaller than the allocated buffers, free the buffers and re-allocate? For the problematic application that I found, *any* policy for when to expire would effectively solve this problem, since the "peak" usage is significantly worse than the "average" usage.

My final suggestion: I would be happy to attempt to revise the javadoc about direct buffers to make it clearer that using heap buffers for I/O will cause copies, and will also cause the JDK to cache native memory. At least then we could argue that this behaviour is "as designed." :)

On Tue, Dec 29, 2015 at 4:17 AM, Alan Bateman < <mailto:Alan.Bateman at oracle.com> Alan.Bateman at oracle.com> wrote:

On 27/12/2015 20:35, Evan Jones wrote:

Summary: nio Util caches an unlimited amount of memory for temporary direct ByteBuffers, which effectively is a native memory leak. Applications that do large I/Os can inadvertently waste gigabytes of native memory or run out of memory. I suggest it should only cache a "small" amount of memory per-thread (e.g. 1 MB), and maybe have a flag to allow changing the limit for applications where this causes a performance regression.

1. Would JDK committers support this change?
2. If so, any suggestions for the default and/or how to override it with a flag?

Tony Printezis (CCed here) added a flag to Twitter's internal JVM/JDK to limit the size of this cache, which we could probably use that as a starting point for a patch.

Limiting the size of the buffer cache might help in some scenarios, it just means a bit more complexity and yet another tuning option.

Do you do scatter/gather I/O? The current implementation will cache up to IOV_MAX buffers per thread but if you aren't doing scatter/gather I/O then caching a maximum of one buffer per thread should reduce the memory usage. It wouldn't be hard to modify the BufferCache implementation to track the number of per-thread buffers in use so that this count is the maximum cached rather than IOV_MAX. I'm curious if you've looked into doing anything along those lines.

-Alan

-- 

Evan Jones

http://evanjones.ca/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20151229/6e9476d7/attachment.html>