Suggested fix for JDK-4724038 (Add unmap method to MappedByteBuffer)

[Uwe Schindler]

Hi,

> As I thought, the problem for some seems to be non-prompt unmapping of
> mapped address space held by otherwise unreachable mapped byte buffers.
> The mapped address space doesn't live in the Java heap and doesn't
> represent a heap memory pressure, so GC doesn't kick-in automatically
> when one would like. One could help by manually triggering GC with
> System.gc() in such situations. The problem is how to detect such situations.

Unfortunately, System#gc() is explicitely disallowed in most environments (because it performs a full GC): You should not use explicit GCs, because this hurts low-latency applications like search engines. So disabling explicit GCs should be done for such installations, e.g. external libraries tend to call System#gc() for no reason...

> Direct byte buffers (ByteBuffer.allocateDirect) maintain a count of bytes
> currently allocated and don't allow allocation of native memory beyond
> certain configured limit (-XX:MaxDirectMemorySize=<size>).
> Before throwing OutOfMemoryError, the  ByteBuffer.allocateDirect()
> request tries it's best to free direct memory allocated by otherwise
> unreachable direct ByteBuffers (using System.gc() to trigger GC and helping
> process references).

This code breaks if you disallow explicit GC. As Dawid says, I don't think the application should take care about GC.

> Would similar approach - configured limit for FileChannel.map()ped address
> space be of any help to Lucene applications? Is it possible to estimate the
> max. amount of address space a particular Lucene application may need at
> any one time so that mapping over such limit could be considered an
> application error?

This does not scale with index sizes going into the hundreds of Gigabytes. We cannot force the users to calculate their index size before using it and set corresponding JVM settings.

Uwe

-----
Uwe Schindler
uschindler at apache.org 
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
http://lucene.apache.org/