Provide API to explicitly unmap MappedByteBuffer?

Tue Mar 20 18:08:47 UTC 2018

Hi & thanks Alan,

I am talking with Andrew every year about that (on FOSDEM). Yes, he has a plan how to solve this, but it requires changes in the memory model. But it is really a misdesign in the original NIO concept to let the Garbage Collector handle the unmapping (I know this was done early in Java’s days when Grabage collectors were much more predicatble and System.gc() did something…). The main reason for the problem is: The ByteBuffer is a small object on heap (like 30 bytes), but holding resources up to 2 Gigabytes. If a service (running for days) is opening a file and mmaps it, the small object on heap is quickly going to old generation and then is stays there forever – the garbage collector has no idea that it is really heavy. No chance to get rid of it anymore, not even System.gc() helps in most configurations!

Until this problem is solved (hopefully *before* sun.misc.Unsafe leaves completely from accessible APIs), the Apache Lucene team continues to use the Cleaner API (in Java 8) and the new Unsafe method (Java 9+). We can not 100% prevent SIGSEGV or SIGBUS if Lucene’s code is used in wrong way (mainly if you close an IndexReader while another thread is searching te index at same time), but the “servers” on top of Lucene (like Apache Solr and Elasticsearch) take care of preventing that.

Nevertheless, I am the person who was responsible for adding this new method to sun.misc.Unsafe before Java 9 – I am so sorry about that! �� But sun.misc.Unsafe seemed to be the best place to keep this “unsafe” unmapping available to experts (guarded by security manager and setAccessible, of course).

I think we should really spend Andrew Haley some time to work on the issue. I am open to review his code and share comments. We (Apache Lucene/Solr & Elasticsearch) have strong requirements about the unmapping and hope to get this solved soon!

Uwe

-----

Uwe Schindler

uschindler at apache.org 

ASF Member, Apache Lucene PMC / Committer

Bremen, Germany

http://lucene.apache.org/

From: nio-dev <nio-dev-bounces at openjdk.java.net> On Behalf Of Alan Bateman
Sent: Tuesday, March 20, 2018 2:59 PM
To: Langer, Christoph <christoph.langer at sap.com>; nio-dev at openjdk.java.net
Subject: Re: Provide API to explicitly unmap MappedByteBuffer?

On 20/03/2018 10:39, Langer, Christoph wrote:

Hi,

I was just handling a customer issue where Java code tried to delete a file on Windows which was mapped into memory vial FileChannel.map() before. It took me a little while to understand the issue but obviously the problem was that Windows still kept a delete lock on the file as long as the ByteBuffer was still mapped. This behavior is also documented cleanly: https://download.java.net/java/jdk10/docs/api/java/nio/MappedByteBuffer.html.

So far so good, I just started wondering why there isn’t an API to force unmapping (cleanup) on a MappedByteBuffer? The app that opens the mapping simply reads a properties file at this place and I guess they should be able to change this place to use standard io easily and without observable performance problems. However, if they had a way to explicitly unmap the MappedByteBuffer it would also help.

Was this topic already discussed? Is there a hard reason why explicit cleanup should not be offered? I guess people using MappedByteBuffers should know anyway what they are doing…

This has been discussed many times, it's a really hard problem that many smart people have spend time on.

Andrew Haley may be able to say more about this. He has (or had) a proposal on this but it involved deep changes in the VM. I don't know he has prototyped it yet to see how feasible it is.

-Alan

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20180320/c613c0d4/attachment.html>