RFR [9] 8148117: Move sun.misc.Cleaner to jdk.internal.ref

[Uwe Schindler]

(reposting with correct sender address)

Hi,

> Note: some popular open source libraries that hack into the internal
> private cleaner field of direct buffers will have to have their code
> updated, if they wish to continue to do this.

I followed parts of this issue, but as representative from Apache Lucene, which really really needs to un-mmap MappedByteBuffers (otherwise Lucene gets unuseable with JDK 9), I have no idea what the fix for us should be?

What is the replacement for the following code (see marked BufferCleaner impl, which is our abstraction): https://goo.gl/DGYZZj

The main reason why sun.misc.Cleaner was on the critical API list is NOT the case that somebody wanted to implement their own cleaner as replacement for finalization. The main reason (and this is *not* solved by the new APIs in java.lang.ref.Cleaner) is to *force* unmapping of direct and mmapped byte buffers.

The unmapping of mmapped buffers is really needed and must be in Java 9, otherwise projects like Lucene will not work with Java 9, or may be slow as hell (if they use non-mmapped I/O) or otherwise run out of disk space (because mmapped files use disk space as they are never ever freed by the GC - see below). On Windows, in addition, files that are mmapped cannot be closed. These problem are the worst desaster that can happen to Lucene - a desaster like the big Hotspot issues back in 2011.

With this patch the above code part would no longer work, because you can still make the getCleaner method available with setAccessible in Java 9 Jigsaw, but invoking the clean() method on the cleaner will horribly fail, because the cleaner is part of an internal package.

So you have 2 possibilities:
- Re-add sun.misc.Cleaner with a limited API (only the public clean() method) and let DirectBuffer return an instance of this sun.misc.Cleaner public API, so it is accessible without breaking encapsulation by adding extra exports
- Or fix the full issue: Add "official" support to unmmap mmapped files. Really! You have to do this, otherwise it's a desaster (see below for the "why").

Mark Reinhold: I will as every year contact you on the FOSDEM dinner about sun.misc.Cleaner and unmapping DirectBuffers :-) Be sure to have a heated discussion!

FYI: The reason why unmapping mmapped buffers is really needed is the following: MappedByteBuffer/DirectBuffers are very small objects (GC wise). Those are created by Lucene on mapping huuuuuuge files (several tens or hundreds of Gigabytes for some large indexes). Those objects live for longer time (up to several hours). But from time to time, Lucene releases the buffers and maps new ones (when updates occur on disk). Because those small - only a few bytes for the Garbage Collector - buffer instances lived so long, there is no reason to free them asap by GC. In fact, they are never freed if your application runs for several days (as Lucene/SOlr/Elasticsearch servers are living). So the small 20 byte objects hang forever on gigabytes of data! No way to realease that!

So Lucene/Solr/Elasticserach's misuse of sun.misc.Cleaner is just to force the DirectBuffers to free the huge amounts of resources they hold. The new API of java.lang.ref.Cleaner does not help at all with this. You are just hiding the only possible workaround!

Sorry for the bad news, but there needs to be a decision! We are great supporters of Jigsaw, but if we have to disable Jigsaw encapsulation, just to get access to jdk.internal.**.Cleaner, you are destroying the additional encapsulation for us!

Please add a new issue to either revert to have sun.misc.Cleaner available just to forcefully free mmapped files, or add an official API to allow unmapping (including Hotspot changes to prevent SIGSEGV/SIGBUS).

Uwe

-----
Uwe Schindler
uschindler at apache.org 
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
http://lucene.apache.org/

> -----Original Message-----
> From: core-libs-dev [mailto:core-libs-dev-bounces at openjdk.java.net] On
> Behalf Of Chris Hegarty
> Sent: Saturday, January 23, 2016 5:17 PM
> To: core-libs-dev <core-libs-dev at openjdk.java.net>
> Subject: RFR [9] 8148117: Move sun.misc.Cleaner to jdk.internal.ref
> 
> sun.misc.Cleaner was previously listed as a critical internal API in JEP
> 260 [1], but on further investigation it has been moved to an open issue,
> for the following reasons:
>   1) its primary use in the JDK is within NIO direct buffers to release
>      native memory. The base module cannot have a dependency on the
>      jdk.unsupported module so will need to be updated to use an
>      alternative cleaner,
>   2) the usage of Cleaner outside the JDK, as determined by corpus
>      analysis, has largely been observed to hack into private fields of
>      the internal NIO direct buffer classes to explicitly release native
>      memory.
> 
> As stated in 1), the type of the cleaner used by NIO direct buffers will
> have to change. Given this, and the fact that JDK 9 has a new general
> purposed cleaner API, java.lang.ref.Cleaner [2] the value of keep
> sun.misc.Cleaner is questionable.
> 
> This issue proposes to simply move Cleaner into an internal non-exported
> package in the base module so that it can be used by the NIO direct
> buffers classes, and small number of other places.
> 
> Webrev:
>   http://cr.openjdk.java.net/~chegar/8148117/
> 
> 
> If, at some point in the future, it is determined that sun.misc.Cleaner
> is in fact a critical internal API ( with static usage ), then it can be
> reinstated as a subtype of jdk.internal.ref.Cleaner, providing the same
> public API.
> 
> Note: some popular open source libraries that hack into the internal
> private cleaner field of direct buffers will have to have their code
> updated, if they wish to continue to do this.
> 
> -Chris.
> 
> [1] https://bugs.openjdk.java.net/browse/JDK-8132928
> [1] https://bugs.openjdk.java.net/browse/JDK-8138696