Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch
Stephen Felts
stephen.felts at oracle.com
Fri Dec 9 23:01:39 UTC 2016
Gradle/groovy are known to have problems with the restricted module access of b148.
You can get around this specific problem by using the environment variable _JAVA_OPTIONS=--add-opens=java.base/java.lang=ALL-UNNAMED
The packages that you need to open depend on what Java methods your gradle/groovy files happen to step on.
In my case, I need to use the following for our gradle build.
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens=java.base/java.net=ALL-UNNAMED
--add-opens=java.base/java.text=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
--add-opens=java.base/java.util.regex=ALL-UNNAMED
--add-opens=java.base/java.util.jar=ALL-UNNAMED
--add-opens=java.base/java.util.zip=ALL-UNNAMED
--add-opens=java.base/javax.net=ALL-UNNAMED
--add-opens=java.base/sun.net.www=ALL-UNNAMED
--add-opens=java.base/sun.net.www.protocol.file=ALL-UNNAMED
--add-opens=java.base/sun.nio.fs=ALL-UNNAMED
--add-opens=java.xml/javax.xml.transform=ALL-UNNAMED
--add-opens=java.xml/javax.xml.transform.stream=ALL-UNNAMED
--add-opens=java.xml/com.sun.org.apache.xalan.internal.xsltc=ALL-UNNAMED
--add-opens=java.xml/com.sun.org.apache.xalan.internal.xsltc.compiler=ALL-UNNAMED
--add-opens=java.xml/com.sun.org.apache.xalan.internal.xsltc.trax=ALL-UNNAMED
-----Original Message-----
From: Uwe Schindler [mailto:uschindler at apache.org]
Sent: Friday, December 09, 2016 5:33 PM
To: jigsaw-dev at openjdk.java.net; Core-Libs-Dev
Subject: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch
Hi,
I updated our Jenkins server for the JDK 9 preview testing to use build 148. Previously we had build 140 and build 147, which both worked without any issues. But after the update the following stuff goes wrong:
(1) Unmapping of direct buffers no longer works, although this API was marked as critical because there is no replacement up to now, so code can unmap memory mapped files, which is one of the most important things Apache Lucene needs to use to access huge random access files while reading the index. Without memory mapping, the slowdown for Lucene users will be huge
This is caused by the recent Jigsaw changes, published in build 148. Unfortunately we did not test the Jigsaw builds, so we would have noticed that earlier. Basically the following peace of code fails now (with or without doPrivileged and with/without security manager):
final Class<?> directBufferClass = Class.forName("java.nio.DirectByteBuffer");
final Method m = directBufferClass.getMethod("cleaner");
m.setAccessible(true);
MethodHandle directBufferCleanerMethod = lookup.unreflect(m);
Class<?> cleanerClass = directBufferCleanerMethod.type().returnType();
// build method handle for unmapping, full code is here: https://goo.gl/TfQWl6
>From my understanding previous lengthy discussions on OpenJDKs mailing lists (and a discussion on last FOSDEM), we agreed, that this unmapping code will still be supported with Java 9 (it was already adapted to work with Java 8 and Java 9 in Lucene, see the full code at https://goo.gl/TfQWl6) and Java 10 might have a new "official" unmapping API. On FOSDEM we already had some discussions how to implement this (with Andrew Haley and Mark Reinholds). We would have a strong interest to start some proposal about this (JEP or similar, but how to proceed). The idea was to allow unmapping in some special Hotspot state and trigger a safepoint, keywords in the discussion were "volatile only on safepoints".
So how to proceed with this. I know we can fix the issue with "--add-opens java.base/java.nio=ALL-UNNAMED", but for a product with huge impact like Elasticsearch or Solr, this is not a good way to go, especially as it will not work out of box anymore. From reading the module documentation, I had the feeling that there were some plans to support those "critical APIs" in the module definition (e.g., add exclusion from the checks for java.nio.DirectByteBuffer). An alternative is of course access to the (public) interface in the internal class sun.nio.ch.DirectBuffer, but we avoided this in our code, because it is also likely to break (I have not yet tried, if you have it declared as critical API).
I just refer to the following issues that explicitely added APIs to allow unmapping (jdk.internal.Cleaner implements Runnable,...):
https://bugs.openjdk.java.net/browse/JDK-8148117
https://bugs.openjdk.java.net/browse/JDK-8132928
http://cr.openjdk.java.net/~chegar/8148117/src/java.base/share/classes/jdk/internal/ref/Cleaner.java.udiff.html
(2) A second thing we noticed is that Groovy no longer works and dies with strange error messages. This does not affect Lucene's program code, but breaks our build system (Ant-based but with some scripts inside). Also Elasticsearch's usage of Gradle is affected.
We see this:
groovy.lang.MissingMethodException: No signature of method: static java.lang.Integer.valueOf() is applicable for argument types: (java.lang.String) values: [54] Possible solutions: plus(java.lang.String)
at groovy.lang.MetaClassImpl.invokeStaticMissingMethod(MetaClassImpl.java:1500)
at groovy.lang.MetaClassImpl.invokeStaticMethod(MetaClassImpl.java:1486)
at org.codehaus.groovy.runtime.callsite.StaticMetaClassSite.call(StaticMetaClassSite.java:53)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at embedded_script_in_C__Users_Uwe_Schindler_Projects_lucene_trunk_lusolr1_lucene_common_build_dot_xml$_run_closure1.doCall(embedded_script_in_C__Users_Uwe_Schindler_Projects_lucene_trunk_lusolr1_lucene_common_build_dot_xml:7)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:538)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1021)
at groovy.lang.Closure.call(Closure.java:426)
at groovy.lang.Closure.call(Closure.java:442)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.callClosureForLine(DefaultGroovyMethods.java:5236)
at org.codehaus.groovy.runtime.IOGroovyMethods.eachLine(IOGroovyMethods.java:487)
at org.codehaus.groovy.runtime.ResourceGroovyMethods.eachLine(ResourceGroovyMethods.java:292)
at org.codehaus.groovy.runtime.ResourceGroovyMethods.eachLine(ResourceGroovyMethods.java:257)
at org.codehaus.groovy.runtime.dgm$945.invoke(Unknown Source)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
at embedded_script_in_C__Users_Uwe_Schindler_Projects_lucene_trunk_lusolr1_lucene_common_build_dot_xml.run(embedded_script_in_C__Users_Uwe_Schindler_Projects_lucene_trunk_lusolr1_lucene_common_build_dot_xml:5)
at groovy.lang.GroovyShell.runScriptOrMainOrTestOrRunnable(GroovyShell.java:263)
at groovy.lang.GroovyShell.run(GroovyShell.java:518)
at groovy.lang.GroovyShell.run(GroovyShell.java:497)
at org.codehaus.groovy.ant.Groovy.parseAndRunScript(Groovy.java:512)
at org.codehaus.groovy.ant.Groovy.execGroovy(Groovy.java:448)
at org.codehaus.groovy.ant.Groovy.execute(Groovy.java:313)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:538)
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:392)
at org.apache.tools.ant.Target.performTasks(Target.java:413)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1399)
at org.apache.tools.ant.Project.executeTarget(Project.java:1368)
at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
at org.apache.tools.ant.Project.executeTargets(Project.java:1251)
at org.apache.tools.ant.Main.runBuild(Main.java:811)
at org.apache.tools.ant.Main.startAnt(Main.java:217)
at org.apache.tools.ant.launch.Launcher.run(Launcher.java:280)
at org.apache.tools.ant.launch.Launcher.main(Launcher.java:109)
I have no idea what is causing this, we are just users of Groovy - it just breaks our build.
For now we had to revert to build 147 of JDK 9 which works as it should. Any suggestions how to proceed? For Lucene the first issue is very important. Without unmapping support, Elasticsearch, Solr and Lucene cannot work with Java 9, sorry - but our users are strongly waiting for it (because of huge speed improvements in ByteBuffers, more security because of Jigsaw,...).
Uwe
-----
Uwe Schindler
HYPERLINK "mailto:uschindler at apache.org"uschindler at apache.org
ASF Member, Apache Lucene PMC / Committer Bremen, Germany http://lucene.apache.org/
More information about the jigsaw-dev
mailing list