Multi-Release JAR file patch as applied to build 108 of Java 9 breaks almost every project out there (Apache Ant, Gradle, partly Apache Maven)

Sat Mar 5 14:03:36 UTC 2016

Hi Claes,

is there a way to just build a new runtime library without compiling a full JDK (including Hotspot). So just replacing the jimage files locally?

Uwe

-----

Uwe Schindler

uschindler at apache.org 

ASF Member, Apache Lucene PMC / Committer

Bremen, Germany

http://lucene.apache.org/

From: Claes Redestad [mailto:claes.redestad at oracle.com] 
Sent: Saturday, March 05, 2016 2:50 PM
To: Uwe Schindler <uschindler at apache.org>; core-libs-dev at openjdk.java.net
Cc: rory.odonnell at oracle.com; dev at ant.apache.org; bodewig at apache.org
Subject: Re: Multi-Release JAR file patch as applied to build 108 of Java 9 breaks almost every project out there (Apache Ant, Gradle, partly Apache Maven)

Hi,

similar issues were discovered too late to stop b108, e.g., https://bugs.openjdk.java.net/browse/JDK-8150920. Fix is already in jdk9/dev, so I think the next build should be more well-behaved and hope we can provide it more promptly than normal.

If you can build OpenJDK from jdk9/dev and report any remaining issues due to the multi-release feature that would be quite helpful!

Thanks!

/Claes

Uwe Schindler <uschindler at apache.org <mailto:uschindler at apache.org> > skrev: (5 mars 2016 14:24:37 CET)

Hi OpenJDK Core Developers,

you may know the Apache Lucene team is testing early access releases of Java 9. We reported many bugs already, but most of them only applied to Hotspot and Lucene itsself. But this problem since build 108 is now really severe, because it breaks the build system already!

To allow further testing of Open Source Projects, I'd suggest to revert the Multi-Release-JAR runtime support patch and provide a new preview build ASAP, because we found out after a night of debugging a build system from which we don't know all internals what is causing the problems and there is no workaround. I am very sorry that I have to say this, but it unfortunately build 108 breaks *ALL* versions of Apache Ant, the grandfather of all Java build systems :-) I know also OpenJDK is using it, too! So with Multi-Release JAR file patch applied (see http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/f9913ea0f95c), any Ant-based build - including the JDK build itsself - would no longer bootstrap. It is impossible to also build Gradle projects, because Gradle uses Ant internally for many tasks). Maven projects may be affected, too.

Now you might have the question: What happened?

We tried to build Lucene on our Jenkins server, but the build itsself failed with a stupid error message:

BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:21: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/common-build.xml:56: not doesn't support the nested "matches" element.

The first idea was: Ah, there were changes in XML parsing (JDK-8149915). So we debugged the build. But it was quite clear that XML parsing was not the issue. It got quite clear when we enabled "-debug" on the build. What happened was that Ant was not loading its internal conditions/tasks/type definitions anymore, so the build system does not know almost any type anymore. The debug log showed that Ant was no longer able to load the resource "/org/apache/tools/ant/antlib.xml" from its own JAR file anymore. Instead it printed some strange debugging output (which looked totally broken).

I spend the whole night digging through their code and found the issue: The commit of Multi-Release-Jar files (see http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/f9913ea0f95c) broke resource handling in Apache Ant. In short: If you call ClassLoader.getResources() / or getResource() you get back an URL from where you can load the Resource - this is all fine and still works. But, with the Multi-Release JAR files patch this now has an URL fragment appended to the URL: '#release' (see http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/f9913ea0f95c); this also applies to non-multi-release JAR files like Apache Ant's "ant.jar".

In Java 7, Java 8,... and Java 9pre-b108, ClassLoader.getResource()/getResources() returned stuff like: 

"jar:file:/C:/Program%20Files/Java/apache-ant-1.9.6/lib/ant.jar!/org/apache/tools/ant/antlib.xml"

Now in Java 9b108 the following is returned:

"jar:file:/C:/Program%20Files/Java/apache-ant-1.9.6/lib/ant.jar!/org/apache/tools/ant/antlib.xml#release"

And here Ant breaks (and I assume many other projects like Maven, too). Ant checks for the file extension of the string (because it may load definitions from both XML and properties files). So it does endsWith(".xml") and of course this now returns false. The effect is that Ant tries to load its own task definitions as a java properties file instead of XML. Of course this fails, because the data behind this URL is XML. The effect is that Ant cannot bootstrap as everything to build is missing.

One might say: Ant's code is broken (I agree, it is not nice because it relies on the string representation of the resource URL - which is a no-go anyways), but it is impossible to fix, because Ant is bundled on most developer computers and those will suddenly break with Java 9! There is also no version out there that works around this, so we cannot test anything anymore!

The problematic line in Ant's code is here: http://grepcode.com/file/repo1.maven.org/maven2/org.apache.ant/ant/1.9.6/org/apache/tools/ant/taskdefs/Definer.java?av=f#259

I'd suggest to please ASAP revert the Multi-Release JAR file patch and provide a new preview build as soon as possible. I think there is more work needed to fix this. If this does not revert to the original state, it will be impossible to build and test Lucene, Elasticsearch,.... (and almost every Java project out there!). So short: We cannot test anymore and it is likely that we cannot support Java 9 anymore because the build system used by most Java projects behind the scenes does not bootstrap itself anymore.

My suggestion would be to investigate other versions for this patch that does *not* modify the resource URLs by appending a fragment to them (at least not for the "standard" case without an actual Multi-Release Jar). For new multi-release JAR files I am fine with appending fragments, but please not for default ones. Maybe change code to handle the URLs from the non-versioned part differently (without fragment). Leaving the fragment inide may break many othe rprojects, because many programmers are very sloppy with handling URLs (well-known issue is calling URL#getFile() of a file:-URL that breaks on Windows systems and spaces in path name). Many people just call toString() on URL and do some test on it (startsWith, endsWith). So appending fragments is a no-go for backwards compatibility with JAR resources!

I posted this to the mailing list and did not open a bug report on http://bugs.java.com/, because this is a more general issue - feel free to open bug reports around this!!! I would be very happy if we could find a quick solution for this problem. Until there is a solution we have to stop testing Java 9 with Apache Lucene/Solr/..., and this is not a good sign, especially as Jigsaw will be merged soon.

Thanks for listening,
Uwe

P.S.: I also CCed the Apache Ant team. They should fix the broken code anyways, but this won't help for many projects already out there (e.g. Apache Lucene still has a minimum requirement of Ant 1.8.2 because MacOSX computers ship with that version since years).

-----
Uwe Schindler
 <mailto:uschindler at apache.org> uschindler at apache.org 
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
 <http://lucene.apache.org> http://lucene.apache.org/

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.