Multi-Release JAR file patch as applied to build 108 of Java 9 breaks almost every project out there (Apache Ant, Gradle, partly Apache Maven)

[David Holmes]

On 5/03/2016 11:50 PM, Claes Redestad wrote:
> Hi,
>
> similar issues were discovered too late to stop b108, e.g., https://bugs.openjdk.java.net/browse/JDK-8150920. Fix is already in jdk9/dev, so I think the next build should be more well-behaved and hope we can provide it more promptly than normal.

As that bug leads to a non-open bug here's the changeset URL:

http://hg.openjdk.java.net/jdk9/dev/jdk/rev/721288127c82

David

> If you can build OpenJDK from jdk9/dev and report any remaining issues due to the multi-release feature that would be quite helpful!
>
> Thanks!
>
> /Claes
>
> Uwe Schindler <uschindler at apache.org> skrev: (5 mars 2016 14:24:37 CET)
>> Hi OpenJDK Core Developers,
>>
>> you may know the Apache Lucene team is testing early access releases of
>> Java 9. We reported many bugs already, but most of them only applied to
>> Hotspot and Lucene itsself. But this problem since build 108 is now
>> really severe, because it breaks the build system already!
>>
>> To allow further testing of Open Source Projects, I'd suggest to revert
>> the Multi-Release-JAR runtime support patch and provide a new preview
>> build ASAP, because we found out after a night of debugging a build
>> system from which we don't know all internals what is causing the
>> problems and there is no workaround. I am very sorry that I have to say
>> this, but it unfortunately build 108 breaks *ALL* versions of Apache
>> Ant, the grandfather of all Java build systems :-) I know also OpenJDK
>> is using it, too! So with Multi-Release JAR file patch applied (see
>> http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/f9913ea0f95c), any
>> Ant-based build - including the JDK build itsself - would no longer
>> bootstrap. It is impossible to also build Gradle projects, because
>> Gradle uses Ant internally for many tasks). Maven projects may be
>> affected, too.
>>
>> Now you might have the question: What happened?
>>
>> We tried to build Lucene on our Jenkins server, but the build itsself
>> failed with a stupid error message:
>>
>> BUILD FAILED
>> /home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:21: The
>> following error occurred while executing this line:
>> /home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/common-build.xml:56:
>> not doesn't support the nested "matches" element.
>>
>> The first idea was: Ah, there were changes in XML parsing
>> (JDK-8149915). So we debugged the build. But it was quite clear that
>> XML parsing was not the issue. It got quite clear when we enabled
>> "-debug" on the build. What happened was that Ant was not loading its
>> internal conditions/tasks/type definitions anymore, so the build system
>> does not know almost any type anymore. The debug log showed that Ant
>> was no longer able to load the resource
>> "/org/apache/tools/ant/antlib.xml" from its own JAR file anymore.
>> Instead it printed some strange debugging output (which looked totally
>> broken).
>>
>> I spend the whole night digging through their code and found the issue:
>> The commit of Multi-Release-Jar files (see
>> http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/f9913ea0f95c) broke
>> resource handling in Apache Ant. In short: If you call
>> ClassLoader.getResources() / or getResource() you get back an URL from
>> where you can load the Resource - this is all fine and still works.
>> But, with the Multi-Release JAR files patch this now has an URL
>> fragment appended to the URL: '#release' (see
>> http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/f9913ea0f95c); this also
>> applies to non-multi-release JAR files like Apache Ant's "ant.jar".
>>
>> In Java 7, Java 8,... and Java 9pre-b108,
>> ClassLoader.getResource()/getResources() returned stuff like:
>>
>> "jar:file:/C:/Program%20Files/Java/apache-ant-1.9.6/lib/ant.jar!/org/apache/tools/ant/antlib.xml"
>>
>> Now in Java 9b108 the following is returned:
>>
>> "jar:file:/C:/Program%20Files/Java/apache-ant-1.9.6/lib/ant.jar!/org/apache/tools/ant/antlib.xml#release"
>>
>> And here Ant breaks (and I assume many other projects like Maven, too).
>> Ant checks for the file extension of the string (because it may load
>> definitions from both XML and properties files). So it does
>> endsWith(".xml") and of course this now returns false. The effect is
>> that Ant tries to load its own task definitions as a java properties
>> file instead of XML. Of course this fails, because the data behind this
>> URL is XML. The effect is that Ant cannot bootstrap as everything to
>> build is missing.
>>
>> One might say: Ant's code is broken (I agree, it is not nice because it
>> relies on the string representation of the resource URL - which is a
>> no-go anyways), but it is impossible to fix, because Ant is bundled on
>> most developer computers and those will suddenly break with Java 9!
>> There is also no version out there that works around this, so we cannot
>> test anything anymore!
>>
>> The problematic line in Ant's code is here:
>> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.ant/ant/1.9.6/org/apache/tools/ant/taskdefs/Definer.java?av=f#259
>>
>> I'd suggest to please ASAP revert the Multi-Release JAR file patch and
>> provide a new preview build as soon as possible. I think there is more
>> work needed to fix this. If this does not revert to the original state,
>> it will be impossible to build and test Lucene, Elasticsearch,.... (and
>> almost every Java project out there!). So short: We cannot test anymore
>> and it is likely that we cannot support Java 9 anymore because the
>> build system used by most Java projects behind the scenes does not
>> bootstrap itself anymore.
>>
>> My suggestion would be to investigate other versions for this patch
>> that does *not* modify the resource URLs by appending a fragment to
>> them (at least not for the "standard" case without an actual
>> Multi-Release Jar). For new multi-release JAR files I am fine with
>> appending fragments, but please not for default ones. Maybe change code
>> to handle the URLs from the non-versioned part differently (without
>> fragment). Leaving the fragment inide may break many othe rprojects,
>> because many programmers are very sloppy with handling URLs (well-known
>> issue is calling URL#getFile() of a file:-URL that breaks on Windows
>> systems and spaces in path name). Many people just call toString() on
>> URL and do some test on it (startsWith, endsWith). So appending
>> fragments is a no-go for backwards compatibility with JAR resources!
>>
>> I posted this to the mailing list and did not open a bug report on
>> http://bugs.java.com/, because this is a more general issue - feel free
>> to open bug reports around this!!! I would be very happy if we could
>> find a quick solution for this problem. Until there is a solution we
>> have to stop testing Java 9 with Apache Lucene/Solr/..., and this is
>> not a good sign, especially as Jigsaw will be merged soon.
>>
>> Thanks for listening,
>> Uwe
>>
>> P.S.: I also CCed the Apache Ant team. They should fix the broken code
>> anyways, but this won't help for many projects already out there (e.g.
>> Apache Lucene still has a minimum requirement of Ant 1.8.2 because
>> MacOSX computers ship with that version since years).
>>
>> -----
>> Uwe Schindler
>> uschindler at apache.org
>> ASF Member, Apache Lucene PMC / Committer
>> Bremen, Germany
>> http://lucene.apache.org/
>