Multiple versions of a non-exported dependency

Thu Sep 1 15:34:04 UTC 2016

Trying not to distract this conversation away from the details…  In the end, the concept of “micro services” and things like “Jini” discovery can allow smaller distribution of systems in single class loaders without conflicts.   But still, there is the static data design issue.  I have a Jini discovery based desktop application that I’ve used for more than a decade to “find” services on my network, download their proxy and associated UI, and use them.   Those applications are all separate class loaders and do not interact inside of my desktop application.  If they need to interact, they do so via their services which interact with other services through discovery of that service, download of the proxy and calls out through the jointly know interface/class name which was used in the lookup criteria.   All of this has explicit version management because the Jar file names are well known and include version numbers in them.

In Jini, there is a lot of class loading happening with downloaded code, but it is a well formed tree structure inside of my desktop application because each application is using a specific version of jars in its setting of the exported class details. 

Jigsaw is orthogonal to what I am doing with Jini because Jigsaw is about starting the application, not extending the application as the Jini discovery and downloads do.  However, I am trying to point out how the direct references to version details in the design and operation of the system allow for multiple versions to coexist trivially and be managed explicitly to help the system continue to function correctly.   

If I have an application deployed twice, with two different sets of jars because of version changes, and my desktop application picks one of them at random to use, it won’t matter if I in fact try to open both instances by explicitly knowing how to do that because the class loader design keeps them separated and the exported class loader detail identifies the specifics need to make each function as a separate application within my desktop container.

I don’t know what base of users and what paradigms of deployments the Jigsaw team considered.  It’s clear that they focused first on modularity of the JVM.  It’s not obvious that how other module systems unrelated to pluggable functionality that the JVM is trying to separate were considered.  The separation of detail that is not referenced is a simple modularization detail.  The isolation of detail that should not be referenced is a different modularization detail which just doesn’t seem to be completely considered for any detail other than “exposure” as opposed to “compatibility” which is what “Version” points to.

Gregg

> On Sep 1, 2016, at 10:12 AM, Gregg Wonderly <greggwon at cox.net> wrote:
> 
> Why I was referring to is how will modules find classes from other modules?  How will the different version of the same package namespace that Gili was talking about be hidden?   In other words, you can only make them all visible by using a single class loader, unless there is a DAG of dependencies that naturally falls out, because all we have is ClassLoaders parent as the pointer to other interesting detail.
> 
> How will definedModulesWithManyLoaders make it possible for all of the correct details to be visible? 
> 
> Some explicit example detail:
> 
> module A uses l4j logging in module B
> 
> module C uses l4j logging in module D
> 
> module E is a standalone module
> 
> A and C exchange data objects defined in E
> 
> +— B — A
> |
> + E
> |
> +— D — C
> 
> This graph you can draw today with ClassLoader parent references.   But, there are more complicated graphs that fall out from less simple needs.  It’s this specific issue of still not trying to provide support for versions and for arranging the module dependencies more explicitly through the use of version as part of the package name namespace which Gili is trying to speak to I feel.
> 
> Since B and D are the same package name, you have to hide them from each other in separate class loaders, obviously.   But the Graph from C to E and A to E is not always direct and can in some cases not be possible with a single instance of the jar.  That breaks static class data designs if you create two copies in separate loaders.
> 
> Gregg
> 
>> On Sep 1, 2016, at 8:37 AM, David M. Lloyd <david.lloyd at redhat.com> wrote:
>> 
>> You've missed the point I'm afraid.  I'm just talking about having the launcher use (the equivalent of) java.lang.reflect.Layer#defineModulesWithManyLoaders instead of (the equivalent of) java.lang.reflect.Layer#defineModulesWithOneLoader.  (The launcher actually uses the slightly lower level defineModules() method I think, but really what I'm suggesting is to update the function to assign new class loaders for named modules instead of reusing the same one.)
>> 
>> On 09/01/2016 08:23 AM, Gregg Wonderly wrote:
>>> The important detail for me, is that ClassLoader per module, with the current Class resolution scheme (this ClassLoader and whatever I can find in the parent), provides a lot of issues. The “custom ClassLoaders” or “containers like OSGi” remarks point at the “us and them” attitude that is pretty prevalent in this conversation.  The majority of developers are looking for a module system that is not an “us or them” proposition.   These “all or nothing” compromises are what create the “hell” that dominates conversations here.  What we all want to be able to do, is write software once, target it to “THE Java platform”, and be done.
>>> 
>>> What Sun and now Oracle are continuing to do, is create more stuff that is nothing like what everyone else is doing with modularity and instead create something that is orthogonal to most peoples problem spaces and in the end creates tremendously more “work” for nothing more than compatibility with the new “JVM” environment.
>>> 
>>> The real goal here needs to be making all of the other module and container systems obsolete.  Those systems should “want” to provide support for the awesome, new module system that will make in unnecessary for them to roll their own details any longer.
>>> 
>>> Yes, that is a long road and a tall measure for success.  But frankly, even the lack of any visibility of the style of modules that Netbeans has used for decades makes it clear that this groups view at Oracle is extremely narrow and perhaps even more uninformed about what the community actually needs.
>>> 
>>> Gregg
>>> 
>>>> On Sep 1, 2016, at 7:29 AM, David M. Lloyd <david.lloyd at redhat.com> wrote:
>>>> 
>>>> It seems like there is no good reason why the application modules aren't loaded with classloader-per-module now.  The platform stuff could all be in one, but the application stuff?  Problems like this are going to come up a lot otherwise; let's consider making that change.
>>>> 
>>>> On 08/31/2016 07:45 PM, Neil Bartlett wrote:
>>>>> Remi,
>>>>> 
>>>>> Actually I don’t think that statically linking will work. This would produce modules that have overlapping private (non-exported) packages, and such modules also cannot be used in Java 9 on the modulepath.
>>>>> 
>>>>> I tested this in build 9-ea+126-jigsaw-nightly-h5280-20160713 by creating two modules both containing a private package org.example.util. The following exception resulted:  java.lang.reflect.LayerInstantiationException: Package org.example.util in both module a and module b.
>>>>> 
>>>>> Again this could be “solved” by using custom ClassLoaders or a ClassLoader-based module system like OSGi on Java 9.
>>>>> 
>>>>> Neil
>>>>> 
>>>>> 
>>>>> 
>>>>>> On 31 Aug 2016, at 20:28, Remi Forax <forax at univ-mlv.fr> wrote:
>>>>>> 
>>>>>> The other solution is to statically link the right version of slf4j inside guava and jsoup.
>>>>>> A tool like jarjar can be updated to merge two modular jars (merge two module-info).
>>>>>> 
>>>>>> cheers,
>>>>>> Rémi
>>>>>> 
>>>>>> ----- Mail original -----
>>>>>>> De: "Neil Bartlett" <njbartlett at gmail.com>
>>>>>>> À: cowwoc at bbs.darktech.org, "Alex Buckley" <alex.buckley at oracle.com>
>>>>>>> Cc: "ZML-OpenJDK-Jigsaw-Developers" <jigsaw-dev at openjdk.java.net>
>>>>>>> Envoyé: Mercredi 31 Août 2016 20:54:44
>>>>>>> Objet: Re: Multiple versions of a non-exported dependency
>>>>>> 
>>>>>>> Gili,
>>>>>>> 
>>>>>>> As Alex points out: your use-case can be supported in Java 9 but only with the
>>>>>>> addition of custom ClassLoaders, or by using an existing ClassLoader-based
>>>>>>> module system such as OSGi.
>>>>>>> 
>>>>>>> The same is also true of Java 8, and Java 7, etc.
>>>>>>> 
>>>>>>> Regards,
>>>>>>> Neil
>>>>>>> 
>>>>>>> 
>>>>>>>> On 31 Aug 2016, at 19:29, Alex Buckley <alex.buckley at oracle.com> wrote:
>>>>>>>> 
>>>>>>>> On 8/31/2016 10:56 AM, cowwoc wrote:
>>>>>>>>> I recently became aware of the fact that the Jigsaw specification declared
>>>>>>>>> "version-selection" as a non-goal. While I understand how we ended up here,
>>>>>>>>> I am hoping that you were able to support the following (very common)
>>>>>>>>> use-case:
>>>>>>>>> 
>>>>>>>>> * Module "HelloWorld" depends on modules "Guava" and "JSoup".
>>>>>>>>> * Module "Guava" depends on module slf4j version 1 (requires but does not
>>>>>>>>> export it).
>>>>>>>>> * Module "JSoup" depends on module slf4j version 2 (requires but does not
>>>>>>>>> export it).
>>>>>>>>> * slf4j version 2 and is not backwards-compatible with version 1.
>>>>>>>>> 
>>>>>>>>> What happens at runtime? Will Jigsaw (out of the box, without 3rd-party
>>>>>>>>> tools like Maven or OSGI) be smart enough to provide different versions of
>>>>>>>>> slf4j to "Guava" and "JSoup"?
>>>>>>>> 
>>>>>>>> (You mean Guava/JSoup requires slf4j version 1/2 and does not "re-export" it
>>>>>>>> a.k.a. 'requires public'.)
>>>>>>>> 
>>>>>>>> This use case isn't possible on JDK 8 for JARs on the classpath, and it's not
>>>>>>>> supported on JDK 9 for modular JARs on the modulepath:
>>>>>>>> 
>>>>>>>> - If you have two versions of a modular JAR slf4j.jar in different directories
>>>>>>>> on the modulepath, then the first one to be found will dominate, and that's
>>>>>>>> what will be resolved for both Guava and JSoup.
>>>>>>>> 
>>>>>>>> - If you have two modular JARs slf4j_v1.jar and slf4j_v2.jar on the modulepath,
>>>>>>>> and Guava requires slf4j_v1 and JSoup requires slf4j_v2, then launching 'java
>>>>>>>> -m HelloWorld' will fail. The boot layer will refuse to map the "same" packages
>>>>>>>> from different slf4j_v* modules to the application class loader.
>>>>>>>> 
>>>>>>>> The use case _is_ supported on JDK 9 for modular JARs loaded into custom loaders
>>>>>>>> of custom layers. That is, the Java Platform Module System is perfectly capable
>>>>>>>> of supporting the use case -- please see any of my "Jigsaw: Under The Hood"
>>>>>>>> presentations. The use case just isn't supported "out of the box" by the 'java'
>>>>>>>> launcher for JARs on the modulepath.
>>>>>>>> 
>>>>>>>> Alex
>>>>> 
>>>> 
>>>> --
>>>> - DML
>>> 
>> 
>> -- 
>> - DML
> 