From njbartlett at gmail.com Thu Nov 3 15:20:11 2011 From: njbartlett at gmail.com (Neil Bartlett) Date: Thu, 3 Nov 2011 22:20:11 +0000 Subject: Motivation for type-level exports Message-ID: Dear Jigsaw developers, I hope this is the correct forum to ask a few questions about the design of the current Jigsaw prototype. I'll start with a question about the export clause in module-info.java. In the grammar[1] it states that exports take a qualified identifier indicating a package or type name. Could you please explain why it is a requirement for this to be granular to the level of individual types, when Java already has the package-private ("default") access modifier to restrict access to types from outside the package. It seems that the only possible need for type-level exports could be one or both of the following: 1. To restrict visibility of a type where package-private access would otherwise have allowed access. In other words you're talking about restricting visibility of types to other types in the *same package* (albeit a package that is split across modules). 2. To hide visibility of a type that has been erroneously made public. However it would seem far simpler to correct such errors by making the type non-public. Backwards compatibility cannot be the issue here because restricting visibility would break users of that type just as surely as changing its accessibility would. I would be grateful if you could explain the motivation for this feature, ideally with examples illustrating why it is needed for JDK modularisation. Many thanks, Neil [1] http://openjdk.java.net/projects/jigsaw/doc/topics/grammar.html From jesse.glick at oracle.com Thu Nov 3 16:35:41 2011 From: jesse.glick at oracle.com (Jesse Glick) Date: Thu, 03 Nov 2011 19:35:41 -0400 Subject: Motivation for type-level exports In-Reply-To: References: Message-ID: <4EB3254D.2070700@oracle.com> Not an official response but my guess: On 11/03/2011 06:20 PM, Neil Bartlett wrote: > It seems that the only possible need for type-level exports could be > one or both of the following: 3. To permit the type to be visible to other packages in the same module, but not to other modules. In my experience this is a rather common need. If only package-granularity exports are available, you can still get around this, but it is tricky, requiring a special trampoline [1]. A new access modifier intermediate between public and default would be more intuitive, I think. [1] http://wiki.apidesign.org/wiki/FriendPackages From david.lloyd at redhat.com Thu Nov 3 18:57:26 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 03 Nov 2011 20:57:26 -0500 Subject: Motivation for type-level exports In-Reply-To: <4EB3254D.2070700@oracle.com> References: <4EB3254D.2070700@oracle.com> Message-ID: <4EB34686.1080601@redhat.com> On 11/03/2011 06:35 PM, Jesse Glick wrote: > Not an official response but my guess: > > On 11/03/2011 06:20 PM, Neil Bartlett wrote: >> It seems that the only possible need for type-level exports could be >> one or both of the following: > > 3. To permit the type to be visible to other packages in the same > module, but not to other modules. In my experience this is a rather > common need. If only package-granularity exports are available, you can > still get around this, but it is tricky, requiring a special trampoline > [1]. A new access modifier intermediate between public and default would > be more intuitive, I think. > > [1] http://wiki.apidesign.org/wiki/FriendPackages I think you're confusing the issue of visibility with accessibility. Your #3 is accommodated via a new access level - "module" - which would restrict *accessibility* of a class but not *visibility* of that class. Given the presumed availability of the new module-wide access level, your #3 is satisfied wholly, yet Neil's question is unanswered. Leaving aside the unmitigated disaster that inevitably awaits us in regards to the location and disposition of module metadata in the current prototype, exports (or rather the lack of them) are generally more useful in avoiding linkage problems than as a security mechanism. But our experience has shown that controlling exports at a package level is more than adequate for this task for what we expect to be a typical module library distribution. We've only needed to add hooks for type-level visibility controls for the purposes of our OSGi implementation. -- - DML From alex.buckley at oracle.com Thu Nov 3 19:10:57 2011 From: alex.buckley at oracle.com (Alex Buckley) Date: Thu, 03 Nov 2011 19:10:57 -0700 Subject: Motivation for type-level exports In-Reply-To: <4EB34686.1080601@redhat.com> References: <4EB3254D.2070700@oracle.com> <4EB34686.1080601@redhat.com> Message-ID: <4EB349B1.9070205@oracle.com> On 11/3/2011 6:57 PM, David M. Lloyd wrote: > I think you're confusing the issue of visibility with accessibility. > > Your #3 is accommodated via a new access level - "module" - which would > restrict *accessibility* of a class but not *visibility* of that class. > Given the presumed availability of the new module-wide access level, > your #3 is satisfied wholly, yet Neil's question is unanswered. The pertinent Java Module System requirement is phrased as: "Encapsulation ? If a module contains a type, whether defined therein or imported from some other module, then it must be possible to declare that it makes that type available for use by modules depending upon it, or that it does not do so." (http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#encapsulation) As noted in the change history, this deliberately doesn't mention visibility or accessibility. As noted in the open requirements, it's not settled whether accessibility should be configured by the declaration of a type or member (via a 'module' keyword), or by some other denotation (such as whether a type is listed as an export by its enclosing module). It is possible for the JVM to implement access control either way, since the JVM will be aware of the module system and module declarations therein. This could be in addition to the module system's class loaders restricting visibility to non-exported types. (http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#jsr-294) Alex From njbartlett at gmail.com Fri Nov 4 10:04:24 2011 From: njbartlett at gmail.com (Neil Bartlett) Date: Fri, 4 Nov 2011 17:04:24 +0000 Subject: Motivation for type-level exports In-Reply-To: <4EB3254D.2070700@oracle.com> References: <4EB3254D.2070700@oracle.com> Message-ID: Thank you Jesse. All reasonable responses are welcome, not just "official" ones. Restricting visibility of a type that is currently public does not seem like a requirement that could be motivated by JDK modularisation, since it would break compatibility for any existing consumers of that type. It's possible that there is a need to introduce *new* types in the JDK modules that will be public but non-exported, though that seems unlikely. In application code, why not just have package-level exports, i.e. entire packages are either exported or private? Regards Neil On Thursday, 3 November 2011 at 23:35, Jesse Glick wrote: > Not an official response but my guess: > > On 11/03/2011 06:20 PM, Neil Bartlett wrote: > > It seems that the only possible need for type-level exports could be > > one or both of the following: > > > > > 3. To permit the type to be visible to other packages in the same module, but not to other modules. In my experience this is a rather common need. If only > package-granularity exports are available, you can still get around this, but it is tricky, requiring a special trampoline [1]. A new access modifier intermediate between > public and default would be more intuitive, I think. > > [1] http://wiki.apidesign.org/wiki/FriendPackages From dean.long at oracle.com Fri Nov 4 12:00:53 2011 From: dean.long at oracle.com (dean.long at oracle.com) Date: Fri, 4 Nov 2011 12:00:53 -0700 (PDT) Subject: Auto Reply: jigsaw-dev Digest, Vol 34, Issue 1 Message-ID: <4f6d9a42-6445-442d-a4de-ab27790de6d2@default> This is an auto-replied message. I am out of the office right now. From eric at tibco.com Fri Nov 4 13:21:54 2011 From: eric at tibco.com (Eric Johnson) Date: Fri, 4 Nov 2011 21:21:54 +0100 Subject: Motivation for type-level exports In-Reply-To: References: <4EB3254D.2070700@oracle.com> Message-ID: <4EB44962.9090302@tibco.com> Hi Neil, On 11/4/11 6:04 PM, Neil Bartlett wrote: > Restricting visibility of a type that is currently public does not seem like a requirement that could be motivated by JDK modularisation, since it would break compatibility for any existing consumers of that type. It's possible that there is a need to introduce*new* types in the JDK modules that will be public but non-exported, though that seems unlikely. I haven't studied this question in the context of modularizing the JDK, but I've seen this question arise in the context of OSGi. However, I think you're almost certainly overlooking all of the "com.sun.*" classes in the JDK that are an implementation detail, but that occasionally end up being used by clients. It is exactly this problem - the fact that Java cannot make the distinction between a class which is "public" within a module, and "published" outside of a module. I'm willing to bet that virtually every single "public" com.sun.* class falls into this precise category. On top of that, I *know* I've seen cases within the JDK documentation itself where a class has been marked as "not for use". I've not seen this recently, so I wonder if those classes have been hidden by clever use of the javadoc tool. One way of addressing this question would be to annotate the class itself, but OSGi doesn't do this, I believe, because it would increase load times. > > In application code, why not just have package-level exports, i.e. entire packages are either exported or private? Because there are awkward circumstances in Java where you need to make a class public within a package in order to access details from that package in other packages of the same module, but you just don't want to make the class part of the published API. Not having run into this recently, I cannot give you a good example off the top of my head, my apologies. Typically you're going to see this problem as part of attempting to modularizing existing code bases. -Eric. From david.lloyd at redhat.com Fri Nov 4 13:33:25 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Fri, 04 Nov 2011 15:33:25 -0500 Subject: Motivation for type-level exports In-Reply-To: <4EB44962.9090302@tibco.com> References: <4EB3254D.2070700@oracle.com> <4EB44962.9090302@tibco.com> Message-ID: <4EB44C15.8080204@redhat.com> On 11/04/2011 03:21 PM, Eric Johnson wrote: > Hi Neil, > > On 11/4/11 6:04 PM, Neil Bartlett wrote: >> Restricting visibility of a type that is currently public does not >> seem like a requirement that could be motivated by JDK modularisation, >> since it would break compatibility for any existing consumers of that >> type. It's possible that there is a need to introduce*new* types in >> the JDK modules that will be public but non-exported, though that >> seems unlikely. > > I haven't studied this question in the context of modularizing the JDK, > but I've seen this question arise in the context of OSGi. However, I > think you're almost certainly overlooking all of the "com.sun.*" classes > in the JDK that are an implementation detail, but that occasionally end > up being used by clients. It is exactly this problem - the fact that > Java cannot make the distinction between a class which is "public" > within a module, and "published" outside of a module. I'm willing to bet > that virtually every single "public" com.sun.* class falls into this > precise category. This is easily solved using package-level exports though and need not delve into the layer of per-class exports. In JBoss AS we have a module which represents the standard JDK, and separate modules for the proprietary classes that vary based on vendor. -- - DML From gnormington at vmware.com Fri Nov 4 21:18:08 2011 From: gnormington at vmware.com (Glyn Normington) Date: Sat, 5 Nov 2011 04:18:08 +0000 Subject: Converting plain JARs to Java modules Message-ID: <9E44FD48-21D1-4F4B-81DC-10FC4B57A712@vmware.com> One of the issues any new module system faces is reusing existing JARs. OSGi has tools like bnd and bundler to add OSGi metadata to a JAR with sensible defaults. I was wondering if the Java module system will be capable of having tools that automatically convert Jars to Java modules. I think this will be important if we want to be able to depend on existing JARs in a modular way. One specific issue is that since Java modules express their dependencies on other Java modules rather than packages, it is not possible to infer these dependencies by analysing the byte code of a JAR. Has any thought been given to allowing Java modules to express dependencies on packages rather than on other Java modules? Regards, Glyn From Alan.Bateman at oracle.com Sat Nov 5 03:27:50 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 05 Nov 2011 10:27:50 +0000 Subject: Converting plain JARs to Java modules In-Reply-To: <9E44FD48-21D1-4F4B-81DC-10FC4B57A712@vmware.com> References: <9E44FD48-21D1-4F4B-81DC-10FC4B57A712@vmware.com> Message-ID: <4EB50FA6.8030400@oracle.com> On 05/11/2011 04:18, Glyn Normington wrote: > One of the issues any new module system faces is reusing existing JARs. OSGi has tools like bnd and bundler to add OSGi metadata to a JAR with sensible defaults. > > I was wondering if the Java module system will be capable of having tools that automatically convert Jars to Java modules. I think this will be important if we want to be able to depend on existing JARs in a modular way. > > One specific issue is that since Java modules express their dependencies on other Java modules rather than packages, it is not possible to infer these dependencies by analysing the byte code of a JAR. Has any thought been given to allowing Java modules to express dependencies on packages rather than on other Java modules? > The migration and interop section of the requirements document [1] lists several requirements related to JAR files. Package-level dependencies is listed in the open requirements section. The current prototype has support for modular JAR files (essentially a module-info class in the top-level directory of the JAR file). You can specify a modular JAR file to the jmod command to install it as a module. Also the jar command has been updated with an option to generate a simple module-info.class. Note that it doesn't look at the static dependencies when generating the module-info but does look at the Main-Class and Class-Path attributes in manifest. For more complicated cases then it's better to write the module-info.java yourself and then use the jar command to add it to the JAR file. -Alan. [1] http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#_4 From mandy.chung at oracle.com Sat Nov 5 09:59:45 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Sat, 05 Nov 2011 09:59:45 -0700 Subject: Converting plain JARs to Java modules In-Reply-To: <4EB50FA6.8030400@oracle.com> References: <9E44FD48-21D1-4F4B-81DC-10FC4B57A712@vmware.com> <4EB50FA6.8030400@oracle.com> Message-ID: <4EB56B81.5020004@oracle.com> On 11/5/2011 3:27 AM, Alan Bateman wrote: > On 05/11/2011 04:18, Glyn Normington wrote: >> One of the issues any new module system faces is reusing existing >> JARs. OSGi has tools like bnd and bundler to add OSGi metadata to a >> JAR with sensible defaults. >> >> I was wondering if the Java module system will be capable of having >> tools that automatically convert Jars to Java modules. I think this >> will be important if we want to be able to depend on existing JARs in >> a modular way. >> >> One specific issue is that since Java modules express their >> dependencies on other Java modules rather than packages, it is not >> possible to infer these dependencies by analysing the byte code of a >> JAR. Has any thought been given to allowing Java modules to express >> dependencies on packages rather than on other Java modules? >> > The migration and interop section of the requirements document [1] > lists several requirements related to JAR files. Package-level > dependencies is listed in the open requirements section. > > The current prototype has support for modular JAR files (essentially a > module-info class in the top-level directory of the JAR file). You can > specify a modular JAR file to the jmod command to install it as a > module. Also the jar command has been updated with an option to > generate a simple module-info.class. Note that it doesn't look at the > static dependencies when generating the module-info but does look at > the Main-Class and Class-Path attributes in manifest. For more > complicated cases then it's better to write the module-info.java > yourself and then use the jar command to add it to the JAR file. Additional information: You can refer to [2] about the jar command to transform a plain JAR file to a modular JAR file. There is an internal tool called ClassAnaylzer [3] that analyzes the bytecode to generate the module dependencies. This is currently used by the jdk build. We will likely provide a similar tool to help developers for the dependency analysis. [1] http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#_4 [2] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2011-August/001460.html [3] http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/raw-file/tip/make/tools/classanalyzer/classanalyzer.html From david.holmes at oracle.com Sun Nov 6 17:20:00 2011 From: david.holmes at oracle.com (David Holmes) Date: Mon, 07 Nov 2011 11:20:00 +1000 Subject: Motivation for type-level exports In-Reply-To: References: <4EB3254D.2070700@oracle.com> Message-ID: <4EB73240.6050907@oracle.com> On 5/11/2011 3:04 AM, Neil Bartlett wrote: > Thank you Jesse. All reasonable responses are welcome, not just "official" ones. > > Restricting visibility of a type that is currently public does not seem like a requirement that could be motivated by JDK modularisation, since it would break compatibility for any existing consumers of that type. It's possible that there is a need to introduce *new* types in the JDK modules that will be public but non-exported, though that seems unlikely. One of the use cases has always (AFAIK going way back to early days of Java module/superpackage work) been the ability to hide, within a module, types that are public only because it is the only way they can be shared between related packages. > In application code, why not just have package-level exports, i.e. entire packages are either exported or private? Because that doesn't allow cross-package sharing. David ----- > Regards > Neil > > > > On Thursday, 3 November 2011 at 23:35, Jesse Glick wrote: > >> Not an official response but my guess: >> >> On 11/03/2011 06:20 PM, Neil Bartlett wrote: >>> It seems that the only possible need for type-level exports could be >>> one or both of the following: >>> >> >> >> 3. To permit the type to be visible to other packages in the same module, but not to other modules. In my experience this is a rather common need. If only >> package-granularity exports are available, you can still get around this, but it is tricky, requiring a special trampoline [1]. A new access modifier intermediate between >> public and default would be more intuitive, I think. >> >> [1] http://wiki.apidesign.org/wiki/FriendPackages > From peter.kriens at aqute.biz Mon Nov 7 00:31:27 2011 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 7 Nov 2011 09:31:27 +0100 Subject: Motivation for type-level exports In-Reply-To: <4EB44962.9090302@tibco.com> References: <4EB3254D.2070700@oracle.com> <4EB44962.9090302@tibco.com> Message-ID: OSGi can filter classes from an exported package (Core 3.7.8). Kind regards, Peter Kriens On 4 nov. 2011, at 21:21, Eric Johnson wrote: > Hi Neil, > > On 11/4/11 6:04 PM, Neil Bartlett wrote: >> Restricting visibility of a type that is currently public does not seem like a requirement that could be motivated by JDK modularisation, since it would break compatibility for any existing consumers of that type. It's possible that there is a need to introduce*new* types in the JDK modules that will be public but non-exported, though that seems unlikely. > > I haven't studied this question in the context of modularizing the JDK, but I've seen this question arise in the context of OSGi. However, I think you're almost certainly overlooking all of the "com.sun.*" classes in the JDK that are an implementation detail, but that occasionally end up being used by clients. It is exactly this problem - the fact that Java cannot make the distinction between a class which is "public" within a module, and "published" outside of a module. I'm willing to bet that virtually every single "public" com.sun.* class falls into this precise category. > > On top of that, I *know* I've seen cases within the JDK documentation itself where a class has been marked as "not for use". I've not seen this recently, so I wonder if those classes have been hidden by clever use of the javadoc tool. > > One way of addressing this question would be to annotate the class itself, but OSGi doesn't do this, I believe, because it would increase load times. > >> >> In application code, why not just have package-level exports, i.e. entire packages are either exported or private? > > Because there are awkward circumstances in Java where you need to make a class public within a package in order to access details from that package in other packages of the same module, but you just don't want to make the class part of the published API. Not having run into this recently, I cannot give you a good example off the top of my head, my apologies. Typically you're going to see this problem as part of attempting to modularizing existing code bases. > > -Eric. > From gnormington at vmware.com Mon Nov 7 01:47:37 2011 From: gnormington at vmware.com (Glyn Normington) Date: Mon, 7 Nov 2011 09:47:37 +0000 Subject: Converting plain JARs to Java modules In-Reply-To: <4EB56B81.5020004@oracle.com> References: <9E44FD48-21D1-4F4B-81DC-10FC4B57A712@vmware.com> <4EB50FA6.8030400@oracle.com> <4EB56B81.5020004@oracle.com> Message-ID: Thanks for those responses. I see that semi-automatic conversion of a existing JAR file to a modular JAR file is possible. In case it's not obvious, please note that support for package level dependencies would permit a fully automatic conversion because it would then not be necessary to know the module name corresponding to each package dependency. A fully automatic converstion could be particularly useful in automatically serving up modularised versions of existing JARs, e.g. in a front end to Maven central. Peter Kriens also makes the case for package level dependencies here, in case anyone missed it: http://www.osgi.org/blog/2011/10/what-to-depend-on.html Regards, Glyn On 5 Nov 2011, at 16:59, Mandy Chung wrote: > On 11/5/2011 3:27 AM, Alan Bateman wrote: >> On 05/11/2011 04:18, Glyn Normington wrote: >>> One of the issues any new module system faces is reusing existing JARs. OSGi has tools like bnd and bundler to add OSGi metadata to a JAR with sensible defaults. >>> >>> I was wondering if the Java module system will be capable of having tools that automatically convert Jars to Java modules. I think this will be important if we want to be able to depend on existing JARs in a modular way. >>> >>> One specific issue is that since Java modules express their dependencies on other Java modules rather than packages, it is not possible to infer these dependencies by analysing the byte code of a JAR. Has any thought been given to allowing Java modules to express dependencies on packages rather than on other Java modules? >>> >> The migration and interop section of the requirements document [1] lists several requirements related to JAR files. Package-level dependencies is listed in the open requirements section. >> >> The current prototype has support for modular JAR files (essentially a module-info class in the top-level directory of the JAR file). You can specify a modular JAR file to the jmod command to install it as a module. Also the jar command has been updated with an option to generate a simple module-info.class. Note that it doesn't look at the static dependencies when generating the module-info but does look at the Main-Class and Class-Path attributes in manifest. For more complicated cases then it's better to write the module-info.java yourself and then use the jar command to add it to the JAR file. > > Additional information: > You can refer to [2] about the jar command to transform a plain JAR file to a modular JAR file. > There is an internal tool called ClassAnaylzer [3] that analyzes the bytecode to generate the module dependencies. This is currently used by the jdk build. We will likely provide a similar tool to help developers for the dependency analysis. > > [1] http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#_4 > [2] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2011-August/001460.html > [3] http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/raw-file/tip/make/tools/classanalyzer/classanalyzer.html From gnormington at vmware.com Mon Nov 7 02:10:37 2011 From: gnormington at vmware.com (Glyn Normington) Date: Mon, 7 Nov 2011 10:10:37 +0000 Subject: Ordering of versions Message-ID: <32BCD13E-CDF7-44BD-8F5A-2178C530B8BA@vmware.com> The requirements document description of versioning ([1]) says: "Version strings must be totally ordered but otherwise the module system must not impose any particular structure or semantics upon them." Dalibor Topic's presentation on Jigsaw at EclipseCon Europe gave some interesting examples of versions supported by Jigsaw, but it wasn't clear how a total ordering could be defined that preserved the total ordering of multiple version conventions. For example, what is the order of the following versions, where "-" is taken to delimit a qualifier: 1.2-3,1.2.1-3, 1.2.3? Regards, Glyn [1] http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#versioning From Alan.Bateman at oracle.com Mon Nov 7 02:22:24 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 07 Nov 2011 10:22:24 +0000 Subject: Converting plain JARs to Java modules In-Reply-To: References: <9E44FD48-21D1-4F4B-81DC-10FC4B57A712@vmware.com> <4EB50FA6.8030400@oracle.com> <4EB56B81.5020004@oracle.com> Message-ID: <4EB7B160.9050701@oracle.com> On 07/11/2011 09:47, Glyn Normington wrote: > Thanks for those responses. I see that semi-automatic conversion of a existing JAR file to a modular JAR file is possible. > > In case it's not obvious, please note that support for package level dependencies would permit a fully automatic conversion because it would then not be necessary to know the module name corresponding to each package dependency. A fully automatic converstion could be particularly useful in automatically serving up modularised versions of existing JARs, e.g. in a front end to Maven central. At least for Maven artifacts then you can generate a module name from the maven coordinates. In the current prototype then it uses artifactId at version when installing a Maven artifact as a module. It could be argued that the groupId should be in there too but exact mapping TBD. In any case, you don't know what the exported types are. As regards static analysis then it will clearly be useful as a migration aid but it wouldn't catch everything. Mandy pointed out the class analyzer that we currently use in the build and there is clearly potential to do more. -Alan. From gnormington at vmware.com Mon Nov 7 03:15:04 2011 From: gnormington at vmware.com (Glyn Normington) Date: Mon, 7 Nov 2011 11:15:04 +0000 Subject: Converting plain JARs to Java modules In-Reply-To: <4EB7B160.9050701@oracle.com> References: <9E44FD48-21D1-4F4B-81DC-10FC4B57A712@vmware.com> <4EB50FA6.8030400@oracle.com> <4EB56B81.5020004@oracle.com> <4EB7B160.9050701@oracle.com> Message-ID: <90E45BBA-D4B1-4838-9654-69A3733B6AA2@vmware.com> On 7 Nov 2011, at 10:22, Alan Bateman wrote: > On 07/11/2011 09:47, Glyn Normington wrote: >> Thanks for those responses. I see that semi-automatic conversion of a existing JAR file to a modular JAR file is possible. >> >> In case it's not obvious, please note that support for package level dependencies would permit a fully automatic conversion because it would then not be necessary to know the module name corresponding to each package dependency. A fully automatic converstion could be particularly useful in automatically serving up modularised versions of existing JARs, e.g. in a front end to Maven central. > At least for Maven artifacts then you can generate a module name from the maven coordinates. In the current prototype then it uses artifactId at version when installing a Maven artifact as a module. It could be argued that the groupId should be in there too but exact mapping TBD. In any case, you don't know what the exported types are. Good point about using the maven coordinates to infer module names. I wonder if dependencies expressed in Maven are sufficient to generate a module's dependencies? Sounds plausible. As for exported types, a reasonable starting point is to export all the types of an existing JAR. > > As regards static analysis then it will clearly be useful as a migration aid but it wouldn't catch everything. Mandy pointed out the class analyzer that we currently use in the build and there is clearly potential to do more. > > -Alan. Regards, Glyn From mark.reinhold at oracle.com Tue Nov 8 13:40:53 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 08 Nov 2011 13:40:53 -0800 Subject: Motivation for type-level exports In-Reply-To: njbartlett@gmail.com; Fri, 04 Nov 2011 17:04:24 -0000; Message-ID: <20111108214053.4687013EE@eggemoggin.niobe.net> 2011/11/4 10:04 -0700, njbartlett at gmail.com: > ... > > Restricting visibility of a type that is currently public does not seem > like a requirement that could be motivated by JDK modularisation, since > it would break compatibility for any existing consumers of that > type. Correct. > It's possible that there is a need to introduce *new* types in > the JDK modules that will be public but non-exported, though that seems > unlikely. Agreed. We haven't, in fact, run across any situations in the JDK modularization work that absolutely require type-level exports. > In application code, why not just have package-level exports, > i.e. entire packages are either exported or private? Package-level imports are certainly simpler, and they do appear adequate for many, if not most, use cases both in the JDK and in applications. When we introduced export clauses into the prototype Alex argued, and I agreed [1], that we should use type granularity rather than package granularity. That approach is theoretically appealing but it turns out to be difficult to implement efficiently (as we learned from Mandy's prototype work last summer); it also makes it more difficult to reason about the semantics of module graphs. I strongly suspect that we'll drop type-level exports unless some actual use cases crop up. - Mark [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2011-April/001224.html From mark.reinhold at oracle.com Tue Nov 8 13:50:39 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 08 Nov 2011 13:50:39 -0800 Subject: Motivation for type-level exports In-Reply-To: mark.reinhold@oracle.com; Tue, 08 Nov 2011 13:40:53 PST; <20111108214053.4687013EE@eggemoggin.niobe.net> Message-ID: <20111108215039.178C113EE@eggemoggin.niobe.net> 2011/11/8 13:40 -0800, mark.reinhold at oracle.com: > 2011/11/4 10:04 -0700, njbartlett at gmail.com: >> ... >> >> Restricting visibility of a type that is currently public does not seem >> like a requirement that could be motivated by JDK modularisation, since >> it would break compatibility for any existing consumers of that >> type. > > Correct. > >> It's possible that there is a need to introduce *new* types in >> the JDK modules that will be public but non-exported, though that seems >> unlikely. > > Agreed. > > We haven't, in fact, run across any situations in the JDK modularization > work that absolutely require type-level exports. > >> In application code, why not just have package-level exports, >> i.e. entire packages are either exported or private? > > Package-level imports are certainly simpler, and they do appear adequate s/imports/exports/ (!) > for many, if not most, use cases both in the JDK and in applications. > > When we introduced export clauses into the prototype Alex argued, and > I agreed [1], that we should use type granularity rather than package > granularity. That approach is theoretically appealing but it turns out > to be difficult to implement efficiently (as we learned from Mandy's > prototype work last summer); it also makes it more difficult to reason > about the semantics of module graphs. > > I strongly suspect that we'll drop type-level exports unless some actual > use cases crop up. > > - Mark > > > [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2011-April/001224.html From mark.reinhold at oracle.com Tue Nov 8 14:32:56 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 08 Nov 2011 14:32:56 -0800 Subject: Converting plain JARs to Java modules In-Reply-To: gnormington@vmware.com; Mon, 07 Nov 2011 09:47:37 GMT; Message-ID: <20111108223256.D33CA13EE@eggemoggin.niobe.net> 2011/11/7 1:47 -0800, gnormington at vmware.com: > ... > > In case it's not obvious, please note that support for package level > dependencies would permit a fully automatic conversion because it would > then not be necessary to know the module name corresponding to each > package dependency. Yes, I think we understand that. > A fully automatic converstion could be particularly > useful in automatically serving up modularised versions of existing > JARs, e.g. in a front end to Maven central. Maven POMs already contain what is, essentially, module-level dependence information, so the value of package-level dependences for that case is far from clear. Alan has prototyped a tool which constructs a Jigsaw module declaration from a POM; it works well, at least for simple cases, though there does seem to be a lot of noise in the POMs published on Maven Central. - Mark From mark.reinhold at oracle.com Tue Nov 8 14:54:03 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 08 Nov 2011 14:54:03 -0800 Subject: Ordering of versions In-Reply-To: gnormington@vmware.com; Mon, 07 Nov 2011 10:10:37 GMT; <32BCD13E-CDF7-44BD-8F5A-2178C530B8BA@vmware.com> Message-ID: <20111108225403.3425713EE@eggemoggin.niobe.net> 2011/11/7 2:10 -0800, gnormington at vmware.com: > The requirements document description of versioning ([1]) says: > > "Version strings must be totally ordered but otherwise the module > system must not impose any particular structure or semantics upon > them." > > Dalibor Topic's presentation on Jigsaw at EclipseCon Europe gave some > interesting examples of versions supported by Jigsaw, but it wasn't > clear how a total ordering could be defined that preserved the total > ordering of multiple version conventions. > > For example, what is the order of the following versions, where "-" is > taken to delimit a qualifier: 1.2-3,1.2.1-3, 1.2.3? Exactly the order you gave: 1.2-3 < 1.2.1-3 < 1.2.3 Jigsaw presently treats the last '-' as introducing a qualifier or branch version. When comparing two version strings the qualifiers are tested only if the segments preceding them are identical. This design is imperfect. If nothing else, a direct mapping to Debian package versions violates the Debian versioning policy [1]. - Mark [1] http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Version From sjlee0 at gmail.com Tue Nov 8 15:15:08 2011 From: sjlee0 at gmail.com (Sangjin Lee) Date: Tue, 8 Nov 2011 15:15:08 -0800 Subject: Converting plain JARs to Java modules In-Reply-To: <20111108223256.D33CA13EE@eggemoggin.niobe.net> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> Message-ID: Translating maven POMs into module dependency information is not quite the same as generating it based on the bytecode in my view. We're well too aware that most of the POMs do not contain correct (direct) dependencies for this to be accurate. The generated module dependency information would be as good as the underlying POMs, which is generally not so good. Generating OSGi manifests using package-level dependencies can be very accurate, and does not depend much on user input. Furthermore, it is a local operation. Regards, Sangjin On Tue, Nov 8, 2011 at 2:32 PM, wrote: > 2011/11/7 1:47 -0800, gnormington at vmware.com: > > ... > > > > In case it's not obvious, please note that support for package level > > dependencies would permit a fully automatic conversion because it would > > then not be necessary to know the module name corresponding to each > > package dependency. > > Yes, I think we understand that. > > > A fully automatic converstion could be particularly > > useful in automatically serving up modularised versions of existing > > JARs, e.g. in a front end to Maven central. > > Maven POMs already contain what is, essentially, module-level dependence > information, so the value of package-level dependences for that case is > far from clear. Alan has prototyped a tool which constructs a Jigsaw > module declaration from a POM; it works well, at least for simple cases, > though there does seem to be a lot of noise in the POMs published on > Maven Central. > > - Mark > From mandy.chung at oracle.com Tue Nov 8 19:15:30 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Tue, 08 Nov 2011 19:15:30 -0800 Subject: Review request: module graph clean up Message-ID: <4EB9F052.2030805@oracle.com> Webrev at: http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/modules-cleanup.00/ These changes should make the module graph cleaner and separate out some components as its own module (currently grouped with some other module): 1) separate java.util.prefs into its own module 2) separate the cosnaming jndi provider as its own module and management-iiop as it own module - this removes corba's dependency to jndi and management 3) separate management-snmp as its own module 4) remove jdk.ext but break it into sctp, zipfs, httpserver modules 5) rename jsse to ssl Thanks Mandy From richard.s.hall at oracle.com Tue Nov 8 19:56:12 2011 From: richard.s.hall at oracle.com (Richard S. Hall) Date: Tue, 08 Nov 2011 22:56:12 -0500 Subject: Motivation for type-level exports In-Reply-To: <20111108214053.4687013EE@eggemoggin.niobe.net> References: <20111108214053.4687013EE@eggemoggin.niobe.net> Message-ID: <4EB9F9DC.9030205@oracle.com> On 11/8/11 16:40, mark.reinhold at oracle.com wrote: > 2011/11/4 10:04 -0700, njbartlett at gmail.com: >> ... >> >> Restricting visibility of a type that is currently public does not seem >> like a requirement that could be motivated by JDK modularisation, since >> it would break compatibility for any existing consumers of that >> type. > Correct. > >> It's possible that there is a need to introduce *new* types in >> the JDK modules that will be public but non-exported, though that seems >> unlikely. > Agreed. > > We haven't, in fact, run across any situations in the JDK modularization > work that absolutely require type-level exports. > >> In application code, why not just have package-level exports, >> i.e. entire packages are either exported or private? > Package-level imports are certainly simpler, and they do appear adequate > for many, if not most, use cases both in the JDK and in applications. > > When we introduced export clauses into the prototype Alex argued, and > I agreed [1], that we should use type granularity rather than package > granularity. That approach is theoretically appealing but it turns out > to be difficult to implement efficiently (as we learned from Mandy's > prototype work last summer); it also makes it more difficult to reason > about the semantics of module graphs. > > I strongly suspect that we'll drop type-level exports unless some actual > use cases crop up. Even though OSGi supports type-level filtering from exported packages, I think it is safe to say that it is rarely used and few would miss it if it never existed in the first place. -> richard > > - Mark > > > [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2011-April/001224.html From mark.reinhold at oracle.com Tue Nov 8 21:54:47 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 08 Nov 2011 21:54:47 -0800 Subject: Review request: module graph clean up In-Reply-To: mandy.chung@oracle.com; Tue, 08 Nov 2011 19:15:30 PST; <4EB9F052.2030805@oracle.com> Message-ID: <20111109055447.807F81442@eggemoggin.niobe.net> 2011/11/8 19:15 -0800, mandy.chung at oracle.com: > Webrev at: > http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/modules-cleanup.00/ > > ... Thanks -- this looks good to me, but Alan should review it too. > ... > 5) rename jsse to ssl How about also renaming "jce" to "crypto", and "jaas" to "auth"? - Mark From mandy.chung at oracle.com Tue Nov 8 22:26:07 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Tue, 08 Nov 2011 22:26:07 -0800 Subject: Review request: module graph clean up In-Reply-To: <20111109055447.807F81442@eggemoggin.niobe.net> References: <20111109055447.807F81442@eggemoggin.niobe.net> Message-ID: <4EBA1CFF.5060303@oracle.com> On 11/8/11 9:54 PM, mark.reinhold at oracle.com wrote: > How about also renaming "jce" to "crypto", and "jaas" to "auth"? done. Updated webrev at: http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/modules-cleanup.01/ Mandy From peter.kriens at aqute.biz Wed Nov 9 00:51:24 2011 From: peter.kriens at aqute.biz (Peter Kriens) Date: Wed, 9 Nov 2011 09:51:24 +0100 Subject: Converting plain JARs to Java modules In-Reply-To: <20111108223256.D33CA13EE@eggemoggin.niobe.net> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> Message-ID: I have seen Alan's demo of converting poms to Jigsaw modules on JavaOne. I recall that Alan had to fixup most of the poms manually to make this work for Jigsaw? Even after fixing them up, there was no link into the type system or were these dependencies verifiable, you still had to hope that the poms were correct. The problem is that module dependencies, like poms, introduce an alternate namespace which is fully disconnected from the Java type system. This requires manual maintenance and opens a large space for errors that are impossible with package dependencies. Package dependencies are directly connected the Java type system and therefore imports can be automatically calculated from the code, creating a chain of verification. Kind regards, Peter Kriens On 8 nov. 2011, at 23:32, mark.reinhold at oracle.com wrote: > 2011/11/7 1:47 -0800, gnormington at vmware.com: >> ... >> >> In case it's not obvious, please note that support for package level >> dependencies would permit a fully automatic conversion because it would >> then not be necessary to know the module name corresponding to each >> package dependency. > > Yes, I think we understand that. > >> A fully automatic converstion could be particularly >> useful in automatically serving up modularised versions of existing >> JARs, e.g. in a front end to Maven central. > > Maven POMs already contain what is, essentially, module-level dependence > information, so the value of package-level dependences for that case is > far from clear. Alan has prototyped a tool which constructs a Jigsaw > module declaration from a POM; it works well, at least for simple cases, > though there does seem to be a lot of noise in the POMs published on > Maven Central. > > - Mark From gnormington at vmware.com Wed Nov 9 01:06:28 2011 From: gnormington at vmware.com (Glyn Normington) Date: Wed, 9 Nov 2011 09:06:28 +0000 Subject: Converting plain JARs to Java modules In-Reply-To: References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> Message-ID: <34BD9188-FCF3-42EB-A32A-768AEDE8D849@vmware.com> Those points of unreliability of Maven POMs and the locality of the conversion are strong arguments in favour of support for package-level dependencies and a JAR->module converter generating such dependencies. The only downside I am aware of is that a fully automatic tool needs to leave the package import version ranges wide open as it can't guess the correct versions needed. I think this is acceptable for a fully automatic conversion. Regards, Glyn On 8 Nov 2011, at 23:15, Sangjin Lee wrote: > Translating maven POMs into module dependency information is not quite the same as generating it based on the bytecode in my view. We're well too aware that most of the POMs do not contain correct (direct) dependencies for this to be accurate. The generated module dependency information would be as good as the underlying POMs, which is generally not so good. > > Generating OSGi manifests using package-level dependencies can be very accurate, and does not depend much on user input. Furthermore, it is a local operation. > > Regards, > Sangjin > > On Tue, Nov 8, 2011 at 2:32 PM, wrote: > 2011/11/7 1:47 -0800, gnormington at vmware.com: > > ... > > > > In case it's not obvious, please note that support for package level > > dependencies would permit a fully automatic conversion because it would > > then not be necessary to know the module name corresponding to each > > package dependency. > > Yes, I think we understand that. > > > A fully automatic converstion could be particularly > > useful in automatically serving up modularised versions of existing > > JARs, e.g. in a front end to Maven central. > > Maven POMs already contain what is, essentially, module-level dependence > information, so the value of package-level dependences for that case is > far from clear. Alan has prototyped a tool which constructs a Jigsaw > module declaration from a POM; it works well, at least for simple cases, > though there does seem to be a lot of noise in the POMs published on > Maven Central. > > - Mark > From gnormington at vmware.com Wed Nov 9 01:08:45 2011 From: gnormington at vmware.com (Glyn Normington) Date: Wed, 9 Nov 2011 09:08:45 +0000 Subject: Ordering of versions In-Reply-To: <20111108225403.3425713EE@eggemoggin.niobe.net> References: <20111108225403.3425713EE@eggemoggin.niobe.net> Message-ID: <0AFBDA9F-F04D-4B82-90DB-56ABDF8B128B@vmware.com> Thanks Mark - that's helpful. One other thing: are qualifiers treated as strings from an ordering perspective? Regards, Glyn On 8 Nov 2011, at 22:54, mark.reinhold at oracle.com wrote: > 2011/11/7 2:10 -0800, gnormington at vmware.com: >> The requirements document description of versioning ([1]) says: >> >> "Version strings must be totally ordered but otherwise the module >> system must not impose any particular structure or semantics upon >> them." >> >> Dalibor Topic's presentation on Jigsaw at EclipseCon Europe gave some >> interesting examples of versions supported by Jigsaw, but it wasn't >> clear how a total ordering could be defined that preserved the total >> ordering of multiple version conventions. >> >> For example, what is the order of the following versions, where "-" is >> taken to delimit a qualifier: 1.2-3,1.2.1-3, 1.2.3? > > Exactly the order you gave: 1.2-3 < 1.2.1-3 < 1.2.3 > > Jigsaw presently treats the last '-' as introducing a qualifier or branch > version. When comparing two version strings the qualifiers are tested > only if the segments preceding them are identical. > > This design is imperfect. If nothing else, a direct mapping to Debian > package versions violates the Debian versioning policy [1]. > > - Mark > > > [1] http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Version From peter.kriens at aqute.biz Wed Nov 9 01:12:04 2011 From: peter.kriens at aqute.biz (Peter Kriens) Date: Wed, 9 Nov 2011 10:12:04 +0100 Subject: Converting plain JARs to Java modules In-Reply-To: <34BD9188-FCF3-42EB-A32A-768AEDE8D849@vmware.com> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <34BD9188-FCF3-42EB-A32A-768AEDE8D849@vmware.com> Message-ID: An annotation in the package-info.java source can easily provide a package version that could then be encoded in the class files by the compiler. Notice that due to their smaller granularity packages are much easier to version semantically than jars. Kind regards, Peter Kriens On 9 nov. 2011, at 10:06, Glyn Normington wrote: > Those points of unreliability of Maven POMs and the locality of the conversion are strong arguments in favour of support for package-level dependencies and a JAR->module converter generating such dependencies. The only downside I am aware of is that a fully automatic tool needs to leave the package import version ranges wide open as it can't guess the correct versions needed. I think this is acceptable for a fully automatic conversion. > > Regards, > Glyn > > On 8 Nov 2011, at 23:15, Sangjin Lee wrote: > >> Translating maven POMs into module dependency information is not quite the same as generating it based on the bytecode in my view. We're well too aware that most of the POMs do not contain correct (direct) dependencies for this to be accurate. The generated module dependency information would be as good as the underlying POMs, which is generally not so good. >> >> Generating OSGi manifests using package-level dependencies can be very accurate, and does not depend much on user input. Furthermore, it is a local operation. >> >> Regards, >> Sangjin >> >> On Tue, Nov 8, 2011 at 2:32 PM, wrote: >> 2011/11/7 1:47 -0800, gnormington at vmware.com: >>> ... >>> >>> In case it's not obvious, please note that support for package level >>> dependencies would permit a fully automatic conversion because it would >>> then not be necessary to know the module name corresponding to each >>> package dependency. >> >> Yes, I think we understand that. >> >>> A fully automatic converstion could be particularly >>> useful in automatically serving up modularised versions of existing >>> JARs, e.g. in a front end to Maven central. >> >> Maven POMs already contain what is, essentially, module-level dependence >> information, so the value of package-level dependences for that case is >> far from clear. Alan has prototyped a tool which constructs a Jigsaw >> module declaration from a POM; it works well, at least for simple cases, >> though there does seem to be a lot of noise in the POMs published on >> Maven Central. >> >> - Mark >> > From gnormington at vmware.com Wed Nov 9 01:15:00 2011 From: gnormington at vmware.com (Glyn Normington) Date: Wed, 9 Nov 2011 09:15:00 +0000 Subject: Converting plain JARs to Java modules In-Reply-To: References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <34BD9188-FCF3-42EB-A32A-768AEDE8D849@vmware.com> Message-ID: <0B9F545A-B7ED-493F-B7C9-7C7E9199C862@vmware.com> Actually, the scenario I had in mind was generating, fully automatically, Java modules from today's content of Maven central. So we shouldn't assume we can recompile the dependencies in that scenario. Regards, Glyn On 9 Nov 2011, at 09:12, Peter Kriens wrote: > An annotation in the package-info.java source can easily provide a package version that could then be encoded in the class files by the compiler. > > Notice that due to their smaller granularity packages are much easier to version semantically than jars. > > Kind regards, > > Peter Kriens > > > On 9 nov. 2011, at 10:06, Glyn Normington wrote: > >> Those points of unreliability of Maven POMs and the locality of the conversion are strong arguments in favour of support for package-level dependencies and a JAR->module converter generating such dependencies. The only downside I am aware of is that a fully automatic tool needs to leave the package import version ranges wide open as it can't guess the correct versions needed. I think this is acceptable for a fully automatic conversion. >> >> Regards, >> Glyn >> >> On 8 Nov 2011, at 23:15, Sangjin Lee wrote: >> >>> Translating maven POMs into module dependency information is not quite the same as generating it based on the bytecode in my view. We're well too aware that most of the POMs do not contain correct (direct) dependencies for this to be accurate. The generated module dependency information would be as good as the underlying POMs, which is generally not so good. >>> >>> Generating OSGi manifests using package-level dependencies can be very accurate, and does not depend much on user input. Furthermore, it is a local operation. >>> >>> Regards, >>> Sangjin >>> >>> On Tue, Nov 8, 2011 at 2:32 PM, wrote: >>> 2011/11/7 1:47 -0800, gnormington at vmware.com: >>>> ... >>>> >>>> In case it's not obvious, please note that support for package level >>>> dependencies would permit a fully automatic conversion because it would >>>> then not be necessary to know the module name corresponding to each >>>> package dependency. >>> >>> Yes, I think we understand that. >>> >>>> A fully automatic converstion could be particularly >>>> useful in automatically serving up modularised versions of existing >>>> JARs, e.g. in a front end to Maven central. >>> >>> Maven POMs already contain what is, essentially, module-level dependence >>> information, so the value of package-level dependences for that case is >>> far from clear. Alan has prototyped a tool which constructs a Jigsaw >>> module declaration from a POM; it works well, at least for simple cases, >>> though there does seem to be a lot of noise in the POMs published on >>> Maven Central. >>> >>> - Mark >>> >> > From peter.kriens at aqute.biz Wed Nov 9 01:19:04 2011 From: peter.kriens at aqute.biz (Peter Kriens) Date: Wed, 9 Nov 2011 10:19:04 +0100 Subject: Converting plain JARs to Java modules In-Reply-To: <0B9F545A-B7ED-493F-B7C9-7C7E9199C862@vmware.com> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <34BD9188-FCF3-42EB-A32A-768AEDE8D849@vmware.com> <0B9F545A-B7ED-493F-B7C9-7C7E9199C862@vmware.com> Message-ID: <55680B43-09F6-4411-B559-63FA8CA98E8B@aQute.biz> Yup, then you're screwed ... :-) However, for non OSGi bundles I think you could calculate an educated guess of their package versions based on the existing poms if you do it once for the whole of maven central. Kind regards, Peter Kriens On 9 nov. 2011, at 10:15, Glyn Normington wrote: > Actually, the scenario I had in mind was generating, fully automatically, Java modules from today's content of Maven central. So we shouldn't assume we can recompile the dependencies in that scenario. > > Regards, > Glyn > > On 9 Nov 2011, at 09:12, Peter Kriens wrote: > >> An annotation in the package-info.java source can easily provide a package version that could then be encoded in the class files by the compiler. >> >> Notice that due to their smaller granularity packages are much easier to version semantically than jars. >> >> Kind regards, >> >> Peter Kriens >> >> >> On 9 nov. 2011, at 10:06, Glyn Normington wrote: >> >>> Those points of unreliability of Maven POMs and the locality of the conversion are strong arguments in favour of support for package-level dependencies and a JAR->module converter generating such dependencies. The only downside I am aware of is that a fully automatic tool needs to leave the package import version ranges wide open as it can't guess the correct versions needed. I think this is acceptable for a fully automatic conversion. >>> >>> Regards, >>> Glyn >>> >>> On 8 Nov 2011, at 23:15, Sangjin Lee wrote: >>> >>>> Translating maven POMs into module dependency information is not quite the same as generating it based on the bytecode in my view. We're well too aware that most of the POMs do not contain correct (direct) dependencies for this to be accurate. The generated module dependency information would be as good as the underlying POMs, which is generally not so good. >>>> >>>> Generating OSGi manifests using package-level dependencies can be very accurate, and does not depend much on user input. Furthermore, it is a local operation. >>>> >>>> Regards, >>>> Sangjin >>>> >>>> On Tue, Nov 8, 2011 at 2:32 PM, wrote: >>>> 2011/11/7 1:47 -0800, gnormington at vmware.com: >>>>> ... >>>>> >>>>> In case it's not obvious, please note that support for package level >>>>> dependencies would permit a fully automatic conversion because it would >>>>> then not be necessary to know the module name corresponding to each >>>>> package dependency. >>>> >>>> Yes, I think we understand that. >>>> >>>>> A fully automatic converstion could be particularly >>>>> useful in automatically serving up modularised versions of existing >>>>> JARs, e.g. in a front end to Maven central. >>>> >>>> Maven POMs already contain what is, essentially, module-level dependence >>>> information, so the value of package-level dependences for that case is >>>> far from clear. Alan has prototyped a tool which constructs a Jigsaw >>>> module declaration from a POM; it works well, at least for simple cases, >>>> though there does seem to be a lot of noise in the POMs published on >>>> Maven Central. >>>> >>>> - Mark >>>> >>> >> > From Alan.Bateman at oracle.com Wed Nov 9 02:20:51 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 09 Nov 2011 10:20:51 +0000 Subject: Converting plain JARs to Java modules In-Reply-To: References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> Message-ID: <4EBA5403.5030809@oracle.com> On 09/11/2011 08:51, Peter Kriens wrote: > I have seen Alan's demo of converting poms to Jigsaw modules on JavaOne. I recall that Alan had to fixup most of the poms manually to make this work for Jigsaw? Even after fixing them up, there was no link into the type system or were these dependencies verifiable, you still had to hope that the poms were correct. > > The problem is that module dependencies, like poms, introduce an alternate namespace which is fully disconnected from the Java type system. This requires manual maintenance and opens a large space for errors that are impossible with package dependencies. > > Package dependencies are directly connected the Java type system and therefore imports can be automatically calculated from the code, creating a chain of verification. > There wasn't any manual editing of POMs. The demo was real in that it connected to the specified repository (I think we specified a URL to maven central) to locate the requested artifact, downloaded its POM and the POMs of all of the transitive dependencies, worked out what needed to be downloaded, generated a module-info.class for each one and then installed them as modules in the specified module library. Converting artifacts from maven repositories to modules isn't without its issues . As I'm sure you know, POMs tend to specify specific versions of their dependencies which periodically results in conflicts. It's easy to find examples where an application ends up depending on several different versions of commons-io or log4j or other commonly used libraries. Maven has various "strategies" for resolving these, the default as I understand it, being where it chooses the "nearest declaration" (essentially the linear class path). Jigsaw detects these conflicts at install when generating the configuration for the application module and so installation of the application module (the module with the entry point) correctly fails. In the demo we did have a check box called "Resolve Conflicts" to resolve conflicts with the same strategy and I think I remarked a couple of times that installing modules in this manner would result in running with something that is unlikely to have been tested and highly unsuitable for installing into a module library that is used by other applications. There are several other issues too. One annoying one is artifacts that contain classes in packages that are exported by JDK modules. Other issues included JAR files with duplicate entries, JAR files with classes that fail verification, and other garbage like this. I think it's too early to say exactly how we will integrate with Maven repositories. There is clearly a need (and a requirement) to be able to download and install artifacts from maven repositories as if they were modules. It's easy to envisage updating "jmod add-repo" to allow a URL to a maven repository be specified but it needs a bit more thinking about some of the issues first. As regards POMs that have incorrect dependency information then I think that needs consideration too. As was mentioned earlier in the thread then static analysis of the class files could help. It's not going to detect all issues, reflection usages, etc. Ultimately though one would hope that that once we have a Java Module System that the artifacts uploaded to these repositories will that module declarations. -Alan. From peter.kriens at aqute.biz Wed Nov 9 05:04:02 2011 From: peter.kriens at aqute.biz (Peter Kriens) Date: Wed, 9 Nov 2011 14:04:02 +0100 Subject: Converting plain JARs to Java modules In-Reply-To: <4EBA5403.5030809@oracle.com> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> Message-ID: <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> The issue is that maven problems are not caused because maven is bad or that pom authors are stupid. The reason is that the module-to-module dependency architecture in maven (and Jigsaw) is error prone ... Kind regards, Peter Kriens On 9 nov. 2011, at 11:20, Alan Bateman wrote: > On 09/11/2011 08:51, Peter Kriens wrote: >> I have seen Alan's demo of converting poms to Jigsaw modules on JavaOne. I recall that Alan had to fixup most of the poms manually to make this work for Jigsaw? Even after fixing them up, there was no link into the type system or were these dependencies verifiable, you still had to hope that the poms were correct. >> >> The problem is that module dependencies, like poms, introduce an alternate namespace which is fully disconnected from the Java type system. This requires manual maintenance and opens a large space for errors that are impossible with package dependencies. >> >> Package dependencies are directly connected the Java type system and therefore imports can be automatically calculated from the code, creating a chain of verification. >> > There wasn't any manual editing of POMs. The demo was real in that it connected to the specified repository (I think we specified a URL to maven central) to locate the requested artifact, downloaded its POM and the POMs of all of the transitive dependencies, worked out what needed to be downloaded, generated a module-info.class for each one and then installed them as modules in the specified module library. > > Converting artifacts from maven repositories to modules isn't without its issues . As I'm sure you know, POMs tend to specify specific versions of their dependencies which periodically results in conflicts. It's easy to find examples where an application ends up depending on several different versions of commons-io or log4j or other commonly used libraries. Maven has various "strategies" for resolving these, the default as I understand it, being where it chooses the "nearest declaration" (essentially the linear class path). Jigsaw detects these conflicts at install when generating the configuration for the application module and so installation of the application module (the module with the entry point) correctly fails. In the demo we did have a check box called "Resolve Conflicts" to resolve conflicts with the same strategy and I think I remarked a couple of times that installing modules in this manner would result in running with something that is unlikely to have been tested and highly unsuitable for installing into a module library that is used by other applications. > > There are several other issues too. One annoying one is artifacts that contain classes in packages that are exported by JDK modules. Other issues included JAR files with duplicate entries, JAR files with classes that fail verification, and other garbage like this. > > I think it's too early to say exactly how we will integrate with Maven repositories. There is clearly a need (and a requirement) to be able to download and install artifacts from maven repositories as if they were modules. It's easy to envisage updating "jmod add-repo" to allow a URL to a maven repository be specified but it needs a bit more thinking about some of the issues first. > > As regards POMs that have incorrect dependency information then I think that needs consideration too. As was mentioned earlier in the thread then static analysis of the class files could help. It's not going to detect all issues, reflection usages, etc. Ultimately though one would hope that that once we have a Java Module System that the artifacts uploaded to these repositories will that module declarations. > > -Alan. > > From Alan.Bateman at oracle.com Wed Nov 9 05:49:25 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 09 Nov 2011 13:49:25 +0000 Subject: Converting plain JARs to Java modules In-Reply-To: <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> Message-ID: <4EBA84E5.7060409@oracle.com> On 09/11/2011 13:04, Peter Kriens wrote: > The issue is that maven problems are not caused because maven is bad or that pom authors are stupid. The reason is that the module-to-module dependency architecture in maven (and Jigsaw) is error prone ... This thread started out with someone asking about adding module declarations to existing JAR files, and in that context, I agree it can be error prone without good tools. I think things should be a lot better when modules are compiled. -Alan. From peter.kriens at aqute.biz Wed Nov 9 06:02:01 2011 From: peter.kriens at aqute.biz (Peter Kriens) Date: Wed, 9 Nov 2011 15:02:01 +0100 Subject: Converting plain JARs to Java modules In-Reply-To: <4EBA84E5.7060409@oracle.com> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> Message-ID: I agree that tools are needed but we must be careful to not expect tools to stopgap an architectural issue. I think it is important to first do good architectural design leveraging existing tools (e.g. the Java type system) before you try to add new tools. It is such a pity (but all to common) that a design allows for classes of errors that would be impossible with a slightly different design. Kind regards, Peter Kriens On 9 nov. 2011, at 14:49, Alan Bateman wrote: > On 09/11/2011 13:04, Peter Kriens wrote: >> The issue is that maven problems are not caused because maven is bad or that pom authors are stupid. The reason is that the module-to-module dependency architecture in maven (and Jigsaw) is error prone ... > This thread started out with someone asking about adding module declarations to existing JAR files, and in that context, I agree it can be error prone without good tools. I think things should be a lot better when modules are compiled. > > -Alan. > From david.lloyd at redhat.com Wed Nov 9 06:04:53 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 09 Nov 2011 08:04:53 -0600 Subject: Converting plain JARs to Java modules In-Reply-To: References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> Message-ID: <4EBA8885.6020808@redhat.com> I'll just state now that using packages as a dependency unit is a terrible idea, and not some architectural revelation. That way, Peter's wrath will be largely directed at me. :-) On 11/09/2011 08:02 AM, Peter Kriens wrote: > I agree that tools are needed but we must be careful to not expect tools to stopgap an architectural issue. I think it is important to first do good architectural design leveraging existing tools (e.g. the Java type system) before you try to add new tools. It is such a pity (but all to common) that a design allows for classes of errors that would be impossible with a slightly different design. > > Kind regards, > > Peter Kriens > > > > > > > > On 9 nov. 2011, at 14:49, Alan Bateman wrote: > >> On 09/11/2011 13:04, Peter Kriens wrote: >>> The issue is that maven problems are not caused because maven is bad or that pom authors are stupid. The reason is that the module-to-module dependency architecture in maven (and Jigsaw) is error prone ... >> This thread started out with someone asking about adding module declarations to existing JAR files, and in that context, I agree it can be error prone without good tools. I think things should be a lot better when modules are compiled. >> >> -Alan. >> > -- - DML From Alan.Bateman at oracle.com Wed Nov 9 06:07:25 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 09 Nov 2011 14:07:25 +0000 Subject: Review request: module graph clean up In-Reply-To: <4EBA1CFF.5060303@oracle.com> References: <20111109055447.807F81442@eggemoggin.niobe.net> <4EBA1CFF.5060303@oracle.com> Message-ID: <4EBA891D.5070203@oracle.com> On 09/11/2011 06:26, Mandy Chung wrote: > On 11/8/11 9:54 PM, mark.reinhold at oracle.com wrote: >> How about also renaming "jce" to "crypto", and "jaas" to "auth"? > > done. > > Updated webrev at: > http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/modules-cleanup.01/ Mandy and I chatted about some of these tweaks some time ago and so they look fine to me. Overall I think the module graph in the current prototype is starting to look good (ignoring the jdk.foo/sun.foo module pairs). -Alan. From sean.mullan at oracle.com Wed Nov 9 06:08:54 2011 From: sean.mullan at oracle.com (Sean Mullan) Date: Wed, 09 Nov 2011 09:08:54 -0500 Subject: Review request: module graph clean up In-Reply-To: <20111109055447.807F81442@eggemoggin.niobe.net> References: <20111109055447.807F81442@eggemoggin.niobe.net> Message-ID: <4EBA8976.6000400@oracle.com> On 11/9/11 12:54 AM, mark.reinhold at oracle.com wrote: > 2011/11/8 19:15 -0800, mandy.chung at oracle.com: >> Webrev at: >> http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/modules-cleanup.00/ >> >> ... > > Thanks -- this looks good to me, but Alan should review it too. > >> ... >> 5) rename jsse to ssl > > How about also renaming "jce" to "crypto", and "jaas" to "auth"? I would also suggest tls instead of ssl. tls is the standard name. --Sean From Alan.Bateman at oracle.com Wed Nov 9 06:36:11 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 09 Nov 2011 14:36:11 +0000 Subject: Converting plain JARs to Java modules In-Reply-To: References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> Message-ID: <4EBA8FDB.2090706@oracle.com> On 09/11/2011 14:02, Peter Kriens wrote: > I agree that tools are needed but we must be careful to not expect tools to stopgap an architectural issue. I think it is important to first do good architectural design leveraging existing tools (e.g. the Java type system) before you try to add new tools. It is such a pity (but all to common) that a design allows for classes of errors that would be impossible with a slightly different design. > Ignoring the solely-for-migration case where module declarations are added as a post-build step, then what errors are you most concerned about and could they be caught at compile time? -Alan. From mark.reinhold at oracle.com Wed Nov 9 07:21:38 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 09 Nov 2011 07:21:38 -0800 Subject: Review request: module graph clean up In-Reply-To: mandy.chung@oracle.com; Tue, 08 Nov 2011 19:15:30 PST; <4EB9F052.2030805@oracle.com> Message-ID: <20111109152138.8B423DD3@eggemoggin.niobe.net> 2011/11/8 19:15 -0800, mandy.chung at oracle.com: > Webrev at: > http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/modules-cleanup.00/ > > These changes should make the module graph cleaner and separate out some > components as its own module (currently grouped with some other module): > > 1) separate java.util.prefs into its own module On second thought, what's the motivation for this item? The java.util.prefs package is pretty small. Why split it out? Its implementation includes some native code, at least on Posix and Windows. Are we going to move that to a separate shared library as well? - Mark From mark.reinhold at oracle.com Wed Nov 9 07:42:01 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 09 Nov 2011 07:42:01 -0800 Subject: Ordering of versions In-Reply-To: gnormington@vmware.com; Wed, 09 Nov 2011 09:08:45 GMT; <0AFBDA9F-F04D-4B82-90DB-56ABDF8B128B@vmware.com> Message-ID: <20111109154201.CDEA1DD3@eggemoggin.niobe.net> 2011/11/9 1:08 -0800, gnormington at vmware.com: > Thanks Mark - that's helpful. One other thing: are qualifiers treated > as strings from an ordering perspective? Not quite. They're tokenized and compared just like the initial part of a version string. - Mark From Alan.Bateman at oracle.com Wed Nov 9 07:47:12 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 09 Nov 2011 15:47:12 +0000 Subject: Review request: module graph clean up In-Reply-To: <20111109152138.8B423DD3@eggemoggin.niobe.net> References: <20111109152138.8B423DD3@eggemoggin.niobe.net> Message-ID: <4EBAA080.7080204@oracle.com> On 09/11/2011 15:21, mark.reinhold at oracle.com wrote: > : > On second thought, what's the motivation for this item? The > java.util.prefs package is pretty small. Why split it out? > > Its implementation includes some native code, at least on > Posix and Windows. Are we going to move that to a separate > shared library as well? > > - Mark I have to own up here and say that I mentioned to Mandy recently that I didn't think that the desktop module was the right place for the preferences API. I don't think it's an API that even got too much traction but I think should be usable in contexts where the desktop module is not installed. It is small and a good candidate for aggregation. The native methods are currently in libjava, so in the base module. -Alan. From gnormington at vmware.com Wed Nov 9 07:48:37 2011 From: gnormington at vmware.com (Glyn Normington) Date: Wed, 9 Nov 2011 15:48:37 +0000 Subject: Ordering of versions References: <21AAFBC1-3217-4FF6-BCE2-05230C45701D@vmware.com> Message-ID: <116F3FB6-2127-46CA-9C7E-7400F3101B0C@vmware.com> Forgot to reply to the list. Regards, Glyn Begin forwarded message: > From: Glyn Normington > Subject: Re: Ordering of versions > Date: 9 November 2011 15:47:58 GMT > To: mark.reinhold at oracle.com > > I see. How are the tokens of a qualifier compared? If they are compared just like the initial part, then that would be a numeric comparison for numeric tokens, but what about non-numeric tokens like BETA and GA? Are tokens compared as strings if at least one of the tokens being compared is non-numeric? > > Or I can read the code if you prefer to send me a link. ;-) > > Regards, > Glyn > > On 9 Nov 2011, at 15:42, mark.reinhold at oracle.com wrote: > >> 2011/11/9 1:08 -0800, gnormington at vmware.com: >>> Thanks Mark - that's helpful. One other thing: are qualifiers treated >>> as strings from an ordering perspective? >> >> Not quite. They're tokenized and compared just like the initial part >> of a version string. >> >> - Mark > From Roger.Riggs at oracle.com Wed Nov 9 07:52:48 2011 From: Roger.Riggs at oracle.com (Roger Riggs) Date: Wed, 09 Nov 2011 10:52:48 -0500 Subject: Review request: module graph clean up In-Reply-To: <4EBAA080.7080204@oracle.com> References: <20111109152138.8B423DD3@eggemoggin.niobe.net> <4EBAA080.7080204@oracle.com> Message-ID: <4EBAA1D0.4050808@oracle.com> Hi Alan, Perhaps the desktop module isn't the right place for prefs, but it does not belong in the base module. The base module is for essential APIs that all developers need regardless of the deployment target. For embedded target devices, the preferences API has no use and is just dead-code, even if it is small. The base module is IMHO already too large and includes APIs not needed for all target devices and is over budget. Roger On 11/09/2011 10:47 AM, Alan Bateman wrote: > On 09/11/2011 15:21, mark.reinhold at oracle.com wrote: >> : >> On second thought, what's the motivation for this item? The >> java.util.prefs package is pretty small. Why split it out? >> >> Its implementation includes some native code, at least on >> Posix and Windows. Are we going to move that to a separate >> shared library as well? >> >> - Mark > I have to own up here and say that I mentioned to Mandy recently that > I didn't think that the desktop module was the right place for the > preferences API. I don't think it's an API that even got too much > traction but I think should be usable in contexts where the desktop > module is not installed. It is small and a good candidate for > aggregation. The native methods are currently in libjava, so in the > base module. > > -Alan. From mark.reinhold at oracle.com Wed Nov 9 07:55:49 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 09 Nov 2011 07:55:49 -0800 Subject: Review request: module graph clean up In-Reply-To: alan.bateman@oracle.com; Wed, 09 Nov 2011 15:47:12 GMT; <4EBAA080.7080204@oracle.com> Message-ID: <20111109155549.3586FDD3@eggemoggin.niobe.net> 2011/11/9 7:47 -0800, alan.bateman at oracle.com: > On 09/11/2011 15:21, mark.reinhold at oracle.com wrote: >> On second thought, what's the motivation for this item? The >> java.util.prefs package is pretty small. Why split it out? > > I have to own up here and say that I mentioned to Mandy recently that I didn't > think that the desktop module was the right place for the preferences API. I > don't think it's an API that even got too much traction but I think should be > usable in contexts where the desktop module is not installed. Agreed. > It is small and a > good candidate for aggregation. The native methods are currently in libjava, so > in the base module. Sounds right to me. - Mark From eric at tibco.com Wed Nov 9 08:02:20 2011 From: eric at tibco.com (Eric Johnson) Date: Wed, 9 Nov 2011 17:02:20 +0100 Subject: Converting plain JARs to Java modules In-Reply-To: <4EBA8885.6020808@redhat.com> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> Message-ID: <4EBAA40C.9020403@tibco.com> We've struggled with this question in conjunction with OSGi. We started out using OSGi's require-bundle - and got badly burned as certain Java packages moved around from bundle to bundle as we switched operating environments. Trouble really stems from Java's language design, more than anything else - language namespaces (a.k.a. packages) aren't associated with a JAR, but have a separate logical existence. This means that at runtime at least, you need dependency information at a package level. You may want less fine-grained dependencies at other times (say, for example, packaging for the Linux platform). At least if you have package-level information, you can likely generate correct dependency information at the module level, but definitely *not* the other way around. -Eric. On 11/9/11 3:04 PM, David M. Lloyd wrote: > I'll just state now that using packages as a dependency unit is a > terrible idea, and not some architectural revelation. That way, > Peter's wrath will be largely directed at me. :-) > > On 11/09/2011 08:02 AM, Peter Kriens wrote: >> I agree that tools are needed but we must be careful to not expect >> tools to stopgap an architectural issue. I think it is important to >> first do good architectural design leveraging existing tools (e.g. >> the Java type system) before you try to add new tools. It is such a >> pity (but all to common) that a design allows for classes of errors >> that would be impossible with a slightly different design. >> >> Kind regards, >> >> Peter Kriens >> >> >> >> >> >> >> >> On 9 nov. 2011, at 14:49, Alan Bateman wrote: >> >>> On 09/11/2011 13:04, Peter Kriens wrote: >>>> The issue is that maven problems are not caused because maven is >>>> bad or that pom authors are stupid. The reason is that the >>>> module-to-module dependency architecture in maven (and Jigsaw) is >>>> error prone ... >>> This thread started out with someone asking about adding module >>> declarations to existing JAR files, and in that context, I agree it >>> can be error prone without good tools. I think things should be a >>> lot better when modules are compiled. >>> >>> -Alan. >>> >> > > From Alan.Bateman at oracle.com Wed Nov 9 08:11:16 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 09 Nov 2011 16:11:16 +0000 Subject: Review request: module graph clean up In-Reply-To: <4EBAA1D0.4050808@oracle.com> References: <20111109152138.8B423DD3@eggemoggin.niobe.net> <4EBAA080.7080204@oracle.com> <4EBAA1D0.4050808@oracle.com> Message-ID: <4EBAA624.9020701@oracle.com> On 09/11/2011 15:52, Roger Riggs wrote: > Hi Alan, > > Perhaps the desktop module isn't the right place for prefs, > but it does not belong in the base module. The base module > is for essential APIs that all developers need regardless of the > deployment target. For embedded target devices, the preferences > API has no use and is just dead-code, even if it is small. > The base module is IMHO already too large and includes APIs > not needed for all target devices and is over budget. The preferences API isn't commonly used and I'm not suggesting it be moved into the base module, I was just mentioning that this is where the native code is now. Mandy's change puts the preferences API into its own module, which I think is okay for now but we do need I think we do need to look aggregation of some of the very fine grain modules at some point. -Alan. From mark.reinhold at oracle.com Wed Nov 9 08:28:20 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 09 Nov 2011 08:28:20 -0800 Subject: Review request: module graph clean up In-Reply-To: roger.riggs@oracle.com; Wed, 09 Nov 2011 10:52:48 EST; <4EBAA1D0.4050808@oracle.com> Message-ID: <20111109162820.E1FFEDD3@eggemoggin.niobe.net> 2011/11/9 7:52 -0800, roger.riggs at oracle.com: > Perhaps the desktop module isn't the right place for prefs, > but it does not belong in the base module. The base module > is for essential APIs that all developers need regardless of the > deployment target. For embedded target devices, the preferences > API has no use and is just dead-code, even if it is small. > The base module is IMHO already too large and includes APIs > not needed for all target devices and is over budget. 86KB of class files + 3KB native code (uncompressed on Posix; Windows sizes are probably similar) hardly seems worth splitting out. We definitely want to make sure that JDK 8 is useful on small devices, but we must balance that with the overall complexity of the module graph. The module graph is, essentially, a kind of API. Developers are going to have to learn its structure and be able to reason about it. The more complex we make it, the harder that's going to be. In general I think we should do everything we can to optimize for space systemically before we consider splitting modules much beyond what we have now. Simply compressing class files when installing modules, e.g., reduces the class content of the current base module from 10 to 5.3MB. For now, java.util.prefs should remain in the base module. - Mark From njbartlett at gmail.com Wed Nov 9 08:35:51 2011 From: njbartlett at gmail.com (Neil Bartlett) Date: Wed, 9 Nov 2011 16:35:51 +0000 Subject: Converting plain JARs to Java modules In-Reply-To: <4EBA8885.6020808@redhat.com> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> Message-ID: And I'll just state that David is wrong, but that doesn't really get us anywhere. Neil On Wed, Nov 9, 2011 at 2:04 PM, David M. Lloyd wrote: > I'll just state now that using packages as a dependency unit is a terrible > idea, and not some architectural revelation. ?That way, Peter's wrath will > be largely directed at me. :-) > > On 11/09/2011 08:02 AM, Peter Kriens wrote: >> >> I agree that tools are needed but we must be careful to not expect tools >> to stopgap an architectural issue. I think it is important to first do good >> architectural design leveraging existing tools (e.g. the Java type system) >> before you try to add new tools. It is such a pity (but all to common) that >> a design allows for classes of errors that would be impossible with a >> slightly different design. >> >> Kind regards, >> >> ? ? ? ?Peter Kriens >> >> >> >> >> >> >> >> On 9 nov. 2011, at 14:49, Alan Bateman wrote: >> >>> On 09/11/2011 13:04, Peter Kriens wrote: >>>> >>>> The issue is that maven problems are not caused because maven is bad or >>>> that pom authors are stupid. The reason is that the module-to-module >>>> dependency architecture in maven (and Jigsaw) is error prone ... >>> >>> This thread started out with someone asking about adding module >>> declarations to existing JAR files, and in that context, I agree it can be >>> error prone without good tools. I think things should be a lot better when >>> modules are compiled. >>> >>> -Alan. >>> >> > > > -- > - DML > From mark.reinhold at oracle.com Wed Nov 9 08:37:04 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 09 Nov 2011 08:37:04 -0800 Subject: Ordering of versions In-Reply-To: gnormington@vmware.com; Wed, 09 Nov 2011 15:47:58 GMT; <21AAFBC1-3217-4FF6-BCE2-05230C45701D@vmware.com> Message-ID: <20111109163704.44A48DD3@eggemoggin.niobe.net> 2011/11/9 7:47 -0800, gnormington at vmware.com: > I see. How are the tokens of a qualifier compared? If they are compared > just like the initial part, Yes. > then that would be a numeric comparison for > numeric tokens, but what about non-numeric tokens like BETA and GA? They are compared lexicographically, so BETA < GA. > Are > tokens compared as strings if at least one of the tokens being compared > is non-numeric? Yes. > Or I can read the code if you prefer to send me a link. ;-) http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/file/960eb03d1270/src/share/classes/org/openjdk/jigsaw/JigsawVersion.java - Mark From gnormington at vmware.com Wed Nov 9 08:43:30 2011 From: gnormington at vmware.com (Glyn Normington) Date: Wed, 9 Nov 2011 16:43:30 +0000 Subject: Ordering of versions In-Reply-To: <20111109163704.44A48DD3@eggemoggin.niobe.net> References: <20111109163704.44A48DD3@eggemoggin.niobe.net> Message-ID: Gotcha. Thanks for your patience. Regards, Glyn On 9 Nov 2011, at 16:37, mark.reinhold at oracle.com wrote: > 2011/11/9 7:47 -0800, gnormington at vmware.com: >> I see. How are the tokens of a qualifier compared? If they are compared >> just like the initial part, > > Yes. > >> then that would be a numeric comparison for >> numeric tokens, but what about non-numeric tokens like BETA and GA? > > They are compared lexicographically, so BETA < GA. > >> Are >> tokens compared as strings if at least one of the tokens being compared >> is non-numeric? > > Yes. > >> Or I can read the code if you prefer to send me a link. ;-) > > http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/file/960eb03d1270/src/share/classes/org/openjdk/jigsaw/JigsawVersion.java > > - Mark From njbartlett at gmail.com Wed Nov 9 08:46:54 2011 From: njbartlett at gmail.com (Neil Bartlett) Date: Wed, 9 Nov 2011 16:46:54 +0000 Subject: Converting plain JARs to Java modules In-Reply-To: <4EBAA40C.9020403@tibco.com> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> <4EBAA40C.9020403@tibco.com> Message-ID: I think this point is key: runtime dependencies are not necessarily the same as build-time dependencies. In fact they are usually NOT the same for various reasons, not least of which is that running a module requires all of its transitive dependencies to be present whereas building it only requires its first-level dependencies. At runtime, dependencies MUST be based on packages... experience in OSGi and Eclipse has shown that again and again. On the other hand, build-time dependencies can be written at module level, indeed most developers I have worked with tend to think about the "libraries" they are using rather than the Java packages. The mistake made in tools like Eclipse PDE was to tie these two together: if the developer wanted to use package-level imports then they had to maintain those package imports manually in the tool. This became unmanageable so they switched to using module-level dependencies, which made life easier at build time but was the wrong answer for runtime. The Java 8 Module System requirements document appears to be making the same mistake when it talks about "fidelity across all phases", though that may be a matter of interpretation. Neil On Wed, Nov 9, 2011 at 4:02 PM, Eric Johnson wrote: > We've struggled with this question in conjunction with OSGi. > > We started out using OSGi's require-bundle - and got badly burned as certain > Java packages moved around from bundle to bundle as we switched operating > environments. Trouble really stems from Java's language design, more than > anything else - language namespaces (a.k.a. packages) aren't associated with > a JAR, but have a separate logical existence. This means that at runtime at > least, you need dependency information at a package level. > > You may want less fine-grained dependencies at other times (say, for > example, packaging for the Linux platform). At least if you have > package-level information, you can likely generate correct dependency > information at the module level, but definitely *not* the other way around. > > -Eric. > > On 11/9/11 3:04 PM, David M. Lloyd wrote: >> >> I'll just state now that using packages as a dependency unit is a terrible >> idea, and not some architectural revelation. ?That way, Peter's wrath will >> be largely directed at me. :-) >> >> On 11/09/2011 08:02 AM, Peter Kriens wrote: >>> >>> I agree that tools are needed but we must be careful to not expect tools >>> to stopgap an architectural issue. I think it is important to first do good >>> architectural design leveraging existing tools (e.g. the Java type system) >>> before you try to add new tools. It is such a pity (but all to common) that >>> a design allows for classes of errors that would be impossible with a >>> slightly different design. >>> >>> Kind regards, >>> >>> ? ?Peter Kriens >>> >>> >>> >>> >>> >>> >>> >>> On 9 nov. 2011, at 14:49, Alan Bateman wrote: >>> >>>> On 09/11/2011 13:04, Peter Kriens wrote: >>>>> >>>>> The issue is that maven problems are not caused because maven is bad or >>>>> that pom authors are stupid. The reason is that the module-to-module >>>>> dependency architecture in maven (and Jigsaw) is error prone ... >>>> >>>> This thread started out with someone asking about adding module >>>> declarations to existing JAR files, and in that context, I agree it can be >>>> error prone without good tools. I think things should be a lot better when >>>> modules are compiled. >>>> >>>> -Alan. >>>> >>> >> >> > From njbartlett at gmail.com Wed Nov 9 08:59:21 2011 From: njbartlett at gmail.com (Neil Bartlett) Date: Wed, 9 Nov 2011 16:59:21 +0000 Subject: Ordering of versions In-Reply-To: <20111109163704.44A48DD3@eggemoggin.niobe.net> References: <21AAFBC1-3217-4FF6-BCE2-05230C45701D@vmware.com> <20111109163704.44A48DD3@eggemoggin.niobe.net> Message-ID: Forgive me if I have missed something, but if a version segment/qualifier can be interpreted as either a number or a string, depending on whether it is compared against another number or string, then versions are not totally ordered. For example: "1B" is greater than "10" 10 is greater than 2 "2" is greater than "1B" Therefore "1B" is greater than "1B". Neil On Wed, Nov 9, 2011 at 4:37 PM, wrote: > 2011/11/9 7:47 -0800, gnormington at vmware.com: >> I see. How are the tokens of a qualifier compared? If they are compared >> just like the initial part, > > Yes. > >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? then that would be a numeric comparison for >> numeric tokens, but what about non-numeric tokens like BETA and GA? > > They are compared lexicographically, so BETA < GA. > >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Are >> tokens compared as strings if at least one of the tokens being compared >> is non-numeric? > > Yes. > >> Or I can read the code if you prefer to send me a link. ;-) > > http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/file/960eb03d1270/src/share/classes/org/openjdk/jigsaw/JigsawVersion.java > > - Mark > From mark.reinhold at oracle.com Wed Nov 9 09:20:25 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 09 Nov 2011 09:20:25 -0800 Subject: Ordering of versions In-Reply-To: njbartlett@gmail.com; Wed, 09 Nov 2011 16:59:21 GMT; Message-ID: <20111109172025.3F1C8DD3@eggemoggin.niobe.net> 2011/11/9 8:59 -0800, njbartlett at gmail.com: > Forgive me if I have missed something, but if a version > segment/qualifier can be interpreted as either a number or a string, > depending on whether it is compared against another number or string, > then versions are not totally ordered. > > For example: > > "1B" is greater than "10" No, "1B" < "10" because "1B" is parsed as two tokens, "1" and "B", and "1" < "10". - Mark From Alan.Bateman at oracle.com Wed Nov 9 10:21:07 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 09 Nov 2011 18:21:07 +0000 Subject: Compressing class files in the module library Message-ID: <4EBAC493.8080307@oracle.com> I have an initial patch [1] to compress the classes of modules in the module library. It essentially adds an option to "jmod create" so that the classes of any modules installed into the library get compressed. The patch also adds a build option to enable compression in the system libraries of the images that the build generates. Before finishing this I'd like to get some input on a few points. The first is whether it's reasonable to enable compression when creating the module library? The alternative is install time so that it's on a per module basis. The former is less flexible but as the module library format evolves it may not be feasible to have a mix of compressed and uncompressed modules. The other question is suggestions for the option name. For now it's -9 or --enable-compression (-9 influenced by the zip -9 option for compress better). I'm looking for better suggestions. Thanks, -Alan. [1] http://cr.openjdk.java.net/~alanb/ModuleLibraryCompression/webrev/ From njbartlett at gmail.com Wed Nov 9 10:45:16 2011 From: njbartlett at gmail.com (Neil Bartlett) Date: Wed, 9 Nov 2011 18:45:16 +0000 Subject: Ordering of versions In-Reply-To: <20111109172025.3F1C8DD3@eggemoggin.niobe.net> References: <20111109172025.3F1C8DD3@eggemoggin.niobe.net> Message-ID: What determines the split into 2 tokens, i.e. why is "1B" parsed as 2 tokens whereas "10" is just 1 token? Does this mean that 1B is equivalent to 1.B? Is "beta" parsed as 4 tokens, b.e.t.a? Perhaps I should read the code as Glyn is doing, but a proper description of how the versioning is *intended* to work would be better. Rgds Neil On Wed, Nov 9, 2011 at 5:20 PM, wrote: > 2011/11/9 8:59 -0800, njbartlett at gmail.com: >> Forgive me if I have missed something, but if a version >> segment/qualifier can be interpreted as either a number or a string, >> depending on whether it is compared against another number or string, >> then versions are not totally ordered. >> >> For example: >> >> "1B" is greater than "10" > > No, "1B" < "10" because "1B" is parsed as two tokens, "1" and "B", > and "1" < "10". > > - Mark > From brian at pontarelli.com Wed Nov 9 11:02:42 2011 From: brian at pontarelli.com (Brian Pontarelli) Date: Wed, 9 Nov 2011 12:02:42 -0700 Subject: Ordering of versions In-Reply-To: <20111109163704.44A48DD3@eggemoggin.niobe.net> References: <20111109163704.44A48DD3@eggemoggin.niobe.net> Message-ID: <1535DFDE-DD79-4490-9C5C-14FBA276D12F@pontarelli.com> It seems obvious from your example that lexicographic is probably going to cause issues and confusion. Perhaps it makes sense to define a set of well know and commonly used names and the order they have. I would assume this would be simple to get a majority on. Sent from my iPhone On Nov 9, 2011, at 9:37 AM, mark.reinhold at oracle.com wrote: > 2011/11/9 7:47 -0800, gnormington at vmware.com: >> I see. How are the tokens of a qualifier compared? If they are compared >> just like the initial part, > > Yes. > >> then that would be a numeric comparison for >> numeric tokens, but what about non-numeric tokens like BETA and GA? > > They are compared lexicographically, so BETA < GA. > >> Are >> tokens compared as strings if at least one of the tokens being compared >> is non-numeric? > > Yes. > >> Or I can read the code if you prefer to send me a link. ;-) > > http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/file/960eb03d1270/src/share/classes/org/openjdk/jigsaw/JigsawVersion.java > > - Mark From mandy.chung at oracle.com Wed Nov 9 11:17:30 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 09 Nov 2011 11:17:30 -0800 Subject: Review request: module graph clean up In-Reply-To: <20111109162820.E1FFEDD3@eggemoggin.niobe.net> References: <20111109162820.E1FFEDD3@eggemoggin.niobe.net> Message-ID: <4EBAD1CA.8060008@oracle.com> On 11/9/11 8:28 AM, mark.reinhold at oracle.com wrote: > For now, java.util.prefs should remain in the base module. Looking at the preferences implementation further, I realize that preferences for Unix are stored in the file system in XML format. This might require further work to eliminate this XML dependency before we can move it back to the base module. For now, we could move it out from the desktop module and keep it as a separate module and move it back to the base module when we eliminate its XML dependency. Mandy From eric at tibco.com Wed Nov 9 11:22:58 2011 From: eric at tibco.com (Eric Johnson) Date: Wed, 9 Nov 2011 20:22:58 +0100 Subject: Ordering of versions In-Reply-To: <1535DFDE-DD79-4490-9C5C-14FBA276D12F@pontarelli.com> References: <20111109163704.44A48DD3@eggemoggin.niobe.net> <1535DFDE-DD79-4490-9C5C-14FBA276D12F@pontarelli.com> Message-ID: <4EBAD312.1000405@tibco.com> Is it even possible to solve this problem with just a version string? Maven and OSGi (currently) have incompatible version schemes. In some cases, what Maven would consider less than another version number, OSGi would consider greater (the "snapshot" problem). The only general way that I can think of to solve this is to indicate somewhere with metadata which version string semantics are in effect, and apply those semantics within the scope of the indicated metadata. -Eric. On 11/9/11 8:02 PM, Brian Pontarelli wrote: > It seems obvious from your example that lexicographic is probably going to cause issues and confusion. Perhaps it makes sense to define a set of well know and commonly used names and the order they have. I would assume this would be simple to get a majority on. > > Sent from my iPhone > > On Nov 9, 2011, at 9:37 AM, mark.reinhold at oracle.com wrote: > >> 2011/11/9 7:47 -0800, gnormington at vmware.com: >>> I see. How are the tokens of a qualifier compared? If they are compared >>> just like the initial part, >> Yes. >> >>> then that would be a numeric comparison for >>> numeric tokens, but what about non-numeric tokens like BETA and GA? >> They are compared lexicographically, so BETA< GA. >> >>> Are >>> tokens compared as strings if at least one of the tokens being compared >>> is non-numeric? >> Yes. >> >>> Or I can read the code if you prefer to send me a link. ;-) >> http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/file/960eb03d1270/src/share/classes/org/openjdk/jigsaw/JigsawVersion.java >> >> - Mark From njbartlett at gmail.com Wed Nov 9 11:45:01 2011 From: njbartlett at gmail.com (Neil Bartlett) Date: Wed, 9 Nov 2011 19:45:01 +0000 Subject: Ordering of versions In-Reply-To: <4EBAD312.1000405@tibco.com> References: <20111109163704.44A48DD3@eggemoggin.niobe.net> <1535DFDE-DD79-4490-9C5C-14FBA276D12F@pontarelli.com> <4EBAD312.1000405@tibco.com> Message-ID: Right: there is no general way to tell the order of two versions *just* by looking at the version strings. You need to know the scheme that is force, and there are multiple possible schemes, some of which are more sane and intuitive than others. The Java 8 module system needs to choose a version scheme that is, at the very least, totally ordered... otherwise the version string is meaningless, and is just a part of the module identity. The next most important concern is that the scheme should be intuitive with respect to ordering, and follow the principle of least surprise. Next I would suggest that the scheme should support sensible semantics for the meaning of each version segment with regard to signalling the scope of a change. The final concern might be compatibility with existing practice, which essentially boils down to OSGi and Maven (incidentally, I hear that these will be compatible in the next OSGi specification revision, with the introduction of "negative qualifiers".. basically the same as snapshots). I would submit that Mark's description of the Jigsaw versions and tokens, though I have not fully understood it yet, fails the intuition test. That is unless my intuition has been distorted by 6-7 years of OSGi. Rgds, Neil On Wed, Nov 9, 2011 at 7:22 PM, Eric Johnson wrote: > Is it even possible to solve this problem with just a version string? > > Maven and OSGi (currently) have incompatible version schemes. In some cases, > what Maven would consider less than another version number, OSGi would > consider greater (the "snapshot" problem). > > The only general way that I can think of to solve this is to indicate > somewhere with metadata which version string semantics are in effect, and > apply those semantics within the scope of the indicated metadata. > > -Eric. > > On 11/9/11 8:02 PM, Brian Pontarelli wrote: >> >> It seems obvious from your example that lexicographic is probably going to >> cause issues and confusion. Perhaps it makes sense to define a set of well >> know and commonly used names and the order they have. I would assume this >> would be simple to get a majority on. >> >> Sent from my iPhone >> >> On Nov 9, 2011, at 9:37 AM, mark.reinhold at oracle.com wrote: >> >>> 2011/11/9 7:47 -0800, gnormington at vmware.com: >>>> >>>> I see. How are the tokens of a qualifier compared? If they are compared >>>> just like the initial part, >>> >>> Yes. >>> >>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ?then that would be a numeric comparison for >>>> numeric tokens, but what about non-numeric tokens like BETA and GA? >>> >>> They are compared lexicographically, so BETA< ?GA. >>> >>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Are >>>> tokens compared as strings if at least one of the tokens being compared >>>> is non-numeric? >>> >>> Yes. >>> >>>> Or I can read the code if you prefer to send me a link. ;-) >>> >>> >>> http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/file/960eb03d1270/src/share/classes/org/openjdk/jigsaw/JigsawVersion.java >>> >>> - Mark > From peter.jensen at oracle.com Wed Nov 9 12:18:27 2011 From: peter.jensen at oracle.com (Peter Jensen) Date: Wed, 09 Nov 2011 12:18:27 -0800 Subject: Runtime impact of compressing class files in the module library In-Reply-To: <4EBAC60B.7090603@oracle.com> References: <4EBAC60B.7090603@oracle.com> Message-ID: <4EBAE013.3070905@oracle.com> Just joining the alias. I wrote a test program that measures the time to 1/ read all API classes in a jar file, in jar entry order 2/ load all the same classes by name, in order of Hashset iteration (set populated during read test). Gary Adam's helped run this on a BeagleBoard, at different CPU speed, using an Embedded SE build, with a compressed vs. uncompressed rt.jar. Note: Embedded SE compress by default. I'm not sure if it's 'zip -9' or standard zip compression. Both test show linear dependency between time and cpu speed. That is, the workload consist of a component inversely proportional to CPU frequency, and a constant component (e.g. IO wait time). The CPU sensitive component shows a worst-case class-loading overhead of 7 to 6 (17%) (worst-case is for an infinitely fast storage. With high CPU to IO performance you get better results). This for an approx. 50% reduction in jar file size. Results and regression analysis attached. -------- Original Message -------- Subject: Fwd: Compressing class files in the module library Date: Wed, 09 Nov 2011 13:27:23 -0500 From: Gary Adams fyi -------- Original Message -------- Subject: Compressing class files in the module library Date: Wed, 09 Nov 2011 18:21:07 +0000 From: Alan Bateman To: jigsaw-dev I have an initial patch [1] to compress the classes of modules in the module library. It essentially adds an option to "jmod create" so that the classes of any modules installed into the library get compressed. The patch also adds a build option to enable compression in the system libraries of the images that the build generates. Before finishing this I'd like to get some input on a few points. The first is whether it's reasonable to enable compression when creating the module library? The alternative is install time so that it's on a per module basis. The former is less flexible but as the module library format evolves it may not be feasible to have a mix of compressed and uncompressed modules. The other question is suggestions for the option name. For now it's -9 or --enable-compression (-9 influenced by the zip -9 option for compress better). I'm looking for better suggestions. Thanks, -Alan. [1]http://cr.openjdk.java.net/~alanb/ModuleLibraryCompression/webrev/ From peter.jensen at oracle.com Wed Nov 9 12:51:43 2011 From: peter.jensen at oracle.com (Peter Jensen) Date: Wed, 09 Nov 2011 12:51:43 -0800 Subject: Runtime impact of compressing class files in the module library In-Reply-To: <4EBAE013.3070905@oracle.com> References: <4EBAC60B.7090603@oracle.com> <4EBAE013.3070905@oracle.com> Message-ID: <4EBAE7DF.3050808@oracle.com> No attachments, okay! Here's data in csv format (without the nice diagrams:-) "Read jar entries sequentially",,"Compressed",,"Uncompressed",,"Read Overhead" "CPU Speed (GHz)","msec/M-CPU","Read (msec)","Read Model","Read 0 (msec)","Read 0 Model", 0.3,3.33,8735,8728,4011,4006,118% 0.6,1.67,4668,4689,2156,2177,115% 0.8,1.25,3674,3679,1731,1720,114% 1,1.00,3093,3073,1451,1446,113% ,0,,650,,348, "Linear regression",,,,,, "Constant (msec)",,,650,,348, "M-CPU cycles",,,2424,,1097, "Correlation",,,0.99998,,0.99992, ,,,,,, ,,,,,, "Load Classes ? Hashset order",,"Compressed",,"Uncompressed",,"Load Overhead" "CPU Speed (GHz)","msec/M-CPU","Load (msec)","Load Model","Load 0 (msec)","Load 0 Model", 0.3,3.33,27904,27888,25010,24939,12% 0.6,1.67,15891,15944,14702,14865,7% 0.8,1.25,12947,12958,12116,12347,5% 1,1.00,11213,11166,11158,10836,3% ,0,,3999,,4791, "Linear regression",,,,,, "Constant (msec)",,,3999,,4791, "M-CPU cycles",,,7167,,6044, "Correlation",,,0.99998,,0.99922, On 11/09/11 12:18, Peter Jensen wrote: > Just joining the alias. > > I wrote a test program that measures the time to > 1/ read all API classes in a jar file, in jar entry order > 2/ load all the same classes by name, in order of Hashset iteration > (set populated during read test). > > Gary Adam's helped run this on a BeagleBoard, at different CPU speed, > using an Embedded SE build, > with a compressed vs. uncompressed rt.jar. Note: Embedded SE compress > by default. I'm not sure > if it's 'zip -9' or standard zip compression. > > Both test show linear dependency between time and cpu speed. That is, > the workload > consist of a component inversely proportional to CPU frequency, and a > constant component > (e.g. IO wait time). > > The CPU sensitive component shows a worst-case class-loading overhead > of 7 to 6 (17%) > (worst-case is for an infinitely fast storage. With high CPU to IO > performance you get better results). > > This for an approx. 50% reduction in jar file size. > > Results and regression analysis attached. > > -------- Original Message -------- > Subject: Fwd: Compressing class files in the module library > Date: Wed, 09 Nov 2011 13:27:23 -0500 > From: Gary Adams > > > > > > fyi > > -------- Original Message -------- > Subject: Compressing class files in the module library > Date: Wed, 09 Nov 2011 18:21:07 +0000 > From: Alan Bateman > To: jigsaw-dev > > > > I have an initial patch [1] to compress the classes of modules in the > module library. It essentially adds an option to "jmod create" so that > the classes of any modules installed into the library get compressed. > The patch also adds a build option to enable compression in the system > libraries of the images that the build generates. > > Before finishing this I'd like to get some input on a few points. The > first is whether it's reasonable to enable compression when creating the > module library? The alternative is install time so that it's on a per > module basis. The former is less flexible but as the module library > format evolves it may not be feasible to have a mix of compressed and > uncompressed modules. > > The other question is suggestions for the option name. For now it's -9 > or --enable-compression (-9 influenced by the zip -9 option for > compress better). I'm looking for better suggestions. > > Thanks, > > -Alan. > > [1]http://cr.openjdk.java.net/~alanb/ModuleLibraryCompression/webrev/ > From david.lloyd at redhat.com Wed Nov 9 12:58:16 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 09 Nov 2011 14:58:16 -0600 Subject: Ordering of versions In-Reply-To: References: <20111109163704.44A48DD3@eggemoggin.niobe.net> <1535DFDE-DD79-4490-9C5C-14FBA276D12F@pontarelli.com> <4EBAD312.1000405@tibco.com> Message-ID: <4EBAE968.8040408@redhat.com> Agreed 110% about version schemes. Rather than trying to encompass all schemes, the version scheme should be dependent on the library implementation... in fact it may not even make sense to require modules to even have a version in all cases. The Java module system must embrace the fact that there are many approaches to versioning, and while its default library implementation(s) may support a specific scheme, other schemes should be allowed. On 11/09/2011 01:45 PM, Neil Bartlett wrote: > Right: there is no general way to tell the order of two versions > *just* by looking at the version strings. You need to know the scheme > that is force, and there are multiple possible schemes, some of which > are more sane and intuitive than others. > > The Java 8 module system needs to choose a version scheme that is, at > the very least, totally ordered... otherwise the version string is > meaningless, and is just a part of the module identity. The next most > important concern is that the scheme should be intuitive with respect > to ordering, and follow the principle of least surprise. Next I would > suggest that the scheme should support sensible semantics for the > meaning of each version segment with regard to signalling the scope of > a change. The final concern might be compatibility with existing > practice, which essentially boils down to OSGi and Maven > (incidentally, I hear that these will be compatible in the next OSGi > specification revision, with the introduction of "negative > qualifiers".. basically the same as snapshots). > > I would submit that Mark's description of the Jigsaw versions and > tokens, though I have not fully understood it yet, fails the intuition > test. That is unless my intuition has been distorted by 6-7 years of > OSGi. > > Rgds, > Neil > > > > On Wed, Nov 9, 2011 at 7:22 PM, Eric Johnson wrote: >> Is it even possible to solve this problem with just a version string? >> >> Maven and OSGi (currently) have incompatible version schemes. In some cases, >> what Maven would consider less than another version number, OSGi would >> consider greater (the "snapshot" problem). >> >> The only general way that I can think of to solve this is to indicate >> somewhere with metadata which version string semantics are in effect, and >> apply those semantics within the scope of the indicated metadata. >> >> -Eric. >> >> On 11/9/11 8:02 PM, Brian Pontarelli wrote: >>> >>> It seems obvious from your example that lexicographic is probably going to >>> cause issues and confusion. Perhaps it makes sense to define a set of well >>> know and commonly used names and the order they have. I would assume this >>> would be simple to get a majority on. >>> >>> Sent from my iPhone >>> >>> On Nov 9, 2011, at 9:37 AM, mark.reinhold at oracle.com wrote: >>> >>>> 2011/11/9 7:47 -0800, gnormington at vmware.com: >>>>> >>>>> I see. How are the tokens of a qualifier compared? If they are compared >>>>> just like the initial part, >>>> >>>> Yes. >>>> >>>>> then that would be a numeric comparison for >>>>> numeric tokens, but what about non-numeric tokens like BETA and GA? >>>> >>>> They are compared lexicographically, so BETA< GA. >>>> >>>>> Are >>>>> tokens compared as strings if at least one of the tokens being compared >>>>> is non-numeric? >>>> >>>> Yes. >>>> >>>>> Or I can read the code if you prefer to send me a link. ;-) >>>> >>>> >>>> http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/file/960eb03d1270/src/share/classes/org/openjdk/jigsaw/JigsawVersion.java >>>> >>>> - Mark >> -- - DML From mark.reinhold at oracle.com Wed Nov 9 13:57:22 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 09 Nov 2011 13:57:22 -0800 Subject: Ordering of versions In-Reply-To: njbartlett@gmail.com; Wed, 09 Nov 2011 18:45:16 GMT; Message-ID: <20111109215722.4684E111B@eggemoggin.niobe.net> 2011/11/9 10:45 -0800, njbartlett at gmail.com: > What determines the split into 2 tokens, i.e. why is "1B" parsed as 2 > tokens whereas "10" is just 1 token? Version strings are tokenized at alpha/numeric/punctuation boundaries, and punctuation is discarded. > Does this mean that 1B is > equivalent to 1.B? Yes. > Is "beta" parsed as 4 tokens, b.e.t.a? No, it's just "beta". > Perhaps I should read the code as Glyn is doing, but a proper > description of how the versioning is *intended* to work would be > better. No argument there. The prototype is inspired by, though not identical to, Debian's version-comparison algorithm [1]. We've intentionally not spent a lot of time designing and specifying this yet because we know there is such a wide variety of opinions and experiences to be taken into account, as demonstrated by this very thread. - Mark [1] http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Version From mark.reinhold at oracle.com Wed Nov 9 14:19:47 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 09 Nov 2011 14:19:47 -0800 Subject: Ordering of versions In-Reply-To: njbartlett@gmail.com; Wed, 09 Nov 2011 19:45:01 GMT; Message-ID: <20111109221947.E48FD111B@eggemoggin.niobe.net> 2011/11/9 11:45 -0800, njbartlett at gmail.com: > Right: there is no general way to tell the order of two versions > *just* by looking at the version strings. You need to know the scheme > that is force, and there are multiple possible schemes, some of which > are more sane and intuitive than others. ... and some of which effectively contain others. > The Java 8 module system needs to choose a version scheme that is, at > the very least, totally ordered... otherwise the version string is > meaningless, and is just a part of the module identity. Yep. > The next most > important concern is that the scheme should be intuitive with respect > to ordering, and follow the principle of least surprise. Yep. > Next I would > suggest that the scheme should support sensible semantics for the > meaning of each version segment with regard to signalling the scope of > a change. Support such semantics, sure, but I'm not sure it should prescribe them. OSGi and Maven have established some useful and widely-used conventions around version semantics but I've seen developers and customers use all sorts of other wild and woolly version schemes over the years. I'm not convinced that a standard Java module system should require people to abandon such schemes, unless they're just really totally broken. > The final concern might be compatibility with existing > practice, which essentially boils down to OSGi and Maven > (incidentally, I hear that these will be compatible in the next OSGi > specification revision, with the introduction of "negative > qualifiers".. basically the same as snapshots). The Java standard module system should accommodate the OSGi and Maven schemes if at all possible, but that's not the same thing as adopting precisely one or the other. > I would submit that Mark's description of the Jigsaw versions and > tokens, though I have not fully understood it yet, fails the intuition > test. As I said, it's just an initial prototype. - Mark From brian at pontarelli.com Wed Nov 9 14:20:29 2011 From: brian at pontarelli.com (Brian Pontarelli) Date: Wed, 9 Nov 2011 15:20:29 -0700 Subject: Ordering of versions In-Reply-To: <4EBAE968.8040408@redhat.com> References: <20111109163704.44A48DD3@eggemoggin.niobe.net> <1535DFDE-DD79-4490-9C5C-14FBA276D12F@pontarelli.com> <4EBAD312.1000405@tibco.com> <4EBAE968.8040408@redhat.com> Message-ID: <7D8AB2E8-0E12-47E3-A57E-FAE9305E5A9C@pontarelli.com> This should work even with large graphs. A single module can define its scheme based on some implementation class or "well-known named" scheme. The key is that 99% of Java devs will follow the standard implementation, which should be simple and cover the most common forms. My suggestions for the baseline form is: ::= ("." ("." ("-" )*)*)* ::= digit ::= digit ::= digit ::= digit | digit ::= "alpha" | "a" | "beta" | "b" | "rc" The order of the cycles would be: - Alpha - Beta - Release Candidate The only cycle that I would contend isn't a good idea to allow is "GA". In my opinion a GA is the same as the version without the "extra" element. Therefore, an order would be: 1.0-A1 1.0-A2 1.0-B1 1.0-B2 1.0-RC1 1.0-RC2 1.0 1.0.1 1.0.2 1.1 ? -bp On Nov 9, 2011, at 1:58 PM, David M. Lloyd wrote: > Agreed 110% about version schemes. Rather than trying to encompass all schemes, the version scheme should be dependent on the library implementation... in fact it may not even make sense to require modules to even have a version in all cases. > > The Java module system must embrace the fact that there are many approaches to versioning, and while its default library implementation(s) may support a specific scheme, other schemes should be allowed. > > On 11/09/2011 01:45 PM, Neil Bartlett wrote: >> Right: there is no general way to tell the order of two versions >> *just* by looking at the version strings. You need to know the scheme >> that is force, and there are multiple possible schemes, some of which >> are more sane and intuitive than others. >> >> The Java 8 module system needs to choose a version scheme that is, at >> the very least, totally ordered... otherwise the version string is >> meaningless, and is just a part of the module identity. The next most >> important concern is that the scheme should be intuitive with respect >> to ordering, and follow the principle of least surprise. Next I would >> suggest that the scheme should support sensible semantics for the >> meaning of each version segment with regard to signalling the scope of >> a change. The final concern might be compatibility with existing >> practice, which essentially boils down to OSGi and Maven >> (incidentally, I hear that these will be compatible in the next OSGi >> specification revision, with the introduction of "negative >> qualifiers".. basically the same as snapshots). >> >> I would submit that Mark's description of the Jigsaw versions and >> tokens, though I have not fully understood it yet, fails the intuition >> test. That is unless my intuition has been distorted by 6-7 years of >> OSGi. >> >> Rgds, >> Neil >> >> >> >> On Wed, Nov 9, 2011 at 7:22 PM, Eric Johnson wrote: >>> Is it even possible to solve this problem with just a version string? >>> >>> Maven and OSGi (currently) have incompatible version schemes. In some cases, >>> what Maven would consider less than another version number, OSGi would >>> consider greater (the "snapshot" problem). >>> >>> The only general way that I can think of to solve this is to indicate >>> somewhere with metadata which version string semantics are in effect, and >>> apply those semantics within the scope of the indicated metadata. >>> >>> -Eric. >>> >>> On 11/9/11 8:02 PM, Brian Pontarelli wrote: >>>> >>>> It seems obvious from your example that lexicographic is probably going to >>>> cause issues and confusion. Perhaps it makes sense to define a set of well >>>> know and commonly used names and the order they have. I would assume this >>>> would be simple to get a majority on. >>>> >>>> Sent from my iPhone >>>> >>>> On Nov 9, 2011, at 9:37 AM, mark.reinhold at oracle.com wrote: >>>> >>>>> 2011/11/9 7:47 -0800, gnormington at vmware.com: >>>>>> >>>>>> I see. How are the tokens of a qualifier compared? If they are compared >>>>>> just like the initial part, >>>>> >>>>> Yes. >>>>> >>>>>> then that would be a numeric comparison for >>>>>> numeric tokens, but what about non-numeric tokens like BETA and GA? >>>>> >>>>> They are compared lexicographically, so BETA< GA. >>>>> >>>>>> Are >>>>>> tokens compared as strings if at least one of the tokens being compared >>>>>> is non-numeric? >>>>> >>>>> Yes. >>>>> >>>>>> Or I can read the code if you prefer to send me a link. ;-) >>>>> >>>>> >>>>> http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/file/960eb03d1270/src/share/classes/org/openjdk/jigsaw/JigsawVersion.java >>>>> >>>>> - Mark >>> > > > -- > - DML From brian at pontarelli.com Wed Nov 9 14:26:50 2011 From: brian at pontarelli.com (Brian Pontarelli) Date: Wed, 9 Nov 2011 15:26:50 -0700 Subject: Email threading issues Message-ID: <494CCAC3-A214-43A7-A178-DC950569956D@pontarelli.com> Completely unrelated to Jigsaw, I'm seeing a strange behavior with emails on this list and I'm wondering if others are seeing it as well. Whenever Mark and other Oracle folks reply to an email, it seems to start a new thread on my side. I looked at the headers and their messages appear to be fine, but I'm not an email expert so I'm not sure why my client starts a new thread for their emails. Anyone else seeing that behavior? -bp From sean.mullan at oracle.com Wed Nov 9 14:41:30 2011 From: sean.mullan at oracle.com (Sean Mullan) Date: Wed, 09 Nov 2011 17:41:30 -0500 Subject: Email threading issues In-Reply-To: <494CCAC3-A214-43A7-A178-DC950569956D@pontarelli.com> References: <494CCAC3-A214-43A7-A178-DC950569956D@pontarelli.com> Message-ID: <4EBB019A.4020903@oracle.com> On 11/9/11 5:26 PM, Brian Pontarelli wrote: > Completely unrelated to Jigsaw, I'm seeing a strange behavior with emails on > this list and I'm wondering if others are seeing it as well. Whenever Mark > and other Oracle folks reply to an email, it seems to start a new thread on > my side. I looked at the headers and their messages appear to be fine, but > I'm not an email expert so I'm not sure why my client starts a new thread for > their emails. > > Anyone else seeing that behavior? I have noticed that when using Thunderbird. If you select "reply list" instead of "reply all" it creates a new thread. I don't know exactly why. --Sean From brian at pontarelli.com Wed Nov 9 16:09:28 2011 From: brian at pontarelli.com (Brian Pontarelli) Date: Wed, 9 Nov 2011 17:09:28 -0700 Subject: Email threading issues In-Reply-To: <4EBB019A.4020903@oracle.com> References: <494CCAC3-A214-43A7-A178-DC950569956D@pontarelli.com> <4EBB019A.4020903@oracle.com> Message-ID: <7CC2F15F-3B55-499A-89E6-791B112B608E@pontarelli.com> On Nov 9, 2011, at 3:41 PM, Sean Mullan wrote: > On 11/9/11 5:26 PM, Brian Pontarelli wrote: >> Completely unrelated to Jigsaw, I'm seeing a strange behavior with emails on >> this list and I'm wondering if others are seeing it as well. Whenever Mark >> and other Oracle folks reply to an email, it seems to start a new thread on >> my side. I looked at the headers and their messages appear to be fine, but >> I'm not an email expert so I'm not sure why my client starts a new thread for >> their emails. >> >> Anyone else seeing that behavior? > > I have noticed that when using Thunderbird. If you select "reply list" instead > of "reply all" it creates a new thread. I don't know exactly why. > > --Sean One thing that would be _really nice_ would be to change the listserv settings so that the reply-to is the list rather than the sender. It might fix some of these types of issues with threading. -bp From mandy.chung at oracle.com Wed Nov 9 20:48:02 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 09 Nov 2011 20:48:02 -0800 Subject: Review request: jpkg -m option takes both classes and resources Message-ID: <4EBB5782.2050904@oracle.com> The jpkg command currently takes the -m option to specify the directory containing the classes for a module and takes the -r option to specify the directory containing the resources. Classes and resources must be in separate directory which is different than what we (Java developers) generally do (compiled classes and resources are placed in the same build directory). To create a jmod file from an existing Java project, it has to separate the classes and resources from a build directory to run the jpkg command which is rather annoying. This change removes the -r option and have the -m option to take both classes and resources. Webrev: http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/jpkg-dash-r-option.00/ Many files and tests are touched due to the rename of ModuleFileFormat to ModuleFile and ModuleFileFormat.Writer to ModuleFileWriter. Since several tests are updated for this change, I took the chance to change -source 7 to -source 8 and update other tests as well. The main change is: 1. ModuleFileWriter (was ModuleFileFormat.Writer) - I refactored the existing implementation so that it can process a given directory of both classes and resources in a clean way. 2. Rename ModuleFileFormat to ModuleFile and I leave the Reader implementation as it is. Sean mentioned in the past that the ModuleFile class should be made immutable and need some clean up. Some work to be done in the future. 3. Rename the ModuleFormatTest01.java to ModuleFileTest.java. I rewrote half of the code and keep some existing utility functions. The new ModuleFileWriter implementation only writes a CLASSES section if there is at least one class file other than module-info.class. I had to fix this test as the existing implementation always creates a CLASSES section which is a bug. But there is no clean solution. Also initially the test tests both regular jmod files and signed jmod files which duplicates the code. I changed ModuleFileTest.java to take an input argument and if set, it will run the same set of test cases but sign the jmod file. So ModuleFileTest.java is now a jtreg test to test regular jmod file and SignedModuleFileTest.sh will run ModuleFileTest with an input argument to test the sign jmod file case. Thanks Mandy From gnormington at vmware.com Wed Nov 9 23:42:39 2011 From: gnormington at vmware.com (Glyn Normington) Date: Thu, 10 Nov 2011 07:42:39 +0000 Subject: Fwd: Email threading issues References: Message-ID: I didn't do that on purpose, which is why I voted for this suggestion! Regards, Glyn Begin forwarded message: > From: Glyn Normington > Subject: Re: Email threading issues > Date: 10 November 2011 07:41:39 GMT > To: Brian Pontarelli > > On 10 Nov 2011, at 00:09, Brian Pontarelli wrote: > >> One thing that would be _really nice_ would be to change the listserv settings so that the reply-to is the list rather than the sender. > > +1 From chris.hegarty at oracle.com Thu Nov 10 03:49:08 2011 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 10 Nov 2011 11:49:08 +0000 Subject: Compressing class files in the module library In-Reply-To: <4EBAC493.8080307@oracle.com> References: <4EBAC493.8080307@oracle.com> Message-ID: <4EBBBA34.4060005@oracle.com> On 09/11/2011 18:21, Alan Bateman wrote: > > I have an initial patch [1] to compress the classes of modules in the > module library. It essentially adds an option to "jmod create" so that > the classes of any modules installed into the library get compressed. > The patch also adds a build option to enable compression in the system > libraries of the images that the build generates. Wow, this look good. So we will always be creating module library images that use compression when we build. Do we need an option or environment variable to override this? I'm just wondering if you could see a scenario where compression wouldn't be desirable by default. > Before finishing this I'd like to get some input on a few points. The > first is whether it's reasonable to enable compression when creating the > module library? The alternative is install time so that it's on a per > module basis. The former is less flexible but as the module library > format evolves it may not be feasible to have a mix of compressed and > uncompressed modules. I do like the idea of being able to select this type of compression on a per module basis at install time ( lesser used modules could be compressed, while more startup performance sensitive ones may not ). But I do understand your reasoning about the evolution of the module library format. And this approach does not preclude the possibility of adding per module compression to an uncompressed library in the future. > The other question is suggestions for the option name. For now it's -9 > or --enable-compression (-9 influenced by the zip -9 option for compress > better). I'm looking for better suggestions. I understand the reasoning behind the option name -9 'compress better', from the module library point of view, but this gives the impression that the level of compression can be selected -0 -> -9 ( or that it is using the best compression available) when really I don't think it can be, right? The pack200 API doesn't seem to give this fine grain level of control over the compression level (or maybe I just missed it), which would be nice. -Chris. > > Thanks, > > -Alan. > > [1] http://cr.openjdk.java.net/~alanb/ModuleLibraryCompression/webrev/ From Alan.Bateman at oracle.com Thu Nov 10 04:25:15 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 10 Nov 2011 12:25:15 +0000 Subject: Compressing class files in the module library In-Reply-To: <4EBBBA34.4060005@oracle.com> References: <4EBAC493.8080307@oracle.com> <4EBBBA34.4060005@oracle.com> Message-ID: <4EBBC2AB.5050807@oracle.com> On 10/11/2011 11:49, Chris Hegarty wrote: > > So we will always be creating module library images that use > compression when we build. Do we need an option or environment > variable to override this? I'm just wondering if you could see a > scenario where compression wouldn't be desirable by default. The default build isn't changed, it requires building with COMPRESS_MODULE_LIBRARY=true to enable. In time I would expect options like this to be subsumed or implied by other build options. > : > > I understand the reasoning behind the option name -9 'compress > better', from the module library point of view, but this gives the > impression that the level of compression can be selected -0 -> -9 ( or > that it is using the best compression available) when really I don't > think it can be, right? The pack200 API doesn't seem to give this fine > grain level of control over the compression level (or maybe I just > missed it), which would be nice. -9 was an opening bid until we find something better (jmod create -C -L mlib isn't too bad). Suggestions welcome. -Alan From brian at pontarelli.com Thu Nov 10 09:33:50 2011 From: brian at pontarelli.com (Brian Pontarelli) Date: Thu, 10 Nov 2011 10:33:50 -0700 Subject: Email threading issues In-Reply-To: References: Message-ID: Who is the list owner and can they configure it to use the list for the reply-to? On Nov 10, 2011, at 12:42 AM, Glyn Normington wrote: > I didn't do that on purpose, which is why I voted for this suggestion! > > Regards, > Glyn > > Begin forwarded message: > >> From: Glyn Normington >> Subject: Re: Email threading issues >> Date: 10 November 2011 07:41:39 GMT >> To: Brian Pontarelli >> >> On 10 Nov 2011, at 00:09, Brian Pontarelli wrote: >> >>> One thing that would be _really nice_ would be to change the listserv settings so that the reply-to is the list rather than the sender. >> >> +1 > From dalibor.topic at oracle.com Thu Nov 10 09:47:47 2011 From: dalibor.topic at oracle.com (Dalibor Topic) Date: Thu, 10 Nov 2011 18:47:47 +0100 Subject: Email threading issues In-Reply-To: References: Message-ID: <4EBC0E43.1010908@oracle.com> On 11/10/11 6:33 PM, Brian Pontarelli wrote: > Who is the list owner http://mail.openjdk.java.net/mailman/listinfo/jigsaw-dev -> bottom of page. Same as for any other OpenJDK mailing list, fwiw. cheers, dalibor topic -- Oracle Dalibor Topic | Java F/OSS Ambassador Phone: +494023646738 | Mobile: +491772664192 Oracle Java Platform Group ORACLE Deutschland B.V. & Co. KG | Nagelsweg 55 | 20097 Hamburg ORACLE Deutschland B.V. & Co. KG Hauptverwaltung: Riesstr. 25, D-80992 M?nchen Registergericht: Amtsgericht M?nchen, HRA 95603 Komplement?rin: ORACLE Deutschland Verwaltung B.V. Hertogswetering 163/167, 3543 AS Utrecht, Niederlande Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697 Gesch?ftsf?hrer: J?rgen Kunz, Marcel van de Molen, Alexander van der Ven Green Oracle Oracle is committed to developing practices and products that help protect the environment From dalibor.topic at oracle.com Thu Nov 10 10:05:57 2011 From: dalibor.topic at oracle.com (Dalibor Topic) Date: Thu, 10 Nov 2011 19:05:57 +0100 Subject: Email threading issues In-Reply-To: <7CC2F15F-3B55-499A-89E6-791B112B608E@pontarelli.com> References: <494CCAC3-A214-43A7-A178-DC950569956D@pontarelli.com> <4EBB019A.4020903@oracle.com> <7CC2F15F-3B55-499A-89E6-791B112B608E@pontarelli.com> Message-ID: <4EBC1285.4090100@oracle.com> On 11/10/11 1:09 AM, Brian Pontarelli wrote: > One thing that would be _really nice_ would be to change the listserv settings so that the reply-to is the list rather than the sender. A classic on that subject is http://www.unicom.com/pw/reply-to-harmful.html . I prefer no reply-to munging myself, fwiw - it's much easier to forward a reply that was sent privately to the list, then to unsend a reply that was meant to be private but got reply-to vacuumed onto a list. cheers, dalibor topic -- Oracle Dalibor Topic | Java F/OSS Ambassador Phone: +494023646738 | Mobile: +491772664192 Oracle Java Platform Group ORACLE Deutschland B.V. & Co. KG | Nagelsweg 55 | 20097 Hamburg ORACLE Deutschland B.V. & Co. KG Hauptverwaltung: Riesstr. 25, D-80992 M?nchen Registergericht: Amtsgericht M?nchen, HRA 95603 Komplement?rin: ORACLE Deutschland Verwaltung B.V. Hertogswetering 163/167, 3543 AS Utrecht, Niederlande Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697 Gesch?ftsf?hrer: J?rgen Kunz, Marcel van de Molen, Alexander van der Ven Green Oracle Oracle is committed to developing practices and products that help protect the environment From mark.reinhold at oracle.com Thu Nov 10 10:32:31 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 10 Nov 2011 10:32:31 -0800 Subject: Email threading issues In-Reply-To: dalibor.topic@oracle.com; Thu, 10 Nov 2011 19:05:57 +0100; <4EBC1285.4090100@oracle.com> Message-ID: <20111110183231.67D3B2DE4@eggemoggin.niobe.net> 2011/11/10 10:05 -0800, dalibor.topic at oracle.com: > On 11/10/11 1:09 AM, Brian Pontarelli wrote: >> One thing that would be _really nice_ would be to change the listserv >> settings so that the reply-to is the list rather than the sender. > > A classic on that subject is http://www.unicom.com/pw/reply-to-harmful.html . > > I prefer no reply-to munging myself, fwiw - it's much easier to forward a > reply that was sent privately to the list, then to unsend a reply that was > meant to be private but got reply-to vacuumed onto a list. Yep. When we set up the OpenJDK Mailman back in 2007 we made the decision not to munge reply-to headers. It's the safest approach for those of us who work on both the open and closed sides of the fence. - Mark From hinkmond.wong at oracle.com Thu Nov 10 11:25:37 2011 From: hinkmond.wong at oracle.com (Hinkmond Wong) Date: Thu, 10 Nov 2011 11:25:37 -0800 Subject: Compressing class files in the module library In-Reply-To: <4EBBC2AB.5050807@oracle.com> References: <4EBAC493.8080307@oracle.com> <4EBBBA34.4060005@oracle.com> <4EBBC2AB.5050807@oracle.com> Message-ID: <4EBC2531.2040009@oracle.com> On 11/10/2011 4:25 AM, Alan Bateman wrote: > On 10/11/2011 11:49, Chris Hegarty wrote: >> >> So we will always be creating module library images that use >> compression when we build. Do we need an option or environment >> variable to override this? I'm just wondering if you could see a >> scenario where compression wouldn't be desirable by default. > The default build isn't changed, it requires building with > COMPRESS_MODULE_LIBRARY=true to enable. In time I would expect options > like this to be subsumed or implied by other build options. > >> : >> >> I understand the reasoning behind the option name -9 'compress >> better', from the module library point of view, but this gives the >> impression that the level of compression can be selected -0 -> -9 ( >> or that it is using the best compression available) when really I >> don't think it can be, right? The pack200 API doesn't seem to give >> this fine grain level of control over the compression level (or maybe >> I just missed it), which would be nice. > -9 was an opening bid until we find something better (jmod create -C > -L mlib isn't too bad). Suggestions welcome. In order to keep monitoring that the Base Module is meeting the <10MB max static footprint size, we probably should keep at least one of the default builds at -9 to have an automated post-build action warn when the size of the Base Module has exceeded the 10MB max threshold. Thanks, Hinkmond From mandy.chung at oracle.com Thu Nov 10 13:07:38 2011 From: mandy.chung at oracle.com (mandy.chung at oracle.com) Date: Thu, 10 Nov 2011 21:07:38 +0000 Subject: hg: jigsaw/jigsaw/jdk: /export/mchung/jigsaw/ws/commit.msg Message-ID: <20111110210748.44B5947308@hg.openjdk.java.net> Changeset: fd61ac05b32f Author: mchung Date: 2011-11-10 12:32 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/fd61ac05b32f /export/mchung/jigsaw/ws/commit.msg ! make/modules/Makefile + make/modules/jdk.depconfig - make/modules/jdk7.depconfig ! make/modules/modules.config ! make/modules/modules.group ! make/modules/optional.depconfig From mandy.chung at oracle.com Thu Nov 10 13:11:36 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Thu, 10 Nov 2011 13:11:36 -0800 Subject: hg: jigsaw/jigsaw/jdk: /export/mchung/jigsaw/ws/commit.msg In-Reply-To: <20111110210748.44B5947308@hg.openjdk.java.net> References: <20111110210748.44B5947308@hg.openjdk.java.net> Message-ID: <4EBC3E08.5090909@oracle.com> On 11/10/11 13:07, mandy.chung at oracle.com wrote: > Changeset: fd61ac05b32f > Author: mchung > Date: 2011-11-10 12:32 -0800 > URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/fd61ac05b32f > > /export/mchung/jigsaw/ws/commit.msg > Oops... my mistake using the wrong option. The intended changeset comment is: Add zipfs, httpserver, sctp and a few other modules and also rename a few names Reviewed-by: alanb, mr Mandy From mandy.chung at oracle.com Thu Nov 10 14:21:05 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Thu, 10 Nov 2011 14:21:05 -0800 Subject: Compressing class files in the module library In-Reply-To: <4EBAC493.8080307@oracle.com> References: <4EBAC493.8080307@oracle.com> Message-ID: <4EBC4E51.3020202@oracle.com> On 11/09/11 10:21, Alan Bateman wrote: > > I have an initial patch [1] to compress the classes of modules in the > module library. It essentially adds an option to "jmod create" so that > the classes of any modules installed into the library get compressed. > The patch also adds a build option to enable compression in the system > libraries of the images that the build generates. > > Before finishing this I'd like to get some input on a few points. The > first is whether it's reasonable to enable compression when creating > the module library? The alternative is install time so that it's on a > per module basis. The former is less flexible but as the module > library format evolves it may not be feasible to have a mix of > compressed and uncompressed modules. > It sounds reasonable to enable compression at the module library creation time. This is a footprint / startup time tradeoff that would be applicable for most, if not all, applications to run on a type of platform. I do like the per-module basis alternative that gives the flexibility that perhaps could be considered as the module library implementation-specific support. Anyway, what you have seems reasonable for now. > The other question is suggestions for the option name. For now it's -9 > or --enable-compression (-9 influenced by the zip -9 option for > compress better). I'm looking for better suggestions. I agree with Chris that -9 gives the impression that it supports a range of compression level (0-9). If that's what we support in the future, -9 is a good option. Regarding Chris's question on the Pack200 compression level, the spec for Pack200.Packer.EFFORT says: Level 1 may produce somewhat larger size and faster compression speed, while level 9 will take much longer but may produce better compression. The use of "may" which means it's not a guarantee. It does accept to specify a compression level from 0-9. Other than -0...-9, would "-z" be a good alternative? i.e. jmod create -z -L mlib Mandy > > Thanks, > > -Alan. > > [1] http://cr.openjdk.java.net/~alanb/ModuleLibraryCompression/webrev/ From mark.reinhold at oracle.com Thu Nov 10 21:59:53 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 10 Nov 2011 21:59:53 -0800 Subject: Compressing class files in the module library In-Reply-To: alan.bateman@oracle.com; Wed, 09 Nov 2011 18:21:07 GMT; <4EBAC493.8080307@oracle.com> Message-ID: <20111111055953.1438E7E1@eggemoggin.niobe.net> 2011/11/9 10:21 -0800, alan.bateman at oracle.com: > ... > > Before finishing this I'd like to get some input on a few points. The first is > whether it's reasonable to enable compression when creating the module library? That makes sense to me. > The alternative is install time so that it's on a per module basis. The former > is less flexible but as the module library format evolves it may not be > feasible to have a mix of compressed and uncompressed modules. Being able to select per-module compression at install time seems like needless flexibility at this point. We can always add it later if required. > The other question is suggestions for the option name. For now it's -9 > or --enable-compression (-9 influenced by the zip -9 option for compress > better). I'm looking for better suggestions. I like Mandy's "-z" suggestion. Reserve digits for actual compression levels, as with most other tools. - Mark From peter.kriens at aqute.biz Sat Nov 12 03:59:32 2011 From: peter.kriens at aqute.biz (Peter Kriens) Date: Sat, 12 Nov 2011 12:59:32 +0100 Subject: Converting plain JARs to Java modules In-Reply-To: <4EBA8885.6020808@redhat.com> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> Message-ID: <67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz> Neither my wrath, nor the fact that I rarely if ever get angry is relevant in this discussion ... This is a technical argument that are solvable by technical people that share the same goals. I prefer package dependencies because they address the excessive type coupling problem in object oriented systems, not because they're part of OSGi. Let me argue my case. Module-to-package dependencies (m2p) are preferable over module-to-module dependencies (m2m) for many reasons but these are the most important reasons: M2P is leverages the Java type system unlike m2m that must introduce new namespaces outside the Java type system. M2P can be used to break the transitive dependency chain, m2m suffers of excessive coupling Since the first bullet's benefit should be clear I only argue the more complex second bullet. A module is in many regards like a class. A class encapsulates members, depends on other members/classes, and makes a few members accessible outside the class. A module has a similar structure but then with types/packages as members. After the initial success of Object Oriented Programming (OO) it was quickly learned that reuse did not take place at the expected scale due to excessive type coupling. The problem was that a class aggregated many dependencies to simplify its implementation but these dependencies were unrelated to the contract it implemented. Since class dependencies are transitive most applications disappointingly became an almost fully connected graph. Java's great innovation was the interface because it broke both the transitivity and aggregation of dependencies. A class could now express its dependency (use or implement) on a contract (the interface) and was therefore fully type decoupled from the opposite site. An interface can act as a contract because it names the signature of a set of methods so that the compiler can verify the client and the implementer. Since a module has a very similar structure to a class it suffers from exactly the same transitive aggregation of dependencies. This is not a theory, look at the experiences with Maven (http://www.sonatype.com/people/2011/04/how-not-to-download-the-internet/) Again, this is not that maven is bad or developers are stupid, it is the same underlying force that finally resulted in the Java interface. The parallel for the class' interface for modules is a named set of interfaces. This concept already exists in Java: a package. Looking at almost all JSRs it is clear that our industry already uses packages as "interfaces" to provider implementations. Therefore, just like a class should not depend on other implementation types, a module should preferably not depend on other modules. A module should instead depend on contracts. Since modules will be used to provide components from different sources managed with different life cycles the excessive type coupling caused by m2m is even more damaging than in c2c. Proper use of m2p creates significantly less type coupled systems than m2m, the benefits should be obvious. Since there are use cases for m2m (non-type safe languages for example) I do believe that Jigsaw should still support m2m. However, it would be greatly beneficial to our industry if we could take advantage of the lessons learned with the Java interface and realize how surprisingly important the Java package actually is in our eco system. Kind regards, Peter Kriens On 9 nov. 2011, at 15:04, David M. Lloyd wrote: > I'll just state now that using packages as a dependency unit is a terrible idea, and not some architectural revelation. That way, Peter's wrath will be largely directed at me. :-) > > On 11/09/2011 08:02 AM, Peter Kriens wrote: >> I agree that tools are needed but we must be careful to not expect tools to stopgap an architectural issue. I think it is important to first do good architectural design leveraging existing tools (e.g. the Java type system) before you try to add new tools. It is such a pity (but all to common) that a design allows for classes of errors that would be impossible with a slightly different design. >> >> Kind regards, >> >> Peter Kriens >> >> >> >> >> >> >> >> On 9 nov. 2011, at 14:49, Alan Bateman wrote: >> >>> On 09/11/2011 13:04, Peter Kriens wrote: >>>> The issue is that maven problems are not caused because maven is bad or that pom authors are stupid. The reason is that the module-to-module dependency architecture in maven (and Jigsaw) is error prone ... >>> This thread started out with someone asking about adding module declarations to existing JAR files, and in that context, I agree it can be error prone without good tools. I think things should be a lot better when modules are compiled. >>> >>> -Alan. >>> >> > > > -- > - DML From cowwoc at bbs.darktech.org Sat Nov 12 19:59:33 2011 From: cowwoc at bbs.darktech.org (cowwoc) Date: Sat, 12 Nov 2011 19:59:33 -0800 (PST) Subject: Ordering of versions In-Reply-To: <20111109221947.E48FD111B@eggemoggin.niobe.net> References: <32BCD13E-CDF7-44BD-8F5A-2178C530B8BA@vmware.com> <20111109163704.44A48DD3@eggemoggin.niobe.net> <1535DFDE-DD79-4490-9C5C-14FBA276D12F@pontarelli.com> <4EBAD312.1000405@tibco.com> <20111109221947.E48FD111B@eggemoggin.niobe.net> Message-ID: <1321156773451-4988030.post@n5.nabble.com> mark.reinhold wrote: > > Support such semantics, sure, but I'm not sure it should prescribe them. > OSGi and Maven have established some useful and widely-used conventions > around version semantics but I've seen developers and customers use all > sorts of other wild and woolly version schemes over the years. I'm not > convinced that a standard Java module system should require people to > abandon such schemes, unless they're just really totally broken. > Mark, If at all possible, please consider allowing users to specify a Comparator (or Comparator depending on your taste) on a per-module basis. This would allow users to use any versioning scheme they see fit. Gili -- View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Ordering-of-versions-tp4970773p4988030.html Sent from the jigsaw-dev mailing list archive at Nabble.com. From gnormington at vmware.com Sun Nov 13 23:49:38 2011 From: gnormington at vmware.com (Glyn Normington) Date: Mon, 14 Nov 2011 07:49:38 +0000 Subject: Converting plain JARs to Java modules In-Reply-To: <67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> <67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz> Message-ID: I look forward to David's elaboration of why he thinks "using packages as a dependency unit is a terrible idea" to balance Peter's clear explanation of the benefits of m2p. Meanwhile, it's worth noting that, according to the requirements document, Jigsaw is aimed at platform modularisation and the platform being modularised has some non-optimal division of types across packages (see the package subsets requirement) which favour m2m dependencies. (Note that Apache Harmony was developed with modularity in mind and was able to exploit m2p, so platform modularisation per se needn't be limited to m2m.) So if Jigsaw excludes m2p, it will then be applicable to certain kinds of legacy code modularisation and less applicable to new module development and modularisation of existing code whose division into packages suits m2p. IIRC this was the original positioning of Jigsaw: for use primarily within the OpenJDK codebase and only exposed for application use because it was too inconvenient to hide it. Regards, Glyn On 12 Nov 2011, at 11:59, Peter Kriens wrote: > Neither my wrath, nor the fact that I rarely if ever get angry is relevant in this discussion ... This is a technical argument that are solvable by technical people that share the same goals. I prefer package dependencies because they address the excessive type coupling problem in object oriented systems, not because they're part of OSGi. Let me argue my case. > > Module-to-package dependencies (m2p) are preferable over module-to-module dependencies (m2m) for many reasons but these are the most important reasons: > > M2P is leverages the Java type system unlike m2m that must introduce new namespaces outside the Java type system. > M2P can be used to break the transitive dependency chain, m2m suffers of excessive coupling > > Since the first bullet's benefit should be clear I only argue the more complex second bullet. > > A module is in many regards like a class. A class encapsulates members, depends on other members/classes, and makes a few members accessible outside the class. A module has a similar structure but then with types/packages as members. > > After the initial success of Object Oriented Programming (OO) it was quickly learned that reuse did not take place at the expected scale due to excessive type coupling. The problem was that a class aggregated many dependencies to simplify its implementation but these dependencies were unrelated to the contract it implemented. Since class dependencies are transitive most applications disappointingly became an almost fully connected graph. > > Java's great innovation was the interface because it broke both the transitivity and aggregation of dependencies. A class could now express its dependency (use or implement) on a contract (the interface) and was therefore fully type decoupled from the opposite site. > > An interface can act as a contract because it names the signature of a set of methods so that the compiler can verify the client and the implementer. > > Since a module has a very similar structure to a class it suffers from exactly the same transitive aggregation of dependencies. This is not a theory, look at the experiences with Maven (http://www.sonatype.com/people/2011/04/how-not-to-download-the-internet/) Again, this is not that maven is bad or developers are stupid, it is the same underlying force that finally resulted in the Java interface. > > The parallel for the class' interface for modules is a named set of interfaces. This concept already exists in Java: a package. Looking at almost all JSRs it is clear that our industry already uses packages as "interfaces" to provider implementations. > > Therefore, just like a class should not depend on other implementation types, a module should preferably not depend on other modules. A module should instead depend on contracts. Since modules will be used to provide components from different sources managed with different life cycles the excessive type coupling caused by m2m is even more damaging than in c2c. Proper use of m2p creates significantly less type coupled systems than m2m, the benefits should be obvious. > > Since there are use cases for m2m (non-type safe languages for example) I do believe that Jigsaw should still support m2m. However, it would be greatly beneficial to our industry if we could take advantage of the lessons learned with the Java interface and realize how surprisingly important the Java package actually is in our eco system. > > Kind regards, > > Peter Kriens > > > > > > > > > On 9 nov. 2011, at 15:04, David M. Lloyd wrote: > >> I'll just state now that using packages as a dependency unit is a terrible idea, and not some architectural revelation. That way, Peter's wrath will be largely directed at me. :-) >> >> On 11/09/2011 08:02 AM, Peter Kriens wrote: >>> I agree that tools are needed but we must be careful to not expect tools to stopgap an architectural issue. I think it is important to first do good architectural design leveraging existing tools (e.g. the Java type system) before you try to add new tools. It is such a pity (but all to common) that a design allows for classes of errors that would be impossible with a slightly different design. >>> >>> Kind regards, >>> >>> Peter Kriens >>> >>> >>> >>> >>> >>> >>> >>> On 9 nov. 2011, at 14:49, Alan Bateman wrote: >>> >>>> On 09/11/2011 13:04, Peter Kriens wrote: >>>>> The issue is that maven problems are not caused because maven is bad or that pom authors are stupid. The reason is that the module-to-module dependency architecture in maven (and Jigsaw) is error prone ... >>>> This thread started out with someone asking about adding module declarations to existing JAR files, and in that context, I agree it can be error prone without good tools. I think things should be a lot better when modules are compiled. >>>> >>>> -Alan. >>>> >>> >> >> >> -- >> - DML > From Alan.Bateman at oracle.com Mon Nov 14 01:24:02 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 14 Nov 2011 09:24:02 +0000 Subject: Compressing class files in the module library In-Reply-To: <4EBC4E51.3020202@oracle.com> References: <4EBAC493.8080307@oracle.com> <4EBC4E51.3020202@oracle.com> Message-ID: <4EC0DE32.6060202@oracle.com> On 10/11/2011 22:21, Mandy Chung wrote: > > I agree with Chris that -9 gives the impression that it supports a > range of compression level (0-9). If that's what we support in the > future, -9 is a good option. Regarding Chris's question on the > Pack200 compression level, the spec for Pack200.Packer.EFFORT says: > Level 1 may produce somewhat larger size and faster compression > speed, while level 9 will take much longer but may produce better > compression. > > The use of "may" which means it's not a guarantee. It does accept to > specify a compression level from 0-9. > > Other than -0...-9, would "-z" be a good alternative? i.e. jmod > create -z -L mlib -z is a good idea, I see Mark likes it too. I've updated the webrev [1]. I've left it as a module create time option for now. If we really need it on a per-module basis then we can add it later. -Alan [1] http://cr.openjdk.java.net/~alanb/ModuleLibraryCompression/webrev/ From chris.hegarty at oracle.com Mon Nov 14 02:39:27 2011 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Mon, 14 Nov 2011 10:39:27 +0000 Subject: Compressing class files in the module library In-Reply-To: <4EC0DE32.6060202@oracle.com> References: <4EBAC493.8080307@oracle.com> <4EBC4E51.3020202@oracle.com> <4EC0DE32.6060202@oracle.com> Message-ID: <4EC0EFDF.2070003@oracle.com> Thanks Alan, I also like the -z option. One more minor comment on the code itself, otherwise looks fine. - with the changes in ClassInfo the given InputStream is no longer closed. Was this intentional? -Chris. On 14/11/2011 09:24, Alan Bateman wrote: > On 10/11/2011 22:21, Mandy Chung wrote: >> >> I agree with Chris that -9 gives the impression that it supports a >> range of compression level (0-9). If that's what we support in the >> future, -9 is a good option. Regarding Chris's question on the Pack200 >> compression level, the spec for Pack200.Packer.EFFORT says: >> Level 1 may produce somewhat larger size and faster compression speed, >> while level 9 will take much longer but may produce better compression. >> >> The use of "may" which means it's not a guarantee. It does accept to >> specify a compression level from 0-9. >> >> Other than -0...-9, would "-z" be a good alternative? i.e. jmod create >> -z -L mlib > -z is a good idea, I see Mark likes it too. I've updated the webrev [1]. > I've left it as a module create time option for now. If we really need > it on a per-module basis then we can add it later. > > -Alan > > [1] http://cr.openjdk.java.net/~alanb/ModuleLibraryCompression/webrev/ From Alan.Bateman at oracle.com Mon Nov 14 02:57:22 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 14 Nov 2011 10:57:22 +0000 Subject: Compressing class files in the module library In-Reply-To: <4EC0EFDF.2070003@oracle.com> References: <4EBAC493.8080307@oracle.com> <4EBC4E51.3020202@oracle.com> <4EC0DE32.6060202@oracle.com> <4EC0EFDF.2070003@oracle.com> Message-ID: <4EC0F412.6070500@oracle.com> On 14/11/2011 10:39, Chris Hegarty wrote: > Thanks Alan, I also like the -z option. > > One more minor comment on the code itself, otherwise looks fine. > > - with the changes in ClassInfo the given InputStream is no longer > closed. Was this intentional? It is closed, maybe just now obvious now because it is using try-with-resources. -Alan. From peter.kriens at aqute.biz Mon Nov 14 06:49:13 2011 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 14 Nov 2011 15:49:13 +0100 Subject: Ordering of versions In-Reply-To: <1321156773451-4988030.post@n5.nabble.com> References: <32BCD13E-CDF7-44BD-8F5A-2178C530B8BA@vmware.com> <20111109163704.44A48DD3@eggemoggin.niobe.net> <1535DFDE-DD79-4490-9C5C-14FBA276D12F@pontarelli.com> <4EBAD312.1000405@tibco.com> <20111109221947.E48FD111B@eggemoggin.niobe.net> <1321156773451-4988030.post@n5.nabble.com> Message-ID: <95C7B0D1-522B-478A-8C5C-971F2DC6792F@aQute.biz> Gili, Can you explain how this would work when my modules use a different scheme than yours? Kind regards, Peter Kriens On 13 nov. 2011, at 04:59, cowwoc wrote: > > mark.reinhold wrote: >> >> Support such semantics, sure, but I'm not sure it should prescribe them. >> OSGi and Maven have established some useful and widely-used conventions >> around version semantics but I've seen developers and customers use all >> sorts of other wild and woolly version schemes over the years. I'm not >> convinced that a standard Java module system should require people to >> abandon such schemes, unless they're just really totally broken. >> > > Mark, > > If at all possible, please consider allowing users to specify a > Comparator (or Comparator depending on your taste) on a > per-module basis. This would allow users to use any versioning scheme they > see fit. > > Gili > > -- > View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Ordering-of-versions-tp4970773p4988030.html > Sent from the jigsaw-dev mailing list archive at Nabble.com. From brian at pontarelli.com Mon Nov 14 08:04:47 2011 From: brian at pontarelli.com (Brian Pontarelli) Date: Mon, 14 Nov 2011 09:04:47 -0700 Subject: Ordering of versions In-Reply-To: <95C7B0D1-522B-478A-8C5C-971F2DC6792F@aQute.biz> References: <32BCD13E-CDF7-44BD-8F5A-2178C530B8BA@vmware.com> <20111109163704.44A48DD3@eggemoggin.niobe.net> <1535DFDE-DD79-4490-9C5C-14FBA276D12F@pontarelli.com> <4EBAD312.1000405@tibco.com> <20111109221947.E48FD111B@eggemoggin.niobe.net> <1321156773451-4988030.post@n5.nabble.com> <95C7B0D1-522B-478A-8C5C-971F2DC6792F@aQute.biz> Message-ID: <954F8B43-A6CE-4D9C-8C41-CBFBE4126CA8@pontarelli.com> It seems to me that the version scheme for a single module only impacts that module and not its dependencies. When the dependency graph is completely built and then traversed, modules are compared based on their scheme with all the other instances of themselves in the graph. Therefore, I don't think it impacts others at all. Thoughts? -bp On Nov 14, 2011, at 7:49 AM, Peter Kriens wrote: > Gili, > Can you explain how this would work when my modules use a different scheme than yours? > > Kind regards, > > Peter Kriens > > On 13 nov. 2011, at 04:59, cowwoc wrote: > >> >> mark.reinhold wrote: >>> >>> Support such semantics, sure, but I'm not sure it should prescribe them. >>> OSGi and Maven have established some useful and widely-used conventions >>> around version semantics but I've seen developers and customers use all >>> sorts of other wild and woolly version schemes over the years. I'm not >>> convinced that a standard Java module system should require people to >>> abandon such schemes, unless they're just really totally broken. >>> >> >> Mark, >> >> If at all possible, please consider allowing users to specify a >> Comparator (or Comparator depending on your taste) on a >> per-module basis. This would allow users to use any versioning scheme they >> see fit. >> >> Gili >> >> -- >> View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Ordering-of-versions-tp4970773p4988030.html >> Sent from the jigsaw-dev mailing list archive at Nabble.com. > From chris.hegarty at oracle.com Mon Nov 14 08:15:59 2011 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Mon, 14 Nov 2011 16:15:59 +0000 Subject: Review request: jpkg -m option takes both classes and resources In-Reply-To: <4EBB5782.2050904@oracle.com> References: <4EBB5782.2050904@oracle.com> Message-ID: <4EC13EBF.3030201@oracle.com> Mandy, Can the sun.java.launcher.module.boot property (in java.c) be cleaned up to remove the resources directory/jar? I don't think this has been used for a while, unless it is part of exploded modules? I scanned the webrev and it looks fine. I'll try to spend some more time reviewing it tomorrow. -Chris. On 10/11/2011 04:48, Mandy Chung wrote: > The jpkg command currently takes the -m option to specify the directory > containing the classes for a module and takes the -r option to specify > the directory containing the resources. Classes and resources must be in > separate directory which is different than what we (Java developers) > generally do (compiled classes and resources are placed in the same > build directory). To create a jmod file from an existing Java project, > it has to separate the classes and resources from a build directory to > run the jpkg command which is rather annoying. > > This change removes the -r option and have the -m option to take both > classes and resources. > > Webrev: > http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/jpkg-dash-r-option.00/ > > Many files and tests are touched due to the rename of ModuleFileFormat > to ModuleFile and ModuleFileFormat.Writer to ModuleFileWriter. Since > several tests are updated for this change, I took the chance to change > -source 7 to -source 8 and update other tests as well. > > The main change is: > 1. ModuleFileWriter (was ModuleFileFormat.Writer) - I refactored the > existing implementation so that it can process a given directory of both > classes and resources in a clean way. > 2. Rename ModuleFileFormat to ModuleFile and I leave the Reader > implementation as it is. Sean mentioned in the past that the ModuleFile > class should be made immutable and need some clean up. Some work to be > done in the future. > 3. Rename the ModuleFormatTest01.java to ModuleFileTest.java. I rewrote > half of the code and keep some existing utility functions. The new > ModuleFileWriter implementation only writes a CLASSES section if there > is at least one class file other than module-info.class. I had to fix > this test as the existing implementation always creates a CLASSES > section which is a bug. But there is no clean solution. Also initially > the test tests both regular jmod files and signed jmod files which > duplicates the code. I changed ModuleFileTest.java to take an input > argument and if set, it will run the same set of test cases but sign the > jmod file. So ModuleFileTest.java is now a jtreg test to test regular > jmod file and SignedModuleFileTest.sh will run ModuleFileTest with an > input argument to test the sign jmod file case. > > Thanks > Mandy > > From njbartlett at gmail.com Mon Nov 14 08:15:49 2011 From: njbartlett at gmail.com (Neil Bartlett) Date: Mon, 14 Nov 2011 16:15:49 +0000 Subject: Ordering of versions In-Reply-To: <954F8B43-A6CE-4D9C-8C41-CBFBE4126CA8@pontarelli.com> References: <32BCD13E-CDF7-44BD-8F5A-2178C530B8BA@vmware.com> <20111109163704.44A48DD3@eggemoggin.niobe.net> <1535DFDE-DD79-4490-9C5C-14FBA276D12F@pontarelli.com> <4EBAD312.1000405@tibco.com> <20111109221947.E48FD111B@eggemoggin.niobe.net> <1321156773451-4988030.post@n5.nabble.com> <95C7B0D1-522B-478A-8C5C-971F2DC6792F@aQute.biz> <954F8B43-A6CE-4D9C-8C41-CBFBE4126CA8@pontarelli.com> Message-ID: Brian, You're right that a module need not be compared against other modules for versioning purposes. But for this idea to work, each module would have to provide its own implementation of a Version class, which has a number of very challenging implications. First, where would that class be loaded from? It could not be loaded from the module itself because it is needed at build, install and resolve time, i.e. well before any class from the module could possibly be executed. So it would have to be loaded via some means external to or alongside the module system. Then how would the implementation of the Version itself be evolved? Suppose I found a bug in my compareTo() method, how would I provide a fix to the bug? Potentially each version of a module would end up with its own version *scheme*, which would be nonsense. The whole idea reminds me of the "module resolution scripts" that were suggested back in the days of JSR 277. They would have been a disaster as well, not least because they would have made it impossible to statically reason about the dependency graph. So, I don't think any reasonable module system can allow each module to decide version *ordering* arbitrarily. There has to be one ordering rule used by all modules. I would also like there to be one set of semantics for version bumps in each segment... but I can see that the chances of getting that in the Java 8 Module System are currently slim. Regards, Neil On Mon, Nov 14, 2011 at 4:04 PM, Brian Pontarelli wrote: > It seems to me that the version scheme for a single module only impacts that module and not its dependencies. When the dependency graph is completely built and then traversed, modules are compared based on their scheme with all the other instances of themselves in the graph. Therefore, I don't think it impacts others at all. > > Thoughts? > > -bp > > > On Nov 14, 2011, at 7:49 AM, Peter Kriens wrote: > >> Gili, >> Can you explain how this would work when my modules use a different scheme than yours? >> >> Kind regards, >> >> ? ? ? Peter Kriens >> >> On 13 nov. 2011, at 04:59, cowwoc wrote: >> >>> >>> mark.reinhold wrote: >>>> >>>> Support such semantics, sure, but I'm not sure it should prescribe them. >>>> OSGi and Maven have established some useful and widely-used conventions >>>> around version semantics but I've seen developers and customers use all >>>> sorts of other wild and woolly version schemes over the years. ?I'm not >>>> convinced that a standard Java module system should require people to >>>> abandon such schemes, unless they're just really totally broken. >>>> >>> >>> Mark, >>> >>> If at all possible, please consider allowing users to specify a >>> Comparator (or Comparator depending on your taste) on a >>> per-module basis. This would allow users to use any versioning scheme they >>> see fit. >>> >>> Gili >>> >>> -- >>> View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Ordering-of-versions-tp4970773p4988030.html >>> Sent from the jigsaw-dev mailing list archive at Nabble.com. >> > > From mcculls at gmail.com Mon Nov 14 08:20:20 2011 From: mcculls at gmail.com (Stuart McCulloch) Date: Mon, 14 Nov 2011 16:20:20 +0000 Subject: Ordering of versions In-Reply-To: <954F8B43-A6CE-4D9C-8C41-CBFBE4126CA8@pontarelli.com> References: <32BCD13E-CDF7-44BD-8F5A-2178C530B8BA@vmware.com> <20111109163704.44A48DD3@eggemoggin.niobe.net> <1535DFDE-DD79-4490-9C5C-14FBA276D12F@pontarelli.com> <4EBAD312.1000405@tibco.com> <20111109221947.E48FD111B@eggemoggin.niobe.net> <1321156773451-4988030.post@n5.nabble.com> <95C7B0D1-522B-478A-8C5C-971F2DC6792F@aQute.biz> <954F8B43-A6CE-4D9C-8C41-CBFBE4126CA8@pontarelli.com> Message-ID: <9A54FC40-DD93-42EF-9B8F-9A7E7773A7A8@gmail.com> On 14 Nov 2011, at 16:04, Brian Pontarelli wrote: > It seems to me that the version scheme for a single module only impacts that module and not its dependencies. When the dependency graph is completely built and then traversed, modules are compared based on their scheme with all the other instances of themselves in the graph. Therefore, I don't think it impacts others at all. > > Thoughts? What happens if someone changes the Comparator implementation used in their module over time? (either by design or mistake) You could end up with some situations where the ordering was not defined, or there were conflicting/ambiguous results. The other downside of an executable versioning scheme is that you probably would have to download a full bundle (maybe with all its dependencies) in order to run its comparison method, only to find out that you didn't actually need that version. Executable schemes also make it harder to do static analysis in other languages and could introduce security issues (since you're running unknown code even before installing the bundle). -- Cheers, Stuart > -bp > > > On Nov 14, 2011, at 7:49 AM, Peter Kriens wrote: > >> Gili, >> Can you explain how this would work when my modules use a different scheme than yours? >> >> Kind regards, >> >> Peter Kriens >> >> On 13 nov. 2011, at 04:59, cowwoc wrote: >> >>> >>> mark.reinhold wrote: >>>> >>>> Support such semantics, sure, but I'm not sure it should prescribe them. >>>> OSGi and Maven have established some useful and widely-used conventions >>>> around version semantics but I've seen developers and customers use all >>>> sorts of other wild and woolly version schemes over the years. I'm not >>>> convinced that a standard Java module system should require people to >>>> abandon such schemes, unless they're just really totally broken. >>>> >>> >>> Mark, >>> >>> If at all possible, please consider allowing users to specify a >>> Comparator (or Comparator depending on your taste) on a >>> per-module basis. This would allow users to use any versioning scheme they >>> see fit. >>> >>> Gili >>> >>> -- >>> View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Ordering-of-versions-tp4970773p4988030.html >>> Sent from the jigsaw-dev mailing list archive at Nabble.com. >> > From brian at pontarelli.com Mon Nov 14 08:29:40 2011 From: brian at pontarelli.com (Brian Pontarelli) Date: Mon, 14 Nov 2011 09:29:40 -0700 Subject: Ordering of versions In-Reply-To: References: <32BCD13E-CDF7-44BD-8F5A-2178C530B8BA@vmware.com> <20111109163704.44A48DD3@eggemoggin.niobe.net> <1535DFDE-DD79-4490-9C5C-14FBA276D12F@pontarelli.com> <4EBAD312.1000405@tibco.com> <20111109221947.E48FD111B@eggemoggin.niobe.net> <1321156773451-4988030.post@n5.nabble.com> <95C7B0D1-522B-478A-8C5C-971F2DC6792F@aQute.biz> <954F8B43-A6CE-4D9C-8C41-CBFBE4126CA8@pontarelli.com> Message-ID: <96BD4960-7EA1-46D7-920F-C56D8399A199@pontarelli.com> I'm actually not a huge advocate for multiple version schemes, but I think it is still possible to define well-known version schemes and version them via a module (or other external version concept). However, I still think that the EBNF I sent out (or something similar) is going to cover most cases. I also feel like those that have internal version schemes that don't fit can switch since they will have to change up a lot more than just the version scheme to start using modules. -bp On Nov 14, 2011, at 9:15 AM, Neil Bartlett wrote: > Brian, > > You're right that a module need not be compared against other modules > for versioning purposes. But for this idea to work, each module would > have to provide its own implementation of a Version class, which has a > number of very challenging implications. First, where would that class > be loaded from? > > It could not be loaded from the module itself because it is needed at > build, install and resolve time, i.e. well before any class from the > module could possibly be executed. So it would have to be loaded via > some means external to or alongside the module system. Then how would > the implementation of the Version itself be evolved? Suppose I found a > bug in my compareTo() method, how would I provide a fix to the bug? > Potentially each version of a module would end up with its own version > *scheme*, which would be nonsense. > > The whole idea reminds me of the "module resolution scripts" that were > suggested back in the days of JSR 277. They would have been a disaster > as well, not least because they would have made it impossible to > statically reason about the dependency graph. > > So, I don't think any reasonable module system can allow each module > to decide version *ordering* arbitrarily. There has to be one ordering > rule used by all modules. I would also like there to be one set of > semantics for version bumps in each segment... but I can see that the > chances of getting that in the Java 8 Module System are currently > slim. > > Regards, > Neil > > On Mon, Nov 14, 2011 at 4:04 PM, Brian Pontarelli wrote: >> It seems to me that the version scheme for a single module only impacts that module and not its dependencies. When the dependency graph is completely built and then traversed, modules are compared based on their scheme with all the other instances of themselves in the graph. Therefore, I don't think it impacts others at all. >> >> Thoughts? >> >> -bp >> >> >> On Nov 14, 2011, at 7:49 AM, Peter Kriens wrote: >> >>> Gili, >>> Can you explain how this would work when my modules use a different scheme than yours? >>> >>> Kind regards, >>> >>> Peter Kriens >>> >>> On 13 nov. 2011, at 04:59, cowwoc wrote: >>> >>>> >>>> mark.reinhold wrote: >>>>> >>>>> Support such semantics, sure, but I'm not sure it should prescribe them. >>>>> OSGi and Maven have established some useful and widely-used conventions >>>>> around version semantics but I've seen developers and customers use all >>>>> sorts of other wild and woolly version schemes over the years. I'm not >>>>> convinced that a standard Java module system should require people to >>>>> abandon such schemes, unless they're just really totally broken. >>>>> >>>> >>>> Mark, >>>> >>>> If at all possible, please consider allowing users to specify a >>>> Comparator (or Comparator depending on your taste) on a >>>> per-module basis. This would allow users to use any versioning scheme they >>>> see fit. >>>> >>>> Gili >>>> >>>> -- >>>> View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Ordering-of-versions-tp4970773p4988030.html >>>> Sent from the jigsaw-dev mailing list archive at Nabble.com. >>> >> >> From Alan.Bateman at oracle.com Mon Nov 14 08:33:58 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 14 Nov 2011 16:33:58 +0000 Subject: Review request: jpkg -m option takes both classes and resources In-Reply-To: <4EC13EBF.3030201@oracle.com> References: <4EBB5782.2050904@oracle.com> <4EC13EBF.3030201@oracle.com> Message-ID: <4EC142F6.5040902@oracle.com> On 14/11/2011 16:15, Chris Hegarty wrote: > Mandy, > > Can the sun.java.launcher.module.boot property (in java.c) be cleaned > up to remove the resources directory/jar? I don't think this has been > used for a while, unless it is part of exploded modules? I cleaned this up in my patch that adds the exploded module support (just need to get to re-based, cleaned it and pushed one of these days). -Alan. From david.lloyd at redhat.com Mon Nov 14 09:05:23 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 14 Nov 2011 11:05:23 -0600 Subject: Why package deps won't work (Was: Re: Converting plain JARs to Java modules) In-Reply-To: References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> <67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz> Message-ID: <4EC14A53.7070907@redhat.com> The explanation is quite simple, really - each point can be pretty much wiped out by a heavy dose of reality. 1. "M2P is leverages the Java type system unlike m2m that must introduce new namespaces outside the Java type system." - this is just fluffy buzzwordology. Considering packages part of the Java type system is a pretty liberal interpretation of the term "type system" as they're basically just arbitrary name spaces. That said, there's nothing particularly "new" about namespacing modules. JARs have names. Projects have names. Maven uses its own namespaces for artifacts. Even the primitive JDK extension mechanism uses symbolic names. Even a simple convention of the parent-most package name for the name of a module is very simple to grasp and is in fact the solution we've used quite effectively thus far. Thus implying that module names are some new alien concept is really not valid. 2. "M2P can be used to break the transitive dependency chain, m2m suffers of excessive coupling." - How about some facts to back this up? I've found "m2m" coupling to be just right. In JBoss Modules, we do not export transitive dependencies by default. This results in a very simple and clean dependency graph between modules. Using package dependencies results in a *far more complex* dependency graph, because you need edges for every package even though *most of the time you want the whole module anyway*. Again, real world here. If we're just going to throw dogma around, I'll put it the other way: m2p is a design error by the OSGi spec designers which has since been embraced as a religion. It offers no significant benefit, other than a couple of edge cases which are frankly just as well handled by m2m simply by adding package filters. Which, by the way, I haven't seen the need to do yet in our 200+ module environment, but which we do have the capability to do. M2P is a solution just itching for a problem. But you're going to have a tough time convincing me that users *want* to have to use special tooling because they want to depend on a module which has too many packages to list out by hand. And before you cry "wildcards", be sure to consider that some modules use package names which are subordinate to other modules' packages, which is a perfectly normal and allowable scenario. Using wildcards for package matching could cause amazing levels of havoc. Using package dependencies means you either must have a master package index for linking, or you need a resolver which has to have analyzed every module you ever plan to load. Otherwise, O(1) loading of modules is impossible, which is absolutely 100% a deal-breaker for JDK modules which must be incrementally installable. And it forbids having packages with the same name in more than one JAR without bringing run-time versioning into the fold, which is a terrible, terrible can of worms. Finally it should be perfectly clear to anyone who has read the original requirements document that nothing in this module system should prevent OSGi from functioning as it is, so there is absolutely no reason to assume that any OSGi implementation is so threatened - especially if m2p linking is as superior as has been expressed. Our module system (which is conceptually similar to Jigsaw in many regards) in fact does support our OSGi implementation quite effectively without itself implementing OSGi's package-to-package resolution (which like I said throws O(1) out the window). On 11/14/2011 01:49 AM, Glyn Normington wrote: > I look forward to David's elaboration of why he thinks "using packages as a dependency unit is a terrible idea" to balance Peter's clear explanation of the benefits of m2p. > > Meanwhile, it's worth noting that, according to the requirements document, Jigsaw is aimed at platform modularisation and the platform being modularised has some non-optimal division of types across packages (see the package subsets requirement) which favour m2m dependencies. (Note that Apache Harmony was developed with modularity in mind and was able to exploit m2p, so platform modularisation per se needn't be limited to m2m.) > > So if Jigsaw excludes m2p, it will then be applicable to certain kinds of legacy code modularisation and less applicable to new module development and modularisation of existing code whose division into packages suits m2p. IIRC this was the original positioning of Jigsaw: for use primarily within the OpenJDK codebase and only exposed for application use because it was too inconvenient to hide it. > > Regards, > Glyn > > On 12 Nov 2011, at 11:59, Peter Kriens wrote: > >> Neither my wrath, nor the fact that I rarely if ever get angry is relevant in this discussion ... This is a technical argument that are solvable by technical people that share the same goals. I prefer package dependencies because they address the excessive type coupling problem in object oriented systems, not because they're part of OSGi. Let me argue my case. >> >> Module-to-package dependencies (m2p) are preferable over module-to-module dependencies (m2m) for many reasons but these are the most important reasons: >> >> M2P is leverages the Java type system unlike m2m that must introduce new namespaces outside the Java type system. >> M2P can be used to break the transitive dependency chain, m2m suffers of excessive coupling >> >> Since the first bullet's benefit should be clear I only argue the more complex second bullet. >> >> A module is in many regards like a class. A class encapsulates members, depends on other members/classes, and makes a few members accessible outside the class. A module has a similar structure but then with types/packages as members. >> >> After the initial success of Object Oriented Programming (OO) it was quickly learned that reuse did not take place at the expected scale due to excessive type coupling. The problem was that a class aggregated many dependencies to simplify its implementation but these dependencies were unrelated to the contract it implemented. Since class dependencies are transitive most applications disappointingly became an almost fully connected graph. >> >> Java's great innovation was the interface because it broke both the transitivity and aggregation of dependencies. A class could now express its dependency (use or implement) on a contract (the interface) and was therefore fully type decoupled from the opposite site. >> >> An interface can act as a contract because it names the signature of a set of methods so that the compiler can verify the client and the implementer. >> >> Since a module has a very similar structure to a class it suffers from exactly the same transitive aggregation of dependencies. This is not a theory, look at the experiences with Maven (http://www.sonatype.com/people/2011/04/how-not-to-download-the-internet/) Again, this is not that maven is bad or developers are stupid, it is the same underlying force that finally resulted in the Java interface. >> >> The parallel for the class' interface for modules is a named set of interfaces. This concept already exists in Java: a package. Looking at almost all JSRs it is clear that our industry already uses packages as "interfaces" to provider implementations. >> >> Therefore, just like a class should not depend on other implementation types, a module should preferably not depend on other modules. A module should instead depend on contracts. Since modules will be used to provide components from different sources managed with different life cycles the excessive type coupling caused by m2m is even more damaging than in c2c. Proper use of m2p creates significantly less type coupled systems than m2m, the benefits should be obvious. >> >> Since there are use cases for m2m (non-type safe languages for example) I do believe that Jigsaw should still support m2m. However, it would be greatly beneficial to our industry if we could take advantage of the lessons learned with the Java interface and realize how surprisingly important the Java package actually is in our eco system. >> >> Kind regards, >> >> Peter Kriens >> >> >> >> >> >> >> >> >> On 9 nov. 2011, at 15:04, David M. Lloyd wrote: >> >>> I'll just state now that using packages as a dependency unit is a terrible idea, and not some architectural revelation. That way, Peter's wrath will be largely directed at me. :-) >>> >>> On 11/09/2011 08:02 AM, Peter Kriens wrote: >>>> I agree that tools are needed but we must be careful to not expect tools to stopgap an architectural issue. I think it is important to first do good architectural design leveraging existing tools (e.g. the Java type system) before you try to add new tools. It is such a pity (but all to common) that a design allows for classes of errors that would be impossible with a slightly different design. >>>> >>>> Kind regards, >>>> >>>> Peter Kriens >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On 9 nov. 2011, at 14:49, Alan Bateman wrote: >>>> >>>>> On 09/11/2011 13:04, Peter Kriens wrote: >>>>>> The issue is that maven problems are not caused because maven is bad or that pom authors are stupid. The reason is that the module-to-module dependency architecture in maven (and Jigsaw) is error prone ... >>>>> This thread started out with someone asking about adding module declarations to existing JAR files, and in that context, I agree it can be error prone without good tools. I think things should be a lot better when modules are compiled. >>>>> >>>>> -Alan. >>>>> >>>> >>> >>> >>> -- >>> - DML >> > -- - DML From david.lloyd at redhat.com Mon Nov 14 09:08:09 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 14 Nov 2011 11:08:09 -0600 Subject: Ordering of versions In-Reply-To: <96BD4960-7EA1-46D7-920F-C56D8399A199@pontarelli.com> References: <32BCD13E-CDF7-44BD-8F5A-2178C530B8BA@vmware.com> <20111109163704.44A48DD3@eggemoggin.niobe.net> <1535DFDE-DD79-4490-9C5C-14FBA276D12F@pontarelli.com> <4EBAD312.1000405@tibco.com> <20111109221947.E48FD111B@eggemoggin.niobe.net> <1321156773451-4988030.post@n5.nabble.com> <95C7B0D1-522B-478A-8C5C-971F2DC6792F@aQute.biz> <954F8B43-A6CE-4D9C-8C41-CBFBE4126CA8@pontarelli.com> <96BD4960-7EA1-46D7-920F-C56D8399A199@pontarelli.com> Message-ID: <4EC14AF9.7000207@redhat.com> Multiple version schemes should definitely be supported, but it should be done on a per-library basis, not a per-module basis (which is basically impossible anyway, at least if any standard of quality is applied). For example, Fedora will want to use Fedora's versioning scheme for their OS-managed module repository. Just as I'm sure Oracle will want to use a Solaris-friendly versioning scheme for theirs. On 11/14/2011 10:29 AM, Brian Pontarelli wrote: > I'm actually not a huge advocate for multiple version schemes, but I think it is still possible to define well-known version schemes and version them via a module (or other external version concept). > > However, I still think that the EBNF I sent out (or something similar) is going to cover most cases. I also feel like those that have internal version schemes that don't fit can switch since they will have to change up a lot more than just the version scheme to start using modules. > > -bp > > > On Nov 14, 2011, at 9:15 AM, Neil Bartlett wrote: > >> Brian, >> >> You're right that a module need not be compared against other modules >> for versioning purposes. But for this idea to work, each module would >> have to provide its own implementation of a Version class, which has a >> number of very challenging implications. First, where would that class >> be loaded from? >> >> It could not be loaded from the module itself because it is needed at >> build, install and resolve time, i.e. well before any class from the >> module could possibly be executed. So it would have to be loaded via >> some means external to or alongside the module system. Then how would >> the implementation of the Version itself be evolved? Suppose I found a >> bug in my compareTo() method, how would I provide a fix to the bug? >> Potentially each version of a module would end up with its own version >> *scheme*, which would be nonsense. >> >> The whole idea reminds me of the "module resolution scripts" that were >> suggested back in the days of JSR 277. They would have been a disaster >> as well, not least because they would have made it impossible to >> statically reason about the dependency graph. >> >> So, I don't think any reasonable module system can allow each module >> to decide version *ordering* arbitrarily. There has to be one ordering >> rule used by all modules. I would also like there to be one set of >> semantics for version bumps in each segment... but I can see that the >> chances of getting that in the Java 8 Module System are currently >> slim. >> >> Regards, >> Neil >> >> On Mon, Nov 14, 2011 at 4:04 PM, Brian Pontarelli wrote: >>> It seems to me that the version scheme for a single module only impacts that module and not its dependencies. When the dependency graph is completely built and then traversed, modules are compared based on their scheme with all the other instances of themselves in the graph. Therefore, I don't think it impacts others at all. >>> >>> Thoughts? >>> >>> -bp >>> >>> >>> On Nov 14, 2011, at 7:49 AM, Peter Kriens wrote: >>> >>>> Gili, >>>> Can you explain how this would work when my modules use a different scheme than yours? >>>> >>>> Kind regards, >>>> >>>> Peter Kriens >>>> >>>> On 13 nov. 2011, at 04:59, cowwoc wrote: >>>> >>>>> >>>>> mark.reinhold wrote: >>>>>> >>>>>> Support such semantics, sure, but I'm not sure it should prescribe them. >>>>>> OSGi and Maven have established some useful and widely-used conventions >>>>>> around version semantics but I've seen developers and customers use all >>>>>> sorts of other wild and woolly version schemes over the years. I'm not >>>>>> convinced that a standard Java module system should require people to >>>>>> abandon such schemes, unless they're just really totally broken. >>>>>> >>>>> >>>>> Mark, >>>>> >>>>> If at all possible, please consider allowing users to specify a >>>>> Comparator (or Comparator depending on your taste) on a >>>>> per-module basis. This would allow users to use any versioning scheme they >>>>> see fit. >>>>> >>>>> Gili >>>>> >>>>> -- >>>>> View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Ordering-of-versions-tp4970773p4988030.html >>>>> Sent from the jigsaw-dev mailing list archive at Nabble.com. >>>> >>> >>> > -- - DML From cowwoc at bbs.darktech.org Mon Nov 14 10:10:22 2011 From: cowwoc at bbs.darktech.org (cowwoc) Date: Mon, 14 Nov 2011 10:10:22 -0800 (PST) Subject: Ordering of versions In-Reply-To: <9A54FC40-DD93-42EF-9B8F-9A7E7773A7A8@gmail.com> References: <32BCD13E-CDF7-44BD-8F5A-2178C530B8BA@vmware.com> <20111109163704.44A48DD3@eggemoggin.niobe.net> <1535DFDE-DD79-4490-9C5C-14FBA276D12F@pontarelli.com> <4EBAD312.1000405@tibco.com> <20111109221947.E48FD111B@eggemoggin.niobe.net> <1321156773451-4988030.post@n5.nabble.com> <95C7B0D1-522B-478A-8C5C-971F2DC6792F@aQute.biz> <954F8B43-A6CE-4D9C-8C41-CBFBE4126CA8@pontarelli.com> <9A54FC40-DD93-42EF-9B8F-9A7E7773A7A8@gmail.com> Message-ID: <4EC15984.4010604@bbs.darktech.org> On 14/11/2011 11:21 AM, Stuart McCulloch [via jigsaw-dev] wrote: > On 14 Nov 2011, at 16:04, Brian Pontarelli wrote: > > > It seems to me that the version scheme for a single module only > impacts that module and not its dependencies. When the dependency > graph is completely built and then traversed, modules are compared > based on their scheme with all the other instances of themselves in > the graph. Therefore, I don't think it impacts others at all. > > > > Thoughts? > > What happens if someone changes the Comparator implementation > used in their module over time? (either by design or mistake) > > You could end up with some situations where the ordering was not > defined, or there were conflicting/ambiguous results. That's true for any implementation of Comparator that already exists today. Should we throw out the Comparator interface because people could implement it wrong? If a developer pushes a bad Comparator implementation, he will simply fix the bug and push the new revision. > The other downside of an executable versioning scheme is that you > probably would have to download a full bundle (maybe with all its > dependencies) in order to run its comparison method, only to find out > that you didn't actually need that version. I don't think any of us are advocating this. The idea would be for the version comparator to play a similar role to pom.xml in Maven. You'd download the Comparator independently of the module and only proceed if you desire that version. I don't see why a Comparator would need to depend on other classes in the Module anyway. We're talking about a simple String comparison here. > Executable schemes also make it harder to do static analysis in other > languages and could introduce security issues (since you're running > unknown code even before installing the bundle). Regarding static analysis, as I mentioned in other posts you could simply mention that results are cached so the Comparator must be deterministic. Regarding running untrusted code, the way I see it you're downloading a module with the intent of running it. Whether you run untrusted code at dependency-resolution time or at runtime makes little difference. Either way you are expected to verify that you only run code that you trust. When you install a native application on Windows you get a warning before the installer even launches (because you don't necessarily trust the installer). We'd do the same thing (for any untrusted dependency). Gili -- View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Ordering-of-versions-tp4970773p4991902.html Sent from the jigsaw-dev mailing list archive at Nabble.com. From cowwoc at bbs.darktech.org Mon Nov 14 10:20:37 2011 From: cowwoc at bbs.darktech.org (cowwoc) Date: Mon, 14 Nov 2011 10:20:37 -0800 (PST) Subject: Ordering of versions In-Reply-To: References: <32BCD13E-CDF7-44BD-8F5A-2178C530B8BA@vmware.com> <20111109163704.44A48DD3@eggemoggin.niobe.net> <1535DFDE-DD79-4490-9C5C-14FBA276D12F@pontarelli.com> <4EBAD312.1000405@tibco.com> <20111109221947.E48FD111B@eggemoggin.niobe.net> <1321156773451-4988030.post@n5.nabble.com> <95C7B0D1-522B-478A-8C5C-971F2DC6792F@aQute.biz> <954F8B43-A6CE-4D9C-8C41-CBFBE4126CA8@pontarelli.com> Message-ID: <4EC156B4.4010305@bbs.darktech.org> On 14/11/2011 11:17 AM, Neil Bartlett [via jigsaw-dev] wrote: > Brian, > > You're right that a module need not be compared against other modules > for versioning purposes. But for this idea to work, each module would > have to provide its own implementation of a Version class, which has a > number of very challenging implications. First, where would that class > be loaded from? > > It could not be loaded from the module itself because it is needed at > build, install and resolve time, i.e. well before any class from the > module could possibly be executed. So it would have to be loaded via > some means external to or alongside the module system. Borrowing an analogy from Maven, you'd load the pom.xml file separately from the main artifact. The pom.xml provides you with the module metadata (version comparison in this case) and if you decide to proceed to fetch the rest of the module. > Then how would > the implementation of the Version itself be evolved? Suppose I found a > bug in my compareTo() method, how would I provide a fix to the bug? > Potentially each version of a module would end up with its own version > *scheme*, which would be nonsense. Two solutions come to mind: 1) Treat the version comparator as a first-class citizen. There is one comparator across all versions of a module and developers may replace it as needed. 2) Each version has its own comparator. The first version that matches the desired version-range gets used. In my opinion, the ability to specify arbitrary version schemes is worth the risk of asymmetric results similar to Objects.equals(). > The whole idea reminds me of the "module resolution scripts" that were > suggested back in the days of JSR 277. They would have been a disaster > as well, not least because they would have made it impossible to > statically reason about the dependency graph. You could mandate that the Comparator operate deterministically, and then cache the results. Gili -- View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Ordering-of-versions-tp4970773p4991929.html Sent from the jigsaw-dev mailing list archive at Nabble.com. From Alan.Bateman at oracle.com Mon Nov 14 10:34:55 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 14 Nov 2011 18:34:55 +0000 Subject: Review request: jpkg -m option takes both classes and resources In-Reply-To: <4EBB5782.2050904@oracle.com> References: <4EBB5782.2050904@oracle.com> Message-ID: <4EC15F4F.8080802@oracle.com> On 10/11/2011 04:48, Mandy Chung wrote: > The jpkg command currently takes the -m option to specify the > directory containing the classes for a module and takes the -r option > to specify the directory containing the resources. Classes and > resources must be in separate directory which is different than what > we (Java developers) generally do (compiled classes and resources are > placed in the same build directory). To create a jmod file from an > existing Java project, it has to separate the classes and resources > from a build directory to run the jpkg command which is rather annoying. > > This change removes the -r option and have the -m option to take both > classes and resources. > > Webrev: > > http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/jpkg-dash-r-option.00/ > > Many files and tests are touched due to the rename of ModuleFileFormat > to ModuleFile and ModuleFileFormat.Writer to ModuleFileWriter. Since > several tests are updated for this change, I took the chance to change > -source 7 to -source 8 and update other tests as well. > > The main change is: > 1. ModuleFileWriter (was ModuleFileFormat.Writer) - I refactored the > existing implementation so that it can process a given directory of > both classes and resources in a clean way. > 2. Rename ModuleFileFormat to ModuleFile and I leave the Reader > implementation as it is. Sean mentioned in the past that the > ModuleFile class should be made immutable and need some clean up. > Some work to be done in the future. > 3. Rename the ModuleFormatTest01.java to ModuleFileTest.java. I > rewrote half of the code and keep some existing utility functions. > The new ModuleFileWriter implementation only writes a CLASSES section > if there is at least one class file other than module-info.class. I > had to fix this test as the existing implementation always creates a > CLASSES section which is a bug. But there is no clean solution. Also > initially the test tests both regular jmod files and signed jmod files > which duplicates the code. I changed ModuleFileTest.java to take an > input argument and if set, it will run the same set of test cases but > sign the jmod file. So ModuleFileTest.java is now a jtreg test to > test regular jmod file and SignedModuleFileTest.sh will run > ModuleFileTest with an input argument to test the sign jmod file case. I went through the webrev, except that I only skimmed the "new" ModuleFileWriter (as refactoring ModuleFileFormat.Writer and moving into its own source file at the same time makes this a bit awkward to review). I don't see any any obvious issue. FileCompressor and ClassFileCompressor extending ByteArrayOutputStream is a bit odd because they are output stream but don't have OutputStream in the name. Also getInstance suggests that they can return any instance but a new instance is required each time. One question, does classanalyzer's Modularizer need resDir now? -Alan. From mandy.chung at oracle.com Mon Nov 14 10:35:22 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Mon, 14 Nov 2011 10:35:22 -0800 Subject: Review request: jpkg -m option takes both classes and resources In-Reply-To: <4EC142F6.5040902@oracle.com> References: <4EBB5782.2050904@oracle.com> <4EC13EBF.3030201@oracle.com> <4EC142F6.5040902@oracle.com> Message-ID: <4EC15F6A.9060006@oracle.com> On 11/14/11 08:33, Alan Bateman wrote: >> Mandy, >> >> Can the sun.java.launcher.module.boot property (in java.c) be cleaned >> up to remove the resources directory/jar? I don't think this has been >> used for a while, unless it is part of exploded modules? > I cleaned this up in my patch that adds the exploded module support > (just need to get to re-based, cleaned it and pushed one of these days). The VM also has hardcoded the resources directory/jar as an interim solution that Alan probably also has cleaned up in his exploded module support patch. Mandy From mandy.chung at oracle.com Mon Nov 14 10:55:17 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Mon, 14 Nov 2011 10:55:17 -0800 Subject: Review request: jpkg -m option takes both classes and resources In-Reply-To: <4EC15F4F.8080802@oracle.com> References: <4EBB5782.2050904@oracle.com> <4EC15F4F.8080802@oracle.com> Message-ID: <4EC16415.3010905@oracle.com> On 11/14/11 10:34, Alan Bateman wrote: > I went through the webrev, except that I only skimmed the "new" > ModuleFileWriter (as refactoring ModuleFileFormat.Writer and moving > into its own source file at the same time makes this a bit awkward to > review). I don't see any any obvious issue. FileCompressor and > ClassFileCompressor extending ByteArrayOutputStream is a bit odd > because they are output stream but don't have OutputStream in the > name. Also getInstance suggests that they can return any instance but > a new instance is required each time. > I agree that the class names and the getInstance method should be renamed. How about renaming the class names to CompressorOutputStream and CompressedClassFileOutputStream and renaming getInstance to newInstance? > One question, does classanalyzer's Modularizer need resDir now? It doesn't need resDir now. I should clean that up. Thanks for the review Mandy From Alan.Bateman at oracle.com Mon Nov 14 11:14:11 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 14 Nov 2011 19:14:11 +0000 Subject: Review request: jpkg -m option takes both classes and resources In-Reply-To: <4EC16415.3010905@oracle.com> References: <4EBB5782.2050904@oracle.com> <4EC15F4F.8080802@oracle.com> <4EC16415.3010905@oracle.com> Message-ID: <4EC16883.3070604@oracle.com> On 14/11/2011 18:55, Mandy Chung wrote: > > I agree that the class names and the getInstance method should be > renamed. How about renaming the class names to CompressorOutputStream > and CompressedClassFileOutputStream and renaming getInstance to > newInstance? Yes, that would be better (alternatively CompressedClassOutputStream). -Alan From brian at pontarelli.com Mon Nov 14 15:13:19 2011 From: brian at pontarelli.com (Brian Pontarelli) Date: Mon, 14 Nov 2011 16:13:19 -0700 Subject: Ordering of versions In-Reply-To: <4EC14AF9.7000207@redhat.com> References: <32BCD13E-CDF7-44BD-8F5A-2178C530B8BA@vmware.com> <20111109163704.44A48DD3@eggemoggin.niobe.net> <1535DFDE-DD79-4490-9C5C-14FBA276D12F@pontarelli.com> <4EBAD312.1000405@tibco.com> <20111109221947.E48FD111B@eggemoggin.niobe.net> <1321156773451-4988030.post@n5.nabble.com> <95C7B0D1-522B-478A-8C5C-971F2DC6792F@aQute.biz> <954F8B43-A6CE-4D9C-8C41-CBFBE4126CA8@pontarelli.com> <96BD4960-7EA1-46D7-920F-C56D8399A199@pontarelli.com> <4EC14AF9.7000207@redhat.com> Message-ID: <12D1B294-F698-4707-A77C-432366E7A80F@pontarelli.com> Just to be clear, I wasn't stating that multiple version schemes should not be supported. I was stating that the default, which will cover 99% or more of cases, should be simple and well defined. However, I still don't think that a system that encourages multiple version schemes is a good idea. I think this will end up causing massive confusion and annoyance. I think the ability to have a custom version scheme should be there, but only used if absolutely necessary (read "rarely"). -bp On Nov 14, 2011, at 10:08 AM, David M. Lloyd wrote: > Multiple version schemes should definitely be supported, but it should be done on a per-library basis, not a per-module basis (which is basically impossible anyway, at least if any standard of quality is applied). > > For example, Fedora will want to use Fedora's versioning scheme for their OS-managed module repository. Just as I'm sure Oracle will want to use a Solaris-friendly versioning scheme for theirs. > > On 11/14/2011 10:29 AM, Brian Pontarelli wrote: >> I'm actually not a huge advocate for multiple version schemes, but I think it is still possible to define well-known version schemes and version them via a module (or other external version concept). >> >> However, I still think that the EBNF I sent out (or something similar) is going to cover most cases. I also feel like those that have internal version schemes that don't fit can switch since they will have to change up a lot more than just the version scheme to start using modules. >> >> -bp >> >> >> On Nov 14, 2011, at 9:15 AM, Neil Bartlett wrote: >> >>> Brian, >>> >>> You're right that a module need not be compared against other modules >>> for versioning purposes. But for this idea to work, each module would >>> have to provide its own implementation of a Version class, which has a >>> number of very challenging implications. First, where would that class >>> be loaded from? >>> >>> It could not be loaded from the module itself because it is needed at >>> build, install and resolve time, i.e. well before any class from the >>> module could possibly be executed. So it would have to be loaded via >>> some means external to or alongside the module system. Then how would >>> the implementation of the Version itself be evolved? Suppose I found a >>> bug in my compareTo() method, how would I provide a fix to the bug? >>> Potentially each version of a module would end up with its own version >>> *scheme*, which would be nonsense. >>> >>> The whole idea reminds me of the "module resolution scripts" that were >>> suggested back in the days of JSR 277. They would have been a disaster >>> as well, not least because they would have made it impossible to >>> statically reason about the dependency graph. >>> >>> So, I don't think any reasonable module system can allow each module >>> to decide version *ordering* arbitrarily. There has to be one ordering >>> rule used by all modules. I would also like there to be one set of >>> semantics for version bumps in each segment... but I can see that the >>> chances of getting that in the Java 8 Module System are currently >>> slim. >>> >>> Regards, >>> Neil >>> >>> On Mon, Nov 14, 2011 at 4:04 PM, Brian Pontarelli wrote: >>>> It seems to me that the version scheme for a single module only impacts that module and not its dependencies. When the dependency graph is completely built and then traversed, modules are compared based on their scheme with all the other instances of themselves in the graph. Therefore, I don't think it impacts others at all. >>>> >>>> Thoughts? >>>> >>>> -bp >>>> >>>> >>>> On Nov 14, 2011, at 7:49 AM, Peter Kriens wrote: >>>> >>>>> Gili, >>>>> Can you explain how this would work when my modules use a different scheme than yours? >>>>> >>>>> Kind regards, >>>>> >>>>> Peter Kriens >>>>> >>>>> On 13 nov. 2011, at 04:59, cowwoc wrote: >>>>> >>>>>> >>>>>> mark.reinhold wrote: >>>>>>> >>>>>>> Support such semantics, sure, but I'm not sure it should prescribe them. >>>>>>> OSGi and Maven have established some useful and widely-used conventions >>>>>>> around version semantics but I've seen developers and customers use all >>>>>>> sorts of other wild and woolly version schemes over the years. I'm not >>>>>>> convinced that a standard Java module system should require people to >>>>>>> abandon such schemes, unless they're just really totally broken. >>>>>>> >>>>>> >>>>>> Mark, >>>>>> >>>>>> If at all possible, please consider allowing users to specify a >>>>>> Comparator (or Comparator depending on your taste) on a >>>>>> per-module basis. This would allow users to use any versioning scheme they >>>>>> see fit. >>>>>> >>>>>> Gili >>>>>> >>>>>> -- >>>>>> View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Ordering-of-versions-tp4970773p4988030.html >>>>>> Sent from the jigsaw-dev mailing list archive at Nabble.com. >>>>> >>>> >>>> >> > > > -- > - DML From cowwoc at bbs.darktech.org Mon Nov 14 16:48:01 2011 From: cowwoc at bbs.darktech.org (cowwoc) Date: Mon, 14 Nov 2011 16:48:01 -0800 (PST) Subject: Ordering of versions In-Reply-To: <12D1B294-F698-4707-A77C-432366E7A80F@pontarelli.com> References: <4EBAD312.1000405@tibco.com> <20111109221947.E48FD111B@eggemoggin.niobe.net> <1321156773451-4988030.post@n5.nabble.com> <95C7B0D1-522B-478A-8C5C-971F2DC6792F@aQute.biz> <954F8B43-A6CE-4D9C-8C41-CBFBE4126CA8@pontarelli.com> <96BD4960-7EA1-46D7-920F-C56D8399A199@pontarelli.com> <4EC14AF9.7000207@redhat.com> <12D1B294-F698-4707-A77C-432366E7A80F@pontarelli.com> Message-ID: <4EC1B6B2.3020600@bbs.darktech.org> On 14/11/2011 6:14 PM, Brian Pontarelli [via jigsaw-dev] wrote: > Just to be clear, I wasn't stating that multiple version schemes > should not be supported. I was stating that the default, which will > cover 99% or more of cases, should be simple and well defined. Here is a default we can all identify with :) public final class NaturalOrdering> implements Comparator { @Override public int compare(T left, T right) { if (right == null) throw new NullPointerException("right may not be null"); if (left == right) return 0; return left.compareTo(right); } } > However, I still don't think that a system that encourages multiple > version schemes is a good idea. I think this will end up causing > massive confusion and annoyance. I think the ability to have a custom > version scheme should be there, but only used if absolutely necessary > (read "rarely"). People are already using multiple version schemes today and programming tools, by and large, allow it. Allowing developers to continue using custom schemes will not make the world any more chaotic than it already is. This is a subjective matter and as such there is no "right answer". That's one of the things that rubs me the wrong way about Maven. They assume that there is a "right way" of doing everything and if you fall into the "wrong way" category using their system is a virtual hell. Project Jigsaw should be inclusive on these kinds of subjective matters. PS: I don't see why arbitrary versioning schemes would complicate life for end-users. They specify a dependency on FooLib 1.0.5+ and it's up to the system to figure out what "greater than" means in this case. The whole thing is quite transparent to end-users. Gili -- View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Ordering-of-versions-tp4970773p4993029.html Sent from the jigsaw-dev mailing list archive at Nabble.com. From njbartlett at gmail.com Mon Nov 14 17:19:41 2011 From: njbartlett at gmail.com (Neil Bartlett) Date: Tue, 15 Nov 2011 01:19:41 +0000 Subject: Ordering of versions In-Reply-To: <4EC156B4.4010305@bbs.darktech.org> References: <32BCD13E-CDF7-44BD-8F5A-2178C530B8BA@vmware.com> <20111109163704.44A48DD3@eggemoggin.niobe.net> <1535DFDE-DD79-4490-9C5C-14FBA276D12F@pontarelli.com> <4EBAD312.1000405@tibco.com> <20111109221947.E48FD111B@eggemoggin.niobe.net> <1321156773451-4988030.post@n5.nabble.com> <95C7B0D1-522B-478A-8C5C-971F2DC6792F@aQute.biz> <954F8B43-A6CE-4D9C-8C41-CBFBE4126CA8@pontarelli.com> <4EC156B4.4010305@bbs.darktech.org> Message-ID: On Mon, Nov 14, 2011 at 6:20 PM, cowwoc wrote: > On 14/11/2011 11:17 AM, Neil Bartlett [via jigsaw-dev] wrote: > [snip] > ? ? Borrowing an analogy from Maven, you'd load the pom.xml file > separately from the main artifact. The pom.xml provides you with the > module metadata (version comparison in this case) and if you decide to > proceed to fetch the rest of the module. The analogy is bogus. Maven's POM -- like OSGi's manifest and Jigsaw's module-info.java -- is a declarative XML format, not an executable. We don't need to run any part of the module before we resolve the model, we only need to read the metadata and execute the comparison which exists inside the module system. > 1) Treat the version comparator as a first-class citizen. There is one > comparator across all versions of a module and developers may replace it > as needed. How do I replace the comparator when I find it has a bug? Presumably until my new module with its new comparator can be installed, it must be compared against the current installed module using the current installed comparator. If the bug is as serious as, say, compareTo() returning +1 irrespective of the arguments, then the current module will always be considered higher than any replacement. > 2) Each version has its own comparator. The first version that matches > the desired version-range gets used. In my opinion, the ability to > specify arbitrary version schemes is worth the risk of asymmetric > results similar to Objects.equals(). Well I'm glad you see the risk, but I don't yet see the benefit. Is it really useful for different modules to be able to redefine ordering of versions? This is almost like each module wanting its own set of natural numbers. > ? ? You could mandate that the Comparator operate deterministically, > and then cache the results. Please forgive the following diversion: a Russian friend once told me this joke that was popular during the old Soviet days. Stalin, Khrushchev and Brezhnev are riding a train together, and the train breaks down. Stalin goes and shoots the driver and engineer, but this doesn't solve the problem, the train doesn't move. Khrushchev offers the (new) driver and engineer a nice flat in the suburbs of Moscow, but this doesn't work either, and the train still doesn't move. Brezhnev stands up and declares, "I will fix this problem". He closes all the curtains in the train carriage, sits down and states "see, now we are moving". Mandating that a comparator must be deterministic is far less reliable than using pure declarations and controlling the comparison within the module system. Again, what's the benefit to balance this risk? > Gili > > > -- > View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Ordering-of-versions-tp4970773p4991929.html > Sent from the jigsaw-dev mailing list archive at Nabble.com. > From cowwoc at bbs.darktech.org Mon Nov 14 17:36:41 2011 From: cowwoc at bbs.darktech.org (cowwoc) Date: Mon, 14 Nov 2011 20:36:41 -0500 Subject: Ordering of versions In-Reply-To: References: <32BCD13E-CDF7-44BD-8F5A-2178C530B8BA@vmware.com> <20111109163704.44A48DD3@eggemoggin.niobe.net> <1535DFDE-DD79-4490-9C5C-14FBA276D12F@pontarelli.com> <4EBAD312.1000405@tibco.com> <20111109221947.E48FD111B@eggemoggin.niobe.net> <1321156773451-4988030.post@n5.nabble.com> <95C7B0D1-522B-478A-8C5C-971F2DC6792F@aQute.biz> <954F8B43-A6CE-4D9C-8C41-CBFBE4126CA8@pontarelli.com> <4EC156B4.4010305@bbs.darktech.org> Message-ID: <4EC1C229.8060503@bbs.darktech.org> Hey Neil, On 14/11/2011 8:19 PM, Neil Bartlett wrote: > On Mon, Nov 14, 2011 at 6:20 PM, cowwoc wrote: >> On 14/11/2011 11:17 AM, Neil Bartlett [via jigsaw-dev] wrote: >> [snip] >> Borrowing an analogy from Maven, you'd load the pom.xml file >> separately from the main artifact. The pom.xml provides you with the >> module metadata (version comparison in this case) and if you decide to >> proceed to fetch the rest of the module. > The analogy is bogus. Maven's POM -- like OSGi's manifest and Jigsaw's > module-info.java -- is a declarative XML format, not an executable. We > don't need to run any part of the module before we resolve the model, > we only need to read the metadata and execute the comparison which > exists inside the module system. It doesn't matter whether it's a declarative XML format or an executable. So long as the Comparator does not depend on any part of the Module, it can be downloaded and executed independently just like the pom.xml file. >> 1) Treat the version comparator as a first-class citizen. There is one >> comparator across all versions of a module and developers may replace it >> as needed. > How do I replace the comparator when I find it has a bug? Presumably > until my new module with its new comparator can be installed, it must > be compared against the current installed module using the current > installed comparator. If the bug is as serious as, say, compareTo() > returning +1 irrespective of the arguments, then the current module > will always be considered higher than any replacement. You're assuming that the Comparator is only replaced by the Comparator from a higher version. I'm saying that the Comparator exists outside the scope of any specific version. It applies module-wide. As such, you can replace the Comparator without needing to evaluate the old Comparator. It doesn't matter whether your version is "newer" or "older", you just replace whatever is there. >> 2) Each version has its own comparator. The first version that matches >> the desired version-range gets used. In my opinion, the ability to >> specify arbitrary version schemes is worth the risk of asymmetric >> results similar to Objects.equals(). > Well I'm glad you see the risk, but I don't yet see the benefit. Is it > really useful for different modules to be able to redefine ordering of > versions? This is almost like each module wanting its own set of > natural numbers. Take a look at software projects in the world. Almost no two use the same versioning scheme. The "benefit" is that we support the de-facto standard which is to say these people have a good reason for using different version schemes and there is no reason to believe we know better than them. I consider version schemes to be the same as code-formatting rules. They're highly subjective, there is no "right answer" and enforcing a rigid "standard" is bound to piss many people off. >> You could mandate that the Comparator operate deterministically, >> and then cache the results. > Please forgive the following diversion: a Russian friend once told me > this joke that was popular during the old Soviet days. Stalin, > Khrushchev and Brezhnev are riding a train together, and the train > breaks down. Stalin goes and shoots the driver and engineer, but this > doesn't solve the problem, the train doesn't move. Khrushchev offers > the (new) driver and engineer a nice flat in the suburbs of Moscow, > but this doesn't work either, and the train still doesn't move. > Brezhnev stands up and declares, "I will fix this problem". He closes > all the curtains in the train carriage, sits down and states "see, now > we are moving". > > Mandating that a comparator must be deterministic is far less reliable > than using pure declarations and controlling the comparison within the > module system. Again, what's the benefit to balance this risk? Heh. Good joke ;) Okay, so I guess we have to agree to disagree. I completely see your point about the safety of a declarative model, but I personally prefer the flexibility of an imperative model. I respect your views on the matter. Gili From njbartlett at gmail.com Mon Nov 14 18:33:45 2011 From: njbartlett at gmail.com (Neil Bartlett) Date: Tue, 15 Nov 2011 02:33:45 +0000 Subject: Why package deps won't work (Was: Re: Converting plain JARs to Java modules) In-Reply-To: <4EC14A53.7070907@redhat.com> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> <67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz> <4EC14A53.7070907@redhat.com> Message-ID: Hi David, More than happy to talk about reality, though Peter already was doing so. Nothing he said was theoretical, it came directly from his experiences and the experiences of developers who have been using OSGi for around 12 years. Before digging into the points in your message, I need to address the subject line "Why package deps won't work". Package dependencies will work and they do work, as is proven every day by the large number of applications running on OSGi. To claim otherwise would be wilfully ignorant, so I will assume you are being hyperbolic and really meant to assert that module deps are just better than package deps. Therefore I intend to argue on that basis and claim that package deps are better than module deps; note that I don't claim that module dependencies "won't work" because JBoss Modules is a proof point that they do. On Mon, Nov 14, 2011 at 5:05 PM, David M. Lloyd wrote: > The explanation is quite simple, really - each point can be pretty much > wiped out by a heavy dose of reality. > > 1. "M2P is leverages the Java type system unlike m2m that must introduce new > namespaces outside the Java type system." - this is just fluffy > buzzwordology. ?Considering packages part of the Java type system is a > pretty liberal interpretation of the term "type system" as they're basically > just arbitrary name spaces. ?That said, there's nothing particularly "new" > about namespacing modules. ?JARs have names. Projects have names. ?Maven > uses its own namespaces for artifacts. ?Even the primitive JDK extension > mechanism uses symbolic names. >From Wikipedia (http://en.wikipedia.org/wiki/Type_system): "A type system associates a type with each computed value ... the aim is to prevent operations expecting a certain kind of value being used with values for which that operation does not make sense." The following will not compile: import java.awt.List; // ... List list = new ArrayList(); list.iterator(); // etc whereas the following will: import java.util.List; // ... List list = new ArrayList(); list.iterator(); // etc Package names are part of the type system because using an incorrect package name in Java source can result in type errors during compilation, and because the type and available operations associated with each value varies with the package name used. In contrast, it is impossible to obtain a type error at compilation by incorrectly naming a JAR, project, Maven artefact etc, because none of those things are ever referenced in Java source code. > Even a simple convention of the parent-most package name for the name of a > module is very simple to grasp and is in fact the solution we've used quite > effectively thus far. ?Thus implying that module names are some new alien > concept is really not valid. The claim is not that they are a new alien concept. The claim is that they are not part of the Java language and type system, and as a result of this (along with other problems) are less useful for depending upon than packages. > 2. "M2P can be used to break the transitive dependency chain, m2m suffers of > excessive coupling." - How about some facts to back this up? ?I've found > "m2m" coupling to be just right. ?In JBoss Modules, we do not export > transitive dependencies by default. ?This results in a very simple and clean > dependency graph between modules. ?Using package dependencies results in a > *far more complex* dependency graph, because you need edges for every > package even though *most of the time you want the whole module anyway*. > ?Again, real world here. Peter is definitely also talking about the real world and so am I. In an m2m dependency model you cannot avoid transitive dependencies, whether you expose them to consumers or not. An importer of a module must be assumed to depend on the whole functionality of that module whether or not that is actually the case, and therefore all of the transitive dependencies must be present at runtime. In an m2p world we have the opportunity to split modules and break apart dependency graphs, because the unit of coupling is more granular. > If we're just going to throw dogma around, I'll put it the other way: m2p is > a design error by the OSGi spec designers which has since been embraced as a > religion. Pejorative accusations about "dogma" or "religion" have no place in a technical discussion. They are also untrue. All of the practising OSGi developers I know have arrived at their support for package dependencies as a result of real world experience, not because of some willingness to bow down to the Almighty CPEG. I don't know of any practising OSGi developer who has used both Require-Bundle (m2m) and Import-Package (m2p) and actually prefers the former. I do know several who started as strong supporters of Require-Bundle and switched to being supporters of Import-Package, not because of Peter's wrathful pontificating but because they encountered the specific problems that he described and found that they were fixed by using Import-Package, and indeed that everything worked so much more cleanly that way. I'm in this camp myself, and so was Jeff McAffer before he went over to the Dark Side (Microsoft). > It offers no significant benefit, other than a couple of edge > cases which are frankly just as well handled by m2m simply by adding package > filters. ?Which, by the way, I haven't seen the need to do yet in our 200+ > module environment, but which we do have the capability to do. Substitution, module refactoring and transitive decoupling are hardly edge cases. However I can see how these issues might not yet have come to the fore in a module system designed for a single product with a small number of modules, and where that product has not yet been through the mill of multiple version evolutions. > M2P is a solution just itching for a problem. This is also not a very useful statement, I assure you that the problem came before the solution. Better to show why you think the problem is invalid or should have been solved differently. > But you're going to have a > tough time convincing me that users *want* to have to use special tooling > because they want to depend on a module which has too many packages to list > out by hand. ?And before you cry "wildcards", be sure to consider that some > modules use package names which are subordinate to other modules' packages, > which is a perfectly normal and allowable scenario. ?Using wildcards for > package matching could cause amazing levels of havoc. OSGi never uses wildcards at runtime, and tools such as bnd do not need wildcards in order to express package-level dependencies. They extract the set of packages that were actually used by the code, all of which are available in the class files. This is possible because packages are part of the type system (see first point above). So it's not that I don't want to list dependencies by hand, rather I only want to do it once. I am already forced to do it in the import statements of my Java sources. If I had to repeat that dependency information -- whether in m2p or m2m form -- then I would run the risk of it getting out of step with the real dependencies. > Using package dependencies means you either must have a master package index > for linking, or you need a resolver which has to have analyzed every module > you ever plan to load. ?Otherwise, O(1) loading of modules is impossible, > which is absolutely 100% a deal-breaker for JDK modules which must be > incrementally installable. ?And it forbids having packages with the same > name in more than one JAR without bringing run-time versioning into the > fold, which is a terrible, terrible can of worms. Could you please explain why O(1) is the only acceptable complexity for installing modules. OSGi does indeed support incremental install and while I accept it is probably not O(1) for each module, it would likely be no more than O(N), though I haven't done the maths yet to prove this. Bear in mind that in practice, for small N, the constant factors can result in O(1) being more expensive than O(N). I have seen OSGi used with many thousands of modules, so unless you have some data and a use-case showings package-level resolution as unacceptably slow, your concern just sounds like premature optimisation. There are two reasons to have a packages with the same name in more than one JAR. The first is a situation called split packages, and it is highly undesirable because it causes the runtime model of the package to diverge from the compile-time model, and therefore things like package-private types and members stop working correctly. For this reason, OSGi's m2p imports support depending only upon a single exporter of a particular package, i.e. we do not aggregate all exports of that package. Unfortunately split packages are sometimes unavoidable in legacy code that cannot be refactored, e.g. the JDK. To support such scenarios OSGi has Require-Bundle, i.e. m2m. This does not negate the problems associated with m2m, it is simply a trade-off that we face with poorly factored legacy code. The second reason for multiple packages with the same name is when you explicitly want to install multiple versions of a library/API and have them all available within the same runtime. I wouldn't call this a "can of worms" exactly because it can be done without too much trouble, though for the sake of a simple life I personally avoid this situation unless it's necessary. > Finally it should be perfectly clear to anyone who has read the original > requirements document that nothing in this module system should prevent OSGi > from functioning as it is, so there is absolutely no reason to assume that > any OSGi implementation is so threatened - especially if m2p linking is as > superior as has been expressed. ?Our module system (which is conceptually > similar to Jigsaw in many regards) in fact does support our OSGi > implementation quite effectively without itself implementing OSGi's > package-to-package resolution (which like I said throws O(1) out the > window). I agree that Jigsaw's existence doesn't threaten OSGi's, so long as Java 8 doesn't actually break OSGi (and if it did so, it would likewise break many other applications and could not be considered backwards compatible with Java 7). The two can interoperate through m2m-type dependencies. Tim Ellison started Project Penrose for the purpose of investigating, testing and deepening this collaboration. Neverthless, the point that I believe Glyn was making is the following. We accept that m2m dependencies are probably required for the JDK, which implies a module system like Jigsaw or OSGi/Require-Bundle rather than OSGi/Import-Package. However is it intended to be used for application modularisation as well? This is of course a question for the Jigsaw team rather than you, David. As a result of experience in developing and evolving large real-world applications using a module system that supports BOTH m2m and m2p dependencies, I believe it would be very unfortunate if a module system that supports ONLY m2m were to become widely used in the application space... not because OSGi can't handle the competition, but because those applications will be fragile and hard to evolve. My question for you David is as follows. I understand that you prefer module dependencies, but do you believe that package dependencies have no value whatsoever and therefore should not be available to application developers in the Java 8 module system? If so, why did Red Hat create an OSGi implementation? Kind regards Neil > > On 11/14/2011 01:49 AM, Glyn Normington wrote: >> >> I look forward to David's elaboration of why he thinks "using packages as >> a dependency unit is a terrible idea" to balance Peter's clear explanation >> of the benefits of m2p. >> >> Meanwhile, it's worth noting that, according to the requirements document, >> Jigsaw is aimed at platform modularisation and the platform being >> modularised has some non-optimal division of types across packages (see the >> package subsets requirement) which favour m2m dependencies. (Note that >> Apache Harmony was developed with modularity in mind and was able to exploit >> m2p, so platform modularisation per se needn't be limited to m2m.) >> >> So if Jigsaw excludes m2p, it will then be applicable to certain kinds of >> legacy code modularisation and less applicable to new module development and >> modularisation of existing code whose division into packages suits m2p. IIRC >> this was the original positioning of Jigsaw: for use primarily within the >> OpenJDK codebase and only exposed for application use because it was too >> inconvenient to hide it. >> >> Regards, >> Glyn >> >> On 12 Nov 2011, at 11:59, Peter Kriens wrote: >> >>> Neither my wrath, nor the fact that I rarely if ever get angry is >>> relevant in this discussion ... This is a technical argument that are >>> solvable by technical people that share the same goals. I prefer package >>> dependencies because they address the excessive type coupling problem in >>> object oriented systems, not because they're part of OSGi. Let me argue my >>> case. >>> >>> Module-to-package dependencies (m2p) are preferable over module-to-module >>> dependencies (m2m) for many reasons but these are the most important >>> reasons: >>> >>> M2P is leverages the Java type system unlike m2m that must introduce new >>> namespaces outside the Java type system. >>> M2P can be used to break the transitive dependency chain, m2m suffers of >>> excessive coupling >>> >>> Since the first bullet's benefit should be clear I only argue the more >>> complex second bullet. >>> >>> A module is in many regards like a class. A class encapsulates members, >>> depends on other members/classes, and makes a few members accessible outside >>> the class. A module has a similar structure but then with types/packages as >>> members. >>> >>> After the initial success of Object Oriented Programming (OO) it was >>> quickly learned that reuse did not take place at the expected scale due to >>> excessive type coupling. The problem was that a class aggregated many >>> dependencies to simplify its implementation but these dependencies were >>> unrelated to the contract it implemented. Since class dependencies are >>> transitive most applications disappointingly became an almost fully >>> connected graph. >>> >>> Java's great innovation was the interface because it broke both the >>> transitivity and aggregation of dependencies. A class could now express its >>> dependency (use or implement) on a contract (the interface) and was >>> therefore fully type decoupled from the opposite site. >>> >>> An interface can act as a contract because it names the signature of a >>> set of methods so that the compiler can verify the client and the >>> implementer. >>> >>> Since a module has a very similar structure to a class it suffers from >>> exactly the same transitive aggregation of dependencies. This is not a >>> theory, look at the experiences with Maven >>> (http://www.sonatype.com/people/2011/04/how-not-to-download-the-internet/) >>> Again, this is not that maven is bad or developers are stupid, it is the >>> same underlying force that finally resulted in the Java interface. >>> >>> The parallel for the class' interface for modules is a named set of >>> interfaces. This concept already exists in Java: a package. Looking at >>> almost all JSRs it is clear that our industry already uses packages as >>> "interfaces" to provider implementations. >>> >>> Therefore, just like a class should not depend on other implementation >>> types, a module should preferably not depend on other modules. A module >>> should instead depend on contracts. Since modules will be used to provide >>> components from different sources managed with different life cycles the >>> excessive type coupling caused by m2m is even more damaging than in c2c. >>> Proper use of m2p creates significantly less type coupled systems than m2m, >>> the benefits should be obvious. >>> >>> Since there are use cases for m2m (non-type safe languages for example) I >>> do believe that Jigsaw should still support m2m. However, it would be >>> greatly beneficial to our industry if we could take advantage of the lessons >>> learned with the Java interface and realize how surprisingly important the >>> Java package actually is in our eco system. >>> >>> Kind regards, >>> >>> ? ? ? ?Peter Kriens >>> >>> >>> >>> >>> >>> >>> >>> >>> On 9 nov. 2011, at 15:04, David M. Lloyd wrote: >>> >>>> I'll just state now that using packages as a dependency unit is a >>>> terrible idea, and not some architectural revelation. ?That way, Peter's >>>> wrath will be largely directed at me. :-) >>>> >>>> On 11/09/2011 08:02 AM, Peter Kriens wrote: >>>>> >>>>> I agree that tools are needed but we must be careful to not expect >>>>> tools to stopgap an architectural issue. I think it is important to first do >>>>> good architectural design leveraging existing tools (e.g. the Java type >>>>> system) before you try to add new tools. It is such a pity (but all to >>>>> common) that a design allows for classes of errors that would be impossible >>>>> with a slightly different design. >>>>> >>>>> Kind regards, >>>>> >>>>> ? ? ? ?Peter Kriens >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On 9 nov. 2011, at 14:49, Alan Bateman wrote: >>>>> >>>>>> On 09/11/2011 13:04, Peter Kriens wrote: >>>>>>> >>>>>>> The issue is that maven problems are not caused because maven is bad >>>>>>> or that pom authors are stupid. The reason is that the module-to-module >>>>>>> dependency architecture in maven (and Jigsaw) is error prone ... >>>>>> >>>>>> This thread started out with someone asking about adding module >>>>>> declarations to existing JAR files, and in that context, I agree it can be >>>>>> error prone without good tools. I think things should be a lot better when >>>>>> modules are compiled. >>>>>> >>>>>> -Alan. >>>>>> >>>>> >>>> >>>> >>>> -- >>>> - DML >>> >> > > > -- > - DML > From mandy.chung at oracle.com Mon Nov 14 20:14:01 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Mon, 14 Nov 2011 20:14:01 -0800 Subject: Review request: jpkg -m option takes both classes and resources In-Reply-To: <4EC16883.3070604@oracle.com> References: <4EBB5782.2050904@oracle.com> <4EC15F4F.8080802@oracle.com> <4EC16415.3010905@oracle.com> <4EC16883.3070604@oracle.com> Message-ID: <4EC1E709.9010807@oracle.com> On 11/14/2011 11:14 AM, Alan Bateman wrote: > On 14/11/2011 18:55, Mandy Chung wrote: >> >> I agree that the class names and the getInstance method should be >> renamed. How about renaming the class names to >> CompressorOutputStream and CompressedClassFileOutputStream and >> renaming getInstance to newInstance? > Yes, that would be better (alternatively CompressedClassOutputStream). Rename done. I also changed the newInstance method to do the compression before returning the compressed output stream instance (this removes the call to the compress method from several callers). Upated webrev: http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/jpkg-dash-r-option.01/ Mandy From david.lloyd at redhat.com Mon Nov 14 21:14:37 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 14 Nov 2011 23:14:37 -0600 Subject: Why package deps won't work (Was: Re: Converting plain JARs to Java modules) In-Reply-To: References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> <67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz> <4EC14A53.7070907@redhat.com> Message-ID: <4EC1F53D.7040309@redhat.com> On 11/14/2011 08:33 PM, Neil Bartlett wrote: > Hi David, > > More than happy to talk about reality, though Peter already was doing > so. Nothing he said was theoretical, it came directly from his > experiences and the experiences of developers who have been using OSGi > for around 12 years. > Before digging into the points in your message, I need to address the > subject line "Why package deps won't work". Package dependencies will > work and they do work, as is proven every day by the large number of > applications running on OSGi. To claim otherwise would be wilfully > ignorant, so I will assume you are being hyperbolic and really meant > to assert that module deps are just better than package deps. > Therefore I intend to argue on that basis and claim that package deps > are better than module deps; note that I don't claim that module > dependencies "won't work" because JBoss Modules is a proof point that > they do. There was an implied "... for Java SE 8 modules" after "won't work" (this is why I said "won't" rather than "don't") which apparently wasn't implied enough. Of course people use OSGi, or we wouldn't be arguing, but that alone isn't enough to make OSGi's dependency system preferable for the SE case (after all, far more people use plain old class paths). I believe that JBoss Modules is a very accurate representation of how an SE module system could function, right down to executing JARs on the command line with dependencies and modularizing certain JDK APIs. This is not a proof point, but strong evidence that it is an effective solution for the actual problem on the table, and that similar architectures are likely to succeed for the same problem domain. As to why package deps are not an example of an effective solution for this problem, I intend to illustrate in greater detail. Further responses and elaboration below. > On Mon, Nov 14, 2011 at 5:05 PM, David M. Lloyd wrote: >> The explanation is quite simple, really - each point can be pretty much >> wiped out by a heavy dose of reality. >> >> 1. "M2P is leverages the Java type system unlike m2m that must introduce new >> namespaces outside the Java type system." - this is just fluffy >> buzzwordology. Considering packages part of the Java type system is a >> pretty liberal interpretation of the term "type system" as they're basically >> just arbitrary name spaces. That said, there's nothing particularly "new" >> about namespacing modules. JARs have names. Projects have names. Maven >> uses its own namespaces for artifacts. Even the primitive JDK extension >> mechanism uses symbolic names. > > From Wikipedia (http://en.wikipedia.org/wiki/Type_system): "A type > system associates a type with each computed value ... the aim is to > prevent operations expecting a certain kind of value being used with > values for which that operation does not make sense." > > The following will not compile: > > import java.awt.List; > // ... > List list = new ArrayList(); > list.iterator(); // etc > > whereas the following will: > > import java.util.List; > // ... > List list = new ArrayList(); > list.iterator(); // etc > > Package names are part of the type system because using an incorrect > package name in Java source can result in type errors during > compilation, and because the type and available operations associated > with each value varies with the package name used. > > In contrast, it is impossible to obtain a type error at compilation by > incorrectly naming a JAR, project, Maven artefact etc, because none of > those things are ever referenced in Java source code. I stand by my statement. A package is part of the name of the type; it is not a type, nor a meta-type, nor anything like a type in and of itself. This makes them part of the language, sure, but not part of the type system, in my opinion. Thus, "a stretch". I don't feel any particular need to have the world agree with me on this though, other than as I state below... >> Even a simple convention of the parent-most package name for the name of a >> module is very simple to grasp and is in fact the solution we've used quite >> effectively thus far. Thus implying that module names are some new alien >> concept is really not valid. > > The claim is not that they are a new alien concept. The claim is that > they are not part of the Java language and type system, and as a > result of this (along with other problems) are less useful for > depending upon than packages. I don't see how not being part of the Java language makes module identifiers "less useful". I think that this argument artificially elevates the significance of the relationship between packages and the actual type system, and then uses that elevation as a proof that a system based on other namespace principles is deficient, without offering any real examples which actually relate to the type system. >> 2. "M2P can be used to break the transitive dependency chain, m2m suffers of >> excessive coupling." - How about some facts to back this up? I've found >> "m2m" coupling to be just right. In JBoss Modules, we do not export >> transitive dependencies by default. This results in a very simple and clean >> dependency graph between modules. Using package dependencies results in a >> *far more complex* dependency graph, because you need edges for every >> package even though *most of the time you want the whole module anyway*. >> Again, real world here. > > Peter is definitely also talking about the real world and so am I. In > an m2m dependency model you cannot avoid transitive dependencies, > whether you expose them to consumers or not. In *any* model you cannot avoid transitive dependencies. If module A exports module B as part of its API, then you get A and B, and that's life. If you only want part of B, then you filter it down. But with most existing libraries out there (and they easily number in the thousands), they're intended to be used whole, so this doesn't really happen too much in practice. > An importer of a module must be assumed to depend on the whole functionality of that module > whether or not that is actually the case, and therefore all of the > transitive dependencies must be present at runtime. In an m2p world we > have the opportunity to split modules and break apart dependency > graphs, because the unit of coupling is more granular. You could break things down by actual classes if granularity is the goal. But it isn't; it is so much less important than, say, performance, conceptual simplicity, usability, efficiency, memory footprint, etc. The fact is that the vast majority of modules you'll find published at, say Maven central, are designed to be used completely, not partially, and can and are used as such without issue. People just don't usually design modules to be imported by package, except perhaps as an afterthought concession to OSGi, and they don't use them that way at runtime either. It is so seldom a real problem in the SE world (or EE for that matter) once you step away from flat class paths. The simple fact is that most users think of their dependencies in the same terms that their IDEs and their build scripts do: by module. This is truly a case of creating a lot of complexity and extra work in the common case to avoid problems which are not a common case. I think it is much better to optimize for the common case, where people know exactly what dependencies they want (in terms of artifacts), and they just want to declare them and be done without (a) hunting down package lists or (b) running extra tooling. >> If we're just going to throw dogma around, I'll put it the other way: m2p is >> a design error by the OSGi spec designers which has since been embraced as a >> religion. > > Pejorative accusations about "dogma" or "religion" have no place in a > technical discussion. They are also untrue. All of the practising OSGi > developers I know have arrived at their support for package > dependencies as a result of real world experience, not because of some > willingness to bow down to the Almighty CPEG. I don't know of any > practising OSGi developer who has used both Require-Bundle (m2m) and > Import-Package (m2p) and actually prefers the former. Sure, but Require-Bundle is *not* the same thing as using module-to-module dependencies in a non-OSGi system. All OSGi life revolves around the resolver. In the SE world, you cannot have a resolver without introducing a lot of machinery which will definitely negatively impact boot performance in the case that the resolver runs at module load time, or add the complexity of centralized package index management in the case that the resolver would run at install time (as in, install into a module library/repository, not install in the OSGi sense). If in a typical installation a user has to do more than simply delete a file to remove a module, or drop a JAR into a directory to add a module, I believe we've run quite far off-track usability-wise. And if a user has to sit through a lengthy resolution process at runtime (even if only the first time) then we're off-track performance-wise, especially if such resolution must expand to include modules in the module repository which are not even loaded. > I do know several who started as strong supporters of Require-Bundle > and switched to being supporters of Import-Package, not because of > Peter's wrathful pontificating but because they encountered the > specific problems that he described and found that they were fixed by > using Import-Package, and indeed that everything worked so much more > cleanly that way. I'm in this camp myself, and so was Jeff McAffer > before he went over to the Dark Side (Microsoft). There are other, simpler ways to fix package conflict issues when they arise. Like simply excluding the offending package from the import. And as I've said before, I do not believe that Require-Bundle in the context of OSGi bundles is comparable to loading modules by name in the context of a module system such as Jigsaw or JBoss Modules and the types of modules that could be loaded thereby (the aforementioned thousands of existing modules). >> It offers no significant benefit, other than a couple of edge >> cases which are frankly just as well handled by m2m simply by adding package >> filters. Which, by the way, I haven't seen the need to do yet in our 200+ >> module environment, but which we do have the capability to do. > > Substitution, module refactoring and transitive decoupling are hardly > edge cases. Substitution isn't solely addressed by package dependencies. Nor is module refactoring. And additional transitive decoupling isn't something people actually want - as I said, the common case is that users want to be able to use any class from a module once they import that module. In the real world, it is very rare to only want to import part of a module! If you need this routinely, you've done something very unorthodox. Thus I stand by my statement. > However I can see how these issues might not yet have come > to the fore in a module system designed for a single product with a > small number of modules, and where that product has not yet been > through the mill of multiple version evolutions. Cute. I assure you that we have been around through a long, long history of a huge number of Java users doing everything under the sun to their class paths and class loading structures. We also have experience with operating system distribution and managing module distribution across a wide range of other programming languages, not just Java. Our module system has already stood up to reasonably large deployments (200+ modules) with excellent memory footprint and performance numbers and we've been processing usability feedback, including feedback from those with OSGi experience, which has been mostly positive. Furthermore when I was at JavaOne giving booth talks about JBoss Modules and AS7, I found that users expressed quite a bit of frustration at the OSGi model and were quite receptive to a more orthodox m2m system, for what that's worth. >> M2P is a solution just itching for a problem. > > This is also not a very useful statement, I assure you that the > problem came before the solution. Better to show why you think the > problem is invalid or should have been solved differently. There is no way you can convince me that this solution was invented with general modularity in mind. Back when OSGi was first formed, it was designed for the embedded market. The world of Java today is completely different. People just don't bundle their libraries by package, as was possibly expected in the early days of Java. If they did we wouldn't be having this discussion because packages and modules would already be one and the same. OSGi evolved to where it is today. It was not designed from the ground up with actual requirements which pertained to the problem at hand which is modularity of the SE platform and applications which run on it. So yeah I'm going to say, solution before problem. Taking a solution and trying to apply it retroactively to a different problem like this has never failed to bite me in the ass, personally. But I can't speak to everyone's experience. >> But you're going to have a >> tough time convincing me that users *want* to have to use special tooling >> because they want to depend on a module which has too many packages to list >> out by hand. And before you cry "wildcards", be sure to consider that some >> modules use package names which are subordinate to other modules' packages, >> which is a perfectly normal and allowable scenario. Using wildcards for >> package matching could cause amazing levels of havoc. > > OSGi never uses wildcards at runtime, and tools such as bnd do not > need wildcards in order to express package-level dependencies. They > extract the set of packages that were actually used by the code, all > of which are available in the class files. This is possible because > packages are part of the type system (see first point above). > > So it's not that I don't want to list dependencies by hand, rather I > only want to do it once. I am already forced to do it in the import > statements of my Java sources. If I had to repeat that dependency > information -- whether in m2p or m2m form -- then I would run the risk > of it getting out of step with the real dependencies. One of the critical logical gaps here which m2p ignores at a fundamental level is that *packages are not unique*. In the real world, packages are repeated across more than one module all the time, and not just due to versioning. There is *no way* you could take every JAR at Maven Central and plug it in to an m2p system. You'd have to break apart and repackage every single one, at least, and at worst you'd have to rewrite an awful lot of them under different package names to account for this restriction. You'd need to enforce package uniqueness across your whole library, for all time. I don't think this is a sane option. You're adding significant overhead to management, installation, and execution, for what? So you can have the privilege of needing a special tool to extract your dependencies for you? >> Using package dependencies means you either must have a master package index >> for linking, or you need a resolver which has to have analyzed every module >> you ever plan to load. Otherwise, O(1) loading of modules is impossible, >> which is absolutely 100% a deal-breaker for JDK modules which must be >> incrementally installable. And it forbids having packages with the same >> name in more than one JAR without bringing run-time versioning into the >> fold, which is a terrible, terrible can of worms. > > Could you please explain why O(1) is the only acceptable complexity > for installing modules. Because this is Java SE we're talking about. You're going to have a potentially huge number of modules installed in your system. Even if you get around the versioning issues somehow and fully externalize the index of all packages, you still have to load or traverse the centralized index for every module load. I say that not only is ?(1) the only acceptable complexity for installing modules but also loading them at run time. Linking modules at run time should be no worse than O(n) complexity for the number of dependencies the module has, including transitive dependencies. Loading classes and resources from modules should be ?(k) where k is the number of modules which contain a package or directory which matches the class or resource being loaded (almost always one, in my experience). Loading a module should normally be limited to a one-time O(1) disk access without searching or loading any other files or modules than the module artifact itself (and possibly its descriptor if they are held externally by a particular module repository implementation, which is useful but not necessary). Linking a module should be limited in disk access to loading only the modules it imports via direct dependency, or by the relatively rare partial or full re-export of a transitive dependency by a direct dependency. In particular, the expectation of accessing a central index has some potentially serious implications, facing either possible file lock contention by multiple threads or memory overhead of loading in the complete index in advance. It should be expected that modules are loaded concurrently, and such concurrency should not be hindered any more than necessary. Other possible implementations (such as file- or directory-per-package) have their own drawbacks as well or violate what I consider to be core requirements. If there are implementations of package-based resolution which don't involve either a centralized index of one or many files or a resolver which must rove all installed modules in advance, I'd like to hear about it. > OSGi does indeed support incremental install > and while I accept it is probably not O(1) for each module, it would > likely be no more than O(N), though I haven't done the maths yet to > prove this. Bear in mind that in practice, for small N, the constant > factors can result in O(1) being more expensive than O(N). I have seen > OSGi used with many thousands of modules, so unless you have some data > and a use-case showings package-level resolution as unacceptably slow, > your concern just sounds like premature optimisation. Identifying algorithmic complexity during design phase is *never* premature optimization AFAICT. If you don't understand the complexity of the algorithms you're about to implement, and how they are expected to be applied to the problem at hand, then you're not ready for implementation yet. However, postulating that O(n) is okay because can sometimes be faster than O(1) without measuring it for the specific problem in question is definitely premature de-optimization. :-) > There are two reasons to have a packages with the same name in more > than one JAR. The first is a situation called split packages, and it > is highly undesirable because it causes the runtime model of the > package to diverge from the compile-time model, and therefore things > like package-private types and members stop working correctly. For > this reason, OSGi's m2p imports support depending only upon a single > exporter of a particular package, i.e. we do not aggregate all exports > of that package. > > Unfortunately split packages are sometimes unavoidable in legacy code > that cannot be refactored, e.g. the JDK. To support such scenarios > OSGi has Require-Bundle, i.e. m2m. This does not negate the problems > associated with m2m, it is simply a trade-off that we face with poorly > factored legacy code. > > The second reason for multiple packages with the same name is when you > explicitly want to install multiple versions of a library/API and have > them all available within the same runtime. I wouldn't call this a > "can of worms" exactly because it can be done without too much > trouble, though for the sake of a simple life I personally avoid this > situation unless it's necessary. Yes, and it is actually fairly common in practice to want two versions of something in the sense of the two versions being wholly different implementations (think apache commons logging versus jcl-over-slf4j for a trivial example which crops up a lot). There are reasons to use one or the other. Multiplicity of versions (in this respect) is something the module system has to handle gracefully and simply. >> Finally it should be perfectly clear to anyone who has read the original >> requirements document that nothing in this module system should prevent OSGi >> from functioning as it is, so there is absolutely no reason to assume that >> any OSGi implementation is so threatened - especially if m2p linking is as >> superior as has been expressed. Our module system (which is conceptually >> similar to Jigsaw in many regards) in fact does support our OSGi >> implementation quite effectively without itself implementing OSGi's >> package-to-package resolution (which like I said throws O(1) out the >> window). > > I agree that Jigsaw's existence doesn't threaten OSGi's, so long as > Java 8 doesn't actually break OSGi (and if it did so, it would > likewise break many other applications and could not be considered > backwards compatible with Java 7). The two can interoperate through > m2m-type dependencies. Tim Ellison started Project Penrose for the > purpose of investigating, testing and deepening this collaboration. > > Neverthless, the point that I believe Glyn was making is the > following. We accept that m2m dependencies are probably required for > the JDK, which implies a module system like Jigsaw or > OSGi/Require-Bundle rather than OSGi/Import-Package. However is it > intended to be used for application modularisation as well? This is of > course a question for the Jigsaw team rather than you, David. The question of whether Java SE 8 modules are intended to be used for application modularization is a question for the EG, not the Jigsaw team. The job of the Jigsaw team is really to implement a prototype which meets the requirements set forth by the EG, which may become the reference implementation at a future point. The question of whether the greater Java community will embrace the SE module system for applications is to be answered by the community only. I personally believe that to develop a module system which is not intended to be usable by the average standalone application is foolhardy, a great waste of effort, and is doomed to mediocrity, given the success we have had with such a system. However until there is an EG, these are all just as much questions for me as for anybody, and I can and will answer to the best of my knowledge and belief. > As a result of experience in developing and evolving large real-world > applications using a module system that supports BOTH m2m and m2p > dependencies, I believe it would be very unfortunate if a module > system that supports ONLY m2m were to become widely used in the > application space... not because OSGi can't handle the competition, > but because those applications will be fragile and hard to evolve. I don't believe this to be the case. I think that the status quo doesn't result in particularly fragile applications, and such can be easy to evolve or hard depending on the quality of the application components and frameworks involved. I think that m2m dependencies enhance the status quo such that applications are somewhat less fragile (owing chiefly to the simple measure of preventing transitive dependencies from being exported by default), and quite easy to evolve as well (even large and complex frameworks rarely have more than a few dependencies, unless they themselves are overly fragmented (e.g. CXF as a nightmare example which yet still works fine under an m2m system)). > My question for you David is as follows. I understand that you prefer > module dependencies, but do you believe that package dependencies have > no value whatsoever and therefore should not be available to > application developers in the Java 8 module system? If so, why did Red > Hat create an OSGi implementation? I personally believe that they have some value, but that value is limited to interoperability with OSGi. I do not believe that it is a preferable model for most users or standalone applications, nor for larger applications such as our application server. I think that the requirement for extra tooling and the increased run-time complexity which is exposed to users nullifies the benefits. If someone really wants this variety of modularity, they should simply use OSGi. Red Hat created an OSGi implementation for many reasons, but the only significant one to me is that there are people who want to use their OSGi applications with our application server. I don't believe any other reason is even necessary. It's the same reason we strive to support any spec, from EJB 1.0 to Java EE 6 and beyond. Someone wants the functionality, so we deliver it to the best of our ability. > Kind regards > Neil > >> >> On 11/14/2011 01:49 AM, Glyn Normington wrote: >>> >>> I look forward to David's elaboration of why he thinks "using packages as >>> a dependency unit is a terrible idea" to balance Peter's clear explanation >>> of the benefits of m2p. >>> >>> Meanwhile, it's worth noting that, according to the requirements document, >>> Jigsaw is aimed at platform modularisation and the platform being >>> modularised has some non-optimal division of types across packages (see the >>> package subsets requirement) which favour m2m dependencies. (Note that >>> Apache Harmony was developed with modularity in mind and was able to exploit >>> m2p, so platform modularisation per se needn't be limited to m2m.) >>> >>> So if Jigsaw excludes m2p, it will then be applicable to certain kinds of >>> legacy code modularisation and less applicable to new module development and >>> modularisation of existing code whose division into packages suits m2p. IIRC >>> this was the original positioning of Jigsaw: for use primarily within the >>> OpenJDK codebase and only exposed for application use because it was too >>> inconvenient to hide it. >>> >>> Regards, >>> Glyn >>> >>> On 12 Nov 2011, at 11:59, Peter Kriens wrote: >>> >>>> Neither my wrath, nor the fact that I rarely if ever get angry is >>>> relevant in this discussion ... This is a technical argument that are >>>> solvable by technical people that share the same goals. I prefer package >>>> dependencies because they address the excessive type coupling problem in >>>> object oriented systems, not because they're part of OSGi. Let me argue my >>>> case. >>>> >>>> Module-to-package dependencies (m2p) are preferable over module-to-module >>>> dependencies (m2m) for many reasons but these are the most important >>>> reasons: >>>> >>>> M2P is leverages the Java type system unlike m2m that must introduce new >>>> namespaces outside the Java type system. >>>> M2P can be used to break the transitive dependency chain, m2m suffers of >>>> excessive coupling >>>> >>>> Since the first bullet's benefit should be clear I only argue the more >>>> complex second bullet. >>>> >>>> A module is in many regards like a class. A class encapsulates members, >>>> depends on other members/classes, and makes a few members accessible outside >>>> the class. A module has a similar structure but then with types/packages as >>>> members. >>>> >>>> After the initial success of Object Oriented Programming (OO) it was >>>> quickly learned that reuse did not take place at the expected scale due to >>>> excessive type coupling. The problem was that a class aggregated many >>>> dependencies to simplify its implementation but these dependencies were >>>> unrelated to the contract it implemented. Since class dependencies are >>>> transitive most applications disappointingly became an almost fully >>>> connected graph. >>>> >>>> Java's great innovation was the interface because it broke both the >>>> transitivity and aggregation of dependencies. A class could now express its >>>> dependency (use or implement) on a contract (the interface) and was >>>> therefore fully type decoupled from the opposite site. >>>> >>>> An interface can act as a contract because it names the signature of a >>>> set of methods so that the compiler can verify the client and the >>>> implementer. >>>> >>>> Since a module has a very similar structure to a class it suffers from >>>> exactly the same transitive aggregation of dependencies. This is not a >>>> theory, look at the experiences with Maven >>>> (http://www.sonatype.com/people/2011/04/how-not-to-download-the-internet/) >>>> Again, this is not that maven is bad or developers are stupid, it is the >>>> same underlying force that finally resulted in the Java interface. >>>> >>>> The parallel for the class' interface for modules is a named set of >>>> interfaces. This concept already exists in Java: a package. Looking at >>>> almost all JSRs it is clear that our industry already uses packages as >>>> "interfaces" to provider implementations. >>>> >>>> Therefore, just like a class should not depend on other implementation >>>> types, a module should preferably not depend on other modules. A module >>>> should instead depend on contracts. Since modules will be used to provide >>>> components from different sources managed with different life cycles the >>>> excessive type coupling caused by m2m is even more damaging than in c2c. >>>> Proper use of m2p creates significantly less type coupled systems than m2m, >>>> the benefits should be obvious. >>>> >>>> Since there are use cases for m2m (non-type safe languages for example) I >>>> do believe that Jigsaw should still support m2m. However, it would be >>>> greatly beneficial to our industry if we could take advantage of the lessons >>>> learned with the Java interface and realize how surprisingly important the >>>> Java package actually is in our eco system. >>>> >>>> Kind regards, >>>> >>>> Peter Kriens >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On 9 nov. 2011, at 15:04, David M. Lloyd wrote: >>>> >>>>> I'll just state now that using packages as a dependency unit is a >>>>> terrible idea, and not some architectural revelation. That way, Peter's >>>>> wrath will be largely directed at me. :-) >>>>> >>>>> On 11/09/2011 08:02 AM, Peter Kriens wrote: >>>>>> >>>>>> I agree that tools are needed but we must be careful to not expect >>>>>> tools to stopgap an architectural issue. I think it is important to first do >>>>>> good architectural design leveraging existing tools (e.g. the Java type >>>>>> system) before you try to add new tools. It is such a pity (but all to >>>>>> common) that a design allows for classes of errors that would be impossible >>>>>> with a slightly different design. >>>>>> >>>>>> Kind regards, >>>>>> >>>>>> Peter Kriens >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On 9 nov. 2011, at 14:49, Alan Bateman wrote: >>>>>> >>>>>>> On 09/11/2011 13:04, Peter Kriens wrote: >>>>>>>> >>>>>>>> The issue is that maven problems are not caused because maven is bad >>>>>>>> or that pom authors are stupid. The reason is that the module-to-module >>>>>>>> dependency architecture in maven (and Jigsaw) is error prone ... >>>>>>> >>>>>>> This thread started out with someone asking about adding module >>>>>>> declarations to existing JAR files, and in that context, I agree it can be >>>>>>> error prone without good tools. I think things should be a lot better when >>>>>>> modules are compiled. >>>>>>> >>>>>>> -Alan. >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> - DML >>>> >>> >> >> >> -- >> - DML >> -- - DML From brian at pontarelli.com Tue Nov 15 06:13:00 2011 From: brian at pontarelli.com (Brian Pontarelli) Date: Tue, 15 Nov 2011 07:13:00 -0700 Subject: Why package deps won't work (Was: Re: Converting plain JARs to Java modules) In-Reply-To: <4EC1F53D.7040309@redhat.com> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> <67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz> <4EC14A53.7070907@redhat.com> <4EC1F53D.7040309@redhat.com> Message-ID: <38B01C7D-9072-43F7-8398-0D1A3C2F114A@pontarelli.com> I'm not at a computer to reply properly, but a few things about packages that come to mind are: - package visibility - package-info.java - the Package class - package versioning Yank these out, specifically package-private visibility, and packages are just names. Otherwise, it seems to me that they are part of the type system. Sent from my iPad On Nov 14, 2011, at 10:14 PM, "David M. Lloyd" wrote: > On 11/14/2011 08:33 PM, Neil Bartlett wrote: >> Hi David, >> >> More than happy to talk about reality, though Peter already was doing >> so. Nothing he said was theoretical, it came directly from his >> experiences and the experiences of developers who have been using OSGi >> for around 12 years. > >> Before digging into the points in your message, I need to address the >> subject line "Why package deps won't work". Package dependencies will >> work and they do work, as is proven every day by the large number of >> applications running on OSGi. To claim otherwise would be wilfully >> ignorant, so I will assume you are being hyperbolic and really meant >> to assert that module deps are just better than package deps. >> Therefore I intend to argue on that basis and claim that package deps >> are better than module deps; note that I don't claim that module >> dependencies "won't work" because JBoss Modules is a proof point that >> they do. > > There was an implied "... for Java SE 8 modules" after "won't work" (this is why I said "won't" rather than "don't") which apparently wasn't implied enough. > > Of course people use OSGi, or we wouldn't be arguing, but that alone isn't enough to make OSGi's dependency system preferable for the SE case (after all, far more people use plain old class paths). I believe that JBoss Modules is a very accurate representation of how an SE module system could function, right down to executing JARs on the command line with dependencies and modularizing certain JDK APIs. This is not a proof point, but strong evidence that it is an effective solution for the actual problem on the table, and that similar architectures are likely to succeed for the same problem domain. As to why package deps are not an example of an effective solution for this problem, I intend to illustrate in greater detail. > > Further responses and elaboration below. > >> On Mon, Nov 14, 2011 at 5:05 PM, David M. Lloyd wrote: >>> The explanation is quite simple, really - each point can be pretty much >>> wiped out by a heavy dose of reality. >>> >>> 1. "M2P is leverages the Java type system unlike m2m that must introduce new >>> namespaces outside the Java type system." - this is just fluffy >>> buzzwordology. Considering packages part of the Java type system is a >>> pretty liberal interpretation of the term "type system" as they're basically >>> just arbitrary name spaces. That said, there's nothing particularly "new" >>> about namespacing modules. JARs have names. Projects have names. Maven >>> uses its own namespaces for artifacts. Even the primitive JDK extension >>> mechanism uses symbolic names. >> >> From Wikipedia (http://en.wikipedia.org/wiki/Type_system): "A type >> system associates a type with each computed value ... the aim is to >> prevent operations expecting a certain kind of value being used with >> values for which that operation does not make sense." >> >> The following will not compile: >> >> import java.awt.List; >> // ... >> List list = new ArrayList(); >> list.iterator(); // etc >> >> whereas the following will: >> >> import java.util.List; >> // ... >> List list = new ArrayList(); >> list.iterator(); // etc >> >> Package names are part of the type system because using an incorrect >> package name in Java source can result in type errors during >> compilation, and because the type and available operations associated >> with each value varies with the package name used. >> >> In contrast, it is impossible to obtain a type error at compilation by >> incorrectly naming a JAR, project, Maven artefact etc, because none of >> those things are ever referenced in Java source code. > > I stand by my statement. A package is part of the name of the type; it is not a type, nor a meta-type, nor anything like a type in and of itself. This makes them part of the language, sure, but not part of the type system, in my opinion. Thus, "a stretch". I don't feel any particular need to have the world agree with me on this though, other than as I state below... > >>> Even a simple convention of the parent-most package name for the name of a >>> module is very simple to grasp and is in fact the solution we've used quite >>> effectively thus far. Thus implying that module names are some new alien >>> concept is really not valid. >> >> The claim is not that they are a new alien concept. The claim is that >> they are not part of the Java language and type system, and as a >> result of this (along with other problems) are less useful for >> depending upon than packages. > > I don't see how not being part of the Java language makes module identifiers "less useful". I think that this argument artificially elevates the significance of the relationship between packages and the actual type system, and then uses that elevation as a proof that a system based on other namespace principles is deficient, without offering any real examples which actually relate to the type system. > >>> 2. "M2P can be used to break the transitive dependency chain, m2m suffers of >>> excessive coupling." - How about some facts to back this up? I've found >>> "m2m" coupling to be just right. In JBoss Modules, we do not export >>> transitive dependencies by default. This results in a very simple and clean >>> dependency graph between modules. Using package dependencies results in a >>> *far more complex* dependency graph, because you need edges for every >>> package even though *most of the time you want the whole module anyway*. >>> Again, real world here. >> >> Peter is definitely also talking about the real world and so am I. In >> an m2m dependency model you cannot avoid transitive dependencies, >> whether you expose them to consumers or not. > > In *any* model you cannot avoid transitive dependencies. If module A exports module B as part of its API, then you get A and B, and that's life. If you only want part of B, then you filter it down. But with most existing libraries out there (and they easily number in the thousands), they're intended to be used whole, so this doesn't really happen too much in practice. > >> An importer of a module must be assumed to depend on the whole functionality of that module >> whether or not that is actually the case, and therefore all of the >> transitive dependencies must be present at runtime. In an m2p world we >> have the opportunity to split modules and break apart dependency >> graphs, because the unit of coupling is more granular. > > You could break things down by actual classes if granularity is the goal. But it isn't; it is so much less important than, say, performance, conceptual simplicity, usability, efficiency, memory footprint, etc. The fact is that the vast majority of modules you'll find published at, say Maven central, are designed to be used completely, not partially, and can and are used as such without issue. > > People just don't usually design modules to be imported by package, except perhaps as an afterthought concession to OSGi, and they don't use them that way at runtime either. It is so seldom a real problem in the SE world (or EE for that matter) once you step away from flat class paths. The simple fact is that most users think of their dependencies in the same terms that their IDEs and their build scripts do: by module. > > This is truly a case of creating a lot of complexity and extra work in the common case to avoid problems which are not a common case. I think it is much better to optimize for the common case, where people know exactly what dependencies they want (in terms of artifacts), and they just want to declare them and be done without (a) hunting down package lists or (b) running extra tooling. > >>> If we're just going to throw dogma around, I'll put it the other way: m2p is >>> a design error by the OSGi spec designers which has since been embraced as a >>> religion. >> >> Pejorative accusations about "dogma" or "religion" have no place in a >> technical discussion. They are also untrue. All of the practising OSGi >> developers I know have arrived at their support for package >> dependencies as a result of real world experience, not because of some >> willingness to bow down to the Almighty CPEG. I don't know of any >> practising OSGi developer who has used both Require-Bundle (m2m) and >> Import-Package (m2p) and actually prefers the former. > > Sure, but Require-Bundle is *not* the same thing as using module-to-module dependencies in a non-OSGi system. All OSGi life revolves around the resolver. In the SE world, you cannot have a resolver without introducing a lot of machinery which will definitely negatively impact boot performance in the case that the resolver runs at module load time, or add the complexity of centralized package index management in the case that the resolver would run at install time (as in, install into a module library/repository, not install in the OSGi sense). > > If in a typical installation a user has to do more than simply delete a file to remove a module, or drop a JAR into a directory to add a module, I believe we've run quite far off-track usability-wise. And if a user has to sit through a lengthy resolution process at runtime (even if only the first time) then we're off-track performance-wise, especially if such resolution must expand to include modules in the module repository which are not even loaded. > >> I do know several who started as strong supporters of Require-Bundle >> and switched to being supporters of Import-Package, not because of >> Peter's wrathful pontificating but because they encountered the >> specific problems that he described and found that they were fixed by >> using Import-Package, and indeed that everything worked so much more >> cleanly that way. I'm in this camp myself, and so was Jeff McAffer >> before he went over to the Dark Side (Microsoft). > > There are other, simpler ways to fix package conflict issues when they arise. Like simply excluding the offending package from the import. > > And as I've said before, I do not believe that Require-Bundle in the context of OSGi bundles is comparable to loading modules by name in the context of a module system such as Jigsaw or JBoss Modules and the types of modules that could be loaded thereby (the aforementioned thousands of existing modules). > >>> It offers no significant benefit, other than a couple of edge >>> cases which are frankly just as well handled by m2m simply by adding package >>> filters. Which, by the way, I haven't seen the need to do yet in our 200+ >>> module environment, but which we do have the capability to do. >> >> Substitution, module refactoring and transitive decoupling are hardly >> edge cases. > > Substitution isn't solely addressed by package dependencies. Nor is module refactoring. And additional transitive decoupling isn't something people actually want - as I said, the common case is that users want to be able to use any class from a module once they import that module. In the real world, it is very rare to only want to import part of a module! If you need this routinely, you've done something very unorthodox. Thus I stand by my statement. > >> However I can see how these issues might not yet have come >> to the fore in a module system designed for a single product with a >> small number of modules, and where that product has not yet been >> through the mill of multiple version evolutions. > > Cute. I assure you that we have been around through a long, long history of a huge number of Java users doing everything under the sun to their class paths and class loading structures. We also have experience with operating system distribution and managing module distribution across a wide range of other programming languages, not just Java. Our module system has already stood up to reasonably large deployments (200+ modules) with excellent memory footprint and performance numbers and we've been processing usability feedback, including feedback from those with OSGi experience, which has been mostly positive. > > Furthermore when I was at JavaOne giving booth talks about JBoss Modules and AS7, I found that users expressed quite a bit of frustration at the OSGi model and were quite receptive to a more orthodox m2m system, for what that's worth. > >>> M2P is a solution just itching for a problem. >> >> This is also not a very useful statement, I assure you that the >> problem came before the solution. Better to show why you think the >> problem is invalid or should have been solved differently. > > There is no way you can convince me that this solution was invented with general modularity in mind. Back when OSGi was first formed, it was designed for the embedded market. The world of Java today is completely different. People just don't bundle their libraries by package, as was possibly expected in the early days of Java. If they did we wouldn't be having this discussion because packages and modules would already be one and the same. > > OSGi evolved to where it is today. It was not designed from the ground up with actual requirements which pertained to the problem at hand which is modularity of the SE platform and applications which run on it. So yeah I'm going to say, solution before problem. Taking a solution and trying to apply it retroactively to a different problem like this has never failed to bite me in the ass, personally. But I can't speak to everyone's experience. > >>> But you're going to have a >>> tough time convincing me that users *want* to have to use special tooling >>> because they want to depend on a module which has too many packages to list >>> out by hand. And before you cry "wildcards", be sure to consider that some >>> modules use package names which are subordinate to other modules' packages, >>> which is a perfectly normal and allowable scenario. Using wildcards for >>> package matching could cause amazing levels of havoc. >> >> OSGi never uses wildcards at runtime, and tools such as bnd do not >> need wildcards in order to express package-level dependencies. They >> extract the set of packages that were actually used by the code, all >> of which are available in the class files. This is possible because >> packages are part of the type system (see first point above). >> >> So it's not that I don't want to list dependencies by hand, rather I >> only want to do it once. I am already forced to do it in the import >> statements of my Java sources. If I had to repeat that dependency >> information -- whether in m2p or m2m form -- then I would run the risk >> of it getting out of step with the real dependencies. > > One of the critical logical gaps here which m2p ignores at a fundamental level is that *packages are not unique*. In the real world, packages are repeated across more than one module all the time, and not just due to versioning. There is *no way* you could take every JAR at Maven Central and plug it in to an m2p system. You'd have to break apart and repackage every single one, at least, and at worst you'd have to rewrite an awful lot of them under different package names to account for this restriction. You'd need to enforce package uniqueness across your whole library, for all time. I don't think this is a sane option. You're adding significant overhead to management, installation, and execution, for what? So you can have the privilege of needing a special tool to extract your dependencies for you? > >>> Using package dependencies means you either must have a master package index >>> for linking, or you need a resolver which has to have analyzed every module >>> you ever plan to load. Otherwise, O(1) loading of modules is impossible, >>> which is absolutely 100% a deal-breaker for JDK modules which must be >>> incrementally installable. And it forbids having packages with the same >>> name in more than one JAR without bringing run-time versioning into the >>> fold, which is a terrible, terrible can of worms. >> >> Could you please explain why O(1) is the only acceptable complexity >> for installing modules. > > Because this is Java SE we're talking about. You're going to have a potentially huge number of modules installed in your system. Even if you get around the versioning issues somehow and fully externalize the index of all packages, you still have to load or traverse the centralized index for every module load. > > I say that not only is ?(1) the only acceptable complexity for installing modules but also loading them at run time. Linking modules at run time should be no worse than O(n) complexity for the number of dependencies the module has, including transitive dependencies. Loading classes and resources from modules should be ?(k) where k is the number of modules which contain a package or directory which matches the class or resource being loaded (almost always one, in my experience). > > Loading a module should normally be limited to a one-time O(1) disk access without searching or loading any other files or modules than the module artifact itself (and possibly its descriptor if they are held externally by a particular module repository implementation, which is useful but not necessary). Linking a module should be limited in disk access to loading only the modules it imports via direct dependency, or by the relatively rare partial or full re-export of a transitive dependency by a direct dependency. > > In particular, the expectation of accessing a central index has some potentially serious implications, facing either possible file lock contention by multiple threads or memory overhead of loading in the complete index in advance. It should be expected that modules are loaded concurrently, and such concurrency should not be hindered any more than necessary. Other possible implementations (such as file- or directory-per-package) have their own drawbacks as well or violate what I consider to be core requirements. > > If there are implementations of package-based resolution which don't involve either a centralized index of one or many files or a resolver which must rove all installed modules in advance, I'd like to hear about it. > >> OSGi does indeed support incremental install >> and while I accept it is probably not O(1) for each module, it would >> likely be no more than O(N), though I haven't done the maths yet to >> prove this. Bear in mind that in practice, for small N, the constant >> factors can result in O(1) being more expensive than O(N). I have seen >> OSGi used with many thousands of modules, so unless you have some data >> and a use-case showings package-level resolution as unacceptably slow, >> your concern just sounds like premature optimisation. > > Identifying algorithmic complexity during design phase is *never* premature optimization AFAICT. If you don't understand the complexity of the algorithms you're about to implement, and how they are expected to be applied to the problem at hand, then you're not ready for implementation yet. > > However, postulating that O(n) is okay because can sometimes be faster than O(1) without measuring it for the specific problem in question is definitely premature de-optimization. :-) > >> There are two reasons to have a packages with the same name in more >> than one JAR. The first is a situation called split packages, and it >> is highly undesirable because it causes the runtime model of the >> package to diverge from the compile-time model, and therefore things >> like package-private types and members stop working correctly. For >> this reason, OSGi's m2p imports support depending only upon a single >> exporter of a particular package, i.e. we do not aggregate all exports >> of that package. >> >> Unfortunately split packages are sometimes unavoidable in legacy code >> that cannot be refactored, e.g. the JDK. To support such scenarios >> OSGi has Require-Bundle, i.e. m2m. This does not negate the problems >> associated with m2m, it is simply a trade-off that we face with poorly >> factored legacy code. >> >> The second reason for multiple packages with the same name is when you >> explicitly want to install multiple versions of a library/API and have >> them all available within the same runtime. I wouldn't call this a >> "can of worms" exactly because it can be done without too much >> trouble, though for the sake of a simple life I personally avoid this >> situation unless it's necessary. > > Yes, and it is actually fairly common in practice to want two versions of something in the sense of the two versions being wholly different implementations (think apache commons logging versus jcl-over-slf4j for a trivial example which crops up a lot). There are reasons to use one or the other. Multiplicity of versions (in this respect) is something the module system has to handle gracefully and simply. > >>> Finally it should be perfectly clear to anyone who has read the original >>> requirements document that nothing in this module system should prevent OSGi >>> from functioning as it is, so there is absolutely no reason to assume that >>> any OSGi implementation is so threatened - especially if m2p linking is as >>> superior as has been expressed. Our module system (which is conceptually >>> similar to Jigsaw in many regards) in fact does support our OSGi >>> implementation quite effectively without itself implementing OSGi's >>> package-to-package resolution (which like I said throws O(1) out the >>> window). >> >> I agree that Jigsaw's existence doesn't threaten OSGi's, so long as >> Java 8 doesn't actually break OSGi (and if it did so, it would >> likewise break many other applications and could not be considered >> backwards compatible with Java 7). The two can interoperate through >> m2m-type dependencies. Tim Ellison started Project Penrose for the >> purpose of investigating, testing and deepening this collaboration. >> >> Neverthless, the point that I believe Glyn was making is the >> following. We accept that m2m dependencies are probably required for >> the JDK, which implies a module system like Jigsaw or >> OSGi/Require-Bundle rather than OSGi/Import-Package. However is it >> intended to be used for application modularisation as well? This is of >> course a question for the Jigsaw team rather than you, David. > > The question of whether Java SE 8 modules are intended to be used for application modularization is a question for the EG, not the Jigsaw team. The job of the Jigsaw team is really to implement a prototype which meets the requirements set forth by the EG, which may become the reference implementation at a future point. > > The question of whether the greater Java community will embrace the SE module system for applications is to be answered by the community only. I personally believe that to develop a module system which is not intended to be usable by the average standalone application is foolhardy, a great waste of effort, and is doomed to mediocrity, given the success we have had with such a system. > > However until there is an EG, these are all just as much questions for me as for anybody, and I can and will answer to the best of my knowledge and belief. > >> As a result of experience in developing and evolving large real-world >> applications using a module system that supports BOTH m2m and m2p >> dependencies, I believe it would be very unfortunate if a module >> system that supports ONLY m2m were to become widely used in the >> application space... not because OSGi can't handle the competition, >> but because those applications will be fragile and hard to evolve. > > I don't believe this to be the case. I think that the status quo doesn't result in particularly fragile applications, and such can be easy to evolve or hard depending on the quality of the application components and frameworks involved. I think that m2m dependencies enhance the status quo such that applications are somewhat less fragile (owing chiefly to the simple measure of preventing transitive dependencies from being exported by default), and quite easy to evolve as well (even large and complex frameworks rarely have more than a few dependencies, unless they themselves are overly fragmented (e.g. CXF as a nightmare example which yet still works fine under an m2m system)). > >> My question for you David is as follows. I understand that you prefer >> module dependencies, but do you believe that package dependencies have >> no value whatsoever and therefore should not be available to >> application developers in the Java 8 module system? If so, why did Red >> Hat create an OSGi implementation? > > I personally believe that they have some value, but that value is limited to interoperability with OSGi. I do not believe that it is a preferable model for most users or standalone applications, nor for larger applications such as our application server. I think that the requirement for extra tooling and the increased run-time complexity which is exposed to users nullifies the benefits. If someone really wants this variety of modularity, they should simply use OSGi. > > Red Hat created an OSGi implementation for many reasons, but the only significant one to me is that there are people who want to use their OSGi applications with our application server. I don't believe any other reason is even necessary. It's the same reason we strive to support any spec, from EJB 1.0 to Java EE 6 and beyond. Someone wants the functionality, so we deliver it to the best of our ability. > >> Kind regards >> Neil >> >>> >>> On 11/14/2011 01:49 AM, Glyn Normington wrote: >>>> >>>> I look forward to David's elaboration of why he thinks "using packages as >>>> a dependency unit is a terrible idea" to balance Peter's clear explanation >>>> of the benefits of m2p. >>>> >>>> Meanwhile, it's worth noting that, according to the requirements document, >>>> Jigsaw is aimed at platform modularisation and the platform being >>>> modularised has some non-optimal division of types across packages (see the >>>> package subsets requirement) which favour m2m dependencies. (Note that >>>> Apache Harmony was developed with modularity in mind and was able to exploit >>>> m2p, so platform modularisation per se needn't be limited to m2m.) >>>> >>>> So if Jigsaw excludes m2p, it will then be applicable to certain kinds of >>>> legacy code modularisation and less applicable to new module development and >>>> modularisation of existing code whose division into packages suits m2p. IIRC >>>> this was the original positioning of Jigsaw: for use primarily within the >>>> OpenJDK codebase and only exposed for application use because it was too >>>> inconvenient to hide it. >>>> >>>> Regards, >>>> Glyn >>>> >>>> On 12 Nov 2011, at 11:59, Peter Kriens wrote: >>>> >>>>> Neither my wrath, nor the fact that I rarely if ever get angry is >>>>> relevant in this discussion ... This is a technical argument that are >>>>> solvable by technical people that share the same goals. I prefer package >>>>> dependencies because they address the excessive type coupling problem in >>>>> object oriented systems, not because they're part of OSGi. Let me argue my >>>>> case. >>>>> >>>>> Module-to-package dependencies (m2p) are preferable over module-to-module >>>>> dependencies (m2m) for many reasons but these are the most important >>>>> reasons: >>>>> >>>>> M2P is leverages the Java type system unlike m2m that must introduce new >>>>> namespaces outside the Java type system. >>>>> M2P can be used to break the transitive dependency chain, m2m suffers of >>>>> excessive coupling >>>>> >>>>> Since the first bullet's benefit should be clear I only argue the more >>>>> complex second bullet. >>>>> >>>>> A module is in many regards like a class. A class encapsulates members, >>>>> depends on other members/classes, and makes a few members accessible outside >>>>> the class. A module has a similar structure but then with types/packages as >>>>> members. >>>>> >>>>> After the initial success of Object Oriented Programming (OO) it was >>>>> quickly learned that reuse did not take place at the expected scale due to >>>>> excessive type coupling. The problem was that a class aggregated many >>>>> dependencies to simplify its implementation but these dependencies were >>>>> unrelated to the contract it implemented. Since class dependencies are >>>>> transitive most applications disappointingly became an almost fully >>>>> connected graph. >>>>> >>>>> Java's great innovation was the interface because it broke both the >>>>> transitivity and aggregation of dependencies. A class could now express its >>>>> dependency (use or implement) on a contract (the interface) and was >>>>> therefore fully type decoupled from the opposite site. >>>>> >>>>> An interface can act as a contract because it names the signature of a >>>>> set of methods so that the compiler can verify the client and the >>>>> implementer. >>>>> >>>>> Since a module has a very similar structure to a class it suffers from >>>>> exactly the same transitive aggregation of dependencies. This is not a >>>>> theory, look at the experiences with Maven >>>>> (http://www.sonatype.com/people/2011/04/how-not-to-download-the-internet/) >>>>> Again, this is not that maven is bad or developers are stupid, it is the >>>>> same underlying force that finally resulted in the Java interface. >>>>> >>>>> The parallel for the class' interface for modules is a named set of >>>>> interfaces. This concept already exists in Java: a package. Looking at >>>>> almost all JSRs it is clear that our industry already uses packages as >>>>> "interfaces" to provider implementations. >>>>> >>>>> Therefore, just like a class should not depend on other implementation >>>>> types, a module should preferably not depend on other modules. A module >>>>> should instead depend on contracts. Since modules will be used to provide >>>>> components from different sources managed with different life cycles the >>>>> excessive type coupling caused by m2m is even more damaging than in c2c. >>>>> Proper use of m2p creates significantly less type coupled systems than m2m, >>>>> the benefits should be obvious. >>>>> >>>>> Since there are use cases for m2m (non-type safe languages for example) I >>>>> do believe that Jigsaw should still support m2m. However, it would be >>>>> greatly beneficial to our industry if we could take advantage of the lessons >>>>> learned with the Java interface and realize how surprisingly important the >>>>> Java package actually is in our eco system. >>>>> >>>>> Kind regards, >>>>> >>>>> Peter Kriens >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On 9 nov. 2011, at 15:04, David M. Lloyd wrote: >>>>> >>>>>> I'll just state now that using packages as a dependency unit is a >>>>>> terrible idea, and not some architectural revelation. That way, Peter's >>>>>> wrath will be largely directed at me. :-) >>>>>> >>>>>> On 11/09/2011 08:02 AM, Peter Kriens wrote: >>>>>>> >>>>>>> I agree that tools are needed but we must be careful to not expect >>>>>>> tools to stopgap an architectural issue. I think it is important to first do >>>>>>> good architectural design leveraging existing tools (e.g. the Java type >>>>>>> system) before you try to add new tools. It is such a pity (but all to >>>>>>> common) that a design allows for classes of errors that would be impossible >>>>>>> with a slightly different design. >>>>>>> >>>>>>> Kind regards, >>>>>>> >>>>>>> Peter Kriens >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 9 nov. 2011, at 14:49, Alan Bateman wrote: >>>>>>> >>>>>>>> On 09/11/2011 13:04, Peter Kriens wrote: >>>>>>>>> >>>>>>>>> The issue is that maven problems are not caused because maven is bad >>>>>>>>> or that pom authors are stupid. The reason is that the module-to-module >>>>>>>>> dependency architecture in maven (and Jigsaw) is error prone ... >>>>>>>> >>>>>>>> This thread started out with someone asking about adding module >>>>>>>> declarations to existing JAR files, and in that context, I agree it can be >>>>>>>> error prone without good tools. I think things should be a lot better when >>>>>>>> modules are compiled. >>>>>>>> >>>>>>>> -Alan. >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> - DML >>>>> >>>> >>> >>> >>> -- >>> - DML >>> > > > -- > - DML From forax at univ-mlv.fr Tue Nov 15 06:42:42 2011 From: forax at univ-mlv.fr (=?UTF-8?B?UsOpbWkgRm9yYXg=?=) Date: Tue, 15 Nov 2011 15:42:42 +0100 Subject: Why package deps won't work (Was: Re: Converting plain JARs to Java modules) In-Reply-To: <38B01C7D-9072-43F7-8398-0D1A3C2F114A@pontarelli.com> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> <67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz> <4EC14A53.7070907@redhat.com> <4EC1F53D.7040309@redhat.com> <38B01C7D-9072-43F7-8398-0D1A3C2F114A@pontarelli.com> Message-ID: <4EC27A62.60902@univ-mlv.fr> On 11/15/2011 03:13 PM, Brian Pontarelli wrote: > I'm not at a computer to reply properly, but a few things about packages that come to mind are: > > - package visibility > - package-info.java > - the Package class > - package versioning > > Yank these out, specifically package-private visibility, and packages are just names. Otherwise, it seems to me that they are part of the type system. > > Sent from my iPad No apart the package visibility that impact the type system, all other items are just here to provide a way to put/get annotations on package thus are not part of the type system. R?mi From nicolas.lalevee at hibnet.org Tue Nov 15 08:00:51 2011 From: nicolas.lalevee at hibnet.org (=?iso-8859-1?Q?Nicolas_Lalev=E9e?=) Date: Tue, 15 Nov 2011 17:00:51 +0100 Subject: Why package deps won't work (Was: Re: Converting plain JARs to Java modules) In-Reply-To: <4EC1F53D.7040309@redhat.com> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> <67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz> <4EC14A53.7070907@redhat.com> <4EC1F53D.7040309@redhat.com> Message-ID: Hi guys, Thank you for this very interesting discussion. I'll just react on a particular point and an example you raised. Le 15 nov. 2011 ? 06:14, David M. Lloyd a ?crit : > Yes, and it is actually fairly common in practice to want two versions of something in the sense of the two versions being wholly different implementations (think apache commons logging versus jcl-over-slf4j for a trivial example which crops up a lot). There are reasons to use one or the other. Multiplicity of versions (in this respect) is something the module system has to handle gracefully and simply. Then how a m2m solution is expected to resolve such case where a module is declaring a dependency on "commons-logging", whereas at deployment time I want "jcl-over-slf4j" instead ? With Maven/Ivy, some exclude rule can be set to build the classpath. Then at runtime the jvm just see one classpath and not the "module" declaration, so everything work then fine. An OSGi runtime being aware of the modules/bundle, you cannot really trick it with exclude rules. If you do, then what's the point of declaring them in the first place ? It is like tricking the jvm and declare that ch.qos.logback.classic.Logger is the substitute of org.apache.commons.logging.Log. Expressing dependencies on APIs, package in the OSGi space, then the modules can correctly bind each other without any overridden declaration. I think that was the main point of Peter's mail, API vs implementation. "Package" and "Bundle" is the way of declaring module level API and Implementation in OSGi. I can imagine than in a m2m solution it would be possible to invent special kind of module which would be about declaring APIs, just like Java Interfaces are to Java Classes. I don't know much about JBoss Modules, but such "Module Interface" doesn't exist there, does it ? How the slf4j libraries are handled in JBoss Modules world ? Nicolas From david.lloyd at redhat.com Tue Nov 15 08:40:58 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 15 Nov 2011 10:40:58 -0600 Subject: Why package deps won't work (Was: Re: Converting plain JARs to Java modules) In-Reply-To: References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> <67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz> <4EC14A53.7070907@redhat.com> <4EC1F53D.7040309@redhat.com> Message-ID: <4EC2961A.6030305@redhat.com> On 11/15/2011 10:00 AM, Nicolas Lalev?e wrote: > Hi guys, > > Thank you for this very interesting discussion. I'll just react on a particular point and an example you raised. > > Le 15 nov. 2011 ? 06:14, David M. Lloyd a ?crit : > >> Yes, and it is actually fairly common in practice to want two versions of something in the sense of the two versions being wholly different implementations (think apache commons logging versus jcl-over-slf4j for a trivial example which crops up a lot). There are reasons to use one or the other. Multiplicity of versions (in this respect) is something the module system has to handle gracefully and simply. > > Then how a m2m solution is expected to resolve such case where a module is declaring a dependency on "commons-logging", whereas at deployment time I want "jcl-over-slf4j" instead ? > > With Maven/Ivy, some exclude rule can be set to build the classpath. Then at runtime the jvm just see one classpath and not the "module" declaration, so everything work then fine. > An OSGi runtime being aware of the modules/bundle, you cannot really trick it with exclude rules. If you do, then what's the point of declaring them in the first place ? It is like tricking the jvm and declare that ch.qos.logback.classic.Logger is the substitute of org.apache.commons.logging.Log. Expressing dependencies on APIs, package in the OSGi space, then the modules can correctly bind each other without any overridden declaration. > > I think that was the main point of Peter's mail, API vs implementation. "Package" and "Bundle" is the way of declaring module level API and Implementation in OSGi. I can imagine than in a m2m solution it would be possible to invent special kind of module which would be about declaring APIs, just like Java Interfaces are to Java Classes. I don't know much about JBoss Modules, but such "Module Interface" doesn't exist there, does it ? How the slf4j libraries are handled in JBoss Modules world ? In our case we never want apache commons logging because of a long history of memory leaks and other such bugs, so we created a module alias called "org.apache.commons.logging" which points to "org.slf4j.jcl-over-slf4j". Thus the user can depend on the commons-logging module and get the API they expect to get. -- - DML From brian at pontarelli.com Tue Nov 15 08:43:16 2011 From: brian at pontarelli.com (Brian Pontarelli) Date: Tue, 15 Nov 2011 09:43:16 -0700 Subject: Why package deps won't work (Was: Re: Converting plain JARs to Java modules) In-Reply-To: <4EC27A62.60902@univ-mlv.fr> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> <67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz> <4EC14A53.7070907@redhat.com> <4EC1F53D.7040309@redhat.com> <38B01C7D-9072-43F7-8398-0D1A3C2F114A@pontarelli.com> <4EC27A62.60902@univ-mlv.fr> Message-ID: <63268B0D-00BE-42BB-BBAB-58F06E3E3430@pontarelli.com> On Nov 15, 2011, at 7:42 AM, R?mi Forax wrote: > On 11/15/2011 03:13 PM, Brian Pontarelli wrote: >> I'm not at a computer to reply properly, but a few things about packages that come to mind are: >> >> - package visibility >> - package-info.java >> - the Package class >> - package versioning >> >> Yank these out, specifically package-private visibility, and packages are just names. Otherwise, it seems to me that they are part of the type system. >> >> Sent from my iPad > > No apart the package visibility that impact the type system, > all other items are just here to provide a way to put/get annotations on package > thus are not part of the type system. You also have sealing of packages, package versioning (in the manifest) and annotations, like you mentioned. So, not 100% part of the type system, but part of the platform for sure. -bp From nicolas.lalevee at hibnet.org Tue Nov 15 08:54:46 2011 From: nicolas.lalevee at hibnet.org (=?iso-8859-1?Q?Nicolas_Lalev=E9e?=) Date: Tue, 15 Nov 2011 17:54:46 +0100 Subject: Why package deps won't work (Was: Re: Converting plain JARs to Java modules) In-Reply-To: <4EC2961A.6030305@redhat.com> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> <67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz> <4EC14A53.7070907@redhat.com> <4EC1F53D.7040309@redhat.com> <4EC2961A.6030305@redhat.com> Message-ID: <74FE3867-6780-4D95-981F-5F91067CAC91@hibnet.org> Le 15 nov. 2011 ? 17:40, David M. Lloyd a ?crit : > On 11/15/2011 10:00 AM, Nicolas Lalev?e wrote: >> Hi guys, >> >> Thank you for this very interesting discussion. I'll just react on a particular point and an example you raised. >> >> Le 15 nov. 2011 ? 06:14, David M. Lloyd a ?crit : >> >>> Yes, and it is actually fairly common in practice to want two versions of something in the sense of the two versions being wholly different implementations (think apache commons logging versus jcl-over-slf4j for a trivial example which crops up a lot). There are reasons to use one or the other. Multiplicity of versions (in this respect) is something the module system has to handle gracefully and simply. >> >> Then how a m2m solution is expected to resolve such case where a module is declaring a dependency on "commons-logging", whereas at deployment time I want "jcl-over-slf4j" instead ? >> >> With Maven/Ivy, some exclude rule can be set to build the classpath. Then at runtime the jvm just see one classpath and not the "module" declaration, so everything work then fine. >> An OSGi runtime being aware of the modules/bundle, you cannot really trick it with exclude rules. If you do, then what's the point of declaring them in the first place ? It is like tricking the jvm and declare that ch.qos.logback.classic.Logger is the substitute of org.apache.commons.logging.Log. Expressing dependencies on APIs, package in the OSGi space, then the modules can correctly bind each other without any overridden declaration. >> >> I think that was the main point of Peter's mail, API vs implementation. "Package" and "Bundle" is the way of declaring module level API and Implementation in OSGi. I can imagine than in a m2m solution it would be possible to invent special kind of module which would be about declaring APIs, just like Java Interfaces are to Java Classes. I don't know much about JBoss Modules, but such "Module Interface" doesn't exist there, does it ? How the slf4j libraries are handled in JBoss Modules world ? > > In our case we never want apache commons logging because of a long history of memory leaks and other such bugs, so we created a module alias called "org.apache.commons.logging" which points to "org.slf4j.jcl-over-slf4j". Thus the user can depend on the commons-logging module and get the API they expect to get. That's not the use case I was talking about. To be more precise: * "Team A" release "Module A" and declare a dependency on "commons-logging" * "Team B" want to use "Module A" but with "jcl-over-slf4j" Are we stuck with third party declared dependency ? Nicolas From david.lloyd at redhat.com Tue Nov 15 09:06:56 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 15 Nov 2011 11:06:56 -0600 Subject: Why package deps won't work (Was: Re: Converting plain JARs to Java modules) In-Reply-To: <74FE3867-6780-4D95-981F-5F91067CAC91@hibnet.org> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> <67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz> <4EC14A53.7070907@redhat.com> <4EC1F53D.7040309@redhat.com> <4EC2961A.6030305@redhat.com> <74FE3867-6780-4D95-981F-5F91067CAC91@hibnet.org> Message-ID: <4EC29C30.6080908@redhat.com> On 11/15/2011 10:54 AM, Nicolas Lalev?e wrote: > > Le 15 nov. 2011 ? 17:40, David M. Lloyd a ?crit : > >> On 11/15/2011 10:00 AM, Nicolas Lalev?e wrote: >>> Hi guys, >>> >>> Thank you for this very interesting discussion. I'll just react on a particular point and an example you raised. >>> >>> Le 15 nov. 2011 ? 06:14, David M. Lloyd a ?crit : >>> >>>> Yes, and it is actually fairly common in practice to want two versions of something in the sense of the two versions being wholly different implementations (think apache commons logging versus jcl-over-slf4j for a trivial example which crops up a lot). There are reasons to use one or the other. Multiplicity of versions (in this respect) is something the module system has to handle gracefully and simply. >>> >>> Then how a m2m solution is expected to resolve such case where a module is declaring a dependency on "commons-logging", whereas at deployment time I want "jcl-over-slf4j" instead ? >>> >>> With Maven/Ivy, some exclude rule can be set to build the classpath. Then at runtime the jvm just see one classpath and not the "module" declaration, so everything work then fine. >>> An OSGi runtime being aware of the modules/bundle, you cannot really trick it with exclude rules. If you do, then what's the point of declaring them in the first place ? It is like tricking the jvm and declare that ch.qos.logback.classic.Logger is the substitute of org.apache.commons.logging.Log. Expressing dependencies on APIs, package in the OSGi space, then the modules can correctly bind each other without any overridden declaration. >>> >>> I think that was the main point of Peter's mail, API vs implementation. "Package" and "Bundle" is the way of declaring module level API and Implementation in OSGi. I can imagine than in a m2m solution it would be possible to invent special kind of module which would be about declaring APIs, just like Java Interfaces are to Java Classes. I don't know much about JBoss Modules, but such "Module Interface" doesn't exist there, does it ? How the slf4j libraries are handled in JBoss Modules world ? >> >> In our case we never want apache commons logging because of a long history of memory leaks and other such bugs, so we created a module alias called "org.apache.commons.logging" which points to "org.slf4j.jcl-over-slf4j". Thus the user can depend on the commons-logging module and get the API they expect to get. > > That's not the use case I was talking about. To be more precise: > * "Team A" release "Module A" and declare a dependency on "commons-logging" > * "Team B" want to use "Module A" but with "jcl-over-slf4j" > > Are we stuck with third party declared dependency ? I suppose it depends on who administers the module repository. Note that the same question applies to m2m versus m2p. I think that the owner of the module repository gets to make that choice as the administrator for the system. To take the JCL case in point: the administrator is the one who determines which implementation is appropriate for the platform. You never really would want to support both in this case. This is as opposed to another case: Say one user is using Hibernate 3 and one is using Hibernate 4. We solve this type of versioning issue in JBoss Modules with a version "slot". Put simply, this is a disambiguating identifier used to allow two same-named modules to exist in one repository. The "latest version" progression of each "slot" advances independently of the other. As with the equivalent situation on an operating system distribution, the long term goal is always to phase out old versions and introduce new versions over a period of time which is managed by a social process. This type of setup has been proven to work in a number of operating system distributions to date, and seems to work well for our case as well. Like the operating system case, I believe that run-time dependencies are the domain of the module distributor. The dependencies are often slightly varied depending on who is installing the library, where, and why. I think it's better to maintain a simple system which is flexible enough to be customized by distribution than to try to come up with a centralized or globalized dependency scheme. Hard realities like licensing, implementation selection, and defective software will make a pure globalized scheme unworkable. Take the simple division of philosophy between Debian and Ubuntu or Fedora, where licensing ideology makes a substantial difference in what is distributed and how. Consider also the case where a proprietary shop wants to maintain its own distribution, but certain libraries are considered disallowed or are reimplemented with security or other concerns in mind. The original build dependencies of a module might not be directly translatable into run-time dependencies in these cases, yet to ignore these cases is to put an undue burden on the user. -- - DML From eric at tibco.com Tue Nov 15 14:18:29 2011 From: eric at tibco.com (Eric Johnson) Date: Tue, 15 Nov 2011 23:18:29 +0100 Subject: Why package deps won't work (Was: Re: Converting plain JARs to Java modules) In-Reply-To: <4EC29C30.6080908@redhat.com> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> <67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz> <4EC14A53.7070907@redhat.com> <4EC1F53D.7040309@redhat.com> <4EC2961A.6030305@redhat.com> <74FE3867-6780-4D95-981F-5F91067CAC91@hibnet.org> <4EC29C30.6080908@redhat.com> Message-ID: <4EC2E535.6050009@tibco.com> On 11/15/11 6:06 PM, David M. Lloyd wrote: > As with the equivalent situation on an operating system distribution, > the long term goal is always to phase out old versions and introduce > new versions over a period of time which is managed by a social > process. This type of setup has been proven to work in a number of > operating system distributions to date, and seems to work well for our > case as well. The logical conclusion from this, though is that with future JDKs, each Linux distributor will have their own modularization of the JDK, so as to fit better with their package system, or their version definition, or the legacy accommodations that they've made. Wouldn't we all be better off if the "official" modularization of the JDK could mostly be used as-is, and only tweaked around the edges by distributions? The situation we ran into as an actual use case with OSGi, and the point I think Peter was making, is that if you use M2P dependencies, then you, as the modularization distributor, can make changes to the core modules without that fanning out to the entire dependency tree, because now you can move package X to module Y from original module O, and no dependent modules need to change. If you just use M2M dependencies, then when you try to remodularize for whatever reason, and do the same package move, then you have unpleasant options. Either you make a bad compromise for compatibility (O now depends on Y), or you have to update all the dependent modules. Yuck, or yuck. > > Like the operating system case, I believe that run-time dependencies > are the domain of the module distributor. The dependencies are often > slightly varied depending on who is installing the library, where, and > why. I think it's better to maintain a simple system which is > flexible enough to be customized by distribution than to try to come > up with a centralized or globalized dependency scheme. Hard realities > like licensing, implementation selection, and defective software will > make a pure globalized scheme unworkable. Take the simple division of > philosophy between Debian and Ubuntu or Fedora, where licensing > ideology makes a substantial difference in what is distributed and how. Hmmm. "runtime dependencies are the domain of the module distributor". Maybe we're just arguing terminology? I say runtime dependencies are sometimes the domain of the *deployer*, and to me, that's not the same as a module distributor (which I interpret as my Linux provider, or enterprise software provider). Example: If my application requires an implementation of the javax.xml.parsers API, as an application developer, there are at least three possibilities: (a) determined by the OS distribution, (b) determined by other components of the system that feel a need to constrain this, or (c) specifically determined by my launch configuration. I might, as an example of the latter, at least from my IDE, wish to run test suites using a variety of different implementations of the API. If I start recording M2M dependencies, I might be over-constrained, and complete stuck if some existing module happens to include both javax.xml.parsers, as well as an implementation of said APIs (as they used to do!). -Eric. From david.lloyd at redhat.com Tue Nov 15 14:56:07 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 15 Nov 2011 16:56:07 -0600 Subject: Why package deps won't work (Was: Re: Converting plain JARs to Java modules) In-Reply-To: <4EC2E535.6050009@tibco.com> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> <67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz> <4EC14A53.7070907@redhat.com> <4EC1F53D.7040309@redhat.com> <4EC2961A.6030305@redhat.com> <74FE3867-6780-4D95-981F-5F91067CAC91@hibnet.org> <4EC29C30.6080908@redhat.com> <4EC2E535.6050009@tibco.com> Message-ID: <4EC2EE07.3000600@redhat.com> On 11/15/2011 04:18 PM, Eric Johnson wrote: > On 11/15/11 6:06 PM, David M. Lloyd wrote: >> As with the equivalent situation on an operating system distribution, >> the long term goal is always to phase out old versions and introduce >> new versions over a period of time which is managed by a social >> process. This type of setup has been proven to work in a number of >> operating system distributions to date, and seems to work well for our >> case as well. > > The logical conclusion from this, though is that with future JDKs, each > Linux distributor will have their own modularization of the JDK, so as > to fit better with their package system, or their version definition, or > the legacy accommodations that they've made. Wouldn't we all be better > off if the "official" modularization of the JDK could mostly be used > as-is, and only tweaked around the edges by distributions? Sure, theoretically. This hasn't happened for any other language though, and I don't think it's realistic to expect that Java will be any different. For most languages there are OS-level packages for each module that the OS distributes but there is by no means any guarantee that a given OS would ship an entire module environment for a language. IIRC though one thing you could do with Perl in, say, Fedora is use CPAN locally to add your own collection of modules over the operating system module library. This type of overlay can be an effective solution. > The situation we ran into as an actual use case with OSGi, and the point > I think Peter was making, is that if you use M2P dependencies, then you, > as the modularization distributor, can make changes to the core modules > without that fanning out to the entire dependency tree, because now you > can move package X to module Y from original module O, and no dependent > modules need to change. > > If you just use M2M dependencies, then when you try to remodularize for > whatever reason, and do the same package move, then you have unpleasant > options. Either you make a bad compromise for compatibility (O now > depends on Y), or you have to update all the dependent modules. Yuck, or > yuck. If you're, say, splitting a module or reorganizing and you want to avoid updating dependents, you always have the option of adding compatibility aggregate modules and so on. But this type of refactor isn't too common in a static module repository because libraries are generally built for long-term backwards compatibility which is tough to maintain when you're moving packages around. So while m2p might solve moving packages more elegantly (assuming single versions of all packages of course), it's not really a common case in an SE situation and not a requirement (it's a nice-to-have at best). >> Like the operating system case, I believe that run-time dependencies >> are the domain of the module distributor. The dependencies are often >> slightly varied depending on who is installing the library, where, and >> why. I think it's better to maintain a simple system which is flexible >> enough to be customized by distribution than to try to come up with a >> centralized or globalized dependency scheme. Hard realities like >> licensing, implementation selection, and defective software will make >> a pure globalized scheme unworkable. Take the simple division of >> philosophy between Debian and Ubuntu or Fedora, where licensing >> ideology makes a substantial difference in what is distributed and how. > > Hmmm. "runtime dependencies are the domain of the module distributor". > Maybe we're just arguing terminology? > > I say runtime dependencies are sometimes the domain of the *deployer*, > and to me, that's not the same as a module distributor (which I > interpret as my Linux provider, or enterprise software provider). Yeah definitely. There is an overlap for sure. I think that in the OS distribution case for example there will always be a case where the user wants to develop against either a wholly local repository, a hybrid of locally installed and globally installed modules, or purely against the global install. Also there is a case where a user wants to develop against a hybrid of their own local install which is overlaying a vendor distribution of modules (like an application server or a large proprietary application). Overall I think we definitely need the ability to support hybrid module root configurations. And you shouldn't have to be a wizard to be able to assemble your own module distribution. > Example: If my application requires an implementation of the > javax.xml.parsers API, as an application developer, there are at least > three possibilities: (a) determined by the OS distribution, (b) > determined by other components of the system that feel a need to > constrain this, or (c) specifically determined by my launch > configuration. I might, as an example of the latter, at least from my > IDE, wish to run test suites using a variety of different > implementations of the API. > > If I start recording M2M dependencies, I might be over-constrained, and > complete stuck if some existing module happens to include both > javax.xml.parsers, as well as an implementation of said APIs (as they > used to do!). If you use M2P dependencies for this then you may not even be able to launch without constructing or managing package indexes or creating a number of extra package files, or running tooling to rebuild these. One advantage to the M2M system is that you can overlay module repositories fairly easily by simply loading modules from each one in sequence. -- - DML From peter.kriens at aqute.biz Wed Nov 16 09:06:01 2011 From: peter.kriens at aqute.biz (Peter Kriens) Date: Wed, 16 Nov 2011 18:06:01 +0100 Subject: Why package deps work (Was: Re: Converting plain JARs to Java modules) In-Reply-To: <4EC1F53D.7040309@redhat.com> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> <67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz> <4EC14A53.7070907@redhat.com> <4EC1F53D.7040309@redhat.com> Message-ID: <328B38A3-EB53-4AD2-970B-CE84F5C26F96@aQute.biz> David, Thanks for the elaboration. I won't go into all the details because I do not want to make the same points again and Neil and others seemed to have made many points more eloquently than I can. However, I notice some misconceptions: > In *any* model you cannot avoid transitive dependencies. My point is actually that we actually can ... This actually what interfaces do for classes and (specification) packages do for modules. They postpone the instance binding to runtime/deployment time but for this extra complexity we get the benefit of disconnecting the transitive dependency graph. For example, my code depends on javax.persistence package and not Eclipse Link; this way I do not inherit Eclipse Links transitive dependencies. The log hell described in some other mails clearly shows the advantage of this model since you can defer what logging implementation to use to deployment time. > Maintenance of dependencies with m2m I understand that AS7 keeps the wiring separate and this is maintained by hand; Pom dependencies are maintained by hand and it Jigsaw will also require manual handling of dependencies because they require a new namespace unconnected to Java sources. This clearly lacks scalability since humans are really bad in maintaining this information for more than a handful of modules. In contrast, packages are part of the Java language and the dependencies are therefore already present in the class files; they therefore do not require the manual maintenance necessary for m2m. Kind regards, Peter Kriens On 15 nov. 2011, at 06:14, David M. Lloyd wrote: > On 11/14/2011 08:33 PM, Neil Bartlett wrote: >> Hi David, >> >> More than happy to talk about reality, though Peter already was doing >> so. Nothing he said was theoretical, it came directly from his >> experiences and the experiences of developers who have been using OSGi >> for around 12 years. > >> Before digging into the points in your message, I need to address the >> subject line "Why package deps won't work". Package dependencies will >> work and they do work, as is proven every day by the large number of >> applications running on OSGi. To claim otherwise would be wilfully >> ignorant, so I will assume you are being hyperbolic and really meant >> to assert that module deps are just better than package deps. >> Therefore I intend to argue on that basis and claim that package deps >> are better than module deps; note that I don't claim that module >> dependencies "won't work" because JBoss Modules is a proof point that >> they do. > > There was an implied "... for Java SE 8 modules" after "won't work" (this is why I said "won't" rather than "don't") which apparently wasn't implied enough. > > Of course people use OSGi, or we wouldn't be arguing, but that alone isn't enough to make OSGi's dependency system preferable for the SE case (after all, far more people use plain old class paths). I believe that JBoss Modules is a very accurate representation of how an SE module system could function, right down to executing JARs on the command line with dependencies and modularizing certain JDK APIs. This is not a proof point, but strong evidence that it is an effective solution for the actual problem on the table, and that similar architectures are likely to succeed for the same problem domain. As to why package deps are not an example of an effective solution for this problem, I intend to illustrate in greater detail. > > Further responses and elaboration below. > >> On Mon, Nov 14, 2011 at 5:05 PM, David M. Lloyd wrote: >>> The explanation is quite simple, really - each point can be pretty much >>> wiped out by a heavy dose of reality. >>> >>> 1. "M2P is leverages the Java type system unlike m2m that must introduce new >>> namespaces outside the Java type system." - this is just fluffy >>> buzzwordology. Considering packages part of the Java type system is a >>> pretty liberal interpretation of the term "type system" as they're basically >>> just arbitrary name spaces. That said, there's nothing particularly "new" >>> about namespacing modules. JARs have names. Projects have names. Maven >>> uses its own namespaces for artifacts. Even the primitive JDK extension >>> mechanism uses symbolic names. >> >> From Wikipedia (http://en.wikipedia.org/wiki/Type_system): "A type >> system associates a type with each computed value ... the aim is to >> prevent operations expecting a certain kind of value being used with >> values for which that operation does not make sense." >> >> The following will not compile: >> >> import java.awt.List; >> // ... >> List list = new ArrayList(); >> list.iterator(); // etc >> >> whereas the following will: >> >> import java.util.List; >> // ... >> List list = new ArrayList(); >> list.iterator(); // etc >> >> Package names are part of the type system because using an incorrect >> package name in Java source can result in type errors during >> compilation, and because the type and available operations associated >> with each value varies with the package name used. >> >> In contrast, it is impossible to obtain a type error at compilation by >> incorrectly naming a JAR, project, Maven artefact etc, because none of >> those things are ever referenced in Java source code. > > I stand by my statement. A package is part of the name of the type; it is not a type, nor a meta-type, nor anything like a type in and of itself. This makes them part of the language, sure, but not part of the type system, in my opinion. Thus, "a stretch". I don't feel any particular need to have the world agree with me on this though, other than as I state below... > >>> Even a simple convention of the parent-most package name for the name of a >>> module is very simple to grasp and is in fact the solution we've used quite >>> effectively thus far. Thus implying that module names are some new alien >>> concept is really not valid. >> >> The claim is not that they are a new alien concept. The claim is that >> they are not part of the Java language and type system, and as a >> result of this (along with other problems) are less useful for >> depending upon than packages. > > I don't see how not being part of the Java language makes module identifiers "less useful". I think that this argument artificially elevates the significance of the relationship between packages and the actual type system, and then uses that elevation as a proof that a system based on other namespace principles is deficient, without offering any real examples which actually relate to the type system. > >>> 2. "M2P can be used to break the transitive dependency chain, m2m suffers of >>> excessive coupling." - How about some facts to back this up? I've found >>> "m2m" coupling to be just right. In JBoss Modules, we do not export >>> transitive dependencies by default. This results in a very simple and clean >>> dependency graph between modules. Using package dependencies results in a >>> *far more complex* dependency graph, because you need edges for every >>> package even though *most of the time you want the whole module anyway*. >>> Again, real world here. >> >> Peter is definitely also talking about the real world and so am I. In >> an m2m dependency model you cannot avoid transitive dependencies, >> whether you expose them to consumers or not. > > In *any* model you cannot avoid transitive dependencies. If module A exports module B as part of its API, then you get A and B, and that's life. If you only want part of B, then you filter it down. But with most existing libraries out there (and they easily number in the thousands), they're intended to be used whole, so this doesn't really happen too much in practice. > >> An importer of a module must be assumed to depend on the whole functionality of that module >> whether or not that is actually the case, and therefore all of the >> transitive dependencies must be present at runtime. In an m2p world we >> have the opportunity to split modules and break apart dependency >> graphs, because the unit of coupling is more granular. > > You could break things down by actual classes if granularity is the goal. But it isn't; it is so much less important than, say, performance, conceptual simplicity, usability, efficiency, memory footprint, etc. The fact is that the vast majority of modules you'll find published at, say Maven central, are designed to be used completely, not partially, and can and are used as such without issue. > > People just don't usually design modules to be imported by package, except perhaps as an afterthought concession to OSGi, and they don't use them that way at runtime either. It is so seldom a real problem in the SE world (or EE for that matter) once you step away from flat class paths. The simple fact is that most users think of their dependencies in the same terms that their IDEs and their build scripts do: by module. > > This is truly a case of creating a lot of complexity and extra work in the common case to avoid problems which are not a common case. I think it is much better to optimize for the common case, where people know exactly what dependencies they want (in terms of artifacts), and they just want to declare them and be done without (a) hunting down package lists or (b) running extra tooling. > >>> If we're just going to throw dogma around, I'll put it the other way: m2p is >>> a design error by the OSGi spec designers which has since been embraced as a >>> religion. >> >> Pejorative accusations about "dogma" or "religion" have no place in a >> technical discussion. They are also untrue. All of the practising OSGi >> developers I know have arrived at their support for package >> dependencies as a result of real world experience, not because of some >> willingness to bow down to the Almighty CPEG. I don't know of any >> practising OSGi developer who has used both Require-Bundle (m2m) and >> Import-Package (m2p) and actually prefers the former. > > Sure, but Require-Bundle is *not* the same thing as using module-to-module dependencies in a non-OSGi system. All OSGi life revolves around the resolver. In the SE world, you cannot have a resolver without introducing a lot of machinery which will definitely negatively impact boot performance in the case that the resolver runs at module load time, or add the complexity of centralized package index management in the case that the resolver would run at install time (as in, install into a module library/repository, not install in the OSGi sense). > > If in a typical installation a user has to do more than simply delete a file to remove a module, or drop a JAR into a directory to add a module, I believe we've run quite far off-track usability-wise. And if a user has to sit through a lengthy resolution process at runtime (even if only the first time) then we're off-track performance-wise, especially if such resolution must expand to include modules in the module repository which are not even loaded. > >> I do know several who started as strong supporters of Require-Bundle >> and switched to being supporters of Import-Package, not because of >> Peter's wrathful pontificating but because they encountered the >> specific problems that he described and found that they were fixed by >> using Import-Package, and indeed that everything worked so much more >> cleanly that way. I'm in this camp myself, and so was Jeff McAffer >> before he went over to the Dark Side (Microsoft). > > There are other, simpler ways to fix package conflict issues when they arise. Like simply excluding the offending package from the import. > > And as I've said before, I do not believe that Require-Bundle in the context of OSGi bundles is comparable to loading modules by name in the context of a module system such as Jigsaw or JBoss Modules and the types of modules that could be loaded thereby (the aforementioned thousands of existing modules). > >>> It offers no significant benefit, other than a couple of edge >>> cases which are frankly just as well handled by m2m simply by adding package >>> filters. Which, by the way, I haven't seen the need to do yet in our 200+ >>> module environment, but which we do have the capability to do. >> >> Substitution, module refactoring and transitive decoupling are hardly >> edge cases. > > Substitution isn't solely addressed by package dependencies. Nor is module refactoring. And additional transitive decoupling isn't something people actually want - as I said, the common case is that users want to be able to use any class from a module once they import that module. In the real world, it is very rare to only want to import part of a module! If you need this routinely, you've done something very unorthodox. Thus I stand by my statement. > >> However I can see how these issues might not yet have come >> to the fore in a module system designed for a single product with a >> small number of modules, and where that product has not yet been >> through the mill of multiple version evolutions. > > Cute. I assure you that we have been around through a long, long history of a huge number of Java users doing everything under the sun to their class paths and class loading structures. We also have experience with operating system distribution and managing module distribution across a wide range of other programming languages, not just Java. Our module system has already stood up to reasonably large deployments (200+ modules) with excellent memory footprint and performance numbers and we've been processing usability feedback, including feedback from those with OSGi experience, which has been mostly positive. > > Furthermore when I was at JavaOne giving booth talks about JBoss Modules and AS7, I found that users expressed quite a bit of frustration at the OSGi model and were quite receptive to a more orthodox m2m system, for what that's worth. > >>> M2P is a solution just itching for a problem. >> >> This is also not a very useful statement, I assure you that the >> problem came before the solution. Better to show why you think the >> problem is invalid or should have been solved differently. > > There is no way you can convince me that this solution was invented with general modularity in mind. Back when OSGi was first formed, it was designed for the embedded market. The world of Java today is completely different. People just don't bundle their libraries by package, as was possibly expected in the early days of Java. If they did we wouldn't be having this discussion because packages and modules would already be one and the same. > > OSGi evolved to where it is today. It was not designed from the ground up with actual requirements which pertained to the problem at hand which is modularity of the SE platform and applications which run on it. So yeah I'm going to say, solution before problem. Taking a solution and trying to apply it retroactively to a different problem like this has never failed to bite me in the ass, personally. But I can't speak to everyone's experience. > >>> But you're going to have a >>> tough time convincing me that users *want* to have to use special tooling >>> because they want to depend on a module which has too many packages to list >>> out by hand. And before you cry "wildcards", be sure to consider that some >>> modules use package names which are subordinate to other modules' packages, >>> which is a perfectly normal and allowable scenario. Using wildcards for >>> package matching could cause amazing levels of havoc. >> >> OSGi never uses wildcards at runtime, and tools such as bnd do not >> need wildcards in order to express package-level dependencies. They >> extract the set of packages that were actually used by the code, all >> of which are available in the class files. This is possible because >> packages are part of the type system (see first point above). >> >> So it's not that I don't want to list dependencies by hand, rather I >> only want to do it once. I am already forced to do it in the import >> statements of my Java sources. If I had to repeat that dependency >> information -- whether in m2p or m2m form -- then I would run the risk >> of it getting out of step with the real dependencies. > > One of the critical logical gaps here which m2p ignores at a fundamental level is that *packages are not unique*. In the real world, packages are repeated across more than one module all the time, and not just due to versioning. There is *no way* you could take every JAR at Maven Central and plug it in to an m2p system. You'd have to break apart and repackage every single one, at least, and at worst you'd have to rewrite an awful lot of them under different package names to account for this restriction. You'd need to enforce package uniqueness across your whole library, for all time. I don't think this is a sane option. You're adding significant overhead to management, installation, and execution, for what? So you can have the privilege of needing a special tool to extract your dependencies for you? > >>> Using package dependencies means you either must have a master package index >>> for linking, or you need a resolver which has to have analyzed every module >>> you ever plan to load. Otherwise, O(1) loading of modules is impossible, >>> which is absolutely 100% a deal-breaker for JDK modules which must be >>> incrementally installable. And it forbids having packages with the same >>> name in more than one JAR without bringing run-time versioning into the >>> fold, which is a terrible, terrible can of worms. >> >> Could you please explain why O(1) is the only acceptable complexity >> for installing modules. > > Because this is Java SE we're talking about. You're going to have a potentially huge number of modules installed in your system. Even if you get around the versioning issues somehow and fully externalize the index of all packages, you still have to load or traverse the centralized index for every module load. > > I say that not only is ?(1) the only acceptable complexity for installing modules but also loading them at run time. Linking modules at run time should be no worse than O(n) complexity for the number of dependencies the module has, including transitive dependencies. Loading classes and resources from modules should be ?(k) where k is the number of modules which contain a package or directory which matches the class or resource being loaded (almost always one, in my experience). > > Loading a module should normally be limited to a one-time O(1) disk access without searching or loading any other files or modules than the module artifact itself (and possibly its descriptor if they are held externally by a particular module repository implementation, which is useful but not necessary). Linking a module should be limited in disk access to loading only the modules it imports via direct dependency, or by the relatively rare partial or full re-export of a transitive dependency by a direct dependency. > > In particular, the expectation of accessing a central index has some potentially serious implications, facing either possible file lock contention by multiple threads or memory overhead of loading in the complete index in advance. It should be expected that modules are loaded concurrently, and such concurrency should not be hindered any more than necessary. Other possible implementations (such as file- or directory-per-package) have their own drawbacks as well or violate what I consider to be core requirements. > > If there are implementations of package-based resolution which don't involve either a centralized index of one or many files or a resolver which must rove all installed modules in advance, I'd like to hear about it. > >> OSGi does indeed support incremental install >> and while I accept it is probably not O(1) for each module, it would >> likely be no more than O(N), though I haven't done the maths yet to >> prove this. Bear in mind that in practice, for small N, the constant >> factors can result in O(1) being more expensive than O(N). I have seen >> OSGi used with many thousands of modules, so unless you have some data >> and a use-case showings package-level resolution as unacceptably slow, >> your concern just sounds like premature optimisation. > > Identifying algorithmic complexity during design phase is *never* premature optimization AFAICT. If you don't understand the complexity of the algorithms you're about to implement, and how they are expected to be applied to the problem at hand, then you're not ready for implementation yet. > > However, postulating that O(n) is okay because can sometimes be faster than O(1) without measuring it for the specific problem in question is definitely premature de-optimization. :-) > >> There are two reasons to have a packages with the same name in more >> than one JAR. The first is a situation called split packages, and it >> is highly undesirable because it causes the runtime model of the >> package to diverge from the compile-time model, and therefore things >> like package-private types and members stop working correctly. For >> this reason, OSGi's m2p imports support depending only upon a single >> exporter of a particular package, i.e. we do not aggregate all exports >> of that package. >> >> Unfortunately split packages are sometimes unavoidable in legacy code >> that cannot be refactored, e.g. the JDK. To support such scenarios >> OSGi has Require-Bundle, i.e. m2m. This does not negate the problems >> associated with m2m, it is simply a trade-off that we face with poorly >> factored legacy code. >> >> The second reason for multiple packages with the same name is when you >> explicitly want to install multiple versions of a library/API and have >> them all available within the same runtime. I wouldn't call this a >> "can of worms" exactly because it can be done without too much >> trouble, though for the sake of a simple life I personally avoid this >> situation unless it's necessary. > > Yes, and it is actually fairly common in practice to want two versions of something in the sense of the two versions being wholly different implementations (think apache commons logging versus jcl-over-slf4j for a trivial example which crops up a lot). There are reasons to use one or the other. Multiplicity of versions (in this respect) is something the module system has to handle gracefully and simply. > >>> Finally it should be perfectly clear to anyone who has read the original >>> requirements document that nothing in this module system should prevent OSGi >>> from functioning as it is, so there is absolutely no reason to assume that >>> any OSGi implementation is so threatened - especially if m2p linking is as >>> superior as has been expressed. Our module system (which is conceptually >>> similar to Jigsaw in many regards) in fact does support our OSGi >>> implementation quite effectively without itself implementing OSGi's >>> package-to-package resolution (which like I said throws O(1) out the >>> window). >> >> I agree that Jigsaw's existence doesn't threaten OSGi's, so long as >> Java 8 doesn't actually break OSGi (and if it did so, it would >> likewise break many other applications and could not be considered >> backwards compatible with Java 7). The two can interoperate through >> m2m-type dependencies. Tim Ellison started Project Penrose for the >> purpose of investigating, testing and deepening this collaboration. >> >> Neverthless, the point that I believe Glyn was making is the >> following. We accept that m2m dependencies are probably required for >> the JDK, which implies a module system like Jigsaw or >> OSGi/Require-Bundle rather than OSGi/Import-Package. However is it >> intended to be used for application modularisation as well? This is of >> course a question for the Jigsaw team rather than you, David. > > The question of whether Java SE 8 modules are intended to be used for application modularization is a question for the EG, not the Jigsaw team. The job of the Jigsaw team is really to implement a prototype which meets the requirements set forth by the EG, which may become the reference implementation at a future point. > > The question of whether the greater Java community will embrace the SE module system for applications is to be answered by the community only. I personally believe that to develop a module system which is not intended to be usable by the average standalone application is foolhardy, a great waste of effort, and is doomed to mediocrity, given the success we have had with such a system. > > However until there is an EG, these are all just as much questions for me as for anybody, and I can and will answer to the best of my knowledge and belief. > >> As a result of experience in developing and evolving large real-world >> applications using a module system that supports BOTH m2m and m2p >> dependencies, I believe it would be very unfortunate if a module >> system that supports ONLY m2m were to become widely used in the >> application space... not because OSGi can't handle the competition, >> but because those applications will be fragile and hard to evolve. > > I don't believe this to be the case. I think that the status quo doesn't result in particularly fragile applications, and such can be easy to evolve or hard depending on the quality of the application components and frameworks involved. I think that m2m dependencies enhance the status quo such that applications are somewhat less fragile (owing chiefly to the simple measure of preventing transitive dependencies from being exported by default), and quite easy to evolve as well (even large and complex frameworks rarely have more than a few dependencies, unless they themselves are overly fragmented (e.g. CXF as a nightmare example which yet still works fine under an m2m system)). > >> My question for you David is as follows. I understand that you prefer >> module dependencies, but do you believe that package dependencies have >> no value whatsoever and therefore should not be available to >> application developers in the Java 8 module system? If so, why did Red >> Hat create an OSGi implementation? > > I personally believe that they have some value, but that value is limited to interoperability with OSGi. I do not believe that it is a preferable model for most users or standalone applications, nor for larger applications such as our application server. I think that the requirement for extra tooling and the increased run-time complexity which is exposed to users nullifies the benefits. If someone really wants this variety of modularity, they should simply use OSGi. > > Red Hat created an OSGi implementation for many reasons, but the only significant one to me is that there are people who want to use their OSGi applications with our application server. I don't believe any other reason is even necessary. It's the same reason we strive to support any spec, from EJB 1.0 to Java EE 6 and beyond. Someone wants the functionality, so we deliver it to the best of our ability. > >> Kind regards >> Neil >> >>> >>> On 11/14/2011 01:49 AM, Glyn Normington wrote: >>>> >>>> I look forward to David's elaboration of why he thinks "using packages as >>>> a dependency unit is a terrible idea" to balance Peter's clear explanation >>>> of the benefits of m2p. >>>> >>>> Meanwhile, it's worth noting that, according to the requirements document, >>>> Jigsaw is aimed at platform modularisation and the platform being >>>> modularised has some non-optimal division of types across packages (see the >>>> package subsets requirement) which favour m2m dependencies. (Note that >>>> Apache Harmony was developed with modularity in mind and was able to exploit >>>> m2p, so platform modularisation per se needn't be limited to m2m.) >>>> >>>> So if Jigsaw excludes m2p, it will then be applicable to certain kinds of >>>> legacy code modularisation and less applicable to new module development and >>>> modularisation of existing code whose division into packages suits m2p. IIRC >>>> this was the original positioning of Jigsaw: for use primarily within the >>>> OpenJDK codebase and only exposed for application use because it was too >>>> inconvenient to hide it. >>>> >>>> Regards, >>>> Glyn >>>> >>>> On 12 Nov 2011, at 11:59, Peter Kriens wrote: >>>> >>>>> Neither my wrath, nor the fact that I rarely if ever get angry is >>>>> relevant in this discussion ... This is a technical argument that are >>>>> solvable by technical people that share the same goals. I prefer package >>>>> dependencies because they address the excessive type coupling problem in >>>>> object oriented systems, not because they're part of OSGi. Let me argue my >>>>> case. >>>>> >>>>> Module-to-package dependencies (m2p) are preferable over module-to-module >>>>> dependencies (m2m) for many reasons but these are the most important >>>>> reasons: >>>>> >>>>> M2P is leverages the Java type system unlike m2m that must introduce new >>>>> namespaces outside the Java type system. >>>>> M2P can be used to break the transitive dependency chain, m2m suffers of >>>>> excessive coupling >>>>> >>>>> Since the first bullet's benefit should be clear I only argue the more >>>>> complex second bullet. >>>>> >>>>> A module is in many regards like a class. A class encapsulates members, >>>>> depends on other members/classes, and makes a few members accessible outside >>>>> the class. A module has a similar structure but then with types/packages as >>>>> members. >>>>> >>>>> After the initial success of Object Oriented Programming (OO) it was >>>>> quickly learned that reuse did not take place at the expected scale due to >>>>> excessive type coupling. The problem was that a class aggregated many >>>>> dependencies to simplify its implementation but these dependencies were >>>>> unrelated to the contract it implemented. Since class dependencies are >>>>> transitive most applications disappointingly became an almost fully >>>>> connected graph. >>>>> >>>>> Java's great innovation was the interface because it broke both the >>>>> transitivity and aggregation of dependencies. A class could now express its >>>>> dependency (use or implement) on a contract (the interface) and was >>>>> therefore fully type decoupled from the opposite site. >>>>> >>>>> An interface can act as a contract because it names the signature of a >>>>> set of methods so that the compiler can verify the client and the >>>>> implementer. >>>>> >>>>> Since a module has a very similar structure to a class it suffers from >>>>> exactly the same transitive aggregation of dependencies. This is not a >>>>> theory, look at the experiences with Maven >>>>> (http://www.sonatype.com/people/2011/04/how-not-to-download-the-internet/) >>>>> Again, this is not that maven is bad or developers are stupid, it is the >>>>> same underlying force that finally resulted in the Java interface. >>>>> >>>>> The parallel for the class' interface for modules is a named set of >>>>> interfaces. This concept already exists in Java: a package. Looking at >>>>> almost all JSRs it is clear that our industry already uses packages as >>>>> "interfaces" to provider implementations. >>>>> >>>>> Therefore, just like a class should not depend on other implementation >>>>> types, a module should preferably not depend on other modules. A module >>>>> should instead depend on contracts. Since modules will be used to provide >>>>> components from different sources managed with different life cycles the >>>>> excessive type coupling caused by m2m is even more damaging than in c2c. >>>>> Proper use of m2p creates significantly less type coupled systems than m2m, >>>>> the benefits should be obvious. >>>>> >>>>> Since there are use cases for m2m (non-type safe languages for example) I >>>>> do believe that Jigsaw should still support m2m. However, it would be >>>>> greatly beneficial to our industry if we could take advantage of the lessons >>>>> learned with the Java interface and realize how surprisingly important the >>>>> Java package actually is in our eco system. >>>>> >>>>> Kind regards, >>>>> >>>>> Peter Kriens >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On 9 nov. 2011, at 15:04, David M. Lloyd wrote: >>>>> >>>>>> I'll just state now that using packages as a dependency unit is a >>>>>> terrible idea, and not some architectural revelation. That way, Peter's >>>>>> wrath will be largely directed at me. :-) >>>>>> >>>>>> On 11/09/2011 08:02 AM, Peter Kriens wrote: >>>>>>> >>>>>>> I agree that tools are needed but we must be careful to not expect >>>>>>> tools to stopgap an architectural issue. I think it is important to first do >>>>>>> good architectural design leveraging existing tools (e.g. the Java type >>>>>>> system) before you try to add new tools. It is such a pity (but all to >>>>>>> common) that a design allows for classes of errors that would be impossible >>>>>>> with a slightly different design. >>>>>>> >>>>>>> Kind regards, >>>>>>> >>>>>>> Peter Kriens >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 9 nov. 2011, at 14:49, Alan Bateman wrote: >>>>>>> >>>>>>>> On 09/11/2011 13:04, Peter Kriens wrote: >>>>>>>>> >>>>>>>>> The issue is that maven problems are not caused because maven is bad >>>>>>>>> or that pom authors are stupid. The reason is that the module-to-module >>>>>>>>> dependency architecture in maven (and Jigsaw) is error prone ... >>>>>>>> >>>>>>>> This thread started out with someone asking about adding module >>>>>>>> declarations to existing JAR files, and in that context, I agree it can be >>>>>>>> error prone without good tools. I think things should be a lot better when >>>>>>>> modules are compiled. >>>>>>>> >>>>>>>> -Alan. >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> - DML >>>>> >>>> >>> >>> >>> -- >>> - DML >>> > > > -- > - DML From brian at pontarelli.com Wed Nov 16 10:23:57 2011 From: brian at pontarelli.com (Brian Pontarelli) Date: Wed, 16 Nov 2011 11:23:57 -0700 Subject: Why package deps work (Was: Re: Converting plain JARs to Java modules) In-Reply-To: <328B38A3-EB53-4AD2-970B-CE84F5C26F96@aQute.biz> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> <67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz> <4EC14A53.7070907@redhat.com> <4EC1F53D.7040309@redhat.com> <328B38A3-EB53-4AD2-970B-CE84F5C26F96@aQute.biz> Message-ID: <9B28CE0A-85DC-4F53-8A88-5830759DA4BC@pontarelli.com> On Nov 16, 2011, at 10:06 AM, Peter Kriens wrote: > David, > Thanks for the elaboration. I won't go into all the details because I do not want to make the same points again and Neil and others seemed to have made many points more eloquently than I can. However, I notice some misconceptions: > >> In *any* model you cannot avoid transitive dependencies. > > My point is actually that we actually can ... This actually what interfaces do for classes and (specification) packages do for modules. They postpone the instance binding to runtime/deployment time but for this extra complexity we get the benefit of disconnecting the transitive dependency graph. > > For example, my code depends on javax.persistence package and not Eclipse Link; this way I do not inherit Eclipse Links transitive dependencies. The log hell described in some other mails clearly shows the advantage of this model since you can defer what logging implementation to use to deployment time. Are you talking about build time here? In practice transitive dependencies can always be avoided at build time. If you are talking about runtime, transitive dependencies are definitely required. I'm not sure exactly what your JPA vs. Eclipse Link example is getting at. > >> Maintenance of dependencies with m2m > I understand that AS7 keeps the wiring separate and this is maintained by hand; Pom dependencies are maintained by hand and it Jigsaw will also require manual handling of dependencies because they require a new namespace unconnected to Java sources. This clearly lacks scalability since humans are really bad in maintaining this information for more than a handful of modules. > > In contrast, packages are part of the Java language and the dependencies are therefore already present in the class files; they therefore do not require the manual maintenance necessary for m2m. Again, are you talking about build time or runtime? Also, are you suggesting that Java determine your dependencies automatically including the version? Or is this merely to help developers find missing dependencies and remove unused dependencies? -bp > > Kind regards, > > Peter Kriens > > > > > > > > On 15 nov. 2011, at 06:14, David M. Lloyd wrote: > >> On 11/14/2011 08:33 PM, Neil Bartlett wrote: >>> Hi David, >>> >>> More than happy to talk about reality, though Peter already was doing >>> so. Nothing he said was theoretical, it came directly from his >>> experiences and the experiences of developers who have been using OSGi >>> for around 12 years. >> >>> Before digging into the points in your message, I need to address the >>> subject line "Why package deps won't work". Package dependencies will >>> work and they do work, as is proven every day by the large number of >>> applications running on OSGi. To claim otherwise would be wilfully >>> ignorant, so I will assume you are being hyperbolic and really meant >>> to assert that module deps are just better than package deps. >>> Therefore I intend to argue on that basis and claim that package deps >>> are better than module deps; note that I don't claim that module >>> dependencies "won't work" because JBoss Modules is a proof point that >>> they do. >> >> There was an implied "... for Java SE 8 modules" after "won't work" (this is why I said "won't" rather than "don't") which apparently wasn't implied enough. >> >> Of course people use OSGi, or we wouldn't be arguing, but that alone isn't enough to make OSGi's dependency system preferable for the SE case (after all, far more people use plain old class paths). I believe that JBoss Modules is a very accurate representation of how an SE module system could function, right down to executing JARs on the command line with dependencies and modularizing certain JDK APIs. This is not a proof point, but strong evidence that it is an effective solution for the actual problem on the table, and that similar architectures are likely to succeed for the same problem domain. As to why package deps are not an example of an effective solution for this problem, I intend to illustrate in greater detail. >> >> Further responses and elaboration below. >> >>> On Mon, Nov 14, 2011 at 5:05 PM, David M. Lloyd wrote: >>>> The explanation is quite simple, really - each point can be pretty much >>>> wiped out by a heavy dose of reality. >>>> >>>> 1. "M2P is leverages the Java type system unlike m2m that must introduce new >>>> namespaces outside the Java type system." - this is just fluffy >>>> buzzwordology. Considering packages part of the Java type system is a >>>> pretty liberal interpretation of the term "type system" as they're basically >>>> just arbitrary name spaces. That said, there's nothing particularly "new" >>>> about namespacing modules. JARs have names. Projects have names. Maven >>>> uses its own namespaces for artifacts. Even the primitive JDK extension >>>> mechanism uses symbolic names. >>> >>> From Wikipedia (http://en.wikipedia.org/wiki/Type_system): "A type >>> system associates a type with each computed value ... the aim is to >>> prevent operations expecting a certain kind of value being used with >>> values for which that operation does not make sense." >>> >>> The following will not compile: >>> >>> import java.awt.List; >>> // ... >>> List list = new ArrayList(); >>> list.iterator(); // etc >>> >>> whereas the following will: >>> >>> import java.util.List; >>> // ... >>> List list = new ArrayList(); >>> list.iterator(); // etc >>> >>> Package names are part of the type system because using an incorrect >>> package name in Java source can result in type errors during >>> compilation, and because the type and available operations associated >>> with each value varies with the package name used. >>> >>> In contrast, it is impossible to obtain a type error at compilation by >>> incorrectly naming a JAR, project, Maven artefact etc, because none of >>> those things are ever referenced in Java source code. >> >> I stand by my statement. A package is part of the name of the type; it is not a type, nor a meta-type, nor anything like a type in and of itself. This makes them part of the language, sure, but not part of the type system, in my opinion. Thus, "a stretch". I don't feel any particular need to have the world agree with me on this though, other than as I state below... >> >>>> Even a simple convention of the parent-most package name for the name of a >>>> module is very simple to grasp and is in fact the solution we've used quite >>>> effectively thus far. Thus implying that module names are some new alien >>>> concept is really not valid. >>> >>> The claim is not that they are a new alien concept. The claim is that >>> they are not part of the Java language and type system, and as a >>> result of this (along with other problems) are less useful for >>> depending upon than packages. >> >> I don't see how not being part of the Java language makes module identifiers "less useful". I think that this argument artificially elevates the significance of the relationship between packages and the actual type system, and then uses that elevation as a proof that a system based on other namespace principles is deficient, without offering any real examples which actually relate to the type system. >> >>>> 2. "M2P can be used to break the transitive dependency chain, m2m suffers of >>>> excessive coupling." - How about some facts to back this up? I've found >>>> "m2m" coupling to be just right. In JBoss Modules, we do not export >>>> transitive dependencies by default. This results in a very simple and clean >>>> dependency graph between modules. Using package dependencies results in a >>>> *far more complex* dependency graph, because you need edges for every >>>> package even though *most of the time you want the whole module anyway*. >>>> Again, real world here. >>> >>> Peter is definitely also talking about the real world and so am I. In >>> an m2m dependency model you cannot avoid transitive dependencies, >>> whether you expose them to consumers or not. >> >> In *any* model you cannot avoid transitive dependencies. If module A exports module B as part of its API, then you get A and B, and that's life. If you only want part of B, then you filter it down. But with most existing libraries out there (and they easily number in the thousands), they're intended to be used whole, so this doesn't really happen too much in practice. >> >>> An importer of a module must be assumed to depend on the whole functionality of that module >>> whether or not that is actually the case, and therefore all of the >>> transitive dependencies must be present at runtime. In an m2p world we >>> have the opportunity to split modules and break apart dependency >>> graphs, because the unit of coupling is more granular. >> >> You could break things down by actual classes if granularity is the goal. But it isn't; it is so much less important than, say, performance, conceptual simplicity, usability, efficiency, memory footprint, etc. The fact is that the vast majority of modules you'll find published at, say Maven central, are designed to be used completely, not partially, and can and are used as such without issue. >> >> People just don't usually design modules to be imported by package, except perhaps as an afterthought concession to OSGi, and they don't use them that way at runtime either. It is so seldom a real problem in the SE world (or EE for that matter) once you step away from flat class paths. The simple fact is that most users think of their dependencies in the same terms that their IDEs and their build scripts do: by module. >> >> This is truly a case of creating a lot of complexity and extra work in the common case to avoid problems which are not a common case. I think it is much better to optimize for the common case, where people know exactly what dependencies they want (in terms of artifacts), and they just want to declare them and be done without (a) hunting down package lists or (b) running extra tooling. >> >>>> If we're just going to throw dogma around, I'll put it the other way: m2p is >>>> a design error by the OSGi spec designers which has since been embraced as a >>>> religion. >>> >>> Pejorative accusations about "dogma" or "religion" have no place in a >>> technical discussion. They are also untrue. All of the practising OSGi >>> developers I know have arrived at their support for package >>> dependencies as a result of real world experience, not because of some >>> willingness to bow down to the Almighty CPEG. I don't know of any >>> practising OSGi developer who has used both Require-Bundle (m2m) and >>> Import-Package (m2p) and actually prefers the former. >> >> Sure, but Require-Bundle is *not* the same thing as using module-to-module dependencies in a non-OSGi system. All OSGi life revolves around the resolver. In the SE world, you cannot have a resolver without introducing a lot of machinery which will definitely negatively impact boot performance in the case that the resolver runs at module load time, or add the complexity of centralized package index management in the case that the resolver would run at install time (as in, install into a module library/repository, not install in the OSGi sense). >> >> If in a typical installation a user has to do more than simply delete a file to remove a module, or drop a JAR into a directory to add a module, I believe we've run quite far off-track usability-wise. And if a user has to sit through a lengthy resolution process at runtime (even if only the first time) then we're off-track performance-wise, especially if such resolution must expand to include modules in the module repository which are not even loaded. >> >>> I do know several who started as strong supporters of Require-Bundle >>> and switched to being supporters of Import-Package, not because of >>> Peter's wrathful pontificating but because they encountered the >>> specific problems that he described and found that they were fixed by >>> using Import-Package, and indeed that everything worked so much more >>> cleanly that way. I'm in this camp myself, and so was Jeff McAffer >>> before he went over to the Dark Side (Microsoft). >> >> There are other, simpler ways to fix package conflict issues when they arise. Like simply excluding the offending package from the import. >> >> And as I've said before, I do not believe that Require-Bundle in the context of OSGi bundles is comparable to loading modules by name in the context of a module system such as Jigsaw or JBoss Modules and the types of modules that could be loaded thereby (the aforementioned thousands of existing modules). >> >>>> It offers no significant benefit, other than a couple of edge >>>> cases which are frankly just as well handled by m2m simply by adding package >>>> filters. Which, by the way, I haven't seen the need to do yet in our 200+ >>>> module environment, but which we do have the capability to do. >>> >>> Substitution, module refactoring and transitive decoupling are hardly >>> edge cases. >> >> Substitution isn't solely addressed by package dependencies. Nor is module refactoring. And additional transitive decoupling isn't something people actually want - as I said, the common case is that users want to be able to use any class from a module once they import that module. In the real world, it is very rare to only want to import part of a module! If you need this routinely, you've done something very unorthodox. Thus I stand by my statement. >> >>> However I can see how these issues might not yet have come >>> to the fore in a module system designed for a single product with a >>> small number of modules, and where that product has not yet been >>> through the mill of multiple version evolutions. >> >> Cute. I assure you that we have been around through a long, long history of a huge number of Java users doing everything under the sun to their class paths and class loading structures. We also have experience with operating system distribution and managing module distribution across a wide range of other programming languages, not just Java. Our module system has already stood up to reasonably large deployments (200+ modules) with excellent memory footprint and performance numbers and we've been processing usability feedback, including feedback from those with OSGi experience, which has been mostly positive. >> >> Furthermore when I was at JavaOne giving booth talks about JBoss Modules and AS7, I found that users expressed quite a bit of frustration at the OSGi model and were quite receptive to a more orthodox m2m system, for what that's worth. >> >>>> M2P is a solution just itching for a problem. >>> >>> This is also not a very useful statement, I assure you that the >>> problem came before the solution. Better to show why you think the >>> problem is invalid or should have been solved differently. >> >> There is no way you can convince me that this solution was invented with general modularity in mind. Back when OSGi was first formed, it was designed for the embedded market. The world of Java today is completely different. People just don't bundle their libraries by package, as was possibly expected in the early days of Java. If they did we wouldn't be having this discussion because packages and modules would already be one and the same. >> >> OSGi evolved to where it is today. It was not designed from the ground up with actual requirements which pertained to the problem at hand which is modularity of the SE platform and applications which run on it. So yeah I'm going to say, solution before problem. Taking a solution and trying to apply it retroactively to a different problem like this has never failed to bite me in the ass, personally. But I can't speak to everyone's experience. >> >>>> But you're going to have a >>>> tough time convincing me that users *want* to have to use special tooling >>>> because they want to depend on a module which has too many packages to list >>>> out by hand. And before you cry "wildcards", be sure to consider that some >>>> modules use package names which are subordinate to other modules' packages, >>>> which is a perfectly normal and allowable scenario. Using wildcards for >>>> package matching could cause amazing levels of havoc. >>> >>> OSGi never uses wildcards at runtime, and tools such as bnd do not >>> need wildcards in order to express package-level dependencies. They >>> extract the set of packages that were actually used by the code, all >>> of which are available in the class files. This is possible because >>> packages are part of the type system (see first point above). >>> >>> So it's not that I don't want to list dependencies by hand, rather I >>> only want to do it once. I am already forced to do it in the import >>> statements of my Java sources. If I had to repeat that dependency >>> information -- whether in m2p or m2m form -- then I would run the risk >>> of it getting out of step with the real dependencies. >> >> One of the critical logical gaps here which m2p ignores at a fundamental level is that *packages are not unique*. In the real world, packages are repeated across more than one module all the time, and not just due to versioning. There is *no way* you could take every JAR at Maven Central and plug it in to an m2p system. You'd have to break apart and repackage every single one, at least, and at worst you'd have to rewrite an awful lot of them under different package names to account for this restriction. You'd need to enforce package uniqueness across your whole library, for all time. I don't think this is a sane option. You're adding significant overhead to management, installation, and execution, for what? So you can have the privilege of needing a special tool to extract your dependencies for you? >> >>>> Using package dependencies means you either must have a master package index >>>> for linking, or you need a resolver which has to have analyzed every module >>>> you ever plan to load. Otherwise, O(1) loading of modules is impossible, >>>> which is absolutely 100% a deal-breaker for JDK modules which must be >>>> incrementally installable. And it forbids having packages with the same >>>> name in more than one JAR without bringing run-time versioning into the >>>> fold, which is a terrible, terrible can of worms. >>> >>> Could you please explain why O(1) is the only acceptable complexity >>> for installing modules. >> >> Because this is Java SE we're talking about. You're going to have a potentially huge number of modules installed in your system. Even if you get around the versioning issues somehow and fully externalize the index of all packages, you still have to load or traverse the centralized index for every module load. >> >> I say that not only is ?(1) the only acceptable complexity for installing modules but also loading them at run time. Linking modules at run time should be no worse than O(n) complexity for the number of dependencies the module has, including transitive dependencies. Loading classes and resources from modules should be ?(k) where k is the number of modules which contain a package or directory which matches the class or resource being loaded (almost always one, in my experience). >> >> Loading a module should normally be limited to a one-time O(1) disk access without searching or loading any other files or modules than the module artifact itself (and possibly its descriptor if they are held externally by a particular module repository implementation, which is useful but not necessary). Linking a module should be limited in disk access to loading only the modules it imports via direct dependency, or by the relatively rare partial or full re-export of a transitive dependency by a direct dependency. >> >> In particular, the expectation of accessing a central index has some potentially serious implications, facing either possible file lock contention by multiple threads or memory overhead of loading in the complete index in advance. It should be expected that modules are loaded concurrently, and such concurrency should not be hindered any more than necessary. Other possible implementations (such as file- or directory-per-package) have their own drawbacks as well or violate what I consider to be core requirements. >> >> If there are implementations of package-based resolution which don't involve either a centralized index of one or many files or a resolver which must rove all installed modules in advance, I'd like to hear about it. >> >>> OSGi does indeed support incremental install >>> and while I accept it is probably not O(1) for each module, it would >>> likely be no more than O(N), though I haven't done the maths yet to >>> prove this. Bear in mind that in practice, for small N, the constant >>> factors can result in O(1) being more expensive than O(N). I have seen >>> OSGi used with many thousands of modules, so unless you have some data >>> and a use-case showings package-level resolution as unacceptably slow, >>> your concern just sounds like premature optimisation. >> >> Identifying algorithmic complexity during design phase is *never* premature optimization AFAICT. If you don't understand the complexity of the algorithms you're about to implement, and how they are expected to be applied to the problem at hand, then you're not ready for implementation yet. >> >> However, postulating that O(n) is okay because can sometimes be faster than O(1) without measuring it for the specific problem in question is definitely premature de-optimization. :-) >> >>> There are two reasons to have a packages with the same name in more >>> than one JAR. The first is a situation called split packages, and it >>> is highly undesirable because it causes the runtime model of the >>> package to diverge from the compile-time model, and therefore things >>> like package-private types and members stop working correctly. For >>> this reason, OSGi's m2p imports support depending only upon a single >>> exporter of a particular package, i.e. we do not aggregate all exports >>> of that package. >>> >>> Unfortunately split packages are sometimes unavoidable in legacy code >>> that cannot be refactored, e.g. the JDK. To support such scenarios >>> OSGi has Require-Bundle, i.e. m2m. This does not negate the problems >>> associated with m2m, it is simply a trade-off that we face with poorly >>> factored legacy code. >>> >>> The second reason for multiple packages with the same name is when you >>> explicitly want to install multiple versions of a library/API and have >>> them all available within the same runtime. I wouldn't call this a >>> "can of worms" exactly because it can be done without too much >>> trouble, though for the sake of a simple life I personally avoid this >>> situation unless it's necessary. >> >> Yes, and it is actually fairly common in practice to want two versions of something in the sense of the two versions being wholly different implementations (think apache commons logging versus jcl-over-slf4j for a trivial example which crops up a lot). There are reasons to use one or the other. Multiplicity of versions (in this respect) is something the module system has to handle gracefully and simply. >> >>>> Finally it should be perfectly clear to anyone who has read the original >>>> requirements document that nothing in this module system should prevent OSGi >>>> from functioning as it is, so there is absolutely no reason to assume that >>>> any OSGi implementation is so threatened - especially if m2p linking is as >>>> superior as has been expressed. Our module system (which is conceptually >>>> similar to Jigsaw in many regards) in fact does support our OSGi >>>> implementation quite effectively without itself implementing OSGi's >>>> package-to-package resolution (which like I said throws O(1) out the >>>> window). >>> >>> I agree that Jigsaw's existence doesn't threaten OSGi's, so long as >>> Java 8 doesn't actually break OSGi (and if it did so, it would >>> likewise break many other applications and could not be considered >>> backwards compatible with Java 7). The two can interoperate through >>> m2m-type dependencies. Tim Ellison started Project Penrose for the >>> purpose of investigating, testing and deepening this collaboration. >>> >>> Neverthless, the point that I believe Glyn was making is the >>> following. We accept that m2m dependencies are probably required for >>> the JDK, which implies a module system like Jigsaw or >>> OSGi/Require-Bundle rather than OSGi/Import-Package. However is it >>> intended to be used for application modularisation as well? This is of >>> course a question for the Jigsaw team rather than you, David. >> >> The question of whether Java SE 8 modules are intended to be used for application modularization is a question for the EG, not the Jigsaw team. The job of the Jigsaw team is really to implement a prototype which meets the requirements set forth by the EG, which may become the reference implementation at a future point. >> >> The question of whether the greater Java community will embrace the SE module system for applications is to be answered by the community only. I personally believe that to develop a module system which is not intended to be usable by the average standalone application is foolhardy, a great waste of effort, and is doomed to mediocrity, given the success we have had with such a system. >> >> However until there is an EG, these are all just as much questions for me as for anybody, and I can and will answer to the best of my knowledge and belief. >> >>> As a result of experience in developing and evolving large real-world >>> applications using a module system that supports BOTH m2m and m2p >>> dependencies, I believe it would be very unfortunate if a module >>> system that supports ONLY m2m were to become widely used in the >>> application space... not because OSGi can't handle the competition, >>> but because those applications will be fragile and hard to evolve. >> >> I don't believe this to be the case. I think that the status quo doesn't result in particularly fragile applications, and such can be easy to evolve or hard depending on the quality of the application components and frameworks involved. I think that m2m dependencies enhance the status quo such that applications are somewhat less fragile (owing chiefly to the simple measure of preventing transitive dependencies from being exported by default), and quite easy to evolve as well (even large and complex frameworks rarely have more than a few dependencies, unless they themselves are overly fragmented (e.g. CXF as a nightmare example which yet still works fine under an m2m system)). >> >>> My question for you David is as follows. I understand that you prefer >>> module dependencies, but do you believe that package dependencies have >>> no value whatsoever and therefore should not be available to >>> application developers in the Java 8 module system? If so, why did Red >>> Hat create an OSGi implementation? >> >> I personally believe that they have some value, but that value is limited to interoperability with OSGi. I do not believe that it is a preferable model for most users or standalone applications, nor for larger applications such as our application server. I think that the requirement for extra tooling and the increased run-time complexity which is exposed to users nullifies the benefits. If someone really wants this variety of modularity, they should simply use OSGi. >> >> Red Hat created an OSGi implementation for many reasons, but the only significant one to me is that there are people who want to use their OSGi applications with our application server. I don't believe any other reason is even necessary. It's the same reason we strive to support any spec, from EJB 1.0 to Java EE 6 and beyond. Someone wants the functionality, so we deliver it to the best of our ability. >> >>> Kind regards >>> Neil >>> >>>> >>>> On 11/14/2011 01:49 AM, Glyn Normington wrote: >>>>> >>>>> I look forward to David's elaboration of why he thinks "using packages as >>>>> a dependency unit is a terrible idea" to balance Peter's clear explanation >>>>> of the benefits of m2p. >>>>> >>>>> Meanwhile, it's worth noting that, according to the requirements document, >>>>> Jigsaw is aimed at platform modularisation and the platform being >>>>> modularised has some non-optimal division of types across packages (see the >>>>> package subsets requirement) which favour m2m dependencies. (Note that >>>>> Apache Harmony was developed with modularity in mind and was able to exploit >>>>> m2p, so platform modularisation per se needn't be limited to m2m.) >>>>> >>>>> So if Jigsaw excludes m2p, it will then be applicable to certain kinds of >>>>> legacy code modularisation and less applicable to new module development and >>>>> modularisation of existing code whose division into packages suits m2p. IIRC >>>>> this was the original positioning of Jigsaw: for use primarily within the >>>>> OpenJDK codebase and only exposed for application use because it was too >>>>> inconvenient to hide it. >>>>> >>>>> Regards, >>>>> Glyn >>>>> >>>>> On 12 Nov 2011, at 11:59, Peter Kriens wrote: >>>>> >>>>>> Neither my wrath, nor the fact that I rarely if ever get angry is >>>>>> relevant in this discussion ... This is a technical argument that are >>>>>> solvable by technical people that share the same goals. I prefer package >>>>>> dependencies because they address the excessive type coupling problem in >>>>>> object oriented systems, not because they're part of OSGi. Let me argue my >>>>>> case. >>>>>> >>>>>> Module-to-package dependencies (m2p) are preferable over module-to-module >>>>>> dependencies (m2m) for many reasons but these are the most important >>>>>> reasons: >>>>>> >>>>>> M2P is leverages the Java type system unlike m2m that must introduce new >>>>>> namespaces outside the Java type system. >>>>>> M2P can be used to break the transitive dependency chain, m2m suffers of >>>>>> excessive coupling >>>>>> >>>>>> Since the first bullet's benefit should be clear I only argue the more >>>>>> complex second bullet. >>>>>> >>>>>> A module is in many regards like a class. A class encapsulates members, >>>>>> depends on other members/classes, and makes a few members accessible outside >>>>>> the class. A module has a similar structure but then with types/packages as >>>>>> members. >>>>>> >>>>>> After the initial success of Object Oriented Programming (OO) it was >>>>>> quickly learned that reuse did not take place at the expected scale due to >>>>>> excessive type coupling. The problem was that a class aggregated many >>>>>> dependencies to simplify its implementation but these dependencies were >>>>>> unrelated to the contract it implemented. Since class dependencies are >>>>>> transitive most applications disappointingly became an almost fully >>>>>> connected graph. >>>>>> >>>>>> Java's great innovation was the interface because it broke both the >>>>>> transitivity and aggregation of dependencies. A class could now express its >>>>>> dependency (use or implement) on a contract (the interface) and was >>>>>> therefore fully type decoupled from the opposite site. >>>>>> >>>>>> An interface can act as a contract because it names the signature of a >>>>>> set of methods so that the compiler can verify the client and the >>>>>> implementer. >>>>>> >>>>>> Since a module has a very similar structure to a class it suffers from >>>>>> exactly the same transitive aggregation of dependencies. This is not a >>>>>> theory, look at the experiences with Maven >>>>>> (http://www.sonatype.com/people/2011/04/how-not-to-download-the-internet/) >>>>>> Again, this is not that maven is bad or developers are stupid, it is the >>>>>> same underlying force that finally resulted in the Java interface. >>>>>> >>>>>> The parallel for the class' interface for modules is a named set of >>>>>> interfaces. This concept already exists in Java: a package. Looking at >>>>>> almost all JSRs it is clear that our industry already uses packages as >>>>>> "interfaces" to provider implementations. >>>>>> >>>>>> Therefore, just like a class should not depend on other implementation >>>>>> types, a module should preferably not depend on other modules. A module >>>>>> should instead depend on contracts. Since modules will be used to provide >>>>>> components from different sources managed with different life cycles the >>>>>> excessive type coupling caused by m2m is even more damaging than in c2c. >>>>>> Proper use of m2p creates significantly less type coupled systems than m2m, >>>>>> the benefits should be obvious. >>>>>> >>>>>> Since there are use cases for m2m (non-type safe languages for example) I >>>>>> do believe that Jigsaw should still support m2m. However, it would be >>>>>> greatly beneficial to our industry if we could take advantage of the lessons >>>>>> learned with the Java interface and realize how surprisingly important the >>>>>> Java package actually is in our eco system. >>>>>> >>>>>> Kind regards, >>>>>> >>>>>> Peter Kriens >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On 9 nov. 2011, at 15:04, David M. Lloyd wrote: >>>>>> >>>>>>> I'll just state now that using packages as a dependency unit is a >>>>>>> terrible idea, and not some architectural revelation. That way, Peter's >>>>>>> wrath will be largely directed at me. :-) >>>>>>> >>>>>>> On 11/09/2011 08:02 AM, Peter Kriens wrote: >>>>>>>> >>>>>>>> I agree that tools are needed but we must be careful to not expect >>>>>>>> tools to stopgap an architectural issue. I think it is important to first do >>>>>>>> good architectural design leveraging existing tools (e.g. the Java type >>>>>>>> system) before you try to add new tools. It is such a pity (but all to >>>>>>>> common) that a design allows for classes of errors that would be impossible >>>>>>>> with a slightly different design. >>>>>>>> >>>>>>>> Kind regards, >>>>>>>> >>>>>>>> Peter Kriens >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 9 nov. 2011, at 14:49, Alan Bateman wrote: >>>>>>>> >>>>>>>>> On 09/11/2011 13:04, Peter Kriens wrote: >>>>>>>>>> >>>>>>>>>> The issue is that maven problems are not caused because maven is bad >>>>>>>>>> or that pom authors are stupid. The reason is that the module-to-module >>>>>>>>>> dependency architecture in maven (and Jigsaw) is error prone ... >>>>>>>>> >>>>>>>>> This thread started out with someone asking about adding module >>>>>>>>> declarations to existing JAR files, and in that context, I agree it can be >>>>>>>>> error prone without good tools. I think things should be a lot better when >>>>>>>>> modules are compiled. >>>>>>>>> >>>>>>>>> -Alan. >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> - DML >>>>>> >>>>> >>>> >>>> >>>> -- >>>> - DML >>>> >> >> >> -- >> - DML > From peter.kriens at aqute.biz Wed Nov 16 11:05:11 2011 From: peter.kriens at aqute.biz (Peter Kriens) Date: Wed, 16 Nov 2011 20:05:11 +0100 Subject: Why package deps work (Was: Re: Converting plain JARs to Java modules) In-Reply-To: <9B28CE0A-85DC-4F53-8A88-5830759DA4BC@pontarelli.com> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> <67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz> <4EC14A53.7070907@redhat.com> <4EC1F53D.7040309@redhat.com> <328B38A3-EB53-4AD2-970B-CE84F5C26F96@aQute.biz> <9B28CE0A-85DC-4F53-8A88-5830759DA4BC@pontarelli.com> Message-ID: On 16 nov. 2011, at 19:23, Brian Pontarelli wrote: > Are you talking about build time here? In practice transitive dependencies can always be avoided at build time. If you are talking about runtime, transitive dependencies are definitely required. I'm not sure exactly what your JPA vs. Eclipse Link example is getting at. There is type coupling and instance coupling. Interfaces break the transitive type coupling but then require that you bind an instance for that interface in runtime, there is a trade off. Spring/Guice/CDI/Services, etc all handle this instance coupling. With a module system, packages type-decouple you from an implementation module just like an interface type decouples a class from its dependencies. In the Eclipse Link example, my code is not type coupled to Eclipse Link, it is only type coupled to its specification package javax.persistence. This allows the deployer to choose either Eclipse Link or OpenJPA they both provide the implementation to the javax.transaction package. However, my code is not having a type coupling on either leaving flexibility to the deployer. So we make a trade off between type decoupling and runtime complexity. This is the same trade off we make in IOC/DI engines, annotations, and factories. > Again, are you talking about build time or runtime? Also, are you suggesting that Java determine your dependencies automatically including the version? Or is this merely to help developers find missing dependencies and remove unused dependencies? If you compile class com.bar.Foo that refers to com.bar.Bar then the Foo.class and Bar.class file contain all their type dependencies and by implication their package dependencies. The version of the package can then for example be found from package-info.java. This technique is used extensively in bnd, an open source library used in Eclipse, IDEA, ant, maven, SBT, and others. It reads the class files in a JAR and generates the OSGi dependency information automatically. Hope this clarifies. Kind regards, Peter Kriens > > -bp > > > > > > >> >> Kind regards, >> >> Peter Kriens >> >> >> >> >> >> >> >> On 15 nov. 2011, at 06:14, David M. Lloyd wrote: >> >>> On 11/14/2011 08:33 PM, Neil Bartlett wrote: >>>> Hi David, >>>> >>>> More than happy to talk about reality, though Peter already was doing >>>> so. Nothing he said was theoretical, it came directly from his >>>> experiences and the experiences of developers who have been using OSGi >>>> for around 12 years. >>> >>>> Before digging into the points in your message, I need to address the >>>> subject line "Why package deps won't work". Package dependencies will >>>> work and they do work, as is proven every day by the large number of >>>> applications running on OSGi. To claim otherwise would be wilfully >>>> ignorant, so I will assume you are being hyperbolic and really meant >>>> to assert that module deps are just better than package deps. >>>> Therefore I intend to argue on that basis and claim that package deps >>>> are better than module deps; note that I don't claim that module >>>> dependencies "won't work" because JBoss Modules is a proof point that >>>> they do. >>> >>> There was an implied "... for Java SE 8 modules" after "won't work" (this is why I said "won't" rather than "don't") which apparently wasn't implied enough. >>> >>> Of course people use OSGi, or we wouldn't be arguing, but that alone isn't enough to make OSGi's dependency system preferable for the SE case (after all, far more people use plain old class paths). I believe that JBoss Modules is a very accurate representation of how an SE module system could function, right down to executing JARs on the command line with dependencies and modularizing certain JDK APIs. This is not a proof point, but strong evidence that it is an effective solution for the actual problem on the table, and that similar architectures are likely to succeed for the same problem domain. As to why package deps are not an example of an effective solution for this problem, I intend to illustrate in greater detail. >>> >>> Further responses and elaboration below. >>> >>>> On Mon, Nov 14, 2011 at 5:05 PM, David M. Lloyd wrote: >>>>> The explanation is quite simple, really - each point can be pretty much >>>>> wiped out by a heavy dose of reality. >>>>> >>>>> 1. "M2P is leverages the Java type system unlike m2m that must introduce new >>>>> namespaces outside the Java type system." - this is just fluffy >>>>> buzzwordology. Considering packages part of the Java type system is a >>>>> pretty liberal interpretation of the term "type system" as they're basically >>>>> just arbitrary name spaces. That said, there's nothing particularly "new" >>>>> about namespacing modules. JARs have names. Projects have names. Maven >>>>> uses its own namespaces for artifacts. Even the primitive JDK extension >>>>> mechanism uses symbolic names. >>>> >>>> From Wikipedia (http://en.wikipedia.org/wiki/Type_system): "A type >>>> system associates a type with each computed value ... the aim is to >>>> prevent operations expecting a certain kind of value being used with >>>> values for which that operation does not make sense." >>>> >>>> The following will not compile: >>>> >>>> import java.awt.List; >>>> // ... >>>> List list = new ArrayList(); >>>> list.iterator(); // etc >>>> >>>> whereas the following will: >>>> >>>> import java.util.List; >>>> // ... >>>> List list = new ArrayList(); >>>> list.iterator(); // etc >>>> >>>> Package names are part of the type system because using an incorrect >>>> package name in Java source can result in type errors during >>>> compilation, and because the type and available operations associated >>>> with each value varies with the package name used. >>>> >>>> In contrast, it is impossible to obtain a type error at compilation by >>>> incorrectly naming a JAR, project, Maven artefact etc, because none of >>>> those things are ever referenced in Java source code. >>> >>> I stand by my statement. A package is part of the name of the type; it is not a type, nor a meta-type, nor anything like a type in and of itself. This makes them part of the language, sure, but not part of the type system, in my opinion. Thus, "a stretch". I don't feel any particular need to have the world agree with me on this though, other than as I state below... >>> >>>>> Even a simple convention of the parent-most package name for the name of a >>>>> module is very simple to grasp and is in fact the solution we've used quite >>>>> effectively thus far. Thus implying that module names are some new alien >>>>> concept is really not valid. >>>> >>>> The claim is not that they are a new alien concept. The claim is that >>>> they are not part of the Java language and type system, and as a >>>> result of this (along with other problems) are less useful for >>>> depending upon than packages. >>> >>> I don't see how not being part of the Java language makes module identifiers "less useful". I think that this argument artificially elevates the significance of the relationship between packages and the actual type system, and then uses that elevation as a proof that a system based on other namespace principles is deficient, without offering any real examples which actually relate to the type system. >>> >>>>> 2. "M2P can be used to break the transitive dependency chain, m2m suffers of >>>>> excessive coupling." - How about some facts to back this up? I've found >>>>> "m2m" coupling to be just right. In JBoss Modules, we do not export >>>>> transitive dependencies by default. This results in a very simple and clean >>>>> dependency graph between modules. Using package dependencies results in a >>>>> *far more complex* dependency graph, because you need edges for every >>>>> package even though *most of the time you want the whole module anyway*. >>>>> Again, real world here. >>>> >>>> Peter is definitely also talking about the real world and so am I. In >>>> an m2m dependency model you cannot avoid transitive dependencies, >>>> whether you expose them to consumers or not. >>> >>> In *any* model you cannot avoid transitive dependencies. If module A exports module B as part of its API, then you get A and B, and that's life. If you only want part of B, then you filter it down. But with most existing libraries out there (and they easily number in the thousands), they're intended to be used whole, so this doesn't really happen too much in practice. >>> >>>> An importer of a module must be assumed to depend on the whole functionality of that module >>>> whether or not that is actually the case, and therefore all of the >>>> transitive dependencies must be present at runtime. In an m2p world we >>>> have the opportunity to split modules and break apart dependency >>>> graphs, because the unit of coupling is more granular. >>> >>> You could break things down by actual classes if granularity is the goal. But it isn't; it is so much less important than, say, performance, conceptual simplicity, usability, efficiency, memory footprint, etc. The fact is that the vast majority of modules you'll find published at, say Maven central, are designed to be used completely, not partially, and can and are used as such without issue. >>> >>> People just don't usually design modules to be imported by package, except perhaps as an afterthought concession to OSGi, and they don't use them that way at runtime either. It is so seldom a real problem in the SE world (or EE for that matter) once you step away from flat class paths. The simple fact is that most users think of their dependencies in the same terms that their IDEs and their build scripts do: by module. >>> >>> This is truly a case of creating a lot of complexity and extra work in the common case to avoid problems which are not a common case. I think it is much better to optimize for the common case, where people know exactly what dependencies they want (in terms of artifacts), and they just want to declare them and be done without (a) hunting down package lists or (b) running extra tooling. >>> >>>>> If we're just going to throw dogma around, I'll put it the other way: m2p is >>>>> a design error by the OSGi spec designers which has since been embraced as a >>>>> religion. >>>> >>>> Pejorative accusations about "dogma" or "religion" have no place in a >>>> technical discussion. They are also untrue. All of the practising OSGi >>>> developers I know have arrived at their support for package >>>> dependencies as a result of real world experience, not because of some >>>> willingness to bow down to the Almighty CPEG. I don't know of any >>>> practising OSGi developer who has used both Require-Bundle (m2m) and >>>> Import-Package (m2p) and actually prefers the former. >>> >>> Sure, but Require-Bundle is *not* the same thing as using module-to-module dependencies in a non-OSGi system. All OSGi life revolves around the resolver. In the SE world, you cannot have a resolver without introducing a lot of machinery which will definitely negatively impact boot performance in the case that the resolver runs at module load time, or add the complexity of centralized package index management in the case that the resolver would run at install time (as in, install into a module library/repository, not install in the OSGi sense). >>> >>> If in a typical installation a user has to do more than simply delete a file to remove a module, or drop a JAR into a directory to add a module, I believe we've run quite far off-track usability-wise. And if a user has to sit through a lengthy resolution process at runtime (even if only the first time) then we're off-track performance-wise, especially if such resolution must expand to include modules in the module repository which are not even loaded. >>> >>>> I do know several who started as strong supporters of Require-Bundle >>>> and switched to being supporters of Import-Package, not because of >>>> Peter's wrathful pontificating but because they encountered the >>>> specific problems that he described and found that they were fixed by >>>> using Import-Package, and indeed that everything worked so much more >>>> cleanly that way. I'm in this camp myself, and so was Jeff McAffer >>>> before he went over to the Dark Side (Microsoft). >>> >>> There are other, simpler ways to fix package conflict issues when they arise. Like simply excluding the offending package from the import. >>> >>> And as I've said before, I do not believe that Require-Bundle in the context of OSGi bundles is comparable to loading modules by name in the context of a module system such as Jigsaw or JBoss Modules and the types of modules that could be loaded thereby (the aforementioned thousands of existing modules). >>> >>>>> It offers no significant benefit, other than a couple of edge >>>>> cases which are frankly just as well handled by m2m simply by adding package >>>>> filters. Which, by the way, I haven't seen the need to do yet in our 200+ >>>>> module environment, but which we do have the capability to do. >>>> >>>> Substitution, module refactoring and transitive decoupling are hardly >>>> edge cases. >>> >>> Substitution isn't solely addressed by package dependencies. Nor is module refactoring. And additional transitive decoupling isn't something people actually want - as I said, the common case is that users want to be able to use any class from a module once they import that module. In the real world, it is very rare to only want to import part of a module! If you need this routinely, you've done something very unorthodox. Thus I stand by my statement. >>> >>>> However I can see how these issues might not yet have come >>>> to the fore in a module system designed for a single product with a >>>> small number of modules, and where that product has not yet been >>>> through the mill of multiple version evolutions. >>> >>> Cute. I assure you that we have been around through a long, long history of a huge number of Java users doing everything under the sun to their class paths and class loading structures. We also have experience with operating system distribution and managing module distribution across a wide range of other programming languages, not just Java. Our module system has already stood up to reasonably large deployments (200+ modules) with excellent memory footprint and performance numbers and we've been processing usability feedback, including feedback from those with OSGi experience, which has been mostly positive. >>> >>> Furthermore when I was at JavaOne giving booth talks about JBoss Modules and AS7, I found that users expressed quite a bit of frustration at the OSGi model and were quite receptive to a more orthodox m2m system, for what that's worth. >>> >>>>> M2P is a solution just itching for a problem. >>>> >>>> This is also not a very useful statement, I assure you that the >>>> problem came before the solution. Better to show why you think the >>>> problem is invalid or should have been solved differently. >>> >>> There is no way you can convince me that this solution was invented with general modularity in mind. Back when OSGi was first formed, it was designed for the embedded market. The world of Java today is completely different. People just don't bundle their libraries by package, as was possibly expected in the early days of Java. If they did we wouldn't be having this discussion because packages and modules would already be one and the same. >>> >>> OSGi evolved to where it is today. It was not designed from the ground up with actual requirements which pertained to the problem at hand which is modularity of the SE platform and applications which run on it. So yeah I'm going to say, solution before problem. Taking a solution and trying to apply it retroactively to a different problem like this has never failed to bite me in the ass, personally. But I can't speak to everyone's experience. >>> >>>>> But you're going to have a >>>>> tough time convincing me that users *want* to have to use special tooling >>>>> because they want to depend on a module which has too many packages to list >>>>> out by hand. And before you cry "wildcards", be sure to consider that some >>>>> modules use package names which are subordinate to other modules' packages, >>>>> which is a perfectly normal and allowable scenario. Using wildcards for >>>>> package matching could cause amazing levels of havoc. >>>> >>>> OSGi never uses wildcards at runtime, and tools such as bnd do not >>>> need wildcards in order to express package-level dependencies. They >>>> extract the set of packages that were actually used by the code, all >>>> of which are available in the class files. This is possible because >>>> packages are part of the type system (see first point above). >>>> >>>> So it's not that I don't want to list dependencies by hand, rather I >>>> only want to do it once. I am already forced to do it in the import >>>> statements of my Java sources. If I had to repeat that dependency >>>> information -- whether in m2p or m2m form -- then I would run the risk >>>> of it getting out of step with the real dependencies. >>> >>> One of the critical logical gaps here which m2p ignores at a fundamental level is that *packages are not unique*. In the real world, packages are repeated across more than one module all the time, and not just due to versioning. There is *no way* you could take every JAR at Maven Central and plug it in to an m2p system. You'd have to break apart and repackage every single one, at least, and at worst you'd have to rewrite an awful lot of them under different package names to account for this restriction. You'd need to enforce package uniqueness across your whole library, for all time. I don't think this is a sane option. You're adding significant overhead to management, installation, and execution, for what? So you can have the privilege of needing a special tool to extract your dependencies for you? >>> >>>>> Using package dependencies means you either must have a master package index >>>>> for linking, or you need a resolver which has to have analyzed every module >>>>> you ever plan to load. Otherwise, O(1) loading of modules is impossible, >>>>> which is absolutely 100% a deal-breaker for JDK modules which must be >>>>> incrementally installable. And it forbids having packages with the same >>>>> name in more than one JAR without bringing run-time versioning into the >>>>> fold, which is a terrible, terrible can of worms. >>>> >>>> Could you please explain why O(1) is the only acceptable complexity >>>> for installing modules. >>> >>> Because this is Java SE we're talking about. You're going to have a potentially huge number of modules installed in your system. Even if you get around the versioning issues somehow and fully externalize the index of all packages, you still have to load or traverse the centralized index for every module load. >>> >>> I say that not only is ?(1) the only acceptable complexity for installing modules but also loading them at run time. Linking modules at run time should be no worse than O(n) complexity for the number of dependencies the module has, including transitive dependencies. Loading classes and resources from modules should be ?(k) where k is the number of modules which contain a package or directory which matches the class or resource being loaded (almost always one, in my experience). >>> >>> Loading a module should normally be limited to a one-time O(1) disk access without searching or loading any other files or modules than the module artifact itself (and possibly its descriptor if they are held externally by a particular module repository implementation, which is useful but not necessary). Linking a module should be limited in disk access to loading only the modules it imports via direct dependency, or by the relatively rare partial or full re-export of a transitive dependency by a direct dependency. >>> >>> In particular, the expectation of accessing a central index has some potentially serious implications, facing either possible file lock contention by multiple threads or memory overhead of loading in the complete index in advance. It should be expected that modules are loaded concurrently, and such concurrency should not be hindered any more than necessary. Other possible implementations (such as file- or directory-per-package) have their own drawbacks as well or violate what I consider to be core requirements. >>> >>> If there are implementations of package-based resolution which don't involve either a centralized index of one or many files or a resolver which must rove all installed modules in advance, I'd like to hear about it. >>> >>>> OSGi does indeed support incremental install >>>> and while I accept it is probably not O(1) for each module, it would >>>> likely be no more than O(N), though I haven't done the maths yet to >>>> prove this. Bear in mind that in practice, for small N, the constant >>>> factors can result in O(1) being more expensive than O(N). I have seen >>>> OSGi used with many thousands of modules, so unless you have some data >>>> and a use-case showings package-level resolution as unacceptably slow, >>>> your concern just sounds like premature optimisation. >>> >>> Identifying algorithmic complexity during design phase is *never* premature optimization AFAICT. If you don't understand the complexity of the algorithms you're about to implement, and how they are expected to be applied to the problem at hand, then you're not ready for implementation yet. >>> >>> However, postulating that O(n) is okay because can sometimes be faster than O(1) without measuring it for the specific problem in question is definitely premature de-optimization. :-) >>> >>>> There are two reasons to have a packages with the same name in more >>>> than one JAR. The first is a situation called split packages, and it >>>> is highly undesirable because it causes the runtime model of the >>>> package to diverge from the compile-time model, and therefore things >>>> like package-private types and members stop working correctly. For >>>> this reason, OSGi's m2p imports support depending only upon a single >>>> exporter of a particular package, i.e. we do not aggregate all exports >>>> of that package. >>>> >>>> Unfortunately split packages are sometimes unavoidable in legacy code >>>> that cannot be refactored, e.g. the JDK. To support such scenarios >>>> OSGi has Require-Bundle, i.e. m2m. This does not negate the problems >>>> associated with m2m, it is simply a trade-off that we face with poorly >>>> factored legacy code. >>>> >>>> The second reason for multiple packages with the same name is when you >>>> explicitly want to install multiple versions of a library/API and have >>>> them all available within the same runtime. I wouldn't call this a >>>> "can of worms" exactly because it can be done without too much >>>> trouble, though for the sake of a simple life I personally avoid this >>>> situation unless it's necessary. >>> >>> Yes, and it is actually fairly common in practice to want two versions of something in the sense of the two versions being wholly different implementations (think apache commons logging versus jcl-over-slf4j for a trivial example which crops up a lot). There are reasons to use one or the other. Multiplicity of versions (in this respect) is something the module system has to handle gracefully and simply. >>> >>>>> Finally it should be perfectly clear to anyone who has read the original >>>>> requirements document that nothing in this module system should prevent OSGi >>>>> from functioning as it is, so there is absolutely no reason to assume that >>>>> any OSGi implementation is so threatened - especially if m2p linking is as >>>>> superior as has been expressed. Our module system (which is conceptually >>>>> similar to Jigsaw in many regards) in fact does support our OSGi >>>>> implementation quite effectively without itself implementing OSGi's >>>>> package-to-package resolution (which like I said throws O(1) out the >>>>> window). >>>> >>>> I agree that Jigsaw's existence doesn't threaten OSGi's, so long as >>>> Java 8 doesn't actually break OSGi (and if it did so, it would >>>> likewise break many other applications and could not be considered >>>> backwards compatible with Java 7). The two can interoperate through >>>> m2m-type dependencies. Tim Ellison started Project Penrose for the >>>> purpose of investigating, testing and deepening this collaboration. >>>> >>>> Neverthless, the point that I believe Glyn was making is the >>>> following. We accept that m2m dependencies are probably required for >>>> the JDK, which implies a module system like Jigsaw or >>>> OSGi/Require-Bundle rather than OSGi/Import-Package. However is it >>>> intended to be used for application modularisation as well? This is of >>>> course a question for the Jigsaw team rather than you, David. >>> >>> The question of whether Java SE 8 modules are intended to be used for application modularization is a question for the EG, not the Jigsaw team. The job of the Jigsaw team is really to implement a prototype which meets the requirements set forth by the EG, which may become the reference implementation at a future point. >>> >>> The question of whether the greater Java community will embrace the SE module system for applications is to be answered by the community only. I personally believe that to develop a module system which is not intended to be usable by the average standalone application is foolhardy, a great waste of effort, and is doomed to mediocrity, given the success we have had with such a system. >>> >>> However until there is an EG, these are all just as much questions for me as for anybody, and I can and will answer to the best of my knowledge and belief. >>> >>>> As a result of experience in developing and evolving large real-world >>>> applications using a module system that supports BOTH m2m and m2p >>>> dependencies, I believe it would be very unfortunate if a module >>>> system that supports ONLY m2m were to become widely used in the >>>> application space... not because OSGi can't handle the competition, >>>> but because those applications will be fragile and hard to evolve. >>> >>> I don't believe this to be the case. I think that the status quo doesn't result in particularly fragile applications, and such can be easy to evolve or hard depending on the quality of the application components and frameworks involved. I think that m2m dependencies enhance the status quo such that applications are somewhat less fragile (owing chiefly to the simple measure of preventing transitive dependencies from being exported by default), and quite easy to evolve as well (even large and complex frameworks rarely have more than a few dependencies, unless they themselves are overly fragmented (e.g. CXF as a nightmare example which yet still works fine under an m2m system)). >>> >>>> My question for you David is as follows. I understand that you prefer >>>> module dependencies, but do you believe that package dependencies have >>>> no value whatsoever and therefore should not be available to >>>> application developers in the Java 8 module system? If so, why did Red >>>> Hat create an OSGi implementation? >>> >>> I personally believe that they have some value, but that value is limited to interoperability with OSGi. I do not believe that it is a preferable model for most users or standalone applications, nor for larger applications such as our application server. I think that the requirement for extra tooling and the increased run-time complexity which is exposed to users nullifies the benefits. If someone really wants this variety of modularity, they should simply use OSGi. >>> >>> Red Hat created an OSGi implementation for many reasons, but the only significant one to me is that there are people who want to use their OSGi applications with our application server. I don't believe any other reason is even necessary. It's the same reason we strive to support any spec, from EJB 1.0 to Java EE 6 and beyond. Someone wants the functionality, so we deliver it to the best of our ability. >>> >>>> Kind regards >>>> Neil >>>> >>>>> >>>>> On 11/14/2011 01:49 AM, Glyn Normington wrote: >>>>>> >>>>>> I look forward to David's elaboration of why he thinks "using packages as >>>>>> a dependency unit is a terrible idea" to balance Peter's clear explanation >>>>>> of the benefits of m2p. >>>>>> >>>>>> Meanwhile, it's worth noting that, according to the requirements document, >>>>>> Jigsaw is aimed at platform modularisation and the platform being >>>>>> modularised has some non-optimal division of types across packages (see the >>>>>> package subsets requirement) which favour m2m dependencies. (Note that >>>>>> Apache Harmony was developed with modularity in mind and was able to exploit >>>>>> m2p, so platform modularisation per se needn't be limited to m2m.) >>>>>> >>>>>> So if Jigsaw excludes m2p, it will then be applicable to certain kinds of >>>>>> legacy code modularisation and less applicable to new module development and >>>>>> modularisation of existing code whose division into packages suits m2p. IIRC >>>>>> this was the original positioning of Jigsaw: for use primarily within the >>>>>> OpenJDK codebase and only exposed for application use because it was too >>>>>> inconvenient to hide it. >>>>>> >>>>>> Regards, >>>>>> Glyn >>>>>> >>>>>> On 12 Nov 2011, at 11:59, Peter Kriens wrote: >>>>>> >>>>>>> Neither my wrath, nor the fact that I rarely if ever get angry is >>>>>>> relevant in this discussion ... This is a technical argument that are >>>>>>> solvable by technical people that share the same goals. I prefer package >>>>>>> dependencies because they address the excessive type coupling problem in >>>>>>> object oriented systems, not because they're part of OSGi. Let me argue my >>>>>>> case. >>>>>>> >>>>>>> Module-to-package dependencies (m2p) are preferable over module-to-module >>>>>>> dependencies (m2m) for many reasons but these are the most important >>>>>>> reasons: >>>>>>> >>>>>>> M2P is leverages the Java type system unlike m2m that must introduce new >>>>>>> namespaces outside the Java type system. >>>>>>> M2P can be used to break the transitive dependency chain, m2m suffers of >>>>>>> excessive coupling >>>>>>> >>>>>>> Since the first bullet's benefit should be clear I only argue the more >>>>>>> complex second bullet. >>>>>>> >>>>>>> A module is in many regards like a class. A class encapsulates members, >>>>>>> depends on other members/classes, and makes a few members accessible outside >>>>>>> the class. A module has a similar structure but then with types/packages as >>>>>>> members. >>>>>>> >>>>>>> After the initial success of Object Oriented Programming (OO) it was >>>>>>> quickly learned that reuse did not take place at the expected scale due to >>>>>>> excessive type coupling. The problem was that a class aggregated many >>>>>>> dependencies to simplify its implementation but these dependencies were >>>>>>> unrelated to the contract it implemented. Since class dependencies are >>>>>>> transitive most applications disappointingly became an almost fully >>>>>>> connected graph. >>>>>>> >>>>>>> Java's great innovation was the interface because it broke both the >>>>>>> transitivity and aggregation of dependencies. A class could now express its >>>>>>> dependency (use or implement) on a contract (the interface) and was >>>>>>> therefore fully type decoupled from the opposite site. >>>>>>> >>>>>>> An interface can act as a contract because it names the signature of a >>>>>>> set of methods so that the compiler can verify the client and the >>>>>>> implementer. >>>>>>> >>>>>>> Since a module has a very similar structure to a class it suffers from >>>>>>> exactly the same transitive aggregation of dependencies. This is not a >>>>>>> theory, look at the experiences with Maven >>>>>>> (http://www.sonatype.com/people/2011/04/how-not-to-download-the-internet/) >>>>>>> Again, this is not that maven is bad or developers are stupid, it is the >>>>>>> same underlying force that finally resulted in the Java interface. >>>>>>> >>>>>>> The parallel for the class' interface for modules is a named set of >>>>>>> interfaces. This concept already exists in Java: a package. Looking at >>>>>>> almost all JSRs it is clear that our industry already uses packages as >>>>>>> "interfaces" to provider implementations. >>>>>>> >>>>>>> Therefore, just like a class should not depend on other implementation >>>>>>> types, a module should preferably not depend on other modules. A module >>>>>>> should instead depend on contracts. Since modules will be used to provide >>>>>>> components from different sources managed with different life cycles the >>>>>>> excessive type coupling caused by m2m is even more damaging than in c2c. >>>>>>> Proper use of m2p creates significantly less type coupled systems than m2m, >>>>>>> the benefits should be obvious. >>>>>>> >>>>>>> Since there are use cases for m2m (non-type safe languages for example) I >>>>>>> do believe that Jigsaw should still support m2m. However, it would be >>>>>>> greatly beneficial to our industry if we could take advantage of the lessons >>>>>>> learned with the Java interface and realize how surprisingly important the >>>>>>> Java package actually is in our eco system. >>>>>>> >>>>>>> Kind regards, >>>>>>> >>>>>>> Peter Kriens >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 9 nov. 2011, at 15:04, David M. Lloyd wrote: >>>>>>> >>>>>>>> I'll just state now that using packages as a dependency unit is a >>>>>>>> terrible idea, and not some architectural revelation. That way, Peter's >>>>>>>> wrath will be largely directed at me. :-) >>>>>>>> >>>>>>>> On 11/09/2011 08:02 AM, Peter Kriens wrote: >>>>>>>>> >>>>>>>>> I agree that tools are needed but we must be careful to not expect >>>>>>>>> tools to stopgap an architectural issue. I think it is important to first do >>>>>>>>> good architectural design leveraging existing tools (e.g. the Java type >>>>>>>>> system) before you try to add new tools. It is such a pity (but all to >>>>>>>>> common) that a design allows for classes of errors that would be impossible >>>>>>>>> with a slightly different design. >>>>>>>>> >>>>>>>>> Kind regards, >>>>>>>>> >>>>>>>>> Peter Kriens >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 9 nov. 2011, at 14:49, Alan Bateman wrote: >>>>>>>>> >>>>>>>>>> On 09/11/2011 13:04, Peter Kriens wrote: >>>>>>>>>>> >>>>>>>>>>> The issue is that maven problems are not caused because maven is bad >>>>>>>>>>> or that pom authors are stupid. The reason is that the module-to-module >>>>>>>>>>> dependency architecture in maven (and Jigsaw) is error prone ... >>>>>>>>>> >>>>>>>>>> This thread started out with someone asking about adding module >>>>>>>>>> declarations to existing JAR files, and in that context, I agree it can be >>>>>>>>>> error prone without good tools. I think things should be a lot better when >>>>>>>>>> modules are compiled. >>>>>>>>>> >>>>>>>>>> -Alan. >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> - DML >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> - DML >>>>> >>> >>> >>> -- >>> - DML >> > From brian at pontarelli.com Wed Nov 16 13:16:43 2011 From: brian at pontarelli.com (Brian Pontarelli) Date: Wed, 16 Nov 2011 14:16:43 -0700 Subject: Why package deps work (Was: Re: Converting plain JARs to Java modules) In-Reply-To: References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> <67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz> <4EC14A53.7070907@redhat.com> <4EC1F53D.7040309@redhat.com> <328B38A3-EB53-4AD2-970B-CE84F5C26F96@aQute.biz> <9B28CE0A-85DC-4F53-8A88-5830759DA4BC@pontarelli.com> Message-ID: On Nov 16, 2011, at 12:05 PM, Peter Kriens wrote: > On 16 nov. 2011, at 19:23, Brian Pontarelli wrote: >> Are you talking about build time here? In practice transitive dependencies can always be avoided at build time. If you are talking about runtime, transitive dependencies are definitely required. I'm not sure exactly what your JPA vs. Eclipse Link example is getting at. > There is type coupling and instance coupling. Interfaces break the transitive type coupling but then require that you bind an instance for that interface in runtime, there is a trade off. Spring/Guice/CDI/Services, etc all handle this instance coupling. With a module system, packages type-decouple you from an implementation module just like an interface type decouples a class from its dependencies. In the Eclipse Link example, my code is not type coupled to Eclipse Link, it is only type coupled to its specification package javax.persistence. This allows the deployer to choose either Eclipse Link or OpenJPA they both provide the implementation to the javax.transaction package. However, my code is not having a type coupling on either leaving flexibility to the deployer. > > So we make a trade off between type decoupling and runtime complexity. This is the same trade off we make in IOC/DI engines, annotations, and factories. Right, I'm very familiar with this as a pattern and a design decision, but I'm missing how this can be blanket applied to modules. I would also throw out there that this is unlikely to work in practice. Let's use an example. Let's say you write a module to provide a user management system. It uses JPA only for persistence and does not use any hibernate or Eclipse link specific class, which BTW is highly unlikely. You write your module and then unit test it. Although it would be really neat if you could ship your module to the world and not really care what implementation of JPA is used, this is unlikely. In fact, whatever JPA implementation you unit tested against is probably the only one that you can definitely say your module will run with. Therefore, you would really need to unit test your module against all variations of JPA and that again is unlikely. In other cases, it might work though. For example SLF4J is probably a safe bet to code against the API and then be able to drop in any of the implementations. I guess my point is that it isn't always as simple as you are implying. > >> Again, are you talking about build time or runtime? Also, are you suggesting that Java determine your dependencies automatically including the version? Or is this merely to help developers find missing dependencies and remove unused dependencies? > If you compile class com.bar.Foo that refers to com.bar.Bar then the Foo.class and Bar.class file contain all their type dependencies and by implication their package dependencies. The version of the package can then for example be found from package-info.java. > > This technique is used extensively in bnd, an open source library used in Eclipse, IDEA, ant, maven, SBT, and others. It reads the class files in a JAR and generates the OSGi dependency information automatically. This is true unless there is reflection, which is common. JEE is based entirely on reflecting the top level entry points (servlets, filters, etc). There are many cases of configuration files defining implementations and transitive dependencies. This makes using a bytecode analyzer only a partial solution. -bp > > Hope this clarifies. Kind regards, > > Peter Kriens > > > >> >> -bp >> >> >> >> >> >> >>> >>> Kind regards, >>> >>> Peter Kriens >>> >>> >>> >>> >>> >>> >>> >>> On 15 nov. 2011, at 06:14, David M. Lloyd wrote: >>> >>>> On 11/14/2011 08:33 PM, Neil Bartlett wrote: >>>>> Hi David, >>>>> >>>>> More than happy to talk about reality, though Peter already was doing >>>>> so. Nothing he said was theoretical, it came directly from his >>>>> experiences and the experiences of developers who have been using OSGi >>>>> for around 12 years. >>>> >>>>> Before digging into the points in your message, I need to address the >>>>> subject line "Why package deps won't work". Package dependencies will >>>>> work and they do work, as is proven every day by the large number of >>>>> applications running on OSGi. To claim otherwise would be wilfully >>>>> ignorant, so I will assume you are being hyperbolic and really meant >>>>> to assert that module deps are just better than package deps. >>>>> Therefore I intend to argue on that basis and claim that package deps >>>>> are better than module deps; note that I don't claim that module >>>>> dependencies "won't work" because JBoss Modules is a proof point that >>>>> they do. >>>> >>>> There was an implied "... for Java SE 8 modules" after "won't work" (this is why I said "won't" rather than "don't") which apparently wasn't implied enough. >>>> >>>> Of course people use OSGi, or we wouldn't be arguing, but that alone isn't enough to make OSGi's dependency system preferable for the SE case (after all, far more people use plain old class paths). I believe that JBoss Modules is a very accurate representation of how an SE module system could function, right down to executing JARs on the command line with dependencies and modularizing certain JDK APIs. This is not a proof point, but strong evidence that it is an effective solution for the actual problem on the table, and that similar architectures are likely to succeed for the same problem domain. As to why package deps are not an example of an effective solution for this problem, I intend to illustrate in greater detail. >>>> >>>> Further responses and elaboration below. >>>> >>>>> On Mon, Nov 14, 2011 at 5:05 PM, David M. Lloyd wrote: >>>>>> The explanation is quite simple, really - each point can be pretty much >>>>>> wiped out by a heavy dose of reality. >>>>>> >>>>>> 1. "M2P is leverages the Java type system unlike m2m that must introduce new >>>>>> namespaces outside the Java type system." - this is just fluffy >>>>>> buzzwordology. Considering packages part of the Java type system is a >>>>>> pretty liberal interpretation of the term "type system" as they're basically >>>>>> just arbitrary name spaces. That said, there's nothing particularly "new" >>>>>> about namespacing modules. JARs have names. Projects have names. Maven >>>>>> uses its own namespaces for artifacts. Even the primitive JDK extension >>>>>> mechanism uses symbolic names. >>>>> >>>>> From Wikipedia (http://en.wikipedia.org/wiki/Type_system): "A type >>>>> system associates a type with each computed value ... the aim is to >>>>> prevent operations expecting a certain kind of value being used with >>>>> values for which that operation does not make sense." >>>>> >>>>> The following will not compile: >>>>> >>>>> import java.awt.List; >>>>> // ... >>>>> List list = new ArrayList(); >>>>> list.iterator(); // etc >>>>> >>>>> whereas the following will: >>>>> >>>>> import java.util.List; >>>>> // ... >>>>> List list = new ArrayList(); >>>>> list.iterator(); // etc >>>>> >>>>> Package names are part of the type system because using an incorrect >>>>> package name in Java source can result in type errors during >>>>> compilation, and because the type and available operations associated >>>>> with each value varies with the package name used. >>>>> >>>>> In contrast, it is impossible to obtain a type error at compilation by >>>>> incorrectly naming a JAR, project, Maven artefact etc, because none of >>>>> those things are ever referenced in Java source code. >>>> >>>> I stand by my statement. A package is part of the name of the type; it is not a type, nor a meta-type, nor anything like a type in and of itself. This makes them part of the language, sure, but not part of the type system, in my opinion. Thus, "a stretch". I don't feel any particular need to have the world agree with me on this though, other than as I state below... >>>> >>>>>> Even a simple convention of the parent-most package name for the name of a >>>>>> module is very simple to grasp and is in fact the solution we've used quite >>>>>> effectively thus far. Thus implying that module names are some new alien >>>>>> concept is really not valid. >>>>> >>>>> The claim is not that they are a new alien concept. The claim is that >>>>> they are not part of the Java language and type system, and as a >>>>> result of this (along with other problems) are less useful for >>>>> depending upon than packages. >>>> >>>> I don't see how not being part of the Java language makes module identifiers "less useful". I think that this argument artificially elevates the significance of the relationship between packages and the actual type system, and then uses that elevation as a proof that a system based on other namespace principles is deficient, without offering any real examples which actually relate to the type system. >>>> >>>>>> 2. "M2P can be used to break the transitive dependency chain, m2m suffers of >>>>>> excessive coupling." - How about some facts to back this up? I've found >>>>>> "m2m" coupling to be just right. In JBoss Modules, we do not export >>>>>> transitive dependencies by default. This results in a very simple and clean >>>>>> dependency graph between modules. Using package dependencies results in a >>>>>> *far more complex* dependency graph, because you need edges for every >>>>>> package even though *most of the time you want the whole module anyway*. >>>>>> Again, real world here. >>>>> >>>>> Peter is definitely also talking about the real world and so am I. In >>>>> an m2m dependency model you cannot avoid transitive dependencies, >>>>> whether you expose them to consumers or not. >>>> >>>> In *any* model you cannot avoid transitive dependencies. If module A exports module B as part of its API, then you get A and B, and that's life. If you only want part of B, then you filter it down. But with most existing libraries out there (and they easily number in the thousands), they're intended to be used whole, so this doesn't really happen too much in practice. >>>> >>>>> An importer of a module must be assumed to depend on the whole functionality of that module >>>>> whether or not that is actually the case, and therefore all of the >>>>> transitive dependencies must be present at runtime. In an m2p world we >>>>> have the opportunity to split modules and break apart dependency >>>>> graphs, because the unit of coupling is more granular. >>>> >>>> You could break things down by actual classes if granularity is the goal. But it isn't; it is so much less important than, say, performance, conceptual simplicity, usability, efficiency, memory footprint, etc. The fact is that the vast majority of modules you'll find published at, say Maven central, are designed to be used completely, not partially, and can and are used as such without issue. >>>> >>>> People just don't usually design modules to be imported by package, except perhaps as an afterthought concession to OSGi, and they don't use them that way at runtime either. It is so seldom a real problem in the SE world (or EE for that matter) once you step away from flat class paths. The simple fact is that most users think of their dependencies in the same terms that their IDEs and their build scripts do: by module. >>>> >>>> This is truly a case of creating a lot of complexity and extra work in the common case to avoid problems which are not a common case. I think it is much better to optimize for the common case, where people know exactly what dependencies they want (in terms of artifacts), and they just want to declare them and be done without (a) hunting down package lists or (b) running extra tooling. >>>> >>>>>> If we're just going to throw dogma around, I'll put it the other way: m2p is >>>>>> a design error by the OSGi spec designers which has since been embraced as a >>>>>> religion. >>>>> >>>>> Pejorative accusations about "dogma" or "religion" have no place in a >>>>> technical discussion. They are also untrue. All of the practising OSGi >>>>> developers I know have arrived at their support for package >>>>> dependencies as a result of real world experience, not because of some >>>>> willingness to bow down to the Almighty CPEG. I don't know of any >>>>> practising OSGi developer who has used both Require-Bundle (m2m) and >>>>> Import-Package (m2p) and actually prefers the former. >>>> >>>> Sure, but Require-Bundle is *not* the same thing as using module-to-module dependencies in a non-OSGi system. All OSGi life revolves around the resolver. In the SE world, you cannot have a resolver without introducing a lot of machinery which will definitely negatively impact boot performance in the case that the resolver runs at module load time, or add the complexity of centralized package index management in the case that the resolver would run at install time (as in, install into a module library/repository, not install in the OSGi sense). >>>> >>>> If in a typical installation a user has to do more than simply delete a file to remove a module, or drop a JAR into a directory to add a module, I believe we've run quite far off-track usability-wise. And if a user has to sit through a lengthy resolution process at runtime (even if only the first time) then we're off-track performance-wise, especially if such resolution must expand to include modules in the module repository which are not even loaded. >>>> >>>>> I do know several who started as strong supporters of Require-Bundle >>>>> and switched to being supporters of Import-Package, not because of >>>>> Peter's wrathful pontificating but because they encountered the >>>>> specific problems that he described and found that they were fixed by >>>>> using Import-Package, and indeed that everything worked so much more >>>>> cleanly that way. I'm in this camp myself, and so was Jeff McAffer >>>>> before he went over to the Dark Side (Microsoft). >>>> >>>> There are other, simpler ways to fix package conflict issues when they arise. Like simply excluding the offending package from the import. >>>> >>>> And as I've said before, I do not believe that Require-Bundle in the context of OSGi bundles is comparable to loading modules by name in the context of a module system such as Jigsaw or JBoss Modules and the types of modules that could be loaded thereby (the aforementioned thousands of existing modules). >>>> >>>>>> It offers no significant benefit, other than a couple of edge >>>>>> cases which are frankly just as well handled by m2m simply by adding package >>>>>> filters. Which, by the way, I haven't seen the need to do yet in our 200+ >>>>>> module environment, but which we do have the capability to do. >>>>> >>>>> Substitution, module refactoring and transitive decoupling are hardly >>>>> edge cases. >>>> >>>> Substitution isn't solely addressed by package dependencies. Nor is module refactoring. And additional transitive decoupling isn't something people actually want - as I said, the common case is that users want to be able to use any class from a module once they import that module. In the real world, it is very rare to only want to import part of a module! If you need this routinely, you've done something very unorthodox. Thus I stand by my statement. >>>> >>>>> However I can see how these issues might not yet have come >>>>> to the fore in a module system designed for a single product with a >>>>> small number of modules, and where that product has not yet been >>>>> through the mill of multiple version evolutions. >>>> >>>> Cute. I assure you that we have been around through a long, long history of a huge number of Java users doing everything under the sun to their class paths and class loading structures. We also have experience with operating system distribution and managing module distribution across a wide range of other programming languages, not just Java. Our module system has already stood up to reasonably large deployments (200+ modules) with excellent memory footprint and performance numbers and we've been processing usability feedback, including feedback from those with OSGi experience, which has been mostly positive. >>>> >>>> Furthermore when I was at JavaOne giving booth talks about JBoss Modules and AS7, I found that users expressed quite a bit of frustration at the OSGi model and were quite receptive to a more orthodox m2m system, for what that's worth. >>>> >>>>>> M2P is a solution just itching for a problem. >>>>> >>>>> This is also not a very useful statement, I assure you that the >>>>> problem came before the solution. Better to show why you think the >>>>> problem is invalid or should have been solved differently. >>>> >>>> There is no way you can convince me that this solution was invented with general modularity in mind. Back when OSGi was first formed, it was designed for the embedded market. The world of Java today is completely different. People just don't bundle their libraries by package, as was possibly expected in the early days of Java. If they did we wouldn't be having this discussion because packages and modules would already be one and the same. >>>> >>>> OSGi evolved to where it is today. It was not designed from the ground up with actual requirements which pertained to the problem at hand which is modularity of the SE platform and applications which run on it. So yeah I'm going to say, solution before problem. Taking a solution and trying to apply it retroactively to a different problem like this has never failed to bite me in the ass, personally. But I can't speak to everyone's experience. >>>> >>>>>> But you're going to have a >>>>>> tough time convincing me that users *want* to have to use special tooling >>>>>> because they want to depend on a module which has too many packages to list >>>>>> out by hand. And before you cry "wildcards", be sure to consider that some >>>>>> modules use package names which are subordinate to other modules' packages, >>>>>> which is a perfectly normal and allowable scenario. Using wildcards for >>>>>> package matching could cause amazing levels of havoc. >>>>> >>>>> OSGi never uses wildcards at runtime, and tools such as bnd do not >>>>> need wildcards in order to express package-level dependencies. They >>>>> extract the set of packages that were actually used by the code, all >>>>> of which are available in the class files. This is possible because >>>>> packages are part of the type system (see first point above). >>>>> >>>>> So it's not that I don't want to list dependencies by hand, rather I >>>>> only want to do it once. I am already forced to do it in the import >>>>> statements of my Java sources. If I had to repeat that dependency >>>>> information -- whether in m2p or m2m form -- then I would run the risk >>>>> of it getting out of step with the real dependencies. >>>> >>>> One of the critical logical gaps here which m2p ignores at a fundamental level is that *packages are not unique*. In the real world, packages are repeated across more than one module all the time, and not just due to versioning. There is *no way* you could take every JAR at Maven Central and plug it in to an m2p system. You'd have to break apart and repackage every single one, at least, and at worst you'd have to rewrite an awful lot of them under different package names to account for this restriction. You'd need to enforce package uniqueness across your whole library, for all time. I don't think this is a sane option. You're adding significant overhead to management, installation, and execution, for what? So you can have the privilege of needing a special tool to extract your dependencies for you? >>>> >>>>>> Using package dependencies means you either must have a master package index >>>>>> for linking, or you need a resolver which has to have analyzed every module >>>>>> you ever plan to load. Otherwise, O(1) loading of modules is impossible, >>>>>> which is absolutely 100% a deal-breaker for JDK modules which must be >>>>>> incrementally installable. And it forbids having packages with the same >>>>>> name in more than one JAR without bringing run-time versioning into the >>>>>> fold, which is a terrible, terrible can of worms. >>>>> >>>>> Could you please explain why O(1) is the only acceptable complexity >>>>> for installing modules. >>>> >>>> Because this is Java SE we're talking about. You're going to have a potentially huge number of modules installed in your system. Even if you get around the versioning issues somehow and fully externalize the index of all packages, you still have to load or traverse the centralized index for every module load. >>>> >>>> I say that not only is ?(1) the only acceptable complexity for installing modules but also loading them at run time. Linking modules at run time should be no worse than O(n) complexity for the number of dependencies the module has, including transitive dependencies. Loading classes and resources from modules should be ?(k) where k is the number of modules which contain a package or directory which matches the class or resource being loaded (almost always one, in my experience). >>>> >>>> Loading a module should normally be limited to a one-time O(1) disk access without searching or loading any other files or modules than the module artifact itself (and possibly its descriptor if they are held externally by a particular module repository implementation, which is useful but not necessary). Linking a module should be limited in disk access to loading only the modules it imports via direct dependency, or by the relatively rare partial or full re-export of a transitive dependency by a direct dependency. >>>> >>>> In particular, the expectation of accessing a central index has some potentially serious implications, facing either possible file lock contention by multiple threads or memory overhead of loading in the complete index in advance. It should be expected that modules are loaded concurrently, and such concurrency should not be hindered any more than necessary. Other possible implementations (such as file- or directory-per-package) have their own drawbacks as well or violate what I consider to be core requirements. >>>> >>>> If there are implementations of package-based resolution which don't involve either a centralized index of one or many files or a resolver which must rove all installed modules in advance, I'd like to hear about it. >>>> >>>>> OSGi does indeed support incremental install >>>>> and while I accept it is probably not O(1) for each module, it would >>>>> likely be no more than O(N), though I haven't done the maths yet to >>>>> prove this. Bear in mind that in practice, for small N, the constant >>>>> factors can result in O(1) being more expensive than O(N). I have seen >>>>> OSGi used with many thousands of modules, so unless you have some data >>>>> and a use-case showings package-level resolution as unacceptably slow, >>>>> your concern just sounds like premature optimisation. >>>> >>>> Identifying algorithmic complexity during design phase is *never* premature optimization AFAICT. If you don't understand the complexity of the algorithms you're about to implement, and how they are expected to be applied to the problem at hand, then you're not ready for implementation yet. >>>> >>>> However, postulating that O(n) is okay because can sometimes be faster than O(1) without measuring it for the specific problem in question is definitely premature de-optimization. :-) >>>> >>>>> There are two reasons to have a packages with the same name in more >>>>> than one JAR. The first is a situation called split packages, and it >>>>> is highly undesirable because it causes the runtime model of the >>>>> package to diverge from the compile-time model, and therefore things >>>>> like package-private types and members stop working correctly. For >>>>> this reason, OSGi's m2p imports support depending only upon a single >>>>> exporter of a particular package, i.e. we do not aggregate all exports >>>>> of that package. >>>>> >>>>> Unfortunately split packages are sometimes unavoidable in legacy code >>>>> that cannot be refactored, e.g. the JDK. To support such scenarios >>>>> OSGi has Require-Bundle, i.e. m2m. This does not negate the problems >>>>> associated with m2m, it is simply a trade-off that we face with poorly >>>>> factored legacy code. >>>>> >>>>> The second reason for multiple packages with the same name is when you >>>>> explicitly want to install multiple versions of a library/API and have >>>>> them all available within the same runtime. I wouldn't call this a >>>>> "can of worms" exactly because it can be done without too much >>>>> trouble, though for the sake of a simple life I personally avoid this >>>>> situation unless it's necessary. >>>> >>>> Yes, and it is actually fairly common in practice to want two versions of something in the sense of the two versions being wholly different implementations (think apache commons logging versus jcl-over-slf4j for a trivial example which crops up a lot). There are reasons to use one or the other. Multiplicity of versions (in this respect) is something the module system has to handle gracefully and simply. >>>> >>>>>> Finally it should be perfectly clear to anyone who has read the original >>>>>> requirements document that nothing in this module system should prevent OSGi >>>>>> from functioning as it is, so there is absolutely no reason to assume that >>>>>> any OSGi implementation is so threatened - especially if m2p linking is as >>>>>> superior as has been expressed. Our module system (which is conceptually >>>>>> similar to Jigsaw in many regards) in fact does support our OSGi >>>>>> implementation quite effectively without itself implementing OSGi's >>>>>> package-to-package resolution (which like I said throws O(1) out the >>>>>> window). >>>>> >>>>> I agree that Jigsaw's existence doesn't threaten OSGi's, so long as >>>>> Java 8 doesn't actually break OSGi (and if it did so, it would >>>>> likewise break many other applications and could not be considered >>>>> backwards compatible with Java 7). The two can interoperate through >>>>> m2m-type dependencies. Tim Ellison started Project Penrose for the >>>>> purpose of investigating, testing and deepening this collaboration. >>>>> >>>>> Neverthless, the point that I believe Glyn was making is the >>>>> following. We accept that m2m dependencies are probably required for >>>>> the JDK, which implies a module system like Jigsaw or >>>>> OSGi/Require-Bundle rather than OSGi/Import-Package. However is it >>>>> intended to be used for application modularisation as well? This is of >>>>> course a question for the Jigsaw team rather than you, David. >>>> >>>> The question of whether Java SE 8 modules are intended to be used for application modularization is a question for the EG, not the Jigsaw team. The job of the Jigsaw team is really to implement a prototype which meets the requirements set forth by the EG, which may become the reference implementation at a future point. >>>> >>>> The question of whether the greater Java community will embrace the SE module system for applications is to be answered by the community only. I personally believe that to develop a module system which is not intended to be usable by the average standalone application is foolhardy, a great waste of effort, and is doomed to mediocrity, given the success we have had with such a system. >>>> >>>> However until there is an EG, these are all just as much questions for me as for anybody, and I can and will answer to the best of my knowledge and belief. >>>> >>>>> As a result of experience in developing and evolving large real-world >>>>> applications using a module system that supports BOTH m2m and m2p >>>>> dependencies, I believe it would be very unfortunate if a module >>>>> system that supports ONLY m2m were to become widely used in the >>>>> application space... not because OSGi can't handle the competition, >>>>> but because those applications will be fragile and hard to evolve. >>>> >>>> I don't believe this to be the case. I think that the status quo doesn't result in particularly fragile applications, and such can be easy to evolve or hard depending on the quality of the application components and frameworks involved. I think that m2m dependencies enhance the status quo such that applications are somewhat less fragile (owing chiefly to the simple measure of preventing transitive dependencies from being exported by default), and quite easy to evolve as well (even large and complex frameworks rarely have more than a few dependencies, unless they themselves are overly fragmented (e.g. CXF as a nightmare example which yet still works fine under an m2m system)). >>>> >>>>> My question for you David is as follows. I understand that you prefer >>>>> module dependencies, but do you believe that package dependencies have >>>>> no value whatsoever and therefore should not be available to >>>>> application developers in the Java 8 module system? If so, why did Red >>>>> Hat create an OSGi implementation? >>>> >>>> I personally believe that they have some value, but that value is limited to interoperability with OSGi. I do not believe that it is a preferable model for most users or standalone applications, nor for larger applications such as our application server. I think that the requirement for extra tooling and the increased run-time complexity which is exposed to users nullifies the benefits. If someone really wants this variety of modularity, they should simply use OSGi. >>>> >>>> Red Hat created an OSGi implementation for many reasons, but the only significant one to me is that there are people who want to use their OSGi applications with our application server. I don't believe any other reason is even necessary. It's the same reason we strive to support any spec, from EJB 1.0 to Java EE 6 and beyond. Someone wants the functionality, so we deliver it to the best of our ability. >>>> >>>>> Kind regards >>>>> Neil >>>>> >>>>>> >>>>>> On 11/14/2011 01:49 AM, Glyn Normington wrote: >>>>>>> >>>>>>> I look forward to David's elaboration of why he thinks "using packages as >>>>>>> a dependency unit is a terrible idea" to balance Peter's clear explanation >>>>>>> of the benefits of m2p. >>>>>>> >>>>>>> Meanwhile, it's worth noting that, according to the requirements document, >>>>>>> Jigsaw is aimed at platform modularisation and the platform being >>>>>>> modularised has some non-optimal division of types across packages (see the >>>>>>> package subsets requirement) which favour m2m dependencies. (Note that >>>>>>> Apache Harmony was developed with modularity in mind and was able to exploit >>>>>>> m2p, so platform modularisation per se needn't be limited to m2m.) >>>>>>> >>>>>>> So if Jigsaw excludes m2p, it will then be applicable to certain kinds of >>>>>>> legacy code modularisation and less applicable to new module development and >>>>>>> modularisation of existing code whose division into packages suits m2p. IIRC >>>>>>> this was the original positioning of Jigsaw: for use primarily within the >>>>>>> OpenJDK codebase and only exposed for application use because it was too >>>>>>> inconvenient to hide it. >>>>>>> >>>>>>> Regards, >>>>>>> Glyn >>>>>>> >>>>>>> On 12 Nov 2011, at 11:59, Peter Kriens wrote: >>>>>>> >>>>>>>> Neither my wrath, nor the fact that I rarely if ever get angry is >>>>>>>> relevant in this discussion ... This is a technical argument that are >>>>>>>> solvable by technical people that share the same goals. I prefer package >>>>>>>> dependencies because they address the excessive type coupling problem in >>>>>>>> object oriented systems, not because they're part of OSGi. Let me argue my >>>>>>>> case. >>>>>>>> >>>>>>>> Module-to-package dependencies (m2p) are preferable over module-to-module >>>>>>>> dependencies (m2m) for many reasons but these are the most important >>>>>>>> reasons: >>>>>>>> >>>>>>>> M2P is leverages the Java type system unlike m2m that must introduce new >>>>>>>> namespaces outside the Java type system. >>>>>>>> M2P can be used to break the transitive dependency chain, m2m suffers of >>>>>>>> excessive coupling >>>>>>>> >>>>>>>> Since the first bullet's benefit should be clear I only argue the more >>>>>>>> complex second bullet. >>>>>>>> >>>>>>>> A module is in many regards like a class. A class encapsulates members, >>>>>>>> depends on other members/classes, and makes a few members accessible outside >>>>>>>> the class. A module has a similar structure but then with types/packages as >>>>>>>> members. >>>>>>>> >>>>>>>> After the initial success of Object Oriented Programming (OO) it was >>>>>>>> quickly learned that reuse did not take place at the expected scale due to >>>>>>>> excessive type coupling. The problem was that a class aggregated many >>>>>>>> dependencies to simplify its implementation but these dependencies were >>>>>>>> unrelated to the contract it implemented. Since class dependencies are >>>>>>>> transitive most applications disappointingly became an almost fully >>>>>>>> connected graph. >>>>>>>> >>>>>>>> Java's great innovation was the interface because it broke both the >>>>>>>> transitivity and aggregation of dependencies. A class could now express its >>>>>>>> dependency (use or implement) on a contract (the interface) and was >>>>>>>> therefore fully type decoupled from the opposite site. >>>>>>>> >>>>>>>> An interface can act as a contract because it names the signature of a >>>>>>>> set of methods so that the compiler can verify the client and the >>>>>>>> implementer. >>>>>>>> >>>>>>>> Since a module has a very similar structure to a class it suffers from >>>>>>>> exactly the same transitive aggregation of dependencies. This is not a >>>>>>>> theory, look at the experiences with Maven >>>>>>>> (http://www.sonatype.com/people/2011/04/how-not-to-download-the-internet/) >>>>>>>> Again, this is not that maven is bad or developers are stupid, it is the >>>>>>>> same underlying force that finally resulted in the Java interface. >>>>>>>> >>>>>>>> The parallel for the class' interface for modules is a named set of >>>>>>>> interfaces. This concept already exists in Java: a package. Looking at >>>>>>>> almost all JSRs it is clear that our industry already uses packages as >>>>>>>> "interfaces" to provider implementations. >>>>>>>> >>>>>>>> Therefore, just like a class should not depend on other implementation >>>>>>>> types, a module should preferably not depend on other modules. A module >>>>>>>> should instead depend on contracts. Since modules will be used to provide >>>>>>>> components from different sources managed with different life cycles the >>>>>>>> excessive type coupling caused by m2m is even more damaging than in c2c. >>>>>>>> Proper use of m2p creates significantly less type coupled systems than m2m, >>>>>>>> the benefits should be obvious. >>>>>>>> >>>>>>>> Since there are use cases for m2m (non-type safe languages for example) I >>>>>>>> do believe that Jigsaw should still support m2m. However, it would be >>>>>>>> greatly beneficial to our industry if we could take advantage of the lessons >>>>>>>> learned with the Java interface and realize how surprisingly important the >>>>>>>> Java package actually is in our eco system. >>>>>>>> >>>>>>>> Kind regards, >>>>>>>> >>>>>>>> Peter Kriens >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 9 nov. 2011, at 15:04, David M. Lloyd wrote: >>>>>>>> >>>>>>>>> I'll just state now that using packages as a dependency unit is a >>>>>>>>> terrible idea, and not some architectural revelation. That way, Peter's >>>>>>>>> wrath will be largely directed at me. :-) >>>>>>>>> >>>>>>>>> On 11/09/2011 08:02 AM, Peter Kriens wrote: >>>>>>>>>> >>>>>>>>>> I agree that tools are needed but we must be careful to not expect >>>>>>>>>> tools to stopgap an architectural issue. I think it is important to first do >>>>>>>>>> good architectural design leveraging existing tools (e.g. the Java type >>>>>>>>>> system) before you try to add new tools. It is such a pity (but all to >>>>>>>>>> common) that a design allows for classes of errors that would be impossible >>>>>>>>>> with a slightly different design. >>>>>>>>>> >>>>>>>>>> Kind regards, >>>>>>>>>> >>>>>>>>>> Peter Kriens >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 9 nov. 2011, at 14:49, Alan Bateman wrote: >>>>>>>>>> >>>>>>>>>>> On 09/11/2011 13:04, Peter Kriens wrote: >>>>>>>>>>>> >>>>>>>>>>>> The issue is that maven problems are not caused because maven is bad >>>>>>>>>>>> or that pom authors are stupid. The reason is that the module-to-module >>>>>>>>>>>> dependency architecture in maven (and Jigsaw) is error prone ... >>>>>>>>>>> >>>>>>>>>>> This thread started out with someone asking about adding module >>>>>>>>>>> declarations to existing JAR files, and in that context, I agree it can be >>>>>>>>>>> error prone without good tools. I think things should be a lot better when >>>>>>>>>>> modules are compiled. >>>>>>>>>>> >>>>>>>>>>> -Alan. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> - DML >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> - DML >>>>>> >>>> >>>> >>>> -- >>>> - DML >>> >> > From njbartlett at gmail.com Wed Nov 16 15:03:03 2011 From: njbartlett at gmail.com (Neil Bartlett) Date: Wed, 16 Nov 2011 23:03:03 +0000 Subject: Why package deps work (Was: Re: Converting plain JARs to Java modules) In-Reply-To: References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> <67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz> <4EC14A53.7070907@redhat.com> <4EC1F53D.7040309@redhat.com> <328B38A3-EB53-4AD2-970B-CE84F5C26F96@aQute.biz> <9B28CE0A-85DC-4F53-8A88-5830759DA4BC@pontarelli.com> Message-ID: Hi Brian, On Wed, Nov 16, 2011 at 9:16 PM, Brian Pontarelli wrote: > [snip] > Let's use an example. Let's say you write a module to provide a user management system. It uses JPA only for persistence ?and does not use any hibernate or Eclipse link specific class, which BTW is highly unlikely. You write your module and then unit test it. How is this unlikely? On the contrary, it's highly unlikely that it *could* use any Hibernate or EclipseLink specific class. First, the module would be compiled with visibility only of the abstract API, i.e. javax.persistence, and then at runtime we employ the module system to ensure that it cannot see any specific implementation classes, even via reflection. In fact if we expect our modules to be reusable -- so they can be used not only in the project we are working on right now but in subsequent projects over the next several years -- it's very likely that the module will be used with multiple JPA providers. That's not to say we switch JPA providers willy nilly, or without running a QA process over the set of module that we intend to put into production. We also don't claim that we never find and fix bugs in our modules when used against alternative providers. > Although it would be really neat if you could ship your module to the world and not really care what implementation of JPA is used, this is unlikely. In fact, whatever JPA implementation you unit tested against is probably the only one that you can definitely say your module will run with. Therefore, you would really need to unit test your module against all variations of JPA and that again is unlikely. The developer of the module should NOT care what implementation of JPA they use, so long as it is an implementation that complies with the contract. The deployer or assembler of an application is responsible for finding and providing a compliant implementation. Testing of course is done against a specific implementation, and does not guarantee that other implementations can be used; they would have to be tested by whoever wanted to use them. Nevertheless the work required to fix issues arising from differences in interpretation of contracts is much less than the porting effort required if the original module developer had coded directly against an implementation. It's important to stress that none of this is theoretical. Exactly this scenario involving JPA, along with many other similar scenarios, can be done and is done in real world practice. > In other cases, it might work though. For example SLF4J is probably a safe bet to code against the API and then be able to drop in any of the implementations. This seems to contradict your previous statements. What is special about SLF4J? If anything I would expect the JPA example to "safer", since the JPA API was designed very cautiously by a JCP Expert Group, whereas SLF4J came from a single author in the open source world (I intend no slight against the SLF4J author; I simply have no information on the quality and stability of its API). > I guess my point is that it isn't always as simple as you are implying. I agree it is not always simple, but it is possible. Regards Neil > > >> >>> Again, are you talking about build time or runtime? Also, are you suggesting that Java determine your dependencies automatically including the version? Or is this merely to help developers find missing dependencies and remove unused dependencies? >> If you compile class com.bar.Foo that refers to com.bar.Bar then the Foo.class and Bar.class file contain all their type dependencies and by implication their package dependencies. The version of the package can then for example be found from package-info.java. >> >> This technique is used extensively in bnd, an open source library used in Eclipse, IDEA, ant, maven, SBT, and others. It reads the class files in a JAR and generates the OSGi dependency information automatically. > > This is true unless there is reflection, which is common. JEE is based entirely on reflecting the top level entry points (servlets, filters, etc). There are many cases of configuration files defining implementations and transitive dependencies. This makes using a bytecode analyzer only a partial solution. > > -bp > > > > >> >> Hope this clarifies. Kind regards, >> >> ? ? ? Peter Kriens >> >> >> >>> >>> -bp >>> >>> >>> >>> >>> >>> >>>> >>>> Kind regards, >>>> >>>> Peter Kriens >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On 15 nov. 2011, at 06:14, David M. Lloyd wrote: >>>> >>>>> On 11/14/2011 08:33 PM, Neil Bartlett wrote: >>>>>> Hi David, >>>>>> >>>>>> More than happy to talk about reality, though Peter already was doing >>>>>> so. Nothing he said was theoretical, it came directly from his >>>>>> experiences and the experiences of developers who have been using OSGi >>>>>> for around 12 years. >>>>> >>>>>> Before digging into the points in your message, I need to address the >>>>>> subject line "Why package deps won't work". Package dependencies will >>>>>> work and they do work, as is proven every day by the large number of >>>>>> applications running on OSGi. To claim otherwise would be wilfully >>>>>> ignorant, so I will assume you are being hyperbolic and really meant >>>>>> to assert that module deps are just better than package deps. >>>>>> Therefore I intend to argue on that basis and claim that package deps >>>>>> are better than module deps; note that I don't claim that module >>>>>> dependencies "won't work" because JBoss Modules is a proof point that >>>>>> they do. >>>>> >>>>> There was an implied "... for Java SE 8 modules" after "won't work" (this is why I said "won't" rather than "don't") which apparently wasn't implied enough. >>>>> >>>>> Of course people use OSGi, or we wouldn't be arguing, but that alone isn't enough to make OSGi's dependency system preferable for the SE case (after all, far more people use plain old class paths). ?I believe that JBoss Modules is a very accurate representation of how an SE module system could function, right down to executing JARs on the command line with dependencies and modularizing certain JDK APIs. ?This is not a proof point, but strong evidence that it is an effective solution for the actual problem on the table, and that similar architectures are likely to succeed for the same problem domain. ?As to why package deps are not an example of an effective solution for this problem, I intend to illustrate in greater detail. >>>>> >>>>> Further responses and elaboration below. >>>>> >>>>>> On Mon, Nov 14, 2011 at 5:05 PM, David M. Lloyd ?wrote: >>>>>>> The explanation is quite simple, really - each point can be pretty much >>>>>>> wiped out by a heavy dose of reality. >>>>>>> >>>>>>> 1. "M2P is leverages the Java type system unlike m2m that must introduce new >>>>>>> namespaces outside the Java type system." - this is just fluffy >>>>>>> buzzwordology. ?Considering packages part of the Java type system is a >>>>>>> pretty liberal interpretation of the term "type system" as they're basically >>>>>>> just arbitrary name spaces. ?That said, there's nothing particularly "new" >>>>>>> about namespacing modules. ?JARs have names. Projects have names. ?Maven >>>>>>> uses its own namespaces for artifacts. ?Even the primitive JDK extension >>>>>>> mechanism uses symbolic names. >>>>>> >>>>>> From Wikipedia (http://en.wikipedia.org/wiki/Type_system): "A type >>>>>> system associates a type with each computed value ... ?the aim is to >>>>>> prevent operations expecting a certain kind of value being used with >>>>>> values for which that operation does not make sense." >>>>>> >>>>>> The following will not compile: >>>>>> >>>>>> ? import java.awt.List; >>>>>> ? // ... >>>>>> ? List list = new ArrayList(); >>>>>> ? list.iterator(); // etc >>>>>> >>>>>> whereas the following will: >>>>>> >>>>>> ? import java.util.List; >>>>>> ? // ... >>>>>> ? List list = new ArrayList(); >>>>>> ? list.iterator(); // etc >>>>>> >>>>>> Package names are part of the type system because using an incorrect >>>>>> package name in Java source can result in type errors during >>>>>> compilation, and because the type and available operations associated >>>>>> with each value varies with the package name used. >>>>>> >>>>>> In contrast, it is impossible to obtain a type error at compilation by >>>>>> incorrectly naming a JAR, project, Maven artefact etc, because none of >>>>>> those things are ever referenced in Java source code. >>>>> >>>>> I stand by my statement. ?A package is part of the name of the type; it is not a type, nor a meta-type, nor anything like a type in and of itself. ?This makes them part of the language, sure, but not part of the type system, in my opinion. ?Thus, "a stretch". ?I don't feel any particular need to have the world agree with me on this though, other than as I state below... >>>>> >>>>>>> Even a simple convention of the parent-most package name for the name of a >>>>>>> module is very simple to grasp and is in fact the solution we've used quite >>>>>>> effectively thus far. ?Thus implying that module names are some new alien >>>>>>> concept is really not valid. >>>>>> >>>>>> The claim is not that they are a new alien concept. The claim is that >>>>>> they are not part of the Java language and type system, and as a >>>>>> result of this (along with other problems) are less useful for >>>>>> depending upon than packages. >>>>> >>>>> I don't see how not being part of the Java language makes module identifiers "less useful". ?I think that this argument artificially elevates the significance of the relationship between packages and the actual type system, and then uses that elevation as a proof that a system based on other namespace principles is deficient, without offering any real examples which actually relate to the type system. >>>>> >>>>>>> 2. "M2P can be used to break the transitive dependency chain, m2m suffers of >>>>>>> excessive coupling." - How about some facts to back this up? ?I've found >>>>>>> "m2m" coupling to be just right. ?In JBoss Modules, we do not export >>>>>>> transitive dependencies by default. ?This results in a very simple and clean >>>>>>> dependency graph between modules. ?Using package dependencies results in a >>>>>>> *far more complex* dependency graph, because you need edges for every >>>>>>> package even though *most of the time you want the whole module anyway*. >>>>>>> Again, real world here. >>>>>> >>>>>> Peter is definitely also talking about the real world and so am I. In >>>>>> an m2m dependency model you cannot avoid transitive dependencies, >>>>>> whether you expose them to consumers or not. >>>>> >>>>> In *any* model you cannot avoid transitive dependencies. ?If module A exports module B as part of its API, then you get A and B, and that's life. ?If you only want part of B, then you filter it down. ?But with most existing libraries out there (and they easily number in the thousands), they're intended to be used whole, so this doesn't really happen too much in practice. >>>>> >>>>>> An importer of a module must be assumed to depend on the whole functionality of that module >>>>>> whether or not that is actually the case, and therefore all of the >>>>>> transitive dependencies must be present at runtime. In an m2p world we >>>>>> have the opportunity to split modules and break apart dependency >>>>>> graphs, because the unit of coupling is more granular. >>>>> >>>>> You could break things down by actual classes if granularity is the goal. ?But it isn't; it is so much less important than, say, performance, conceptual simplicity, usability, efficiency, memory footprint, etc. ?The fact is that the vast majority of modules you'll find published at, say Maven central, are designed to be used completely, not partially, and can and are used as such without issue. >>>>> >>>>> People just don't usually design modules to be imported by package, except perhaps as an afterthought concession to OSGi, and they don't use them that way at runtime either. ?It is so seldom a real problem in the SE world (or EE for that matter) once you step away from flat class paths. ?The simple fact is that most users think of their dependencies in the same terms that their IDEs and their build scripts do: by module. >>>>> >>>>> This is truly a case of creating a lot of complexity and extra work in the common case to avoid problems which are not a common case. ?I think it is much better to optimize for the common case, where people know exactly what dependencies they want (in terms of artifacts), and they just want to declare them and be done without (a) hunting down package lists or (b) running extra tooling. >>>>> >>>>>>> If we're just going to throw dogma around, I'll put it the other way: m2p is >>>>>>> a design error by the OSGi spec designers which has since been embraced as a >>>>>>> religion. >>>>>> >>>>>> Pejorative accusations about "dogma" or "religion" have no place in a >>>>>> technical discussion. They are also untrue. All of the practising OSGi >>>>>> developers I know have arrived at their support for package >>>>>> dependencies as a result of real world experience, not because of some >>>>>> willingness to bow down to the Almighty CPEG. I don't know of any >>>>>> practising OSGi developer who has used both Require-Bundle (m2m) and >>>>>> Import-Package (m2p) and actually prefers the former. >>>>> >>>>> Sure, but Require-Bundle is *not* the same thing as using module-to-module dependencies in a non-OSGi system. ?All OSGi life revolves around the resolver. ?In the SE world, you cannot have a resolver without introducing a lot of machinery which will definitely negatively impact boot performance in the case that the resolver runs at module load time, or add the complexity of centralized package index management in the case that the resolver would run at install time (as in, install into a module library/repository, not install in the OSGi sense). >>>>> >>>>> If in a typical installation a user has to do more than simply delete a file to remove a module, or drop a JAR into a directory to add a module, I believe we've run quite far off-track usability-wise. ?And if a user has to sit through a lengthy resolution process at runtime (even if only the first time) then we're off-track performance-wise, especially if such resolution must expand to include modules in the module repository which are not even loaded. >>>>> >>>>>> I do know several who started as strong supporters of Require-Bundle >>>>>> and switched to being supporters of Import-Package, not because of >>>>>> Peter's wrathful pontificating but because they encountered the >>>>>> specific problems that he described and found that they were fixed by >>>>>> using Import-Package, and indeed that everything worked so much more >>>>>> cleanly that way. I'm in this camp myself, and so was Jeff McAffer >>>>>> before he went over to the Dark Side (Microsoft). >>>>> >>>>> There are other, simpler ways to fix package conflict issues when they arise. ?Like simply excluding the offending package from the import. >>>>> >>>>> And as I've said before, I do not believe that Require-Bundle in the context of OSGi bundles is comparable to loading modules by name in the context of a module system such as Jigsaw or JBoss Modules and the types of modules that could be loaded thereby (the aforementioned thousands of existing modules). >>>>> >>>>>>> It offers no significant benefit, other than a couple of edge >>>>>>> cases which are frankly just as well handled by m2m simply by adding package >>>>>>> filters. ?Which, by the way, I haven't seen the need to do yet in our 200+ >>>>>>> module environment, but which we do have the capability to do. >>>>>> >>>>>> Substitution, module refactoring and transitive decoupling are hardly >>>>>> edge cases. >>>>> >>>>> Substitution isn't solely addressed by package dependencies. ?Nor is module refactoring. ?And additional transitive decoupling isn't something people actually want - as I said, the common case is that users want to be able to use any class from a module once they import that module. ?In the real world, it is very rare to only want to import part of a module! ?If you need this routinely, you've done something very unorthodox. ?Thus I stand by my statement. >>>>> >>>>>> However I can see how these issues might not yet have come >>>>>> to the fore in a module system designed for a single product with a >>>>>> small number of modules, and where that product has not yet been >>>>>> through the mill of multiple version evolutions. >>>>> >>>>> Cute. ?I assure you that we have been around through a long, long history of a huge number of Java users doing everything under the sun to their class paths and class loading structures. ?We also have experience with operating system distribution and managing module distribution across a wide range of other programming languages, not just Java. ?Our module system has already stood up to reasonably large deployments (200+ modules) with excellent memory footprint and performance numbers and we've been processing usability feedback, including feedback from those with OSGi experience, which has been mostly positive. >>>>> >>>>> Furthermore when I was at JavaOne giving booth talks about JBoss Modules and AS7, I found that users expressed quite a bit of frustration at the OSGi model and were quite receptive to a more orthodox m2m system, for what that's worth. >>>>> >>>>>>> M2P is a solution just itching for a problem. >>>>>> >>>>>> This is also not a very useful statement, I assure you that the >>>>>> problem came before the solution. Better to show why you think the >>>>>> problem is invalid or should have been solved differently. >>>>> >>>>> There is no way you can convince me that this solution was invented with general modularity in mind. ?Back when OSGi was first formed, it was designed for the embedded market. ?The world of Java today is completely different. ?People just don't bundle their libraries by package, as was possibly expected in the early days of Java. ?If they did we wouldn't be having this discussion because packages and modules would already be one and the same. >>>>> >>>>> OSGi evolved to where it is today. ?It was not designed from the ground up with actual requirements which pertained to the problem at hand which is modularity of the SE platform and applications which run on it. ?So yeah I'm going to say, solution before problem. ?Taking a solution and trying to apply it retroactively to a different problem like this has never failed to bite me in the ass, personally. ?But I can't speak to everyone's experience. >>>>> >>>>>>> But you're going to have a >>>>>>> tough time convincing me that users *want* to have to use special tooling >>>>>>> because they want to depend on a module which has too many packages to list >>>>>>> out by hand. ?And before you cry "wildcards", be sure to consider that some >>>>>>> modules use package names which are subordinate to other modules' packages, >>>>>>> which is a perfectly normal and allowable scenario. ?Using wildcards for >>>>>>> package matching could cause amazing levels of havoc. >>>>>> >>>>>> OSGi never uses wildcards at runtime, and tools such as bnd do not >>>>>> need wildcards in order to express package-level dependencies. They >>>>>> extract the set of packages that were actually used by the code, all >>>>>> of which are available in the class files. This is possible because >>>>>> packages are part of the type system (see first point above). >>>>>> >>>>>> So it's not that I don't want to list dependencies by hand, rather I >>>>>> only want to do it once. I am already forced to do it in the import >>>>>> statements of my Java sources. If I had to repeat that dependency >>>>>> information -- whether in m2p or m2m form -- then I would run the risk >>>>>> of it getting out of step with the real dependencies. >>>>> >>>>> One of the critical logical gaps here which m2p ignores at a fundamental level is that *packages are not unique*. ?In the real world, packages are repeated across more than one module all the time, and not just due to versioning. ?There is *no way* you could take every JAR at Maven Central and plug it in to an m2p system. ?You'd have to break apart and repackage every single one, at least, and at worst you'd have to rewrite an awful lot of them under different package names to account for this restriction. ?You'd need to enforce package uniqueness across your whole library, for all time. ?I don't think this is a sane option. ?You're adding significant overhead to management, installation, and execution, for what? ?So you can have the privilege of needing a special tool to extract your dependencies for you? >>>>> >>>>>>> Using package dependencies means you either must have a master package index >>>>>>> for linking, or you need a resolver which has to have analyzed every module >>>>>>> you ever plan to load. ?Otherwise, O(1) loading of modules is impossible, >>>>>>> which is absolutely 100% a deal-breaker for JDK modules which must be >>>>>>> incrementally installable. ?And it forbids having packages with the same >>>>>>> name in more than one JAR without bringing run-time versioning into the >>>>>>> fold, which is a terrible, terrible can of worms. >>>>>> >>>>>> Could you please explain why O(1) is the only acceptable complexity >>>>>> for installing modules. >>>>> >>>>> Because this is Java SE we're talking about. ?You're going to have a potentially huge number of modules installed in your system. ?Even if you get around the versioning issues somehow and fully externalize the index of all packages, you still have to load or traverse the centralized index for every module load. >>>>> >>>>> I say that not only is ?(1) the only acceptable complexity for installing modules but also loading them at run time. ?Linking modules at run time should be no worse than O(n) complexity for the number of dependencies the module has, including transitive dependencies. ?Loading classes and resources from modules should be ?(k) where k is the number of modules which contain a package or directory which matches the class or resource being loaded (almost always one, in my experience). >>>>> >>>>> Loading a module should normally be limited to a one-time O(1) disk access without searching or loading any other files or modules than the module artifact itself (and possibly its descriptor if they are held externally by a particular module repository implementation, which is useful but not necessary). ?Linking a module should be limited in disk access to loading only the modules it imports via direct dependency, or by the relatively rare partial or full re-export of a transitive dependency by a direct dependency. >>>>> >>>>> In particular, the expectation of accessing a central index has some potentially serious implications, facing either possible file lock contention by multiple threads or memory overhead of loading in the complete index in advance. ?It should be expected that modules are loaded concurrently, and such concurrency should not be hindered any more than necessary. ?Other possible implementations (such as file- or directory-per-package) have their own drawbacks as well or violate what I consider to be core requirements. >>>>> >>>>> If there are implementations of package-based resolution which don't involve either a centralized index of one or many files or a resolver which must rove all installed modules in advance, I'd like to hear about it. >>>>> >>>>>> OSGi does indeed support incremental install >>>>>> and while I accept it is probably not O(1) for each module, it would >>>>>> likely be no more than O(N), though I haven't done the maths yet to >>>>>> prove this. Bear in mind that in practice, for small N, the constant >>>>>> factors can result in O(1) being more expensive than O(N). I have seen >>>>>> OSGi used with many thousands of modules, so unless you have some data >>>>>> and a use-case showings package-level resolution as unacceptably slow, >>>>>> your concern just sounds like premature optimisation. >>>>> >>>>> Identifying algorithmic complexity during design phase is *never* premature optimization AFAICT. ?If you don't understand the complexity of the algorithms you're about to implement, and how they are expected to be applied to the problem at hand, then you're not ready for implementation yet. >>>>> >>>>> However, postulating that O(n) is okay because can sometimes be faster than O(1) without measuring it for the specific problem in question is definitely premature de-optimization. :-) >>>>> >>>>>> There are two reasons to have a packages with the same name in more >>>>>> than one JAR. The first is a situation called split packages, and it >>>>>> is highly undesirable because it causes the runtime model of the >>>>>> package to diverge from the compile-time model, and therefore things >>>>>> like package-private types and members stop working correctly. For >>>>>> this reason, OSGi's m2p imports support depending only upon a single >>>>>> exporter of a particular package, i.e. we do not aggregate all exports >>>>>> of that package. >>>>>> >>>>>> Unfortunately split packages are sometimes unavoidable in legacy code >>>>>> that cannot be refactored, e.g. the JDK. To support such scenarios >>>>>> OSGi has Require-Bundle, i.e. m2m. This does not negate the problems >>>>>> associated with m2m, it is simply a trade-off that we face with poorly >>>>>> factored legacy code. >>>>>> >>>>>> The second reason for multiple packages with the same name is when you >>>>>> explicitly want to install multiple versions of a library/API and have >>>>>> them all available within the same runtime. I wouldn't call this a >>>>>> "can of worms" exactly because it can be done without too much >>>>>> trouble, though for the sake of a simple life I personally avoid this >>>>>> situation unless it's necessary. >>>>> >>>>> Yes, and it is actually fairly common in practice to want two versions of something in the sense of the two versions being wholly different implementations (think apache commons logging versus jcl-over-slf4j for a trivial example which crops up a lot). ?There are reasons to use one or the other. ?Multiplicity of versions (in this respect) is something the module system has to handle gracefully and simply. >>>>> >>>>>>> Finally it should be perfectly clear to anyone who has read the original >>>>>>> requirements document that nothing in this module system should prevent OSGi >>>>>>> from functioning as it is, so there is absolutely no reason to assume that >>>>>>> any OSGi implementation is so threatened - especially if m2p linking is as >>>>>>> superior as has been expressed. ?Our module system (which is conceptually >>>>>>> similar to Jigsaw in many regards) in fact does support our OSGi >>>>>>> implementation quite effectively without itself implementing OSGi's >>>>>>> package-to-package resolution (which like I said throws O(1) out the >>>>>>> window). >>>>>> >>>>>> I agree that Jigsaw's existence doesn't threaten OSGi's, so long as >>>>>> Java 8 doesn't actually break OSGi (and if it did so, it would >>>>>> likewise break many other applications and could not be considered >>>>>> backwards compatible with Java 7). ?The two can interoperate through >>>>>> m2m-type dependencies. Tim Ellison started Project Penrose for the >>>>>> purpose of investigating, testing and deepening this collaboration. >>>>>> >>>>>> Neverthless, the point that I believe Glyn was making is the >>>>>> following. We accept that m2m dependencies are probably required for >>>>>> the JDK, which implies a module system like Jigsaw or >>>>>> OSGi/Require-Bundle rather than OSGi/Import-Package. However is it >>>>>> intended to be used for application modularisation as well? This is of >>>>>> course a question for the Jigsaw team rather than you, David. >>>>> >>>>> The question of whether Java SE 8 modules are intended to be used for application modularization is a question for the EG, not the Jigsaw team. ?The job of the Jigsaw team is really to implement a prototype which meets the requirements set forth by the EG, which may become the reference implementation at a future point. >>>>> >>>>> The question of whether the greater Java community will embrace the SE module system for applications is to be answered by the community only. ?I personally believe that to develop a module system which is not intended to be usable by the average standalone application is foolhardy, a great waste of effort, and is doomed to mediocrity, given the success we have had with such a system. >>>>> >>>>> However until there is an EG, these are all just as much questions for me as for anybody, and I can and will answer to the best of my knowledge and belief. >>>>> >>>>>> As a result of experience in developing and evolving large real-world >>>>>> applications using a module system that supports BOTH m2m and m2p >>>>>> dependencies, I believe it would be very unfortunate if a module >>>>>> system that supports ONLY m2m were to become widely used in the >>>>>> application space... not because OSGi can't handle the competition, >>>>>> but because those applications will be fragile and hard to evolve. >>>>> >>>>> I don't believe this to be the case. ?I think that the status quo doesn't result in particularly fragile applications, and such can be easy to evolve or hard depending on the quality of the application components and frameworks involved. ?I think that m2m dependencies enhance the status quo such that applications are somewhat less fragile (owing chiefly to the simple measure of preventing transitive dependencies from being exported by default), and quite easy to evolve as well (even large and complex frameworks rarely have more than a few dependencies, unless they themselves are overly fragmented (e.g. CXF as a nightmare example which yet still works fine under an m2m system)). >>>>> >>>>>> My question for you David is as follows. I understand that you prefer >>>>>> module dependencies, but do you believe that package dependencies have >>>>>> no value whatsoever and therefore should not be available to >>>>>> application developers in the Java 8 module system? If so, why did Red >>>>>> Hat create an OSGi implementation? >>>>> >>>>> I personally believe that they have some value, but that value is limited to interoperability with OSGi. ?I do not believe that it is a preferable model for most users or standalone applications, nor for larger applications such as our application server. ?I think that the requirement for extra tooling and the increased run-time complexity which is exposed to users nullifies the benefits. ?If someone really wants this variety of modularity, they should simply use OSGi. >>>>> >>>>> Red Hat created an OSGi implementation for many reasons, but the only significant one to me is that there are people who want to use their OSGi applications with our application server. ?I don't believe any other reason is even necessary. ?It's the same reason we strive to support any spec, from EJB 1.0 to Java EE 6 and beyond. ?Someone wants the functionality, so we deliver it to the best of our ability. >>>>> >>>>>> Kind regards >>>>>> Neil >>>>>> >>>>>>> >>>>>>> On 11/14/2011 01:49 AM, Glyn Normington wrote: >>>>>>>> >>>>>>>> I look forward to David's elaboration of why he thinks "using packages as >>>>>>>> a dependency unit is a terrible idea" to balance Peter's clear explanation >>>>>>>> of the benefits of m2p. >>>>>>>> >>>>>>>> Meanwhile, it's worth noting that, according to the requirements document, >>>>>>>> Jigsaw is aimed at platform modularisation and the platform being >>>>>>>> modularised has some non-optimal division of types across packages (see the >>>>>>>> package subsets requirement) which favour m2m dependencies. (Note that >>>>>>>> Apache Harmony was developed with modularity in mind and was able to exploit >>>>>>>> m2p, so platform modularisation per se needn't be limited to m2m.) >>>>>>>> >>>>>>>> So if Jigsaw excludes m2p, it will then be applicable to certain kinds of >>>>>>>> legacy code modularisation and less applicable to new module development and >>>>>>>> modularisation of existing code whose division into packages suits m2p. IIRC >>>>>>>> this was the original positioning of Jigsaw: for use primarily within the >>>>>>>> OpenJDK codebase and only exposed for application use because it was too >>>>>>>> inconvenient to hide it. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Glyn >>>>>>>> >>>>>>>> On 12 Nov 2011, at 11:59, Peter Kriens wrote: >>>>>>>> >>>>>>>>> Neither my wrath, nor the fact that I rarely if ever get angry is >>>>>>>>> relevant in this discussion ... This is a technical argument that are >>>>>>>>> solvable by technical people that share the same goals. I prefer package >>>>>>>>> dependencies because they address the excessive type coupling problem in >>>>>>>>> object oriented systems, not because they're part of OSGi. Let me argue my >>>>>>>>> case. >>>>>>>>> >>>>>>>>> Module-to-package dependencies (m2p) are preferable over module-to-module >>>>>>>>> dependencies (m2m) for many reasons but these are the most important >>>>>>>>> reasons: >>>>>>>>> >>>>>>>>> M2P is leverages the Java type system unlike m2m that must introduce new >>>>>>>>> namespaces outside the Java type system. >>>>>>>>> M2P can be used to break the transitive dependency chain, m2m suffers of >>>>>>>>> excessive coupling >>>>>>>>> >>>>>>>>> Since the first bullet's benefit should be clear I only argue the more >>>>>>>>> complex second bullet. >>>>>>>>> >>>>>>>>> A module is in many regards like a class. A class encapsulates members, >>>>>>>>> depends on other members/classes, and makes a few members accessible outside >>>>>>>>> the class. A module has a similar structure but then with types/packages as >>>>>>>>> members. >>>>>>>>> >>>>>>>>> After the initial success of Object Oriented Programming (OO) it was >>>>>>>>> quickly learned that reuse did not take place at the expected scale due to >>>>>>>>> excessive type coupling. The problem was that a class aggregated many >>>>>>>>> dependencies to simplify its implementation but these dependencies were >>>>>>>>> unrelated to the contract it implemented. Since class dependencies are >>>>>>>>> transitive most applications disappointingly became an almost fully >>>>>>>>> connected graph. >>>>>>>>> >>>>>>>>> Java's great innovation was the interface because it broke both the >>>>>>>>> transitivity and aggregation of dependencies. A class could now express its >>>>>>>>> dependency (use or implement) on a contract (the interface) and was >>>>>>>>> therefore fully type decoupled from the opposite site. >>>>>>>>> >>>>>>>>> An interface can act as a contract because it names the signature of a >>>>>>>>> set of methods so that the compiler can verify the client and the >>>>>>>>> implementer. >>>>>>>>> >>>>>>>>> Since a module has a very similar structure to a class it suffers from >>>>>>>>> exactly the same transitive aggregation of dependencies. This is not a >>>>>>>>> theory, look at the experiences with Maven >>>>>>>>> (http://www.sonatype.com/people/2011/04/how-not-to-download-the-internet/) >>>>>>>>> Again, this is not that maven is bad or developers are stupid, it is the >>>>>>>>> same underlying force that finally resulted in the Java interface. >>>>>>>>> >>>>>>>>> The parallel for the class' interface for modules is a named set of >>>>>>>>> interfaces. This concept already exists in Java: a package. Looking at >>>>>>>>> almost all JSRs it is clear that our industry already uses packages as >>>>>>>>> "interfaces" to provider implementations. >>>>>>>>> >>>>>>>>> Therefore, just like a class should not depend on other implementation >>>>>>>>> types, a module should preferably not depend on other modules. A module >>>>>>>>> should instead depend on contracts. Since modules will be used to provide >>>>>>>>> components from different sources managed with different life cycles the >>>>>>>>> excessive type coupling caused by m2m is even more damaging than in c2c. >>>>>>>>> Proper use of m2p creates significantly less type coupled systems than m2m, >>>>>>>>> the benefits should be obvious. >>>>>>>>> >>>>>>>>> Since there are use cases for m2m (non-type safe languages for example) I >>>>>>>>> do believe that Jigsaw should still support m2m. However, it would be >>>>>>>>> greatly beneficial to our industry if we could take advantage of the lessons >>>>>>>>> learned with the Java interface and realize how surprisingly important the >>>>>>>>> Java package actually is in our eco system. >>>>>>>>> >>>>>>>>> Kind regards, >>>>>>>>> >>>>>>>>> ? ? Peter Kriens >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 9 nov. 2011, at 15:04, David M. Lloyd wrote: >>>>>>>>> >>>>>>>>>> I'll just state now that using packages as a dependency unit is a >>>>>>>>>> terrible idea, and not some architectural revelation. ?That way, Peter's >>>>>>>>>> wrath will be largely directed at me. :-) >>>>>>>>>> >>>>>>>>>> On 11/09/2011 08:02 AM, Peter Kriens wrote: >>>>>>>>>>> >>>>>>>>>>> I agree that tools are needed but we must be careful to not expect >>>>>>>>>>> tools to stopgap an architectural issue. I think it is important to first do >>>>>>>>>>> good architectural design leveraging existing tools (e.g. the Java type >>>>>>>>>>> system) before you try to add new tools. It is such a pity (but all to >>>>>>>>>>> common) that a design allows for classes of errors that would be impossible >>>>>>>>>>> with a slightly different design. >>>>>>>>>>> >>>>>>>>>>> Kind regards, >>>>>>>>>>> >>>>>>>>>>> ? ? Peter Kriens >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 9 nov. 2011, at 14:49, Alan Bateman wrote: >>>>>>>>>>> >>>>>>>>>>>> On 09/11/2011 13:04, Peter Kriens wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> The issue is that maven problems are not caused because maven is bad >>>>>>>>>>>>> or that pom authors are stupid. The reason is that the module-to-module >>>>>>>>>>>>> dependency architecture in maven (and Jigsaw) is error prone ... >>>>>>>>>>>> >>>>>>>>>>>> This thread started out with someone asking about adding module >>>>>>>>>>>> declarations to existing JAR files, and in that context, I agree it can be >>>>>>>>>>>> error prone without good tools. I think things should be a lot better when >>>>>>>>>>>> modules are compiled. >>>>>>>>>>>> >>>>>>>>>>>> -Alan. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> - DML >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> - DML >>>>>>> >>>>> >>>>> >>>>> -- >>>>> - DML >>>> >>> >> > > From brian at pontarelli.com Wed Nov 16 15:39:19 2011 From: brian at pontarelli.com (Brian Pontarelli) Date: Wed, 16 Nov 2011 16:39:19 -0700 Subject: Why package deps work (Was: Re: Converting plain JARs to Java modules) In-Reply-To: References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> <67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz> <4EC14A53.7070907@redhat.com> <4EC1F53D.7040309@redhat.com> <328B38A3-EB53-4AD2-970B-CE84F5C26F96@aQute.biz> <9B28CE0A-85DC-4F53-8A88-5830759DA4BC@pontarelli.com> Message-ID: <46755271-0738-4DD5-8FE0-54C5CD29FD29@pontarelli.com> On Nov 16, 2011, at 4:03 PM, Neil Bartlett wrote: > Hi Brian, > > On Wed, Nov 16, 2011 at 9:16 PM, Brian Pontarelli wrote: >> [snip] >> Let's use an example. Let's say you write a module to provide a user management system. It uses JPA only for persistence and does not use any hibernate or Eclipse link specific class, which BTW is highly unlikely. You write your module and then unit test it. > > How is this unlikely? On the contrary, it's highly unlikely that it > *could* use any Hibernate or EclipseLink specific class. First, the > module would be compiled with visibility only of the abstract API, > i.e. javax.persistence, and then at runtime we employ the module > system to ensure that it cannot see any specific implementation > classes, even via reflection. This is not how it works in practice. @Type is commonly used and not part of JPA. This happens A LOT. > > In fact if we expect our modules to be reusable -- so they can be used > not only in the project we are working on right now but in subsequent > projects over the next several years -- it's very likely that the > module will be used with multiple JPA providers. That's not to say we > switch JPA providers willy nilly, or without running a QA process over > the set of module that we intend to put into production. We also don't > claim that we never find and fix bugs in our modules when used against > alternative providers. I think a QA and testing process is great, but in practice I just don't see many developers doing this. That's my opinion and based on what I've seen in the Java world for many years. > >> Although it would be really neat if you could ship your module to the world and not really care what implementation of JPA is used, this is unlikely. In fact, whatever JPA implementation you unit tested against is probably the only one that you can definitely say your module will run with. Therefore, you would really need to unit test your module against all variations of JPA and that again is unlikely. > > The developer of the module should NOT care what implementation of JPA > they use, so long as it is an implementation that complies with the > contract. The deployer or assembler of an application is responsible > for finding and providing a compliant implementation. Testing of > course is done against a specific implementation, and does not > guarantee that other implementations can be used; they would have to > be tested by whoever wanted to use them. Nevertheless the work > required to fix issues arising from differences in interpretation of > contracts is much less than the porting effort required if the > original module developer had coded directly against an > implementation. Again, I disagree. In most cases, JPA implementations aren't portable. I've seen many companies try to switch and fail over and over again. This happens a lot because specifications tend to cover far fewer requirements than the implementations do. > > It's important to stress that none of this is theoretical. Exactly > this scenario involving JPA, along with many other similar scenarios, > can be done and is done in real world practice. All my examples of reflection and non-portable implementations are not theoretical, but my experience on real-world projects, working with a wide variety of companies and knowing many engineers who run into these issues all the time. > >> In other cases, it might work though. For example SLF4J is probably a safe bet to code against the API and then be able to drop in any of the implementations. > > This seems to contradict your previous statements. What is special > about SLF4J? If anything I would expect the JPA example to "safer", > since the JPA API was designed very cautiously by a JCP Expert Group, > whereas SLF4J came from a single author in the open source world (I > intend no slight against the SLF4J author; I simply have no > information on the quality and stability of its API). It doesn't really contradict it, I'm just stating that some cases of APIs and implementations work. The difference here is that SLF4J isn't a JSR specification that is implemented by multiple vendors with dramatically different goals with respect to their implementations (i.e. $$). SLF4j implementations are all written by the same group and all tested under the same set of unit tests to ensure that the API works perfectly (or as good as can be expected) across all implementations. I've found that JSRs and JCP Expert Groups tend not to do this as well as open source projects. JPA is a classic example of an API that lacks a number of key features and has numerous implementations all with different semantics and handling. Another classic example is @Inject. That thing is sorta a joke at this point. > >> I guess my point is that it isn't always as simple as you are implying. > > I agree it is not always simple, but it is possible. I tend to think of it as possible, but in many cases not practical. -bp > > Regards > Neil > >> >> >>> >>>> Again, are you talking about build time or runtime? Also, are you suggesting that Java determine your dependencies automatically including the version? Or is this merely to help developers find missing dependencies and remove unused dependencies? >>> If you compile class com.bar.Foo that refers to com.bar.Bar then the Foo.class and Bar.class file contain all their type dependencies and by implication their package dependencies. The version of the package can then for example be found from package-info.java. >>> >>> This technique is used extensively in bnd, an open source library used in Eclipse, IDEA, ant, maven, SBT, and others. It reads the class files in a JAR and generates the OSGi dependency information automatically. >> >> This is true unless there is reflection, which is common. JEE is based entirely on reflecting the top level entry points (servlets, filters, etc). There are many cases of configuration files defining implementations and transitive dependencies. This makes using a bytecode analyzer only a partial solution. >> >> -bp >> >> >> >> >>> >>> Hope this clarifies. Kind regards, >>> >>> Peter Kriens >>> >>> >>> >>>> >>>> -bp >>>> >>>> >>>> >>>> >>>> >>>> >>>>> >>>>> Kind regards, >>>>> >>>>> Peter Kriens >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On 15 nov. 2011, at 06:14, David M. Lloyd wrote: >>>>> >>>>>> On 11/14/2011 08:33 PM, Neil Bartlett wrote: >>>>>>> Hi David, >>>>>>> >>>>>>> More than happy to talk about reality, though Peter already was doing >>>>>>> so. Nothing he said was theoretical, it came directly from his >>>>>>> experiences and the experiences of developers who have been using OSGi >>>>>>> for around 12 years. >>>>>> >>>>>>> Before digging into the points in your message, I need to address the >>>>>>> subject line "Why package deps won't work". Package dependencies will >>>>>>> work and they do work, as is proven every day by the large number of >>>>>>> applications running on OSGi. To claim otherwise would be wilfully >>>>>>> ignorant, so I will assume you are being hyperbolic and really meant >>>>>>> to assert that module deps are just better than package deps. >>>>>>> Therefore I intend to argue on that basis and claim that package deps >>>>>>> are better than module deps; note that I don't claim that module >>>>>>> dependencies "won't work" because JBoss Modules is a proof point that >>>>>>> they do. >>>>>> >>>>>> There was an implied "... for Java SE 8 modules" after "won't work" (this is why I said "won't" rather than "don't") which apparently wasn't implied enough. >>>>>> >>>>>> Of course people use OSGi, or we wouldn't be arguing, but that alone isn't enough to make OSGi's dependency system preferable for the SE case (after all, far more people use plain old class paths). I believe that JBoss Modules is a very accurate representation of how an SE module system could function, right down to executing JARs on the command line with dependencies and modularizing certain JDK APIs. This is not a proof point, but strong evidence that it is an effective solution for the actual problem on the table, and that similar architectures are likely to succeed for the same problem domain. As to why package deps are not an example of an effective solution for this problem, I intend to illustrate in greater detail. >>>>>> >>>>>> Further responses and elaboration below. >>>>>> >>>>>>> On Mon, Nov 14, 2011 at 5:05 PM, David M. Lloyd wrote: >>>>>>>> The explanation is quite simple, really - each point can be pretty much >>>>>>>> wiped out by a heavy dose of reality. >>>>>>>> >>>>>>>> 1. "M2P is leverages the Java type system unlike m2m that must introduce new >>>>>>>> namespaces outside the Java type system." - this is just fluffy >>>>>>>> buzzwordology. Considering packages part of the Java type system is a >>>>>>>> pretty liberal interpretation of the term "type system" as they're basically >>>>>>>> just arbitrary name spaces. That said, there's nothing particularly "new" >>>>>>>> about namespacing modules. JARs have names. Projects have names. Maven >>>>>>>> uses its own namespaces for artifacts. Even the primitive JDK extension >>>>>>>> mechanism uses symbolic names. >>>>>>> >>>>>>> From Wikipedia (http://en.wikipedia.org/wiki/Type_system): "A type >>>>>>> system associates a type with each computed value ... the aim is to >>>>>>> prevent operations expecting a certain kind of value being used with >>>>>>> values for which that operation does not make sense." >>>>>>> >>>>>>> The following will not compile: >>>>>>> >>>>>>> import java.awt.List; >>>>>>> // ... >>>>>>> List list = new ArrayList(); >>>>>>> list.iterator(); // etc >>>>>>> >>>>>>> whereas the following will: >>>>>>> >>>>>>> import java.util.List; >>>>>>> // ... >>>>>>> List list = new ArrayList(); >>>>>>> list.iterator(); // etc >>>>>>> >>>>>>> Package names are part of the type system because using an incorrect >>>>>>> package name in Java source can result in type errors during >>>>>>> compilation, and because the type and available operations associated >>>>>>> with each value varies with the package name used. >>>>>>> >>>>>>> In contrast, it is impossible to obtain a type error at compilation by >>>>>>> incorrectly naming a JAR, project, Maven artefact etc, because none of >>>>>>> those things are ever referenced in Java source code. >>>>>> >>>>>> I stand by my statement. A package is part of the name of the type; it is not a type, nor a meta-type, nor anything like a type in and of itself. This makes them part of the language, sure, but not part of the type system, in my opinion. Thus, "a stretch". I don't feel any particular need to have the world agree with me on this though, other than as I state below... >>>>>> >>>>>>>> Even a simple convention of the parent-most package name for the name of a >>>>>>>> module is very simple to grasp and is in fact the solution we've used quite >>>>>>>> effectively thus far. Thus implying that module names are some new alien >>>>>>>> concept is really not valid. >>>>>>> >>>>>>> The claim is not that they are a new alien concept. The claim is that >>>>>>> they are not part of the Java language and type system, and as a >>>>>>> result of this (along with other problems) are less useful for >>>>>>> depending upon than packages. >>>>>> >>>>>> I don't see how not being part of the Java language makes module identifiers "less useful". I think that this argument artificially elevates the significance of the relationship between packages and the actual type system, and then uses that elevation as a proof that a system based on other namespace principles is deficient, without offering any real examples which actually relate to the type system. >>>>>> >>>>>>>> 2. "M2P can be used to break the transitive dependency chain, m2m suffers of >>>>>>>> excessive coupling." - How about some facts to back this up? I've found >>>>>>>> "m2m" coupling to be just right. In JBoss Modules, we do not export >>>>>>>> transitive dependencies by default. This results in a very simple and clean >>>>>>>> dependency graph between modules. Using package dependencies results in a >>>>>>>> *far more complex* dependency graph, because you need edges for every >>>>>>>> package even though *most of the time you want the whole module anyway*. >>>>>>>> Again, real world here. >>>>>>> >>>>>>> Peter is definitely also talking about the real world and so am I. In >>>>>>> an m2m dependency model you cannot avoid transitive dependencies, >>>>>>> whether you expose them to consumers or not. >>>>>> >>>>>> In *any* model you cannot avoid transitive dependencies. If module A exports module B as part of its API, then you get A and B, and that's life. If you only want part of B, then you filter it down. But with most existing libraries out there (and they easily number in the thousands), they're intended to be used whole, so this doesn't really happen too much in practice. >>>>>> >>>>>>> An importer of a module must be assumed to depend on the whole functionality of that module >>>>>>> whether or not that is actually the case, and therefore all of the >>>>>>> transitive dependencies must be present at runtime. In an m2p world we >>>>>>> have the opportunity to split modules and break apart dependency >>>>>>> graphs, because the unit of coupling is more granular. >>>>>> >>>>>> You could break things down by actual classes if granularity is the goal. But it isn't; it is so much less important than, say, performance, conceptual simplicity, usability, efficiency, memory footprint, etc. The fact is that the vast majority of modules you'll find published at, say Maven central, are designed to be used completely, not partially, and can and are used as such without issue. >>>>>> >>>>>> People just don't usually design modules to be imported by package, except perhaps as an afterthought concession to OSGi, and they don't use them that way at runtime either. It is so seldom a real problem in the SE world (or EE for that matter) once you step away from flat class paths. The simple fact is that most users think of their dependencies in the same terms that their IDEs and their build scripts do: by module. >>>>>> >>>>>> This is truly a case of creating a lot of complexity and extra work in the common case to avoid problems which are not a common case. I think it is much better to optimize for the common case, where people know exactly what dependencies they want (in terms of artifacts), and they just want to declare them and be done without (a) hunting down package lists or (b) running extra tooling. >>>>>> >>>>>>>> If we're just going to throw dogma around, I'll put it the other way: m2p is >>>>>>>> a design error by the OSGi spec designers which has since been embraced as a >>>>>>>> religion. >>>>>>> >>>>>>> Pejorative accusations about "dogma" or "religion" have no place in a >>>>>>> technical discussion. They are also untrue. All of the practising OSGi >>>>>>> developers I know have arrived at their support for package >>>>>>> dependencies as a result of real world experience, not because of some >>>>>>> willingness to bow down to the Almighty CPEG. I don't know of any >>>>>>> practising OSGi developer who has used both Require-Bundle (m2m) and >>>>>>> Import-Package (m2p) and actually prefers the former. >>>>>> >>>>>> Sure, but Require-Bundle is *not* the same thing as using module-to-module dependencies in a non-OSGi system. All OSGi life revolves around the resolver. In the SE world, you cannot have a resolver without introducing a lot of machinery which will definitely negatively impact boot performance in the case that the resolver runs at module load time, or add the complexity of centralized package index management in the case that the resolver would run at install time (as in, install into a module library/repository, not install in the OSGi sense). >>>>>> >>>>>> If in a typical installation a user has to do more than simply delete a file to remove a module, or drop a JAR into a directory to add a module, I believe we've run quite far off-track usability-wise. And if a user has to sit through a lengthy resolution process at runtime (even if only the first time) then we're off-track performance-wise, especially if such resolution must expand to include modules in the module repository which are not even loaded. >>>>>> >>>>>>> I do know several who started as strong supporters of Require-Bundle >>>>>>> and switched to being supporters of Import-Package, not because of >>>>>>> Peter's wrathful pontificating but because they encountered the >>>>>>> specific problems that he described and found that they were fixed by >>>>>>> using Import-Package, and indeed that everything worked so much more >>>>>>> cleanly that way. I'm in this camp myself, and so was Jeff McAffer >>>>>>> before he went over to the Dark Side (Microsoft). >>>>>> >>>>>> There are other, simpler ways to fix package conflict issues when they arise. Like simply excluding the offending package from the import. >>>>>> >>>>>> And as I've said before, I do not believe that Require-Bundle in the context of OSGi bundles is comparable to loading modules by name in the context of a module system such as Jigsaw or JBoss Modules and the types of modules that could be loaded thereby (the aforementioned thousands of existing modules). >>>>>> >>>>>>>> It offers no significant benefit, other than a couple of edge >>>>>>>> cases which are frankly just as well handled by m2m simply by adding package >>>>>>>> filters. Which, by the way, I haven't seen the need to do yet in our 200+ >>>>>>>> module environment, but which we do have the capability to do. >>>>>>> >>>>>>> Substitution, module refactoring and transitive decoupling are hardly >>>>>>> edge cases. >>>>>> >>>>>> Substitution isn't solely addressed by package dependencies. Nor is module refactoring. And additional transitive decoupling isn't something people actually want - as I said, the common case is that users want to be able to use any class from a module once they import that module. In the real world, it is very rare to only want to import part of a module! If you need this routinely, you've done something very unorthodox. Thus I stand by my statement. >>>>>> >>>>>>> However I can see how these issues might not yet have come >>>>>>> to the fore in a module system designed for a single product with a >>>>>>> small number of modules, and where that product has not yet been >>>>>>> through the mill of multiple version evolutions. >>>>>> >>>>>> Cute. I assure you that we have been around through a long, long history of a huge number of Java users doing everything under the sun to their class paths and class loading structures. We also have experience with operating system distribution and managing module distribution across a wide range of other programming languages, not just Java. Our module system has already stood up to reasonably large deployments (200+ modules) with excellent memory footprint and performance numbers and we've been processing usability feedback, including feedback from those with OSGi experience, which has been mostly positive. >>>>>> >>>>>> Furthermore when I was at JavaOne giving booth talks about JBoss Modules and AS7, I found that users expressed quite a bit of frustration at the OSGi model and were quite receptive to a more orthodox m2m system, for what that's worth. >>>>>> >>>>>>>> M2P is a solution just itching for a problem. >>>>>>> >>>>>>> This is also not a very useful statement, I assure you that the >>>>>>> problem came before the solution. Better to show why you think the >>>>>>> problem is invalid or should have been solved differently. >>>>>> >>>>>> There is no way you can convince me that this solution was invented with general modularity in mind. Back when OSGi was first formed, it was designed for the embedded market. The world of Java today is completely different. People just don't bundle their libraries by package, as was possibly expected in the early days of Java. If they did we wouldn't be having this discussion because packages and modules would already be one and the same. >>>>>> >>>>>> OSGi evolved to where it is today. It was not designed from the ground up with actual requirements which pertained to the problem at hand which is modularity of the SE platform and applications which run on it. So yeah I'm going to say, solution before problem. Taking a solution and trying to apply it retroactively to a different problem like this has never failed to bite me in the ass, personally. But I can't speak to everyone's experience. >>>>>> >>>>>>>> But you're going to have a >>>>>>>> tough time convincing me that users *want* to have to use special tooling >>>>>>>> because they want to depend on a module which has too many packages to list >>>>>>>> out by hand. And before you cry "wildcards", be sure to consider that some >>>>>>>> modules use package names which are subordinate to other modules' packages, >>>>>>>> which is a perfectly normal and allowable scenario. Using wildcards for >>>>>>>> package matching could cause amazing levels of havoc. >>>>>>> >>>>>>> OSGi never uses wildcards at runtime, and tools such as bnd do not >>>>>>> need wildcards in order to express package-level dependencies. They >>>>>>> extract the set of packages that were actually used by the code, all >>>>>>> of which are available in the class files. This is possible because >>>>>>> packages are part of the type system (see first point above). >>>>>>> >>>>>>> So it's not that I don't want to list dependencies by hand, rather I >>>>>>> only want to do it once. I am already forced to do it in the import >>>>>>> statements of my Java sources. If I had to repeat that dependency >>>>>>> information -- whether in m2p or m2m form -- then I would run the risk >>>>>>> of it getting out of step with the real dependencies. >>>>>> >>>>>> One of the critical logical gaps here which m2p ignores at a fundamental level is that *packages are not unique*. In the real world, packages are repeated across more than one module all the time, and not just due to versioning. There is *no way* you could take every JAR at Maven Central and plug it in to an m2p system. You'd have to break apart and repackage every single one, at least, and at worst you'd have to rewrite an awful lot of them under different package names to account for this restriction. You'd need to enforce package uniqueness across your whole library, for all time. I don't think this is a sane option. You're adding significant overhead to management, installation, and execution, for what? So you can have the privilege of needing a special tool to extract your dependencies for you? >>>>>> >>>>>>>> Using package dependencies means you either must have a master package index >>>>>>>> for linking, or you need a resolver which has to have analyzed every module >>>>>>>> you ever plan to load. Otherwise, O(1) loading of modules is impossible, >>>>>>>> which is absolutely 100% a deal-breaker for JDK modules which must be >>>>>>>> incrementally installable. And it forbids having packages with the same >>>>>>>> name in more than one JAR without bringing run-time versioning into the >>>>>>>> fold, which is a terrible, terrible can of worms. >>>>>>> >>>>>>> Could you please explain why O(1) is the only acceptable complexity >>>>>>> for installing modules. >>>>>> >>>>>> Because this is Java SE we're talking about. You're going to have a potentially huge number of modules installed in your system. Even if you get around the versioning issues somehow and fully externalize the index of all packages, you still have to load or traverse the centralized index for every module load. >>>>>> >>>>>> I say that not only is ?(1) the only acceptable complexity for installing modules but also loading them at run time. Linking modules at run time should be no worse than O(n) complexity for the number of dependencies the module has, including transitive dependencies. Loading classes and resources from modules should be ?(k) where k is the number of modules which contain a package or directory which matches the class or resource being loaded (almost always one, in my experience). >>>>>> >>>>>> Loading a module should normally be limited to a one-time O(1) disk access without searching or loading any other files or modules than the module artifact itself (and possibly its descriptor if they are held externally by a particular module repository implementation, which is useful but not necessary). Linking a module should be limited in disk access to loading only the modules it imports via direct dependency, or by the relatively rare partial or full re-export of a transitive dependency by a direct dependency. >>>>>> >>>>>> In particular, the expectation of accessing a central index has some potentially serious implications, facing either possible file lock contention by multiple threads or memory overhead of loading in the complete index in advance. It should be expected that modules are loaded concurrently, and such concurrency should not be hindered any more than necessary. Other possible implementations (such as file- or directory-per-package) have their own drawbacks as well or violate what I consider to be core requirements. >>>>>> >>>>>> If there are implementations of package-based resolution which don't involve either a centralized index of one or many files or a resolver which must rove all installed modules in advance, I'd like to hear about it. >>>>>> >>>>>>> OSGi does indeed support incremental install >>>>>>> and while I accept it is probably not O(1) for each module, it would >>>>>>> likely be no more than O(N), though I haven't done the maths yet to >>>>>>> prove this. Bear in mind that in practice, for small N, the constant >>>>>>> factors can result in O(1) being more expensive than O(N). I have seen >>>>>>> OSGi used with many thousands of modules, so unless you have some data >>>>>>> and a use-case showings package-level resolution as unacceptably slow, >>>>>>> your concern just sounds like premature optimisation. >>>>>> >>>>>> Identifying algorithmic complexity during design phase is *never* premature optimization AFAICT. If you don't understand the complexity of the algorithms you're about to implement, and how they are expected to be applied to the problem at hand, then you're not ready for implementation yet. >>>>>> >>>>>> However, postulating that O(n) is okay because can sometimes be faster than O(1) without measuring it for the specific problem in question is definitely premature de-optimization. :-) >>>>>> >>>>>>> There are two reasons to have a packages with the same name in more >>>>>>> than one JAR. The first is a situation called split packages, and it >>>>>>> is highly undesirable because it causes the runtime model of the >>>>>>> package to diverge from the compile-time model, and therefore things >>>>>>> like package-private types and members stop working correctly. For >>>>>>> this reason, OSGi's m2p imports support depending only upon a single >>>>>>> exporter of a particular package, i.e. we do not aggregate all exports >>>>>>> of that package. >>>>>>> >>>>>>> Unfortunately split packages are sometimes unavoidable in legacy code >>>>>>> that cannot be refactored, e.g. the JDK. To support such scenarios >>>>>>> OSGi has Require-Bundle, i.e. m2m. This does not negate the problems >>>>>>> associated with m2m, it is simply a trade-off that we face with poorly >>>>>>> factored legacy code. >>>>>>> >>>>>>> The second reason for multiple packages with the same name is when you >>>>>>> explicitly want to install multiple versions of a library/API and have >>>>>>> them all available within the same runtime. I wouldn't call this a >>>>>>> "can of worms" exactly because it can be done without too much >>>>>>> trouble, though for the sake of a simple life I personally avoid this >>>>>>> situation unless it's necessary. >>>>>> >>>>>> Yes, and it is actually fairly common in practice to want two versions of something in the sense of the two versions being wholly different implementations (think apache commons logging versus jcl-over-slf4j for a trivial example which crops up a lot). There are reasons to use one or the other. Multiplicity of versions (in this respect) is something the module system has to handle gracefully and simply. >>>>>> >>>>>>>> Finally it should be perfectly clear to anyone who has read the original >>>>>>>> requirements document that nothing in this module system should prevent OSGi >>>>>>>> from functioning as it is, so there is absolutely no reason to assume that >>>>>>>> any OSGi implementation is so threatened - especially if m2p linking is as >>>>>>>> superior as has been expressed. Our module system (which is conceptually >>>>>>>> similar to Jigsaw in many regards) in fact does support our OSGi >>>>>>>> implementation quite effectively without itself implementing OSGi's >>>>>>>> package-to-package resolution (which like I said throws O(1) out the >>>>>>>> window). >>>>>>> >>>>>>> I agree that Jigsaw's existence doesn't threaten OSGi's, so long as >>>>>>> Java 8 doesn't actually break OSGi (and if it did so, it would >>>>>>> likewise break many other applications and could not be considered >>>>>>> backwards compatible with Java 7). The two can interoperate through >>>>>>> m2m-type dependencies. Tim Ellison started Project Penrose for the >>>>>>> purpose of investigating, testing and deepening this collaboration. >>>>>>> >>>>>>> Neverthless, the point that I believe Glyn was making is the >>>>>>> following. We accept that m2m dependencies are probably required for >>>>>>> the JDK, which implies a module system like Jigsaw or >>>>>>> OSGi/Require-Bundle rather than OSGi/Import-Package. However is it >>>>>>> intended to be used for application modularisation as well? This is of >>>>>>> course a question for the Jigsaw team rather than you, David. >>>>>> >>>>>> The question of whether Java SE 8 modules are intended to be used for application modularization is a question for the EG, not the Jigsaw team. The job of the Jigsaw team is really to implement a prototype which meets the requirements set forth by the EG, which may become the reference implementation at a future point. >>>>>> >>>>>> The question of whether the greater Java community will embrace the SE module system for applications is to be answered by the community only. I personally believe that to develop a module system which is not intended to be usable by the average standalone application is foolhardy, a great waste of effort, and is doomed to mediocrity, given the success we have had with such a system. >>>>>> >>>>>> However until there is an EG, these are all just as much questions for me as for anybody, and I can and will answer to the best of my knowledge and belief. >>>>>> >>>>>>> As a result of experience in developing and evolving large real-world >>>>>>> applications using a module system that supports BOTH m2m and m2p >>>>>>> dependencies, I believe it would be very unfortunate if a module >>>>>>> system that supports ONLY m2m were to become widely used in the >>>>>>> application space... not because OSGi can't handle the competition, >>>>>>> but because those applications will be fragile and hard to evolve. >>>>>> >>>>>> I don't believe this to be the case. I think that the status quo doesn't result in particularly fragile applications, and such can be easy to evolve or hard depending on the quality of the application components and frameworks involved. I think that m2m dependencies enhance the status quo such that applications are somewhat less fragile (owing chiefly to the simple measure of preventing transitive dependencies from being exported by default), and quite easy to evolve as well (even large and complex frameworks rarely have more than a few dependencies, unless they themselves are overly fragmented (e.g. CXF as a nightmare example which yet still works fine under an m2m system)). >>>>>> >>>>>>> My question for you David is as follows. I understand that you prefer >>>>>>> module dependencies, but do you believe that package dependencies have >>>>>>> no value whatsoever and therefore should not be available to >>>>>>> application developers in the Java 8 module system? If so, why did Red >>>>>>> Hat create an OSGi implementation? >>>>>> >>>>>> I personally believe that they have some value, but that value is limited to interoperability with OSGi. I do not believe that it is a preferable model for most users or standalone applications, nor for larger applications such as our application server. I think that the requirement for extra tooling and the increased run-time complexity which is exposed to users nullifies the benefits. If someone really wants this variety of modularity, they should simply use OSGi. >>>>>> >>>>>> Red Hat created an OSGi implementation for many reasons, but the only significant one to me is that there are people who want to use their OSGi applications with our application server. I don't believe any other reason is even necessary. It's the same reason we strive to support any spec, from EJB 1.0 to Java EE 6 and beyond. Someone wants the functionality, so we deliver it to the best of our ability. >>>>>> >>>>>>> Kind regards >>>>>>> Neil >>>>>>> >>>>>>>> >>>>>>>> On 11/14/2011 01:49 AM, Glyn Normington wrote: >>>>>>>>> >>>>>>>>> I look forward to David's elaboration of why he thinks "using packages as >>>>>>>>> a dependency unit is a terrible idea" to balance Peter's clear explanation >>>>>>>>> of the benefits of m2p. >>>>>>>>> >>>>>>>>> Meanwhile, it's worth noting that, according to the requirements document, >>>>>>>>> Jigsaw is aimed at platform modularisation and the platform being >>>>>>>>> modularised has some non-optimal division of types across packages (see the >>>>>>>>> package subsets requirement) which favour m2m dependencies. (Note that >>>>>>>>> Apache Harmony was developed with modularity in mind and was able to exploit >>>>>>>>> m2p, so platform modularisation per se needn't be limited to m2m.) >>>>>>>>> >>>>>>>>> So if Jigsaw excludes m2p, it will then be applicable to certain kinds of >>>>>>>>> legacy code modularisation and less applicable to new module development and >>>>>>>>> modularisation of existing code whose division into packages suits m2p. IIRC >>>>>>>>> this was the original positioning of Jigsaw: for use primarily within the >>>>>>>>> OpenJDK codebase and only exposed for application use because it was too >>>>>>>>> inconvenient to hide it. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Glyn >>>>>>>>> >>>>>>>>> On 12 Nov 2011, at 11:59, Peter Kriens wrote: >>>>>>>>> >>>>>>>>>> Neither my wrath, nor the fact that I rarely if ever get angry is >>>>>>>>>> relevant in this discussion ... This is a technical argument that are >>>>>>>>>> solvable by technical people that share the same goals. I prefer package >>>>>>>>>> dependencies because they address the excessive type coupling problem in >>>>>>>>>> object oriented systems, not because they're part of OSGi. Let me argue my >>>>>>>>>> case. >>>>>>>>>> >>>>>>>>>> Module-to-package dependencies (m2p) are preferable over module-to-module >>>>>>>>>> dependencies (m2m) for many reasons but these are the most important >>>>>>>>>> reasons: >>>>>>>>>> >>>>>>>>>> M2P is leverages the Java type system unlike m2m that must introduce new >>>>>>>>>> namespaces outside the Java type system. >>>>>>>>>> M2P can be used to break the transitive dependency chain, m2m suffers of >>>>>>>>>> excessive coupling >>>>>>>>>> >>>>>>>>>> Since the first bullet's benefit should be clear I only argue the more >>>>>>>>>> complex second bullet. >>>>>>>>>> >>>>>>>>>> A module is in many regards like a class. A class encapsulates members, >>>>>>>>>> depends on other members/classes, and makes a few members accessible outside >>>>>>>>>> the class. A module has a similar structure but then with types/packages as >>>>>>>>>> members. >>>>>>>>>> >>>>>>>>>> After the initial success of Object Oriented Programming (OO) it was >>>>>>>>>> quickly learned that reuse did not take place at the expected scale due to >>>>>>>>>> excessive type coupling. The problem was that a class aggregated many >>>>>>>>>> dependencies to simplify its implementation but these dependencies were >>>>>>>>>> unrelated to the contract it implemented. Since class dependencies are >>>>>>>>>> transitive most applications disappointingly became an almost fully >>>>>>>>>> connected graph. >>>>>>>>>> >>>>>>>>>> Java's great innovation was the interface because it broke both the >>>>>>>>>> transitivity and aggregation of dependencies. A class could now express its >>>>>>>>>> dependency (use or implement) on a contract (the interface) and was >>>>>>>>>> therefore fully type decoupled from the opposite site. >>>>>>>>>> >>>>>>>>>> An interface can act as a contract because it names the signature of a >>>>>>>>>> set of methods so that the compiler can verify the client and the >>>>>>>>>> implementer. >>>>>>>>>> >>>>>>>>>> Since a module has a very similar structure to a class it suffers from >>>>>>>>>> exactly the same transitive aggregation of dependencies. This is not a >>>>>>>>>> theory, look at the experiences with Maven >>>>>>>>>> (http://www.sonatype.com/people/2011/04/how-not-to-download-the-internet/) >>>>>>>>>> Again, this is not that maven is bad or developers are stupid, it is the >>>>>>>>>> same underlying force that finally resulted in the Java interface. >>>>>>>>>> >>>>>>>>>> The parallel for the class' interface for modules is a named set of >>>>>>>>>> interfaces. This concept already exists in Java: a package. Looking at >>>>>>>>>> almost all JSRs it is clear that our industry already uses packages as >>>>>>>>>> "interfaces" to provider implementations. >>>>>>>>>> >>>>>>>>>> Therefore, just like a class should not depend on other implementation >>>>>>>>>> types, a module should preferably not depend on other modules. A module >>>>>>>>>> should instead depend on contracts. Since modules will be used to provide >>>>>>>>>> components from different sources managed with different life cycles the >>>>>>>>>> excessive type coupling caused by m2m is even more damaging than in c2c. >>>>>>>>>> Proper use of m2p creates significantly less type coupled systems than m2m, >>>>>>>>>> the benefits should be obvious. >>>>>>>>>> >>>>>>>>>> Since there are use cases for m2m (non-type safe languages for example) I >>>>>>>>>> do believe that Jigsaw should still support m2m. However, it would be >>>>>>>>>> greatly beneficial to our industry if we could take advantage of the lessons >>>>>>>>>> learned with the Java interface and realize how surprisingly important the >>>>>>>>>> Java package actually is in our eco system. >>>>>>>>>> >>>>>>>>>> Kind regards, >>>>>>>>>> >>>>>>>>>> Peter Kriens >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 9 nov. 2011, at 15:04, David M. Lloyd wrote: >>>>>>>>>> >>>>>>>>>>> I'll just state now that using packages as a dependency unit is a >>>>>>>>>>> terrible idea, and not some architectural revelation. That way, Peter's >>>>>>>>>>> wrath will be largely directed at me. :-) >>>>>>>>>>> >>>>>>>>>>> On 11/09/2011 08:02 AM, Peter Kriens wrote: >>>>>>>>>>>> >>>>>>>>>>>> I agree that tools are needed but we must be careful to not expect >>>>>>>>>>>> tools to stopgap an architectural issue. I think it is important to first do >>>>>>>>>>>> good architectural design leveraging existing tools (e.g. the Java type >>>>>>>>>>>> system) before you try to add new tools. It is such a pity (but all to >>>>>>>>>>>> common) that a design allows for classes of errors that would be impossible >>>>>>>>>>>> with a slightly different design. >>>>>>>>>>>> >>>>>>>>>>>> Kind regards, >>>>>>>>>>>> >>>>>>>>>>>> Peter Kriens >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 9 nov. 2011, at 14:49, Alan Bateman wrote: >>>>>>>>>>>> >>>>>>>>>>>>> On 09/11/2011 13:04, Peter Kriens wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> The issue is that maven problems are not caused because maven is bad >>>>>>>>>>>>>> or that pom authors are stupid. The reason is that the module-to-module >>>>>>>>>>>>>> dependency architecture in maven (and Jigsaw) is error prone ... >>>>>>>>>>>>> >>>>>>>>>>>>> This thread started out with someone asking about adding module >>>>>>>>>>>>> declarations to existing JAR files, and in that context, I agree it can be >>>>>>>>>>>>> error prone without good tools. I think things should be a lot better when >>>>>>>>>>>>> modules are compiled. >>>>>>>>>>>>> >>>>>>>>>>>>> -Alan. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> - DML >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> - DML >>>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> - DML >>>>> >>>> >>> >> >> From peter.kriens at aqute.biz Thu Nov 17 06:24:16 2011 From: peter.kriens at aqute.biz (Peter Kriens) Date: Thu, 17 Nov 2011 15:24:16 +0100 Subject: Why package deps work (Was: Re: Converting plain JARs to Java modules) In-Reply-To: <46755271-0738-4DD5-8FE0-54C5CD29FD29@pontarelli.com> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> <67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz> <4EC14A53.7070907@redhat.com> <4EC1F53D.7040309@redhat.com> <328B38A3-EB53-4AD2-970B-CE84F5C26F96@aQute.biz> <9B28CE0A-85DC-4F53-8A88-5830759DA4BC@pontarelli.com> <46755271-0738-4DD5-8FE0-54C5CD29FD29@pontarelli.com> Message-ID: On 17 nov. 2011, at 00:39, Brian Pontarelli wrote: > > I tend to think of it as possible, but in many cases not practical. Fair enough, but should a Java module system not at least try to support this possibility? Kind regards, Peter Kriens On 17 nov. 2011, at 00:39, Brian Pontarelli wrote: > > On Nov 16, 2011, at 4:03 PM, Neil Bartlett wrote: > >> Hi Brian, >> >> On Wed, Nov 16, 2011 at 9:16 PM, Brian Pontarelli wrote: >>> [snip] >>> Let's use an example. Let's say you write a module to provide a user management system. It uses JPA only for persistence and does not use any hibernate or Eclipse link specific class, which BTW is highly unlikely. You write your module and then unit test it. >> >> How is this unlikely? On the contrary, it's highly unlikely that it >> *could* use any Hibernate or EclipseLink specific class. First, the >> module would be compiled with visibility only of the abstract API, >> i.e. javax.persistence, and then at runtime we employ the module >> system to ensure that it cannot see any specific implementation >> classes, even via reflection. > > This is not how it works in practice. @Type is commonly used and not part of JPA. This happens A LOT. > >> >> In fact if we expect our modules to be reusable -- so they can be used >> not only in the project we are working on right now but in subsequent >> projects over the next several years -- it's very likely that the >> module will be used with multiple JPA providers. That's not to say we >> switch JPA providers willy nilly, or without running a QA process over >> the set of module that we intend to put into production. We also don't >> claim that we never find and fix bugs in our modules when used against >> alternative providers. > > I think a QA and testing process is great, but in practice I just don't see many developers doing this. That's my opinion and based on what I've seen in the Java world for many years. > > >> >>> Although it would be really neat if you could ship your module to the world and not really care what implementation of JPA is used, this is unlikely. In fact, whatever JPA implementation you unit tested against is probably the only one that you can definitely say your module will run with. Therefore, you would really need to unit test your module against all variations of JPA and that again is unlikely. >> >> The developer of the module should NOT care what implementation of JPA >> they use, so long as it is an implementation that complies with the >> contract. The deployer or assembler of an application is responsible >> for finding and providing a compliant implementation. Testing of >> course is done against a specific implementation, and does not >> guarantee that other implementations can be used; they would have to >> be tested by whoever wanted to use them. Nevertheless the work >> required to fix issues arising from differences in interpretation of >> contracts is much less than the porting effort required if the >> original module developer had coded directly against an >> implementation. > > Again, I disagree. In most cases, JPA implementations aren't portable. I've seen many companies try to switch and fail over and over again. This happens a lot because specifications tend to cover far fewer requirements than the implementations do. > > >> >> It's important to stress that none of this is theoretical. Exactly >> this scenario involving JPA, along with many other similar scenarios, >> can be done and is done in real world practice. > > All my examples of reflection and non-portable implementations are not theoretical, but my experience on real-world projects, working with a wide variety of companies and knowing many engineers who run into these issues all the time. > > >> >>> In other cases, it might work though. For example SLF4J is probably a safe bet to code against the API and then be able to drop in any of the implementations. >> >> This seems to contradict your previous statements. What is special >> about SLF4J? If anything I would expect the JPA example to "safer", >> since the JPA API was designed very cautiously by a JCP Expert Group, >> whereas SLF4J came from a single author in the open source world (I >> intend no slight against the SLF4J author; I simply have no >> information on the quality and stability of its API). > > It doesn't really contradict it, I'm just stating that some cases of APIs and implementations work. The difference here is that SLF4J isn't a JSR specification that is implemented by multiple vendors with dramatically different goals with respect to their implementations (i.e. $$). SLF4j implementations are all written by the same group and all tested under the same set of unit tests to ensure that the API works perfectly (or as good as can be expected) across all implementations. > > I've found that JSRs and JCP Expert Groups tend not to do this as well as open source projects. JPA is a classic example of an API that lacks a number of key features and has numerous implementations all with different semantics and handling. > > Another classic example is @Inject. That thing is sorta a joke at this point. > > >> >>> I guess my point is that it isn't always as simple as you are implying. >> >> I agree it is not always simple, but it is possible. > > I tend to think of it as possible, but in many cases not practical. > > -bp > > > > > > >> >> Regards >> Neil >> >>> >>> >>>> >>>>> Again, are you talking about build time or runtime? Also, are you suggesting that Java determine your dependencies automatically including the version? Or is this merely to help developers find missing dependencies and remove unused dependencies? >>>> If you compile class com.bar.Foo that refers to com.bar.Bar then the Foo.class and Bar.class file contain all their type dependencies and by implication their package dependencies. The version of the package can then for example be found from package-info.java. >>>> >>>> This technique is used extensively in bnd, an open source library used in Eclipse, IDEA, ant, maven, SBT, and others. It reads the class files in a JAR and generates the OSGi dependency information automatically. >>> >>> This is true unless there is reflection, which is common. JEE is based entirely on reflecting the top level entry points (servlets, filters, etc). There are many cases of configuration files defining implementations and transitive dependencies. This makes using a bytecode analyzer only a partial solution. >>> >>> -bp >>> >>> >>> >>> >>>> >>>> Hope this clarifies. Kind regards, >>>> >>>> Peter Kriens >>>> >>>> >>>> >>>>> >>>>> -bp >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> >>>>>> Kind regards, >>>>>> >>>>>> Peter Kriens >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On 15 nov. 2011, at 06:14, David M. Lloyd wrote: >>>>>> >>>>>>> On 11/14/2011 08:33 PM, Neil Bartlett wrote: >>>>>>>> Hi David, >>>>>>>> >>>>>>>> More than happy to talk about reality, though Peter already was doing >>>>>>>> so. Nothing he said was theoretical, it came directly from his >>>>>>>> experiences and the experiences of developers who have been using OSGi >>>>>>>> for around 12 years. >>>>>>> >>>>>>>> Before digging into the points in your message, I need to address the >>>>>>>> subject line "Why package deps won't work". Package dependencies will >>>>>>>> work and they do work, as is proven every day by the large number of >>>>>>>> applications running on OSGi. To claim otherwise would be wilfully >>>>>>>> ignorant, so I will assume you are being hyperbolic and really meant >>>>>>>> to assert that module deps are just better than package deps. >>>>>>>> Therefore I intend to argue on that basis and claim that package deps >>>>>>>> are better than module deps; note that I don't claim that module >>>>>>>> dependencies "won't work" because JBoss Modules is a proof point that >>>>>>>> they do. >>>>>>> >>>>>>> There was an implied "... for Java SE 8 modules" after "won't work" (this is why I said "won't" rather than "don't") which apparently wasn't implied enough. >>>>>>> >>>>>>> Of course people use OSGi, or we wouldn't be arguing, but that alone isn't enough to make OSGi's dependency system preferable for the SE case (after all, far more people use plain old class paths). I believe that JBoss Modules is a very accurate representation of how an SE module system could function, right down to executing JARs on the command line with dependencies and modularizing certain JDK APIs. This is not a proof point, but strong evidence that it is an effective solution for the actual problem on the table, and that similar architectures are likely to succeed for the same problem domain. As to why package deps are not an example of an effective solution for this problem, I intend to illustrate in greater detail. >>>>>>> >>>>>>> Further responses and elaboration below. >>>>>>> >>>>>>>> On Mon, Nov 14, 2011 at 5:05 PM, David M. Lloyd wrote: >>>>>>>>> The explanation is quite simple, really - each point can be pretty much >>>>>>>>> wiped out by a heavy dose of reality. >>>>>>>>> >>>>>>>>> 1. "M2P is leverages the Java type system unlike m2m that must introduce new >>>>>>>>> namespaces outside the Java type system." - this is just fluffy >>>>>>>>> buzzwordology. Considering packages part of the Java type system is a >>>>>>>>> pretty liberal interpretation of the term "type system" as they're basically >>>>>>>>> just arbitrary name spaces. That said, there's nothing particularly "new" >>>>>>>>> about namespacing modules. JARs have names. Projects have names. Maven >>>>>>>>> uses its own namespaces for artifacts. Even the primitive JDK extension >>>>>>>>> mechanism uses symbolic names. >>>>>>>> >>>>>>>> From Wikipedia (http://en.wikipedia.org/wiki/Type_system): "A type >>>>>>>> system associates a type with each computed value ... the aim is to >>>>>>>> prevent operations expecting a certain kind of value being used with >>>>>>>> values for which that operation does not make sense." >>>>>>>> >>>>>>>> The following will not compile: >>>>>>>> >>>>>>>> import java.awt.List; >>>>>>>> // ... >>>>>>>> List list = new ArrayList(); >>>>>>>> list.iterator(); // etc >>>>>>>> >>>>>>>> whereas the following will: >>>>>>>> >>>>>>>> import java.util.List; >>>>>>>> // ... >>>>>>>> List list = new ArrayList(); >>>>>>>> list.iterator(); // etc >>>>>>>> >>>>>>>> Package names are part of the type system because using an incorrect >>>>>>>> package name in Java source can result in type errors during >>>>>>>> compilation, and because the type and available operations associated >>>>>>>> with each value varies with the package name used. >>>>>>>> >>>>>>>> In contrast, it is impossible to obtain a type error at compilation by >>>>>>>> incorrectly naming a JAR, project, Maven artefact etc, because none of >>>>>>>> those things are ever referenced in Java source code. >>>>>>> >>>>>>> I stand by my statement. A package is part of the name of the type; it is not a type, nor a meta-type, nor anything like a type in and of itself. This makes them part of the language, sure, but not part of the type system, in my opinion. Thus, "a stretch". I don't feel any particular need to have the world agree with me on this though, other than as I state below... >>>>>>> >>>>>>>>> Even a simple convention of the parent-most package name for the name of a >>>>>>>>> module is very simple to grasp and is in fact the solution we've used quite >>>>>>>>> effectively thus far. Thus implying that module names are some new alien >>>>>>>>> concept is really not valid. >>>>>>>> >>>>>>>> The claim is not that they are a new alien concept. The claim is that >>>>>>>> they are not part of the Java language and type system, and as a >>>>>>>> result of this (along with other problems) are less useful for >>>>>>>> depending upon than packages. >>>>>>> >>>>>>> I don't see how not being part of the Java language makes module identifiers "less useful". I think that this argument artificially elevates the significance of the relationship between packages and the actual type system, and then uses that elevation as a proof that a system based on other namespace principles is deficient, without offering any real examples which actually relate to the type system. >>>>>>> >>>>>>>>> 2. "M2P can be used to break the transitive dependency chain, m2m suffers of >>>>>>>>> excessive coupling." - How about some facts to back this up? I've found >>>>>>>>> "m2m" coupling to be just right. In JBoss Modules, we do not export >>>>>>>>> transitive dependencies by default. This results in a very simple and clean >>>>>>>>> dependency graph between modules. Using package dependencies results in a >>>>>>>>> *far more complex* dependency graph, because you need edges for every >>>>>>>>> package even though *most of the time you want the whole module anyway*. >>>>>>>>> Again, real world here. >>>>>>>> >>>>>>>> Peter is definitely also talking about the real world and so am I. In >>>>>>>> an m2m dependency model you cannot avoid transitive dependencies, >>>>>>>> whether you expose them to consumers or not. >>>>>>> >>>>>>> In *any* model you cannot avoid transitive dependencies. If module A exports module B as part of its API, then you get A and B, and that's life. If you only want part of B, then you filter it down. But with most existing libraries out there (and they easily number in the thousands), they're intended to be used whole, so this doesn't really happen too much in practice. >>>>>>> >>>>>>>> An importer of a module must be assumed to depend on the whole functionality of that module >>>>>>>> whether or not that is actually the case, and therefore all of the >>>>>>>> transitive dependencies must be present at runtime. In an m2p world we >>>>>>>> have the opportunity to split modules and break apart dependency >>>>>>>> graphs, because the unit of coupling is more granular. >>>>>>> >>>>>>> You could break things down by actual classes if granularity is the goal. But it isn't; it is so much less important than, say, performance, conceptual simplicity, usability, efficiency, memory footprint, etc. The fact is that the vast majority of modules you'll find published at, say Maven central, are designed to be used completely, not partially, and can and are used as such without issue. >>>>>>> >>>>>>> People just don't usually design modules to be imported by package, except perhaps as an afterthought concession to OSGi, and they don't use them that way at runtime either. It is so seldom a real problem in the SE world (or EE for that matter) once you step away from flat class paths. The simple fact is that most users think of their dependencies in the same terms that their IDEs and their build scripts do: by module. >>>>>>> >>>>>>> This is truly a case of creating a lot of complexity and extra work in the common case to avoid problems which are not a common case. I think it is much better to optimize for the common case, where people know exactly what dependencies they want (in terms of artifacts), and they just want to declare them and be done without (a) hunting down package lists or (b) running extra tooling. >>>>>>> >>>>>>>>> If we're just going to throw dogma around, I'll put it the other way: m2p is >>>>>>>>> a design error by the OSGi spec designers which has since been embraced as a >>>>>>>>> religion. >>>>>>>> >>>>>>>> Pejorative accusations about "dogma" or "religion" have no place in a >>>>>>>> technical discussion. They are also untrue. All of the practising OSGi >>>>>>>> developers I know have arrived at their support for package >>>>>>>> dependencies as a result of real world experience, not because of some >>>>>>>> willingness to bow down to the Almighty CPEG. I don't know of any >>>>>>>> practising OSGi developer who has used both Require-Bundle (m2m) and >>>>>>>> Import-Package (m2p) and actually prefers the former. >>>>>>> >>>>>>> Sure, but Require-Bundle is *not* the same thing as using module-to-module dependencies in a non-OSGi system. All OSGi life revolves around the resolver. In the SE world, you cannot have a resolver without introducing a lot of machinery which will definitely negatively impact boot performance in the case that the resolver runs at module load time, or add the complexity of centralized package index management in the case that the resolver would run at install time (as in, install into a module library/repository, not install in the OSGi sense). >>>>>>> >>>>>>> If in a typical installation a user has to do more than simply delete a file to remove a module, or drop a JAR into a directory to add a module, I believe we've run quite far off-track usability-wise. And if a user has to sit through a lengthy resolution process at runtime (even if only the first time) then we're off-track performance-wise, especially if such resolution must expand to include modules in the module repository which are not even loaded. >>>>>>> >>>>>>>> I do know several who started as strong supporters of Require-Bundle >>>>>>>> and switched to being supporters of Import-Package, not because of >>>>>>>> Peter's wrathful pontificating but because they encountered the >>>>>>>> specific problems that he described and found that they were fixed by >>>>>>>> using Import-Package, and indeed that everything worked so much more >>>>>>>> cleanly that way. I'm in this camp myself, and so was Jeff McAffer >>>>>>>> before he went over to the Dark Side (Microsoft). >>>>>>> >>>>>>> There are other, simpler ways to fix package conflict issues when they arise. Like simply excluding the offending package from the import. >>>>>>> >>>>>>> And as I've said before, I do not believe that Require-Bundle in the context of OSGi bundles is comparable to loading modules by name in the context of a module system such as Jigsaw or JBoss Modules and the types of modules that could be loaded thereby (the aforementioned thousands of existing modules). >>>>>>> >>>>>>>>> It offers no significant benefit, other than a couple of edge >>>>>>>>> cases which are frankly just as well handled by m2m simply by adding package >>>>>>>>> filters. Which, by the way, I haven't seen the need to do yet in our 200+ >>>>>>>>> module environment, but which we do have the capability to do. >>>>>>>> >>>>>>>> Substitution, module refactoring and transitive decoupling are hardly >>>>>>>> edge cases. >>>>>>> >>>>>>> Substitution isn't solely addressed by package dependencies. Nor is module refactoring. And additional transitive decoupling isn't something people actually want - as I said, the common case is that users want to be able to use any class from a module once they import that module. In the real world, it is very rare to only want to import part of a module! If you need this routinely, you've done something very unorthodox. Thus I stand by my statement. >>>>>>> >>>>>>>> However I can see how these issues might not yet have come >>>>>>>> to the fore in a module system designed for a single product with a >>>>>>>> small number of modules, and where that product has not yet been >>>>>>>> through the mill of multiple version evolutions. >>>>>>> >>>>>>> Cute. I assure you that we have been around through a long, long history of a huge number of Java users doing everything under the sun to their class paths and class loading structures. We also have experience with operating system distribution and managing module distribution across a wide range of other programming languages, not just Java. Our module system has already stood up to reasonably large deployments (200+ modules) with excellent memory footprint and performance numbers and we've been processing usability feedback, including feedback from those with OSGi experience, which has been mostly positive. >>>>>>> >>>>>>> Furthermore when I was at JavaOne giving booth talks about JBoss Modules and AS7, I found that users expressed quite a bit of frustration at the OSGi model and were quite receptive to a more orthodox m2m system, for what that's worth. >>>>>>> >>>>>>>>> M2P is a solution just itching for a problem. >>>>>>>> >>>>>>>> This is also not a very useful statement, I assure you that the >>>>>>>> problem came before the solution. Better to show why you think the >>>>>>>> problem is invalid or should have been solved differently. >>>>>>> >>>>>>> There is no way you can convince me that this solution was invented with general modularity in mind. Back when OSGi was first formed, it was designed for the embedded market. The world of Java today is completely different. People just don't bundle their libraries by package, as was possibly expected in the early days of Java. If they did we wouldn't be having this discussion because packages and modules would already be one and the same. >>>>>>> >>>>>>> OSGi evolved to where it is today. It was not designed from the ground up with actual requirements which pertained to the problem at hand which is modularity of the SE platform and applications which run on it. So yeah I'm going to say, solution before problem. Taking a solution and trying to apply it retroactively to a different problem like this has never failed to bite me in the ass, personally. But I can't speak to everyone's experience. >>>>>>> >>>>>>>>> But you're going to have a >>>>>>>>> tough time convincing me that users *want* to have to use special tooling >>>>>>>>> because they want to depend on a module which has too many packages to list >>>>>>>>> out by hand. And before you cry "wildcards", be sure to consider that some >>>>>>>>> modules use package names which are subordinate to other modules' packages, >>>>>>>>> which is a perfectly normal and allowable scenario. Using wildcards for >>>>>>>>> package matching could cause amazing levels of havoc. >>>>>>>> >>>>>>>> OSGi never uses wildcards at runtime, and tools such as bnd do not >>>>>>>> need wildcards in order to express package-level dependencies. They >>>>>>>> extract the set of packages that were actually used by the code, all >>>>>>>> of which are available in the class files. This is possible because >>>>>>>> packages are part of the type system (see first point above). >>>>>>>> >>>>>>>> So it's not that I don't want to list dependencies by hand, rather I >>>>>>>> only want to do it once. I am already forced to do it in the import >>>>>>>> statements of my Java sources. If I had to repeat that dependency >>>>>>>> information -- whether in m2p or m2m form -- then I would run the risk >>>>>>>> of it getting out of step with the real dependencies. >>>>>>> >>>>>>> One of the critical logical gaps here which m2p ignores at a fundamental level is that *packages are not unique*. In the real world, packages are repeated across more than one module all the time, and not just due to versioning. There is *no way* you could take every JAR at Maven Central and plug it in to an m2p system. You'd have to break apart and repackage every single one, at least, and at worst you'd have to rewrite an awful lot of them under different package names to account for this restriction. You'd need to enforce package uniqueness across your whole library, for all time. I don't think this is a sane option. You're adding significant overhead to management, installation, and execution, for what? So you can have the privilege of needing a special tool to extract your dependencies for you? >>>>>>> >>>>>>>>> Using package dependencies means you either must have a master package index >>>>>>>>> for linking, or you need a resolver which has to have analyzed every module >>>>>>>>> you ever plan to load. Otherwise, O(1) loading of modules is impossible, >>>>>>>>> which is absolutely 100% a deal-breaker for JDK modules which must be >>>>>>>>> incrementally installable. And it forbids having packages with the same >>>>>>>>> name in more than one JAR without bringing run-time versioning into the >>>>>>>>> fold, which is a terrible, terrible can of worms. >>>>>>>> >>>>>>>> Could you please explain why O(1) is the only acceptable complexity >>>>>>>> for installing modules. >>>>>>> >>>>>>> Because this is Java SE we're talking about. You're going to have a potentially huge number of modules installed in your system. Even if you get around the versioning issues somehow and fully externalize the index of all packages, you still have to load or traverse the centralized index for every module load. >>>>>>> >>>>>>> I say that not only is ?(1) the only acceptable complexity for installing modules but also loading them at run time. Linking modules at run time should be no worse than O(n) complexity for the number of dependencies the module has, including transitive dependencies. Loading classes and resources from modules should be ?(k) where k is the number of modules which contain a package or directory which matches the class or resource being loaded (almost always one, in my experience). >>>>>>> >>>>>>> Loading a module should normally be limited to a one-time O(1) disk access without searching or loading any other files or modules than the module artifact itself (and possibly its descriptor if they are held externally by a particular module repository implementation, which is useful but not necessary). Linking a module should be limited in disk access to loading only the modules it imports via direct dependency, or by the relatively rare partial or full re-export of a transitive dependency by a direct dependency. >>>>>>> >>>>>>> In particular, the expectation of accessing a central index has some potentially serious implications, facing either possible file lock contention by multiple threads or memory overhead of loading in the complete index in advance. It should be expected that modules are loaded concurrently, and such concurrency should not be hindered any more than necessary. Other possible implementations (such as file- or directory-per-package) have their own drawbacks as well or violate what I consider to be core requirements. >>>>>>> >>>>>>> If there are implementations of package-based resolution which don't involve either a centralized index of one or many files or a resolver which must rove all installed modules in advance, I'd like to hear about it. >>>>>>> >>>>>>>> OSGi does indeed support incremental install >>>>>>>> and while I accept it is probably not O(1) for each module, it would >>>>>>>> likely be no more than O(N), though I haven't done the maths yet to >>>>>>>> prove this. Bear in mind that in practice, for small N, the constant >>>>>>>> factors can result in O(1) being more expensive than O(N). I have seen >>>>>>>> OSGi used with many thousands of modules, so unless you have some data >>>>>>>> and a use-case showings package-level resolution as unacceptably slow, >>>>>>>> your concern just sounds like premature optimisation. >>>>>>> >>>>>>> Identifying algorithmic complexity during design phase is *never* premature optimization AFAICT. If you don't understand the complexity of the algorithms you're about to implement, and how they are expected to be applied to the problem at hand, then you're not ready for implementation yet. >>>>>>> >>>>>>> However, postulating that O(n) is okay because can sometimes be faster than O(1) without measuring it for the specific problem in question is definitely premature de-optimization. :-) >>>>>>> >>>>>>>> There are two reasons to have a packages with the same name in more >>>>>>>> than one JAR. The first is a situation called split packages, and it >>>>>>>> is highly undesirable because it causes the runtime model of the >>>>>>>> package to diverge from the compile-time model, and therefore things >>>>>>>> like package-private types and members stop working correctly. For >>>>>>>> this reason, OSGi's m2p imports support depending only upon a single >>>>>>>> exporter of a particular package, i.e. we do not aggregate all exports >>>>>>>> of that package. >>>>>>>> >>>>>>>> Unfortunately split packages are sometimes unavoidable in legacy code >>>>>>>> that cannot be refactored, e.g. the JDK. To support such scenarios >>>>>>>> OSGi has Require-Bundle, i.e. m2m. This does not negate the problems >>>>>>>> associated with m2m, it is simply a trade-off that we face with poorly >>>>>>>> factored legacy code. >>>>>>>> >>>>>>>> The second reason for multiple packages with the same name is when you >>>>>>>> explicitly want to install multiple versions of a library/API and have >>>>>>>> them all available within the same runtime. I wouldn't call this a >>>>>>>> "can of worms" exactly because it can be done without too much >>>>>>>> trouble, though for the sake of a simple life I personally avoid this >>>>>>>> situation unless it's necessary. >>>>>>> >>>>>>> Yes, and it is actually fairly common in practice to want two versions of something in the sense of the two versions being wholly different implementations (think apache commons logging versus jcl-over-slf4j for a trivial example which crops up a lot). There are reasons to use one or the other. Multiplicity of versions (in this respect) is something the module system has to handle gracefully and simply. >>>>>>> >>>>>>>>> Finally it should be perfectly clear to anyone who has read the original >>>>>>>>> requirements document that nothing in this module system should prevent OSGi >>>>>>>>> from functioning as it is, so there is absolutely no reason to assume that >>>>>>>>> any OSGi implementation is so threatened - especially if m2p linking is as >>>>>>>>> superior as has been expressed. Our module system (which is conceptually >>>>>>>>> similar to Jigsaw in many regards) in fact does support our OSGi >>>>>>>>> implementation quite effectively without itself implementing OSGi's >>>>>>>>> package-to-package resolution (which like I said throws O(1) out the >>>>>>>>> window). >>>>>>>> >>>>>>>> I agree that Jigsaw's existence doesn't threaten OSGi's, so long as >>>>>>>> Java 8 doesn't actually break OSGi (and if it did so, it would >>>>>>>> likewise break many other applications and could not be considered >>>>>>>> backwards compatible with Java 7). The two can interoperate through >>>>>>>> m2m-type dependencies. Tim Ellison started Project Penrose for the >>>>>>>> purpose of investigating, testing and deepening this collaboration. >>>>>>>> >>>>>>>> Neverthless, the point that I believe Glyn was making is the >>>>>>>> following. We accept that m2m dependencies are probably required for >>>>>>>> the JDK, which implies a module system like Jigsaw or >>>>>>>> OSGi/Require-Bundle rather than OSGi/Import-Package. However is it >>>>>>>> intended to be used for application modularisation as well? This is of >>>>>>>> course a question for the Jigsaw team rather than you, David. >>>>>>> >>>>>>> The question of whether Java SE 8 modules are intended to be used for application modularization is a question for the EG, not the Jigsaw team. The job of the Jigsaw team is really to implement a prototype which meets the requirements set forth by the EG, which may become the reference implementation at a future point. >>>>>>> >>>>>>> The question of whether the greater Java community will embrace the SE module system for applications is to be answered by the community only. I personally believe that to develop a module system which is not intended to be usable by the average standalone application is foolhardy, a great waste of effort, and is doomed to mediocrity, given the success we have had with such a system. >>>>>>> >>>>>>> However until there is an EG, these are all just as much questions for me as for anybody, and I can and will answer to the best of my knowledge and belief. >>>>>>> >>>>>>>> As a result of experience in developing and evolving large real-world >>>>>>>> applications using a module system that supports BOTH m2m and m2p >>>>>>>> dependencies, I believe it would be very unfortunate if a module >>>>>>>> system that supports ONLY m2m were to become widely used in the >>>>>>>> application space... not because OSGi can't handle the competition, >>>>>>>> but because those applications will be fragile and hard to evolve. >>>>>>> >>>>>>> I don't believe this to be the case. I think that the status quo doesn't result in particularly fragile applications, and such can be easy to evolve or hard depending on the quality of the application components and frameworks involved. I think that m2m dependencies enhance the status quo such that applications are somewhat less fragile (owing chiefly to the simple measure of preventing transitive dependencies from being exported by default), and quite easy to evolve as well (even large and complex frameworks rarely have more than a few dependencies, unless they themselves are overly fragmented (e.g. CXF as a nightmare example which yet still works fine under an m2m system)). >>>>>>> >>>>>>>> My question for you David is as follows. I understand that you prefer >>>>>>>> module dependencies, but do you believe that package dependencies have >>>>>>>> no value whatsoever and therefore should not be available to >>>>>>>> application developers in the Java 8 module system? If so, why did Red >>>>>>>> Hat create an OSGi implementation? >>>>>>> >>>>>>> I personally believe that they have some value, but that value is limited to interoperability with OSGi. I do not believe that it is a preferable model for most users or standalone applications, nor for larger applications such as our application server. I think that the requirement for extra tooling and the increased run-time complexity which is exposed to users nullifies the benefits. If someone really wants this variety of modularity, they should simply use OSGi. >>>>>>> >>>>>>> Red Hat created an OSGi implementation for many reasons, but the only significant one to me is that there are people who want to use their OSGi applications with our application server. I don't believe any other reason is even necessary. It's the same reason we strive to support any spec, from EJB 1.0 to Java EE 6 and beyond. Someone wants the functionality, so we deliver it to the best of our ability. >>>>>>> >>>>>>>> Kind regards >>>>>>>> Neil >>>>>>>> >>>>>>>>> >>>>>>>>> On 11/14/2011 01:49 AM, Glyn Normington wrote: >>>>>>>>>> >>>>>>>>>> I look forward to David's elaboration of why he thinks "using packages as >>>>>>>>>> a dependency unit is a terrible idea" to balance Peter's clear explanation >>>>>>>>>> of the benefits of m2p. >>>>>>>>>> >>>>>>>>>> Meanwhile, it's worth noting that, according to the requirements document, >>>>>>>>>> Jigsaw is aimed at platform modularisation and the platform being >>>>>>>>>> modularised has some non-optimal division of types across packages (see the >>>>>>>>>> package subsets requirement) which favour m2m dependencies. (Note that >>>>>>>>>> Apache Harmony was developed with modularity in mind and was able to exploit >>>>>>>>>> m2p, so platform modularisation per se needn't be limited to m2m.) >>>>>>>>>> >>>>>>>>>> So if Jigsaw excludes m2p, it will then be applicable to certain kinds of >>>>>>>>>> legacy code modularisation and less applicable to new module development and >>>>>>>>>> modularisation of existing code whose division into packages suits m2p. IIRC >>>>>>>>>> this was the original positioning of Jigsaw: for use primarily within the >>>>>>>>>> OpenJDK codebase and only exposed for application use because it was too >>>>>>>>>> inconvenient to hide it. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Glyn >>>>>>>>>> >>>>>>>>>> On 12 Nov 2011, at 11:59, Peter Kriens wrote: >>>>>>>>>> >>>>>>>>>>> Neither my wrath, nor the fact that I rarely if ever get angry is >>>>>>>>>>> relevant in this discussion ... This is a technical argument that are >>>>>>>>>>> solvable by technical people that share the same goals. I prefer package >>>>>>>>>>> dependencies because they address the excessive type coupling problem in >>>>>>>>>>> object oriented systems, not because they're part of OSGi. Let me argue my >>>>>>>>>>> case. >>>>>>>>>>> >>>>>>>>>>> Module-to-package dependencies (m2p) are preferable over module-to-module >>>>>>>>>>> dependencies (m2m) for many reasons but these are the most important >>>>>>>>>>> reasons: >>>>>>>>>>> >>>>>>>>>>> M2P is leverages the Java type system unlike m2m that must introduce new >>>>>>>>>>> namespaces outside the Java type system. >>>>>>>>>>> M2P can be used to break the transitive dependency chain, m2m suffers of >>>>>>>>>>> excessive coupling >>>>>>>>>>> >>>>>>>>>>> Since the first bullet's benefit should be clear I only argue the more >>>>>>>>>>> complex second bullet. >>>>>>>>>>> >>>>>>>>>>> A module is in many regards like a class. A class encapsulates members, >>>>>>>>>>> depends on other members/classes, and makes a few members accessible outside >>>>>>>>>>> the class. A module has a similar structure but then with types/packages as >>>>>>>>>>> members. >>>>>>>>>>> >>>>>>>>>>> After the initial success of Object Oriented Programming (OO) it was >>>>>>>>>>> quickly learned that reuse did not take place at the expected scale due to >>>>>>>>>>> excessive type coupling. The problem was that a class aggregated many >>>>>>>>>>> dependencies to simplify its implementation but these dependencies were >>>>>>>>>>> unrelated to the contract it implemented. Since class dependencies are >>>>>>>>>>> transitive most applications disappointingly became an almost fully >>>>>>>>>>> connected graph. >>>>>>>>>>> >>>>>>>>>>> Java's great innovation was the interface because it broke both the >>>>>>>>>>> transitivity and aggregation of dependencies. A class could now express its >>>>>>>>>>> dependency (use or implement) on a contract (the interface) and was >>>>>>>>>>> therefore fully type decoupled from the opposite site. >>>>>>>>>>> >>>>>>>>>>> An interface can act as a contract because it names the signature of a >>>>>>>>>>> set of methods so that the compiler can verify the client and the >>>>>>>>>>> implementer. >>>>>>>>>>> >>>>>>>>>>> Since a module has a very similar structure to a class it suffers from >>>>>>>>>>> exactly the same transitive aggregation of dependencies. This is not a >>>>>>>>>>> theory, look at the experiences with Maven >>>>>>>>>>> (http://www.sonatype.com/people/2011/04/how-not-to-download-the-internet/) >>>>>>>>>>> Again, this is not that maven is bad or developers are stupid, it is the >>>>>>>>>>> same underlying force that finally resulted in the Java interface. >>>>>>>>>>> >>>>>>>>>>> The parallel for the class' interface for modules is a named set of >>>>>>>>>>> interfaces. This concept already exists in Java: a package. Looking at >>>>>>>>>>> almost all JSRs it is clear that our industry already uses packages as >>>>>>>>>>> "interfaces" to provider implementations. >>>>>>>>>>> >>>>>>>>>>> Therefore, just like a class should not depend on other implementation >>>>>>>>>>> types, a module should preferably not depend on other modules. A module >>>>>>>>>>> should instead depend on contracts. Since modules will be used to provide >>>>>>>>>>> components from different sources managed with different life cycles the >>>>>>>>>>> excessive type coupling caused by m2m is even more damaging than in c2c. >>>>>>>>>>> Proper use of m2p creates significantly less type coupled systems than m2m, >>>>>>>>>>> the benefits should be obvious. >>>>>>>>>>> >>>>>>>>>>> Since there are use cases for m2m (non-type safe languages for example) I >>>>>>>>>>> do believe that Jigsaw should still support m2m. However, it would be >>>>>>>>>>> greatly beneficial to our industry if we could take advantage of the lessons >>>>>>>>>>> learned with the Java interface and realize how surprisingly important the >>>>>>>>>>> Java package actually is in our eco system. >>>>>>>>>>> >>>>>>>>>>> Kind regards, >>>>>>>>>>> >>>>>>>>>>> Peter Kriens >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 9 nov. 2011, at 15:04, David M. Lloyd wrote: >>>>>>>>>>> >>>>>>>>>>>> I'll just state now that using packages as a dependency unit is a >>>>>>>>>>>> terrible idea, and not some architectural revelation. That way, Peter's >>>>>>>>>>>> wrath will be largely directed at me. :-) >>>>>>>>>>>> >>>>>>>>>>>> On 11/09/2011 08:02 AM, Peter Kriens wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> I agree that tools are needed but we must be careful to not expect >>>>>>>>>>>>> tools to stopgap an architectural issue. I think it is important to first do >>>>>>>>>>>>> good architectural design leveraging existing tools (e.g. the Java type >>>>>>>>>>>>> system) before you try to add new tools. It is such a pity (but all to >>>>>>>>>>>>> common) that a design allows for classes of errors that would be impossible >>>>>>>>>>>>> with a slightly different design. >>>>>>>>>>>>> >>>>>>>>>>>>> Kind regards, >>>>>>>>>>>>> >>>>>>>>>>>>> Peter Kriens >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 9 nov. 2011, at 14:49, Alan Bateman wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> On 09/11/2011 13:04, Peter Kriens wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The issue is that maven problems are not caused because maven is bad >>>>>>>>>>>>>>> or that pom authors are stupid. The reason is that the module-to-module >>>>>>>>>>>>>>> dependency architecture in maven (and Jigsaw) is error prone ... >>>>>>>>>>>>>> >>>>>>>>>>>>>> This thread started out with someone asking about adding module >>>>>>>>>>>>>> declarations to existing JAR files, and in that context, I agree it can be >>>>>>>>>>>>>> error prone without good tools. I think things should be a lot better when >>>>>>>>>>>>>> modules are compiled. >>>>>>>>>>>>>> >>>>>>>>>>>>>> -Alan. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> - DML >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> - DML >>>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> - DML >>>>>> >>>>> >>>> >>> >>> > From brian at pontarelli.com Thu Nov 17 07:53:57 2011 From: brian at pontarelli.com (Brian Pontarelli) Date: Thu, 17 Nov 2011 08:53:57 -0700 Subject: Why package deps work (Was: Re: Converting plain JARs to Java modules) In-Reply-To: References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> <67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz> <4EC14A53.7070907@redhat.com> <4EC1F53D.7040309@redhat.com> <328B38A3-EB53-4AD2-970B-CE84F5C26F96@aQute.biz> <9B28CE0A-85DC-4F53-8A88-5830759DA4BC@pontarelli.com> <46755271-0738-4DD5-8FE0-54C5CD29FD29@pontarelli.com> Message-ID: On Nov 17, 2011, at 7:24 AM, Peter Kriens wrote: > On 17 nov. 2011, at 00:39, Brian Pontarelli wrote: >> >> I tend to think of it as possible, but in many cases not practical. > Fair enough, but should a Java module system not at least try to support this possibility? Definitely. The support should be there when it is possible and it should also cover all the other cases as well. -bp > > Kind regards, > > Peter Kriens > > > On 17 nov. 2011, at 00:39, Brian Pontarelli wrote: > >> >> On Nov 16, 2011, at 4:03 PM, Neil Bartlett wrote: >> >>> Hi Brian, >>> >>> On Wed, Nov 16, 2011 at 9:16 PM, Brian Pontarelli wrote: >>>> [snip] >>>> Let's use an example. Let's say you write a module to provide a user management system. It uses JPA only for persistence and does not use any hibernate or Eclipse link specific class, which BTW is highly unlikely. You write your module and then unit test it. >>> >>> How is this unlikely? On the contrary, it's highly unlikely that it >>> *could* use any Hibernate or EclipseLink specific class. First, the >>> module would be compiled with visibility only of the abstract API, >>> i.e. javax.persistence, and then at runtime we employ the module >>> system to ensure that it cannot see any specific implementation >>> classes, even via reflection. >> >> This is not how it works in practice. @Type is commonly used and not part of JPA. This happens A LOT. >> >>> >>> In fact if we expect our modules to be reusable -- so they can be used >>> not only in the project we are working on right now but in subsequent >>> projects over the next several years -- it's very likely that the >>> module will be used with multiple JPA providers. That's not to say we >>> switch JPA providers willy nilly, or without running a QA process over >>> the set of module that we intend to put into production. We also don't >>> claim that we never find and fix bugs in our modules when used against >>> alternative providers. >> >> I think a QA and testing process is great, but in practice I just don't see many developers doing this. That's my opinion and based on what I've seen in the Java world for many years. >> >> >>> >>>> Although it would be really neat if you could ship your module to the world and not really care what implementation of JPA is used, this is unlikely. In fact, whatever JPA implementation you unit tested against is probably the only one that you can definitely say your module will run with. Therefore, you would really need to unit test your module against all variations of JPA and that again is unlikely. >>> >>> The developer of the module should NOT care what implementation of JPA >>> they use, so long as it is an implementation that complies with the >>> contract. The deployer or assembler of an application is responsible >>> for finding and providing a compliant implementation. Testing of >>> course is done against a specific implementation, and does not >>> guarantee that other implementations can be used; they would have to >>> be tested by whoever wanted to use them. Nevertheless the work >>> required to fix issues arising from differences in interpretation of >>> contracts is much less than the porting effort required if the >>> original module developer had coded directly against an >>> implementation. >> >> Again, I disagree. In most cases, JPA implementations aren't portable. I've seen many companies try to switch and fail over and over again. This happens a lot because specifications tend to cover far fewer requirements than the implementations do. >> >> >>> >>> It's important to stress that none of this is theoretical. Exactly >>> this scenario involving JPA, along with many other similar scenarios, >>> can be done and is done in real world practice. >> >> All my examples of reflection and non-portable implementations are not theoretical, but my experience on real-world projects, working with a wide variety of companies and knowing many engineers who run into these issues all the time. >> >> >>> >>>> In other cases, it might work though. For example SLF4J is probably a safe bet to code against the API and then be able to drop in any of the implementations. >>> >>> This seems to contradict your previous statements. What is special >>> about SLF4J? If anything I would expect the JPA example to "safer", >>> since the JPA API was designed very cautiously by a JCP Expert Group, >>> whereas SLF4J came from a single author in the open source world (I >>> intend no slight against the SLF4J author; I simply have no >>> information on the quality and stability of its API). >> >> It doesn't really contradict it, I'm just stating that some cases of APIs and implementations work. The difference here is that SLF4J isn't a JSR specification that is implemented by multiple vendors with dramatically different goals with respect to their implementations (i.e. $$). SLF4j implementations are all written by the same group and all tested under the same set of unit tests to ensure that the API works perfectly (or as good as can be expected) across all implementations. >> >> I've found that JSRs and JCP Expert Groups tend not to do this as well as open source projects. JPA is a classic example of an API that lacks a number of key features and has numerous implementations all with different semantics and handling. >> >> Another classic example is @Inject. That thing is sorta a joke at this point. >> >> >>> >>>> I guess my point is that it isn't always as simple as you are implying. >>> >>> I agree it is not always simple, but it is possible. >> >> I tend to think of it as possible, but in many cases not practical. >> >> -bp >> >> >> >> >> >> >>> >>> Regards >>> Neil >>> >>>> >>>> >>>>> >>>>>> Again, are you talking about build time or runtime? Also, are you suggesting that Java determine your dependencies automatically including the version? Or is this merely to help developers find missing dependencies and remove unused dependencies? >>>>> If you compile class com.bar.Foo that refers to com.bar.Bar then the Foo.class and Bar.class file contain all their type dependencies and by implication their package dependencies. The version of the package can then for example be found from package-info.java. >>>>> >>>>> This technique is used extensively in bnd, an open source library used in Eclipse, IDEA, ant, maven, SBT, and others. It reads the class files in a JAR and generates the OSGi dependency information automatically. >>>> >>>> This is true unless there is reflection, which is common. JEE is based entirely on reflecting the top level entry points (servlets, filters, etc). There are many cases of configuration files defining implementations and transitive dependencies. This makes using a bytecode analyzer only a partial solution. >>>> >>>> -bp >>>> >>>> >>>> >>>> >>>>> >>>>> Hope this clarifies. Kind regards, >>>>> >>>>> Peter Kriens >>>>> >>>>> >>>>> >>>>>> >>>>>> -bp >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> Kind regards, >>>>>>> >>>>>>> Peter Kriens >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 15 nov. 2011, at 06:14, David M. Lloyd wrote: >>>>>>> >>>>>>>> On 11/14/2011 08:33 PM, Neil Bartlett wrote: >>>>>>>>> Hi David, >>>>>>>>> >>>>>>>>> More than happy to talk about reality, though Peter already was doing >>>>>>>>> so. Nothing he said was theoretical, it came directly from his >>>>>>>>> experiences and the experiences of developers who have been using OSGi >>>>>>>>> for around 12 years. >>>>>>>> >>>>>>>>> Before digging into the points in your message, I need to address the >>>>>>>>> subject line "Why package deps won't work". Package dependencies will >>>>>>>>> work and they do work, as is proven every day by the large number of >>>>>>>>> applications running on OSGi. To claim otherwise would be wilfully >>>>>>>>> ignorant, so I will assume you are being hyperbolic and really meant >>>>>>>>> to assert that module deps are just better than package deps. >>>>>>>>> Therefore I intend to argue on that basis and claim that package deps >>>>>>>>> are better than module deps; note that I don't claim that module >>>>>>>>> dependencies "won't work" because JBoss Modules is a proof point that >>>>>>>>> they do. >>>>>>>> >>>>>>>> There was an implied "... for Java SE 8 modules" after "won't work" (this is why I said "won't" rather than "don't") which apparently wasn't implied enough. >>>>>>>> >>>>>>>> Of course people use OSGi, or we wouldn't be arguing, but that alone isn't enough to make OSGi's dependency system preferable for the SE case (after all, far more people use plain old class paths). I believe that JBoss Modules is a very accurate representation of how an SE module system could function, right down to executing JARs on the command line with dependencies and modularizing certain JDK APIs. This is not a proof point, but strong evidence that it is an effective solution for the actual problem on the table, and that similar architectures are likely to succeed for the same problem domain. As to why package deps are not an example of an effective solution for this problem, I intend to illustrate in greater detail. >>>>>>>> >>>>>>>> Further responses and elaboration below. >>>>>>>> >>>>>>>>> On Mon, Nov 14, 2011 at 5:05 PM, David M. Lloyd wrote: >>>>>>>>>> The explanation is quite simple, really - each point can be pretty much >>>>>>>>>> wiped out by a heavy dose of reality. >>>>>>>>>> >>>>>>>>>> 1. "M2P is leverages the Java type system unlike m2m that must introduce new >>>>>>>>>> namespaces outside the Java type system." - this is just fluffy >>>>>>>>>> buzzwordology. Considering packages part of the Java type system is a >>>>>>>>>> pretty liberal interpretation of the term "type system" as they're basically >>>>>>>>>> just arbitrary name spaces. That said, there's nothing particularly "new" >>>>>>>>>> about namespacing modules. JARs have names. Projects have names. Maven >>>>>>>>>> uses its own namespaces for artifacts. Even the primitive JDK extension >>>>>>>>>> mechanism uses symbolic names. >>>>>>>>> >>>>>>>>> From Wikipedia (http://en.wikipedia.org/wiki/Type_system): "A type >>>>>>>>> system associates a type with each computed value ... the aim is to >>>>>>>>> prevent operations expecting a certain kind of value being used with >>>>>>>>> values for which that operation does not make sense." >>>>>>>>> >>>>>>>>> The following will not compile: >>>>>>>>> >>>>>>>>> import java.awt.List; >>>>>>>>> // ... >>>>>>>>> List list = new ArrayList(); >>>>>>>>> list.iterator(); // etc >>>>>>>>> >>>>>>>>> whereas the following will: >>>>>>>>> >>>>>>>>> import java.util.List; >>>>>>>>> // ... >>>>>>>>> List list = new ArrayList(); >>>>>>>>> list.iterator(); // etc >>>>>>>>> >>>>>>>>> Package names are part of the type system because using an incorrect >>>>>>>>> package name in Java source can result in type errors during >>>>>>>>> compilation, and because the type and available operations associated >>>>>>>>> with each value varies with the package name used. >>>>>>>>> >>>>>>>>> In contrast, it is impossible to obtain a type error at compilation by >>>>>>>>> incorrectly naming a JAR, project, Maven artefact etc, because none of >>>>>>>>> those things are ever referenced in Java source code. >>>>>>>> >>>>>>>> I stand by my statement. A package is part of the name of the type; it is not a type, nor a meta-type, nor anything like a type in and of itself. This makes them part of the language, sure, but not part of the type system, in my opinion. Thus, "a stretch". I don't feel any particular need to have the world agree with me on this though, other than as I state below... >>>>>>>> >>>>>>>>>> Even a simple convention of the parent-most package name for the name of a >>>>>>>>>> module is very simple to grasp and is in fact the solution we've used quite >>>>>>>>>> effectively thus far. Thus implying that module names are some new alien >>>>>>>>>> concept is really not valid. >>>>>>>>> >>>>>>>>> The claim is not that they are a new alien concept. The claim is that >>>>>>>>> they are not part of the Java language and type system, and as a >>>>>>>>> result of this (along with other problems) are less useful for >>>>>>>>> depending upon than packages. >>>>>>>> >>>>>>>> I don't see how not being part of the Java language makes module identifiers "less useful". I think that this argument artificially elevates the significance of the relationship between packages and the actual type system, and then uses that elevation as a proof that a system based on other namespace principles is deficient, without offering any real examples which actually relate to the type system. >>>>>>>> >>>>>>>>>> 2. "M2P can be used to break the transitive dependency chain, m2m suffers of >>>>>>>>>> excessive coupling." - How about some facts to back this up? I've found >>>>>>>>>> "m2m" coupling to be just right. In JBoss Modules, we do not export >>>>>>>>>> transitive dependencies by default. This results in a very simple and clean >>>>>>>>>> dependency graph between modules. Using package dependencies results in a >>>>>>>>>> *far more complex* dependency graph, because you need edges for every >>>>>>>>>> package even though *most of the time you want the whole module anyway*. >>>>>>>>>> Again, real world here. >>>>>>>>> >>>>>>>>> Peter is definitely also talking about the real world and so am I. In >>>>>>>>> an m2m dependency model you cannot avoid transitive dependencies, >>>>>>>>> whether you expose them to consumers or not. >>>>>>>> >>>>>>>> In *any* model you cannot avoid transitive dependencies. If module A exports module B as part of its API, then you get A and B, and that's life. If you only want part of B, then you filter it down. But with most existing libraries out there (and they easily number in the thousands), they're intended to be used whole, so this doesn't really happen too much in practice. >>>>>>>> >>>>>>>>> An importer of a module must be assumed to depend on the whole functionality of that module >>>>>>>>> whether or not that is actually the case, and therefore all of the >>>>>>>>> transitive dependencies must be present at runtime. In an m2p world we >>>>>>>>> have the opportunity to split modules and break apart dependency >>>>>>>>> graphs, because the unit of coupling is more granular. >>>>>>>> >>>>>>>> You could break things down by actual classes if granularity is the goal. But it isn't; it is so much less important than, say, performance, conceptual simplicity, usability, efficiency, memory footprint, etc. The fact is that the vast majority of modules you'll find published at, say Maven central, are designed to be used completely, not partially, and can and are used as such without issue. >>>>>>>> >>>>>>>> People just don't usually design modules to be imported by package, except perhaps as an afterthought concession to OSGi, and they don't use them that way at runtime either. It is so seldom a real problem in the SE world (or EE for that matter) once you step away from flat class paths. The simple fact is that most users think of their dependencies in the same terms that their IDEs and their build scripts do: by module. >>>>>>>> >>>>>>>> This is truly a case of creating a lot of complexity and extra work in the common case to avoid problems which are not a common case. I think it is much better to optimize for the common case, where people know exactly what dependencies they want (in terms of artifacts), and they just want to declare them and be done without (a) hunting down package lists or (b) running extra tooling. >>>>>>>> >>>>>>>>>> If we're just going to throw dogma around, I'll put it the other way: m2p is >>>>>>>>>> a design error by the OSGi spec designers which has since been embraced as a >>>>>>>>>> religion. >>>>>>>>> >>>>>>>>> Pejorative accusations about "dogma" or "religion" have no place in a >>>>>>>>> technical discussion. They are also untrue. All of the practising OSGi >>>>>>>>> developers I know have arrived at their support for package >>>>>>>>> dependencies as a result of real world experience, not because of some >>>>>>>>> willingness to bow down to the Almighty CPEG. I don't know of any >>>>>>>>> practising OSGi developer who has used both Require-Bundle (m2m) and >>>>>>>>> Import-Package (m2p) and actually prefers the former. >>>>>>>> >>>>>>>> Sure, but Require-Bundle is *not* the same thing as using module-to-module dependencies in a non-OSGi system. All OSGi life revolves around the resolver. In the SE world, you cannot have a resolver without introducing a lot of machinery which will definitely negatively impact boot performance in the case that the resolver runs at module load time, or add the complexity of centralized package index management in the case that the resolver would run at install time (as in, install into a module library/repository, not install in the OSGi sense). >>>>>>>> >>>>>>>> If in a typical installation a user has to do more than simply delete a file to remove a module, or drop a JAR into a directory to add a module, I believe we've run quite far off-track usability-wise. And if a user has to sit through a lengthy resolution process at runtime (even if only the first time) then we're off-track performance-wise, especially if such resolution must expand to include modules in the module repository which are not even loaded. >>>>>>>> >>>>>>>>> I do know several who started as strong supporters of Require-Bundle >>>>>>>>> and switched to being supporters of Import-Package, not because of >>>>>>>>> Peter's wrathful pontificating but because they encountered the >>>>>>>>> specific problems that he described and found that they were fixed by >>>>>>>>> using Import-Package, and indeed that everything worked so much more >>>>>>>>> cleanly that way. I'm in this camp myself, and so was Jeff McAffer >>>>>>>>> before he went over to the Dark Side (Microsoft). >>>>>>>> >>>>>>>> There are other, simpler ways to fix package conflict issues when they arise. Like simply excluding the offending package from the import. >>>>>>>> >>>>>>>> And as I've said before, I do not believe that Require-Bundle in the context of OSGi bundles is comparable to loading modules by name in the context of a module system such as Jigsaw or JBoss Modules and the types of modules that could be loaded thereby (the aforementioned thousands of existing modules). >>>>>>>> >>>>>>>>>> It offers no significant benefit, other than a couple of edge >>>>>>>>>> cases which are frankly just as well handled by m2m simply by adding package >>>>>>>>>> filters. Which, by the way, I haven't seen the need to do yet in our 200+ >>>>>>>>>> module environment, but which we do have the capability to do. >>>>>>>>> >>>>>>>>> Substitution, module refactoring and transitive decoupling are hardly >>>>>>>>> edge cases. >>>>>>>> >>>>>>>> Substitution isn't solely addressed by package dependencies. Nor is module refactoring. And additional transitive decoupling isn't something people actually want - as I said, the common case is that users want to be able to use any class from a module once they import that module. In the real world, it is very rare to only want to import part of a module! If you need this routinely, you've done something very unorthodox. Thus I stand by my statement. >>>>>>>> >>>>>>>>> However I can see how these issues might not yet have come >>>>>>>>> to the fore in a module system designed for a single product with a >>>>>>>>> small number of modules, and where that product has not yet been >>>>>>>>> through the mill of multiple version evolutions. >>>>>>>> >>>>>>>> Cute. I assure you that we have been around through a long, long history of a huge number of Java users doing everything under the sun to their class paths and class loading structures. We also have experience with operating system distribution and managing module distribution across a wide range of other programming languages, not just Java. Our module system has already stood up to reasonably large deployments (200+ modules) with excellent memory footprint and performance numbers and we've been processing usability feedback, including feedback from those with OSGi experience, which has been mostly positive. >>>>>>>> >>>>>>>> Furthermore when I was at JavaOne giving booth talks about JBoss Modules and AS7, I found that users expressed quite a bit of frustration at the OSGi model and were quite receptive to a more orthodox m2m system, for what that's worth. >>>>>>>> >>>>>>>>>> M2P is a solution just itching for a problem. >>>>>>>>> >>>>>>>>> This is also not a very useful statement, I assure you that the >>>>>>>>> problem came before the solution. Better to show why you think the >>>>>>>>> problem is invalid or should have been solved differently. >>>>>>>> >>>>>>>> There is no way you can convince me that this solution was invented with general modularity in mind. Back when OSGi was first formed, it was designed for the embedded market. The world of Java today is completely different. People just don't bundle their libraries by package, as was possibly expected in the early days of Java. If they did we wouldn't be having this discussion because packages and modules would already be one and the same. >>>>>>>> >>>>>>>> OSGi evolved to where it is today. It was not designed from the ground up with actual requirements which pertained to the problem at hand which is modularity of the SE platform and applications which run on it. So yeah I'm going to say, solution before problem. Taking a solution and trying to apply it retroactively to a different problem like this has never failed to bite me in the ass, personally. But I can't speak to everyone's experience. >>>>>>>> >>>>>>>>>> But you're going to have a >>>>>>>>>> tough time convincing me that users *want* to have to use special tooling >>>>>>>>>> because they want to depend on a module which has too many packages to list >>>>>>>>>> out by hand. And before you cry "wildcards", be sure to consider that some >>>>>>>>>> modules use package names which are subordinate to other modules' packages, >>>>>>>>>> which is a perfectly normal and allowable scenario. Using wildcards for >>>>>>>>>> package matching could cause amazing levels of havoc. >>>>>>>>> >>>>>>>>> OSGi never uses wildcards at runtime, and tools such as bnd do not >>>>>>>>> need wildcards in order to express package-level dependencies. They >>>>>>>>> extract the set of packages that were actually used by the code, all >>>>>>>>> of which are available in the class files. This is possible because >>>>>>>>> packages are part of the type system (see first point above). >>>>>>>>> >>>>>>>>> So it's not that I don't want to list dependencies by hand, rather I >>>>>>>>> only want to do it once. I am already forced to do it in the import >>>>>>>>> statements of my Java sources. If I had to repeat that dependency >>>>>>>>> information -- whether in m2p or m2m form -- then I would run the risk >>>>>>>>> of it getting out of step with the real dependencies. >>>>>>>> >>>>>>>> One of the critical logical gaps here which m2p ignores at a fundamental level is that *packages are not unique*. In the real world, packages are repeated across more than one module all the time, and not just due to versioning. There is *no way* you could take every JAR at Maven Central and plug it in to an m2p system. You'd have to break apart and repackage every single one, at least, and at worst you'd have to rewrite an awful lot of them under different package names to account for this restriction. You'd need to enforce package uniqueness across your whole library, for all time. I don't think this is a sane option. You're adding significant overhead to management, installation, and execution, for what? So you can have the privilege of needing a special tool to extract your dependencies for you? >>>>>>>> >>>>>>>>>> Using package dependencies means you either must have a master package index >>>>>>>>>> for linking, or you need a resolver which has to have analyzed every module >>>>>>>>>> you ever plan to load. Otherwise, O(1) loading of modules is impossible, >>>>>>>>>> which is absolutely 100% a deal-breaker for JDK modules which must be >>>>>>>>>> incrementally installable. And it forbids having packages with the same >>>>>>>>>> name in more than one JAR without bringing run-time versioning into the >>>>>>>>>> fold, which is a terrible, terrible can of worms. >>>>>>>>> >>>>>>>>> Could you please explain why O(1) is the only acceptable complexity >>>>>>>>> for installing modules. >>>>>>>> >>>>>>>> Because this is Java SE we're talking about. You're going to have a potentially huge number of modules installed in your system. Even if you get around the versioning issues somehow and fully externalize the index of all packages, you still have to load or traverse the centralized index for every module load. >>>>>>>> >>>>>>>> I say that not only is ?(1) the only acceptable complexity for installing modules but also loading them at run time. Linking modules at run time should be no worse than O(n) complexity for the number of dependencies the module has, including transitive dependencies. Loading classes and resources from modules should be ?(k) where k is the number of modules which contain a package or directory which matches the class or resource being loaded (almost always one, in my experience). >>>>>>>> >>>>>>>> Loading a module should normally be limited to a one-time O(1) disk access without searching or loading any other files or modules than the module artifact itself (and possibly its descriptor if they are held externally by a particular module repository implementation, which is useful but not necessary). Linking a module should be limited in disk access to loading only the modules it imports via direct dependency, or by the relatively rare partial or full re-export of a transitive dependency by a direct dependency. >>>>>>>> >>>>>>>> In particular, the expectation of accessing a central index has some potentially serious implications, facing either possible file lock contention by multiple threads or memory overhead of loading in the complete index in advance. It should be expected that modules are loaded concurrently, and such concurrency should not be hindered any more than necessary. Other possible implementations (such as file- or directory-per-package) have their own drawbacks as well or violate what I consider to be core requirements. >>>>>>>> >>>>>>>> If there are implementations of package-based resolution which don't involve either a centralized index of one or many files or a resolver which must rove all installed modules in advance, I'd like to hear about it. >>>>>>>> >>>>>>>>> OSGi does indeed support incremental install >>>>>>>>> and while I accept it is probably not O(1) for each module, it would >>>>>>>>> likely be no more than O(N), though I haven't done the maths yet to >>>>>>>>> prove this. Bear in mind that in practice, for small N, the constant >>>>>>>>> factors can result in O(1) being more expensive than O(N). I have seen >>>>>>>>> OSGi used with many thousands of modules, so unless you have some data >>>>>>>>> and a use-case showings package-level resolution as unacceptably slow, >>>>>>>>> your concern just sounds like premature optimisation. >>>>>>>> >>>>>>>> Identifying algorithmic complexity during design phase is *never* premature optimization AFAICT. If you don't understand the complexity of the algorithms you're about to implement, and how they are expected to be applied to the problem at hand, then you're not ready for implementation yet. >>>>>>>> >>>>>>>> However, postulating that O(n) is okay because can sometimes be faster than O(1) without measuring it for the specific problem in question is definitely premature de-optimization. :-) >>>>>>>> >>>>>>>>> There are two reasons to have a packages with the same name in more >>>>>>>>> than one JAR. The first is a situation called split packages, and it >>>>>>>>> is highly undesirable because it causes the runtime model of the >>>>>>>>> package to diverge from the compile-time model, and therefore things >>>>>>>>> like package-private types and members stop working correctly. For >>>>>>>>> this reason, OSGi's m2p imports support depending only upon a single >>>>>>>>> exporter of a particular package, i.e. we do not aggregate all exports >>>>>>>>> of that package. >>>>>>>>> >>>>>>>>> Unfortunately split packages are sometimes unavoidable in legacy code >>>>>>>>> that cannot be refactored, e.g. the JDK. To support such scenarios >>>>>>>>> OSGi has Require-Bundle, i.e. m2m. This does not negate the problems >>>>>>>>> associated with m2m, it is simply a trade-off that we face with poorly >>>>>>>>> factored legacy code. >>>>>>>>> >>>>>>>>> The second reason for multiple packages with the same name is when you >>>>>>>>> explicitly want to install multiple versions of a library/API and have >>>>>>>>> them all available within the same runtime. I wouldn't call this a >>>>>>>>> "can of worms" exactly because it can be done without too much >>>>>>>>> trouble, though for the sake of a simple life I personally avoid this >>>>>>>>> situation unless it's necessary. >>>>>>>> >>>>>>>> Yes, and it is actually fairly common in practice to want two versions of something in the sense of the two versions being wholly different implementations (think apache commons logging versus jcl-over-slf4j for a trivial example which crops up a lot). There are reasons to use one or the other. Multiplicity of versions (in this respect) is something the module system has to handle gracefully and simply. >>>>>>>> >>>>>>>>>> Finally it should be perfectly clear to anyone who has read the original >>>>>>>>>> requirements document that nothing in this module system should prevent OSGi >>>>>>>>>> from functioning as it is, so there is absolutely no reason to assume that >>>>>>>>>> any OSGi implementation is so threatened - especially if m2p linking is as >>>>>>>>>> superior as has been expressed. Our module system (which is conceptually >>>>>>>>>> similar to Jigsaw in many regards) in fact does support our OSGi >>>>>>>>>> implementation quite effectively without itself implementing OSGi's >>>>>>>>>> package-to-package resolution (which like I said throws O(1) out the >>>>>>>>>> window). >>>>>>>>> >>>>>>>>> I agree that Jigsaw's existence doesn't threaten OSGi's, so long as >>>>>>>>> Java 8 doesn't actually break OSGi (and if it did so, it would >>>>>>>>> likewise break many other applications and could not be considered >>>>>>>>> backwards compatible with Java 7). The two can interoperate through >>>>>>>>> m2m-type dependencies. Tim Ellison started Project Penrose for the >>>>>>>>> purpose of investigating, testing and deepening this collaboration. >>>>>>>>> >>>>>>>>> Neverthless, the point that I believe Glyn was making is the >>>>>>>>> following. We accept that m2m dependencies are probably required for >>>>>>>>> the JDK, which implies a module system like Jigsaw or >>>>>>>>> OSGi/Require-Bundle rather than OSGi/Import-Package. However is it >>>>>>>>> intended to be used for application modularisation as well? This is of >>>>>>>>> course a question for the Jigsaw team rather than you, David. >>>>>>>> >>>>>>>> The question of whether Java SE 8 modules are intended to be used for application modularization is a question for the EG, not the Jigsaw team. The job of the Jigsaw team is really to implement a prototype which meets the requirements set forth by the EG, which may become the reference implementation at a future point. >>>>>>>> >>>>>>>> The question of whether the greater Java community will embrace the SE module system for applications is to be answered by the community only. I personally believe that to develop a module system which is not intended to be usable by the average standalone application is foolhardy, a great waste of effort, and is doomed to mediocrity, given the success we have had with such a system. >>>>>>>> >>>>>>>> However until there is an EG, these are all just as much questions for me as for anybody, and I can and will answer to the best of my knowledge and belief. >>>>>>>> >>>>>>>>> As a result of experience in developing and evolving large real-world >>>>>>>>> applications using a module system that supports BOTH m2m and m2p >>>>>>>>> dependencies, I believe it would be very unfortunate if a module >>>>>>>>> system that supports ONLY m2m were to become widely used in the >>>>>>>>> application space... not because OSGi can't handle the competition, >>>>>>>>> but because those applications will be fragile and hard to evolve. >>>>>>>> >>>>>>>> I don't believe this to be the case. I think that the status quo doesn't result in particularly fragile applications, and such can be easy to evolve or hard depending on the quality of the application components and frameworks involved. I think that m2m dependencies enhance the status quo such that applications are somewhat less fragile (owing chiefly to the simple measure of preventing transitive dependencies from being exported by default), and quite easy to evolve as well (even large and complex frameworks rarely have more than a few dependencies, unless they themselves are overly fragmented (e.g. CXF as a nightmare example which yet still works fine under an m2m system)). >>>>>>>> >>>>>>>>> My question for you David is as follows. I understand that you prefer >>>>>>>>> module dependencies, but do you believe that package dependencies have >>>>>>>>> no value whatsoever and therefore should not be available to >>>>>>>>> application developers in the Java 8 module system? If so, why did Red >>>>>>>>> Hat create an OSGi implementation? >>>>>>>> >>>>>>>> I personally believe that they have some value, but that value is limited to interoperability with OSGi. I do not believe that it is a preferable model for most users or standalone applications, nor for larger applications such as our application server. I think that the requirement for extra tooling and the increased run-time complexity which is exposed to users nullifies the benefits. If someone really wants this variety of modularity, they should simply use OSGi. >>>>>>>> >>>>>>>> Red Hat created an OSGi implementation for many reasons, but the only significant one to me is that there are people who want to use their OSGi applications with our application server. I don't believe any other reason is even necessary. It's the same reason we strive to support any spec, from EJB 1.0 to Java EE 6 and beyond. Someone wants the functionality, so we deliver it to the best of our ability. >>>>>>>> >>>>>>>>> Kind regards >>>>>>>>> Neil >>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 11/14/2011 01:49 AM, Glyn Normington wrote: >>>>>>>>>>> >>>>>>>>>>> I look forward to David's elaboration of why he thinks "using packages as >>>>>>>>>>> a dependency unit is a terrible idea" to balance Peter's clear explanation >>>>>>>>>>> of the benefits of m2p. >>>>>>>>>>> >>>>>>>>>>> Meanwhile, it's worth noting that, according to the requirements document, >>>>>>>>>>> Jigsaw is aimed at platform modularisation and the platform being >>>>>>>>>>> modularised has some non-optimal division of types across packages (see the >>>>>>>>>>> package subsets requirement) which favour m2m dependencies. (Note that >>>>>>>>>>> Apache Harmony was developed with modularity in mind and was able to exploit >>>>>>>>>>> m2p, so platform modularisation per se needn't be limited to m2m.) >>>>>>>>>>> >>>>>>>>>>> So if Jigsaw excludes m2p, it will then be applicable to certain kinds of >>>>>>>>>>> legacy code modularisation and less applicable to new module development and >>>>>>>>>>> modularisation of existing code whose division into packages suits m2p. IIRC >>>>>>>>>>> this was the original positioning of Jigsaw: for use primarily within the >>>>>>>>>>> OpenJDK codebase and only exposed for application use because it was too >>>>>>>>>>> inconvenient to hide it. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Glyn >>>>>>>>>>> >>>>>>>>>>> On 12 Nov 2011, at 11:59, Peter Kriens wrote: >>>>>>>>>>> >>>>>>>>>>>> Neither my wrath, nor the fact that I rarely if ever get angry is >>>>>>>>>>>> relevant in this discussion ... This is a technical argument that are >>>>>>>>>>>> solvable by technical people that share the same goals. I prefer package >>>>>>>>>>>> dependencies because they address the excessive type coupling problem in >>>>>>>>>>>> object oriented systems, not because they're part of OSGi. Let me argue my >>>>>>>>>>>> case. >>>>>>>>>>>> >>>>>>>>>>>> Module-to-package dependencies (m2p) are preferable over module-to-module >>>>>>>>>>>> dependencies (m2m) for many reasons but these are the most important >>>>>>>>>>>> reasons: >>>>>>>>>>>> >>>>>>>>>>>> M2P is leverages the Java type system unlike m2m that must introduce new >>>>>>>>>>>> namespaces outside the Java type system. >>>>>>>>>>>> M2P can be used to break the transitive dependency chain, m2m suffers of >>>>>>>>>>>> excessive coupling >>>>>>>>>>>> >>>>>>>>>>>> Since the first bullet's benefit should be clear I only argue the more >>>>>>>>>>>> complex second bullet. >>>>>>>>>>>> >>>>>>>>>>>> A module is in many regards like a class. A class encapsulates members, >>>>>>>>>>>> depends on other members/classes, and makes a few members accessible outside >>>>>>>>>>>> the class. A module has a similar structure but then with types/packages as >>>>>>>>>>>> members. >>>>>>>>>>>> >>>>>>>>>>>> After the initial success of Object Oriented Programming (OO) it was >>>>>>>>>>>> quickly learned that reuse did not take place at the expected scale due to >>>>>>>>>>>> excessive type coupling. The problem was that a class aggregated many >>>>>>>>>>>> dependencies to simplify its implementation but these dependencies were >>>>>>>>>>>> unrelated to the contract it implemented. Since class dependencies are >>>>>>>>>>>> transitive most applications disappointingly became an almost fully >>>>>>>>>>>> connected graph. >>>>>>>>>>>> >>>>>>>>>>>> Java's great innovation was the interface because it broke both the >>>>>>>>>>>> transitivity and aggregation of dependencies. A class could now express its >>>>>>>>>>>> dependency (use or implement) on a contract (the interface) and was >>>>>>>>>>>> therefore fully type decoupled from the opposite site. >>>>>>>>>>>> >>>>>>>>>>>> An interface can act as a contract because it names the signature of a >>>>>>>>>>>> set of methods so that the compiler can verify the client and the >>>>>>>>>>>> implementer. >>>>>>>>>>>> >>>>>>>>>>>> Since a module has a very similar structure to a class it suffers from >>>>>>>>>>>> exactly the same transitive aggregation of dependencies. This is not a >>>>>>>>>>>> theory, look at the experiences with Maven >>>>>>>>>>>> (http://www.sonatype.com/people/2011/04/how-not-to-download-the-internet/) >>>>>>>>>>>> Again, this is not that maven is bad or developers are stupid, it is the >>>>>>>>>>>> same underlying force that finally resulted in the Java interface. >>>>>>>>>>>> >>>>>>>>>>>> The parallel for the class' interface for modules is a named set of >>>>>>>>>>>> interfaces. This concept already exists in Java: a package. Looking at >>>>>>>>>>>> almost all JSRs it is clear that our industry already uses packages as >>>>>>>>>>>> "interfaces" to provider implementations. >>>>>>>>>>>> >>>>>>>>>>>> Therefore, just like a class should not depend on other implementation >>>>>>>>>>>> types, a module should preferably not depend on other modules. A module >>>>>>>>>>>> should instead depend on contracts. Since modules will be used to provide >>>>>>>>>>>> components from different sources managed with different life cycles the >>>>>>>>>>>> excessive type coupling caused by m2m is even more damaging than in c2c. >>>>>>>>>>>> Proper use of m2p creates significantly less type coupled systems than m2m, >>>>>>>>>>>> the benefits should be obvious. >>>>>>>>>>>> >>>>>>>>>>>> Since there are use cases for m2m (non-type safe languages for example) I >>>>>>>>>>>> do believe that Jigsaw should still support m2m. However, it would be >>>>>>>>>>>> greatly beneficial to our industry if we could take advantage of the lessons >>>>>>>>>>>> learned with the Java interface and realize how surprisingly important the >>>>>>>>>>>> Java package actually is in our eco system. >>>>>>>>>>>> >>>>>>>>>>>> Kind regards, >>>>>>>>>>>> >>>>>>>>>>>> Peter Kriens >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 9 nov. 2011, at 15:04, David M. Lloyd wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I'll just state now that using packages as a dependency unit is a >>>>>>>>>>>>> terrible idea, and not some architectural revelation. That way, Peter's >>>>>>>>>>>>> wrath will be largely directed at me. :-) >>>>>>>>>>>>> >>>>>>>>>>>>> On 11/09/2011 08:02 AM, Peter Kriens wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> I agree that tools are needed but we must be careful to not expect >>>>>>>>>>>>>> tools to stopgap an architectural issue. I think it is important to first do >>>>>>>>>>>>>> good architectural design leveraging existing tools (e.g. the Java type >>>>>>>>>>>>>> system) before you try to add new tools. It is such a pity (but all to >>>>>>>>>>>>>> common) that a design allows for classes of errors that would be impossible >>>>>>>>>>>>>> with a slightly different design. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Kind regards, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Peter Kriens >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 9 nov. 2011, at 14:49, Alan Bateman wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 09/11/2011 13:04, Peter Kriens wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The issue is that maven problems are not caused because maven is bad >>>>>>>>>>>>>>>> or that pom authors are stupid. The reason is that the module-to-module >>>>>>>>>>>>>>>> dependency architecture in maven (and Jigsaw) is error prone ... >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This thread started out with someone asking about adding module >>>>>>>>>>>>>>> declarations to existing JAR files, and in that context, I agree it can be >>>>>>>>>>>>>>> error prone without good tools. I think things should be a lot better when >>>>>>>>>>>>>>> modules are compiled. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -Alan. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> - DML >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> - DML >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> - DML >>>>>>> >>>>>> >>>>> >>>> >>>> >> > From mcconnell at dpml.net Thu Nov 17 08:19:23 2011 From: mcconnell at dpml.net (Stephen McConnell) Date: Fri, 18 Nov 2011 02:49:23 +1030 Subject: Why package deps work (Was: Re: Converting plain JARs to Javamodules) In-Reply-To: References: <20111108223256.D33CA13EE@eggemoggin.niobe.net><4EBA5403.5030809@oracle.com><31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz><4EBA84E5.7060409@oracle.com><4EBA8885.6020808@redhat.com><67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz><4EC14A53.7070907@redhat.com><4EC1F53D.7040309@redhat.com><328B38A3-EB53-4AD2-970B-CE84F5C26F96@aQute.biz><9B28CE0A-85DC-4F53-8A88-5830759DA4BC@pontarelli.com><46755271-0738-4DD5-8FE0-54C5CD29FD29@pontarelli.com> Message-ID: <538719122E7B4BB08D3C0729C70C714E@GLORIA> Seems to me that two rather different discussions are going on here. * One discussion deals with the practical realities of managing the deployment of a system. * Another discussion deals with the management of solutions based on composition of systems. I don't see these two viewpoints as conflicting. In my mind I can build and package an API as a module (involving multiple packages). I can build and package a different system which happens to be an implementation of that API as a module (also involving multiple packages). My implementation module declares a dependency on my API module. Providing I have runtime access to the discovery and deployment of modules - I can deal with runtime composition independently of Jigsaw. Have I missed anything? Cheers, Steve. -----Original Message----- From: Brian Pontarelli Sent: Friday, November 18, 2011 2:23 AM To: Peter Kriens Cc: jigsaw-dev at openjdk.java.net Subject: Re: Why package deps work (Was: Re: Converting plain JARs to Javamodules) On Nov 17, 2011, at 7:24 AM, Peter Kriens wrote: > On 17 nov. 2011, at 00:39, Brian Pontarelli wrote: >> >> I tend to think of it as possible, but in many cases not practical. > Fair enough, but should a Java module system not at least try to support > this possibility? Definitely. The support should be there when it is possible and it should also cover all the other cases as well. -bp > > Kind regards, > > Peter Kriens > > > On 17 nov. 2011, at 00:39, Brian Pontarelli wrote: > >> >> On Nov 16, 2011, at 4:03 PM, Neil Bartlett wrote: >> >>> Hi Brian, >>> >>> On Wed, Nov 16, 2011 at 9:16 PM, Brian Pontarelli >>> wrote: >>>> [snip] >>>> Let's use an example. Let's say you write a module to provide a user >>>> management system. It uses JPA only for persistence and does not use >>>> any hibernate or Eclipse link specific class, which BTW is highly >>>> unlikely. You write your module and then unit test it. >>> >>> How is this unlikely? On the contrary, it's highly unlikely that it >>> *could* use any Hibernate or EclipseLink specific class. First, the >>> module would be compiled with visibility only of the abstract API, >>> i.e. javax.persistence, and then at runtime we employ the module >>> system to ensure that it cannot see any specific implementation >>> classes, even via reflection. >> >> This is not how it works in practice. @Type is commonly used and not part >> of JPA. This happens A LOT. >> >>> >>> In fact if we expect our modules to be reusable -- so they can be used >>> not only in the project we are working on right now but in subsequent >>> projects over the next several years -- it's very likely that the >>> module will be used with multiple JPA providers. That's not to say we >>> switch JPA providers willy nilly, or without running a QA process over >>> the set of module that we intend to put into production. We also don't >>> claim that we never find and fix bugs in our modules when used against >>> alternative providers. >> >> I think a QA and testing process is great, but in practice I just don't >> see many developers doing this. That's my opinion and based on what I've >> seen in the Java world for many years. >> >> >>> >>>> Although it would be really neat if you could ship your module to the >>>> world and not really care what implementation of JPA is used, this is >>>> unlikely. In fact, whatever JPA implementation you unit tested against >>>> is probably the only one that you can definitely say your module will >>>> run with. Therefore, you would really need to unit test your module >>>> against all variations of JPA and that again is unlikely. >>> >>> The developer of the module should NOT care what implementation of JPA >>> they use, so long as it is an implementation that complies with the >>> contract. The deployer or assembler of an application is responsible >>> for finding and providing a compliant implementation. Testing of >>> course is done against a specific implementation, and does not >>> guarantee that other implementations can be used; they would have to >>> be tested by whoever wanted to use them. Nevertheless the work >>> required to fix issues arising from differences in interpretation of >>> contracts is much less than the porting effort required if the >>> original module developer had coded directly against an >>> implementation. >> >> Again, I disagree. In most cases, JPA implementations aren't portable. >> I've seen many companies try to switch and fail over and over again. This >> happens a lot because specifications tend to cover far fewer requirements >> than the implementations do. >> >> >>> >>> It's important to stress that none of this is theoretical. Exactly >>> this scenario involving JPA, along with many other similar scenarios, >>> can be done and is done in real world practice. >> >> All my examples of reflection and non-portable implementations are not >> theoretical, but my experience on real-world projects, working with a >> wide variety of companies and knowing many engineers who run into these >> issues all the time. >> >> >>> >>>> In other cases, it might work though. For example SLF4J is probably a >>>> safe bet to code against the API and then be able to drop in any of the >>>> implementations. >>> >>> This seems to contradict your previous statements. What is special >>> about SLF4J? If anything I would expect the JPA example to "safer", >>> since the JPA API was designed very cautiously by a JCP Expert Group, >>> whereas SLF4J came from a single author in the open source world (I >>> intend no slight against the SLF4J author; I simply have no >>> information on the quality and stability of its API). >> >> It doesn't really contradict it, I'm just stating that some cases of APIs >> and implementations work. The difference here is that SLF4J isn't a JSR >> specification that is implemented by multiple vendors with dramatically >> different goals with respect to their implementations (i.e. $$). SLF4j >> implementations are all written by the same group and all tested under >> the same set of unit tests to ensure that the API works perfectly (or as >> good as can be expected) across all implementations. >> >> I've found that JSRs and JCP Expert Groups tend not to do this as well as >> open source projects. JPA is a classic example of an API that lacks a >> number of key features and has numerous implementations all with >> different semantics and handling. >> >> Another classic example is @Inject. That thing is sorta a joke at this >> point. >> >> >>> >>>> I guess my point is that it isn't always as simple as you are implying. >>> >>> I agree it is not always simple, but it is possible. >> >> I tend to think of it as possible, but in many cases not practical. >> >> -bp >> >> >> >> >> >> >>> >>> Regards >>> Neil >>> >>>> >>>> >>>>> >>>>>> Again, are you talking about build time or runtime? Also, are you >>>>>> suggesting that Java determine your dependencies automatically >>>>>> including the version? Or is this merely to help developers find >>>>>> missing dependencies and remove unused dependencies? >>>>> If you compile class com.bar.Foo that refers to com.bar.Bar then the >>>>> Foo.class and Bar.class file contain all their type dependencies and >>>>> by implication their package dependencies. The version of the package >>>>> can then for example be found from package-info.java. >>>>> >>>>> This technique is used extensively in bnd, an open source library used >>>>> in Eclipse, IDEA, ant, maven, SBT, and others. It reads the class >>>>> files in a JAR and generates the OSGi dependency information >>>>> automatically. >>>> >>>> This is true unless there is reflection, which is common. JEE is based >>>> entirely on reflecting the top level entry points (servlets, filters, >>>> etc). There are many cases of configuration files defining >>>> implementations and transitive dependencies. This makes using a >>>> bytecode analyzer only a partial solution. >>>> >>>> -bp >>>> >>>> >>>> >>>> >>>>> >>>>> Hope this clarifies. Kind regards, >>>>> >>>>> Peter Kriens >>>>> >>>>> >>>>> >>>>>> >>>>>> -bp >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> Kind regards, >>>>>>> >>>>>>> Peter Kriens >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 15 nov. 2011, at 06:14, David M. Lloyd wrote: >>>>>>> >>>>>>>> On 11/14/2011 08:33 PM, Neil Bartlett wrote: >>>>>>>>> Hi David, >>>>>>>>> >>>>>>>>> More than happy to talk about reality, though Peter already was >>>>>>>>> doing >>>>>>>>> so. Nothing he said was theoretical, it came directly from his >>>>>>>>> experiences and the experiences of developers who have been using >>>>>>>>> OSGi >>>>>>>>> for around 12 years. >>>>>>>> >>>>>>>>> Before digging into the points in your message, I need to address >>>>>>>>> the >>>>>>>>> subject line "Why package deps won't work". Package dependencies >>>>>>>>> will >>>>>>>>> work and they do work, as is proven every day by the large number >>>>>>>>> of >>>>>>>>> applications running on OSGi. To claim otherwise would be wilfully >>>>>>>>> ignorant, so I will assume you are being hyperbolic and really >>>>>>>>> meant >>>>>>>>> to assert that module deps are just better than package deps. >>>>>>>>> Therefore I intend to argue on that basis and claim that package >>>>>>>>> deps >>>>>>>>> are better than module deps; note that I don't claim that module >>>>>>>>> dependencies "won't work" because JBoss Modules is a proof point >>>>>>>>> that >>>>>>>>> they do. >>>>>>>> >>>>>>>> There was an implied "... for Java SE 8 modules" after "won't work" >>>>>>>> (this is why I said "won't" rather than "don't") which apparently >>>>>>>> wasn't implied enough. >>>>>>>> >>>>>>>> Of course people use OSGi, or we wouldn't be arguing, but that >>>>>>>> alone isn't enough to make OSGi's dependency system preferable for >>>>>>>> the SE case (after all, far more people use plain old class paths). >>>>>>>> I believe that JBoss Modules is a very accurate representation of >>>>>>>> how an SE module system could function, right down to executing >>>>>>>> JARs on the command line with dependencies and modularizing certain >>>>>>>> JDK APIs. This is not a proof point, but strong evidence that it >>>>>>>> is an effective solution for the actual problem on the table, and >>>>>>>> that similar architectures are likely to succeed for the same >>>>>>>> problem domain. As to why package deps are not an example of an >>>>>>>> effective solution for this problem, I intend to illustrate in >>>>>>>> greater detail. >>>>>>>> >>>>>>>> Further responses and elaboration below. >>>>>>>> >>>>>>>>> On Mon, Nov 14, 2011 at 5:05 PM, David M. >>>>>>>>> Lloyd wrote: >>>>>>>>>> The explanation is quite simple, really - each point can be >>>>>>>>>> pretty much >>>>>>>>>> wiped out by a heavy dose of reality. >>>>>>>>>> >>>>>>>>>> 1. "M2P is leverages the Java type system unlike m2m that must >>>>>>>>>> introduce new >>>>>>>>>> namespaces outside the Java type system." - this is just fluffy >>>>>>>>>> buzzwordology. Considering packages part of the Java type system >>>>>>>>>> is a >>>>>>>>>> pretty liberal interpretation of the term "type system" as >>>>>>>>>> they're basically >>>>>>>>>> just arbitrary name spaces. That said, there's nothing >>>>>>>>>> particularly "new" >>>>>>>>>> about namespacing modules. JARs have names. Projects have names. >>>>>>>>>> Maven >>>>>>>>>> uses its own namespaces for artifacts. Even the primitive JDK >>>>>>>>>> extension >>>>>>>>>> mechanism uses symbolic names. >>>>>>>>> >>>>>>>>> From Wikipedia (http://en.wikipedia.org/wiki/Type_system): "A type >>>>>>>>> system associates a type with each computed value ... the aim is >>>>>>>>> to >>>>>>>>> prevent operations expecting a certain kind of value being used >>>>>>>>> with >>>>>>>>> values for which that operation does not make sense." >>>>>>>>> >>>>>>>>> The following will not compile: >>>>>>>>> >>>>>>>>> import java.awt.List; >>>>>>>>> // ... >>>>>>>>> List list = new ArrayList(); >>>>>>>>> list.iterator(); // etc >>>>>>>>> >>>>>>>>> whereas the following will: >>>>>>>>> >>>>>>>>> import java.util.List; >>>>>>>>> // ... >>>>>>>>> List list = new ArrayList(); >>>>>>>>> list.iterator(); // etc >>>>>>>>> >>>>>>>>> Package names are part of the type system because using an >>>>>>>>> incorrect >>>>>>>>> package name in Java source can result in type errors during >>>>>>>>> compilation, and because the type and available operations >>>>>>>>> associated >>>>>>>>> with each value varies with the package name used. >>>>>>>>> >>>>>>>>> In contrast, it is impossible to obtain a type error at >>>>>>>>> compilation by >>>>>>>>> incorrectly naming a JAR, project, Maven artefact etc, because >>>>>>>>> none of >>>>>>>>> those things are ever referenced in Java source code. >>>>>>>> >>>>>>>> I stand by my statement. A package is part of the name of the >>>>>>>> type; it is not a type, nor a meta-type, nor anything like a type >>>>>>>> in and of itself. This makes them part of the language, sure, but >>>>>>>> not part of the type system, in my opinion. Thus, "a stretch". I >>>>>>>> don't feel any particular need to have the world agree with me on >>>>>>>> this though, other than as I state below... >>>>>>>> >>>>>>>>>> Even a simple convention of the parent-most package name for the >>>>>>>>>> name of a >>>>>>>>>> module is very simple to grasp and is in fact the solution we've >>>>>>>>>> used quite >>>>>>>>>> effectively thus far. Thus implying that module names are some >>>>>>>>>> new alien >>>>>>>>>> concept is really not valid. >>>>>>>>> >>>>>>>>> The claim is not that they are a new alien concept. The claim is >>>>>>>>> that >>>>>>>>> they are not part of the Java language and type system, and as a >>>>>>>>> result of this (along with other problems) are less useful for >>>>>>>>> depending upon than packages. >>>>>>>> >>>>>>>> I don't see how not being part of the Java language makes module >>>>>>>> identifiers "less useful". I think that this argument artificially >>>>>>>> elevates the significance of the relationship between packages and >>>>>>>> the actual type system, and then uses that elevation as a proof >>>>>>>> that a system based on other namespace principles is deficient, >>>>>>>> without offering any real examples which actually relate to the >>>>>>>> type system. >>>>>>>> >>>>>>>>>> 2. "M2P can be used to break the transitive dependency chain, m2m >>>>>>>>>> suffers of >>>>>>>>>> excessive coupling." - How about some facts to back this up? >>>>>>>>>> I've found >>>>>>>>>> "m2m" coupling to be just right. In JBoss Modules, we do not >>>>>>>>>> export >>>>>>>>>> transitive dependencies by default. This results in a very >>>>>>>>>> simple and clean >>>>>>>>>> dependency graph between modules. Using package dependencies >>>>>>>>>> results in a >>>>>>>>>> *far more complex* dependency graph, because you need edges for >>>>>>>>>> every >>>>>>>>>> package even though *most of the time you want the whole module >>>>>>>>>> anyway*. >>>>>>>>>> Again, real world here. >>>>>>>>> >>>>>>>>> Peter is definitely also talking about the real world and so am I. >>>>>>>>> In >>>>>>>>> an m2m dependency model you cannot avoid transitive dependencies, >>>>>>>>> whether you expose them to consumers or not. >>>>>>>> >>>>>>>> In *any* model you cannot avoid transitive dependencies. If module >>>>>>>> A exports module B as part of its API, then you get A and B, and >>>>>>>> that's life. If you only want part of B, then you filter it down. >>>>>>>> But with most existing libraries out there (and they easily number >>>>>>>> in the thousands), they're intended to be used whole, so this >>>>>>>> doesn't really happen too much in practice. >>>>>>>> >>>>>>>>> An importer of a module must be assumed to depend on the whole >>>>>>>>> functionality of that module >>>>>>>>> whether or not that is actually the case, and therefore all of the >>>>>>>>> transitive dependencies must be present at runtime. In an m2p >>>>>>>>> world we >>>>>>>>> have the opportunity to split modules and break apart dependency >>>>>>>>> graphs, because the unit of coupling is more granular. >>>>>>>> >>>>>>>> You could break things down by actual classes if granularity is the >>>>>>>> goal. But it isn't; it is so much less important than, say, >>>>>>>> performance, conceptual simplicity, usability, efficiency, memory >>>>>>>> footprint, etc. The fact is that the vast majority of modules >>>>>>>> you'll find published at, say Maven central, are designed to be >>>>>>>> used completely, not partially, and can and are used as such >>>>>>>> without issue. >>>>>>>> >>>>>>>> People just don't usually design modules to be imported by package, >>>>>>>> except perhaps as an afterthought concession to OSGi, and they >>>>>>>> don't use them that way at runtime either. It is so seldom a real >>>>>>>> problem in the SE world (or EE for that matter) once you step away >>>>>>>> from flat class paths. The simple fact is that most users think of >>>>>>>> their dependencies in the same terms that their IDEs and their >>>>>>>> build scripts do: by module. >>>>>>>> >>>>>>>> This is truly a case of creating a lot of complexity and extra work >>>>>>>> in the common case to avoid problems which are not a common case. >>>>>>>> I think it is much better to optimize for the common case, where >>>>>>>> people know exactly what dependencies they want (in terms of >>>>>>>> artifacts), and they just want to declare them and be done without >>>>>>>> (a) hunting down package lists or (b) running extra tooling. >>>>>>>> >>>>>>>>>> If we're just going to throw dogma around, I'll put it the other >>>>>>>>>> way: m2p is >>>>>>>>>> a design error by the OSGi spec designers which has since been >>>>>>>>>> embraced as a >>>>>>>>>> religion. >>>>>>>>> >>>>>>>>> Pejorative accusations about "dogma" or "religion" have no place >>>>>>>>> in a >>>>>>>>> technical discussion. They are also untrue. All of the practising >>>>>>>>> OSGi >>>>>>>>> developers I know have arrived at their support for package >>>>>>>>> dependencies as a result of real world experience, not because of >>>>>>>>> some >>>>>>>>> willingness to bow down to the Almighty CPEG. I don't know of any >>>>>>>>> practising OSGi developer who has used both Require-Bundle (m2m) >>>>>>>>> and >>>>>>>>> Import-Package (m2p) and actually prefers the former. >>>>>>>> >>>>>>>> Sure, but Require-Bundle is *not* the same thing as using >>>>>>>> module-to-module dependencies in a non-OSGi system. All OSGi life >>>>>>>> revolves around the resolver. In the SE world, you cannot have a >>>>>>>> resolver without introducing a lot of machinery which will >>>>>>>> definitely negatively impact boot performance in the case that the >>>>>>>> resolver runs at module load time, or add the complexity of >>>>>>>> centralized package index management in the case that the resolver >>>>>>>> would run at install time (as in, install into a module >>>>>>>> library/repository, not install in the OSGi sense). >>>>>>>> >>>>>>>> If in a typical installation a user has to do more than simply >>>>>>>> delete a file to remove a module, or drop a JAR into a directory to >>>>>>>> add a module, I believe we've run quite far off-track >>>>>>>> usability-wise. And if a user has to sit through a lengthy >>>>>>>> resolution process at runtime (even if only the first time) then >>>>>>>> we're off-track performance-wise, especially if such resolution >>>>>>>> must expand to include modules in the module repository which are >>>>>>>> not even loaded. >>>>>>>> >>>>>>>>> I do know several who started as strong supporters of >>>>>>>>> Require-Bundle >>>>>>>>> and switched to being supporters of Import-Package, not because of >>>>>>>>> Peter's wrathful pontificating but because they encountered the >>>>>>>>> specific problems that he described and found that they were fixed >>>>>>>>> by >>>>>>>>> using Import-Package, and indeed that everything worked so much >>>>>>>>> more >>>>>>>>> cleanly that way. I'm in this camp myself, and so was Jeff McAffer >>>>>>>>> before he went over to the Dark Side (Microsoft). >>>>>>>> >>>>>>>> There are other, simpler ways to fix package conflict issues when >>>>>>>> they arise. Like simply excluding the offending package from the >>>>>>>> import. >>>>>>>> >>>>>>>> And as I've said before, I do not believe that Require-Bundle in >>>>>>>> the context of OSGi bundles is comparable to loading modules by >>>>>>>> name in the context of a module system such as Jigsaw or JBoss >>>>>>>> Modules and the types of modules that could be loaded thereby (the >>>>>>>> aforementioned thousands of existing modules). >>>>>>>> >>>>>>>>>> It offers no significant benefit, other than a couple of edge >>>>>>>>>> cases which are frankly just as well handled by m2m simply by >>>>>>>>>> adding package >>>>>>>>>> filters. Which, by the way, I haven't seen the need to do yet in >>>>>>>>>> our 200+ >>>>>>>>>> module environment, but which we do have the capability to do. >>>>>>>>> >>>>>>>>> Substitution, module refactoring and transitive decoupling are >>>>>>>>> hardly >>>>>>>>> edge cases. >>>>>>>> >>>>>>>> Substitution isn't solely addressed by package dependencies. Nor >>>>>>>> is module refactoring. And additional transitive decoupling isn't >>>>>>>> something people actually want - as I said, the common case is that >>>>>>>> users want to be able to use any class from a module once they >>>>>>>> import that module. In the real world, it is very rare to only >>>>>>>> want to import part of a module! If you need this routinely, >>>>>>>> you've done something very unorthodox. Thus I stand by my >>>>>>>> statement. >>>>>>>> >>>>>>>>> However I can see how these issues might not yet have come >>>>>>>>> to the fore in a module system designed for a single product with >>>>>>>>> a >>>>>>>>> small number of modules, and where that product has not yet been >>>>>>>>> through the mill of multiple version evolutions. >>>>>>>> >>>>>>>> Cute. I assure you that we have been around through a long, long >>>>>>>> history of a huge number of Java users doing everything under the >>>>>>>> sun to their class paths and class loading structures. We also >>>>>>>> have experience with operating system distribution and managing >>>>>>>> module distribution across a wide range of other programming >>>>>>>> languages, not just Java. Our module system has already stood up >>>>>>>> to reasonably large deployments (200+ modules) with excellent >>>>>>>> memory footprint and performance numbers and we've been processing >>>>>>>> usability feedback, including feedback from those with OSGi >>>>>>>> experience, which has been mostly positive. >>>>>>>> >>>>>>>> Furthermore when I was at JavaOne giving booth talks about JBoss >>>>>>>> Modules and AS7, I found that users expressed quite a bit of >>>>>>>> frustration at the OSGi model and were quite receptive to a more >>>>>>>> orthodox m2m system, for what that's worth. >>>>>>>> >>>>>>>>>> M2P is a solution just itching for a problem. >>>>>>>>> >>>>>>>>> This is also not a very useful statement, I assure you that the >>>>>>>>> problem came before the solution. Better to show why you think the >>>>>>>>> problem is invalid or should have been solved differently. >>>>>>>> >>>>>>>> There is no way you can convince me that this solution was invented >>>>>>>> with general modularity in mind. Back when OSGi was first formed, >>>>>>>> it was designed for the embedded market. The world of Java today >>>>>>>> is completely different. People just don't bundle their libraries >>>>>>>> by package, as was possibly expected in the early days of Java. If >>>>>>>> they did we wouldn't be having this discussion because packages and >>>>>>>> modules would already be one and the same. >>>>>>>> >>>>>>>> OSGi evolved to where it is today. It was not designed from the >>>>>>>> ground up with actual requirements which pertained to the problem >>>>>>>> at hand which is modularity of the SE platform and applications >>>>>>>> which run on it. So yeah I'm going to say, solution before >>>>>>>> problem. Taking a solution and trying to apply it retroactively to >>>>>>>> a different problem like this has never failed to bite me in the >>>>>>>> ass, personally. But I can't speak to everyone's experience. >>>>>>>> >>>>>>>>>> But you're going to have a >>>>>>>>>> tough time convincing me that users *want* to have to use special >>>>>>>>>> tooling >>>>>>>>>> because they want to depend on a module which has too many >>>>>>>>>> packages to list >>>>>>>>>> out by hand. And before you cry "wildcards", be sure to consider >>>>>>>>>> that some >>>>>>>>>> modules use package names which are subordinate to other modules' >>>>>>>>>> packages, >>>>>>>>>> which is a perfectly normal and allowable scenario. Using >>>>>>>>>> wildcards for >>>>>>>>>> package matching could cause amazing levels of havoc. >>>>>>>>> >>>>>>>>> OSGi never uses wildcards at runtime, and tools such as bnd do not >>>>>>>>> need wildcards in order to express package-level dependencies. >>>>>>>>> They >>>>>>>>> extract the set of packages that were actually used by the code, >>>>>>>>> all >>>>>>>>> of which are available in the class files. This is possible >>>>>>>>> because >>>>>>>>> packages are part of the type system (see first point above). >>>>>>>>> >>>>>>>>> So it's not that I don't want to list dependencies by hand, rather >>>>>>>>> I >>>>>>>>> only want to do it once. I am already forced to do it in the >>>>>>>>> import >>>>>>>>> statements of my Java sources. If I had to repeat that dependency >>>>>>>>> information -- whether in m2p or m2m form -- then I would run the >>>>>>>>> risk >>>>>>>>> of it getting out of step with the real dependencies. >>>>>>>> >>>>>>>> One of the critical logical gaps here which m2p ignores at a >>>>>>>> fundamental level is that *packages are not unique*. In the real >>>>>>>> world, packages are repeated across more than one module all the >>>>>>>> time, and not just due to versioning. There is *no way* you could >>>>>>>> take every JAR at Maven Central and plug it in to an m2p system. >>>>>>>> You'd have to break apart and repackage every single one, at least, >>>>>>>> and at worst you'd have to rewrite an awful lot of them under >>>>>>>> different package names to account for this restriction. You'd >>>>>>>> need to enforce package uniqueness across your whole library, for >>>>>>>> all time. I don't think this is a sane option. You're adding >>>>>>>> significant overhead to management, installation, and execution, >>>>>>>> for what? So you can have the privilege of needing a special tool >>>>>>>> to extract your dependencies for you? >>>>>>>> >>>>>>>>>> Using package dependencies means you either must have a master >>>>>>>>>> package index >>>>>>>>>> for linking, or you need a resolver which has to have analyzed >>>>>>>>>> every module >>>>>>>>>> you ever plan to load. Otherwise, O(1) loading of modules is >>>>>>>>>> impossible, >>>>>>>>>> which is absolutely 100% a deal-breaker for JDK modules which >>>>>>>>>> must be >>>>>>>>>> incrementally installable. And it forbids having packages with >>>>>>>>>> the same >>>>>>>>>> name in more than one JAR without bringing run-time versioning >>>>>>>>>> into the >>>>>>>>>> fold, which is a terrible, terrible can of worms. >>>>>>>>> >>>>>>>>> Could you please explain why O(1) is the only acceptable >>>>>>>>> complexity >>>>>>>>> for installing modules. >>>>>>>> >>>>>>>> Because this is Java SE we're talking about. You're going to have >>>>>>>> a potentially huge number of modules installed in your system. >>>>>>>> Even if you get around the versioning issues somehow and fully >>>>>>>> externalize the index of all packages, you still have to load or >>>>>>>> traverse the centralized index for every module load. >>>>>>>> >>>>>>>> I say that not only is ?(1) the only acceptable complexity for >>>>>>>> installing modules but also loading them at run time. Linking >>>>>>>> modules at run time should be no worse than O(n) complexity for the >>>>>>>> number of dependencies the module has, including transitive >>>>>>>> dependencies. Loading classes and resources from modules should be >>>>>>>> ?(k) where k is the number of modules which contain a package or >>>>>>>> directory which matches the class or resource being loaded (almost >>>>>>>> always one, in my experience). >>>>>>>> >>>>>>>> Loading a module should normally be limited to a one-time O(1) disk >>>>>>>> access without searching or loading any other files or modules than >>>>>>>> the module artifact itself (and possibly its descriptor if they are >>>>>>>> held externally by a particular module repository implementation, >>>>>>>> which is useful but not necessary). Linking a module should be >>>>>>>> limited in disk access to loading only the modules it imports via >>>>>>>> direct dependency, or by the relatively rare partial or full >>>>>>>> re-export of a transitive dependency by a direct dependency. >>>>>>>> >>>>>>>> In particular, the expectation of accessing a central index has >>>>>>>> some potentially serious implications, facing either possible file >>>>>>>> lock contention by multiple threads or memory overhead of loading >>>>>>>> in the complete index in advance. It should be expected that >>>>>>>> modules are loaded concurrently, and such concurrency should not be >>>>>>>> hindered any more than necessary. Other possible implementations >>>>>>>> (such as file- or directory-per-package) have their own drawbacks >>>>>>>> as well or violate what I consider to be core requirements. >>>>>>>> >>>>>>>> If there are implementations of package-based resolution which >>>>>>>> don't involve either a centralized index of one or many files or a >>>>>>>> resolver which must rove all installed modules in advance, I'd like >>>>>>>> to hear about it. >>>>>>>> >>>>>>>>> OSGi does indeed support incremental install >>>>>>>>> and while I accept it is probably not O(1) for each module, it >>>>>>>>> would >>>>>>>>> likely be no more than O(N), though I haven't done the maths yet >>>>>>>>> to >>>>>>>>> prove this. Bear in mind that in practice, for small N, the >>>>>>>>> constant >>>>>>>>> factors can result in O(1) being more expensive than O(N). I have >>>>>>>>> seen >>>>>>>>> OSGi used with many thousands of modules, so unless you have some >>>>>>>>> data >>>>>>>>> and a use-case showings package-level resolution as unacceptably >>>>>>>>> slow, >>>>>>>>> your concern just sounds like premature optimisation. >>>>>>>> >>>>>>>> Identifying algorithmic complexity during design phase is *never* >>>>>>>> premature optimization AFAICT. If you don't understand the >>>>>>>> complexity of the algorithms you're about to implement, and how >>>>>>>> they are expected to be applied to the problem at hand, then you're >>>>>>>> not ready for implementation yet. >>>>>>>> >>>>>>>> However, postulating that O(n) is okay because can sometimes be >>>>>>>> faster than O(1) without measuring it for the specific problem in >>>>>>>> question is definitely premature de-optimization. :-) >>>>>>>> >>>>>>>>> There are two reasons to have a packages with the same name in >>>>>>>>> more >>>>>>>>> than one JAR. The first is a situation called split packages, and >>>>>>>>> it >>>>>>>>> is highly undesirable because it causes the runtime model of the >>>>>>>>> package to diverge from the compile-time model, and therefore >>>>>>>>> things >>>>>>>>> like package-private types and members stop working correctly. For >>>>>>>>> this reason, OSGi's m2p imports support depending only upon a >>>>>>>>> single >>>>>>>>> exporter of a particular package, i.e. we do not aggregate all >>>>>>>>> exports >>>>>>>>> of that package. >>>>>>>>> >>>>>>>>> Unfortunately split packages are sometimes unavoidable in legacy >>>>>>>>> code >>>>>>>>> that cannot be refactored, e.g. the JDK. To support such scenarios >>>>>>>>> OSGi has Require-Bundle, i.e. m2m. This does not negate the >>>>>>>>> problems >>>>>>>>> associated with m2m, it is simply a trade-off that we face with >>>>>>>>> poorly >>>>>>>>> factored legacy code. >>>>>>>>> >>>>>>>>> The second reason for multiple packages with the same name is when >>>>>>>>> you >>>>>>>>> explicitly want to install multiple versions of a library/API and >>>>>>>>> have >>>>>>>>> them all available within the same runtime. I wouldn't call this a >>>>>>>>> "can of worms" exactly because it can be done without too much >>>>>>>>> trouble, though for the sake of a simple life I personally avoid >>>>>>>>> this >>>>>>>>> situation unless it's necessary. >>>>>>>> >>>>>>>> Yes, and it is actually fairly common in practice to want two >>>>>>>> versions of something in the sense of the two versions being wholly >>>>>>>> different implementations (think apache commons logging versus >>>>>>>> jcl-over-slf4j for a trivial example which crops up a lot). There >>>>>>>> are reasons to use one or the other. Multiplicity of versions (in >>>>>>>> this respect) is something the module system has to handle >>>>>>>> gracefully and simply. >>>>>>>> >>>>>>>>>> Finally it should be perfectly clear to anyone who has read the >>>>>>>>>> original >>>>>>>>>> requirements document that nothing in this module system should >>>>>>>>>> prevent OSGi >>>>>>>>>> from functioning as it is, so there is absolutely no reason to >>>>>>>>>> assume that >>>>>>>>>> any OSGi implementation is so threatened - especially if m2p >>>>>>>>>> linking is as >>>>>>>>>> superior as has been expressed. Our module system (which is >>>>>>>>>> conceptually >>>>>>>>>> similar to Jigsaw in many regards) in fact does support our OSGi >>>>>>>>>> implementation quite effectively without itself implementing >>>>>>>>>> OSGi's >>>>>>>>>> package-to-package resolution (which like I said throws O(1) out >>>>>>>>>> the >>>>>>>>>> window). >>>>>>>>> >>>>>>>>> I agree that Jigsaw's existence doesn't threaten OSGi's, so long >>>>>>>>> as >>>>>>>>> Java 8 doesn't actually break OSGi (and if it did so, it would >>>>>>>>> likewise break many other applications and could not be considered >>>>>>>>> backwards compatible with Java 7). The two can interoperate >>>>>>>>> through >>>>>>>>> m2m-type dependencies. Tim Ellison started Project Penrose for the >>>>>>>>> purpose of investigating, testing and deepening this >>>>>>>>> collaboration. >>>>>>>>> >>>>>>>>> Neverthless, the point that I believe Glyn was making is the >>>>>>>>> following. We accept that m2m dependencies are probably required >>>>>>>>> for >>>>>>>>> the JDK, which implies a module system like Jigsaw or >>>>>>>>> OSGi/Require-Bundle rather than OSGi/Import-Package. However is it >>>>>>>>> intended to be used for application modularisation as well? This >>>>>>>>> is of >>>>>>>>> course a question for the Jigsaw team rather than you, David. >>>>>>>> >>>>>>>> The question of whether Java SE 8 modules are intended to be used >>>>>>>> for application modularization is a question for the EG, not the >>>>>>>> Jigsaw team. The job of the Jigsaw team is really to implement a >>>>>>>> prototype which meets the requirements set forth by the EG, which >>>>>>>> may become the reference implementation at a future point. >>>>>>>> >>>>>>>> The question of whether the greater Java community will embrace the >>>>>>>> SE module system for applications is to be answered by the >>>>>>>> community only. I personally believe that to develop a module >>>>>>>> system which is not intended to be usable by the average standalone >>>>>>>> application is foolhardy, a great waste of effort, and is doomed to >>>>>>>> mediocrity, given the success we have had with such a system. >>>>>>>> >>>>>>>> However until there is an EG, these are all just as much questions >>>>>>>> for me as for anybody, and I can and will answer to the best of my >>>>>>>> knowledge and belief. >>>>>>>> >>>>>>>>> As a result of experience in developing and evolving large >>>>>>>>> real-world >>>>>>>>> applications using a module system that supports BOTH m2m and m2p >>>>>>>>> dependencies, I believe it would be very unfortunate if a module >>>>>>>>> system that supports ONLY m2m were to become widely used in the >>>>>>>>> application space... not because OSGi can't handle the >>>>>>>>> competition, >>>>>>>>> but because those applications will be fragile and hard to evolve. >>>>>>>> >>>>>>>> I don't believe this to be the case. I think that the status quo >>>>>>>> doesn't result in particularly fragile applications, and such can >>>>>>>> be easy to evolve or hard depending on the quality of the >>>>>>>> application components and frameworks involved. I think that m2m >>>>>>>> dependencies enhance the status quo such that applications are >>>>>>>> somewhat less fragile (owing chiefly to the simple measure of >>>>>>>> preventing transitive dependencies from being exported by default), >>>>>>>> and quite easy to evolve as well (even large and complex frameworks >>>>>>>> rarely have more than a few dependencies, unless they themselves >>>>>>>> are overly fragmented (e.g. CXF as a nightmare example which yet >>>>>>>> still works fine under an m2m system)). >>>>>>>> >>>>>>>>> My question for you David is as follows. I understand that you >>>>>>>>> prefer >>>>>>>>> module dependencies, but do you believe that package dependencies >>>>>>>>> have >>>>>>>>> no value whatsoever and therefore should not be available to >>>>>>>>> application developers in the Java 8 module system? If so, why did >>>>>>>>> Red >>>>>>>>> Hat create an OSGi implementation? >>>>>>>> >>>>>>>> I personally believe that they have some value, but that value is >>>>>>>> limited to interoperability with OSGi. I do not believe that it is >>>>>>>> a preferable model for most users or standalone applications, nor >>>>>>>> for larger applications such as our application server. I think >>>>>>>> that the requirement for extra tooling and the increased run-time >>>>>>>> complexity which is exposed to users nullifies the benefits. If >>>>>>>> someone really wants this variety of modularity, they should simply >>>>>>>> use OSGi. >>>>>>>> >>>>>>>> Red Hat created an OSGi implementation for many reasons, but the >>>>>>>> only significant one to me is that there are people who want to use >>>>>>>> their OSGi applications with our application server. I don't >>>>>>>> believe any other reason is even necessary. It's the same reason >>>>>>>> we strive to support any spec, from EJB 1.0 to Java EE 6 and >>>>>>>> beyond. Someone wants the functionality, so we deliver it to the >>>>>>>> best of our ability. >>>>>>>> >>>>>>>>> Kind regards >>>>>>>>> Neil >>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 11/14/2011 01:49 AM, Glyn Normington wrote: >>>>>>>>>>> >>>>>>>>>>> I look forward to David's elaboration of why he thinks "using >>>>>>>>>>> packages as >>>>>>>>>>> a dependency unit is a terrible idea" to balance Peter's clear >>>>>>>>>>> explanation >>>>>>>>>>> of the benefits of m2p. >>>>>>>>>>> >>>>>>>>>>> Meanwhile, it's worth noting that, according to the requirements >>>>>>>>>>> document, >>>>>>>>>>> Jigsaw is aimed at platform modularisation and the platform >>>>>>>>>>> being >>>>>>>>>>> modularised has some non-optimal division of types across >>>>>>>>>>> packages (see the >>>>>>>>>>> package subsets requirement) which favour m2m dependencies. >>>>>>>>>>> (Note that >>>>>>>>>>> Apache Harmony was developed with modularity in mind and was >>>>>>>>>>> able to exploit >>>>>>>>>>> m2p, so platform modularisation per se needn't be limited to >>>>>>>>>>> m2m.) >>>>>>>>>>> >>>>>>>>>>> So if Jigsaw excludes m2p, it will then be applicable to certain >>>>>>>>>>> kinds of >>>>>>>>>>> legacy code modularisation and less applicable to new module >>>>>>>>>>> development and >>>>>>>>>>> modularisation of existing code whose division into packages >>>>>>>>>>> suits m2p. IIRC >>>>>>>>>>> this was the original positioning of Jigsaw: for use primarily >>>>>>>>>>> within the >>>>>>>>>>> OpenJDK codebase and only exposed for application use because it >>>>>>>>>>> was too >>>>>>>>>>> inconvenient to hide it. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Glyn >>>>>>>>>>> >>>>>>>>>>> On 12 Nov 2011, at 11:59, Peter Kriens wrote: >>>>>>>>>>> >>>>>>>>>>>> Neither my wrath, nor the fact that I rarely if ever get angry >>>>>>>>>>>> is >>>>>>>>>>>> relevant in this discussion ... This is a technical argument >>>>>>>>>>>> that are >>>>>>>>>>>> solvable by technical people that share the same goals. I >>>>>>>>>>>> prefer package >>>>>>>>>>>> dependencies because they address the excessive type coupling >>>>>>>>>>>> problem in >>>>>>>>>>>> object oriented systems, not because they're part of OSGi. Let >>>>>>>>>>>> me argue my >>>>>>>>>>>> case. >>>>>>>>>>>> >>>>>>>>>>>> Module-to-package dependencies (m2p) are preferable over >>>>>>>>>>>> module-to-module >>>>>>>>>>>> dependencies (m2m) for many reasons but these are the most >>>>>>>>>>>> important >>>>>>>>>>>> reasons: >>>>>>>>>>>> >>>>>>>>>>>> M2P is leverages the Java type system unlike m2m that must >>>>>>>>>>>> introduce new >>>>>>>>>>>> namespaces outside the Java type system. >>>>>>>>>>>> M2P can be used to break the transitive dependency chain, m2m >>>>>>>>>>>> suffers of >>>>>>>>>>>> excessive coupling >>>>>>>>>>>> >>>>>>>>>>>> Since the first bullet's benefit should be clear I only argue >>>>>>>>>>>> the more >>>>>>>>>>>> complex second bullet. >>>>>>>>>>>> >>>>>>>>>>>> A module is in many regards like a class. A class encapsulates >>>>>>>>>>>> members, >>>>>>>>>>>> depends on other members/classes, and makes a few members >>>>>>>>>>>> accessible outside >>>>>>>>>>>> the class. A module has a similar structure but then with >>>>>>>>>>>> types/packages as >>>>>>>>>>>> members. >>>>>>>>>>>> >>>>>>>>>>>> After the initial success of Object Oriented Programming (OO) >>>>>>>>>>>> it was >>>>>>>>>>>> quickly learned that reuse did not take place at the expected >>>>>>>>>>>> scale due to >>>>>>>>>>>> excessive type coupling. The problem was that a class >>>>>>>>>>>> aggregated many >>>>>>>>>>>> dependencies to simplify its implementation but these >>>>>>>>>>>> dependencies were >>>>>>>>>>>> unrelated to the contract it implemented. Since class >>>>>>>>>>>> dependencies are >>>>>>>>>>>> transitive most applications disappointingly became an almost >>>>>>>>>>>> fully >>>>>>>>>>>> connected graph. >>>>>>>>>>>> >>>>>>>>>>>> Java's great innovation was the interface because it broke both >>>>>>>>>>>> the >>>>>>>>>>>> transitivity and aggregation of dependencies. A class could now >>>>>>>>>>>> express its >>>>>>>>>>>> dependency (use or implement) on a contract (the interface) and >>>>>>>>>>>> was >>>>>>>>>>>> therefore fully type decoupled from the opposite site. >>>>>>>>>>>> >>>>>>>>>>>> An interface can act as a contract because it names the >>>>>>>>>>>> signature of a >>>>>>>>>>>> set of methods so that the compiler can verify the client and >>>>>>>>>>>> the >>>>>>>>>>>> implementer. >>>>>>>>>>>> >>>>>>>>>>>> Since a module has a very similar structure to a class it >>>>>>>>>>>> suffers from >>>>>>>>>>>> exactly the same transitive aggregation of dependencies. This >>>>>>>>>>>> is not a >>>>>>>>>>>> theory, look at the experiences with Maven >>>>>>>>>>>> (http://www.sonatype.com/people/2011/04/how-not-to-download-the-internet/) >>>>>>>>>>>> Again, this is not that maven is bad or developers are stupid, >>>>>>>>>>>> it is the >>>>>>>>>>>> same underlying force that finally resulted in the Java >>>>>>>>>>>> interface. >>>>>>>>>>>> >>>>>>>>>>>> The parallel for the class' interface for modules is a named >>>>>>>>>>>> set of >>>>>>>>>>>> interfaces. This concept already exists in Java: a package. >>>>>>>>>>>> Looking at >>>>>>>>>>>> almost all JSRs it is clear that our industry already uses >>>>>>>>>>>> packages as >>>>>>>>>>>> "interfaces" to provider implementations. >>>>>>>>>>>> >>>>>>>>>>>> Therefore, just like a class should not depend on other >>>>>>>>>>>> implementation >>>>>>>>>>>> types, a module should preferably not depend on other modules. >>>>>>>>>>>> A module >>>>>>>>>>>> should instead depend on contracts. Since modules will be used >>>>>>>>>>>> to provide >>>>>>>>>>>> components from different sources managed with different life >>>>>>>>>>>> cycles the >>>>>>>>>>>> excessive type coupling caused by m2m is even more damaging >>>>>>>>>>>> than in c2c. >>>>>>>>>>>> Proper use of m2p creates significantly less type coupled >>>>>>>>>>>> systems than m2m, >>>>>>>>>>>> the benefits should be obvious. >>>>>>>>>>>> >>>>>>>>>>>> Since there are use cases for m2m (non-type safe languages for >>>>>>>>>>>> example) I >>>>>>>>>>>> do believe that Jigsaw should still support m2m. However, it >>>>>>>>>>>> would be >>>>>>>>>>>> greatly beneficial to our industry if we could take advantage >>>>>>>>>>>> of the lessons >>>>>>>>>>>> learned with the Java interface and realize how surprisingly >>>>>>>>>>>> important the >>>>>>>>>>>> Java package actually is in our eco system. >>>>>>>>>>>> >>>>>>>>>>>> Kind regards, >>>>>>>>>>>> >>>>>>>>>>>> Peter Kriens >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 9 nov. 2011, at 15:04, David M. Lloyd wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I'll just state now that using packages as a dependency unit >>>>>>>>>>>>> is a >>>>>>>>>>>>> terrible idea, and not some architectural revelation. That >>>>>>>>>>>>> way, Peter's >>>>>>>>>>>>> wrath will be largely directed at me. :-) >>>>>>>>>>>>> >>>>>>>>>>>>> On 11/09/2011 08:02 AM, Peter Kriens wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> I agree that tools are needed but we must be careful to not >>>>>>>>>>>>>> expect >>>>>>>>>>>>>> tools to stopgap an architectural issue. I think it is >>>>>>>>>>>>>> important to first do >>>>>>>>>>>>>> good architectural design leveraging existing tools (e.g. the >>>>>>>>>>>>>> Java type >>>>>>>>>>>>>> system) before you try to add new tools. It is such a pity >>>>>>>>>>>>>> (but all to >>>>>>>>>>>>>> common) that a design allows for classes of errors that would >>>>>>>>>>>>>> be impossible >>>>>>>>>>>>>> with a slightly different design. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Kind regards, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Peter Kriens >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 9 nov. 2011, at 14:49, Alan Bateman wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 09/11/2011 13:04, Peter Kriens wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The issue is that maven problems are not caused because >>>>>>>>>>>>>>>> maven is bad >>>>>>>>>>>>>>>> or that pom authors are stupid. The reason is that the >>>>>>>>>>>>>>>> module-to-module >>>>>>>>>>>>>>>> dependency architecture in maven (and Jigsaw) is error >>>>>>>>>>>>>>>> prone ... >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This thread started out with someone asking about adding >>>>>>>>>>>>>>> module >>>>>>>>>>>>>>> declarations to existing JAR files, and in that context, I >>>>>>>>>>>>>>> agree it can be >>>>>>>>>>>>>>> error prone without good tools. I think things should be a >>>>>>>>>>>>>>> lot better when >>>>>>>>>>>>>>> modules are compiled. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -Alan. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> - DML >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> - DML >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> - DML >>>>>>> >>>>>> >>>>> >>>> >>>> >> > From david.lloyd at redhat.com Thu Nov 17 08:31:31 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 17 Nov 2011 10:31:31 -0600 Subject: Why package deps work (Was: Re: Converting plain JARs to Java modules) In-Reply-To: References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> <67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz> <4EC14A53.7070907@redhat.com> <4EC1F53D.7040309@redhat.com> <328B38A3-EB53-4AD2-970B-CE84F5C26F96@aQute.biz> <9B28CE0A-85DC-4F53-8A88-5830759DA4BC@pontarelli.com> Message-ID: <4EC536E3.8010003@redhat.com> On 11/16/2011 01:05 PM, Peter Kriens wrote: > On 16 nov. 2011, at 19:23, Brian Pontarelli wrote: >> Are you talking about build time here? In practice transitive >> dependencies can always be avoided at build time. If you are >> talking about runtime, transitive dependencies are definitely >> required. I'm not sure exactly what your JPA vs. Eclipse Link >> example is getting at. > > There is type coupling and instance coupling. Interfaces break the > transitive type coupling but then require that you bind an instance > for that interface in runtime, there is a trade off. > Spring/Guice/CDI/Services, etc all handle this instance coupling. > With a module system, packages type-decouple you from an > implementation module just like an interface type decouples a class > from its dependencies. In the Eclipse Link example, my code is not > type coupled to Eclipse Link, it is only type coupled to its > specification package javax.persistence. This allows the deployer to > choose either Eclipse Link or OpenJPA they both provide the > implementation to the javax.transaction package. However, my code is > not having a type coupling on either leaving flexibility to the > deployer. > > So we make a trade off between type decoupling and runtime > complexity. This is the same trade off we make in IOC/DI engines, > annotations, and factories. But I don't see why m2p gives this coupling any better than m2m? I can just as easily have a javax.transaction module with m2m as I could have a package with m2p. In a static module repository, there are really only two ways to choose an implementation of some given API: statically, such that there is only one implementation for an API which is globally accessed at the API's module "address", or at runtime, where the user selects their desired implementation based on their own dependencies or other selection tools. There is no special mechanism which becomes available solely due to m2p. >> Again, are you talking about build time or runtime? Also, are you >> suggesting that Java determine your dependencies automatically >> including the version? Or is this merely to help developers find >> missing dependencies and remove unused dependencies? > If you compile class com.bar.Foo that refers to com.bar.Bar then the > Foo.class and Bar.class file contain all their type dependencies and > by implication their package dependencies. The version of the package > can then for example be found from package-info.java. > > This technique is used extensively in bnd, an open source library > used in Eclipse, IDEA, ant, maven, SBT, and others. It reads the > class files in a JAR and generates the OSGi dependency information > automatically. This doesn't actually help. The user already *knows* what their dependency is - i.e. the javax.transaction API *module*, which is listed in their IDE dependencies as well as in their build - but now they must also employ special tooling in order to have the runtime module system figure it out too. This is a significant "impedance mismatch", to use a popular metaphor. -- - DML From cowwoc at bbs.darktech.org Thu Nov 17 15:52:18 2011 From: cowwoc at bbs.darktech.org (cowwoc) Date: Thu, 17 Nov 2011 15:52:18 -0800 (PST) Subject: Use-cases for version ranges? Message-ID: <1321573938450-5002801.post@n5.nabble.com> Can someone please explain why modules need to be able to specify version ranges for dependencies? I believe OSGI allows the specification of version ranges while Maven allows the specification of individual versions. The only thing that comes to mind is when module C depends on A and B, A depends on log4j 1.0, and B depends on log4j 1.1. What does C do? Is this the main use-case for version ranges? By the sound of it, this is a trust model where developers are told that log4j 1.x won't break compatibility so they depend on that range without actually testing against each version (newer versions may be released after their own software). I question whether such a mechanism is better or worse than depending on individual versions which may be overridden at a later time (a la Maven). On the one hand, you don't need to release a new version of the application each time a dependency is updated. On the other hand, no one is actually running tests to ensure that the versions are really compatible. Is there a way to get module A to see log4j 1.0 and module B to see log4j 1.1 (using separate ClassLoaders)? Thanks, Gili -- View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html Sent from the jigsaw-dev mailing list archive at Nabble.com. From njbartlett at gmail.com Thu Nov 17 23:10:48 2011 From: njbartlett at gmail.com (Neil Bartlett) Date: Fri, 18 Nov 2011 07:10:48 +0000 Subject: Use-cases for version ranges? In-Reply-To: <1321573938450-5002801.post@n5.nabble.com> References: <1321573938450-5002801.post@n5.nabble.com> Message-ID: Suppose as the developer of module A, I declare a dependency on log4j, exactly version 1.0.0 because I have not tested against log4j 1.0.1, 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* being used with log4j version 1.0.1 even if this combinations is later tested and proven to work by somebody else. In other words, testing is important but it doesn't necessarily have to always be done by the original developer of each module. On the other hand let's say I state my dependency using the following range: [1.2.14, 2.0). This is OSGi syntax and I believe Jigsaw is following it, and it simply means I accept version 1.2.14 up to but not including 2.0. Anybody can see that I compiled and tested against 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 etc. It does not mean that I *guarantee* my module will work with log4j 1.3 because that obviously depends on whether the log4j authors accept and follow the common semantics of indicating backwards-incompatible changes with a bump to the first version segment. The consequence of trying to lock down imports to a narrow range or even a point version is that assembling an application becomes very difficult, and we are forced to deploy many versions of common libraries concurrently. This is non-optimal, though we can handle it to some degree via per-module classloaders as in OSGi. Regards, Neil On Thu, Nov 17, 2011 at 11:52 PM, cowwoc wrote: > Can someone please explain why modules need to be able to specify version > ranges for dependencies? I believe OSGI allows the specification of version > ranges while Maven allows the specification of individual versions. > > The only thing that comes to mind is when module C depends on A and B, A > depends on log4j 1.0, and B depends on log4j 1.1. What does C do? Is this > the main use-case for version ranges? > > By the sound of it, this is a trust model where developers are told that > log4j 1.x won't break compatibility so they depend on that range without > actually testing against each version (newer versions may be released after > their own software). I question whether such a mechanism is better or worse > than depending on individual versions which may be overridden at a later > time (a la Maven). On the one hand, you don't need to release a new version > of the application each time a dependency is updated. On the other hand, no > one is actually running tests to ensure that the versions are really > compatible. > > Is there a way to get module A to see log4j 1.0 and module B to see log4j > 1.1 (using separate ClassLoaders)? > > Thanks, > Gili > > -- > View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html > Sent from the jigsaw-dev mailing list archive at Nabble.com. > From njbartlett at gmail.com Thu Nov 17 23:23:30 2011 From: njbartlett at gmail.com (Neil Bartlett) Date: Fri, 18 Nov 2011 07:23:30 +0000 Subject: Use-cases for version ranges? In-Reply-To: References: <1321573938450-5002801.post@n5.nabble.com> Message-ID: I noticed that I failed to address your point about Maven using point versions. Maven is a build tool. At build time we need to compile against a single specific version so that we have repeatable builds. In general we should build each module against the lowest version of the library that it can possibly use, and there are no major negative consequences of having several versions of a library at build time (except that Maven has to download a lot!). At runtime however we need to have the flexibility to substitute a single compatible version. Neil On Fri, Nov 18, 2011 at 7:10 AM, Neil Bartlett wrote: > Suppose as the developer of module A, I declare a dependency on log4j, > exactly version 1.0.0 because I have not tested against log4j 1.0.1, > 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* being > used with log4j version 1.0.1 even if this combinations is later > tested and proven to work by somebody else. In other words, testing is > important but it doesn't necessarily have to always be done by the > original developer of each module. > > On the other hand let's say I state my dependency using the following > range: [1.2.14, 2.0). This is OSGi syntax and I believe Jigsaw is > following it, and it simply means I accept version 1.2.14 up to but > not including 2.0. Anybody can see that I compiled and tested against > 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 etc. It > does not mean that I *guarantee* my module will work with log4j 1.3 > because that obviously depends on whether the log4j authors accept and > follow the common semantics of indicating backwards-incompatible > changes with a bump to the first version segment. > > The consequence of trying to lock down imports to a narrow range or > even a point version is that assembling an application becomes very > difficult, and we are forced to deploy many versions of common > libraries concurrently. This is non-optimal, though we can handle it > to some degree via per-module classloaders as in OSGi. > > Regards, > Neil > > On Thu, Nov 17, 2011 at 11:52 PM, cowwoc wrote: >> Can someone please explain why modules need to be able to specify version >> ranges for dependencies? I believe OSGI allows the specification of version >> ranges while Maven allows the specification of individual versions. >> >> The only thing that comes to mind is when module C depends on A and B, A >> depends on log4j 1.0, and B depends on log4j 1.1. What does C do? Is this >> the main use-case for version ranges? >> >> By the sound of it, this is a trust model where developers are told that >> log4j 1.x won't break compatibility so they depend on that range without >> actually testing against each version (newer versions may be released after >> their own software). I question whether such a mechanism is better or worse >> than depending on individual versions which may be overridden at a later >> time (a la Maven). On the one hand, you don't need to release a new version >> of the application each time a dependency is updated. On the other hand, no >> one is actually running tests to ensure that the versions are really >> compatible. >> >> Is there a way to get module A to see log4j 1.0 and module B to see log4j >> 1.1 (using separate ClassLoaders)? >> >> Thanks, >> Gili >> >> -- >> View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html >> Sent from the jigsaw-dev mailing list archive at Nabble.com. >> > From Alan.Bateman at oracle.com Fri Nov 18 03:40:53 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 18 Nov 2011 11:40:53 +0000 Subject: Review request: jpkg -m option takes both classes and resources In-Reply-To: <4EC1E709.9010807@oracle.com> References: <4EBB5782.2050904@oracle.com> <4EC15F4F.8080802@oracle.com> <4EC16415.3010905@oracle.com> <4EC16883.3070604@oracle.com> <4EC1E709.9010807@oracle.com> Message-ID: <4EC64445.9030607@oracle.com> On 15/11/2011 04:14, Mandy Chung wrote: > > Rename done. I also changed the newInstance method to do the > compression before returning the compressed output stream instance > (this removes the call to the compress method from several callers). > > Upated webrev: > > http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/jpkg-dash-r-option.01/ Thanks, I think this looks much better. -Alan. From thomas.diesler at jboss.com Fri Nov 18 07:26:02 2011 From: thomas.diesler at jboss.com (Thomas Diesler) Date: Fri, 18 Nov 2011 16:26:02 +0100 Subject: Use-cases for version ranges? In-Reply-To: References: <1321573938450-5002801.post@n5.nabble.com> Message-ID: <4EC6790A.5000208@jboss.com> Yes OSGi uses a version scheme with semantic meaning (see the whitepaper on Semantic Versioning for details). This works very well as long as the world conforms to the semantic meaning of that scheme. In reality we see a wide vararity of versioning schemes being used - not all of them make sense let alone can be used to reason about compatibility. At jboss we use a model whereby the identity of a module is defined by 'identifier' plus 'slot'. A slot is not necessarily a version/range but a compatibility expression. The default slot is 'main' and would initially contain the version of the jar that you use to begin with org.log4j:main => log4j-1.2.14.jar Modules that have a dependency on log4j would use 'org.log4j:main' to express that. When there is an update to log4j, we can replace the binary in the same slot with the newer version as long as it is compatible (QA would check that). An incompatible version (e.g. log4j-2.0) can be installed on a new slot (e.g. 2x) and clients can be updated accordingly. Unlike OSGi, the dependency may be expressed within the deployment that has the dependency or externally as we do for all system components. OSGi requires that the unit of deployment is the 'bundle' for which it mandates that it uses semantic versioning mainly for its package import/exports. Bottom line: dependencies based on version ranges makes sense if there is a compatibility contract associated with version strings. (i.e. you must have some fidelity that 1.2 is backward compatible to 1.1 but 2.0 is not) -thomas On 11/18/2011 08:10 AM, Neil Bartlett wrote: > Suppose as the developer of module A, I declare a dependency on log4j, > exactly version 1.0.0 because I have not tested against log4j 1.0.1, > 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* being > used with log4j version 1.0.1 even if this combinations is later > tested and proven to work by somebody else. In other words, testing is > important but it doesn't necessarily have to always be done by the > original developer of each module. > > On the other hand let's say I state my dependency using the following > range: [1.2.14, 2.0). This is OSGi syntax and I believe Jigsaw is > following it, and it simply means I accept version 1.2.14 up to but > not including 2.0. Anybody can see that I compiled and tested against > 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 etc. It > does not mean that I *guarantee* my module will work with log4j 1.3 > because that obviously depends on whether the log4j authors accept and > follow the common semantics of indicating backwards-incompatible > changes with a bump to the first version segment. > > The consequence of trying to lock down imports to a narrow range or > even a point version is that assembling an application becomes very > difficult, and we are forced to deploy many versions of common > libraries concurrently. This is non-optimal, though we can handle it > to some degree via per-module classloaders as in OSGi. > > Regards, > Neil > > On Thu, Nov 17, 2011 at 11:52 PM, cowwoc wrote: >> Can someone please explain why modules need to be able to specify version >> ranges for dependencies? I believe OSGI allows the specification of version >> ranges while Maven allows the specification of individual versions. >> >> The only thing that comes to mind is when module C depends on A and B, A >> depends on log4j 1.0, and B depends on log4j 1.1. What does C do? Is this >> the main use-case for version ranges? >> >> By the sound of it, this is a trust model where developers are told that >> log4j 1.x won't break compatibility so they depend on that range without >> actually testing against each version (newer versions may be released after >> their own software). I question whether such a mechanism is better or worse >> than depending on individual versions which may be overridden at a later >> time (a la Maven). On the one hand, you don't need to release a new version >> of the application each time a dependency is updated. On the other hand, no >> one is actually running tests to ensure that the versions are really >> compatible. >> >> Is there a way to get module A to see log4j 1.0 and module B to see log4j >> 1.1 (using separate ClassLoaders)? >> >> Thanks, >> Gili >> >> -- >> View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html >> Sent from the jigsaw-dev mailing list archive at Nabble.com. >> -- xxxxxxxxxxxxxxxxxxxxxxxxxxxx Thomas Diesler JBoss OSGi Lead JBoss, a division of Red Hat xxxxxxxxxxxxxxxxxxxxxxxxxxxx From sjlee0 at gmail.com Fri Nov 18 08:16:30 2011 From: sjlee0 at gmail.com (Sangjin Lee) Date: Fri, 18 Nov 2011 08:16:30 -0800 Subject: Use-cases for version ranges? In-Reply-To: <4EC6790A.5000208@jboss.com> References: <1321573938450-5002801.post@n5.nabble.com> <4EC6790A.5000208@jboss.com> Message-ID: Fixed versions become problematic if your runtime consists of modules that were not created by you and cannot be modified to coordinate their versions. And that is a fairly common occurrence. For example, suppose your module A depends on a third-part module B of version 1.2.3 (and that only). But A also depends on another (unrelated) third-party library module C of version 1.0.0. Furthermore, suppose C also depends on B but of version 1.1.0 (and that only). If we do not allow version ranges, this runtime would have to bring in both versions of B into it. Depending on how B is used, this may lead to runtime problems such as ClassCastExceptions. And this can quickly proliferate especially if B is a popular library. If you can modify C's module metadata to agree to whatever a common version of B is, that would address this issue. However, that may not be always doable or practical. Version ranges are useful if there is actually room for different consumers to agree on a common version of the provider. Yes, semantic versions play a big part here, but IMO the alternatives are not as attractive. BTW, the current JDK behavior can be considered as having a completely open version range (0 to infinity). Once you're on the classpath, you're willing (and forced) to accept any version. Using a fixed version is the other extreme. Regards, Sangjin On Fri, Nov 18, 2011 at 7:26 AM, Thomas Diesler wrote: > Yes OSGi uses a version scheme with semantic meaning (see the whitepaper > on Semantic Versioning SemanticVersioning.pdf> > for details). This works very well as long as the world conforms to the > semantic meaning of that scheme. In reality we see a wide vararity of > versioning schemes being used - not all of them make sense let alone can be > used to reason about compatibility. > > At jboss we use a model whereby the identity of a module is defined by > 'identifier' plus 'slot'. A slot is not necessarily a version/range but a > compatibility expression. The default slot is 'main' and would initially > contain the version of the jar that you use to begin with > > org.log4j:main => log4j-1.2.14.jar > > Modules that have a dependency on log4j would use 'org.log4j:main' to > express that. When there is an update to log4j, we can replace the binary > in the same slot with the newer version as long as it is compatible (QA > would check that). An incompatible version (e.g. log4j-2.0) can be > installed on a new slot (e.g. 2x) and clients can be updated accordingly. > > Unlike OSGi, the dependency may be expressed within the deployment that > has the dependency or externally as we do for all system components. > > OSGi requires that the unit of deployment is the 'bundle' for which it > mandates that it uses semantic versioning mainly for its package > import/exports. > > Bottom line: dependencies based on version ranges makes sense if there is > a compatibility contract associated with version strings. (i.e. you must > have some fidelity that 1.2 is backward compatible to 1.1 but 2.0 is not) > > -thomas > > > On 11/18/2011 08:10 AM, Neil Bartlett wrote: > >> Suppose as the developer of module A, I declare a dependency on log4j, >> exactly version 1.0.0 because I have not tested against log4j 1.0.1, >> 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* being >> used with log4j version 1.0.1 even if this combinations is later >> tested and proven to work by somebody else. In other words, testing is >> important but it doesn't necessarily have to always be done by the >> original developer of each module. >> >> On the other hand let's say I state my dependency using the following >> range: [1.2.14, 2.0). This is OSGi syntax and I believe Jigsaw is >> following it, and it simply means I accept version 1.2.14 up to but >> not including 2.0. Anybody can see that I compiled and tested against >> 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 etc. It >> does not mean that I *guarantee* my module will work with log4j 1.3 >> because that obviously depends on whether the log4j authors accept and >> follow the common semantics of indicating backwards-incompatible >> changes with a bump to the first version segment. >> >> The consequence of trying to lock down imports to a narrow range or >> even a point version is that assembling an application becomes very >> difficult, and we are forced to deploy many versions of common >> libraries concurrently. This is non-optimal, though we can handle it >> to some degree via per-module classloaders as in OSGi. >> >> Regards, >> Neil >> >> On Thu, Nov 17, 2011 at 11:52 PM, cowwoc >> wrote: >> >>> Can someone please explain why modules need to be able to specify version >>> ranges for dependencies? I believe OSGI allows the specification of >>> version >>> ranges while Maven allows the specification of individual versions. >>> >>> The only thing that comes to mind is when module C depends on A and B, A >>> depends on log4j 1.0, and B depends on log4j 1.1. What does C do? Is this >>> the main use-case for version ranges? >>> >>> By the sound of it, this is a trust model where developers are told that >>> log4j 1.x won't break compatibility so they depend on that range without >>> actually testing against each version (newer versions may be released >>> after >>> their own software). I question whether such a mechanism is better or >>> worse >>> than depending on individual versions which may be overridden at a later >>> time (a la Maven). On the one hand, you don't need to release a new >>> version >>> of the application each time a dependency is updated. On the other hand, >>> no >>> one is actually running tests to ensure that the versions are really >>> compatible. >>> >>> Is there a way to get module A to see log4j 1.0 and module B to see log4j >>> 1.1 (using separate ClassLoaders)? >>> >>> Thanks, >>> Gili >>> >>> -- >>> View this message in context: http://jigsaw-dev.1059479.n5.** >>> nabble.com/Use-cases-for-**version-ranges-**tp5002801p5002801.html >>> Sent from the jigsaw-dev mailing list archive at Nabble.com. >>> >>> > -- > xxxxxxxxxxxxxxxxxxxxxxxxxxxx > Thomas Diesler > JBoss OSGi Lead > JBoss, a division of Red Hat > xxxxxxxxxxxxxxxxxxxxxxxxxxxx > > From dalibor.topic at oracle.com Fri Nov 18 08:51:29 2011 From: dalibor.topic at oracle.com (Dalibor Topic) Date: Fri, 18 Nov 2011 17:51:29 +0100 Subject: Use-cases for version ranges? In-Reply-To: <1321573938450-5002801.post@n5.nabble.com> References: <1321573938450-5002801.post@n5.nabble.com> Message-ID: <4EC68D11.4040603@oracle.com> On 11/18/11 12:52 AM, cowwoc wrote: > I question whether such a mechanism is better or worse > than depending on individual versions which may be overridden at a later > time (a la Maven). On the one hand, you don't need to release a new version > of the application each time a dependency is updated. On the other hand, no > one is actually running tests to ensure that the versions are really > compatible. I think that it's better. Consider a directed dependency graph with N nodes. One of the nodes has a security issue, which gets fixed in a new release with a new version, so it needs to be updated to some later version. Typically, the ABI doesn't change in a security fix. If you depend on the individual version, you in addition need to update all nodes with an edge going to the updated node, potentially recursively (since you may need to update the metadata to point to the updated metadata, at least). In other words, you have a domino effect. Consider a large N, where security updates for various third party components don't happen on a synchronized schedule, and you potentially have multiple domino effects happily cascading with each other ... https://www.youtube.com/watch?v=qybUFnY7Y8w . cheers, dalibor topic -- Oracle Dalibor Topic | Java F/OSS Ambassador Phone: +494023646738 | Mobile: +491772664192 Oracle Java Platform Group ORACLE Deutschland B.V. & Co. KG | Nagelsweg 55 | 20097 Hamburg ORACLE Deutschland B.V. & Co. KG Hauptverwaltung: Riesstr. 25, D-80992 M?nchen Registergericht: Amtsgericht M?nchen, HRA 95603 Komplement?rin: ORACLE Deutschland Verwaltung B.V. Hertogswetering 163/167, 3543 AS Utrecht, Niederlande Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697 Gesch?ftsf?hrer: J?rgen Kunz, Marcel van de Molen, Alexander van der Ven Green Oracle Oracle is committed to developing practices and products that help protect the environment From dalibor.topic at oracle.com Fri Nov 18 09:54:14 2011 From: dalibor.topic at oracle.com (Dalibor Topic) Date: Fri, 18 Nov 2011 18:54:14 +0100 Subject: Review request: jpkg -m option takes both classes and resources In-Reply-To: <4EC1E709.9010807@oracle.com> References: <4EBB5782.2050904@oracle.com> <4EC15F4F.8080802@oracle.com> <4EC16415.3010905@oracle.com> <4EC16883.3070604@oracle.com> <4EC1E709.9010807@oracle.com> Message-ID: <4EC69BC6.6070104@oracle.com> On 11/15/11 5:14 AM, Mandy Chung wrote: > On 11/14/2011 11:14 AM, Alan Bateman wrote: >> On 14/11/2011 18:55, Mandy Chung wrote: >>> >>> I agree that the class names and the getInstance method should be renamed. How about renaming the class names to CompressorOutputStream and CompressedClassFileOutputStream and renaming getInstance to newInstance? >> Yes, that would be better (alternatively CompressedClassOutputStream). > > Rename done. I also changed the newInstance method to do the compression before returning the compressed output stream instance (this removes the call to the compress method from several callers). > > Upated webrev: > http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/jpkg-dash-r-option.01/ This is a very nice change! Do you plan eventually to factor out a module file reader as well? cheers, dalibor topic -- Oracle Dalibor Topic | Java F/OSS Ambassador Phone: +494023646738 | Mobile: +491772664192 Oracle Java Platform Group ORACLE Deutschland B.V. & Co. KG | Nagelsweg 55 | 20097 Hamburg ORACLE Deutschland B.V. & Co. KG Hauptverwaltung: Riesstr. 25, D-80992 M?nchen Registergericht: Amtsgericht M?nchen, HRA 95603 Komplement?rin: ORACLE Deutschland Verwaltung B.V. Hertogswetering 163/167, 3543 AS Utrecht, Niederlande Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697 Gesch?ftsf?hrer: J?rgen Kunz, Marcel van de Molen, Alexander van der Ven Green Oracle Oracle is committed to developing practices and products that help protect the environment From brian at pontarelli.com Fri Nov 18 10:48:42 2011 From: brian at pontarelli.com (Brian Pontarelli) Date: Fri, 18 Nov 2011 11:48:42 -0700 Subject: Use-cases for version ranges? In-Reply-To: <4EC68D11.4040603@oracle.com> References: <1321573938450-5002801.post@n5.nabble.com> <4EC68D11.4040603@oracle.com> Message-ID: <4C3634AC-251D-4C74-96EC-634C4B2E691B@pontarelli.com> On Nov 18, 2011, at 9:51 AM, Dalibor Topic wrote: > On 11/18/11 12:52 AM, cowwoc wrote: >> I question whether such a mechanism is better or worse >> than depending on individual versions which may be overridden at a later >> time (a la Maven). On the one hand, you don't need to release a new version >> of the application each time a dependency is updated. On the other hand, no >> one is actually running tests to ensure that the versions are really >> compatible. > > I think that it's better. Consider a directed dependency graph with N nodes. One > of the nodes has a security issue, which gets fixed in a new release with a new > version, so it needs to be updated to some later version. Typically, the ABI > doesn't change in a security fix. > > If you depend on the individual version, you in addition need to update all nodes > with an edge going to the updated node, potentially recursively (since you may need > to update the metadata to point to the updated metadata, at least). In other words, > you have a domino effect. Consider a large N, where security updates for various third > party components don't happen on a synchronized schedule, and you potentially have > multiple domino effects happily cascading with each other ... > https://www.youtube.com/watch?v=qybUFnY7Y8w . If modules define their version compatibility, you don't have these issues. Someone can pull in new versions of a module or tell a module to use a different version of a dependency and as long as it doesn't break the version compatibility. This is how Savant works and it was built for Orbitz, which had an extremely large dependency graph. I used to have good wiki page on version compatibility, but I can't find it anymore. The way it works is that the module developer defines how they want to maintain runtime compatibility across versions. In most cases, things are compatible across minor or patch versions. "Minor" compatibility means 1.1 and 1.2 are are compatible but not 2.0. "Patch" compatibility means 1.1.1 and 1.1.2 are compatible but not 1.2. I think the two fundamental issues I have with ranges are: 1. At any given time I can only be certain my code will work with one version of a dependency. That is the version I compile and test against. Therefore, as a developer, I should only define that version in my dependencies meta-data. 2. There should not be any magic upgrading of my dependencies at run or build time. Someone should be forced to specifically upgrade one of my dependencies and only for a really good reason. They should also be forced to test that upgrade to ensure it will work. Without module isolation, this problem was much more difficult because there could only be one version of a JAR on the classpath and therefore build tools needed to understand how to safely upgrade JARs. Some build tools did this better than others. With true isolation, this is less of an issue, making single versions even easier to use. Issues come back into play when modules export modules or with circular dependencies. These mimic the classpath behavior, which in my opinion is best solved via compatibility rather than ranges. -bp From mandy.chung at oracle.com Fri Nov 18 13:09:15 2011 From: mandy.chung at oracle.com (mandy.chung at oracle.com) Date: Fri, 18 Nov 2011 21:09:15 +0000 Subject: hg: jigsaw/jigsaw/langtools: Remove jpkg -r option and -m accepts classes and resources Message-ID: <20111118210922.9C655473DE@hg.openjdk.java.net> Changeset: 1e829edfb340 Author: mchung Date: 2011-11-18 12:02 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/1e829edfb340 Remove jpkg -r option and -m accepts classes and resources Reviewed-by: alanb, chegar ! make/tools/Jigsaw/FpkgTask.java From mandy.chung at oracle.com Fri Nov 18 13:15:53 2011 From: mandy.chung at oracle.com (mandy.chung at oracle.com) Date: Fri, 18 Nov 2011 21:15:53 +0000 Subject: hg: jigsaw/jigsaw/jdk: 2 new changesets Message-ID: <20111118211620.2620C473DF@hg.openjdk.java.net> Changeset: 638a21385dfa Author: mchung Date: 2011-11-18 12:01 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/638a21385dfa Remove jpkg -r option and -m accepts classes and resources Reviewed-by: alanb, chegar ! make/common/BuildNativePackages.gmk ! make/java/java/FILES_java.gmk ! make/modules/Makefile ! make/tools/classanalyzer/src/com/sun/classanalyzer/Modularizer.java ! src/share/classes/org/openjdk/jigsaw/Manifest.java + src/share/classes/org/openjdk/jigsaw/ModuleFile.java - src/share/classes/org/openjdk/jigsaw/ModuleFileFormat.java ! src/share/classes/org/openjdk/jigsaw/ModuleFileSigner.java ! src/share/classes/org/openjdk/jigsaw/ModuleFileVerifier.java + src/share/classes/org/openjdk/jigsaw/ModuleFileWriter.java ! src/share/classes/org/openjdk/jigsaw/SignedModule.java ! src/share/classes/org/openjdk/jigsaw/SimpleLibrary.java ! src/share/classes/org/openjdk/jigsaw/cli/Librarian.java ! src/share/classes/org/openjdk/jigsaw/cli/Packager.java ! src/share/classes/org/openjdk/jigsaw/cli/Signer.java ! test/org/openjdk/jigsaw/circular-deps.sh ! test/org/openjdk/jigsaw/cli/JpkgArgsTest.java + test/org/openjdk/jigsaw/cli/ModuleFileTest.java ! test/org/openjdk/jigsaw/cli/ModuleFormatHeaderHashTest.java - test/org/openjdk/jigsaw/cli/ModuleFormatTest01.java - test/org/openjdk/jigsaw/cli/ModuleFormatTest01.sh ! test/org/openjdk/jigsaw/cli/ModuleFormatTestLeftOverBytes.java + test/org/openjdk/jigsaw/cli/SignedModuleFileTest.sh ! test/org/openjdk/jigsaw/cli/jmod-basic.sh ! test/org/openjdk/jigsaw/cli/jmod-signed.sh ! test/org/openjdk/jigsaw/cli/signed-module.sh ! test/org/openjdk/jigsaw/cli/timestamp-test.sh ! test/org/openjdk/jigsaw/library.sh ! test/org/openjdk/jigsaw/preinstall.sh ! test/org/openjdk/jigsaw/resource.sh Changeset: a07343ae2797 Author: mchung Date: 2011-11-18 13:15 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/a07343ae2797 Merge ! make/modules/Makefile - make/modules/jdk7.depconfig From cowwoc at bbs.darktech.org Fri Nov 18 20:27:17 2011 From: cowwoc at bbs.darktech.org (cowwoc) Date: Fri, 18 Nov 2011 23:27:17 -0500 Subject: Use-cases for version ranges? In-Reply-To: References: <1321573938450-5002801.post@n5.nabble.com> Message-ID: <4EC73025.40402@bbs.darktech.org> Neil, I guess I don't understand why Jigsaw should work differently from Maven on this point. I am expecting developers to specify specific versions that they tested (point versions, not ranges) and end-users may override these "recommendations" as they see fit. Where you see version range [1.2.14, 2.0) as a way of communicating "the developer guarantees 1.2.14 but you may use newer versions up to 2.0 at your own risk" I'd expect the developer to simply specify 1.2.14 and there should be no limit on what version end-users may use if they so wish. Gili On 18/11/2011 2:23 AM, Neil Bartlett wrote: > I noticed that I failed to address your point about Maven using point versions. > > Maven is a build tool. At build time we need to compile against a > single specific version so that we have repeatable builds. In general > we should build each module against the lowest version of the library > that it can possibly use, and there are no major negative consequences > of having several versions of a library at build time (except that > Maven has to download a lot!). At runtime however we need to have the > flexibility to substitute a single compatible version. > > Neil > > On Fri, Nov 18, 2011 at 7:10 AM, Neil Bartlett wrote: >> Suppose as the developer of module A, I declare a dependency on log4j, >> exactly version 1.0.0 because I have not tested against log4j 1.0.1, >> 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* being >> used with log4j version 1.0.1 even if this combinations is later >> tested and proven to work by somebody else. In other words, testing is >> important but it doesn't necessarily have to always be done by the >> original developer of each module. >> >> On the other hand let's say I state my dependency using the following >> range: [1.2.14, 2.0). This is OSGi syntax and I believe Jigsaw is >> following it, and it simply means I accept version 1.2.14 up to but >> not including 2.0. Anybody can see that I compiled and tested against >> 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 etc. It >> does not mean that I *guarantee* my module will work with log4j 1.3 >> because that obviously depends on whether the log4j authors accept and >> follow the common semantics of indicating backwards-incompatible >> changes with a bump to the first version segment. >> >> The consequence of trying to lock down imports to a narrow range or >> even a point version is that assembling an application becomes very >> difficult, and we are forced to deploy many versions of common >> libraries concurrently. This is non-optimal, though we can handle it >> to some degree via per-module classloaders as in OSGi. >> >> Regards, >> Neil >> >> On Thu, Nov 17, 2011 at 11:52 PM, cowwoc wrote: >>> Can someone please explain why modules need to be able to specify version >>> ranges for dependencies? I believe OSGI allows the specification of version >>> ranges while Maven allows the specification of individual versions. >>> >>> The only thing that comes to mind is when module C depends on A and B, A >>> depends on log4j 1.0, and B depends on log4j 1.1. What does C do? Is this >>> the main use-case for version ranges? >>> >>> By the sound of it, this is a trust model where developers are told that >>> log4j 1.x won't break compatibility so they depend on that range without >>> actually testing against each version (newer versions may be released after >>> their own software). I question whether such a mechanism is better or worse >>> than depending on individual versions which may be overridden at a later >>> time (a la Maven). On the one hand, you don't need to release a new version >>> of the application each time a dependency is updated. On the other hand, no >>> one is actually running tests to ensure that the versions are really >>> compatible. >>> >>> Is there a way to get module A to see log4j 1.0 and module B to see log4j >>> 1.1 (using separate ClassLoaders)? >>> >>> Thanks, >>> Gili >>> >>> -- >>> View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html >>> Sent from the jigsaw-dev mailing list archive at Nabble.com. >>> From cowwoc at bbs.darktech.org Fri Nov 18 20:32:43 2011 From: cowwoc at bbs.darktech.org (cowwoc) Date: Fri, 18 Nov 2011 20:32:43 -0800 (PST) Subject: Use-cases for version ranges? In-Reply-To: <4C3634AC-251D-4C74-96EC-634C4B2E691B@pontarelli.com> References: <1321573938450-5002801.post@n5.nabble.com> <4EC68D11.4040603@oracle.com> <4C3634AC-251D-4C74-96EC-634C4B2E691B@pontarelli.com> Message-ID: <1321677163436-5006223.post@n5.nabble.com> Brian Pontarelli wrote: > > I think the two fundamental issues I have with ranges are: > > 1. At any given time I can only be certain my code will work with one > version of a dependency. That is the version I compile and test against. > Therefore, as a developer, I should only define that version in my > dependencies meta-data. > > 2. There should not be any magic upgrading of my dependencies at run or > build time. Someone should be forced to specifically upgrade one of my > dependencies and only for a really good reason. They should also be forced > to test that upgrade to ensure it will work. > > Without module isolation, this problem was much more difficult because > there could only be one version of a JAR on the classpath and therefore > build tools needed to understand how to safely upgrade JARs. Some build > tools did this better than others. > > With true isolation, this is less of an issue, making single versions even > easier to use. > > Issues come back into play when modules export modules or with circular > dependencies. These mimic the classpath behavior, which in my opinion is > best solved via compatibility rather than ranges. > I can't speak for module isolation (care to explain what you mean?) but I agree wholeheartedly with your two points. Whether we support version ranges or end-user overrides, can we all agree that developers can only be expected to certify that their code works with a single version of a dependency? If so, the question becomes: what mechanism is best suited for allowing someone other than the original developer to change the dependency version... One approach is version ranges. Another approach is end-user overrides. I don't understand what module isolation refers to but I suspect that's a third option. Gili -- View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5006223.html Sent from the jigsaw-dev mailing list archive at Nabble.com. From enewcomer at gmail.com Fri Nov 18 21:24:37 2011 From: enewcomer at gmail.com (Eric Newcomer) Date: Sat, 19 Nov 2011 00:24:37 -0500 Subject: Why package deps work (Was: Re: Converting plain JARs to Java modules) In-Reply-To: <4EC536E3.8010003@redhat.com> References: <20111108223256.D33CA13EE@eggemoggin.niobe.net> <4EBA5403.5030809@oracle.com> <31A81CCD-E5D0-478C-8AE5-EB90692CF7E3@aqute.biz> <4EBA84E5.7060409@oracle.com> <4EBA8885.6020808@redhat.com> <67949A96-DF80-43C1-B3A6-23FEADCAF612@aQute.biz> <4EC14A53.7070907@redhat.com> <4EC1F53D.7040309@redhat.com> <328B38A3-EB53-4AD2-970B-CE84F5C26F96@aQute.biz> <9B28CE0A-85DC-4F53-8A88-5830759DA4BC@pontarelli.com> <4EC536E3.8010003@redhat.com> Message-ID: Reading through this thread it looks like the different views are based on two fundamentally different assumptions: that a module system should start with Java SE/EE and fit within it's design constraints (JBoss) and that a module system should be designed from the ground up, and Java just happens to be used for it's implementation (OSGi). A lot of the back and forth seems to be about how modules fit with current practice in Java rather than the way in which a module system should behave, whether or not it's implemented using Java. Eric On Nov 17, 2011, at 11:31 AM, "David M. Lloyd" wrote: > On 11/16/2011 01:05 PM, Peter Kriens wrote: >> On 16 nov. 2011, at 19:23, Brian Pontarelli wrote: >>> Are you talking about build time here? In practice transitive >>> dependencies can always be avoided at build time. If you are >>> talking about runtime, transitive dependencies are definitely >>> required. I'm not sure exactly what your JPA vs. Eclipse Link >>> example is getting at. > > >> There is type coupling and instance coupling. Interfaces break the >> transitive type coupling but then require that you bind an instance >> for that interface in runtime, there is a trade off. >> Spring/Guice/CDI/Services, etc all handle this instance coupling. >> With a module system, packages type-decouple you from an >> implementation module just like an interface type decouples a class >> from its dependencies. In the Eclipse Link example, my code is not >> type coupled to Eclipse Link, it is only type coupled to its >> specification package javax.persistence. This allows the deployer to >> choose either Eclipse Link or OpenJPA they both provide the >> implementation to the javax.transaction package. However, my code is >> not having a type coupling on either leaving flexibility to the >> deployer. >> >> So we make a trade off between type decoupling and runtime >> complexity. This is the same trade off we make in IOC/DI engines, >> annotations, and factories. > > But I don't see why m2p gives this coupling any better than m2m? I can just as easily have a javax.transaction module with m2m as I could have a package with m2p. > > In a static module repository, there are really only two ways to choose an implementation of some given API: statically, such that there is only one implementation for an API which is globally accessed at the API's module "address", or at runtime, where the user selects their desired implementation based on their own dependencies or other selection tools. There is no special mechanism which becomes available solely due to m2p. > >>> Again, are you talking about build time or runtime? Also, are you >>> suggesting that Java determine your dependencies automatically >>> including the version? Or is this merely to help developers find >>> missing dependencies and remove unused dependencies? >> If you compile class com.bar.Foo that refers to com.bar.Bar then the >> Foo.class and Bar.class file contain all their type dependencies and >> by implication their package dependencies. The version of the package >> can then for example be found from package-info.java. >> >> This technique is used extensively in bnd, an open source library >> used in Eclipse, IDEA, ant, maven, SBT, and others. It reads the >> class files in a JAR and generates the OSGi dependency information >> automatically. > > This doesn't actually help. The user already *knows* what their dependency is - i.e. the javax.transaction API *module*, which is listed in their IDE dependencies as well as in their build - but now they must also employ special tooling in order to have the runtime module system figure it out too. This is a significant "impedance mismatch", to use a popular metaphor. > > -- > - DML From njbartlett at gmail.com Fri Nov 18 22:59:30 2011 From: njbartlett at gmail.com (Neil Bartlett) Date: Sat, 19 Nov 2011 06:59:30 +0000 Subject: Use-cases for version ranges? In-Reply-To: <4EC73025.40402@bbs.darktech.org> References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> Message-ID: Gili, I didn't say anything about guarantees, and in this industry I have never heard of anybody providing a guarantees about the performance of their software, especially in the presence of external dependencies. Version ranges are a means of communicating expectations, and we provide both a lower and an upper bound because this is useful information. I expect my module will work with 1.2.14, and I expect it will not work with 2.0. If I were a provider of the API rather than a consumer then I would have a much narrower expectation, e.g. [1.2,1.3), and this would also be useful information to convey. Regards Neil On Sat, Nov 19, 2011 at 4:27 AM, cowwoc wrote: > Neil, > > ? ?I guess I don't understand why Jigsaw should work differently from Maven > on this point. I am expecting developers to specify specific versions that > they tested (point versions, not ranges) and end-users may override these > "recommendations" as they see fit. > > ? ?Where you see version range [1.2.14, 2.0) as a way of communicating "the > developer guarantees 1.2.14 but you may use newer versions up to 2.0 at your > own risk" I'd expect the developer to simply specify 1.2.14 and there should > be no limit on what version end-users may use if they so wish. > > Gili > > On 18/11/2011 2:23 AM, Neil Bartlett wrote: >> >> I noticed that I failed to address your point about Maven using point >> versions. >> >> Maven is a build tool. At build time we need to compile against a >> single specific version so that we have repeatable builds. In general >> we should build each module against the lowest version of the library >> that it can possibly use, and there are no major negative consequences >> of having several versions of a library at build time (except that >> Maven has to download a lot!). At runtime however we need to have the >> flexibility to substitute a single compatible version. >> >> Neil >> >> On Fri, Nov 18, 2011 at 7:10 AM, Neil Bartlett >> ?wrote: >>> >>> Suppose as the developer of module A, I declare a dependency on log4j, >>> exactly version 1.0.0 because I have not tested against log4j 1.0.1, >>> 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* being >>> used with log4j version 1.0.1 even if this combinations is later >>> tested and proven to work by somebody else. In other words, testing is >>> important but it doesn't necessarily have to always be done by the >>> original developer of each module. >>> >>> On the other hand let's say I state my dependency using the following >>> range: [1.2.14, 2.0). This is OSGi syntax and I believe Jigsaw is >>> following it, and it simply means I accept version 1.2.14 up to but >>> not including 2.0. Anybody can see that I compiled and tested against >>> 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 etc. It >>> does not mean that I *guarantee* my module will work with log4j 1.3 >>> because that obviously depends on whether the log4j authors accept and >>> follow the common semantics of indicating backwards-incompatible >>> changes with a bump to the first version segment. >>> >>> The consequence of trying to lock down imports to a narrow range or >>> even a point version is that assembling an application becomes very >>> difficult, and we are forced to deploy many versions of common >>> libraries concurrently. This is non-optimal, though we can handle it >>> to some degree via per-module classloaders as in OSGi. >>> >>> Regards, >>> Neil >>> >>> On Thu, Nov 17, 2011 at 11:52 PM, cowwoc ?wrote: >>>> >>>> Can someone please explain why modules need to be able to specify >>>> version >>>> ranges for dependencies? I believe OSGI allows the specification of >>>> version >>>> ranges while Maven allows the specification of individual versions. >>>> >>>> The only thing that comes to mind is when module C depends on A and B, A >>>> depends on log4j 1.0, and B depends on log4j 1.1. What does C do? Is >>>> this >>>> the main use-case for version ranges? >>>> >>>> By the sound of it, this is a trust model where developers are told that >>>> log4j 1.x won't break compatibility so they depend on that range without >>>> actually testing against each version (newer versions may be released >>>> after >>>> their own software). I question whether such a mechanism is better or >>>> worse >>>> than depending on individual versions which may be overridden at a later >>>> time (a la Maven). On the one hand, you don't need to release a new >>>> version >>>> of the application each time a dependency is updated. On the other hand, >>>> no >>>> one is actually running tests to ensure that the versions are really >>>> compatible. >>>> >>>> Is there a way to get module A to see log4j 1.0 and module B to see >>>> log4j >>>> 1.1 (using separate ClassLoaders)? >>>> >>>> Thanks, >>>> Gili >>>> >>>> -- >>>> View this message in context: >>>> http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html >>>> Sent from the jigsaw-dev mailing list archive at Nabble.com. >>>> > > From nicolas.lalevee at hibnet.org Sat Nov 19 02:49:47 2011 From: nicolas.lalevee at hibnet.org (=?iso-8859-1?Q?Nicolas_Lalev=E9e?=) Date: Sat, 19 Nov 2011 11:49:47 +0100 Subject: Use-cases for version ranges? In-Reply-To: <1321677163436-5006223.post@n5.nabble.com> References: <1321573938450-5002801.post@n5.nabble.com> <4EC68D11.4040603@oracle.com> <4C3634AC-251D-4C74-96EC-634C4B2E691B@pontarelli.com> <1321677163436-5006223.post@n5.nabble.com> Message-ID: <2808B399-A620-4D66-A7A8-537A173F2DE9@hibnet.org> Le 19 nov. 2011 ? 05:32, cowwoc a ?crit : > > Brian Pontarelli wrote: >> >> I think the two fundamental issues I have with ranges are: >> >> 1. At any given time I can only be certain my code will work with one >> version of a dependency. That is the version I compile and test against. >> Therefore, as a developer, I should only define that version in my >> dependencies meta-data. >> >> 2. There should not be any magic upgrading of my dependencies at run or >> build time. Someone should be forced to specifically upgrade one of my >> dependencies and only for a really good reason. They should also be forced >> to test that upgrade to ensure it will work. >> >> Without module isolation, this problem was much more difficult because >> there could only be one version of a JAR on the classpath and therefore >> build tools needed to understand how to safely upgrade JARs. Some build >> tools did this better than others. >> >> With true isolation, this is less of an issue, making single versions even >> easier to use. >> >> Issues come back into play when modules export modules or with circular >> dependencies. These mimic the classpath behavior, which in my opinion is >> best solved via compatibility rather than ranges. >> > > I can't speak for module isolation (care to explain what you mean?) but I > agree wholeheartedly with your two points. Whether we support version ranges > or end-user overrides, can we all agree that developers can only be expected > to certify that their code works with a single version of a dependency? > > If so, the question becomes: what mechanism is best suited for allowing > someone other than the original developer to change the dependency > version... One approach is version ranges. Another approach is end-user > overrides. I don't understand what module isolation refers to but I suspect > that's a third option. I'm not sure either how module isolation would help either. But I think there's solution we don't want to see because we fear duplicates, redundancy, this is error prone: having two dependency descriptors. But as Neil already wrote it: "At build time we need to compile against a single specific version so that we have repeatable builds. [...] At runtime however we need to have the flexibility to substitute a single compatible version." Since we want to declare two very different things, I don't think of no other way than having two kinds of declaration. And note that these two declarations will be used at very different times. The first will be used while we construct the classpath (to build or to deploy), the other by the jvm at runtime to ensure modularity. I think I have a solution. I'm experimenting it with Ivy (which is good at building classpath) and OSGi (which is good at runtime dependencies). A project in development would have an ivy.xml in which I would declare the dependency I want to build, test, run against. It could looks like a dependency on commons-logging 1.0. I make Ivy resolves and retrieves my dependencies against an OSGi repository (I'll explain why later), so I get my jars in a lib folder. Then I need to make my project runnable in an OSGi environment, so I need an other descriptor. I need to redeclare my dependency in a MANIFEST ? Actually no since OSGi dependency model is nicely integrated in the Java environment, no extra namespace, just java packages. So my dependencies are already declared by my java import in every of my classes. And there's nice tools (bnd, bundlor) to compute the MANIFEST automatically. So no burden. In the MANIFEST I would then have a dependency on the package org.apache.commons.logging. Last but not least, the release of the project. Since I don't want the users of my project to be tied to the dependencies I chose in my ivy.xml, I shouldn't publish it. Instead, users should be aware of what my project is requiring, which API must be fullfield: the information is in the MANIFEST. The end users should fulfill the requirement on the package org.apache.commons.logging by either the implementation bundle commons-logging or the bundle jcl-over-slf4j. So I'll publish only the OSGi metadata, hence my project should be publish to an OSGi repository. And the development cycle is closed. There's one thing to note about the Ivy resolve against an OSGi repository. As the OSGi Bundle Repository spec draft nicely state: "SHALL NOT be required to produce identical sets of resolved resources throughout every phase of the development process." Since the inherent "flexibility" nature of the dependency declaration of an OSGi bundle, two resolves at different times may result in two different classpath. Hence the Ivy resolve would be done only one to build a classpath. And only do it again on dependency change. In Eclipse world, this is what is called an target platform. This is the best dependency management I have found so far. I have to admit it is theoretical, Ivy's model is not yet ready to understand OSGi dependency model. A last note on the solution on overriding dependencies. This really looks like a hack to me. The developer of a module A declare a dependency on module B and we say, no, it's actually module C. For me it it like changing the visibility of a method from private to public via introspection. Nicolas From cowwoc at bbs.darktech.org Sat Nov 19 18:55:06 2011 From: cowwoc at bbs.darktech.org (cowwoc) Date: Sat, 19 Nov 2011 18:55:06 -0800 (PST) Subject: Use-cases for version ranges? In-Reply-To: <2808B399-A620-4D66-A7A8-537A173F2DE9@hibnet.org> References: <1321573938450-5002801.post@n5.nabble.com> <4EC68D11.4040603@oracle.com> <4C3634AC-251D-4C74-96EC-634C4B2E691B@pontarelli.com> <1321677163436-5006223.post@n5.nabble.com> <2808B399-A620-4D66-A7A8-537A173F2DE9@hibnet.org> Message-ID: <4EC86BF7.7050105@bbs.darktech.org> On 19/11/2011 5:51 AM, Nicolas Lalev?e [via jigsaw-dev] wrote: > A last note on the solution on overriding dependencies. This really > looks like a hack to me. The developer of a module A declare a > dependency on module B and we say, no, it's actually module C. For me > it it like changing the visibility of a method from private to public > via introspection. We aren't substituting module B for module C. We are substituting module B version 1.0 with module B version 1.1. So long as we substitute for a version which is compatible wiht version 1.0 why should the author care? Gili -- View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5007723.html Sent from the jigsaw-dev mailing list archive at Nabble.com. From cowwoc at bbs.darktech.org Sat Nov 19 19:09:37 2011 From: cowwoc at bbs.darktech.org (cowwoc) Date: Sat, 19 Nov 2011 22:09:37 -0500 Subject: Use-cases for version ranges? In-Reply-To: References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> Message-ID: <4EC86F71.1000707@bbs.darktech.org> I'd like to propose another possibility: the author of the dependency should tell *us* about version compatibility, not the other way around. For example: 1. The author of module A declares a dependency on module B version 1.2 (specific version). 2. The author of module B publishes version 1.3. He declares that version 1.3 is compatible with 1.2 (meaning, the runtime system is allows to substitute version 1.3 for 1.2). The upside of this approach is that the author of B is in a better position to declare compatibility than the author of A. The author of A still only needs to test a single version. What do you think? Gili On 19/11/2011 1:59 AM, Neil Bartlett wrote: > Gili, > > I didn't say anything about guarantees, and in this industry I have > never heard of anybody providing a guarantees about the performance of > their software, especially in the presence of external dependencies. > > Version ranges are a means of communicating expectations, and we > provide both a lower and an upper bound because this is useful > information. I expect my module will work with 1.2.14, and I expect it > will not work with 2.0. If I were a provider of the API rather than a > consumer then I would have a much narrower expectation, e.g. > [1.2,1.3), and this would also be useful information to convey. > > Regards > Neil > > On Sat, Nov 19, 2011 at 4:27 AM, cowwoc wrote: >> Neil, >> >> I guess I don't understand why Jigsaw should work differently from Maven >> on this point. I am expecting developers to specify specific versions that >> they tested (point versions, not ranges) and end-users may override these >> "recommendations" as they see fit. >> >> Where you see version range [1.2.14, 2.0) as a way of communicating "the >> developer guarantees 1.2.14 but you may use newer versions up to 2.0 at your >> own risk" I'd expect the developer to simply specify 1.2.14 and there should >> be no limit on what version end-users may use if they so wish. >> >> Gili >> >> On 18/11/2011 2:23 AM, Neil Bartlett wrote: >>> I noticed that I failed to address your point about Maven using point >>> versions. >>> >>> Maven is a build tool. At build time we need to compile against a >>> single specific version so that we have repeatable builds. In general >>> we should build each module against the lowest version of the library >>> that it can possibly use, and there are no major negative consequences >>> of having several versions of a library at build time (except that >>> Maven has to download a lot!). At runtime however we need to have the >>> flexibility to substitute a single compatible version. >>> >>> Neil >>> >>> On Fri, Nov 18, 2011 at 7:10 AM, Neil Bartlett >>> wrote: >>>> Suppose as the developer of module A, I declare a dependency on log4j, >>>> exactly version 1.0.0 because I have not tested against log4j 1.0.1, >>>> 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* being >>>> used with log4j version 1.0.1 even if this combinations is later >>>> tested and proven to work by somebody else. In other words, testing is >>>> important but it doesn't necessarily have to always be done by the >>>> original developer of each module. >>>> >>>> On the other hand let's say I state my dependency using the following >>>> range: [1.2.14, 2.0). This is OSGi syntax and I believe Jigsaw is >>>> following it, and it simply means I accept version 1.2.14 up to but >>>> not including 2.0. Anybody can see that I compiled and tested against >>>> 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 etc. It >>>> does not mean that I *guarantee* my module will work with log4j 1.3 >>>> because that obviously depends on whether the log4j authors accept and >>>> follow the common semantics of indicating backwards-incompatible >>>> changes with a bump to the first version segment. >>>> >>>> The consequence of trying to lock down imports to a narrow range or >>>> even a point version is that assembling an application becomes very >>>> difficult, and we are forced to deploy many versions of common >>>> libraries concurrently. This is non-optimal, though we can handle it >>>> to some degree via per-module classloaders as in OSGi. >>>> >>>> Regards, >>>> Neil >>>> >>>> On Thu, Nov 17, 2011 at 11:52 PM, cowwoc wrote: >>>>> Can someone please explain why modules need to be able to specify >>>>> version >>>>> ranges for dependencies? I believe OSGI allows the specification of >>>>> version >>>>> ranges while Maven allows the specification of individual versions. >>>>> >>>>> The only thing that comes to mind is when module C depends on A and B, A >>>>> depends on log4j 1.0, and B depends on log4j 1.1. What does C do? Is >>>>> this >>>>> the main use-case for version ranges? >>>>> >>>>> By the sound of it, this is a trust model where developers are told that >>>>> log4j 1.x won't break compatibility so they depend on that range without >>>>> actually testing against each version (newer versions may be released >>>>> after >>>>> their own software). I question whether such a mechanism is better or >>>>> worse >>>>> than depending on individual versions which may be overridden at a later >>>>> time (a la Maven). On the one hand, you don't need to release a new >>>>> version >>>>> of the application each time a dependency is updated. On the other hand, >>>>> no >>>>> one is actually running tests to ensure that the versions are really >>>>> compatible. >>>>> >>>>> Is there a way to get module A to see log4j 1.0 and module B to see >>>>> log4j >>>>> 1.1 (using separate ClassLoaders)? >>>>> >>>>> Thanks, >>>>> Gili >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html >>>>> Sent from the jigsaw-dev mailing list archive at Nabble.com. >>>>> >> From mcconnell at dpml.net Sun Nov 20 04:25:17 2011 From: mcconnell at dpml.net (Stephen McConnell) Date: Sun, 20 Nov 2011 22:55:17 +1030 Subject: Use-cases for version ranges? In-Reply-To: <4EC86F71.1000707@bbs.darktech.org> References: <1321573938450-5002801.post@n5.nabble.com><4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> Message-ID: I could say that the question is academic - but here is the thing - its not even academic. We talk about versions and yet the platform has no computational mechanisms to flag version violation. At least an academic discussion would bring in grounded metrics. In reality - the discussion here about V1 versus V2 and that warm fuzzy feeling when your are just a n.m from an n+1 and it's a waste of breath (and that comment goes to all sides of the debate). Give me an endorsed tool that shows me that module [name]-[build] is computationally backward compatible to a different [name]-[build] and then, only them will this discussion have reasonable grounds to assert any authority on the subject of version compatibility (past or future). Without that tool - versioning is a unqualified statement of trust. And the evidence suggests that the threshold of trust delivery on actual computational compliance is really really thin. Cheers, Steve. -----Original Message----- From: cowwoc Sent: Sunday, November 20, 2011 1:39 PM To: jigsaw-dev at openjdk.java.net Subject: Re: Use-cases for version ranges? I'd like to propose another possibility: the author of the dependency should tell *us* about version compatibility, not the other way around. For example: 1. The author of module A declares a dependency on module B version 1.2 (specific version). 2. The author of module B publishes version 1.3. He declares that version 1.3 is compatible with 1.2 (meaning, the runtime system is allows to substitute version 1.3 for 1.2). The upside of this approach is that the author of B is in a better position to declare compatibility than the author of A. The author of A still only needs to test a single version. What do you think? Gili On 19/11/2011 1:59 AM, Neil Bartlett wrote: > Gili, > > I didn't say anything about guarantees, and in this industry I have > never heard of anybody providing a guarantees about the performance of > their software, especially in the presence of external dependencies. > > Version ranges are a means of communicating expectations, and we > provide both a lower and an upper bound because this is useful > information. I expect my module will work with 1.2.14, and I expect it > will not work with 2.0. If I were a provider of the API rather than a > consumer then I would have a much narrower expectation, e.g. > [1.2,1.3), and this would also be useful information to convey. > > Regards > Neil > > On Sat, Nov 19, 2011 at 4:27 AM, cowwoc wrote: >> Neil, >> >> I guess I don't understand why Jigsaw should work differently from >> Maven >> on this point. I am expecting developers to specify specific versions >> that >> they tested (point versions, not ranges) and end-users may override these >> "recommendations" as they see fit. >> >> Where you see version range [1.2.14, 2.0) as a way of communicating >> "the >> developer guarantees 1.2.14 but you may use newer versions up to 2.0 at >> your >> own risk" I'd expect the developer to simply specify 1.2.14 and there >> should >> be no limit on what version end-users may use if they so wish. >> >> Gili >> >> On 18/11/2011 2:23 AM, Neil Bartlett wrote: >>> I noticed that I failed to address your point about Maven using point >>> versions. >>> >>> Maven is a build tool. At build time we need to compile against a >>> single specific version so that we have repeatable builds. In general >>> we should build each module against the lowest version of the library >>> that it can possibly use, and there are no major negative consequences >>> of having several versions of a library at build time (except that >>> Maven has to download a lot!). At runtime however we need to have the >>> flexibility to substitute a single compatible version. >>> >>> Neil >>> >>> On Fri, Nov 18, 2011 at 7:10 AM, Neil Bartlett >>> wrote: >>>> Suppose as the developer of module A, I declare a dependency on log4j, >>>> exactly version 1.0.0 because I have not tested against log4j 1.0.1, >>>> 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* being >>>> used with log4j version 1.0.1 even if this combinations is later >>>> tested and proven to work by somebody else. In other words, testing is >>>> important but it doesn't necessarily have to always be done by the >>>> original developer of each module. >>>> >>>> On the other hand let's say I state my dependency using the following >>>> range: [1.2.14, 2.0). This is OSGi syntax and I believe Jigsaw is >>>> following it, and it simply means I accept version 1.2.14 up to but >>>> not including 2.0. Anybody can see that I compiled and tested against >>>> 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 etc. It >>>> does not mean that I *guarantee* my module will work with log4j 1.3 >>>> because that obviously depends on whether the log4j authors accept and >>>> follow the common semantics of indicating backwards-incompatible >>>> changes with a bump to the first version segment. >>>> >>>> The consequence of trying to lock down imports to a narrow range or >>>> even a point version is that assembling an application becomes very >>>> difficult, and we are forced to deploy many versions of common >>>> libraries concurrently. This is non-optimal, though we can handle it >>>> to some degree via per-module classloaders as in OSGi. >>>> >>>> Regards, >>>> Neil >>>> >>>> On Thu, Nov 17, 2011 at 11:52 PM, cowwoc >>>> wrote: >>>>> Can someone please explain why modules need to be able to specify >>>>> version >>>>> ranges for dependencies? I believe OSGI allows the specification of >>>>> version >>>>> ranges while Maven allows the specification of individual versions. >>>>> >>>>> The only thing that comes to mind is when module C depends on A and B, >>>>> A >>>>> depends on log4j 1.0, and B depends on log4j 1.1. What does C do? Is >>>>> this >>>>> the main use-case for version ranges? >>>>> >>>>> By the sound of it, this is a trust model where developers are told >>>>> that >>>>> log4j 1.x won't break compatibility so they depend on that range >>>>> without >>>>> actually testing against each version (newer versions may be released >>>>> after >>>>> their own software). I question whether such a mechanism is better or >>>>> worse >>>>> than depending on individual versions which may be overridden at a >>>>> later >>>>> time (a la Maven). On the one hand, you don't need to release a new >>>>> version >>>>> of the application each time a dependency is updated. On the other >>>>> hand, >>>>> no >>>>> one is actually running tests to ensure that the versions are really >>>>> compatible. >>>>> >>>>> Is there a way to get module A to see log4j 1.0 and module B to see >>>>> log4j >>>>> 1.1 (using separate ClassLoaders)? >>>>> >>>>> Thanks, >>>>> Gili >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html >>>>> Sent from the jigsaw-dev mailing list archive at Nabble.com. >>>>> >> From nicolas.lalevee at hibnet.org Sun Nov 20 05:18:40 2011 From: nicolas.lalevee at hibnet.org (=?iso-8859-1?Q?Nicolas_Lalev=E9e?=) Date: Sun, 20 Nov 2011 14:18:40 +0100 Subject: Use-cases for version ranges? In-Reply-To: <4EC86BF7.7050105@bbs.darktech.org> References: <1321573938450-5002801.post@n5.nabble.com> <4EC68D11.4040603@oracle.com> <4C3634AC-251D-4C74-96EC-634C4B2E691B@pontarelli.com> <1321677163436-5006223.post@n5.nabble.com> <2808B399-A620-4D66-A7A8-537A173F2DE9@hibnet.org> <4EC86BF7.7050105@bbs.darktech.org> Message-ID: Le 20 nov. 2011 ? 03:55, cowwoc a ?crit : > On 19/11/2011 5:51 AM, Nicolas Lalev?e [via jigsaw-dev] wrote: >> A last note on the solution on overriding dependencies. This really >> looks like a hack to me. The developer of a module A declare a >> dependency on module B and we say, no, it's actually module C. For me >> it it like changing the visibility of a method from private to public >> via introspection. > > We aren't substituting module B for module C. We are substituting > module B version 1.0 with module B version 1.1. So long as we substitute > for a version which is compatible wiht version 1.0 why should the author > care? The author probably not, but what about the module system in the jvm ? The dependency declarations are for the module system so it can correctly ensure modularity. Nicolas From nicolas.lalevee at hibnet.org Sun Nov 20 05:21:45 2011 From: nicolas.lalevee at hibnet.org (=?iso-8859-1?Q?Nicolas_Lalev=E9e?=) Date: Sun, 20 Nov 2011 14:21:45 +0100 Subject: Use-cases for version ranges? In-Reply-To: References: <1321573938450-5002801.post@n5.nabble.com><4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> Message-ID: Le 20 nov. 2011 ? 13:25, Stephen McConnell a ?crit : > > I could say that the question is academic - but here is the thing - its not even academic. > > We talk about versions and yet the platform has no computational mechanisms to flag version violation. At least an academic discussion would bring in grounded metrics. In reality - the discussion here about V1 versus V2 and that warm fuzzy feeling when your are just a n.m from an n+1 and it's a waste of breath (and that comment goes to all sides of the debate). > > Give me an endorsed tool that shows me that module [name]-[build] is computationally backward compatible to a different [name]-[build] and then, only them will this discussion have reasonable grounds to assert any authority on the subject of version compatibility (past or future). Without that tool - versioning is a unqualified statement of trust. And the evidence suggests that the threshold of trust delivery on actual computational compliance is really really thin. I haven't use it myself here is a tool which is about backward compatibility check automation: http://www.eclipse.org/pde/pde-api-tools/ Nicolas > > Cheers, Steve. > > > -----Original Message----- From: cowwoc > Sent: Sunday, November 20, 2011 1:39 PM > To: jigsaw-dev at openjdk.java.net > Subject: Re: Use-cases for version ranges? > > > I'd like to propose another possibility: the author of the > dependency should tell *us* about version compatibility, not the other > way around. For example: > > 1. The author of module A declares a dependency on module B version 1.2 > (specific version). > 2. The author of module B publishes version 1.3. He declares that > version 1.3 is compatible with 1.2 (meaning, the runtime system is > allows to substitute version 1.3 for 1.2). > > The upside of this approach is that the author of B is in a better > position to declare compatibility than the author of A. The author of A > still only needs to test a single version. What do you think? > > Gili > > On 19/11/2011 1:59 AM, Neil Bartlett wrote: >> Gili, >> >> I didn't say anything about guarantees, and in this industry I have >> never heard of anybody providing a guarantees about the performance of >> their software, especially in the presence of external dependencies. >> >> Version ranges are a means of communicating expectations, and we >> provide both a lower and an upper bound because this is useful >> information. I expect my module will work with 1.2.14, and I expect it >> will not work with 2.0. If I were a provider of the API rather than a >> consumer then I would have a much narrower expectation, e.g. >> [1.2,1.3), and this would also be useful information to convey. >> >> Regards >> Neil >> >> On Sat, Nov 19, 2011 at 4:27 AM, cowwoc wrote: >>> Neil, >>> >>> I guess I don't understand why Jigsaw should work differently from Maven >>> on this point. I am expecting developers to specify specific versions that >>> they tested (point versions, not ranges) and end-users may override these >>> "recommendations" as they see fit. >>> >>> Where you see version range [1.2.14, 2.0) as a way of communicating "the >>> developer guarantees 1.2.14 but you may use newer versions up to 2.0 at your >>> own risk" I'd expect the developer to simply specify 1.2.14 and there should >>> be no limit on what version end-users may use if they so wish. >>> >>> Gili >>> >>> On 18/11/2011 2:23 AM, Neil Bartlett wrote: >>>> I noticed that I failed to address your point about Maven using point >>>> versions. >>>> >>>> Maven is a build tool. At build time we need to compile against a >>>> single specific version so that we have repeatable builds. In general >>>> we should build each module against the lowest version of the library >>>> that it can possibly use, and there are no major negative consequences >>>> of having several versions of a library at build time (except that >>>> Maven has to download a lot!). At runtime however we need to have the >>>> flexibility to substitute a single compatible version. >>>> >>>> Neil >>>> >>>> On Fri, Nov 18, 2011 at 7:10 AM, Neil Bartlett >>>> wrote: >>>>> Suppose as the developer of module A, I declare a dependency on log4j, >>>>> exactly version 1.0.0 because I have not tested against log4j 1.0.1, >>>>> 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* being >>>>> used with log4j version 1.0.1 even if this combinations is later >>>>> tested and proven to work by somebody else. In other words, testing is >>>>> important but it doesn't necessarily have to always be done by the >>>>> original developer of each module. >>>>> >>>>> On the other hand let's say I state my dependency using the following >>>>> range: [1.2.14, 2.0). This is OSGi syntax and I believe Jigsaw is >>>>> following it, and it simply means I accept version 1.2.14 up to but >>>>> not including 2.0. Anybody can see that I compiled and tested against >>>>> 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 etc. It >>>>> does not mean that I *guarantee* my module will work with log4j 1.3 >>>>> because that obviously depends on whether the log4j authors accept and >>>>> follow the common semantics of indicating backwards-incompatible >>>>> changes with a bump to the first version segment. >>>>> >>>>> The consequence of trying to lock down imports to a narrow range or >>>>> even a point version is that assembling an application becomes very >>>>> difficult, and we are forced to deploy many versions of common >>>>> libraries concurrently. This is non-optimal, though we can handle it >>>>> to some degree via per-module classloaders as in OSGi. >>>>> >>>>> Regards, >>>>> Neil >>>>> >>>>> On Thu, Nov 17, 2011 at 11:52 PM, cowwoc wrote: >>>>>> Can someone please explain why modules need to be able to specify >>>>>> version >>>>>> ranges for dependencies? I believe OSGI allows the specification of >>>>>> version >>>>>> ranges while Maven allows the specification of individual versions. >>>>>> >>>>>> The only thing that comes to mind is when module C depends on A and B, A >>>>>> depends on log4j 1.0, and B depends on log4j 1.1. What does C do? Is >>>>>> this >>>>>> the main use-case for version ranges? >>>>>> >>>>>> By the sound of it, this is a trust model where developers are told that >>>>>> log4j 1.x won't break compatibility so they depend on that range without >>>>>> actually testing against each version (newer versions may be released >>>>>> after >>>>>> their own software). I question whether such a mechanism is better or >>>>>> worse >>>>>> than depending on individual versions which may be overridden at a later >>>>>> time (a la Maven). On the one hand, you don't need to release a new >>>>>> version >>>>>> of the application each time a dependency is updated. On the other hand, >>>>>> no >>>>>> one is actually running tests to ensure that the versions are really >>>>>> compatible. >>>>>> >>>>>> Is there a way to get module A to see log4j 1.0 and module B to see >>>>>> log4j >>>>>> 1.1 (using separate ClassLoaders)? >>>>>> >>>>>> Thanks, >>>>>> Gili >>>>>> >>>>>> -- >>>>>> View this message in context: >>>>>> http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html >>>>>> Sent from the jigsaw-dev mailing list archive at Nabble.com. >>>>>> >>> > From njbartlett at gmail.com Sun Nov 20 05:45:38 2011 From: njbartlett at gmail.com (Neil Bartlett) Date: Sun, 20 Nov 2011 13:45:38 +0000 Subject: Use-cases for version ranges? In-Reply-To: References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> Message-ID: Stephen, You again seem to be talking about dependencies on implementation packages. If we depend on APIs consisting primarily of interfaces, then versions and version ranges are reliable and can be completely automated. Regards, Neil On Sun, Nov 20, 2011 at 12:25 PM, Stephen McConnell wrote: > > I could say that the question is academic - but here is the thing - its not > even academic. > > We talk about versions and yet the platform has no computational mechanisms > to flag version violation. At least an academic discussion would bring in > grounded metrics. ?In reality - the discussion here about V1 versus V2 and > that warm fuzzy feeling when your are just a n.m from an n+1 and it's a > waste of breath (and that comment goes to all sides of the debate). > > Give me an endorsed tool that shows me that module [name]-[build] is > computationally backward compatible to a different [name]-[build] and then, > only them will this discussion have reasonable grounds to assert any > authority on the subject of version compatibility (past or future). Without > that tool - versioning is a unqualified statement of trust. And the evidence > suggests that the threshold of trust delivery on actual computational > compliance is really really thin. > > Cheers, Steve. > > > -----Original Message----- From: cowwoc > Sent: Sunday, November 20, 2011 1:39 PM > To: jigsaw-dev at openjdk.java.net > Subject: Re: Use-cases for version ranges? > > > ? ?I'd like to propose another possibility: the author of the > dependency should tell *us* about version compatibility, not the other > way around. For example: > > 1. The author of module A declares a dependency on module B version 1.2 > (specific version). > 2. The author of module B publishes version 1.3. He declares that > version 1.3 is compatible with 1.2 (meaning, the runtime system is > allows to substitute version 1.3 for 1.2). > > ? ?The upside of this approach is that the author of B is in a better > position to declare compatibility than the author of A. The author of A > still only needs to test a single version. What do you think? > > Gili > > On 19/11/2011 1:59 AM, Neil Bartlett wrote: >> >> Gili, >> >> I didn't say anything about guarantees, and in this industry I have >> never heard of anybody providing a guarantees about the performance of >> their software, especially in the presence of external dependencies. >> >> Version ranges are a means of communicating expectations, and we >> provide both a lower and an upper bound because this is useful >> information. I expect my module will work with 1.2.14, and I expect it >> will not work with 2.0. If I were a provider of the API rather than a >> consumer then I would have a much narrower expectation, ?e.g. >> [1.2,1.3), and this would also be useful information to convey. >> >> Regards >> Neil >> >> On Sat, Nov 19, 2011 at 4:27 AM, cowwoc ?wrote: >>> >>> Neil, >>> >>> ? ?I guess I don't understand why Jigsaw should work differently from >>> Maven >>> on this point. I am expecting developers to specify specific versions >>> that >>> they tested (point versions, not ranges) and end-users may override these >>> "recommendations" as they see fit. >>> >>> ? ?Where you see version range [1.2.14, 2.0) as a way of communicating >>> "the >>> developer guarantees 1.2.14 but you may use newer versions up to 2.0 at >>> your >>> own risk" I'd expect the developer to simply specify 1.2.14 and there >>> should >>> be no limit on what version end-users may use if they so wish. >>> >>> Gili >>> >>> On 18/11/2011 2:23 AM, Neil Bartlett wrote: >>>> >>>> I noticed that I failed to address your point about Maven using point >>>> versions. >>>> >>>> Maven is a build tool. At build time we need to compile against a >>>> single specific version so that we have repeatable builds. In general >>>> we should build each module against the lowest version of the library >>>> that it can possibly use, and there are no major negative consequences >>>> of having several versions of a library at build time (except that >>>> Maven has to download a lot!). At runtime however we need to have the >>>> flexibility to substitute a single compatible version. >>>> >>>> Neil >>>> >>>> On Fri, Nov 18, 2011 at 7:10 AM, Neil Bartlett >>>> ?wrote: >>>>> >>>>> Suppose as the developer of module A, I declare a dependency on log4j, >>>>> exactly version 1.0.0 because I have not tested against log4j 1.0.1, >>>>> 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* being >>>>> used with log4j version 1.0.1 even if this combinations is later >>>>> tested and proven to work by somebody else. In other words, testing is >>>>> important but it doesn't necessarily have to always be done by the >>>>> original developer of each module. >>>>> >>>>> On the other hand let's say I state my dependency using the following >>>>> range: [1.2.14, 2.0). This is OSGi syntax and I believe Jigsaw is >>>>> following it, and it simply means I accept version 1.2.14 up to but >>>>> not including 2.0. Anybody can see that I compiled and tested against >>>>> 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 etc. It >>>>> does not mean that I *guarantee* my module will work with log4j 1.3 >>>>> because that obviously depends on whether the log4j authors accept and >>>>> follow the common semantics of indicating backwards-incompatible >>>>> changes with a bump to the first version segment. >>>>> >>>>> The consequence of trying to lock down imports to a narrow range or >>>>> even a point version is that assembling an application becomes very >>>>> difficult, and we are forced to deploy many versions of common >>>>> libraries concurrently. This is non-optimal, though we can handle it >>>>> to some degree via per-module classloaders as in OSGi. >>>>> >>>>> Regards, >>>>> Neil >>>>> >>>>> On Thu, Nov 17, 2011 at 11:52 PM, cowwoc >>>>> wrote: >>>>>> >>>>>> Can someone please explain why modules need to be able to specify >>>>>> version >>>>>> ranges for dependencies? I believe OSGI allows the specification of >>>>>> version >>>>>> ranges while Maven allows the specification of individual versions. >>>>>> >>>>>> The only thing that comes to mind is when module C depends on A and B, >>>>>> A >>>>>> depends on log4j 1.0, and B depends on log4j 1.1. What does C do? Is >>>>>> this >>>>>> the main use-case for version ranges? >>>>>> >>>>>> By the sound of it, this is a trust model where developers are told >>>>>> that >>>>>> log4j 1.x won't break compatibility so they depend on that range >>>>>> without >>>>>> actually testing against each version (newer versions may be released >>>>>> after >>>>>> their own software). I question whether such a mechanism is better or >>>>>> worse >>>>>> than depending on individual versions which may be overridden at a >>>>>> later >>>>>> time (a la Maven). On the one hand, you don't need to release a new >>>>>> version >>>>>> of the application each time a dependency is updated. On the other >>>>>> hand, >>>>>> no >>>>>> one is actually running tests to ensure that the versions are really >>>>>> compatible. >>>>>> >>>>>> Is there a way to get module A to see log4j 1.0 and module B to see >>>>>> log4j >>>>>> 1.1 (using separate ClassLoaders)? >>>>>> >>>>>> Thanks, >>>>>> Gili >>>>>> >>>>>> -- >>>>>> View this message in context: >>>>>> >>>>>> http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html >>>>>> Sent from the jigsaw-dev mailing list archive at Nabble.com. >>>>>> >>> > > From lukas.krecan at gmail.com Sun Nov 20 07:51:54 2011 From: lukas.krecan at gmail.com (Lukas Krecan) Date: Sun, 20 Nov 2011 16:51:54 +0100 Subject: Use-cases for version ranges? In-Reply-To: <4EC86F71.1000707@bbs.darktech.org> References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> Message-ID: <4EC9221A.4020409@gmail.com> Hi, it's obvious that the only one who has all the information is the person DEPLOYING the application (in Java EE it's Application Assembler role). In the enterprise word the application is maintained long after the implementation is finished. Imagine that there was a security problem in a library and a new version has been released. As a person taking care of the application I want just change this dependency without having to recompile the application. So having an automatic resolution of dependency versions is a nice feature, but we will always need a way how to specify the dependencies at the runtime. To reiterate, let's reuse the previous example, where 0 depends on A which in turn depends on B. 1. Author of B can not guarantee that B 1.3 is compatible with 1.2. Well, he can, but he can not be trusted. Some changes can cause problems that can not be envisaged by their author. The compatibility can be tested only by the users of the library! 2. Author of A can not garantee that A will work with future versions of B. She can only test it with existing versions of B. She can say that A 3.0 is compatible with B [1.0,1.9]. 3. Application 0 depends on A 3.0 and B 2.0 has been released. I want to use it, but new version of A has not been released yet. I need to override the A-> B dependency. 4. All programmers have left the project 0 and few months later, B 2.1 has been released. It contains a critical security bug fix. System administrator needs to specify this dependency without having to recompile the application 0. He just needs to specify the new dependency. You see, the dependency resolution can not be fully automated. At the end, some dependencies have to be specified manually regardless the versioning mechanism. The cause is simple. All the components have different lifecycle and only the end-user (application 0) has all the data. Only him can test that the combination works as expected. Regards Lukas On 11/20/2011 04:09 AM, cowwoc wrote: > > I'd like to propose another possibility: the author of the > dependency should tell *us* about version compatibility, not the other > way around. For example: > > 1. The author of module A declares a dependency on module B version > 1.2 (specific version). > 2. The author of module B publishes version 1.3. He declares that > version 1.3 is compatible with 1.2 (meaning, the runtime system is > allows to substitute version 1.3 for 1.2). > > The upside of this approach is that the author of B is in a better > position to declare compatibility than the author of A. The author of > A still only needs to test a single version. What do you think? > > Gili > > On 19/11/2011 1:59 AM, Neil Bartlett wrote: >> Gili, >> >> I didn't say anything about guarantees, and in this industry I have >> never heard of anybody providing a guarantees about the performance of >> their software, especially in the presence of external dependencies. >> >> Version ranges are a means of communicating expectations, and we >> provide both a lower and an upper bound because this is useful >> information. I expect my module will work with 1.2.14, and I expect it >> will not work with 2.0. If I were a provider of the API rather than a >> consumer then I would have a much narrower expectation, e.g. >> [1.2,1.3), and this would also be useful information to convey. >> >> Regards >> Neil >> >> On Sat, Nov 19, 2011 at 4:27 AM, cowwoc wrote: >>> Neil, >>> >>> I guess I don't understand why Jigsaw should work differently >>> from Maven >>> on this point. I am expecting developers to specify specific >>> versions that >>> they tested (point versions, not ranges) and end-users may override >>> these >>> "recommendations" as they see fit. >>> >>> Where you see version range [1.2.14, 2.0) as a way of >>> communicating "the >>> developer guarantees 1.2.14 but you may use newer versions up to 2.0 >>> at your >>> own risk" I'd expect the developer to simply specify 1.2.14 and >>> there should >>> be no limit on what version end-users may use if they so wish. >>> >>> Gili >>> >>> On 18/11/2011 2:23 AM, Neil Bartlett wrote: >>>> I noticed that I failed to address your point about Maven using point >>>> versions. >>>> >>>> Maven is a build tool. At build time we need to compile against a >>>> single specific version so that we have repeatable builds. In general >>>> we should build each module against the lowest version of the library >>>> that it can possibly use, and there are no major negative consequences >>>> of having several versions of a library at build time (except that >>>> Maven has to download a lot!). At runtime however we need to have the >>>> flexibility to substitute a single compatible version. >>>> >>>> Neil >>>> >>>> On Fri, Nov 18, 2011 at 7:10 AM, Neil Bartlett >>>> wrote: >>>>> Suppose as the developer of module A, I declare a dependency on >>>>> log4j, >>>>> exactly version 1.0.0 because I have not tested against log4j 1.0.1, >>>>> 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* being >>>>> used with log4j version 1.0.1 even if this combinations is later >>>>> tested and proven to work by somebody else. In other words, >>>>> testing is >>>>> important but it doesn't necessarily have to always be done by the >>>>> original developer of each module. >>>>> >>>>> On the other hand let's say I state my dependency using the following >>>>> range: [1.2.14, 2.0). This is OSGi syntax and I believe Jigsaw is >>>>> following it, and it simply means I accept version 1.2.14 up to but >>>>> not including 2.0. Anybody can see that I compiled and tested against >>>>> 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 etc. It >>>>> does not mean that I *guarantee* my module will work with log4j 1.3 >>>>> because that obviously depends on whether the log4j authors accept >>>>> and >>>>> follow the common semantics of indicating backwards-incompatible >>>>> changes with a bump to the first version segment. >>>>> >>>>> The consequence of trying to lock down imports to a narrow range or >>>>> even a point version is that assembling an application becomes very >>>>> difficult, and we are forced to deploy many versions of common >>>>> libraries concurrently. This is non-optimal, though we can handle it >>>>> to some degree via per-module classloaders as in OSGi. >>>>> >>>>> Regards, >>>>> Neil >>>>> >>>>> On Thu, Nov 17, 2011 at 11:52 PM, >>>>> cowwoc wrote: >>>>>> Can someone please explain why modules need to be able to specify >>>>>> version >>>>>> ranges for dependencies? I believe OSGI allows the specification of >>>>>> version >>>>>> ranges while Maven allows the specification of individual versions. >>>>>> >>>>>> The only thing that comes to mind is when module C depends on A >>>>>> and B, A >>>>>> depends on log4j 1.0, and B depends on log4j 1.1. What does C do? Is >>>>>> this >>>>>> the main use-case for version ranges? >>>>>> >>>>>> By the sound of it, this is a trust model where developers are >>>>>> told that >>>>>> log4j 1.x won't break compatibility so they depend on that range >>>>>> without >>>>>> actually testing against each version (newer versions may be >>>>>> released >>>>>> after >>>>>> their own software). I question whether such a mechanism is >>>>>> better or >>>>>> worse >>>>>> than depending on individual versions which may be overridden at >>>>>> a later >>>>>> time (a la Maven). On the one hand, you don't need to release a new >>>>>> version >>>>>> of the application each time a dependency is updated. On the >>>>>> other hand, >>>>>> no >>>>>> one is actually running tests to ensure that the versions are really >>>>>> compatible. >>>>>> >>>>>> Is there a way to get module A to see log4j 1.0 and module B to see >>>>>> log4j >>>>>> 1.1 (using separate ClassLoaders)? >>>>>> >>>>>> Thanks, >>>>>> Gili >>>>>> >>>>>> -- >>>>>> View this message in context: >>>>>> http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html >>>>>> >>>>>> Sent from the jigsaw-dev mailing list archive at Nabble.com. >>>>>> >>> > From mcculls at gmail.com Sun Nov 20 07:55:00 2011 From: mcculls at gmail.com (Stuart McCulloch) Date: Sun, 20 Nov 2011 15:55:00 +0000 Subject: Use-cases for version ranges? In-Reply-To: <4EC86F71.1000707@bbs.darktech.org> References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> Message-ID: <1B1D8A39-4A2A-4A62-BDC9-4D1BA99E2298@gmail.com> On 20 Nov 2011, at 03:09, cowwoc wrote: > I'd like to propose another possibility: the author of the dependency should tell *us* about version compatibility, not the other way around. For example: > > 1. The author of module A declares a dependency on module B version 1.2 (specific version). > 2. The author of module B publishes version 1.3. He declares that version 1.3 is compatible with 1.2 (meaning, the runtime system is allows to substitute version 1.3 for 1.2). Note that the author of module B could also do this by following a semantic versioning scheme [1] [2]. > The upside of this approach is that the author of B is in a better position to declare compatibility than the author of A. The author of A still only needs to test a single version. What do you think? You might want to read the Multiple Versions section from this entry on Peter's blog [3] where he raises the question of whether the importer or exporter is best placed to express compatibility. This later blog entry [4] about versioning is also worthwhile reading. PS. while most people use a single version for Maven dependencies, you can in fact use version ranges to restrict the resolution process [5]. Microsoft also decided to use ranges in their NuGet package system [6]. [1] http://semver.org/ [2] http://www.osgi.org/wiki/uploads/Links/SemanticVersioning.pdf [3] http://www.osgi.org/blog/2008/02/research-challenges-for-osgi.html [4] http://www.osgi.org/blog/2009/12/versions.html [5] http://docs.codehaus.org/display/MAVEN/Dependency+Mediation+and+Conflict+Resolution#DependencyMediationandConflictResolution-DependencyVersionRanges [6] http://docs.nuget.org/docs/reference/version-range-specification > Gili > > On 19/11/2011 1:59 AM, Neil Bartlett wrote: >> Gili, >> >> I didn't say anything about guarantees, and in this industry I have >> never heard of anybody providing a guarantees about the performance of >> their software, especially in the presence of external dependencies. >> >> Version ranges are a means of communicating expectations, and we >> provide both a lower and an upper bound because this is useful >> information. I expect my module will work with 1.2.14, and I expect it >> will not work with 2.0. If I were a provider of the API rather than a >> consumer then I would have a much narrower expectation, e.g. >> [1.2,1.3), and this would also be useful information to convey. >> >> Regards >> Neil >> >> On Sat, Nov 19, 2011 at 4:27 AM, cowwoc wrote: >>> Neil, >>> >>> I guess I don't understand why Jigsaw should work differently from Maven >>> on this point. I am expecting developers to specify specific versions that >>> they tested (point versions, not ranges) and end-users may override these >>> "recommendations" as they see fit. >>> >>> Where you see version range [1.2.14, 2.0) as a way of communicating "the >>> developer guarantees 1.2.14 but you may use newer versions up to 2.0 at your >>> own risk" I'd expect the developer to simply specify 1.2.14 and there should >>> be no limit on what version end-users may use if they so wish. >>> >>> Gili >>> >>> On 18/11/2011 2:23 AM, Neil Bartlett wrote: >>>> I noticed that I failed to address your point about Maven using point >>>> versions. >>>> >>>> Maven is a build tool. At build time we need to compile against a >>>> single specific version so that we have repeatable builds. In general >>>> we should build each module against the lowest version of the library >>>> that it can possibly use, and there are no major negative consequences >>>> of having several versions of a library at build time (except that >>>> Maven has to download a lot!). At runtime however we need to have the >>>> flexibility to substitute a single compatible version. >>>> >>>> Neil >>>> >>>> On Fri, Nov 18, 2011 at 7:10 AM, Neil Bartlett >>>> wrote: >>>>> Suppose as the developer of module A, I declare a dependency on log4j, >>>>> exactly version 1.0.0 because I have not tested against log4j 1.0.1, >>>>> 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* being >>>>> used with log4j version 1.0.1 even if this combinations is later >>>>> tested and proven to work by somebody else. In other words, testing is >>>>> important but it doesn't necessarily have to always be done by the >>>>> original developer of each module. >>>>> >>>>> On the other hand let's say I state my dependency using the following >>>>> range: [1.2.14, 2.0). This is OSGi syntax and I believe Jigsaw is >>>>> following it, and it simply means I accept version 1.2.14 up to but >>>>> not including 2.0. Anybody can see that I compiled and tested against >>>>> 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 etc. It >>>>> does not mean that I *guarantee* my module will work with log4j 1.3 >>>>> because that obviously depends on whether the log4j authors accept and >>>>> follow the common semantics of indicating backwards-incompatible >>>>> changes with a bump to the first version segment. >>>>> >>>>> The consequence of trying to lock down imports to a narrow range or >>>>> even a point version is that assembling an application becomes very >>>>> difficult, and we are forced to deploy many versions of common >>>>> libraries concurrently. This is non-optimal, though we can handle it >>>>> to some degree via per-module classloaders as in OSGi. >>>>> >>>>> Regards, >>>>> Neil >>>>> >>>>> On Thu, Nov 17, 2011 at 11:52 PM, cowwoc wrote: >>>>>> Can someone please explain why modules need to be able to specify >>>>>> version >>>>>> ranges for dependencies? I believe OSGI allows the specification of >>>>>> version >>>>>> ranges while Maven allows the specification of individual versions. >>>>>> >>>>>> The only thing that comes to mind is when module C depends on A and B, A >>>>>> depends on log4j 1.0, and B depends on log4j 1.1. What does C do? Is >>>>>> this >>>>>> the main use-case for version ranges? >>>>>> >>>>>> By the sound of it, this is a trust model where developers are told that >>>>>> log4j 1.x won't break compatibility so they depend on that range without >>>>>> actually testing against each version (newer versions may be released >>>>>> after >>>>>> their own software). I question whether such a mechanism is better or >>>>>> worse >>>>>> than depending on individual versions which may be overridden at a later >>>>>> time (a la Maven). On the one hand, you don't need to release a new >>>>>> version >>>>>> of the application each time a dependency is updated. On the other hand, >>>>>> no >>>>>> one is actually running tests to ensure that the versions are really >>>>>> compatible. >>>>>> >>>>>> Is there a way to get module A to see log4j 1.0 and module B to see >>>>>> log4j >>>>>> 1.1 (using separate ClassLoaders)? >>>>>> >>>>>> Thanks, >>>>>> Gili >>>>>> >>>>>> -- >>>>>> View this message in context: >>>>>> http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html >>>>>> Sent from the jigsaw-dev mailing list archive at Nabble.com. >>>>>> >>> > From mcculls at gmail.com Sun Nov 20 08:02:15 2011 From: mcculls at gmail.com (Stuart McCulloch) Date: Sun, 20 Nov 2011 16:02:15 +0000 Subject: Use-cases for version ranges? In-Reply-To: <4EC9221A.4020409@gmail.com> References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> <4EC9221A.4020409@gmail.com> Message-ID: <81B46329-4FA1-486B-AC26-64188E014F94@gmail.com> On 20 Nov 2011, at 15:51, Lukas Krecan wrote: > Hi, > it's obvious that the only one who has all the information is the person DEPLOYING the application (in Java EE it's Application Assembler role). In the enterprise word the application is maintained long after the implementation is finished. Imagine that there was a security problem in a library and a new version has been released. As a person taking care of the application I want just change this dependency without having to recompile the application. > > So having an automatic resolution of dependency versions is a nice feature, but we will always need a way how to specify the dependencies at the runtime. > > To reiterate, let's reuse the previous example, where 0 depends on A which in turn depends on B. > > 1. Author of B can not guarantee that B 1.3 is compatible with 1.2. Well, he can, but he can not be trusted. Some changes can cause problems that can not be envisaged by their author. The compatibility can be tested only by the users of the library! > 2. Author of A can not garantee that A will work with future versions of B. She can only test it with existing versions of B. She can say that A 3.0 is compatible with B [1.0,1.9]. > 3. Application 0 depends on A 3.0 and B 2.0 has been released. I want to use it, but new version of A has not been released yet. I need to override the A-> B dependency. > 4. All programmers have left the project 0 and few months later, B 2.1 has been released. It contains a critical security bug fix. System administrator needs to specify this dependency without having to recompile the application 0. He just needs to specify the new dependency. > > You see, the dependency resolution can not be fully automated. At the end, some dependencies have to be specified manually regardless the versioning mechanism. The cause is simple. All the components have different lifecycle and only the end-user (application 0) has all the data. Only him can test that the combination works as expected. Would you agree that the deployer's life can be made easier if people try to follow the same semantic versioning scheme? That at least gives the deployer some idea of the intent of the developer, which they can confirm by testing. It also means that they are less likely to be surprised by unexpected incompatibilities (I'm not saying this would never occur, just that it should be less likely). -- Cheers, Stuart > Regards > Lukas > > > On 11/20/2011 04:09 AM, cowwoc wrote: >> >> I'd like to propose another possibility: the author of the dependency should tell *us* about version compatibility, not the other way around. For example: >> >> 1. The author of module A declares a dependency on module B version 1.2 (specific version). >> 2. The author of module B publishes version 1.3. He declares that version 1.3 is compatible with 1.2 (meaning, the runtime system is allows to substitute version 1.3 for 1.2). >> >> The upside of this approach is that the author of B is in a better position to declare compatibility than the author of A. The author of A still only needs to test a single version. What do you think? >> >> Gili >> >> On 19/11/2011 1:59 AM, Neil Bartlett wrote: >>> Gili, >>> >>> I didn't say anything about guarantees, and in this industry I have >>> never heard of anybody providing a guarantees about the performance of >>> their software, especially in the presence of external dependencies. >>> >>> Version ranges are a means of communicating expectations, and we >>> provide both a lower and an upper bound because this is useful >>> information. I expect my module will work with 1.2.14, and I expect it >>> will not work with 2.0. If I were a provider of the API rather than a >>> consumer then I would have a much narrower expectation, e.g. >>> [1.2,1.3), and this would also be useful information to convey. >>> >>> Regards >>> Neil >>> >>> On Sat, Nov 19, 2011 at 4:27 AM, cowwoc wrote: >>>> Neil, >>>> >>>> I guess I don't understand why Jigsaw should work differently from Maven >>>> on this point. I am expecting developers to specify specific versions that >>>> they tested (point versions, not ranges) and end-users may override these >>>> "recommendations" as they see fit. >>>> >>>> Where you see version range [1.2.14, 2.0) as a way of communicating "the >>>> developer guarantees 1.2.14 but you may use newer versions up to 2.0 at your >>>> own risk" I'd expect the developer to simply specify 1.2.14 and there should >>>> be no limit on what version end-users may use if they so wish. >>>> >>>> Gili >>>> >>>> On 18/11/2011 2:23 AM, Neil Bartlett wrote: >>>>> I noticed that I failed to address your point about Maven using point >>>>> versions. >>>>> >>>>> Maven is a build tool. At build time we need to compile against a >>>>> single specific version so that we have repeatable builds. In general >>>>> we should build each module against the lowest version of the library >>>>> that it can possibly use, and there are no major negative consequences >>>>> of having several versions of a library at build time (except that >>>>> Maven has to download a lot!). At runtime however we need to have the >>>>> flexibility to substitute a single compatible version. >>>>> >>>>> Neil >>>>> >>>>> On Fri, Nov 18, 2011 at 7:10 AM, Neil Bartlett >>>>> wrote: >>>>>> Suppose as the developer of module A, I declare a dependency on log4j, >>>>>> exactly version 1.0.0 because I have not tested against log4j 1.0.1, >>>>>> 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* being >>>>>> used with log4j version 1.0.1 even if this combinations is later >>>>>> tested and proven to work by somebody else. In other words, testing is >>>>>> important but it doesn't necessarily have to always be done by the >>>>>> original developer of each module. >>>>>> >>>>>> On the other hand let's say I state my dependency using the following >>>>>> range: [1.2.14, 2.0). This is OSGi syntax and I believe Jigsaw is >>>>>> following it, and it simply means I accept version 1.2.14 up to but >>>>>> not including 2.0. Anybody can see that I compiled and tested against >>>>>> 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 etc. It >>>>>> does not mean that I *guarantee* my module will work with log4j 1.3 >>>>>> because that obviously depends on whether the log4j authors accept and >>>>>> follow the common semantics of indicating backwards-incompatible >>>>>> changes with a bump to the first version segment. >>>>>> >>>>>> The consequence of trying to lock down imports to a narrow range or >>>>>> even a point version is that assembling an application becomes very >>>>>> difficult, and we are forced to deploy many versions of common >>>>>> libraries concurrently. This is non-optimal, though we can handle it >>>>>> to some degree via per-module classloaders as in OSGi. >>>>>> >>>>>> Regards, >>>>>> Neil >>>>>> >>>>>> On Thu, Nov 17, 2011 at 11:52 PM, cowwoc wrote: >>>>>>> Can someone please explain why modules need to be able to specify >>>>>>> version >>>>>>> ranges for dependencies? I believe OSGI allows the specification of >>>>>>> version >>>>>>> ranges while Maven allows the specification of individual versions. >>>>>>> >>>>>>> The only thing that comes to mind is when module C depends on A and B, A >>>>>>> depends on log4j 1.0, and B depends on log4j 1.1. What does C do? Is >>>>>>> this >>>>>>> the main use-case for version ranges? >>>>>>> >>>>>>> By the sound of it, this is a trust model where developers are told that >>>>>>> log4j 1.x won't break compatibility so they depend on that range without >>>>>>> actually testing against each version (newer versions may be released >>>>>>> after >>>>>>> their own software). I question whether such a mechanism is better or >>>>>>> worse >>>>>>> than depending on individual versions which may be overridden at a later >>>>>>> time (a la Maven). On the one hand, you don't need to release a new >>>>>>> version >>>>>>> of the application each time a dependency is updated. On the other hand, >>>>>>> no >>>>>>> one is actually running tests to ensure that the versions are really >>>>>>> compatible. >>>>>>> >>>>>>> Is there a way to get module A to see log4j 1.0 and module B to see >>>>>>> log4j >>>>>>> 1.1 (using separate ClassLoaders)? >>>>>>> >>>>>>> Thanks, >>>>>>> Gili >>>>>>> >>>>>>> -- >>>>>>> View this message in context: >>>>>>> http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html >>>>>>> Sent from the jigsaw-dev mailing list archive at Nabble.com. >>>>>>> >>>> >> > From cowwoc at bbs.darktech.org Sun Nov 20 08:53:47 2011 From: cowwoc at bbs.darktech.org (cowwoc) Date: Sun, 20 Nov 2011 08:53:47 -0800 (PST) Subject: Use-cases for version ranges? In-Reply-To: <1B1D8A39-4A2A-4A62-BDC9-4D1BA99E2298@gmail.com> References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> <1B1D8A39-4A2A-4A62-BDC9-4D1BA99E2298@gmail.com> Message-ID: <1321808027023-5008437.post@n5.nabble.com> Stuart McCulloch wrote: > > On 20 Nov 2011, at 03:09, cowwoc wrote: > >> I'd like to propose another possibility: the author of the dependency >> should tell *us* about version compatibility, not the other way around. >> For example: >> >> 1. The author of module A declares a dependency on module B version 1.2 >> (specific version). >> 2. The author of module B publishes version 1.3. He declares that version >> 1.3 is compatible with 1.2 (meaning, the runtime system is allows to >> substitute version 1.3 for 1.2). > > Note that the author of module B could also do this by following a > semantic versioning scheme [1] [2]. > He can, but what I prefer about the mechanism I raised above is that you can achieve the same functionality without locking down the versioning scheme. Each publisher specifies a list of versions (not version ranges) the new release is compatible with. Thanks to transitivity the list of versions only needs to contain one version for the system will figure out the rest. Here is a simple example: I publish Log4j version 1.1 and specify it is compatible with version 1.0 I publish Log4j version 1.2 and specify it is compatible with version 1.1 I publish Log4j version 1.3 and specify it is compatible with version 1.2 I publish Log4j version 1.4 and specify it is compatible with version 1.3 Someone depends on log4j 1.1. By transitivity, this means the log4j author recommends using version 1.1, 1.2, 1.3 (notice this is a list of specific versions, not a version range). In order to address the issue brought up by Lukas ("Some changes can cause problems that can not be envisaged by their author") I'd like to add two caveats to complete the picture: 1. Authors are able to change the compatibility listings after-the-fact. Say I published log4j 1.3, declared it was compatible with 1.2 but found a compatibility problem after-the-fact. I'd have to update version 1.3's meta-data (remove its compatibility with 1.2) and version 1.4's meta-data (change its compatibility from 1.3 to 1.2). 2. Regardless of the author recommendations, end-users should be able to override the recommendations and use whatever version they wish. Stuart McCulloch wrote: > >> The upside of this approach is that the author of B is in a better >> position to declare compatibility than the author of A. The author of A >> still only needs to test a single version. What do you think? > > You might want to read the Multiple Versions section from this entry on > Peter's blog [3] where he raises the question of whether the importer or > exporter is best placed to express compatibility. This later blog entry > [4] about versioning is also worthwhile reading. > He seems to agree with my point "the exporter of a package knows exactly to which version he is compatible, being the responsible person for that package [...] The importer is almost always guessing." Stuart McCulloch wrote: > > PS. while most people use a single version for Maven dependencies, you can > in fact use version ranges to restrict the resolution process [5]. > Microsoft also decided to use ranges in their NuGet package system [6]. > > [1] http://semver.org/ > [2] http://www.osgi.org/wiki/uploads/Links/SemanticVersioning.pdf > [3] http://www.osgi.org/blog/2008/02/research-challenges-for-osgi.html > [4] http://www.osgi.org/blog/2009/12/versions.html > [5] > http://docs.codehaus.org/display/MAVEN/Dependency+Mediation+and+Conflict+Resolution#DependencyMediationandConflictResolution-DependencyVersionRanges > [6] http://docs.nuget.org/docs/reference/version-range-specification > Nice. I didn't know that. Thanks, Gili -- View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5008437.html Sent from the jigsaw-dev mailing list archive at Nabble.com. From cowwoc at bbs.darktech.org Sun Nov 20 09:01:25 2011 From: cowwoc at bbs.darktech.org (cowwoc) Date: Sun, 20 Nov 2011 09:01:25 -0800 (PST) Subject: Use-cases for version ranges? In-Reply-To: <4EC9221A.4020409@gmail.com> References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> <4EC9221A.4020409@gmail.com> Message-ID: <1321808485637-5008449.post@n5.nabble.com> Hi Lukas, Lukas Krecan wrote: > > 1. Author of B can not guarantee that B 1.3 is compatible with 1.2. > Well, he can, but he can not be trusted. Some changes can cause problems > that can not be envisaged by their author. The compatibility can be > tested only by the users of the library! > I disagree with this point. I posted a more detailed explanation of my proposal here: http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tt5002801.html#a5008437 I agree with you that users should be able to override the author recommendations at runtime. However, I still believe each module's author is best suited to declare version compatibility for his/her own module. Lukas Krecan wrote: > > 2. Author of A can not garantee that A will work with future versions of > B. She can only test it with existing versions of B. She can say that A > 3.0 is compatible with B [1.0,1.9]. > I think you're right that neither the author of A or B can declare with absolute confidence about version compatibility. Ideally the system should use the specific version of A that B depends on. I don't think anyone is debating that. I think version substitution comes into play when an application depends on B and C, each of which depend on different versions of A. In such a case, the system should provide users with reasonable defaults (recommendations) which end-users should be able to accept or override. Lukas Krecan wrote: > > 3. Application 0 depends on A 3.0 and B 2.0 has been released. I want to > use it, but new version of A has not been released yet. I need to > override the A-> B dependency. > 4. All programmers have left the project 0 and few months later, B 2.1 > has been released. It contains a critical security bug fix. System > administrator needs to specify this dependency without having to > recompile the application 0. He just needs to specify the new dependency. > > You see, the dependency resolution can not be fully automated. At the > end, some dependencies have to be specified manually regardless the > versioning mechanism. The cause is simple. All the components have > different lifecycle and only the end-user (application 0) has all the > data. Only him can test that the combination works as expected. > As stated above, I agree that ultimately end-users should be able to override any defaults/recommendations. I think end-users would benefit from getting recommendations from the authors of modules they depend upon which is why those developers should be able to specify version compatibility for their own versions (again, this is a recommendation not a hard limit). Gili -- View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5008449.html Sent from the jigsaw-dev mailing list archive at Nabble.com. From mcculls at gmail.com Sun Nov 20 09:10:47 2011 From: mcculls at gmail.com (Stuart McCulloch) Date: Sun, 20 Nov 2011 17:10:47 +0000 Subject: Use-cases for version ranges? In-Reply-To: <1321808027023-5008437.post@n5.nabble.com> References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> <1B1D8A39-4A2A-4A62-BDC9-4D1BA99E2298@gmail.com> <1321808027023-5008437.post@n5.nabble.com> Message-ID: <1263C926-FD8E-4B44-A54E-E8CB10D39F54@gmail.com> On 20 Nov 2011, at 16:53, cowwoc wrote: > Stuart McCulloch wrote: >> >> On 20 Nov 2011, at 03:09, cowwoc wrote: >> >>> I'd like to propose another possibility: the author of the dependency >>> should tell *us* about version compatibility, not the other way around. >>> For example: >>> >>> 1. The author of module A declares a dependency on module B version 1.2 >>> (specific version). >>> 2. The author of module B publishes version 1.3. He declares that version >>> 1.3 is compatible with 1.2 (meaning, the runtime system is allows to >>> substitute version 1.3 for 1.2). >> >> Note that the author of module B could also do this by following a >> semantic versioning scheme [1] [2]. >> > > He can, but what I prefer about the mechanism I raised above is that you can > achieve the same functionality without locking down the versioning scheme. > Each publisher specifies a list of versions (not version ranges) the new > release is compatible with. Thanks to transitivity the list of versions only > needs to contain one version for the system will figure out the rest. Here > is a simple example: > > I publish Log4j version 1.1 and specify it is compatible with version 1.0 > I publish Log4j version 1.2 and specify it is compatible with version 1.1 > I publish Log4j version 1.3 and specify it is compatible with version 1.2 > I publish Log4j version 1.4 and specify it is compatible with version 1.3 > > Someone depends on log4j 1.1. By transitivity, this means the log4j author > recommends using version 1.1, 1.2, 1.3 (notice this is a list of specific > versions, not a version range). > > In order to address the issue brought up by Lukas ("Some changes can cause > problems > that can not be envisaged by their author") I'd like to add two caveats to > complete the picture: > > 1. Authors are able to change the compatibility listings after-the-fact. Say > I published log4j 1.3, declared it was compatible with 1.2 but found a > compatibility problem after-the-fact. I'd have to update version 1.3's > meta-data (remove its compatibility with 1.2) and version 1.4's meta-data > (change its compatibility from 1.3 to 1.2). > > 2. Regardless of the author recommendations, end-users should be able to > override the recommendations and use whatever version they wish. I don't know, it just seems like a lot of work compared with following a simple standard. Just like if everyone chose their own symbols for mathematics; yes it would be flexible and people could share their symbol documents (and change them over time), but it makes interpreting the data and building systems on top much more work, especially when you have modules following radically different schemes. As a deployer I'd have to find the right compatibility listings and interpret them, whereas with a standard version scheme I can simply apply existing knowledge. Note that even just agreeing on the meaning of the major/minor/micro segments and allowing some customization at the qualifier level for specific tags (alpha, beta, etc.) would be a start. -- Cheers, Stuart > Stuart McCulloch wrote: >> >>> The upside of this approach is that the author of B is in a better >>> position to declare compatibility than the author of A. The author of A >>> still only needs to test a single version. What do you think? >> >> You might want to read the Multiple Versions section from this entry on >> Peter's blog [3] where he raises the question of whether the importer or >> exporter is best placed to express compatibility. This later blog entry >> [4] about versioning is also worthwhile reading. >> > > He seems to agree with my point "the exporter of a package knows exactly to > which version he is compatible, being the responsible person for that > package [...] The importer is almost always guessing." > > > Stuart McCulloch wrote: >> >> PS. while most people use a single version for Maven dependencies, you can >> in fact use version ranges to restrict the resolution process [5]. >> Microsoft also decided to use ranges in their NuGet package system [6]. >> >> [1] http://semver.org/ >> [2] http://www.osgi.org/wiki/uploads/Links/SemanticVersioning.pdf >> [3] http://www.osgi.org/blog/2008/02/research-challenges-for-osgi.html >> [4] http://www.osgi.org/blog/2009/12/versions.html >> [5] >> http://docs.codehaus.org/display/MAVEN/Dependency+Mediation+and+Conflict+Resolution#DependencyMediationandConflictResolution-DependencyVersionRanges >> [6] http://docs.nuget.org/docs/reference/version-range-specification >> > > Nice. I didn't know that. > > Thanks, > Gili > > -- > View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5008437.html > Sent from the jigsaw-dev mailing list archive at Nabble.com. From mcculls at gmail.com Sun Nov 20 09:15:39 2011 From: mcculls at gmail.com (Stuart McCulloch) Date: Sun, 20 Nov 2011 17:15:39 +0000 Subject: Use-cases for version ranges? In-Reply-To: <1321808027023-5008437.post@n5.nabble.com> References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> <1B1D8A39-4A2A-4A62-BDC9-4D1BA99E2298@gmail.com> <1321808027023-5008437.post@n5.nabble.com> Message-ID: <5619B1EE-EE8E-4557-BA5F-162A66D081E7@gmail.com> On 20 Nov 2011, at 16:53, cowwoc wrote: > Stuart McCulloch wrote: >> >> On 20 Nov 2011, at 03:09, cowwoc wrote: >>> The upside of this approach is that the author of B is in a better >>> position to declare compatibility than the author of A. The author of A >>> still only needs to test a single version. What do you think? >> >> You might want to read the Multiple Versions section from this entry on >> Peter's blog [3] where he raises the question of whether the importer or >> exporter is best placed to express compatibility. This later blog entry >> [4] about versioning is also worthwhile reading. >> > > He seems to agree with my point "the exporter of a package knows exactly to > which version he is compatible, being the responsible person for that > package [...] The importer is almost always guessing." Yes it wasn't meant as a counter-point, just an FYI that there's some interesting discussions around this topic in the OSGi archives. > Stuart McCulloch wrote: >> >> PS. while most people use a single version for Maven dependencies, you can >> in fact use version ranges to restrict the resolution process [5]. >> Microsoft also decided to use ranges in their NuGet package system [6]. >> >> [1] http://semver.org/ >> [2] http://www.osgi.org/wiki/uploads/Links/SemanticVersioning.pdf >> [3] http://www.osgi.org/blog/2008/02/research-challenges-for-osgi.html >> [4] http://www.osgi.org/blog/2009/12/versions.html >> [5] >> http://docs.codehaus.org/display/MAVEN/Dependency+Mediation+and+Conflict+Resolution#DependencyMediationandConflictResolution-DependencyVersionRanges >> [6] http://docs.nuget.org/docs/reference/version-range-specification >> > > Nice. I didn't know that. > > Thanks, > Gili > > -- > View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5008437.html > Sent from the jigsaw-dev mailing list archive at Nabble.com. From brian at pontarelli.com Sun Nov 20 11:08:01 2011 From: brian at pontarelli.com (Brian Pontarelli) Date: Sun, 20 Nov 2011 12:08:01 -0700 Subject: Use-cases for version ranges? In-Reply-To: <4EC86F71.1000707@bbs.darktech.org> References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> Message-ID: <12C096DE-2B95-44A9-A89A-8F3C428B9EC4@pontarelli.com> That's what I was referring to when I brought up "version compatibility" and it is way that Savant handles this. Only the author should be responsible for knowing how their code is compatible. -bp On Nov 19, 2011, at 8:09 PM, cowwoc wrote: > > I'd like to propose another possibility: the author of the dependency should tell *us* about version compatibility, not the other way around. For example: > > 1. The author of module A declares a dependency on module B version 1.2 (specific version). > 2. The author of module B publishes version 1.3. He declares that version 1.3 is compatible with 1.2 (meaning, the runtime system is allows to substitute version 1.3 for 1.2). > > The upside of this approach is that the author of B is in a better position to declare compatibility than the author of A. The author of A still only needs to test a single version. What do you think? > > Gili > > On 19/11/2011 1:59 AM, Neil Bartlett wrote: >> Gili, >> >> I didn't say anything about guarantees, and in this industry I have >> never heard of anybody providing a guarantees about the performance of >> their software, especially in the presence of external dependencies. >> >> Version ranges are a means of communicating expectations, and we >> provide both a lower and an upper bound because this is useful >> information. I expect my module will work with 1.2.14, and I expect it >> will not work with 2.0. If I were a provider of the API rather than a >> consumer then I would have a much narrower expectation, e.g. >> [1.2,1.3), and this would also be useful information to convey. >> >> Regards >> Neil >> >> On Sat, Nov 19, 2011 at 4:27 AM, cowwoc wrote: >>> Neil, >>> >>> I guess I don't understand why Jigsaw should work differently from Maven >>> on this point. I am expecting developers to specify specific versions that >>> they tested (point versions, not ranges) and end-users may override these >>> "recommendations" as they see fit. >>> >>> Where you see version range [1.2.14, 2.0) as a way of communicating "the >>> developer guarantees 1.2.14 but you may use newer versions up to 2.0 at your >>> own risk" I'd expect the developer to simply specify 1.2.14 and there should >>> be no limit on what version end-users may use if they so wish. >>> >>> Gili >>> >>> On 18/11/2011 2:23 AM, Neil Bartlett wrote: >>>> I noticed that I failed to address your point about Maven using point >>>> versions. >>>> >>>> Maven is a build tool. At build time we need to compile against a >>>> single specific version so that we have repeatable builds. In general >>>> we should build each module against the lowest version of the library >>>> that it can possibly use, and there are no major negative consequences >>>> of having several versions of a library at build time (except that >>>> Maven has to download a lot!). At runtime however we need to have the >>>> flexibility to substitute a single compatible version. >>>> >>>> Neil >>>> >>>> On Fri, Nov 18, 2011 at 7:10 AM, Neil Bartlett >>>> wrote: >>>>> Suppose as the developer of module A, I declare a dependency on log4j, >>>>> exactly version 1.0.0 because I have not tested against log4j 1.0.1, >>>>> 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* being >>>>> used with log4j version 1.0.1 even if this combinations is later >>>>> tested and proven to work by somebody else. In other words, testing is >>>>> important but it doesn't necessarily have to always be done by the >>>>> original developer of each module. >>>>> >>>>> On the other hand let's say I state my dependency using the following >>>>> range: [1.2.14, 2.0). This is OSGi syntax and I believe Jigsaw is >>>>> following it, and it simply means I accept version 1.2.14 up to but >>>>> not including 2.0. Anybody can see that I compiled and tested against >>>>> 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 etc. It >>>>> does not mean that I *guarantee* my module will work with log4j 1.3 >>>>> because that obviously depends on whether the log4j authors accept and >>>>> follow the common semantics of indicating backwards-incompatible >>>>> changes with a bump to the first version segment. >>>>> >>>>> The consequence of trying to lock down imports to a narrow range or >>>>> even a point version is that assembling an application becomes very >>>>> difficult, and we are forced to deploy many versions of common >>>>> libraries concurrently. This is non-optimal, though we can handle it >>>>> to some degree via per-module classloaders as in OSGi. >>>>> >>>>> Regards, >>>>> Neil >>>>> >>>>> On Thu, Nov 17, 2011 at 11:52 PM, cowwoc wrote: >>>>>> Can someone please explain why modules need to be able to specify >>>>>> version >>>>>> ranges for dependencies? I believe OSGI allows the specification of >>>>>> version >>>>>> ranges while Maven allows the specification of individual versions. >>>>>> >>>>>> The only thing that comes to mind is when module C depends on A and B, A >>>>>> depends on log4j 1.0, and B depends on log4j 1.1. What does C do? Is >>>>>> this >>>>>> the main use-case for version ranges? >>>>>> >>>>>> By the sound of it, this is a trust model where developers are told that >>>>>> log4j 1.x won't break compatibility so they depend on that range without >>>>>> actually testing against each version (newer versions may be released >>>>>> after >>>>>> their own software). I question whether such a mechanism is better or >>>>>> worse >>>>>> than depending on individual versions which may be overridden at a later >>>>>> time (a la Maven). On the one hand, you don't need to release a new >>>>>> version >>>>>> of the application each time a dependency is updated. On the other hand, >>>>>> no >>>>>> one is actually running tests to ensure that the versions are really >>>>>> compatible. >>>>>> >>>>>> Is there a way to get module A to see log4j 1.0 and module B to see >>>>>> log4j >>>>>> 1.1 (using separate ClassLoaders)? >>>>>> >>>>>> Thanks, >>>>>> Gili >>>>>> >>>>>> -- >>>>>> View this message in context: >>>>>> http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html >>>>>> Sent from the jigsaw-dev mailing list archive at Nabble.com. >>>>>> >>> > From brian at pontarelli.com Sun Nov 20 11:12:24 2011 From: brian at pontarelli.com (Brian Pontarelli) Date: Sun, 20 Nov 2011 12:12:24 -0700 Subject: Use-cases for version ranges? In-Reply-To: References: <1321573938450-5002801.post@n5.nabble.com><4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> Message-ID: <07039A0B-1E18-4432-BCCD-2E668FD73292@pontarelli.com> There are tools that can determine if two JARs are compatible using the Java specification definition of runtime compatibility (I can google and find them again if you need). There are also tools that provide build time version compatibility checking (Savant, Ivy). The hard part is when compatibility is broken at a pre and post condition level rather than the contract level. For example, the method signature didn't change, but the method stopped throwing some exception (NPE for example). -bp On Nov 20, 2011, at 5:25 AM, Stephen McConnell wrote: > > I could say that the question is academic - but here is the thing - its not even academic. > > We talk about versions and yet the platform has no computational mechanisms to flag version violation. At least an academic discussion would bring in grounded metrics. In reality - the discussion here about V1 versus V2 and that warm fuzzy feeling when your are just a n.m from an n+1 and it's a waste of breath (and that comment goes to all sides of the debate). > > Give me an endorsed tool that shows me that module [name]-[build] is computationally backward compatible to a different [name]-[build] and then, only them will this discussion have reasonable grounds to assert any authority on the subject of version compatibility (past or future). Without that tool - versioning is a unqualified statement of trust. And the evidence suggests that the threshold of trust delivery on actual computational compliance is really really thin. > > Cheers, Steve. > > > -----Original Message----- From: cowwoc > Sent: Sunday, November 20, 2011 1:39 PM > To: jigsaw-dev at openjdk.java.net > Subject: Re: Use-cases for version ranges? > > > I'd like to propose another possibility: the author of the > dependency should tell *us* about version compatibility, not the other > way around. For example: > > 1. The author of module A declares a dependency on module B version 1.2 > (specific version). > 2. The author of module B publishes version 1.3. He declares that > version 1.3 is compatible with 1.2 (meaning, the runtime system is > allows to substitute version 1.3 for 1.2). > > The upside of this approach is that the author of B is in a better > position to declare compatibility than the author of A. The author of A > still only needs to test a single version. What do you think? > > Gili > > On 19/11/2011 1:59 AM, Neil Bartlett wrote: >> Gili, >> >> I didn't say anything about guarantees, and in this industry I have >> never heard of anybody providing a guarantees about the performance of >> their software, especially in the presence of external dependencies. >> >> Version ranges are a means of communicating expectations, and we >> provide both a lower and an upper bound because this is useful >> information. I expect my module will work with 1.2.14, and I expect it >> will not work with 2.0. If I were a provider of the API rather than a >> consumer then I would have a much narrower expectation, e.g. >> [1.2,1.3), and this would also be useful information to convey. >> >> Regards >> Neil >> >> On Sat, Nov 19, 2011 at 4:27 AM, cowwoc wrote: >>> Neil, >>> >>> I guess I don't understand why Jigsaw should work differently from Maven >>> on this point. I am expecting developers to specify specific versions that >>> they tested (point versions, not ranges) and end-users may override these >>> "recommendations" as they see fit. >>> >>> Where you see version range [1.2.14, 2.0) as a way of communicating "the >>> developer guarantees 1.2.14 but you may use newer versions up to 2.0 at your >>> own risk" I'd expect the developer to simply specify 1.2.14 and there should >>> be no limit on what version end-users may use if they so wish. >>> >>> Gili >>> >>> On 18/11/2011 2:23 AM, Neil Bartlett wrote: >>>> I noticed that I failed to address your point about Maven using point >>>> versions. >>>> >>>> Maven is a build tool. At build time we need to compile against a >>>> single specific version so that we have repeatable builds. In general >>>> we should build each module against the lowest version of the library >>>> that it can possibly use, and there are no major negative consequences >>>> of having several versions of a library at build time (except that >>>> Maven has to download a lot!). At runtime however we need to have the >>>> flexibility to substitute a single compatible version. >>>> >>>> Neil >>>> >>>> On Fri, Nov 18, 2011 at 7:10 AM, Neil Bartlett >>>> wrote: >>>>> Suppose as the developer of module A, I declare a dependency on log4j, >>>>> exactly version 1.0.0 because I have not tested against log4j 1.0.1, >>>>> 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* being >>>>> used with log4j version 1.0.1 even if this combinations is later >>>>> tested and proven to work by somebody else. In other words, testing is >>>>> important but it doesn't necessarily have to always be done by the >>>>> original developer of each module. >>>>> >>>>> On the other hand let's say I state my dependency using the following >>>>> range: [1.2.14, 2.0). This is OSGi syntax and I believe Jigsaw is >>>>> following it, and it simply means I accept version 1.2.14 up to but >>>>> not including 2.0. Anybody can see that I compiled and tested against >>>>> 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 etc. It >>>>> does not mean that I *guarantee* my module will work with log4j 1.3 >>>>> because that obviously depends on whether the log4j authors accept and >>>>> follow the common semantics of indicating backwards-incompatible >>>>> changes with a bump to the first version segment. >>>>> >>>>> The consequence of trying to lock down imports to a narrow range or >>>>> even a point version is that assembling an application becomes very >>>>> difficult, and we are forced to deploy many versions of common >>>>> libraries concurrently. This is non-optimal, though we can handle it >>>>> to some degree via per-module classloaders as in OSGi. >>>>> >>>>> Regards, >>>>> Neil >>>>> >>>>> On Thu, Nov 17, 2011 at 11:52 PM, cowwoc wrote: >>>>>> Can someone please explain why modules need to be able to specify >>>>>> version >>>>>> ranges for dependencies? I believe OSGI allows the specification of >>>>>> version >>>>>> ranges while Maven allows the specification of individual versions. >>>>>> >>>>>> The only thing that comes to mind is when module C depends on A and B, A >>>>>> depends on log4j 1.0, and B depends on log4j 1.1. What does C do? Is >>>>>> this >>>>>> the main use-case for version ranges? >>>>>> >>>>>> By the sound of it, this is a trust model where developers are told that >>>>>> log4j 1.x won't break compatibility so they depend on that range without >>>>>> actually testing against each version (newer versions may be released >>>>>> after >>>>>> their own software). I question whether such a mechanism is better or >>>>>> worse >>>>>> than depending on individual versions which may be overridden at a later >>>>>> time (a la Maven). On the one hand, you don't need to release a new >>>>>> version >>>>>> of the application each time a dependency is updated. On the other hand, >>>>>> no >>>>>> one is actually running tests to ensure that the versions are really >>>>>> compatible. >>>>>> >>>>>> Is there a way to get module A to see log4j 1.0 and module B to see >>>>>> log4j >>>>>> 1.1 (using separate ClassLoaders)? >>>>>> >>>>>> Thanks, >>>>>> Gili >>>>>> >>>>>> -- >>>>>> View this message in context: >>>>>> http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html >>>>>> Sent from the jigsaw-dev mailing list archive at Nabble.com. >>>>>> >>> > From brian at pontarelli.com Sun Nov 20 11:18:14 2011 From: brian at pontarelli.com (Brian Pontarelli) Date: Sun, 20 Nov 2011 12:18:14 -0700 Subject: Use-cases for version ranges? In-Reply-To: References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> Message-ID: <0072F87A-48C9-421B-A1E9-D9DE8AB923CB@pontarelli.com> One other way I look at version ranges is that it is like guessing the future. If I publish something that contains a version range with versions that haven't been released yet, I'm essentially guessing that another developer will maintain compatibility according to my version range. For example, if I specify this: log4j (1.0, 2.0] I'm essentially just guessing that the Log4j team will be cool and maintain compatibility from the current version of 1.2.16 all the way through 2.0. This really isn't my job or a good idea in my opinion. -bp On Nov 18, 2011, at 11:59 PM, Neil Bartlett wrote: > Gili, > > I didn't say anything about guarantees, and in this industry I have > never heard of anybody providing a guarantees about the performance of > their software, especially in the presence of external dependencies. > > Version ranges are a means of communicating expectations, and we > provide both a lower and an upper bound because this is useful > information. I expect my module will work with 1.2.14, and I expect it > will not work with 2.0. If I were a provider of the API rather than a > consumer then I would have a much narrower expectation, e.g. > [1.2,1.3), and this would also be useful information to convey. > > Regards > Neil > > On Sat, Nov 19, 2011 at 4:27 AM, cowwoc wrote: >> Neil, >> >> I guess I don't understand why Jigsaw should work differently from Maven >> on this point. I am expecting developers to specify specific versions that >> they tested (point versions, not ranges) and end-users may override these >> "recommendations" as they see fit. >> >> Where you see version range [1.2.14, 2.0) as a way of communicating "the >> developer guarantees 1.2.14 but you may use newer versions up to 2.0 at your >> own risk" I'd expect the developer to simply specify 1.2.14 and there should >> be no limit on what version end-users may use if they so wish. >> >> Gili >> >> On 18/11/2011 2:23 AM, Neil Bartlett wrote: >>> >>> I noticed that I failed to address your point about Maven using point >>> versions. >>> >>> Maven is a build tool. At build time we need to compile against a >>> single specific version so that we have repeatable builds. In general >>> we should build each module against the lowest version of the library >>> that it can possibly use, and there are no major negative consequences >>> of having several versions of a library at build time (except that >>> Maven has to download a lot!). At runtime however we need to have the >>> flexibility to substitute a single compatible version. >>> >>> Neil >>> >>> On Fri, Nov 18, 2011 at 7:10 AM, Neil Bartlett >>> wrote: >>>> >>>> Suppose as the developer of module A, I declare a dependency on log4j, >>>> exactly version 1.0.0 because I have not tested against log4j 1.0.1, >>>> 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* being >>>> used with log4j version 1.0.1 even if this combinations is later >>>> tested and proven to work by somebody else. In other words, testing is >>>> important but it doesn't necessarily have to always be done by the >>>> original developer of each module. >>>> >>>> On the other hand let's say I state my dependency using the following >>>> range: [1.2.14, 2.0). This is OSGi syntax and I believe Jigsaw is >>>> following it, and it simply means I accept version 1.2.14 up to but >>>> not including 2.0. Anybody can see that I compiled and tested against >>>> 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 etc. It >>>> does not mean that I *guarantee* my module will work with log4j 1.3 >>>> because that obviously depends on whether the log4j authors accept and >>>> follow the common semantics of indicating backwards-incompatible >>>> changes with a bump to the first version segment. >>>> >>>> The consequence of trying to lock down imports to a narrow range or >>>> even a point version is that assembling an application becomes very >>>> difficult, and we are forced to deploy many versions of common >>>> libraries concurrently. This is non-optimal, though we can handle it >>>> to some degree via per-module classloaders as in OSGi. >>>> >>>> Regards, >>>> Neil >>>> >>>> On Thu, Nov 17, 2011 at 11:52 PM, cowwoc wrote: >>>>> >>>>> Can someone please explain why modules need to be able to specify >>>>> version >>>>> ranges for dependencies? I believe OSGI allows the specification of >>>>> version >>>>> ranges while Maven allows the specification of individual versions. >>>>> >>>>> The only thing that comes to mind is when module C depends on A and B, A >>>>> depends on log4j 1.0, and B depends on log4j 1.1. What does C do? Is >>>>> this >>>>> the main use-case for version ranges? >>>>> >>>>> By the sound of it, this is a trust model where developers are told that >>>>> log4j 1.x won't break compatibility so they depend on that range without >>>>> actually testing against each version (newer versions may be released >>>>> after >>>>> their own software). I question whether such a mechanism is better or >>>>> worse >>>>> than depending on individual versions which may be overridden at a later >>>>> time (a la Maven). On the one hand, you don't need to release a new >>>>> version >>>>> of the application each time a dependency is updated. On the other hand, >>>>> no >>>>> one is actually running tests to ensure that the versions are really >>>>> compatible. >>>>> >>>>> Is there a way to get module A to see log4j 1.0 and module B to see >>>>> log4j >>>>> 1.1 (using separate ClassLoaders)? >>>>> >>>>> Thanks, >>>>> Gili >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html >>>>> Sent from the jigsaw-dev mailing list archive at Nabble.com. >>>>> >> >> From nicolas.lalevee at hibnet.org Sun Nov 20 13:10:25 2011 From: nicolas.lalevee at hibnet.org (=?iso-8859-1?Q?Nicolas_Lalev=E9e?=) Date: Sun, 20 Nov 2011 22:10:25 +0100 Subject: Use-cases for version ranges? In-Reply-To: <0072F87A-48C9-421B-A1E9-D9DE8AB923CB@pontarelli.com> References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <0072F87A-48C9-421B-A1E9-D9DE8AB923CB@pontarelli.com> Message-ID: Le 20 nov. 2011 ? 20:18, Brian Pontarelli a ?crit : > One other way I look at version ranges is that it is like guessing the future. If I publish something that contains a version range with versions that haven't been released yet, I'm essentially guessing that another developer will maintain compatibility according to my version range. > > For example, if I specify this: > > log4j (1.0, 2.0] > > I'm essentially just guessing that the Log4j team will be cool and maintain compatibility from the current version of 1.2.16 all the way through 2.0. This really isn't my job or a good idea in my opinion. Since there is no declared version scheme, and further more no semantic on each segment, I wouldn't use version range either. But then again, if you make the distinction between the API and the implementation, it is safe to use version range, at least on the API level. In the OSGi world, you can have different version between the one declared on the package and the one declared on the bundle. Whereas the log4j team would bump their version of their bundle the way they want, a strong semantic on the version on the packages (the API) would force them to correctly bump it so they will not annoy any "range"-dependent project with incompatible version. The bundle (implementation) version would be the "marketing" version and the package (API) version would strongly state the backward compatibility. OSGi has a quite limited version scheme, which I think is very good with strong semantic on the package. But on the bundle I think it is a mistake, there's some release cycle which doesn't fit into a 3 segments version scheme. Nicolas > > -bp > > On Nov 18, 2011, at 11:59 PM, Neil Bartlett wrote: > >> Gili, >> >> I didn't say anything about guarantees, and in this industry I have >> never heard of anybody providing a guarantees about the performance of >> their software, especially in the presence of external dependencies. >> >> Version ranges are a means of communicating expectations, and we >> provide both a lower and an upper bound because this is useful >> information. I expect my module will work with 1.2.14, and I expect it >> will not work with 2.0. If I were a provider of the API rather than a >> consumer then I would have a much narrower expectation, e.g. >> [1.2,1.3), and this would also be useful information to convey. >> >> Regards >> Neil >> >> On Sat, Nov 19, 2011 at 4:27 AM, cowwoc wrote: >>> Neil, >>> >>> I guess I don't understand why Jigsaw should work differently from Maven >>> on this point. I am expecting developers to specify specific versions that >>> they tested (point versions, not ranges) and end-users may override these >>> "recommendations" as they see fit. >>> >>> Where you see version range [1.2.14, 2.0) as a way of communicating "the >>> developer guarantees 1.2.14 but you may use newer versions up to 2.0 at your >>> own risk" I'd expect the developer to simply specify 1.2.14 and there should >>> be no limit on what version end-users may use if they so wish. >>> >>> Gili >>> >>> On 18/11/2011 2:23 AM, Neil Bartlett wrote: >>>> >>>> I noticed that I failed to address your point about Maven using point >>>> versions. >>>> >>>> Maven is a build tool. At build time we need to compile against a >>>> single specific version so that we have repeatable builds. In general >>>> we should build each module against the lowest version of the library >>>> that it can possibly use, and there are no major negative consequences >>>> of having several versions of a library at build time (except that >>>> Maven has to download a lot!). At runtime however we need to have the >>>> flexibility to substitute a single compatible version. >>>> >>>> Neil >>>> >>>> On Fri, Nov 18, 2011 at 7:10 AM, Neil Bartlett >>>> wrote: >>>>> >>>>> Suppose as the developer of module A, I declare a dependency on log4j, >>>>> exactly version 1.0.0 because I have not tested against log4j 1.0.1, >>>>> 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* being >>>>> used with log4j version 1.0.1 even if this combinations is later >>>>> tested and proven to work by somebody else. In other words, testing is >>>>> important but it doesn't necessarily have to always be done by the >>>>> original developer of each module. >>>>> >>>>> On the other hand let's say I state my dependency using the following >>>>> range: [1.2.14, 2.0). This is OSGi syntax and I believe Jigsaw is >>>>> following it, and it simply means I accept version 1.2.14 up to but >>>>> not including 2.0. Anybody can see that I compiled and tested against >>>>> 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 etc. It >>>>> does not mean that I *guarantee* my module will work with log4j 1.3 >>>>> because that obviously depends on whether the log4j authors accept and >>>>> follow the common semantics of indicating backwards-incompatible >>>>> changes with a bump to the first version segment. >>>>> >>>>> The consequence of trying to lock down imports to a narrow range or >>>>> even a point version is that assembling an application becomes very >>>>> difficult, and we are forced to deploy many versions of common >>>>> libraries concurrently. This is non-optimal, though we can handle it >>>>> to some degree via per-module classloaders as in OSGi. >>>>> >>>>> Regards, >>>>> Neil >>>>> >>>>> On Thu, Nov 17, 2011 at 11:52 PM, cowwoc wrote: >>>>>> >>>>>> Can someone please explain why modules need to be able to specify >>>>>> version >>>>>> ranges for dependencies? I believe OSGI allows the specification of >>>>>> version >>>>>> ranges while Maven allows the specification of individual versions. >>>>>> >>>>>> The only thing that comes to mind is when module C depends on A and B, A >>>>>> depends on log4j 1.0, and B depends on log4j 1.1. What does C do? Is >>>>>> this >>>>>> the main use-case for version ranges? >>>>>> >>>>>> By the sound of it, this is a trust model where developers are told that >>>>>> log4j 1.x won't break compatibility so they depend on that range without >>>>>> actually testing against each version (newer versions may be released >>>>>> after >>>>>> their own software). I question whether such a mechanism is better or >>>>>> worse >>>>>> than depending on individual versions which may be overridden at a later >>>>>> time (a la Maven). On the one hand, you don't need to release a new >>>>>> version >>>>>> of the application each time a dependency is updated. On the other hand, >>>>>> no >>>>>> one is actually running tests to ensure that the versions are really >>>>>> compatible. >>>>>> >>>>>> Is there a way to get module A to see log4j 1.0 and module B to see >>>>>> log4j >>>>>> 1.1 (using separate ClassLoaders)? >>>>>> >>>>>> Thanks, >>>>>> Gili >>>>>> >>>>>> -- >>>>>> View this message in context: >>>>>> http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html >>>>>> Sent from the jigsaw-dev mailing list archive at Nabble.com. >>>>>> >>> >>> > From eric at tibco.com Mon Nov 21 02:42:22 2011 From: eric at tibco.com (Eric Johnson) Date: Mon, 21 Nov 2011 11:42:22 +0100 Subject: Use-cases for version ranges? In-Reply-To: <4EC86F71.1000707@bbs.darktech.org> References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> Message-ID: <4ECA2B0E.40809@tibco.com> Maybe I missed it, but I want to pick on one particular point, around which I've had discussions within my company. On 11/20/11 4:09 AM, cowwoc wrote: > I'd like to propose another possibility: the author of the > dependency should tell *us* about version compatibility, not the other > way around. For example: "compatibility" as defined by whom? Here's my description for some of the common compatibility notions. "Classpath compatibility" - given a flat classpath, everything must be backwards compatible. You never add methods to existing interfaces. Adding methods to non-final public classes is likely inappropriate, but tolerable (name collision with a subclass will lead to unexpected behavior). "JRE compatibility" - All existing methods still there. This is much weaker than Classpath compatible: my favorite example is the JDBC APIs, where the JDK has consistently added methods to the interfaces over the years, rather than extending the base interfaces. This means that anyone that ever does anything non-trivial with the JDBC API has to fix their code with every single JDK release. Even though it is allegedly compatible. "Eclipse API Tools compatibility" - using annotations, under certain circumstances, I get to add methods to classes and interfaces. Otherwise, no. Versions specified at the module level. "OSGi Compatibility" - applies at the package (well, or module, if you wish) level. As proposed (sorry, I need a reference here, but don't have it handy), a minor bump means you are adding methods to interfaces, and a major bump means backwardly incompatible. Unlike Eclipse API Tools, no annotations to cheat around the edges. My own notion - combine "classpath compatibility" with a notion of version #, so that backwards incompatibility can be flagged. All other aspects of version # are strictly speaking an indication of scope of change (bug fix vs. additional functionality) ------ I personally argue strongly in favor of the "classpath compatibility" notion. That is, Java already has the ability to express a new version of an interface that adds a new method by simply having the new interface extend the old one. No fancy version schemes required, and the granularity works all the way down to the individual class/interface. If you do combine this with a version scheme, you can indicate backwards compatibility, and scope of change. Trouble is, the Java community doesn't seem to like the idea of a new named interface with the additional of even one method to an interface. But is that perhaps just because the JDK has acculturated people to the idea that they don't need to do that? It seems to me that any version scheme that attempts to cover up this problem will likely just add to the confusion, rather than resolve the problem. We either have to stick a stake in the ground and declare what we mean by backward compatible, or admit that there isn't a single notion, and therefore version ranges just mean what people want them to mean. -Eric. From Alan.Bateman at oracle.com Mon Nov 21 05:17:10 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 21 Nov 2011 13:17:10 +0000 Subject: Use-cases for version ranges? In-Reply-To: <1321573938450-5002801.post@n5.nabble.com> References: <1321573938450-5002801.post@n5.nabble.com> Message-ID: <4ECA4F56.7080803@oracle.com> On 17/11/2011 23:52, cowwoc wrote: > : > > Is there a way to get module A to see log4j 1.0 and module B to see log4j > 1.1 (using separate ClassLoaders)? > > With Jigsaw then the conflict will be caught when you attempt to install/configure a module with an entry point and that module ultimately requires more than one version of a module. Such a scenario is considered a "bug", possibly caused by overly restrictive version constraints. The one scenario where it may make sense is the container case, say a test harness or a web container that requires one version of a module and the test or application running in the container requires a different version. This isn't implemented in Jigsaw (not yet anyway) but is a scenario that is called out in the requirements document [1]. -Alan. [1] http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#simultaneous-multiple-versions From brian at pontarelli.com Mon Nov 21 05:34:10 2011 From: brian at pontarelli.com (Brian Pontarelli) Date: Mon, 21 Nov 2011 06:34:10 -0700 Subject: Use-cases for version ranges? In-Reply-To: <4ECA4F56.7080803@oracle.com> References: <1321573938450-5002801.post@n5.nabble.com> <4ECA4F56.7080803@oracle.com> Message-ID: <5B1DA05C-3705-42F2-80E9-A806CAB3331C@pontarelli.com> On Nov 21, 2011, at 6:17 AM, Alan Bateman wrote: > On 17/11/2011 23:52, cowwoc wrote: >> : >> >> Is there a way to get module A to see log4j 1.0 and module B to see log4j >> 1.1 (using separate ClassLoaders)? >> >> > With Jigsaw then the conflict will be caught when you attempt to install/configure a module with an entry point and that module ultimately requires more than one version of a module. Such a scenario is considered a "bug", possibly caused by overly restrictive version constraints. The one scenario where it may make sense is the container case, say a test harness or a web container that requires one version of a module and the test or application running in the container requires a different version. This isn't implemented in Jigsaw (not yet anyway) but is a scenario that is called out in the requirements document [1]. I was referring to this case when talking about module isolation. If module A depends on log4j 1.0 and module B depends on log4j 1.1, they should not conflict if the modules complete isolate that dependency from all other modules. However, if module A or B has a public interface like this one, it breaks down: public interface ModuleStuff { Logger getModuleLogger(); } This is where version compatibility comes into play. The build and runtime dependency management system would select 1.1 as long as the log4j team has certified that it is compatible with 1.0. In Savant we call this "minor" compatibility. Savant uses a well know versioning scheme and compatibility system. -bp From david.lloyd at redhat.com Mon Nov 21 07:15:17 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 21 Nov 2011 09:15:17 -0600 Subject: Use-cases for version ranges? In-Reply-To: References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> Message-ID: <4ECA6B05.8020001@redhat.com> I'm tired of hearing pie-in-the sky, "intro to OOP" wishful crap like this. Mission one should be supporting *existing* code. The fact is that people do *not* always split their API from their implementation, and APIs almost *never* consist solely of interfaces. There are many ways of designing an API which people consider "right" and I don't see anyone on this list having any special insight that would enable them to select the "one true" approach to API design. The fact is that any version scheme or module resolution engine which only works for new code which is designed "properly" for it is *going* to fail. And there's no reason for such constraints anyway. Version ranges are barely useful at all in practice. The only ones who can assert compatibility between two libraries of arbitrary version are the ones who have tested the two together (along with anything else that uses a combination of the two). The best one can do establish *minimum* compatibility based on available features, and test test test. When a module is added to a repository, if it is tested by running both its internal unit tests and all available tests of all its dependents, then that should be adequate to establish compatibility *within a repository*. There are too many factors to establish compatibility beyond it. On 11/20/2011 07:45 AM, Neil Bartlett wrote: > Stephen, > > You again seem to be talking about dependencies on implementation > packages. If we depend on APIs consisting primarily of interfaces, > then versions and version ranges are reliable and can be completely > automated. > > Regards, > Neil > > > On Sun, Nov 20, 2011 at 12:25 PM, Stephen McConnell wrote: >> >> I could say that the question is academic - but here is the thing - its not >> even academic. >> >> We talk about versions and yet the platform has no computational mechanisms >> to flag version violation. At least an academic discussion would bring in >> grounded metrics. In reality - the discussion here about V1 versus V2 and >> that warm fuzzy feeling when your are just a n.m from an n+1 and it's a >> waste of breath (and that comment goes to all sides of the debate). >> >> Give me an endorsed tool that shows me that module [name]-[build] is >> computationally backward compatible to a different [name]-[build] and then, >> only them will this discussion have reasonable grounds to assert any >> authority on the subject of version compatibility (past or future). Without >> that tool - versioning is a unqualified statement of trust. And the evidence >> suggests that the threshold of trust delivery on actual computational >> compliance is really really thin. >> >> Cheers, Steve. >> >> >> -----Original Message----- From: cowwoc >> Sent: Sunday, November 20, 2011 1:39 PM >> To: jigsaw-dev at openjdk.java.net >> Subject: Re: Use-cases for version ranges? >> >> >> I'd like to propose another possibility: the author of the >> dependency should tell *us* about version compatibility, not the other >> way around. For example: >> >> 1. The author of module A declares a dependency on module B version 1.2 >> (specific version). >> 2. The author of module B publishes version 1.3. He declares that >> version 1.3 is compatible with 1.2 (meaning, the runtime system is >> allows to substitute version 1.3 for 1.2). >> >> The upside of this approach is that the author of B is in a better >> position to declare compatibility than the author of A. The author of A >> still only needs to test a single version. What do you think? >> >> Gili >> >> On 19/11/2011 1:59 AM, Neil Bartlett wrote: >>> >>> Gili, >>> >>> I didn't say anything about guarantees, and in this industry I have >>> never heard of anybody providing a guarantees about the performance of >>> their software, especially in the presence of external dependencies. >>> >>> Version ranges are a means of communicating expectations, and we >>> provide both a lower and an upper bound because this is useful >>> information. I expect my module will work with 1.2.14, and I expect it >>> will not work with 2.0. If I were a provider of the API rather than a >>> consumer then I would have a much narrower expectation, e.g. >>> [1.2,1.3), and this would also be useful information to convey. >>> >>> Regards >>> Neil >>> >>> On Sat, Nov 19, 2011 at 4:27 AM, cowwoc wrote: >>>> >>>> Neil, >>>> >>>> I guess I don't understand why Jigsaw should work differently from >>>> Maven >>>> on this point. I am expecting developers to specify specific versions >>>> that >>>> they tested (point versions, not ranges) and end-users may override these >>>> "recommendations" as they see fit. >>>> >>>> Where you see version range [1.2.14, 2.0) as a way of communicating >>>> "the >>>> developer guarantees 1.2.14 but you may use newer versions up to 2.0 at >>>> your >>>> own risk" I'd expect the developer to simply specify 1.2.14 and there >>>> should >>>> be no limit on what version end-users may use if they so wish. >>>> >>>> Gili >>>> >>>> On 18/11/2011 2:23 AM, Neil Bartlett wrote: >>>>> >>>>> I noticed that I failed to address your point about Maven using point >>>>> versions. >>>>> >>>>> Maven is a build tool. At build time we need to compile against a >>>>> single specific version so that we have repeatable builds. In general >>>>> we should build each module against the lowest version of the library >>>>> that it can possibly use, and there are no major negative consequences >>>>> of having several versions of a library at build time (except that >>>>> Maven has to download a lot!). At runtime however we need to have the >>>>> flexibility to substitute a single compatible version. >>>>> >>>>> Neil >>>>> >>>>> On Fri, Nov 18, 2011 at 7:10 AM, Neil Bartlett >>>>> wrote: >>>>>> >>>>>> Suppose as the developer of module A, I declare a dependency on log4j, >>>>>> exactly version 1.0.0 because I have not tested against log4j 1.0.1, >>>>>> 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* being >>>>>> used with log4j version 1.0.1 even if this combinations is later >>>>>> tested and proven to work by somebody else. In other words, testing is >>>>>> important but it doesn't necessarily have to always be done by the >>>>>> original developer of each module. >>>>>> >>>>>> On the other hand let's say I state my dependency using the following >>>>>> range: [1.2.14, 2.0). This is OSGi syntax and I believe Jigsaw is >>>>>> following it, and it simply means I accept version 1.2.14 up to but >>>>>> not including 2.0. Anybody can see that I compiled and tested against >>>>>> 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 etc. It >>>>>> does not mean that I *guarantee* my module will work with log4j 1.3 >>>>>> because that obviously depends on whether the log4j authors accept and >>>>>> follow the common semantics of indicating backwards-incompatible >>>>>> changes with a bump to the first version segment. >>>>>> >>>>>> The consequence of trying to lock down imports to a narrow range or >>>>>> even a point version is that assembling an application becomes very >>>>>> difficult, and we are forced to deploy many versions of common >>>>>> libraries concurrently. This is non-optimal, though we can handle it >>>>>> to some degree via per-module classloaders as in OSGi. >>>>>> >>>>>> Regards, >>>>>> Neil >>>>>> >>>>>> On Thu, Nov 17, 2011 at 11:52 PM, cowwoc >>>>>> wrote: >>>>>>> >>>>>>> Can someone please explain why modules need to be able to specify >>>>>>> version >>>>>>> ranges for dependencies? I believe OSGI allows the specification of >>>>>>> version >>>>>>> ranges while Maven allows the specification of individual versions. >>>>>>> >>>>>>> The only thing that comes to mind is when module C depends on A and B, >>>>>>> A >>>>>>> depends on log4j 1.0, and B depends on log4j 1.1. What does C do? Is >>>>>>> this >>>>>>> the main use-case for version ranges? >>>>>>> >>>>>>> By the sound of it, this is a trust model where developers are told >>>>>>> that >>>>>>> log4j 1.x won't break compatibility so they depend on that range >>>>>>> without >>>>>>> actually testing against each version (newer versions may be released >>>>>>> after >>>>>>> their own software). I question whether such a mechanism is better or >>>>>>> worse >>>>>>> than depending on individual versions which may be overridden at a >>>>>>> later >>>>>>> time (a la Maven). On the one hand, you don't need to release a new >>>>>>> version >>>>>>> of the application each time a dependency is updated. On the other >>>>>>> hand, >>>>>>> no >>>>>>> one is actually running tests to ensure that the versions are really >>>>>>> compatible. >>>>>>> >>>>>>> Is there a way to get module A to see log4j 1.0 and module B to see >>>>>>> log4j >>>>>>> 1.1 (using separate ClassLoaders)? >>>>>>> >>>>>>> Thanks, >>>>>>> Gili >>>>>>> >>>>>>> -- >>>>>>> View this message in context: >>>>>>> >>>>>>> http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html >>>>>>> Sent from the jigsaw-dev mailing list archive at Nabble.com. >>>>>>> >>>> >> >> -- - DML From njbartlett at gmail.com Mon Nov 21 08:11:19 2011 From: njbartlett at gmail.com (Neil Bartlett) Date: Mon, 21 Nov 2011 16:11:19 +0000 Subject: Use-cases for version ranges? In-Reply-To: <4ECA6B05.8020001@redhat.com> References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> <4ECA6B05.8020001@redhat.com> Message-ID: David, I'm likewise tired of hearing these defeatist straw-man arguments. First, nobody is suggesting that you be forced to use version ranges. Second, nobody is suggesting that you be forced to design software by separating interfaces from implementation. Finally, nobody is suggesting that unmodified legacy code can be trivially modularised in *any* module system. The module system must support both existing software and new software, of course. You seem suggest that I believe existing software is unimportant; nothing could be further from the truth. But I also believe that new software is important and that practices like interface/implementation separation and semantic versioning will help make the new software better than the old software. Therefore I believe that these features must be supported (though not mandated) in any new module system. You appear to be suggesting that these features are worthless because they cannot be applied to existing software, and/or will not be used by all Java developers. Well so what?? Regards,Neil On Mon, Nov 21, 2011 at 3:15 PM, David M. Lloyd wrote: > I'm tired of hearing pie-in-the sky, "intro to OOP" wishful crap like this. > ?Mission one should be supporting *existing* code. ?The fact is that people > do *not* always split their API from their implementation, and APIs almost > *never* consist solely of interfaces. ?There are many ways of designing an > API which people consider "right" and I don't see anyone on this list having > any special insight that would enable them to select the "one true" approach > to API design. > > The fact is that any version scheme or module resolution engine which only > works for new code which is designed "properly" for it is *going* to fail. > ?And there's no reason for such constraints anyway. ?Version ranges are > barely useful at all in practice. ?The only ones who can assert > compatibility between two libraries of arbitrary version are the ones who > have tested the two together (along with anything else that uses a > combination of the two). > > The best one can do establish *minimum* compatibility based on available > features, and test test test. ?When a module is added to a repository, if it > is tested by running both its internal unit tests and all available tests of > all its dependents, then that should be adequate to establish compatibility > *within a repository*. ?There are too many factors to establish > compatibility beyond it. > > On 11/20/2011 07:45 AM, Neil Bartlett wrote: >> >> Stephen, >> >> You again seem to be talking about dependencies on implementation >> packages. If we depend on APIs consisting primarily of interfaces, >> then versions and version ranges are reliable and can be completely >> automated. >> >> Regards, >> Neil >> >> >> On Sun, Nov 20, 2011 at 12:25 PM, Stephen McConnell >> ?wrote: >>> >>> I could say that the question is academic - but here is the thing - its >>> not >>> even academic. >>> >>> We talk about versions and yet the platform has no computational >>> mechanisms >>> to flag version violation. At least an academic discussion would bring in >>> grounded metrics. ?In reality - the discussion here about V1 versus V2 >>> and >>> that warm fuzzy feeling when your are just a n.m from an n+1 and it's a >>> waste of breath (and that comment goes to all sides of the debate). >>> >>> Give me an endorsed tool that shows me that module [name]-[build] is >>> computationally backward compatible to a different [name]-[build] and >>> then, >>> only them will this discussion have reasonable grounds to assert any >>> authority on the subject of version compatibility (past or future). >>> Without >>> that tool - versioning is a unqualified statement of trust. And the >>> evidence >>> suggests that the threshold of trust delivery on actual computational >>> compliance is really really thin. >>> >>> Cheers, Steve. >>> >>> >>> -----Original Message----- From: cowwoc >>> Sent: Sunday, November 20, 2011 1:39 PM >>> To: jigsaw-dev at openjdk.java.net >>> Subject: Re: Use-cases for version ranges? >>> >>> >>> ? ?I'd like to propose another possibility: the author of the >>> dependency should tell *us* about version compatibility, not the other >>> way around. For example: >>> >>> 1. The author of module A declares a dependency on module B version 1.2 >>> (specific version). >>> 2. The author of module B publishes version 1.3. He declares that >>> version 1.3 is compatible with 1.2 (meaning, the runtime system is >>> allows to substitute version 1.3 for 1.2). >>> >>> ? ?The upside of this approach is that the author of B is in a better >>> position to declare compatibility than the author of A. The author of A >>> still only needs to test a single version. What do you think? >>> >>> Gili >>> >>> On 19/11/2011 1:59 AM, Neil Bartlett wrote: >>>> >>>> Gili, >>>> >>>> I didn't say anything about guarantees, and in this industry I have >>>> never heard of anybody providing a guarantees about the performance of >>>> their software, especially in the presence of external dependencies. >>>> >>>> Version ranges are a means of communicating expectations, and we >>>> provide both a lower and an upper bound because this is useful >>>> information. I expect my module will work with 1.2.14, and I expect it >>>> will not work with 2.0. If I were a provider of the API rather than a >>>> consumer then I would have a much narrower expectation, ?e.g. >>>> [1.2,1.3), and this would also be useful information to convey. >>>> >>>> Regards >>>> Neil >>>> >>>> On Sat, Nov 19, 2011 at 4:27 AM, cowwoc >>>> ?wrote: >>>>> >>>>> Neil, >>>>> >>>>> ? ?I guess I don't understand why Jigsaw should work differently from >>>>> Maven >>>>> on this point. I am expecting developers to specify specific versions >>>>> that >>>>> they tested (point versions, not ranges) and end-users may override >>>>> these >>>>> "recommendations" as they see fit. >>>>> >>>>> ? ?Where you see version range [1.2.14, 2.0) as a way of communicating >>>>> "the >>>>> developer guarantees 1.2.14 but you may use newer versions up to 2.0 at >>>>> your >>>>> own risk" I'd expect the developer to simply specify 1.2.14 and there >>>>> should >>>>> be no limit on what version end-users may use if they so wish. >>>>> >>>>> Gili >>>>> >>>>> On 18/11/2011 2:23 AM, Neil Bartlett wrote: >>>>>> >>>>>> I noticed that I failed to address your point about Maven using point >>>>>> versions. >>>>>> >>>>>> Maven is a build tool. At build time we need to compile against a >>>>>> single specific version so that we have repeatable builds. In general >>>>>> we should build each module against the lowest version of the library >>>>>> that it can possibly use, and there are no major negative consequences >>>>>> of having several versions of a library at build time (except that >>>>>> Maven has to download a lot!). At runtime however we need to have the >>>>>> flexibility to substitute a single compatible version. >>>>>> >>>>>> Neil >>>>>> >>>>>> On Fri, Nov 18, 2011 at 7:10 AM, Neil Bartlett >>>>>> ?wrote: >>>>>>> >>>>>>> Suppose as the developer of module A, I declare a dependency on >>>>>>> log4j, >>>>>>> exactly version 1.0.0 because I have not tested against log4j 1.0.1, >>>>>>> 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* being >>>>>>> used with log4j version 1.0.1 even if this combinations is later >>>>>>> tested and proven to work by somebody else. In other words, testing >>>>>>> is >>>>>>> important but it doesn't necessarily have to always be done by the >>>>>>> original developer of each module. >>>>>>> >>>>>>> On the other hand let's say I state my dependency using the following >>>>>>> range: [1.2.14, 2.0). This is OSGi syntax and I believe Jigsaw is >>>>>>> following it, and it simply means I accept version 1.2.14 up to but >>>>>>> not including 2.0. Anybody can see that I compiled and tested against >>>>>>> 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 etc. It >>>>>>> does not mean that I *guarantee* my module will work with log4j 1.3 >>>>>>> because that obviously depends on whether the log4j authors accept >>>>>>> and >>>>>>> follow the common semantics of indicating backwards-incompatible >>>>>>> changes with a bump to the first version segment. >>>>>>> >>>>>>> The consequence of trying to lock down imports to a narrow range or >>>>>>> even a point version is that assembling an application becomes very >>>>>>> difficult, and we are forced to deploy many versions of common >>>>>>> libraries concurrently. This is non-optimal, though we can handle it >>>>>>> to some degree via per-module classloaders as in OSGi. >>>>>>> >>>>>>> Regards, >>>>>>> Neil >>>>>>> >>>>>>> On Thu, Nov 17, 2011 at 11:52 PM, cowwoc >>>>>>> wrote: >>>>>>>> >>>>>>>> Can someone please explain why modules need to be able to specify >>>>>>>> version >>>>>>>> ranges for dependencies? I believe OSGI allows the specification of >>>>>>>> version >>>>>>>> ranges while Maven allows the specification of individual versions. >>>>>>>> >>>>>>>> The only thing that comes to mind is when module C depends on A and >>>>>>>> B, >>>>>>>> A >>>>>>>> depends on log4j 1.0, and B depends on log4j 1.1. What does C do? Is >>>>>>>> this >>>>>>>> the main use-case for version ranges? >>>>>>>> >>>>>>>> By the sound of it, this is a trust model where developers are told >>>>>>>> that >>>>>>>> log4j 1.x won't break compatibility so they depend on that range >>>>>>>> without >>>>>>>> actually testing against each version (newer versions may be >>>>>>>> released >>>>>>>> after >>>>>>>> their own software). I question whether such a mechanism is better >>>>>>>> or >>>>>>>> worse >>>>>>>> than depending on individual versions which may be overridden at a >>>>>>>> later >>>>>>>> time (a la Maven). On the one hand, you don't need to release a new >>>>>>>> version >>>>>>>> of the application each time a dependency is updated. On the other >>>>>>>> hand, >>>>>>>> no >>>>>>>> one is actually running tests to ensure that the versions are really >>>>>>>> compatible. >>>>>>>> >>>>>>>> Is there a way to get module A to see log4j 1.0 and module B to see >>>>>>>> log4j >>>>>>>> 1.1 (using separate ClassLoaders)? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Gili >>>>>>>> >>>>>>>> -- >>>>>>>> View this message in context: >>>>>>>> >>>>>>>> >>>>>>>> http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html >>>>>>>> Sent from the jigsaw-dev mailing list archive at Nabble.com. >>>>>>>> >>>>> >>> >>> > > > -- > - DML > From brian at pontarelli.com Mon Nov 21 09:37:57 2011 From: brian at pontarelli.com (Brian Pontarelli) Date: Mon, 21 Nov 2011 10:37:57 -0700 Subject: Use-cases for version ranges? In-Reply-To: References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> <4ECA6B05.8020001@redhat.com> Message-ID: <4957F10E-1272-465C-82B8-762664881958@pontarelli.com> To answer your "well so what??" question, adding a feature simply to have a feature seems really dangerous to me. I'd much rather be pragmatic about what needs to be included to support the current code written in Java and what needs to be included to allow a module system to work. I don't see version ranges and certain other items being necessary features. I also see some features as adding pain points rather than alleviating them. -bp On Nov 21, 2011, at 9:11 AM, Neil Bartlett wrote: > David, > I'm likewise tired of hearing these defeatist straw-man arguments. > First, nobody is suggesting that you be forced to use version ranges. > Second, nobody is suggesting that you be forced to design software by > separating interfaces from implementation. Finally, nobody is > suggesting that unmodified legacy code can be trivially modularised in > *any* module system. > The module system must support both existing software and new > software, of course. You seem suggest that I believe existing software > is unimportant; nothing could be further from the truth. But I also > believe that new software is important and that practices like > interface/implementation separation and semantic versioning will help > make the new software better than the old software. > Therefore I believe that these features must be supported (though not > mandated) in any new module system. You appear to be suggesting that > these features are worthless because they cannot be applied to > existing software, and/or will not be used by all Java developers. > Well so what?? > Regards,Neil > On Mon, Nov 21, 2011 at 3:15 PM, David M. Lloyd wrote: >> I'm tired of hearing pie-in-the sky, "intro to OOP" wishful crap like this. >> Mission one should be supporting *existing* code. The fact is that people >> do *not* always split their API from their implementation, and APIs almost >> *never* consist solely of interfaces. There are many ways of designing an >> API which people consider "right" and I don't see anyone on this list having >> any special insight that would enable them to select the "one true" approach >> to API design. >> >> The fact is that any version scheme or module resolution engine which only >> works for new code which is designed "properly" for it is *going* to fail. >> And there's no reason for such constraints anyway. Version ranges are >> barely useful at all in practice. The only ones who can assert >> compatibility between two libraries of arbitrary version are the ones who >> have tested the two together (along with anything else that uses a >> combination of the two). >> >> The best one can do establish *minimum* compatibility based on available >> features, and test test test. When a module is added to a repository, if it >> is tested by running both its internal unit tests and all available tests of >> all its dependents, then that should be adequate to establish compatibility >> *within a repository*. There are too many factors to establish >> compatibility beyond it. >> >> On 11/20/2011 07:45 AM, Neil Bartlett wrote: >>> >>> Stephen, >>> >>> You again seem to be talking about dependencies on implementation >>> packages. If we depend on APIs consisting primarily of interfaces, >>> then versions and version ranges are reliable and can be completely >>> automated. >>> >>> Regards, >>> Neil >>> >>> >>> On Sun, Nov 20, 2011 at 12:25 PM, Stephen McConnell >>> wrote: >>>> >>>> I could say that the question is academic - but here is the thing - its >>>> not >>>> even academic. >>>> >>>> We talk about versions and yet the platform has no computational >>>> mechanisms >>>> to flag version violation. At least an academic discussion would bring in >>>> grounded metrics. In reality - the discussion here about V1 versus V2 >>>> and >>>> that warm fuzzy feeling when your are just a n.m from an n+1 and it's a >>>> waste of breath (and that comment goes to all sides of the debate). >>>> >>>> Give me an endorsed tool that shows me that module [name]-[build] is >>>> computationally backward compatible to a different [name]-[build] and >>>> then, >>>> only them will this discussion have reasonable grounds to assert any >>>> authority on the subject of version compatibility (past or future). >>>> Without >>>> that tool - versioning is a unqualified statement of trust. And the >>>> evidence >>>> suggests that the threshold of trust delivery on actual computational >>>> compliance is really really thin. >>>> >>>> Cheers, Steve. >>>> >>>> >>>> -----Original Message----- From: cowwoc >>>> Sent: Sunday, November 20, 2011 1:39 PM >>>> To: jigsaw-dev at openjdk.java.net >>>> Subject: Re: Use-cases for version ranges? >>>> >>>> >>>> I'd like to propose another possibility: the author of the >>>> dependency should tell *us* about version compatibility, not the other >>>> way around. For example: >>>> >>>> 1. The author of module A declares a dependency on module B version 1.2 >>>> (specific version). >>>> 2. The author of module B publishes version 1.3. He declares that >>>> version 1.3 is compatible with 1.2 (meaning, the runtime system is >>>> allows to substitute version 1.3 for 1.2). >>>> >>>> The upside of this approach is that the author of B is in a better >>>> position to declare compatibility than the author of A. The author of A >>>> still only needs to test a single version. What do you think? >>>> >>>> Gili >>>> >>>> On 19/11/2011 1:59 AM, Neil Bartlett wrote: >>>>> >>>>> Gili, >>>>> >>>>> I didn't say anything about guarantees, and in this industry I have >>>>> never heard of anybody providing a guarantees about the performance of >>>>> their software, especially in the presence of external dependencies. >>>>> >>>>> Version ranges are a means of communicating expectations, and we >>>>> provide both a lower and an upper bound because this is useful >>>>> information. I expect my module will work with 1.2.14, and I expect it >>>>> will not work with 2.0. If I were a provider of the API rather than a >>>>> consumer then I would have a much narrower expectation, e.g. >>>>> [1.2,1.3), and this would also be useful information to convey. >>>>> >>>>> Regards >>>>> Neil >>>>> >>>>> On Sat, Nov 19, 2011 at 4:27 AM, cowwoc >>>>> wrote: >>>>>> >>>>>> Neil, >>>>>> >>>>>> I guess I don't understand why Jigsaw should work differently from >>>>>> Maven >>>>>> on this point. I am expecting developers to specify specific versions >>>>>> that >>>>>> they tested (point versions, not ranges) and end-users may override >>>>>> these >>>>>> "recommendations" as they see fit. >>>>>> >>>>>> Where you see version range [1.2.14, 2.0) as a way of communicating >>>>>> "the >>>>>> developer guarantees 1.2.14 but you may use newer versions up to 2.0 at >>>>>> your >>>>>> own risk" I'd expect the developer to simply specify 1.2.14 and there >>>>>> should >>>>>> be no limit on what version end-users may use if they so wish. >>>>>> >>>>>> Gili >>>>>> >>>>>> On 18/11/2011 2:23 AM, Neil Bartlett wrote: >>>>>>> >>>>>>> I noticed that I failed to address your point about Maven using point >>>>>>> versions. >>>>>>> >>>>>>> Maven is a build tool. At build time we need to compile against a >>>>>>> single specific version so that we have repeatable builds. In general >>>>>>> we should build each module against the lowest version of the library >>>>>>> that it can possibly use, and there are no major negative consequences >>>>>>> of having several versions of a library at build time (except that >>>>>>> Maven has to download a lot!). At runtime however we need to have the >>>>>>> flexibility to substitute a single compatible version. >>>>>>> >>>>>>> Neil >>>>>>> >>>>>>> On Fri, Nov 18, 2011 at 7:10 AM, Neil Bartlett >>>>>>> wrote: >>>>>>>> >>>>>>>> Suppose as the developer of module A, I declare a dependency on >>>>>>>> log4j, >>>>>>>> exactly version 1.0.0 because I have not tested against log4j 1.0.1, >>>>>>>> 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* being >>>>>>>> used with log4j version 1.0.1 even if this combinations is later >>>>>>>> tested and proven to work by somebody else. In other words, testing >>>>>>>> is >>>>>>>> important but it doesn't necessarily have to always be done by the >>>>>>>> original developer of each module. >>>>>>>> >>>>>>>> On the other hand let's say I state my dependency using the following >>>>>>>> range: [1.2.14, 2.0). This is OSGi syntax and I believe Jigsaw is >>>>>>>> following it, and it simply means I accept version 1.2.14 up to but >>>>>>>> not including 2.0. Anybody can see that I compiled and tested against >>>>>>>> 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 etc. It >>>>>>>> does not mean that I *guarantee* my module will work with log4j 1.3 >>>>>>>> because that obviously depends on whether the log4j authors accept >>>>>>>> and >>>>>>>> follow the common semantics of indicating backwards-incompatible >>>>>>>> changes with a bump to the first version segment. >>>>>>>> >>>>>>>> The consequence of trying to lock down imports to a narrow range or >>>>>>>> even a point version is that assembling an application becomes very >>>>>>>> difficult, and we are forced to deploy many versions of common >>>>>>>> libraries concurrently. This is non-optimal, though we can handle it >>>>>>>> to some degree via per-module classloaders as in OSGi. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Neil >>>>>>>> >>>>>>>> On Thu, Nov 17, 2011 at 11:52 PM, cowwoc >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Can someone please explain why modules need to be able to specify >>>>>>>>> version >>>>>>>>> ranges for dependencies? I believe OSGI allows the specification of >>>>>>>>> version >>>>>>>>> ranges while Maven allows the specification of individual versions. >>>>>>>>> >>>>>>>>> The only thing that comes to mind is when module C depends on A and >>>>>>>>> B, >>>>>>>>> A >>>>>>>>> depends on log4j 1.0, and B depends on log4j 1.1. What does C do? Is >>>>>>>>> this >>>>>>>>> the main use-case for version ranges? >>>>>>>>> >>>>>>>>> By the sound of it, this is a trust model where developers are told >>>>>>>>> that >>>>>>>>> log4j 1.x won't break compatibility so they depend on that range >>>>>>>>> without >>>>>>>>> actually testing against each version (newer versions may be >>>>>>>>> released >>>>>>>>> after >>>>>>>>> their own software). I question whether such a mechanism is better >>>>>>>>> or >>>>>>>>> worse >>>>>>>>> than depending on individual versions which may be overridden at a >>>>>>>>> later >>>>>>>>> time (a la Maven). On the one hand, you don't need to release a new >>>>>>>>> version >>>>>>>>> of the application each time a dependency is updated. On the other >>>>>>>>> hand, >>>>>>>>> no >>>>>>>>> one is actually running tests to ensure that the versions are really >>>>>>>>> compatible. >>>>>>>>> >>>>>>>>> Is there a way to get module A to see log4j 1.0 and module B to see >>>>>>>>> log4j >>>>>>>>> 1.1 (using separate ClassLoaders)? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Gili >>>>>>>>> >>>>>>>>> -- >>>>>>>>> View this message in context: >>>>>>>>> >>>>>>>>> >>>>>>>>> http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html >>>>>>>>> Sent from the jigsaw-dev mailing list archive at Nabble.com. >>>>>>>>> >>>>>> >>>> >>>> >> >> >> -- >> - DML >> From njbartlett at gmail.com Mon Nov 21 09:50:25 2011 From: njbartlett at gmail.com (Neil Bartlett) Date: Mon, 21 Nov 2011 17:50:25 +0000 Subject: Use-cases for version ranges? In-Reply-To: <4957F10E-1272-465C-82B8-762664881958@pontarelli.com> References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> <4ECA6B05.8020001@redhat.com> <4957F10E-1272-465C-82B8-762664881958@pontarelli.com> Message-ID: <04E6880E3AF04DC58FDA3B422DA445C3@gmail.com> Brian, In that case it seems you're prepared to sacrifice a large body of existing code, i.e. the OSGi bundles that use version ranges already. That doesn't seem very consistent. I'm all for pragmatism. Version ranges can always be locked down to a single point if you desire, but you cannot unlock a point-version system and turn it into one with ranges. I also understand that adding excessive features carries conceptual weight that can make a system harder to use. Conversely omitting features that are needed makes it *much* harder to use. Version ranges fall into the latter category because they are actually used in practice, and you're proposing to take them away from everybody just because some people don't want to use them. Regards Neil On Monday, 21 November 2011 at 17:37, Brian Pontarelli wrote: > To answer your "well so what??" question, adding a feature simply to have a feature seems really dangerous to me. I'd much rather be pragmatic about what needs to be included to support the current code written in Java and what needs to be included to allow a module system to work. > > I don't see version ranges and certain other items being necessary features. I also see some features as adding pain points rather than alleviating them. > > -bp > > > On Nov 21, 2011, at 9:11 AM, Neil Bartlett wrote: > > > David, > > I'm likewise tired of hearing these defeatist straw-man arguments. > > First, nobody is suggesting that you be forced to use version ranges. > > Second, nobody is suggesting that you be forced to design software by > > separating interfaces from implementation. Finally, nobody is > > suggesting that unmodified legacy code can be trivially modularised in > > *any* module system. > > The module system must support both existing software and new > > software, of course. You seem suggest that I believe existing software > > is unimportant; nothing could be further from the truth. But I also > > believe that new software is important and that practices like > > interface/implementation separation and semantic versioning will help > > make the new software better than the old software. > > Therefore I believe that these features must be supported (though not > > mandated) in any new module system. You appear to be suggesting that > > these features are worthless because they cannot be applied to > > existing software, and/or will not be used by all Java developers. > > Well so what?? > > Regards,Neil > > On Mon, Nov 21, 2011 at 3:15 PM, David M. Lloyd wrote: > > > I'm tired of hearing pie-in-the sky, "intro to OOP" wishful crap like this. > > > Mission one should be supporting *existing* code. The fact is that people > > > do *not* always split their API from their implementation, and APIs almost > > > *never* consist solely of interfaces. There are many ways of designing an > > > API which people consider "right" and I don't see anyone on this list having > > > any special insight that would enable them to select the "one true" approach > > > to API design. > > > > > > The fact is that any version scheme or module resolution engine which only > > > works for new code which is designed "properly" for it is *going* to fail. > > > And there's no reason for such constraints anyway. Version ranges are > > > barely useful at all in practice. The only ones who can assert > > > compatibility between two libraries of arbitrary version are the ones who > > > have tested the two together (along with anything else that uses a > > > combination of the two). > > > > > > The best one can do establish *minimum* compatibility based on available > > > features, and test test test. When a module is added to a repository, if it > > > is tested by running both its internal unit tests and all available tests of > > > all its dependents, then that should be adequate to establish compatibility > > > *within a repository*. There are too many factors to establish > > > compatibility beyond it. > > > > > > On 11/20/2011 07:45 AM, Neil Bartlett wrote: > > > > > > > > Stephen, > > > > > > > > You again seem to be talking about dependencies on implementation > > > > packages. If we depend on APIs consisting primarily of interfaces, > > > > then versions and version ranges are reliable and can be completely > > > > automated. > > > > > > > > Regards, > > > > Neil > > > > > > > > > > > > On Sun, Nov 20, 2011 at 12:25 PM, Stephen McConnell > > > > wrote: > > > > > > > > > > I could say that the question is academic - but here is the thing - its > > > > > not > > > > > even academic. > > > > > > > > > > We talk about versions and yet the platform has no computational > > > > > mechanisms > > > > > to flag version violation. At least an academic discussion would bring in > > > > > grounded metrics. In reality - the discussion here about V1 versus V2 > > > > > and > > > > > that warm fuzzy feeling when your are just a n.m from an n+1 and it's a > > > > > waste of breath (and that comment goes to all sides of the debate). > > > > > > > > > > Give me an endorsed tool that shows me that module [name]-[build] is > > > > > computationally backward compatible to a different [name]-[build] and > > > > > then, > > > > > only them will this discussion have reasonable grounds to assert any > > > > > authority on the subject of version compatibility (past or future). > > > > > Without > > > > > that tool - versioning is a unqualified statement of trust. And the > > > > > evidence > > > > > suggests that the threshold of trust delivery on actual computational > > > > > compliance is really really thin. > > > > > > > > > > Cheers, Steve. > > > > > > > > > > > > > > > -----Original Message----- From: cowwoc > > > > > Sent: Sunday, November 20, 2011 1:39 PM > > > > > To: jigsaw-dev at openjdk.java.net (mailto:jigsaw-dev at openjdk.java.net) > > > > > Subject: Re: Use-cases for version ranges? > > > > > > > > > > > > > > > I'd like to propose another possibility: the author of the > > > > > dependency should tell *us* about version compatibility, not the other > > > > > way around. For example: > > > > > > > > > > 1. The author of module A declares a dependency on module B version 1.2 > > > > > (specific version). > > > > > 2. The author of module B publishes version 1.3. He declares that > > > > > version 1.3 is compatible with 1.2 (meaning, the runtime system is > > > > > allows to substitute version 1.3 for 1.2). > > > > > > > > > > The upside of this approach is that the author of B is in a better > > > > > position to declare compatibility than the author of A. The author of A > > > > > still only needs to test a single version. What do you think? > > > > > > > > > > Gili > > > > > > > > > > On 19/11/2011 1:59 AM, Neil Bartlett wrote: > > > > > > > > > > > > Gili, > > > > > > > > > > > > I didn't say anything about guarantees, and in this industry I have > > > > > > never heard of anybody providing a guarantees about the performance of > > > > > > their software, especially in the presence of external dependencies. > > > > > > > > > > > > Version ranges are a means of communicating expectations, and we > > > > > > provide both a lower and an upper bound because this is useful > > > > > > information. I expect my module will work with 1.2.14, and I expect it > > > > > > will not work with 2.0. If I were a provider of the API rather than a > > > > > > consumer then I would have a much narrower expectation, e.g. > > > > > > [1.2,1.3), and this would also be useful information to convey. > > > > > > > > > > > > Regards > > > > > > Neil > > > > > > > > > > > > On Sat, Nov 19, 2011 at 4:27 AM, cowwoc > > > > > > wrote: > > > > > > > > > > > > > > Neil, > > > > > > > > > > > > > > I guess I don't understand why Jigsaw should work differently from > > > > > > > Maven > > > > > > > on this point. I am expecting developers to specify specific versions > > > > > > > that > > > > > > > they tested (point versions, not ranges) and end-users may override > > > > > > > these > > > > > > > "recommendations" as they see fit. > > > > > > > > > > > > > > Where you see version range [1.2.14, 2.0) as a way of communicating > > > > > > > "the > > > > > > > developer guarantees 1.2.14 but you may use newer versions up to 2.0 at > > > > > > > your > > > > > > > own risk" I'd expect the developer to simply specify 1.2.14 and there > > > > > > > should > > > > > > > be no limit on what version end-users may use if they so wish. > > > > > > > > > > > > > > Gili > > > > > > > > > > > > > > On 18/11/2011 2:23 AM, Neil Bartlett wrote: > > > > > > > > > > > > > > > > I noticed that I failed to address your point about Maven using point > > > > > > > > versions. > > > > > > > > > > > > > > > > Maven is a build tool. At build time we need to compile against a > > > > > > > > single specific version so that we have repeatable builds. In general > > > > > > > > we should build each module against the lowest version of the library > > > > > > > > that it can possibly use, and there are no major negative consequences > > > > > > > > of having several versions of a library at build time (except that > > > > > > > > Maven has to download a lot!). At runtime however we need to have the > > > > > > > > flexibility to substitute a single compatible version. > > > > > > > > > > > > > > > > Neil > > > > > > > > > > > > > > > > On Fri, Nov 18, 2011 at 7:10 AM, Neil Bartlett > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Suppose as the developer of module A, I declare a dependency on > > > > > > > > > log4j, > > > > > > > > > exactly version 1.0.0 because I have not tested against log4j 1.0.1, > > > > > > > > > 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* being > > > > > > > > > used with log4j version 1.0.1 even if this combinations is later > > > > > > > > > tested and proven to work by somebody else. In other words, testing > > > > > > > > > is > > > > > > > > > important but it doesn't necessarily have to always be done by the > > > > > > > > > original developer of each module. > > > > > > > > > > > > > > > > > > On the other hand let's say I state my dependency using the following > > > > > > > > > range: [1.2.14, 2.0). This is OSGi syntax and I believe Jigsaw is > > > > > > > > > following it, and it simply means I accept version 1.2.14 up to but > > > > > > > > > not including 2.0. Anybody can see that I compiled and tested against > > > > > > > > > 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 etc. It > > > > > > > > > does not mean that I *guarantee* my module will work with log4j 1.3 > > > > > > > > > because that obviously depends on whether the log4j authors accept > > > > > > > > > and > > > > > > > > > follow the common semantics of indicating backwards-incompatible > > > > > > > > > changes with a bump to the first version segment. > > > > > > > > > > > > > > > > > > The consequence of trying to lock down imports to a narrow range or > > > > > > > > > even a point version is that assembling an application becomes very > > > > > > > > > difficult, and we are forced to deploy many versions of common > > > > > > > > > libraries concurrently. This is non-optimal, though we can handle it > > > > > > > > > to some degree via per-module classloaders as in OSGi. > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > Neil > > > > > > > > > > > > > > > > > > On Thu, Nov 17, 2011 at 11:52 PM, cowwoc > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Can someone please explain why modules need to be able to specify > > > > > > > > > > version > > > > > > > > > > ranges for dependencies? I believe OSGI allows the specification of > > > > > > > > > > version > > > > > > > > > > ranges while Maven allows the specification of individual versions. > > > > > > > > > > > > > > > > > > > > The only thing that comes to mind is when module C depends on A and > > > > > > > > > > B, > > > > > > > > > > A > > > > > > > > > > depends on log4j 1.0, and B depends on log4j 1.1. What does C do? Is > > > > > > > > > > this > > > > > > > > > > the main use-case for version ranges? > > > > > > > > > > > > > > > > > > > > By the sound of it, this is a trust model where developers are told > > > > > > > > > > that > > > > > > > > > > log4j 1.x won't break compatibility so they depend on that range > > > > > > > > > > without > > > > > > > > > > actually testing against each version (newer versions may be > > > > > > > > > > released > > > > > > > > > > after > > > > > > > > > > their own software). I question whether such a mechanism is better > > > > > > > > > > or > > > > > > > > > > worse > > > > > > > > > > than depending on individual versions which may be overridden at a > > > > > > > > > > later > > > > > > > > > > time (a la Maven). On the one hand, you don't need to release a new > > > > > > > > > > version > > > > > > > > > > of the application each time a dependency is updated. On the other > > > > > > > > > > hand, > > > > > > > > > > no > > > > > > > > > > one is actually running tests to ensure that the versions are really > > > > > > > > > > compatible. > > > > > > > > > > > > > > > > > > > > Is there a way to get module A to see log4j 1.0 and module B to see > > > > > > > > > > log4j > > > > > > > > > > 1.1 (using separate ClassLoaders)? > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Gili > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > View this message in context: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html > > > > > > > > > > Sent from the jigsaw-dev mailing list archive at Nabble.com (http://Nabble.com). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > - DML > > > > > > > > > > From brian at pontarelli.com Mon Nov 21 10:11:35 2011 From: brian at pontarelli.com (Brian Pontarelli) Date: Mon, 21 Nov 2011 11:11:35 -0700 Subject: Use-cases for version ranges? In-Reply-To: <04E6880E3AF04DC58FDA3B422DA445C3@gmail.com> References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> <4ECA6B05.8020001@redhat.com> <4957F10E-1272-465C-82B8-762664881958@pontarelli.com> <04E6880E3AF04DC58FDA3B422DA445C3@gmail.com> Message-ID: <442B574F-466C-4F91-9BD5-FD25DF67E272@pontarelli.com> This is something I had forgotten about because I haven't used OSGi in quite a while. I'll need to consider it more, but my initial thought is that these might not co-exist well. Although, perhaps there is a way to translate OSGi version ranges to a compatibility verification system under JMS (or any other system that uses compatibility verification as I've defined it). Thanks for reminding me about OSGi's version system. I'll definitely have to go back and think about it some more. -bp On Nov 21, 2011, at 10:50 AM, Neil Bartlett wrote: > Brian, > > In that case it seems you're prepared to sacrifice a large body of existing code, i.e. the OSGi bundles that use version ranges already. That doesn't seem very consistent. > > I'm all for pragmatism. Version ranges can always be locked down to a single point if you desire, but you cannot unlock a point-version system and turn it into one with ranges. > > I also understand that adding excessive features carries conceptual weight that can make a system harder to use. Conversely omitting features that are needed makes it *much* harder to use. Version ranges fall into the latter category because they are actually used in practice, and you're proposing to take them away from everybody just because some people don't want to use them. > > Regards > Neil > > > On Monday, 21 November 2011 at 17:37, Brian Pontarelli wrote: > >> To answer your "well so what??" question, adding a feature simply to have a feature seems really dangerous to me. I'd much rather be pragmatic about what needs to be included to support the current code written in Java and what needs to be included to allow a module system to work. >> >> I don't see version ranges and certain other items being necessary features. I also see some features as adding pain points rather than alleviating them. >> >> -bp >> >> >> On Nov 21, 2011, at 9:11 AM, Neil Bartlett wrote: >> >>> David, >>> I'm likewise tired of hearing these defeatist straw-man arguments. >>> First, nobody is suggesting that you be forced to use version ranges. >>> Second, nobody is suggesting that you be forced to design software by >>> separating interfaces from implementation. Finally, nobody is >>> suggesting that unmodified legacy code can be trivially modularised in >>> *any* module system. >>> The module system must support both existing software and new >>> software, of course. You seem suggest that I believe existing software >>> is unimportant; nothing could be further from the truth. But I also >>> believe that new software is important and that practices like >>> interface/implementation separation and semantic versioning will help >>> make the new software better than the old software. >>> Therefore I believe that these features must be supported (though not >>> mandated) in any new module system. You appear to be suggesting that >>> these features are worthless because they cannot be applied to >>> existing software, and/or will not be used by all Java developers. >>> Well so what?? >>> Regards,Neil >>> On Mon, Nov 21, 2011 at 3:15 PM, David M. Lloyd wrote: >>>> I'm tired of hearing pie-in-the sky, "intro to OOP" wishful crap like this. >>>> Mission one should be supporting *existing* code. The fact is that people >>>> do *not* always split their API from their implementation, and APIs almost >>>> *never* consist solely of interfaces. There are many ways of designing an >>>> API which people consider "right" and I don't see anyone on this list having >>>> any special insight that would enable them to select the "one true" approach >>>> to API design. >>>> >>>> The fact is that any version scheme or module resolution engine which only >>>> works for new code which is designed "properly" for it is *going* to fail. >>>> And there's no reason for such constraints anyway. Version ranges are >>>> barely useful at all in practice. The only ones who can assert >>>> compatibility between two libraries of arbitrary version are the ones who >>>> have tested the two together (along with anything else that uses a >>>> combination of the two). >>>> >>>> The best one can do establish *minimum* compatibility based on available >>>> features, and test test test. When a module is added to a repository, if it >>>> is tested by running both its internal unit tests and all available tests of >>>> all its dependents, then that should be adequate to establish compatibility >>>> *within a repository*. There are too many factors to establish >>>> compatibility beyond it. >>>> >>>> On 11/20/2011 07:45 AM, Neil Bartlett wrote: >>>>> >>>>> Stephen, >>>>> >>>>> You again seem to be talking about dependencies on implementation >>>>> packages. If we depend on APIs consisting primarily of interfaces, >>>>> then versions and version ranges are reliable and can be completely >>>>> automated. >>>>> >>>>> Regards, >>>>> Neil >>>>> >>>>> >>>>> On Sun, Nov 20, 2011 at 12:25 PM, Stephen McConnell >>>>> wrote: >>>>>> >>>>>> I could say that the question is academic - but here is the thing - its >>>>>> not >>>>>> even academic. >>>>>> >>>>>> We talk about versions and yet the platform has no computational >>>>>> mechanisms >>>>>> to flag version violation. At least an academic discussion would bring in >>>>>> grounded metrics. In reality - the discussion here about V1 versus V2 >>>>>> and >>>>>> that warm fuzzy feeling when your are just a n.m from an n+1 and it's a >>>>>> waste of breath (and that comment goes to all sides of the debate). >>>>>> >>>>>> Give me an endorsed tool that shows me that module [name]-[build] is >>>>>> computationally backward compatible to a different [name]-[build] and >>>>>> then, >>>>>> only them will this discussion have reasonable grounds to assert any >>>>>> authority on the subject of version compatibility (past or future). >>>>>> Without >>>>>> that tool - versioning is a unqualified statement of trust. And the >>>>>> evidence >>>>>> suggests that the threshold of trust delivery on actual computational >>>>>> compliance is really really thin. >>>>>> >>>>>> Cheers, Steve. >>>>>> >>>>>> >>>>>> -----Original Message----- From: cowwoc >>>>>> Sent: Sunday, November 20, 2011 1:39 PM >>>>>> To: jigsaw-dev at openjdk.java.net >>>>>> Subject: Re: Use-cases for version ranges? >>>>>> >>>>>> >>>>>> I'd like to propose another possibility: the author of the >>>>>> dependency should tell *us* about version compatibility, not the other >>>>>> way around. For example: >>>>>> >>>>>> 1. The author of module A declares a dependency on module B version 1.2 >>>>>> (specific version). >>>>>> 2. The author of module B publishes version 1.3. He declares that >>>>>> version 1.3 is compatible with 1.2 (meaning, the runtime system is >>>>>> allows to substitute version 1.3 for 1.2). >>>>>> >>>>>> The upside of this approach is that the author of B is in a better >>>>>> position to declare compatibility than the author of A. The author of A >>>>>> still only needs to test a single version. What do you think? >>>>>> >>>>>> Gili >>>>>> >>>>>> On 19/11/2011 1:59 AM, Neil Bartlett wrote: >>>>>>> >>>>>>> Gili, >>>>>>> >>>>>>> I didn't say anything about guarantees, and in this industry I have >>>>>>> never heard of anybody providing a guarantees about the performance of >>>>>>> their software, especially in the presence of external dependencies. >>>>>>> >>>>>>> Version ranges are a means of communicating expectations, and we >>>>>>> provide both a lower and an upper bound because this is useful >>>>>>> information. I expect my module will work with 1.2.14, and I expect it >>>>>>> will not work with 2.0. If I were a provider of the API rather than a >>>>>>> consumer then I would have a much narrower expectation, e.g. >>>>>>> [1.2,1.3), and this would also be useful information to convey. >>>>>>> >>>>>>> Regards >>>>>>> Neil >>>>>>> >>>>>>> On Sat, Nov 19, 2011 at 4:27 AM, cowwoc >>>>>>> wrote: >>>>>>>> >>>>>>>> Neil, >>>>>>>> >>>>>>>> I guess I don't understand why Jigsaw should work differently from >>>>>>>> Maven >>>>>>>> on this point. I am expecting developers to specify specific versions >>>>>>>> that >>>>>>>> they tested (point versions, not ranges) and end-users may override >>>>>>>> these >>>>>>>> "recommendations" as they see fit. >>>>>>>> >>>>>>>> Where you see version range [1.2.14, 2.0) as a way of communicating >>>>>>>> "the >>>>>>>> developer guarantees 1.2.14 but you may use newer versions up to 2.0 at >>>>>>>> your >>>>>>>> own risk" I'd expect the developer to simply specify 1.2.14 and there >>>>>>>> should >>>>>>>> be no limit on what version end-users may use if they so wish. >>>>>>>> >>>>>>>> Gili >>>>>>>> >>>>>>>> On 18/11/2011 2:23 AM, Neil Bartlett wrote: >>>>>>>>> >>>>>>>>> I noticed that I failed to address your point about Maven using point >>>>>>>>> versions. >>>>>>>>> >>>>>>>>> Maven is a build tool. At build time we need to compile against a >>>>>>>>> single specific version so that we have repeatable builds. In general >>>>>>>>> we should build each module against the lowest version of the library >>>>>>>>> that it can possibly use, and there are no major negative consequences >>>>>>>>> of having several versions of a library at build time (except that >>>>>>>>> Maven has to download a lot!). At runtime however we need to have the >>>>>>>>> flexibility to substitute a single compatible version. >>>>>>>>> >>>>>>>>> Neil >>>>>>>>> >>>>>>>>> On Fri, Nov 18, 2011 at 7:10 AM, Neil Bartlett >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Suppose as the developer of module A, I declare a dependency on >>>>>>>>>> log4j, >>>>>>>>>> exactly version 1.0.0 because I have not tested against log4j 1.0.1, >>>>>>>>>> 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* being >>>>>>>>>> used with log4j version 1.0.1 even if this combinations is later >>>>>>>>>> tested and proven to work by somebody else. In other words, testing >>>>>>>>>> is >>>>>>>>>> important but it doesn't necessarily have to always be done by the >>>>>>>>>> original developer of each module. >>>>>>>>>> >>>>>>>>>> On the other hand let's say I state my dependency using the following >>>>>>>>>> range: [1.2.14, 2.0). This is OSGi syntax and I believe Jigsaw is >>>>>>>>>> following it, and it simply means I accept version 1.2.14 up to but >>>>>>>>>> not including 2.0. Anybody can see that I compiled and tested against >>>>>>>>>> 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 etc. It >>>>>>>>>> does not mean that I *guarantee* my module will work with log4j 1.3 >>>>>>>>>> because that obviously depends on whether the log4j authors accept >>>>>>>>>> and >>>>>>>>>> follow the common semantics of indicating backwards-incompatible >>>>>>>>>> changes with a bump to the first version segment. >>>>>>>>>> >>>>>>>>>> The consequence of trying to lock down imports to a narrow range or >>>>>>>>>> even a point version is that assembling an application becomes very >>>>>>>>>> difficult, and we are forced to deploy many versions of common >>>>>>>>>> libraries concurrently. This is non-optimal, though we can handle it >>>>>>>>>> to some degree via per-module classloaders as in OSGi. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Neil >>>>>>>>>> >>>>>>>>>> On Thu, Nov 17, 2011 at 11:52 PM, cowwoc >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Can someone please explain why modules need to be able to specify >>>>>>>>>>> version >>>>>>>>>>> ranges for dependencies? I believe OSGI allows the specification of >>>>>>>>>>> version >>>>>>>>>>> ranges while Maven allows the specification of individual versions. >>>>>>>>>>> >>>>>>>>>>> The only thing that comes to mind is when module C depends on A and >>>>>>>>>>> B, >>>>>>>>>>> A >>>>>>>>>>> depends on log4j 1.0, and B depends on log4j 1.1. What does C do? Is >>>>>>>>>>> this >>>>>>>>>>> the main use-case for version ranges? >>>>>>>>>>> >>>>>>>>>>> By the sound of it, this is a trust model where developers are told >>>>>>>>>>> that >>>>>>>>>>> log4j 1.x won't break compatibility so they depend on that range >>>>>>>>>>> without >>>>>>>>>>> actually testing against each version (newer versions may be >>>>>>>>>>> released >>>>>>>>>>> after >>>>>>>>>>> their own software). I question whether such a mechanism is better >>>>>>>>>>> or >>>>>>>>>>> worse >>>>>>>>>>> than depending on individual versions which may be overridden at a >>>>>>>>>>> later >>>>>>>>>>> time (a la Maven). On the one hand, you don't need to release a new >>>>>>>>>>> version >>>>>>>>>>> of the application each time a dependency is updated. On the other >>>>>>>>>>> hand, >>>>>>>>>>> no >>>>>>>>>>> one is actually running tests to ensure that the versions are really >>>>>>>>>>> compatible. >>>>>>>>>>> >>>>>>>>>>> Is there a way to get module A to see log4j 1.0 and module B to see >>>>>>>>>>> log4j >>>>>>>>>>> 1.1 (using separate ClassLoaders)? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Gili >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> View this message in context: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html >>>>>>>>>>> Sent from the jigsaw-dev mailing list archive at Nabble.com. >>>> >>>> >>>> -- >>>> - DML > From david.lloyd at redhat.com Mon Nov 21 08:33:17 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 21 Nov 2011 10:33:17 -0600 Subject: Use-cases for version ranges? In-Reply-To: References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> <4ECA6B05.8020001@redhat.com> Message-ID: <4ECA7D4D.9030608@redhat.com> On 11/21/2011 10:11 AM, Neil Bartlett wrote: > David, > I'm likewise tired of hearing these defeatist straw-man arguments. > First, nobody is suggesting that you be forced to use version ranges. > Second, nobody is suggesting that you be forced to design software by > separating interfaces from implementation. Finally, nobody is > suggesting that unmodified legacy code can be trivially modularised in > *any* module system. Trivial? Depends on how you define it. Most artifacts should be able to be modularized as-is however, with minimal changes if any. We adopted an external descriptor strategy for the sole purpose of being able to use modules as is, and we were able to package our 200+ modules just that way, no repackaging needed. So far the only problems we have really run into seem to be legitimate bugs, and one or two libraries which assume a flat classpath. I just glanced at the quick stats for Maven Central. There are 267,924 indexed artifacts at the time of this writing - 31,847 unique artifacts. This is a major reason why people even use Java in the first place. Ideally we would be able to start with at least the possibility of having a somewhat pre-populated repository. > The module system must support both existing software and new > software, of course. You seem suggest that I believe existing software > is unimportant; nothing could be further from the truth. But I also > believe that new software is important and that practices like > interface/implementation separation and semantic versioning will help > make the new software better than the old software. > Therefore I believe that these features must be supported (though not > mandated) in any new module system. You appear to be suggesting that > these features are worthless because they cannot be applied to > existing software, and/or will not be used by all Java developers. > Well so what?? Not worthless, but a question of personal choice. I believe that interface and implementation separation has not been objectively shown to be a superior way to write frameworks. In my experience, sometimes it is and sometimes it is not. I believe that "semantic versioning" isn't going to work because most people who develop Java software will not understand it or not get it right, due in no small part to being a complex idea with a lot of subtlety. In think that empirical testing is the only way to ensure compatibility which will scale. > Regards,Neil > On Mon, Nov 21, 2011 at 3:15 PM, David M. Lloyd wrote: >> I'm tired of hearing pie-in-the sky, "intro to OOP" wishful crap like this. >> Mission one should be supporting *existing* code. The fact is that people >> do *not* always split their API from their implementation, and APIs almost >> *never* consist solely of interfaces. There are many ways of designing an >> API which people consider "right" and I don't see anyone on this list having >> any special insight that would enable them to select the "one true" approach >> to API design. >> >> The fact is that any version scheme or module resolution engine which only >> works for new code which is designed "properly" for it is *going* to fail. >> And there's no reason for such constraints anyway. Version ranges are >> barely useful at all in practice. The only ones who can assert >> compatibility between two libraries of arbitrary version are the ones who >> have tested the two together (along with anything else that uses a >> combination of the two). >> >> The best one can do establish *minimum* compatibility based on available >> features, and test test test. When a module is added to a repository, if it >> is tested by running both its internal unit tests and all available tests of >> all its dependents, then that should be adequate to establish compatibility >> *within a repository*. There are too many factors to establish >> compatibility beyond it. >> >> On 11/20/2011 07:45 AM, Neil Bartlett wrote: >>> >>> Stephen, >>> >>> You again seem to be talking about dependencies on implementation >>> packages. If we depend on APIs consisting primarily of interfaces, >>> then versions and version ranges are reliable and can be completely >>> automated. >>> >>> Regards, >>> Neil >>> >>> >>> On Sun, Nov 20, 2011 at 12:25 PM, Stephen McConnell >>> wrote: >>>> >>>> I could say that the question is academic - but here is the thing - its >>>> not >>>> even academic. >>>> >>>> We talk about versions and yet the platform has no computational >>>> mechanisms >>>> to flag version violation. At least an academic discussion would bring in >>>> grounded metrics. In reality - the discussion here about V1 versus V2 >>>> and >>>> that warm fuzzy feeling when your are just a n.m from an n+1 and it's a >>>> waste of breath (and that comment goes to all sides of the debate). >>>> >>>> Give me an endorsed tool that shows me that module [name]-[build] is >>>> computationally backward compatible to a different [name]-[build] and >>>> then, >>>> only them will this discussion have reasonable grounds to assert any >>>> authority on the subject of version compatibility (past or future). >>>> Without >>>> that tool - versioning is a unqualified statement of trust. And the >>>> evidence >>>> suggests that the threshold of trust delivery on actual computational >>>> compliance is really really thin. >>>> >>>> Cheers, Steve. >>>> >>>> >>>> -----Original Message----- From: cowwoc >>>> Sent: Sunday, November 20, 2011 1:39 PM >>>> To: jigsaw-dev at openjdk.java.net >>>> Subject: Re: Use-cases for version ranges? >>>> >>>> >>>> I'd like to propose another possibility: the author of the >>>> dependency should tell *us* about version compatibility, not the other >>>> way around. For example: >>>> >>>> 1. The author of module A declares a dependency on module B version 1.2 >>>> (specific version). >>>> 2. The author of module B publishes version 1.3. He declares that >>>> version 1.3 is compatible with 1.2 (meaning, the runtime system is >>>> allows to substitute version 1.3 for 1.2). >>>> >>>> The upside of this approach is that the author of B is in a better >>>> position to declare compatibility than the author of A. The author of A >>>> still only needs to test a single version. What do you think? >>>> >>>> Gili >>>> >>>> On 19/11/2011 1:59 AM, Neil Bartlett wrote: >>>>> >>>>> Gili, >>>>> >>>>> I didn't say anything about guarantees, and in this industry I have >>>>> never heard of anybody providing a guarantees about the performance of >>>>> their software, especially in the presence of external dependencies. >>>>> >>>>> Version ranges are a means of communicating expectations, and we >>>>> provide both a lower and an upper bound because this is useful >>>>> information. I expect my module will work with 1.2.14, and I expect it >>>>> will not work with 2.0. If I were a provider of the API rather than a >>>>> consumer then I would have a much narrower expectation, e.g. >>>>> [1.2,1.3), and this would also be useful information to convey. >>>>> >>>>> Regards >>>>> Neil >>>>> >>>>> On Sat, Nov 19, 2011 at 4:27 AM, cowwoc >>>>> wrote: >>>>>> >>>>>> Neil, >>>>>> >>>>>> I guess I don't understand why Jigsaw should work differently from >>>>>> Maven >>>>>> on this point. I am expecting developers to specify specific versions >>>>>> that >>>>>> they tested (point versions, not ranges) and end-users may override >>>>>> these >>>>>> "recommendations" as they see fit. >>>>>> >>>>>> Where you see version range [1.2.14, 2.0) as a way of communicating >>>>>> "the >>>>>> developer guarantees 1.2.14 but you may use newer versions up to 2.0 at >>>>>> your >>>>>> own risk" I'd expect the developer to simply specify 1.2.14 and there >>>>>> should >>>>>> be no limit on what version end-users may use if they so wish. >>>>>> >>>>>> Gili >>>>>> >>>>>> On 18/11/2011 2:23 AM, Neil Bartlett wrote: >>>>>>> >>>>>>> I noticed that I failed to address your point about Maven using point >>>>>>> versions. >>>>>>> >>>>>>> Maven is a build tool. At build time we need to compile against a >>>>>>> single specific version so that we have repeatable builds. In general >>>>>>> we should build each module against the lowest version of the library >>>>>>> that it can possibly use, and there are no major negative consequences >>>>>>> of having several versions of a library at build time (except that >>>>>>> Maven has to download a lot!). At runtime however we need to have the >>>>>>> flexibility to substitute a single compatible version. >>>>>>> >>>>>>> Neil >>>>>>> >>>>>>> On Fri, Nov 18, 2011 at 7:10 AM, Neil Bartlett >>>>>>> wrote: >>>>>>>> >>>>>>>> Suppose as the developer of module A, I declare a dependency on >>>>>>>> log4j, >>>>>>>> exactly version 1.0.0 because I have not tested against log4j 1.0.1, >>>>>>>> 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* being >>>>>>>> used with log4j version 1.0.1 even if this combinations is later >>>>>>>> tested and proven to work by somebody else. In other words, testing >>>>>>>> is >>>>>>>> important but it doesn't necessarily have to always be done by the >>>>>>>> original developer of each module. >>>>>>>> >>>>>>>> On the other hand let's say I state my dependency using the following >>>>>>>> range: [1.2.14, 2.0). This is OSGi syntax and I believe Jigsaw is >>>>>>>> following it, and it simply means I accept version 1.2.14 up to but >>>>>>>> not including 2.0. Anybody can see that I compiled and tested against >>>>>>>> 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 etc. It >>>>>>>> does not mean that I *guarantee* my module will work with log4j 1.3 >>>>>>>> because that obviously depends on whether the log4j authors accept >>>>>>>> and >>>>>>>> follow the common semantics of indicating backwards-incompatible >>>>>>>> changes with a bump to the first version segment. >>>>>>>> >>>>>>>> The consequence of trying to lock down imports to a narrow range or >>>>>>>> even a point version is that assembling an application becomes very >>>>>>>> difficult, and we are forced to deploy many versions of common >>>>>>>> libraries concurrently. This is non-optimal, though we can handle it >>>>>>>> to some degree via per-module classloaders as in OSGi. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Neil >>>>>>>> >>>>>>>> On Thu, Nov 17, 2011 at 11:52 PM, cowwoc >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Can someone please explain why modules need to be able to specify >>>>>>>>> version >>>>>>>>> ranges for dependencies? I believe OSGI allows the specification of >>>>>>>>> version >>>>>>>>> ranges while Maven allows the specification of individual versions. >>>>>>>>> >>>>>>>>> The only thing that comes to mind is when module C depends on A and >>>>>>>>> B, >>>>>>>>> A >>>>>>>>> depends on log4j 1.0, and B depends on log4j 1.1. What does C do? Is >>>>>>>>> this >>>>>>>>> the main use-case for version ranges? >>>>>>>>> >>>>>>>>> By the sound of it, this is a trust model where developers are told >>>>>>>>> that >>>>>>>>> log4j 1.x won't break compatibility so they depend on that range >>>>>>>>> without >>>>>>>>> actually testing against each version (newer versions may be >>>>>>>>> released >>>>>>>>> after >>>>>>>>> their own software). I question whether such a mechanism is better >>>>>>>>> or >>>>>>>>> worse >>>>>>>>> than depending on individual versions which may be overridden at a >>>>>>>>> later >>>>>>>>> time (a la Maven). On the one hand, you don't need to release a new >>>>>>>>> version >>>>>>>>> of the application each time a dependency is updated. On the other >>>>>>>>> hand, >>>>>>>>> no >>>>>>>>> one is actually running tests to ensure that the versions are really >>>>>>>>> compatible. >>>>>>>>> >>>>>>>>> Is there a way to get module A to see log4j 1.0 and module B to see >>>>>>>>> log4j >>>>>>>>> 1.1 (using separate ClassLoaders)? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Gili >>>>>>>>> >>>>>>>>> -- >>>>>>>>> View this message in context: >>>>>>>>> >>>>>>>>> >>>>>>>>> http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html >>>>>>>>> Sent from the jigsaw-dev mailing list archive at Nabble.com. >>>>>>>>> >>>>>> >>>> >>>> >> >> >> -- >> - DML >> -- - DML From cowwoc at bbs.darktech.org Mon Nov 21 11:36:20 2011 From: cowwoc at bbs.darktech.org (cowwoc) Date: Mon, 21 Nov 2011 11:36:20 -0800 (PST) Subject: Use-cases for version ranges? In-Reply-To: <442B574F-466C-4F91-9BD5-FD25DF67E272@pontarelli.com> References: <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> <4ECA6B05.8020001@redhat.com> <4957F10E-1272-465C-82B8-762664881958@pontarelli.com> <04E6880E3AF04DC58FDA3B422DA445C3@gmail.com> <442B574F-466C-4F91-9BD5-FD25DF67E272@pontarelli.com> Message-ID: <4ECAA825.6040804@bbs.darktech.org> One way I could see this working is replacing version ranges with a list of specific versions. Since OSGI's versioning scheme is fixed, expanding the range into specific version numbers is easy to do. To touch upon Brian's point, not only do version ranges come with extra conceptual weight, but they also force us to fix the versioning scheme (which I'm strongly against), or introduce a Comparator, or... The point is, they come at a cost. I'm still of the opinion that you cannot certify compatibility on a range of versions, rather you must specify a list of *individual* versions which the author has tested one by one. Version ranges imply (to me at least) that someone made a guess. Gili On 21/11/2011 1:18 PM, Brian Pontarelli [via jigsaw-dev] wrote: > This is something I had forgotten about because I haven't used OSGi in > quite a while. I'll need to consider it more, but my initial thought > is that these might not co-exist well. Although, perhaps there is a > way to translate OSGi version ranges to a compatibility verification > system under JMS (or any other system that uses compatibility > verification as I've defined it). > > Thanks for reminding me about OSGi's version system. I'll definitely > have to go back and think about it some more. > > -bp > > > On Nov 21, 2011, at 10:50 AM, Neil Bartlett wrote: > > > Brian, > > > > In that case it seems you're prepared to sacrifice a large body of > existing code, i.e. the OSGi bundles that use version ranges already. > That doesn't seem very consistent. > > > > I'm all for pragmatism. Version ranges can always be locked down to > a single point if you desire, but you cannot unlock a point-version > system and turn it into one with ranges. > > > > I also understand that adding excessive features carries conceptual > weight that can make a system harder to use. Conversely omitting > features that are needed makes it *much* harder to use. Version ranges > fall into the latter category because they are actually used in > practice, and you're proposing to take them away from everybody just > because some people don't want to use them. > > > > Regards > > Neil > > > > > > On Monday, 21 November 2011 at 17:37, Brian Pontarelli wrote: > > > >> To answer your "well so what??" question, adding a feature simply > to have a feature seems really dangerous to me. I'd much rather be > pragmatic about what needs to be included to support the current code > written in Java and what needs to be included to allow a module system > to work. > >> > >> I don't see version ranges and certain other items being necessary > features. I also see some features as adding pain points rather than > alleviating them. > >> > >> -bp > >> > >> > >> On Nov 21, 2011, at 9:11 AM, Neil Bartlett wrote: > >> > >>> David, > >>> I'm likewise tired of hearing these defeatist straw-man arguments. > >>> First, nobody is suggesting that you be forced to use version ranges. > >>> Second, nobody is suggesting that you be forced to design software by > >>> separating interfaces from implementation. Finally, nobody is > >>> suggesting that unmodified legacy code can be trivially > modularised in > >>> *any* module system. > >>> The module system must support both existing software and new > >>> software, of course. You seem suggest that I believe existing > software > >>> is unimportant; nothing could be further from the truth. But I also > >>> believe that new software is important and that practices like > >>> interface/implementation separation and semantic versioning will help > >>> make the new software better than the old software. > >>> Therefore I believe that these features must be supported (though not > >>> mandated) in any new module system. You appear to be suggesting that > >>> these features are worthless because they cannot be applied to > >>> existing software, and/or will not be used by all Java developers. > >>> Well so what?? > >>> Regards,Neil > >>> On Mon, Nov 21, 2011 at 3:15 PM, David M. Lloyd <[hidden email] > > wrote: > >>>> I'm tired of hearing pie-in-the sky, "intro to OOP" wishful crap > like this. > >>>> Mission one should be supporting *existing* code. The fact is > that people > >>>> do *not* always split their API from their implementation, and > APIs almost > >>>> *never* consist solely of interfaces. There are many ways of > designing an > >>>> API which people consider "right" and I don't see anyone on this > list having > >>>> any special insight that would enable them to select the "one > true" approach > >>>> to API design. > >>>> > >>>> The fact is that any version scheme or module resolution engine > which only > >>>> works for new code which is designed "properly" for it is *going* > to fail. > >>>> And there's no reason for such constraints anyway. Version ranges > are > >>>> barely useful at all in practice. The only ones who can assert > >>>> compatibility between two libraries of arbitrary version are the > ones who > >>>> have tested the two together (along with anything else that uses a > >>>> combination of the two). > >>>> > >>>> The best one can do establish *minimum* compatibility based on > available > >>>> features, and test test test. When a module is added to a > repository, if it > >>>> is tested by running both its internal unit tests and all > available tests of > >>>> all its dependents, then that should be adequate to establish > compatibility > >>>> *within a repository*. There are too many factors to establish > >>>> compatibility beyond it. > >>>> > >>>> On 11/20/2011 07:45 AM, Neil Bartlett wrote: > >>>>> > >>>>> Stephen, > >>>>> > >>>>> You again seem to be talking about dependencies on implementation > >>>>> packages. If we depend on APIs consisting primarily of interfaces, > >>>>> then versions and version ranges are reliable and can be completely > >>>>> automated. > >>>>> > >>>>> Regards, > >>>>> Neil > >>>>> > >>>>> > >>>>> On Sun, Nov 20, 2011 at 12:25 PM, Stephen McConnell<[hidden > email] > > >>>>> wrote: > >>>>>> > >>>>>> I could say that the question is academic - but here is the > thing - its > >>>>>> not > >>>>>> even academic. > >>>>>> > >>>>>> We talk about versions and yet the platform has no computational > >>>>>> mechanisms > >>>>>> to flag version violation. At least an academic discussion > would bring in > >>>>>> grounded metrics. In reality - the discussion here about V1 > versus V2 > >>>>>> and > >>>>>> that warm fuzzy feeling when your are just a n.m from an n+1 > and it's a > >>>>>> waste of breath (and that comment goes to all sides of the > debate). > >>>>>> > >>>>>> Give me an endorsed tool that shows me that module > [name]-[build] is > >>>>>> computationally backward compatible to a different > [name]-[build] and > >>>>>> then, > >>>>>> only them will this discussion have reasonable grounds to > assert any > >>>>>> authority on the subject of version compatibility (past or > future). > >>>>>> Without > >>>>>> that tool - versioning is a unqualified statement of trust. And > the > >>>>>> evidence > >>>>>> suggests that the threshold of trust delivery on actual > computational > >>>>>> compliance is really really thin. > >>>>>> > >>>>>> Cheers, Steve. > >>>>>> > >>>>>> > >>>>>> -----Original Message----- From: cowwoc > >>>>>> Sent: Sunday, November 20, 2011 1:39 PM > >>>>>> To: [hidden email] > > >>>>>> Subject: Re: Use-cases for version ranges? > >>>>>> > >>>>>> > >>>>>> I'd like to propose another possibility: the author of the > >>>>>> dependency should tell *us* about version compatibility, not > the other > >>>>>> way around. For example: > >>>>>> > >>>>>> 1. The author of module A declares a dependency on module B > version 1.2 > >>>>>> (specific version). > >>>>>> 2. The author of module B publishes version 1.3. He declares that > >>>>>> version 1.3 is compatible with 1.2 (meaning, the runtime system is > >>>>>> allows to substitute version 1.3 for 1.2). > >>>>>> > >>>>>> The upside of this approach is that the author of B is in a better > >>>>>> position to declare compatibility than the author of A. The > author of A > >>>>>> still only needs to test a single version. What do you think? > >>>>>> > >>>>>> Gili > >>>>>> > >>>>>> On 19/11/2011 1:59 AM, Neil Bartlett wrote: > >>>>>>> > >>>>>>> Gili, > >>>>>>> > >>>>>>> I didn't say anything about guarantees, and in this industry I > have > >>>>>>> never heard of anybody providing a guarantees about the > performance of > >>>>>>> their software, especially in the presence of external > dependencies. > >>>>>>> > >>>>>>> Version ranges are a means of communicating expectations, and we > >>>>>>> provide both a lower and an upper bound because this is useful > >>>>>>> information. I expect my module will work with 1.2.14, and I > expect it > >>>>>>> will not work with 2.0. If I were a provider of the API rather > than a > >>>>>>> consumer then I would have a much narrower expectation, e.g. > >>>>>>> [1.2,1.3), and this would also be useful information to convey. > >>>>>>> > >>>>>>> Regards > >>>>>>> Neil > >>>>>>> > >>>>>>> On Sat, Nov 19, 2011 at 4:27 AM, cowwoc<[hidden email] > > > >>>>>>> wrote: > >>>>>>>> > >>>>>>>> Neil, > >>>>>>>> > >>>>>>>> I guess I don't understand why Jigsaw should work differently > from > >>>>>>>> Maven > >>>>>>>> on this point. I am expecting developers to specify specific > versions > >>>>>>>> that > >>>>>>>> they tested (point versions, not ranges) and end-users may > override > >>>>>>>> these > >>>>>>>> "recommendations" as they see fit. > >>>>>>>> > >>>>>>>> Where you see version range [1.2.14, 2.0) as a way of > communicating > >>>>>>>> "the > >>>>>>>> developer guarantees 1.2.14 but you may use newer versions up > to 2.0 at > >>>>>>>> your > >>>>>>>> own risk" I'd expect the developer to simply specify 1.2.14 > and there > >>>>>>>> should > >>>>>>>> be no limit on what version end-users may use if they so wish. > >>>>>>>> > >>>>>>>> Gili > >>>>>>>> > >>>>>>>> On 18/11/2011 2:23 AM, Neil Bartlett wrote: > >>>>>>>>> > >>>>>>>>> I noticed that I failed to address your point about Maven > using point > >>>>>>>>> versions. > >>>>>>>>> > >>>>>>>>> Maven is a build tool. At build time we need to compile > against a > >>>>>>>>> single specific version so that we have repeatable builds. > In general > >>>>>>>>> we should build each module against the lowest version of > the library > >>>>>>>>> that it can possibly use, and there are no major negative > consequences > >>>>>>>>> of having several versions of a library at build time > (except that > >>>>>>>>> Maven has to download a lot!). At runtime however we need to > have the > >>>>>>>>> flexibility to substitute a single compatible version. > >>>>>>>>> > >>>>>>>>> Neil > >>>>>>>>> > >>>>>>>>> On Fri, Nov 18, 2011 at 7:10 AM, Neil Bartlett<[hidden > email] > > >>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> Suppose as the developer of module A, I declare a > dependency on > >>>>>>>>>> log4j, > >>>>>>>>>> exactly version 1.0.0 because I have not tested against > log4j 1.0.1, > >>>>>>>>>> 1.0.2, 1.3, 999.999 etc. I effectively prevent my module > *ever* being > >>>>>>>>>> used with log4j version 1.0.1 even if this combinations is > later > >>>>>>>>>> tested and proven to work by somebody else. In other words, > testing > >>>>>>>>>> is > >>>>>>>>>> important but it doesn't necessarily have to always be done > by the > >>>>>>>>>> original developer of each module. > >>>>>>>>>> > >>>>>>>>>> On the other hand let's say I state my dependency using the > following > >>>>>>>>>> range: [1.2.14, 2.0). This is OSGi syntax and I believe > Jigsaw is > >>>>>>>>>> following it, and it simply means I accept version 1.2.14 > up to but > >>>>>>>>>> not including 2.0. Anybody can see that I compiled and > tested against > >>>>>>>>>> 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, > 1.9 etc. It > >>>>>>>>>> does not mean that I *guarantee* my module will work with > log4j 1.3 > >>>>>>>>>> because that obviously depends on whether the log4j authors > accept > >>>>>>>>>> and > >>>>>>>>>> follow the common semantics of indicating > backwards-incompatible > >>>>>>>>>> changes with a bump to the first version segment. > >>>>>>>>>> > >>>>>>>>>> The consequence of trying to lock down imports to a narrow > range or > >>>>>>>>>> even a point version is that assembling an application > becomes very > >>>>>>>>>> difficult, and we are forced to deploy many versions of common > >>>>>>>>>> libraries concurrently. This is non-optimal, though we can > handle it > >>>>>>>>>> to some degree via per-module classloaders as in OSGi. > >>>>>>>>>> > >>>>>>>>>> Regards, > >>>>>>>>>> Neil > >>>>>>>>>> > >>>>>>>>>> On Thu, Nov 17, 2011 at 11:52 PM, cowwoc<[hidden email] > > > >>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Can someone please explain why modules need to be able to > specify > >>>>>>>>>>> version > >>>>>>>>>>> ranges for dependencies? I believe OSGI allows the > specification of > >>>>>>>>>>> version > >>>>>>>>>>> ranges while Maven allows the specification of individual > versions. > >>>>>>>>>>> > >>>>>>>>>>> The only thing that comes to mind is when module C depends > on A and > >>>>>>>>>>> B, > >>>>>>>>>>> A > >>>>>>>>>>> depends on log4j 1.0, and B depends on log4j 1.1. What > does C do? Is > >>>>>>>>>>> this > >>>>>>>>>>> the main use-case for version ranges? > >>>>>>>>>>> > >>>>>>>>>>> By the sound of it, this is a trust model where developers > are told > >>>>>>>>>>> that > >>>>>>>>>>> log4j 1.x won't break compatibility so they depend on that > range > >>>>>>>>>>> without > >>>>>>>>>>> actually testing against each version (newer versions may be > >>>>>>>>>>> released > >>>>>>>>>>> after > >>>>>>>>>>> their own software). I question whether such a mechanism > is better > >>>>>>>>>>> or > >>>>>>>>>>> worse > >>>>>>>>>>> than depending on individual versions which may be > overridden at a > >>>>>>>>>>> later > >>>>>>>>>>> time (a la Maven). On the one hand, you don't need to > release a new > >>>>>>>>>>> version > >>>>>>>>>>> of the application each time a dependency is updated. On > the other > >>>>>>>>>>> hand, > >>>>>>>>>>> no > >>>>>>>>>>> one is actually running tests to ensure that the versions > are really > >>>>>>>>>>> compatible. > >>>>>>>>>>> > >>>>>>>>>>> Is there a way to get module A to see log4j 1.0 and module > B to see > >>>>>>>>>>> log4j > >>>>>>>>>>> 1.1 (using separate ClassLoaders)? > >>>>>>>>>>> > >>>>>>>>>>> Thanks, > >>>>>>>>>>> Gili > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> View this message in context: > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html > >>>>>>>>>>> Sent from the jigsaw-dev mailing list archive at Nabble.com. > >>>> > >>>> > >>>> -- > >>>> - DML > > > > > > ------------------------------------------------------------------------ > If you reply to this email, your message will be added to the > discussion below: > http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5011266.html > > To unsubscribe from Use-cases for version ranges?, click here > . > NAML > > -- View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5011446.html Sent from the jigsaw-dev mailing list archive at Nabble.com. From cowwoc at bbs.darktech.org Mon Nov 21 11:41:43 2011 From: cowwoc at bbs.darktech.org (cowwoc) Date: Mon, 21 Nov 2011 11:41:43 -0800 (PST) Subject: Use-cases for version ranges? In-Reply-To: <4ECA2B0E.40809@tibco.com> References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> <4ECA2B0E.40809@tibco.com> Message-ID: <4ECAA968.6040709@bbs.darktech.org> Hi Eric, On 21/11/2011 5:43 AM, Eric Johnson [via jigsaw-dev] wrote: > Trouble is, the Java community doesn't seem to like the idea of a new > named interface with the additional of even one method to an interface. > But is that perhaps just because the JDK has acculturated people to the > idea that they don't need to do that? It seems to me that any version > scheme that attempts to cover up this problem will likely just add to > the confusion, rather than resolve the problem. We either have to stick > a stake in the ground and declare what we mean by backward compatible, > or admit that there isn't a single notion, and therefore version ranges > just mean what people want them to mean. My problem with this approach is java.util.Collections followed by Collections2 in Guava. Another example, Futures vs MoreFutures in Guava. These are examples of class subclassing, not interface extension, but the problem is the same: once you pick a good name it's extremely difficult to come up with yet another good name for the same thing. How many times do you think you could extend an interface before you run out of names? Twice? Three times maybe? I'm also tired of Java features having their designs crippled in the name of backwards-compatibility. I'm personally in favor of each author declaring a backwards compatibility policy and breaking compatibility as necessary. In the case of the JDK, you can imagine removing deprecated methods after 5 years. Projects like Guava give you 6 months. The point is that you can't keep backwards compatibility forever, especially in light of the requirement you mentioned that developers may not add any new methods ;) Gili -- View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5011460.html Sent from the jigsaw-dev mailing list archive at Nabble.com. From alex.buckley at oracle.com Mon Nov 21 11:49:49 2011 From: alex.buckley at oracle.com (Alex Buckley) Date: Mon, 21 Nov 2011 11:49:49 -0800 Subject: Use-cases for version ranges? In-Reply-To: <4ECAA968.6040709@bbs.darktech.org> References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> <4ECA2B0E.40809@tibco.com> <4ECAA968.6040709@bbs.darktech.org> Message-ID: <4ECAAB5D.2040101@oracle.com> Hi Gili and Eric, Java SE 8 is extremely likely to support the addition of methods to interfaces in a manner which is binary-compatible _and_ source-compatible. See slide 52 onwards in http://blogs.oracle.com/abuckley/resource/QConSF2011-JavaSE78.pdf. How does this affect your view of semantic compatibility and version ranges? Alex On 11/21/2011 11:41 AM, cowwoc wrote: > Hi Eric, > > On 21/11/2011 5:43 AM, Eric Johnson [via jigsaw-dev] wrote: >> Trouble is, the Java community doesn't seem to like the idea of a new >> named interface with the additional of even one method to an interface. >> But is that perhaps just because the JDK has acculturated people to the >> idea that they don't need to do that? It seems to me that any version >> scheme that attempts to cover up this problem will likely just add to >> the confusion, rather than resolve the problem. We either have to stick >> a stake in the ground and declare what we mean by backward compatible, >> or admit that there isn't a single notion, and therefore version ranges >> just mean what people want them to mean. > > My problem with this approach is java.util.Collections followed by > Collections2 in Guava. Another example, Futures vs MoreFutures in Guava. > These are examples of class subclassing, not interface extension, but > the problem is the same: once you pick a good name it's extremely > difficult to come up with yet another good name for the same thing. How > many times do you think you could extend an interface before you run out > of names? Twice? Three times maybe? > > I'm also tired of Java features having their designs crippled in > the name of backwards-compatibility. I'm personally in favor of each > author declaring a backwards compatibility policy and breaking > compatibility as necessary. In the case of the JDK, you can imagine > removing deprecated methods after 5 years. Projects like Guava give you > 6 months. The point is that you can't keep backwards compatibility > forever, especially in light of the requirement you mentioned that > developers may not add any new methods ;) > > Gili From david.bosschaert at gmail.com Mon Nov 21 11:57:02 2011 From: david.bosschaert at gmail.com (David Bosschaert) Date: Mon, 21 Nov 2011 19:57:02 +0000 Subject: Use-cases for version ranges? In-Reply-To: <4ECA7D4D.9030608@redhat.com> References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> <4ECA6B05.8020001@redhat.com> <4ECA7D4D.9030608@redhat.com> Message-ID: I think that the fact that in JBoss Modules the metadata is *outside* of the module (in a separate file that sits alongside the module) and in OSGi the metadata is *inside* the module (embedded in the Jar) is a significant difference. If you have the metadata outside of the module you have additional flexibility. You can change the metadata without changing the module itself, so if there is an issue with the metadata you can easily fix it up (e.g. if you need to depend on another version of your dependency). Also there are obviously impacts of changing the module metadata if its inside the module and you are working with signed jars. On the other side, having the metadata inside the module has the benefit that you only need to worry about a single artifact: the module itself. I'm not advocating one over the other here; I think both have benefits, but I think that using version ranges makes more sense in the case the metadata is embedded in the module. If I'm not mistaken, the model that Jigsaw is going for is currently also the model where the metadata is inside the module. David On 21 November 2011 16:33, David M. Lloyd wrote: > On 11/21/2011 10:11 AM, Neil Bartlett wrote: >> >> David, >> I'm likewise tired of hearing these defeatist straw-man arguments. >> First, nobody is suggesting that you be forced to use version ranges. >> Second, nobody is suggesting that you be forced to design software by >> separating interfaces from implementation. Finally, nobody is >> suggesting that unmodified legacy code can be trivially modularised in >> *any* module system. > > Trivial? ?Depends on how you define it. ?Most artifacts should be able to be > modularized as-is however, with minimal changes if any. ?We adopted an > external descriptor strategy for the sole purpose of being able to use > modules as is, and we were able to package our 200+ modules just that way, > no repackaging needed. ?So far the only problems we have really run into > seem to be legitimate bugs, and one or two libraries which assume a flat > classpath. > > I just glanced at the quick stats for Maven Central. ?There are 267,924 > indexed artifacts at the time of this writing - 31,847 unique artifacts. > ?This is a major reason why people even use Java in the first place. Ideally > we would be able to start with at least the possibility of having a somewhat > pre-populated repository. > >> The module system must support both existing software and new >> software, of course. You seem suggest that I believe existing software >> is unimportant; nothing could be further from the truth. But I also >> believe that new software is important and that practices like >> interface/implementation separation and semantic versioning will help >> make the new software better than the old software. >> Therefore I believe that these features must be supported (though not >> mandated) in any new module system. You appear to be suggesting that >> these features are worthless because they cannot be applied to >> existing software, and/or will not be used by all Java developers. >> Well so what?? > > Not worthless, but a question of personal choice. ?I believe that interface > and implementation separation has not been objectively shown to be a > superior way to write frameworks. ?In my experience, sometimes it is and > sometimes it is not. > > I believe that "semantic versioning" isn't going to work because most people > who develop Java software will not understand it or not get it right, due in > no small part to being a complex idea with a lot of subtlety. ?In think that > empirical testing is the only way to ensure compatibility which will scale. > >> Regards,Neil >> On Mon, Nov 21, 2011 at 3:15 PM, David M. Lloyd >> ?wrote: >>> >>> I'm tired of hearing pie-in-the sky, "intro to OOP" wishful crap like >>> this. >>> ?Mission one should be supporting *existing* code. ?The fact is that >>> people >>> do *not* always split their API from their implementation, and APIs >>> almost >>> *never* consist solely of interfaces. ?There are many ways of designing >>> an >>> API which people consider "right" and I don't see anyone on this list >>> having >>> any special insight that would enable them to select the "one true" >>> approach >>> to API design. >>> >>> The fact is that any version scheme or module resolution engine which >>> only >>> works for new code which is designed "properly" for it is *going* to >>> fail. >>> ?And there's no reason for such constraints anyway. ?Version ranges are >>> barely useful at all in practice. ?The only ones who can assert >>> compatibility between two libraries of arbitrary version are the ones who >>> have tested the two together (along with anything else that uses a >>> combination of the two). >>> >>> The best one can do establish *minimum* compatibility based on available >>> features, and test test test. ?When a module is added to a repository, if >>> it >>> is tested by running both its internal unit tests and all available tests >>> of >>> all its dependents, then that should be adequate to establish >>> compatibility >>> *within a repository*. ?There are too many factors to establish >>> compatibility beyond it. >>> >>> On 11/20/2011 07:45 AM, Neil Bartlett wrote: >>>> >>>> Stephen, >>>> >>>> You again seem to be talking about dependencies on implementation >>>> packages. If we depend on APIs consisting primarily of interfaces, >>>> then versions and version ranges are reliable and can be completely >>>> automated. >>>> >>>> Regards, >>>> Neil >>>> >>>> >>>> On Sun, Nov 20, 2011 at 12:25 PM, Stephen McConnell >>>> ?wrote: >>>>> >>>>> I could say that the question is academic - but here is the thing - its >>>>> not >>>>> even academic. >>>>> >>>>> We talk about versions and yet the platform has no computational >>>>> mechanisms >>>>> to flag version violation. At least an academic discussion would bring >>>>> in >>>>> grounded metrics. ?In reality - the discussion here about V1 versus V2 >>>>> and >>>>> that warm fuzzy feeling when your are just a n.m from an n+1 and it's a >>>>> waste of breath (and that comment goes to all sides of the debate). >>>>> >>>>> Give me an endorsed tool that shows me that module [name]-[build] is >>>>> computationally backward compatible to a different [name]-[build] and >>>>> then, >>>>> only them will this discussion have reasonable grounds to assert any >>>>> authority on the subject of version compatibility (past or future). >>>>> Without >>>>> that tool - versioning is a unqualified statement of trust. And the >>>>> evidence >>>>> suggests that the threshold of trust delivery on actual computational >>>>> compliance is really really thin. >>>>> >>>>> Cheers, Steve. >>>>> >>>>> >>>>> -----Original Message----- From: cowwoc >>>>> Sent: Sunday, November 20, 2011 1:39 PM >>>>> To: jigsaw-dev at openjdk.java.net >>>>> Subject: Re: Use-cases for version ranges? >>>>> >>>>> >>>>> ? ?I'd like to propose another possibility: the author of the >>>>> dependency should tell *us* about version compatibility, not the other >>>>> way around. For example: >>>>> >>>>> 1. The author of module A declares a dependency on module B version 1.2 >>>>> (specific version). >>>>> 2. The author of module B publishes version 1.3. He declares that >>>>> version 1.3 is compatible with 1.2 (meaning, the runtime system is >>>>> allows to substitute version 1.3 for 1.2). >>>>> >>>>> ? ?The upside of this approach is that the author of B is in a better >>>>> position to declare compatibility than the author of A. The author of A >>>>> still only needs to test a single version. What do you think? >>>>> >>>>> Gili >>>>> >>>>> On 19/11/2011 1:59 AM, Neil Bartlett wrote: >>>>>> >>>>>> Gili, >>>>>> >>>>>> I didn't say anything about guarantees, and in this industry I have >>>>>> never heard of anybody providing a guarantees about the performance of >>>>>> their software, especially in the presence of external dependencies. >>>>>> >>>>>> Version ranges are a means of communicating expectations, and we >>>>>> provide both a lower and an upper bound because this is useful >>>>>> information. I expect my module will work with 1.2.14, and I expect it >>>>>> will not work with 2.0. If I were a provider of the API rather than a >>>>>> consumer then I would have a much narrower expectation, ?e.g. >>>>>> [1.2,1.3), and this would also be useful information to convey. >>>>>> >>>>>> Regards >>>>>> Neil >>>>>> >>>>>> On Sat, Nov 19, 2011 at 4:27 AM, cowwoc >>>>>> ?wrote: >>>>>>> >>>>>>> Neil, >>>>>>> >>>>>>> ? ?I guess I don't understand why Jigsaw should work differently from >>>>>>> Maven >>>>>>> on this point. I am expecting developers to specify specific versions >>>>>>> that >>>>>>> they tested (point versions, not ranges) and end-users may override >>>>>>> these >>>>>>> "recommendations" as they see fit. >>>>>>> >>>>>>> ? ?Where you see version range [1.2.14, 2.0) as a way of >>>>>>> communicating >>>>>>> "the >>>>>>> developer guarantees 1.2.14 but you may use newer versions up to 2.0 >>>>>>> at >>>>>>> your >>>>>>> own risk" I'd expect the developer to simply specify 1.2.14 and there >>>>>>> should >>>>>>> be no limit on what version end-users may use if they so wish. >>>>>>> >>>>>>> Gili >>>>>>> >>>>>>> On 18/11/2011 2:23 AM, Neil Bartlett wrote: >>>>>>>> >>>>>>>> I noticed that I failed to address your point about Maven using >>>>>>>> point >>>>>>>> versions. >>>>>>>> >>>>>>>> Maven is a build tool. At build time we need to compile against a >>>>>>>> single specific version so that we have repeatable builds. In >>>>>>>> general >>>>>>>> we should build each module against the lowest version of the >>>>>>>> library >>>>>>>> that it can possibly use, and there are no major negative >>>>>>>> consequences >>>>>>>> of having several versions of a library at build time (except that >>>>>>>> Maven has to download a lot!). At runtime however we need to have >>>>>>>> the >>>>>>>> flexibility to substitute a single compatible version. >>>>>>>> >>>>>>>> Neil >>>>>>>> >>>>>>>> On Fri, Nov 18, 2011 at 7:10 AM, Neil Bartlett >>>>>>>> ?wrote: >>>>>>>>> >>>>>>>>> Suppose as the developer of module A, I declare a dependency on >>>>>>>>> log4j, >>>>>>>>> exactly version 1.0.0 because I have not tested against log4j >>>>>>>>> 1.0.1, >>>>>>>>> 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* >>>>>>>>> being >>>>>>>>> used with log4j version 1.0.1 even if this combinations is later >>>>>>>>> tested and proven to work by somebody else. In other words, testing >>>>>>>>> is >>>>>>>>> important but it doesn't necessarily have to always be done by the >>>>>>>>> original developer of each module. >>>>>>>>> >>>>>>>>> On the other hand let's say I state my dependency using the >>>>>>>>> following >>>>>>>>> range: [1.2.14, 2.0). This is OSGi syntax and I believe Jigsaw is >>>>>>>>> following it, and it simply means I accept version 1.2.14 up to but >>>>>>>>> not including 2.0. Anybody can see that I compiled and tested >>>>>>>>> against >>>>>>>>> 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 etc. >>>>>>>>> It >>>>>>>>> does not mean that I *guarantee* my module will work with log4j 1.3 >>>>>>>>> because that obviously depends on whether the log4j authors accept >>>>>>>>> and >>>>>>>>> follow the common semantics of indicating backwards-incompatible >>>>>>>>> changes with a bump to the first version segment. >>>>>>>>> >>>>>>>>> The consequence of trying to lock down imports to a narrow range or >>>>>>>>> even a point version is that assembling an application becomes very >>>>>>>>> difficult, and we are forced to deploy many versions of common >>>>>>>>> libraries concurrently. This is non-optimal, though we can handle >>>>>>>>> it >>>>>>>>> to some degree via per-module classloaders as in OSGi. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Neil >>>>>>>>> >>>>>>>>> On Thu, Nov 17, 2011 at 11:52 PM, cowwoc >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Can someone please explain why modules need to be able to specify >>>>>>>>>> version >>>>>>>>>> ranges for dependencies? I believe OSGI allows the specification >>>>>>>>>> of >>>>>>>>>> version >>>>>>>>>> ranges while Maven allows the specification of individual >>>>>>>>>> versions. >>>>>>>>>> >>>>>>>>>> The only thing that comes to mind is when module C depends on A >>>>>>>>>> and >>>>>>>>>> B, >>>>>>>>>> A >>>>>>>>>> depends on log4j 1.0, and B depends on log4j 1.1. What does C do? >>>>>>>>>> Is >>>>>>>>>> this >>>>>>>>>> the main use-case for version ranges? >>>>>>>>>> >>>>>>>>>> By the sound of it, this is a trust model where developers are >>>>>>>>>> told >>>>>>>>>> that >>>>>>>>>> log4j 1.x won't break compatibility so they depend on that range >>>>>>>>>> without >>>>>>>>>> actually testing against each version (newer versions may be >>>>>>>>>> released >>>>>>>>>> after >>>>>>>>>> their own software). I question whether such a mechanism is better >>>>>>>>>> or >>>>>>>>>> worse >>>>>>>>>> than depending on individual versions which may be overridden at a >>>>>>>>>> later >>>>>>>>>> time (a la Maven). On the one hand, you don't need to release a >>>>>>>>>> new >>>>>>>>>> version >>>>>>>>>> of the application each time a dependency is updated. On the other >>>>>>>>>> hand, >>>>>>>>>> no >>>>>>>>>> one is actually running tests to ensure that the versions are >>>>>>>>>> really >>>>>>>>>> compatible. >>>>>>>>>> >>>>>>>>>> Is there a way to get module A to see log4j 1.0 and module B to >>>>>>>>>> see >>>>>>>>>> log4j >>>>>>>>>> 1.1 (using separate ClassLoaders)? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Gili >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> View this message in context: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html >>>>>>>>>> Sent from the jigsaw-dev mailing list archive at Nabble.com. >>>>>>>>>> >>>>>>> >>>>> >>>>> >>> >>> >>> -- >>> - DML >>> > > > -- > - DML > From cowwoc at bbs.darktech.org Mon Nov 21 12:16:39 2011 From: cowwoc at bbs.darktech.org (cowwoc) Date: Mon, 21 Nov 2011 12:16:39 -0800 (PST) Subject: Use-cases for version ranges? In-Reply-To: References: <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> <4ECA6B05.8020001@redhat.com> <4ECA7D4D.9030608@redhat.com> Message-ID: <4ECAB192.1010302@bbs.darktech.org> The reason I advocate placing the meta-data outside of the module is that people make mistakes. What happens if you specify a version range that turns out to be incorrect? You need to be able to fix these kinds of things after the fact. Gili On 21/11/2011 3:11 PM, David Bosschaert [via jigsaw-dev] wrote: > I think that the fact that in JBoss Modules the metadata is *outside* > of the module (in a separate file that sits alongside the module) and > in OSGi the metadata is *inside* the module (embedded in the Jar) is a > significant difference. > > If you have the metadata outside of the module you have additional > flexibility. You can change the metadata without changing the module > itself, so if there is an issue with the metadata you can easily fix > it up (e.g. if you need to depend on another version of your > dependency). Also there are obviously impacts of changing the module > metadata if its inside the module and you are working with signed > jars. On the other side, having the metadata inside the module has the > benefit that you only need to worry about a single artifact: the > module itself. > > I'm not advocating one over the other here; I think both have > benefits, but I think that using version ranges makes more sense in > the case the metadata is embedded in the module. > > If I'm not mistaken, the model that Jigsaw is going for is currently > also the model where the metadata is inside the module. > > David > > On 21 November 2011 16:33, David M. Lloyd <[hidden email] > > wrote: > > > On 11/21/2011 10:11 AM, Neil Bartlett wrote: > >> > >> David, > >> I'm likewise tired of hearing these defeatist straw-man arguments. > >> First, nobody is suggesting that you be forced to use version ranges. > >> Second, nobody is suggesting that you be forced to design software by > >> separating interfaces from implementation. Finally, nobody is > >> suggesting that unmodified legacy code can be trivially modularised in > >> *any* module system. > > > > Trivial? Depends on how you define it. Most artifacts should be > able to be > > modularized as-is however, with minimal changes if any. We adopted an > > external descriptor strategy for the sole purpose of being able to use > > modules as is, and we were able to package our 200+ modules just > that way, > > no repackaging needed. So far the only problems we have really run > into > > seem to be legitimate bugs, and one or two libraries which assume a > flat > > classpath. > > > > I just glanced at the quick stats for Maven Central. There are 267,924 > > indexed artifacts at the time of this writing - 31,847 unique > artifacts. > > This is a major reason why people even use Java in the first place. > Ideally > > we would be able to start with at least the possibility of having a > somewhat > > pre-populated repository. > > > >> The module system must support both existing software and new > >> software, of course. You seem suggest that I believe existing software > >> is unimportant; nothing could be further from the truth. But I also > >> believe that new software is important and that practices like > >> interface/implementation separation and semantic versioning will help > >> make the new software better than the old software. > >> Therefore I believe that these features must be supported (though not > >> mandated) in any new module system. You appear to be suggesting that > >> these features are worthless because they cannot be applied to > >> existing software, and/or will not be used by all Java developers. > >> Well so what?? > > > > Not worthless, but a question of personal choice. I believe that > interface > > and implementation separation has not been objectively shown to be a > > superior way to write frameworks. In my experience, sometimes it is > and > > sometimes it is not. > > > > I believe that "semantic versioning" isn't going to work because > most people > > who develop Java software will not understand it or not get it > right, due in > > no small part to being a complex idea with a lot of subtlety. In > think that > > empirical testing is the only way to ensure compatibility which will > scale. > > > >> Regards,Neil > >> On Mon, Nov 21, 2011 at 3:15 PM, David M. Lloyd<[hidden email] > > > >> wrote: > >>> > >>> I'm tired of hearing pie-in-the sky, "intro to OOP" wishful crap like > >>> this. > >>> Mission one should be supporting *existing* code. The fact is that > >>> people > >>> do *not* always split their API from their implementation, and APIs > >>> almost > >>> *never* consist solely of interfaces. There are many ways of > designing > >>> an > >>> API which people consider "right" and I don't see anyone on this list > >>> having > >>> any special insight that would enable them to select the "one true" > >>> approach > >>> to API design. > >>> > >>> The fact is that any version scheme or module resolution engine which > >>> only > >>> works for new code which is designed "properly" for it is *going* to > >>> fail. > >>> And there's no reason for such constraints anyway. Version > ranges are > >>> barely useful at all in practice. The only ones who can assert > >>> compatibility between two libraries of arbitrary version are the > ones who > >>> have tested the two together (along with anything else that uses a > >>> combination of the two). > >>> > >>> The best one can do establish *minimum* compatibility based on > available > >>> features, and test test test. When a module is added to a > repository, if > >>> it > >>> is tested by running both its internal unit tests and all > available tests > >>> of > >>> all its dependents, then that should be adequate to establish > >>> compatibility > >>> *within a repository*. There are too many factors to establish > >>> compatibility beyond it. > >>> > >>> On 11/20/2011 07:45 AM, Neil Bartlett wrote: > >>>> > >>>> Stephen, > >>>> > >>>> You again seem to be talking about dependencies on implementation > >>>> packages. If we depend on APIs consisting primarily of interfaces, > >>>> then versions and version ranges are reliable and can be completely > >>>> automated. > >>>> > >>>> Regards, > >>>> Neil > >>>> > >>>> > >>>> On Sun, Nov 20, 2011 at 12:25 PM, Stephen McConnell<[hidden > email] > > >>>> wrote: > >>>>> > >>>>> I could say that the question is academic - but here is the > thing - its > >>>>> not > >>>>> even academic. > >>>>> > >>>>> We talk about versions and yet the platform has no computational > >>>>> mechanisms > >>>>> to flag version violation. At least an academic discussion would > bring > >>>>> in > >>>>> grounded metrics. In reality - the discussion here about V1 > versus V2 > >>>>> and > >>>>> that warm fuzzy feeling when your are just a n.m from an n+1 and > it's a > >>>>> waste of breath (and that comment goes to all sides of the debate). > >>>>> > >>>>> Give me an endorsed tool that shows me that module > [name]-[build] is > >>>>> computationally backward compatible to a different > [name]-[build] and > >>>>> then, > >>>>> only them will this discussion have reasonable grounds to assert > any > >>>>> authority on the subject of version compatibility (past or future). > >>>>> Without > >>>>> that tool - versioning is a unqualified statement of trust. And the > >>>>> evidence > >>>>> suggests that the threshold of trust delivery on actual > computational > >>>>> compliance is really really thin. > >>>>> > >>>>> Cheers, Steve. > >>>>> > >>>>> > >>>>> -----Original Message----- From: cowwoc > >>>>> Sent: Sunday, November 20, 2011 1:39 PM > >>>>> To: [hidden email] > >>>>> Subject: Re: Use-cases for version ranges? > >>>>> > >>>>> > >>>>> I'd like to propose another possibility: the author of the > >>>>> dependency should tell *us* about version compatibility, not the > other > >>>>> way around. For example: > >>>>> > >>>>> 1. The author of module A declares a dependency on module B > version 1.2 > >>>>> (specific version). > >>>>> 2. The author of module B publishes version 1.3. He declares that > >>>>> version 1.3 is compatible with 1.2 (meaning, the runtime system is > >>>>> allows to substitute version 1.3 for 1.2). > >>>>> > >>>>> The upside of this approach is that the author of B is in a > better > >>>>> position to declare compatibility than the author of A. The > author of A > >>>>> still only needs to test a single version. What do you think? > >>>>> > >>>>> Gili > >>>>> > >>>>> On 19/11/2011 1:59 AM, Neil Bartlett wrote: > >>>>>> > >>>>>> Gili, > >>>>>> > >>>>>> I didn't say anything about guarantees, and in this industry I > have > >>>>>> never heard of anybody providing a guarantees about the > performance of > >>>>>> their software, especially in the presence of external > dependencies. > >>>>>> > >>>>>> Version ranges are a means of communicating expectations, and we > >>>>>> provide both a lower and an upper bound because this is useful > >>>>>> information. I expect my module will work with 1.2.14, and I > expect it > >>>>>> will not work with 2.0. If I were a provider of the API rather > than a > >>>>>> consumer then I would have a much narrower expectation, e.g. > >>>>>> [1.2,1.3), and this would also be useful information to convey. > >>>>>> > >>>>>> Regards > >>>>>> Neil > >>>>>> > >>>>>> On Sat, Nov 19, 2011 at 4:27 AM, cowwoc<[hidden email] > > > >>>>>> wrote: > >>>>>>> > >>>>>>> Neil, > >>>>>>> > >>>>>>> I guess I don't understand why Jigsaw should work > differently from > >>>>>>> Maven > >>>>>>> on this point. I am expecting developers to specify specific > versions > >>>>>>> that > >>>>>>> they tested (point versions, not ranges) and end-users may > override > >>>>>>> these > >>>>>>> "recommendations" as they see fit. > >>>>>>> > >>>>>>> Where you see version range [1.2.14, 2.0) as a way of > >>>>>>> communicating > >>>>>>> "the > >>>>>>> developer guarantees 1.2.14 but you may use newer versions up > to 2.0 > >>>>>>> at > >>>>>>> your > >>>>>>> own risk" I'd expect the developer to simply specify 1.2.14 > and there > >>>>>>> should > >>>>>>> be no limit on what version end-users may use if they so wish. > >>>>>>> > >>>>>>> Gili > >>>>>>> > >>>>>>> On 18/11/2011 2:23 AM, Neil Bartlett wrote: > >>>>>>>> > >>>>>>>> I noticed that I failed to address your point about Maven using > >>>>>>>> point > >>>>>>>> versions. > >>>>>>>> > >>>>>>>> Maven is a build tool. At build time we need to compile > against a > >>>>>>>> single specific version so that we have repeatable builds. In > >>>>>>>> general > >>>>>>>> we should build each module against the lowest version of the > >>>>>>>> library > >>>>>>>> that it can possibly use, and there are no major negative > >>>>>>>> consequences > >>>>>>>> of having several versions of a library at build time (except > that > >>>>>>>> Maven has to download a lot!). At runtime however we need to > have > >>>>>>>> the > >>>>>>>> flexibility to substitute a single compatible version. > >>>>>>>> > >>>>>>>> Neil > >>>>>>>> > >>>>>>>> On Fri, Nov 18, 2011 at 7:10 AM, Neil Bartlett<[hidden email] > > > >>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>> Suppose as the developer of module A, I declare a dependency on > >>>>>>>>> log4j, > >>>>>>>>> exactly version 1.0.0 because I have not tested against log4j > >>>>>>>>> 1.0.1, > >>>>>>>>> 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* > >>>>>>>>> being > >>>>>>>>> used with log4j version 1.0.1 even if this combinations is > later > >>>>>>>>> tested and proven to work by somebody else. In other words, > testing > >>>>>>>>> is > >>>>>>>>> important but it doesn't necessarily have to always be done > by the > >>>>>>>>> original developer of each module. > >>>>>>>>> > >>>>>>>>> On the other hand let's say I state my dependency using the > >>>>>>>>> following > >>>>>>>>> range: [1.2.14, 2.0). This is OSGi syntax and I believe > Jigsaw is > >>>>>>>>> following it, and it simply means I accept version 1.2.14 up > to but > >>>>>>>>> not including 2.0. Anybody can see that I compiled and tested > >>>>>>>>> against > >>>>>>>>> 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 > etc. > >>>>>>>>> It > >>>>>>>>> does not mean that I *guarantee* my module will work with > log4j 1.3 > >>>>>>>>> because that obviously depends on whether the log4j authors > accept > >>>>>>>>> and > >>>>>>>>> follow the common semantics of indicating > backwards-incompatible > >>>>>>>>> changes with a bump to the first version segment. > >>>>>>>>> > >>>>>>>>> The consequence of trying to lock down imports to a narrow > range or > >>>>>>>>> even a point version is that assembling an application > becomes very > >>>>>>>>> difficult, and we are forced to deploy many versions of common > >>>>>>>>> libraries concurrently. This is non-optimal, though we can > handle > >>>>>>>>> it > >>>>>>>>> to some degree via per-module classloaders as in OSGi. > >>>>>>>>> > >>>>>>>>> Regards, > >>>>>>>>> Neil > >>>>>>>>> > >>>>>>>>> On Thu, Nov 17, 2011 at 11:52 PM, cowwoc<[hidden email] > > > >>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> Can someone please explain why modules need to be able to > specify > >>>>>>>>>> version > >>>>>>>>>> ranges for dependencies? I believe OSGI allows the > specification > >>>>>>>>>> of > >>>>>>>>>> version > >>>>>>>>>> ranges while Maven allows the specification of individual > >>>>>>>>>> versions. > >>>>>>>>>> > >>>>>>>>>> The only thing that comes to mind is when module C depends > on A > >>>>>>>>>> and > >>>>>>>>>> B, > >>>>>>>>>> A > >>>>>>>>>> depends on log4j 1.0, and B depends on log4j 1.1. What does > C do? > >>>>>>>>>> Is > >>>>>>>>>> this > >>>>>>>>>> the main use-case for version ranges? > >>>>>>>>>> > >>>>>>>>>> By the sound of it, this is a trust model where developers are > >>>>>>>>>> told > >>>>>>>>>> that > >>>>>>>>>> log4j 1.x won't break compatibility so they depend on that > range > >>>>>>>>>> without > >>>>>>>>>> actually testing against each version (newer versions may be > >>>>>>>>>> released > >>>>>>>>>> after > >>>>>>>>>> their own software). I question whether such a mechanism is > better > >>>>>>>>>> or > >>>>>>>>>> worse > >>>>>>>>>> than depending on individual versions which may be > overridden at a > >>>>>>>>>> later > >>>>>>>>>> time (a la Maven). On the one hand, you don't need to > release a > >>>>>>>>>> new > >>>>>>>>>> version > >>>>>>>>>> of the application each time a dependency is updated. On > the other > >>>>>>>>>> hand, > >>>>>>>>>> no > >>>>>>>>>> one is actually running tests to ensure that the versions are > >>>>>>>>>> really > >>>>>>>>>> compatible. > >>>>>>>>>> > >>>>>>>>>> Is there a way to get module A to see log4j 1.0 and module > B to > >>>>>>>>>> see > >>>>>>>>>> log4j > >>>>>>>>>> 1.1 (using separate ClassLoaders)? > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> Gili > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> View this message in context: > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5002801.html > >>>>>>>>>> Sent from the jigsaw-dev mailing list archive at Nabble.com. > >>>>>>>>>> > >>>>>>> > >>>>> > >>>>> > >>> > >>> > >>> -- > >>> - DML > >>> > > > > > > -- > > - DML > > > > > ------------------------------------------------------------------------ > If you reply to this email, your message will be added to the > discussion below: > http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5011534.html > > To unsubscribe from Use-cases for version ranges?, click here > . > NAML > > -- View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5011558.html Sent from the jigsaw-dev mailing list archive at Nabble.com. From brian at pontarelli.com Mon Nov 21 12:16:33 2011 From: brian at pontarelli.com (Brian Pontarelli) Date: Mon, 21 Nov 2011 13:16:33 -0700 Subject: Use-cases for version ranges? In-Reply-To: <4ECAA968.6040709@bbs.darktech.org> References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> <4ECA2B0E.40809@tibco.com> <4ECAA968.6040709@bbs.darktech.org> Message-ID: On Nov 21, 2011, at 12:41 PM, cowwoc wrote: > Hi Eric, > > On 21/11/2011 5:43 AM, Eric Johnson [via jigsaw-dev] wrote: >> Trouble is, the Java community doesn't seem to like the idea of a new >> named interface with the additional of even one method to an interface. >> But is that perhaps just because the JDK has acculturated people to the >> idea that they don't need to do that? It seems to me that any version >> scheme that attempts to cover up this problem will likely just add to >> the confusion, rather than resolve the problem. We either have to stick >> a stake in the ground and declare what we mean by backward compatible, >> or admit that there isn't a single notion, and therefore version ranges >> just mean what people want them to mean. > > My problem with this approach is java.util.Collections followed by > Collections2 in Guava. Another example, Futures vs MoreFutures in Guava. > These are examples of class subclassing, not interface extension, but > the problem is the same: once you pick a good name it's extremely > difficult to come up with yet another good name for the same thing. How > many times do you think you could extend an interface before you run out > of names? Twice? Three times maybe? The naming convention is version based. It looks like: interface Collections {} interface Collections_1_2 extends Collections {} interface Collections_1_2_1 extends Collections_1_2 {} interface Collections_1_3 extends Collections_1_2_1 {} We attempted this at Orbitz and it was not pretty in practice. -bp From tdiekman at tibco.com Mon Nov 21 12:36:25 2011 From: tdiekman at tibco.com (Tim Diekmann) Date: Mon, 21 Nov 2011 20:36:25 +0000 Subject: Use-cases for version ranges? In-Reply-To: References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> <4ECA2B0E.40809@tibco.com> <4ECAA968.6040709@bbs.darktech.org> Message-ID: > The naming convention is version based. It looks like: > > interface Collections {} > interface Collections_1_2 extends Collections {} > interface Collections_1_2_1 extends Collections_1_2 {} > interface Collections_1_3 extends Collections_1_2_1 {} > > We attempted this at Orbitz and it was not pretty in practice. But did it work or cause other problems? Tim. From cowwoc at bbs.darktech.org Mon Nov 21 13:04:20 2011 From: cowwoc at bbs.darktech.org (cowwoc) Date: Mon, 21 Nov 2011 13:04:20 -0800 (PST) Subject: Use-cases for version ranges? In-Reply-To: References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> <4ECA2B0E.40809@tibco.com> <4ECAA968.6040709@bbs.darktech.org> Message-ID: <4ECABCC0.80104@bbs.darktech.org> I get the feeling that someone doesn't care about readability or usability ;) Computers should be working hard to make humans happy, not the other way around. Gili On 21/11/2011 3:37 PM, Tim Diekmann [via jigsaw-dev] wrote: > > The naming convention is version based. It looks like: > > > > interface Collections {} > > interface Collections_1_2 extends Collections {} > > interface Collections_1_2_1 extends Collections_1_2 {} > > interface Collections_1_3 extends Collections_1_2_1 {} > > > > We attempted this at Orbitz and it was not pretty in practice. > > But did it work or cause other problems? > > Tim. > > > ------------------------------------------------------------------------ > If you reply to this email, your message will be added to the > discussion below: > http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5011613.html > > To unsubscribe from Use-cases for version ranges?, click here > . > NAML > > -- View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5011690.html Sent from the jigsaw-dev mailing list archive at Nabble.com. From cowwoc at bbs.darktech.org Mon Nov 21 13:11:33 2011 From: cowwoc at bbs.darktech.org (cowwoc) Date: Mon, 21 Nov 2011 13:11:33 -0800 (PST) Subject: Use-cases for version ranges? In-Reply-To: <4ECAAB5D.2040101@oracle.com> References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> <4ECA2B0E.40809@tibco.com> <4ECAA968.6040709@bbs.darktech.org> <4ECAAB5D.2040101@oracle.com> Message-ID: <4ECABE77.3050204@bbs.darktech.org> Sorry Alex. You really hit a nerve with these slides. It's amazing how far people will go to stuff lambda expressions down our throats. So now you're killing interfaces and adding multiple inheritance to Java? Great.... just great. Instead of killing interfaces, how about allowing developers to replace an interface with an abstract Class without breaking compatibility? That's essentially what you're doing anyway... Also, shouldn't "spliterable" be called "splitable"? This is precisely what is wrong with the way the Java APIs are being evolved. Instead of breaking compatibility on a *class* level you're breaking compatibility on a *language* level and acting as if nothing is wrong. Needless to say, I am upset. If you were to fork Java into version 2.x and break the necessary classes at the same time the change might actually be less profound than introducing multiple inheritance. You could rewrite class files from version 1.x to 2.x as needed, similar to Retroweaver. To answer your original question, being able to evolve interfaces without breaking backwards compatibility helps but it doesn't really change my view of semantic compatibility. I remember plenty of Swing bugs I reported which were closed as WON'T FIX because doing so would have broken backwards compatibility for existing users. I'm talking about cases where someone made a mistake in the design and it could not be fixed for the rest of time. At the end of the day there needs to be a way for us to make changes that break compatibility in a reasonable manner. Had Sun/Oracle taken this route, they could have slowly replaced deprecated classes and methods over time. Even Microsoft dropped Windows 95 after 10 years. Gili On 21/11/2011 2:52 PM, Alex Buckley [via jigsaw-dev] wrote: > Hi Gili and Eric, > > Java SE 8 is extremely likely to support the addition of methods to > interfaces in a manner which is binary-compatible _and_ > source-compatible. See slide 52 onwards in > http://blogs.oracle.com/abuckley/resource/QConSF2011-JavaSE78.pdf. > > How does this affect your view of semantic compatibility and version > ranges? > > Alex > > On 11/21/2011 11:41 AM, cowwoc wrote: > > > Hi Eric, > > > > On 21/11/2011 5:43 AM, Eric Johnson [via jigsaw-dev] wrote: > >> Trouble is, the Java community doesn't seem to like the idea of a new > >> named interface with the additional of even one method to an > interface. > >> But is that perhaps just because the JDK has acculturated people to > the > >> idea that they don't need to do that? It seems to me that any version > >> scheme that attempts to cover up this problem will likely just add to > >> the confusion, rather than resolve the problem. We either have to > stick > >> a stake in the ground and declare what we mean by backward compatible, > >> or admit that there isn't a single notion, and therefore version > ranges > >> just mean what people want them to mean. > > > > My problem with this approach is java.util.Collections > followed by > > Collections2 in Guava. Another example, Futures vs MoreFutures in > Guava. > > These are examples of class subclassing, not interface extension, but > > the problem is the same: once you pick a good name it's extremely > > difficult to come up with yet another good name for the same thing. How > > many times do you think you could extend an interface before you run > out > > of names? Twice? Three times maybe? > > > > I'm also tired of Java features having their designs crippled in > > the name of backwards-compatibility. I'm personally in favor of each > > author declaring a backwards compatibility policy and breaking > > compatibility as necessary. In the case of the JDK, you can imagine > > removing deprecated methods after 5 years. Projects like Guava give you > > 6 months. The point is that you can't keep backwards compatibility > > forever, especially in light of the requirement you mentioned that > > developers may not add any new methods ;) > > > > Gili > > > ------------------------------------------------------------------------ > If you reply to this email, your message will be added to the > discussion below: > http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5011483.html > > To unsubscribe from Use-cases for version ranges?, click here > . > NAML > > -- View this message in context: http://jigsaw-dev.1059479.n5.nabble.com/Use-cases-for-version-ranges-tp5002801p5011709.html Sent from the jigsaw-dev mailing list archive at Nabble.com. From tdiekman at tibco.com Mon Nov 21 13:44:57 2011 From: tdiekman at tibco.com (Tim Diekmann) Date: Mon, 21 Nov 2011 21:44:57 +0000 Subject: Use-cases for version ranges? In-Reply-To: <4ECABCC0.80104@bbs.darktech.org> References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> <4ECA2B0E.40809@tibco.com> <4ECAA968.6040709@bbs.darktech.org> <4ECABCC0.80104@bbs.darktech.org> Message-ID: <183293FD-1D7D-4D1A-B002-0E0A0AC24CF3@tibco.com> Interesting comment. Working in a company that builds enterprise software, we are more concerned in making the experience using our software easier for our users, which means that we shoulder the burden of the hard stuff for them. In the end, it is our customers that pay our bills. Tim. "There is never enough time to do it right, but there is always enough time to do it over" -- Murphy's Law On Nov 21, 2011, at 1:04 PM, cowwoc wrote: > > I get the feeling that someone doesn't care about readability or > usability ;) Computers should be working hard to make humans happy, not > the other way around. > > Gili > > On 21/11/2011 3:37 PM, Tim Diekmann [via jigsaw-dev] wrote: >>> The naming convention is version based. It looks like: >>> >>> interface Collections {} >>> interface Collections_1_2 extends Collections {} >>> interface Collections_1_2_1 extends Collections_1_2 {} >>> interface Collections_1_3 extends Collections_1_2_1 {} >>> >>> We attempted this at Orbitz and it was not pretty in practice. >> >> But did it work or cause other problems? >> >> Tim. From brian at pontarelli.com Mon Nov 21 13:50:49 2011 From: brian at pontarelli.com (Brian Pontarelli) Date: Mon, 21 Nov 2011 14:50:49 -0700 Subject: Use-cases for version ranges? In-Reply-To: References: <1321573938450-5002801.post@n5.nabble.com> <4EC73025.40402@bbs.darktech.org> <4EC86F71.1000707@bbs.darktech.org> <4ECA6B05.8020001@redhat.com> <4ECA7D4D.9030608@redhat.com> Message-ID: That's an entirely different topic, by my quick opinion is that meta-data should be outside to increase performance. I shouldn't have to download every version of every module in my dependency graph to build/traverse/prune it. -bp On Nov 21, 2011, at 12:57 PM, David Bosschaert wrote: > I think that the fact that in JBoss Modules the metadata is *outside* > of the module (in a separate file that sits alongside the module) and > in OSGi the metadata is *inside* the module (embedded in the Jar) is a > significant difference. > > If you have the metadata outside of the module you have additional > flexibility. You can change the metadata without changing the module > itself, so if there is an issue with the metadata you can easily fix > it up (e.g. if you need to depend on another version of your > dependency). Also there are obviously impacts of changing the module > metadata if its inside the module and you are working with signed > jars. On the other side, having the metadata inside the module has the > benefit that you only need to worry about a single artifact: the > module itself. > > I'm not advocating one over the other here; I think both have > benefits, but I think that using version ranges makes more sense in > the case the metadata is embedded in the module. > > If I'm not mistaken, the model that Jigsaw is going for is currently > also the model where the metadata is inside the module. > > David > > On 21 November 2011 16:33, David M. Lloyd wrote: >> On 11/21/2011 10:11 AM, Neil Bartlett wrote: >>> >>> David, >>> I'm likewise tired of hearing these defeatist straw-man arguments. >>> First, nobody is suggesting that you be forced to use version ranges. >>> Second, nobody is suggesting that you be forced to design software by >>> separating interfaces from implementation. Finally, nobody is >>> suggesting that unmodified legacy code can be trivially modularised in >>> *any* module system. >> >> Trivial? Depends on how you define it. Most artifacts should be able to be >> modularized as-is however, with minimal changes if any. We adopted an >> external descriptor strategy for the sole purpose of being able to use >> modules as is, and we were able to package our 200+ modules just that way, >> no repackaging needed. So far the only problems we have really run into >> seem to be legitimate bugs, and one or two libraries which assume a flat >> classpath. >> >> I just glanced at the quick stats for Maven Central. There are 267,924 >> indexed artifacts at the time of this writing - 31,847 unique artifacts. >> This is a major reason why people even use Java in the first place. Ideally >> we would be able to start with at least the possibility of having a somewhat >> pre-populated repository. >> >>> The module system must support both existing software and new >>> software, of course. You seem suggest that I believe existing software >>> is unimportant; nothing could be further from the truth. But I also >>> believe that new software is important and that practices like >>> interface/implementation separation and semantic versioning will help >>> make the new software better than the old software. >>> Therefore I believe that these features must be supported (though not >>> mandated) in any new module system. You appear to be suggesting that >>> these features are worthless because they cannot be applied to >>> existing software, and/or will not be used by all Java developers. >>> Well so what?? >> >> Not worthless, but a question of personal choice. I believe that interface >> and implementation separation has not been objectively shown to be a >> superior way to write frameworks. In my experience, sometimes it is and >> sometimes it is not. >> >> I believe that "semantic versioning" isn't going to work because most people >> who develop Java software will not understand it or not get it right, due in >> no small part to being a complex idea with a lot of subtlety. In think that >> empirical testing is the only way to ensure compatibility which will scale. >> >>> Regards,Neil >>> On Mon, Nov 21, 2011 at 3:15 PM, David M. Lloyd >>> wrote: >>>> >>>> I'm tired of hearing pie-in-the sky, "intro to OOP" wishful crap like >>>> this. >>>> Mission one should be supporting *existing* code. The fact is that >>>> people >>>> do *not* always split their API from their implementation, and APIs >>>> almost >>>> *never* consist solely of interfaces. There are many ways of designing >>>> an >>>> API which people consider "right" and I don't see anyone on this list >>>> having >>>> any special insight that would enable them to select the "one true" >>>> approach >>>> to API design. >>>> >>>> The fact is that any version scheme or module resolution engine which >>>> only >>>> works for new code which is designed "properly" for it is *going* to >>>> fail. >>>> And there's no reason for such constraints anyway. Version ranges are >>>> barely useful at all in practice. The only ones who can assert >>>> compatibility between two libraries of arbitrary version are the ones who >>>> have tested the two together (along with anything else that uses a >>>> combination of the two). >>>> >>>> The best one can do establish *minimum* compatibility based on available >>>> features, and test test test. When a module is added to a repository, if >>>> it >>>> is tested by running both its internal unit tests and all available tests >>>> of >>>> all its dependents, then that should be adequate to establish >>>> compatibility >>>> *within a repository*. There are too many factors to establish >>>> compatibility beyond it. >>>> >>>> On 11/20/2011 07:45 AM, Neil Bartlett wrote: >>>>> >>>>> Stephen, >>>>> >>>>> You again seem to be talking about dependencies on implementation >>>>> packages. If we depend on APIs consisting primarily of interfaces, >>>>> then versions and version ranges are reliable and can be completely >>>>> automated. >>>>> >>>>> Regards, >>>>> Neil >>>>> >>>>> >>>>> On Sun, Nov 20, 2011 at 12:25 PM, Stephen McConnell >>>>> wrote: >>>>>> >>>>>> I could say that the question is academic - but here is the thing - its >>>>>> not >>>>>> even academic. >>>>>> >>>>>> We talk about versions and yet the platform has no computational >>>>>> mechanisms >>>>>> to flag version violation. At least an academic discussion would bring >>>>>> in >>>>>> grounded metrics. In reality - the discussion here about V1 versus V2 >>>>>> and >>>>>> that warm fuzzy feeling when your are just a n.m from an n+1 and it's a >>>>>> waste of breath (and that comment goes to all sides of the debate). >>>>>> >>>>>> Give me an endorsed tool that shows me that module [name]-[build] is >>>>>> computationally backward compatible to a different [name]-[build] and >>>>>> then, >>>>>> only them will this discussion have reasonable grounds to assert any >>>>>> authority on the subject of version compatibility (past or future). >>>>>> Without >>>>>> that tool - versioning is a unqualified statement of trust. And the >>>>>> evidence >>>>>> suggests that the threshold of trust delivery on actual computational >>>>>> compliance is really really thin. >>>>>> >>>>>> Cheers, Steve. >>>>>> >>>>>> >>>>>> -----Original Message----- From: cowwoc >>>>>> Sent: Sunday, November 20, 2011 1:39 PM >>>>>> To: jigsaw-dev at openjdk.java.net >>>>>> Subject: Re: Use-cases for version ranges? >>>>>> >>>>>> >>>>>> I'd like to propose another possibility: the author of the >>>>>> dependency should tell *us* about version compatibility, not the other >>>>>> way around. For example: >>>>>> >>>>>> 1. The author of module A declares a dependency on module B version 1.2 >>>>>> (specific version). >>>>>> 2. The author of module B publishes version 1.3. He declares that >>>>>> version 1.3 is compatible with 1.2 (meaning, the runtime system is >>>>>> allows to substitute version 1.3 for 1.2). >>>>>> >>>>>> The upside of this approach is that the author of B is in a better >>>>>> position to declare compatibility than the author of A. The author of A >>>>>> still only needs to test a single version. What do you think? >>>>>> >>>>>> Gili >>>>>> >>>>>> On 19/11/2011 1:59 AM, Neil Bartlett wrote: >>>>>>> >>>>>>> Gili, >>>>>>> >>>>>>> I didn't say anything about guarantees, and in this industry I have >>>>>>> never heard of anybody providing a guarantees about the performance of >>>>>>> their software, especially in the presence of external dependencies. >>>>>>> >>>>>>> Version ranges are a means of communicating expectations, and we >>>>>>> provide both a lower and an upper bound because this is useful >>>>>>> information. I expect my module will work with 1.2.14, and I expect it >>>>>>> will not work with 2.0. If I were a provider of the API rather than a >>>>>>> consumer then I would have a much narrower expectation, e.g. >>>>>>> [1.2,1.3), and this would also be useful information to convey. >>>>>>> >>>>>>> Regards >>>>>>> Neil >>>>>>> >>>>>>> On Sat, Nov 19, 2011 at 4:27 AM, cowwoc >>>>>>> wrote: >>>>>>>> >>>>>>>> Neil, >>>>>>>> >>>>>>>> I guess I don't understand why Jigsaw should work differently from >>>>>>>> Maven >>>>>>>> on this point. I am expecting developers to specify specific versions >>>>>>>> that >>>>>>>> they tested (point versions, not ranges) and end-users may override >>>>>>>> these >>>>>>>> "recommendations" as they see fit. >>>>>>>> >>>>>>>> Where you see version range [1.2.14, 2.0) as a way of >>>>>>>> communicating >>>>>>>> "the >>>>>>>> developer guarantees 1.2.14 but you may use newer versions up to 2.0 >>>>>>>> at >>>>>>>> your >>>>>>>> own risk" I'd expect the developer to simply specify 1.2.14 and there >>>>>>>> should >>>>>>>> be no limit on what version end-users may use if they so wish. >>>>>>>> >>>>>>>> Gili >>>>>>>> >>>>>>>> On 18/11/2011 2:23 AM, Neil Bartlett wrote: >>>>>>>>> >>>>>>>>> I noticed that I failed to address your point about Maven using >>>>>>>>> point >>>>>>>>> versions. >>>>>>>>> >>>>>>>>> Maven is a build tool. At build time we need to compile against a >>>>>>>>> single specific version so that we have repeatable builds. In >>>>>>>>> general >>>>>>>>> we should build each module against the lowest version of the >>>>>>>>> library >>>>>>>>> that it can possibly use, and there are no major negative >>>>>>>>> consequences >>>>>>>>> of having several versions of a library at build time (except that >>>>>>>>> Maven has to download a lot!). At runtime however we need to have >>>>>>>>> the >>>>>>>>> flexibility to substitute a single compatible version. >>>>>>>>> >>>>>>>>> Neil >>>>>>>>> >>>>>>>>> On Fri, Nov 18, 2011 at 7:10 AM, Neil Bartlett >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Suppose as the developer of module A, I declare a dependency on >>>>>>>>>> log4j, >>>>>>>>>> exactly version 1.0.0 because I have not tested against log4j >>>>>>>>>> 1.0.1, >>>>>>>>>> 1.0.2, 1.3, 999.999 etc. I effectively prevent my module *ever* >>>>>>>>>> being >>>>>>>>>> used with log4j version 1.0.1 even if this combinations is later >>>>>>>>>> tested and proven to work by somebody else. In other words, testing >>>>>>>>>> is >>>>>>>>>> important but it doesn't necessarily have to always be done by the >>>>>>>>>> original developer of each module. >>>>>>>>>> >>>>>>>>>> On the other hand let's say I state my dependency using the >>>>>>>>>> following >>>>>>>>>> range: [1.2.14, 2.0). This is OSGi syntax and I believe Jigsaw is >>>>>>>>>> following it, and it simply means I accept version 1.2.14 up to but >>>>>>>>>> not including 2.0. Anybody can see that I compiled and tested >>>>>>>>>> against >>>>>>>>>> 1.2.14, but has the option of using 1.2.15, 1.2.16, 1.3, 1.9 etc. >>>>>>>>>> It >>>>>>>>>> does not mean that I *guarantee* my module will work with log4j 1.3 >>>>>>>>>> because that obviously depends on whether the log4j authors accept >>>>>>>>>> and >>>>>>>>>> follow the common semantics of indicating backwards-incompatible >>>>>>>>>> changes with a bump to the first version segment. >>>>>>>>>> >>>>>>>>>> The consequence of trying to lock down imports to a narrow range or >>>>>>>>>> even a point version is that assembling an application becomes very >>>>>>>>>> difficult, and we are forced to deploy many versions of common >>>>>>>>>> libraries concurrently. This is non-optimal, though we can handle >>>>>>>>>> it >>>>>>>>>> to some degree via per-module classloaders as in OSGi. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Neil >>>>>>>>>> >>>>>>>>>> On Thu, Nov 17, 2011 at 11:52 PM, cowwoc >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Can someone please explain why modules need to be able to specify >>>>>>>>>>> version >>>>>>>>>>> ranges for dependencies? I believe OSGI allows the specification >>>>>>>>>>> of >>>>>>>>>>> version >>>>>>>>>>> ranges while Maven allows the specification of individual >>>>>>>>>>> versions. >>>>>>>>>>> >>>>>>>>>>> The only thing that comes to mind is when module C depends on A >>>>>>>>>>> and >>>>>>>>>>> B, >>>>>>>>>>> A >>>>>>>>>>> depends on log4j 1.0, and B depends on log4j 1.1. What does C do? >>>>>>>>>>> Is >>>>>>>>>>> this >>>>>>>>>>> the main use-case for version ranges? >>>>>>>>>>> >>>>>>>>>>> By the sound of it, this is a trust model where developers are >>>>>>>>>>> told >>>>>>>>>>> that >>>>>>>>>>> log4j 1.x won't break compatibility so they depend on that range >>>>>>>>>>> without >>>>>>>>>>> actually testing against each version (newer versions may be >>>>>>>>>>> released >>>>>>>>>>> after >>>>>>>>>>> their own software). I question whether such a mechanism is better >>>>>>>>>>> or >>>>>>>>>>> worse >>>>>>>>>>> than depending on individual versions which may be overridden at a >>>>>>>>>>> later >>>>>>>>>>> time (a la Maven). On the one hand, you don't need to release a >>>>>>>>>>> new >>>>>>>