From rfscholte at apache.org Sat Feb 11 16:18:09 2017 From: rfscholte at apache.org (Robert Scholte) Date: Sat, 11 Feb 2017 17:18:09 +0100 Subject: Automodules Message-ID: There are clearly 2 kinds of jars in play here: dependencies and end-products. To have a reliable end-product you want to specify *every* requirement and in this case it doesn't matter that a dependency has no official module name, as long as you can specify and refer to it as a requirement. To have a reliable dependency you can only depend on named modules. Up until now it is required to specify all requirements or do some command line argument magic to compile or run such project. What I would like to achieve is to have a situation where end-products can never be used as a dependency, because that would allow auto modules, which makes it again unreliable as dependency. Specifying a dependency is done by referring to the name of a module. So what if we say that a end-product doesn't have a name anymore or that it is not a module anymore. In that case it cannot be used as dependency AND we could allow automodules in such case. In case of a dependency or named module you can only refer to other named modules. All other required jars should be picked up from the classpath, preferably by default. This is just a start, but would probably match most requirements from both camps. So an example of the end-product jar: final myapp { // 'module' replaced with 'final' requires mylib; // a named module, assume part of Maven multimodule project requires java.base; // a named module, part of Java requires java.sql; // a named module, part of Java requires jackson.core; // an unnamed module, name extract from filename (jackson-core-2.6.2.jar) requires jackson.databind; // an unnamed module, name extract from filename (jackson-databind-2.6.2.jar) } or module { // remove the name, so can never refer to it requires mylib; // a named module, assume part of Maven multimodule project requires java.base; // a named module, part of Java requires java.sql; // a named module, part of Java requires jackson.core; // an unnamed module, name extract from filename (jackson-core-2.6.2.jar) requires jackson.databind; // an unnamed module, name extract from filename (jackson-databind-2.6.2.jar) } An example of the dependency jar module org.joda.convert { // requires ??guava??; no named module yet, will be picked up from classpath } other projects could already refer to org.joda.convert if they want to. Once guava is a named module, org.joda.convert can specify it. regards, Robert From david.lloyd at redhat.com Mon Feb 13 15:59:44 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 13 Feb 2017 09:59:44 -0600 Subject: Automodules In-Reply-To: References: Message-ID: <85eae21b-8a74-2ae5-d136-c035eefa5c93@redhat.com> On 02/11/2017 10:18 AM, Robert Scholte wrote: > There are clearly 2 kinds of jars in play here: dependencies and > end-products. > > To have a reliable end-product you want to specify *every* requirement > and in this case it doesn't matter that a dependency has no official > module name, as long as you can specify and refer to it as a requirement. > > To have a reliable dependency you can only depend on named modules. Up > until now it is required to specify all requirements or do some command > line argument magic to compile or run such project. > > What I would like to achieve is to have a situation where end-products > can never be used as a dependency, because that would allow auto > modules, which makes it again unreliable as dependency. We accomplished exactly these goals in jboss-modules, simply by having multiple "worlds": The module world itself (of course), and separated worlds for the modularized class path or -jar command line. In the raw class path situation, we create a single module for the entire class path. This is essentially an "unnamed" blob, which can consume available modules but cannot in turn be consumed by them. This allows a first step into the module world by allowing (over time) the user to move modules from the class path into the module space as their dependencies can be correctly and specifically analyzed (in other words, it's one step from sloppy class path to constrained/clean module). Classpath JARs could also be separated into modules which have a mutual dependency mesh (assuming that circular dependencies are allowed), allowing for a better diagnostic identity than just "classpath" (for example). In the -jar situation, we create a module per JAR using the JAR's Class-Path MANIFEST.MF header as the linking mechanism. In the container we also support the old extensions mechanism (which it turns out is actually a pretty good basic module system in its own right). In this way, JARs are isolated from one another, consuming only what is explicitly declared. In the Java EE container, these dependencies are usually transitive due to some unfortunate specification language, but on the command-line they are not, which further improves isolation. In this case as well JARs can depend on modules but not vice-versa. Circularity is also historically allowed in this situation. I have found these mechanisms to be more than adequate for providing an easy on ramp for modularization, and neither one requires module name fudging; in fact, mangling of names is harmful since it takes the intuitive "my-project-1.3.3.jar" that would appear in diagnostics, which can be copied and pasted into a "find" command for example, and turns it into something rather unintuitive. Having a modular classpath in fact is the easiest possible "small" step because it allows code to run under the new module accessibility and (service) loading rules without otherwise affecting class loading behavior in any way. This is why I'm frustrated with automatic modules: they just don't seem necessary, and have several proven problems which also seems to make them insufficient. > Specifying a dependency is done by referring to the name of a module. > So what if we say that a end-product doesn't have a name anymore or that > it is not a module anymore. In that case it cannot be used as dependency > AND we could allow automodules in such case. > > In case of a dependency or named module you can only refer to other > named modules. All other required jars should be picked up from the > classpath, preferably by default. > > This is just a start, but would probably match most requirements from > both camps. > > So an example of the end-product jar: > > final myapp { // 'module' replaced with 'final' > requires mylib; // a named module, assume part of Maven > multimodule project > requires java.base; // a named module, part of Java > requires java.sql; // a named module, part of Java > requires jackson.core; // an unnamed module, name extract from > filename (jackson-core-2.6.2.jar) > requires jackson.databind; // an unnamed module, name extract from > filename (jackson-databind-2.6.2.jar) > } > > or > > module { // remove the name, so can never refer to it > requires mylib; // a named module, assume part of Maven > multimodule project > requires java.base; // a named module, part of Java > requires java.sql; // a named module, part of Java > requires jackson.core; // an unnamed module, name extract from > filename (jackson-core-2.6.2.jar) > requires jackson.databind; // an unnamed module, name extract from > filename (jackson-databind-2.6.2.jar) > } > > An example of the dependency jar > > module org.joda.convert { > // requires ??guava??; no named module yet, will be picked up from > classpath > } > > other projects could already refer to org.joda.convert if they want to. > Once guava is a named module, org.joda.convert can specify it. > > regards, > Robert -- - DML From mark.reinhold at oracle.com Mon Feb 13 18:18:18 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 13 Feb 2017 10:18:18 -0800 Subject: Proposal: #NonHierarchicalLayers In-Reply-To: <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> References: <20161207234117.1E5F1213F1@eggemoggin.niobe.net> <1de06198-930b-4b71-fe5d-1477d9cace80@redhat.com> <20161212232335.A9BFC24B27@eggemoggin.niobe.net> <9b66aa52-877a-e4b2-1458-9ff47dfb690b@redhat.com> <20170126223431.B417255DD9@eggemoggin.niobe.net> <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> Message-ID: <20170213101818.107385081@eggemoggin.niobe.net> 2017/1/30 7:13:31 -0800, david.lloyd at redhat.com: > On 01/26/2017 04:34 PM, mark.reinhold at oracle.com wrote: >> ... >> >> If not, then adding the above methods won't help you anyway. They only >> affect the run-time state of modules, i.e., the state known to the JVM >> and surfaced in `java.lang.reflect.Module`. The JPMS resolver doesn't >> read the run-time state but, rather, the configurations computed by >> earlier invocations of the resolver, and ultimately those are all based >> on module descriptors, over in the `java.lang.module` package. > > Currently the prototype code does not use the JPMS resolver. From my > understanding, the JVM tables mediate access control, but the actual > linkage between classes at run time happens via > java.lang.ClassLoader#findClass(java.lang.String, java.lang.String). As > long as this method resolves Did you mean to say more here? Was your reply somehow truncated? - Mark From mark.reinhold at oracle.com Mon Feb 13 18:21:45 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 13 Feb 2017 10:21:45 -0800 Subject: Proposal: #VersionedDependences In-Reply-To: <1441073057.2123030.1485607201470.JavaMail.zimbra@u-pem.fr> References: <20161209214646.D915621F70@eggemoggin.niobe.net> <1717092100.1687029.1481822451766.JavaMail.zimbra@u-pem.fr> <20170126223631.BE89955DE7@eggemoggin.niobe.net> <1525805814.2065614.1485539603846.JavaMail.zimbra@u-pem.fr> <20170127142421.887524637@eggemoggin.niobe.net> <20170127174547.771927105@eggemoggin.niobe.net> <1441073057.2123030.1485607201470.JavaMail.zimbra@u-pem.fr> Message-ID: <20170213102145.95354985@eggemoggin.niobe.net> 2017/1/28 4:40:01 -0800, forax at univ-mlv.fr: > 2017/1/27 17:45:47 -0800, mark.reinhold at oracle.com: >> ... >> >> I suppose the best we can do here is, wherever a version is exposed, >> define a pair of methods. One will return the raw version string; the >> other will return a `Version` object if the raw string can be parsed >> as such, and an empty `Optional` otherwise. >> >> So instead of the single `compiledVersion` method proposed for the >> `Requires` class we'd have two, >> >> Optional compiledVersionString(); >> Optional compiledVersion(); >> >> and similarly for the `ModuleDescriptor::version()` method itself. >> >> Make sense? > > yes ! > > for the API, if i can nitpick, i prefer: > > Optional rawCompiledVersion() > Optional compiledVersion() Sure, let's go with those. I like that they'll wind up not being nearby the non-raw methods in the Javadoc, thereby making them less obvious. - Mark From rfscholte at apache.org Mon Feb 13 20:54:57 2017 From: rfscholte at apache.org (Robert Scholte) Date: Mon, 13 Feb 2017 21:54:57 +0100 Subject: Automodules In-Reply-To: <85eae21b-8a74-2ae5-d136-c035eefa5c93@redhat.com> References: <85eae21b-8a74-2ae5-d136-c035eefa5c93@redhat.com> Message-ID: On Mon, 13 Feb 2017 16:59:44 +0100, David M. Lloyd wrote: > On 02/11/2017 10:18 AM, Robert Scholte wrote: >> There are clearly 2 kinds of jars in play here: dependencies and >> end-products. >> >> To have a reliable end-product you want to specify *every* requirement >> and in this case it doesn't matter that a dependency has no official >> module name, as long as you can specify and refer to it as a >> requirement. >> >> To have a reliable dependency you can only depend on named modules. Up >> until now it is required to specify all requirements or do some command >> line argument magic to compile or run such project. >> >> What I would like to achieve is to have a situation where end-products >> can never be used as a dependency, because that would allow auto >> modules, which makes it again unreliable as dependency. > > We accomplished exactly these goals in jboss-modules, simply by having > multiple "worlds": The module world itself (of course), and separated > worlds for the modularized class path or -jar command line. > > In the raw class path situation, we create a single module for the > entire class path. This is essentially an "unnamed" blob, which can > consume available modules but cannot in turn be consumed by them. This > allows a first step into the module world by allowing (over time) the > user to move modules from the class path into the module space as their > dependencies can be correctly and specifically analyzed (in other words, > it's one step from sloppy class path to constrained/clean module). > Classpath JARs could also be separated into modules which have a mutual > dependency mesh (assuming that circular dependencies are allowed), > allowing for a better diagnostic identity than just "classpath" (for > example). > > In the -jar situation, we create a module per JAR using the JAR's > Class-Path MANIFEST.MF header as the linking mechanism. In the > container we also support the old extensions mechanism (which it turns > out is actually a pretty good basic module system in its own right). In > this way, JARs are isolated from one another, consuming only what is > explicitly declared. In the Java EE container, these dependencies are > usually transitive due to some unfortunate specification language, but > on the command-line they are not, which further improves isolation. In > this case as well JARs can depend on modules but not vice-versa. > Circularity is also historically allowed in this situation. > > I have found these mechanisms to be more than adequate for providing an > easy on ramp for modularization, and neither one requires module name > fudging; in fact, mangling of names is harmful since it takes the > intuitive "my-project-1.3.3.jar" that would appear in diagnostics, which > can be copied and pasted into a "find" command for example, and turns it > into something rather unintuitive. Having a modular classpath in fact > is the easiest possible "small" step because it allows code to run under > the new module accessibility and (service) loading rules without > otherwise affecting class loading behavior in any way. > > This is why I'm frustrated with automatic modules: they just don't seem > necessary, and have several proven problems which also seems to make > them insufficient. > Nice to know that my thoughts are confirmed with an existing modular system. And the "small" step as adding one requirement at the time to a named module fits very well with the Maven eco system. thanks, Robert >> Specifying a dependency is done by referring to the name of a module. >> So what if we say that a end-product doesn't have a name anymore or that >> it is not a module anymore. In that case it cannot be used as dependency >> AND we could allow automodules in such case. >> >> In case of a dependency or named module you can only refer to other >> named modules. All other required jars should be picked up from the >> classpath, preferably by default. >> >> This is just a start, but would probably match most requirements from >> both camps. >> >> So an example of the end-product jar: >> >> final myapp { // 'module' replaced with 'final' >> requires mylib; // a named module, assume part of Maven >> multimodule project >> requires java.base; // a named module, part of Java >> requires java.sql; // a named module, part of Java >> requires jackson.core; // an unnamed module, name extract from >> filename (jackson-core-2.6.2.jar) >> requires jackson.databind; // an unnamed module, name extract from >> filename (jackson-databind-2.6.2.jar) >> } >> >> or >> >> module { // remove the name, so can never refer to it >> requires mylib; // a named module, assume part of Maven >> multimodule project >> requires java.base; // a named module, part of Java >> requires java.sql; // a named module, part of Java >> requires jackson.core; // an unnamed module, name extract from >> filename (jackson-core-2.6.2.jar) >> requires jackson.databind; // an unnamed module, name extract from >> filename (jackson-databind-2.6.2.jar) >> } >> >> An example of the dependency jar >> >> module org.joda.convert { >> // requires ??guava??; no named module yet, will be picked up from >> classpath >> } >> >> other projects could already refer to org.joda.convert if they want to. >> Once guava is a named module, org.joda.convert can specify it. >> >> regards, >> Robert From mark.reinhold at oracle.com Wed Feb 15 19:51:08 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 15 Feb 2017 11:51:08 -0800 (PST) Subject: Proposal: #NonHierarchicalLayers In-Reply-To: <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> References: <20161207234117.1E5F1213F1@eggemoggin.niobe.net> <1de06198-930b-4b71-fe5d-1477d9cace80@redhat.com> <20161212232335.A9BFC24B27@eggemoggin.niobe.net> <9b66aa52-877a-e4b2-1458-9ff47dfb690b@redhat.com> <20170126223431.B417255DD9@eggemoggin.niobe.net> <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> Message-ID: <20170215195108.C881964E6F@eggemoggin.niobe.net> 2017/1/30 7:13:31 -0800, david.lloyd at redhat.com: > On 01/26/2017 04:34 PM, mark.reinhold at oracle.com wrote: >> 2016/12/13 6:47:53 -0800, david.lloyd at redhat.com: >>> On 12/12/2016 05:23 PM, mark.reinhold at oracle.com wrote: >>>> 2016/12/8 5:46:24 -0800, david.lloyd at redhat.com: >>>>> ... I've added the following methods to Layer.Controller: >>>>> >>>>> public Controller addPackage(Module source, String pn) { ... } >>>>> public Controller addOpens(Module source, String pn, Module target) { ... } >>>>> public Controller addOpensToAll(Module source, String pn) { ... } >>>>> public Controller addOpensToAllUnnamed(Module source, String pn) { ... } >>>>> public Controller addExports(Module source, String pn, Module target) { >>>>> ... } >>>>> public Controller addExportsToAll(Module source, String pn) { ... } >>>>> public Controller addExportsToAllUnnamed(Module source, String pn) { ... } >>>>> public Controller addUses(Module source, Class service) { ... } >>>> >>>> Can you explain exactly why you need all these methods? >>>> >>>> I can see why you might need the qualified `addExports` method, akin >>>> to the existing `addOpens` method, if you're doing some form of module >>>> resolution on your own that's somehow taking named layers, or whatever, >>>> into account. >>> >>> Yeah we're assembling the module structure in a multi-stage lazy >>> resolution process, thus we don't know exactly what we're opening or >>> exporting until after all contents and dependencies are defined (and >>> this can change over time). >> >> Hmm. This seems to be a fundamental difference between JBoss modules and >> OSGi. Once an OSGi bundle is loaded then its exports don't change, at >> least not until it's updated, and this is part of what enables Watson's >> JPMS embedding to work. > > OSGi does support dynamic attachments of fragments. The current > prototype cannot do this but on Jan. 4 Thomas expressed that being able > to add packages would enable this part of the specification. Re-linking > everything is (according to this email) an alternative that comes at the > cost of not supporting this feature. True, but Thomas later recommended against complicating JPMS in order to support this feature of OSGi [1]. He can live with having to re-resolve bundles when fragments add API packages, which is not common and arguably an anti-pattern. >> ... >> >> For any kind of Watson-style embedding of JBoss modules in JPMS to work >> then there has to be a point in time at which you can say "okay, this >> JBoss module is in a stable state, let's spin up a JPMS module descriptor >> for it so that the JPMS resolver can resolve some JPMS modules against >> it." If the JBoss module is updated later on then you create a new JPMS >> module descriptor for it and re-resolve any JPMS modules that used to >> depend upon it. > > In order to do this we have to shut down all existing deployments in the > container that have any kind of dependency relationship, and restart > them. This is not really acceptable as it defeats the purpose of the > mechanism. > >> Can you, in general, identify such points in time for JBoss modules? > > Any time the user deploys additional or replacement content is a point > where a module may need to be updated. > >> If so, then I don't think you need the above methods. The JPMS module >> descriptor that you construct needn't capture all the details of the >> corresponding JBoss module. In fact it can't, since JBoss modules can >> relate to each other in ways that aren't expressible in JPMS. That's >> fine, though -- the module descriptor need only expose the information >> that the JPMS resolver needs for actual JPMS modules, mainly `opens`, >> `exports`, and `requires transitive`. >> >> If not, then adding the above methods won't help you anyway. They only >> affect the run-time state of modules, i.e., the state known to the JVM >> and surfaced in `java.lang.reflect.Module`. The JPMS resolver doesn't >> read the run-time state but, rather, the configurations computed by >> earlier invocations of the resolver, and ultimately those are all based >> on module descriptors, over in the `java.lang.module` package. > > Currently the prototype code does not use the JPMS resolver. From my > understanding, the JVM tables mediate access control, but the actual > linkage between classes at run time happens via > java.lang.ClassLoader#findClass(java.lang.String, java.lang.String). As > long as this method resolves This sentence is incomplete, so I'm not sure what you meant to say here. At any rate, if you're not eventually going to use the JPMS resolver to resolve some JPMS modules against some JBoss modules, in order to obtain bidirectional interoperation as is now possible with OSGi [2], then why bother with any of this? - Mark [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-January/010776.html [2] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000410.html From mark.reinhold at oracle.com Wed Feb 15 22:29:56 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 15 Feb 2017 14:29:56 -0800 (PST) Subject: Closing out yet more open issues Message-ID: <20170215222956.D4DFF64ED1@eggemoggin.niobe.net> Proposals for the following issues have been available for evaluation and experimentation for some time [1]: #AwkwardStrongEncapsulation #ClassFileAccModule #ClassFileModuleName #IndirectQualifiedReflectiveAccess #ModuleNameCharacters #NonHierarchicalLayers #ReadabilityAddedByLayerCreator #ReflectiveAccessToNonExportedTypes #VersionedDependences Responses to these proposals have been either neutral or positive, and in some cases further suggestions have been adopted, so I've marked these issues as closed. We're still discussing the new constraints on module names included in the broader proposal for #VersionedDependences [2], but that's really an amendment to the accepted proposal for #VersionsInModuleNames [3], so I've closed the former issue and re-opened the latter. - Mark [1] http://openjdk.java.net/projects/jigsaw/spec/issues/ [2] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-December/000516.html [3] http://openjdk.java.net/projects/jigsaw/spec/issues/#VersionsInModuleNames From david.lloyd at redhat.com Wed Feb 15 22:32:06 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 15 Feb 2017 16:32:06 -0600 Subject: Proposal: #NonHierarchicalLayers In-Reply-To: <20170215195108.C881964E6F@eggemoggin.niobe.net> References: <20161207234117.1E5F1213F1@eggemoggin.niobe.net> <1de06198-930b-4b71-fe5d-1477d9cace80@redhat.com> <20161212232335.A9BFC24B27@eggemoggin.niobe.net> <9b66aa52-877a-e4b2-1458-9ff47dfb690b@redhat.com> <20170126223431.B417255DD9@eggemoggin.niobe.net> <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> <20170215195108.C881964E6F@eggemoggin.niobe.net> Message-ID: <713a39e0-4b76-0a28-2f91-cd7cb1655fcb@redhat.com> On 02/15/2017 01:51 PM, mark.reinhold at oracle.com wrote: > 2017/1/30 7:13:31 -0800, david.lloyd at redhat.com: >> On 01/26/2017 04:34 PM, mark.reinhold at oracle.com wrote: >>> 2016/12/13 6:47:53 -0800, david.lloyd at redhat.com: >>>> On 12/12/2016 05:23 PM, mark.reinhold at oracle.com wrote: >>>>> 2016/12/8 5:46:24 -0800, david.lloyd at redhat.com: >>>>>> ... I've added the following methods to Layer.Controller: >>>>>> >>>>>> public Controller addPackage(Module source, String pn) { ... } >>>>>> public Controller addOpens(Module source, String pn, Module target) { ... } >>>>>> public Controller addOpensToAll(Module source, String pn) { ... } >>>>>> public Controller addOpensToAllUnnamed(Module source, String pn) { ... } >>>>>> public Controller addExports(Module source, String pn, Module target) { >>>>>> ... } >>>>>> public Controller addExportsToAll(Module source, String pn) { ... } >>>>>> public Controller addExportsToAllUnnamed(Module source, String pn) { ... } >>>>>> public Controller addUses(Module source, Class service) { ... } >>>>> >>>>> Can you explain exactly why you need all these methods? >>>>> >>>>> I can see why you might need the qualified `addExports` method, akin >>>>> to the existing `addOpens` method, if you're doing some form of module >>>>> resolution on your own that's somehow taking named layers, or whatever, >>>>> into account. >>>> >>>> Yeah we're assembling the module structure in a multi-stage lazy >>>> resolution process, thus we don't know exactly what we're opening or >>>> exporting until after all contents and dependencies are defined (and >>>> this can change over time). >>> >>> Hmm. This seems to be a fundamental difference between JBoss modules and >>> OSGi. Once an OSGi bundle is loaded then its exports don't change, at >>> least not until it's updated, and this is part of what enables Watson's >>> JPMS embedding to work. >> >> OSGi does support dynamic attachments of fragments. The current >> prototype cannot do this but on Jan. 4 Thomas expressed that being able >> to add packages would enable this part of the specification. Re-linking >> everything is (according to this email) an alternative that comes at the >> cost of not supporting this feature. > > True, but Thomas later recommended against complicating JPMS in order to > support this feature of OSGi [1]. He can live with having to re-resolve > bundles when fragments add API packages, which is not common and arguably > an anti-pattern. Sure, it's possible that OSGi implementations can get away with it, as few users seem to care much about such strict spec adherence and perhaps the spec will be modified to take the new restrictions into account. >>> For any kind of Watson-style embedding of JBoss modules in JPMS to work >>> then there has to be a point in time at which you can say "okay, this >>> JBoss module is in a stable state, let's spin up a JPMS module descriptor >>> for it so that the JPMS resolver can resolve some JPMS modules against >>> it." If the JBoss module is updated later on then you create a new JPMS >>> module descriptor for it and re-resolve any JPMS modules that used to >>> depend upon it. >> >> In order to do this we have to shut down all existing deployments in the >> container that have any kind of dependency relationship, and restart >> them. This is not really acceptable as it defeats the purpose of the >> mechanism. >> >>> Can you, in general, identify such points in time for JBoss modules? >> >> Any time the user deploys additional or replacement content is a point >> where a module may need to be updated. >> >>> If so, then I don't think you need the above methods. The JPMS module >>> descriptor that you construct needn't capture all the details of the >>> corresponding JBoss module. In fact it can't, since JBoss modules can >>> relate to each other in ways that aren't expressible in JPMS. That's >>> fine, though -- the module descriptor need only expose the information >>> that the JPMS resolver needs for actual JPMS modules, mainly `opens`, >>> `exports`, and `requires transitive`. >>> >>> If not, then adding the above methods won't help you anyway. They only >>> affect the run-time state of modules, i.e., the state known to the JVM >>> and surfaced in `java.lang.reflect.Module`. The JPMS resolver doesn't >>> read the run-time state but, rather, the configurations computed by >>> earlier invocations of the resolver, and ultimately those are all based >>> on module descriptors, over in the `java.lang.module` package. >> >> Currently the prototype code does not use the JPMS resolver. From my >> understanding, the JVM tables mediate access control, but the actual >> linkage between classes at run time happens via >> java.lang.ClassLoader#findClass(java.lang.String, java.lang.String). As >> long as this method resolves > > This sentence is incomplete, so I'm not sure what you meant to say here. Something like "as long as this method resolves classes in modules then we'll be fine". > At any rate, if you're not eventually going to use the JPMS resolver to > resolve some JPMS modules against some JBoss modules, in order to obtain > bidirectional interoperation as is now possible with OSGi [2], then why > bother with any of this? I don't want to. Even with all of our outstanding issues resolved, this end product is not what we'd hoped for and is at best a partial compromise, and in my mind represents far too radical a change in far too short of a time frame. In many ways it runs flatly counter to what we've learned and accomplished these past several years in our own architecture. The reality though is that if the platform includes a new deployment format with new behaviors, some users will soon expect that this format and its corresponding behaviors to be supported by containers, because it will be viewed by many to be a standard. It's possible that we can support this expectation to an extent simply by adding a separated JPMS container, and that would be a good option for that purpose. However that leaves our existing deployment model forever deficient because of its relative lack of encapsulation features, and to a lesser extent, reliance on "fake" module diagnostics, to speak nothing of compatibility oddities for users going from one to the other. If we want to bring our existing deployment model with all its features up to date with the new encapsulation capabilities, we either need to find a way to utilize Jigsaw to do it (which is what I'm trying to accomplish here), or else we need some new API which allows us to establish similar behaviors at a class loader level. Otherwise, you're right: if we can't take advantage of the useful capabilities of JPMS on our own terms, then why would we vote to support what amounts to a competing module architecture and deployment format, which actively undermines alternative systems which can't or won't abide by its restrictions (by the combination of establishing that the JPMS is not a meta-module system, yet also restricting key JVM features to only that system, and ensuring that differing behavioral characteristics (like version numbers or descriptor formats for example) either conform or are relegated to second-class behaviors), and which from our point of view has several critical deficiencies and substantial compatibility problems compared to what we have today? The only reason we would do, for the lack (via the above reasons) of inherently representing a good forward evolution for us, is if it provided a way for us to coexist while also giving us similar capabilities. Otherwise we frankly shouldn't support such a system at all, and I would hope to persuade others of the same. -- - DML From david.lloyd at redhat.com Wed Feb 15 22:57:51 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 15 Feb 2017 16:57:51 -0600 Subject: Closing out yet more open issues In-Reply-To: <20170215222956.D4DFF64ED1@eggemoggin.niobe.net> References: <20170215222956.D4DFF64ED1@eggemoggin.niobe.net> Message-ID: <7a946467-5970-9ce7-db84-523003e826fb@redhat.com> On 02/15/2017 04:29 PM, mark.reinhold at oracle.com wrote: > Proposals for the following issues have been available for evaluation > and experimentation for some time [1]: > > #AwkwardStrongEncapsulation > #ClassFileAccModule > #ClassFileModuleName > #IndirectQualifiedReflectiveAccess > #ModuleNameCharacters > #NonHierarchicalLayers > #ReadabilityAddedByLayerCreator > #ReflectiveAccessToNonExportedTypes > #VersionedDependences > > Responses to these proposals have been either neutral or positive, and > in some cases further suggestions have been adopted, so I've marked > these issues as closed. > > We're still discussing the new constraints on module names included in > the broader proposal for #VersionedDependences [2], but that's really > an amendment to the accepted proposal for #VersionsInModuleNames [3], > so I've closed the former issue and re-opened the latter. The discussion around Layer#addPackage(Module, String) is presently under the heading of #NonHierarchicalLayers and I consider it unresolved as of yet. Does it belong under a different issue? The lines seem a bit blurred as discussion goes on. -- - DML From mark.reinhold at oracle.com Wed Feb 15 23:39:23 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 15 Feb 2017 15:39:23 -0800 Subject: Closing out yet more open issues In-Reply-To: <7a946467-5970-9ce7-db84-523003e826fb@redhat.com> References: <20170215222956.D4DFF64ED1@eggemoggin.niobe.net> <7a946467-5970-9ce7-db84-523003e826fb@redhat.com> Message-ID: <20170215153923.921608933@eggemoggin.niobe.net> 2017/2/15 14:57:51 -0800, david.lloyd at redhat.com: > The discussion around Layer#addPackage(Module, String) is presently > under the heading of #NonHierarchicalLayers and I consider it unresolved > as of yet. Does it belong under a different issue? The lines seem a > bit blurred as discussion goes on. That thread wound up resuming where an earlier thread on a different set of issues (#MutableConfigurations, #LazyConfigurationAndInstantiation, #CyclicDependences, and #DiscardableModules) [1] tailed off. Let's call this general topic #LayerPrimitives; I'll add it to the list. - Mark [1] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-November/000458.html From mark.reinhold at oracle.com Thu Feb 16 16:48:27 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 16 Feb 2017 08:48:27 -0800 (PST) Subject: How to name modules, automatic and otherwise Message-ID: <20170216164827.A309F65E1B@eggemoggin.niobe.net> This note is in reply to the concerns about automatic modules raised by Robert Scholte and Brian Fox [1], and by Stephen Colebourne and others [2]. I've collected my conclusions here rather than in separate messages because there are several distinct yet intertwined issues. Summary: - Module names should not include Maven group identifiers, because modules are more abstract than the artifacts that define them. - Module names should use the reverse-domain-name-prefix convention or, preferably, the project-name-prefix convention. - We should not abandon automatic modules, since they are a key tool for migration and adoption. - We can address the problems of automatic modules with two fairly minor technical enhancements. If any of these points strikes you as controversial, please read on! * * * Module names should not include Maven group identifiers, as Robert Scholte and Brian Fox suggest [1], even for modules declared explicitly in `module-info.java` files. Modules in JPMS are a construct of the Java programming language, implemented in both the compiler and the virtual machine. As such, they are more abstract entities than the artifacts that define them. This distinction is useful, both conceptually and practically, hence module names should remain more abstract. This distinction is useful conceptually because it makes it easier, as we read source code, to think clearly about the nature of a module. We can reason about a module's dependences, exports, services, and so forth without cluttering our minds with the details of group identifiers and version constraints. Today, e.g., we can write, and read: module foo.data { exports com.bar.foo.data; requires hibernate.core; requires hibernate.jcache; requires hibernate.validator; } If we were to extend the syntax of module names to include group identifiers, and encourage people to use them, then we'd be faced with something much more verbose: module com.bar:foo.data { exports com.bar.foo.data; requires org.hibernate:hibernate.core; requires org.hibernate:hibernate.jcache; requires org.hibernate:hibernate.validator; } Group identifiers make perfect sense in the context of a build system such as Maven, where they bring necessary structure to the names of the millions of artifacts available across different repositories. Such structure is superfluous and distracting in the context of a module system, where the number of relevant modules in any particular situation is more likely to be in the tens, or hundreds, or (rarely) thousands. All else being equal, simpler names are better. At a practical level, the distinction between modules and artifacts is useful because it leaves the entire problem of artifact selection to the build system. This allows us to switch from one artifact to another simply by editing a `pom.xml` file to adjust a version constraint or a group identifier; if module names included group identifiers then we'd also have to edit the `module-info.java` file. This flexibility can be helpful if, e.g., a project is forked and a new module with the same name and artifact identifier is published under a different group identifier. We long ago decided not to do version selection in the module system, which surprised some people but has worked out fairly well. We should treat group selection in the same manner. Another practical benefit of the module/artifact distinction is that it keeps the module system independent of any particular build system, so that build systems can continue to improve and evolve independently over time. Maven-style coordinates are the most popular way to name artifacts in repositories today, but that might not be true ten years from now. It would be unwise to adopt Maven's naming convention for module names just because it's popular now, and doubly so to bake Maven's group-identifier concept into the Java programming language. * * * If module names don't include group identifiers, then how should modules be named? What advice should we give to someone who's creating a new module from scratch, or modularizing an existing component by writing a `module-info.java` file for it? (Continue to set aside, for the moment, the problems of automatic modules.) In structuring any particular space of names we must balance (at least) three fundamental tensions: We want names that are long enough to be descriptive, short enough to be memorable, and unique enough to avoid needless conflicts. If you control all of the modules upon which your module depends, and all of the modules that depend upon it, then you can of course name your module whatever you want, and change its name at any time. If, however, you're going to publish your module for use by others -- whether just within your own organization or to a global repository such as Maven Central -- then you should take more care. There are two well-known ways to go about this. - Choose module names that start with the reversed form of an Internet domain name that you control, or are at least associated with. The Java Language Specification has long suggested this convention as a way to minimize conflicts amongst package names, and it has been widely though not universally adopted for that purpose. - Choose module names that start with the name of your project or product. Module (and package) names that start with reversed domain names are less likely to conflict but they're unnecessarily verbose, they start with the least-important information (e.g., `com`, `org`, or `net`), and they don't read well after exogenous changes such as open-source donations or corporate acquisitions (e.g., `com.sun.*`). The reversed domain-name approach was sensible in the early days of Java, before we had development tools sophisticated enough to help us deal with the occasional conflict. We have such tools now, so going forward the superior readability of short module and package names that start with project or product names is preferable to the onerous verbosity of those that start with reversed domain names. This advice will strike some readers as controversial. I respect those who will choose, for the sake of tradition or an abundance of caution, to use the reversed domain-name convention for module names and continue to use that convention for package names. I do know, however, of at least one major, well-known project whose developers intend to adopt the project-name-prefix convention for their module names. * * * If module names don't include group identifiers, then how should automatic modules be named? Or are automatic modules so troublesome that we should remove them from the design? To answer the second question first: It would be a tragic shame to drop automatic modules, since otherwise top-down migration is impossible if you're not willing to modify artifacts that you don't maintain, which most people (quite sensibly) aren't. Even if you limit your use of automatic modules to closed systems, as Stephen Colebourne suggests [2], they're still of significant value. Let's see if we can rescue them. The present algorithm for naming automatic modules has two problems: (A) Conflicts are possible across large artifact repositories, since the name of an automatic module is computed from the name of the artifact that defines it. [1] (B) It's risky to publish a module that `requires` some other module that has not yet been modularized, and hence must be used as an automatic module. If the maintainer of that module later chooses an explicit name different from the automatic name then you must publish a new version of your module with an updated `requires` directive. [2] As to (A), yes, conflicts exist, though it's worth observing that many of the conflicts in the Maven Central data are due to poorly-chosen artifact names: `parent`, `library`, `core`, and `common` top the list, which then falls off in a long-tail distribution. When conflicts are detected then build tools can rename artifacts either automatically or, preferably, to user-specified names that map to sensible automatic-module names. If renaming artifacts in the filesystem proves impractical then we could extend the syntax of the `--module-path` option to allow a module name to be specified for each specifically-named artifact, though strictly speaking that would be a feature of the JDK rather than JPMS. We can address (B) by enabling the maintainers of existing components to specify the module names that should be given to their components when used as automatic modules, without having to write `module-info.java` files. This can be done very simply, with a single new JAR-file manifest `Module-Name` attribute, as first suggested almost a year ago [3]. If we add this one feature then the maintainer of an existing component that, e.g., must still build and run on JDK 7 can choose a module name for that component, record it in the manifest by adding a few lines to the `pom.xml` file, and tell users that they can use it as an automatic module on JDK 9 without fear that the module name will change when the component is properly modularized some years from now. The actual change to the component is small and low-risk, so it can reasonably be done in a patch release. There's no need to write a `module-info.java` file, and in fact doing so may be inadvisable at this point if the component depends on other components that have not yet been given module names. This approach for (B) does add one more (optional) step to the migration path, but it will hopefully lead to a larger number of explicitly-named modules in the world -- and in particular in Maven Central -- sooner rather than later. - Mark [1] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2017-January/000537.html [2] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-January/011106.html [3] http://openjdk.java.net/projects/jigsaw/spec/issues/#ModuleNameInManifest From forax at univ-mlv.fr Fri Feb 17 10:45:14 2017 From: forax at univ-mlv.fr (Remi Forax) Date: Fri, 17 Feb 2017 11:45:14 +0100 (CET) Subject: How to name modules, automatic and otherwise In-Reply-To: <20170216164827.A309F65E1B@eggemoggin.niobe.net> References: <20170216164827.A309F65E1B@eggemoggin.niobe.net> Message-ID: <1305027279.2009761.1487328314502.JavaMail.zimbra@u-pem.fr> I agree that - we should not bake the Maven way of specify identifiers into the language. - we should not drop automatic modules, this is a practical way to do the migration if you are maintaining an application. On the addition of Module-Name attribute, i can understand that it's a simple way to reserve a name. I will agree to introduce that in the spec only if Robert and Brian finds that an automatic module with a Module-Name attribute is something that can be pushed on Maven Central. Otherwise, specifying a Module-Name is not very different of writing a module-info.java with everything exported (i see the fact that a regular module can not access to the classpath unlike an automatic module as a bonus). I fully disagree on the idea of using short names for module, the Java community is too big for this kind of shortcut, the reverse DNS prefix have serve us well and counting the number of keystrokes is not a good metric. I've spend some time with Herve Boutemy last Tuesday night (and early Wednesday) to discuss about existing modules in Maven Central interropt. The Maven group id is something you have to ask before being able to publish on Maven Central, so it's more a Maven Central group id that a Maven group id. We should recommend that existing module on Maven Central should use a name that starts with the Maven Central group id. By example, Google's Guava should be com.google.guava or com.google.guava.core (or anything else that starts with com.google.guava). This will solve the name collisions issues between artifacts of Maven Central at least. R?mi ----- Mail original ----- > De: "mark reinhold" > ?: jpms-spec-experts at openjdk.java.net > Envoy?: Jeudi 16 F?vrier 2017 17:48:27 > Objet: How to name modules, automatic and otherwise > This note is in reply to the concerns about automatic modules raised by > Robert Scholte and Brian Fox [1], and by Stephen Colebourne and others > [2]. I've collected my conclusions here rather than in separate messages > because there are several distinct yet intertwined issues. > > Summary: > > - Module names should not include Maven group identifiers, because > modules are more abstract than the artifacts that define them. > > - Module names should use the reverse-domain-name-prefix convention > or, preferably, the project-name-prefix convention. > > - We should not abandon automatic modules, since they are a key tool > for migration and adoption. > > - We can address the problems of automatic modules with two fairly > minor technical enhancements. > > If any of these points strikes you as controversial, please read on! > > * * * > > Module names should not include Maven group identifiers, as Robert > Scholte and Brian Fox suggest [1], even for modules declared explicitly > in `module-info.java` files. Modules in JPMS are a construct of the Java > programming language, implemented in both the compiler and the virtual > machine. As such, they are more abstract entities than the artifacts > that define them. This distinction is useful, both conceptually and > practically, hence module names should remain more abstract. > > This distinction is useful conceptually because it makes it easier, as > we read source code, to think clearly about the nature of a module. We > can reason about a module's dependences, exports, services, and so forth > without cluttering our minds with the details of group identifiers and > version constraints. Today, e.g., we can write, and read: > > module foo.data { > exports com.bar.foo.data; > requires hibernate.core; > requires hibernate.jcache; > requires hibernate.validator; > } > > If we were to extend the syntax of module names to include group > identifiers, and encourage people to use them, then we'd be faced with > something much more verbose: > > module com.bar:foo.data { > exports com.bar.foo.data; > requires org.hibernate:hibernate.core; > requires org.hibernate:hibernate.jcache; > requires org.hibernate:hibernate.validator; > } > > Group identifiers make perfect sense in the context of a build system > such as Maven, where they bring necessary structure to the names of the > millions of artifacts available across different repositories. Such > structure is superfluous and distracting in the context of a module > system, where the number of relevant modules in any particular situation > is more likely to be in the tens, or hundreds, or (rarely) thousands. > All else being equal, simpler names are better. > > At a practical level, the distinction between modules and artifacts is > useful because it leaves the entire problem of artifact selection to the > build system. This allows us to switch from one artifact to another > simply by editing a `pom.xml` file to adjust a version constraint or a > group identifier; if module names included group identifiers then we'd > also have to edit the `module-info.java` file. This flexibility can be > helpful if, e.g., a project is forked and a new module with the same name > and artifact identifier is published under a different group identifier. > We long ago decided not to do version selection in the module system, > which surprised some people but has worked out fairly well. We should > treat group selection in the same manner. > > Another practical benefit of the module/artifact distinction is that it > keeps the module system independent of any particular build system, so > that build systems can continue to improve and evolve independently over > time. Maven-style coordinates are the most popular way to name artifacts > in repositories today, but that might not be true ten years from now. It > would be unwise to adopt Maven's naming convention for module names just > because it's popular now, and doubly so to bake Maven's group-identifier > concept into the Java programming language. > > * * * > > If module names don't include group identifiers, then how should modules > be named? What advice should we give to someone who's creating a new > module from scratch, or modularizing an existing component by writing a > `module-info.java` file for it? (Continue to set aside, for the moment, > the problems of automatic modules.) > > In structuring any particular space of names we must balance (at least) > three fundamental tensions: We want names that are long enough to be > descriptive, short enough to be memorable, and unique enough to avoid > needless conflicts. > > If you control all of the modules upon which your module depends, and > all of the modules that depend upon it, then you can of course name your > module whatever you want, and change its name at any time. If, however, > you're going to publish your module for use by others -- whether just > within your own organization or to a global repository such as Maven > Central -- then you should take more care. There are two well-known > ways to go about this. > > - Choose module names that start with the reversed form of an Internet > domain name that you control, or are at least associated with. The > Java Language Specification has long suggested this convention as a > way to minimize conflicts amongst package names, and it has been > widely though not universally adopted for that purpose. > > - Choose module names that start with the name of your project or > product. Module (and package) names that start with reversed domain > names are less likely to conflict but they're unnecessarily verbose, > they start with the least-important information (e.g., `com`, `org`, > or `net`), and they don't read well after exogenous changes such as > open-source donations or corporate acquisitions (e.g., `com.sun.*`). > > The reversed domain-name approach was sensible in the early days of Java, > before we had development tools sophisticated enough to help us deal with > the occasional conflict. We have such tools now, so going forward the > superior readability of short module and package names that start with > project or product names is preferable to the onerous verbosity of those > that start with reversed domain names. > > This advice will strike some readers as controversial. I respect those > who will choose, for the sake of tradition or an abundance of caution, to > use the reversed domain-name convention for module names and continue to > use that convention for package names. I do know, however, of at least > one major, well-known project whose developers intend to adopt the > project-name-prefix convention for their module names. > > * * * > > If module names don't include group identifiers, then how should automatic > modules be named? Or are automatic modules so troublesome that we should > remove them from the design? > > To answer the second question first: It would be a tragic shame to drop > automatic modules, since otherwise top-down migration is impossible if > you're not willing to modify artifacts that you don't maintain, which > most people (quite sensibly) aren't. Even if you limit your use of > automatic modules to closed systems, as Stephen Colebourne suggests [2], > they're still of significant value. Let's see if we can rescue them. > > The present algorithm for naming automatic modules has two problems: > > (A) Conflicts are possible across large artifact repositories, since > the name of an automatic module is computed from the name of the > artifact that defines it. [1] > > (B) It's risky to publish a module that `requires` some other module > that has not yet been modularized, and hence must be used as an > automatic module. If the maintainer of that module later chooses > an explicit name different from the automatic name then you must > publish a new version of your module with an updated `requires` > directive. [2] > > As to (A), yes, conflicts exist, though it's worth observing that many of > the conflicts in the Maven Central data are due to poorly-chosen artifact > names: `parent`, `library`, `core`, and `common` top the list, which then > falls off in a long-tail distribution. When conflicts are detected then > build tools can rename artifacts either automatically or, preferably, to > user-specified names that map to sensible automatic-module names. If > renaming artifacts in the filesystem proves impractical then we could > extend the syntax of the `--module-path` option to allow a module name > to be specified for each specifically-named artifact, though strictly > speaking that would be a feature of the JDK rather than JPMS. > > We can address (B) by enabling the maintainers of existing components to > specify the module names that should be given to their components when > used as automatic modules, without having to write `module-info.java` > files. This can be done very simply, with a single new JAR-file manifest > `Module-Name` attribute, as first suggested almost a year ago [3]. > > If we add this one feature then the maintainer of an existing component > that, e.g., must still build and run on JDK 7 can choose a module name > for that component, record it in the manifest by adding a few lines to > the `pom.xml` file, and tell users that they can use it as an automatic > module on JDK 9 without fear that the module name will change when the > component is properly modularized some years from now. The actual change > to the component is small and low-risk, so it can reasonably be done in > a patch release. There's no need to write a `module-info.java` file, > and in fact doing so may be inadvisable at this point if the component > depends on other components that have not yet been given module names. > > This approach for (B) does add one more (optional) step to the migration > path, but it will hopefully lead to a larger number of explicitly-named > modules in the world -- and in particular in Maven Central -- sooner > rather than later. > > - Mark > > > [1] > http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2017-January/000537.html > [2] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-January/011106.html > [3] http://openjdk.java.net/projects/jigsaw/spec/issues/#ModuleNameInManifest From rfscholte at apache.org Mon Feb 20 17:50:38 2017 From: rfscholte at apache.org (Robert Scholte) Date: Mon, 20 Feb 2017 18:50:38 +0100 Subject: How to name modules, automatic and otherwise In-Reply-To: <1305027279.2009761.1487328314502.JavaMail.zimbra@u-pem.fr> References: <20170216164827.A309F65E1B@eggemoggin.niobe.net> <1305027279.2009761.1487328314502.JavaMail.zimbra@u-pem.fr> Message-ID: On Fri, 17 Feb 2017 11:45:14 +0100, Remi Forax wrote: > I agree that > - we should not bake the Maven way of specify identifiers into the > language. > - we should not drop automatic modules, this is a practical way to do > the migration if you are maintaining an application. > > On the addition of Module-Name attribute, i can understand that it's a > simple way to reserve a name. I will agree to introduce that in the spec > only if Robert and Brian finds that an automatic module with a > Module-Name attribute is something that can be pushed on Maven Central. > Otherwise, specifying a Module-Name is not very different of writing a > module-info.java with everything exported (i see the fact that a regular > module can not access to the classpath unlike an automatic module as a > bonus). This looks fine to me. It will close the gap where you want to be able to give your project a module name for others to be used, in case you cannot write a module-info file because it would depend on automodules. I'm investigating some ideas which should prevent projects specifying references to automodules to be published to Central. > > I fully disagree on the idea of using short names for module, the Java > community is too big for this kind of shortcut, the reverse DNS prefix > have serve us well and counting the number of keystrokes is not a good > metric. > I've spend some time with Herve Boutemy last Tuesday night (and early > Wednesday) to discuss about existing modules in Maven Central interropt. > The Maven group id is something you have to ask before being able to > publish on Maven Central, so it's more a Maven Central group id that a > Maven group id. We should recommend that existing module on Maven > Central should use a name that starts with the Maven Central group id. > By example, Google's Guava should be com.google.guava or > com.google.guava.core (or anything else that starts with > com.google.guava). > > This will solve the name collisions issues between artifacts of Maven > Central at least. From JPMS I understand that in the end this is simply an identifier. What it looks like, it doesn't matter, as long as it is uniquely resolved on the module path. There is this kind of trust where developers should pick there module name wisely, otherwise their project won't work in a modular world. In case of classes there are the same issues, but there were these tricks of class path order or package relocation to solve collisions, but that's not possible with modules. Choosing the right name will be very important, and it better not match only the artifactId. thanks, Robert > > R?mi > > ----- Mail original ----- >> De: "mark reinhold" >> ?: jpms-spec-experts at openjdk.java.net >> Envoy?: Jeudi 16 F?vrier 2017 17:48:27 >> Objet: How to name modules, automatic and otherwise > >> This note is in reply to the concerns about automatic modules raised by >> Robert Scholte and Brian Fox [1], and by Stephen Colebourne and others >> [2]. I've collected my conclusions here rather than in separate >> messages >> because there are several distinct yet intertwined issues. >> >> Summary: >> >> - Module names should not include Maven group identifiers, because >> modules are more abstract than the artifacts that define them. >> >> - Module names should use the reverse-domain-name-prefix convention >> or, preferably, the project-name-prefix convention. >> >> - We should not abandon automatic modules, since they are a key tool >> for migration and adoption. >> >> - We can address the problems of automatic modules with two fairly >> minor technical enhancements. >> >> If any of these points strikes you as controversial, please read on! >> >> * * * >> >> Module names should not include Maven group identifiers, as Robert >> Scholte and Brian Fox suggest [1], even for modules declared explicitly >> in `module-info.java` files. Modules in JPMS are a construct of the >> Java >> programming language, implemented in both the compiler and the virtual >> machine. As such, they are more abstract entities than the artifacts >> that define them. This distinction is useful, both conceptually and >> practically, hence module names should remain more abstract. >> >> This distinction is useful conceptually because it makes it easier, as >> we read source code, to think clearly about the nature of a module. We >> can reason about a module's dependences, exports, services, and so forth >> without cluttering our minds with the details of group identifiers and >> version constraints. Today, e.g., we can write, and read: >> >> module foo.data { >> exports com.bar.foo.data; >> requires hibernate.core; >> requires hibernate.jcache; >> requires hibernate.validator; >> } >> >> If we were to extend the syntax of module names to include group >> identifiers, and encourage people to use them, then we'd be faced with >> something much more verbose: >> >> module com.bar:foo.data { >> exports com.bar.foo.data; >> requires org.hibernate:hibernate.core; >> requires org.hibernate:hibernate.jcache; >> requires org.hibernate:hibernate.validator; >> } >> >> Group identifiers make perfect sense in the context of a build system >> such as Maven, where they bring necessary structure to the names of the >> millions of artifacts available across different repositories. Such >> structure is superfluous and distracting in the context of a module >> system, where the number of relevant modules in any particular situation >> is more likely to be in the tens, or hundreds, or (rarely) thousands. >> All else being equal, simpler names are better. >> >> At a practical level, the distinction between modules and artifacts is >> useful because it leaves the entire problem of artifact selection to the >> build system. This allows us to switch from one artifact to another >> simply by editing a `pom.xml` file to adjust a version constraint or a >> group identifier; if module names included group identifiers then we'd >> also have to edit the `module-info.java` file. This flexibility can be >> helpful if, e.g., a project is forked and a new module with the same >> name >> and artifact identifier is published under a different group identifier. >> We long ago decided not to do version selection in the module system, >> which surprised some people but has worked out fairly well. We should >> treat group selection in the same manner. >> >> Another practical benefit of the module/artifact distinction is that it >> keeps the module system independent of any particular build system, so >> that build systems can continue to improve and evolve independently over >> time. Maven-style coordinates are the most popular way to name >> artifacts >> in repositories today, but that might not be true ten years from now. >> It >> would be unwise to adopt Maven's naming convention for module names just >> because it's popular now, and doubly so to bake Maven's group-identifier >> concept into the Java programming language. >> >> * * * >> >> If module names don't include group identifiers, then how should modules >> be named? What advice should we give to someone who's creating a new >> module from scratch, or modularizing an existing component by writing a >> `module-info.java` file for it? (Continue to set aside, for the moment, >> the problems of automatic modules.) >> >> In structuring any particular space of names we must balance (at least) >> three fundamental tensions: We want names that are long enough to be >> descriptive, short enough to be memorable, and unique enough to avoid >> needless conflicts. >> >> If you control all of the modules upon which your module depends, and >> all of the modules that depend upon it, then you can of course name your >> module whatever you want, and change its name at any time. If, however, >> you're going to publish your module for use by others -- whether just >> within your own organization or to a global repository such as Maven >> Central -- then you should take more care. There are two well-known >> ways to go about this. >> >> - Choose module names that start with the reversed form of an Internet >> domain name that you control, or are at least associated with. The >> Java Language Specification has long suggested this convention as a >> way to minimize conflicts amongst package names, and it has been >> widely though not universally adopted for that purpose. >> >> - Choose module names that start with the name of your project or >> product. Module (and package) names that start with reversed domain >> names are less likely to conflict but they're unnecessarily verbose, >> they start with the least-important information (e.g., `com`, `org`, >> or `net`), and they don't read well after exogenous changes such as >> open-source donations or corporate acquisitions (e.g., `com.sun.*`). >> >> The reversed domain-name approach was sensible in the early days of >> Java, >> before we had development tools sophisticated enough to help us deal >> with >> the occasional conflict. We have such tools now, so going forward the >> superior readability of short module and package names that start with >> project or product names is preferable to the onerous verbosity of those >> that start with reversed domain names. >> >> This advice will strike some readers as controversial. I respect those >> who will choose, for the sake of tradition or an abundance of caution, >> to >> use the reversed domain-name convention for module names and continue to >> use that convention for package names. I do know, however, of at least >> one major, well-known project whose developers intend to adopt the >> project-name-prefix convention for their module names. >> >> * * * >> >> If module names don't include group identifiers, then how should >> automatic >> modules be named? Or are automatic modules so troublesome that we >> should >> remove them from the design? >> >> To answer the second question first: It would be a tragic shame to drop >> automatic modules, since otherwise top-down migration is impossible if >> you're not willing to modify artifacts that you don't maintain, which >> most people (quite sensibly) aren't. Even if you limit your use of >> automatic modules to closed systems, as Stephen Colebourne suggests [2], >> they're still of significant value. Let's see if we can rescue them. >> >> The present algorithm for naming automatic modules has two problems: >> >> (A) Conflicts are possible across large artifact repositories, since >> the name of an automatic module is computed from the name of the >> artifact that defines it. [1] >> >> (B) It's risky to publish a module that `requires` some other module >> that has not yet been modularized, and hence must be used as an >> automatic module. If the maintainer of that module later chooses >> an explicit name different from the automatic name then you must >> publish a new version of your module with an updated `requires` >> directive. [2] >> >> As to (A), yes, conflicts exist, though it's worth observing that many >> of >> the conflicts in the Maven Central data are due to poorly-chosen >> artifact >> names: `parent`, `library`, `core`, and `common` top the list, which >> then >> falls off in a long-tail distribution. When conflicts are detected then >> build tools can rename artifacts either automatically or, preferably, to >> user-specified names that map to sensible automatic-module names. If >> renaming artifacts in the filesystem proves impractical then we could >> extend the syntax of the `--module-path` option to allow a module name >> to be specified for each specifically-named artifact, though strictly >> speaking that would be a feature of the JDK rather than JPMS. >> >> We can address (B) by enabling the maintainers of existing components to >> specify the module names that should be given to their components when >> used as automatic modules, without having to write `module-info.java` >> files. This can be done very simply, with a single new JAR-file >> manifest >> `Module-Name` attribute, as first suggested almost a year ago [3]. >> >> If we add this one feature then the maintainer of an existing component >> that, e.g., must still build and run on JDK 7 can choose a module name >> for that component, record it in the manifest by adding a few lines to >> the `pom.xml` file, and tell users that they can use it as an automatic >> module on JDK 9 without fear that the module name will change when the >> component is properly modularized some years from now. The actual >> change >> to the component is small and low-risk, so it can reasonably be done in >> a patch release. There's no need to write a `module-info.java` file, >> and in fact doing so may be inadvisable at this point if the component >> depends on other components that have not yet been given module names. >> >> This approach for (B) does add one more (optional) step to the migration >> path, but it will hopefully lead to a larger number of explicitly-named >> modules in the world -- and in particular in Maven Central -- sooner >> rather than later. >> >> - Mark >> >> >> [1] >> http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2017-January/000537.html >> [2] >> http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-January/011106.html >> [3] >> http://openjdk.java.net/projects/jigsaw/spec/issues/#ModuleNameInManifest From mark.reinhold at oracle.com Tue Feb 21 16:04:42 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 21 Feb 2017 08:04:42 -0800 (PST) Subject: #LayerPrimitives (was Re: Proposal: #NonHierarchicalLayers) In-Reply-To: <713a39e0-4b76-0a28-2f91-cd7cb1655fcb@redhat.com> References: <20161207234117.1E5F1213F1@eggemoggin.niobe.net> <1de06198-930b-4b71-fe5d-1477d9cace80@redhat.com> <20161212232335.A9BFC24B27@eggemoggin.niobe.net> <9b66aa52-877a-e4b2-1458-9ff47dfb690b@redhat.com> <20170126223431.B417255DD9@eggemoggin.niobe.net> <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> <20170215195108.C881964E6F@eggemoggin.niobe.net> <713a39e0-4b76-0a28-2f91-cd7cb1655fcb@redhat.com> Message-ID: <20170221160442.3348866481@eggemoggin.niobe.net> 2017/2/15 14:32:06 -0800, david.lloyd at redhat.com: > On 02/15/2017 01:51 PM, mark.reinhold at oracle.com wrote: >> ... >> >> At any rate, if you're not eventually going to use the JPMS resolver to >> resolve some JPMS modules against some JBoss modules, in order to obtain >> bidirectional interoperation as is now possible with OSGi [2], then why >> bother with any of this? > > I don't want to. Even with all of our outstanding issues resolved, this > end product is not what we'd hoped for and is at best a partial > compromise, and in my mind represents far too radical a change in far > too short of a time frame. In many ways it runs flatly counter to what > we've learned and accomplished these past several years in our own > architecture. > > The reality though is that if the platform includes a new deployment > format with new behaviors, some users will soon expect that this format > and its corresponding behaviors to be supported by containers, because > it will be viewed by many to be a standard. > > It's possible that we can support this expectation to an extent simply > by adding a separated JPMS container, and that would be a good option > for that purpose. However that leaves our existing deployment model > forever deficient because of its relative lack of encapsulation > features, and to a lesser extent, reliance on "fake" module diagnostics, > to speak nothing of compatibility oddities for users going from one to > the other. If we want to bring our existing deployment model with all > its features up to date with the new encapsulation capabilities, we > either need to find a way to utilize Jigsaw to do it (which is what I'm > trying to accomplish here), or else we need some new API which allows us > to establish similar behaviors at a class loader level. > > Otherwise, you're right: if we can't take advantage of the useful > capabilities of JPMS on our own terms, then why would we vote to support > what amounts to a competing module architecture and deployment format, > which actively undermines alternative systems which can't or won't abide > by its restrictions (by the combination of establishing that the JPMS is > not a meta-module system, yet also restricting key JVM features to only > that system, and ensuring that differing behavioral characteristics > (like version numbers or descriptor formats for example) either conform > or are relegated to second-class behaviors), and which from our point of > view has several critical deficiencies and substantial compatibility > problems compared to what we have today? > > The only reason we would do, for the lack (via the above reasons) of > inherently representing a good forward evolution for us, is if it > provided a way for us to coexist while also giving us similar > capabilities. Otherwise we frankly shouldn't support such a system at > all, and I would hope to persuade others of the same. I would be disappointed indeed to see you conclude that it is more important to protect and promote your home-grown, non-standard module system, which appears not to be used much outside of the JBoss/Wildfly ecosystem, than it is to support this JSR, which aims to bring significant benefits in a standard way to the entire Java ecosystem. The goal of this JSR is to design a module system approachable by all Java developers and suitable for the modularization of the platform itself, as stated in the initial JSR submission [1]. It is not a goal to design a "meta" module system via which multiple module systems can interoperate on an intimate basis, as I have observed at least twice in the past [2][3]. That would be a far larger and even more difficult project that is clearly beyond the scope of the JSR, despite your wishes to the contrary. In order to achieve this JSR's goal there is no need to expose the low-level primitives upon which we have implemented this module system, as you request. Those primitives were not designed for use outside of the implementation itself, since designing a meta-module system is not a goal. If that had been a goal then we would have started by studying the internals of many existing module systems in order to identify a workable set of primitives. That is not what we have done, however, and without having done that I cannot be confident that what we have today is the right set of primitives for this non-goal. My caution here is not merely abstract: It is based upon twenty years of sometimes-painful experience. I have too often seen APIs that seemed like a good idea at the time but were, in fact, woefully deficient, baked into the Java Platform where they fester for ages, cause pain to all who use it, and torment those who maintain it. I will not let that happen here -- especially not with primitive, low-level operations that could be difficult to emulate later on if a better set of primitives is found in some future design effort. At this point we have met the agreed requirements with respect to other module systems, namely to ensure that they can run on top of JPMS [4] and that they can read JPMS module metadata in order to resolve JPMS modules on their own terms, if so desired [5]. We have even extended JPMS to enable bidirectional interoperation with OSGi containers [6], a change that goes above and beyond the agreed requirements yet is still within the spirit of the original JSR. On this basis I intend to close the following issues without making the changes that they suggest. Those changes add complexity and risk, and are not necessary to the achievement of this JSR's stated goal. #CyclicDependences #DiscardableModules #LazyConfigurationAndInstantiation #MutableConfigurations #LayerPrimitives - Mark [1] https://www.jcp.org/en/jsr/detail?id=376#2 [2] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-February/000019.html [3] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-July/000377.html [4] http://openjdk.java.net/projects/jigsaw/spec/reqs/#compatible-java-platform-modularization [5] http://openjdk.java.net/projects/jigsaw/spec/reqs/#interoperation [6] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000410.html From david.lloyd at redhat.com Tue Feb 21 17:05:22 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 21 Feb 2017 11:05:22 -0600 Subject: #LayerPrimitives (was Re: Proposal: #NonHierarchicalLayers) In-Reply-To: <20170221160442.3348866481@eggemoggin.niobe.net> References: <20161207234117.1E5F1213F1@eggemoggin.niobe.net> <1de06198-930b-4b71-fe5d-1477d9cace80@redhat.com> <20161212232335.A9BFC24B27@eggemoggin.niobe.net> <9b66aa52-877a-e4b2-1458-9ff47dfb690b@redhat.com> <20170126223431.B417255DD9@eggemoggin.niobe.net> <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> <20170215195108.C881964E6F@eggemoggin.niobe.net> <713a39e0-4b76-0a28-2f91-cd7cb1655fcb@redhat.com> <20170221160442.3348866481@eggemoggin.niobe.net> Message-ID: Logically a non-goal is not an anti-goal, as I have observed many times in the past as well. Not setting out to do something is not the same thing as forbidding it from being done, especially if it is a minimal change. We consider these issue inadequately addressed despite being rejected. By JCP procedures, this obligates us to vote "no". Is there no possible compromise? On 02/21/2017 10:04 AM, mark.reinhold at oracle.com wrote: > 2017/2/15 14:32:06 -0800, david.lloyd at redhat.com: >> On 02/15/2017 01:51 PM, mark.reinhold at oracle.com wrote: >>> ... >>> >>> At any rate, if you're not eventually going to use the JPMS resolver to >>> resolve some JPMS modules against some JBoss modules, in order to obtain >>> bidirectional interoperation as is now possible with OSGi [2], then why >>> bother with any of this? >> >> I don't want to. Even with all of our outstanding issues resolved, this >> end product is not what we'd hoped for and is at best a partial >> compromise, and in my mind represents far too radical a change in far >> too short of a time frame. In many ways it runs flatly counter to what >> we've learned and accomplished these past several years in our own >> architecture. >> >> The reality though is that if the platform includes a new deployment >> format with new behaviors, some users will soon expect that this format >> and its corresponding behaviors to be supported by containers, because >> it will be viewed by many to be a standard. >> >> It's possible that we can support this expectation to an extent simply >> by adding a separated JPMS container, and that would be a good option >> for that purpose. However that leaves our existing deployment model >> forever deficient because of its relative lack of encapsulation >> features, and to a lesser extent, reliance on "fake" module diagnostics, >> to speak nothing of compatibility oddities for users going from one to >> the other. If we want to bring our existing deployment model with all >> its features up to date with the new encapsulation capabilities, we >> either need to find a way to utilize Jigsaw to do it (which is what I'm >> trying to accomplish here), or else we need some new API which allows us >> to establish similar behaviors at a class loader level. >> >> Otherwise, you're right: if we can't take advantage of the useful >> capabilities of JPMS on our own terms, then why would we vote to support >> what amounts to a competing module architecture and deployment format, >> which actively undermines alternative systems which can't or won't abide >> by its restrictions (by the combination of establishing that the JPMS is >> not a meta-module system, yet also restricting key JVM features to only >> that system, and ensuring that differing behavioral characteristics >> (like version numbers or descriptor formats for example) either conform >> or are relegated to second-class behaviors), and which from our point of >> view has several critical deficiencies and substantial compatibility >> problems compared to what we have today? >> >> The only reason we would do, for the lack (via the above reasons) of >> inherently representing a good forward evolution for us, is if it >> provided a way for us to coexist while also giving us similar >> capabilities. Otherwise we frankly shouldn't support such a system at >> all, and I would hope to persuade others of the same. > > I would be disappointed indeed to see you conclude that it is more > important to protect and promote your home-grown, non-standard module > system, which appears not to be used much outside of the JBoss/Wildfly > ecosystem, than it is to support this JSR, which aims to bring > significant benefits in a standard way to the entire Java ecosystem. > > The goal of this JSR is to design a module system approachable by all > Java developers and suitable for the modularization of the platform > itself, as stated in the initial JSR submission [1]. It is not a goal > to design a "meta" module system via which multiple module systems can > interoperate on an intimate basis, as I have observed at least twice in > the past [2][3]. That would be a far larger and even more difficult > project that is clearly beyond the scope of the JSR, despite your > wishes to the contrary. > > In order to achieve this JSR's goal there is no need to expose the > low-level primitives upon which we have implemented this module system, > as you request. Those primitives were not designed for use outside of > the implementation itself, since designing a meta-module system is not a > goal. If that had been a goal then we would have started by studying the > internals of many existing module systems in order to identify a workable > set of primitives. That is not what we have done, however, and without > having done that I cannot be confident that what we have today is the > right set of primitives for this non-goal. > > My caution here is not merely abstract: It is based upon twenty years of > sometimes-painful experience. I have too often seen APIs that seemed > like a good idea at the time but were, in fact, woefully deficient, baked > into the Java Platform where they fester for ages, cause pain to all who > use it, and torment those who maintain it. I will not let that happen > here -- especially not with primitive, low-level operations that could be > difficult to emulate later on if a better set of primitives is found in > some future design effort. > > At this point we have met the agreed requirements with respect to other > module systems, namely to ensure that they can run on top of JPMS [4] and > that they can read JPMS module metadata in order to resolve JPMS modules > on their own terms, if so desired [5]. We have even extended JPMS to > enable bidirectional interoperation with OSGi containers [6], a change > that goes above and beyond the agreed requirements yet is still within > the spirit of the original JSR. > > On this basis I intend to close the following issues without making the > changes that they suggest. Those changes add complexity and risk, and > are not necessary to the achievement of this JSR's stated goal. > > #CyclicDependences > #DiscardableModules > #LazyConfigurationAndInstantiation > #MutableConfigurations > #LayerPrimitives I'm very sorry to see that this is the case. -- - DML From david.lloyd at redhat.com Wed Feb 22 14:57:23 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 22 Feb 2017 08:57:23 -0600 Subject: #LayerPrimitives (was Re: Proposal: #NonHierarchicalLayers) In-Reply-To: References: <20161207234117.1E5F1213F1@eggemoggin.niobe.net> <1de06198-930b-4b71-fe5d-1477d9cace80@redhat.com> <20161212232335.A9BFC24B27@eggemoggin.niobe.net> <9b66aa52-877a-e4b2-1458-9ff47dfb690b@redhat.com> <20170126223431.B417255DD9@eggemoggin.niobe.net> <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> <20170215195108.C881964E6F@eggemoggin.niobe.net> <713a39e0-4b76-0a28-2f91-cd7cb1655fcb@redhat.com> <20170221160442.3348866481@eggemoggin.niobe.net> Message-ID: <8f9b3cf9-7809-dc91-d3b3-af82814447d0@redhat.com> One thing that is very unclear to me is, what is the logic behind decreeing that a module can add "exports" and "uses" on its own but a LayerController cannot do so for a module in a layer it directly controls, unless it generates & injects bytecode to do this? On 02/21/2017 11:05 AM, David M. Lloyd wrote: > Logically a non-goal is not an anti-goal, as I have observed many times > in the past as well. Not setting out to do something is not the same > thing as forbidding it from being done, especially if it is a minimal > change. > > We consider these issue inadequately addressed despite being rejected. > By JCP procedures, this obligates us to vote "no". Is there no possible > compromise? > > On 02/21/2017 10:04 AM, mark.reinhold at oracle.com wrote: >> 2017/2/15 14:32:06 -0800, david.lloyd at redhat.com: >>> On 02/15/2017 01:51 PM, mark.reinhold at oracle.com wrote: >>>> ... >>>> >>>> At any rate, if you're not eventually going to use the JPMS resolver to >>>> resolve some JPMS modules against some JBoss modules, in order to >>>> obtain >>>> bidirectional interoperation as is now possible with OSGi [2], then why >>>> bother with any of this? >>> >>> I don't want to. Even with all of our outstanding issues resolved, this >>> end product is not what we'd hoped for and is at best a partial >>> compromise, and in my mind represents far too radical a change in far >>> too short of a time frame. In many ways it runs flatly counter to what >>> we've learned and accomplished these past several years in our own >>> architecture. >>> >>> The reality though is that if the platform includes a new deployment >>> format with new behaviors, some users will soon expect that this format >>> and its corresponding behaviors to be supported by containers, because >>> it will be viewed by many to be a standard. >>> >>> It's possible that we can support this expectation to an extent simply >>> by adding a separated JPMS container, and that would be a good option >>> for that purpose. However that leaves our existing deployment model >>> forever deficient because of its relative lack of encapsulation >>> features, and to a lesser extent, reliance on "fake" module diagnostics, >>> to speak nothing of compatibility oddities for users going from one to >>> the other. If we want to bring our existing deployment model with all >>> its features up to date with the new encapsulation capabilities, we >>> either need to find a way to utilize Jigsaw to do it (which is what I'm >>> trying to accomplish here), or else we need some new API which allows us >>> to establish similar behaviors at a class loader level. >>> >>> Otherwise, you're right: if we can't take advantage of the useful >>> capabilities of JPMS on our own terms, then why would we vote to support >>> what amounts to a competing module architecture and deployment format, >>> which actively undermines alternative systems which can't or won't abide >>> by its restrictions (by the combination of establishing that the JPMS is >>> not a meta-module system, yet also restricting key JVM features to only >>> that system, and ensuring that differing behavioral characteristics >>> (like version numbers or descriptor formats for example) either conform >>> or are relegated to second-class behaviors), and which from our point of >>> view has several critical deficiencies and substantial compatibility >>> problems compared to what we have today? >>> >>> The only reason we would do, for the lack (via the above reasons) of >>> inherently representing a good forward evolution for us, is if it >>> provided a way for us to coexist while also giving us similar >>> capabilities. Otherwise we frankly shouldn't support such a system at >>> all, and I would hope to persuade others of the same. >> >> I would be disappointed indeed to see you conclude that it is more >> important to protect and promote your home-grown, non-standard module >> system, which appears not to be used much outside of the JBoss/Wildfly >> ecosystem, than it is to support this JSR, which aims to bring >> significant benefits in a standard way to the entire Java ecosystem. >> >> The goal of this JSR is to design a module system approachable by all >> Java developers and suitable for the modularization of the platform >> itself, as stated in the initial JSR submission [1]. It is not a goal >> to design a "meta" module system via which multiple module systems can >> interoperate on an intimate basis, as I have observed at least twice in >> the past [2][3]. That would be a far larger and even more difficult >> project that is clearly beyond the scope of the JSR, despite your >> wishes to the contrary. >> >> In order to achieve this JSR's goal there is no need to expose the >> low-level primitives upon which we have implemented this module system, >> as you request. Those primitives were not designed for use outside of >> the implementation itself, since designing a meta-module system is not a >> goal. If that had been a goal then we would have started by studying the >> internals of many existing module systems in order to identify a workable >> set of primitives. That is not what we have done, however, and without >> having done that I cannot be confident that what we have today is the >> right set of primitives for this non-goal. >> >> My caution here is not merely abstract: It is based upon twenty years of >> sometimes-painful experience. I have too often seen APIs that seemed >> like a good idea at the time but were, in fact, woefully deficient, baked >> into the Java Platform where they fester for ages, cause pain to all who >> use it, and torment those who maintain it. I will not let that happen >> here -- especially not with primitive, low-level operations that could be >> difficult to emulate later on if a better set of primitives is found in >> some future design effort. >> >> At this point we have met the agreed requirements with respect to other >> module systems, namely to ensure that they can run on top of JPMS [4] and >> that they can read JPMS module metadata in order to resolve JPMS modules >> on their own terms, if so desired [5]. We have even extended JPMS to >> enable bidirectional interoperation with OSGi containers [6], a change >> that goes above and beyond the agreed requirements yet is still within >> the spirit of the original JSR. >> >> On this basis I intend to close the following issues without making the >> changes that they suggest. Those changes add complexity and risk, and >> are not necessary to the achievement of this JSR's stated goal. >> >> #CyclicDependences >> #DiscardableModules >> #LazyConfigurationAndInstantiation >> #MutableConfigurations >> #LayerPrimitives > > I'm very sorry to see that this is the case. -- - DML From david.lloyd at redhat.com Wed Feb 22 15:17:09 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 22 Feb 2017 09:17:09 -0600 Subject: #LayerPrimitives (was Re: Proposal: #NonHierarchicalLayers) In-Reply-To: <20170221160442.3348866481@eggemoggin.niobe.net> References: <20161207234117.1E5F1213F1@eggemoggin.niobe.net> <1de06198-930b-4b71-fe5d-1477d9cace80@redhat.com> <20161212232335.A9BFC24B27@eggemoggin.niobe.net> <9b66aa52-877a-e4b2-1458-9ff47dfb690b@redhat.com> <20170126223431.B417255DD9@eggemoggin.niobe.net> <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> <20170215195108.C881964E6F@eggemoggin.niobe.net> <713a39e0-4b76-0a28-2f91-cd7cb1655fcb@redhat.com> <20170221160442.3348866481@eggemoggin.niobe.net> Message-ID: I want to address a few non-technical matters in this response, separately. On 02/21/2017 10:04 AM, mark.reinhold at oracle.com wrote: > I would be disappointed indeed to see you conclude that it is more > important to protect and promote your home-grown, non-standard module > system, which appears not to be used much outside of the JBoss/Wildfly > ecosystem, than it is to support this JSR, which aims to bring > significant benefits in a standard way to the entire Java ecosystem. You are presenting a false dichotomy here, that our "home-grown, non-standard" module system is not aiming to bring significant benefits in a standard way to the entire Java ecosystem. I contend that we do bring significant benefits, and we see adoption not only from our JBoss and WildFly ecosystem (which is itself not as inconsequential as you might make it seem). We have a *very* substantial number of real-world users that already take advantage of the benefits of modular development. And with that user base comes a great deal of experience in the things that work and the things that don't. I've tried again and again to share this experience but it has always been dismissed. > My caution here is not merely abstract: It is based upon twenty years of > sometimes-painful experience. I have too often seen APIs that seemed > like a good idea at the time but were, in fact, woefully deficient, baked > into the Java Platform where they fester for ages, cause pain to all who > use it, and torment those who maintain it. I will not let that happen Can you deny that this ivory tower "daddy knows best" rhetoric is exactly what has lead to the very baked-in, painfully deficient APIs that you refer to? What about real-world experience? > At this point we have met the agreed requirements with respect to other > module systems, namely to ensure that they can run on top of JPMS [4] and > that they can read JPMS module metadata in order to resolve JPMS modules > on their own terms, if so desired [5]. We have even extended JPMS to > enable bidirectional interoperation with OSGi containers [6], a change > that goes above and beyond the agreed requirements yet is still within > the spirit of the original JSR. > > On this basis I intend to close the following issues without making the > changes that they suggest. Those changes add complexity and risk, and > are not necessary to the achievement of this JSR's stated goal. > > #CyclicDependences > #DiscardableModules > #LazyConfigurationAndInstantiation > #MutableConfigurations > #LayerPrimitives I want to point out that #LayerPrimitives adds very, very little complexity and very little risk. The most extensive version of the patch only adds 40-odd lines of quite defensive code, plus JavaDoc. Most of the functions of the proposed API are already possible to do, but require Layer providers to inject bytecode into their Modules, which is unreasonably onerous. And the last function to add a package to a module is the only possible way to enable the ability to supplement deployments or generate bytecode into a predictable package (barring reserving (many?) such packages ahead of time); it's very simple in and of itself and uses functionality and behavior which is already present in JPMS. In what way is this small addition unreasonably complex or risky? -- - DML From david.lloyd at redhat.com Fri Feb 24 15:16:07 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Fri, 24 Feb 2017 09:16:07 -0600 Subject: #LayerPrimitives should be re-opened Message-ID: <3b9b48da-c313-6a82-5f22-dcc91b37cfd8@redhat.com> I am requesting that the #LayerPrimitives issue be re-opened. This issue requested the ability for Layer.Controller to perform the following operations, which Modules can already do for themselves (and, by extension, Layers can do as well using bytecode injection): ? addExports(Module source, String packageName, Module target) - equivalent to Module#addExports(String packageName, Module target), "source" must be in the controller's corresponding layer ? addUses(Module source, Class service) - equivalent to Module#addUses(Class service), "source" must be in the controller's corresponding layer These two methods should be added because requiring bytecode injection to perform these functions is unreasonably onerous, which alone should be enough to justify re-opening the issue. The following self-explanatory operations do not have corresponding methods on Module, and normally can only be done by editing the descriptor or using the Instrumentation#redefineModule(...) API to do so: ? addOpensToAll(Module source, String packageName) ? addExportsToAll(Module source, String packageName) It is not always possible for a container to know what packages need to be exported or opened to all when the module is built, especially in a dynamic deployment environment like ours. Note that Controller#addOpens(Module source, String packageName, Module target) is already an existent method on Controller thus is already deemed "safe". The following operations neither have corresponding methods on Module, nor can be done by editing the descriptor AFAIK: ? addOpensToAllUnnamed(Module source, String packageName) ? addExportsToAllUnnamed(Module source, String packageName) The latter is arguably not necessary since unnamed modules are usually not going to be bound by JPMS linkage, so from what I understand, the exports usually won't apply in this case anyway. It seems to me that the former method might be of use in some cases; having unnamed module code reflecting on module code does not seem very outlandish to me, especially in a container environment. This leaves addPackage(Module source, String packageName). This method is not available on Module and can only otherwise be done by editing the descriptor or using Instrumentation#redefineModule(...). However, if your loader does not yet know the complete set of packages (i.e. it's lazy), or the set of packages might be supplemented after the module is initially created, you either need this method, or you have to redesign your container's linkage strategy from the ground up. We definitely need this method. One OSGi implementer has indicated that they can live without it (by not implementing some optional pieces, and mandating strict behavior where other containers had flexibility previously) but I've also reached out to a few others to see if I can get more data about this. I know that JBoss OSGi does presently rely on this sort of functionality already; I am not sure if it is limited to fragment support or if there is more to it. I have a query out to the maintainer of that project (Arcadiy Ivanov) to get further information, but I understand he is traveling at the moment. However, the potential of the OSGi requirement is not a necessary condition for this method to be justified. We believe that it is also very useful when delivering a Java EE container; we have relied on this functionality for many years. There are not many Java EE certified vendors in the world, so I would consider this alone to be just as substantial a justification for the feature. Logically it is possible to deliver a Java EE container without it, just as it is possible to deliver an OSGi container without it, but this means the end of certain features which a massive number of users have relied upon. This excludes other possible uses, like: dynamic code discovery (via network for example), IDE code hot-deployment features which would require Instrumentation otherwise, etc. The patch itself (still viewable here [1]) is very small, mostly consisting of JavaDoc, and only utilizes existing functions in Jigsaw. It does not introduce any new implementation behavior, and is only available to Layer implementers, therefore I would consider the risk to be very small by any concrete or objective measure. These primitives also have the advantage of enabling an acceptable solution to the following issues (at minimum): ? #NonHierarchicalLayers ? #CyclicDependences (at least, within containers) ? #MutableConfigurations And can be used to provide container-based solutions to several others as well. To me this looks like a very big reward for low risk. My patch also proposed a trivial one-line fix to remove a double null check from the existing Controller#addOpens(Module, String, Module) method. This may have been fixed upstream already, but because a storm has disabled my network connection, I cannot verify this easily at present. If the problem with the issue is the total set of methods, I'm OK with dividing the issue up and arguing the merits of each. I think this issue should be re-opened, and I think that doing so, with a view towards satisfactory resolution, is in the best interests of the community as well as the majority of the members of this expert group. [1] https://gist.github.com/dmlloyd/e3c114f0ac7595a93ecb63b1dbed5e00 -- - DML From mark.reinhold at oracle.com Fri Feb 24 17:10:35 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Fri, 24 Feb 2017 09:10:35 -0800 Subject: Advice + proposals regarding automodule naming In-Reply-To: References: Message-ID: <20170224091035.721333759@eggemoggin.niobe.net> 2017/1/16 1:37:08 -0800, Robert Scholte : > ... > > Proposal 1: Leverage existing coordinates when available. > > Maven is inarguably the most popular build system for Java components, > with Maven Central being the default and largest repository of Java > components in the world. By default, every jar built by Maven > automatically gets a simple properties file inserted into it with its > unique coordinates. Now, not every jar in Central was built with Maven, > however 94% of them were, as we can find the pom.properties file in > 1,806,023 of the 1,913,561 central components . Talk about the default > effect in action! I've been wondering about a discrepancy between this figure and a similar figure that I obtained from a set of Oracle-internal Maven repositories, which by now include a reasonable sampling of what's in Central. My count shows that only about 80% of roughly 350,000 JAR files contain at least one `pom.properties` file. (Some contain more than one, but that's a different problem.) I'm only counting regular JAR files, i.e., ignoring those with non-empty classifiers such as `sources` and `javadoc`. Could the discrepancy between my count and yours be that you're counting all JAR files, regardless of classifier? Or is there some other possibility? Maybe my sample is just skewed ... - Mark From brianf at infinity.nu Fri Feb 24 17:56:20 2017 From: brianf at infinity.nu (Brian Fox) Date: Fri, 24 Feb 2017 12:56:20 -0500 Subject: Advice + proposals regarding automodule naming In-Reply-To: <20170224091035.721333759@eggemoggin.niobe.net> References: <20170224091035.721333759@eggemoggin.niobe.net> Message-ID: Hi Mark, We maintain a separate database of the complete list of all files (recursively unpacked and hashed) that drives our Lifecycle suite of products, so the count of pom.properties came from here. My guess is that since the internal maven isn't likely a full mirror, there's just a sample skew there. On Fri, Feb 24, 2017 at 12:10 PM, wrote: > 2017/1/16 1:37:08 -0800, Robert Scholte : > > ... > > > > Proposal 1: Leverage existing coordinates when available. > > > > Maven is inarguably the most popular build system for Java components, > > with Maven Central being the default and largest repository of Java > > components in the world. By default, every jar built by Maven > > automatically gets a simple properties file inserted into it with its > > unique coordinates. Now, not every jar in Central was built with Maven, > > however 94% of them were, as we can find the pom.properties file in > > 1,806,023 of the 1,913,561 central components . Talk about the default > > effect in action! > > I've been wondering about a discrepancy between this figure and a > similar figure that I obtained from a set of Oracle-internal Maven > repositories, which by now include a reasonable sampling of what's in > Central. > > My count shows that only about 80% of roughly 350,000 JAR files contain > at least one `pom.properties` file. (Some contain more than one, but > that's a different problem.) > > I'm only counting regular JAR files, i.e., ignoring those with non-empty > classifiers such as `sources` and `javadoc`. Could the discrepancy > between my count and yours be that you're counting all JAR files, > regardless of classifier? > > Or is there some other possibility? Maybe my sample is just skewed ... > > - Mark > From mark.reinhold at oracle.com Fri Feb 24 18:27:55 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Fri, 24 Feb 2017 10:27:55 -0800 Subject: Advice + proposals regarding automodule naming In-Reply-To: References: <20170224091035.721333759@eggemoggin.niobe.net> Message-ID: <20170224102755.166316239@eggemoggin.niobe.net> 2017/2/24 9:56:20 -0800, brianf at infinity.nu: > We maintain a separate database of the complete list of all files > (recursively unpacked and hashed) that drives our Lifecycle suite of > products, so the count of pom.properties came from here. My guess is that > since the internal maven isn't likely a full mirror, there's just a sample > skew there. Did you count every JAR file, or did you exclude JAR files with non-empty classifiers (e.g., sources and javadoc)? - Mark From tjwatson at us.ibm.com Fri Feb 24 18:58:17 2017 From: tjwatson at us.ibm.com (Thomas Watson) Date: Fri, 24 Feb 2017 12:58:17 -0600 Subject: #LayerPrimitives should be re-opened In-Reply-To: <3b9b48da-c313-6a82-5f22-dcc91b37cfd8@redhat.com> References: <3b9b48da-c313-6a82-5f22-dcc91b37cfd8@redhat.com> Message-ID: Comments inline. Tom > From: "David M. Lloyd" > I am requesting that the #LayerPrimitives issue be re-opened. > > This issue requested the ability for Layer.Controller to perform the > following operations, which Modules can already do for themselves (and, > by extension, Layers can do as well using bytecode injection): > > ? addExports(Module source, String packageName, Module target) - > equivalent to Module#addExports(String packageName, Module target), > "source" must be in the controller's corresponding layer > ? addUses(Module source, Class service) - equivalent to > Module#addUses(Class service), "source" must be in the controller's > corresponding layer > > These two methods should be added because requiring bytecode injection > to perform these functions is unreasonably onerous, which alone should > be enough to justify re-opening the issue. This seems reasonable to me given that a module can perform these operations themselves it makes sense to allow the controller to do it also. I think the addUses would be more useful if we had a way to know when a module is trying to load a new ServiceLoader class. This way we could dynamically add the required uses just in time before the loader did its check to see if the module can use the service. > > The following self-explanatory operations do not have corresponding > methods on Module, and normally can only be done by editing the > descriptor or using the Instrumentation#redefineModule(...) API to do so: > > ? addOpensToAll(Module source, String packageName) > ? addExportsToAll(Module source, String packageName) > > It is not always possible for a container to know what packages need to > be exported or opened to all when the module is built, especially in a > dynamic deployment environment like ours. Note that > Controller#addOpens(Module source, String packageName, Module target) is > already an existent method on Controller thus is already deemed "safe". > > The following operations neither have corresponding methods on Module, > nor can be done by editing the descriptor AFAIK: > > ? addOpensToAllUnnamed(Module source, String packageName) > ? addExportsToAllUnnamed(Module source, String packageName) > > The latter is arguably not necessary since unnamed modules are usually > not going to be bound by JPMS linkage, so from what I understand, the > exports usually won't apply in this case anyway. This is not my understanding. Unnamed modules can read any other module, but they still can only load classes from packages that are unqualified exports already. But I do question if the addExportsToAll and addOpensToAll include unnamed modules or not. If they do then it seems this extra method to only do it for unnamed modules is not needed. > It seems to me that > the former method might be of use in some cases; having unnamed module > code reflecting on module code does not seem very outlandish to me, > especially in a container environment. Perhaps, but I wonder why the controlling layer did not just make the modules within the layer open modules so it did not have to do this dynamically later. > > This leaves addPackage(Module source, String packageName). This method > is not available on Module and can only otherwise be done by editing the > descriptor or using Instrumentation#redefineModule(...). However, if > your loader does not yet know the complete set of packages (i.e. it's > lazy), or the set of packages might be supplemented after the module is > initially created, you either need this method, or you have to redesign > your container's linkage strategy from the ground up. We definitely > need this method. One OSGi implementer has indicated that they can live > without it (by not implementing some optional pieces, and mandating > strict behavior where other containers had flexibility previously) but > I've also reached out to a few others to see if I can get more data > about this. I know that JBoss OSGi does presently rely on this sort of > functionality already; I am not sure if it is limited to fragment > support or if there is more to it. I have a query out to the maintainer > of that project (Arcadiy Ivanov) to get further information, but I > understand he is traveling at the moment. This method would help for the dynamic fragment attachment case in OSGi. Also required for that would be the ability to addExportToAll. Having addPackage without addExportToAll would be an incomplete solution for the dynamic fragment attachment case in OSGi.. > > However, the potential of the OSGi requirement is not a necessary > condition for this method to be justified. We believe that it is also > very useful when delivering a Java EE container; we have relied on this > functionality for many years. There are not many Java EE certified > vendors in the world, so I would consider this alone to be just as > substantial a justification for the feature. Logically it is possible > to deliver a Java EE container without it, just as it is possible to > deliver an OSGi container without it, but this means the end of certain > features which a massive number of users have relied upon. > > This excludes other possible uses, like: dynamic code discovery (via > network for example), IDE code hot-deployment features which would > require Instrumentation otherwise, etc. > > The patch itself (still viewable here [1]) is very small, mostly > consisting of JavaDoc, and only utilizes existing functions in Jigsaw. > It does not introduce any new implementation behavior, and is only > available to Layer implementers, therefore I would consider the risk to > be very small by any concrete or objective measure. > > These primitives also have the advantage of enabling an acceptable > solution to the following issues (at minimum): > > ? #NonHierarchicalLayers > ? #CyclicDependences (at least, within containers) > ? #MutableConfigurations > > And can be used to provide container-based solutions to several others > as well. To me this looks like a very big reward for low risk. > > My patch also proposed a trivial one-line fix to remove a double null > check from the existing Controller#addOpens(Module, String, Module) > method. This may have been fixed upstream already, but because a storm > has disabled my network connection, I cannot verify this easily at present. > > If the problem with the issue is the total set of methods, I'm OK with > dividing the issue up and arguing the merits of each. > > I think this issue should be re-opened, and I think that doing so, with > a view towards satisfactory resolution, is in the best interests of the > community as well as the majority of the members of this expert group. > > [1] https://gist.github.com/dmlloyd/e3c114f0ac7595a93ecb63b1dbed5e00 > > -- > - DML > From brianf at infinity.nu Fri Feb 24 19:13:29 2017 From: brianf at infinity.nu (Brian Fox) Date: Fri, 24 Feb 2017 14:13:29 -0500 Subject: Advice + proposals regarding automodule naming In-Reply-To: <20170224102755.166316239@eggemoggin.niobe.net> References: <20170224091035.721333759@eggemoggin.niobe.net> <20170224102755.166316239@eggemoggin.niobe.net> Message-ID: Ah good question. I looked back at what my guys sent me: "The counts we provided are for total GA and unique artifactId" which means the classified artifacts (sources and javadoc) were not counted. I'm not sure the Sources and Javadoc plugins insert the properties into there. For projects that do produce executable jars that are classified, the maven-jar-plugin would have been used so those would still have the pom.properities in it. In other words, its a function of which plugin created the jar (jar, sources, javadoc) and not if it has a classifier or not. On Fri, Feb 24, 2017 at 1:27 PM, wrote: > 2017/2/24 9:56:20 -0800, brianf at infinity.nu: > > We maintain a separate database of the complete list of all files > > (recursively unpacked and hashed) that drives our Lifecycle suite of > > products, so the count of pom.properties came from here. My guess is that > > since the internal maven isn't likely a full mirror, there's just a > sample > > skew there. > > Did you count every JAR file, or did you exclude JAR files with > non-empty classifiers (e.g., sources and javadoc)? > > - Mark > From mark.reinhold at oracle.com Fri Feb 24 20:55:09 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Fri, 24 Feb 2017 12:55:09 -0800 Subject: Advice + proposals regarding automodule naming In-Reply-To: References: <20170224091035.721333759@eggemoggin.niobe.net> <20170224102755.166316239@eggemoggin.niobe.net> Message-ID: <20170224125509.534200457@eggemoggin.niobe.net> 2017/2/24 11:13:29 -0800, brianf at infinity.nu: > Ah good question. I looked back at what my guys sent me: "The counts we > provided are for total GA and unique artifactId" which means the classified > artifacts (sources and javadoc) were not counted. I'm not sure the Sources > and Javadoc plugins insert the properties into there. For projects that do > produce executable jars that are classified, the maven-jar-plugin would > have been used so those would still have the pom.properities in it. In > other words, its a function of which plugin created the jar (jar, sources, > javadoc) and not if it has a classifier or not. Okay -- thanks for the clarification. - Mark From mark.reinhold at oracle.com Fri Feb 24 22:43:48 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Fri, 24 Feb 2017 14:43:48 -0800 Subject: Advice + proposals regarding automodule naming In-Reply-To: References: Message-ID: <20170224144348.724345607@eggemoggin.niobe.net> For the record, I've created the new issue http://openjdk.java.net/projects/jigsaw/spec/issues/#AutomaticModuleNames for this topic and added links there to the various discussion threads. - Mark From rfscholte at apache.org Sat Feb 25 14:11:47 2017 From: rfscholte at apache.org (Robert Scholte) Date: Sat, 25 Feb 2017 15:11:47 +0100 Subject: Advice + proposals regarding automodule naming In-Reply-To: <20170224144348.724345607@eggemoggin.niobe.net> References: <20170224144348.724345607@eggemoggin.niobe.net> Message-ID: thanks for adding this as separate issue, really appreciate it. Once there's a resolution, the issues related to module names in general should probably be revised as well. Robert On Fri, 24 Feb 2017 23:43:48 +0100, wrote: > For the record, I've created the new issue > > http://openjdk.java.net/projects/jigsaw/spec/issues/#AutomaticModuleNames > > for this topic and added links there to the various discussion threads. > > - Mark From mark.reinhold at oracle.com Mon Feb 27 10:47:04 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 27 Feb 2017 02:47:04 -0800 (PST) Subject: #LayerPrimitives (was Re: Proposal: #NonHierarchicalLayers) In-Reply-To: References: <20161207234117.1E5F1213F1@eggemoggin.niobe.net> <1de06198-930b-4b71-fe5d-1477d9cace80@redhat.com> <20161212232335.A9BFC24B27@eggemoggin.niobe.net> <9b66aa52-877a-e4b2-1458-9ff47dfb690b@redhat.com> <20170126223431.B417255DD9@eggemoggin.niobe.net> <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> <20170215195108.C881964E6F@eggemoggin.niobe.net> <713a39e0-4b76-0a28-2f91-cd7cb1655fcb@redhat.com> <20170221160442.3348866481@eggemoggin.niobe.net> Message-ID: <20170227104704.28F7D730BD@eggemoggin.niobe.net> 2017/2/21 9:05:22 -0800, david.lloyd at redhat.com: > Logically a non-goal is not an anti-goal, as I have observed many times > in the past as well. Not setting out to do something is not the same > thing as forbidding it from being done Of course not. It was a non-goal to support bidirectional interoperation with OSGi. It turned out that doing so required just a couple of small changes, however, so we went ahead and did it. > , especially if it is a minimal > change. The changes you have requested may appear minimal on the surface today. The impact of those changes over the long term is, however, likely far from minimal, as I have explained. Taking both the immediate and long-term consequences of every change into account is fundamental to responsible platform stewardship. > We consider these issue inadequately addressed despite being rejected. > By JCP procedures, this obligates us to vote "no". There is no rule in the JCP that requires you to vote "no" on a JSR if the JSR fails to achieve a non-goal. You can choose to vote "no" anyway, of course, if you decide that it is more important to protect your own narrow interests than it is to support the broader interests of the entire Java ecosystem. - Mark From neil.bartlett at paremus.com Mon Feb 27 13:08:13 2017 From: neil.bartlett at paremus.com (Neil Bartlett (Paremus)) Date: Mon, 27 Feb 2017 13:08:13 +0000 Subject: #LayerPrimitives (was Re: Proposal: #NonHierarchicalLayers) In-Reply-To: <20170227104704.28F7D730BD@eggemoggin.niobe.net> References: <20161207234117.1E5F1213F1@eggemoggin.niobe.net> <1de06198-930b-4b71-fe5d-1477d9cace80@redhat.com> <20161212232335.A9BFC24B27@eggemoggin.niobe.net> <9b66aa52-877a-e4b2-1458-9ff47dfb690b@redhat.com> <20170126223431.B417255DD9@eggemoggin.niobe.net> <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> <20170215195108.C881964E6F@eggemoggin.niobe.net> <713a39e0-4b76-0a28-2f91-cd7cb1655fcb@redhat.com> <20170221160442.3348866481@eggemoggin.niobe.net> <20170227104704.28F7D730BD@eggemoggin.niobe.net> Message-ID: > On 27 Feb 2017, at 10:47, mark.reinhold at oracle.com wrote: > > 2017/2/21 9:05:22 -0800, david.lloyd at redhat.com: >> Logically a non-goal is not an anti-goal, as I have observed many times >> in the past as well. Not setting out to do something is not the same >> thing as forbidding it from being done > > Of course not. It was a non-goal to support bidirectional interoperation > with OSGi. It turned out that doing so required just a couple of small > changes, however, so we went ahead and did it. > >> , especially if it is a minimal >> change. > > The changes you have requested may appear minimal on the surface today. > The impact of those changes over the long term is, however, likely far > from minimal, as I have explained. Taking both the immediate and > long-term consequences of every change into account is fundamental to > responsible platform stewardship. > >> We consider these issue inadequately addressed despite being rejected. >> By JCP procedures, this obligates us to vote "no". > > There is no rule in the JCP that requires you to vote "no" on a JSR if > the JSR fails to achieve a non-goal. > > You can choose to vote "no" anyway, of course, if you decide that it is > more important to protect your own narrow interests than it is to support > the broader interests of the entire Java ecosystem. Mark, I have to call this out as inappropriate. Everything I have seen from David?s and Red Hat?s behaviour suggests that they are working for the broader interests of the Java ecosystem, and that you each disagree merely on how those interests are best served. Accusations of acting from bad motives will cause this forum to become less than civil. From my perspective I am not clear why the changes under discussion are substantially more risky than those you have already made for OSGi interop. The stated justifications for rejecting #LayerPrimitives make little sense, and I?m concerned that this issue could block (or hinder to the point of impracticability) a fully backwards-compatible OSGi implementation. Regards, Neil > > - Mark From david.lloyd at redhat.com Mon Feb 27 13:43:25 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 27 Feb 2017 07:43:25 -0600 Subject: #CyclicDependences should be re-opened Message-ID: I propose that the #CyclicDependences issue should be re-opened. This issue is primarily about modules which exist on the module path in a regular user application. We have observed that it is quite possible for applications (even simpler ones) to require circularity in common cases. It has been posited that ideologically, circularity is an anti-pattern. However I find this position to be meritless, and experience shows that in practice, this position causes more problems than it solves. The notion that all modules in a cycle are logically one is also flawed. It makes the assumption that all the modules were developed by the same person, at the same time, which is rarely if ever the case. Modern, real world applications, even relatively small ones, now consist of dozens or hundreds of distinct pieces from dozens of sources. Maven is a big part of this: by allowing application dependencies to be managed automatically, the friction of doing so has been greatly reduced, and I've correspondingly observed that including substantial dependency graphs in an application as a common practice has increased apace. Maven forbids circularity during compilation. This circularity prevention extends to cycles in the artifact graph being built, and building does not necessarily require the full graph to be present. There is nothing today preventing cycles at run time, and many, many users are taking advantage of this, either knowingly or unknowingly every time they use their favorite containers and frameworks. Anecdotally, it took us fewer than four months from our initial JBoss Modules prototype in 2010 to discover that without circularity support, we were going to have a very hard time assembling even a basic application module set let alone our entire application server. The primary JPMS remedy for circularity is services. In some cases this can work, but there are limitations that prevent this from being sufficient as a general solution, particularly for existing cases. The problems come in multiple forms. We encountered the following two before we abandoned the idea of restricting circularity as a mistake: 1. A module is compiled against an API (often a spec API); at run time, a different API module is in use which consumes common code which in turn comsumes the module being compiled. From the user perspective, they are not even aware that they have introduced a cycle, and will be surprised and often baffled by the problem at run time. This is the most common case, and services cannot provide a complete solution, particularly with existing code. Even if they could, avoiding cycles in this sort of situation can require extreme care and planning. 2. An API module is designed to be linked statically against a flexible implementation, by providing an ABI which must be satisfied. The implementation must consume the API in turn; without circularity, the two must be joined as one module. Even if somehow they did not, it's typical for implementations to consume libraries that in turn require the original API module. This is the case with slf4j for example, which is generally regarded to be a better solution than its commons-logging predecessor which did things in an arguably more JPMS-friendly way, yet with critical disadvantages. Neither of these cases constitutes a poor development practice, and in fact they often afford quite a lot of flexibility of the type which Java has previously excelled at. It is possible for containers to enable circularity in a limited context by bypassing the JPMS resolver. This is not a good solution, it is merely the minimum solution. It leaves non-container users in the cold and will lead to stressful late nights for developers who are trying to assemble applications but are foiled by a restriction that has no benefit to them and that they may not even understand. And, in-container modules which can manage circularity will, when examining their own module's descriptor, see incorrect information as at least one module in the graph necessarily must not explicitly reference its dependency (as a practical matter, it is difficult to come up with a system wherein only some modules are not expressed so the simple solution is to express none instead). A value-added feature is one thing, but requiring a container for basic common-sense functionality (incomplete as it would be in this case) is too much pain to inflict on users for a flawed ideology used to justify an implementation choice. I think that cutting off these use cases as an anti-pattern is poorly justified and is not in the best interests of the community or the expert group. -- - DML From david.lloyd at redhat.com Mon Feb 27 13:43:30 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 27 Feb 2017 07:43:30 -0600 Subject: #LayerPrimitives should be re-opened In-Reply-To: <3b9b48da-c313-6a82-5f22-dcc91b37cfd8@redhat.com> References: <3b9b48da-c313-6a82-5f22-dcc91b37cfd8@redhat.com> Message-ID: I want to add an addendum to this. We don't consider this to be the *best* solution, rather the very *minimum* to allow workarounds which make up for a subset of what we still consider to be critical deficiencies in the specification in the form of past rejected and dismissed issues, which I will expound upon subsequently. On 02/24/2017 09:16 AM, David M. Lloyd wrote: > I am requesting that the #LayerPrimitives issue be re-opened. > > This issue requested the ability for Layer.Controller to perform the > following operations, which Modules can already do for themselves (and, > by extension, Layers can do as well using bytecode injection): > > ? addExports(Module source, String packageName, Module target) - > equivalent to Module#addExports(String packageName, Module target), > "source" must be in the controller's corresponding layer > ? addUses(Module source, Class service) - equivalent to > Module#addUses(Class service), "source" must be in the controller's > corresponding layer > > These two methods should be added because requiring bytecode injection > to perform these functions is unreasonably onerous, which alone should > be enough to justify re-opening the issue. > > The following self-explanatory operations do not have corresponding > methods on Module, and normally can only be done by editing the > descriptor or using the Instrumentation#redefineModule(...) API to do so: > > ? addOpensToAll(Module source, String packageName) > ? addExportsToAll(Module source, String packageName) > > It is not always possible for a container to know what packages need to > be exported or opened to all when the module is built, especially in a > dynamic deployment environment like ours. Note that > Controller#addOpens(Module source, String packageName, Module target) is > already an existent method on Controller thus is already deemed "safe". > > The following operations neither have corresponding methods on Module, > nor can be done by editing the descriptor AFAIK: > > ? addOpensToAllUnnamed(Module source, String packageName) > ? addExportsToAllUnnamed(Module source, String packageName) > > The latter is arguably not necessary since unnamed modules are usually > not going to be bound by JPMS linkage, so from what I understand, the > exports usually won't apply in this case anyway. It seems to me that > the former method might be of use in some cases; having unnamed module > code reflecting on module code does not seem very outlandish to me, > especially in a container environment. > > This leaves addPackage(Module source, String packageName). This method > is not available on Module and can only otherwise be done by editing the > descriptor or using Instrumentation#redefineModule(...). However, if > your loader does not yet know the complete set of packages (i.e. it's > lazy), or the set of packages might be supplemented after the module is > initially created, you either need this method, or you have to redesign > your container's linkage strategy from the ground up. We definitely > need this method. One OSGi implementer has indicated that they can live > without it (by not implementing some optional pieces, and mandating > strict behavior where other containers had flexibility previously) but > I've also reached out to a few others to see if I can get more data > about this. I know that JBoss OSGi does presently rely on this sort of > functionality already; I am not sure if it is limited to fragment > support or if there is more to it. I have a query out to the maintainer > of that project (Arcadiy Ivanov) to get further information, but I > understand he is traveling at the moment. > > However, the potential of the OSGi requirement is not a necessary > condition for this method to be justified. We believe that it is also > very useful when delivering a Java EE container; we have relied on this > functionality for many years. There are not many Java EE certified > vendors in the world, so I would consider this alone to be just as > substantial a justification for the feature. Logically it is possible > to deliver a Java EE container without it, just as it is possible to > deliver an OSGi container without it, but this means the end of certain > features which a massive number of users have relied upon. > > This excludes other possible uses, like: dynamic code discovery (via > network for example), IDE code hot-deployment features which would > require Instrumentation otherwise, etc. > > The patch itself (still viewable here [1]) is very small, mostly > consisting of JavaDoc, and only utilizes existing functions in Jigsaw. > It does not introduce any new implementation behavior, and is only > available to Layer implementers, therefore I would consider the risk to > be very small by any concrete or objective measure. > > These primitives also have the advantage of enabling an acceptable > solution to the following issues (at minimum): > > ? #NonHierarchicalLayers > ? #CyclicDependences (at least, within containers) > ? #MutableConfigurations > > And can be used to provide container-based solutions to several others > as well. To me this looks like a very big reward for low risk. > > My patch also proposed a trivial one-line fix to remove a double null > check from the existing Controller#addOpens(Module, String, Module) > method. This may have been fixed upstream already, but because a storm > has disabled my network connection, I cannot verify this easily at present. > > If the problem with the issue is the total set of methods, I'm OK with > dividing the issue up and arguing the merits of each. > > I think this issue should be re-opened, and I think that doing so, with > a view towards satisfactory resolution, is in the best interests of the > community as well as the majority of the members of this expert group. > > [1] https://gist.github.com/dmlloyd/e3c114f0ac7595a93ecb63b1dbed5e00 > -- - DML From david.lloyd at redhat.com Mon Feb 27 13:44:30 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 27 Feb 2017 07:44:30 -0600 Subject: Preemptively rejected issue: Modules, ClassLoaders, and compatibility Message-ID: <1f13eb07-0465-19e1-8b23-ea2f3eb6f458@redhat.com> Since the early days of this JSR, in fact since 2012 (if not earlier), long before it was officially formed (which wasn't until the end of 2014), the design decision in Jigsaw of creating a new concept for encapsulation has been the subject of friction and something we've been fighting with both technically and politically. We still hold that this decision was in error, and that it is becoming more apparent as time wears on and more problems arise with the fundamental architecture of Jigsaw. The justification of this design decision was chiefly made around compatibility: having the same types and arrangements of class loaders was something that was assumed that many applications would count on. Splitting the JDK means that some modules that used to have a "null" class loader now no longer would. JDK modules which had elevated privileges would lose them, resulting in breakage and other problems. Over time, more new rules were introduced regarding reflection, service loader, and other behaviors. These rules have broken compatibility in many ways which are, by any evidence, far more widespread than those which would result from changing the class loader arrangement of the core platform or even class path applications. Since that time, a laudable initiative has been undertaken to de-privilege the core platform, module by module, in the interest of security. This change inherently breaks the previously established compatibility criteria in a much more direct manner; each of these modules has a class loader value that it did not have before. The logical conclusion of this effort is in fact that every module would be de-privileged up to, but not including, java.base. Any program depending on null class loader values would be broken; any JDK code relying on this as a security check would need to be updated. I believe that it was also suggested that forward compatibility for Java EE containers which rely on a special class loader structure was a justification for this as well. However, other issues we have raised in the name of meeting what we view as the bare minimum for our Java EE container's compatibility have been closed, which would invalidate this justification as far as I can tell. Several existing issues, resolved and unresolved, arise solely as a consequence of this design choice, including #MultipleModuleVersions and #ConcealedPackageConflicts, both of which are still open. Neither of these issues would necessarily exist had this choice been made; on the contrary, the question would have been about supporting such restrictions as a matter of policy, rather than as a consequence of the implementation. I don't think I am overstepping when I suggest that in such a hypothetical case, most developers would express that such restrictions should not be added, or that they should only apply on an opt-in basis, intuitively seeing the necessity of such things. In addition, there would have been no need to have disparate behavior between named and unnamed modules, as this division has clearly arisen as an implementation artifact, not a first principle of design (who would imagine such a thing themselves?). Each of these things represents a cognitive friction between existing and future code. Millions upon millions of lines of code of Java exist. In fact it's probably not even countable anymore at this point. By introducing a separated abstraction for the module concept when a largely equivalent concept already exists in the form of class loading, every single one of these things that relies on traditional class loading behavior is needlessly compromised in ways that seem hard to predict and harder to justify. A layer of complexity is added wherein there are now yet more different ways in which classes and resources can be found and resolved, with more rules layered on top, when the old rules were already sufficient as they were. The Java community deserves a simple, clean design that grows intuitively from what they are familiar with. Allowing "old" code and "new" code to coexist is not a good substitute when we could have easily had a new system wherein the vast majority of "old" code would have been able to function without change, even were it using service loading, class loading, and reflection. Our opposition to this design choice persists, despite the preemptive dismissal of our original and subsequent objections to it. At least one issue that was just recently closed, #LayerPrimitives, existed as part of a mitigation strategy to cope with this problem. -- - DML From david.lloyd at redhat.com Mon Feb 27 17:32:47 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 27 Feb 2017 11:32:47 -0600 Subject: #LayerPrimitives should be re-opened In-Reply-To: References: <3b9b48da-c313-6a82-5f22-dcc91b37cfd8@redhat.com> Message-ID: <0342a07b-41e5-a246-6d3d-21317a81c107@redhat.com> On 02/24/2017 12:58 PM, Thomas Watson wrote: > Comments inline. > > Tom > >> From: "David M. Lloyd" >> I am requesting that the #LayerPrimitives issue be re-opened. >> >> This issue requested the ability for Layer.Controller to perform the >> following operations, which Modules can already do for themselves (and, >> by extension, Layers can do as well using bytecode injection): >> >> ? addExports(Module source, String packageName, Module target) - >> equivalent to Module#addExports(String packageName, Module target), >> "source" must be in the controller's corresponding layer >> ? addUses(Module source, Class service) - equivalent to >> Module#addUses(Class service), "source" must be in the controller's >> corresponding layer >> >> These two methods should be added because requiring bytecode injection >> to perform these functions is unreasonably onerous, which alone should >> be enough to justify re-opening the issue. > > This seems reasonable to me given that a module can perform these > operations themselves it makes sense to allow the controller to do it also. > > I think the addUses would be more useful if we had a way to know when a > module is trying to load a new ServiceLoader class. This way we could > dynamically add the required uses just in time before the loader did its > check to see if the module can use the service. I think that service loading in general might be an unresolved issue. Earlier I recall we discussed letting the Layer have some jurisdiction over what services were available and how to find them, which relates closely to this, but I don't see anything non-internal that could enable either this feature or being able to detect when a service is being loaded. And, discovering undeclared service consumers is non-trivial. >> The following self-explanatory operations do not have corresponding >> methods on Module, and normally can only be done by editing the >> descriptor or using the Instrumentation#redefineModule(...) API to do so: >> >> ? addOpensToAll(Module source, String packageName) >> ? addExportsToAll(Module source, String packageName) >> >> It is not always possible for a container to know what packages need to >> be exported or opened to all when the module is built, especially in a >> dynamic deployment environment like ours. Note that >> Controller#addOpens(Module source, String packageName, Module target) is >> already an existent method on Controller thus is already deemed "safe". >> >> The following operations neither have corresponding methods on Module, >> nor can be done by editing the descriptor AFAIK: >> >> ? addOpensToAllUnnamed(Module source, String packageName) >> ? addExportsToAllUnnamed(Module source, String packageName) >> >> The latter is arguably not necessary since unnamed modules are usually >> not going to be bound by JPMS linkage, so from what I understand, the >> exports usually won't apply in this case anyway. > > This is not my understanding. Unnamed modules can read any other > module, but they still can only load classes from packages that are > unqualified exports already. But I do question if the addExportsToAll > and addOpensToAll include unnamed modules or not. If they do then it > seems this extra method to only do it for unnamed modules is not needed. AFAIK they do include unnamed modules. The hypothetical case for unnamed modules would relate to dealing with mixed systems with pre-existing class loaders. It might be possible in some cases to single out the unnamed module for class loaders in such cases and use addExports directly in this case. >> It seems to me that >> the former method might be of use in some cases; having unnamed module >> code reflecting on module code does not seem very outlandish to me, >> especially in a container environment. > > Perhaps, but I wonder why the controlling layer did not just make the > modules within the layer open modules so it did not have to do this > dynamically later. This is the case where the layer does not know that the module needed to be open until some of the module was already loaded. >> This leaves addPackage(Module source, String packageName). This method >> is not available on Module and can only otherwise be done by editing the >> descriptor or using Instrumentation#redefineModule(...). However, if >> your loader does not yet know the complete set of packages (i.e. it's >> lazy), or the set of packages might be supplemented after the module is >> initially created, you either need this method, or you have to redesign >> your container's linkage strategy from the ground up. We definitely >> need this method. One OSGi implementer has indicated that they can live >> without it (by not implementing some optional pieces, and mandating >> strict behavior where other containers had flexibility previously) but >> I've also reached out to a few others to see if I can get more data >> about this. I know that JBoss OSGi does presently rely on this sort of >> functionality already; I am not sure if it is limited to fragment >> support or if there is more to it. I have a query out to the maintainer >> of that project (Arcadiy Ivanov) to get further information, but I >> understand he is traveling at the moment. > > This method would help for the dynamic fragment attachment case in OSGi. > Also required for that would be the ability to addExportToAll. Having > addPackage without addExportToAll would be an incomplete solution for > the dynamic fragment attachment case in OSGi.. -- - DML From tjwatson at us.ibm.com Mon Feb 27 17:47:25 2017 From: tjwatson at us.ibm.com (Thomas Watson) Date: Mon, 27 Feb 2017 11:47:25 -0600 Subject: Proposal: #NonHierarchicalLayers In-Reply-To: <20170215195108.C881964E6F@eggemoggin.niobe.net> References: <20161207234117.1E5F1213F1@eggemoggin.niobe.net> <1de06198-930b-4b71-fe5d-1477d9cace80@redhat.com> <20161212232335.A9BFC24B27@eggemoggin.niobe.net> <9b66aa52-877a-e4b2-1458-9ff47dfb690b@redhat.com> <20170126223431.B417255DD9@eggemoggin.niobe.net> <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> <20170215195108.C881964E6F@eggemoggin.niobe.net> Message-ID: > From: mark.reinhold at oracle.com > > OSGi does support dynamic attachments of fragments. The current > > prototype cannot do this but on Jan. 4 Thomas expressed that being able > > to add packages would enable this part of the specification. Re-linking > > everything is (according to this email) an alternative that comes at the > > cost of not supporting this feature. > > True, but Thomas later recommended against complicating JPMS in order to > support this feature of OSGi [1]. He can live with having to re-resolve > bundles when fragments add API packages, which is not common and arguably > an anti-pattern. > The addPackage method would be beneficial for other reasons when adhering an existing module system into a JPMS Layer. - addPackage would allow existing module systems to avoid aggressive discovery of all private packages for its own modules when building a JPMS ModuleDescriptor. For the existing module systems I know about the class loader is used extensively as the module primitive. Requirements wire up the class loaders for proper class loader delegation and typically the APIs are declared capabilities to indicate what the modular class loaders can load and expose to another module class loader. But nowhere was it required to aggressively discover all private packages and resources up front. Having an addPackage would allow such module systems to grow the list of private packages lazily as it is defining classes in the packages that are private to the module. - In my prototype I also have a boot strapping issue that addPackage would be helpful for. In my prototype I have a launcher that assembles a module Layer which then holds the OSGi framework implementation. In this case the launcher is using Java 9 Layer API and then is loading up a standard OSGi framework implementation using only OSGi standard API. The launcher then assigns the framework implementation a ModuleDescriptor to represent the OSGi framework. In OSGi the framework is represented by an OSGi bundle called the "system.bundle". So in this case the ModuleDescriptor also has a name "system.bundle". And the ModuleDescriptor has a well known list of OSGi APIs it exports. But the launcher does not know the private implementation packages of the framework implementation. In my prototype I just hack in the list of private package for the equinox framework, but this is far from ideal. I did say that I could live without supporting dynamic attachment of fragments which provided additional packages. But I do not want that compromise from me to justify not adding the method addPackage to the controller. I think it would be beneficial to existing module systems that need to live within a proper JPMS Layer. It would help to understand what complications addPackage bring to JPMS above the other dynamic add methods already on the Module API and the others being proposed for the Controller API here. Tom From david.lloyd at redhat.com Tue Feb 28 05:10:27 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 27 Feb 2017 23:10:27 -0600 Subject: #LayerPrimitives (was Re: Proposal: #NonHierarchicalLayers) In-Reply-To: <20170227104704.28F7D730BD@eggemoggin.niobe.net> References: <20161207234117.1E5F1213F1@eggemoggin.niobe.net> <1de06198-930b-4b71-fe5d-1477d9cace80@redhat.com> <20161212232335.A9BFC24B27@eggemoggin.niobe.net> <9b66aa52-877a-e4b2-1458-9ff47dfb690b@redhat.com> <20170126223431.B417255DD9@eggemoggin.niobe.net> <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> <20170215195108.C881964E6F@eggemoggin.niobe.net> <713a39e0-4b76-0a28-2f91-cd7cb1655fcb@redhat.com> <20170221160442.3348866481@eggemoggin.niobe.net> <20170227104704.28F7D730BD@eggemoggin.niobe.net> Message-ID: <58B50643.4050004@redhat.com> On 2/27/17 4:47 AM, mark.reinhold at oracle.com wrote: > The changes you have requested may appear minimal on the surface today. > The impact of those changes over the long term is, however, likely far > from minimal, as I have explained. Taking both the immediate and > long-term consequences of every change into account is fundamental to > responsible platform stewardship. You did state that this was the case, but there was never an explanation as to what the actual long term impact would be and why that impact is unacceptable. Could you please elaborate on that? >> We consider these issue inadequately addressed despite being rejected. >> By JCP procedures, this obligates us to vote "no". > > There is no rule in the JCP that requires you to vote "no" on a JSR if > the JSR fails to achieve a non-goal. This is what I'm referring to: "EC members should vote 'no' if they believe that the Spec Lead or Maintenance Lead has not adequately addressed all Issues including those that have been rejected or otherwise closed by the Expert Group." "Non-goal" is your phrasing, but objectively speaking we're talking about an issue (or rather, multiple issues) that you have rejected but we consider inadequately resolved, which is why I think this language (and the associated spirit) is applicable. Obviously I'd prefer that we work to resolve the issues. My intention was to point out that with all further paths for discussion or compromise closed off, we are stuck without options save one, and that's not really a position we want to be in. On a personal note, I want to reiterate that I am very committed to serving the best interests of the Java ecosystem. In fact I think that each expert in the group brings a different valuable and valid perspective. I may use our own projects as an example an laboratory subject for experimentation, but that does not mean that I am seeking only to serve our own interests; merely that I find it a convenient way to explore the boundaries of the specification and find problem areas. For me it's not too hard to extrapolate these issues to other libraries and existing use cases, nor to see the potential value in the corresponding capabilities for new use cases. -- - DML