From david.lloyd at redhat.com Wed Mar 1 22:37:00 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 1 Mar 2017 16:37:00 -0600 Subject: Fwd: Re: Invoking default methods from a Proxy's InvocationHandler in JDK9 In-Reply-To: References: Message-ID: <56948df6-1d91-62de-d67b-975f979cc78b@redhat.com> This keeps coming up. I think we'd better raise it as an issue because the workaround used in Java 8 is specifically broken by JPMS. In that regard, Peter Levart's workaround might be considered to fall under the FC extension umbrella. -------- Forwarded Message -------- Subject: Re: Invoking default methods from a Proxy's InvocationHandler in JDK9 Date: Tue, 3 Jan 2017 11:14:31 -0800 From: Mandy Chung To: Peter Levart CC: jigsaw-dev at openjdk.java.net > On Jan 2, 2017, at 11:20 PM, Peter Levart wrote: > > Hi Matthew, > > On 01/03/2017 04:28 AM, Matthew Hall wrote: >> I'm a member of the JDBI [1] project, an open source SQL access library >> atop JDBC. >> >> A major part of our API provides implementations of declarative interfaces >> defined by users (similar to MyBatis). Interface methods may be default (in >> which case the default method implementation is used) or abstract (in which >> case JDBI provides the implementation). >> >> We're using JDK 8, and we use java.lang.reflect.Proxy and InvocationHandler >> to provide declarative interface implementations for our users. >> >> Prior to release of JDK 8, it appears that the subject of enhancing Proxies >> for default methods was discussed [2], but ultimately did not make it into >> the release. >> >> The root of the problem is that the generated Proxy class overrides all >> methods in the proxy interfaces. This means that the interface default >> methods are now superclass methods to the proxy class, which makes them >> un-invokable from outside code--including the InvocationHandler. >> >> As far as we can tell, the _only_ way to invoke a default method--on a >> proxy, from an InvocationHandler--is to resort to some horrible >> setAccessible shenanigans [3]. >> >> Specifically, we have to: >> 1. Call Constructor.setAccessible(true) to make the private constructor >> MethodHandles.Lookup(Class lookupClass, int allowedModes) accessible, >> 2. Invoke that private constructor to instantiate a MethodHandles.Lookup >> with private (!) access to all the things, >> 3. Call Lookup.unreflectSpecial to get the MethodHandle for the default >> method, and >> 4. Invoke the method through the MethodHandle. >> >> This is ugly, but it works--for now. If JDK 9 takes away access to >> setAccessible, it may cease to work. >> >> I found some discussion about this last summer [4], but it's hard to tell >> if anything came out of the discussion. >> >> Is there anything on the roadmap for JDK9 to allow a proxy's >> InvocationHandler to invoke default methods on proxies? Are there any >> existing proposals I should be aware of? > > I have created a prototype last year: > > http://mail.openjdk.java.net/pipermail/core-libs-dev/2016-June/041629.html > > But I think the JDK 9 train has already left the station. So perhaps in JDK 10... https://bugs.openjdk.java.net/browse/JDK-8159746 is the JBS issue for this. One other possibility is to fix proxy generated class not to override default methods but there would require more investigation to tease out before we can be certain if this can make JDK 9. Mandy -- - DML From david.lloyd at redhat.com Thu Mar 2 01:56:05 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 1 Mar 2017 19:56:05 -0600 Subject: Fwd: Re: Invoking default methods from a Proxy's InvocationHandler in JDK9 In-Reply-To: <56948df6-1d91-62de-d67b-975f979cc78b@redhat.com> References: <56948df6-1d91-62de-d67b-975f979cc78b@redhat.com> Message-ID: <4ff152eb-5302-33a9-724a-bad12bda1bba@redhat.com> According to the JDBI people, it seems that the security checks on unreflectSpecial were relaxed for default methods. If this is the case then I think we're OK here. On 03/01/2017 04:37 PM, David M. Lloyd wrote: > This keeps coming up. I think we'd better raise it as an issue because > the workaround used in Java 8 is specifically broken by JPMS. In that > regard, Peter Levart's workaround might be considered to fall under the > FC extension umbrella. > > -------- Forwarded Message -------- > Subject: Re: Invoking default methods from a Proxy's InvocationHandler > in JDK9 > Date: Tue, 3 Jan 2017 11:14:31 -0800 > From: Mandy Chung > To: Peter Levart > CC: jigsaw-dev at openjdk.java.net > > >> On Jan 2, 2017, at 11:20 PM, Peter Levart wrote: >> >> Hi Matthew, >> >> On 01/03/2017 04:28 AM, Matthew Hall wrote: >>> I'm a member of the JDBI [1] project, an open source SQL access library >>> atop JDBC. >>> >>> A major part of our API provides implementations of declarative >>> interfaces >>> defined by users (similar to MyBatis). Interface methods may be >>> default (in >>> which case the default method implementation is used) or abstract (in >>> which >>> case JDBI provides the implementation). >>> >>> We're using JDK 8, and we use java.lang.reflect.Proxy and >>> InvocationHandler >>> to provide declarative interface implementations for our users. >>> >>> Prior to release of JDK 8, it appears that the subject of enhancing >>> Proxies >>> for default methods was discussed [2], but ultimately did not make it >>> into >>> the release. >>> >>> The root of the problem is that the generated Proxy class overrides all >>> methods in the proxy interfaces. This means that the interface default >>> methods are now superclass methods to the proxy class, which makes them >>> un-invokable from outside code--including the InvocationHandler. >>> >>> As far as we can tell, the _only_ way to invoke a default method--on a >>> proxy, from an InvocationHandler--is to resort to some horrible >>> setAccessible shenanigans [3]. >>> >>> Specifically, we have to: >>> 1. Call Constructor.setAccessible(true) to make the private constructor >>> MethodHandles.Lookup(Class lookupClass, int allowedModes) accessible, >>> 2. Invoke that private constructor to instantiate a MethodHandles.Lookup >>> with private (!) access to all the things, >>> 3. Call Lookup.unreflectSpecial to get the MethodHandle for the default >>> method, and >>> 4. Invoke the method through the MethodHandle. >>> >>> This is ugly, but it works--for now. If JDK 9 takes away access to >>> setAccessible, it may cease to work. >>> >>> I found some discussion about this last summer [4], but it's hard to >>> tell >>> if anything came out of the discussion. >>> >>> Is there anything on the roadmap for JDK9 to allow a proxy's >>> InvocationHandler to invoke default methods on proxies? Are there any >>> existing proposals I should be aware of? >> >> I have created a prototype last year: >> >> http://mail.openjdk.java.net/pipermail/core-libs-dev/2016-June/041629.html >> >> >> But I think the JDK 9 train has already left the station. So perhaps >> in JDK 10... > > > https://bugs.openjdk.java.net/browse/JDK-8159746 is the JBS issue for this. > > One other possibility is to fix proxy generated class not to override > default methods but there would require more investigation to tease out > before we can be certain if this can make JDK 9. > > Mandy > > -- - DML From david.lloyd at redhat.com Fri Mar 3 16:43:09 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Fri, 3 Mar 2017 10:43:09 -0600 Subject: The easy solution for #AvoidConcealedPackageConflicts, #MultipleModuleVersions Message-ID: I want to propose (again) that the only practical answer for these two issues is to divide application modules (not platform modules) into separated class loaders, always. Any other solution requires packages to be split among modules in one class loader, and requires users to know more about the content of their modules than they should need to care about. Any previous compatibility argument regarding class loader arrangement is already moot, since class loading behavior is different enough now that user code already does not work as it did. There is no longer any reasonable expectation that user code which uses its own class loader to locate resources and classes will find classes that that code cannot find via class resolution, which indeed is the entire point of modularity. Making this optional would only increase confusion and is unlikely to solve any actual problem. Is there any technical reason why we should not just do this? -- - DML From Alan.Bateman at oracle.com Sun Mar 5 12:13:28 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sun, 5 Mar 2017 12:13:28 +0000 Subject: The easy solution for #AvoidConcealedPackageConflicts, #MultipleModuleVersions In-Reply-To: References: Message-ID: <98870728-3308-5ffa-d2ef-1e06d55c3ee0@oracle.com> On 03/03/2017 16:43, David M. Lloyd wrote: > : > > Any previous compatibility argument regarding class loader arrangement > is already moot, since class loading behavior is different enough now > that user code already does not work as it did. There is no longer > any reasonable expectation that user code which uses its own class > loader to locate resources and classes will find classes that that > code cannot find via class resolution, which indeed is the entire > point of modularity. I'm not sure what you mean by "does not work as it did". There is an area where great care has been taken to avoid significant changes as it has the potential to break too much existing code. Specifically, the arrangement of the 3 built-in class loaders and class loading has not changed. The limited changes in this area with compatibility impact are: 1. The JDK's application class loader is no longer a URLClassLoader that can potentially break code that assumes the implementation type. 2. The modules for a few non-core components (e.g. java.corba and java.sql) are defined to the platform class loader rather than the boot loader. This potentially breaks a few cases where code creates a custom class loader with `null` as the parent and assumes that all Java SE types are visible via that loader. The potential for breakage with these implementation (not JPMS) changes is why they have been in JDK 9 for a long time and in EA builds since 2015 to give any impacted libraries and tools as much time as possible to adjust. -Alan From mark.reinhold at oracle.com Mon Mar 6 02:02:44 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Sun, 5 Mar 2017 18:02:44 -0800 (PST) Subject: Proposal: #NonHierarchicalLayers (+ #LayerPrimitives) In-Reply-To: References: <20161207234117.1E5F1213F1@eggemoggin.niobe.net> <1de06198-930b-4b71-fe5d-1477d9cace80@redhat.com> <20161212232335.A9BFC24B27@eggemoggin.niobe.net> <9b66aa52-877a-e4b2-1458-9ff47dfb690b@redhat.com> <20170126223431.B417255DD9@eggemoggin.niobe.net> <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> <20170215195108.C881964E6F@eggemoggin.niobe.net> Message-ID: <20170306020244.3B3D5760B4@eggemoggin.niobe.net> 2017/2/27 17:47:25 +0000, tjwatson at us.ibm.com: > From: mark.reinhold at oracle.com >> From: david.lloyd at redhat.com >>> ... >>> >>> OSGi does support dynamic attachments of fragments. The current >>> prototype cannot do this but on Jan. 4 Thomas expressed that being able >>> to add packages would enable this part of the specification. Re-linking >>> everything is (according to this email) an alternative that comes at the >>> cost of not supporting this feature. >> >> True, but Thomas later recommended against complicating JPMS in order to >> support this feature of OSGi [1]. He can live with having to re-resolve >> bundles when fragments add API packages, which is not common and arguably >> an anti-pattern. > > The addPackage method would be beneficial for other reasons when adhering > an existing module system into a JPMS Layer. > > - addPackage would allow existing module systems to avoid aggressive > discovery of all private packages for its own modules when building a JPMS > ModuleDescriptor. For the existing module systems I know about the class > loader is used extensively as the module primitive. Requirements wire up > the class loaders for proper class loader delegation and typically the > APIs are declared capabilities to indicate what the modular class loaders > can load and expose to another module class loader. But nowhere was it > required to aggressively discover all private packages and resources up > front. Having an addPackage would allow such module systems to grow the > list of private packages lazily as it is defining classes in the packages > that are private to the module. Understood, but for context: I previously suggested that you could use the `Private-Package` manifest header added by bndtools, when present, in order to avoid scanning a bundle for private packages [1]. In reply you wrote that you planned to use that approach for bundles built by bndtools, and that "even for the bundles where we have to scan for all packages the framework can easily cache the private-package information so that upon restart the scan is not needed again" [2]. So, I see how an `addPackage` method could help OSGi interoperation, but it doesn't appear to be a hard requirement. (As a side note, scanning a JAR file to determine its private packages is pretty fast these days, even without caching. That's what we do for automatic modules in the JPMS implementation, and so far it hasn't been a significant performance problem.) > - In my prototype I also have a boot strapping issue that addPackage would > be helpful for. In my prototype I have a launcher that assembles a module > Layer which then holds the OSGi framework implementation. In this case > the launcher is using Java 9 Layer API and then is loading up a standard > OSGi framework implementation using only OSGi standard API. The launcher > then assigns the framework implementation a ModuleDescriptor to represent > the OSGi framework. In OSGi the framework is represented by an OSGi > bundle called the "system.bundle". So in this case the ModuleDescriptor > also has a name "system.bundle". And the ModuleDescriptor has a well > known list of OSGi APIs it exports. But the launcher does not know the > private implementation packages of the framework implementation. In my > prototype I just hack in the list of private package for the equinox > framework, but this is far from ideal. Aren't the solutions for the previous problem also applicable here? If such a launcher must be able to launch an arbitrary OSGi framework implementation then can't it use the `Private-Package` manifest header, when available, or else scan the framework's bundle for private packages and cache that result for later use? A higher-level question is, how important is it for an OSGi-on-JPMS launcher to be completely independent of the OSGi framework that it's launching? If every framework provides its own such launcher then, in each one, a little bit of build logic could be used to embed a list of the system bundle's private packages in the launcher itself, so that there's no need to duplicate that list manually. > I did say that I could live without supporting dynamic attachment of > fragments which provided additional packages. But I do not want that > compromise from me to justify not adding the method addPackage to the > controller. I think it would be beneficial to existing module systems > that need to live within a proper JPMS Layer. Understood. > It would help to understand what complications addPackage bring to JPMS > above the other dynamic add methods already on the Module API and the > others being proposed for the Controller API here. My concern is less about how these additional methods would complicate JPMS -- although they would -- than about the constraints they would place on the future evolution of JPMS, the rest of the platform, and its implementations. The `addReads` and `addOpens` methods that we've already added to the `Layer.Controller` class require essentially the same support from the run-time library and the JVM as the corresponding methods in the `Module` class. Those methods in the `Module` class are essential to the design, so adding them to the `Controller` class presented little risk. The same argument can be made for the qualified `addExports` method and the `addUses` method, though whether adding just these two methods to the `Controller` API would benefit anyone is unclear. The main problem with the remainder of the proposed methods is that they vastly increase the space of situations in which the run-time system must be prepared to update the definitions of modules. That can, in turn, limit both the space of potential implementations and the bounds of practical performance. The current, internal versions of these methods are used only in limited circumstances. Some are invoked only in the process of instantiating a module, and never thereafter. Others exist only to support white-box testing, and are never invoked during normal run-time operations. These limitations on use simplify both the specification and the implementation of the rest of the system, which can be written under the assumption that module definitions change in these ways only rarely, if ever. To generalize these methods and expose them for use at arbitrary times would add complexity to both the specification and the RI, and would very likely degrade performance. To make these methods part of the Java SE API would -- more importantly -- mandate that all future specifications and implementations support such updates to module definitions at arbitrary times. It is not necessary to expose these methods in order to achieve the goal of this JSR. Designing such low-level primitives via which different module systems can interoperate on an intimate basis was never a goal of this JSR. To avoid adding complexity to the specification and its implementations in the short term, and to impose as few constraints as possible on the evolution of the specification and its implementations in the long term, these methods should remain internal. - Mark [1] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000409.html [2] http://mail.openjdk.java.net/pipermail/jpms-spec-comments/2016-October/000070.html From tjwatson at us.ibm.com Mon Mar 6 16:08:19 2017 From: tjwatson at us.ibm.com (Thomas Watson) Date: Mon, 6 Mar 2017 10:08:19 -0600 Subject: Proposal: #NonHierarchicalLayers (+ #LayerPrimitives) In-Reply-To: <20170306020244.3B3D5760B4@eggemoggin.niobe.net> References: <20161207234117.1E5F1213F1@eggemoggin.niobe.net> <1de06198-930b-4b71-fe5d-1477d9cace80@redhat.com> <20161212232335.A9BFC24B27@eggemoggin.niobe.net> <9b66aa52-877a-e4b2-1458-9ff47dfb690b@redhat.com> <20170126223431.B417255DD9@eggemoggin.niobe.net> <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> <20170215195108.C881964E6F@eggemoggin.niobe.net> <20170306020244.3B3D5760B4@eggemoggin.niobe.net> Message-ID: > From: mark.reinhold at oracle.com > > The addPackage method would be beneficial for other reasons when adhering > > an existing module system into a JPMS Layer. > > > > - addPackage would allow existing module systems to avoid aggressive > > discovery of all private packages for its own modules when building a JPMS > > ModuleDescriptor. For the existing module systems I know about the class > > loader is used extensively as the module primitive. Requirements wire up > > the class loaders for proper class loader delegation and typically the > > APIs are declared capabilities to indicate what the modular class loaders > > can load and expose to another module class loader. But nowhere was it > > required to aggressively discover all private packages and resources up > > front. Having an addPackage would allow such module systems to grow the > > list of private packages lazily as it is defining classes in the packages > > that are private to the module. > > Understood, but for context: I previously suggested that you could use > the `Private-Package` manifest header added by bndtools, when present, > in order to avoid scanning a bundle for private packages [1]. In reply > you wrote that you planned to use that approach for bundles built by > bndtools, and that "even for the bundles where we have to scan for all > packages the framework can easily cache the private-package information > so that upon restart the scan is not needed again" [2]. So, I see how an > `addPackage` method could help OSGi interoperation, but it doesn't appear > to be a hard requirement. > > (As a side note, scanning a JAR file to determine its private packages > is pretty fast these days, even without caching. That's what we do for > automatic modules in the JPMS implementation, and so far it hasn't been > a significant performance problem.) The Private-Package cannot be relied on from BND. A very large number of existing bundles are not built with BND. The option to have Private-Package header in BND can be turned off. In many cases the exhaustive scan of the jars will be needed. As a data point, to scan my current Eclipse installation (~360 bundles at well over 310 MB of jars) it is taking over 1000 ms to do the full scan (on a pretty new MacBook Pro). Some optimization may speed this up. In OSGi the artifact the bundle is stored in on disk is abstracted away. There is no independent way to get hold of the jar file directly. I am using OSGi APIs to do the scanning of the bundle wiring. It is possible this is not as speedy as it can be, but the cost will always be measurable and it will linear O(n) with the number of bundles installed. Without caching the results this also pays an upfront cost of having to open each and every jar file. In Equinox we delay the opening of the jars until a class or resource is actually going to be loaded from the bundle. This upfront scanning forces us to open all installed bundle jars every startup even though no classes or resources may be loaded from the jar. The caching comes with its own price in code complication and performance cost to load. Doing this lazily would greatly simplify this and should result in better performance for really large installations. We have common cases with 1000s of bundles installed. > > > - In my prototype I also have a boot strapping issue that addPackage would > > be helpful for. In my prototype I have a launcher that assembles a module > > Layer which then holds the OSGi framework implementation. In this case > > the launcher is using Java 9 Layer API and then is loading up a standard > > OSGi framework implementation using only OSGi standard API. The launcher > > then assigns the framework implementation a ModuleDescriptor to represent > > the OSGi framework. In OSGi the framework is represented by an OSGi > > bundle called the "system.bundle". So in this case the ModuleDescriptor > > also has a name "system.bundle". And the ModuleDescriptor has a well > > known list of OSGi APIs it exports. But the launcher does not know the > > private implementation packages of the framework implementation. In my > > prototype I just hack in the list of private package for the equinox > > framework, but this is far from ideal. > > Aren't the solutions for the previous problem also applicable here? If > such a launcher must be able to launch an arbitrary OSGi framework > implementation then can't it use the `Private-Package` manifest header, > when available, or else scan the framework's bundle for private packages > and cache that result for later use? Perhaps, but there is still a complication with system bundle fragments. I mentioned this in a past thread about how important it is to support dynamic fragment attachment. In general I don't think there is a large number of OSGi users depending on dynamic fragment attachment to normal host bundles. But system bundle fragments are a different story. These are dynamically attached to the framework implementation class loader. These require special support by the launcher in order to provide the framework the ability to add content to its own class loader. In fact, this is how my JPMS POC is implemented, in its own system bundle framework fragment. To make this work the list of equinox framework packages and the list of packages for framework fragment must be specified for the framework module descriptor by the launcher. For my POC I just hardcoded these packages directly into the launcher. In the general case the launcher will not know what framework fragments will be installed during the lifetime of the framework instance. The addPackage and addExports method would allow such a launcher to handle when new packages are added from a framework fragment. > > A higher-level question is, how important is it for an OSGi-on-JPMS > launcher to be completely independent of the OSGi framework that it's > launching? If every framework provides its own such launcher then, in > each one, a little bit of build logic could be used to embed a list of > the system bundle's private packages in the launcher itself, so that > there's no need to duplicate that list manually. There are framework independent launchers that only use OSGi APIs (BND tools has one such example). It would be beneficial if these launchers could do the right thing when running on JPMS. The addPackage and addExportToAll method would make this possible. > > > I did say that I could live without supporting dynamic attachment of > > fragments which provided additional packages. But I do not want that > > compromise from me to justify not adding the method addPackage to the > > controller. I think it would be beneficial to existing module systems > > that need to live within a proper JPMS Layer. > > Understood. > > > It would help to understand what complications addPackage bring to JPMS > > above the other dynamic add methods already on the Module API and the > > others being proposed for the Controller API here. > > My concern is less about how these additional methods would complicate > JPMS -- although they would -- than about the constraints they would > place on the future evolution of JPMS, the rest of the platform, and its > implementations. > > The `addReads` and `addOpens` methods that we've already added to the > `Layer.Controller` class require essentially the same support from the > run-time library and the JVM as the corresponding methods in the `Module` > class. Those methods in the `Module` class are essential to the design, > so adding them to the `Controller` class presented little risk. > > The same argument can be made for the qualified `addExports` method and > the `addUses` method, though whether adding just these two methods to the > `Controller` API would benefit anyone is unclear. > > The main problem with the remainder of the proposed methods is that they > vastly increase the space of situations in which the run-time system must > be prepared to update the definitions of modules. That can, in turn, > limit both the space of potential implementations and the bounds of > practical performance. I obviously don't know the internal details of the java implementation, I can understand your potential concerns. I do wonder if the performance improvements you speak of can still be done in the vast majority of cases where the controller is not exposed at all. The controller is only made available when using one of the static defineModule methods on the Layer API. The static defineModule methods are useful for cases where an existing module system is trying to adhere their modules into JPMS and the additional dynamics are needed by the controller. But for the typical case where it is JPMS only there is no need for the controller at all, so you should be able to count on things remaining static. I say this knowing it does not simplify the spec or RI for the controller case, but it should allow for you to optimize the common case more at some point. > > The current, internal versions of these methods are used only in limited > circumstances. Some are invoked only in the process of instantiating a > module, and never thereafter. Others exist only to support white-box > testing, and are never invoked during normal run-time operations. These > limitations on use simplify both the specification and the implementation > of the rest of the system, which can be written under the assumption that > module definitions change in these ways only rarely, if ever. > > To generalize these methods and expose them for use at arbitrary times > would add complexity to both the specification and the RI, and would very > likely degrade performance. To make these methods part of the Java SE > API would -- more importantly -- mandate that all future specifications > and implementations support such updates to module definitions at > arbitrary times. > > It is not necessary to expose these methods in order to achieve the goal > of this JSR. Designing such low-level primitives via which different > module systems can interoperate on an intimate basis was never a goal > of this JSR. To avoid adding complexity to the specification and its > implementations in the short term, and to impose as few constraints as > possible on the evolution of the specification and its implementations > in the long term, these methods should remain internal. > > - Mark > > > [1] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016- > October/000409.html > [2] http://mail.openjdk.java.net/pipermail/jpms-spec-comments/2016- > October/000070.html > From mark.reinhold at oracle.com Mon Mar 6 17:16:25 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 6 Mar 2017 09:16:25 -0800 (PST) Subject: Proposal: #AvoidConcealedPackageConflicts, #StaticLayerConfiguration, & #MultipleModuleVersions Message-ID: <20170306171625.D38FF768C7@eggemoggin.niobe.net> Issue summaries --------------- #AvoidConcealedPackageConflicts -- Provide a simple means to load two modules, without using reflection, when they contain non-exported packages of the same name. This could be done by loading them via different class loaders or, alternatively, via #StaticLayerConfiguration. [1] #StaticLayerConfiguration -- Layers can, at present, only be created dynamically, via reflection. If there were a way to specify them statically, i.e., at startup time, then they could be used to solve the kinds of version conflicts that are presently addressed via shading or shadowing. [2] #MultipleModuleVersions -- Allow multiple distinct modules of a given name to be loaded in a convenient fashion, without using reflection. This could be done by creating new layers automatically, or by relaxing the constraints on multiple versions within a layer, or by some other means (cf. #StaticLayerConfiguration, #AvoidConcealedPackageConflicts). Addressing this issue may entail reconsidering the multiple versions non-requirement. [3] Proposal -------- These overlapping issues do reflect actual, practical problems. There are, however, already effective -- if somewhat crude -- solutions to these problems via techniques such as shading (in Maven) and shadowing (Gradle). More sophisticated solutions could be designed and implemented in a future release. The lack of immediate solutions to these problems should not block a developer who wants to modularize an existing class-path application. If such an application works properly on the class path today then it likely does not have conflicting packages anyway, since conflicting packages on the class path almost always lead to trouble. [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#AvoidConcealedPackageConflicts [2] http://openjdk.java.net/projects/jigsaw/spec/issues/#StaticLayerConfiguration [3] http://openjdk.java.net/projects/jigsaw/spec/issues/#MultipleModuleVersions From mark.reinhold at oracle.com Mon Mar 6 17:18:25 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 6 Mar 2017 09:18:25 -0800 (PST) Subject: Proposal: #MultiModuleExecutableJARs, #MultiModuleJARs, & #ReifiedModuleGraphs Message-ID: <20170306171825.E3A19768CD@eggemoggin.niobe.net> Issue summaries --------------- #MultiModuleExecutableJARs -- Provide a means to create an executable modular "uber-JAR" that contains more than one module, preserving module identities and boundaries, so that an entire application can be shipped as a single artifact. [1] #MultiModuleJARs -- Provide a means to package more than one module in a single artifact, while preserving module identities and boundaries. (This is a generalization of #MultiModuleExecutableJARs.) [2] #ReifiedModuleGraphs -- Define a means to record, in some type of artifact, the module graph that results from resolving a given main module against a given application module path. The artifact could contain full copies of the corresponding module artifacts, or references to such artifacts, or some combination of the two. At run time such an artifact could be used to create a new layer. [3] Proposal -------- The features suggested by these issues could indeed be useful, but they are not required for the initial release and could be added in a later revision. [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#MultiModuleExecutableJARs [2] http://openjdk.java.net/projects/jigsaw/spec/issues/#MultiModuleJARs [3] http://openjdk.java.net/projects/jigsaw/spec/issues/#ReifiedModuleGraphs From mark.reinhold at oracle.com Mon Mar 6 17:19:25 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 6 Mar 2017 09:19:25 -0800 (PST) Subject: Proposal: #ExportAnnotation Message-ID: <20170306171925.EF1FD768CF@eggemoggin.niobe.net> Issue summary ------------- #ExportAnnotation -- Define a standard, source-retention `Export` annotation which can be used in a `package-info.java` file to indicate that the package is to be exported. Tools that create `module-info.java` files could then interpret this annotation in order to generate corresponding `exports` clauses. [1] Proposal -------- Do not implement this feature. The declaration of a module is centralized in a single file in order to make it easy to see exactly what a module exports, requires, uses, and so forth. Distributing package-export information across a module's `package-info.java` files would just make the related `module-info.java` file more difficult to read. This is not a practice that we should encourage. If someone really wants to do this then they can easily define their own annotation for this purpose. [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ExportAnnotation From mark.reinhold at oracle.com Mon Mar 6 17:20:25 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 6 Mar 2017 09:20:25 -0800 (PST) Subject: Proposal: #ResourceExistenceAndSize Message-ID: <20170306172026.00DCB768D1@eggemoggin.niobe.net> Issue summary ------------- #ResourceExistenceAndSize -- Given a `Module` object and a resource name, should there be a way to determine whether that resource exists in that module and, if it does exist, what its size is, without actually opening the resource as a stream? One way to do this would be to introduce a new abstraction to represent resources. [1] Proposal -------- This could be a useful feature, but it is not required for the initial release and could be added in a later revision. [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ResourceExistenceAndSize From david.lloyd at redhat.com Mon Mar 6 19:44:23 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 6 Mar 2017 13:44:23 -0600 Subject: Proposal: #AvoidConcealedPackageConflicts, #StaticLayerConfiguration, & #MultipleModuleVersions In-Reply-To: <20170306171625.D38FF768C7@eggemoggin.niobe.net> References: <20170306171625.D38FF768C7@eggemoggin.niobe.net> Message-ID: On 03/06/2017 11:16 AM, mark.reinhold at oracle.com wrote: > Issue summaries > --------------- > > #AvoidConcealedPackageConflicts -- Provide a simple means to load two > modules, without using reflection, when they contain non-exported > packages of the same name. This could be done by loading them via > different class loaders or, alternatively, via > #StaticLayerConfiguration. [1] > > #StaticLayerConfiguration -- Layers can, at present, only be created > dynamically, via reflection. If there were a way to specify them > statically, i.e., at startup time, then they could be used to solve the > kinds of version conflicts that are presently addressed via shading or > shadowing. [2] > > #MultipleModuleVersions -- Allow multiple distinct modules of a given > name to be loaded in a convenient fashion, without using reflection. > This could be done by creating new layers automatically, or by relaxing > the constraints on multiple versions within a layer, or by some other > means (cf. #StaticLayerConfiguration, #AvoidConcealedPackageConflicts). > Addressing this issue may entail reconsidering the multiple versions > non-requirement. [3] > > Proposal > -------- > > These overlapping issues do reflect actual, practical problems. There > are, however, already effective -- if somewhat crude -- solutions to > these problems via techniques such as shading (in Maven) and shadowing > (Gradle). More sophisticated solutions could be designed and implemented > in a future release. > > The lack of immediate solutions to these problems should not block a > developer who wants to modularize an existing class-path application. > If such an application works properly on the class path today then it > likely does not have conflicting packages anyway, since conflicting > packages on the class path almost always lead to trouble. We maintain that #AvoidConcealedPackageConflicts and #MultipleModuleVersions are essential for developing medium to large applications. Today developers of such applications do not use the class path; they use existing systems like OSGi or ours, or they roll their own plugin systems. I don't see how JPMS can be considered to fulfill the primary goal stated in its submission if it does not meet these simple needs, particularly since they are so easily solved; I believe it may be or be nearly a single-line change at this point with little if any technical downside that I know of. The bar is and should be higher than modularizing only existing class-path applications. For most users, the entire value proposition of modularity is to easily solve issues like #AvoidConcealedPackageConflicts; if they can't solve things like this, then why would they move to modules in the first place? I don't think I've ever received a non-surprised response when presenting the issues of multiple versions and package conflicts to end users. As for #StaticLayerConfiguration... that is essentially a proposed solution to the other two problems, but I think the more obvious solution of multiple class loaders is much simpler both to implement and to use, so if this issue were closed in favor of that resolution we would be satisfied I think. Your email didn't specifically say that these issues were closed; does that mean that you are still open to discussion on these three issues? -- - DML From mark.reinhold at oracle.com Mon Mar 6 20:35:49 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 6 Mar 2017 12:35:49 -0800 (PST) Subject: Proposal: #NonHierarchicalLayers (+ #LayerPrimitives) In-Reply-To: References: <20161207234117.1E5F1213F1@eggemoggin.niobe.net> <1de06198-930b-4b71-fe5d-1477d9cace80@redhat.com> <20161212232335.A9BFC24B27@eggemoggin.niobe.net> <9b66aa52-877a-e4b2-1458-9ff47dfb690b@redhat.com> <20170126223431.B417255DD9@eggemoggin.niobe.net> <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> <20170215195108.C881964E6F@eggemoggin.niobe.net> <20170306020244.3B3D5760B4@eggemoggin.niobe.net> Message-ID: <20170306203549.15ADF76921@eggemoggin.niobe.net> 2017/3/6 8:08:19 -0800, tjwatson at us.ibm.com: > 2017/3/5 18:02:44 -0800, mark.reinhold at oracle.com: >> 2017/2/27 17:47:25 +0000, tjwatson at us.ibm.com: >>> ... >>> >>> The addPackage method would be beneficial for other reasons when adhering >>> an existing module system into a JPMS Layer. >>> >>> - addPackage would allow existing module systems to avoid aggressive >>> discovery of all private packages for its own modules when building a JPMS >>> ModuleDescriptor. For the existing module systems I know about the class >>> loader is used extensively as the module primitive. Requirements wire up >>> the class loaders for proper class loader delegation and typically the >>> APIs are declared capabilities to indicate what the modular class loaders >>> can load and expose to another module class loader. But nowhere was it >>> required to aggressively discover all private packages and resources up >>> front. Having an addPackage would allow such module systems to grow the >>> list of private packages lazily as it is defining classes in the packages >>> that are private to the module. >> >> Understood, but for context: I previously suggested that you could use >> the `Private-Package` manifest header added by bndtools, when present, >> in order to avoid scanning a bundle for private packages [1]. In reply >> you wrote that you planned to use that approach for bundles built by >> bndtools, and that "even for the bundles where we have to scan for all >> packages the framework can easily cache the private-package information >> so that upon restart the scan is not needed again" [2]. So, I see how an >> `addPackage` method could help OSGi interoperation, but it doesn't appear >> to be a hard requirement. >> >> (As a side note, scanning a JAR file to determine its private packages >> is pretty fast these days, even without caching. That's what we do for >> automatic modules in the JPMS implementation, and so far it hasn't been >> a significant performance problem.) > > The Private-Package cannot be relied on from BND. A very large number of > existing bundles are not built with BND. The option to have > Private-Package header in BND can be turned off. In many cases the > exhaustive scan of the jars will be needed. As a data point, to scan my > current Eclipse installation (~360 bundles at well over 310 MB of jars) it > is taking over 1000 ms to do the full scan (on a pretty new MacBook Pro). > > Some optimization may speed this up. In OSGi the artifact the bundle is > stored in on disk is abstracted away. There is no independent way to get > hold of the jar file directly. I am using OSGi APIs to do the scanning of > the bundle wiring. It is possible this is not as speedy as it can be, but > the cost will always be measurable and it will linear O(n) with the number > of bundles installed. Without caching the results this also pays an > upfront cost of having to open each and every jar file. In Equinox we > delay the opening of the jars until a class or resource is actually going > to be loaded from the bundle. This upfront scanning forces us to open all > installed bundle jars every startup even though no classes or resources > may be loaded from the jar. > > The caching comes with its own price in code complication and performance > cost to load. Doing this lazily would greatly simplify this and should > result in better performance for really large installations. We have > common cases with 1000s of bundles installed. Thanks for the additional background, but this still sounds like a performance problem that can be solved at a higher level rather than in the JVM itself. (Perhaps, in the long run, a future revision of the OSGi specification should add an API to support the caching of bundle package lists.) >>> - In my prototype I also have a boot strapping issue that addPackage would >>> be helpful for. In my prototype I have a launcher that assembles a module >>> Layer which then holds the OSGi framework implementation. In this case >>> the launcher is using Java 9 Layer API and then is loading up a standard >>> OSGi framework implementation using only OSGi standard API. The launcher >>> then assigns the framework implementation a ModuleDescriptor to represent >>> the OSGi framework. In OSGi the framework is represented by an OSGi >>> bundle called the "system.bundle". So in this case the ModuleDescriptor >>> also has a name "system.bundle". And the ModuleDescriptor has a well >>> known list of OSGi APIs it exports. But the launcher does not know the >>> private implementation packages of the framework implementation. In my >>> prototype I just hack in the list of private package for the equinox >>> framework, but this is far from ideal. >> >> Aren't the solutions for the previous problem also applicable here? If >> such a launcher must be able to launch an arbitrary OSGi framework >> implementation then can't it use the `Private-Package` manifest header, >> when available, or else scan the framework's bundle for private packages >> and cache that result for later use? > > Perhaps, but there is still a complication with system bundle fragments. I > mentioned this in a past thread about how important it is to support > dynamic fragment attachment. In general I don't think there is a large > number of OSGi users depending on dynamic fragment attachment to normal > host bundles. But system bundle fragments are a different story. These > are dynamically attached to the framework implementation class loader. > These require special support by the launcher in order to provide the > framework the ability to add content to its own class loader. > > In fact, this is how my JPMS POC is implemented, in its own system bundle > framework fragment. To make this work the list of equinox framework > packages and the list of packages for framework fragment must be specified > for the framework module descriptor by the launcher. For my POC I just > hardcoded these packages directly into the launcher. In the general case > the launcher will not know what framework fragments will be installed > during the lifetime of the framework instance. The addPackage and > addExports method would allow such a launcher to handle when new packages > are added from a framework fragment. How critical is this general case? How often are framework fragments actually used in a way in which they cannot be identified when the framework is launched? If I understand ??3.15 of the OSGi Core R6 specification [6] correctly then a framework implementation is not required to resolve framework fragments (a.k.a. "extension bundles") dynamically, though it is allowed to do so (see steps 2 and 3 of the lifecycle description at the bottom of p. 85). If an OSGi implementation can always restart when a framework fragment is resolved or refreshed then isn't that a natural time to scan the fragment's bundle for its packages, possibly caching that information for later use? Do framework fragments typically add exported packages to the system bundle, or do they only carry non-exported implementation code? >> A higher-level question is, how important is it for an OSGi-on-JPMS >> launcher to be completely independent of the OSGi framework that it's >> launching? If every framework provides its own such launcher then, in >> each one, a little bit of build logic could be used to embed a list of >> the system bundle's private packages in the launcher itself, so that >> there's no need to duplicate that list manually. > > There are framework independent launchers that only use OSGi APIs (BND > tools has one such example). It would be beneficial if these launchers > could do the right thing when running on JPMS. The addPackage and > addExportToAll method would make this possible. Yes, but could this problem be addressed in some other way? Would modest additions to the OSGi API allow a launcher to do a bit of introspection on the framework being launched? >> ... >> >> The main problem with the remainder of the proposed methods is that they >> vastly increase the space of situations in which the run-time system must >> be prepared to update the definitions of modules. That can, in turn, >> limit both the space of potential implementations and the bounds of >> practical performance. > > I obviously don't know the internal details of the java implementation, I > can understand your potential concerns. I do wonder if the performance > improvements you speak of can still be done in the vast majority of cases > where the controller is not exposed at all. The controller is only made > available when using one of the static defineModule methods on the Layer > API. The static defineModule methods are useful for cases where an > existing module system is trying to adhere their modules into JPMS and the > additional dynamics are needed by the controller. But for the typical > case where it is JPMS only there is no need for the controller at all, so > you should be able to count on things remaining static. > > I say this knowing it does not simplify the spec or RI for the controller > case, but it should allow for you to optimize the common case more at some > point. This approach could allow optimization of the common case but, as you note, it would not simplify the specification or its implementations. It would, in fact, likely make implementations even more complex since optimizing only in some cases, even if they are common, is more complex than optimizing all of the time. - Mark [1] https://osgi.org/download/r6/osgi.core-6.0.0.pdf From david.lloyd at redhat.com Mon Mar 6 20:47:18 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 6 Mar 2017 14:47:18 -0600 Subject: Proposal: #NonHierarchicalLayers (+ #LayerPrimitives) In-Reply-To: <20170306203549.15ADF76921@eggemoggin.niobe.net> References: <20161207234117.1E5F1213F1@eggemoggin.niobe.net> <1de06198-930b-4b71-fe5d-1477d9cace80@redhat.com> <20161212232335.A9BFC24B27@eggemoggin.niobe.net> <9b66aa52-877a-e4b2-1458-9ff47dfb690b@redhat.com> <20170126223431.B417255DD9@eggemoggin.niobe.net> <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> <20170215195108.C881964E6F@eggemoggin.niobe.net> <20170306020244.3B3D5760B4@eggemoggin.niobe.net> <20170306203549.15ADF76921@eggemoggin.niobe.net> Message-ID: <9bc026d6-c949-8baf-2407-5a71e2f2873f@redhat.com> On 03/06/2017 02:35 PM, mark.reinhold at oracle.com wrote: > 2017/3/6 8:08:19 -0800, tjwatson at us.ibm.com: >> 2017/3/5 18:02:44 -0800, mark.reinhold at oracle.com: >>> The main problem with the remainder of the proposed methods is that they >>> vastly increase the space of situations in which the run-time system must >>> be prepared to update the definitions of modules. That can, in turn, >>> limit both the space of potential implementations and the bounds of >>> practical performance. >> >> I obviously don't know the internal details of the java implementation, I >> can understand your potential concerns. I do wonder if the performance >> improvements you speak of can still be done in the vast majority of cases >> where the controller is not exposed at all. The controller is only made >> available when using one of the static defineModule methods on the Layer >> API. The static defineModule methods are useful for cases where an >> existing module system is trying to adhere their modules into JPMS and the >> additional dynamics are needed by the controller. But for the typical >> case where it is JPMS only there is no need for the controller at all, so >> you should be able to count on things remaining static. >> >> I say this knowing it does not simplify the spec or RI for the controller >> case, but it should allow for you to optimize the common case more at some >> point. > > This approach could allow optimization of the common case but, as you > note, it would not simplify the specification or its implementations. > It would, in fact, likely make implementations even more complex since > optimizing only in some cases, even if they are common, is more complex > than optimizing all of the time. I don't think that any argument was made to the contrary by Thomas. However I would argue that gains to many (which include but are definitely not limited to OSGi implementations and other containers) more than offset this additional cost to specialty compiler developers, which seems likely to be very small in any event (but would still be worthwhile even if it were substantial). The main objection stated above seems highly theoretical (and easily solved), whereas the benefits to existing code are quite real and measurable. Is there evidence to the contrary? -- - DML From tjwatson at us.ibm.com Mon Mar 6 21:46:20 2017 From: tjwatson at us.ibm.com (Thomas Watson) Date: Mon, 6 Mar 2017 15:46:20 -0600 Subject: Proposal: #NonHierarchicalLayers (+ #LayerPrimitives) In-Reply-To: <20170306203549.15ADF76921@eggemoggin.niobe.net> References: <20161207234117.1E5F1213F1@eggemoggin.niobe.net> <1de06198-930b-4b71-fe5d-1477d9cace80@redhat.com> <20161212232335.A9BFC24B27@eggemoggin.niobe.net> <9b66aa52-877a-e4b2-1458-9ff47dfb690b@redhat.com> <20170126223431.B417255DD9@eggemoggin.niobe.net> <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> <20170215195108.C881964E6F@eggemoggin.niobe.net> <20170306020244.3B3D5760B4@eggemoggin.niobe.net> <20170306203549.15ADF76921@eggemoggin.niobe.net> Message-ID: > From: mark.reinhold at oracle.com > > The caching comes with its own price in code complication and performance > > cost to load. Doing this lazily would greatly simplify this and should > > result in better performance for really large installations. We have > > common cases with 1000s of bundles installed. > > Thanks for the additional background, but this still sounds like a > performance problem that can be solved at a higher level rather than > in the JVM itself. > > (Perhaps, in the long run, a future revision of the OSGi specification > should add an API to support the caching of bundle package lists.) I have my doubts the Private-Package header will become anything standard in OSGi. Similar to how I would not like to list private packages in a module-info.java file, I would not like having to specify the private package list in the bundle manifest. Of coarse that is not really what you are suggesting. Instead you are suggesting the OSGi wiring API could be enhanced to have something similar to the ModuleDescriptor.packages() method to return a cached set of package names. The issue is that information is not needed by the framework (at least as of yet) and should not be needed by any other bundle installed in the framework. I would have a hard time proposing such a method back to the specification. > > In fact, this is how my JPMS POC is implemented, in its own system bundle > > framework fragment. To make this work the list of equinox framework > > packages and the list of packages for framework fragment must be specified > > for the framework module descriptor by the launcher. For my POC I just > > hardcoded these packages directly into the launcher. In the general case > > the launcher will not know what framework fragments will be installed > > during the lifetime of the framework instance. The addPackage and > > addExports method would allow such a launcher to handle when new packages > > are added from a framework fragment. > > How critical is this general case? How often are framework fragments > actually used in a way in which they cannot be identified when the > framework is launched? I know this is important for Eclipse because there are many framework extensions implemented by various eclipse projects as well as others. ObjectTeams and aspectJ weaving projects are two that come to mind. Equinox itself has something called Equinox regions which runs on any OSGi framework as a framework extension. These framework fragments almost always have their own private packages and often have additional APIs that are exported by the system bundle. The way Eclipse provisions these extensions depends on the ability to resolve them dynamically, without a restart. > > If I understand ??3.15 of the OSGi Core R6 specification [6] correctly > then a framework implementation is not required to resolve framework > fragments (a.k.a. "extension bundles") dynamically, though it is allowed > to do so (see steps 2 and 3 of the lifecycle description at the bottom of > p. 85). If an OSGi implementation can always restart when a framework > fragment is resolved or refreshed then isn't that a natural time to scan > the fragment's bundle for its packages, possibly caching that information > for later use? Any practical framework will support attaching system.bundle fragments dynamically. I'm most familiar with Equinox and Felix, and I know both these frameworks support this for system.bundle fragments. I also know a framework cannot pass the compliance tests for system.bundle fragments without supporting dynamic attachment, despite the spec stating that it is not required. > > > > There are framework independent launchers that only use OSGi APIs (BND > > tools has one such example). It would be beneficial if these launchers > > could do the right thing when running on JPMS. The addPackage and > > addExportToAll method would make this possible. > > Yes, but could this problem be addressed in some other way? Would modest > additions to the OSGi API allow a launcher to do a bit of introspection > on the framework being launched? Perhaps, but for my usecases the launcher simply cannot know what will be installed by a management agent later. And to be a practical implementation the framework really needs to support dynamic attachment of system.bundle fragments. Tom From mark.reinhold at oracle.com Mon Mar 6 23:45:25 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 6 Mar 2017 15:45:25 -0800 (PST) Subject: Proposal: #NonHierarchicalLayers (+ #LayerPrimitives) In-Reply-To: References: <20161207234117.1E5F1213F1@eggemoggin.niobe.net> <1de06198-930b-4b71-fe5d-1477d9cace80@redhat.com> <20161212232335.A9BFC24B27@eggemoggin.niobe.net> <9b66aa52-877a-e4b2-1458-9ff47dfb690b@redhat.com> <20170126223431.B417255DD9@eggemoggin.niobe.net> <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> <20170215195108.C881964E6F@eggemoggin.niobe.net> <20170306020244.3B3D5760B4@eggemoggin.niobe.net> <20170306203549.15ADF76921@eggemoggin.niobe.net> Message-ID: <20170306234525.C37CD76960@eggemoggin.niobe.net> 2017/3/6 13:46:20 -0800, tjwatson at us.ibm.com: > 2017/03/06 12:35:49 -0800, mark.reinhold at oracle.com: >> >> ... >> >> Thanks for the additional background, but this still sounds like a >> performance problem that can be solved at a higher level rather than >> in the JVM itself. >> >> (Perhaps, in the long run, a future revision of the OSGi specification >> should add an API to support the caching of bundle package lists.) > > I have my doubts the Private-Package header will become anything standard > in OSGi. Similar to how I would not like to list private packages in a > module-info.java file, I would not like having to specify the private > package list in the bundle manifest. > > Of coarse that is not really what you are suggesting. No, I think it's an idea worth considering. As an optimization it's not clear that it needs to be part of the OSGi standard, though maybe that would be useful. In JPMS we have a similar optimization based upon the `ModulePackages` class-file attribute [1]. We don't ask people to list all packages in `module-info.java` files, but the `jar` tool and others will compute package lists and store them as the value of this attribute in `module-info.class` files. If the module system finds this attribute then it uses its value, otherwise it scans the JAR file to determine the full list of packages. Isn't `Private-Package` of a similar nature? In practice isn't it (or can't it be) emitted by whatever tool you use to construct your manifest, without human intervention, rather than written out by hand? Framework implementations would use this header when present, otherwise they'd scan JAR files as needed (and hopefully cache the results). I understand that this might not be common today, but it could be enabled and encouraged going forward. > Instead you are > suggesting the OSGi wiring API could be enhanced to have something similar > to the ModuleDescriptor.packages() method to return a cached set of > package names. That's one interpretation of my second suggestion, which could be used alone or in combination with the `Private-Package` header. Another form of that suggestion would be a simpler, lower-level API which would allow a framework implementation to obtain a list of the packages in a bundle without the framework reading the bundle's JAR file directly, so that it could implement a package-list cache on its own. > The issue is that information is not needed by the > framework (at least as of yet) and should not be needed by any other > bundle installed in the framework. I would have a hard time proposing > such a method back to the specification. Perhaps better interoperation with JPMS would be adequate justification. >> ... >> >> How critical is this general case? How often are framework fragments >> actually used in a way in which they cannot be identified when the >> framework is launched? > > I know this is important for Eclipse because there are many framework > extensions implemented by various eclipse projects as well as others. > ObjectTeams and aspectJ weaving projects are two that come to mind. > Equinox itself has something called Equinox regions which runs on any OSGi > framework as a framework extension. These framework fragments almost > always have their own private packages and often have additional APIs that > are exported by the system bundle. The way Eclipse provisions these > extensions depends on the ability to resolve them dynamically, without a > restart. Hmm, interesting data. >> If I understand ??3.15 of the OSGi Core R6 specification [6] correctly >> then a framework implementation is not required to resolve framework >> fragments (a.k.a. "extension bundles") dynamically, though it is allowed >> to do so (see steps 2 and 3 of the lifecycle description at the bottom of >> p. 85). If an OSGi implementation can always restart when a framework >> fragment is resolved or refreshed then isn't that a natural time to scan >> the fragment's bundle for its packages, possibly caching that information >> for later use? > > Any practical framework will support attaching system.bundle fragments > dynamically. I'm most familiar with Equinox and Felix, and I know both > these frameworks support this for system.bundle fragments. I also know a > framework cannot pass the compliance tests for system.bundle fragments > without supporting dynamic attachment, despite the spec stating that it is > not required. Sounds like there's a bug in the specification. >> 2017/03/06 08:08:19 -0800, tjwatson at us.ibm.com: >>> ... >>> >>> There are framework independent launchers that only use OSGi APIs (BND >>> tools has one such example). It would be beneficial if these launchers >>> could do the right thing when running on JPMS. The addPackage and >>> addExportToAll method would make this possible. >> >> Yes, but could this problem be addressed in some other way? Would modest >> additions to the OSGi API allow a launcher to do a bit of introspection >> on the framework being launched? > > Perhaps, but for my usecases the launcher simply cannot know what will be > installed by a management agent later. And to be a practical > implementation the framework really needs to support dynamic attachment of > system.bundle fragments. So, here's an even higher-level question: Why is it necessary to make the OSGi system bundle appear as a JPMS module in the first place? The scheme I previously suggested [2] maps an OSGi bundle wiring to a JPMS module and layer so that layers of JPMS modules can be resolved against the bundle directly. If a particular bundle will never be used by a JPMS module, however, then there's no need to handle its wirings in that way, since there's no reason it needs to appear to be a JPMS module. In a mixed OSGi/JPMS system would we realistically expect a JPMS module to require the OSGi system bundle, masquerading as a JPMS module, and use its exported APIs? That seems pretty unlikely. If a component uses an OSGi-specific API then shouldn't it just be written as an actual OSGi bundle in the first place? Could your proof-of-concept implementation be made to work without putting the system bundle in a JPMS layer? If so then it could continue to implement dynamic framework extensions just as it does today, support the dependence of JPMS modules upon OSGi bundles that export APIs that aren't specific to OSGi, and yet still not require the exposure of the low-level internal methods proposed by #LayerPrimitives. - Mark [1] http://cr.openjdk.java.net/~mr/jigsaw/spec/lang-vm.html#jigsaw-2.3 [2] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000410.html From tjwatson at us.ibm.com Tue Mar 7 16:09:59 2017 From: tjwatson at us.ibm.com (Thomas Watson) Date: Tue, 7 Mar 2017 10:09:59 -0600 Subject: Proposal: #NonHierarchicalLayers (+ #LayerPrimitives) In-Reply-To: <20170306234525.C37CD76960@eggemoggin.niobe.net> References: <20161207234117.1E5F1213F1@eggemoggin.niobe.net> <1de06198-930b-4b71-fe5d-1477d9cace80@redhat.com> <20161212232335.A9BFC24B27@eggemoggin.niobe.net> <9b66aa52-877a-e4b2-1458-9ff47dfb690b@redhat.com> <20170126223431.B417255DD9@eggemoggin.niobe.net> <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> <20170215195108.C881964E6F@eggemoggin.niobe.net> <20170306020244.3B3D5760B4@eggemoggin.niobe.net> <20170306203549.15ADF76921@eggemoggin.niobe.net> <20170306234525.C37CD76960@eggemoggin.niobe.net> Message-ID: mark.reinhold at oracle.com wrote on 03/06/2017 05:45:25 PM: > So, here's an even higher-level question: Why is it necessary to make > the OSGi system bundle appear as a JPMS module in the first place? > > The scheme I previously suggested [2] maps an OSGi bundle wiring to a > JPMS module and layer so that layers of JPMS modules can be resolved > against the bundle directly. If a particular bundle will never be used > by a JPMS module, however, then there's no need to handle its wirings in > that way, since there's no reason it needs to appear to be a JPMS module. > > In a mixed OSGi/JPMS system would we realistically expect a JPMS module > to require the OSGi system bundle, masquerading as a JPMS module, and use > its exported APIs? That seems pretty unlikely. If a component uses an > OSGi-specific API then shouldn't it just be written as an actual OSGi > bundle in the first place? > > Could your proof-of-concept implementation be made to work without > putting the system bundle in a JPMS layer? If so then it could continue > to implement dynamic framework extensions just as it does today, support > the dependence of JPMS modules upon OSGi bundles that export APIs that > aren't specific to OSGi, and yet still not require the exposure of the > low-level internal methods proposed by #LayerPrimitives. No that cannot work without hacks to allow other modules to depend on the unnamed system.bundle module. The reason is because we have bundles today that make extensive use of org.osgi.framework packages. These bundles then also export packages that we want to make available to JPMS modules on top. These bundles have to be loaded in to a named module in order to allow these JPMS modules to require them. I think it would be an unfortunate work around to have to link an edge to an unnamed system.bundle module from a bundle module in order to get this functionality to work. It also implies that it will be more difficult in the future to ship an osgi framework implementation jar as a real module since we must load the implementation of the framework into an unnamed module to be fully functional with respect to framework extensions. FYI, for the most part I see your points about other solutions are possible to solve the performance issue with private packages. I'm just pointing out that it was not a necessary concern before Java 9, so we need to be prepared to react. The private-package scanning issue may well have to be solved outside of JPMS. Particularly if the internals of addPackage consider the method to be rarely called (if ever) and therefore there is no need for it to be super fast. But for supporting system.bundle fragment attachment the addPackage and addExportToAll methods are needed. The number of calls to these methods to support system.bundle fragments should be small and not as concerned with how speedy they are. On the other hand, if addPackage is used to solve the private package scanning issue then addPackage must be exceptionally fast and thread safe. Early on in my POC I had the idea of allowing the Layer creation to be able to map a class loader to a to a default module name. Basically the reverse of the existing clf function. Instead of mapping a module name to a class loader this would map a class loader to a default module name. Any package that gets defined in a package that is unknown to the modules associated with that class loader would get defined as part of the default module instead of the current unnamed module. Would this be an acceptable solution for addPackage? We still would need the addExportToAll though. Tom > > - Mark > > > [1] http://cr.openjdk.java.net/~mr/jigsaw/spec/lang-vm.html#jigsaw-2.3 > [2] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016- > October/000410.html > From mark.reinhold at oracle.com Tue Mar 7 16:50:04 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 7 Mar 2017 08:50:04 -0800 (PST) Subject: Draft JPMS Public Review specification Message-ID: <20170307165004.472677715F@eggemoggin.niobe.net> At this point in time the draft specification appears to achieve the goal of this JSR. We have a small and shrinking number of open issues on the list [1], and I expect the resolution of those issues to require at most modest adjustments to the final draft. I therefore intend to submit the following draft to the JCP PMO to be posted for Public Review: http://cr.openjdk.java.net/~mr/jigsaw/spec/ Please let me know by 17:00 UTC next Tuesday, 14 March if you think any changes are required, or if you need more time for review. - Mark [1] http://openjdk.java.net/projects/jigsaw/spec/issues From david.lloyd at redhat.com Tue Mar 7 19:27:08 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 7 Mar 2017 13:27:08 -0600 Subject: Proposal: #NonHierarchicalLayers (+ #LayerPrimitives) In-Reply-To: References: <20161207234117.1E5F1213F1@eggemoggin.niobe.net> <1de06198-930b-4b71-fe5d-1477d9cace80@redhat.com> <20161212232335.A9BFC24B27@eggemoggin.niobe.net> <9b66aa52-877a-e4b2-1458-9ff47dfb690b@redhat.com> <20170126223431.B417255DD9@eggemoggin.niobe.net> <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> <20170215195108.C881964E6F@eggemoggin.niobe.net> <20170306020244.3B3D5760B4@eggemoggin.niobe.net> <20170306203549.15ADF76921@eggemoggin.niobe.net> <20170306234525.C37CD76960@eggemoggin.niobe.net> Message-ID: <1d131755-dfa5-90ec-e332-9290293f5c54@redhat.com> On 03/07/2017 10:09 AM, Thomas Watson wrote: > mark.reinhold at oracle.com wrote on 03/06/2017 05:45:25 PM: >> So, here's an even higher-level question: Why is it necessary to make >> the OSGi system bundle appear as a JPMS module in the first place? >> >> The scheme I previously suggested [2] maps an OSGi bundle wiring to a >> JPMS module and layer so that layers of JPMS modules can be resolved >> against the bundle directly. If a particular bundle will never be used >> by a JPMS module, however, then there's no need to handle its wirings in >> that way, since there's no reason it needs to appear to be a JPMS > module. >> >> In a mixed OSGi/JPMS system would we realistically expect a JPMS module >> to require the OSGi system bundle, masquerading as a JPMS module, and > use >> its exported APIs? That seems pretty unlikely. If a component uses an >> OSGi-specific API then shouldn't it just be written as an actual OSGi >> bundle in the first place? >> >> Could your proof-of-concept implementation be made to work without >> putting the system bundle in a JPMS layer? If so then it could continue >> to implement dynamic framework extensions just as it does today, support >> the dependence of JPMS modules upon OSGi bundles that export APIs that >> aren't specific to OSGi, and yet still not require the exposure of the >> low-level internal methods proposed by #LayerPrimitives. > > No that cannot work without hacks to allow other modules to depend on the > unnamed system.bundle module. The reason is because we have bundles today > that make extensive use of org.osgi.framework packages. These bundles > then also export packages that we want to make available to JPMS modules > on top. These bundles have to be loaded in to a named module in order to > allow these JPMS modules to require them. I think it would be an > unfortunate work around to have to link an edge to an unnamed > system.bundle module from a bundle module in order to get this > functionality to work. It also implies that it will be more difficult in > the future to ship an osgi framework implementation jar as a real module > since we must load the implementation of the framework into an unnamed > module to be fully functional with respect to framework extensions. > > > FYI, for the most part I see your points about other solutions are > possible to solve the performance issue with private packages. I'm just > pointing out that it was not a necessary concern before Java 9, so we need > to be prepared to react. The private-package scanning issue may well have > to be solved outside of JPMS. Particularly if the internals of addPackage > consider the method to be rarely called (if ever) and therefore there is > no need for it to be super fast. But for supporting system.bundle > fragment attachment the addPackage and addExportToAll methods are needed. > The number of calls to these methods to support system.bundle fragments > should be small and not as concerned with how speedy they are. On the > other hand, if addPackage is used to solve the private package scanning > issue then addPackage must be exceptionally fast and thread safe. > > Early on in my POC I had the idea of allowing the Layer creation to be > able to map a class loader to a to a default module name. Basically the > reverse of the existing clf function. Instead of mapping a module name to > a class loader this would map a class loader to a default module name. Any > package that gets defined in a package that is unknown to the modules > associated with that class loader would get defined as part of the default > module instead of the current unnamed module. Would this be an acceptable > solution for addPackage? We still would need the addExportToAll though. This would work for us as well as a replacement for addPackage. -- - DML From tjwatson at us.ibm.com Wed Mar 8 14:09:11 2017 From: tjwatson at us.ibm.com (Thomas Watson) Date: Wed, 8 Mar 2017 08:09:11 -0600 Subject: Concerns mapping existing dynamic module systems to a 1-1 Module->Layer In-Reply-To: <20170307165004.472677715F@eggemoggin.niobe.net> References: <20170307165004.472677715F@eggemoggin.niobe.net> Message-ID: There have been several topics recently discussing concerns in one context or another when trying to use JPMS with an existing module system (like OSGi or JBoss Modules) where the existing module system supports dynamic modules. At a high level, the solution involves a 1-1 mapping for Module->Layer. This allows such a dyamic module system to make use of the dynamic Layer creation to support the existing module system. From my experience there are some performance concerns that come in three different categories when using a 1-1 mapping between the Module and Layer. 1) The need to discover private packages for the ModuleDescriptor. Not specifically related to the 1-1 mapping but required to correctly create the Module for the Layer. The extra work to scan for private packages could be handled outside of JPMS with some kind of caching within the dynamic module system. 2) The need to aggressively create every class loader for every module when creating a layer. The Layer API requires the layer management code to supply a function to map a module name to a class loader and this function is called for each module included in the Layer at construction time. This eliminates any possibility of lazy class loader creation. Creating a class loader is not as expensive as it used to be on older VMs, but it is measurable, particularly if you need to aggressively create 1000s of them all up front. In Equinox we delay class loader creation until just before the first class load for the bundle. This layer function eliminates that possibility. 3) 1-1 mapping of of Module->Layer means that we have to do a separate JPMS resolution for each and every module in our system. This JPMS resolution has no value to our existing module systems because we already did the resolution outside of JPMS. We are just trying to map our resolution into JPMS. This design forces us to double resolve. I need to adjust my POC to see how expensive it is to do the resolution in many Layers vs one Layer (at the cost of no dynamics). This would not be a real solution, it would just pinpoint that the overhead can be attributed to the extra JPMS resolution we have to do for each Layer we have to create. Right now resolution of the Layer is taking nearly 1ms per Layer (that is not counting the time it takes to create the class loaders). Altogether the overhead appears to be 4-5 ms per module (on a pretty snappy machine). Caching the discovery of private packages does cut that by at least half. Most of the remaining overhead can be attributed to the fact that we are using 1-1 Module->Layer mappings in order to support dynamics as well as hiding our own resolution wires that do not fit into the strict rules of JPMS when dealing with split packages or cycles. Most of the additional methods David Lloyd is suggesting is to get the functionality needed to do the complete mapping of existing module systems into JPMS. Essentially I think we are trying to work with JPMS in order to solve the following requirement [1]: Interoperation ? It must be possible for another module system, such as OSGi, to locate Java modules and resolve them using its own resolver, except possibly for core system modules. While the solution we have today achieves this to some extent it is not obvious to me that we have done this in the most useful of ways. In particular, all the additions David is suggesting are being resisted because they potentially are exposing internal implementation details of the JVM which is not desired by the JVM team. To be a complete solution for the JBoss case all (most?) of the methods David suggests are needed. As I stated already, OSGi really only needs the addPackage and addExportsToAll methods to be complete. But this still requires double resolution work to get us back to the functionality we were at with Java 8 and that is only if the suggested methods make it into java 9. Taking a certain interpretation of the above requirement seems to imply that we should be able to have a solution that does not require double resolution at the JPMS Layer. It seems to implies that we should be able to use our own resolver to wire the modules for our own module system how ever we see fit. But I don't think that is really what we got with the current solution which requires JPMS module resolution and then us using the controller to provide the read edges required by our module system class loaders. I was not in on the early discussions of JPMS, but was there any thread of discussion about an approach that provided a pre-resolved graph to JPMS? It seems like such a solution would naturally fit into the spirit of the interoperation requirement. I acknowledge that such an idea is a drastic turn in the current design and I don't see how they could be contained in any reasonable way. I think we are left with the Layer Controller now for better or worse. I hope the overhead of creating and resolving 100s of Layers can be optimized in the future so we can avoid having to do this largely unnecessary resolution work when dealing with existing module systems. But if the Controller is what we are left with I think the solution needs to be as complete as possible for the usecases it is used for. Namely to do the adhering of JBoss Modules and OSGi into JPMS. The next suggestion could be to implement this by always hiding our module system requirements, that should speed up resolution because there is nothing for JPMS to resolve. I would have to confirm this actually helps, but I would rather not do this when OSGi resolution wires can fit into the JPMS rules (i.e. does not contain any cycles or split packages). Here I would rather have a correctly reflected resolution of the Modules within the hierarchical Layer graph when it is possible, and only do the addReads edges trick post Layer resolution when we actually have cycles or split packages to deal with. The reason this is important is because it allows us to more naturally fit into JPMS when it is possible. For example, I think that is the easy way for us to ensure transitive dependencies work when we are mapping require-bundle (with visibility reexport) into JPMS modules which other real JPMS modules can depend on. This way JPMS module resolution for the regular JPMS layers works as usual for requires transitive. My TODO here is still to prove that the double resolution and required extra Layer per Module is contributing to the extra overhead. I suspect that it is, but I guess it could be possible that just creating and resolve 100s of Modules in one Layer also has the same overhead. Tom [1] http://openjdk.java.net/projects/jigsaw/spec/reqs/2015-04-01#interoperation From mark.reinhold at oracle.com Wed Mar 8 20:07:39 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 8 Mar 2017 12:07:39 -0800 (PST) Subject: Proposal: #NonHierarchicalLayers (+ #LayerPrimitives) In-Reply-To: References: <20161207234117.1E5F1213F1@eggemoggin.niobe.net> <1de06198-930b-4b71-fe5d-1477d9cace80@redhat.com> <20161212232335.A9BFC24B27@eggemoggin.niobe.net> <9b66aa52-877a-e4b2-1458-9ff47dfb690b@redhat.com> <20170126223431.B417255DD9@eggemoggin.niobe.net> <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> <20170215195108.C881964E6F@eggemoggin.niobe.net> <20170306020244.3B3D5760B4@eggemoggin.niobe.net> <20170306203549.15ADF76921@eggemoggin.niobe.net> <20170306234525.C37CD76960@eggemoggin.niobe.net> Message-ID: <20170308200739.18F0B815D9@eggemoggin.niobe.net> 2017/3/7 8:09:59 -0800, tjwatson at us.ibm.com: > mark.reinhold at oracle.com wrote on 03/06/2017 05:45:25 PM: >> So, here's an even higher-level question: Why is it necessary to make >> the OSGi system bundle appear as a JPMS module in the first place? >> >> ... >> >> Could your proof-of-concept implementation be made to work without >> putting the system bundle in a JPMS layer? If so then it could continue >> to implement dynamic framework extensions just as it does today, support >> the dependence of JPMS modules upon OSGi bundles that export APIs that >> aren't specific to OSGi, and yet still not require the exposure of the >> low-level internal methods proposed by #LayerPrimitives. > > No that cannot work without hacks to allow other modules to depend on the > unnamed system.bundle module. The reason is because we have bundles today > that make extensive use of org.osgi.framework packages. These bundles > then also export packages that we want to make available to JPMS modules > on top. These bundles have to be loaded in to a named module in order to > allow these JPMS modules to require them. I think it would be an > unfortunate work around to have to link an edge to an unnamed > system.bundle module from a bundle module in order to get this > functionality to work. Okay, so it's possible, but it's unpleasant. > It also implies that it will be more difficult in > the future to ship an osgi framework implementation jar as a real module > since we must load the implementation of the framework into an unnamed > module to be fully functional with respect to framework extensions. More difficult, perhaps, but still possible. An OSGi framework could be embedded in a JPMS module as a JAR-file resource that its launcher, which is regular code in that JPMS module, loads directly into the unnamed module of an appropriate class loader. I don't think it's unreasonable for an OSGi/JPMS interoperation solution to have to do these things, especially when the alternative is to commit the entire Java SE Platform to otherwise-unnecessary complexity for the long term. > FYI, for the most part I see your points about other solutions are > possible to solve the performance issue with private packages. I'm just > pointing out that it was not a necessary concern before Java 9, so we need > to be prepared to react. The private-package scanning issue may well have > to be solved outside of JPMS. Particularly if the internals of addPackage > consider the method to be rarely called (if ever) and therefore there is > no need for it to be super fast. But for supporting system.bundle > fragment attachment the addPackage and addExportToAll methods are needed. > The number of calls to these methods to support system.bundle fragments > should be small and not as concerned with how speedy they are. On the > other hand, if addPackage is used to solve the private package scanning > issue then addPackage must be exceptionally fast and thread safe. All of these low-level methods must be thread-safe in any case, no matter how often they're used, and that's a big part of the inherent complexity. > Early on in my POC I had the idea of allowing the Layer creation to be > able to map a class loader to a to a default module name. Basically the > reverse of the existing clf function. Instead of mapping a module name to > a class loader this would map a class loader to a default module name. Any > package that gets defined in a package that is unknown to the modules > associated with that class loader would get defined as part of the default > module instead of the current unnamed module. Would this be an acceptable > solution for addPackage? We still would need the addExportToAll though. In implementation terms this would be little different from `addPackage`, since the JVM would still have to be prepared to add packages to a named module at any time. - Mark From mark.reinhold at oracle.com Wed Mar 8 20:08:39 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 8 Mar 2017 12:08:39 -0800 (PST) Subject: Concerns mapping existing dynamic module systems to a 1-1 Module->Layer In-Reply-To: References: Message-ID: <20170308200839.1CC9E815DD@eggemoggin.niobe.net> 2017/3/8 6:09:11 -0800, tjwatson at us.ibm.com: > There have been several topics recently discussing concerns in one context > or another when trying to use JPMS with an existing module system (like > OSGi or JBoss Modules) where the existing module system supports dynamic > modules. > > At a high level, the solution involves a 1-1 mapping for Module->Layer. > This allows such a dyamic module system to make use of the dynamic Layer > creation to support the existing module system. > > From my experience there are some performance concerns that come in three > different categories when using a 1-1 mapping between the Module and > Layer. > > ... > > Altogether the overhead appears to be 4-5 ms per module (on a pretty > snappy machine). Caching the discovery of private packages does cut that > by at least half. Most of the remaining overhead can be attributed to the > fact that we are using 1-1 Module->Layer mappings in order to support > dynamics as well as hiding our own resolution wires that do not fit into > the strict rules of JPMS when dealing with split packages or cycles. > > Most of the additional methods David Lloyd is suggesting is to get the > functionality needed to do the complete mapping of existing module systems > into JPMS. Essentially I think we are trying to work with JPMS in order > to solve the following requirement [1]: > > Interoperation ??? It must be possible for another module system, such as > OSGi, to locate Java modules and resolve them using its own resolver, > except possibly for core system modules. To clarify, this requirement does not imply bidirectional interoperation with dynamic module systems, as we've been discussing here. All that this requirement means is that another module system must be able to read JPMS module metadata in order to resolve JPMS modules on its own terms. In the case of OSGi, e.g., a suitably-enhanced framework implementation could use the `java.lang.module.ModuleFinder` API to locate a module's definition, and then use the `ModuleReader` API to read its metadata and content to synthesize a corresponding OSGi bundle that's loaded by an otherwise-ordinary OSGi class loader. Support for bidirectional interoperation is not mandated by the agreed requirements nor by the original JSR submission, as I've noted previously [2]. I thought we'd found a way to achieve this stretch goal for OSGi, but this recent discussion suggests that further research and exploration is required. I hope you'll understand that I'm reluctant to delay this JSR any further in order to accommodate such work. > While the solution we have today achieves this to some extent it is not > obvious to me that we have done this in the most useful of ways. In > particular, all the additions David is suggesting are being resisted > because they potentially are exposing internal implementation details of > the JVM which is not desired by the JVM team. To be a complete solution > for the JBoss case all (most?) of the methods David suggests are needed. > As I stated already, OSGi really only needs the addPackage and > addExportsToAll methods to be complete. But this still requires double > resolution work to get us back to the functionality we were at with Java 8 > and that is only if the suggested methods make it into java 9. No changes to OSGi implementations, nor to JPMS, should be required in order for OSGi implementations to achieve the level of functionality that they have on Java 8. An OSGi implementation should run just fine on top of Java 9, without interacting directly with JPMS in any way. (If it doesn't then there's a bug somewhere, in which case please report it and we'll fix it.) What we've been discussing here goes well beyond the functionality available on Java 8. > Taking a certain interpretation of the above requirement seems to imply > that we should be able to have a solution that does not require double > resolution at the JPMS Layer. It seems to implies that we should be able > to use our own resolver to wire the modules for our own module system how > ever we see fit. But I don't think that is really what we got with the > current solution which requires JPMS module resolution and then us using > the controller to provide the read edges required by our module system > class loaders. I was not in on the early discussions of JPMS, but was > there any thread of discussion about an approach that provided a > pre-resolved graph to JPMS? No, not that I recall. > It seems like such a solution would naturally > fit into the spirit of the interoperation requirement. I acknowledge that > such an idea is a drastic turn in the current design and I don't see how > they could be contained in any reasonable way. That would indeed be an enormous change at this point in time, and it would be justified only by a stretch goal rather than by an agreed requirement. > I think we are left with the Layer Controller now for better or worse. I > hope the overhead of creating and resolving 100s of Layers can be > optimized in the future so we can avoid having to do this largely > unnecessary resolution work when dealing with existing module systems. > But if the Controller is what we are left with I think the solution needs > to be as complete as possible for the usecases it is used for. Namely to > do the adhering of JBoss Modules and OSGi into JPMS. Sorry, but I remain unwilling to expose the requested low-level primitive operations, for the reasons previously stated. If the `Layer.Controller` API is insufficient as it stands today then we should defer support for bidirectional interoperation to a future release. - Mark [1] http://openjdk.java.net/projects/jigsaw/spec/reqs/2015-04-01#interoperation [2] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000410.html From david.lloyd at redhat.com Wed Mar 8 21:05:01 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 8 Mar 2017 15:05:01 -0600 Subject: Draft JPMS Public Review specification In-Reply-To: <20170307165004.472677715F@eggemoggin.niobe.net> References: <20170307165004.472677715F@eggemoggin.niobe.net> Message-ID: <67eda315-7bbd-5249-c5fd-5838b04a33a1@redhat.com> Mark, Right now a Public Review definitely does not seem appropriate given that at least some of the EG still sees major problems with the draft, whereas it sends a message that there is consensus when there is not. Given the fact that there still are significant issues outstanding and disagreement within the EG, it seems likely to me that a at least a second Public Review would need to be submitted anyway, one way or another. In this light, I think it makes sense to wait at least until there's some semblance of consensus among the EG before posting the public review. On 03/07/2017 10:50 AM, mark.reinhold at oracle.com wrote: > At this point in time the draft specification appears to achieve the goal > of this JSR. We have a small and shrinking number of open issues on the > list [1], and I expect the resolution of those issues to require at most > modest adjustments to the final draft. I therefore intend to submit the > following draft to the JCP PMO to be posted for Public Review: > > http://cr.openjdk.java.net/~mr/jigsaw/spec/ > > Please let me know by 17:00 UTC next Tuesday, 14 March if you think any > changes are required, or if you need more time for review. > > - Mark > > > [1] http://openjdk.java.net/projects/jigsaw/spec/issues > -- - DML From tjwatson at us.ibm.com Wed Mar 8 21:33:09 2017 From: tjwatson at us.ibm.com (Thomas Watson) Date: Wed, 8 Mar 2017 15:33:09 -0600 Subject: Concerns mapping existing dynamic module systems to a 1-1 Module->Layer In-Reply-To: <20170308200839.1CC9E815DD@eggemoggin.niobe.net> References: <20170308200839.1CC9E815DD@eggemoggin.niobe.net> Message-ID: > From: mark.reinhold at oracle.com > To: Thomas Watson/Austin/IBM at IBMUS > Cc: jpms-spec-experts at openjdk.java.net > Date: 03/08/2017 02:08 PM > Subject: Re: Concerns mapping existing dynamic module systems to a > 1-1 Module->Layer > > 2017/3/8 6:09:11 -0800, tjwatson at us.ibm.com: > > There have been several topics recently discussing concerns in one context > > or another when trying to use JPMS with an existing module system (like > > OSGi or JBoss Modules) where the existing module system supports dynamic > > modules. > > > > At a high level, the solution involves a 1-1 mapping for Module->Layer. > > This allows such a dyamic module system to make use of the dynamic Layer > > creation to support the existing module system. > > > > From my experience there are some performance concerns that come in three > > different categories when using a 1-1 mapping between the Module and > > Layer. > > > > ... > > > > Altogether the overhead appears to be 4-5 ms per module (on a pretty > > snappy machine). Caching the discovery of private packages does cut that > > by at least half. Most of the remaining overhead can be attributed to the > > fact that we are using 1-1 Module->Layer mappings in order to support > > dynamics as well as hiding our own resolution wires that do not fit into > > the strict rules of JPMS when dealing with split packages or cycles. > > > > Most of the additional methods David Lloyd is suggesting is to get the > > functionality needed to do the complete mapping of existing module systems > > into JPMS. Essentially I think we are trying to work with JPMS in order > > to solve the following requirement [1]: > > > > Interoperation ??? It must be possible for another module system, such as > > OSGi, to locate Java modules and resolve them using its own resolver, > > except possibly for core system modules. > > To clarify, this requirement does not imply bidirectional interoperation > with dynamic module systems, as we've been discussing here. All that > this requirement means is that another module system must be able to read > JPMS module metadata in order to resolve JPMS modules on its own terms. > In the case of OSGi, e.g., a suitably-enhanced framework implementation > could use the `java.lang.module.ModuleFinder` API to locate a module's > definition, and then use the `ModuleReader` API to read its metadata and > content to synthesize a corresponding OSGi bundle that's loaded by an > otherwise-ordinary OSGi class loader. I can confirm that this is possible, I have implemented something along these lines. But I imagine there are going to be JPMS modules out there that will not function correctly when their own classes start returning an unnamed module from Class.getModule(). In my opinion, for a JPMS module to be loaded as a bundle in OSGi the classes for that bundle must be associated with a named module. > > Support for bidirectional interoperation is not mandated by the agreed > requirements nor by the original JSR submission, as I've noted previously > [2]. I thought we'd found a way to achieve this stretch goal for OSGi, > but this recent discussion suggests that further research and exploration > is required. I hope you'll understand that I'm reluctant to delay this > JSR any further in order to accommodate such work. > > > While the solution we have today achieves this to some extent it is not > > obvious to me that we have done this in the most useful of ways. In > > particular, all the additions David is suggesting are being resisted > > because they potentially are exposing internal implementation details of > > the JVM which is not desired by the JVM team. To be a complete solution > > for the JBoss case all (most?) of the methods David suggests are needed. > > As I stated already, OSGi really only needs the addPackage and > > addExportsToAll methods to be complete. But this still requires double > > resolution work to get us back to the functionality we were at with Java 8 > > and that is only if the suggested methods make it into java 9. > > No changes to OSGi implementations, nor to JPMS, should be required in > order for OSGi implementations to achieve the level of functionality that > they have on Java 8. An OSGi implementation should run just fine on top > of Java 9, without interacting directly with JPMS in any way. (If it > doesn't then there's a bug somewhere, in which case please report it and > we'll fix it.) What we've been discussing here goes well beyond the > functionality available on Java 8. Understood. Let me clarify. With an enhanced OSGi framework that loads OSGi bundles as JPMS modules/layers we cannot achieve the same level of OSGi functionality as would be possible when running in the compatibility unnamed module space (or on top of Java 8). > > > Taking a certain interpretation of the above requirement seems to imply > > that we should be able to have a solution that does not require double > > resolution at the JPMS Layer. It seems to implies that we should be able > > to use our own resolver to wire the modules for our own module system how > > ever we see fit. But I don't think that is really what we got with the > > current solution which requires JPMS module resolution and then us using > > the controller to provide the read edges required by our module system > > class loaders. I was not in on the early discussions of JPMS, but was > > there any thread of discussion about an approach that provided a > > pre-resolved graph to JPMS? > > No, not that I recall. > > > It seems like such a solution would naturally > > fit into the spirit of the interoperation requirement. I acknowledge that > > such an idea is a drastic turn in the current design and I don't see how > > they could be contained in any reasonable way. > > That would indeed be an enormous change at this point in time, and it > would be justified only by a stretch goal rather than by an agreed > requirement. > > > I think we are left with the Layer Controller now for better or worse. I > > hope the overhead of creating and resolving 100s of Layers can be > > optimized in the future so we can avoid having to do this largely > > unnecessary resolution work when dealing with existing module systems. > > But if the Controller is what we are left with I think the solution needs > > to be as complete as possible for the usecases it is used for. Namely to > > do the adhering of JBoss Modules and OSGi into JPMS. > > Sorry, but I remain unwilling to expose the requested low-level primitive > operations, for the reasons previously stated. If the `Layer.Controller` > API is insufficient as it stands today then we should defer support for > bidirectional interoperation to a future release. The Layer.Controller gives us the tools to necessary to achieve a form bidirectional support. The Controller should not be dropped from Java 9. > > - Mark > > > [1] http://openjdk.java.net/projects/jigsaw/spec/reqs/ > 2015-04-01#interoperation > [2] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016- > October/000410.html > Tom From tjwatson at us.ibm.com Wed Mar 8 21:22:32 2017 From: tjwatson at us.ibm.com (Thomas Watson) Date: Wed, 8 Mar 2017 15:22:32 -0600 Subject: Proposal: #NonHierarchicalLayers (+ #LayerPrimitives) In-Reply-To: <20170308200739.18F0B815D9@eggemoggin.niobe.net> References: <20161207234117.1E5F1213F1@eggemoggin.niobe.net> <20161212232335.A9BFC24B27@eggemoggin.niobe.net> <9b66aa52-877a-e4b2-1458-9ff47dfb690b@redhat.com> <20170126223431.B417255DD9@eggemoggin.niobe.net> <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> <20170215195108.C881964E6F@eggemoggin.niobe.net> <20170306020244.3B3D5760B4@eggemoggin.niobe.net> <20170306203549.15ADF76921@eggemoggin.niobe.net> <20170306234525.C37CD76960@eggemoggin.niobe.net> Message-ID: mark.reinhold at oracle.com wrote on 03/08/2017 02:07:39 PM: > > No that cannot work without hacks to allow other modules to depend on the > > unnamed system.bundle module. The reason is because we have bundles today > > that make extensive use of org.osgi.framework packages. These bundles > > then also export packages that we want to make available to JPMS modules > > on top. These bundles have to be loaded in to a named module in order to > > allow these JPMS modules to require them. I think it would be an > > unfortunate work around to have to link an edge to an unnamed > > system.bundle module from a bundle module in order to get this > > functionality to work. > > Okay, so it's possible, but it's unpleasant. That is one way to put it! I think I would look for other ways to load the extension bundles into JPMS. For example, perhaps I can place them in a new Layer/Module that is separate from the framework system.bundle module but maps to the same class loader. This would deviate from how normal fragments are handled in the framework, but I think that would be better than hacking in an unnamed module. Care would need to be taken not to claim the extension module contained packages already contained by the system.bundle, but that is possible to detect. > > > It also implies that it will be more difficult in > > the future to ship an osgi framework implementation jar as a real module > > since we must load the implementation of the framework into an unnamed > > module to be fully functional with respect to framework extensions. > > More difficult, perhaps, but still possible. An OSGi framework could be > embedded in a JPMS module as a JAR-file resource that its launcher, which > is regular code in that JPMS module, loads directly into the unnamed > module of an appropriate class loader. > > I don't think it's unreasonable for an OSGi/JPMS interoperation solution > to have to do these things, especially when the alternative is to commit > the entire Java SE Platform to otherwise-unnecessary complexity for the > long term. If it was only for this limited case in the support of OSGi framework extensions then I tend to agree. I am under the impression that others have more general usecases for these methods. I am only giving feedback for my usecase to justify the need for the methods. There may be some ways around this for OSGi. I have to investigate to see how bad or difficult it makes life for us. > > > FYI, for the most part I see your points about other solutions are > > possible to solve the performance issue with private packages. I'm just > > pointing out that it was not a necessary concern before Java 9, so we need > > to be prepared to react. The private-package scanning issue may well have > > to be solved outside of JPMS. Particularly if the internals of addPackage > > consider the method to be rarely called (if ever) and therefore there is > > no need for it to be super fast. But for supporting system.bundle > > fragment attachment the addPackage and addExportToAll methods are needed. > > The number of calls to these methods to support system.bundle fragments > > should be small and not as concerned with how speedy they are. On the > > other hand, if addPackage is used to solve the private package scanning > > issue then addPackage must be exceptionally fast and thread safe. > > All of these low-level methods must be thread-safe in any case, no matter > how often they're used, and that's a big part of the inherent complexity. Understood. > > > Early on in my POC I had the idea of allowing the Layer creation to be > > able to map a class loader to a to a default module name. Basically the > > reverse of the existing clf function. Instead of mapping a module name to > > a class loader this would map a class loader to a default module name. Any > > package that gets defined in a package that is unknown to the modules > > associated with that class loader would get defined as part of the default > > module instead of the current unnamed module. Would this be an acceptable > > solution for addPackage? We still would need the addExportToAll though. > > In implementation terms this would be little different from `addPackage`, > since the JVM would still have to be prepared to add packages to a named > module at any time. I figured as much. > > - Mark > Tom From david.lloyd at redhat.com Wed Mar 8 22:17:03 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 8 Mar 2017 16:17:03 -0600 Subject: Concerns mapping existing dynamic module systems to a 1-1 Module->Layer In-Reply-To: <20170308200839.1CC9E815DD@eggemoggin.niobe.net> References: <20170308200839.1CC9E815DD@eggemoggin.niobe.net> Message-ID: <740acf92-552c-6760-9bee-391f712e3da5@redhat.com> On 03/08/2017 02:08 PM, mark.reinhold at oracle.com wrote: > Sorry, but I remain unwilling to expose the requested low-level primitive > operations, for the reasons previously stated. If the `Layer.Controller` > API is insufficient as it stands today then we should defer support for > bidirectional interoperation to a future release. I can't see Red Hat supporting this JSR without what has been categorized as bidirectional OSGi interoperation. Not just because of our so-called "narrow needs" but because overall this is the very last-ditch path through a maze of constraints that we have needed to traverse in order to work around some critical problems and limitations of JPMS itself. This in turn is going to impact community users whose development and execution model does not match whatever highly specific case it is that JPMS is trying to model, and it will do so in very real, well-understood ways. And the only other option (not using or supporting JPMS) is also a non-option given that it is being positioned as a standard. Essentially you are asking the EG to ratify an unproven, clean-room module model *as a standard* when many other proven, real-world models exist, to which the proposed model bears little resemblance. I just don't see how that is a reasonable expectation. This additional complexity that these extra methods is supposed to introduce has never been quantified. The only concrete measure we have right now is the dozen-odd lines of code that the proposed patch would add. If we can't concretely quantify this complexity, and it is the justification for denying this request, then we cannot reach consensus because there's no way to have a rational discussion about it. -- - DML From mark.reinhold at oracle.com Wed Mar 8 22:31:45 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 08 Mar 2017 14:31:45 -0800 Subject: Proposal: #NonHierarchicalLayers (+ #LayerPrimitives) In-Reply-To: References: <20161207234117.1E5F1213F1@eggemoggin.niobe.net> <20161212232335.A9BFC24B27@eggemoggin.niobe.net> <9b66aa52-877a-e4b2-1458-9ff47dfb690b@redhat.com> <20170126223431.B417255DD9@eggemoggin.niobe.net> <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> <20170215195108.C881964E6F@eggemoggin.niobe.net> <20170306020244.3B3D5760B4@eggemoggin.niobe.net> <20170306203549.15ADF76921@eggemoggin.niobe.net> <20170306234525.C37CD76960@eggemoggin.niobe.net> <20170308200739.18F0B815D9@eggemoggin.niobe.net> Message-ID: <20170308143145.414784649@eggemoggin.niobe.net> 2017/3/8 13:22:32 -0800, tjwatson at us.ibm.com: > mark.reinhold at oracle.com wrote on 03/08/2017 02:07:39 PM: >> ... >> >> I don't think it's unreasonable for an OSGi/JPMS interoperation solution >> to have to do these things, especially when the alternative is to commit >> the entire Java SE Platform to otherwise-unnecessary complexity for the >> long term. > > If it was only for this limited case in the support of OSGi framework > extensions then I tend to agree. I am under the impression that others > have more general usecases for these methods. I am only giving feedback > for my usecase to justify the need for the methods. There may be some > ways around this for OSGi. I have to investigate to see how bad or > difficult it makes life for us. I appreciate all your feedback on OSGi so far. It'd be helpful to know sooner, rather than later, whether you can work through these issues in your proof-of-concept implementation. - Mark From rfscholte at apache.org Wed Mar 8 22:48:13 2017 From: rfscholte at apache.org (Robert Scholte) Date: Wed, 08 Mar 2017 23:48:13 +0100 Subject: Draft JPMS Public Review specification In-Reply-To: <67eda315-7bbd-5249-c5fd-5838b04a33a1@redhat.com> References: <20170307165004.472677715F@eggemoggin.niobe.net> <67eda315-7bbd-5249-c5fd-5838b04a33a1@redhat.com> Message-ID: I agree with David. There are still quite some topics marked as "Proposal posted or discussion active" of which I don't know what the final resolution would be. And for a few I know they will have impact on other topics as well. I will go through the list and return with a concrete list of remarks. Robert On Wed, 08 Mar 2017 22:05:01 +0100, David M. Lloyd wrote: > Mark, > > Right now a Public Review definitely does not seem appropriate given > that at least some of the EG still sees major problems with the draft, > whereas it sends a message that there is consensus when there is not. > Given the fact that there still are significant issues outstanding and > disagreement within the EG, it seems likely to me that a at least a > second Public Review would need to be submitted anyway, one way or > another. In this light, I think it makes sense to wait at least until > there's some semblance of consensus among the EG before posting the > public review. > > On 03/07/2017 10:50 AM, mark.reinhold at oracle.com wrote: >> At this point in time the draft specification appears to achieve the >> goal >> of this JSR. We have a small and shrinking number of open issues on the >> list [1], and I expect the resolution of those issues to require at most >> modest adjustments to the final draft. I therefore intend to submit the >> following draft to the JCP PMO to be posted for Public Review: >> >> http://cr.openjdk.java.net/~mr/jigsaw/spec/ >> >> Please let me know by 17:00 UTC next Tuesday, 14 March if you think any >> changes are required, or if you need more time for review. >> >> - Mark >> >> >> [1] http://openjdk.java.net/projects/jigsaw/spec/issues >> From mark.reinhold at oracle.com Wed Mar 8 22:48:18 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 08 Mar 2017 14:48:18 -0800 Subject: Concerns mapping existing dynamic module systems to a 1-1 Module->Layer In-Reply-To: References: <20170308200839.1CC9E815DD@eggemoggin.niobe.net> Message-ID: <20170308144818.107997596@eggemoggin.niobe.net> 2017/3/8 13:33:09 -0800, tjwatson at us.ibm.com: > 2017/3/8 12:08:39 -0800, mark.reinhold at oracle.com: >> 2017/3/8 6:09:11 -0800, tjwatson at us.ibm.com: >>> ... >>> >>> Most of the additional methods David Lloyd is suggesting is to get the >>> functionality needed to do the complete mapping of existing module systems >>> into JPMS. Essentially I think we are trying to work with JPMS in order >>> to solve the following requirement [1]: >>> >>> Interoperation -- It must be possible for another module system, such as >>> OSGi, to locate Java modules and resolve them using its own resolver, >>> except possibly for core system modules. >> >> To clarify, this requirement does not imply bidirectional interoperation >> with dynamic module systems, as we've been discussing here. All that >> this requirement means is that another module system must be able to read >> JPMS module metadata in order to resolve JPMS modules on its own terms. >> In the case of OSGi, e.g., a suitably-enhanced framework implementation >> could use the `java.lang.module.ModuleFinder` API to locate a module's >> definition, and then use the `ModuleReader` API to read its metadata and >> content to synthesize a corresponding OSGi bundle that's loaded by an >> otherwise-ordinary OSGi class loader. > > I can confirm that this is possible, I have implemented something along > these lines. But I imagine there are going to be JPMS modules out there > that will not function correctly when their own classes start returning an > unnamed module from Class.getModule(). In my opinion, for a JPMS module > to be loaded as a bundle in OSGi the classes for that bundle must be > associated with a named module. There may, eventually, be a few such JPMS modules out there, but I can't believe that they'll be all that common. Unless the code in a JAR file is going to do deep introspection on its run-time module then it can remain blissfully ignorant of whether that module is named or unnamed. This is exactly why modular JAR files can be used on both the module path and the class path, and even on the class path of pre-9 releases with a little bit of care. >>> ... >>> >>> As I stated already, OSGi really only needs the addPackage and >>> addExportsToAll methods to be complete. But this still requires double >>> resolution work to get us back to the functionality we were at with Java 8 >>> and that is only if the suggested methods make it into java 9. >> >> No changes to OSGi implementations, nor to JPMS, should be required in >> order for OSGi implementations to achieve the level of functionality that >> they have on Java 8. An OSGi implementation should run just fine on top >> of Java 9, without interacting directly with JPMS in any way. (If it >> doesn't then there's a bug somewhere, in which case please report it and >> we'll fix it.) What we've been discussing here goes well beyond the >> functionality available on Java 8. > > Understood. Let me clarify. With an enhanced OSGi framework that loads > OSGi bundles as JPMS modules/layers we cannot achieve the same level of > OSGi functionality as would be possible when running in the compatibility > unnamed module space (or on top of Java 8). Okay, got it. - Mark From neil.bartlett at paremus.com Thu Mar 9 01:44:24 2017 From: neil.bartlett at paremus.com (Neil Bartlett (Paremus)) Date: Thu, 9 Mar 2017 01:44:24 +0000 Subject: Draft JPMS Public Review specification In-Reply-To: References: <20170307165004.472677715F@eggemoggin.niobe.net> <67eda315-7bbd-5249-c5fd-5838b04a33a1@redhat.com> Message-ID: <4602D010-D440-45DF-B60A-6350E63CF957@paremus.com> I also agree with David. There is not sufficient consensus, including on those issues that have been prematurely marked as ?Resolved?, for example #LayerPrimitives. Neil > On 8 Mar 2017, at 22:48, Robert Scholte wrote: > > I agree with David. There are still quite some topics marked as "Proposal posted or discussion active" of which I don't know what the final resolution would be. And for a few I know they will have impact on other topics as well. > > I will go through the list and return with a concrete list of remarks. > > Robert > > On Wed, 08 Mar 2017 22:05:01 +0100, David M. Lloyd wrote: > >> Mark, >> >> Right now a Public Review definitely does not seem appropriate given that at least some of the EG still sees major problems with the draft, whereas it sends a message that there is consensus when there is not. Given the fact that there still are significant issues outstanding and disagreement within the EG, it seems likely to me that a at least a second Public Review would need to be submitted anyway, one way or another. In this light, I think it makes sense to wait at least until there's some semblance of consensus among the EG before posting the public review. >> >> On 03/07/2017 10:50 AM, mark.reinhold at oracle.com wrote: >>> At this point in time the draft specification appears to achieve the goal >>> of this JSR. We have a small and shrinking number of open issues on the >>> list [1], and I expect the resolution of those issues to require at most >>> modest adjustments to the final draft. I therefore intend to submit the >>> following draft to the JCP PMO to be posted for Public Review: >>> >>> http://cr.openjdk.java.net/~mr/jigsaw/spec/ >>> >>> Please let me know by 17:00 UTC next Tuesday, 14 March if you think any >>> changes are required, or if you need more time for review. >>> >>> - Mark >>> >>> >>> [1] http://openjdk.java.net/projects/jigsaw/spec/issues >>> From tjwatson at us.ibm.com Thu Mar 9 13:47:24 2017 From: tjwatson at us.ibm.com (Thomas Watson) Date: Thu, 9 Mar 2017 07:47:24 -0600 Subject: Concerns mapping existing dynamic module systems to a 1-1 Module->Layer In-Reply-To: <20170308144818.107997596@eggemoggin.niobe.net> References: <20170308200839.1CC9E815DD@eggemoggin.niobe.net> <20170308144818.107997596@eggemoggin.niobe.net> Message-ID: > From: mark.reinhold at oracle.com > >> To clarify, this requirement does not imply bidirectional interoperation > >> with dynamic module systems, as we've been discussing here. All that > >> this requirement means is that another module system must be able to read > >> JPMS module metadata in order to resolve JPMS modules on its own terms. > >> In the case of OSGi, e.g., a suitably-enhanced framework implementation > >> could use the `java.lang.module.ModuleFinder` API to locate a module's > >> definition, and then use the `ModuleReader` API to read its metadata and > >> content to synthesize a corresponding OSGi bundle that's loaded by an > >> otherwise-ordinary OSGi class loader. > > > > I can confirm that this is possible, I have implemented something along > > these lines. But I imagine there are going to be JPMS modules out there > > that will not function correctly when their own classes start returning an > > unnamed module from Class.getModule(). In my opinion, for a JPMS module > > to be loaded as a bundle in OSGi the classes for that bundle must be > > associated with a named module. > > There may, eventually, be a few such JPMS modules out there, but I can't > believe that they'll be all that common. Unless the code in a JAR file > is going to do deep introspection on its run-time module then it can > remain blissfully ignorant of whether that module is named or unnamed. > This is exactly why modular JAR files can be used on both the module > path and the class path, and even on the class path of pre-9 releases > with a little bit of care. > > > - Mark > My other concern is about perception. It will not look good if an existing module system specification cannot take advantage of the new Java language features that are targeting modularity front and center. I think it is in the best interests of our communities to make this possible. Tom From mark.reinhold at oracle.com Thu Mar 9 16:54:53 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 09 Mar 2017 08:54:53 -0800 Subject: Draft JPMS Public Review specification In-Reply-To: References: <20170307165004.472677715F@eggemoggin.niobe.net> <67eda315-7bbd-5249-c5fd-5838b04a33a1@redhat.com> Message-ID: <20170309085453.594712597@eggemoggin.niobe.net> 2017/3/8 14:48:13 -0800, Robert Scholte : > I agree with David. There are still quite some topics marked as "Proposal > posted or discussion active" of which I don't know what the final > resolution would be. And for a few I know they will have impact on other > topics as well. The JCP does not require that every open issue be resolved prior to Public Review, only that issues be documented and tracked. As I said, I expect the resolution of the remaining open issues to require at most modest adjustments to the final draft, but I'm open to arguments to the contrary. > I will go through the list and return with a concrete list of remarks. I look forward to your feedback. - Mark From mark.reinhold at oracle.com Thu Mar 9 17:33:55 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 09 Mar 2017 09:33:55 -0800 Subject: Concerns mapping existing dynamic module systems to a 1-1 Module->Layer In-Reply-To: References: <20170308200839.1CC9E815DD@eggemoggin.niobe.net> <20170308144818.107997596@eggemoggin.niobe.net> Message-ID: <20170309093355.177166587@eggemoggin.niobe.net> 2017/3/9 5:47:24 -0800, tjwatson at us.ibm.com: > ... > > My other concern is about perception. It will not look good if an > existing module system specification cannot take advantage of the new Java > language features that are targeting modularity front and center. I think > it is in the best interests of our communities to make this possible. I think it is in the best interest of the entire Java ecosystem to deliver on our commitments rather than argue, ad infinitum, about features which are not required in order to meet those commitments and which could impose undesirable constraints on the long-term evolution of the Java SE Platform. - Mark From david.lloyd at redhat.com Thu Mar 9 18:26:40 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 9 Mar 2017 12:26:40 -0600 Subject: Concerns mapping existing dynamic module systems to a 1-1 Module->Layer In-Reply-To: <20170309093355.177166587@eggemoggin.niobe.net> References: <20170308200839.1CC9E815DD@eggemoggin.niobe.net> <20170308144818.107997596@eggemoggin.niobe.net> <20170309093355.177166587@eggemoggin.niobe.net> Message-ID: On 03/09/2017 11:33 AM, mark.reinhold at oracle.com wrote: > 2017/3/9 5:47:24 -0800, tjwatson at us.ibm.com: >> ... >> >> My other concern is about perception. It will not look good if an >> existing module system specification cannot take advantage of the new Java >> language features that are targeting modularity front and center. I think >> it is in the best interests of our communities to make this possible. > > I think it is in the best interest of the entire Java ecosystem to > deliver on our commitments rather than argue, ad infinitum, about > features which are not required in order to meet those commitments > and which could impose undesirable constraints on the long-term > evolution of the Java SE Platform. There are plenty of behaviors in the implementation that are not in the specification, and there are things missing from the specification, which is why we have raised issues. We also have a commitment to attempt to achieve some form of consensus, and to represent the needs of the community. And the constraints still have not been quantified. -- - DML From mark.reinhold at oracle.com Thu Mar 9 19:06:13 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 9 Mar 2017 11:06:13 -0800 (PST) Subject: Concerns mapping existing dynamic module systems to a 1-1 Module->Layer In-Reply-To: <740acf92-552c-6760-9bee-391f712e3da5@redhat.com> References: <20170308200839.1CC9E815DD@eggemoggin.niobe.net> <740acf92-552c-6760-9bee-391f712e3da5@redhat.com> Message-ID: <20170309190613.8D7CC81787@eggemoggin.niobe.net> 2017/3/8 14:17:03 -0800, david.lloyd at redhat.com: > ... > > This additional complexity that these extra methods is supposed to > introduce has never been quantified. The only concrete measure we have > right now is the dozen-odd lines of code that the proposed patch would > add. If we can't concretely quantify this complexity, and it is the > justification for denying this request, then we cannot reach consensus > because there's no way to have a rational discussion about it. You are asking for the impossible. I could quantify this complexity today, but that would tell us very little about the future. I cannot peer into the future, inspect all plausible evolutionary timelines of the Java SE Platform, and come back to you with any kind of measurement of the potential long-term impact of these additional methods. What I can do is apply my judgement and experience, together with that of engineers who have vastly more experience than I -- or you -- in the implementation of high-performance, production-quality JVMs. You are free to disagree with the conclusion that I have reached by this method, but that does not mean that rational discussion is not possible. - Mark From david.lloyd at redhat.com Thu Mar 9 19:16:08 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 9 Mar 2017 13:16:08 -0600 Subject: Concerns mapping existing dynamic module systems to a 1-1 Module->Layer In-Reply-To: <20170309190613.8D7CC81787@eggemoggin.niobe.net> References: <20170308200839.1CC9E815DD@eggemoggin.niobe.net> <740acf92-552c-6760-9bee-391f712e3da5@redhat.com> <20170309190613.8D7CC81787@eggemoggin.niobe.net> Message-ID: <54f907a4-24d2-670e-554c-70feecccbe3f@redhat.com> On 03/09/2017 01:06 PM, mark.reinhold at oracle.com wrote: > 2017/3/8 14:17:03 -0800, david.lloyd at redhat.com: >> ... >> >> This additional complexity that these extra methods is supposed to >> introduce has never been quantified. The only concrete measure we have >> right now is the dozen-odd lines of code that the proposed patch would >> add. If we can't concretely quantify this complexity, and it is the >> justification for denying this request, then we cannot reach consensus >> because there's no way to have a rational discussion about it. > > You are asking for the impossible. I could quantify this complexity > today, but that would tell us very little about the future. I cannot > peer into the future, inspect all plausible evolutionary timelines of > the Java SE Platform, and come back to you with any kind of measurement > of the potential long-term impact of these additional methods. > > What I can do is apply my judgement and experience, together with that > of engineers who have vastly more experience than I -- or you -- in the > implementation of high-performance, production-quality JVMs. > > You are free to disagree with the conclusion that I have reached by this > method, but that does not mean that rational discussion is not possible. Then there must be *some* concrete basis from which discussion can commence. Can it be shown that adding these methods can even *theoretically* cause problematic deoptimizations that are not already caused just by nature of having dynamic Layers to begin with? Can it be shown that it's even remotely possible to perform AOT or any other kind of specialized optimizations on dynamically-generated Layers... even *without* the patch? Are there even any hypotheses as to what problems could occur once these methods are in play, bearing in mind that neither the JDK nor the application module path have exposed Controllers and thus cannot be said to be constrained by them? Is there some other categorical consideration beyond AOT and JIT optimizations that your experience or those around you is suggesting makes this a specific problem? -- - DML From tjwatson at us.ibm.com Fri Mar 10 17:37:41 2017 From: tjwatson at us.ibm.com (Thomas Watson) Date: Fri, 10 Mar 2017 11:37:41 -0600 Subject: Proposal: #NonHierarchicalLayers (+ #LayerPrimitives) In-Reply-To: <20170308143145.414784649@eggemoggin.niobe.net> References: <20161207234117.1E5F1213F1@eggemoggin.niobe.net> <9b66aa52-877a-e4b2-1458-9ff47dfb690b@redhat.com> <20170126223431.B417255DD9@eggemoggin.niobe.net> <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> <20170215195108.C881964E6F@eggemoggin.niobe.net> <20170306020244.3B3D5760B4@eggemoggin.niobe.net> <20170306203549.15ADF76921@eggemoggin.niobe.net> <20170306234525.C37CD76960@eggemoggin.niobe.net> <20170308143145.414784649@eggemoggin.niobe.net> Message-ID: > From: mark.reinhold at oracle.com > To: Thomas Watson/Austin/IBM at IBMUS > Cc: jpms-spec-experts at openjdk.java.net > Date: 03/08/2017 04:31 PM > Subject: Re: Proposal: #NonHierarchicalLayers (+ #LayerPrimitives) > > 2017/3/8 13:22:32 -0800, tjwatson at us.ibm.com: > > mark.reinhold at oracle.com wrote on 03/08/2017 02:07:39 PM: > >> ... > >> > >> I don't think it's unreasonable for an OSGi/JPMS interoperation solution > >> to have to do these things, especially when the alternative is to commit > >> the entire Java SE Platform to otherwise-unnecessary complexity for the > >> long term. > > > > If it was only for this limited case in the support of OSGi framework > > extensions then I tend to agree. I am under the impression that others > > have more general usecases for these methods. I am only giving feedback > > for my usecase to justify the need for the methods. There may be some > > ways around this for OSGi. I have to investigate to see how bad or > > difficult it makes life for us. > > I appreciate all your feedback on OSGi so far. It'd be helpful to know > sooner, rather than later, whether you can work through these issues in > your proof-of-concept implementation. > > - Mark > The approach of taking each dynamic system.bundle fragment that adds packages and placing them in their own module can be made to work. It is ugly because it doesn't fit well within the model used by OSGi to resolve fragments. Outside of OSGi and JPMS resolution there will need to be a separate sort of resolution that adds the necessary read edges to the system.bundle fragment module instance when other modules in the system are resolved to the packages provided by the fragment. In OSGi the read edge naturally goes to the host the fragment is attached. This is necessary because a fragment in OSGi can be attached to multiple host bundles at the same time. So the packages get merged into the host and provided by the host as separate capabilities from the host's class loader. One difference for system.bundle fragments is that they will ever only have one host so we can do this side wiring for the reads to the single module representing the fragment without having to worry which class loader to use for the fragment module. When I have adequate time I could see a way to expand this to the general case for dynamic attachment of fragments to normal bundles also. The fragments would live in there own Layer so we could generate a new module/layer to represent each instance of the fragment that is attached to a separate host. If that was made to work then I would always model fragments this way in JPMS even when they are statically resolved with their host so that we have a consistent behavior. There is my feedback. It is a bit unpleasant from my perspective, but possible. It would would be less unpleasant if we had addPackage and addExportToAll which would fit better into our module system. It does seem others do have this need for addPackage outside of existing module systems [1]. Tom [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-March/011603.html From david.lloyd at redhat.com Fri Mar 10 18:53:16 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Fri, 10 Mar 2017 12:53:16 -0600 Subject: Concerns mapping existing dynamic module systems to a 1-1 Module->Layer In-Reply-To: <54f907a4-24d2-670e-554c-70feecccbe3f@redhat.com> References: <20170308200839.1CC9E815DD@eggemoggin.niobe.net> <740acf92-552c-6760-9bee-391f712e3da5@redhat.com> <20170309190613.8D7CC81787@eggemoggin.niobe.net> <54f907a4-24d2-670e-554c-70feecccbe3f@redhat.com> Message-ID: <087f0b2e-c0c9-c277-df8c-95505241f61b@redhat.com> On 03/09/2017 01:16 PM, David M. Lloyd wrote: > On 03/09/2017 01:06 PM, mark.reinhold at oracle.com wrote: >> 2017/3/8 14:17:03 -0800, david.lloyd at redhat.com: >>> ... >>> >>> This additional complexity that these extra methods is supposed to >>> introduce has never been quantified. The only concrete measure we have >>> right now is the dozen-odd lines of code that the proposed patch would >>> add. If we can't concretely quantify this complexity, and it is the >>> justification for denying this request, then we cannot reach consensus >>> because there's no way to have a rational discussion about it. >> >> You are asking for the impossible. I could quantify this complexity >> today, but that would tell us very little about the future. I cannot >> peer into the future, inspect all plausible evolutionary timelines of >> the Java SE Platform, and come back to you with any kind of measurement >> of the potential long-term impact of these additional methods. >> >> What I can do is apply my judgement and experience, together with that >> of engineers who have vastly more experience than I -- or you -- in the >> implementation of high-performance, production-quality JVMs. >> >> You are free to disagree with the conclusion that I have reached by this >> method, but that does not mean that rational discussion is not possible. > > Then there must be *some* concrete basis from which discussion can > commence. Can it be shown that adding these methods can even > *theoretically* cause problematic deoptimizations that are not already > caused just by nature of having dynamic Layers to begin with? > > Can it be shown that it's even remotely possible to perform AOT or any > other kind of specialized optimizations on dynamically-generated > Layers... even *without* the patch? Are there even any hypotheses as > to what problems could occur once these methods are in play, bearing in > mind that neither the JDK nor the application module path have exposed > Controllers and thus cannot be said to be constrained by them? > > Is there some other categorical consideration beyond AOT and JIT > optimizations that your experience or those around you is suggesting > makes this a specific problem? Put another way. It's provable that adding layer controller methods cannot impact either boot layer modules or application modules, whether they are in the same or differing layers. This derives from the simple fact that the controllers are not available for these layers. It's already unlikely that dynamic layers can work with AOT compilation, simply because the AOT compiler cannot predict how dynamic layers are going to interact with lower layers. Solving this would require magic, as far as I can tell. So AOT and optimization cannot be the reason to reject this. No JVM people I've interacted with have provided any clues as to why this is even potentially harmful to the future of the JVM. You've argued that this is not a core requirement. However I've argued that it *is* a core requirement. Other experts have also argued that it is a core requirement. Members of the community are arguing that it's a core requirement. Basically everyone who has discussed this publicly has agreed that this is needed and useful except for you. We've made it very clear that we consider this a necessity to satisfy dynamic configuration. You've argued that this adds unknown unknowns in the future. But in light of the many other changes which are at least equally as impactful, there appears to be nothing that makes this particular change that crosses any concrete line into blanket unacceptability, or at least nothing that you've shared thus far. Obviously you cannot look into the future. But that doesn't lighten the burden on you to explain why you oppose this change. I understand that this is not something you find useful. I understand that this issue (and others as well) go against your personal feelings about how software ought to work. But there is a lot of valid knowledge and experience that does not necessarily align with yours. And I don't think I'm out of line to ask to stow the hand-waving/appeals to authority and at least discuss these ideas on their *actual* merits and drawbacks. Every time you unilaterally decide that an issue has no need for further discussion, when clearly more discussion is warranted, the process gets subverted a little bit more. Every time you've unilaterally rejected an issue, we've come back with a new way to attempt to solve the same problems that we see, which we've been talking about for many years now. We've given ground on real mutability in favor of the second-class approach of using Layers everywhere for everything. Our best case is far from optimal but we (and Thomas and IBM, I think) have been willing to explore other proposals, but you've relentlessly rejected every workable idea. The JCP is not a rubber stamp, nor is it a free community design review for already-completed software projects. It's a collaborative process that is meant to take *advantage* of the broad experience of the community to arrive at better, more powerful software than one single group alone could produce. We're all trying to make this specification *better* and suitable for real-world use cases, and I wish I could find a way to make you see that. -- - DML From rfscholte at apache.org Sun Mar 12 12:40:00 2017 From: rfscholte at apache.org (Robert Scholte) Date: Sun, 12 Mar 2017 13:40:00 +0100 Subject: Draft JPMS Public Review specification In-Reply-To: <20170307165004.472677715F@eggemoggin.niobe.net> References: <20170307165004.472677715F@eggemoggin.niobe.net> Message-ID: Hi Mark, As promised a small set of concrete examples: In the JAR File Specification[1] I'm missing the option to add the `Module-Name` attribute to the new JAR-file manifest.[2] It is the choice of the jigsaw team that every jar used in a modular project must be specified as a requirement in the module declaration. For me a subset of requirements would already be fine, while slowly increasing the number of requirements with every release of the project to make it more and more stable. Anyhow, I can live with this decision, but is comes at a price. Maven won't allow references to automodules in case of libraries for reasons discussed in other threads. Our definition of a library is that is has at least 1 export-statement. Referring to named modules, either by the name in the module-info file or the `Module-Name` attribute, is fine. For applications (without any export-statement) referring to automodules can be done with minimal danger. For ease of transition, the `Module-Name` is simply a requirement, otherwise libraries can only start adding the module declaration once every required module is named. Until that moment this project could not claim a module name, nor can it be referred to by other libraries. I have the feeling that VersionsInModuleNames[3], or actually modulenames ending with numbers should be allowed. In case of automodules, there are cases where this would help, but also a same amount of modules where this won't work. The resolution for automodules should not be the (only) reason to decline numbers at the end. David suggested to reopen #CyclicDependences and based on his story I agree that this one should be reconsidered. The problem can arise at runtime when a combination of versions of different modules come together and create a cycle, whereas at compile time (with a different set of modules) everything looks fine. I don't think that it is the responsibility of the JVM to prevent cycles, there are other tools which can detect cycles and warn about it and where you can suppress it, because it is intended. Based on the Java Platform Module System: Requirements[4] I don't see any goal specification regarding cyclic dependencies. It looks more like a bonus you achieved with the JDK/JRE itself to make it possible to link small assemblies, but the outer world sometimes needs this trick. best, Robert [1] http://cr.openjdk.java.net/~mr/jigsaw/spec/jar.html#Modular [2] http://openjdk.java.net/projects/jigsaw/spec/issues/#ModuleNameInManifest [3] http://openjdk.java.net/projects/jigsaw/spec/issues/#VersionsInModuleNames [4] http://openjdk.java.net/projects/jigsaw/spec/reqs/ On Tue, 07 Mar 2017 17:50:04 +0100, wrote: > At this point in time the draft specification appears to achieve the goal > of this JSR. We have a small and shrinking number of open issues on the > list [1], and I expect the resolution of those issues to require at most > modest adjustments to the final draft. I therefore intend to submit the > following draft to the JCP PMO to be posted for Public Review: > > http://cr.openjdk.java.net/~mr/jigsaw/spec/ > > Please let me know by 17:00 UTC next Tuesday, 14 March if you think any > changes are required, or if you need more time for review. > > - Mark > > > [1] http://openjdk.java.net/projects/jigsaw/spec/issues From Tim_Ellison at uk.ibm.com Sun Mar 12 22:54:42 2017 From: Tim_Ellison at uk.ibm.com (Tim Ellison) Date: Sun, 12 Mar 2017 22:54:42 +0000 Subject: Draft JPMS Public Review specification In-Reply-To: <20170307165004.472677715F@eggemoggin.niobe.net> References: <20170307165004.472677715F@eggemoggin.niobe.net> Message-ID: It is good to see that the number of open issues being addressed is shrinking; and while a number of those issues can rightfully be deferred for consideration in future revisions, there is still ongoing healthy debate focused on important technical details of the dynamic layers and associated layer primitives. Although I appreciate that not all issues need be addressed by the time of Public Review, I think it is somewhat premature to call for a review until the EG has let the current discussion run its course and come to a decision about how to proceed, and conclude that the goals of the JSR are being met. The JSR materials and discussion have been made widely available, so the purpose of the submission would be a signal that the EG is content, or at least accepting, of the current situation. I feel we are close to that point. Regards, Tim "jpms-spec-experts" wrote on 07/03/2017 16:50:04: > At this point in time the draft specification appears to achieve the goal > of this JSR. We have a small and shrinking number of open issues on the > list [1], and I expect the resolution of those issues to require at most > modest adjustments to the final draft. I therefore intend to submit the > following draft to the JCP PMO to be posted for Public Review: > > http://cr.openjdk.java.net/~mr/jigsaw/spec/ > > Please let me know by 17:00 UTC next Tuesday, 14 March if you think any > changes are required, or if you need more time for review. > > - Mark > > > [1] http://openjdk.java.net/projects/jigsaw/spec/issues > Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU From Tim_Ellison at uk.ibm.com Sun Mar 12 22:54:41 2017 From: Tim_Ellison at uk.ibm.com (Tim Ellison) Date: Sun, 12 Mar 2017 22:54:41 +0000 Subject: Draft JPMS Public Review specification In-Reply-To: References: <20170307165004.472677715F@eggemoggin.niobe.net> Message-ID: "jpms-spec-experts" wrote on 12/03/2017 12:40:00: > Hi Mark, > > As promised a small set of concrete examples: > > In the JAR File Specification[1] I'm missing the option to add the > `Module-Name` attribute to the new JAR-file manifest.[2] > > It is the choice of the jigsaw team that every jar used in a modular > project must be specified as a requirement in the module declaration. For > me a subset of requirements would already be fine, while slowly increasing > the number of requirements with every release of the project to make it > more and more stable. Anyhow, I can live with this decision, but is comes > at a price. > > Maven won't allow references to automodules in case of libraries for > reasons discussed in other threads. > Our definition of a library is that is has at least 1 export-statement. > Referring to named modules, either by the name in the module-info file or > the `Module-Name` attribute, is fine. > For applications (without any export-statement) referring to automodules > can be done with minimal danger. > > For ease of transition, the `Module-Name` is simply a requirement, > otherwise libraries can only start adding the module declaration once > every required module is named. Until that moment this project could not > claim a module name, nor can it be referred to by other libraries. > > I have the feeling that VersionsInModuleNames[3], or actually modulenames > ending with numbers should be allowed. In case of automodules, there are > cases where this would help, but also a same amount of modules where this > won't work. The resolution for automodules should not be the (only) reason > to decline numbers at the end. I agree that we should drop the proposal addressing #VersionsInModuleNames, that module names must end with a Java letter. Based on practical experience there are a number of libraries that have attempted to use a number legitimately (i.e. not as a version identifier) and been caught out by this. There are any number of bad practices that could be accomplished within the current design, and attempting to spec them out of existence is quite futile. This proposal introduces friction to adoption for a very limited gain. Regards, Tim > David suggested to reopen #CyclicDependences and based on his story I > agree that this one should be reconsidered. The problem can arise at > runtime when a combination of versions of different modules come together > and create a cycle, whereas at compile time (with a different set of > modules) everything looks fine. I don't think that it is the > responsibility of the JVM to prevent cycles, there are other tools which > can detect cycles and warn about it and where you can suppress it, because > it is intended. > > Based on the Java Platform Module System: Requirements[4] I don't see any > goal specification regarding cyclic dependencies. It looks more like a > bonus you achieved with the JDK/JRE itself to make it possible to link > small assemblies, but the outer world sometimes needs this trick. > > best, > Robert > > [1] http://cr.openjdk.java.net/~mr/jigsaw/spec/jar.html#Modular > [2] > http://openjdk.java.net/projects/jigsaw/spec/issues/#ModuleNameInManifest > [3] > http://openjdk.java.net/projects/jigsaw/spec/issues/#VersionsInModuleNames > [4] http://openjdk.java.net/projects/jigsaw/spec/reqs/ > > On Tue, 07 Mar 2017 17:50:04 +0100, wrote: > > > At this point in time the draft specification appears to achieve the goal > > of this JSR. We have a small and shrinking number of open issues on the > > list [1], and I expect the resolution of those issues to require at most > > modest adjustments to the final draft. I therefore intend to submit the > > following draft to the JCP PMO to be posted for Public Review: > > > > http://cr.openjdk.java.net/~mr/jigsaw/spec/ > > > > Please let me know by 17:00 UTC next Tuesday, 14 March if you think any > > changes are required, or if you need more time for review. > > > > - Mark > > > > > > [1] http://openjdk.java.net/projects/jigsaw/spec/issues > Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU From mark.reinhold at oracle.com Tue Mar 14 15:58:10 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 14 Mar 2017 08:58:10 -0700 (PDT) Subject: Proposal: #NonHierarchicalLayers (+ #LayerPrimitives) In-Reply-To: References: <20161207234117.1E5F1213F1@eggemoggin.niobe.net> <9b66aa52-877a-e4b2-1458-9ff47dfb690b@redhat.com> <20170126223431.B417255DD9@eggemoggin.niobe.net> <7454db4e-8eeb-49a8-f7b6-d4821ddb9170@redhat.com> <20170215195108.C881964E6F@eggemoggin.niobe.net> <20170306020244.3B3D5760B4@eggemoggin.niobe.net> <20170306203549.15ADF76921@eggemoggin.niobe.net> <20170306234525.C37CD76960@eggemoggin.niobe.net> <20170308143145.414784649@eggemoggin.niobe.net> Message-ID: <20170314155810.2924A81EBF@eggemoggin.niobe.net> 2017/3/10 9:37:41 -0800, tjwatson at us.ibm.com: > 2017/3/8 14:31:45 -0800, mark.reinhold at oracle.com: >> 2017/3/8 13:22:32 -0800, tjwatson at us.ibm.com: >>> mark.reinhold at oracle.com wrote on 03/08/2017 02:07:39 PM: >>>> ... >>>> >>>> I don't think it's unreasonable for an OSGi/JPMS interoperation solution >>>> to have to do these things, especially when the alternative is to commit >>>> the entire Java SE Platform to otherwise-unnecessary complexity for the >>>> long term. >>> >>> If it was only for this limited case in the support of OSGi framework >>> extensions then I tend to agree. I am under the impression that others >>> have more general usecases for these methods. I am only giving feedback >>> for my usecase to justify the need for the methods. There may be some >>> ways around this for OSGi. I have to investigate to see how bad or >>> difficult it makes life for us. >> >> I appreciate all your feedback on OSGi so far. It'd be helpful to know >> sooner, rather than later, whether you can work through these issues in >> your proof-of-concept implementation. > > The approach of taking each dynamic system.bundle fragment that adds > packages and placing them in their own module can be made to work. It is > ugly because it doesn't fit well within the model used by OSGi to resolve > fragments. Outside of OSGi and JPMS resolution there will need to be a > separate sort of resolution that adds the necessary read edges to the > system.bundle fragment module instance when other modules in the system > are resolved to the packages provided by the fragment. In OSGi the read > edge naturally goes to the host the fragment is attached. This is > necessary because a fragment in OSGi can be attached to multiple host > bundles at the same time. So the packages get merged into the host and > provided by the host as separate capabilities from the host's class > loader. One difference for system.bundle fragments is that they will ever > only have one host so we can do this side wiring for the reads to the > single module representing the fragment without having to worry which > class loader to use for the fragment module. > > When I have adequate time I could see a way to expand this to the general > case for dynamic attachment of fragments to normal bundles also. The > fragments would live in there own Layer so we could generate a new > module/layer to represent each instance of the fragment that is attached > to a separate host. If that was made to work then I would always model > fragments this way in JPMS even when they are statically resolved with > their host so that we have a consistent behavior. > > There is my feedback. It is a bit unpleasant from my perspective, but > possible. Good -- thanks for the confirmation. > It would would be less unpleasant if we had addPackage and > addExportToAll which would fit better into our module system. It does seem > others do have this need for addPackage outside of existing module systems > [1]. [1] suggests that the `java.lang.instrument.Instrumentation` interface should allow packages to be added to an existing module at run time. That interface is intended only to support the concurrent instrumentation of existing code, which does not require that capability; it was never meant to support the style of fine-grained incremental development assumed in [1]. Comprehensive platform-level support for that might be desirable one day, but it's well beyond the scope of this JSR. - Mark > [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-March/011603.html From mark.reinhold at oracle.com Tue Mar 14 15:59:10 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 14 Mar 2017 08:59:10 -0700 (PDT) Subject: Draft JPMS Public Review specification In-Reply-To: References: <20170307165004.472677715F@eggemoggin.niobe.net> Message-ID: <20170314155910.2E7F781EC4@eggemoggin.niobe.net> 2017/3/12 5:40:00 -0700, rfscholte at apache.org: > As promised a small set of concrete examples: > > In the JAR File Specification[1] I'm missing the option to add the > `Module-Name` attribute to the new JAR-file manifest.[2] I haven't yet added that attribute to the JAR-file specification because #ModuleNameInManifest is tightly related to #AutomaticModuleNames [5], and while we've made progress on that issue I'm not yet sure that we've found the best solution. I'll have more to say on that topic shortly, but either way I suspect that whatever we conclude will result in most minor changes, so this issue needn't hold up the Public Review submission. > ... > > Maven won't allow references to automodules in case of libraries for > reasons discussed in other threads. > Our definition of a library is that is has at least 1 export-statement. > > Referring to named modules, either by the name in the module-info file or > the `Module-Name` attribute, is fine. > For applications (without any export-statement) referring to automodules > can be done with minimal danger. > > For ease of transition, the `Module-Name` is simply a requirement, > otherwise libraries can only start adding the module declaration once > every required module is named. Until that moment this project could not > claim a module name, nor can it be referred to by other libraries. Okay, I understand this to mean that you strongly want the `Module-Name` attribute. Noted for future discussion. > I have the feeling that VersionsInModuleNames[3], or actually modulenames > ending with numbers should be allowed. In case of automodules, there are > cases where this would help, but also a same amount of modules where this > won't work. The resolution for automodules should not be the (only) reason > to decline numbers at the end. I understand the objections to this constraint, but I think it's essential to provide some sort of mechanical advice to developers to guide them away from encoding version numbers in module names. Perhaps there's some other way to do that? If not, then aside from `fabric8` and `commons-lang3` would this constraint actually impact any other well-known projects? (Perhaps a scan of Central could guide us here.) > David suggested to reopen #CyclicDependences and based on his story I > agree that this one should be reconsidered. The problem can arise at > runtime when a combination of versions of different modules come together > and create a cycle, whereas at compile time (with a different set of > modules) everything looks fine. I don't think that it is the > responsibility of the JVM to prevent cycles, there are other tools which > can detect cycles and warn about it and where you can suppress it, because > it is intended. > > Based on the Java Platform Module System: Requirements[4] I don't see any > goal specification regarding cyclic dependencies. It looks more like a > bonus you achieved with the JDK/JRE itself to make it possible to link > small assemblies, but the outer world sometimes needs this trick. It's true that the agreed requirements don't say anything about whether or not cycles should be allowed. Disallowing them does, however, bring a number of significant advantages, as noted in the issue statement [6]. The designers of some other module systems have found it useful to allow cycles, but JPMS differs from those in a couple of essential ways that bear upon this issue: - JPMS is not highly dynamic, so if a cycle arises while configuring a layer then that fact will be reported early, when it's easier to diagnose, rather than at some (possibly much) later point in time. - JPMS aims to provide a clear migration path for existing components, but it assumes that components can and will be revised during that process. If a cyclic module graph arises then the maintainers of the relevant components should consider whether a service should be introduced in order to break the cycle. Taken together with the compelling arguments against cyclic module relationships made by at least one noted expert in the field [7], on balance I think it's best to disallow cycles in JPMS for now. If actual experience with JPMS as used directly by developers -- rather than as an embedding target for other module systems -- suggests that we should allow cycles then we could enable them in a future release. If we enable them now, however, then we can never take them back. - Mark > [1] http://cr.openjdk.java.net/~mr/jigsaw/spec/jar.html#Modular > [2] http://openjdk.java.net/projects/jigsaw/spec/issues/#ModuleNameInManifest > [3] http://openjdk.java.net/projects/jigsaw/spec/issues/#VersionsInModuleNames > [4] http://openjdk.java.net/projects/jigsaw/spec/reqs/ [5] http://openjdk.java.net/projects/jigsaw/spec/issues/#AutomaticModuleNames [6] http://openjdk.java.net/projects/jigsaw/spec/issues/#CyclicDependences [7] http://www.kirkk.com/modularity/2009/12/acyclic-relationships/ From mark.reinhold at oracle.com Tue Mar 14 16:05:29 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 14 Mar 2017 09:05:29 -0700 (PDT) Subject: Draft JPMS Public Review specification In-Reply-To: References: <20170307165004.472677715F@eggemoggin.niobe.net> Message-ID: <20170314160529.620AB81ECF@eggemoggin.niobe.net> 2017/3/12 15:54:41 -0700, tim_ellison at uk.ibm.com: > ... > > I agree that we should drop the proposal addressing #VersionsInModuleNames, > that module names must end with a Java letter. Based on practical experience > there are a number of libraries that have attempted to use a number > legitimately (i.e. not as a version identifier) and been caught out by this. Examples, please, other than `commons-lang3` and `fabric8`? > There are any number of bad practices that could be accomplished within the > current design, and attempting to spec them out of existence is quite futile. > This proposal introduces friction to adoption for a very limited gain. If only a couple of projects are affected by this constraint then perhaps the gain outweighs the friction. Otherwise, is there some other way to discourage developers from encoding version numbers in module names? - Mark From david.lloyd at redhat.com Tue Mar 14 16:34:23 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 14 Mar 2017 11:34:23 -0500 Subject: Draft JPMS Public Review specification In-Reply-To: <20170314160529.620AB81ECF@eggemoggin.niobe.net> References: <20170307165004.472677715F@eggemoggin.niobe.net> <20170314160529.620AB81ECF@eggemoggin.niobe.net> Message-ID: <905d7a89-e13c-73d6-fe7c-311f9bacfcfd@redhat.com> On 03/14/2017 11:05 AM, mark.reinhold at oracle.com wrote: > 2017/3/12 15:54:41 -0700, tim_ellison at uk.ibm.com: >> ... >> >> I agree that we should drop the proposal addressing #VersionsInModuleNames, >> that module names must end with a Java letter. Based on practical experience >> there are a number of libraries that have attempted to use a number >> legitimately (i.e. not as a version identifier) and been caught out by this. > > Examples, please, other than `commons-lang3` and `fabric8`? You may recall I posted a number of other examples in the email thread about that subject. But if your answer is always going to be "other than that, what else?" then I guess there's no more discussion possible there. >> There are any number of bad practices that could be accomplished within the >> current design, and attempting to spec them out of existence is quite futile. >> This proposal introduces friction to adoption for a very limited gain. > > If only a couple of projects are affected by this constraint then perhaps the > gain outweighs the friction. > > Otherwise, is there some other way to discourage developers from encoding > version numbers in module names? I think the point is that AFAICT nobody else agrees that encoding version numbers in module names is a bad practice on its own merits. Whatever it is you are trying to discourage can almost certainly be accomplished in other (substantially worse) ways. -- - DML From david.lloyd at redhat.com Tue Mar 14 18:53:43 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 14 Mar 2017 13:53:43 -0500 Subject: Draft JPMS Public Review specification In-Reply-To: <20170314155910.2E7F781EC4@eggemoggin.niobe.net> References: <20170307165004.472677715F@eggemoggin.niobe.net> <20170314155910.2E7F781EC4@eggemoggin.niobe.net> Message-ID: On 03/14/2017 10:59 AM, mark.reinhold at oracle.com wrote: > 2017/3/12 5:40:00 -0700, rfscholte at apache.org: >> David suggested to reopen #CyclicDependences and based on his story I >> agree that this one should be reconsidered. The problem can arise at >> runtime when a combination of versions of different modules come together >> and create a cycle, whereas at compile time (with a different set of >> modules) everything looks fine. I don't think that it is the >> responsibility of the JVM to prevent cycles, there are other tools which >> can detect cycles and warn about it and where you can suppress it, because >> it is intended. >> >> Based on the Java Platform Module System: Requirements[4] I don't see any >> goal specification regarding cyclic dependencies. It looks more like a >> bonus you achieved with the JDK/JRE itself to make it possible to link >> small assemblies, but the outer world sometimes needs this trick. > > It's true that the agreed requirements don't say anything about whether > or not cycles should be allowed. Disallowing them does, however, bring a > number of significant advantages, as noted in the issue statement [6]. > The designers of some other module systems have found it useful to allow > cycles, but JPMS differs from those in a couple of essential ways that > bear upon this issue: > > - JPMS is not highly dynamic, so if a cycle arises while configuring > a layer then that fact will be reported early, when it's easier to > diagnose, rather than at some (possibly much) later point in time. This is only true if you're building all your modules at one time, in the exact versions and arrangement in which they will run. Experience shows that Maven has become ubiquitous specifically because this is almost never true in practice (except for a few specifically monolithic and self-contained projects, like the JDK itself for example); at test/run time, environments are usually an accumulation of modules built at many different times and places. Therefore enforced prevention of cycles beyond compile-time is likely to *cause* unexpected problems, not avoid them, for reasons I've explained in detail previously. A user will not be thankful that we have held their hand if their project only fails in production after some third-party dependency has been updated and unexpectedly introduces a long cycle - something we've observed in the wild quite frequently. > - JPMS aims to provide a clear migration path for existing components, > but it assumes that components can and will be revised during that > process. If a cyclic module graph arises then the maintainers of > the relevant components should consider whether a service should > be introduced in order to break the cycle. It does not follow that users will want to (or even be able to) substantially refactor the modules in their deployment; editing a descriptor is one thing but a possibly major refactor is something completely different. > Taken together with the compelling arguments against cyclic module > relationships made by at least one noted expert in the field [7], on > balance I think it's best to disallow cycles in JPMS for now. If actual > experience with JPMS as used directly by developers -- rather than as > an embedding target for other module systems -- suggests that we should > allow cycles then we could enable them in a future release. If we enable > them now, however, then we can never take them back. Why not compromise by disallowing cycles at compilation time, but not enforcing the restriction at run time? This satisfies your desire to report early, and matches the behavior of current build tools, but also avoids the inevitable packaging issues in large-application environments, and follows the principle of least surprise for end users. Put another way, what would the late/run-time part of cycle enforcement gain anyone? -- - DML From mark.reinhold at oracle.com Tue Mar 14 23:57:23 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 14 Mar 2017 16:57:23 -0700 (PDT) Subject: Draft JPMS Public Review specification In-Reply-To: <20170307165004.472677715F@eggemoggin.niobe.net> Message-ID: <20170314235723.CF91782005@eggemoggin.niobe.net> 2017/3/7 8:50:04 -0800, mark.reinhold at oracle.com: > At this point in time the draft specification appears to achieve the goal > of this JSR. We have a small and shrinking number of open issues on the > list [1], and I expect the resolution of those issues to require at most > modest adjustments to the final draft. I therefore intend to submit the > following draft to the JCP PMO to be posted for Public Review: > > http://cr.openjdk.java.net/~mr/jigsaw/spec/ > > Please let me know by 17:00 UTC next Tuesday, 14 March if you think any > changes are required, or if you need more time for review. Thanks for the feedback on the draft; I have updated the issue list [1] with links to the recent discussion. No changes to the content of the specification are needed prior to submission, and it still appears to be the case that resolution of the remaining open issues will require at most modest adjustments to the final draft. We do not have consensus in this EG on moving forward to Public Review. That is only because some members continue to insist that we must address goals that were neither mentioned in the original JSR submission [2] nor recorded in the agreed requirements [3]. It is, however, in the best interest of the wider Java ecosystem to proceed so that we can deliver on our actual committed goal [4]. I have therefore submitted the specification for Public Review despite this lack of consensus. - Mark [1] http://openjdk.java.net/projects/jigsaw/spec/issues [2] https://www.jcp.org/en/jsr/detail?id=376 [3] http://openjdk.java.net/projects/jigsaw/spec/reqs/ [4] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2017-February/000585.html From Tim_Ellison at uk.ibm.com Wed Mar 15 13:26:03 2017 From: Tim_Ellison at uk.ibm.com (Tim Ellison) Date: Wed, 15 Mar 2017 13:26:03 +0000 Subject: Draft JPMS Public Review specification In-Reply-To: <905d7a89-e13c-73d6-fe7c-311f9bacfcfd@redhat.com> References: <20170307165004.472677715F@eggemoggin.niobe.net> <20170314160529.620AB81ECF@eggemoggin.niobe.net> <905d7a89-e13c-73d6-fe7c-311f9bacfcfd@redhat.com> Message-ID: "jpms-spec-experts" wrote on 14/03/2017 16:34:23: > On 03/14/2017 11:05 AM, mark.reinhold at oracle.com wrote: > > 2017/3/12 15:54:41 -0700, tim_ellison at uk.ibm.com: > >> ... > >> > >> I agree that we should drop the proposal addressing #VersionsInModuleNames, > >> that module names must end with a Java letter. Based on > practical experience > >> there are a number of libraries that have attempted to use a number > >> legitimately (i.e. not as a version identifier) and been caught > out by this. > > > > Examples, please, other than `commons-lang3` and `fabric8`? > > You may recall I posted a number of other examples in the email thread > about that subject. But if your answer is always going to be "other > than that, what else?" then I guess there's no more discussion possible > there. My examples are from teams in IBM who have tried adopting early builds, and hit this as a problem. While they can work around it by renaming their modules to something they find unnatural, I find it hard to justify this constraint. > >> There are any number of bad practices that could be accomplished within the > >> current design, and attempting to spec them out of existence is > quite futile. > >> This proposal introduces friction to adoption for a very limited gain. > > > > If only a couple of projects are affected by this constraint then > perhaps the > > gain outweighs the friction. > > > > Otherwise, is there some other way to discourage developers from encoding > > version numbers in module names? > > I think the point is that AFAICT nobody else agrees that encoding > version numbers in module names is a bad practice on its own merits. > Whatever it is you are trying to discourage can almost certainly be > accomplished in other (substantially worse) ways. Indeed. Regards, Tim Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU From neil.bartlett at paremus.com Wed Mar 15 19:04:07 2017 From: neil.bartlett at paremus.com (Neil Bartlett (Paremus)) Date: Wed, 15 Mar 2017 19:04:07 +0000 Subject: #VersionsInModuleNames [was: Re: Draft JPMS Public Review specification] In-Reply-To: References: <20170307165004.472677715F@eggemoggin.niobe.net> <20170314160529.620AB81ECF@eggemoggin.niobe.net> <905d7a89-e13c-73d6-fe7c-311f9bacfcfd@redhat.com> Message-ID: <3ADFED90-9C09-4F05-92C0-6C2BC8BD3D1F@paremus.com> I believe the restriction on #VersionsInModuleNames should not be implemented. In OSGi it is a common practice ? though by no means universal ? for the bundle identity to be derived from the name of the ?top level? package it contains. I expect this will be a common practice under JPMS as well. Therefore the set of allowed module names should be the same as the set of allowed package names. On the other hand if it is logical to disallow versions in module names then they should likewise be disallowed in package names for all the same reasons. Backwards compatibility would require this change to be limited to packages of named modules. In my experience, when you attempt to take a ?best practice? and mandate it universally by technical means, you fail to guarantee good behaviour and you frustrate people who must deal with genuine edge cases. Module authors will still have plenty of opportunities to shoot themselves in the foot, whether or not this particular restriction is enforced. What is really achieved by forcing a module author to write ?guava_eighteen? versus ?guava_18?? except perhaps confusing non-English speakers? I would also call the experts? attention to an email thread on the jigsaw-dev mailing list (http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-March/011657.html ) and the arguments made by Stephen Colebourne. Neil > On 15 Mar 2017, at 13:26, Tim Ellison wrote: > > "jpms-spec-experts" > wrote on > 14/03/2017 16:34:23: >> On 03/14/2017 11:05 AM, mark.reinhold at oracle.com wrote: >>> 2017/3/12 15:54:41 -0700, tim_ellison at uk.ibm.com: >>>> ... >>>> >>>> I agree that we should drop the proposal addressing > #VersionsInModuleNames, >>>> that module names must end with a Java letter. Based on >> practical experience >>>> there are a number of libraries that have attempted to use a number >>>> legitimately (i.e. not as a version identifier) and been caught >> out by this. >>> >>> Examples, please, other than `commons-lang3` and `fabric8`? >> >> You may recall I posted a number of other examples in the email thread >> about that subject. But if your answer is always going to be "other >> than that, what else?" then I guess there's no more discussion possible >> there. > > My examples are from teams in IBM who have tried adopting early builds, > and > hit this as a problem. While they can work around it by renaming their > modules > to something they find unnatural, I find it hard to justify this > constraint. > >>>> There are any number of bad practices that could be accomplished > within the >>>> current design, and attempting to spec them out of existence is >> quite futile. >>>> This proposal introduces friction to adoption for a very limited > gain. >>> >>> If only a couple of projects are affected by this constraint then >> perhaps the >>> gain outweighs the friction. >>> >>> Otherwise, is there some other way to discourage developers from > encoding >>> version numbers in module names? >> >> I think the point is that AFAICT nobody else agrees that encoding >> version numbers in module names is a bad practice on its own merits. >> Whatever it is you are trying to discourage can almost certainly be >> accomplished in other (substantially worse) ways. > > Indeed. > > Regards, > Tim > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU From david.lloyd at redhat.com Wed Mar 15 20:00:37 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 15 Mar 2017 15:00:37 -0500 Subject: How to name modules, automatic and otherwise In-Reply-To: <1305027279.2009761.1487328314502.JavaMail.zimbra@u-pem.fr> References: <20170216164827.A309F65E1B@eggemoggin.niobe.net> <1305027279.2009761.1487328314502.JavaMail.zimbra@u-pem.fr> Message-ID: <7cf59089-8b10-c221-1f3b-f1c9b79c0367@redhat.com> R?mi, AFIACT this automatic mutation of name based on groupId could only be done by Maven or the Maven Central repository software. And it would essentially entail baking the generated name into the module... at which point we have a module-info.class file anyway (i.e. not automatic modules!). Did I miss something, or are you actually arguing to drop automatic modules from the spec and push it up to the tooling instead? On 02/17/2017 04:45 AM, Remi Forax wrote: > I agree that > - we should not bake the Maven way of specify identifiers into the language. > - we should not drop automatic modules, this is a practical way to do the migration if you are maintaining an application. > > On the addition of Module-Name attribute, i can understand that it's a simple way to reserve a name. I will agree to introduce that in the spec only if Robert and Brian finds that an automatic module with a Module-Name attribute is something that can be pushed on Maven Central. Otherwise, specifying a Module-Name is not very different of writing a module-info.java with everything exported (i see the fact that a regular module can not access to the classpath unlike an automatic module as a bonus). > > I fully disagree on the idea of using short names for module, the Java community is too big for this kind of shortcut, the reverse DNS prefix have serve us well and counting the number of keystrokes is not a good metric. > I've spend some time with Herve Boutemy last Tuesday night (and early Wednesday) to discuss about existing modules in Maven Central interropt. The Maven group id is something you have to ask before being able to publish on Maven Central, so it's more a Maven Central group id that a Maven group id. We should recommend that existing module on Maven Central should use a name that starts with the Maven Central group id. > By example, Google's Guava should be com.google.guava or com.google.guava.core (or anything else that starts with com.google.guava). > > This will solve the name collisions issues between artifacts of Maven Central at least. > > R?mi > > ----- Mail original ----- >> De: "mark reinhold" >> ?: jpms-spec-experts at openjdk.java.net >> Envoy?: Jeudi 16 F?vrier 2017 17:48:27 >> Objet: How to name modules, automatic and otherwise > >> This note is in reply to the concerns about automatic modules raised by >> Robert Scholte and Brian Fox [1], and by Stephen Colebourne and others >> [2]. I've collected my conclusions here rather than in separate messages >> because there are several distinct yet intertwined issues. >> >> Summary: >> >> - Module names should not include Maven group identifiers, because >> modules are more abstract than the artifacts that define them. >> >> - Module names should use the reverse-domain-name-prefix convention >> or, preferably, the project-name-prefix convention. >> >> - We should not abandon automatic modules, since they are a key tool >> for migration and adoption. >> >> - We can address the problems of automatic modules with two fairly >> minor technical enhancements. >> >> If any of these points strikes you as controversial, please read on! >> >> * * * >> >> Module names should not include Maven group identifiers, as Robert >> Scholte and Brian Fox suggest [1], even for modules declared explicitly >> in `module-info.java` files. Modules in JPMS are a construct of the Java >> programming language, implemented in both the compiler and the virtual >> machine. As such, they are more abstract entities than the artifacts >> that define them. This distinction is useful, both conceptually and >> practically, hence module names should remain more abstract. >> >> This distinction is useful conceptually because it makes it easier, as >> we read source code, to think clearly about the nature of a module. We >> can reason about a module's dependences, exports, services, and so forth >> without cluttering our minds with the details of group identifiers and >> version constraints. Today, e.g., we can write, and read: >> >> module foo.data { >> exports com.bar.foo.data; >> requires hibernate.core; >> requires hibernate.jcache; >> requires hibernate.validator; >> } >> >> If we were to extend the syntax of module names to include group >> identifiers, and encourage people to use them, then we'd be faced with >> something much more verbose: >> >> module com.bar:foo.data { >> exports com.bar.foo.data; >> requires org.hibernate:hibernate.core; >> requires org.hibernate:hibernate.jcache; >> requires org.hibernate:hibernate.validator; >> } >> >> Group identifiers make perfect sense in the context of a build system >> such as Maven, where they bring necessary structure to the names of the >> millions of artifacts available across different repositories. Such >> structure is superfluous and distracting in the context of a module >> system, where the number of relevant modules in any particular situation >> is more likely to be in the tens, or hundreds, or (rarely) thousands. >> All else being equal, simpler names are better. >> >> At a practical level, the distinction between modules and artifacts is >> useful because it leaves the entire problem of artifact selection to the >> build system. This allows us to switch from one artifact to another >> simply by editing a `pom.xml` file to adjust a version constraint or a >> group identifier; if module names included group identifiers then we'd >> also have to edit the `module-info.java` file. This flexibility can be >> helpful if, e.g., a project is forked and a new module with the same name >> and artifact identifier is published under a different group identifier. >> We long ago decided not to do version selection in the module system, >> which surprised some people but has worked out fairly well. We should >> treat group selection in the same manner. >> >> Another practical benefit of the module/artifact distinction is that it >> keeps the module system independent of any particular build system, so >> that build systems can continue to improve and evolve independently over >> time. Maven-style coordinates are the most popular way to name artifacts >> in repositories today, but that might not be true ten years from now. It >> would be unwise to adopt Maven's naming convention for module names just >> because it's popular now, and doubly so to bake Maven's group-identifier >> concept into the Java programming language. >> >> * * * >> >> If module names don't include group identifiers, then how should modules >> be named? What advice should we give to someone who's creating a new >> module from scratch, or modularizing an existing component by writing a >> `module-info.java` file for it? (Continue to set aside, for the moment, >> the problems of automatic modules.) >> >> In structuring any particular space of names we must balance (at least) >> three fundamental tensions: We want names that are long enough to be >> descriptive, short enough to be memorable, and unique enough to avoid >> needless conflicts. >> >> If you control all of the modules upon which your module depends, and >> all of the modules that depend upon it, then you can of course name your >> module whatever you want, and change its name at any time. If, however, >> you're going to publish your module for use by others -- whether just >> within your own organization or to a global repository such as Maven >> Central -- then you should take more care. There are two well-known >> ways to go about this. >> >> - Choose module names that start with the reversed form of an Internet >> domain name that you control, or are at least associated with. The >> Java Language Specification has long suggested this convention as a >> way to minimize conflicts amongst package names, and it has been >> widely though not universally adopted for that purpose. >> >> - Choose module names that start with the name of your project or >> product. Module (and package) names that start with reversed domain >> names are less likely to conflict but they're unnecessarily verbose, >> they start with the least-important information (e.g., `com`, `org`, >> or `net`), and they don't read well after exogenous changes such as >> open-source donations or corporate acquisitions (e.g., `com.sun.*`). >> >> The reversed domain-name approach was sensible in the early days of Java, >> before we had development tools sophisticated enough to help us deal with >> the occasional conflict. We have such tools now, so going forward the >> superior readability of short module and package names that start with >> project or product names is preferable to the onerous verbosity of those >> that start with reversed domain names. >> >> This advice will strike some readers as controversial. I respect those >> who will choose, for the sake of tradition or an abundance of caution, to >> use the reversed domain-name convention for module names and continue to >> use that convention for package names. I do know, however, of at least >> one major, well-known project whose developers intend to adopt the >> project-name-prefix convention for their module names. >> >> * * * >> >> If module names don't include group identifiers, then how should automatic >> modules be named? Or are automatic modules so troublesome that we should >> remove them from the design? >> >> To answer the second question first: It would be a tragic shame to drop >> automatic modules, since otherwise top-down migration is impossible if >> you're not willing to modify artifacts that you don't maintain, which >> most people (quite sensibly) aren't. Even if you limit your use of >> automatic modules to closed systems, as Stephen Colebourne suggests [2], >> they're still of significant value. Let's see if we can rescue them. >> >> The present algorithm for naming automatic modules has two problems: >> >> (A) Conflicts are possible across large artifact repositories, since >> the name of an automatic module is computed from the name of the >> artifact that defines it. [1] >> >> (B) It's risky to publish a module that `requires` some other module >> that has not yet been modularized, and hence must be used as an >> automatic module. If the maintainer of that module later chooses >> an explicit name different from the automatic name then you must >> publish a new version of your module with an updated `requires` >> directive. [2] >> >> As to (A), yes, conflicts exist, though it's worth observing that many of >> the conflicts in the Maven Central data are due to poorly-chosen artifact >> names: `parent`, `library`, `core`, and `common` top the list, which then >> falls off in a long-tail distribution. When conflicts are detected then >> build tools can rename artifacts either automatically or, preferably, to >> user-specified names that map to sensible automatic-module names. If >> renaming artifacts in the filesystem proves impractical then we could >> extend the syntax of the `--module-path` option to allow a module name >> to be specified for each specifically-named artifact, though strictly >> speaking that would be a feature of the JDK rather than JPMS. >> >> We can address (B) by enabling the maintainers of existing components to >> specify the module names that should be given to their components when >> used as automatic modules, without having to write `module-info.java` >> files. This can be done very simply, with a single new JAR-file manifest >> `Module-Name` attribute, as first suggested almost a year ago [3]. >> >> If we add this one feature then the maintainer of an existing component >> that, e.g., must still build and run on JDK 7 can choose a module name >> for that component, record it in the manifest by adding a few lines to >> the `pom.xml` file, and tell users that they can use it as an automatic >> module on JDK 9 without fear that the module name will change when the >> component is properly modularized some years from now. The actual change >> to the component is small and low-risk, so it can reasonably be done in >> a patch release. There's no need to write a `module-info.java` file, >> and in fact doing so may be inadvisable at this point if the component >> depends on other components that have not yet been given module names. >> >> This approach for (B) does add one more (optional) step to the migration >> path, but it will hopefully lead to a larger number of explicitly-named >> modules in the world -- and in particular in Maven Central -- sooner >> rather than later. >> >> - Mark >> >> >> [1] >> http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2017-January/000537.html >> [2] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-January/011106.html >> [3] http://openjdk.java.net/projects/jigsaw/spec/issues/#ModuleNameInManifest -- - DML From forax at univ-mlv.fr Thu Mar 16 10:51:48 2017 From: forax at univ-mlv.fr (Remi Forax) Date: Thu, 16 Mar 2017 11:51:48 +0100 (CET) Subject: How to name modules, automatic and otherwise In-Reply-To: <7cf59089-8b10-c221-1f3b-f1c9b79c0367@redhat.com> References: <20170216164827.A309F65E1B@eggemoggin.niobe.net> <1305027279.2009761.1487328314502.JavaMail.zimbra@u-pem.fr> <7cf59089-8b10-c221-1f3b-f1c9b79c0367@redhat.com> Message-ID: <693885663.787752.1489661508196.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "David M. Lloyd" > ?: jpms-spec-experts at openjdk.java.net > Envoy?: Mercredi 15 Mars 2017 21:00:37 > Objet: Re: How to name modules, automatic and otherwise > R?mi, AFIACT this automatic mutation of name based on groupId could only > be done by Maven or the Maven Central repository software. And it would > essentially entail baking the generated name into the module... at which > point we have a module-info.class file anyway (i.e. not automatic modules!). > > Did I miss something, or are you actually arguing to drop automatic > modules from the spec and push it up to the tooling instead? No, there are two different things, 1) We should have automatic module to solve the issue of people wanting to have there application (not their library) to works with module. Here, i think i agree with the recent mail of Stephen Colebourne, that if there is no ModuleName in the Manifest, we should have the simplest/boldest algorithm to compute the name of the module from name of the jar, perhaps not something as extrem as using the name of the jar directly but something along taking the characters from the start to the last occurrence of '-'. 2) How to avoid the npm mess when someone will want to upload a modular jar on Maven Central. Here, i think that as EG, we should recommend that the module name should starts with the Maven group id. I know that this will not be a specification but just a recommandation, because it a problem of whom is managing Maven Central. But i think it's important to provide guidelines on that subject because it's a question that had come often during the session i've done to explain JPMS at different JUGs. R?mi > > On 02/17/2017 04:45 AM, Remi Forax wrote: >> I agree that >> - we should not bake the Maven way of specify identifiers into the language. >> - we should not drop automatic modules, this is a practical way to do the >> migration if you are maintaining an application. >> >> On the addition of Module-Name attribute, i can understand that it's a simple >> way to reserve a name. I will agree to introduce that in the spec only if >> Robert and Brian finds that an automatic module with a Module-Name attribute is >> something that can be pushed on Maven Central. Otherwise, specifying a >> Module-Name is not very different of writing a module-info.java with everything >> exported (i see the fact that a regular module can not access to the classpath >> unlike an automatic module as a bonus). >> >> I fully disagree on the idea of using short names for module, the Java community >> is too big for this kind of shortcut, the reverse DNS prefix have serve us well >> and counting the number of keystrokes is not a good metric. >> I've spend some time with Herve Boutemy last Tuesday night (and early Wednesday) >> to discuss about existing modules in Maven Central interropt. The Maven group >> id is something you have to ask before being able to publish on Maven Central, >> so it's more a Maven Central group id that a Maven group id. We should >> recommend that existing module on Maven Central should use a name that starts >> with the Maven Central group id. >> By example, Google's Guava should be com.google.guava or com.google.guava.core >> (or anything else that starts with com.google.guava). >> >> This will solve the name collisions issues between artifacts of Maven Central at >> least. >> >> R?mi >> >> ----- Mail original ----- >>> De: "mark reinhold" >>> ?: jpms-spec-experts at openjdk.java.net >>> Envoy?: Jeudi 16 F?vrier 2017 17:48:27 >>> Objet: How to name modules, automatic and otherwise >> >>> This note is in reply to the concerns about automatic modules raised by >>> Robert Scholte and Brian Fox [1], and by Stephen Colebourne and others >>> [2]. I've collected my conclusions here rather than in separate messages >>> because there are several distinct yet intertwined issues. >>> >>> Summary: >>> >>> - Module names should not include Maven group identifiers, because >>> modules are more abstract than the artifacts that define them. >>> >>> - Module names should use the reverse-domain-name-prefix convention >>> or, preferably, the project-name-prefix convention. >>> >>> - We should not abandon automatic modules, since they are a key tool >>> for migration and adoption. >>> >>> - We can address the problems of automatic modules with two fairly >>> minor technical enhancements. >>> >>> If any of these points strikes you as controversial, please read on! >>> >>> * * * >>> >>> Module names should not include Maven group identifiers, as Robert >>> Scholte and Brian Fox suggest [1], even for modules declared explicitly >>> in `module-info.java` files. Modules in JPMS are a construct of the Java >>> programming language, implemented in both the compiler and the virtual >>> machine. As such, they are more abstract entities than the artifacts >>> that define them. This distinction is useful, both conceptually and >>> practically, hence module names should remain more abstract. >>> >>> This distinction is useful conceptually because it makes it easier, as >>> we read source code, to think clearly about the nature of a module. We >>> can reason about a module's dependences, exports, services, and so forth >>> without cluttering our minds with the details of group identifiers and >>> version constraints. Today, e.g., we can write, and read: >>> >>> module foo.data { >>> exports com.bar.foo.data; >>> requires hibernate.core; >>> requires hibernate.jcache; >>> requires hibernate.validator; >>> } >>> >>> If we were to extend the syntax of module names to include group >>> identifiers, and encourage people to use them, then we'd be faced with >>> something much more verbose: >>> >>> module com.bar:foo.data { >>> exports com.bar.foo.data; >>> requires org.hibernate:hibernate.core; >>> requires org.hibernate:hibernate.jcache; >>> requires org.hibernate:hibernate.validator; >>> } >>> >>> Group identifiers make perfect sense in the context of a build system >>> such as Maven, where they bring necessary structure to the names of the >>> millions of artifacts available across different repositories. Such >>> structure is superfluous and distracting in the context of a module >>> system, where the number of relevant modules in any particular situation >>> is more likely to be in the tens, or hundreds, or (rarely) thousands. >>> All else being equal, simpler names are better. >>> >>> At a practical level, the distinction between modules and artifacts is >>> useful because it leaves the entire problem of artifact selection to the >>> build system. This allows us to switch from one artifact to another >>> simply by editing a `pom.xml` file to adjust a version constraint or a >>> group identifier; if module names included group identifiers then we'd >>> also have to edit the `module-info.java` file. This flexibility can be >>> helpful if, e.g., a project is forked and a new module with the same name >>> and artifact identifier is published under a different group identifier. >>> We long ago decided not to do version selection in the module system, >>> which surprised some people but has worked out fairly well. We should >>> treat group selection in the same manner. >>> >>> Another practical benefit of the module/artifact distinction is that it >>> keeps the module system independent of any particular build system, so >>> that build systems can continue to improve and evolve independently over >>> time. Maven-style coordinates are the most popular way to name artifacts >>> in repositories today, but that might not be true ten years from now. It >>> would be unwise to adopt Maven's naming convention for module names just >>> because it's popular now, and doubly so to bake Maven's group-identifier >>> concept into the Java programming language. >>> >>> * * * >>> >>> If module names don't include group identifiers, then how should modules >>> be named? What advice should we give to someone who's creating a new >>> module from scratch, or modularizing an existing component by writing a >>> `module-info.java` file for it? (Continue to set aside, for the moment, >>> the problems of automatic modules.) >>> >>> In structuring any particular space of names we must balance (at least) >>> three fundamental tensions: We want names that are long enough to be >>> descriptive, short enough to be memorable, and unique enough to avoid >>> needless conflicts. >>> >>> If you control all of the modules upon which your module depends, and >>> all of the modules that depend upon it, then you can of course name your >>> module whatever you want, and change its name at any time. If, however, >>> you're going to publish your module for use by others -- whether just >>> within your own organization or to a global repository such as Maven >>> Central -- then you should take more care. There are two well-known >>> ways to go about this. >>> >>> - Choose module names that start with the reversed form of an Internet >>> domain name that you control, or are at least associated with. The >>> Java Language Specification has long suggested this convention as a >>> way to minimize conflicts amongst package names, and it has been >>> widely though not universally adopted for that purpose. >>> >>> - Choose module names that start with the name of your project or >>> product. Module (and package) names that start with reversed domain >>> names are less likely to conflict but they're unnecessarily verbose, >>> they start with the least-important information (e.g., `com`, `org`, >>> or `net`), and they don't read well after exogenous changes such as >>> open-source donations or corporate acquisitions (e.g., `com.sun.*`). >>> >>> The reversed domain-name approach was sensible in the early days of Java, >>> before we had development tools sophisticated enough to help us deal with >>> the occasional conflict. We have such tools now, so going forward the >>> superior readability of short module and package names that start with >>> project or product names is preferable to the onerous verbosity of those >>> that start with reversed domain names. >>> >>> This advice will strike some readers as controversial. I respect those >>> who will choose, for the sake of tradition or an abundance of caution, to >>> use the reversed domain-name convention for module names and continue to >>> use that convention for package names. I do know, however, of at least >>> one major, well-known project whose developers intend to adopt the >>> project-name-prefix convention for their module names. >>> >>> * * * >>> >>> If module names don't include group identifiers, then how should automatic >>> modules be named? Or are automatic modules so troublesome that we should >>> remove them from the design? >>> >>> To answer the second question first: It would be a tragic shame to drop >>> automatic modules, since otherwise top-down migration is impossible if >>> you're not willing to modify artifacts that you don't maintain, which >>> most people (quite sensibly) aren't. Even if you limit your use of >>> automatic modules to closed systems, as Stephen Colebourne suggests [2], >>> they're still of significant value. Let's see if we can rescue them. >>> >>> The present algorithm for naming automatic modules has two problems: >>> >>> (A) Conflicts are possible across large artifact repositories, since >>> the name of an automatic module is computed from the name of the >>> artifact that defines it. [1] >>> >>> (B) It's risky to publish a module that `requires` some other module >>> that has not yet been modularized, and hence must be used as an >>> automatic module. If the maintainer of that module later chooses >>> an explicit name different from the automatic name then you must >>> publish a new version of your module with an updated `requires` >>> directive. [2] >>> >>> As to (A), yes, conflicts exist, though it's worth observing that many of >>> the conflicts in the Maven Central data are due to poorly-chosen artifact >>> names: `parent`, `library`, `core`, and `common` top the list, which then >>> falls off in a long-tail distribution. When conflicts are detected then >>> build tools can rename artifacts either automatically or, preferably, to >>> user-specified names that map to sensible automatic-module names. If >>> renaming artifacts in the filesystem proves impractical then we could >>> extend the syntax of the `--module-path` option to allow a module name >>> to be specified for each specifically-named artifact, though strictly >>> speaking that would be a feature of the JDK rather than JPMS. >>> >>> We can address (B) by enabling the maintainers of existing components to >>> specify the module names that should be given to their components when >>> used as automatic modules, without having to write `module-info.java` >>> files. This can be done very simply, with a single new JAR-file manifest >>> `Module-Name` attribute, as first suggested almost a year ago [3]. >>> >>> If we add this one feature then the maintainer of an existing component >>> that, e.g., must still build and run on JDK 7 can choose a module name >>> for that component, record it in the manifest by adding a few lines to >>> the `pom.xml` file, and tell users that they can use it as an automatic >>> module on JDK 9 without fear that the module name will change when the >>> component is properly modularized some years from now. The actual change >>> to the component is small and low-risk, so it can reasonably be done in >>> a patch release. There's no need to write a `module-info.java` file, >>> and in fact doing so may be inadvisable at this point if the component >>> depends on other components that have not yet been given module names. >>> >>> This approach for (B) does add one more (optional) step to the migration >>> path, but it will hopefully lead to a larger number of explicitly-named >>> modules in the world -- and in particular in Maven Central -- sooner >>> rather than later. >>> >>> - Mark >>> >>> >>> [1] >>> http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2017-January/000537.html >>> [2] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-January/011106.html >>> [3] http://openjdk.java.net/projects/jigsaw/spec/issues/#ModuleNameInManifest > > -- > - DML From forax at univ-mlv.fr Thu Mar 16 11:33:27 2017 From: forax at univ-mlv.fr (Remi Forax) Date: Thu, 16 Mar 2017 12:33:27 +0100 (CET) Subject: #LayerPrimitives aka allowing to add private package at runtime to a module ? Message-ID: <259607006.840452.1489664007763.JavaMail.zimbra@u-pem.fr> This is currently a point where several of us disagree, here is my opinion on it. First, we have to all recognize that making something mutable, the content of a module or anything else, makes maintenance, debugging, etc harder than having the same thing non mutable. So the default choice should to choosing the immutable solution apart if there are good argument. Also, in term of implementation, making the content of a module mutable can be seen as a cache invalidation issue, and we all know that cache invalidation is a hard problem especially when it's on global states. With JPMS, a module is now part of the Java language, so it seems weird to make it mutable the same way, a java.lang.Class is not mutable in Java. But that's not true, in fact a java.lang.Class is mutable in Java, but you can only mutate it using an agent. I'm pretty sure that if we do not make the content of a module mutable, people will hack the JPMS implementation to make it mutable, because during the development you want to be able to add private package like you want to add new method in an existing class without having to restart the program (think things like JRebel or Vert.x here). So in my opinion, we should have a mechanism to add private package to a module, there real question is should this mechanism only visible from an agent or should it be also available from the LayerController ? Aside, there is something that make me uncomfortable in the mails from David or Thomas, I'm maybe wrong, but it seems to me that you are thinking that you will be able to propose to your users to use the JPMS runtime with the current existing modules. I think your are chasing an unicorn/trying to solve a problem nobody care, i maintain several OSGI bundles and i do not want an implementation of OSGI to see them as JPMS modules, not by default because they will not work, the encapsulation model of JPMS is far stricter than the current model (i think it's the same issue with JBoss Module but i've no experience with them). As a user, if i create an OSGI module with a module-info or a bundle manifest that let me specified that this is a JPMS module, then i want it to be seen as a JPMS module but not by default. We have introduced a modulepath which is not a replacement of the classpath exactly for the same reason. So the interropt between OSGI and JPMS or JBoss Module and JPMS should be seen in my opinion with that idea in mind. In that case, supporting fragment that have also a JPMS module can be seen as a corner case, having to iterate over private packages is also not an issue because the existing manifest and the manifest of a JPMS module can be different, having more info, so at runtime the private packages will be available. R?mi From david.lloyd at redhat.com Thu Mar 16 13:39:25 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 16 Mar 2017 08:39:25 -0500 Subject: #LayerPrimitives aka allowing to add private package at runtime to a module ? In-Reply-To: <259607006.840452.1489664007763.JavaMail.zimbra@u-pem.fr> References: <259607006.840452.1489664007763.JavaMail.zimbra@u-pem.fr> Message-ID: On 03/16/2017 06:33 AM, Remi Forax wrote: > This is currently a point where several of us disagree, here is my opinion on it. Thanks for the reply. > First, we have to all recognize that making something mutable, the content of a module or anything else, makes maintenance, debugging, etc harder than having the same thing non mutable. > So the default choice should to choosing the immutable solution apart if there are good argument. This is a good argument. In my initial JBoss Modules implementation, modules were immutable. Our first service container was also immutable. But these models provided too many restrictions to us and to end users and in the end we found these ideals to be unrealistic, even misguided, when applied to this kind of thing. Yes maintenance and debug cost for us is important - more important than you know, even. But, a framework has one (or a constant number of) developer(s), whereas it has many, many users - a number which grows without bound, if a framework is successful. In this light, saving cost to the developer(s) of the framework (even in this hypothetical situation) is meaningless if that cost savings means your users have to make up the difference. In any event though, it's just an academic debate because modules in the Jigsaw implementation are not, and never have been, immutable. The methods on Module and Layer.Controller that do exist could never function otherwise, and there's the matter of dynamic proxies of course, which also generate packages at run time. If the JDK must do it for its framework support classes, it's a safe bet that others need to as well. To produce an obvious and trivial example, any EJB-compliant implementation must also have the ability to generate dynamic proxy classes in the same way that the JDK does (but with a different base class). JPA uses all kinds of proxies. Many frameworks and specs require generation of classes at run time. And so on. Every one of these things have the same security requirements as JDK proxies. So in order to support the immutability argument, now containers have to guess every package they might need to generate or load into up front, and frankly most containers don't work that way. Why? Because until now, class loading was 100% lazy, and this is has proven to be a *good model*. > Also, in term of implementation, making the content of a module mutable can be seen as a cache invalidation issue, and we all know that cache invalidation is a hard problem especially when it's on global states. This I find to be a weaker argument. What cache are we talking about, and what are the practical problems that make this intractable? > With JPMS, a module is now part of the Java language, so it seems weird to make it mutable the same way, a java.lang.Class is not mutable in Java. > But that's not true, in fact a java.lang.Class is mutable in Java, but you can only mutate it using an agent. It's clear that modules are not to be treated like classes, because (for example) circularity among modules is considered "bad" whereas circularity among classes has been shown to be indispensable, and classes within a class loader or module are loaded and resolved lazily whereas modules are loaded and resolved aggressively, etc. But I see your point that only "special" citizens ought to mutate a module. I agree, but I do not think this should be limited to agents: containers also have a similar need for similar reasons. > I'm pretty sure that if we do not make the content of a module mutable, people will hack the JPMS implementation to make it mutable, because during the development you want to be able to add private package like you want to add new method in an existing class without having to restart the program (think things like JRebel or Vert.x here). > > So in my opinion, we should have a mechanism to add private package to a module, there real question is should this mechanism only visible from an agent or should it be also available from the LayerController ? Even considering Java EE alone requires the framework to have the ability to do these things. It's not so hard to imagine that there are many related use cases. > Aside, there is something that make me uncomfortable in the mails from David or Thomas, I'm maybe wrong, but it seems to me that you are thinking that you will be able to propose to your users to use the JPMS runtime with the current existing modules. I think your are chasing an unicorn/trying to solve a problem nobody care, i maintain several OSGI bundles and i do not want an implementation of OSGI to see them as JPMS modules, not by default because they will not work, the encapsulation model of JPMS is far stricter than the current model (i think it's the same issue with JBoss Module but i've no experience with them). As a user, if i create an OSGI module with a module-info or a bundle manifest that let me specified that this is a JPMS module, then i want it to be seen as a JPMS module but not by default. We have introduced a modulepath which is not a replacement of the classpath exactly for the same reason. So the interropt between OSGI and JPMS or JBoss Module and JPMS should be seen in my opinion with that idea in mind. In that case, supporting fragment that have also a JPMS module can be seen as a corner case, having to iterate over private packages is also not an issue because the existing manifest and the manifest of a JPMS module can be different, having more info, so at runtime the private packages will be available. There are multiple answers. First, JPMS justified the new reflection capabilities on a security basis. And believe me, I argued hard against that. :-) If this basis is to hold up, then containers need to be able to extend these guarantees and options to their deployments. To quote Mark, "The lead of Oracle's Java Vulnerability Team estimates that at least a third of all the vulnerabilities reported since JDK 7 GA would have been prevented if we'd had the ability to strongly encapsulate JDK-internal packages." We (JBoss) have had our own share of CVEs as well, and the same arguments could be applied here just as easily. Why should containers be second-class when it comes to security? How does it impact users to say "we've solved a major problem with security but the solution is only available to code and containers that have not been written yet, sorry"? Second, JPMS introduces a new deployment and packaging model. If users use them, containers will need to support them. And AFAICT Java EE 9 is still operating under the assumption that this packaging model will be used by deployments. So in order to interoperate with them, existing systems (the same existing systems that have the dynamicity requirements above) have to be able to synthesize a way for such modules to be deployed and interoperate with the existing systems. The best way is to have the actual JPMS implementation do this in the normal way. These problems (and most others that I've raised) are derived from the idea we have chosen the implementation, and then are deciding what use cases are valid based on the design parameters of the chosen implementation. Instead we *must* have a design and implementation which meets the use cases. Any pretense of being guided by the requirements has long since been exposed as being applied inconsistently (at best) as it suites the implementation. But we can't decide that the use case of interoperability is invalid just because the implementation does not presently make it convenient. -- - DML From tjwatson at us.ibm.com Thu Mar 16 13:52:10 2017 From: tjwatson at us.ibm.com (Thomas Watson) Date: Thu, 16 Mar 2017 07:52:10 -0600 Subject: #LayerPrimitives aka allowing to add private package at runtime to a module ? In-Reply-To: <259607006.840452.1489664007763.JavaMail.zimbra@u-pem.fr> References: <259607006.840452.1489664007763.JavaMail.zimbra@u-pem.fr> Message-ID: > From: Remi Forax > > Aside, there is something that make me uncomfortable in the mails > from David or Thomas, I'm maybe wrong, but it seems to me that you > are thinking that you will be able to propose to your users to use > the JPMS runtime with the current existing modules. I'm not sure what you mean by existing modules here. I assume you mean the existing modules from our own module systems (aka OSGi bundes). > I think your are > chasing an unicorn/trying to solve a problem nobody care, i maintain > several OSGI bundles and i do not want an implementation of OSGI to > see them as JPMS modules, not by default because they will not work, > the encapsulation model of JPMS is far stricter than the current > model (i think it's the same issue with JBoss Module but i've no > experience with them). I'm not proposing users of OSGi today will want to have their bundles loaded as JPMS modules by default. I have a very specific usecase in mind that does not apply to the scenarios OSGi is used for today because JPMS does not exist yet. We have a JEE runtime that is built out of OSGi bundles. If I am correct JBoss is similar but built out of JBoss modules. Today we can deploy JEE applications and control the access to APIs exported by our runtime from the application class loader. Our APIs are loaded out of OSGi bundles. The typical applications are not OSGi bundles. They are EARs, WARs, EJBs etc. Now I am assuming at some point JEE will be updated to consider delivering applications as JPMS modules, or perhaps something outside of the JEE spec will become popular and it will be developed as JPMS modules and we will want to support that. Here I assume the developers of these application JPMS modules will want their modules loaded as real JPMS modules. As such they will likely require standard named modules which we provide today as OSGi bundles (e.g. java.servlet). > As a user, if i create an OSGI module with a > module-info or a bundle manifest that let me specified that this is > a JPMS module, then i want it to be seen as a JPMS module but not by > default. We have introduced a modulepath which is not a replacement > of the classpath exactly for the same reason. So the interropt > between OSGI and JPMS or JBoss Module and JPMS should be seen in my > opinion with that idea in mind. In that case, supporting fragment > that have also a JPMS module can be seen as a corner case, having to > iterate over private packages is also not an issue because the > existing manifest and the manifest of a JPMS module can be > different, having more info, so at runtime the private packages will > be available. > > R?mi Private packages - I've already agreed that scanning for private packages likely is something that should be solved outside of JPMS. Using a dynamic addPackage to do this lazily probably is not the right solution. You are correct, JPMS has some more strict rules. - In JPMS a module is not open by default - In OSGi a bundle is always open (perhaps in the future there will be a way to close a bundle) - In JPMS no cycles are allowed - In OSGi cycles are allowed - In JPMS no split packages are allowed - In OSGi split packages are allowed (albeit not as a best practice) But JPMS also does allow for a module to be mutable in certain ways. - read edges are dynamic - service uses are dynamic - adding targeted open packages are dynamic - adding targeted export packages are dynamic All of which I assume can invalidate the cache you referred to. I use this dynamic nature of modules to wire OSGi modules when the OSGi modules break the strict rules you mention. But if there are no cycles or split packages we are able to create an accurate Layer/Module hierarchy to represent the bundles as open modules. In our case the runtime has very few, if any, cycles and absolutely has no split packages so I anticipate our bundles will accurately map into proper module descriptions. With this in mind you may ask why we do not migrate directly over to JPMS modules and run like any other JPMS module would. We depend heavily on the most important parts of OSGi, which is not the actual module system and class loading, instead it is the activation lifecycle, service registry and configuration. It will be a massive undertaking to migrate such things over to JPMS directly. But if you are developing OSGi bundles that simply Export-Package, Import-Package and Require-Bundle and do not participate in the rest of what OSGi offers for component style development then JPMS is a perfectly suitable alternative. With that said, I do not anticipate many will care to run their existing OSGi applications as JPMS modules like we will want to if we have to support applications built out of JPMS modules in the future. Tom From tjwatson at us.ibm.com Thu Mar 16 13:55:01 2017 From: tjwatson at us.ibm.com (Thomas Watson) Date: Thu, 16 Mar 2017 07:55:01 -0600 Subject: #LayerPrimitives aka allowing to add private package at runtime to a module ? In-Reply-To: <259607006.840452.1489664007763.JavaMail.zimbra@u-pem.fr> References: <259607006.840452.1489664007763.JavaMail.zimbra@u-pem.fr> Message-ID: Sorry for the double email, I sent the first message to the observers list by mistake. > From: Remi Forax > > Aside, there is something that make me uncomfortable in the mails > from David or Thomas, I'm maybe wrong, but it seems to me that you > are thinking that you will be able to propose to your users to use > the JPMS runtime with the current existing modules. I'm not sure what you mean by existing modules here. I assume you mean the existing modules from our own module systems (aka OSGi bundes). > I think your are > chasing an unicorn/trying to solve a problem nobody care, i maintain > several OSGI bundles and i do not want an implementation of OSGI to > see them as JPMS modules, not by default because they will not work, > the encapsulation model of JPMS is far stricter than the current > model (i think it's the same issue with JBoss Module but i've no > experience with them). I'm not proposing users of OSGi today will want to have their bundles loaded as JPMS modules by default. I have a very specific usecase in mind that does not apply to the scenarios OSGi is used for today because JPMS does not exist yet. We have a JEE runtime that is built out of OSGi bundles. If I am correct JBoss is similar but built out of JBoss modules. Today we can deploy JEE applications and control the access to APIs exported by our runtime from the application class loader. Our APIs are loaded out of OSGi bundles. The typical applications are not OSGi bundles. They are EARs, WARs, EJBs etc. Now I am assuming at some point JEE will be updated to consider delivering applications as JPMS modules, or perhaps something outside of the JEE spec will become popular and it will be developed as JPMS modules and we will want to support that. Here I assume the developers of these application JPMS modules will want their modules loaded as real JPMS modules. As such they will likely require standard named modules which we provide today as OSGi bundles (e.g. java.servlet). > As a user, if i create an OSGI module with a > module-info or a bundle manifest that let me specified that this is > a JPMS module, then i want it to be seen as a JPMS module but not by > default. We have introduced a modulepath which is not a replacement > of the classpath exactly for the same reason. So the interropt > between OSGI and JPMS or JBoss Module and JPMS should be seen in my > opinion with that idea in mind. In that case, supporting fragment > that have also a JPMS module can be seen as a corner case, having to > iterate over private packages is also not an issue because the > existing manifest and the manifest of a JPMS module can be > different, having more info, so at runtime the private packages will > be available. > > R?mi Private packages - I've already agreed that scanning for private packages likely is something that should be solved outside of JPMS. Using a dynamic addPackage to do this lazily probably is not the right solution. You are correct, JPMS has some more strict rules. - In JPMS a module is not open by default - In OSGi a bundle is always open (perhaps in the future there will be a way to close a bundle) - In JPMS no cycles are allowed - In OSGi cycles are allowed - In JPMS no split packages are allowed - In OSGi split packages are allowed (albeit not as a best practice) But JPMS also does allow for a module to be mutable in certain ways. - read edges are dynamic - service uses are dynamic - adding targeted open packages are dynamic - adding targeted export packages are dynamic All of which I assume can invalidate the cache you referred to. I use this dynamic nature of modules to wire OSGi modules when the OSGi modules break the strict rules you mention. But if there are no cycles or split packages we are able to create an accurate Layer/Module hierarchy to represent the bundles as open modules. In our case the runtime has very few, if any, cycles and absolutely has no split packages so I anticipate our bundles will accurately map into proper module descriptions. With this in mind you may ask why we do not migrate directly over to JPMS modules and run like any other JPMS module would. We depend heavily on the most important parts of OSGi, which is not the actual module system and class loading, instead it is the activation lifecycle, service registry and configuration. It will be a massive undertaking to migrate such things over to JPMS directly. But if you are developing OSGi bundles that simply Export-Package, Import-Package and Require-Bundle and do not participate in the rest of what OSGi offers for component style development then JPMS is a perfectly suitable alternative. With that said, I do not anticipate many will care to run their existing OSGi applications as JPMS modules like we will want to if we have to support applications built out of JPMS modules in the future. Tom From stef at epardaud.fr Thu Mar 16 14:33:35 2017 From: stef at epardaud.fr (Stephane Epardaud) Date: Thu, 16 Mar 2017 15:33:35 +0100 Subject: #LayerPrimitives aka allowing to add private package at runtime to a module ? In-Reply-To: <259607006.840452.1489664007763.JavaMail.zimbra@u-pem.fr> References: <259607006.840452.1489664007763.JavaMail.zimbra@u-pem.fr> Message-ID: <58CAA23F.6040206@epardaud.fr> On 16/03/2017 12:33, Remi Forax wrote: > With JPMS, a module is now part of the Java language, so it seems weird to make it mutable the same way, a java.lang.Class is not mutable in Java. > Actually, I was more thinking that it was akin to "packages" in reflection, where they are mutable in the sense that you cannot mutate classes once loaded, but you can load classes one after the other, and you can keep adding classes to packages as long as you want. If packages in reflection would let you access its member list, that would be mutable. Doesn't strike me as different with modules which are just package containers. I can totally imagine class loaders adding new packages to modules at run-time the same way we can add classes to packages. From mark.reinhold at oracle.com Thu Mar 16 14:55:32 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 16 Mar 2017 07:55:32 -0700 Subject: #LayerPrimitives aka allowing to add private package at runtime to a module ? In-Reply-To: <259607006.840452.1489664007763.JavaMail.zimbra@u-pem.fr> References: <259607006.840452.1489664007763.JavaMail.zimbra@u-pem.fr> Message-ID: <20170316075532.233647927@eggemoggin.niobe.net> 2017/3/16 4:33:27 -0700, forax at univ-mlv.fr: > ... > > With JPMS, a module is now part of the Java language, so it seems > weird to make it mutable the same way, a java.lang.Class is not > mutable in Java. But that's not true, in fact a java.lang.Class is > mutable in Java, but you can only mutate it using an agent. > > I'm pretty sure that if we do not make the content of a module > mutable, people will hack the JPMS implementation to make it mutable, > because during the development you want to be able to add private > package like you want to add new method in an existing class without > having to restart the program (think things like JRebel or Vert.x > here). JVM TI and the `java.lang.instrument` API cannot be used to add methods to an existing class, or really do very much at all except redefine existing methods. The specification of the `RetransformClasses` function [1] is very clear about this: The retransformation may change method bodies, the constant pool and attributes. The retransformation must not add, remove or rename fields or methods, change the signatures of methods, change modifiers, or change inheritance. There is equivalent wording in the specification of the `RedefineClasses` function [2]. The JVM TI specification also states that "these restrictions may be lifted in future versions". If that ever comes to pass [3] then that would be a reasonable time to consider extending JVM TI also to support the addition of packages to modules by agents at run time. JRebel and similar tools work by doing load-time instrumentation to add a level of indirection between a class and its content, so that methods can appear to be added on-the-fly [4]. - Mark [1] http://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html#RetransformClasses [2] http://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html#RedefineClasses [3] http://openjdk.java.net/jeps/159 [4] https://zeroturnaround.com/rebellabs/reloading_java_classes_401_hotswap_jrebel From mark.reinhold at oracle.com Thu Mar 23 18:15:24 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 23 Mar 2017 11:15:24 -0700 (PDT) Subject: Proposal (revised): #VersionsInModuleNames Message-ID: <20170323181524.ADFFE882C1@eggemoggin.niobe.net> Issue summary ------------- #VersionsInModuleNames -- Some have argued that library maintainers will be tempted to encode major version numbers, or even full version numbers, in module names. Is there some way we can guide people away from doing that? [1] Proposal -------- Based on extensive feedback, from EG members and others, make two changes: - Abandon the current proposal [2] to mandate that module names used anywhere in source-form module declarations both start and end with "Java letters". (This restriction was never implemented in the prototype RI.) - Revise the algorithm that computes the names of automatic modules so that it preserves digits at the end of module names [3]. The name for `commons-lang3-3.0.jar`, e.g., will now be `commons.lang3`. (Whether that algorithm is changed in other ways, or automatic modules are abandoned altogether, is the subject of the separate #AutomaticModuleNames issue [4].) In order to discourage the encoding of version numbers in module names a Java language compiler may issue a lint warning when a module with such a name is compiled, but whether or not a compiler does so is beyond the scope of this specification. [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#VersionsInModuleNames [2] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-December/000516.html [3] http://cr.openjdk.java.net/~mr/jigsaw/spec/api/java/lang/module/ModuleFinder.html#of-java.nio.file.Path...- [4] http://openjdk.java.net/projects/jigsaw/spec/issues/#AutomaticModuleNames From david.lloyd at redhat.com Thu Mar 23 18:44:48 2017 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 23 Mar 2017 13:44:48 -0500 Subject: Proposal (revised): #VersionsInModuleNames In-Reply-To: <20170323181524.ADFFE882C1@eggemoggin.niobe.net> References: <20170323181524.ADFFE882C1@eggemoggin.niobe.net> Message-ID: <60702d8c-7fe9-2a63-6855-b7efe22142f0@redhat.com> Looks good, thanks! On 03/23/2017 01:15 PM, mark.reinhold at oracle.com wrote: > Issue summary > ------------- > > #VersionsInModuleNames -- Some have argued that library maintainers > will be tempted to encode major version numbers, or even full version > numbers, in module names. Is there some way we can guide people away > from doing that? [1] > > Proposal > -------- > > Based on extensive feedback, from EG members and others, make two > changes: > > - Abandon the current proposal [2] to mandate that module names used > anywhere in source-form module declarations both start and end with > "Java letters". > > (This restriction was never implemented in the prototype RI.) > > - Revise the algorithm that computes the names of automatic modules so > that it preserves digits at the end of module names [3]. The name > for `commons-lang3-3.0.jar`, e.g., will now be `commons.lang3`. > > (Whether that algorithm is changed in other ways, or automatic > modules are abandoned altogether, is the subject of the separate > #AutomaticModuleNames issue [4].) > > In order to discourage the encoding of version numbers in module names a > Java language compiler may issue a lint warning when a module with such > a name is compiled, but whether or not a compiler does so is beyond the > scope of this specification. > > > [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#VersionsInModuleNames > [2] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-December/000516.html > [3] http://cr.openjdk.java.net/~mr/jigsaw/spec/api/java/lang/module/ModuleFinder.html#of-java.nio.file.Path...- > [4] http://openjdk.java.net/projects/jigsaw/spec/issues/#AutomaticModuleNames > -- - DML From rfscholte at apache.org Fri Mar 24 09:20:08 2017 From: rfscholte at apache.org (Robert Scholte) Date: Fri, 24 Mar 2017 10:20:08 +0100 Subject: Proposal (revised): #VersionsInModuleNames In-Reply-To: <20170323181524.ADFFE882C1@eggemoggin.niobe.net> References: <20170323181524.ADFFE882C1@eggemoggin.niobe.net> Message-ID: Hi Mark, I like these adjustments. thanks, Robert On Thu, 23 Mar 2017 19:15:24 +0100, wrote: > Issue summary > ------------- > > #VersionsInModuleNames -- Some have argued that library maintainers > will be tempted to encode major version numbers, or even full version > numbers, in module names. Is there some way we can guide people away > from doing that? [1] > > Proposal > -------- > > Based on extensive feedback, from EG members and others, make two > changes: > > - Abandon the current proposal [2] to mandate that module names used > anywhere in source-form module declarations both start and end with > "Java letters". > > (This restriction was never implemented in the prototype RI.) > > - Revise the algorithm that computes the names of automatic modules so > that it preserves digits at the end of module names [3]. The name > for `commons-lang3-3.0.jar`, e.g., will now be `commons.lang3`. > > (Whether that algorithm is changed in other ways, or automatic > modules are abandoned altogether, is the subject of the separate > #AutomaticModuleNames issue [4].) > > In order to discourage the encoding of version numbers in module names a > Java language compiler may issue a lint warning when a module with such > a name is compiled, but whether or not a compiler does so is beyond the > scope of this specification. > > > [1] > http://openjdk.java.net/projects/jigsaw/spec/issues/#VersionsInModuleNames > [2] > http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-December/000516.html > [3] > http://cr.openjdk.java.net/~mr/jigsaw/spec/api/java/lang/module/ModuleFinder.html#of-java.nio.file.Path...- > [4] > http://openjdk.java.net/projects/jigsaw/spec/issues/#AutomaticModuleNames From forax at univ-mlv.fr Fri Mar 24 09:35:25 2017 From: forax at univ-mlv.fr (Remi Forax) Date: Fri, 24 Mar 2017 10:35:25 +0100 (CET) Subject: Proposal (revised): #VersionsInModuleNames In-Reply-To: References: <20170323181524.ADFFE882C1@eggemoggin.niobe.net> Message-ID: <1157854927.1623459.1490348125689.JavaMail.zimbra@u-pem.fr> So am i. R?mi ----- Mail original ----- > De: "Robert Scholte" > ?: jpms-spec-experts at openjdk.java.net > Envoy?: Vendredi 24 Mars 2017 10:20:08 > Objet: Re: Proposal (revised): #VersionsInModuleNames > Hi Mark, > > I like these adjustments. > > thanks, > Robert > > On Thu, 23 Mar 2017 19:15:24 +0100, wrote: > >> Issue summary >> ------------- >> >> #VersionsInModuleNames -- Some have argued that library maintainers >> will be tempted to encode major version numbers, or even full version >> numbers, in module names. Is there some way we can guide people away >> from doing that? [1] >> >> Proposal >> -------- >> >> Based on extensive feedback, from EG members and others, make two >> changes: >> >> - Abandon the current proposal [2] to mandate that module names used >> anywhere in source-form module declarations both start and end with >> "Java letters". >> >> (This restriction was never implemented in the prototype RI.) >> >> - Revise the algorithm that computes the names of automatic modules so >> that it preserves digits at the end of module names [3]. The name >> for `commons-lang3-3.0.jar`, e.g., will now be `commons.lang3`. >> >> (Whether that algorithm is changed in other ways, or automatic >> modules are abandoned altogether, is the subject of the separate >> #AutomaticModuleNames issue [4].) >> >> In order to discourage the encoding of version numbers in module names a >> Java language compiler may issue a lint warning when a module with such >> a name is compiled, but whether or not a compiler does so is beyond the >> scope of this specification. >> >> >> [1] >> http://openjdk.java.net/projects/jigsaw/spec/issues/#VersionsInModuleNames >> [2] >> http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-December/000516.html >> [3] >> http://cr.openjdk.java.net/~mr/jigsaw/spec/api/java/lang/module/ModuleFinder.html#of-java.nio.file.Path...- >> [4] > > http://openjdk.java.net/projects/jigsaw/spec/issues/#AutomaticModuleNames From mark.reinhold at oracle.com Mon Mar 27 15:10:42 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 27 Mar 2017 08:10:42 -0700 (PDT) Subject: Proposal: Move the `Module` and `Layer` classes from `java.lang.reflect` to `java.lang` Message-ID: <20170327151042.C577B89BC8@eggemoggin.niobe.net> (I'm proposing this API change via a new issue in order to make sure that everyone is aware of it. Please let me know ASAP if you have any concerns; I'd like to merge it into an update of the Public Review draft.) Issue summary ------------- #MoveModuleAndLayerClasses -- Move the `Module` and `Layer` classes, and a related exception class, from the `java.lang.reflect` package up into the `java.lang` package. This will improve conceptual clarity, be consistent with the past, and leave room for the future. [1] Proposal -------- Move and rename these classes as follows: java.lang.reflect.Module -> java.lang.Module java.lang.reflect.Layer -> java.lang.ModuleLayer java.lang.reflect.LayerInstantiationException -> java.lang.ModuleLayerInstantiationException The `Layer` class is renamed to `ModuleLayer` so that it's easy for readers to skip over that class by name rather than have to read its Javadoc in order to learn that it's not what they're looking for. The related exception is renamed to match. (The `Module` class is not renamed, about which more below.) Motivation ---------- Experience shows that the best high-level shape for a non-trivial API is often not apparent until it's nearly finished. Looking at the totality of classes and interfaces in the draft specification [2] and considering the history and potential future of the platform leads to several reasons for this proposed change. - For explanatory purposes there is a very useful and pleasant high-level conceptual analogy: class : class loader :: module : module layer That is, classes are instantiated via class loaders, and modules are instantiated via layers. The types corresponding to these concepts should all be in the same package. - Existing constructs with keyword declarations in the language already have corresponding types in the `java.lang` package: `Class` for `class` and `interface`, `Package` for `package`, and `Enum` for `enum`. Placing `Module` in the same package is consistent with these past choices. - The concepts of modules and layers will be with us for a very long time, and are just as fundamental as those of classes and class loaders. The core reflection API in `java.lang.reflect` will be with us for a very long time too but it's not as fundamental, and it stands a good chance of being de-emphasized over time as alternatives are developed (e.g., Maurizio Cimadamore's mirror proposal [3]). - Long after `Class` was introduced we extended the language with the concept of class literals (e.g., `Object.class`). Peering into the future, at some point we may want to extend the language to allow modules to be mentioned outside of module declarations, and in ways that relate language-level expressions to run-time `Module` objects. It would be unfortunate if such constructs had to be specified in the JLS to refer to a `Module` type in a legacy, de-emphasized `java.lang.reflect` package. Impact ------ This is a cosmetic change. No functionality will be added or removed. This change is not, however, source-compatible relative to Java SE 8. It adds new types to the `java.lang` package, which is implicitly imported on demand (i.e., `import java.lang.*`). If code in an existing source file imports some other package on demand, and that package declares a `Module` type, and the existing code refers to that type, then the file will not compile without change. This could affect, among others, users of the popular Guice framework, which declares its own `Module` type [4]. The impact of moving these three types from `java.lang.reflect` up into `java.lang` is ameliorated by two factors: - A very common best practice is to use precise, type-specific `import` statements (e.g., `import com.google.inject.Module`) rather than import-on-demand wildcards (e.g., `import com.google.inject.*`). Code that follows this practice, which is well-supported by all the major IDEs, will compile without error [5]. - If you declare a `Module` type in your own package then other code in your package that refers to that type will not be affected by the existence of `java.lang.Module`. That's because an unqualified type name in source code resolves to a type in the same package in preference to a type in an import-on-demand package. It may sometimes be necessary to add or adjust `import` statements in order to compile existing source code for Java SE 9 but, unlike some other kinds of changes required for migration, these kinds of changes will compile on earlier releases. Source compatibility from release to release is, in general, highly desirable. Unlike binary or behavioral compatibility, however, it has never been a total commitment of the Java Platform. Over the years we have from time to time broken it in order to move forward, and this is one of those times. [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#MoveModuleAndLayerClasses [2] http://cr.openjdk.java.net/~mr/jigsaw/spec/api/java.base-summary.html [3] http://cr.openjdk.java.net/~mcimadamore/reflection-manifesto.html [4] https://google.github.io/guice/api-docs/latest/javadoc/com/google/inject/Module.html [5] http://docs.oracle.com/javase/specs/jls/se8/html/jls-6.html#jls-6.4.1 From forax at univ-mlv.fr Tue Mar 28 06:15:59 2017 From: forax at univ-mlv.fr (Remi Forax) Date: Tue, 28 Mar 2017 06:15:59 +0000 Subject: Proposal: Move the `Module` and `Layer` classes from `java.lang.reflect` to `java.lang` In-Reply-To: <20170327151042.C577B89BC8@eggemoggin.niobe.net> References: <20170327151042.C577B89BC8@eggemoggin.niobe.net> Message-ID: One problem I see is that Module is an annotated elements and usually those belong to java.lang.reflect. So the idea that Module can be moved to java.lang with no dependency on java.lang.reflect is flawed because of that. I still think it's a good move, modules are now first class citizens so Module belongs to java.lang. The ties between java.lang and java.lang.reflect are strong and I do not see how to untangle them in the future, but that's a discussion we can have in the future when we will want to introduce generics over primitives/class values. Remi On March 27, 2017 5:10:42 PM GMT+02:00, mark.reinhold at oracle.com wrote: >(I'm proposing this API change via a new issue in order to make sure > that everyone is aware of it. Please let me know ASAP if you have any > concerns; I'd like to merge it into an update of the Public Review > draft.) > >Issue summary >------------- > > #MoveModuleAndLayerClasses -- Move the `Module` and `Layer` classes, > and a related exception class, from the `java.lang.reflect` package up >into the `java.lang` package. This will improve conceptual clarity, be > consistent with the past, and leave room for the future. [1] > >Proposal >-------- > >Move and rename these classes as follows: > > java.lang.reflect.Module -> java.lang.Module > java.lang.reflect.Layer -> java.lang.ModuleLayer > java.lang.reflect.LayerInstantiationException > -> java.lang.ModuleLayerInstantiationException > >The `Layer` class is renamed to `ModuleLayer` so that it's easy for >readers to skip over that class by name rather than have to read its >Javadoc in order to learn that it's not what they're looking for. >The related exception is renamed to match. (The `Module` class is >not renamed, about which more below.) > >Motivation >---------- > >Experience shows that the best high-level shape for a non-trivial API >is >often not apparent until it's nearly finished. Looking at the totality >of classes and interfaces in the draft specification [2] and >considering >the history and potential future of the platform leads to several >reasons >for this proposed change. > > - For explanatory purposes there is a very useful and pleasant > high-level conceptual analogy: > > class : class loader :: module : module layer > > That is, classes are instantiated via class loaders, and modules are > instantiated via layers. The types corresponding to these concepts > should all be in the same package. > >- Existing constructs with keyword declarations in the language already > have corresponding types in the `java.lang` package: `Class` for > `class` and `interface`, `Package` for `package`, and `Enum` for > `enum`. Placing `Module` in the same package is consistent with > these past choices. > > - The concepts of modules and layers will be with us for a very long > time, and are just as fundamental as those of classes and class > loaders. The core reflection API in `java.lang.reflect` will be > with us for a very long time too but it's not as fundamental, and it > stands a good chance of being de-emphasized over time as alternatives > are developed (e.g., Maurizio Cimadamore's mirror proposal [3]). > > - Long after `Class` was introduced we extended the language with the > concept of class literals (e.g., `Object.class`). Peering into the > future, at some point we may want to extend the language to allow > modules to be mentioned outside of module declarations, and in ways > that relate language-level expressions to run-time `Module` objects. > It would be unfortunate if such constructs had to be specified in the > JLS to refer to a `Module` type in a legacy, de-emphasized > `java.lang.reflect` package. > >Impact >------ > >This is a cosmetic change. No functionality will be added or removed. > >This change is not, however, source-compatible relative to Java SE 8. >It >adds new types to the `java.lang` package, which is implicitly imported >on demand (i.e., `import java.lang.*`). If code in an existing source >file imports some other package on demand, and that package declares a >`Module` type, and the existing code refers to that type, then the file >will not compile without change. This could affect, among others, >users >of the popular Guice framework, which declares its own `Module` type >[4]. > >The impact of moving these three types from `java.lang.reflect` up into >`java.lang` is ameliorated by two factors: > >- A very common best practice is to use precise, type-specific `import` > statements (e.g., `import com.google.inject.Module`) rather than > import-on-demand wildcards (e.g., `import com.google.inject.*`). > Code that follows this practice, which is well-supported by all the > major IDEs, will compile without error [5]. > > - If you declare a `Module` type in your own package then other code > in your package that refers to that type will not be affected by the > existence of `java.lang.Module`. That's because an unqualified type > name in source code resolves to a type in the same package in > preference to a type in an import-on-demand package. > >It may sometimes be necessary to add or adjust `import` statements in >order to compile existing source code for Java SE 9 but, unlike some >other kinds of changes required for migration, these kinds of changes >will compile on earlier releases. > >Source compatibility from release to release is, in general, highly >desirable. Unlike binary or behavioral compatibility, however, it has >never been a total commitment of the Java Platform. Over the years we >have from time to time broken it in order to move forward, and this is >one of those times. > > >[1] >http://openjdk.java.net/projects/jigsaw/spec/issues/#MoveModuleAndLayerClasses >[2] >http://cr.openjdk.java.net/~mr/jigsaw/spec/api/java.base-summary.html >[3] http://cr.openjdk.java.net/~mcimadamore/reflection-manifesto.html >[4] >https://google.github.io/guice/api-docs/latest/javadoc/com/google/inject/Module.html >[5] >http://docs.oracle.com/javase/specs/jls/se8/html/jls-6.html#jls-6.4.1 -- Sent from my Android device with K-9 Mail. Please excuse my brevity. From mark.reinhold at oracle.com Wed Mar 29 20:56:40 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 29 Mar 2017 13:56:40 -0700 Subject: Proposal: Move the `Module` and `Layer` classes from `java.lang.reflect` to `java.lang` In-Reply-To: References: <20170327151042.C577B89BC8@eggemoggin.niobe.net> Message-ID: <20170329135640.910569839@eggemoggin.niobe.net> 2017/3/27 23:15:59 -0700, Remi Forax : > One problem I see is that Module is an annotated elements and usually > those belong to java.lang.reflect. So the idea that Module can be > moved to java.lang with no dependency on java.lang.reflect is flawed > because of that. I didn't claim that `java.lang.Module` wouldn't depend upon anything in `java.lang.reflect`. Other types in the `java.lang` package already do this. `Class` and `Package` implement `AnnotatedElement`, and `Class` further implements both `GenericDeclaration` and `Type` (and `java.io.Serializable`, ugh). There are also, of course, many uses of non-`java.lang` types in the signatures of methods and fields in types in that package. The `java.lang` package has never been able to stand on its own, as aesthetically pleasing as that might have been -- not even in JDK 1.0. > I still think it's a good move, modules are now first class citizens > so Module belongs to java.lang. > > The ties between java.lang and java.lang.reflect are strong and I do > not see how to untangle them in the future, but that's a discussion we > can have in the future when we will want to introduce generics over > primitives/class values. Agreed. - Mark