From peter.kriens at aqute.biz Thu Oct 6 12:16:09 2016 From: peter.kriens at aqute.biz (Peter Kriens) Date: Thu, 6 Oct 2016 14:16:09 +0200 Subject: Resign from JPMS Expert Group Message-ID: <42676DF8-513D-4B86-9978-EA95AD9F8151@aQute.biz> Mark, I would like to resign from the JSR 376 Expert Group since I have unfortunately too little time to participate on the level that is required. Since this leaves the EG without OSGi expertise I would like to recommend Neil Bartlett as the ideal candidate. Neil has confirmed that he will be happy to replace me in the JSR Expert Group. I strongly encourage you to accept him because his OSGi expertise will be an excellent addition to the EG. Kind regards, Peter Kriens From mark.reinhold at oracle.com Thu Oct 6 17:06:59 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 6 Oct 2016 10:06:59 -0700 (PDT) Subject: Article on JPMS integration with existing module systems In-Reply-To: References: Message-ID: <20161006170659.B2351D38BA@eggemoggin.niobe.net> 2016/9/7 6:57:55 -0700, tim_ellison at uk.ibm.com: > Obviously Tom has come at this integration challenge from his deep > knowledge of a particular OSGi implementation. There are some > important observations, and minor tweaks to the current model that > will improve JPMS. > > Thomas Watson wrote on 18/08/2016 14:45:06: >> Below is the plain text format from an article I wrote on using JPMS >> layers with an existing module system (OSGi). ... [1] > > ... > >> Conclusion > > Ok, this is where there are a few issues and recommendations that this > group can debate... > >> This approach allows for a pretty accurate representation of a static set >> of resolved OSGi bundles as JPMS modules. But we are left with several >> issues that need to be addressed before this can be considered a truly >> viable solution. Some may decide these are permanent restrictions of JPMS >> that we will have to live with going forward. But I believe there are >> some tweaks to JPMS that could go a long ways to making this approach >> close to a complete solution. Listed below are some changes that would >> help. I listed them in the order of importance, but I think 1 and 2 are a >> close tie for most important. >> >> 1. Allow for code that manages a JPMS layer to have more control for >> establishing read access for the modules contained in the managed layer. >> The Module addReads method allows for read access to be added for a module >> dynamically at runtime. But it has a restriction that it must be called >> by a class defined by the module that wants new read access. It would be >> a great help if we could call addReads from the management code that >> created the layer. Perhaps an addReads(Module wantsRead, Module toTarget) >> method on Layer that checks the caller module is the same module get >> created the Layer? This could be used to solve a large set of issues >> outlined above: > > That sounds like a reasonable request. This has come up in past discussions. I don't think an API for this is strictly necessary. Code that manages a layer presumably controls the class loaders in that layer, and hence already has the power to load arbitrary classes. It can use that power to load tiny synthetic classes to add additional read edges as required, as Tom realized in his first experiment. That's all a bit clunky, however, and we could make it easier, so for the record I'll enter a new issue: #ReadabilityAddedByLayerCreator -- Provide a means by which the code that creates a layer can add readability edges from the modules in that layer to other modules, whether those modules are in that layer or in other layers. > ... > >> 2. Allow for reflection on classes from concealed packages. Many >> dependency injection containers depend on being able to act upon concealed >> classes in order to construct objects and inject the objects with >> dependencies. Forcing implementation details to be exported so that these >> classes can be acted upon by DI containers is wrong. > > I believe this is already covered by #ReflectiveAccessToNonExportedTypes > http://openjdk.java.net/projects/jigsaw/spec/issues/#ReflectiveAccessToNonExportedTypes Yes, I think that's right. > ... > >> 3. Allow for sub-graphs of modules to be discarded within a layer. > > Again, this sounds like a reasonable request. > >> - JPMS-ISSUE-007 - This would allow us to flush out the "dead" >> modules which should never be used anymore. This is a narrower form of the existing #MutableConfigurations issue [2]. It could be easier to implement, however, so I'll enter it separately as #DiscardableModules -- Provide a means by which a sub-graph of modules within a layer can be discarded, so that both those modules and their class loaders can be reclaimed. >> 4. Allow a layer to map a class loader to a default named module. Any >> classes from unknown packages to the JPMS would be assigned this named >> module instead of the unnamed module. >> >> - JPMS-ISSUE-002 - This would allow us to avoid having to scan for >> private packages. Instead we would map the bundle classloader to a module >> and that module could be used for the private packages. Rather than introduce a notion of default named modules, I wonder if this problem could instead be addressed by tooling. In JPMS we avoid scanning a modular JAR file for concealed packages by having the `jar` tool add a list of all the module's packages to the module descriptor as the JAR file is created. This list is optional, so if it's missing we do scan the JAR file, but we expect that to be a rare occurrence. If I understand the documentation correctly, bndtools does something similar by adding a `Private-Package` manifest header to record the list of non-exported packages in a bundle [3]. Could your OSGi-to-JPMS adaptation use that manifest header, when available, to avoid scanning for private packages? >> 5. Allow the JPMS requires statement to specify a module version. > > Consideration of versioning was explicitly excluded from the original > requirements, and I don't see it coming in at this stage. If you are > controlling the resolution of modules then you would be able to take account > of version numbers in the integration. Right. If you create your own ModuleFinder then you can arrange for the preferred version of each module to appear ahead of any other versions. >> - JPMS-ISSUE-006 - This would allow us to represent multiple >> versions of a bundle within the bundle layer and give JPMS modules the >> ability to specify which version they want. I think the first part of this, i.e., representing multiple versions of a bundle within a single layer, can be avoided entirely, as I'll describe in a separate message. >> My hope is that this experiment is useful in providing constructive >> feedback to the JPMS expert group. I hope they consider enhancing JPMS to >> make JPMS layers more usable with existing module systems like OSGi. > > Most of the suggestions are modest enhancements to the current model, so it > is encouraging to see that a good level of interoperability can be obtained > with a few minor tweaks. > > Thanks for doing this experiment Tom. I look forward to others' comments > too. Tom -- thanks again for this detailed feedback; it's been very helpful. Thinking about your second experiment led me to a slightly different approach to supporting bidirectional interoperation between OSGi and JPMS. It's a big topic, though, so rather than present it here I'll describe it in a separate message. - Mark [1] http://blog.osgi.org/2016/08/osgi-with-java-modules-all-way-down.html [2] http://openjdk.java.net/projects/jigsaw/spec/issues/#MutableConfigurations [3] http://bnd.bndtools.org/heads/private_package.html From mark.reinhold at oracle.com Thu Oct 6 17:07:59 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 6 Oct 2016 10:07:59 -0700 (PDT) Subject: Dynamic modules and bidirectional interoperation Message-ID: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net> The agreed requirements for this JSR mandate a modest degree of dynamic configuration [1]. They do not mandate _dynamic modules_, i.e., the fine-grained dynamic configuration or "hot deployment" that is possible with other module systems, such as OSGi and JBoss, in which individual modules can be installed, updated, and uninstalled at any time. The requirements also mandate that it "be possible for another module system, such as OSGi, to locate Java modules and resolve them using its own resolver, except possibly for core system modules" [2]. They do not mandate _bidirectional interoperation_, which in the case of OSGi would be the ability to arrange for bundles to refer to JPMS modules and also vice versa, i.e., for JPMS modules to refer to bundles. As things stand, the agreed dynamic-configuration and interoperation requirements appear to be satisfied by the proposed design. We have several open issues, however, among them #NonHierarchicalLayers, #MutableConfigurations, and #LazyConfigurationAndInstantiation, arising from the desire for dynamic modules expressed by David Lloyd. More recently, Thomas Watson reported on his attempt to achieve bidirectional interoperation between OSGi and JPMS [3], in which he devised a way to for a layer of JPMS modules to reference a layer of OSGi bundles. For his approach to be practical, however, he needs to be able to discard sub-graphs of modules within a layer so that they and their class loaders can be reclaimed earlier than the layer itself (#DiscardableModules [4]). At this stage we could proceed to adopt dynamic modules and bidirectional interoperation as two additional requirements, to be supported directly by the module system. Doing so would, however, lead to a very different and more complex design, with correspondingly complex implementations, since it would need to provide all of the features mentioned above, and quite likely more. As I've argued elsewhere this would, in my judgement, make it very difficult -- if not impossible -- to achieve the stated goals of this JSR, most importantly the goal that the resulting module system be approachable by Java developers of all skill levels. Fortunately, I don't think we need to go that far. Rather than move toward a design that supports these requirements directly we can instead support them with the present design indirectly, by modeling a dynamic module as a sequence of JPMS modules over time. This key idea is latent in Watson's OSGi adaptation, in which he models OSGi bundle wirings -- rather than bundles themselves -- as JPMS modules. When bundles are installed, updated, or uninstalled then he synthesizes new JPMS modules for the resultant new wirings and configures them into a single new JPMS layer, using the new class loaders created by the OSGi framework itself. That _bundle layer_ refers to the previous bundle layer, if any, so that over time a stack of bundle layers builds up in which the topmost layer always represents the current bundle wirings. JPMS modules can be configured and resolved in additional layers above the current OSGi bundle layer. If a bundle upon which a JPMS module depends is updated then the layer that contains that JPMS module is discarded and a new layer is instantiated, configured to refer to the new topmost bundle layer and (I assume) from the same set of root JPMS modules as the original. A big problem with this scheme, as Watson points out, is that the JPMS modules corresponding to discarded OSGi bundle wirings can be retained in memory for a long time, along with their class loaders, class objects, and related data structures. That's because every bundle layer remains in the stack of bundle layers, and that stack is not discarded until the OSGi framework itself is stopped and discarded (JPMS-ISSUE-007 in [3]). Watson suggests solving this problem by allowing sub-graphs of modules within a layer to be discarded (#DiscardableModules [4]). That would, however, require layers to mutable, whereas today they are not. That layers are immutable brings a number of advantages, including a simpler conceptual model, a stronger degree of reliable configuration, and safer implementations. Here, then, is an alternative approach: - Map each bundle wiring to a unique JPMS module, configured and instantiated in its own unique bundle-wiring layer. An OSGi bundle will then be associated not only with a sequence of bundle wirings over time, but also with a sequence of JPMS module descriptors, configurations, modules, and layers over time. A layer can be reclaimed after no strongly-reachable layers refer to it, so discarded bundle-wiring layers -- and their class loaders -- won't be retained unnecessarily. There's no need to represent multiple versions of the same bundle within a single layer, since every bundle wiring has its own layer (part of JPMS-ISSUE-006 in [3]). - Extend JPMS with #NonHierarchicalLayers, so that a set of JPMS modules can be resolved against modules in more than one parent layer. An ordinary JPMS layer can then have multiple bundle-wiring layers and ordinary JPMS layers as parents. To illustrate, here's a diagram that elaborates upon the second diagram in Watson's note: http://cr.openjdk.java.net/~mr/jigsaw/notes/osgi-on-jpms.png The left-hand side shows a layer containing two ordinary JPMS modules, jpms.a and jpms.b, resolved against two OSGi bundle-wiring layers, for bundle.a and bundle.b. (The blue boxes represent JPMS modules; the orange boxes are the class loaders created by the OSGi framework.) The right-hand side depicts some later time after which bundle.b has been updated, so its original JPMS module and layer were discarded and a new module and layer were created for it (bundle.b layer 2). The JPMS module layer had the original bundle.b layer as one of its parents so it, too, was discarded and a new JPMS layer was created with the second bundle.b layer -- along with the original, unchanged bundle.a layer -- as its parents. This implementation of OSGi bundles in terms of JPMS modules leverages the fact that OSGi already associates a bundle with a sequence of bundle wirings over time, and each new bundle wiring has its own unique class loader. All we are doing here is wrapping each of these class loaders in its own module and layer, so that ordinary JPMS modules can be configured to refer to them. There could be many such layers but these layers are very lightweight, since their sole function is to associate a single module with an existing class loader for the purpose of access control. This approach should generalize to other module systems that support dynamic modules in a similar fashion, i.e., by creating a new class loader each time a dynamic module is updated. There are, no doubt, further details to work out, such as exactly how to synthesize the module descriptors for bundle wirings so that they can sensibly be referenced by ordinary JPMS modules. Cyclic relationships amongst OSGi bundles and split packages across bundles can't be modeled directly in module descriptors, but I suspect they can be accommodated by inserting the necessary additional readability edges after the relevant layers are instantiated. Overall this does, however, appear to be a promising approach to supporting dynamic modules and bidirectional interoperation without adding undue complexity to the platform module system itself. To enable this with the present design we do need #NonHierarchicalLayers; that's a significant change, but I think it's feasible and will explore it further. In order to model the cyclic graphs and split packages that are possible in other module systems it'd be convenient, but not strictly necessary, to have #ReadabilityAddedByLayerCreator [5]. We have no need for #MutableConfigurations, #LazyConfigurationAndInstantiation, or #DiscardableModules, which I think is a big win for simplicity. Comments? - Mark [1] http://openjdk.java.net/projects/jigsaw/spec/reqs/#dynamic-configuration [2] http://openjdk.java.net/projects/jigsaw/spec/reqs/#interoperation [3] http://mail.openjdk.java.net/pipermail/jpms-spec-comments/2016-August/000062.html, or http://blog.osgi.org/2016/08/osgi-with-java-modules-all-way-down.html [4] http://openjdk.java.net/projects/jigsaw/spec/issues/#DiscardableModules [5] http://openjdk.java.net/projects/jigsaw/spec/issues/#ReadabilityAddedByLayerCreator From david.lloyd at redhat.com Thu Oct 6 17:17:46 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 6 Oct 2016 12:17:46 -0500 Subject: Dynamic modules and bidirectional interoperation In-Reply-To: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net> References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net> Message-ID: <75a57d20-0099-31aa-57dc-ff36875d4b0f@redhat.com> A quick question... On 10/06/2016 12:07 PM, mark.reinhold at oracle.com wrote: > Cyclic relationships > amongst OSGi bundles and split packages across bundles can't be modeled > directly in module descriptors, but I suspect they can be accommodated by > inserting the necessary additional readability edges after the relevant > layers are instantiated. Do read edges added at run time affect the subsequent linkage of classes in the affected module? I was under the impression that that was not the case. -- - DML From tjwatson at us.ibm.com Thu Oct 6 18:22:16 2016 From: tjwatson at us.ibm.com (Thomas Watson) Date: Thu, 6 Oct 2016 13:22:16 -0500 Subject: Dynamic modules and bidirectional interoperation In-Reply-To: <75a57d20-0099-31aa-57dc-ff36875d4b0f@redhat.com> References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net> <75a57d20-0099-31aa-57dc-ff36875d4b0f@redhat.com> Message-ID: > From: "David M. Lloyd" > A quick question... > > On 10/06/2016 12:07 PM, mark.reinhold at oracle.com wrote: > > Cyclic relationships > > amongst OSGi bundles and split packages across bundles can't be modeled > > directly in module descriptors, but I suspect they can be accommodated by > > inserting the necessary additional readability edges after the relevant > > layers are instantiated. > > Do read edges added at run time affect the subsequent linkage of classes > in the affected module? I was under the impression that that was not the > case. > > -- > - DML > My assumption is no, but it also is not relevant to the scenario where the read edges are being added to a Module representing an entity (e.g. bundle) from another module system. In such a case I assume the other module system (OSGi or Jboss) is in complete control over the class loader implementation and delegation of the resolution graph for the respective module system. For example, the OSGi bundle class loaders will not all of a sudden start delegating to other module class loaders because of a new edge represented in JPMS. But I would be interested to know if dynamically adding read edges for the built-in class loaders of JMPS Layers affect class loading/linkage delegation at runtime. Tom From mark.reinhold at oracle.com Thu Oct 6 18:36:04 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 06 Oct 2016 11:36:04 -0700 Subject: Dynamic modules and bidirectional interoperation In-Reply-To: <75a57d20-0099-31aa-57dc-ff36875d4b0f@redhat.com> References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net> <75a57d20-0099-31aa-57dc-ff36875d4b0f@redhat.com> Message-ID: <20161006113604.775141470eggemoggin.niobe.net> 2016/10/6 10:17:46 -0700, david.lloyd at redhat.com: > A quick question... > > On 10/06/2016 12:07 PM, mark.reinhold at oracle.com wrote: >> ... Cyclic relationships >> amongst OSGi bundles and split packages across bundles can't be modeled >> directly in module descriptors, but I suspect they can be accommodated by >> inserting the necessary additional readability edges after the relevant >> layers are instantiated. > > Do read edges added at run time affect the subsequent linkage of classes > in the affected module? Read edges added at run time only affect run-time access-control checks. They do not change the configuration previously computed for the relevant layer. Subsequent configurations resolved against that configuration will not take the new read edge into account. - Mark From david.lloyd at redhat.com Thu Oct 6 18:56:01 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 6 Oct 2016 13:56:01 -0500 Subject: Dynamic modules and bidirectional interoperation In-Reply-To: <20161006113604.775141470eggemoggin.niobe.net> References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net> <75a57d20-0099-31aa-57dc-ff36875d4b0f@redhat.com> <20161006113604.775141470eggemoggin.niobe.net> Message-ID: <87aa8663-99f2-d1ff-ec85-1e3739cb8e4d@redhat.com> On 10/06/2016 01:36 PM, mark.reinhold at oracle.com wrote: > 2016/10/6 10:17:46 -0700, david.lloyd at redhat.com: >> A quick question... >> >> On 10/06/2016 12:07 PM, mark.reinhold at oracle.com wrote: >>> ... Cyclic relationships >>> amongst OSGi bundles and split packages across bundles can't be modeled >>> directly in module descriptors, but I suspect they can be accommodated by >>> inserting the necessary additional readability edges after the relevant >>> layers are instantiated. >> >> Do read edges added at run time affect the subsequent linkage of classes >> in the affected module? > > Read edges added at run time only affect run-time access-control > checks. They do not change the configuration previously computed > for the relevant layer. Subsequent configurations resolved against > that configuration will not take the new read edge into account. Then I do not believe that adding read edges alone will be sufficient as a resolution to #CyclicDependencies for either OSGi or our use cases, unfortunately, or at least I don't see how it could be. -- - DML From tjwatson at us.ibm.com Thu Oct 6 19:20:17 2016 From: tjwatson at us.ibm.com (Thomas Watson) Date: Thu, 6 Oct 2016 14:20:17 -0500 Subject: Dynamic modules and bidirectional interoperation In-Reply-To: <87aa8663-99f2-d1ff-ec85-1e3739cb8e4d@redhat.com> References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net><75a57d20-0099-31aa-57dc-ff36875d4b0f@redhat.com><20161006113604.775141470eggemoggin.niobe.net> <87aa8663-99f2-d1ff-ec85-1e3739cb8e4d@redhat.com> Message-ID: > From: "David M. Lloyd" > > On 10/06/2016 01:36 PM, mark.reinhold at oracle.com wrote: > > 2016/10/6 10:17:46 -0700, david.lloyd at redhat.com: > >> A quick question... > >> > >> On 10/06/2016 12:07 PM, mark.reinhold at oracle.com wrote: > >>> ... Cyclic relationships > >>> amongst OSGi bundles and split packages across bundles can't be modeled > >>> directly in module descriptors, but I suspect they can be accommodated by > >>> inserting the necessary additional readability edges after the relevant > >>> layers are instantiated. > >> > >> Do read edges added at run time affect the subsequent linkage of classes > >> in the affected module? > > > > Read edges added at run time only affect run-time access-control > > checks. They do not change the configuration previously computed > > for the relevant layer. Subsequent configurations resolved against > > that configuration will not take the new read edge into account. > > Then I do not believe that adding read edges alone will be sufficient as > a resolution to #CyclicDependencies for either OSGi or our use cases, > unfortunately, or at least I don't see how it could be. > -- > - DML > My understanding is that read edges are needed to get past the accessibility checks that are done by JPMS when a class is loaded by a class loader which is associated with a Module known to JPMS itself. In my experiment I attempted to represent the OSGi class loaders (wirings) with a generated module which lives inside a bundle JMPS Layer. Once I did this all the accessibility rules (for execution not class loading) are enabled at the JPMS layer according to the read edges known by JPMS. But the class loader delegation is still completely under the control of the OSGi module system (framework). The read edges are needed so we can represent the possible cycles, and even access to split packages from the OSGi resolution graph between the Modules within the bundle layer(s). In my experiment I chose to not express any dependencies in the ModuleDesciptors which represent the OSGi bundles. Instead I create the Layer holding the Modules and then after the Layer is "resolved" from JPMS point of view I do addReads to wire the modules up according to the delegation rules of OSGi resolution. This way I can get passed the cycle and split package restrictions that are in place during JPMS Layer resolution. Tom. From david.lloyd at redhat.com Thu Oct 6 19:30:54 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 6 Oct 2016 14:30:54 -0500 Subject: Dynamic modules and bidirectional interoperation In-Reply-To: References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net> <75a57d20-0099-31aa-57dc-ff36875d4b0f@redhat.com> <20161006113604.775141470eggemoggin.niobe.net> <87aa8663-99f2-d1ff-ec85-1e3739cb8e4d@redhat.com> Message-ID: <66e629d2-0378-0058-18c9-e9d2704fd603@redhat.com> On 10/06/2016 02:20 PM, Thomas Watson wrote: > > >> From: "David M. Lloyd" >> > On 10/06/2016 01:36 PM, mark.reinhold at oracle.com wrote: >> > 2016/10/6 10:17:46 -0700, david.lloyd at redhat.com: >> >> A quick question... >> >> >> >> On 10/06/2016 12:07 PM, mark.reinhold at oracle.com wrote: >> >>> ... Cyclic relationships >> >>> amongst OSGi bundles and split packages across bundles can't be modeled >> >>> directly in module descriptors, but I suspect they can be accommodated by >> >>> inserting the necessary additional readability edges after the relevant >> >>> layers are instantiated. >> >> >> >> Do read edges added at run time affect the subsequent linkage of classes >> >> in the affected module? >> > >> > Read edges added at run time only affect run-time access-control >> > checks. They do not change the configuration previously computed >> > for the relevant layer. Subsequent configurations resolved against >> > that configuration will not take the new read edge into account. >> >> Then I do not believe that adding read edges alone will be sufficient as >> a resolution to #CyclicDependencies for either OSGi or our use cases, >> unfortunately, or at least I don't see how it could be. >> -- >> - DML >> > > My understanding is that read edges are needed to get past the > accessibility checks that are done by JPMS when a class is loaded by a > class loader which is associated with a Module known to JPMS itself. In > my experiment I attempted to represent the OSGi class loaders (wirings) > with a generated module which lives inside a bundle JMPS Layer. Once I > did this all the accessibility rules (for execution not class loading) > are enabled at the JPMS layer according to the read edges known by JPMS. > But the class loader delegation is still completely under the control > of the OSGi module system (framework). The read edges are needed so we > can represent the possible cycles, and even access to split packages > from the OSGi resolution graph between the Modules within the bundle > layer(s). In my experiment I chose to not express any dependencies in > the ModuleDesciptors which represent the OSGi bundles. Instead I create > the Layer holding the Modules and then after the Layer is "resolved" > from JPMS point of view I do addReads to wire the modules up according > to the delegation rules of OSGi resolution. This way I can get passed > the cycle and split package restrictions that are in place during JPMS > Layer resolution. I see; but, doesn't that defeat the security benefits of the JPMS? If the classes defined by the bundles themselves are outside of modules (i.e. in a class loader's unnamed module) then they are globally accessible via reflection/setAccessible. If the security requirements are worthwhile, and we've all been proceeding on the assumption that they are, then it seems to me that this will ensure the obsolescence of OSGi (not to mention our deployment models). This is the crux of many of our issues: if you accept that the JPMS security measures are worthwhile, then we either have a situation where the user has to choose between security or flexibility, or else we have to sacrifice flexibility to get the added security across the board, losing features in the process. I continue to maintain that this is a false choice. -- - DML From tjwatson at us.ibm.com Thu Oct 6 20:17:41 2016 From: tjwatson at us.ibm.com (Thomas Watson) Date: Thu, 6 Oct 2016 15:17:41 -0500 Subject: Dynamic modules and bidirectional interoperation In-Reply-To: <66e629d2-0378-0058-18c9-e9d2704fd603@redhat.com> References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net> <75a57d20-0099-31aa-57dc-ff36875d4b0f@redhat.com> <20161006113604.775141470eggemoggin.niobe.net> <87aa8663-99f2-d1ff-ec85-1e3739cb8e4d@redhat.com> <66e629d2-0378-0058-18c9-e9d2704fd603@redhat.com> Message-ID: "David M. Lloyd" wrote on 10/06/2016 02:30:54 PM: > > > > My understanding is that read edges are needed to get past the > > accessibility checks that are done by JPMS when a class is loaded by a > > class loader which is associated with a Module known to JPMS itself. In > > my experiment I attempted to represent the OSGi class loaders (wirings) > > with a generated module which lives inside a bundle JMPS Layer. Once I > > did this all the accessibility rules (for execution not class loading) > > are enabled at the JPMS layer according to the read edges known by JPMS. > > But the class loader delegation is still completely under the control > > of the OSGi module system (framework). The read edges are needed so we > > can represent the possible cycles, and even access to split packages > > from the OSGi resolution graph between the Modules within the bundle > > layer(s). In my experiment I chose to not express any dependencies in > > the ModuleDesciptors which represent the OSGi bundles. Instead I create > > the Layer holding the Modules and then after the Layer is "resolved" > > from JPMS point of view I do addReads to wire the modules up according > > to the delegation rules of OSGi resolution. This way I can get passed > > the cycle and split package restrictions that are in place during JPMS > > Layer resolution. > > I see; but, doesn't that defeat the security benefits of the JPMS? If > the classes defined by the bundles themselves are outside of modules > (i.e. in a class loader's unnamed module) then they are globally > accessible via reflection/setAccessible. I am not trying to define classes using the unnamed module. My prototype was about trying to get out of the unnamed module space for the classes defined by OSGi class loaders. Instead I am mapping an OSGi class loader to a real module that I generated within a JPMS Layer API, by using the custom class loader mapping features of JPMS. The read edges are needed to continue to grant the runtime accessibility needed for the packages exported by the Module in the OSGi Layer to properly mimic the delegation rules OSGi resolves in the framework. This does not open up deep reflection by or to the classes defined by OSGi class loaders. These classes still follow the rules of JMPS. And that is where the issues outlined in the #ReflectiveAccessToNonExportedTypes thread pop up. Ideally the ModuleDescriptors representing OSGi bundles will only export the packages which the OSGi bundle manifest exports (using Export-Package). Depending on how the #ReflectiveAccessToNonExportedTypes works out we also would likely need to 'expose' all the private packages from OSGi bundles for deep reflection by default. My assumption is that the 'exposed' packages are not available for class loader delegation, only for reflection. That is if we want to maintain the behavior of OSGi today which allows deep reflection on any types private or public, exported or not. > If the security requirements > are worthwhile, and we've all been proceeding on the assumption that > they are, then it seems to me that this will ensure the obsolescence of > OSGi (not to mention our deployment models). I don't disagree. I would hope that JPMS Layer creation is flexible enough to allow our existing module systems to be represented in a custom Layer of sorts and maintain a high level of functionality which we enjoy today on Java 8. > > This is the crux of many of our issues: if you accept that the JPMS > security measures are worthwhile, then we either have a situation where > the user has to choose between security or flexibility, or else we have > to sacrifice flexibility to get the added security across the board, > losing features in the process. I continue to maintain that this is a > false choice. I don't disagree, but if we have a good outcome to #ReflectiveAccessToNonExportedTypes which allows us to 'expose' packages for deep reflection then we can at least maintain functionality of our existing module systems in a JPMS Layer world. Perhaps over time the existing module systems will be enhanced such that deep reflection is not needed as much or at least not assumed available by default. But such a change to existing module systems to restrict such things is going to be just as disruptive as the #ReflectiveAccessToNonExportedTypes issue is for JPMS, so that will not come quickly in my opinion. Tom From tjwatson at us.ibm.com Fri Oct 7 15:47:53 2016 From: tjwatson at us.ibm.com (Thomas Watson) Date: Fri, 7 Oct 2016 10:47:53 -0500 Subject: Dynamic modules and bidirectional interoperation In-Reply-To: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net> References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net> Message-ID: Overall I think this approach with #NonHierarchicalLayers is worth pursuing. A couple of observations. 1) Any OSGi dependencies that influence class loader delegation will not be declared in the ModuleDescriptors which represent the OSGi bundle wirings. Instead we will depend on the ability to #ReadabilityAddedByLayerCreator to wire up the modules within the bundle Layer according to OSGi resolution rules. It has a drawback if there are cases where some system code is trying to reason about modular dependencies by using the JPMS module API. I think this will have to be acceptable because I don't think we will be able to fully converge on all the possible dependency types available in the existing module systems. For example, OSGi has Require-Bundle which maps closing to JPMS requires, but OSGi also has Import-Package dependencies which cannot be mapped properly into JMPS module Layers. 2) I assume #NonHierarchicalLayers will be restricted to form a directed acyclic graph. As such I assume each node in the graph of Layers would be visited only once when resolving new Layers on top even if there are multiple paths to the layer from the new Layer being resolved. Tom "jpms-spec-observers" wrote on 10/06/2016 12:07:59 PM: > From: mark.reinhold at oracle.com > > Here, then, is an alternative approach: > > - Map each bundle wiring to a unique JPMS module, configured and > instantiated in its own unique bundle-wiring layer. An OSGi bundle > will then be associated not only with a sequence of bundle wirings > over time, but also with a sequence of JPMS module descriptors, > configurations, modules, and layers over time. A layer can be > reclaimed after no strongly-reachable layers refer to it, so > discarded bundle-wiring layers -- and their class loaders -- won't > be retained unnecessarily. There's no need to represent multiple > versions of the same bundle within a single layer, since every > bundle wiring has its own layer (part of JPMS-ISSUE-006 in [3]). > > - Extend JPMS with #NonHierarchicalLayers, so that a set of JPMS > modules can be resolved against modules in more than one parent > layer. An ordinary JPMS layer can then have multiple bundle-wiring > layers and ordinary JPMS layers as parents. > > To illustrate, here's a diagram that elaborates upon the second diagram > in Watson's note: > > http://cr.openjdk.java.net/~mr/jigsaw/notes/osgi-on-jpms.png > > The left-hand side shows a layer containing two ordinary JPMS modules, > jpms.a and jpms.b, resolved against two OSGi bundle-wiring layers, for > bundle.a and bundle.b. (The blue boxes represent JPMS modules; the > orange boxes are the class loaders created by the OSGi framework.) The > right-hand side depicts some later time after which bundle.b has been > updated, so its original JPMS module and layer were discarded and a new > module and layer were created for it (bundle.b layer 2). The JPMS module > layer had the original bundle.b layer as one of its parents so it, too, > was discarded and a new JPMS layer was created with the second bundle.b > layer -- along with the original, unchanged bundle.a layer -- as its > parents. > > This implementation of OSGi bundles in terms of JPMS modules leverages > the fact that OSGi already associates a bundle with a sequence of bundle > wirings over time, and each new bundle wiring has its own unique class > loader. All we are doing here is wrapping each of these class loaders in > its own module and layer, so that ordinary JPMS modules can be configured > to refer to them. There could be many such layers but these layers are > very lightweight, since their sole function is to associate a single > module with an existing class loader for the purpose of access control. > This approach should generalize to other module systems that support > dynamic modules in a similar fashion, i.e., by creating a new class > loader each time a dynamic module is updated. > > There are, no doubt, further details to work out, such as exactly how to > synthesize the module descriptors for bundle wirings so that they can > sensibly be referenced by ordinary JPMS modules. Cyclic relationships > amongst OSGi bundles and split packages across bundles can't be modeled > directly in module descriptors, but I suspect they can be accommodated by > inserting the necessary additional readability edges after the relevant > layers are instantiated. Overall this does, however, appear to be a > promising approach to supporting dynamic modules and bidirectional > interoperation without adding undue complexity to the platform module > system itself. > > To enable this with the present design we do need #NonHierarchicalLayers; > that's a significant change, but I think it's feasible and will explore > it further. In order to model the cyclic graphs and split packages that > are possible in other module systems it'd be convenient, but not strictly > necessary, to have #ReadabilityAddedByLayerCreator [5]. We have no need > for #MutableConfigurations, #LazyConfigurationAndInstantiation, or > #DiscardableModules, which I think is a big win for simplicity. > > Comments? > > - Mark > > > [1] http://openjdk.java.net/projects/jigsaw/spec/reqs/#dynamic-configuration > [2] http://openjdk.java.net/projects/jigsaw/spec/reqs/#interoperation > [3] http://mail.openjdk.java.net/pipermail/jpms-spec-comments/2016- > August/000062.html, > or http://blog.osgi.org/2016/08/osgi-with-java-modules-all-way-down.html > [4] http://openjdk.java.net/projects/jigsaw/spec/issues/#DiscardableModules > [5] http://openjdk.java.net/projects/jigsaw/spec/issues/ > #ReadabilityAddedByLayerCreator > From mark.reinhold at oracle.com Tue Oct 11 15:14:14 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 11 Oct 2016 08:14:14 -0700 (PDT) Subject: Proposal: #ReflectiveAccessToNonExportedTypes (revised) & #AwkwardStrongEncapsulation: Weak modules & private exports In-Reply-To: <617353007.2424567.1474033457532.JavaMail.zimbra@u-pem.fr> References: <20160912150801.AD8AECD07F@eggemoggin.niobe.net> <617353007.2424567.1474033457532.JavaMail.zimbra@u-pem.fr> Message-ID: <20161011151414.9D5E9D50F5@eggemoggin.niobe.net> 2016/9/16 6:44:17 -0700, forax at univ-mlv.fr: > Hi all, > the goals of this proposal is > - to explicitly mark packages that allow 'deep' reflection, > - disable 'deep' reflection by default even for exported packages > - ease the transition by adding a new migration state between automatic > module and strong module (weak module). > > so 3 goals with one stone :) Indeed! > Have i miss something obvious but the last goal, easing migration is not > listed in the issue summary ? While adding a new kind of module to ease > migration is maybe a good idea, it also add a new kind of modules when > thinking about the modules and their relationship, which seems a terrible > idea. We are loosing the KISS principle of the current module system. I didn't list ease of migration in the issue summary but I didn't think that necessary, since it's an overarching goal of this entire effort. I understand your reluctance to introduce what appears to be another kind of module, but semantically one can read weak module foo.bar { requires hibernate.core; requires hibernate.entitymanager; } as a simple shorthand for the more verbose module foo.bar { exports private com.foo.bar.model; exports private ...; // for other packages as needed requires hibernate.core; requires hibernate.entitymanager; } or, if we were to add wildcards, then module foo.bar { exports private *; requires hibernate.core; requires hibernate.entitymanager; } The advantage of introducing weak modules as an explicit concept is that they nicely capture the reliable-configuration part of our story and, by their very name, explicitly do not promise strong encapsulation. I think weak modules are, moreover, easier to explain to developers just starting to work with modules. Telling someone that they have to type module foo.bar { exports private *; } in order to make a simple module's packages available for deep reflection raises all kinds of questions. What does `exports` mean? What does `private` mean? What does `*` range over? It's all a bit like telling them that they need to type public class X { public static void main(String... args) { ... } } in order to print "Hello, world!". With weak modules you can simply explain that a weak module has no strong encapsulation and so will work whether it has a programmatic API or, as is the case here, it just contains model classes to be manipulated by a reflective framework. Strong encapsulation can be added later. > I fully agree that 'deep' reflection should be disallowed by default and can > be allowed package by package. But as Stephen Colebourne, i fail to see why > enabling 'deep' reflection on a package is related to the export > keyword. While 'deep' reflection is somewhat related to visibility, it's not (I assume you mean accessibility here, rather than visibility.) > only about visibility of a class/method/field only, by example you can change > the value of a public final field of a public class using setAccessible. In > my opinion, exposing the API to 'deep' reflection is a different concern as > exporting a package, thus using 'private' as a modifier on export is a bad > idea (it also doesn't work well with the notion of restricted export (export > to ...)). Adding a new keyword 'expose' as Stephen propose to enable 'deep' > reflection is a better way to answer to the goal 1 and 2. Reflective access to the elements of packages, deep or otherwise, is intimately associated with language/bytecode-level access because the reflection API has always aimed to be an API-level simulation of what's possible at the language/bytecode level. This is an extremely valuable property, since it makes it easy to reason about the correctness of reflective code. (There are, obviously, some exceptions to this analogy, namely the long-standing `setAccessible` method and, more recently, the changes we made for #ReflectionWithoutReadability.) Both the language and the reflection API are subject to the access-control rules of the module system, so using different keywords to express how they're made subject to those rules strikes me as confusing. I've read Colebourne's proposal [1], and I can see its intuitive appeal to developers (like us!) who are used to thinking about all these things. To a new developer it might be easier to explain his `exposes *` than the above `exports private *`, but I think explaining weak modules is even easier. A new developer is likely to hear the "reliable configuration + strong encapsulation" mantra fairly early on, and weak modules are simply the former, without the latter. A couple of other problems I see with Colebourne's approach: - The only way to allow an exported API package to be used reflectively is to expose it for deep reflection. That requires typing another directive and, worse, makes the internals of that package available for deep reflection, which is probably not what was intended. - The interactions between `exports`, `exposes`, and the qualified forms of these directives turns out to be pretty complicated once you work out all the cases. (We've done this internally, for a somewhat similar proposal that allows both `private` and `dynamic` modifiers on the `exports` directive.) They wind up being nearly as bad as the existing rules for protected members [2]. I'm not convinced that this relatively small bit of expressive power is worth the additional complexity. > About the other proposed items: > - using transitive instead of public for require, i fully agree. Good. > - about removing the dynamic exports feature, a package declared as non > exported but declared as exposed will have the same meaning, no ? if yes, > i agree that the notion dynamic export should be removed. I'm not sure if you're thinking of my proposal here, or Colebourne's. In my proposal you can't "expose" a package for reflection without also exporting it. If you export a package then it's available for shallow reflection at run time; if you add the `private` modifier then it's also available for deep reflection via `setAccessible`. As I mentioned in one of the notes in the proposal, removing `dynamic` does remove some expressive power, since you can no longer export a package only at run time. This means that an IDE or a compiler cannot (as easily) prevent you from writing code against a module that contains, e.g., JPA POJOs that are not also APIs (which most aren't). I understand the use case, but the cost of supporting it by retaining the `dynamic` modifier (or its semantic equivalent) is rather high, as I noted above. If experience shows that it's really necessary, we could add it in a future release. - Mark [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-September/009370.html [2] http://docs.oracle.com/javase/specs/jls/se8/html/jls-6.html#jls-6.6.2 From mark.reinhold at oracle.com Tue Oct 11 15:15:14 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 11 Oct 2016 08:15:14 -0700 (PDT) Subject: Proposal: #ReflectiveAccessToNonExportedTypes (revised) & #AwkwardStrongEncapsulation: Weak modules & private exports In-Reply-To: <5b9b1929-eb92-09ab-2997-d7e2395a1d9d@redhat.com> References: <20160912150801.AD8AECD07F@eggemoggin.niobe.net> <5b9b1929-eb92-09ab-2997-d7e2395a1d9d@redhat.com> Message-ID: <20161011151514.A04ECD50FA@eggemoggin.niobe.net> 2016/9/16 8:30:41 -0700, david.lloyd at redhat.com: > After a fairly detailed review of this proposal, we have determined that > it is not acceptable to Red Hat in its present form. I will list the > primary problems here, and then I'll start up discussion on jigsaw-dev > of several possible solutions that could work for us. I'll number these > in case anyone wants to respond piece-wise. > > #1) The "weak" designation appears to be pejorative > > Under this solution, many existing frameworks and/or the modules which > consume them must be marked "weak" in order to work correctly, or else a > much more complex module descriptor must be used. While this seems to > be a stepping stone for migration, we find that it rather makes > middleware appear as a second-class citizen. No insult is intended; "weak" is used here purely in a technical sense, as the obvious antonym of "strong". Suggestions for a better word for this concept are welcome. > #2) Weak modules are, in fact, weak > > The idea of retaining the Java 8 rules for transition modules appears > good on the surface, however the problem is that users may be forced to > use weak modules indefinitely in order to interoperate with certain > existing middleware and libraries, or else be faced with a potentially > complex migration. If the security benefits of "strong" modules are > real, then users will be rightly inclined to move from weak modules to > strong modules. This can be severely problematic if existing > middleware, libraries, and large applications cannot easily operate as > or with "strong" modules or cannot do so in an "easy to learn"/"easy to > use" fashion. This also means that "weak" modules exist only for the > purposes of maintaining a transition period - at some point they will > inevitably transition from "convenience" to "attractive nuisance". Weak modules do fall between automatic and strong modules on the overall migration path but they're not intended, necessarily, to be transitory. There's nothing wrong with evolving an existing JAR file into a weak module and stopping there. There's also nothing wrong with creating a new weak module from scratch, as one might do for a module that contains nothing but JPA model classes, and never evolving it into a strong module -- there's no benefit in doing that, so don't do it. > ... > > #3) Controlling reflection access is always bound up with controlling > exports > > The proposed exports system is simplistic but not exactly simple, yet > several useful modes remain left unaccounted-for. Internally we > examined the usefulness of controlling access to public and so-called > "deep" reflection in combination with whether or not a member was > exported, and we came up with this logical table: > > +------------------+---------------------------+ > | | Module is... | > | Reflection is... +------------+--------------+ > | | Exported | Not Exported | > +----------------------------------------------+ > | Forbidden | Not Useful | Useful | > +----------------------------------------------+ > | Public Only | Useful | Useful | > +----------------------------------------------+ > | Public & Private | Useful | Useful | > +------------------+------------+--------------+ I'm not sure how to read this table. Should the cell containing the text "Module is ..." instead contain "Member is ..."? By "exported" I'll assume that you mean non-reflective access, i.e., from the language at compile time and bytecodes at run time. Do you have example use cases for each of the "Useful" cells? Some are obvious, but others are not. > ... > > #4) There isn't always a container > > The justification for not providing an easy mechanism to restrict > private reflective access to a trusted framework is that a container can > always rewrite a module or inject classes. However there isn't always a > container present - something which the existence of weak modules > acknowledges. Some frameworks and middleware are meant to be usable > with or without a container, and it is undesirable to force these use > cases into a weak module situation or to require a strong coupling in > the module descriptor in this case. Some other solution should be found > that works both for containers and "regular" applications. If you're running outside of a container then how important is it to precisely restrict access to packages meant for deep reflection? If you're outside of a container then you have to trust the person (you?) who sets up your module path and/or class path. To take the canonical case of a module that just contains JPA model classes, if you make it a weak module or else `exports private` the relevant packages without qualification so that the hibernate.core (or whatever) module can reflect into it, then the scope for trouble is limited. Sure, some other module on the module path or even code on the class path could reflect into your module when it's not supposed to, but is it worth complicating the module system just to handle this scenario? Don't get me wrong: Strong encapsulation is generally a very good thing. It's not always absolutely necessary, however, and we need to balance the number of ways in which it can be used against the overall complexity of the design. (It looks like there was no #5 in your list.) > #6) Not all reflection is 100% "friendly", and that's OK > > Serialization frameworks generally need private reflective access to the > module which contains the classes being serialized. For some > frameworks, this access will already be adequate. Some existing > frameworks utilize Unsafe to create uninitialized instances. However > other frameworks, including frameworks that seek to comply with the > Serialization Specification [1], use ReflectionFactory, as that class > provides (exclusively, as far as I know) the ability to acquire a > constructor that produces a _partially_-initialized class where only the > constructors of superclasses to a certain depth are called. It is our > view that it would be a regression to force such frameworks to use > --add-exports or JVMTI, and using Unsafe for this purpose, or continuing > to use it, (as I understand it) puts at risk the ability to perform > optimizations which depend on entry point control, which I am given to > understand are a strong motivator for many of the design constraints > that exist on the Jigsaw implementation. It should be possible for > these frameworks to continue to use ReflectionFactory (unsupported > though it may be) if suitably privileged (in the security manager > sense), as long as all of the module(s) upon which the framework > reflects have granted private reflective access to it (in the module > declaration sense). So far as I know, this is already the case. The `ReflectionFactory` API is specifically retained in JDK 9 for this very purpose, per JEP 260 [1]. Do you have an example for which this does not work? > #7) More loose coupling seems necessary and useful > > In order for typical applications to function with modern middleware as > modules, without compromising security, it may be necessary to enhance > the loose coupling mechanism (uses/provides), or to provide an > additional, similar mechanism, which allows a symbolic coupling which > would allow modules to declare (in an abstract manner) modules which > need to have reflective access to it in a "friendly" manner. > Implementation ideas are forthcoming on jigsaw-dev. > > Providing a way within the system to grant public and/or private > reflection access in a specific manner, and doing so in a manner which > is easy to use and not prejudicial against existing or future middleware > and small or large applications which consume such middleware in the > Java SE or EE space, is our most basic requirement for satisfactory > resolution of this issue. To confirm my understanding, would it be correct to say that your main concern here is scenarios in which some of the classes in a user module are intended to be manipulated reflectively by some trusted framework module, but that framework module is either not known at compile time or is otherwise inconvenient to name in the declaration of the user module? > I will follow up on jigsaw-dev with some > implementation thoughts and more specific requirements. Discussions of issues in the proposed design, and specific proposals for changing the design, belong here on the EG list. The jigsaw-dev list is meant primarily for the discussion of issues outside the scope of the JSR such as implementation details, the design and implementation of JDK-specific APIs and tools, and feedback and questions arising from practical use of the prototype implementation. - Mark [1] http://openjdk.java.net/jeps/260 From Tim_Ellison at uk.ibm.com Wed Oct 12 08:45:16 2016 From: Tim_Ellison at uk.ibm.com (Tim Ellison) Date: Wed, 12 Oct 2016 09:45:16 +0100 Subject: Proposal: #ReflectiveAccessToNonExportedTypes (revised) & #AwkwardStrongEncapsulation: Weak modules & private exports In-Reply-To: <20161011151414.9D5E9D50F5@eggemoggin.niobe.net> References: <20160912150801.AD8AECD07F@eggemoggin.niobe.net> <617353007.2424567.1474033457532.JavaMail.zimbra@u-pem.fr> <20161011151414.9D5E9D50F5@eggemoggin.niobe.net> Message-ID: mark.reinhold at oracle.com wrote on 11/10/2016 16:14:14: > 2016/9/16 6:44:17 -0700, forax at univ-mlv.fr: > > Hi all, > > the goals of this proposal is > > - to explicitly mark packages that allow 'deep' reflection, > > - disable 'deep' reflection by default even for exported packages > > - ease the transition by adding a new migration state between automatic > > module and strong module (weak module). > > > > so 3 goals with one stone :) > > Indeed! > > > Have i miss something obvious but the last goal, easing migration is not > > listed in the issue summary ? While adding a new kind of module to ease > > migration is maybe a good idea, it also add a new kind of modules when > > thinking about the modules and their relationship, which seems a terrible > > idea. We are loosing the KISS principle of the current module system. > > I didn't list ease of migration in the issue summary but I didn't think > that necessary, since it's an overarching goal of this entire effort. Overall the concept of weak module does not provide an attractive migration step for existing module systems because it requires each module to "export private" all of its packages in order to retain existing behavior with respect to reflection. This will effectively expose all of the internals of the existing module as if it were API at runtime and at compile/tool time. It can be argued that even in existing modular systems there is a limited set of packages that must be allowed for deep reflection and the wholesale enablement of all packages concealed and exported for deep reflection is not needed. But this is not the case when mapping an existing modular system's custom class loaders into a JPMS Layer (e.g. OSGi). In that case we are motivated to preserve the existing behavior for the modules that represent the pre-existing modules/bundles. > I understand your reluctance to introduce what appears to be another kind > of module, but semantically one can read > > weak module foo.bar { > requires hibernate.core; > requires hibernate.entitymanager; > } > > as a simple shorthand for the more verbose > > module foo.bar { > exports private com.foo.bar.model; > exports private ...; // for other packages as needed > requires hibernate.core; > requires hibernate.entitymanager; > } > > or, if we were to add wildcards, then > > module foo.bar { > exports private *; > requires hibernate.core; > requires hibernate.entitymanager; > } > > The advantage of introducing weak modules as an explicit concept is that > they nicely capture the reliable-configuration part of our story and, by > their very name, explicitly do not promise strong encapsulation. > > I think weak modules are, moreover, easier to explain to developers just > starting to work with modules. Telling someone that they have to type > > module foo.bar { > exports private *; > } > > in order to make a simple module's packages available for deep reflection > raises all kinds of questions. What does `exports` mean? What does > `private` mean? What does `*` range over? It's all a bit like telling > them that they need to type > > public class X { > public static void main(String... args) { ... } > } > > in order to print "Hello, world!". With weak modules you can simply > explain that a weak module has no strong encapsulation and so will work > whether it has a programmatic API or, as is the case here, it just > contains model classes to be manipulated by a reflective framework. > Strong encapsulation can be added later. The declaration of a package as exported for tool-time linkage/compilation and normal runtime class loader delegation is orthogonal to the declaration of a package as available for 'deep' and/or 'shallow' reflection. The proposal mandates that 'shallow' reflection is only allowed on exported packages, and any concealed packages are not available for any type of reflection, so the use of the keyword 'private' also is confusing. Here the 'private' keyword is used to allow broader access to an exported package which is counter intuitive. One would think a private export would be less accessible than a 'public' export, but in general the proposed semantics are troublesome regardless of the keywords used. > > I fully agree that 'deep' reflection should be disallowed by default and can > > be allowed package by package. But as Stephen Colebourne, i fail to see why > > enabling 'deep' reflection on a package is related to the export > > keyword. While 'deep' reflection is somewhat related to visibility, it's not > > (I assume you mean accessibility here, rather than visibility.) > > > only about visibility of a class/method/field only, by example youcan change > > the value of a public final field of a public class using setAccessible. In > > my opinion, exposing the API to 'deep' reflection is a different concern as > > exporting a package, thus using 'private' as a modifier on export is a bad > > idea (it also doesn't work well with the notion of restricted export (export > > to ...)). Adding a new keyword 'expose' as Stephen propose to enable 'deep' > > reflection is a better way to answer to the goal 1 and 2. > > Reflective access to the elements of packages, deep or otherwise, is > intimately associated with language/bytecode-level access because the > reflection API has always aimed to be an API-level simulation of what's > possible at the language/bytecode level. This is an extremely valuable > property, since it makes it easy to reason about the correctness of > reflective code. (There are, obviously, some exceptions to this analogy, > namely the long-standing `setAccessible` method and, more recently, the > changes we made for #ReflectionWithoutReadability.) Both the language > and the reflection API are subject to the access-control rules of the > module system, so using different keywords to express how they're made > subject to those rules strikes me as confusing. > > I've read Colebourne's proposal [1], and I can see its intuitive appeal > to developers (like us!) who are used to thinking about all these things. > To a new developer it might be easier to explain his `exposes *` than the > above `exports private *`, but I think explaining weak modules is even > easier. A new developer is likely to hear the "reliable configuration + > strong encapsulation" mantra fairly early on, and weak modules are simply > the former, without the latter. I disagree. I like the Colebourne's approach because it keeps to two declarations ('exports' and 'exposes') as orthogonal concepts. However, I don't see the need to open the syntax up for wildcards '*' at this time. > A couple of other problems I see with Colebourne's approach: > > - The only way to allow an exported API package to be used reflectively > is to expose it for deep reflection. That requires typing another > directive and, worse, makes the internals of that package available > for deep reflection, which is probably not what was intended. > > - The interactions between `exports`, `exposes`, and the qualified > forms of these directives turns out to be pretty complicated once you > work out all the cases. (We've done this internally, for a somewhat > similar proposal that allows both `private` and `dynamic` modifiers > on the `exports` directive.) They wind up being nearly as bad as the > existing rules for protected members [2]. I'm not convinced that > this relatively small bit of expressive power is worth the additional > complexity. It would be interesting to see that exploration exposed, because as you say the Colebourne expression appears intuitive. > > About the other proposed items: > > - using transitive instead of public for require, i fully agree. > > Good. > > > - about removing the dynamic exports feature, a package declared as non > > exported but declared as exposed will have the same meaning, > no ? if yes, > > i agree that the notion dynamic export should be removed. > > I'm not sure if you're thinking of my proposal here, or Colebourne's. > > In my proposal you can't "expose" a package for reflection without also > exporting it. If you export a package then it's available for shallow > reflection at run time; if you add the `private` modifier then it's also > available for deep reflection via `setAccessible`. As discussed, it would be useful to explore the options for changing the proposal's constraint that you can't "expose" a package for reflection without also exporting it. Regards, Tim > As I mentioned in one of the notes in the proposal, removing `dynamic` > does remove some expressive power, since you can no longer export a > package only at run time. This means that an IDE or a compiler cannot > (as easily) prevent you from writing code against a module that contains, > e.g., JPA POJOs that are not also APIs (which most aren't). I understand > the use case, but the cost of supporting it by retaining the `dynamic` > modifier (or its semantic equivalent) is rather high, as I noted above. > If experience shows that it's really necessary, we could add it in a > future release. > > - Mark > > > [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016- > September/009370.html > [2] http://docs.oracle.com/javase/specs/jls/se8/html/jls-6.html#jls-6.6.2 > Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU From Tim_Ellison at uk.ibm.com Wed Oct 12 08:48:06 2016 From: Tim_Ellison at uk.ibm.com (Tim Ellison) Date: Wed, 12 Oct 2016 09:48:06 +0100 Subject: Resign from JPMS Expert Group In-Reply-To: <42676DF8-513D-4B86-9978-EA95AD9F8151@aQute.biz> References: <42676DF8-513D-4B86-9978-EA95AD9F8151@aQute.biz> Message-ID: Peter Kriens wrote on 06/10/2016 13:16:09: > Mark, > > I would like to resign from the JSR 376 Expert Group since I have > unfortunately too little time to participate on the level that is required. > > Since this leaves the EG without OSGi expertise I would like to > recommend Neil Bartlett as the ideal candidate. Neil has confirmed > that he will be happy to replace me in the JSR Expert Group. I > strongly encourage you to accept him because his OSGi expertise will > be an excellent addition to the EG. Peter, I'm sorry that you have not been able to find the time to contribute some of your valuable experience to the JSR. I've worked with Neil in the past and concur that he would be a fitting replacement on the EG. Of course, it is Mark's prerogative to permit the substitution. Regards, Tim Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU From mark.reinhold at oracle.com Wed Oct 12 19:30:04 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 12 Oct 2016 12:30:04 -0700 Subject: Resign from JPMS Expert Group In-Reply-To: <42676DF8-513D-4B86-9978-EA95AD9F8151@aQute.biz> References: <42676DF8-513D-4B86-9978-EA95AD9F8151@aQute.biz> Message-ID: <20161012123004.820359933eggemoggin.niobe.net> 2016/10/6 5:16:09 -0700, peter.kriens at aqute.biz: > Mark, > > I would like to resign from the JSR 376 Expert Group since I have > unfortunately too little time to participate on the level that is > required. > > Since this leaves the EG without OSGi expertise I would like to > recommend Neil Bartlett as the ideal candidate. Neil has confirmed > that he will be happy to replace me in the JSR Expert Group. I > strongly encourage you to accept him because his OSGi expertise will > be an excellent addition to the EG. Peter, I'm sorry to hear that you no longer have time to participate. Thanks for your past contributions. I'll certainly give Neal Bartlett's nomination due consideration. - Mark From forax at univ-mlv.fr Fri Oct 21 13:24:38 2016 From: forax at univ-mlv.fr (Remi Forax) Date: Fri, 21 Oct 2016 15:24:38 +0200 (CEST) Subject: Dynamic modules and bidirectional interoperation In-Reply-To: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net> References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net> Message-ID: <1316230997.272548.1477056278273.JavaMail.zimbra@u-pem.fr> Hi Mark, I see an other way to solve the wiring issue, use VM anonymous classes (via unsafe.defineAnonymousClass) to generate the glue code (as lambda meta factory does to implement the functional interface at runtime) and method handles as wires (once created a method handle do not do any security check anymore, so the requirement to be in a module with specific parents can be lifted). With that, you don't need to generate the wiring code inside a new module and when a new bundle need to be re-wire, you can mutate the wire (the method handle) by using a mutable callsite. Said differently, loading/unloading an OSGI bundle is equivalent to doing monkey patching in a dynamic language, so there is no reason to not use the tools we provide for dynamic languages to wire OSGI bundles. so i vote strongly against that proposal mostly because it's too late to change requirements and because some obvious design options to implement OSGI on top of JPMS have not been tested. regards, R?mi ----- Mail original ----- > De: "mark reinhold" > ?: jpms-spec-experts at openjdk.java.net > Envoy?: Jeudi 6 Octobre 2016 19:07:59 > Objet: Dynamic modules and bidirectional interoperation > The agreed requirements for this JSR mandate a modest degree of dynamic > configuration [1]. They do not mandate _dynamic modules_, i.e., the > fine-grained dynamic configuration or "hot deployment" that is possible > with other module systems, such as OSGi and JBoss, in which individual > modules can be installed, updated, and uninstalled at any time. > > The requirements also mandate that it "be possible for another module > system, such as OSGi, to locate Java modules and resolve them using its > own resolver, except possibly for core system modules" [2]. They do not > mandate _bidirectional interoperation_, which in the case of OSGi would > be the ability to arrange for bundles to refer to JPMS modules and also > vice versa, i.e., for JPMS modules to refer to bundles. > > As things stand, the agreed dynamic-configuration and interoperation > requirements appear to be satisfied by the proposed design. We have > several open issues, however, among them #NonHierarchicalLayers, > #MutableConfigurations, and #LazyConfigurationAndInstantiation, arising > from the desire for dynamic modules expressed by David Lloyd. More > recently, Thomas Watson reported on his attempt to achieve bidirectional > interoperation between OSGi and JPMS [3], in which he devised a way to > for a layer of JPMS modules to reference a layer of OSGi bundles. For > his approach to be practical, however, he needs to be able to discard > sub-graphs of modules within a layer so that they and their class loaders > can be reclaimed earlier than the layer itself (#DiscardableModules [4]). > > At this stage we could proceed to adopt dynamic modules and bidirectional > interoperation as two additional requirements, to be supported directly > by the module system. Doing so would, however, lead to a very different > and more complex design, with correspondingly complex implementations, > since it would need to provide all of the features mentioned above, and > quite likely more. As I've argued elsewhere this would, in my judgement, > make it very difficult -- if not impossible -- to achieve the stated > goals of this JSR, most importantly the goal that the resulting module > system be approachable by Java developers of all skill levels. > > Fortunately, I don't think we need to go that far. > > Rather than move toward a design that supports these requirements > directly we can instead support them with the present design indirectly, > by modeling a dynamic module as a sequence of JPMS modules over time. > > This key idea is latent in Watson's OSGi adaptation, in which he models > OSGi bundle wirings -- rather than bundles themselves -- as JPMS modules. > When bundles are installed, updated, or uninstalled then he synthesizes > new JPMS modules for the resultant new wirings and configures them into > a single new JPMS layer, using the new class loaders created by the OSGi > framework itself. That _bundle layer_ refers to the previous bundle > layer, if any, so that over time a stack of bundle layers builds up in > which the topmost layer always represents the current bundle wirings. > > JPMS modules can be configured and resolved in additional layers above > the current OSGi bundle layer. If a bundle upon which a JPMS module > depends is updated then the layer that contains that JPMS module is > discarded and a new layer is instantiated, configured to refer to the > new topmost bundle layer and (I assume) from the same set of root JPMS > modules as the original. > > A big problem with this scheme, as Watson points out, is that the JPMS > modules corresponding to discarded OSGi bundle wirings can be retained > in memory for a long time, along with their class loaders, class objects, > and related data structures. That's because every bundle layer remains > in the stack of bundle layers, and that stack is not discarded until the > OSGi framework itself is stopped and discarded (JPMS-ISSUE-007 in [3]). > > Watson suggests solving this problem by allowing sub-graphs of modules > within a layer to be discarded (#DiscardableModules [4]). That would, > however, require layers to mutable, whereas today they are not. That > layers are immutable brings a number of advantages, including a simpler > conceptual model, a stronger degree of reliable configuration, and safer > implementations. > > Here, then, is an alternative approach: > > - Map each bundle wiring to a unique JPMS module, configured and > instantiated in its own unique bundle-wiring layer. An OSGi bundle > will then be associated not only with a sequence of bundle wirings > over time, but also with a sequence of JPMS module descriptors, > configurations, modules, and layers over time. A layer can be > reclaimed after no strongly-reachable layers refer to it, so > discarded bundle-wiring layers -- and their class loaders -- won't > be retained unnecessarily. There's no need to represent multiple > versions of the same bundle within a single layer, since every > bundle wiring has its own layer (part of JPMS-ISSUE-006 in [3]). > > - Extend JPMS with #NonHierarchicalLayers, so that a set of JPMS > modules can be resolved against modules in more than one parent > layer. An ordinary JPMS layer can then have multiple bundle-wiring > layers and ordinary JPMS layers as parents. > > To illustrate, here's a diagram that elaborates upon the second diagram > in Watson's note: > > http://cr.openjdk.java.net/~mr/jigsaw/notes/osgi-on-jpms.png > > The left-hand side shows a layer containing two ordinary JPMS modules, > jpms.a and jpms.b, resolved against two OSGi bundle-wiring layers, for > bundle.a and bundle.b. (The blue boxes represent JPMS modules; the > orange boxes are the class loaders created by the OSGi framework.) The > right-hand side depicts some later time after which bundle.b has been > updated, so its original JPMS module and layer were discarded and a new > module and layer were created for it (bundle.b layer 2). The JPMS module > layer had the original bundle.b layer as one of its parents so it, too, > was discarded and a new JPMS layer was created with the second bundle.b > layer -- along with the original, unchanged bundle.a layer -- as its > parents. > > This implementation of OSGi bundles in terms of JPMS modules leverages > the fact that OSGi already associates a bundle with a sequence of bundle > wirings over time, and each new bundle wiring has its own unique class > loader. All we are doing here is wrapping each of these class loaders in > its own module and layer, so that ordinary JPMS modules can be configured > to refer to them. There could be many such layers but these layers are > very lightweight, since their sole function is to associate a single > module with an existing class loader for the purpose of access control. > This approach should generalize to other module systems that support > dynamic modules in a similar fashion, i.e., by creating a new class > loader each time a dynamic module is updated. > > There are, no doubt, further details to work out, such as exactly how to > synthesize the module descriptors for bundle wirings so that they can > sensibly be referenced by ordinary JPMS modules. Cyclic relationships > amongst OSGi bundles and split packages across bundles can't be modeled > directly in module descriptors, but I suspect they can be accommodated by > inserting the necessary additional readability edges after the relevant > layers are instantiated. Overall this does, however, appear to be a > promising approach to supporting dynamic modules and bidirectional > interoperation without adding undue complexity to the platform module > system itself. > > To enable this with the present design we do need #NonHierarchicalLayers; > that's a significant change, but I think it's feasible and will explore > it further. In order to model the cyclic graphs and split packages that > are possible in other module systems it'd be convenient, but not strictly > necessary, to have #ReadabilityAddedByLayerCreator [5]. We have no need > for #MutableConfigurations, #LazyConfigurationAndInstantiation, or > #DiscardableModules, which I think is a big win for simplicity. > > Comments? > > - Mark > > > [1] http://openjdk.java.net/projects/jigsaw/spec/reqs/#dynamic-configuration > [2] http://openjdk.java.net/projects/jigsaw/spec/reqs/#interoperation > [3] > http://mail.openjdk.java.net/pipermail/jpms-spec-comments/2016-August/000062.html, > or http://blog.osgi.org/2016/08/osgi-with-java-modules-all-way-down.html > [4] http://openjdk.java.net/projects/jigsaw/spec/issues/#DiscardableModules > [5] > http://openjdk.java.net/projects/jigsaw/spec/issues/#ReadabilityAddedByLayerCreator From david.lloyd at redhat.com Mon Oct 24 13:46:11 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 24 Oct 2016 08:46:11 -0500 Subject: Proposal: #ReflectiveAccessToNonExportedTypes (revised) & #AwkwardStrongEncapsulation: Weak modules & private exports In-Reply-To: <20161011151514.A04ECD50FA@eggemoggin.niobe.net> References: <20160912150801.AD8AECD07F@eggemoggin.niobe.net> <5b9b1929-eb92-09ab-2997-d7e2395a1d9d@redhat.com> <20161011151514.A04ECD50FA@eggemoggin.niobe.net> Message-ID: <279bd57d-515a-a983-ffbb-6f569c29495e@redhat.com> On 10/11/2016 10:15 AM, mark.reinhold at oracle.com wrote: > 2016/9/16 8:30:41 -0700, david.lloyd at redhat.com: >> After a fairly detailed review of this proposal, we have determined that >> it is not acceptable to Red Hat in its present form. I will list the >> primary problems here, and then I'll start up discussion on jigsaw-dev >> of several possible solutions that could work for us. I'll number these >> in case anyone wants to respond piece-wise. >> >> #1) The "weak" designation appears to be pejorative >> >> Under this solution, many existing frameworks and/or the modules which >> consume them must be marked "weak" in order to work correctly, or else a >> much more complex module descriptor must be used. While this seems to >> be a stepping stone for migration, we find that it rather makes >> middleware appear as a second-class citizen. > > No insult is intended; "weak" is used here purely in a technical sense, > as the obvious antonym of "strong". Suggestions for a better word for > this concept are welcome. It's the concept itself that is problematic, I think. Really, if I understand correctly, the "weak" notion exists merely as shorthand to say "grant permission on all packages to all modules for private reflection", which is the sort of thing that a user ought to be saying explicitly. Many users will use weak modules without understanding what they are; some will come to understand it and immediately press for best practices to get rid of them. I think that regardless of what the concept is named, it's going to spell trouble in the end. Our position is that grants of reflective access could (and perhaps should) be separated completely from exports, though as stated previously, exporting should imply public reflection access merely because it doesn't really make any sense not to do so. >> #2) Weak modules are, in fact, weak >> >> The idea of retaining the Java 8 rules for transition modules appears >> good on the surface, however the problem is that users may be forced to >> use weak modules indefinitely in order to interoperate with certain >> existing middleware and libraries, or else be faced with a potentially >> complex migration. If the security benefits of "strong" modules are >> real, then users will be rightly inclined to move from weak modules to >> strong modules. This can be severely problematic if existing >> middleware, libraries, and large applications cannot easily operate as >> or with "strong" modules or cannot do so in an "easy to learn"/"easy to >> use" fashion. This also means that "weak" modules exist only for the >> purposes of maintaining a transition period - at some point they will >> inevitably transition from "convenience" to "attractive nuisance". > > Weak modules do fall between automatic and strong modules on the overall > migration path but they're not intended, necessarily, to be transitory. > There's nothing wrong with evolving an existing JAR file into a weak > module and stopping there. There's also nothing wrong with creating a > new weak module from scratch, as one might do for a module that contains > nothing but JPA model classes, and never evolving it into a strong > module -- there's no benefit in doing that, so don't do it. I definitely understand the intent, but I don't think intent will truly matter for this case as I've said above. >> #3) Controlling reflection access is always bound up with controlling >> exports >> >> The proposed exports system is simplistic but not exactly simple, yet >> several useful modes remain left unaccounted-for. Internally we >> examined the usefulness of controlling access to public and so-called >> "deep" reflection in combination with whether or not a member was >> exported, and we came up with this logical table: >> >> +------------------+---------------------------+ >> | | Module is... | >> | Reflection is... +------------+--------------+ >> | | Exported | Not Exported | >> +----------------------------------------------+ >> | Forbidden | Not Useful | Useful | >> +----------------------------------------------+ >> | Public Only | Useful | Useful | >> +----------------------------------------------+ >> | Public & Private | Useful | Useful | >> +------------------+------------+--------------+ > > I'm not sure how to read this table. Should the cell containing the text > "Module is ..." instead contain "Member is ..."? Right. > By "exported" I'll assume that you mean non-reflective access, i.e., from > the language at compile time and bytecodes at run time. Yes. > Do you have example use cases for each of the "Useful" cells? Some are > obvious, but others are not. Sure, I'll go over each one by one with some example use cases. Forbidden/Not Exported: Obviously the foundation case for access-oriented encapsulation, this is the default mode. Exported modes: Public Only/Exported: It doesn't make sense to disallow reflection on public types and members that can already be accessed directly, so this would be the default access mode to imported members when one module imports another. Public & Private/Exported: This is the JDK 8 compatibility mode, but a module should have to opt into this mode, and it should be restricted to specific targets whenever possible, else the mechanism is essentially defeated. Not exported modes: Public Only/Not Exported: This is the mode that would be used (for example) by DI and ORM frameworks and containers which operate on public members. Normally a module which imports such a framework would grant the necessary reflective access to it as well (though a full export is in this case unnecessary); the framework wouldn't contain requires clauses for modules which consume it. Like I suggested in my first email, a way to "handshake" this agreement seems like it would be necessary: the framework would not allow itself to be imported without the importing module acknowledging the access in its configuration. Public & Private/Not Exported: This is the mode that would be used by serialization and DI/ORM frameworks and containers which operate on private members (e.g. field-level injection). Like the above though, this level of access is usually granted by the module importing the framework in question and should be explicitly opted into if possible. The better situation however is to separate the grant from the export. >> #4) There isn't always a container >> >> The justification for not providing an easy mechanism to restrict >> private reflective access to a trusted framework is that a container can >> always rewrite a module or inject classes. However there isn't always a >> container present - something which the existence of weak modules >> acknowledges. Some frameworks and middleware are meant to be usable >> with or without a container, and it is undesirable to force these use >> cases into a weak module situation or to require a strong coupling in >> the module descriptor in this case. Some other solution should be found >> that works both for containers and "regular" applications. > > If you're running outside of a container then how important is it to > precisely restrict access to packages meant for deep reflection? If > you're outside of a container then you have to trust the person (you?) > who sets up your module path and/or class path. To take the canonical > case of a module that just contains JPA model classes, if you make it a > weak module or else `exports private` the relevant packages without > qualification so that the hibernate.core (or whatever) module can reflect > into it, then the scope for trouble is limited. Sure, some other module > on the module path or even code on the class path could reflect into your > module when it's not supposed to, but is it worth complicating the module > system just to handle this scenario? In either situation, the person (deployer) who sets up the class path or installation can currently do whatever they want. Nothing in the new (or existing) security framework truly makes it possible to run untrusted code safely in this regard, nor to safely run trusted code in an untrusted environment, nor is it a sufficient solution for secure sandboxing even with a security manager present, with or without a container. At best it's a mitigation strategy. That said: the rise of microservices and reactive programming suggests that such standalone (containerless) deployments are, if anything, more popular than ever, but they may suffer from security problems which are very similar to, if not identical to, those which applications running in containers are subject to. In fact jlink (and the new AOT work for that matter) seem particularly and specifically suited to this type of use case. Finally: a JPA model class module is indeed probably harmless in any practical security sense, and locking it down is probably not useful from a security perspective, but even in this case it may yet be useful to restrict package exports such that only _supported_ classes are visible (especially when a module is published within an organization and contracts must be maintained); i.e. encapsulation is not solely a security concern, it also establishes intent as clearly as any interface declaration does. > Don't get me wrong: Strong encapsulation is generally a very good thing. > It's not always absolutely necessary, however, and we need to balance the > number of ways in which it can be used against the overall complexity of > the design. > > (It looks like there was no #5 in your list.) Another genius stroke of... um, well, yeah. >> #6) Not all reflection is 100% "friendly", and that's OK >> >> Serialization frameworks generally need private reflective access to the >> module which contains the classes being serialized. For some >> frameworks, this access will already be adequate. Some existing >> frameworks utilize Unsafe to create uninitialized instances. However >> other frameworks, including frameworks that seek to comply with the >> Serialization Specification [1], use ReflectionFactory, as that class >> provides (exclusively, as far as I know) the ability to acquire a >> constructor that produces a _partially_-initialized class where only the >> constructors of superclasses to a certain depth are called. It is our >> view that it would be a regression to force such frameworks to use >> --add-exports or JVMTI, and using Unsafe for this purpose, or continuing >> to use it, (as I understand it) puts at risk the ability to perform >> optimizations which depend on entry point control, which I am given to >> understand are a strong motivator for many of the design constraints >> that exist on the Jigsaw implementation. It should be possible for >> these frameworks to continue to use ReflectionFactory (unsupported >> though it may be) if suitably privileged (in the security manager >> sense), as long as all of the module(s) upon which the framework >> reflects have granted private reflective access to it (in the module >> declaration sense). > > So far as I know, this is already the case. The `ReflectionFactory` API > is specifically retained in JDK 9 for this very purpose, per JEP 260 [1]. > Do you have an example for which this does not work? No, I had just interpreted previous messages as raising the possibility of closing this off so I wanted to capture the requirement. Since this is not the case though, that's one less thing to worry about. The recent enhancements to this class have also gone a long way to ease my concerns on this. >> #7) More loose coupling seems necessary and useful >> >> In order for typical applications to function with modern middleware as >> modules, without compromising security, it may be necessary to enhance >> the loose coupling mechanism (uses/provides), or to provide an >> additional, similar mechanism, which allows a symbolic coupling which >> would allow modules to declare (in an abstract manner) modules which >> need to have reflective access to it in a "friendly" manner. >> Implementation ideas are forthcoming on jigsaw-dev. >> >> Providing a way within the system to grant public and/or private >> reflection access in a specific manner, and doing so in a manner which >> is easy to use and not prejudicial against existing or future middleware >> and small or large applications which consume such middleware in the >> Java SE or EE space, is our most basic requirement for satisfactory >> resolution of this issue. > > To confirm my understanding, would it be correct to say that your main > concern here is scenarios in which some of the classes in a user module > are intended to be manipulated reflectively by some trusted framework > module, but that framework module is either not known at compile time or > is otherwise inconvenient to name in the declaration of the user module? Correct. We have observed over the past several years that the arrangement and structure of modules at deployment/run time generally diverges substantially from the structure of modules at compile time. In order for compilation to have any kind of regularity, decoupling API from implementation wherever possible seems like a necessity, especially when you consider that JSR and other standards will want to be able to publish a simple contract for interfacing which is agnostic to implementation. So really you can extend this notion beyond reflection; it's just that reflection is a particularly interesting case of the more general requirement. In a practical sense, the most useful unit of coupling between modules is ABI. When you depend on a module, that is really what you're relying on: an ABI which adheres to some contract. Therefore it is useful to have names which correspond to ABIs as well as implementations (whether this is the same or a separate namespace doesn't really matter from this perspective). >> I will follow up on jigsaw-dev with some >> implementation thoughts and more specific requirements. > > Discussions of issues in the proposed design, and specific proposals > for changing the design, belong here on the EG list. The jigsaw-dev > list is meant primarily for the discussion of issues outside the scope > of the JSR such as implementation details, the design and implementation > of JDK-specific APIs and tools, and feedback and questions arising from > practical use of the prototype implementation. > > - Mark > > > [1] http://openjdk.java.net/jeps/260 > -- - DML From tjwatson at us.ibm.com Mon Oct 24 19:40:06 2016 From: tjwatson at us.ibm.com (Thomas Watson) Date: Mon, 24 Oct 2016 14:40:06 -0500 Subject: Dynamic modules and bidirectional interoperation In-Reply-To: <1316230997.272548.1477056278273.JavaMail.zimbra@u-pem.fr> References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net> <1316230997.272548.1477056278273.JavaMail.zimbra@u-pem.fr> Message-ID: Can you explain more about how monkey patching would be done in a JVM implementation neutral way? If I understand correctly your suggestion would require monkey patching in order to add functionality to the module classes provided by the boot layer (Module, Layer, Configuration etc.). This additional functionality would allow an OSGi Framework implementation to represent the dynamic set of bundle wirings in an OSGi Framework as JPMS modules. In order to do that it seems we would need extensive knowledge of the internal implementation details of the classes we are monkey patching in order to modify their internal structures. I can see how using Unsafe would give us the power to hack into the implementation details of jigsaw, but I don't view that as a satisfactory solution to the problem. It would bind us to a particular internal implementation of Java 9 in order to work correctly. It places a new dependency on Unsafe class which is from the jdk.unsupported module. I don't think new dependencies on the jdk.unsupported module should be advocated. Doing so has a high probability of breaking on a future version of Java if Unsafe is removed by default or the implementation details of the module classes change. Tom "jpms-spec-observers" wrote on 10/21/2016 08:24:38 AM: > From: Remi Forax > To: mark reinhold > Cc: jpms-spec-experts at openjdk.java.net > Date: 10/21/2016 08:25 AM > Subject: Re: Dynamic modules and bidirectional interoperation > Sent by: "jpms-spec-observers" > > Hi Mark, > I see an other way to solve the wiring issue, > use VM anonymous classes (via unsafe.defineAnonymousClass) to > generate the glue code (as lambda meta factory does to implement the > functional interface at runtime) and method handles as wires (once > created a method handle do not do any security check anymore, so the > requirement to be in a module with specific parents can be lifted). > With that, you don't need to generate the wiring code inside a new > module and when a new bundle need to be re-wire, you can mutate the > wire (the method handle) by using a mutable callsite. > > Said differently, loading/unloading an OSGI bundle is equivalent to > doing monkey patching in a dynamic language, so there is no reason > to not use the tools we provide for dynamic languages to wire OSGI bundles. > > so i vote strongly against that proposal mostly because it's too > late to change requirements and because some obvious design options > to implement OSGI on top of JPMS have not been tested. > > regards, > R?mi > > ----- Mail original ----- > > De: "mark reinhold" > > ?: jpms-spec-experts at openjdk.java.net > > Envoy?: Jeudi 6 Octobre 2016 19:07:59 > > Objet: Dynamic modules and bidirectional interoperation > > > The agreed requirements for this JSR mandate a modest degree of dynamic > > configuration [1]. They do not mandate _dynamic modules_, i.e., the > > fine-grained dynamic configuration or "hot deployment" that is possible > > with other module systems, such as OSGi and JBoss, in which individual > > modules can be installed, updated, and uninstalled at any time. > > > > The requirements also mandate that it "be possible for another module > > system, such as OSGi, to locate Java modules and resolve them using its > > own resolver, except possibly for core system modules" [2]. They do not > > mandate _bidirectional interoperation_, which in the case of OSGi would > > be the ability to arrange for bundles to refer to JPMS modules and also > > vice versa, i.e., for JPMS modules to refer to bundles. > > From mark.reinhold at oracle.com Tue Oct 25 21:08:49 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 25 Oct 2016 14:08:49 -0700 Subject: Please welcome Neil Bartlett to JSR 376 Message-ID: <20161025140849.350853791eggemoggin.niobe.net> Neil Bartlett, a recognized expert on OSGi, has asked to join this EG. Ordinarily I'm extremely reluctant to accept new EG members this late in the life of a JSR, long after we've settled the requirements and after we've been working on a design for over a year. Peter Kriens's recent departure, however, leaves the EG with no resident OSGi expertise, and that's an important gap to fill. Neil has agreed to accept the requirements as they stand and, further, acknowledges that our primary goal at this stage is to finish the design that we have rather than come up with something significantly different. I'm confident that he'll bring a valuable perspective to our discussions and help to ensure the overall success of the JSR. (Neil -- if there's anything further you'd like to say by way of introduction, please do so!) - Mark From Tim_Ellison at uk.ibm.com Tue Oct 25 21:22:58 2016 From: Tim_Ellison at uk.ibm.com (Tim Ellison) Date: Tue, 25 Oct 2016 22:22:58 +0100 Subject: Please welcome Neil Bartlett to JSR 376 In-Reply-To: <20161025140849.350853791eggemoggin.niobe.net> References: <20161025140849.350853791eggemoggin.niobe.net> Message-ID: mark.reinhold at oracle.com wrote on 25/10/2016 22:08:49: > Neil Bartlett, a recognized expert on OSGi, has asked to join this EG. > > Ordinarily I'm extremely reluctant to accept new EG members this late in > the life of a JSR, long after we've settled the requirements and after > we've been working on a design for over a year. Peter Kriens's recent > departure, however, leaves the EG with no resident OSGi expertise, and > that's an important gap to fill. > > Neil has agreed to accept the requirements as they stand and, further, > acknowledges that our primary goal at this stage is to finish the design > that we have rather than come up with something significantly different. > I'm confident that he'll bring a valuable perspective to our discussions > and help to ensure the overall success of the JSR. > > (Neil -- if there's anything further you'd like to say by way of > introduction, please do so!) Welcome Neil! Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU From david.lloyd at redhat.com Tue Oct 25 21:49:30 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 25 Oct 2016 16:49:30 -0500 Subject: Please welcome Neil Bartlett to JSR 376 In-Reply-To: <20161025140849.350853791eggemoggin.niobe.net> References: <20161025140849.350853791eggemoggin.niobe.net> Message-ID: <0d21c865-414d-0593-f8c1-0c51bdd416f6@redhat.com> On 10/25/2016 04:08 PM, mark.reinhold at oracle.com wrote: > Neil Bartlett, a recognized expert on OSGi, has asked to join this EG. > > Ordinarily I'm extremely reluctant to accept new EG members this late in > the life of a JSR, long after we've settled the requirements and after > we've been working on a design for over a year. Peter Kriens's recent > departure, however, leaves the EG with no resident OSGi expertise, and > that's an important gap to fill. > > Neil has agreed to accept the requirements as they stand and, further, > acknowledges that our primary goal at this stage is to finish the design > that we have rather than come up with something significantly different. > I'm confident that he'll bring a valuable perspective to our discussions > and help to ensure the overall success of the JSR. > > (Neil -- if there's anything further you'd like to say by way of > introduction, please do so!) Welcome aboard, Neil! -- - DML From mark.reinhold at oracle.com Wed Oct 26 16:15:45 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 26 Oct 2016 09:15:45 -0700 Subject: Dynamic modules and bidirectional interoperation In-Reply-To: References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net> Message-ID: <20161026091545.152155830eggemoggin.niobe.net> 2016/10/7 8:47:53 -0700, Thomas Watson , jpms-spec-observers: > Overall I think this approach with #NonHierarchicalLayers is worth > pursuing. Glad to hear it! > A couple of observations. > > 1) Any OSGi dependencies that influence class loader delegation will not > be declared in the ModuleDescriptors which represent the OSGi bundle > wirings. Instead we will depend on the ability to > #ReadabilityAddedByLayerCreator to wire up the modules within the bundle > Layer according to OSGi resolution rules. It has a drawback if there are > cases where some system code is trying to reason about modular > dependencies by using the JPMS module API. I think this will have to be > acceptable because I don't think we will be able to fully converge on all > the possible dependency types available in the existing module systems. > For example, OSGi has Require-Bundle which maps closing to JPMS requires, > but OSGi also has Import-Package dependencies which cannot be mapped > properly into JMPS module Layers. If by "system code" you mean code in the JDK itself, I don't think this will be a big problem at run time. Except for diagnostic purposes, module descriptors are really only interpreted during resolution and configuration. All other run-time reasoning about module readability is based upon the reads edges in the module graph as reported by the `java.lang.reflect.Module::canRead` method, and that will include any reads edges that you add to support bundle wirings. > 2) I assume #NonHierarchicalLayers will be restricted to form a directed > acyclic graph. Yes. > As such I assume each node in the graph of Layers would be > visited only once when resolving new Layers on top even if there are > multiple paths to the layer from the new Layer being resolved. Yes. - Mark From mark.reinhold at oracle.com Wed Oct 26 17:24:47 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 26 Oct 2016 10:24:47 -0700 Subject: Dynamic modules and bidirectional interoperation In-Reply-To: References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net> <20161026091545.152155830eggemoggin.niobe.net> Message-ID: <20161026102447.63677664eggemoggin.niobe.net> 2016/10/26 9:32:42 -0700, Thomas Watson , jpms-spec-comments: > 2016/10/26 09:15:45 -0700, mark.reinhold at oracle.com, jpms-spec-experts: >> 2016/10/7 8:47:53 -0700, Thomas Watson , jpms-spec-observers: >>> ... >>> >>> 1) Any OSGi dependencies that influence class loader delegation will not >>> be declared in the ModuleDescriptors which represent the OSGi bundle >>> wirings. Instead we will depend on the ability to >>> #ReadabilityAddedByLayerCreator to wire up the modules within the bundle >>> Layer according to OSGi resolution rules. It has a drawback if there are >>> cases where some system code is trying to reason about modular >>> dependencies by using the JPMS module API. I think this will have to be >>> acceptable because I don't think we will be able to fully converge on all >>> the possible dependency types available in the existing module systems. >>> For example, OSGi has Require-Bundle which maps closing to JPMS requires, >>> but OSGi also has Import-Package dependencies which cannot be mapped >>> properly into JMPS module Layers. >> >> If by "system code" you mean code in the JDK itself, I don't think this >> will be a big problem at run time. Except for diagnostic purposes, >> module descriptors are really only interpreted during resolution and >> configuration. All other run-time reasoning about module readability >> is based upon the reads edges in the module graph as reported by the >> `java.lang.reflect.Module::canRead` method, and that will include any >> reads edges that you add to support bundle wirings. > > Here I am using "system code" in the general sense, but I think any > "system code" that cares about wiring will need to pay attention to > Module::canRead like you suggest. I can think of a few frameworks that > care about how modules are wired up at runtime. For example, in OSGi we > have the extender pattern. Without going into much detail here, basically > an extender implementation module will only act on extendee modules which > are wired to the extender module. If such a pattern exists using JPMS > then the "system code" would be the extender implementation. Okay, I see. >>> 2) I assume #NonHierarchicalLayers will be restricted to form a directed >>> acyclic graph. >> >> Yes. >> >>> As such I assume each node in the graph of Layers would be >>> visited only once when resolving new Layers on top even if there are >>> multiple paths to the layer from the new Layer being resolved. >> >> Yes. > > The motivation for my question was to ask some follow-ons. > > Will there will be a specified order in which the nodes are traversed > when there are multiple paths to a node? More specifically will the > parents have a specified order in which they are delegated to? Yes, the parents of a new configuration will be specified in an ordered list and during resolution they'll be searched depth-first, in the order in which they occur in the list. Depth-first order makes it easy to reason about resolution, since it works just like a search path. - Mark From mark.reinhold at oracle.com Wed Oct 26 20:15:24 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 26 Oct 2016 13:15:24 -0700 Subject: Dynamic modules and bidirectional interoperation In-Reply-To: References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net> Message-ID: <20161026131524.319958580eggemoggin.niobe.net> 2016/10/7 8:47:53 -0700, tjwatson at us.ibm.com: > Overall I think this approach with #NonHierarchicalLayers is worth > pursuing. A couple of observations. > > 1) Any OSGi dependencies that influence class loader delegation will not > be declared in the ModuleDescriptors which represent the OSGi bundle > wirings. Instead we will depend on the ability to > #ReadabilityAddedByLayerCreator to wire up the modules within the bundle > Layer according to OSGi resolution rules. A question for you about #ReadabilityAddedByLayerCreator ... In this approach, will the code that creates a per-bundle layer be able to specify all of the required reads edges at the time at which that layer is created? Or will it need to be able to add reads edges at one or more later times? If the answer is the former then the resulting API is likely to be much simpler. - Mark From tjwatson at us.ibm.com Wed Oct 26 20:46:07 2016 From: tjwatson at us.ibm.com (Thomas Watson) Date: Wed, 26 Oct 2016 15:46:07 -0500 Subject: Dynamic modules and bidirectional interoperation In-Reply-To: <20161026131524.319958580eggemoggin.niobe.net> References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net> <20161026131524.319958580eggemoggin.niobe.net> Message-ID: > From: mark.reinhold at oracle.com > To: Thomas Watson/Austin/IBM at IBMUS > Cc: jpms-spec-experts at openjdk.java.net > Date: 10/26/2016 03:15 PM > Subject: Re: Dynamic modules and bidirectional interoperation > > 2016/10/7 8:47:53 -0700, tjwatson at us.ibm.com: > > Overall I think this approach with #NonHierarchicalLayers is worth > > pursuing. A couple of observations. > > > > 1) Any OSGi dependencies that influence class loader delegation will not > > be declared in the ModuleDescriptors which represent the OSGi bundle > > wirings. Instead we will depend on the ability to > > #ReadabilityAddedByLayerCreator to wire up the modules within the bundle > > Layer according to OSGi resolution rules. > > A question for you about #ReadabilityAddedByLayerCreator ... > > In this approach, will the code that creates a per-bundle layer be > able to specify all of the required reads edges at the time at which > that layer is created? Or will it need to be able to add reads edges > at one or more later times? > > If the answer is the former then the resulting API is likely to be > much simpler. > > - Mark > For static wires we could provide the read edges up front at Layer creation. The vast majority of wires would be static, but in OSGi we also have something called dynamic package resolution. Dynamic package resolution would happen after layer creation. Unfortunately the dynamic package resolution could result in the need to add a new read edge to a previously unknown module. I suspect there will be an ordering issue for static wires though. The proposal is to have a triplet for each Layer->Module->BundleWiring. I'm not sure what you have in mind for inputs to layer creation, but if the static mapping is needed at layer creation (aka Function>) then each Layer->Module->BundleWiring triplet will need to be created in a specific order to ensure all required modules are created to satisfy the call to the mapping function. This will also prevent the support for cycles which are allowed in OSGi resolution. I would prefer a dynamic API here. Tom From mark.reinhold at oracle.com Wed Oct 26 21:47:41 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 26 Oct 2016 14:47:41 -0700 Subject: Dynamic modules and bidirectional interoperation In-Reply-To: References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net> <20161026131524.319958580eggemoggin.niobe.net> Message-ID: <20161026144741.136621558eggemoggin.niobe.net> 2016/10/26 13:46:07 -0700, Thomas Watson : > 2016/10/26 13:15:24 -0700, mark.reinhold at oracle.com: >> ... >> >> In this approach, will the code that creates a per-bundle layer be >> able to specify all of the required reads edges at the time at which >> that layer is created? Or will it need to be able to add reads edges >> at one or more later times? >> >> If the answer is the former then the resulting API is likely to be >> much simpler. > > For static wires we could provide the read edges up front at Layer > creation. The vast majority of wires would be static, but in OSGi we also > have something called dynamic package resolution. Dynamic package > resolution would happen after layer creation. Unfortunately the dynamic > package resolution could result in the need to add a new read edge to a > previously unknown module. > > I suspect there will be an ordering issue for static wires though. The > proposal is to have a triplet for each Layer->Module->BundleWiring. I'm > not sure what you have in mind for inputs to layer creation, but if the > static mapping is needed at layer creation (aka Function Collection>) then each Layer->Module->BundleWiring triplet will > need to be created in a specific order to ensure all required modules are > created to satisfy the call to the mapping function. This will also > prevent the support for cycles which are allowed in OSGi resolution. I > would prefer a dynamic API here. Okay, between dynamic package resolution and readability cycles it seems that you do need a dynamic API. Thanks for the quick answer. - Mark From mark.reinhold at oracle.com Thu Oct 27 15:52:36 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 27 Oct 2016 08:52:36 -0700 (PDT) Subject: Proposal: #ReflectiveAccessToNonExportedTypes (revised) & #AwkwardStrongEncapsulation: Weak modules & private exports In-Reply-To: References: <20160912150801.AD8AECD07F@eggemoggin.niobe.net> <617353007.2424567.1474033457532.JavaMail.zimbra@u-pem.fr> <20161011151414.9D5E9D50F5@eggemoggin.niobe.net> Message-ID: <20161027155236.D7069DA324@eggemoggin.niobe.net> 2016/10/12 1:45:16 -0700, Tim Ellison : > mark.reinhold at oracle.com wrote on 11/10/2016 16:14:14: >> ... >> >> I've read Colebourne's proposal [1], and I can see its intuitive appeal >> to developers (like us!) who are used to thinking about all these things. >> To a new developer it might be easier to explain his `exposes *` than the >> above `exports private *`, but I think explaining weak modules is even >> easier. A new developer is likely to hear the "reliable configuration + >> strong encapsulation" mantra fairly early on, and weak modules are simply >> the former, without the latter. > > I disagree. I like the Colebourne's approach because it keeps to two > declarations ('exports' and 'exposes') as orthogonal concepts. However, > I don't see the need to open the syntax up for wildcards '*' at this time. > >> A couple of other problems I see with Colebourne's approach: >> >> - The only way to allow an exported API package to be used reflectively >> is to expose it for deep reflection. That requires typing another >> directive and, worse, makes the internals of that package available >> for deep reflection, which is probably not what was intended. (For the record, Colebourne later pointed out that this is not what he intended [1].) >> - The interactions between `exports`, `exposes`, and the qualified >> forms of these directives turns out to be pretty complicated once you >> work out all the cases. (We've done this internally, for a somewhat >> similar proposal that allows both `private` and `dynamic` modifiers >> on the `exports` directive.) They wind up being nearly as bad as the >> existing rules for protected members [2]. I'm not convinced that >> this relatively small bit of expressive power is worth the additional >> complexity. > > It would be interesting to see that exploration exposed, because as you > say the Colebourne expression appears intuitive. On further examination my second statement above turns out not to be true, especially if you simplify things by avoiding wildcards. > As discussed, it would be useful to explore the options for changing the > proposal's constraint that you can't "expose" a package for reflection > without also exporting it. I've written up a new proposal intended to address this which I'll post in a moment, along with an analysis of the proposals presented thus far. - Mark [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-October/009638.html From mark.reinhold at oracle.com Thu Oct 27 15:53:36 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 27 Oct 2016 08:53:36 -0700 (PDT) Subject: Proposal (revised^2): #ReflectiveAccessToNonExportedTypes & #AwkwardStrongEncapsulation: Open modules & open packages Message-ID: <20161027155336.DBCE7DA329@eggemoggin.niobe.net> Issue summary ------------- #ReflectiveAccessToNonExportedTypes --- Some kinds of framework libraries require reflective access to members of the non-exported types of other modules; examples include dependency injection (Guice), persistence (JPA), debugging tools, code-automation tools, and serialization (XStream). In some cases the particular library to be used is not known until run time (e.g., Hibernate and EclipseLink both implement JPA). This capability is also sometimes used to work around bugs in unchangeable code. Access to non-exported packages can, at present, only be done via command-line flags, which is extremely awkward. Provide an easier way for reflective code to access such non-exported types. [1] #AwkwardStrongEncapsulation --- A non-public element of an exported package can still be accessed via the `AccessibleObject::setAccessible` method of the core reflection API. The only way to strongly encapsulate such an element is to move it to a non-exported package. This makes it awkward, at best, to encapsulate the internals of a package that defines a public API. [2] Proposal -------- (This is the third proposal for #ReflectiveAccessToNonExportedTypes. The note that follows [3] summarizes the background and relates the alternatives presented thus far.) Extend the language of module declarations with the concept of _open_ modules. Open modules make it easy to define modules whose internals will be accessed at run time by reflection-based frameworks. Every package in an open module has the following properties, by default: (1) At compile time it is not exported. (2) At run time it is exported without regard to split-package conflicts or other inconsistencies, i.e., it does not affect the operation of the resolver, nor the construction of layer configurations, but is simply exported as if by invoking the `Module::addExports` method during layer instantiation. (3) At run time it is available for _deep reflection_, i.e., all of its elements are accessible via the core reflection API including non-public elements, which can be accessed via the `AccessibleObject::setAccessible` method. A package in an open module can be exported explicitly, for use at compile time, via the familiar `exports` directive, so property (1) will not hold. In that case it will also partake in run-time resolution, so property (2) will not hold either. Suppose we have a module `foo.bar` that contains an API package `com.foo.bar`, whose public types are intended for use by other modules, and a non-API package `com.foo.bar.model`, that contains entity classes to be manipulated by Hibernate via core reflection. Then the module declaration open module foo.bar { exports com.foo.bar; requires hibernate.core; requires hibernate.entitymanager; } makes all elements, public and otherwise, of all packages available for deep reflection at run time, but makes only the public and protected types in `com.foo.bar` accessible at compile time. Thus Hibernate can access non-public elements of the `com.foo.bar.model` package via the `setAccessible` method, and some other module that `requires foo.bar` can be compiled against the API in the `com.foo.bar` package but not against the types in any other package. Open modules can be considered an intermediate step on the migration path, between automatic modules and normal modules: - An automatic module offers the traditional level of encapsulation: All packages are both open for deep reflective access and exported for ordinary compile-time and run-time access to their public types. - An open module offers a modest degree of stronger encapsulation: All packages are open for deep reflective access, but the module's author must specify which packages, if any, are exported for ordinary compile-time and run-time access. - A normal module offers the strongest encapsulation: The module's author must specify which packages, if any, are open, or exported, or both. An open module is a good starting point for application code that's being modularized by its author. It affords a separation of concerns, so that the author can focus on defining the module's domain-specific API without having to worry about how the module will be inspected or manipulated at run time by reflective frameworks. * * * To open specific packages in normal module declarations we introduce a new per-package directive, `opens`. A package in a normal module can be opened, or exported, or both, as follows: - If a package is only opened (`opens p;`) then properties (1), (2), and (3) hold, i.e., its types are not accessible at compile time, it does not affect resolution, and all of its elements are available for deep reflection at run time. - If a package is only exported (`exports p;`) then those properties do not hold, and it is exported exactly as it is today. - If a package is both exported and opened (`exports p; opens p;`) then it is exported as today and is, additionally, available for deep reflection at run time (3). We can use the `opens` directive to refine the previous example so that only the `com.foo.bar` package is accessible at compile time, only the `com.foo.bar.model` package is available for deep reflection at run time, and the non-public elements of all other packages are strongly encapsulated: module foo.bar { exports com.foo.bar; opens com.foo.bar.model; requires hibernate.core; requires hibernate.entitymanager; } It's possible to both open and export a package in a normal module, but it's often inadvisable. This is especially so for API packages, since normally an API's internal implementation details should be strongly encapsulated. This combination of directives may, however, be useful for legacy APIs whose internals are known to be accessed by existing code. To ensure the integrity of the platform we expect that all the modules of the JDK itself will be normal modules and that very few, if any, packages will be opened. The `opens` directive cannot be used in an `open` module, since all packages in such modules are implicitly open. * * * Both the `exports` and `opens` directives can be qualified, so that a package is exported or opened only to certain other named modules. As before, duplicate directives are not permitted in order to ensure easy readability. At most one `exports` directive is relevant to any given package, and at most one `opens` directive is relevant to any given package. There is no syntax for wildcards. If the defaults are not sufficient for a package then the package must be named explicitly. * * * The existing syntax of `requires public` has long been confusing, so we here take the opportunity to fix that problem by renaming the `public` modifier in `requires` directives to `transitive`. Thus the declaration module foo.bar { exports com.foo.bar; requires public java.sql; } is now written module foo.bar { exports com.foo.bar; requires transitive java.sql; } This is potentially confusing in a different way, since in mathematics the term "transitive" is usually applied to an entire relation rather than to three specific elements of a set. Its use here does not, in particular, mean that the resolver does not interpret plain `requires` directives when computing the transitive closure of a set of root modules. "Transitive" as used here is in the more abstract sense, expressing the notion of conveying a property -- in this case, the readability of the required module -- from one thing to another. On balance, however, `transitive` does appear superior to `public`. Notes ----- - This proposal is significantly different from both the first and second proposals [4][5]. It recognizes that, in practice, it will be common to have open modules, all of whose content is available for deep reflection but otherwise inaccessible at compile time unless explicitly exported. If finer-grained control over whether a package is opened, exported, or both is required then a normal module declaration can be used. - This proposal is similar to one made by Stephen Colebourne [6], in which he uses the term `exposes`, and to various unpublished proposals (of which there have been many!). We here adopt the term `opens` rather than `exposes` in order to avoid confusion between the phonetically-similar `exports` and `exposes`, especially for non-native speakers of English. We also use `open` to modify the `module` keyword rather than introduce wildcards for the `opens` directive, since the former is easier to explain. The latter would, further, lead people to expect wildcards for the `exports` directive, but we'd prefer to avoid that since it could encourage the careless exportation of all of a module's packages. Exporting a package for use as an API should always be an explicit, affirmative choice. - A separate `opens` directive, and the avoidance of wildcards, simplifies the interactions amongst qualified and unqualified package-level directives. We no longer need complex rules such as those shown for the `private` modifier in the previous proposal [5]. - This proposal implies corresponding changes to the `ModuleDescriptor` and `Module` APIs, but these are reasonably straightforward. - This proposal amends the proposal for #ResourceEncapsulation [6] to replace the use of private exports with open modules and open packages. Wherever that proposal indicates that a resource can be located when its effective package name identifies a package that's exported privately, assume instead that the resource can be located when that package is either an element of an open module or an open package in a normal module. - If a container is to ensure that a package in an application module is available for deep reflection only by a trusted framework module then it can arrange for that by rewriting that module's descriptor, as suggested previously [8], to insert the appropriate qualified `opens` directive. This works even in the case, pointed out by Jason Greene [9], of two modules that open a package of the same name to the same framework module, since opened and unexported packages are not subject to the usual package-consistency checks. There is no need any longer for the resolution algorithm to take this type of scenario into account [a]. A more flexible way to arrange for such precisely-constrained access is a separate issue and may be addressed by a future proposal. - This proposal primarily addresses "friendly" uses of reflection, such as dependency injection and persistence, in which the author of a module knows in advance that one or more, or possibly all, packages must be opened at run time for deep reflective access by frameworks. Intrusive access to arbitrary packages of arbitrary modules by, e.g., serialization frameworks or debugging tools, will still require the use of sharp knives such as `--add-exports` or `--add-opens` command-line options, the legacy unsupported `sun.misc.Unsafe` API and related APIs, or JVM TI. - Using `--add-exports` or `--add-opens` command-line options, or their equivalents, remains awkward, but sometimes they're the only way out. To ease migration it's worth considering some way for an application packaged as a JAR file to include such options in its `MANIFEST.MF` file, as suggested by Simon Nash [b]. This is addressed in the proposal for #AddExportsInManifest [c], which is hereby amended to rename the `Add-Exports-Private` attribute to `Add-Opens`. [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ReflectiveAccessToNonExportedTypes [2] http://openjdk.java.net/projects/jigsaw/spec/issues/#AwkwardStrongEncapsulation [3] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000431.html [4] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-June/000307.html [5] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-September/000390.html [6] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-September/000392.html [7] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-September/009370.html [8] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008637.html [9] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008641.html [a] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008727.html [b] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-December/005745.html [c] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-September/000391.html From mark.reinhold at oracle.com Thu Oct 27 15:54:36 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 27 Oct 2016 08:54:36 -0700 (PDT) Subject: Varieties of reflective access Message-ID: <20161027155436.E080EDA32D@eggemoggin.niobe.net> This note categorizes the different kinds of inter-module access operations relevant to #ReflectiveAccessToNonExportedTypes, analyzes the possible combinations, and then uses those combinations to relate the presented proposals to each other. Categories of inter-module access --------------------------------- Access operations from code in one module to types in another module can be divided into three categories (for simplicity we ignore the details of protected elements): - _Static access_ is ordinary access at compile time via language constructs, and at run time via constant-pool resolution. (This may include access from bytecodes generated at run time.) Only public elements in exported packages can be accessed. An API package intended to be used by other modules should be exported for static access. - _Shallow reflective access_ is access at run time via the Core Reflection API (`java.lang.reflect`) without using `setAccessible`. Only public elements in exported packages can be accessed. - _Deep reflective access_ is access at run time via the Core Reflection API, using the `setAccessible` method to break into non-public elements. Any element, whether public or not, in any package, whether exported or not, can be accessed. Deep reflective access implies shallow reflective access. A key difference between static access and either kind of reflective access is that, for static access, the module system ensures reliable configuration. If the types in a package are accessible statically then the post-resolution consistency checks are in full force, so there are no duplicate or conflicting packages. If only reflective access is provided then the relevant readability relationships may have been established only at run time, so there could be split packages and, also, cycles in the run-time readability graph. Access via constant-pool resolution at run time without reliable configuration is possible, but only in the presence of class loaders that do not delegate to each other in accordance with the readability graph constructed by the resolver. The class loaders provided by the module system itself always delegate correctly. If custom loaders provided by, e.g., a container, do not delegate correctly then chaos is likely to ensue -- and that's not something that the module system can prevent. Method handles and method-handle lookup objects provide static access, checked at the point of creation rather than the point of use. As such, the notions of shallow vs. deep reflective access don't appear to be relevant. In the long term we want to move toward a world in which clients of frameworks use such lookup objects to grant the access to internals that is needed by those frameworks. Whether that's practical remains to be seen; if it proves not to be so then the shallow vs. deep dichotomy could be leveraged to govern the creation of the privileged method handles needed by frameworks that move away from the more brittle core reflection API. Combinations & use cases ------------------------ Here are all the combinations of categories, along with some use cases. A shorter form of this table will be used below to summarize the various proposals. Static Shallow Deep Use cases N N N Fully-encapsulated internal packages N N Y (N/A) N Y N Partly-encapsulated framework clients N Y Y Non-encapsulated framework clients Y N N API that disallows all reflective access Y N Y (N/A) Y Y N API that allows shallow reflection Y Y Y API that allows deep reflection (= Java SE 8) Notes and observations: - Fully-encapsulated internal packages are fundamental, and are the default for packages that are not listed in a module declaration. This row ("N N N") will be omitted henceforth. - A partly-encapsulated framework client forbids static access and allows only shallow reflective access. This could be used for a non-API package whose public POJOs will be manipulated by a framework via public methods and fields, while keeping its non-public elements strongly encapsulated. This distinction seems not worth making in practice, since reflective frameworks generally override existing language-level access controls as needed. This row ("N Y N") will be omitted henceforth. - A non-encapsulated framework client forbids static access and allows deep reflective access. This could be used for a non-API package whose internals, including private methods and fields, will be manipulated by a framework. This is another fundamental case. - The "N/A" rows are labeled as such because deep reflective access implies shallow reflective access. These rows will be omitted henceforth. - The ability to define an API that disallows all reflective access is theoretically interesting but of questionable utility. This row ("Y N N") will be omitted henceforth. Omitting the aforementioned rows and compressing horizontally, where "S" in the "Refl?" column means "shallow" and "D" means "deep": Stat Refl? Use N D Non-encapsulated framework clients Y S API that allows shallow reflection Y D API that allows deep reflection (= Java SE 8) We now need to decide which combinations are truly useful, and figure out how to express those combinations in module declarations. There are many tradeoffs to be made, the principal one being between expressive power and approachability. (1) SotMS, 2016/3/8 [1] ----------------------- The SotMS document is out of date with respect to the proposals currently being discussed, but it reflects what's been in the JDK 9 mainline EA builds for a long time now. Stat Refl? Declaration N D N/A Y S N/A Y D `exports p` (2) `exports dynamic`, 2016/6/28 [2] ------------------------------------ Stat Refl? Declaration N D N/A Y S N/A Y D `exports p` OR `exports dynamic p` This `exports dynamic` construct provides static access, and thus ensures reliable configuration, but it is ignored at compile time. (3) Weak modules, 2016/9/12 [3] ------------------------------- Stat Refl? Declaration N D N/A Y S `exports p` Y D `exports private p` OR `weak module { ... }` A weak module cannot contain any `exports` directives, and is thus a useful waypoint on the migration trail. This proposal was an attempt to simplify (2), by removing `dynamic` and making weak modules an explicit mechanism rather than a shorthand for something more complex. Several commentators have suggested that this is not just simple but too simple, since a package can be exported for deep reflection only if it is also exported for static access at compile time. (4) Colebourne #1 ----------------- Stat Refl? Declaration N D `exposes p` Y S `exports p` Y D `exports p` AND `exposes p` Colebourne's first proposal [4], as amended by his later comment [5] so that exported packages are available for shallow reflection. This proposal allows both `*` wildcards and qualified forms of both the `exports` and `exposes` directives. If you use a wildcard with either directive then you cannot use that directive in any other way. Wildcards cannot be used with qualified directives. (5) Colebourne #2 [5] --------------------- Stat Refl? Declaration N D `exposed module { ... }` ONLY Y S `exports p` (only in non-weak modules) Y D `weak module { ... }` ONLY Here Colebourne simplifies his first proposal by removing the ability to export a single package for anything other than both static access and shallow reflection. This does not allow precise control of reflective access via qualified exports to trusted framework modules. (6) Open modules & open packages [6] ------------------------------------ Stat Refl? Declaration N D `opens p` in a `module { ... }`, OR `open module { ... }` Y S `exports p` in a `module { ... }` Y D `exports p` AND `opens p`, in a `module { ... }`, OR `exports p` in an `open module { ... }` This proposal is similar to Colebourne #1 (5), and to various unpublished proposals. It uses `opens` instead of `exposes`, `open module { ... }` rather than wildcards with `exposes`, and does not allow wildcards with `exports`. [1] http://openjdk.java.net/projects/jigsaw/spec/sotms/2016-03-08 [2] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-June/000307.html [3] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-September/000390.html [4] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-September/009370.html [5] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-October/009638.html [6] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000430.html From forax at univ-mlv.fr Thu Oct 27 17:04:56 2016 From: forax at univ-mlv.fr (Remi Forax) Date: Thu, 27 Oct 2016 17:04:56 +0000 Subject: Proposal (revised^2): #ReflectiveAccessToNonExportedTypes & #AwkwardStrongEncapsulation: Open modules & open packages In-Reply-To: <20161027155336.DBCE7DA329@eggemoggin.niobe.net> References: <20161027155336.DBCE7DA329@eggemoggin.niobe.net> Message-ID: <4E125A2F-4C11-4F56-92E7-E08E4041DD04@univ-mlv.fr> I really like this proposal. Remi On October 27, 2016 5:53:36 PM GMT+02:00, mark.reinhold at oracle.com wrote: >Issue summary >------------- > > #ReflectiveAccessToNonExportedTypes --- Some kinds of framework > libraries require reflective access to members of the non-exported > types of other modules; examples include dependency injection (Guice), > persistence (JPA), debugging tools, code-automation tools, and > serialization (XStream). In some cases the particular library to be > used is not known until run time (e.g., Hibernate and EclipseLink both > implement JPA). This capability is also sometimes used to work around > bugs in unchangeable code. Access to non-exported packages can, at > present, only be done via command-line flags, which is extremely > awkward. Provide an easier way for reflective code to access such > non-exported types. [1] > > #AwkwardStrongEncapsulation --- A non-public element of an exported >package can still be accessed via the `AccessibleObject::setAccessible` > method of the core reflection API. The only way to strongly > encapsulate such an element is to move it to a non-exported package. > This makes it awkward, at best, to encapsulate the internals of a > package that defines a public API. [2] > >Proposal >-------- > >(This is the third proposal for #ReflectiveAccessToNonExportedTypes. > The note that follows [3] summarizes the background and relates the > alternatives presented thus far.) > >Extend the language of module declarations with the concept of _open_ >modules. Open modules make it easy to define modules whose internals >will be accessed at run time by reflection-based frameworks. Every >package in an open module has the following properties, by default: > > (1) At compile time it is not exported. > > (2) At run time it is exported without regard to split-package > conflicts or other inconsistencies, i.e., it does not affect > the operation of the resolver, nor the construction of layer > configurations, but is simply exported as if by invoking the > `Module::addExports` method during layer instantiation. > > (3) At run time it is available for _deep reflection_, i.e., all > of its elements are accessible via the core reflection API > including non-public elements, which can be accessed via the > `AccessibleObject::setAccessible` method. > >A package in an open module can be exported explicitly, for use at >compile time, via the familiar `exports` directive, so property (1) >will >not hold. In that case it will also partake in run-time resolution, so >property (2) will not hold either. > >Suppose we have a module `foo.bar` that contains an API package >`com.foo.bar`, whose public types are intended for use by other >modules, >and a non-API package `com.foo.bar.model`, that contains entity classes >to be manipulated by Hibernate via core reflection. Then the module >declaration > > open module foo.bar { > exports com.foo.bar; > requires hibernate.core; > requires hibernate.entitymanager; > } > >makes all elements, public and otherwise, of all packages available for >deep reflection at run time, but makes only the public and protected >types in `com.foo.bar` accessible at compile time. Thus Hibernate can >access non-public elements of the `com.foo.bar.model` package via the >`setAccessible` method, and some other module that `requires foo.bar` >can be compiled against the API in the `com.foo.bar` package but not >against the types in any other package. > >Open modules can be considered an intermediate step on the migration >path, between automatic modules and normal modules: > > - An automatic module offers the traditional level of encapsulation: > All packages are both open for deep reflective access and exported > for ordinary compile-time and run-time access to their public types. > > - An open module offers a modest degree of stronger encapsulation: All > packages are open for deep reflective access, but the module's author > must specify which packages, if any, are exported for ordinary > compile-time and run-time access. > > - A normal module offers the strongest encapsulation: The module's > author must specify which packages, if any, are open, or exported, > or both. > >An open module is a good starting point for application code that's >being >modularized by its author. It affords a separation of concerns, so >that >the author can focus on defining the module's domain-specific API >without >having to worry about how the module will be inspected or manipulated >at >run time by reflective frameworks. > > * * * > >To open specific packages in normal module declarations we introduce a >new per-package directive, `opens`. A package in a normal module can >be >opened, or exported, or both, as follows: > > - If a package is only opened (`opens p;`) then properties (1), (2), > and (3) hold, i.e., its types are not accessible at compile time, > it does not affect resolution, and all of its elements are available > for deep reflection at run time. > > - If a package is only exported (`exports p;`) then those properties > do not hold, and it is exported exactly as it is today. > > - If a package is both exported and opened (`exports p; opens p;`) > then it is exported as today and is, additionally, available for > deep reflection at run time (3). > >We can use the `opens` directive to refine the previous example so that >only the `com.foo.bar` package is accessible at compile time, only the >`com.foo.bar.model` package is available for deep reflection at run >time, >and the non-public elements of all other packages are strongly >encapsulated: > > module foo.bar { > exports com.foo.bar; > opens com.foo.bar.model; > requires hibernate.core; > requires hibernate.entitymanager; > } > >It's possible to both open and export a package in a normal module, but >it's often inadvisable. This is especially so for API packages, since >normally an API's internal implementation details should be strongly >encapsulated. This combination of directives may, however, be useful >for >legacy APIs whose internals are known to be accessed by existing code. > >To ensure the integrity of the platform we expect that all the modules >of >the JDK itself will be normal modules and that very few, if any, >packages >will be opened. > >The `opens` directive cannot be used in an `open` module, since all >packages in such modules are implicitly open. > > * * * > >Both the `exports` and `opens` directives can be qualified, so that a >package is exported or opened only to certain other named modules. As >before, duplicate directives are not permitted in order to ensure easy >readability. At most one `exports` directive is relevant to any given >package, and at most one `opens` directive is relevant to any given >package. > >There is no syntax for wildcards. If the defaults are not sufficient >for >a package then the package must be named explicitly. > > * * * > >The existing syntax of `requires public` has long been confusing, so we >here take the opportunity to fix that problem by renaming the `public` >modifier in `requires` directives to `transitive`. Thus the >declaration > > module foo.bar { > exports com.foo.bar; > requires public java.sql; > } > >is now written > > module foo.bar { > exports com.foo.bar; > requires transitive java.sql; > } > >This is potentially confusing in a different way, since in mathematics >the term "transitive" is usually applied to an entire relation rather >than to three specific elements of a set. Its use here does not, in >particular, mean that the resolver does not interpret plain `requires` >directives when computing the transitive closure of a set of root >modules. "Transitive" as used here is in the more abstract sense, >expressing the notion of conveying a property -- in this case, the >readability of the required module -- from one thing to another. On >balance, however, `transitive` does appear superior to `public`. > >Notes >----- > > - This proposal is significantly different from both the first and > second proposals [4][5]. It recognizes that, in practice, it will > be common to have open modules, all of whose content is available > for deep reflection but otherwise inaccessible at compile time unless > explicitly exported. If finer-grained control over whether a package > is opened, exported, or both is required then a normal module > declaration can be used. > > - This proposal is similar to one made by Stephen Colebourne [6], > in which he uses the term `exposes`, and to various unpublished > proposals (of which there have been many!). We here adopt the term > `opens` rather than `exposes` in order to avoid confusion between > the phonetically-similar `exports` and `exposes`, especially for > non-native speakers of English. We also use `open` to modify the > `module` keyword rather than introduce wildcards for the `opens` > directive, since the former is easier to explain. The latter would, > further, lead people to expect wildcards for the `exports` directive, > but we'd prefer to avoid that since it could encourage the careless > exportation of all of a module's packages. Exporting a package for > use as an API should always be an explicit, affirmative choice. > > - A separate `opens` directive, and the avoidance of wildcards, > simplifies the interactions amongst qualified and unqualified > package-level directives. We no longer need complex rules such as > those shown for the `private` modifier in the previous proposal [5]. > >- This proposal implies corresponding changes to the `ModuleDescriptor` > and `Module` APIs, but these are reasonably straightforward. > > - This proposal amends the proposal for #ResourceEncapsulation [6] > to replace the use of private exports with open modules and open > packages. Wherever that proposal indicates that a resource can be > located when its effective package name identifies a package that's > exported privately, assume instead that the resource can be located > when that package is either an element of an open module or an open > package in a normal module. > > - If a container is to ensure that a package in an application module > is available for deep reflection only by a trusted framework module > then it can arrange for that by rewriting that module's descriptor, > as suggested previously [8], to insert the appropriate qualified > `opens` directive. This works even in the case, pointed out by Jason > Greene [9], of two modules that open a package of the same name to > the same framework module, since opened and unexported packages are > not subject to the usual package-consistency checks. There is no > need any longer for the resolution algorithm to take this type of > scenario into account [a]. A more flexible way to arrange for such > precisely-constrained access is a separate issue and may be addressed > by a future proposal. > >- This proposal primarily addresses "friendly" uses of reflection, such > as dependency injection and persistence, in which the author of a > module knows in advance that one or more, or possibly all, packages > must be opened at run time for deep reflective access by frameworks. > Intrusive access to arbitrary packages of arbitrary modules by, e.g., > serialization frameworks or debugging tools, will still require the > use of sharp knives such as `--add-exports` or `--add-opens` > command-line options, the legacy unsupported `sun.misc.Unsafe` API > and related APIs, or JVM TI. > >- Using `--add-exports` or `--add-opens` command-line options, or their > equivalents, remains awkward, but sometimes they're the only way out. > To ease migration it's worth considering some way for an application > packaged as a JAR file to include such options in its `MANIFEST.MF` > file, as suggested by Simon Nash [b]. This is addressed in the > proposal for #AddExportsInManifest [c], which is hereby amended to > rename the `Add-Exports-Private` attribute to `Add-Opens`. > > >[1] >http://openjdk.java.net/projects/jigsaw/spec/issues/#ReflectiveAccessToNonExportedTypes >[2] >http://openjdk.java.net/projects/jigsaw/spec/issues/#AwkwardStrongEncapsulation >[3] >http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000431.html >[4] >http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-June/000307.html >[5] >http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-September/000390.html >[6] >http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-September/000392.html >[7] >http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-September/009370.html >[8] >http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008637.html >[9] >http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008641.html >[a] >http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008727.html >[b] >http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-December/005745.html >[c] >http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-September/000391.html -- Sent from my Android device with K-9 Mail. Please excuse my brevity. From tjwatson at us.ibm.com Thu Oct 27 17:49:41 2016 From: tjwatson at us.ibm.com (Thomas Watson) Date: Thu, 27 Oct 2016 12:49:41 -0500 Subject: Proposal (revised^2): #ReflectiveAccessToNonExportedTypes&#AwkwardStrongEncapsulation: Open modules & open packages In-Reply-To: <4E125A2F-4C11-4F56-92E7-E08E4041DD04@univ-mlv.fr> References: <20161027155336.DBCE7DA329@eggemoggin.niobe.net> <4E125A2F-4C11-4F56-92E7-E08E4041DD04@univ-mlv.fr> Message-ID: I like this proposal also. It nicely solves the usecases I have in mind. Thanks. Tom > From: Remi Forax > To: mark.reinhold at oracle.com, jpms-spec-experts at openjdk.java.net > Date: 10/27/2016 12:05 PM > Subject: Re: Proposal (revised^2): > #ReflectiveAccessToNonExportedTypes & #AwkwardStrongEncapsulation: > Open modules & open packages > Sent by: "jpms-spec-observers" > > I really like this proposal. > > Remi > > On October 27, 2016 5:53:36 PM GMT+02:00, mark.reinhold at oracle.com wrote: > >Issue summary > >------------- > > > > #ReflectiveAccessToNonExportedTypes --- Some kinds of framework > > libraries require reflective access to members of the non-exported > > types of other modules; examples include dependency injection (Guice), > > persistence (JPA), debugging tools, code-automation tools, and > > serialization (XStream). In some cases the particular library to be > > used is not known until run time (e.g., Hibernate and EclipseLink both > > implement JPA). This capability is also sometimes used to work around > > bugs in unchangeable code. Access to non-exported packages can, at > > present, only be done via command-line flags, which is extremely > > awkward. Provide an easier way for reflective code to access such > > non-exported types. [1] > > > > #AwkwardStrongEncapsulation --- A non-public element of an exported > >package can still be accessed via the `AccessibleObject::setAccessible` > > method of the core reflection API. The only way to strongly > > encapsulate such an element is to move it to a non-exported package. > > This makes it awkward, at best, to encapsulate the internals of a > > package that defines a public API. [2] > > > >Proposal > >-------- > > > >(This is the third proposal for #ReflectiveAccessToNonExportedTypes. > > The note that follows [3] summarizes the background and relates the > > alternatives presented thus far.) > > > >Extend the language of module declarations with the concept of _open_ > >modules. Open modules make it easy to define modules whose internals > >will be accessed at run time by reflection-based frameworks. Every > >package in an open module has the following properties, by default: > > > > (1) At compile time it is not exported. > > > > (2) At run time it is exported without regard to split-package > > conflicts or other inconsistencies, i.e., it does not affect > > the operation of the resolver, nor the construction of layer > > configurations, but is simply exported as if by invoking the > > `Module::addExports` method during layer instantiation. > > > > (3) At run time it is available for _deep reflection_, i.e., all > > of its elements are accessible via the core reflection API > > including non-public elements, which can be accessed via the > > `AccessibleObject::setAccessible` method. > > > >A package in an open module can be exported explicitly, for use at > >compile time, via the familiar `exports` directive, so property (1) > >will > >not hold. In that case it will also partake in run-time resolution, so > >property (2) will not hold either. > > > >Suppose we have a module `foo.bar` that contains an API package > >`com.foo.bar`, whose public types are intended for use by other > >modules, > >and a non-API package `com.foo.bar.model`, that contains entity classes > >to be manipulated by Hibernate via core reflection. Then the module > >declaration > > > > open module foo.bar { > > exports com.foo.bar; > > requires hibernate.core; > > requires hibernate.entitymanager; > > } > > > >makes all elements, public and otherwise, of all packages available for > >deep reflection at run time, but makes only the public and protected > >types in `com.foo.bar` accessible at compile time. Thus Hibernate can > >access non-public elements of the `com.foo.bar.model` package via the > >`setAccessible` method, and some other module that `requires foo.bar` > >can be compiled against the API in the `com.foo.bar` package but not > >against the types in any other package. > > > >Open modules can be considered an intermediate step on the migration > >path, between automatic modules and normal modules: > > > > - An automatic module offers the traditional level of encapsulation: > > All packages are both open for deep reflective access and exported > > for ordinary compile-time and run-time access to their public types. > > > > - An open module offers a modest degree of stronger encapsulation: All > > packages are open for deep reflective access, but the module's author > > must specify which packages, if any, are exported for ordinary > > compile-time and run-time access. > > > > - A normal module offers the strongest encapsulation: The module's > > author must specify which packages, if any, are open, or exported, > > or both. > > > >An open module is a good starting point for application code that's > >being > >modularized by its author. It affords a separation of concerns, so > >that > >the author can focus on defining the module's domain-specific API > >without > >having to worry about how the module will be inspected or manipulated > >at > >run time by reflective frameworks. > > > > * * * > > > >To open specific packages in normal module declarations we introduce a > >new per-package directive, `opens`. A package in a normal module can > >be > >opened, or exported, or both, as follows: > > > > - If a package is only opened (`opens p;`) then properties (1), (2), > > and (3) hold, i.e., its types are not accessible at compile time, > > it does not affect resolution, and all of its elements are available > > for deep reflection at run time. > > > > - If a package is only exported (`exports p;`) then those properties > > do not hold, and it is exported exactly as it is today. > > > > - If a package is both exported and opened (`exports p; opens p;`) > > then it is exported as today and is, additionally, available for > > deep reflection at run time (3). > > > >We can use the `opens` directive to refine the previous example so that > >only the `com.foo.bar` package is accessible at compile time, only the > >`com.foo.bar.model` package is available for deep reflection at run > >time, > >and the non-public elements of all other packages are strongly > >encapsulated: > > > > module foo.bar { > > exports com.foo.bar; > > opens com.foo.bar.model; > > requires hibernate.core; > > requires hibernate.entitymanager; > > } > > > >It's possible to both open and export a package in a normal module, but > >it's often inadvisable. This is especially so for API packages, since > >normally an API's internal implementation details should be strongly > >encapsulated. This combination of directives may, however, be useful > >for > >legacy APIs whose internals are known to be accessed by existing code. > > > >To ensure the integrity of the platform we expect that all the modules > >of > >the JDK itself will be normal modules and that very few, if any, > >packages > >will be opened. > > > >The `opens` directive cannot be used in an `open` module, since all > >packages in such modules are implicitly open. > > > > * * * > > > >Both the `exports` and `opens` directives can be qualified, so that a > >package is exported or opened only to certain other named modules. As > >before, duplicate directives are not permitted in order to ensure easy > >readability. At most one `exports` directive is relevant to any given > >package, and at most one `opens` directive is relevant to any given > >package. > > > >There is no syntax for wildcards. If the defaults are not sufficient > >for > >a package then the package must be named explicitly. > > > > * * * > > > >The existing syntax of `requires public` has long been confusing, so we > >here take the opportunity to fix that problem by renaming the `public` > >modifier in `requires` directives to `transitive`. Thus the > >declaration > > > > module foo.bar { > > exports com.foo.bar; > > requires public java.sql; > > } > > > >is now written > > > > module foo.bar { > > exports com.foo.bar; > > requires transitive java.sql; > > } > > > >This is potentially confusing in a different way, since in mathematics > >the term "transitive" is usually applied to an entire relation rather > >than to three specific elements of a set. Its use here does not, in > >particular, mean that the resolver does not interpret plain `requires` > >directives when computing the transitive closure of a set of root > >modules. "Transitive" as used here is in the more abstract sense, > >expressing the notion of conveying a property -- in this case, the > >readability of the required module -- from one thing to another. On > >balance, however, `transitive` does appear superior to `public`. > > > >Notes > >----- > > > > - This proposal is significantly different from both the first and > > second proposals [4][5]. It recognizes that, in practice, it will > > be common to have open modules, all of whose content is available > > for deep reflection but otherwise inaccessible at compile time unless > > explicitly exported. If finer-grained control over whether a package > > is opened, exported, or both is required then a normal module > > declaration can be used. > > > > - This proposal is similar to one made by Stephen Colebourne [6], > > in which he uses the term `exposes`, and to various unpublished > > proposals (of which there have been many!). We here adopt the term > > `opens` rather than `exposes` in order to avoid confusion between > > the phonetically-similar `exports` and `exposes`, especially for > > non-native speakers of English. We also use `open` to modify the > > `module` keyword rather than introduce wildcards for the `opens` > > directive, since the former is easier to explain. The latter would, > > further, lead people to expect wildcards for the `exports` directive, > > but we'd prefer to avoid that since it could encourage the careless > > exportation of all of a module's packages. Exporting a package for > > use as an API should always be an explicit, affirmative choice. > > > > - A separate `opens` directive, and the avoidance of wildcards, > > simplifies the interactions amongst qualified and unqualified > > package-level directives. We no longer need complex rules such as > > those shown for the `private` modifier in the previous proposal [5]. > > > >- This proposal implies corresponding changes to the `ModuleDescriptor` > > and `Module` APIs, but these are reasonably straightforward. > > > > - This proposal amends the proposal for #ResourceEncapsulation [6] > > to replace the use of private exports with open modules and open > > packages. Wherever that proposal indicates that a resource can be > > located when its effective package name identifies a package that's > > exported privately, assume instead that the resource can be located > > when that package is either an element of an open module or an open > > package in a normal module. > > > > - If a container is to ensure that a package in an application module > > is available for deep reflection only by a trusted framework module > > then it can arrange for that by rewriting that module's descriptor, > > as suggested previously [8], to insert the appropriate qualified > > `opens` directive. This works even in the case, pointed out by Jason > > Greene [9], of two modules that open a package of the same name to > > the same framework module, since opened and unexported packages are > > not subject to the usual package-consistency checks. There is no > > need any longer for the resolution algorithm to take this type of > > scenario into account [a]. A more flexible way to arrange for such > > precisely-constrained access is a separate issue and may be addressed > > by a future proposal. > > > >- This proposal primarily addresses "friendly" uses of reflection, such > > as dependency injection and persistence, in which the author of a > > module knows in advance that one or more, or possibly all, packages > > must be opened at run time for deep reflective access by frameworks. > > Intrusive access to arbitrary packages of arbitrary modules by, e.g., > > serialization frameworks or debugging tools, will still require the > > use of sharp knives such as `--add-exports` or `--add-opens` > > command-line options, the legacy unsupported `sun.misc.Unsafe` API > > and related APIs, or JVM TI. > > > >- Using `--add-exports` or `--add-opens` command-line options, or their > > equivalents, remains awkward, but sometimes they're the only way out. > > To ease migration it's worth considering some way for an application > > packaged as a JAR file to include such options in its `MANIFEST.MF` > > file, as suggested by Simon Nash [b]. This is addressed in the > > proposal for #AddExportsInManifest [c], which is hereby amended to > > rename the `Add-Exports-Private` attribute to `Add-Opens`. > > > > > >[1] > >http://openjdk.java.net/projects/jigsaw/spec/issues/ > #ReflectiveAccessToNonExportedTypes > >[2] > >http://openjdk.java.net/projects/jigsaw/spec/issues/ > #AwkwardStrongEncapsulation > >[3] > >http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016- > October/000431.html > >[4] > >http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-June/ > 000307.html > >[5] > >http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016- > September/000390.html > >[6] > >http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016- > September/000392.html > >[7] > > http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-September/009370.html > >[8] > >http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008637.html > >[9] > >http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008641.html > >[a] > >http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008727.html > >[b] > > http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-December/005745.html > >[c] > >http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016- > September/000391.html > > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. > From mark.reinhold at oracle.com Thu Oct 27 23:15:37 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 27 Oct 2016 16:15:37 -0700 Subject: Proposal: #ReflectiveAccessToNonExportedTypes (revised) & #AwkwardStrongEncapsulation: Weak modules & private exports In-Reply-To: <279bd57d-515a-a983-ffbb-6f569c29495e@redhat.com> References: <20160912150801.AD8AECD07F@eggemoggin.niobe.net> <5b9b1929-eb92-09ab-2997-d7e2395a1d9d@redhat.com> <20161011151514.A04ECD50FA@eggemoggin.niobe.net> <279bd57d-515a-a983-ffbb-6f569c29495e@redhat.com> Message-ID: <20161027161537.655286378eggemoggin.niobe.net> 2016/10/24 6:46:11 -0700, david.lloyd at redhat.com: > On 10/11/2016 10:15 AM, mark.reinhold at oracle.com wrote: >> 2016/9/16 8:30:41 -0700, david.lloyd at redhat.com: >>> ... >>> >>> #1) The "weak" designation appears to be pejorative >>> >>> Under this solution, many existing frameworks and/or the modules which >>> consume them must be marked "weak" in order to work correctly, or else a >>> much more complex module descriptor must be used. While this seems to >>> be a stepping stone for migration, we find that it rather makes >>> middleware appear as a second-class citizen. >> >> No insult is intended; "weak" is used here purely in a technical sense, >> as the obvious antonym of "strong". Suggestions for a better word for >> this concept are welcome. > > It's the concept itself that is problematic, I think. Really, if I > understand correctly, the "weak" notion exists merely as shorthand to > say "grant permission on all packages to all modules for private > reflection", which is the sort of thing that a user ought to be saying > explicitly. Many users will use weak modules without understanding what > they are; some will come to understand it and immediately press for best > practices to get rid of them. I think that regardless of what the > concept is named, it's going to spell trouble in the end. Many others had similar reactions, for a variety of reasons, so please have a look at the open modules & open packages proposal I posted earlier today [1]. >>> ... >>> >>> #3) Controlling reflection access is always bound up with controlling >>> exports >>> >>> The proposed exports system is simplistic but not exactly simple, yet >>> several useful modes remain left unaccounted-for. Internally we >>> examined the usefulness of controlling access to public and so-called >>> "deep" reflection in combination with whether or not a member was >>> exported, and we came up with this logical table: >>> >>> +------------------+---------------------------+ >>> | | Module is... | >>> | Reflection is... +------------+--------------+ >>> | | Exported | Not Exported | >>> +----------------------------------------------+ >>> | Forbidden | Not Useful | Useful | >>> +----------------------------------------------+ >>> | Public Only | Useful | Useful | >>> +----------------------------------------------+ >>> | Public & Private | Useful | Useful | >>> +------------------+------------+--------------+ >> >> I'm not sure how to read this table. Should the cell containing the text >> "Module is ..." instead contain "Member is ..."? > > Right. > >> By "exported" I'll assume that you mean non-reflective access, i.e., from >> the language at compile time and bytecodes at run time. > > Yes. > >> Do you have example use cases for each of the "Useful" cells? Some are >> obvious, but others are not. > > Sure, I'll go over each one by one with some example use cases. > > Forbidden/Not Exported: Obviously the foundation case for > access-oriented encapsulation, this is the default mode. > > Exported modes: > > Public Only/Exported: It doesn't make sense to disallow reflection on > public types and members that can already be accessed directly, so this > would be the default access mode to imported members when one module > imports another. > > Public & Private/Exported: This is the JDK 8 compatibility mode, but a > module should have to opt into this mode, and it should be restricted to > specific targets whenever possible, else the mechanism is essentially > defeated. > > Not exported modes: > > Public Only/Not Exported: This is the mode that would be used (for > example) by DI and ORM frameworks and containers which operate on public > members. Normally a module which imports such a framework would grant > the necessary reflective access to it as well (though a full export is > in this case unnecessary); the framework wouldn't contain requires > clauses for modules which consume it. Like I suggested in my first > email, a way to "handshake" this agreement seems like it would be > necessary: the framework would not allow itself to be imported without > the importing module acknowledging the access in its configuration. > > Public & Private/Not Exported: This is the mode that would be used by > serialization and DI/ORM frameworks and containers which operate on > private members (e.g. field-level injection). Like the above though, > this level of access is usually granted by the module importing the > framework in question and should be explicitly opted into if possible. Now that I understand it, I think your table maps fairly directly to the analysis that's driven my current and past proposals, which I summarized earlier today [2]. Please let me know if you think otherwise. The open-modules/packages proposal does not support your "Public Only / Not Exported" mode, which corresponds to the missing "N S" row in the compact table in [2], i.e., the "partly-encapsulated framework client" case. Most reflective frameworks in use today will just reach in via `setAccessible` whenever they need to, so whether the members of an unexported package in a client module are public or private doesn't really matter. The new proposal intentionally does not support this case, which results in a simpler design. > The better situation however is to separate the grant from the export. That's exactly what the new `opens` directive does [1]. >>> #4) There isn't always a container >>> >>> The justification for not providing an easy mechanism to restrict >>> private reflective access to a trusted framework is that a container can >>> always rewrite a module or inject classes. However there isn't always a >>> container present - something which the existence of weak modules >>> acknowledges. Some frameworks and middleware are meant to be usable >>> with or without a container, and it is undesirable to force these use >>> cases into a weak module situation or to require a strong coupling in >>> the module descriptor in this case. Some other solution should be found >>> that works both for containers and "regular" applications. >> >> If you're running outside of a container then how important is it to >> precisely restrict access to packages meant for deep reflection? If >> you're outside of a container then you have to trust the person (you?) >> who sets up your module path and/or class path. To take the canonical >> case of a module that just contains JPA model classes, if you make it a >> weak module or else `exports private` the relevant packages without >> qualification so that the hibernate.core (or whatever) module can reflect >> into it, then the scope for trouble is limited. Sure, some other module >> on the module path or even code on the class path could reflect into your >> module when it's not supposed to, but is it worth complicating the module >> system just to handle this scenario? > > In either situation, the person (deployer) who sets up the class path or > installation can currently do whatever they want. Nothing in the new > (or existing) security framework truly makes it possible to run > untrusted code safely in this regard, nor to safely run trusted code in > an untrusted environment, nor is it a sufficient solution for secure > sandboxing even with a security manager present, with or without a > container. At best it's a mitigation strategy. Exactly. > That said: the rise of microservices and reactive programming suggests > that such standalone (containerless) deployments are, if anything, more > popular than ever, but they may suffer from security problems which are > very similar to, if not identical to, those which applications running > in containers are subject to. That could be, and it deserves further thought. > In fact jlink (and the new AOT work for > that matter) seem particularly and specifically suited to this type of > use case. Agreed. > Finally: a JPA model class module is indeed probably harmless in any > practical security sense, and locking it down is probably not useful > from a security perspective, but even in this case it may yet be useful > to restrict package exports such that only _supported_ classes are > visible (especially when a module is published within an organization > and contracts must be maintained); i.e. encapsulation is not solely a > security concern, it also establishes intent as clearly as any interface > declaration does. I completely agree that encapsulation is not just for security -- it's also about helping developers build reliable systems by defining clear boundaries between elements at all levels (classes, package, modules) so that systems are understandable, testable, and so forth. The open-modules/packages proposal does address your last point: In contrast to the weak-modules proposal, you can prevent some or all of the packages of a framework-client module from being compiled against, i.e., treated as API, by opening them but not exporting them. >>> ... >>> >>> #7) More loose coupling seems necessary and useful >>> >>> In order for typical applications to function with modern middleware as >>> modules, without compromising security, it may be necessary to enhance >>> the loose coupling mechanism (uses/provides), or to provide an >>> additional, similar mechanism, which allows a symbolic coupling which >>> would allow modules to declare (in an abstract manner) modules which >>> need to have reflective access to it in a "friendly" manner. ... >> >> To confirm my understanding, would it be correct to say that your main >> concern here is scenarios in which some of the classes in a user module >> are intended to be manipulated reflectively by some trusted framework >> module, but that framework module is either not known at compile time or >> is otherwise inconvenient to name in the declaration of the user module? > > Correct. Good. I'll enter this as a new issue: #IndirectQualifiedReflectiveAccess --- Provide a means by which a client module can grant qualified reflective access to a framework module that is not known at compile time, assuming that the name of some other module that represents the framework module is known. A canonical example of this case is a client POJO module compiled against a module that defines the JPA API. At run time the client module, or a container or some other code acting on its behalf, must grant qualified reflective access to its POJO packages to the JPA implementation in actual use. - Mark [1] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000430.html [2] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-October/000431.html From forax at univ-mlv.fr Fri Oct 28 15:35:35 2016 From: forax at univ-mlv.fr (Remi Forax) Date: Fri, 28 Oct 2016 17:35:35 +0200 (CEST) Subject: Please welcome Neil Bartlett to JSR 376 In-Reply-To: <0d21c865-414d-0593-f8c1-0c51bdd416f6@redhat.com> References: <20161025140849.350853791eggemoggin.niobe.net> <0d21c865-414d-0593-f8c1-0c51bdd416f6@redhat.com> Message-ID: <1025055751.2163401.1477668935510.JavaMail.zimbra@u-pem.fr> Welcome Neil :) R?mi ----- Mail original ----- > De: "David M. Lloyd" > ?: jpms-spec-experts at openjdk.java.net > Envoy?: Mardi 25 Octobre 2016 23:49:30 > Objet: Re: Please welcome Neil Bartlett to JSR 376 > On 10/25/2016 04:08 PM, mark.reinhold at oracle.com wrote: >> Neil Bartlett, a recognized expert on OSGi, has asked to join this EG. >> >> Ordinarily I'm extremely reluctant to accept new EG members this late in >> the life of a JSR, long after we've settled the requirements and after >> we've been working on a design for over a year. Peter Kriens's recent >> departure, however, leaves the EG with no resident OSGi expertise, and >> that's an important gap to fill. >> >> Neil has agreed to accept the requirements as they stand and, further, >> acknowledges that our primary goal at this stage is to finish the design >> that we have rather than come up with something significantly different. >> I'm confident that he'll bring a valuable perspective to our discussions >> and help to ensure the overall success of the JSR. >> >> (Neil -- if there's anything further you'd like to say by way of >> introduction, please do so!) > > Welcome aboard, Neil! > -- > - DML From forax at univ-mlv.fr Fri Oct 28 15:41:41 2016 From: forax at univ-mlv.fr (Remi Forax) Date: Fri, 28 Oct 2016 17:41:41 +0200 (CEST) Subject: Dynamic modules and bidirectional interoperation In-Reply-To: <20161026102447.63677664eggemoggin.niobe.net> References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net> <20161026091545.152155830eggemoggin.niobe.net> <20161026102447.63677664eggemoggin.niobe.net> Message-ID: <960310567.2164815.1477669301699.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "mark reinhold" > ?: "Thomas Watson" > Cc: jpms-spec-experts at openjdk.java.net > Envoy?: Mercredi 26 Octobre 2016 19:24:47 > Objet: Re: Dynamic modules and bidirectional interoperation > 2016/10/26 9:32:42 -0700, Thomas Watson , > jpms-spec-comments: >> 2016/10/26 09:15:45 -0700, mark.reinhold at oracle.com, jpms-spec-experts: >>> 2016/10/7 8:47:53 -0700, Thomas Watson , >>> jpms-spec-observers: >>>> ... >>>> >>>> 1) Any OSGi dependencies that influence class loader delegation will not >>>> be declared in the ModuleDescriptors which represent the OSGi bundle >>>> wirings. Instead we will depend on the ability to >>>> #ReadabilityAddedByLayerCreator to wire up the modules within the bundle >>>> Layer according to OSGi resolution rules. It has a drawback if there are >>>> cases where some system code is trying to reason about modular >>>> dependencies by using the JPMS module API. I think this will have to be >>>> acceptable because I don't think we will be able to fully converge on all >>>> the possible dependency types available in the existing module systems. >>>> For example, OSGi has Require-Bundle which maps closing to JPMS requires, >>>> but OSGi also has Import-Package dependencies which cannot be mapped >>>> properly into JMPS module Layers. >>> >>> If by "system code" you mean code in the JDK itself, I don't think this >>> will be a big problem at run time. Except for diagnostic purposes, >>> module descriptors are really only interpreted during resolution and >>> configuration. All other run-time reasoning about module readability >>> is based upon the reads edges in the module graph as reported by the >>> `java.lang.reflect.Module::canRead` method, and that will include any >>> reads edges that you add to support bundle wirings. >> >> Here I am using "system code" in the general sense, but I think any >> "system code" that cares about wiring will need to pay attention to >> Module::canRead like you suggest. I can think of a few frameworks that >> care about how modules are wired up at runtime. For example, in OSGi we >> have the extender pattern. Without going into much detail here, basically >> an extender implementation module will only act on extendee modules which >> are wired to the extender module. If such a pattern exists using JPMS >> then the "system code" would be the extender implementation. > > Okay, I see. > >>>> 2) I assume #NonHierarchicalLayers will be restricted to form a directed >>>> acyclic graph. >>> >>> Yes. >>> >>>> As such I assume each node in the graph of Layers would be >>>> visited only once when resolving new Layers on top even if there are >>>> multiple paths to the layer from the new Layer being resolved. >>> >>> Yes. >> >> The motivation for my question was to ask some follow-ons. >> >> Will there will be a specified order in which the nodes are traversed >> when there are multiple paths to a node? More specifically will the >> parents have a specified order in which they are delegated to? > > Yes, the parents of a new configuration will be specified in an ordered > list and during resolution they'll be searched depth-first, in the order > in which they occur in the list. Depth-first order makes it easy to > reason about resolution, since it works just like a search path. I fail to seen how this is not a mini-classpath with exactly the same issues as the classpath. > > - Mark R?mi From forax at univ-mlv.fr Fri Oct 28 16:01:31 2016 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Fri, 28 Oct 2016 18:01:31 +0200 (CEST) Subject: Dynamic modules and bidirectional interoperation In-Reply-To: References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net> <1316230997.272548.1477056278273.JavaMail.zimbra@u-pem.fr> Message-ID: <531946364.2170171.1477670491677.JavaMail.zimbra@u-pem.fr> Hi Thomas, first, sorry, I'm not able to parse my own mail correctly. Let's try again, currently a way to solve your issue is to allow a Layer to have several parents, i really dislike this idea because it introduces an arbitrary order that is hidden deep in the code thus makes really hard to reason about which Layer resolve what, for me it's not different to add a kind of classpath resolution in the middle of the module resolution. Currently, you control the classloaders, so you can load a hand crafted class in the module (you just have to patch the class name to have the right package, you can use the patching capabilities of defineAnonymousClass for that, but you can also call defineClass on the right classloader as again, you control the classloaders). Once you are able to execute arbitrary code in a module you can call addReads, addExports (and soon addOpens) on the j.l.r.Module. regards, R?mi > De: "Thomas Watson" > ?: forax at univ-mlv.fr, "mark reinhold" , > "jpms-spec-observers" > Envoy?: Lundi 24 Octobre 2016 21:40:06 > Objet: Re: Dynamic modules and bidirectional interoperation > Can you explain more about how monkey patching would be done in a JVM > implementation neutral way? If I understand correctly your suggestion would > require monkey patching in order to add functionality to the module classes > provided by the boot layer (Module, Layer, Configuration etc.). This additional > functionality would allow an OSGi Framework implementation to represent the > dynamic set of bundle wirings in an OSGi Framework as JPMS modules. > In order to do that it seems we would need extensive knowledge of the internal > implementation details of the classes we are monkey patching in order to modify > their internal structures. I can see how using Unsafe would give us the power > to hack into the implementation details of jigsaw, but I don't view that as a > satisfactory solution to the problem. It would bind us to a particular internal > implementation of Java 9 in order to work correctly. It places a new dependency > on Unsafe class which is from the jdk.unsupported module. I don't think new > dependencies on the jdk.unsupported module should be advocated. Doing so has a > high probability of breaking on a future version of Java if Unsafe is removed > by default or the implementation details of the module classes change. > Tom > "jpms-spec-observers" wrote on > 10/21/2016 08:24:38 AM: > > From: Remi Forax > > To: mark reinhold > > Cc: jpms-spec-experts at openjdk.java.net > > Date: 10/21/2016 08:25 AM > > Subject: Re: Dynamic modules and bidirectional interoperation > > Sent by: "jpms-spec-observers" > > Hi Mark, > > I see an other way to solve the wiring issue, > > use VM anonymous classes (via unsafe.defineAnonymousClass) to > > generate the glue code (as lambda meta factory does to implement the > > functional interface at runtime) and method handles as wires (once > > created a method handle do not do any security check anymore, so the > > requirement to be in a module with specific parents can be lifted). > > With that, you don't need to generate the wiring code inside a new > > module and when a new bundle need to be re-wire, you can mutate the > > wire (the method handle) by using a mutable callsite. > > Said differently, loading/unloading an OSGI bundle is equivalent to > > doing monkey patching in a dynamic language, so there is no reason > > to not use the tools we provide for dynamic languages to wire OSGI bundles. > > so i vote strongly against that proposal mostly because it's too > > late to change requirements and because some obvious design options > > to implement OSGI on top of JPMS have not been tested. > > regards, > > R?mi > > ----- Mail original ----- > > > De: "mark reinhold" > > > ?: jpms-spec-experts at openjdk.java.net > > > Envoy?: Jeudi 6 Octobre 2016 19:07:59 > > > Objet: Dynamic modules and bidirectional interoperation > > > The agreed requirements for this JSR mandate a modest degree of dynamic > > > configuration [1]. They do not mandate _dynamic modules_, i.e., the > > > fine-grained dynamic configuration or "hot deployment" that is possible > > > with other module systems, such as OSGi and JBoss, in which individual > > > modules can be installed, updated, and uninstalled at any time. > > > The requirements also mandate that it "be possible for another module > > > system, such as OSGi, to locate Java modules and resolve them using its > > > own resolver, except possibly for core system modules" [2]. They do not > > > mandate _bidirectional interoperation_, which in the case of OSGi would > > > be the ability to arrange for bundles to refer to JPMS modules and also > > > vice versa, i.e., for JPMS modules to refer to bundles. From mark.reinhold at oracle.com Fri Oct 28 16:16:49 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Fri, 28 Oct 2016 09:16:49 -0700 Subject: Dynamic modules and bidirectional interoperation In-Reply-To: <960310567.2164815.1477669301699.JavaMail.zimbra@u-pem.fr> References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net> <20161026091545.152155830eggemoggin.niobe.net> <20161026102447.63677664eggemoggin.niobe.net> <960310567.2164815.1477669301699.JavaMail.zimbra@u-pem.fr> Message-ID: <20161028091649.617098323eggemoggin.niobe.net> 2016/10/28 8:41:41 -0700, Remi Forax : > De: "mark reinhold" > Envoy?: Mercredi 26 Octobre 2016 19:24:47 >> 2016/10/26 9:32:42 -0700, Thomas Watson , >> jpms-spec-comments: >>> ... >>> >>> Will there will be a specified order in which the nodes are traversed >>> when there are multiple paths to a node? More specifically will the >>> parents have a specified order in which they are delegated to? >> >> Yes, the parents of a new configuration will be specified in an ordered >> list and during resolution they'll be searched depth-first, in the order >> in which they occur in the list. Depth-first order makes it easy to >> reason about resolution, since it works just like a search path. > > I fail to seen how this is not a mini-classpath with exactly the same > issues as the classpath. It's not a "mini" class path; it's more like a "mini" or "run time" module path. It's better than a class path because it's a means to locate whole modules rather than individual types, as discussed in SotMS: http://openjdk.java.net/projects/jigsaw/spec/sotms/#the-module-path - Mark From forax at univ-mlv.fr Fri Oct 28 17:01:17 2016 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Fri, 28 Oct 2016 19:01:17 +0200 (CEST) Subject: Dynamic modules and bidirectional interoperation In-Reply-To: <20161028091649.617098323eggemoggin.niobe.net> References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net> <20161026091545.152155830eggemoggin.niobe.net> <20161026102447.63677664eggemoggin.niobe.net> <960310567.2164815.1477669301699.JavaMail.zimbra@u-pem.fr> <20161028091649.617098323eggemoggin.niobe.net> Message-ID: <507939175.2182737.1477674077829.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "mark reinhold" > ?: "Remi Forax" > Cc: jpms-spec-experts at openjdk.java.net > Envoy?: Vendredi 28 Octobre 2016 18:16:49 > Objet: Re: Dynamic modules and bidirectional interoperation > 2016/10/28 8:41:41 -0700, Remi Forax : >> De: "mark reinhold" >> Envoy?: Mercredi 26 Octobre 2016 19:24:47 >>> 2016/10/26 9:32:42 -0700, Thomas Watson , >>> jpms-spec-comments: >>>> ... >>>> >>>> Will there will be a specified order in which the nodes are traversed >>>> when there are multiple paths to a node? More specifically will the >>>> parents have a specified order in which they are delegated to? >>> >>> Yes, the parents of a new configuration will be specified in an ordered >>> list and during resolution they'll be searched depth-first, in the order >>> in which they occur in the list. Depth-first order makes it easy to >>> reason about resolution, since it works just like a search path. >> >> I fail to seen how this is not a mini-classpath with exactly the same >> issues as the classpath. > > It's not a "mini" class path; it's more like a "mini" or "run time" > module path. It's better than a class path because it's a means to > locate whole modules rather than individual types, as discussed in > SotMS: > > http://openjdk.java.net/projects/jigsaw/spec/sotms/#the-module-path it's not a modulepath it's a moduletreepath, so i agree it does not have exactly same issue as the classpath, but the main issue is very similar, if you take a look to the list of parents, it's not enough to see which module will be chosen, like with the classpath, if you open all the jars you will find your cookie, with the moduletreepath, if you crawle the tree in deep-first order, you will find your module, in my opinion, in both cases, if you take a look to the list of jars or the list of parents, you have no idea where to find what you're looking for. And here, it's even worst, because at least the classpath has a string representation. > > - Mark R?mi From tjwatson at us.ibm.com Fri Oct 28 19:20:51 2016 From: tjwatson at us.ibm.com (Thomas Watson) Date: Fri, 28 Oct 2016 14:20:51 -0500 Subject: Dynamic modules and bidirectional interoperation In-Reply-To: <531946364.2170171.1477670491677.JavaMail.zimbra@u-pem.fr> References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net> <1316230997.272548.1477056278273.JavaMail.zimbra@u-pem.fr> <531946364.2170171.1477670491677.JavaMail.zimbra@u-pem.fr> Message-ID: > From: forax at univ-mlv.fr > To: Thomas Watson/Austin/IBM at IBMUS > Cc: mark reinhold , jpms-spec-observers > > Date: 10/28/2016 11:01 AM > Subject: Re: Dynamic modules and bidirectional interoperation > > Hi Thomas, > first, sorry, > I'm not able to parse my own mail correctly. > > Let's try again, > currently a way to solve your issue is to allow a Layer to have > several parents, > i really dislike this idea because it introduces an arbitrary order > that is hidden deep in the code thus makes really hard to reason > about which Layer resolve what, > for me it's not different to add a kind of classpath resolution in > the middle of the module resolution. I don't have near the history with JPMS as this group. I'm going out on a limb talking about the definitive behavior of the JMPS runtime, but here goes. Today layers are searched for required modules in a specific single parent order. First the local layer is searched, if not found look in the parent layer, do this until the empty layer is reached. For example, look at the following layers: A -> B -> C These are simply searched in order A, B, C. Now we introduce multiple parents: A -> B -> C \-> D -> E These are searched in order A, B, C, D, E. From the perspective of layer A I would expect a consistent resolution result as if you had a single parent hierarchy. A -> B -> C -> D -> E The problem I do see though is what happens if you have the following: layer B has modules X and Z layer D has modules Y and Z both modules X and Y requires public Z layer A has module W which requires both X and Y This better result in some kind of error regardless if single or multi-parent layers are used because in both cases module W is getting exposed to two copies of the Z module (through dependencies X.B->Z.B and Y.D->Z.D). I'm basing this on the javadoc of Configuration for findModule(String name) and resolveRequires(ModuleFinder before, ModuleFinder after, Collection roots). In my opinion Z.B is the only Z module which module W should ever be able to wire to because if it requires Z directly it could only ever wire to Z.B. But this may be over-restrictive. For example, what if W only requires Y? In that case it seems perfectly reasonable to allow W to wire to Y.D and Z.D. Is this problem made worse with multiple parent layers? This approach of using #NonHierarchicalLayers is suggested as a tool to solve the problem of pinning modules that are dynamically updated or uninstalled in a dynamic module system like OSGi. Another possible solution to that is to use a solution to #DiscardableModules, but that does require an ever growing chain of parent Layers to be created as modules are updated in the dynamic module system. > > Currently, you control the classloaders, so you can load a hand > crafted class in the module (you just have to patch the class name > to have the right package, you can use the patching capabilities of > defineAnonymousClass for that, but you can also call defineClass on > the right classloader as again, you control the classloaders). > Once you are able to execute arbitrary code in a module you can call > addReads, addExports (and soon addOpens) on the j.l.r.Module. I agree with you this could be used instead of a solution to #ReadabilityAddedByLayerCreator, but I think it is a bit messy when it doesn't have to be. A solution to #ReadabilityAddedByLayerCreator should not be too controversial. I also think this is an orthogonal issue to the pinning stale modules issue. I don't think you are suggesting this approach to somehow reach into the Layer implementation to pluck out the stale modules ourselves. That is where the monkey-patching feels more like gorilla-patching to me. > > regards, > R?mi > Tom From mark.reinhold at oracle.com Fri Oct 28 22:48:29 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Fri, 28 Oct 2016 15:48:29 -0700 Subject: Dynamic modules and bidirectional interoperation In-Reply-To: <507939175.2182737.1477674077829.JavaMail.zimbra@u-pem.fr> References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net> <20161026091545.152155830eggemoggin.niobe.net> <20161028091649.617098323eggemoggin.niobe.net> <507939175.2182737.1477674077829.JavaMail.zimbra@u-pem.fr> Message-ID: <20161028154829.70438218eggemoggin.niobe.net> 2016/10/28 10:01:17 -0700, Remi Forax : >> De: mark.reinhold at oracle.com >> Envoy?: Vendredi 28 Octobre 2016 18:16:49 >> 2016/10/28 8:41:41 -0700, Remi Forax : >>> De: mark.reinhold at oracle.com >>> Envoy?: Mercredi 26 Octobre 2016 19:24:47 >>>> ... >>>> >>>> Yes, the parents of a new configuration will be specified in an ordered >>>> list and during resolution they'll be searched depth-first, in the order >>>> in which they occur in the list. Depth-first order makes it easy to >>>> reason about resolution, since it works just like a search path. >>> >>> I fail to seen how this is not a mini-classpath with exactly the same >>> issues as the classpath. >> >> It's not a "mini" class path; it's more like a "mini" or "run time" >> module path. It's better than a class path because it's a means to >> locate whole modules rather than individual types, as discussed in >> SotMS: >> >> http://openjdk.java.net/projects/jigsaw/spec/sotms/#the-module-path > > it's not a modulepath it's a moduletreepath, so i agree it does not > have exactly same issue as the classpath, but the main issue is very > similar, if you take a look to the list of parents, it's not enough to > see which module will be chosen, like with the classpath, if you open > all the jars you will find your cookie, with the moduletreepath, if > you crawle the tree in deep-first order, you will find your module, in > my opinion, in both cases, if you take a look to the list of jars or > the list of parents, you have no idea where to find what you're > looking for. Ah, now I think I see what you're getting at, which is a completely different issue. Yes, it's true that all of the parents, direct and indirect, of a new configuration are logically arranged in a tree rather than a linear path, owing to the depth-first search. This is a behavior shared with the class path (cf. the `Class-Path` attribute in `MANIFEST.MF`). As long as we specify it clearly then I'm not convinced that the complexity is a serious problem, since only a few experts are likely ever to use this API. > And here, it's even worst, because at least the classpath > has a string representation. Why is that interesting? These lists of parent configurations (and any related lists of layers) will only ever be specified programmatically. We're not proposing this for use on the command line, or in system properties, or in any other context that needs a string representation. - Mark From mark.reinhold at oracle.com Mon Oct 31 18:31:57 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 31 Oct 2016 11:31:57 -0700 Subject: Dynamic modules and bidirectional interoperation In-Reply-To: References: <20161006170759.B4FFCD38BC@eggemoggin.niobe.net> <1316230997.272548.1477056278273.JavaMail.zimbra@u-pem.fr> <531946364.2170171.1477670491677.JavaMail.zimbra@u-pem.fr> Message-ID: <20161031113157.642562730eggemoggin.niobe.net> 2016/10/28 12:20:51 -0700, Thomas Watson : > ... > > I don't have near the history with JPMS as this group. I'm going out on a > limb talking about the definitive behavior of the JMPS runtime, but here > goes. Today layers are searched for required modules in a specific single > parent order. First the local layer is searched, if not found look in the > parent layer, do this until the empty layer is reached. This isn't quite right ... > For example, look > at the following layers: > > A -> B -> C > > These are simply searched in order A, B, C. > Now we introduce multiple parents: > > A -> B -> C > \-> D -> E > > These are searched in order A, B, C, D, E. From the perspective of layer > A I would expect a consistent resolution result as if you had a single > parent hierarchy. > > A -> B -> C -> D -> E Yes, that's what I expect also. > The problem I do see though is what happens if you have the following: > > layer B has modules X and Z > layer D has modules Y and Z > both modules X and Y requires public Z > layer A has module W which requires both X and Y (Reminder: `requires public` is being renamed to `requires transitive`.) Graphically: A { W requires X & Y } -> B { X requires transitive Z, Z } -> D { Y requires transitive Z, Z } > This better result in some kind of error regardless if single or > multi-parent layers are used because in both cases module W is getting > exposed to two copies of the Z module (through dependencies X.B->Z.B and > Y.D-> Z.D). I'm basing this on the javadoc of Configuration for > findModule(String name) and resolveRequires(ModuleFinder before, > ModuleFinder after, Collection roots). Whether or not this results in an error depends upon how you configure layer B. To compute a configuration the resolver takes two module finders, a "before" finder and an "after" finder. To locate a module by name it consults the "before" finder, the parent configuration (and then its parents recursively), and the "after" finder, in that order, until it locates a module or fails. If Z.B is in B's "after" finder (the common case) then no error will be reported. Module X in layer B will read the copy of Z in layer D (X.B -> Z.D), i.e., there will be no Z.B in the resulting module graph. Even if you ask for Z explicitly when configuring B, i.e., specify it as a root, you'll get Z.D rather than Z.D. If Z.B is in B's "before" finder (the uncommon case, principally to support `--upgrade-module-path`) then an error will be reported since W would have to read both Z.B and Z.D due to the read edges implied by the `requires transitive Z` directives in the definitions of modules X and Y. (Corner case: If the two Z modules do not define any packages of the same name then right now no error is reported; this is probably a bug.) > In my opinion Z.B is the > only Z module which module W should ever be able to wire to because if it > requires Z directly it could only ever wire to Z.B. No, if W required Z directly then it would still get Z.D since there wouldn't be a Z.B in B's configuration. > But this may be > over-restrictive. For example, what if W only requires Y? In that case it > seems perfectly reasonable to allow W to wire to Y.D and Z.D. If W only requires Y then you'd still get W -> Y.D, Z.D. > Is this > problem made worse with multiple parent layers? The semantics seem clear enough; whether they're "worse" depends upon whether you consider the current behavior to be a problem. The error scenario described above can now occur between two modules of the same name in different parents, so in theory at least there would be more opportunities for this kind of situation to arise. Whether that would turn out to be true in practice, I'm not sure. > This approach of using #NonHierarchicalLayers is suggested as a tool to > solve the problem of pinning modules that are dynamically updated or > uninstalled in a dynamic module system like OSGi. Another possible > solution to that is to use a solution to #DiscardableModules, but that > does require an ever growing chain of parent Layers to be created as > modules are updated in the dynamic module system. Right. > 2016/10/21 6:24:38 -0700, Remi Forax , > jpms-spec-observers: >> ... >> >> Let's try again, currently a way to solve your issue is to allow a >> Layer to have several parents, i really dislike this idea because it >> introduces an arbitrary order that is hidden deep in the code thus >> makes really hard to reason about which Layer resolve what, for me >> it's not different to add a kind of classpath resolution in the >> middle of the module resolution. >> >> Currently, you control the classloaders, so you can load a hand >> crafted class in the module (you just have to patch the class name >> to have the right package, you can use the patching capabilities of >> defineAnonymousClass for that, but you can also call defineClass on >> the right classloader as again, you control the classloaders). Once >> you are able to execute arbitrary code in a module you can call >> addReads, addExports (and soon addOpens) on the j.l.r.Module. > > I agree with you this could be used instead of a solution to > #ReadabilityAddedByLayerCreator, but I think it is a bit messy when it > doesn't have to be. A solution to #ReadabilityAddedByLayerCreator should > not be too controversial. I also think this is an orthogonal issue to the > pinning stale modules issue. I don't think you are suggesting this > approach to somehow reach into the Layer implementation to pluck out the > stale modules ourselves. That is where the monkey-patching feels more like > gorilla-patching to me. Agreed, though I'm not sure that's what R?mi is trying to suggest. Perhaps a more pertinent problem with R?mi's general idea is that it would only modify the run-time module graph, as instantiated via layers and updated via the j.l.r.Module API. Such updates would have no effect on the later resolution of JPMS modules since resolution refers to the unchanging configurations of the relevant layers rather than to the run-time module graph. Depending on the details I suspect this would make it difficult, if not impossible, to resolve JPMS modules against OSGi bundles surfaced as JPMS modules in the way that I suggested. - Mark From mark.reinhold at oracle.com Mon Oct 31 20:22:38 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 31 Oct 2016 13:22:38 -0700 (PDT) Subject: Proposal: #NonHierarchicalLayers Message-ID: <20161031202238.D3E5DDB037@eggemoggin.niobe.net> Issue summary ------------- #NonHierarchicalLayers --- Layers are presently constrained to be hierarchical, i.e., each layer has at most one parent. Should this restriction be relaxed so that a layer can have more than one parent? Some have argued that this will be essential to the adoption of the module system by a future version of the Java EE Platform. It would also enable bidirectional interoperation with existing module systems such as OSGi. [1] Proposal -------- Revise `java.lang.module.Configuration` and `java.lang.reflect.Layer` to augment the existing instance methods that treat `this` as the parent of the child being created with corresponding static methods that take a list of parents. The list of the parents of a configuration can be thought of as a search path. To construct a new configuration the resolver searches for modules in the given parent configurations in the order in which they occur in the list, and for any given parent it searches its parents recursively in the same fashion, before moving on to the next parent in the list. This search order is, thus, not only depth-first but also determined by the order of the parents in each list. The latter property makes it easy to tell when one candidate module will shadow another. To instantiate a new layer from a given configuration, the list of the configurations of its parent layers must be identical to the list of the parents of that configuration. The service providers for a module in a given layer are located in the same way that the resolver searches for modules, i.e., depth-first and in list order. [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#NonHierarchicalLayers From mark.reinhold at oracle.com Mon Oct 31 21:31:39 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 31 Oct 2016 14:31:39 -0700 Subject: Issue list and draft JLS/JVMS text updated Message-ID: <20161031143139.779016730eggemoggin.niobe.net> FYI, I've updated the issue list to link to the latest proposals for #ReflectiveAccessToNonExportedTypes and #AwkwardStrongEncapsulation, to link to the first proposal for #NonHierarchicalLayers, and to include the new issue #IndirectQualifiedReflectiveAccess. (I added #DiscardableModules and #ReadabilityAddedByLayerCreator last week.) http://openjdk.java.net/projects/jigsaw/spec/issues/ I've also updated the draft JLS/JVMS text to include changes for the open-modules/packages proposal: http://cr.openjdk.java.net/~mr/jigsaw/spec/lang-vm.html - Mark