From mark.reinhold at oracle.com Fri Jul 1 00:36:20 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 30 Jun 2016 17:36:20 -0700 Subject: Proposal: #ResourceEncapsulation and #ClassFilesAsResources In-Reply-To: <5773CB63.6030308@redhat.com> References: <20160628212015.99A06B59CC@eggemoggin.niobe.net> <5773CB63.6030308@redhat.com> Message-ID: <20160630173620.105142642eggemoggin.niobe.net> 2016/6/29 6:21:39 -0700, david.lloyd at redhat.com: > On 06/28/2016 04:20 PM, Mark Reinhold wrote: >> ... >> >> Make the following changes to the various `getResource*` methods: >> >> - The `ClassLoader::getResource*` methods will first delegate to the >> loader's parent, if any, and will then search for resources in the >> named modules defined to the loader, and will then search in the >> loader's unnamed module, typically defined by a class path. In the >> case of the `getResources` method the order of the resources in the >> returned enumeration will be unspecified, except that the values of >> `cl.getResources(name).nextElement()` and `cl.getResource(name)` >> will be equal. > > I am really not a fan of unspecified resource order, especially when you > can easily derive a stable order from the graph topology. > > Resources however should always load from the child-most loader first. The `getResource*` methods in java.lang.ClassLoader have delegated to the parent first since JDK 1.2, so for compatibility they'll have to keep doing that. Custom loaders can, of course, continue to do whatever they want. > This isn't a problem in Jigsaw for classes because packages cannot be > duplicated between modules (in a visible manner), but it becomes a > problem with resources, especially if/when run-time cycles are > permitted: you can run into an awkward situation where two mutually > dependent modules each may see the other's resource "first" before their > own, which is almost always undesirable. This suggests that deriving a resource order from the graph topology is maybe not as easy as you think. Is resource order, within a single class loader, really so important that we should go to the trouble of analyzing module relationships within that loader? If a container application really needs fine-grained control over resource order then it could always create a separate class loader for every module. >> - The `Class::getResource*` methods will only search for resources in >> the module that defines the class. In the case of a class in an >> unnamed module this will typically result in searching the class >> path of the corresponding class loader. > > Does this also apply to automatic modules? Yes, of course. >> The `java.lang.reflect.Module::getResourceAsStream` method will remain, >> so that it's possible to look up a resource in a module without having a >> reference to a class defined in that module. It will always be the case >> that, for a given a class `c`, `c.getResourceAsStream(name)` will be >> equivalent to `c.getModule().getResourceAsStream(name)`. > > I believe that in the former case, the "name" was historically relative > to the location of the class. Is that behavior changing? Good catch -- that's an error in my description. >> Rationale >> --------- >> >> The encapsulation of resources was intended to be part of the overall >> encapsulation story, on the view that a module's resources are just >> internal implementation details that need not, and should not, be >> available to code in other modules. Allowing external code to depend >> upon a module's internal resources is just as problematic as allowing >> such code to depend upon internal APIs. >> >> The reality of existing code today, however, is rather different, with >> two common use cases: >> >> - Resources are often used intentionally to convey information to >> external code, e.g., as a way to publish configuration files such as >> `persistence.xml`. This sort of thing should, ideally, be done via >> services [4][5], but that is not the current practice and it will >> take years to migrate to it. > > In many cases services are too limiting. It is useful for users to have > a more powerful, general facility available that can use the same basic > mechanism as services, but for arbitrary resources. For example, one > case that came up quite recently (on Jigsaw-dev I think) was that it is > useful to acquire not the service instance, but the class name, so that > it could be constructed in various ways. Services as they are now > require another front-end class to encapsulate this kind of variation. > Also it is not uncommon for a situation to arise where many classes must > be produced for variant services which all have the same basic behavior > but differ only in details, which is awkward as well as it can result in > large quantities of similar classes. > > It would be nice if service loading and resource loading continued to > follow a similar set of rules. When Jigsaw switched ServiceLoader from > a convenience API over resources to a separate concept, it lost the > concept of using resources *as* an API component or participant, which > is a powerful feature. This change will restore that to some degree, > but it is still strange that loading a META-INF/services/com.foo.Service > file will have different search semantics than using ServiceLoader. > Basically we are taking this capability from users unless they do it the > way that Jigsaw wants you to: via ServiceLoader, and no other way; it > almost seems like Jigsaw is attempting to take on the characteristics of > a container (not a very powerful one though because it only supports > linking via services (a pretty thin concept when you think about it), > not by (for example) annotations or anything else). > > Anyway I hope you can extract some kind of cogency from all that. Um, no, not really. At any rate this proposal is about resource loading, not services. If you have a specific issue to raise or proposal to make about services then please do so in a separate thread. - Mark From mark.reinhold at oracle.com Fri Jul 1 00:45:41 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 30 Jun 2016 17:45:41 -0700 Subject: Proposal: #BootstrapClassLoaderSearchInJVMTI In-Reply-To: References: <20160628212215.9E7D9B59D4@eggemoggin.niobe.net> <20160629163100.841006773eggemoggin.niobe.net> Message-ID: <20160630174541.601998191eggemoggin.niobe.net> 2016/6/30 7:11:36 -0700, tim_ellison at uk.ibm.com: > mark.reinhold at oracle.com wrote on 30/06/2016 00:31:00: >> 2016/6/29 3:45:53 -0700, tim_ellison at uk.ibm.com: >>> ... >>> >>> What would be the effect calling this function on a modular runtime when >>> passed a directory search path segment? a modular JAR? a regular JAR? >> >> It will work just as it does today. >> >> If the bootstrap class loader cannot locate a suitable class file within >> the run-time system then it will consult the path defined by calls to >> this function. Each element of the path can be either a JAR file or an >> "exploded" directory tree corresponding to a package hierarchy. Whether >> a JAR file is a modular JAR is of no import -- any module descriptor >> within it will be ignored. > > Just to check I fully understand, this function will affect the search path > of the unnamed module associated with the bootstrap class loader, correct? Correct. > If that's so then I'm happy with the proposed solution. Good! - Mark From forax at univ-mlv.fr Fri Jul 1 07:39:00 2016 From: forax at univ-mlv.fr (Remi Forax) Date: Fri, 1 Jul 2016 09:39:00 +0200 (CEST) Subject: Proposal: #FixClassFileFormat In-Reply-To: <1813643129.1709819.1467357511504.JavaMail.zimbra@u-pem.fr> Message-ID: <1751220489.1722677.1467358740078.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "mark reinhold" > ?: "Remi Forax" > Cc: jpms-spec-experts at openjdk.java.net > Envoy?: Jeudi 30 Juin 2016 02:35:59 > Objet: Re: Proposal: #ReflectiveAccessToNonExportedTypes: `exports dynamic` > > 2016/6/29 1:51:16 -0700, Remi Forax : > > I would like to agree on that proposal only on the ground that it > > requires to change the classfile format thus allows in the same time > > to fix the issues of the current classfile spec (encoding of the name > > module as a Class instead of UTF8, wrong flag for requires public), > > I think that these are completely distinct issues (though your > class-file issues are not yet in the issue list -- would you care > to suggest some text?). yes, sure. when implementing the support of module into ASM, i've discovered several issues in the classfile format as defined by [1] - module name is not encoded as UTF8. The spec makes great cares to encode reference to module in 'requires' by example as "UTF8" constant pool entry, a name with no other constraints, but fail to do the same thing for the name of the module itself. The name of the module is derived from the name of the class which is encoded as a "Class" constant pool entry. The way to solve that is to avoid to derive the name of the module from the class, the name of the class should be module-info, with no package, which seems logical given that the source file is module-info.java (not prefixed by a package) and the name of the module should be encodedat the beginning of the "Module" attribute as a reference to the constant pool entry of type UTF8. This format can also be more compact despite adding two bytes in the "Module" attribute in the fairly common case where one exported package as the same name as the module name because of the constant poll sharing the names. - there is what i believe is a typo in the current spec when specifying the modifier of a required module (requires_flags), the modifier ACC_PUBLIC is said to have the value 0x0020 instead of 0x0001 in the JVM spec. (0x0020 is either ACC_SUPER or ACC_SYNCHRONIZED). - the attributes for specifying the main-class, the module-version, a concealed module, etc are not specified, and i don't know if there is a reason to not specifying them. R?mi [1] http://cr.openjdk.java.net/~mr/jigsaw/spec/lang-vm.html From forax at univ-mlv.fr Fri Jul 1 07:52:56 2016 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Fri, 1 Jul 2016 09:52:56 +0200 (CEST) Subject: Proposal: #ModuleAnnotations and #ModuleDeprecation In-Reply-To: <20160629163718.727221014eggemoggin.niobe.net> References: <20160628211915.9706BB59C8@eggemoggin.niobe.net> <1683195038.1023897.1467191318948.JavaMail.zimbra@u-pem.fr> <20160629163718.727221014eggemoggin.niobe.net> Message-ID: <699821914.1730784.1467359576240.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "mark reinhold" > ?: "Remi Forax" > Cc: jpms-spec-experts at openjdk.java.net > Envoy?: Jeudi 30 Juin 2016 01:37:18 > Objet: Re: Proposal: #ModuleAnnotations and #ModuleDeprecation > > 2016/6/29 2:08:38 -0700, Remi Forax : > > ... > > > > I like both proposals, adding annotations and imports on module. > > > > I disagree that those annotations should not be visible at runtime, i > > don't see why we need a special case here, it's not complex to > > implement because the VM already provide the hook to access to those > > annotations at runtime, @Deprecated is an annotation visible at > > runtime for a good reason, it allows by example to implement policy > > like disallowing or emit a warning when calling a deprecated code at > > runtime. > > I completely agree, and the intent of the proposal is to make > annotations on module declarations available at run time -- that's > why it specifically mentions extending the `java.lang.reflect` API. > > If there's text in the proposal that suggests otherwise then please > point it out and I'll fix it. Thanks. > > - Mark > sorry, it's me not being able to parse "This proposal will not add support to ModuleDescriptor". So it seems we all agree on this issue :) I understand why you don't want to add support of annotations to ModuleDescriptor because creating a piece of metadata at runtime that will be seem as an annotation is something several people have tried to do (by example CDI/Weld) but the result is not very elegant. R?mi From forax at univ-mlv.fr Fri Jul 1 08:48:07 2016 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Fri, 1 Jul 2016 10:48:07 +0200 (CEST) Subject: Proposal: #ReflectiveAccessToNonExportedTypes: `exports dynamic` In-Reply-To: <20160629173559.18960061eggemoggin.niobe.net> References: <20160628211815.87E88B59C4@eggemoggin.niobe.net> <1254086925.1013150.1467190276220.JavaMail.zimbra@u-pem.fr> <20160629173559.18960061eggemoggin.niobe.net> Message-ID: <1720977211.1766068.1467362887039.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "mark reinhold" > ?: "Remi Forax" > Cc: jpms-spec-experts at openjdk.java.net > Envoy?: Jeudi 30 Juin 2016 02:35:59 > Objet: Re: Proposal: #ReflectiveAccessToNonExportedTypes: `exports dynamic` > [...] > > > but i'm not sure this proposal support it's own weight. > > > > Without dynamic, a 'classical' export works the same way at runtime so > > adding 'dynamic' to an export only restrict the usage at compile time. > > Given that we can already selectively export packages to some chosen > > modules which hide them for the rest of the other modules at compile > > time, adding dynamic to the spec adds little value. > > Qualified exports aren't really a satisfactory substitute for dynamic > exports. They still export packages at compile time, thereby tempting > unwanted usage. They're also really meant only for use within a group > of tightly-associated modules, where hashing can be used to ensure > integrity, rather than amongst unrelated modules, between which names > can all too easily be spoofed. > > More to the point, qualified exports are not adequate for all use cases. > If you want to write a module that uses JPA but is independent of JPA > implementation then you'll need to make your entity classes available > to whichever JPA implementation you do wind up using (Hibernate or > EclipseLink or ...), but you don't know the name of that > implementation's module. In other words, you can't write > > exports com.foo.app.model to jpa; > > because `jpa` is not an actual module, it's a stand-in. > > We could introduce a level of indirection so that you could do that sort > of thing, say some notion of "abstract module", but that'd be a very, > very deep change and it doesn't fit at all well with existing practices > around JAR files. Ok, qualified export to a specific module is a bad idea here, export to anybody seems a better fit. > > The nice thing about `exports dynamic` is that it lets you say "just > make the types in this package available for reflection by anybody, I'm > okay with that". There are two differences between unqualified export and exports dynamic: - exported types are only available at runtime not at compile time, - exported types are available by anybody so a module like hibernate doesn't have to requires a user module. I fail to see why these two guys are bundled together, i may want to have exported types available to anybody and visible at compile time. but i understand that having a way to say i want to export a package to anybody is interesting especially for library like hibernate. Moreover, i don't like the fact that Java the language has a new way to interact with the reflection. The more we introduce such coupling the less we will be able to move in the future. so instead of exports dynamic i propose to introduce a way to say i want to export a package to anybody, an exports to * (star). Using the example of the initial proposal, the module-info will be module com.foo.app { requires hibernate.core; requires hibernate.entitymanager; exports com.foo.app.model to *; } > > > And it has several drawbacks, the main one is that it weaken the > > compiletime/runtime fidelity story, with a dynamic export the runtime > > world is not the compiletime world anymore. The main motivation to > > add dynamic seems to be for backward compatibility during the > > transition between the classpath world and the full module world, for > > several other cases, we use the command line for that, i don't see why > > this case is so special that it should not work like the other > > compatibility issues. > > Yes, it does weaken the fidelity story in order to accommodate existing > practice, but at least it does so in an explicit manner so that what's > going on is apparent just from reading the code. > > I don't think, however, that this change is just about compatibility and > migration. Even in an idealistic, Utopian, fully-modular world I expect > that we'll still want to use frameworks that use reflection on our own > types to help us be more productive. yes, that's very true. Reflection allows to write meta object protocols but making a special case for the reflection seems wrong to me. > > > Moreover, an agent, an annotation processor and a jlink plugin are all > > able to implement the same trick, add export after the compilation and > > before the runtime, so i don't think it's a good idea to change the > > spec for that. > > Then how do you express, in your source code, that you want an export to > be added at run time? > > You could use an annotation, but that would require anybody who wants > to do this sort of thing to configure their environment to include the > module that defines that annotation and to add the required annotation > processor at the appropriate point. Do we really want to tell people > to do that when we can solve the problem for them so simply? > > - Mark > R?mi From mark.reinhold at oracle.com Fri Jul 1 17:10:55 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Fri, 01 Jul 2016 10:10:55 -0700 Subject: Proposal: #ReflectiveAccessToNonExportedTypes: `exports dynamic` In-Reply-To: <1720977211.1766068.1467362887039.JavaMail.zimbra@u-pem.fr> References: <20160628211815.87E88B59C4@eggemoggin.niobe.net> <1254086925.1013150.1467190276220.JavaMail.zimbra@u-pem.fr> <20160629173559.18960061eggemoggin.niobe.net> <1720977211.1766068.1467362887039.JavaMail.zimbra@u-pem.fr> Message-ID: <20160701101055.285920310eggemoggin.niobe.net> 2016/7/1 1:48:07 -0700, Remi Forax : > mark.reinhold at oracle.com: >> ... >> >> The nice thing about `exports dynamic` is that it lets you say "just >> make the types in this package available for reflection by anybody, I'm >> okay with that". > > There are two differences between unqualified export and exports dynamic: > - exported types are only available at runtime not at compile time, > - exported types are available by anybody so a module like hibernate > doesn't have to requires a user module. This second point is not correct: Using `exports dynamic` in a module declaration does not mean that all other modules will automatically read that module. In module declarations, `exports` directives do not affect readability; only `requires` does that. Both normal, unqualified exports and dynamic exports make the public types in a package available to modules that read the exporting module. In the case of a framework like Hibernate, which uses reflection, the required readability will come "for free" thanks to the resolution of the #ReflectionWithoutReadability issue [1] (your suggestion). > I fail to see why these two guys are bundled together, i may want to > have exported types available to anybody and visible at compile time. > but i understand that having a way to say i want to export a package > to anybody is interesting especially for library like hibernate. > > Moreover, i don't like the fact that Java the language has a new way > to interact with the reflection. The more we introduce such coupling > the less we will be able to move in the future. We've already introduced new ways for the language to interact with reflection, in fact in the solution to #ReflectionWithoutReadability. At any rate, the relationship between the language and the various reflective APIs is, almost by definition, very intimate, so I don't think we should be surprised that revisions to the language introduce new interactions in this relationship. Do you have specific examples in mind of undesirable coupling due to this new interaction? > so instead of exports dynamic i propose to introduce a way to say i > want to export a package to anybody, an exports to * (star). > > Using the example of the initial proposal, the module-info will be > > module com.foo.app { > requires hibernate.core; > requires hibernate.entitymanager; > exports com.foo.app.model to *; > } This is exactly what `exports com.foo.app.model` already does today. It exports the public types in that package to any module that reads the `com.foo.app` module. There's no need to add `to *`. Whether some other module reads `com.foo.app` is, as I wrote above, an issue that's orthogonal to whether and how any particular package is exported, and in the case of reflective access is a non-issue since reflection now simply assumes readability. >> Remi Forax : >>> ... >>> >>> And it has several drawbacks, the main one is that it weaken the >>> compiletime/runtime fidelity story, with a dynamic export the runtime >>> world is not the compiletime world anymore. The main motivation to >>> add dynamic seems to be for backward compatibility during the >>> transition between the classpath world and the full module world, for >>> several other cases, we use the command line for that, i don't see why >>> this case is so special that it should not work like the other >>> compatibility issues. >> >> Yes, it does weaken the fidelity story in order to accommodate existing >> practice, but at least it does so in an explicit manner so that what's >> going on is apparent just from reading the code. >> >> I don't think, however, that this change is just about compatibility and >> migration. Even in an idealistic, Utopian, fully-modular world I expect >> that we'll still want to use frameworks that use reflection on our own >> types to help us be more productive. > > yes, that's very true. Reflection allows to write meta object > protocols but making a special case for the reflection seems wrong to > me. Just to be clear, the `exports dynamic` idea is motivated by existing uses of reflection, but it's not intimately tied to reflection. The specification won't directly tie `exports dynamic` to the reflection APIs. Those APIs will, however, work more naturally as a consequence of the semantics of `exports dynamic`, which will be defined entirely in terms of the modular access-control concepts we already have. - Mark [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ReflectionWithoutReadability From mark.reinhold at oracle.com Fri Jul 1 17:43:15 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Fri, 01 Jul 2016 10:43:15 -0700 Subject: Proposal: #FixClassFileFormat In-Reply-To: <1751220489.1722677.1467358740078.JavaMail.zimbra@u-pem.fr> References: <1813643129.1709819.1467357511504.JavaMail.zimbra@u-pem.fr> <1751220489.1722677.1467358740078.JavaMail.zimbra@u-pem.fr> Message-ID: <20160701104315.789031669eggemoggin.niobe.net> 2016/7/1 0:39:00 -0700, Remi Forax : > mark.reinhold at oracle.com: >> 2016/6/29 1:51:16 -0700, Remi Forax : >>> I would like to agree on that proposal only on the ground that it >>> requires to change the classfile format thus allows in the same time >>> to fix the issues of the current classfile spec (encoding of the name >>> module as a Class instead of UTF8, wrong flag for requires public), >> >> I think that these are completely distinct issues (though your >> class-file issues are not yet in the issue list -- would you care >> to suggest some text?). > > yes, sure. > > when implementing the support of module into ASM, i've discovered > several issues in the classfile format as defined by [1] > > - module name is not encoded as UTF8. > > The spec makes great cares to encode reference to module in > 'requires' by example as "UTF8" constant pool entry, a name with no > other constraints, but fail to do the same thing for the name of the > module itself. > > The name of the module is derived from the name of the class which > is encoded as a "Class" constant pool entry. The way to solve that > is to avoid to derive the name of the module from the class, the > name of the class should be module-info, with no package, which > seems logical given that the source file is module-info.java (not > prefixed by a package) and the name of the module should be > encoded at the beginning of the "Module" attribute as a reference to > the constant pool entry of type UTF8. This format can also be more > compact despite adding two bytes in the "Module" attribute in the > fairly common case where one exported package as the same name as > the module name because of the constant poll sharing the names. > > - there is what i believe is a typo in the current spec when > specifying the modifier of a required module (requires_flags), the > modifier ACC_PUBLIC is said to have the value 0x0020 instead of > 0x0001 in the JVM spec. (0x0020 is either ACC_SUPER or > ACC_SYNCHRONIZED). > > - the attributes for specifying the main-class, the module-version, a > concealed module, etc are not specified, and i don't know if there > is a reason to not specifying them. On this last point: They should be specified, and will be specified in an upcoming revision to the spec. Before digging into the other two issues I'd like to try just to record them, without any bias toward a particular solution, so I'll add the following two items to the issue list: #ClassFileModuleName --- The name of a module is not a simple UTF-8 string but is, rather, derived from the `this_class` field of the `ClassFile` structure, which is awkward. #ClassFileRequires --- The `ACC_PUBLIC` constant in a `requires_flags` should be encoded as `0x0001`, as it is elsewhere in the JVMS, rather than as `0x0020`, which has different meanings in other contexts. - Mark From forax at univ-mlv.fr Fri Jul 1 20:20:25 2016 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Fri, 1 Jul 2016 22:20:25 +0200 (CEST) Subject: Proposal: #FixClassFileFormat In-Reply-To: <20160701104315.789031669eggemoggin.niobe.net> References: <1813643129.1709819.1467357511504.JavaMail.zimbra@u-pem.fr> <1751220489.1722677.1467358740078.JavaMail.zimbra@u-pem.fr> <20160701104315.789031669eggemoggin.niobe.net> Message-ID: <321002664.2003394.1467404425232.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "mark reinhold" > ?: "Remi Forax" > Cc: jpms-spec-experts at openjdk.java.net > Envoy?: Vendredi 1 Juillet 2016 19:43:15 > Objet: Re: Proposal: #FixClassFileFormat > > 2016/7/1 0:39:00 -0700, Remi Forax : > > mark.reinhold at oracle.com: > >> 2016/6/29 1:51:16 -0700, Remi Forax : > >>> I would like to agree on that proposal only on the ground that it > >>> requires to change the classfile format thus allows in the same time > >>> to fix the issues of the current classfile spec (encoding of the name > >>> module as a Class instead of UTF8, wrong flag for requires public), > >> > >> I think that these are completely distinct issues (though your > >> class-file issues are not yet in the issue list -- would you care > >> to suggest some text?). > > > > yes, sure. > > > > when implementing the support of module into ASM, i've discovered > > several issues in the classfile format as defined by [1] > > > > - module name is not encoded as UTF8. > > > > The spec makes great cares to encode reference to module in > > 'requires' by example as "UTF8" constant pool entry, a name with no > > other constraints, but fail to do the same thing for the name of the > > module itself. > > > > The name of the module is derived from the name of the class which > > is encoded as a "Class" constant pool entry. The way to solve that > > is to avoid to derive the name of the module from the class, the > > name of the class should be module-info, with no package, which > > seems logical given that the source file is module-info.java (not > > prefixed by a package) and the name of the module should be > > encoded at the beginning of the "Module" attribute as a reference to > > the constant pool entry of type UTF8. This format can also be more > > compact despite adding two bytes in the "Module" attribute in the > > fairly common case where one exported package as the same name as > > the module name because of the constant poll sharing the names. > > > > - there is what i believe is a typo in the current spec when > > specifying the modifier of a required module (requires_flags), the > > modifier ACC_PUBLIC is said to have the value 0x0020 instead of > > 0x0001 in the JVM spec. (0x0020 is either ACC_SUPER or > > ACC_SYNCHRONIZED). > > > > - the attributes for specifying the main-class, the module-version, a > > concealed module, etc are not specified, and i don't know if there > > is a reason to not specifying them. > > On this last point: They should be specified, and will be specified in > an upcoming revision to the spec. > > Before digging into the other two issues I'd like to try just to record > them, without any bias toward a particular solution, so I'll add the > following two items to the issue list: > > #ClassFileModuleName --- The name of a module is not a simple UTF-8 > string but is, rather, derived from the `this_class` field of the > `ClassFile` structure, which is awkward. > > #ClassFileRequires --- The `ACC_PUBLIC` constant in a `requires_flags` > should be encoded as `0x0001`, as it is elsewhere in the JVMS, rather > than as `0x0020`, which has different meanings in other contexts. > yes ! and i've forgotten a third issue, ACC_MODULE is specified as 0x8000, 0x8000 is very special value because it's the only one that left which is available on class, method and field. Historically, in the first specs, "module" was a modifier like public or private thus using the same value for all the elements was making sense, but now ACC_MODULE can only applied on class so using a value available on all the elements make less sense, the value 0x8000 should be free in order to be used later. The possible value for ACC_MODULE are: 0x0008 (ACC_STATIC), 0x0040 (ACC_VOLATILE or ACC_BRIDGE), 0x0080 (ACC_VARARGS or ACC_TRANSIENT), 0x0100 (ACC_NATIVE), 0x0800 (ACC_STRICT). So either, 0x0040 or 0x0080 are the best candidate because the other values are only valid either on field or on method but not on both. And now, because 0x0080 is a permutation of 0x8000 in term of characters , i think 0x0040 is the best value for ACC_MODULE. Changing the value of ACC_MODULE is interesting if it's done at the same time as the other changes because the compiler and the VM can easily detect and reject the classfile encoded with what will become an old format. > - Mark > R?mi From david.lloyd at redhat.com Tue Jul 5 14:41:52 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 5 Jul 2016 09:41:52 -0500 Subject: Proposal: #DefaultModule Message-ID: <577BC730.1040706@redhat.com> I propose that the concept of "unnamed module" be dropped in favor of "default module". The main difference is that the class loader (or module finder or layer configuration or someone else) would be allowed to (but not required to) assign a free-form name and version string to this module. This would allow existing module systems to bring their module concept into some form of consonance with Jigsaw without compromising any of the restrictions that Jigsaw-style modules have. Effecting this change would suggest the addition of an "isDefault()" method on Module, possibly replacing "isNamed()" (which is arguably already somewhat redundant with respect to getName()). Also at some stage, something would have to establish the default module name and version strings, probably defaulting to the (current) null strings to keep a stable status quo. -- - DML From mark.reinhold at oracle.com Wed Jul 6 14:49:17 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 06 Jul 2016 07:49:17 -0700 Subject: Proposal: #FixClassFileFormat In-Reply-To: <321002664.2003394.1467404425232.JavaMail.zimbra@u-pem.fr> References: <1813643129.1709819.1467357511504.JavaMail.zimbra@u-pem.fr> <1751220489.1722677.1467358740078.JavaMail.zimbra@u-pem.fr> <20160701104315.789031669eggemoggin.niobe.net> <321002664.2003394.1467404425232.JavaMail.zimbra@u-pem.fr> Message-ID: <20160706074917.217942295eggemoggin.niobe.net> 2016/7/1 13:20:25 -0700, Remi Forax : > ... > > and i've forgotten a third issue, ACC_MODULE is specified as 0x8000, > 0x8000 is very special value because it's the only one that left which > is available on class, method and field. Historically, in the first > specs, "module" was a modifier like public or private thus using the > same value for all the elements was making sense, but now ACC_MODULE > can only applied on class so using a value available on all the > elements make less sense, the value 0x8000 should be free in order to > be used later. The possible value for ACC_MODULE are: 0x0008 > (ACC_STATIC), 0x0040 (ACC_VOLATILE or ACC_BRIDGE), 0x0080 (ACC_VARARGS > or ACC_TRANSIENT), 0x0100 (ACC_NATIVE), 0x0800 (ACC_STRICT). So > either, 0x0040 or 0x0080 are the best candidate because the other > values are only valid either on field or on method but not on > both. And now, because 0x0080 is a permutation of 0x8000 in term of > characters , i think 0x0040 is the best value for ACC_MODULE. > > Changing the value of ACC_MODULE is interesting if it's done at the > same time as the other changes because the compiler and the VM can > easily detect and reject the classfile encoded with what will become > an old format. I'll enter this in the issue list as: #ClassFileAccModule_ --- The `ACC_MODULE` constant is currently specified to have the value `0x8000`. This is the last available bit remaining across all of the various `access_flags` fields of a class file, and thus should be reserved for some unspecified future purpose where it may be useful to use the same value in all such fields. Alternative candidates for `ACC_MODULE` include `0x0040` (overlaps with `ACC_VOLATILE` and `ACC_BRIDGE`) and `0x0080` (`ACC_TRANSIENT` and `ACC_VARARGS`). - Mark From mark.reinhold at oracle.com Wed Jul 6 15:15:36 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 06 Jul 2016 08:15:36 -0700 Subject: Issue list updated Message-ID: <20160706081536.856940838eggemoggin.niobe.net> FYI, I've updated the issue list to add #AddExportsInManifest and R?mi's various class-file issues (#ClassFileModuleName, #ClassFileAccPublic, and #ClassFileAccModule): http://openjdk.java.net/projects/jigsaw/spec/issues/ - Mark From mark.reinhold at oracle.com Wed Jul 6 15:32:36 2016 From: mark.reinhold at oracle.com (Mark Reinhold) Date: Wed, 6 Jul 2016 08:32:36 -0700 (PDT) Subject: Proposal: #ClassFileAccPublic Message-ID: <20160706153236.A62DBB6B74@eggemoggin.niobe.net> Issue summary ------------- #ClassFileAccPublic --- The `ACC_PUBLIC` constant in a `requires_flags` should be encoded as `0x0001`, as it is elsewhere in the JVMS, rather than as `0x0020`, which has different meanings in other contexts. Proposal -------- Unlike the other `ACC_PUBLIC` constants, this constant does not express an access mode but, rather, the addition of an implied transitive readability edge from modules that depend upon this module to some other module required by this module. To avoid confusion at the class-file level, rename this constant to `ACC_TRANSITIVE`. (Whether the syntax of module declarations should use a keyword other than `public` for this purpose would be a separate issue.) [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ClassFileAccPublic From forax at univ-mlv.fr Wed Jul 6 16:23:52 2016 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 6 Jul 2016 18:23:52 +0200 (CEST) Subject: Proposal: #ClassFileAccPublic In-Reply-To: <20160706153236.A62DBB6B74@eggemoggin.niobe.net> References: <20160706153236.A62DBB6B74@eggemoggin.niobe.net> Message-ID: <823657001.1414112.1467822232509.JavaMail.zimbra@u-pem.fr> ACC_TRANSITIVE looks fine to me, even if it really means transitive at one hop only. R?mi ----- Mail original ----- > De: "Mark Reinhold" > ?: jpms-spec-experts at openjdk.java.net > Envoy?: Mercredi 6 Juillet 2016 17:32:36 > Objet: Proposal: #ClassFileAccPublic > > Issue summary > ------------- > > #ClassFileAccPublic --- The `ACC_PUBLIC` constant in a `requires_flags` > should be encoded as `0x0001`, as it is elsewhere in the JVMS, rather > than as `0x0020`, which has different meanings in other contexts. > > Proposal > -------- > > Unlike the other `ACC_PUBLIC` constants, this constant does not express > an access mode but, rather, the addition of an implied transitive > readability edge from modules that depend upon this module to some other > module required by this module. To avoid confusion at the class-file > level, rename this constant to `ACC_TRANSITIVE`. > > (Whether the syntax of module declarations should use a keyword other > than `public` for this purpose would be a separate issue.) > > > [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ClassFileAccPublic > From forax at univ-mlv.fr Wed Jul 6 16:24:12 2016 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Wed, 6 Jul 2016 18:24:12 +0200 (CEST) Subject: Proposal: #FixClassFileFormat In-Reply-To: <20160706074917.217942295eggemoggin.niobe.net> References: <1813643129.1709819.1467357511504.JavaMail.zimbra@u-pem.fr> <1751220489.1722677.1467358740078.JavaMail.zimbra@u-pem.fr> <20160701104315.789031669eggemoggin.niobe.net> <321002664.2003394.1467404425232.JavaMail.zimbra@u-pem.fr> <20160706074917.217942295eggemoggin.niobe.net> Message-ID: <1055247880.1414152.1467822252693.JavaMail.zimbra@u-pem.fr> thanks, R?mi ----- Mail original ----- > De: "mark reinhold" > ?: forax at univ-mlv.fr > Cc: jpms-spec-experts at openjdk.java.net > Envoy?: Mercredi 6 Juillet 2016 16:49:17 > Objet: Re: Proposal: #FixClassFileFormat > > 2016/7/1 13:20:25 -0700, Remi Forax : > > ... > > > > and i've forgotten a third issue, ACC_MODULE is specified as 0x8000, > > 0x8000 is very special value because it's the only one that left which > > is available on class, method and field. Historically, in the first > > specs, "module" was a modifier like public or private thus using the > > same value for all the elements was making sense, but now ACC_MODULE > > can only applied on class so using a value available on all the > > elements make less sense, the value 0x8000 should be free in order to > > be used later. The possible value for ACC_MODULE are: 0x0008 > > (ACC_STATIC), 0x0040 (ACC_VOLATILE or ACC_BRIDGE), 0x0080 (ACC_VARARGS > > or ACC_TRANSIENT), 0x0100 (ACC_NATIVE), 0x0800 (ACC_STRICT). So > > either, 0x0040 or 0x0080 are the best candidate because the other > > values are only valid either on field or on method but not on > > both. And now, because 0x0080 is a permutation of 0x8000 in term of > > characters , i think 0x0040 is the best value for ACC_MODULE. > > > > Changing the value of ACC_MODULE is interesting if it's done at the > > same time as the other changes because the compiler and the VM can > > easily detect and reject the classfile encoded with what will become > > an old format. > > I'll enter this in the issue list as: > > #ClassFileAccModule_ --- The `ACC_MODULE` constant is currently > specified to have the value `0x8000`. This is the last available bit > remaining across all of the various `access_flags` fields of a class > file, and thus should be reserved for some unspecified future purpose > where it may be useful to use the same value in all such fields. > Alternative candidates for `ACC_MODULE` include `0x0040` (overlaps > with `ACC_VOLATILE` and `ACC_BRIDGE`) and `0x0080` (`ACC_TRANSIENT` > and `ACC_VARARGS`). > > - Mark > From forax at univ-mlv.fr Wed Jul 6 16:48:19 2016 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 6 Jul 2016 18:48:19 +0200 (CEST) Subject: Proposal: #DefaultModule In-Reply-To: <577BC730.1040706@redhat.com> References: <577BC730.1040706@redhat.com> Message-ID: <1007392272.1419315.1467823699221.JavaMail.zimbra@u-pem.fr> Hi David, Correct me if i'm wrong, it seems like the proposal to be able to specify how to find the name and the version of an automatic module (i.e. #CustomizableAutomaticModuleNameMapping) but for the default module. The idea is that an existing module systems will be able to provide a name and a version for the default module of the layers it controls. so this issue should be named #CustomizableDefaultModuleNameMapping and i'm fine with it (obviously the devil is in the detail, i.e. how to do implement that ?) and the name "default module" seems to be a better name that the unamed module. When naming something, avoid name that refers to a property and use name that refers to the concept said an old professor of me. R?mi ----- Mail original ----- > De: "David M. Lloyd" > ?: jpms-spec-experts at openjdk.java.net > Cc: "jigsaw-dev" > Envoy?: Mardi 5 Juillet 2016 16:41:52 > Objet: Proposal: #DefaultModule > > I propose that the concept of "unnamed module" be dropped in favor of > "default module". The main difference is that the class loader (or > module finder or layer configuration or someone else) would be allowed > to (but not required to) assign a free-form name and version string to > this module. This would allow existing module systems to bring their > module concept into some form of consonance with Jigsaw without > compromising any of the restrictions that Jigsaw-style modules have. > > Effecting this change would suggest the addition of an "isDefault()" > method on Module, possibly replacing "isNamed()" (which is arguably > already somewhat redundant with respect to getName()). Also at some > stage, something would have to establish the default module name and > version strings, probably defaulting to the (current) null strings to > keep a stable status quo. > > -- > - DML > From mark.reinhold at oracle.com Wed Jul 6 23:13:26 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 06 Jul 2016 16:13:26 -0700 Subject: Updated EA builds with initial implementations of current proposals Message-ID: <20160706161326.25590246eggemoggin.niobe.net> FYI: http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008467.html - Mark From mark.reinhold at oracle.com Wed Jul 6 23:37:11 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 06 Jul 2016 16:37:11 -0700 Subject: Draft JLS and JVMS changes updated Message-ID: <20160706163711.319138800eggemoggin.niobe.net> http://cr.openjdk.java.net/~mr/jigsaw/spec/lang-vm.html This version includes changes for the proposals for: #CompileTimeDependences (`requires static`) #ReflectiveAccessToNonExportedTypes (`exports dynamic`) #ModuleAnnotations #ClassFileAccPublic It also defines class-file attributes for concealed packages, version strings, main classes, and target platform information. - Mark From forax at univ-mlv.fr Thu Jul 7 08:09:33 2016 From: forax at univ-mlv.fr (Remi Forax) Date: Thu, 7 Jul 2016 10:09:33 +0200 (CEST) Subject: Draft JLS and JVMS changes updated In-Reply-To: <20160706163711.319138800eggemoggin.niobe.net> References: <20160706163711.319138800eggemoggin.niobe.net> Message-ID: <123878428.1541468.1467878973727.JavaMail.zimbra@u-pem.fr> Hi Mark, There is a slight difference between the implementation and the spec, the current implementation in the jake repository allows exports_flags to be a combination of ACC_MANDATED and ACC_SYNTHETIC too. R?mi ----- Mail original ----- > De: "mark reinhold" > ?: jpms-spec-experts at openjdk.java.net > Envoy?: Jeudi 7 Juillet 2016 01:37:11 > Objet: Draft JLS and JVMS changes updated > > > http://cr.openjdk.java.net/~mr/jigsaw/spec/lang-vm.html > > This version includes changes for the proposals for: > > #CompileTimeDependences (`requires static`) > #ReflectiveAccessToNonExportedTypes (`exports dynamic`) > #ModuleAnnotations > #ClassFileAccPublic > > It also defines class-file attributes for concealed packages, version > strings, main classes, and target platform information. > > - Mark > From mark.reinhold at oracle.com Wed Jul 13 14:28:13 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 13 Jul 2016 07:28:13 -0700 (PDT) Subject: Discussion: #NonHierarchicalLayers In-Reply-To: <56D7802A.9070305@redhat.com> References: <56D7802A.9070305@redhat.com> Message-ID: <20160713142813.C0075B78C2@eggemoggin.niobe.net> Reference: http://openjdk.java.net/projects/jigsaw/spec/issues/#NonHierarchicalLayers 2016/3/2 16:07:06 -0800, david.lloyd at redhat.com: > If there is any expectation of moving existing systems over to Jigsaw, I > think that the strict hierarchical layer system should be abandoned in > favor of a graph-oriented system. This allows containers to maintain > separate module graphs with separate namespaces which nevertheless > interconnect. I suspect that generalizing the design to allow non-hierarchical layers is possible, but tricky. Both configurations and layers would be able to have multiple parents, so the API would become more complex, but perhaps that's okay since this is more for expert than casual use. To avoid having to deal with conflicts between parents (i.e., the diamond-import problem), it'd likely be best to make the collections of parents be lists rather than sets, so that they're searched in a determined order. > I believe that this is a necessary requirement if (in particular) there > is an expectation that Java EE 9 (or another Java EE specification) > would be reframed in terms of Jigsaw-style modules, since Java EE module > names have many fewer restrictions, and it is possible for (for example) > more than one application to contain modules of the same name. I don't understand how these two observations motivate non-hierarchical layers, but maybe I'm missing something. Module names are an entirely distinct issue [1]. The constraints on module names could be relaxed regardless of whether or not layers are hierarchical. It's already possible to have multiple modules of the same name with simple hierarchical layers. Two hosted applications can contain a module of the same name as long as they're loaded into distinct, unrelated layers. (Even two layers that are related can contain modules of the same name, in which case the resolver chooses the module in the nearest layer.) Are there other reasons why a hypothetical future Java EE specification, which allows applications to be built from collections of JPMS modules, might necessitate non-hierarchical layers? > If on the other hand there is an expectation that Java EE is not going > to be retrofitted in a way which maps EE applications and modules to > Jigsaw modules, then this should be clearly declared. As I wrote earlier, we definitely do have that expectation -- that's why the requirements include an entire section on dynamic configuration. - Mark [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ModuleNameCharacters From mark.reinhold at oracle.com Wed Jul 13 14:29:13 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 13 Jul 2016 07:29:13 -0700 (PDT) Subject: Discussion: #ResourceExistenceAndSize In-Reply-To: <56D85B77.4040607@redhat.com> References: <56D85B77.4040607@redhat.com> Message-ID: <20160713142913.C1C1FB78C7@eggemoggin.niobe.net> Reference: http://openjdk.java.net/projects/jigsaw/spec/issues/#ResourceExistenceAndSize 2016/3/3 7:42:47 -0800, david.lloyd at redhat.com: > Module.getResourceAsStream() is the only way a resource can be loaded > from a module. This makes it impossible to do either of the following: > > - Determine the size of a resource in advance of loading it > - Determine the existence of a resource without opening it > > Loading resources as java.net.URLs allowed these things to be done > (among other things), however it is also disadvantageous due to the > heavyweight nature of that class. What are some use cases that require determining the existence or size of a resource without loading it? > > I had earlier come up with a simple patch [1], (I'm not going to read your patch on GitHub since the comment on your repo clearly states that it is not a contribution to this JSR nor to OpenJDK. If you wish the EG to consider it then please either post it here or contribute it explicitly to OpenJDK via some other means.) > initially as a way to > address JDK-6865375, which introduced the idea of a Resource class > that is associated with a ClassLoader, with a name, URL, size, and > stream factory method, which would also be a clean solution here (though > with the addition of a Module property as well since in Jigsaw they're > separate). The issues laid out in JDK-6865375 [1] are entirely different from those of resource existence and size, though I do see how a common solution could address all these problems. > The idea is to put resources on a similar footing to classes, whereby > you can (if suitably permitted) acquire their class loaders just like > Class.getClassLoader() allows you to do for classes. This also allows > things like ServiceLoader to load classes directly from the module which > contained the corresponding service descriptor. I'm reluctant to introduce a brand-new general "Resource" abstraction, along with all the complexity that would entail. There is, among other things, a tricky compatibility issue here, as one of the comments to JDK-6865375 points out: Any custom class loader that today overrides the various getResource* methods would have to be retrofitted to override all new "Resource"-returning methods. At any rate, I don't see anything here that's inherently part of the design of the module system, so I intend to close this issue as unrelated. - Mark [1] https://bugs.openjdk.java.net/browse/JDK-6865375 From mark.reinhold at oracle.com Wed Jul 13 14:30:13 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 13 Jul 2016 07:30:13 -0700 (PDT) Subject: Discussion: #MutableConfigurations In-Reply-To: <56D79D56.7040801@redhat.com> References: <56D79D56.7040801@redhat.com> Message-ID: <20160713143013.C3C3AB78CC@eggemoggin.niobe.net> Reference: http://openjdk.java.net/projects/jigsaw/spec/issues/#MutableConfigurations 2016/3/2 18:11:34 -0800, david.lloyd at redhat.com: > It appears from what I can see in the Jigsaw code, that once a > Configuration is calculated, it cannot be changed in any way, That's true. The overall model in the present design is that, given a set of modules, you first compute a configuration, which captures the resolution of all the modules' dependences in a consistent fashion. You then instantiate that as a Layer (or even more than one Layer, if you want). This model is motivated by one of our primary goals, namely reliable configuration. Resolving a complete configuration, rather than doing so incrementally, allows the early detection of missing, duplicate, and conflicting dependencies. > which > makes them unsuitable for use in containers which add and remove modules > at run time. It is not clear how such containers are expected to > function. They're expected to create Layers, which can be related hierarchically as needed (or perhaps more generally, cf. #NonHierarchicalLayers [1]). > If there is a deliberate intention that the Jigsaw system > will not support dynamic modification of layers/configurations > (including the dynamic addition and removal of modules at run time), > then that should be explicitly stated. I don't think it makes sense for Configurations to be mutable, for the reasons stated above. It may be that the Layer concept needs to be made more dynamic in certain ways, but that's a different matter. I intend to close this issue unless there are strong objections from other EG members. - Mark [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#NonHierarchicalLayers From mark.reinhold at oracle.com Wed Jul 13 14:31:13 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 13 Jul 2016 07:31:13 -0700 (PDT) Subject: Discussion: #LazyConfigurationAndInstantiation In-Reply-To: <56E33EEC.20706@redhat.com> References: <56D7AE09.1030003@redhat.com> <20160311174751.5399C9E891@eggemoggin.niobe.net> <56E33EEC.20706@redhat.com> Message-ID: <20160713143113.C5B45B78D2@eggemoggin.niobe.net> Reference: http://openjdk.java.net/projects/jigsaw/spec/issues/#LazyConfigurationAndInstantiation 2016/3/11 13:55:56 -0800, david.lloyd at redhat.com: > On 03/11/2016 11:47 AM, mark.reinhold at oracle.com wrote: >> 2016/3/2 19:22 -0800, david.lloyd at redhat.com: >>> It looks as though the instantiation of a Layer causes a complete load >>> of all the modules in the layer. >> >> What do you mean by "complete load"? Loading all the classes? > > No I mean locating and loading all of the descriptors and building all > the wiring information for every module. Yes, that's what happens. Resolving a Configuration requires reading all of the relevant module descriptors. Instantiating a Configuration into a Layer builds a complete run-time module graph. Both of these operations are linear in the number of modules in the graph. >>> Apart from the consequent lack of >>> dynamicism, I think this is very likely to cause problems for large >>> applications, particularly if such an application might have many >>> operation modes, some of which do not require *all* of the bundled >>> modules to be loaded every single time. >>> >>> Can there not instead be an incremental resolution algorithm, akin to >>> how classes are lazily loaded? >> >> Configuring a set of modules and instantiating that configuration as a >> layer requires no more than reading the modules' descriptors. Nothing >> else from any module definition will be read until it's actually needed. > > But it does require that all the module descriptors from a given layer > be available, and that the load time for the first load of a module in a > layer will always be bounded by the size of the layer, rather than just > by the dependency subgraph of the module being loaded. Correct. > Based on > application server deployments that I know about, I think the far upper > bound for a realistic number of modules in a layer will probably lie in > the thousands to ten thousands range (though there is always the outlier > case where someone has to push it to see how far it goes...), which > might translate into a substantial startup overhead. In our experience with the prototype so far, the time required to resolve a Configuration and instantiate it as a Layer is dominated by the time required to locate and read module descriptors, typically from inside individual artifacts in the filesystem. One fairly straightforward way to speed that up is to link your modules into a custom run-time image, so that the descriptors are all in one optimized artifact [1]. If that's not feasible then you could achieve much the same effect in the build, installation, or startup process of a large application: Construct an optimized cache of the descriptors of all your modules, and use a custom ModuleFinder to load the descriptors from the cache but other module content from more-traditional artifacts. > Another potential > issue is requiring that all modules be present, though this is more > closely related to #MutableConfigurations now I think; I suspect this > issue could be mitigated with an acceptable solution to that issue. As I wrote in my reply re. #MutableConfigurations [2], I think this approach is at odds with our goal to provide reliable configuration. >> If you really want to avoid configuring all modules in certain operation >> modes, do layers not provide sufficient flexibility? Load the core of >> your application into the boot layer, figure out which modules you need >> for the requested operation mode, and then create an additional layer to >> load those modules. > > That can work in some cases, but only if things are loaded one > additional time (rather than on-demand as configuration changes, for > example) or in such a way that you could just keep adding layers. But > in the event of one large intermeshed layer, this won't work without > #MutableConfigurations and/or #NonHierarchicalLayers and something to > coordinate the load. Suppose that we allow #NonHierarchicalLayers [3]. Are there realistic use cases in which layers would not then provide sufficient flexibility? - Mark [1] http://openjdk.java.net/jeps/282 [2] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-July/000359.html [3] http://openjdk.java.net/projects/jigsaw/spec/issues/#NonHierarchicalLayers From mark.reinhold at oracle.com Wed Jul 13 14:32:13 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 13 Jul 2016 07:32:13 -0700 (PDT) Subject: Proposal: #DefaultModule In-Reply-To: <577BC730.1040706@redhat.com> References: <577BC730.1040706@redhat.com> Message-ID: <20160713143213.CF094B78D7@eggemoggin.niobe.net> 2016/7/5 7:41:52 -0700, david.lloyd at redhat.com: > I propose that the concept of "unnamed module" be dropped in favor of > "default module". The main difference is that the class loader (or > module finder or layer configuration or someone else) would be allowed > to (but not required to) assign a free-form name and version string to > this module. This would allow existing module systems to bring their > module concept into some form of consonance with Jigsaw without > compromising any of the restrictions that Jigsaw-style modules have. > > Effecting this change would suggest the addition of an "isDefault()" > method on Module, possibly replacing "isNamed()" (which is arguably > already somewhat redundant with respect to getName()). Also at some > stage, something would have to establish the default module name and > version strings, probably defaulting to the (current) null strings to > keep a stable status quo. The concept of unnamed modules has worked out pretty well so far. The analogy with unnamed packages, while imprecise, has made it fairly easy to explain to people. The fact that an unnamed module has no name is not, as you correctly point out elsewhere, what keeps it from being a candidate for resolution. If these modules could have names, however, then people would naturally ask why they can't participate in resolution, and we'd have yet more explaining to do. I'm therefore reluctant to change unnamed modules into "default" modules, so I suggest that we instead focus on your goal rather than a specific potential solution. Your goal seems to be to enable module systems other than JPMS to place their own diagnostic information in stack traces and certain exception messages. Such systems typically (always?) create a unique class loader for each loaded module, and so in JPMS terms each such "external" module would be represented by a class loader that loads all of its classes into that loader's unnamed JPMS module. If this is correct then there's a simpler way to reach your goal: Enhance class loaders to have optional names. When the run-time system generates a stack trace or an exception message that mentions a module name and version, if present, then it could also insert the name of that module's class loader, if present. This would allow an external module system to provide better diagnostics without having to change the JPMS design in a fundamental way. Would that meet your needs? - Mark From david.lloyd at redhat.com Wed Jul 13 15:23:38 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 13 Jul 2016 10:23:38 -0500 Subject: Discussion: #NonHierarchicalLayers In-Reply-To: <20160713142813.C0075B78C2@eggemoggin.niobe.net> References: <56D7802A.9070305@redhat.com> <20160713142813.C0075B78C2@eggemoggin.niobe.net> Message-ID: <57865CFA.1010208@redhat.com> On 07/13/2016 09:28 AM, mark.reinhold at oracle.com wrote: > Reference: http://openjdk.java.net/projects/jigsaw/spec/issues/#NonHierarchicalLayers > > 2016/3/2 16:07:06 -0800, david.lloyd at redhat.com: >> If there is any expectation of moving existing systems over to Jigsaw, I >> think that the strict hierarchical layer system should be abandoned in >> favor of a graph-oriented system. This allows containers to maintain >> separate module graphs with separate namespaces which nevertheless >> interconnect. > > I suspect that generalizing the design to allow non-hierarchical layers > is possible, but tricky. Both configurations and layers would be able to > have multiple parents, so the API would become more complex, but perhaps > that's okay since this is more for expert than casual use. To avoid > having to deal with conflicts between parents (i.e., the diamond-import > problem), it'd likely be best to make the collections of parents be lists > rather than sets, so that they're searched in a determined order. That's good news. >> I believe that this is a necessary requirement if (in particular) there >> is an expectation that Java EE 9 (or another Java EE specification) >> would be reframed in terms of Jigsaw-style modules, since Java EE module >> names have many fewer restrictions, and it is possible for (for example) >> more than one application to contain modules of the same name. > > I don't understand how these two observations motivate non-hierarchical > layers, but maybe I'm missing something. > > Module names are an entirely distinct issue [1]. The constraints on > module names could be relaxed regardless of whether or not layers are > hierarchical. Right, I shouldn't have conflated these issues. Names are relevant to this issue only insofar as sibling layers allow for module name duplication between them. > It's already possible to have multiple modules of the same name with > simple hierarchical layers. Two hosted applications can contain a module > of the same name as long as they're loaded into distinct, unrelated > layers. (Even two layers that are related can contain modules of the > same name, in which case the resolver chooses the module in the nearest > layer.) > > Are there other reasons why a hypothetical future Java EE specification, > which allows applications to be built from collections of JPMS modules, > might necessitate non-hierarchical layers? It's not the specification that would require them, but the implementation, which may allow OSGi-style dynamic addition and changing of installed dependencies, especially from different (possibly overlapping) module namespaces. Such dynamic changes can currently only be modeled using incrementally-added layers, but this fails if a previously loaded module must be updated to refer to a later one, or if there is more than one module namespace. Another case is where there are two separate deployment systems with isolated name spaces (for example, OSGi and Java EE) which may be cross-linked. In our current solution, we don't actually layer the namespaces - in the internal APIs we actually identify what module space the proposed dependency exists in, along with the name of the module to find. This allows us to (for example) have more than one declaration syntax for dependencies. In pre-Java 9 terms, a good metaphor is Extension-list: versus Class-Path: MANIFEST headers - each comes from a separate namespace, but each declares where the dependency information comes from. We also add a Dependencies: header, which allows (the equivalent of boot layer) module names to be specified as dependencies for a deployment, separately from the old extension list and class path headers (which we use to resolve modules from different namespaces). I believe that it is presently our intention to continue to support extension lists even after that is dropped from the spec; the namespace for extensions is isolated from the namespace of static modules, but deployment modules may depend on either or both (and/or additional module spaces like OSGi, which may also have conflicting namespaces) at their option using the appropriate MANIFEST header or other deployment descriptor. So I would hypothetically imagine a layer for our static modules (though this depends on the resolution or lack thereof of a couple other issues), then another for extensions, another for EE deployments, another sibling for filesystem-based "automatic" modules that we generate for absolute class-path refs, and another sibling for each auxiliary add-on system (like OSGi), all of which have the ability to arbitrarily cross-reference to the other sibling layers. Some possible configurations can be interpreted as a tree, but some configurations must be a DAG or even a fully cyclic graph in order to work. Note also that this might all only be meaningful hand-in-hand with a resolution to #MutableConfigurations and #LazyConfigurationAndInstantiation. Note that we could actually get by *without* non-hierarchical layers, *if* I can explicitly establish module dependences which link between sibling custom layers: I actually view this as a better, more general approach, and one which we use internally. So that's another option. >> If on the other hand there is an expectation that Java EE is not going >> to be retrofitted in a way which maps EE applications and modules to >> Jigsaw modules, then this should be clearly declared. > > As I wrote earlier, we definitely do have that expectation -- that's why > the requirements include an entire section on dynamic configuration. > > - Mark > > > [1] http://openjdk.java.net/projects/jigsaw/spec/issues/#ModuleNameCharacters > -- - DML From david.lloyd at redhat.com Wed Jul 13 16:22:43 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 13 Jul 2016 11:22:43 -0500 Subject: Discussion: #ResourceExistenceAndSize In-Reply-To: <20160713142913.C1C1FB78C7@eggemoggin.niobe.net> References: <56D85B77.4040607@redhat.com> <20160713142913.C1C1FB78C7@eggemoggin.niobe.net> Message-ID: <57866AD3.4020807@redhat.com> On 07/13/2016 09:29 AM, mark.reinhold at oracle.com wrote: > Reference: http://openjdk.java.net/projects/jigsaw/spec/issues/#ResourceExistenceAndSize > > 2016/3/3 7:42:47 -0800, david.lloyd at redhat.com: >> Module.getResourceAsStream() is the only way a resource can be loaded >> from a module. This makes it impossible to do either of the following: >> >> - Determine the size of a resource in advance of loading it >> - Determine the existence of a resource without opening it >> >> Loading resources as java.net.URLs allowed these things to be done >> (among other things), however it is also disadvantageous due to the >> heavyweight nature of that class. > > What are some use cases that require determining the existence or size of > a resource without loading it? You may need to pre-size data structures like arrays, or files on the filesystem, or you may need the information to produce a "Content-Length" style header before reading any data, or to know if you are able to fulfill a request if the request has a minimum or maximum size requirement. Testing for the existence of a resource is useful for making decisions about (for example) how to handle a deployment, as some specs require, though opening, testing, and (if not null) closing the resource does suffice to solve this problem, as long as doing so doesn't incur any kind of undue overhead (and in the modular system, it *probably* won't). >> I had earlier come up with a simple patch [1], > > (I'm not going to read your patch on GitHub since the comment on your > repo clearly states that it is not a contribution to this JSR nor to > OpenJDK. If you wish the EG to consider it then please either post > it here or contribute it explicitly to OpenJDK via some other means.) >> initially as a way to >> address JDK-6865375, which introduced the idea of a Resource class >> that is associated with a ClassLoader, with a name, URL, size, and >> stream factory method, which would also be a clean solution here (though >> with the addition of a Module property as well since in Jigsaw they're >> separate). > > The issues laid out in JDK-6865375 [1] are entirely different from those > of resource existence and size, though I do see how a common solution > could address all these problems. Yeah, I think that publishing the origin of the resource is an important part of resource management as well, and I probably should have made it all one; I suppose I left it off because the of the prior existence of the JDK issue. >> The idea is to put resources on a similar footing to classes, whereby >> you can (if suitably permitted) acquire their class loaders just like >> Class.getClassLoader() allows you to do for classes. This also allows >> things like ServiceLoader to load classes directly from the module which >> contained the corresponding service descriptor. > > I'm reluctant to introduce a brand-new general "Resource" abstraction, > along with all the complexity that would entail. There is, among other > things, a tricky compatibility issue here, as one of the comments to > JDK-6865375 points out: Any custom class loader that today overrides the > various getResource* methods would have to be retrofitted to override all > new "Resource"-returning methods. Yeah the workaround for this in my patch is to provide default implementations which use URL to back the Resource object, which allows the current status quo to be maintained (though the usual issues with adding methods to a base class do apply). > At any rate, I don't see anything here that's inherently part of the > design of the module system, so I intend to close this issue as > unrelated. The relation comes in because Class and ClassLoader have a way to open a resource with a URL, but Module does not, providing only a getResourceAsStream() method. Adding a getResource() method would be one solution, but the underlying requirement is really that the resource existence and size (and origin) could/should be queriable as well, so that was the issue I registered, especially as URL is really not a great solution to this problem. > [1] https://bugs.openjdk.java.net/browse/JDK-6865375 -- - DML From david.lloyd at redhat.com Wed Jul 13 16:25:54 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 13 Jul 2016 11:25:54 -0500 Subject: Proposal: #DefaultModule In-Reply-To: <20160713143213.CF094B78D7@eggemoggin.niobe.net> References: <577BC730.1040706@redhat.com> <20160713143213.CF094B78D7@eggemoggin.niobe.net> Message-ID: <57866B92.200@redhat.com> On 07/13/2016 09:32 AM, mark.reinhold at oracle.com wrote: > 2016/7/5 7:41:52 -0700, david.lloyd at redhat.com: >> I propose that the concept of "unnamed module" be dropped in favor of >> "default module". The main difference is that the class loader (or >> module finder or layer configuration or someone else) would be allowed >> to (but not required to) assign a free-form name and version string to >> this module. This would allow existing module systems to bring their >> module concept into some form of consonance with Jigsaw without >> compromising any of the restrictions that Jigsaw-style modules have. >> >> Effecting this change would suggest the addition of an "isDefault()" >> method on Module, possibly replacing "isNamed()" (which is arguably >> already somewhat redundant with respect to getName()). Also at some >> stage, something would have to establish the default module name and >> version strings, probably defaulting to the (current) null strings to >> keep a stable status quo. > > The concept of unnamed modules has worked out pretty well so far. The > analogy with unnamed packages, while imprecise, has made it fairly easy > to explain to people. The fact that an unnamed module has no name is > not, as you correctly point out elsewhere, what keeps it from being a > candidate for resolution. If these modules could have names, however, > then people would naturally ask why they can't participate in resolution, > and we'd have yet more explaining to do. > > I'm therefore reluctant to change unnamed modules into "default" modules, > so I suggest that we instead focus on your goal rather than a specific > potential solution. > > Your goal seems to be to enable module systems other than JPMS to place > their own diagnostic information in stack traces and certain exception > messages. Such systems typically (always?) create a unique class loader > for each loaded module, and so in JPMS terms each such "external" module > would be represented by a class loader that loads all of its classes into > that loader's unnamed JPMS module. > > If this is correct then there's a simpler way to reach your goal: Enhance > class loaders to have optional names. When the run-time system generates > a stack trace or an exception message that mentions a module name and > version, if present, then it could also insert the name of that module's > class loader, if present. This would allow an external module system to > provide better diagnostics without having to change the JPMS design in a > fundamental way. > > Would that meet your needs? I think that would probably work very well, and fits in nicely with the existing architecture of stack trace assembly as I understand it (since the class loader can be directly referenced from the Class, which is already present). Thanks! -- - DML From david.lloyd at redhat.com Wed Jul 13 16:42:56 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 13 Jul 2016 11:42:56 -0500 Subject: Discussion: #MutableConfigurations In-Reply-To: <20160713143013.C3C3AB78CC@eggemoggin.niobe.net> References: <56D79D56.7040801@redhat.com> <20160713143013.C3C3AB78CC@eggemoggin.niobe.net> Message-ID: <57866F90.5020703@redhat.com> On 07/13/2016 09:30 AM, mark.reinhold at oracle.com wrote: > Reference: http://openjdk.java.net/projects/jigsaw/spec/issues/#MutableConfigurations > > 2016/3/2 18:11:34 -0800, david.lloyd at redhat.com: >> It appears from what I can see in the Jigsaw code, that once a >> Configuration is calculated, it cannot be changed in any way, > > That's true. > > The overall model in the present design is that, given a set of modules, > you first compute a configuration, which captures the resolution of all > the modules' dependences in a consistent fashion. You then instantiate > that as a Layer (or even more than one Layer, if you want). > > This model is motivated by one of our primary goals, namely reliable > configuration. Resolving a complete configuration, rather than doing > so incrementally, allows the early detection of missing, duplicate, > and conflicting dependencies. > >> which >> makes them unsuitable for use in containers which add and remove modules >> at run time. It is not clear how such containers are expected to >> function. > > They're expected to create Layers, which can be related hierarchically as > needed (or perhaps more generally, cf. #NonHierarchicalLayers [1]). > >> If there is a deliberate intention that the Jigsaw system >> will not support dynamic modification of layers/configurations >> (including the dynamic addition and removal of modules at run time), >> then that should be explicitly stated. > > I don't think it makes sense for Configurations to be mutable, for the > reasons stated above. It may be that the Layer concept needs to be made > more dynamic in certain ways, but that's a different matter. > > I intend to close this issue unless there are strong objections from > other EG members. In that case, other EG members take note: this means that hot deployment as the world knows it is over as of Java EE 9, since a module can never delete old or establish new dependence relationships. This also means that OSGi loses this capability as well, so either OSGi has to be redefined with the new restriction, or else OSGi bundles cannot be modules. -- - DML From david.lloyd at redhat.com Wed Jul 13 17:39:42 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 13 Jul 2016 12:39:42 -0500 Subject: Discussion: #LazyConfigurationAndInstantiation In-Reply-To: <20160713143113.C5B45B78D2@eggemoggin.niobe.net> References: <56D7AE09.1030003@redhat.com> <20160311174751.5399C9E891@eggemoggin.niobe.net> <56E33EEC.20706@redhat.com> <20160713143113.C5B45B78D2@eggemoggin.niobe.net> Message-ID: <57867CDE.8000206@redhat.com> On 07/13/2016 09:31 AM, mark.reinhold at oracle.com wrote: > Reference: http://openjdk.java.net/projects/jigsaw/spec/issues/#LazyConfigurationAndInstantiation > > 2016/3/11 13:55:56 -0800, david.lloyd at redhat.com: >> On 03/11/2016 11:47 AM, mark.reinhold at oracle.com wrote: >>> 2016/3/2 19:22 -0800, david.lloyd at redhat.com: >>>> It looks as though the instantiation of a Layer causes a complete load >>>> of all the modules in the layer. >>> >>> What do you mean by "complete load"? Loading all the classes? >> >> No I mean locating and loading all of the descriptors and building all >> the wiring information for every module. > > Yes, that's what happens. Resolving a Configuration requires reading all > of the relevant module descriptors. Instantiating a Configuration into a > Layer builds a complete run-time module graph. Both of these operations > are linear in the number of modules in the graph. > >>>> Apart from the consequent lack of >>>> dynamicism, I think this is very likely to cause problems for large >>>> applications, particularly if such an application might have many >>>> operation modes, some of which do not require *all* of the bundled >>>> modules to be loaded every single time. >>>> >>>> Can there not instead be an incremental resolution algorithm, akin to >>>> how classes are lazily loaded? >>> >>> Configuring a set of modules and instantiating that configuration as a >>> layer requires no more than reading the modules' descriptors. Nothing >>> else from any module definition will be read until it's actually needed. >> >> But it does require that all the module descriptors from a given layer >> be available, and that the load time for the first load of a module in a >> layer will always be bounded by the size of the layer, rather than just >> by the dependency subgraph of the module being loaded. > > Correct. > >> Based on >> application server deployments that I know about, I think the far upper >> bound for a realistic number of modules in a layer will probably lie in >> the thousands to ten thousands range (though there is always the outlier >> case where someone has to push it to see how far it goes...), which >> might translate into a substantial startup overhead. > > In our experience with the prototype so far, the time required to resolve > a Configuration and instantiate it as a Layer is dominated by the time > required to locate and read module descriptors, typically from inside > individual artifacts in the filesystem. > > One fairly straightforward way to speed that up is to link your modules > into a custom run-time image, so that the descriptors are all in one > optimized artifact [1]. If that's not feasible then you could achieve > much the same effect in the build, installation, or startup process of a > large application: Construct an optimized cache of the descriptors of all > your modules, and use a custom ModuleFinder to load the descriptors from > the cache but other module content from more-traditional artifacts. >> Another potenti >> issue is requiring that all modules be present, though this is more >> closely related to #MutableConfigurations now I think; I suspect this >> issue could be mitigated with an acceptable solution to that issue. > > As I wrote in my reply re. #MutableConfigurations [2], I think this > approach is at odds with our goal to provide reliable configuration. I disagree. "Reliable" is only defined in the agreed-upon requirements by stipulating that modules may have dependence relationships which are declared. I interpreted that goal simply as an explicit rejection of various poorly-defined customized class loader behaviors, and a move towards clearer and more predictable behavior. By my interpretation and experience, I consider, for example, the ability to change the explicit dependence relationships to still be "reliable" as long as the effects of doing so are well-defined, just as I consider the behavior of lazy class loading to be "reliable" in that it is well-defined and predictable, and has rules which make sense (referring to the way that classes are loaded, resolved, and initialized in separate phases, which allows for lazy on-demand-only progression through those phases but also allows for almost completely arbitrary interconnection of classes, while remaining basically predictable). The first time that "reliable" is specified in terms of concrete behavior is in the SOTMS document, and these issues are being raised exactly against the state of the module system, so I think that this issue as well as #MutableConfigurations serve to directly challenge the validity of the definition of "reliable" which is used by the proposed implementation, rather than being invalid due to the presumed validity of that definition (which was not agreed upon by the EG). Even the JSR definition (which I acknowledge is quite out of date at this point) states that OSGi does address the problem of "reliable" configuration, despite the fact that the OSGi solution allows for dynamic loading and relinking. This is contrary to the definition in the SOTMS, and also despite the fact that Jigsaw's more strict interpretation of this actually invalidates those behaviors of OSGi, were OSGi to try to merge the bundle concept with the Jigsaw module concept somehow. >>> If you really want to avoid configuring all modules in certain operation >>> modes, do layers not provide sufficient flexibility? Load the core of >>> your application into the boot layer, figure out which modules you need >>> for the requested operation mode, and then create an additional layer to >>> load those modules. >> >> That can work in some cases, but only if things are loaded one >> additional time (rather than on-demand as configuration changes, for >> example) or in such a way that you could just keep adding layers. But >> in the event of one large intermeshed layer, this won't work without >> #MutableConfigurations and/or #NonHierarchicalLayers and something to >> coordinate the load. > > Suppose that we allow #NonHierarchicalLayers [3]. Are there realistic > use cases in which layers would not then provide sufficient flexibility? I'm not sure that it will (in all cases). In our current system, a module isn't loaded until it is referenced, and it is not linked until it is used, like the aforementioned class linking rules. Even if we have separate (non-hierarchical) layers for every single module, because of the way modules are defined in Jigsaw, I think, given module graph G, we'd still have to aggressively traverse all of G in order to load any module that is a part of G, even if nodes in that graph are never actually referenced at run time, which puts us back at square one. -- - DML From forax at univ-mlv.fr Wed Jul 13 20:09:54 2016 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 13 Jul 2016 22:09:54 +0200 Subject: Discussion: #MutableConfigurations In-Reply-To: <57866F90.5020703@redhat.com> References: <56D79D56.7040801@redhat.com> <20160713143013.C3C3AB78CC@eggemoggin.niobe.net> <57866F90.5020703@redhat.com> Message-ID: I may say something stupid but if you have one Layer by ClassLoader, why it's not enough ? Remi On July 13, 2016 6:42:56 PM GMT+02:00, "David M. Lloyd" wrote: >On 07/13/2016 09:30 AM, mark.reinhold at oracle.com wrote: >> Reference: >http://openjdk.java.net/projects/jigsaw/spec/issues/#MutableConfigurations >> >> 2016/3/2 18:11:34 -0800, david.lloyd at redhat.com: >>> It appears from what I can see in the Jigsaw code, that once a >>> Configuration is calculated, it cannot be changed in any way, >> >> That's true. >> >> The overall model in the present design is that, given a set of >modules, >> you first compute a configuration, which captures the resolution of >all >> the modules' dependences in a consistent fashion. You then >instantiate >> that as a Layer (or even more than one Layer, if you want). >> >> This model is motivated by one of our primary goals, namely reliable >> configuration. Resolving a complete configuration, rather than doing >> so incrementally, allows the early detection of missing, duplicate, >> and conflicting dependencies. >> >>> which >>> makes them unsuitable for use in containers which add and remove >modules >>> at run time. It is not clear how such containers are expected to >>> function. >> >> They're expected to create Layers, which can be related >hierarchically as >> needed (or perhaps more generally, cf. #NonHierarchicalLayers [1]). >> >>> If there is a deliberate intention that the Jigsaw >system >>> will not support dynamic modification of layers/configurations >>> (including the dynamic addition and removal of modules at run time), >>> then that should be explicitly stated. >> >> I don't think it makes sense for Configurations to be mutable, for >the >> reasons stated above. It may be that the Layer concept needs to be >made >> more dynamic in certain ways, but that's a different matter. >> >> I intend to close this issue unless there are strong objections from >> other EG members. > >In that case, other EG members take note: this means that hot >deployment >as the world knows it is over as of Java EE 9, since a module can never > >delete old or establish new dependence relationships. This also means >that OSGi loses this capability as well, so either OSGi has to be >redefined with the new restriction, or else OSGi bundles cannot be >modules. > >-- >- DML -- Sent from my Android device with K-9 Mail. Please excuse my brevity. From david.lloyd at redhat.com Wed Jul 13 20:27:21 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 13 Jul 2016 15:27:21 -0500 Subject: Discussion: #MutableConfigurations In-Reply-To: References: <56D79D56.7040801@redhat.com> <20160713143013.C3C3AB78CC@eggemoggin.niobe.net> <57866F90.5020703@redhat.com> Message-ID: <5786A429.1040601@redhat.com> On 07/13/2016 03:09 PM, Remi Forax wrote: > I may say something stupid but if you have one Layer by ClassLoader, why > it's not enough ? Even if you have one layer per classloader per module, you can add modules but you cannot update them when they are removed. > > Remi > > > On July 13, 2016 6:42:56 PM GMT+02:00, "David M. Lloyd" > wrote: > > On 07/13/2016 09:30 AM, mark.reinhold at oracle.com wrote: > > Reference: > http://openjdk.java.net/projects/jigsaw/spec/issues/#MutableConfigurations > > 2016/3/2 18:11:34 -0800, david.lloyd at redhat.com: > > It appears from what I can see in the Jigsaw code, that once a > Configuration is calculated, it cannot be changed in any way, > > > That's true. > > The overall model in the present design is that, given a set of > modules, > you first compute a configuration, which captures the resolution > of all > the modules' dependences in a consistent fashion. You then > instantiate > that as a Lay! er (or even more than one Layer, if you want). > > This model is motivated by one of our primary goals, namely reliable > configuration. Resolving a complete configuration, rather than doing > so incrementally, allows the early detection of missing, duplicate, > and conflicting dependencies. > > which > makes them unsuitable for use in containers which add and > remove modules > at run time. It is not clear how such containers are expected to > function. > > > They're expected to create Layers, which can be related > hierarchically as > needed (or perhaps more generally, cf. #NonHierarchicalLayers [1]). > > I! f there is a deliberate intention that the Jigsaw system > will not support dynamic modification of layers/configurations > (including the dynamic addition and removal of modules at > run time), > then that should be explicitly stated. > > > I don't think it makes sense for Configurations to be mutable, > for the > reasons stated above. It may be that the Layer concept needs to > be made > more dynamic in certain ways, but that's a different matter. > > I intend to close this issue unless there are strong objections from > other EG members. > > > In that case, other EG members take note: this means that hot deployment > as the world knows it is over as of Java EE 9, since a module can never > delete old or establish new dependence relationships. This also means > that OSGi loses this capability as well, so either OSGi has to be > redefined with the new restriction, or else OSGi bundles cannot be modules! > . > > > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. -- - DML From forax at univ-mlv.fr Thu Jul 14 15:06:36 2016 From: forax at univ-mlv.fr (Remi Forax) Date: Thu, 14 Jul 2016 17:06:36 +0200 (CEST) Subject: Me trying to explain the problem behind #ReflectiveAccessToNonExportedTypes In-Reply-To: <1627574708.1456733.1468507103919.JavaMail.zimbra@u-pem.fr> Message-ID: <1105959088.1485118.1468508796360.JavaMail.zimbra@u-pem.fr> Let me try to explain again the problem and the possible solution. Let say i have two modules, module1 and module2. In module1, I have two packages module1.exported and module1.nonexported, so module1 has this module-info.java module module1 { exports module1.exported; } in module1.nonexported, I have a class A that's implement an interface I defined like this: package module1.nonexported; import module1.exported.I; public class A implements I { @Override public void foo() { System.out.println("foo"); } } I is defined in module1.exported like this: package module1.exported; import module1.nonexported.A; public interface I { public void foo(); static I instance() { return new A(); } } Now, in module2, i want to use I so module2 requires module1, like this: module module2 { requires module1; } and I have a main defined in the package modul2.main, like that: package module2.main; import java.lang.reflect.Method; import module1.exported.I; public class Main { public static void main(String[] args) throws ReflectiveOperationException { System.out.println(Class.forName("module1.nonexported.A")); I i = I.instance(); Method m = i.getClass().getMethod("foo"); m.invoke(i); } } If i run this code in with the two modules in the classpath, everything work and the code prints class module1.nonexported.A foo Now, if if the two modules are in the modulepath, Class.forName works but m.invoke() throws an IllegalAccessException because A is in a non exported package java.lang.IllegalAccessException: class module2.main.Main (in module module2) cannot access class module1.nonexported.A (in module module1) because module module1 does not export module1.nonexported to module module2 If i try with m.setAccessible(true), before m.invoke(), setAccessible throws an exception InaccessibleObjectException java.lang.reflect.InaccessibleObjectException: Unable to make member of class module1.nonexported.A accessible: module module1 does not export module1.nonexported to module module2 A way to solve this issue, is to use the interface I to call foo instead of using the class A, Method m = I.class.getMethod("foo"); m.invoke(i); in that case, everything is Ok, because i don't try to use the method foo of A anymore. What Mark as proposed is to introduce a new "export dynamic" semantics which doesn't allow the package to be visible at compile time but to be visible at runtime. This solve the issue by supposing that everyone will write export dynamic on every non exported package to be backward compatible with the Java 8 behavior. While i agree that "export dynamic" is a semantics that jigsaw should provide, i disagree with the syntax "export dynamic" because for me, the a non exported package should have this semantics in order to be backward compatible with existing code. So we should have 3 way to export or not a package, 1. don't export at compile time, don't export at runtime 2. don't export at compile time, export at runtime 3. export at compile time, export at runtime. 2 should be the default, 1 should be use by the package of the JDK (or any other libs) that want strong security, 3 is for the package that defines API (that will be maintained forever). R?mi From david.lloyd at redhat.com Thu Jul 14 15:31:38 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 14 Jul 2016 10:31:38 -0500 Subject: Me trying to explain the problem behind #ReflectiveAccessToNonExportedTypes In-Reply-To: <1105959088.1485118.1468508796360.JavaMail.zimbra@u-pem.fr> References: <1105959088.1485118.1468508796360.JavaMail.zimbra@u-pem.fr> Message-ID: <5787B05A.7090506@redhat.com> On 07/14/2016 10:06 AM, Remi Forax wrote: > Let me try to explain again the problem and the possible solution. > > Let say i have two modules, module1 and module2. > In module1, I have two packages module1.exported and module1.nonexported, > so module1 has this module-info.java > > module module1 { > exports module1.exported; > } > > in module1.nonexported, I have a class A that's implement an interface I defined like this: > package module1.nonexported; > > import module1.exported.I; > > public class A implements I { > @Override > public void foo() { > System.out.println("foo"); > } > } > > I is defined in module1.exported like this: > package module1.exported; > > import module1.nonexported.A; > > public interface I { > public void foo(); > > static I instance() { > return new A(); > } > } > > Now, in module2, i want to use I so module2 requires module1, like this: > module module2 { > requires module1; > } > > and I have a main defined in the package modul2.main, like that: > package module2.main; > > import java.lang.reflect.Method; > > import module1.exported.I; > > public class Main { > public static void main(String[] args) throws ReflectiveOperationException { > System.out.println(Class.forName("module1.nonexported.A")); > > I i = I.instance(); > Method m = i.getClass().getMethod("foo"); > m.invoke(i); > } > } > > If i run this code in with the two modules in the classpath, > everything work and the code prints > class module1.nonexported.A > foo > > Now, if if the two modules are in the modulepath, > Class.forName works but m.invoke() throws an IllegalAccessException because A is in a non exported package > java.lang.IllegalAccessException: class module2.main.Main (in module module2) cannot access class module1.nonexported.A (in module module1) because module module1 does not export module1.nonexported to module module2 > > If i try with m.setAccessible(true), before m.invoke(), setAccessible throws an exception InaccessibleObjectException > java.lang.reflect.InaccessibleObjectException: Unable to make member of class module1.nonexported.A accessible: module module1 does not export module1.nonexported to module module2 > > A way to solve this issue, is to use the interface I to call foo instead of using the class A, > Method m = I.class.getMethod("foo"); > m.invoke(i); > > in that case, everything is Ok, because i don't try to use the method foo of A anymore. > > What Mark as proposed is to introduce a new "export dynamic" semantics which doesn't allow the package to be visible at compile time but to be visible at runtime. > This solve the issue by supposing that everyone will write export dynamic on every non exported package to be backward compatible with the Java 8 behavior. > > While i agree that "export dynamic" is a semantics that jigsaw should provide, i disagree with the syntax "export dynamic" because for me, the a non exported package should have this semantics in order to be backward compatible with existing code. > So we should have 3 way to export or not a package, > 1. don't export at compile time, don't export at runtime > 2. don't export at compile time, export at runtime > 3. export at compile time, export at runtime. > > 2 should be the default, 1 should be use by the package of the JDK (or any other libs) that want strong security, 3 is for the package that defines API (that will be maintained forever). This is a good summary and I think matches my understanding. But the basic problem is that if everyone is doing this "export dynamic" then the safety proposed by restricting "public" is a false promise: we are providing the feature only to take it away again, resulting in the exact same situation we have today, but with no alternatives. If we have to do this then I think the feature itself needs to be re-examined. What I was suggesting was that we re-separate the notion of exports from accessibility altogether, so you again have three different situations rather than two: A. Compile time B. Class link at run time C. Reflection at run time This changes the basic "export" back to mean "make available for linkage", with no bearing on accessibility. Then you have (at minimum) the following ways to export (or not) a package: 1. No export at compile time, no export but reflection-only at run time (but only for public types & members) 2. Yes export at compile time, yes export at run time (but only for public types & members) 3. Yes export at compile time and run time, including specific grants to access package-private members from targeted modules and/or packages Note: We could add a 1? which is an export at run time but not compile time if the EG agrees; I don't think that is particularly important though as build tools can accomplish this easily enough. I don't want to derail the core point of the discussion on this topic, so I'm leaving it aside for now. This way the Java 8 behavior is retained with no "export dynamic" anywhere in sight. Rather you'd have something like "export private", which you'd only grant for those special cases where a package must be shared to a specific friend. Things not intended to be shared can be safely migrated to the new friend package feature. No new access limits are introduced - instead old ones are enhanced in what I think is a simple and intuitive way. All the compatibility concerns go away, but we can still use the mechanism to seal up the problems in the JDK with shared code and things which should not be public. We should retain setAccessible() for now for compatibility. We could also introduce a "--safe-mode" flag which disables setAccessible() *across the board* (that is, not just for modular code) without controversy, because all those public members will remain accessible without "help", plus a way is provided to allow access to be volunteered from one package to another (along much the same philosophical lines as MethodHandles.Lookup IMO). The platform EG can then later decide if this safer mode is good for the platform, and if so, could move to making this the default and possibly adding an "--unsafe-mode" for a limited time. -- - DML From mark.reinhold at oracle.com Thu Jul 14 16:33:04 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 14 Jul 2016 09:33:04 -0700 Subject: Me trying to explain the problem behind #ReflectiveAccessToNonExportedTypes In-Reply-To: <1105959088.1485118.1468508796360.JavaMail.zimbra@u-pem.fr> References: <1627574708.1456733.1468507103919.JavaMail.zimbra@u-pem.fr> <1105959088.1485118.1468508796360.JavaMail.zimbra@u-pem.fr> Message-ID: <20160714093304.841168761eggemoggin.niobe.net> 2016/7/14 8:06:36 -0700, forax at univ-mlv.fr: > Let me try to explain again the problem and the possible solution. > > ... > > What Mark as proposed is to introduce a new "export dynamic" semantics > which doesn't allow the package to be visible at compile time but to > be visible at runtime. > This solve the issue by supposing that everyone will write export > dynamic on every non exported package to be backward compatible with > the Java 8 behavior. That wasn't my assumption. The intent here is that the author of a module should be able to declare which of the module's packages are exported, and when, and which are not. The author should also be able to be reasonably sure that these declarations will not be overridden lightly, by any random code that uses reflection. (Overriding encapsulation via second-class mechanisms such as command-line options or container-based module-descriptor rewriting, however, is fine.) This is what strong encapsulation is all about. > While i agree that "export dynamic" is a semantics that jigsaw should > provide, i disagree with the syntax "export dynamic" because for me, > the a non exported package should have this semantics in order to be > backward compatible with existing code. > So we should have 3 way to export or not a package, > 1. don't export at compile time, don't export at runtime > 2. don't export at compile time, export at runtime > 3. export at compile time, export at runtime. > > 2 should be the default, 1 should be use by the package of the JDK (or > any other libs) that want strong security, 3 is for the package that > defines API (that will be maintained forever). A problem with changing the default is that many developers will never think to do anything else, and leave all their packages unnecessarily exported. We have examples of this sort of thing elsewhere in Java, e.g., default public constructors, which have been problematic for years. Do we really want to repeat that kind of mistake? What do you think of allowing wildcards, i.e., `exports dynamic *`? That way a developer could explicitly choose to export everything at run time, but could do so in a very compact way. - Mark From forax at univ-mlv.fr Thu Jul 14 21:46:21 2016 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Thu, 14 Jul 2016 23:46:21 +0200 (CEST) Subject: Me trying to explain the problem behind #ReflectiveAccessToNonExportedTypes In-Reply-To: <20160714093304.841168761eggemoggin.niobe.net> References: <1627574708.1456733.1468507103919.JavaMail.zimbra@u-pem.fr> <1105959088.1485118.1468508796360.JavaMail.zimbra@u-pem.fr> <20160714093304.841168761eggemoggin.niobe.net> Message-ID: <1028346570.1564026.1468532781557.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "mark reinhold" > ?: "Remi Forax" > Cc: jpms-spec-experts at openjdk.java.net > Envoy?: Jeudi 14 Juillet 2016 18:33:04 > Objet: Re: Me trying to explain the problem behind #ReflectiveAccessToNonExportedTypes > > 2016/7/14 8:06:36 -0700, forax at univ-mlv.fr: > > Let me try to explain again the problem and the possible solution. > > > > ... > > > > What Mark as proposed is to introduce a new "export dynamic" semantics > > which doesn't allow the package to be visible at compile time but to > > be visible at runtime. > > This solve the issue by supposing that everyone will write export > > dynamic on every non exported package to be backward compatible with > > the Java 8 behavior. > > That wasn't my assumption. > > The intent here is that the author of a module should be able to declare > which of the module's packages are exported, and when, and which are > not. The author should also be able to be reasonably sure that these > declarations will not be overridden lightly, by any random code that > uses reflection. (Overriding encapsulation via second-class mechanisms > such as command-line options or container-based module-descriptor > rewriting, however, is fine.) This is what strong encapsulation is > all about. The choice for the author seems to be strong encapsulation vs users being able to use their favorite reflection library. > > > While i agree that "export dynamic" is a semantics that jigsaw should > > provide, i disagree with the syntax "export dynamic" because for me, > > the a non exported package should have this semantics in order to be > > backward compatible with existing code. > > So we should have 3 way to export or not a package, > > 1. don't export at compile time, don't export at runtime > > 2. don't export at compile time, export at runtime > > 3. export at compile time, export at runtime. > > > > 2 should be the default, 1 should be use by the package of the JDK (or > > any other libs) that want strong security, 3 is for the package that > > defines API (that will be maintained forever). > > A problem with changing the default is that many developers will never > think to do anything else, and leave all their packages unnecessarily > exported. We have examples of this sort of thing elsewhere in Java, > e.g., default public constructors, which have been problematic for > years. Do we really want to repeat that kind of mistake? > > What do you think of allowing wildcards, i.e., `exports dynamic *`? > That way a developer could explicitly choose to export everything at > run time, but could do so in a very compact way. Wildcards seems a great idea. But if we want authors to make a decision, instead of choosing a default for them, we should let them to make a decision. So wildcards should exist in both way 'export dynamic *' or 'export noexport *' (or whatever we call it) and we should mandate that a module to have an export wildcard declaration. It's more verbose that the current settings but at least this make the choice explicit. > > - Mark > R?mi From forax at univ-mlv.fr Fri Jul 15 08:18:42 2016 From: forax at univ-mlv.fr (Remi Forax) Date: Fri, 15 Jul 2016 10:18:42 +0200 (CEST) Subject: Me trying to explain the problem behind #ReflectiveAccessToNonExportedTypes In-Reply-To: <5787B05A.7090506@redhat.com> References: <1105959088.1485118.1468508796360.JavaMail.zimbra@u-pem.fr> <5787B05A.7090506@redhat.com> Message-ID: <1363010886.1592154.1468570722083.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "David M. Lloyd" > ?: jpms-spec-experts at openjdk.java.net > Envoy?: Jeudi 14 Juillet 2016 17:31:38 > Objet: Re: Me trying to explain the problem behind #ReflectiveAccessToNonExportedTypes > > On 07/14/2016 10:06 AM, Remi Forax wrote: > > Let me try to explain again the problem and the possible solution. > > > > Let say i have two modules, module1 and module2. > > In module1, I have two packages module1.exported and module1.nonexported, > > so module1 has this module-info.java > > > > module module1 { > > exports module1.exported; > > } > > > > in module1.nonexported, I have a class A that's implement an interface I > > defined like this: > > package module1.nonexported; > > > > import module1.exported.I; > > > > public class A implements I { > > @Override > > public void foo() { > > System.out.println("foo"); > > } > > } > > > > I is defined in module1.exported like this: > > package module1.exported; > > > > import module1.nonexported.A; > > > > public interface I { > > public void foo(); > > > > static I instance() { > > return new A(); > > } > > } > > > > Now, in module2, i want to use I so module2 requires module1, like this: > > module module2 { > > requires module1; > > } > > > > and I have a main defined in the package modul2.main, like that: > > package module2.main; > > > > import java.lang.reflect.Method; > > > > import module1.exported.I; > > > > public class Main { > > public static void main(String[] args) throws > > ReflectiveOperationException { > > System.out.println(Class.forName("module1.nonexported.A")); > > > > I i = I.instance(); > > Method m = i.getClass().getMethod("foo"); > > m.invoke(i); > > } > > } > > > > If i run this code in with the two modules in the classpath, > > everything work and the code prints > > class module1.nonexported.A > > foo > > > > Now, if if the two modules are in the modulepath, > > Class.forName works but m.invoke() throws an IllegalAccessException because > > A is in a non exported package > > java.lang.IllegalAccessException: class module2.main.Main (in module > > module2) cannot access class module1.nonexported.A (in module module1) > > because module module1 does not export module1.nonexported to module > > module2 > > > > If i try with m.setAccessible(true), before m.invoke(), setAccessible > > throws an exception InaccessibleObjectException > > java.lang.reflect.InaccessibleObjectException: Unable to make member of > > class module1.nonexported.A accessible: module module1 does not export > > module1.nonexported to module module2 > > > > A way to solve this issue, is to use the interface I to call foo instead of > > using the class A, > > Method m = I.class.getMethod("foo"); > > m.invoke(i); > > > > in that case, everything is Ok, because i don't try to use the method foo > > of A anymore. > > > > What Mark as proposed is to introduce a new "export dynamic" semantics > > which doesn't allow the package to be visible at compile time but to be > > visible at runtime. > > This solve the issue by supposing that everyone will write export dynamic > > on every non exported package to be backward compatible with the Java 8 > > behavior. > > > > While i agree that "export dynamic" is a semantics that jigsaw should > > provide, i disagree with the syntax "export dynamic" because for me, the a > > non exported package should have this semantics in order to be backward > > compatible with existing code. > > So we should have 3 way to export or not a package, > > 1. don't export at compile time, don't export at runtime > > 2. don't export at compile time, export at runtime > > 3. export at compile time, export at runtime. > > > > 2 should be the default, 1 should be use by the package of the JDK (or any > > other libs) that want strong security, 3 is for the package that defines > > API (that will be maintained forever). > > This is a good summary and I think matches my understanding. But the > basic problem is that if everyone is doing this "export dynamic" then > the safety proposed by restricting "public" is a false promise: we are > providing the feature only to take it away again, resulting in the exact > same situation we have today, but with no alternatives. If we have to > do this then I think the feature itself needs to be re-examined. > > What I was suggesting was that we re-separate the notion of exports from > accessibility altogether, so you again have three different situations > rather than two: > > A. Compile time > B. Class link at run time > C. Reflection at run time B and C may be indistinguishable. For the OpenJdk + Hotspot, you can distinguish between access control done by the VM in C++ and access control done by the reflection API done at runtime. For OpenJDK + IBM J9, there is only one entry point for the the access checks. When specifying the JSR292 (java.lang.invoke + invokedynamic), we spend a lot of time to be sure that reflection behavior and bytecode behavior were aligned. I don't think that trying to separate B and C is possible that late in the game. > > This changes the basic "export" back to mean "make available for > linkage", with no bearing on accessibility. Then you have (at minimum) > the following ways to export (or not) a package: > > 1. No export at compile time, no export but reflection-only at run time > (but only for public types & members) > 2. Yes export at compile time, yes export at run time (but only for > public types & members) > 3. Yes export at compile time and run time, including specific grants to > access package-private members from targeted modules and/or packages > > Note: We could add a 1? which is an export at run time but not compile > time if the EG agrees; I don't think that is particularly important > though as build tools can accomplish this easily enough. I don't want > to derail the core point of the discussion on this topic, so I'm leaving > it aside for now. friend accesssibility doesn't solve the issue here, when you want 'strong encapsulation' i.e. no way to use reflection to access members of a not-exported class, you should not have to provide the list of all possible existing module. I think we want 'export private' (strong encapsulation) and 'export dynamic' to be available without having a friend list. And again, changing the meaning of package private is a no go for me, by example for a package like java.lang.invoke, i want the package to be exported, i want strong encapsulation and i want some classes inside the package by example the class that implements the whole method handle hierarchy to share secret (using the package private access). > > This way the Java 8 behavior is retained with no "export dynamic" > anywhere in sight. Rather you'd have something like "export private", > which you'd only grant for those special cases where a package must be > shared to a specific friend. Things not intended to be shared can be > safely migrated to the new friend package feature. I agree that we should have a way to have Java 8 behavior without having to declare every packages with export dynamic, see my discussion with mark about specifying a default export behavior. > No new access limits > are introduced - instead old ones are enhanced in what I think is a > simple and intuitive way. All the compatibility concerns go away, but > we can still use the mechanism to seal up the problems in the JDK with > shared code and things which should not be public. > > We should retain setAccessible() for now for compatibility. We could > also introduce a "--safe-mode" flag which disables setAccessible() > *across the board* (that is, not just for modular code) without > controversy, because all those public members will remain accessible > without "help", plus a way is provided to allow access to be volunteered > from one package to another (along much the same philosophical lines as > MethodHandles.Lookup IMO). Deprecating setAccessible is another story for me. > > The platform EG can then later decide if this safer mode is good for the > platform, and if so, could move to making this the default and possibly > adding an "--unsafe-mode" for a limited time. > > -- > - DML > regards, R?mi From david.lloyd at redhat.com Fri Jul 15 12:42:26 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Fri, 15 Jul 2016 07:42:26 -0500 Subject: Me trying to explain the problem behind #ReflectiveAccessToNonExportedTypes In-Reply-To: <1363010886.1592154.1468570722083.JavaMail.zimbra@u-pem.fr> References: <1105959088.1485118.1468508796360.JavaMail.zimbra@u-pem.fr> <5787B05A.7090506@redhat.com> <1363010886.1592154.1468570722083.JavaMail.zimbra@u-pem.fr> Message-ID: <5788DA32.1000402@redhat.com> On 7/15/16 3:18 AM, Remi Forax wrote: > ----- Mail original ----- >> De: "David M. Lloyd" >> ?: jpms-spec-experts at openjdk.java.net >> Envoy?: Jeudi 14 Juillet 2016 17:31:38 >> Objet: Re: Me trying to explain the problem behind #ReflectiveAccessToNonExportedTypes >> >> On 07/14/2016 10:06 AM, Remi Forax wrote: >>> Let me try to explain again the problem and the possible solution. >>> >>> Let say i have two modules, module1 and module2. >>> In module1, I have two packages module1.exported and module1.nonexported, >>> so module1 has this module-info.java >>> >>> module module1 { >>> exports module1.exported; >>> } >>> >>> in module1.nonexported, I have a class A that's implement an interface I >>> defined like this: >>> package module1.nonexported; >>> >>> import module1.exported.I; >>> >>> public class A implements I { >>> @Override >>> public void foo() { >>> System.out.println("foo"); >>> } >>> } >>> >>> I is defined in module1.exported like this: >>> package module1.exported; >>> >>> import module1.nonexported.A; >>> >>> public interface I { >>> public void foo(); >>> >>> static I instance() { >>> return new A(); >>> } >>> } >>> >>> Now, in module2, i want to use I so module2 requires module1, like this: >>> module module2 { >>> requires module1; >>> } >>> >>> and I have a main defined in the package modul2.main, like that: >>> package module2.main; >>> >>> import java.lang.reflect.Method; >>> >>> import module1.exported.I; >>> >>> public class Main { >>> public static void main(String[] args) throws >>> ReflectiveOperationException { >>> System.out.println(Class.forName("module1.nonexported.A")); >>> >>> I i = I.instance(); >>> Method m = i.getClass().getMethod("foo"); >>> m.invoke(i); >>> } >>> } >>> >>> If i run this code in with the two modules in the classpath, >>> everything work and the code prints >>> class module1.nonexported.A >>> foo >>> >>> Now, if if the two modules are in the modulepath, >>> Class.forName works but m.invoke() throws an IllegalAccessException because >>> A is in a non exported package >>> java.lang.IllegalAccessException: class module2.main.Main (in module >>> module2) cannot access class module1.nonexported.A (in module module1) >>> because module module1 does not export module1.nonexported to module >>> module2 >>> >>> If i try with m.setAccessible(true), before m.invoke(), setAccessible >>> throws an exception InaccessibleObjectException >>> java.lang.reflect.InaccessibleObjectException: Unable to make member of >>> class module1.nonexported.A accessible: module module1 does not export >>> module1.nonexported to module module2 >>> >>> A way to solve this issue, is to use the interface I to call foo instead of >>> using the class A, >>> Method m = I.class.getMethod("foo"); >>> m.invoke(i); >>> >>> in that case, everything is Ok, because i don't try to use the method foo >>> of A anymore. >>> >>> What Mark as proposed is to introduce a new "export dynamic" semantics >>> which doesn't allow the package to be visible at compile time but to be >>> visible at runtime. >>> This solve the issue by supposing that everyone will write export dynamic >>> on every non exported package to be backward compatible with the Java 8 >>> behavior. >>> >>> While i agree that "export dynamic" is a semantics that jigsaw should >>> provide, i disagree with the syntax "export dynamic" because for me, the a >>> non exported package should have this semantics in order to be backward >>> compatible with existing code. >>> So we should have 3 way to export or not a package, >>> 1. don't export at compile time, don't export at runtime >>> 2. don't export at compile time, export at runtime >>> 3. export at compile time, export at runtime. >>> >>> 2 should be the default, 1 should be use by the package of the JDK (or any >>> other libs) that want strong security, 3 is for the package that defines >>> API (that will be maintained forever). >> >> This is a good summary and I think matches my understanding. But the >> basic problem is that if everyone is doing this "export dynamic" then >> the safety proposed by restricting "public" is a false promise: we are >> providing the feature only to take it away again, resulting in the exact >> same situation we have today, but with no alternatives. If we have to >> do this then I think the feature itself needs to be re-examined. >> >> What I was suggesting was that we re-separate the notion of exports from >> accessibility altogether, so you again have three different situations >> rather than two: >> >> A. Compile time >> B. Class link at run time >> C. Reflection at run time > > B and C may be indistinguishable. For the OpenJdk + Hotspot, you can distinguish between access control done by the VM in C++ and access control done by the reflection API done at runtime. For OpenJDK + IBM J9, there is only one entry point for the the access checks. When specifying the JSR292 (java.lang.invoke + invokedynamic), we spend a lot of time to be sure that reflection behavior and bytecode behavior were aligned. I don't think that trying to separate B and C is possible that late in the game. How could they possibly be indistinguishable? All the code for causing public to be restricted in any way has been added for Jigsaw. >> This changes the basic "export" back to mean "make available for >> linkage", with no bearing on accessibility. Then you have (at minimum) >> the following ways to export (or not) a package: >> >> 1. No export at compile time, no export but reflection-only at run time >> (but only for public types & members) >> 2. Yes export at compile time, yes export at run time (but only for >> public types & members) >> 3. Yes export at compile time and run time, including specific grants to >> access package-private members from targeted modules and/or packages >> >> Note: We could add a 1? which is an export at run time but not compile >> time if the EG agrees; I don't think that is particularly important >> though as build tools can accomplish this easily enough. I don't want >> to derail the core point of the discussion on this topic, so I'm leaving >> it aside for now. > > friend accesssibility doesn't solve the issue here, when you want 'strong encapsulation' i.e. no way to use reflection to access members of a not-exported class You literally *never* want that. By making a class public, users are saying "I do not want to encapsulate this class, I want everyone to be able to use it". That is literally what it means. This is impossible under the current Jigsaw solution. If you want to prevent reflection being used, then package-private is the one you want. >, you should not have to provide the list of all possible existing module. > I think we want 'export private' (strong encapsulation) and 'export dynamic' to be available without having a friend list. Sure, and the same can be said for the Jigsaw solution, but in the majority of cases, your public classes would be public and for the others, you definitely want to control who specifically has access. > And again, changing the meaning of package private is a no go for me, by example for a package like java.lang.invoke, i want the package to be exported, i want strong encapsulation and i want some classes inside the package by example the class that implements the whole method handle hierarchy to share secret (using the package private access). I'm not sure I follow what you mean here. If you have an internal package like java.lang.invoke, and you want to share non-public things to specific JDK-internal targets, you would just list them as friends. >> This way the Java 8 behavior is retained with no "export dynamic" >> anywhere in sight. Rather you'd have something like "export private", >> which you'd only grant for those special cases where a package must be >> shared to a specific friend. Things not intended to be shared can be >> safely migrated to the new friend package feature. > > I agree that we should have a way to have Java 8 behavior without having to declare every packages with export dynamic, > see my discussion with mark about specifying a default export behavior. Yeah, the problem with that is that if you do this, you have defeated the security mechanism completely. With the friend package solution, you don't have to defeat the mechanism to have Java 8 behavior, allowing all existing reflection code to continue to work, which is the entire point. It still lets you solve the same use cases, just in a different way. >> No new access limits >> are introduced - instead old ones are enhanced in what I think is a >> simple and intuitive way. All the compatibility concerns go away, but >> we can still use the mechanism to seal up the problems in the JDK with >> shared code and things which should not be public. >> >> We should retain setAccessible() for now for compatibility. We could >> also introduce a "--safe-mode" flag which disables setAccessible() >> *across the board* (that is, not just for modular code) without >> controversy, because all those public members will remain accessible >> without "help", plus a way is provided to allow access to be volunteered >> from one package to another (along much the same philosophical lines as >> MethodHandles.Lookup IMO). > > Deprecating setAccessible is another story for me. I was under the impression that doing so is the whole point of this encapsulation feature. -- - DML From mark.reinhold at oracle.com Tue Jul 19 21:08:08 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 19 Jul 2016 14:08:08 -0700 (PDT) Subject: Discussion: #ResourceExistenceAndSize In-Reply-To: <57866AD3.4020807@redhat.com> References: <56D85B77.4040607@redhat.com> <20160713142913.C1C1FB78C7@eggemoggin.niobe.net> <57866AD3.4020807@redhat.com> Message-ID: <20160719210808.6AA72B9510@eggemoggin.niobe.net> 2016/7/13 9:22:43 -0700, david.lloyd at redhat.com: > On 07/13/2016 09:29 AM, mark.reinhold at oracle.com wrote: >> Reference: http://openjdk.java.net/projects/jigsaw/spec/issues/#ResourceExistenceAndSize >> >> ... >> >> At any rate, I don't see anything here that's inherently part of the >> design of the module system, so I intend to close this issue as >> unrelated. > > The relation comes in because Class and ClassLoader have a way to open a > resource with a URL, but Module does not, providing only a > getResourceAsStream() method. Adding a getResource() method would be > one solution, but the underlying requirement is really that the resource > existence and size (and origin) could/should be queriable as well, so > that was the issue I registered, especially as URL is really not a great > solution to this problem. So would a simple Module::getResourceSize(String name) method, which returns a long value to indicate the size of the named resource, or else -1 if it doesn't exist, be sufficient? We could consider adding a similar method alongside the other existing getResource* methods in java.lang.Class and ClassLoader. - Mark From mark.reinhold at oracle.com Tue Jul 19 21:09:08 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 19 Jul 2016 14:09:08 -0700 (PDT) Subject: Proposal: #DefaultModule In-Reply-To: <57866B92.200@redhat.com> References: <577BC730.1040706@redhat.com> <20160713143213.CF094B78D7@eggemoggin.niobe.net> <57866B92.200@redhat.com> Message-ID: <20160719210908.74B4EB9512@eggemoggin.niobe.net> 2016/7/13 9:25:54 -0700, david.lloyd at redhat.com: > On 07/13/2016 09:32 AM, mark.reinhold at oracle.com wrote: >> ... >> >> If this is correct then there's a simpler way to reach your goal: Enhance >> class loaders to have optional names. When the run-time system generates >> a stack trace or an exception message that mentions a module name and >> version, if present, then it could also insert the name of that module's >> class loader, if present. This would allow an external module system to >> provide better diagnostics without having to change the JPMS design in a >> fundamental way. >> >> Would that meet your needs? > > I think that would probably work very well, and fits in nicely with the > existing architecture of stack trace assembly as I understand it (since > the class loader can be directly referenced from the Class, which is > already present). Thanks! New issue: http://openjdk.java.net/projects/jigsaw/spec/issues/#ClassLoaderNames - Mark From mark.reinhold at oracle.com Tue Jul 19 21:10:08 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 19 Jul 2016 14:10:08 -0700 (PDT) Subject: Discussion: #MutableConfigurations In-Reply-To: <57866F90.5020703@redhat.com> References: <56D79D56.7040801@redhat.com> <20160713143013.C3C3AB78CC@eggemoggin.niobe.net> <57866F90.5020703@redhat.com> Message-ID: <20160719211008.7690BB9514@eggemoggin.niobe.net> 2016/7/13 9:42:56 -0700, david.lloyd at redhat.com: > On 07/13/2016 09:30 AM, mark.reinhold at oracle.com wrote: >> ... >> >> I don't think it makes sense for Configurations to be mutable, for the >> reasons stated above. It may be that the Layer concept needs to be made >> more dynamic in certain ways, but that's a different matter. >> >> I intend to close this issue unless there are strong objections from >> other EG members. > > In that case, other EG members take note: this means that hot deployment > as the world knows it is over as of Java EE 9, since a module can never > delete old or establish new dependence relationships. This also means > that OSGi loses this capability as well, so either OSGi has to be > redefined with the new restriction, or else OSGi bundles cannot be modules. First, neither OSGi nor any other existing module system will lose any capability at all. They can continue to operate just as they do today. We already have a requirement [1] to support interoperation with such systems, and so far as I can see it's satisfied by the present design. Going further to design a "meta" module system that allows other module systems to be retrofitted to interoperate on an intimate basis with the platform module system is, as I've written before, a research project that's clearly outside the scope of this JSR [2]. Second, it's certainly true that the present design does not support fully-general "hot deployment" in which arbitrary modules can come and go and be reconfigured at any time. That's intentional, because doing so would entail enormous complexity and make it incredibly difficult, if not impossible, to achieve the stated goals of this JSR. That's also why the agreed requirements mandate not this general capability but, rather, a more constrained dynamic-configuration capability, which the present design addresses with the concept of layers. - Mark [1] http://openjdk.java.net/projects/jigsaw/spec/reqs/#interoperation [2] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-February/000019.html From mark.reinhold at oracle.com Tue Jul 19 21:11:08 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 19 Jul 2016 14:11:08 -0700 (PDT) Subject: Discussion: #LazyConfigurationAndInstantiation In-Reply-To: <57867CDE.8000206@redhat.com> References: <56D7AE09.1030003@redhat.com> <20160311174751.5399C9E891@eggemoggin.niobe.net> <56E33EEC.20706@redhat.com> <20160713143113.C5B45B78D2@eggemoggin.niobe.net> <57867CDE.8000206@redhat.com> Message-ID: <20160719211108.78701B9516@eggemoggin.niobe.net> 2016/7/13 10:39:42 -0700, david.lloyd at redhat.com: > On 07/13/2016 09:31 AM, mark.reinhold at oracle.com wrote: >> Reference: http://openjdk.java.net/projects/jigsaw/spec/issues/#LazyConfigurationAndInstantiation >> >> 2016/3/11 13:55:56 -0800, david.lloyd at redhat.com: >>> ... >>> Another potenti >>> issue is requiring that all modules be present, though this is more >>> closely related to #MutableConfigurations now I think; I suspect this >>> issue could be mitigated with an acceptable solution to that issue. >> >> As I wrote in my reply re. #MutableConfigurations [2], I think this >> approach is at odds with our goal to provide reliable configuration. > > I disagree. "Reliable" is only defined in the agreed-upon requirements > by stipulating that modules may have dependence relationships which are > declared. Fair enough. I'll restate my position to say that #MutableConfigurations is at odds with the way in which the present design provides reliable configuration. > I interpreted that goal simply as an explicit rejection of > various poorly-defined customized class loader behaviors, and a move > towards clearer and more predictable behavior. By my interpretation and > experience, I consider, for example, the ability to change the explicit > dependence relationships to still be "reliable" as long as the effects > of doing so are well-defined, just as I consider the behavior of lazy > class loading to be "reliable" in that it is well-defined and > predictable, and has rules which make sense (referring to the way that > classes are loaded, resolved, and initialized in separate phases, which > allows for lazy on-demand-only progression through those phases but also > allows for almost completely arbitrary interconnection of classes, while > remaining basically predictable). These are all reasonable interpretations of the high-level goal of reliable configuration. I make no claim that the present design is the only way to achieve reliable configuration, for some reasonable definition of "reliable". It is, however, one way to achieve it. It might be possible to support reliable configuration, and to satisfy all of our other agreed requirements, with a design that configures and instantiates layers lazily rather than eagerly -- though I'm skeptical. At any rate, that is not the design that we have today. > The first time that "reliable" is specified in terms of concrete > behavior is in the SOTMS document, and these issues are being raised > exactly against the state of the module system, so I think that this > issue as well as #MutableConfigurations serve to directly challenge the > validity of the definition of "reliable" which is used by the proposed > implementation, rather than being invalid due to the presumed validity > of that definition (which was not agreed upon by the EG). The present design is meant to satisfy the requirements previously agreed by this EG. Those requirements do not include any constraints along the lines of #MutableConfigurations or #LazyConfigurationAndInstantiation. Your desire for these properties now is no basis for a claim that the design is unacceptable with respect to previous agreements of the EG. > Even the JSR definition (which I acknowledge is quite out of date at > this point) states that OSGi does address the problem of "reliable" > configuration, despite the fact that the OSGi solution allows for > dynamic loading and relinking. This is contrary to the definition in > the SOTMS, and also despite the fact that Jigsaw's more strict > interpretation of this actually invalidates those behaviors of OSGi, > were OSGi to try to merge the bundle concept with the Jigsaw module > concept somehow. The present design invalidates nothing. OSGi can continue to operate just as it does today. There is no agreed requirement to support a hypothetical future merge of OSGi's bundle concept with the platform's module concept, which is a research project that is clearly outside the scope of this JSR [1]. - Mark [1] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-February/000019.html From mark.reinhold at oracle.com Wed Jul 20 14:33:29 2016 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 20 Jul 2016 07:33:29 -0700 Subject: Me trying to explain the problem behind #ReflectiveAccessToNonExportedTypes In-Reply-To: <1028346570.1564026.1468532781557.JavaMail.zimbra@u-pem.fr> References: <1627574708.1456733.1468507103919.JavaMail.zimbra@u-pem.fr> <1105959088.1485118.1468508796360.JavaMail.zimbra@u-pem.fr> <20160714093304.841168761eggemoggin.niobe.net> <1028346570.1564026.1468532781557.JavaMail.zimbra@u-pem.fr> Message-ID: <20160720073329.361423832eggemoggin.niobe.net> 2016/7/14 14:46:21 -0700, Remi Forax : > 2016/07/14 09:33:04 -0700, mark.reinhold at oracle.com: >> 2016/7/14 8:06:36 -0700, Remi Forax : >>> ... >>> >>> While i agree that "export dynamic" is a semantics that jigsaw should >>> provide, i disagree with the syntax "export dynamic" because for me, >>> the a non exported package should have this semantics in order to be >>> backward compatible with existing code. >>> So we should have 3 way to export or not a package, >>> 1. don't export at compile time, don't export at runtime >>> 2. don't export at compile time, export at runtime >>> 3. export at compile time, export at runtime. >>> >>> 2 should be the default, 1 should be use by the package of the JDK (or >>> any other libs) that want strong security, 3 is for the package that >>> defines API (that will be maintained forever). >> >> A problem with changing the default is that many developers will never >> think to do anything else, and leave all their packages unnecessarily >> exported. We have examples of this sort of thing elsewhere in Java, >> e.g., default public constructors, which have been problematic for >> years. Do we really want to repeat that kind of mistake? >> >> What do you think of allowing wildcards, i.e., `exports dynamic *`? >> That way a developer could explicitly choose to export everything at >> run time, but could do so in a very compact way. > > Wildcards seems a great idea. > > But if we want authors to make a decision, instead of choosing a > default for them, we should let them to make a decision. > > So wildcards should exist in both way 'export dynamic *' or 'export > noexport *' (or whatever we call it) and we should mandate that a > module to have an export wildcard declaration. > > It's more verbose that the current settings but at least this make the > choice explicit. Requiring something to be declared explicitly one way or the other, rather than offer a safe and sensible default, seems very unlike the rest of the language. - Mark From forax at univ-mlv.fr Wed Jul 20 16:40:23 2016 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Wed, 20 Jul 2016 18:40:23 +0200 (CEST) Subject: Me trying to explain the problem behind #ReflectiveAccessToNonExportedTypes In-Reply-To: <20160720073329.361423832eggemoggin.niobe.net> References: <1627574708.1456733.1468507103919.JavaMail.zimbra@u-pem.fr> <1105959088.1485118.1468508796360.JavaMail.zimbra@u-pem.fr> <20160714093304.841168761eggemoggin.niobe.net> <1028346570.1564026.1468532781557.JavaMail.zimbra@u-pem.fr> <20160720073329.361423832eggemoggin.niobe.net> Message-ID: <2141349622.931070.1469032823905.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "mark reinhold" > ?: forax at univ-mlv.fr > Cc: jpms-spec-experts at openjdk.java.net > Envoy?: Mercredi 20 Juillet 2016 16:33:29 > Objet: Re: Me trying to explain the problem behind #ReflectiveAccessToNonExportedTypes > > 2016/7/14 14:46:21 -0700, Remi Forax : > > 2016/07/14 09:33:04 -0700, mark.reinhold at oracle.com: > >> 2016/7/14 8:06:36 -0700, Remi Forax : > >>> ... > >>> > >>> While i agree that "export dynamic" is a semantics that jigsaw should > >>> provide, i disagree with the syntax "export dynamic" because for me, > >>> the a non exported package should have this semantics in order to be > >>> backward compatible with existing code. > >>> So we should have 3 way to export or not a package, > >>> 1. don't export at compile time, don't export at runtime > >>> 2. don't export at compile time, export at runtime > >>> 3. export at compile time, export at runtime. > >>> > >>> 2 should be the default, 1 should be use by the package of the JDK (or > >>> any other libs) that want strong security, 3 is for the package that > >>> defines API (that will be maintained forever). > >> > >> A problem with changing the default is that many developers will never > >> think to do anything else, and leave all their packages unnecessarily > >> exported. We have examples of this sort of thing elsewhere in Java, > >> e.g., default public constructors, which have been problematic for > >> years. Do we really want to repeat that kind of mistake? > >> > >> What do you think of allowing wildcards, i.e., `exports dynamic *`? > >> That way a developer could explicitly choose to export everything at > >> run time, but could do so in a very compact way. > > > > Wildcards seems a great idea. > > > > But if we want authors to make a decision, instead of choosing a > > default for them, we should let them to make a decision. > > > > So wildcards should exist in both way 'export dynamic *' or 'export > > noexport *' (or whatever we call it) and we should mandate that a > > module to have an export wildcard declaration. > > > > It's more verbose that the current settings but at least this make the > > choice explicit. > > Requiring something to be declared explicitly one way or the other, > rather than offer a safe and sensible default, seems very unlike the > rest of the language. Choisir c'est renoncer (Andr?e Gide) - To choose is to forsake There are parts of Java that have no defaults, method calls resolution, default methods. There are parts were the defaults was great in 1995 (or later) but that are not great anymore, fields non-private/non-final by default, inference of type variable defaulting to Object instead of reporting an error. And you have defaults that are still great, unlike C++ you don't have to opt-in to enable reflection. A great strength of Java, is to have resisted to make backward incompatible changes even for good reason, we have renounced to great features to keep backward compatibility, fully reified generics, structural typing of lambdas. In Java, backward compatibility trump all other considerations. And we have all witnesses of the Python2/Python3 debacle, we even know the consequence of not being backward compatible. So the only safe and sensible choice is to be to have non-exported packages classes to be available at runtime by default. And obviously to have a way to declare for a given module that all accesses to a classe of a non exported package are denied at runtime. > > - Mark > R?mi From david.lloyd at redhat.com Thu Jul 21 16:05:55 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 21 Jul 2016 11:05:55 -0500 Subject: Discussion: #MutableConfigurations In-Reply-To: <20160719211008.7690BB9514@eggemoggin.niobe.net> References: <56D79D56.7040801@redhat.com> <20160713143013.C3C3AB78CC@eggemoggin.niobe.net> <57866F90.5020703@redhat.com> <20160719211008.7690BB9514@eggemoggin.niobe.net> Message-ID: <6fcfaccf-f81a-9fb3-bb1a-cdff9c60f85f@redhat.com> On 07/19/2016 04:10 PM, mark.reinhold at oracle.com wrote: > 2016/7/13 9:42:56 -0700, david.lloyd at redhat.com: >> On 07/13/2016 09:30 AM, mark.reinhold at oracle.com wrote: >>> ... >>> >>> I don't think it makes sense for Configurations to be mutable, for the >>> reasons stated above. It may be that the Layer concept needs to be made >>> more dynamic in certain ways, but that's a different matter. >>> >>> I intend to close this issue unless there are strong objections from >>> other EG members. >> >> In that case, other EG members take note: this means that hot deployment >> as the world knows it is over as of Java EE 9, since a module can never >> delete old or establish new dependence relationships. This also means >> that OSGi loses this capability as well, so either OSGi has to be >> redefined with the new restriction, or else OSGi bundles cannot be modules. > > First, neither OSGi nor any other existing module system will lose any > capability at all. They can continue to operate just as they do today. > We already have a requirement [1] to support interoperation with such > systems, and so far as I can see it's satisfied by the present design. I would like this to be true, but unfortunately, operating as we do today is not going to be a long-term option for us given previously-discussed future requirements relating to Java EE 9 (see below). > Going further to design a "meta" module system that allows other module > systems to be retrofitted to interoperate on an intimate basis with the > platform module system is, as I've written before, a research project > that's clearly outside the scope of this JSR [2]. I disagree that it's "clearly" out of scope; what you've delineated as "research project" versus design parameters seems arbitrary to me. It's certainly been made to be out of scope for the current implementation but it could have easily been a part of the design to begin with. > Second, it's certainly true that the present design does not support > fully-general "hot deployment" in which arbitrary modules can come and go > and be reconfigured at any time. We cannot support this restriction. Our Java EE container is built on the ability to support hot deployment; this pronouncement, in combination with the earlier pronouncement that Java EE 9 will almost certainly require deployment units to be modules, is an unacceptable result for us. There must be some way for us to achieve this, either by adding the necessary flexibility to configurations or by allowing another way to define and redefine modules outside of the constraints of the current implementation. > That's intentional, because doing so > would entail enormous complexity and make it incredibly difficult, if not > impossible, to achieve the stated goals of this JSR. I completely disagree. Class files, as defined since the very start of Java, meet the exact same goals for classes as are stated in the JPMS requirements for modules, and they do so by using an incremental algorithm that allows for classes to be loaded and unloaded at any time. Our (JBoss) module system does the exact same thing, in a very similar way, for the same reasons, and still manages to provide the same requirements (sans accessibility restrictions, which can only be part of the VM, but which could be accomplished using techniques I've previously proposed). It's clearly possible to do. > That's also why the > agreed requirements mandate not this general capability but, rather, a > more constrained dynamic-configuration capability, which the present > design addresses with the concept of layers. The problem is that with layers being strictly hierarchical and configurations immutable, it will not be possible for us to deliver a Java EE 9 compliant server without essentially starting over, and giving up important capabilities in the process. I don't see how we can possibly agree to this. We simply must have some way to reconfigure modules at run time, *and* some way to introduce dependence relationships with modules from extra-hierarchical layers - in addition to resolutions to certain other previously-raised issues - in order to move our technology towards Java EE 9, or else we need the guarantee that there will be no requirement for Java EE 9 or any future Java EE or related specification which involves mapping Java EE applications, modules, and/or their constituent pieces directly to named JDK-style modules. I more than anyone want to find a way to make this work but I just don't see any way around it. > [1] http://openjdk.java.net/projects/jigsaw/spec/reqs/#interoperation > [2] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-February/000019.html -- - DML From david.lloyd at redhat.com Thu Jul 21 16:45:56 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 21 Jul 2016 11:45:56 -0500 Subject: Discussion: #LazyConfigurationAndInstantiation In-Reply-To: <20160719211108.78701B9516@eggemoggin.niobe.net> References: <56D7AE09.1030003@redhat.com> <20160311174751.5399C9E891@eggemoggin.niobe.net> <56E33EEC.20706@redhat.com> <20160713143113.C5B45B78D2@eggemoggin.niobe.net> <57867CDE.8000206@redhat.com> <20160719211108.78701B9516@eggemoggin.niobe.net> Message-ID: <1be6cb2a-4ef3-e37c-5e34-ca777cc7ecb4@redhat.com> On 07/19/2016 04:11 PM, mark.reinhold at oracle.com wrote: > 2016/7/13 10:39:42 -0700, david.lloyd at redhat.com: >> On 07/13/2016 09:31 AM, mark.reinhold at oracle.com wrote: >>> Reference: http://openjdk.java.net/projects/jigsaw/spec/issues/#LazyConfigurationAndInstantiation >>> >>> 2016/3/11 13:55:56 -0800, david.lloyd at redhat.com: >>>> ... >>>> Another potenti >>>> issue is requiring that all modules be present, though this is more >>>> closely related to #MutableConfigurations now I think; I suspect this >>>> issue could be mitigated with an acceptable solution to that issue. >>> >>> As I wrote in my reply re. #MutableConfigurations [2], I think this >>> approach is at odds with our goal to provide reliable configuration. >> >> I disagree. "Reliable" is only defined in the agreed-upon requirements >> by stipulating that modules may have dependence relationships which are >> declared. > > Fair enough. I'll restate my position to say that #MutableConfigurations > is at odds with the way in which the present design provides reliable > configuration. Indeed, and this is why these issues have been raised: the current design is not acceptable as-is. >> I interpreted that goal simply as an explicit rejection of >> various poorly-defined customized class loader behaviors, and a move >> towards clearer and more predictable behavior. By my interpretation and >> experience, I consider, for example, the ability to change the explicit >> dependence relationships to still be "reliable" as long as the effects >> of doing so are well-defined, just as I consider the behavior of lazy >> class loading to be "reliable" in that it is well-defined and >> predictable, and has rules which make sense (referring to the way that >> classes are loaded, resolved, and initialized in separate phases, which >> allows for lazy on-demand-only progression through those phases but also >> allows for almost completely arbitrary interconnection of classes, while >> remaining basically predictable). > > These are all reasonable interpretations of the high-level goal of > reliable configuration. > > I make no claim that the present design is the only way to achieve > reliable configuration, for some reasonable definition of "reliable". > It is, however, one way to achieve it. > > It might be possible to support reliable configuration, and to satisfy > all of our other agreed requirements, with a design that configures and > instantiates layers lazily rather than eagerly -- though I'm skeptical. I think it ought to be discussed. It's my estimation that many of the parameters of the current design are in excess of the agreed-upon requirements and can be rolled back to an extent without compromising what was already agreed upon. > At any rate, that is not the design that we have today. Agreed, and that is why we have raised these issues. >> The first time that "reliable" is specified in terms of concrete >> behavior is in the SOTMS document, and these issues are being raised >> exactly against the state of the module system, so I think that this >> issue as well as #MutableConfigurations serve to directly challenge the >> validity of the definition of "reliable" which is used by the proposed >> implementation, rather than being invalid due to the presumed validity >> of that definition (which was not agreed upon by the EG). > > The present design is meant to satisfy the requirements previously agreed > by this EG. Those requirements do not include any constraints along the > lines of #MutableConfigurations or #LazyConfigurationAndInstantiation. > Your desire for these properties now is no basis for a claim that the > design is unacceptable with respect to previous agreements of the EG. I can't agree with this logic: it could just as easily be said that various behaviors of the current design (which exceed the agreed-upon requirements in various ways) also cannot be justified with respect to previous agreements, and that no claim that such decisions are immutable can be valid. In fact my agreement to the requirements was based on the fact that I found nothing in the requirements that invalidates our current module system. I do not think it was unreasonable to expect that we would then have had a subsequent design discussion and agreement, let alone that requirements may be revised as new information becomes available, though none of that ever transpired; the issues process is now the only vehicle for raising design issues or problems with requirements that may have only become obvious after a prototype JDK integration has occurred, or requirements that have overreached into design specifics, and therefore that is the process I am using for all of these cases. >> Even the JSR definition (which I acknowledge is quite out of date at >> this point) states that OSGi does address the problem of "reliable" >> configuration, despite the fact that the OSGi solution allows for >> dynamic loading and relinking. This is contrary to the definition in >> the SOTMS, and also despite the fact that Jigsaw's more strict >> interpretation of this actually invalidates those behaviors of OSGi, >> were OSGi to try to merge the bundle concept with the Jigsaw module >> concept somehow. > > The present design invalidates nothing. OSGi can continue to operate > just as it does today. There is no agreed requirement to support a > hypothetical future merge of OSGi's bundle concept with the platform's > module concept, which is a research project that is clearly outside > the scope of this JSR [1]. Sure, but it is pure disingenuity to imply that such a thing might somehow be possible in the future when it obviously will not be (without resolution to these issues). It is more honest to be upfront and say "this module system is designed in such a way that OSGi-style bundles can never be mapped to it," i.e. it seems quite likely to me that we are consciously giving OSGi and other flexible classloader-based module systems an eventual death sentence if using Jigsaw-style modules will eventually considered (for example) a security necessity, not to mention the previously mentioned problems that arise if such a system is the basis for a Java EE application server. -- - DML