From mark.reinhold at oracle.com Mon Mar 9 20:54:17 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 9 Mar 2015 13:54:17 -0700 (PDT) Subject: Interoperation (Was: Re: Module-system requirements) In-Reply-To: <54E1F75C.4070009@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, <20150215172424.772730@eggemoggin.niobe.net>, <54E1F75C.4070009@redhat.com> Message-ID: <20150309205417.DE06C511E7@eggemoggin.niobe.net> 2015/2/16 5:57 -0800, david.lloyd at redhat.com: > On 02/15/2015 07:24 PM, mark.reinhold at oracle.com wrote: >> 2015/2/11 11:56 -0800, david.lloyd at redhat.com: >>> I'm concerned that the interoperation point doesn't quite go far enough. >>> It says "It must be possible for another module system, such as OSGi, >>> to locate Java modules and resolve them using its own resolver, except >>> possibly for core system modules." >>> >>> I agree with this requirement, such as it is, but I would also assert >>> that it should be possible to retrofit not only OSGi but Java EE, Maven, >>> and even legacy "modules" like extensions and so on as modules. The >>> reason I make this assertion is that we have done all of these things >>> with success and (I dare say) relative ease, even to the extent of >>> achieving both EE and OSGi certification with such a retrofit. >> >> Can you explain what you mean, exactly, by "retrofit"? > > We have implemented both a certified OSGi implementation as well as a > certified Java EE implementation which maps these specifications' module > concepts onto our own concept of a module. ... > >> Are you saying that the JBoss module system can consume Java EE WAR >> files, OSGi bundles, and Maven projects, as they stand and without >> change? > > Yes*. This is because of implementation choices we've made - each > module is a class loader, and we can carefully control which classes > *and* resources are exported from one to another. That said, I do think > it's still worth exploring a different approach than ours regarding > conceptually combining resource directories and packages, which is what > we do today and is not without disadvantage. > > ... > >>> ... >>> >>> To achieve this additional requirement does require a few very specific >>> implementation considerations but I do not believe that any of these >>> considerations contraindicate any of the other requirements stated here. >> >> Can you be more specific as to what those considerations are? Can they >> be expressed as module-system requirements? > > In the case of both Java EE and OSGi, I think it's safe to say that you > need to apply a strict 1:1 mapping between modules and class loaders for > this to work. I don't think we can mandate this. To divide the SE platform into modules in a compatible way requires the ability to load multiple modules via the same class loader, so as to respect long-standing assumptions about which platform classes are loaded by the built-in class loaders (i.e., the bootstrap and extension loaders). In an early Jigsaw prototype we tried the 1:1 module/loader mapping for platform modules and found it unworkable from a compatibility perspective. Aside from the SE platform itself, I don't think we can force all applications to adopt the unique-class-loader-per-module approach. Oracle's WebLogic application server, e.g., doesn't do that today and it'd be difficult, if not impossible, to modify it to do so without breaking existing applications. I suggest the module system be as neutral as possible with respect to how modules and class loaders are related. A complex application should be able to spin up a class loader for every module it loads, if needed (excepting non-upgradeable platform classes), but not all applications should be forced to do so. New requirement, for the Dynamic Configuration section: - _Control of class loaders_ --- An application that loads a module into a dynamic configuration must be able to specify the class loader to be used to load that module. That loader need not be the same as the loader used to load any other module in the configuration. > Also you need an API to allow run-time systems to create > their own configurations (we call them "module loaders", akin in purpose > and function to class loaders) and assemble modules from resources > (JARs) and dependency specifications, and such APIs have to be able to > relink modules or deregister modules to allow them to be GC'd (though > the boot module loader in our case does not expose relink or unload > functionality for obvious reasons). These needs are already captured, I think, by the existing requirements related to dynamic configurations. > OSGi specifically requires that bundle class loaders implement an OSGi > spec interface, implying that the module class loader class must be > extensible (as I said a couple of years back though, we make most of our > methods final on the module class loader to disallow any really bizarre > behavior, and this works well in practice). Which OSGi interface are you thinking of here? > We also had to implement a > pre- and/or post-class-define hook per module to allow some OSGi magic > to happen. These are non-final protected methods in your module class loader? Could you forward a description of these methods to the list? > Finally there is a per-module fallback resource loader that > our OSGi implementation used for one particular part of the spec. Which part, in particular? To what extent does OSGi assume that bundles "export" their resources? If we decide that a module's resources are not available outside the module itself then will the approach of extending a module class loader to be an OSGi bundle loader even work? > Java EE's requirements are much simpler and are satisfied by > module-granularity dependencies and (in certain cases) resource importing. Aside from things like deployment descriptors, which are interpreted by Java EE implementations, does the EE specification require EE "modules" to export resources to other modules? > The APIs to do these things are all pretty simple though. > >>> Thus I feel this requirement should be amended to something along the >>> lines of: >>> >>> Interoperation - It must be possible for another module system, such as >>> OSGi, to locate Java modules and resolve them using its own resolver, >>> except possibly for core system modules. It must be possible for other >>> module systems to reuse the framework provided by this specification in >>> order to facilitate a standard, well-defined mechanism by which >>> different present and future module systems may interoperate. >> >> Unless I've misunderstood something, this sounds more like a research >> project -- a "meta" module system, in fact -- which is well outside the >> scope of this JSR. > > Maybe, but I contend that a lot (enough, insofar as any amount is ever > enough) of research has been done, and that in any event there will have > to be an API which implements the functionality of this JSR anyway, and > it will be used as such no matter our intent so IMO it's best to use > what has been learned to solve the problem in the best manner possible. I agree, but I think your proposed amendment is problematic -- it's too difficult to test, since it's worded in terms of unspecified "present and future module systems". Are there other, more-specific requirements we can identify which would enable this stronger kind of interoperation? - Mark From mark.reinhold at oracle.com Mon Mar 9 20:55:17 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 9 Mar 2015 13:55:17 -0700 (PDT) Subject: Configurations... (Was: Re: Module-system requirements) In-Reply-To: <54E1FAE2.2000605@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, <20150214133307.760777@eggemoggin.niobe.net>, <54E1FAE2.2000605@redhat.com> Message-ID: <20150309205517.E1578511E9@eggemoggin.niobe.net> 2015/2/16 6:12 -0800, david.lloyd at redhat.com: > On 02/14/2015 03:33 PM, mark.reinhold at oracle.com wrote: >> 2015/2/12 7:35 -0800, david.lloyd at redhat.com: >>> I'm a little concerned that the configurations section implies a strict >>> hierarchy: >>> >>> "Nested dynamic configurations ??? An application must be able to create a >>> dynamic configuration that relates to an existing dynamic configuration >>> rather than to the application???s initial configuration." >>> >>> Why can a configuration not relate to more than one other configurations? >> >> A tree of configurations rather than a simple linear list is more >> difficult to reason about and trickier to implement (you must, e.g., >> cope with the diamond-import problem). >> >> The feedback we've received so far has not suggested that this >> limitation will be a problem in practice. Do you have use cases >> which require this flexibility? > > We implement not just a tree, but a general directed graph (even > allowing cycles) among modules and among their configurations (module > loaders), A configuration, as defined in the requirements document, is a group of modules related by resolution to each other and, if needed, to modules in one or more enclosing configurations. If you look just at the graph of modules in a running system then it's at least a DAG, and possibly even cyclic, but that doesn't necessarily mean that the configurations themselves need to be related to each other in a structure any richer than a tree. > which enables us to easily support OSGi and Java EE, as well > as tying these things to filesystem JARs (e.g. for EE deployments which > have Class-Path declarations that have an absolute path name), and the > legacy extensions mechanism which is presently also a part of Java EE > (though this is going/has gone away, I understand), and also allows > tying in other systems as well. Does doing these things require configurations to be related in a directed graph, or is a tree sufficient? > Due to the way we support imports, diamonds are really no problem. Each > module's dependency list is interpreted in order, so the set of imported > resources is always the union of everything exported by each dependency. > So if you acquire a different set of items via two different "paths" > to a module, you get the union of those sets in the end. Duplicated > paths are removed such that the first occurrence is used. Ah, so the order of dependence declarations is significant in your system. Interesting, but I wonder if it's wise. Anyway, the diamond-import problem I was thinking of has to do with configurations rather than modules. It can be solved in a manner similar to what you describe for resources, i.e., by forcing an application that wants to do this to place an order on the configurations to which a new configuration is to be related. When looking for a module the resolver would consider the configurations in that order, and since only one module can satisfy any particular dependence it would just stop at the first one that it finds. I'm still concerned, however, that this could lead to some unpleasant implementation complexities, and also be more difficult to use in practice, so unless there's a clear need to relate configurations in graphs then I'd prefer to limit them to trees. - Mark From mark.reinhold at oracle.com Mon Mar 9 20:56:17 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 9 Mar 2015 13:56:17 -0700 (PDT) Subject: Module-system requirements In-Reply-To: References: <20150211103831.528761@eggemoggin.niobe.net>, , <20150215172507.827110@eggemoggin.niobe.net>, Message-ID: <20150309205617.E54AB511F0@eggemoggin.niobe.net> 2015/2/15 10:12 -0800, Tim Boudreau : > mark.reinhold at oracle.com: >> As I wrote in my reply to David, I think resources (in the traditional >> ClassLoader::getResource sense) should be treated as being strictly >> internal to modules. ... > > There's a common case that I'd like to understand how you solve without at > least META-INF or its equivalent visible to other modules: It's fairly > common to see applications that dynamically compose things they find on the > classpath. Spring and a few other things accomplish this through > classpath-scanning, which is a fairly horrible pattern (you can wind up > visiting every class on the classpath, or a large subset, just to look for > a runtime annotation that is not there for the vast majority). > > An alternative approach I've used many times, which I think a few > frameworks do as well (dagger?) is: Write an annotation processor that > generates a flat file in META-INF that lists the classes that have the > annotation in question; runtime code loads all such files on the classpath > and uses that to initialize things that should be present at runtime. It's > really the same pattern as ServiceLoader. > > Is it safe to assume that, whatever happens, that approach will still work, > whether by special-casing META-INF/** or some equivalent? For JAR files on the legacy class path that approach will, of course, continue to work. I don't expect actual modules to be on the class path (about which more below), which raises a question: Will a framework such as Spring need to discover the classes bearing certain annotations in modules that aren't yet configured and loaded, or in loaded modules, or both? In the former case it would inspect module artifacts, while in the latter it would inspect some reflective run-time representation of a loaded module, since the original module artifact might not be available. Either way, this use case suggests a new requirement for the Development section: - _Annotation digests_ --- When packaging a module into an artifact it must be possible to augment its definition with a summary of all of the type- and package-level annotations that are present in the module, together with an indication of the types and packages to which each annotation applies. This digest must be easily readable from module artifacts, and at run time those parts of it related to annotations retained for run time must be available through an appropriate reflective API. Comments? >> Having said that, I do agree that the module system must provide some >> kind of escape hatch or "promiscuous mode", as you call it, so here's >> a stab at an additional requirement for the "Modularizing Java SE" >> section: >> >> - _Overrideable encapsulation_ --- It must be possible to force a >> module to export one or more of its packages to all other modules, >> or to some specific modules, even if its definition does not do so. > > One sort of low-hanging-fruit approach to this would be: > - Modules do not use the traditional $CLASSPATH > - The contents of $CLASSPATH are treated by the system as a single module > (so everything can see each other, and legacy code behaves as expected) That's looking ahead to a design decision, I think, but in fact the approach I intend to propose treats the legacy class path as the content of a special "unnamed" module, analogous to the unnamed package, which implicitly depends upon all other modules so that it works essentially as you describe. - Mark From mark.reinhold at oracle.com Mon Mar 9 20:57:17 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 9 Mar 2015 13:57:17 -0700 (PDT) Subject: Module-system requirements In-Reply-To: References: <20150211103831.528761@eggemoggin.niobe.net>, , <20150215172507.827110@eggemoggin.niobe.net>, , , Message-ID: <20150309205717.E8E62511F2@eggemoggin.niobe.net> 2015/2/16 12:32 -0800, Tim Boudreau : > tim_ellison at uk.ibm.com: >> Tim Boudreau : >>> One sort of low-hanging-fruit approach to this would be: >>> - Modules do not use the traditional $CLASSPATH >>> - The contents of $CLASSPATH are treated by the system as a single >>> module (so everything can see each other, and legacy code behaves >>> as expected) >> >> Mark's new requirement would do it - though I had to swallow hard after >> reading the definition, as I heard all the safety-catches being clicked >> off. >> >> I'm struggling to see how treating the contents of the classpath as a >> single module would help here? The problem is with platform modules >> hiding previously accessible code. > > I'm not sure it does help with that. I don't think it does -- treating the content of the class path as a module is just a convenient (maybe even elegant) way to relate legacy, non-modular code to modular code. > I'm sure in this JSR there will be > lots of different problems that each of us thinks of as The Most Important > Problem to Solve. No doubt ... > The idea I was trying to get across is that, if you want to partition the > world up into modules and do that in a compatible way, then start by > modularizing the JDK, and giving *it* a new way to link. At the > application layer, keep the default behavior by treating everything linked > the traditional way as, for the purposes of the framework, a single large > "default" module where all the old rules apply. As libraries modularize, > at their own pace, they move out of that "default' module and become > modules in their own right, and new linking rules apply. Yes -- this is roughly the migration strategy I have in mind. - Mark From mark.reinhold at oracle.com Mon Mar 9 20:58:17 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 9 Mar 2015 13:58:17 -0700 (PDT) Subject: Non-interference (was: Re: Module-system requirements) In-Reply-To: References: <20150211103831.528761@eggemoggin.niobe.net>, <20150215172507.827110@eggemoggin.niobe.net>, Message-ID: <20150309205817.EC748511F4@eggemoggin.niobe.net> 2015/2/16 9:04 -0800, tim_ellison at uk.ibm.com: > mark.reinhold at oracle.com wrote on 16/02/2015 01:25:07: >> 2015/2/13 6:06 -0800, tim_ellison at uk.ibm.com: >>> Can you explain "interfere"? Is this referring to ambiguous class and >>> linkage error type problems, or something else here? >> >> The former. If two distinct modules contain packages of the same name >> then code in that package in one module must not be able to access >> package-private types or members in that package in the other module. >> If two distinct modules contain classes of the same name (i.e., with the >> same binary name) then one must not be able to stand in for the other. > > Ok, so the module name partitions the package name space. Yes. > Consider the following change for the Exports fundamental definition: > > Exports ??? A module???s definition must be able to indicate which > of its packages are exported for use by other modules, and which > are not. Multiple modules may export packages of the same name. Good -- I'll revise as you suggest. - Mark From mark.reinhold at oracle.com Mon Mar 9 20:59:17 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 9 Mar 2015 13:59:17 -0700 (PDT) Subject: Exporting things (Was: Re: Module-system requirements) In-Reply-To: <54E1ED8C.8010208@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, <20150215172405.90174@eggemoggin.niobe.net>, <54E1ED8C.8010208@redhat.com> Message-ID: <20150309205917.F049C511F6@eggemoggin.niobe.net> 2015/2/16 5:15 -0800, david.lloyd at redhat.com: > On 02/15/2015 07:24 PM, mark.reinhold at oracle.com wrote: >> 2015/2/11 11:27 -0800, david.lloyd at redhat.com: >>> I notice that under Exports, it is defined that only packages (i.e. >>> classes within those packages) are considered to be exportable, and not >>> resources. I'm not certain how I feel about that... >>> >>> ... To sum up though, I'd be glad to see (from my perspective) this >>> behavior go away in favor of the simpler model of restricting exports to >>> packages (and isolating classes and resources), >> >> I tend to agree. >> >> Exposing all of a module's resources, as JAR files do today, is wrong >> from the perspective of encapsulation. ... >> >> This suggests an additional requirement: >> >> - _Resource encapsulation_ --- The run-time system must ensure that the >> static resource files within a module are directly accessible only by >> code within that module. > > What you are saying here implies (to me) inspecting the call stack to > ensure that the caller actually exists within the module being > referenced. Yes. > I'm not certain we really want to go this far, if my > interpretation is accurate. Such a rule is reasonable on the face of > it, but it does more or less break all the code in existence which > currently uses resources as a sort of API into a library, and this > includes very common patterns for loading configuration (among many > other things). True. To what degree should it be a goal to preserve such mechanisms? Must Java modules be forever constrained to look like JAR files? >>> but only if we can be >>> reasonably sure that the use cases solved thereby are somehow sorted >>> out, which include: >>> >>> ??? Supporting this case (or determining that it is not relevant): A >>> module privately uses services provided by one or more specific peer >>> module(s) (via ServiceLoader), the arrangement of which is determined >>> statically by distribution. >> >> One way to accomplish this is to make the service interface accessible >> only by the relevant modules, via qualified exports. Then only those >> modules can provide or use implementations of the service. >> >> Another possibility is to add a requirement that the binding of providers >> to service interfaces can be controlled manually during the configuration >> process. Is that more along the lines of what you mean when you say >> "determined statically by the distribution"? > > Yes it *could* be, assuming that configurations end up being > sufficiently flexible (e.g. I have one distribution configuration, and > then I have N application configurations for each installed application, > which is in turn an extension of the distribution configuration). Yes, that's exactly the intent of nested configurations. To ensure that a distributor of a set of modules can control how service dependences are resolved amongst those modules here's a new requirement, for the Fundamentals section: - _Selective binding_ --- It must be possible to control the binding process so that specific services are provided only by specific providers, and in a specific order. Does that make sense? > It's a question, I guess, of how the configurations work and how much > power the application configuration has to customize/mess up the > distribution configuration. The assumption is that a nested configuration, of itself, cannot change the configuration(s) in which it's nested. That's worth making explicit in the existing requirement for - _Isolated dynamic configurations_ --- An application must be able to isolate the code in different dynamic configurations at least as well as is possible today, where this is typically done by using multiple class loaders. ADD: An application must be able to ensure that code in a dynamic configuration does not modify other configurations. > In a setup like we have, where one module > can just statically import another's services, it's pretty simple for > the distributor to set it up and really difficult for an application to > mess it up. So a JBoss module can specify that it depends upon service providers from one or more specific modules rather than whichever modules happen to provide implementations of that service? That seems contrary to the nature of services, if I understand you correctly. >>> ??? Having (generally) the same visibility to .class files as to the >>> classes themselves, from a given resource-loading entry point (Class or >>> ClassLoader), i.e. if I have visibility to >>> classLoader.loadClass("SomeClass"), I also have visibility to >>> classLoader.getResource("SomeClass.class"), and vice-versa (any security >>> restrictions aside). >> >> Can you describe some use cases that require this property? > > Just doing a quick spin through libraries used or referenced by WildFly > and its dependencies: Libraries such as CGLIB and BCEL use this to > modify existing classes. Ant apparently uses it to find JavaCC, though > this odd use is probably better done another way. Hibernate uses it to > get bytes to instrument in its instrumenting class loader, as does our > JPA SPI. We also use this trick to generate various fast indexes of > classes without actually loading the classes themselves for various > tools and deployment operations. I believe we have some tooling using > this as well to do various types of analysis on loaded code. When an application is assembled by putting JAR files on the class path then yes, you definitely need CL::getResource("SomeClass.class"), or something like it, in order to do things like instrumentation. That's because the class-path mechanism (mostly) hides the identities of the original JAR files -- there's no way find the JAR file that will contain a particular class file and then load that file yourself, rewriting its bytecodes along the way. When, in the future, an application is assembled by resolving a set of actual modules into a configuration, then we won't necessarily have to use class loaders to mediate resource access, so I'd rather not bake that into the requirements. A configuration ought to be able to relate classes, modules, and module artifacts such that, given a module and a class name, you can read the corresponding class file (or any other kind of resource) from the module's defining artifact and load it however you want. To capture this as a requirement, in the Development section: - _Access to class files and resources_ --- If a module is defined by an artifact that contains class files and resources then it must be possible, once that module is added to a configuration, for suitably-privileged code to access those class files and resources. (The "if a module is defined by an artifact" precondition allows for ahead-of-time-compilation scenarios.) This requirement also addresses the problem of how to locate and read EE deployment descriptors and similar kinds of configuration files. > ... > >>> ??? A consistent ruling on the right behavior of general resources which >>> are co-packaged with classes that use them (typically loaded via e.g. >>> getClass().getResourceAsStream("MyResource.properties") or similar). >> >> If we take the view that resources are module-internal then the answer >> is that code within a module can access that module's resources but code >> outside of the module cannot (with the usual exception for reflection, >> which would require a security check). > > This brings us close to a previous point of contention regarding having > the implementation require a class loader for each module (or not); the > problem never came up for us (even in security manager environments) > because acquiring a class loader already entails a security check. As I wrote earlier, I don't think we can mandate one class loader per module. >>> ??? A consistent ruling on whether it makes sense generally to be able to >>> export resources to dependents. >> >> If we take the view that resources are module-internal then the answer is >> no, this does not make sense. A module can export packages of classes >> and interfaces, and can publish service providers; it cannot also export >> files in a string-based hierarchical namespace. > > I guess the salient underlying question is, "why not?". I'm being (for > the most part) a devil's advocate here, but I think this question really > has to be answered succinctly, because we're really trying to change a > basic (de facto) principle of Java library interoperation (that they > often use resources as a sort of API, even if only primarily for service > loader-style and configuration use cases). > > I can easily imagine an "export resources" feature which specifies > resource directories to make exportable (due in no small part to the > fact that such a feature has been used extensively by us with good > results). Yes, we could do this, but it doesn't seem necessary. Applications that need to access a module's class files and resources can do so directly, as described above. I don't see a need to complicate the module system itself with a notion of exportable resources. > Another use case beyond those mentioned above that is > satisfied by such a capability could be bundling supplementary i18n/l10n > or time zone data or similar as a module. The approach we've taken in the JDK for things like time-zone and locale data is to define appropriate services and providers thereof. That way all the information goes through a proper statically-checked interface, and the layout and format of the actual data can evolve without having to update its consumers. - Mark From mark.reinhold at oracle.com Mon Mar 9 21:00:17 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 9 Mar 2015 14:00:17 -0700 (PDT) Subject: Exporting things (Was: Re: Module-system requirements) In-Reply-To: <54E745C0.80603@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, <54E1ED8C.8010208@redhat.com>, <54E745C0.80603@redhat.com> Message-ID: <20150309210017.F3C89511F8@eggemoggin.niobe.net> 2015/2/20 6:33 -0800, david.lloyd at redhat.com: > On 02/16/2015 07:15 AM, David M. Lloyd wrote: >> ... >> >> What you are saying here implies (to me) inspecting the call stack to >> ensure that the caller actually exists within the module being >> referenced. I'm not certain we really want to go this far, if my >> interpretation is accurate. Such a rule is reasonable on the face of >> it, but it does more or less break all the code in existence which >> currently uses resources as a sort of API into a library, and this >> includes very common patterns for loading configuration (among many >> other things). > > Going into a little more detail on this, I picture a number of scenarios: > > 1) A resource is fully internal to a module, and there's no reason that > any external code ought to need or get access to this data. The module > could restrict access to the resource by caller as inferred above. > > 2) A resource is an internal part of the module, but other things might > need access to the resource for specific purposes (like direct .class > access for instrumentation). The access restriction of this use case is > "solved" by existing containers today by isolating class loaders, so the > user has to have access to the target class loader in order to read the > resource(s) in question. You could subdivide this into (2a) the .class > files themselves and (2b) anything else that might fall under this > category, as I believe the use cases for (2a) are fairly specific to > run-time analysis and instrumentation tooling and similar uses. > > 3) A resource is a part of the module's self-configuration, but is not > of interest to importers. This would include configuration files (like > Java EE deployment descriptors, and other framework-XML-ish things). > Access is typically presently solved similarly to #2 above, or by simply > accessing the target resource archive directly, bypassing class loading. > > 4) A resource is a part of a public API intended to be imported and used > by consumers. Examples include framework configurations like logging > and some other framework-XML-ish things, ServiceLoader, etc. Today this > is solved as an unavoidable consequence of flat classpaths, but also (in > containers) by implicitly or explicitly exporting and importing some or > all of a class loader's resource file/directory structure. I think cases (2) and (3) are supported by the requirement I suggested in my previous reply, which allows suitably-privileged code to read class and resource files from a module's defining artifact. As to (4), I wonder if it's really all that different from (3). If a logging framework, e.g., is invoked by some module and then needs to read that module's logging-configuration file then can't it just read it out of the module's artifact? (I suspect it's reasonable to assume that a logging framework will have the permissions required to do so.) > While JBoss Modules does in fact solve cases 2-4 well enough, there is > no provision for things falling under (1). This is not however due to > any fundamental ideal; quite the opposite in fact. I simply could not > work out a consistent (and simple) set of rules for treating classes and > packages separately from resources and their directories that didn't > simply break everything*. > > The questions are: Are there yet more use cases that I have failed to > identify? Is it possible to come up with a consistent set of rules > which allows all these primary use cases to continue to function cleanly > going forward? Can we do so in a minimally-impacting manner that would > allow most existing code to continue to function with (at most) only > packaging changes? If not, which use cases can or should be sacrificed, > and why? If you can read class files and resources from a module's artifact then existing components that publish information in that way can continue to do so. Components that read such information will require some minor changes, since the ClassLoader::getResource* methods might not work for a module in the way that it does for the class path, or might work slowly, but this change should be fairly simple, e.g., going from object.getClass().getClassLoader().getResourceAsStream("foo") to something like object.getClass().getModule().getModuleArtifact().getStream("foo") In return for mandating (or strongly suggesting) this minor change in resource consumers we get resource encapsulation, and we can also avoid having a complicated "exports resources" feature. That seems like a worthwhile tradeoff. > I could easily imagine adding a separate public resources concept which > is wholly separate from the module internal content, where importers > could import some or all of these resources into their own world, which > would easily solve (4), but (2) and (3) are still left "high and dry". > An "unexported resources" area of a module could address (3), leaving > (2), which could be solved by co-locating those resources with the > classes themselves. But by this point there is some additional > complexity (maybe not *too* much, but definitely some). Yes, this is getting rather hairy ... > Module imports > would specify the module they want to import (possibly to package > granularity), as well as the resource directories they want to import > (which could be re-exported just like module packages could). So from > that perspective the additional complexity is manageable. But when it > comes to build and packaging of a module itself, users would now have > face possibly fairly complex decision regarding which category a > resource belongs to (not to mention that packaging tooling would have to > support this decision), I agree that this seems like too much. > whereas in today's world they just "throw it in > there" and hope for the best. Well, hopefully we can do better than that! - Mark From mark.reinhold at oracle.com Mon Mar 9 21:01:18 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 9 Mar 2015 14:01:18 -0700 (PDT) Subject: Security (was: Re: Module-system requirements) In-Reply-To: References: <20150211103831.528761@eggemoggin.niobe.net>, <20150215172507.827110@eggemoggin.niobe.net>, Message-ID: <20150309210118.032A1511FA@eggemoggin.niobe.net> 2015/2/16 9:04 -0800, tim_ellison at uk.ibm.com: > Presently, JAR files fulfil a number of different roles, including being > one of the structured container types for resources. As modules are > introduced there is an opportunity to use this artifact for some security > concepts that are container-based. For example, while it may be implied > by the 'Non-interference' requirement, it may be desirable to consider > modules to be sealed, thereby guaranteeing that all classes in a package > come from the same module. I think that is implied by the non-interference requirement, but just to make it clear: - _Non-interference_ --- The Java compiler, virtual machine, and run-time system must ensure that modules that contain packages of the same name do not interfere with each other. If two distinct modules contain packages of the same name then, from the perspective of each module, all of the types and members in that package are defined only by that module. Code in that package in one module must not be able to access package-private types or members in that package in the other module. > I was going to suggest that modules be considered a CodeSource so that > they can be signed, or form a protection domain with a configurable > security policy, without consideration for their (file system) container - > but I may have talked myself out of it . (I'll assume that by "container" here you mean "artifact" rather than, say, a "container-style" application such as an IDE or an app server.) > Perhaps these concepts should > remain with the container, notwithstanding the fact that a linked custom > binary run-time image may well collapse the containers and invalidate any > signing/policies associated with the originals. Yes, I suspect these concepts should remain container/artifact-specific, except possibly for signing, but I think we can leave this as a design decision to be worked out later. > A security algorithm implementation provider may always have to be part of > a separate verifiable container to maintain the current level of > integrity. Yes. - Mark From mark.reinhold at oracle.com Mon Mar 9 21:02:18 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 9 Mar 2015 14:02:18 -0700 (PDT) Subject: Security In-Reply-To: <54E7AB98.3030005@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, , <54E7AB98.3030005@redhat.com> Message-ID: <20150309210218.0706B51211@eggemoggin.niobe.net> 2015/2/20 1:48 -0800, david.lloyd at redhat.com: > On 02/16/2015 11:04 AM, Tim Ellison wrote: >> ... >> >> I was going to suggest that modules be considered a CodeSource so that >> they can be signed, or form a protection domain with a configurable >> security policy, without consideration for their (file system) container - >> but I may have talked myself out of it . Perhaps these concepts should >> remain with the container, notwithstanding the fact that a linked custom >> binary run-time image may well collapse the containers and invalidate any >> signing/policies associated with the originals. > > In JBoss Modules, we did this very thing, with a simple per-module > protection domain concept where the permissions are actually established > within the module description itself. ... > > I think there is definitely value in the module knowing what permissions > it needs to function, and to be shipped with those permissions. I think > that if this is combined with a configuration-specific verification > mechanism, this could allow users to express a level of trust the way > they do today for signed JARs, and/or perhaps be able to verify (at > install time) whether or not they want to go ahead with installing a > module with certain permissions. I can imagine building something like this, but would anyone use it? I've seen little evidence over the years of broad use of the fine-grained security model introduced way back in JDK 1.2. Do we really need to complicate the module system with permission declarations? - Mark From mark.reinhold at oracle.com Mon Mar 9 21:03:18 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 9 Mar 2015 14:03:18 -0700 (PDT) Subject: Security In-Reply-To: <54E7B3C8.30709@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, , <54E7B3C8.30709@redhat.com> Message-ID: <20150309210318.0AA3A51213@eggemoggin.niobe.net> 2015/2/20 2:23 -0800, david.lloyd at redhat.com: > ... > > Relatedly, the Encapsulation point is somewhat weaker than some of its > elder cousins in the old Jigsaw draft requirements, in that it doesn't > come out and say that the language needs a per-module class/member > access level. I'm not sure if it's just something that we need to visit > once we start talking about implementation, or if the concept was > abandoned in favor of visibility-style controls as the requirements > presently seem to imply, by my reading anyway. The Encapsulation requirement, as stated, is very much about access control rather than visibility: - _Encapsulation_ --- The access-control mechanisms of the Java language and virtual machine must prevent code from accessing classes and interfaces in packages that are not exported by their containing modules, or in packages whose containing modules are not required by the module containing the code. These mechanisms must be operative even when a security manager is not present. - Mark From mark.reinhold at oracle.com Mon Mar 9 21:04:18 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 9 Mar 2015 14:04:18 -0700 (PDT) Subject: Module, Class, and Resource Iteration (Was: Re: Module-system requirements) In-Reply-To: <54EE1C1D.3020400@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, <54EE1C1D.3020400@redhat.com> Message-ID: <20150309210418.0DD6D51215@eggemoggin.niobe.net> 2015/2/25 11:01 -0800, david.lloyd at redhat.com: > One thing that is missing from the requirements is the ability to > iterate or enumerate modules, classes, and/or resources. This is one > capability that people seem to be assuming will be magically provided by > a module solution. > > While it seems to me that resource or resource directory iteration is > (modulo the discussion on the "Exporting things" sub-thread) easily > accomplished, defining the meaning of module iteration and class > iteration is a little bit more involved. Yes ... > To iterate modules implies that the configuration has a complete picture > of all the modules it can load, and that it can iterate over this > picture in a relatively inexpensive manner. This is probably > reasonable, though in my imagination I see that the set of available > modules may change over time within the life of a single VM process. It should be straightforward to iterate through the modules already present in a configuration. Iterating through the modules available when creating a new configuration, or extending an existing one, should be possible but might not be very efficient since it will most likely require walking the filesystem. Do we need a requirement to capture this? It seems pretty low-level. > Iterating classes is a stranger expectation though, because while it is > a reasonable inclination to want to try this, it is possibly actively > harmful to start loading previously-unloaded classes. So such a > function would either need to restrict its output set to loaded classes, > or do something else like just enumerate class names. In addition, > there are security considerations to take into account; a caller should > only "see" classes that are accessible to it (though things like > frameworks may need an elevated capability in order to introspect > non-public classes). It wouldn't be hard to iterate through the class files in a module artifact. Iterating through the set of classes already loaded for a module (or by class loader) is possible but trickier. JVM TI already supports this [1]; does it need to be possible in a direct reflective API? Are there use cases which require that? > Another potentially iterable "thing" is packages, either in the form of > names or Package objects. No comment on the applicability or lack > thereof of such a capability though. Again, are there use cases for this kind of query? - Mark [1] http://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html#class From mark.reinhold at oracle.com Mon Mar 9 21:05:18 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 9 Mar 2015 14:05:18 -0700 (PDT) Subject: Module, Class, and Resource Iteration (Was: Re: Module-system requirements) In-Reply-To: References: <20150211103831.528761@eggemoggin.niobe.net>, <54EE1C1D.3020400@redhat.com>, Message-ID: <20150309210518.113C551217@eggemoggin.niobe.net> 2015/2/26 1:11 -0800, Tim Boudreau : > Should a module be able to iterate modules it does not depend on (or not in > the transitive closure of its dependencies) and could not load? > > Or is "framework" or bootstrap code logically a module which depends on all > other modules in the application? > > I know that's a very abstract sounding question, but it's cleaner to have a > world where it's modules all the way down, than a special "framework code > over here" case. So if an "application" is effectively a module that > depends, directly or indirectly, on everything else that makes up the > application, the module abstraction is far less leaky. It's true that an application is a module that depends, either directly or indirectly, upon all the other modules that make up the application. I don't see how it makes sense, however, to conceive of "framework" or bootstrap code as depending upon all the other modules in an application, and to limit that code's ability to manipulate modules. Such code will have to interact with the module system, and maybe even inspect module artifacts and do some reflection, in order to set things up on behalf of the application itself. In what way is that a problem? Or am I misunderstanding something? >> Iterating classes is a stranger expectation though, because while it is a >> reasonable inclination to want to try this, it is possibly actively harmful >> to start loading prevriously-unloaded classes. > > I think part of the goal is to make exploits like [1] impossible (using a > MethodHandle to call into JMX to get it to load an untrusted byte array as > bytecode) by making classes in modules you can't "see" via the closure of > your dependencies *really* not visible from call sites that can't see > them. The goal is even stronger: To make such classes inaccessible, not merely invisible. > ... > > For the "iterate all classes" thing mentioned earlier, that is typically > used as a nasty hack to accomplish service discovery that can be done with > metadata, and doesn't seem a use case worth encouraging. Certain things > like profilers and agents might need that, however. Yes, and they can already use JVM TI to get that information. - Mark From mark.reinhold at oracle.com Mon Mar 9 21:06:18 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 9 Mar 2015 14:06:18 -0700 (PDT) Subject: Enhanced ServiceLoader? In-Reply-To: References: , <20150215172631.519378@eggemoggin.niobe.net>, Message-ID: <20150309210618.1C82051227@eggemoggin.niobe.net> 2015/2/15 10:32 -0800, Tim Boudreau : > mark.reinhold at oracle.com: >> Well, at the very least it means, "enhanced so as to integrate it >> with the module system so that service bindings can be based upon >> the available modules rather than (or in addition to) whatever >> META-INF/services files happen to be found on the class path." >> >> The ordering of service providers is a tricky topic. >> >> I agree with Peter when he says that explicit ordering requires a global >> view of the run-time system. That's not possible from the perspective of >> a single module, which is why an @Ordered annotation makes no sense, but >> sometimes you actually do have a global (or sufficiently-global) view. > > Happy to hear that. I agree, in principle, in a modular system, the pieces > don't know about each other and so ordering seems at odds with the > concept. But reality is rarely so pure ... > > ... > > NetBeans did the ordering hints for years by enhancing META-INF/services > files with magic comments, e.g. > > #position 1000 > com.foo.Bar > > It would be good if ServiceLoader could grow enough heft to handle that > role - not necessarily identically - but the feature satisfies a proven > need. How is this any different than an @Ordered annotation? Doesn't it still assume that the author of a module somehow has global knowledge of how that module will be used in every possible scenario? - Mark From mark.reinhold at oracle.com Mon Mar 9 21:07:18 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 9 Mar 2015 14:07:18 -0700 (PDT) Subject: Summary of new and revised requirements Message-ID: <20150309210718.1FC2A5122D@eggemoggin.niobe.net> For convenient reference, here are all of the new and revised requirements suggested, so far. Fundamentals ------------ - _Exports_ --- A module's definition must be able to indicate which of its packages are _exported_ for use by other modules, and which are not. ADD: Multiple modules may export packages of the same name. - _Non-interference_ --- The Java compiler, virtual machine, and run-time system must ensure that modules that contain packages of the same name do not interfere with each other. ADD: If two distinct modules contain packages of the same name then, from the perspective of each module, all of the types and members in that package are defined only by that module. Code in that package in one module must not be able to access package-private types or members in that package in the other module. - _Selective binding (NEW)_ --- It must be possible to control the binding process so that specific services are provided only by specific providers, and in a specific order. Development ----------- - _Annotation digests (NEW)_ --- When packaging a module into an artifact it must be possible to augment its definition with a summary of all of the type- and package-level annotations that are present in the module, together with an indication of the types and packages to which each annotation applies. This digest must be easily readable from module artifacts, and at run time those parts of it related to annotations retained for run time must be available through an appropriate reflective API. - _Access to class files and resources (NEW)_ --- If a module is defined by an artifact that contains class files and resources then it must be possible, once that module is added to a configuration, for suitably-privileged code to access those class files and resources. Dynamic Configuration --------------------- - _Control of class loaders (NEW)_ --- An application that loads a module into a dynamic configuration must be able to specify the class loader to be used to load that module. That loader need not be the same as the loader used to load any other module in the configuration. - _Isolated dynamic configurations_ --- An application must be able to isolate the code in different dynamic configurations at least as well as is possible today, where this is typically done by using multiple class loaders. ADD: An application must be able to ensure that code in a dynamic configuration does not modify other configurations. - Mark From mark.reinhold at oracle.com Mon Mar 9 21:08:18 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 9 Mar 2015 14:08:18 -0700 (PDT) Subject: Summary of new and revised requirements Message-ID: <20150309210818.235505122F@eggemoggin.niobe.net> Due to circumstances beyond my control I was offline for much of the past two weeks, which is why it's taken me until today to catch up with the requirements discussion. I'll post a second draft of the document shortly, and if possible I'd like to finalize it by the end of next week (3/20). - Mark From mark.reinhold at oracle.com Mon Mar 9 21:11:07 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 9 Mar 2015 14:11:07 -0700 (PDT) Subject: Finishing up the requirements discussion Message-ID: <20150309211107.BBFEA51239@eggemoggin.niobe.net> (resending with proper subject line) Due to circumstances beyond my control I was offline for much of the past two weeks, which is why it's taken me until today to catch up with the requirements discussion. I'll post a second draft of the document shortly, and if possible I'd like to finalize it by the end of next week (3/20). - Mark From niftiness at gmail.com Tue Mar 10 02:51:52 2015 From: niftiness at gmail.com (Tim Boudreau) Date: Mon, 9 Mar 2015 22:51:52 -0400 Subject: Security In-Reply-To: <20150309210218.0706B51211@eggemoggin.niobe.net> References: <20150211103831.528761@eggemoggin.niobe.net> <54E7AB98.3030005@redhat.com> <20150309210218.0706B51211@eggemoggin.niobe.net> Message-ID: > > > I think there is definitely value in the module knowing what permissions > > it needs to function, and to be shipped with those permissions. I think > > that if this is combined with a configuration-specific verification > > mechanism, this could allow users to express a level of trust the way > > they do today for signed JARs, and/or perhaps be able to verify (at > > install time) whether or not they want to go ahead with installing a > > module with certain permissions. > > I can imagine building something like this, but would anyone use it? > Umm... no. David, could you give a concrete example of what you're looking for? Permissions like creating classloaders or accessing files or network - i.e. security-manager applet-style stuff, or something else? I don't think it works to discuss permissions in the abstract without an explicit use case or two to understand what you're after and why. > I've seen little evidence over the years of broad use of the fine-grained > security model introduced way back in JDK 1.2. Do we really need to > complicate the module system with permission declarations? > My instinct would be to make sure a module can provide metadata beyond just what's in the spec, as you can with module manifests, so if someone wants that they can implement it and do a tree of per-module SecurityManagers or whatever; if it proves of broad value, it'd be a fine subject for its own subsequent JSR. Happy to be convinced otherwise. -Tim http://timboudreau.com From niftiness at gmail.com Tue Mar 10 03:23:32 2015 From: niftiness at gmail.com (Tim Boudreau) Date: Mon, 9 Mar 2015 23:23:32 -0400 Subject: Enhanced ServiceLoader? In-Reply-To: <20150309210618.1C82051227@eggemoggin.niobe.net> References: <20150215172631.519378@eggemoggin.niobe.net> <20150309210618.1C82051227@eggemoggin.niobe.net> Message-ID: > > > NetBeans did the ordering hints for years by enhancing META-INF/services > > files with magic comments, e.g. > > > > #position 1000 > > com.foo.Bar > > > > It would be good if ServiceLoader could grow enough heft to handle that > > role - not necessarily identically - but the feature satisfies a proven > > need. > > How is this any different than an @Ordered annotation? It isn't, other than that as a performance optimization, you don't have to load the class to determine its order. > Doesn't it still > assume that the author of a module somehow has global knowledge of how > that module will be used in every possible scenario? > It assumes the author of a module has at least *partial* knowledge of how that module will be used in its *intended* scenario: /** * Foos are ordered from Integer.MIN_VALUE to Integer.MAX_VALUE. More * expensive-to-run Foos should be placed close to last. */ public class Foo { ... } @ServiceProvider(service=Foo.class, position=Integer.MIN_VALUE) public class CheapFoo { ... } @ServiceProvider(service=Foo.class, position=0) public class MediumFoo { ... } @ServiceProvider(service=Foo.class, position=Integer.MAX_VALUE) public class ReallyExpensiveFoo { ... } An application is free to use these in some alternate order if its scenario is in fact not the intended scenario. Since the API author has defined the bounds and the meaning of the low and high bounds, the API implementer does not need global knowledge, only knowledge about one or more other implementations. While it is possible for that knowledge to be wrong - there might be an UberExpensiveFoo that should definitely come after ReallyExpensiveFoo - most of the time people are not operating with a total lack of knowledge of the environment their code will run in, so having an order provides quite a bit of value. If there is really something that does have a global view, it is free to ignore the hinted order. I've seen code be accidentally dependent on listener order (this is, in fact, why you can't change Swing Look and Feels without a restart in NetBeans - it reorders look and feel listeners to be called after application listeners, and finding all the sites where that assumption was accidentally made and preventing new ones is a near impossibility). I've seen it happen with service registration order too. In general, if you're going to have an API that purports to provide an "unordered" collection, sooner or later someone is going to inadvertently depend on the order they encounter at runtime - and service registrations are definitely things where that's likely. So in the interest of people writing less-likely-to-fail code, having a way to make that order explicit is a good thing. -Tim -- http://timboudreau.com From stef at epardaud.fr Tue Mar 10 10:59:30 2015 From: stef at epardaud.fr (Stephane Epardaud) Date: Tue, 10 Mar 2015 11:59:30 +0100 Subject: Module, Class, and Resource Iteration (Was: Re: Module-system requirements) In-Reply-To: <20150309210418.0DD6D51215@eggemoggin.niobe.net> References: <20150211103831.528761@eggemoggin.niobe.net>, <54EE1C1D.3020400@redhat.com> <20150309210418.0DD6D51215@eggemoggin.niobe.net> Message-ID: <54FECE92.7000600@epardaud.fr> On 09/03/15 22:04, mark.reinhold at oracle.com wrote: > It wouldn't be hard to iterate through the class files in a module > artifact. Iterating through the set of classes already loaded for a > module (or by class loader) is possible but trickier. JVM TI already > supports this [1]; does it need to be possible in a direct reflective > API? Are there use cases which require that? Yes for auto-discovery if not using annotations (I see you proposed adding annotation->class mapping to the artifacts earlier, so that would solve discovery using annotations), such as inheritance: "give me all types that inherit this interface". >> Another potentially iterable "thing" is packages, either in the form of >> names or Package objects. No comment on the applicability or lack >> thereof of such a capability though. > Again, are there use cases for this kind of query? > Yes: iterating classes is not enough: packages also hold useful information and annotations we may want to scan. Plus, it feels more logical that you iterate classes by iterating packages within a module, encapsulation-wise. From david.lloyd at redhat.com Tue Mar 10 13:37:52 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 10 Mar 2015 08:37:52 -0500 Subject: Security In-Reply-To: <20150309210218.0706B51211@eggemoggin.niobe.net> References: <20150211103831.528761@eggemoggin.niobe.net>, , <54E7AB98.3030005@redhat.com> <20150309210218.0706B51211@eggemoggin.niobe.net> Message-ID: <54FEF3B0.4060903@redhat.com> On 3/9/15 4:02 PM, mark.reinhold at oracle.com wrote: > 2015/2/20 1:48 -0800, david.lloyd at redhat.com: >> On 02/16/2015 11:04 AM, Tim Ellison wrote: >>> ... >>> >>> I was going to suggest that modules be considered a CodeSource so that >>> they can be signed, or form a protection domain with a configurable >>> security policy, without consideration for their (file system) container - >>> but I may have talked myself out of it . Perhaps these concepts should >>> remain with the container, notwithstanding the fact that a linked custom >>> binary run-time image may well collapse the containers and invalidate any >>> signing/policies associated with the originals. >> >> In JBoss Modules, we did this very thing, with a simple per-module >> protection domain concept where the permissions are actually established >> within the module description itself. ... >> >> I think there is definitely value in the module knowing what permissions >> it needs to function, and to be shipped with those permissions. I think >> that if this is combined with a configuration-specific verification >> mechanism, this could allow users to express a level of trust the way >> they do today for signed JARs, and/or perhaps be able to verify (at >> install time) whether or not they want to go ahead with installing a >> module with certain permissions. > > I can imagine building something like this, but would anyone use it? We've seen a reasonably significant usage of security managers and their associated permissions, especially now that it is a part of Java EE, so the functionality would at least need to be present to bring Java EE forward into this world. The fact is that many environments (corporate, educational, government) require the use of a security manager for their Java based applications; it is my belief that if we each reached out to our associated support organizations, we'd find that this is generally true. Without a convenient way to establish permissions per module, it's really a step backwards in this area, and (at least at present) it is so simple to implement that there doesn't seem to be a compelling argument *not* to do it. And I suspect that whether or not we provide a mechanism for distributors to assign permissions to modules, there still needs to be some kind of trust decision by the installer to decide whether (s)he wants to install the module in question based on where it comes from. > I've seen little evidence over the years of broad use of the fine-grained > security model introduced way back in JDK 1.2. Do we really need to > complicate the module system with permission declarations? I think we really should. -- - DML From david.lloyd at redhat.com Tue Mar 10 13:50:10 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 10 Mar 2015 08:50:10 -0500 Subject: Security In-Reply-To: References: <20150211103831.528761@eggemoggin.niobe.net> <54E7AB98.3030005@redhat.com> <20150309210218.0706B51211@eggemoggin.niobe.net> Message-ID: <54FEF692.5050008@redhat.com> On 3/9/15 9:51 PM, Tim Boudreau wrote: > > I think there is definitely value in the module knowing what permissions > > it needs to function, and to be shipped with those permissions. I think > > that if this is combined with a configuration-specific verification > > mechanism, this could allow users to express a level of trust the way > > they do today for signed JARs, and/or perhaps be able to verify (at > > install time) whether or not they want to go ahead with installing a > > module with certain permissions. > > I can imagine building something like this, but would anyone use it? > > > Umm... no. > > David, could you give a concrete example of what you're looking for? > Permissions like creating classloaders or accessing files or network - > i.e. security-manager applet-style stuff, or something else? I'm talking about the J2SE security manager architecture and java.security.Permission and its subclasses. > I don't think it works to discuss permissions in the abstract without an > explicit use case or two to understand what you're after and why. In addition to being able to support Java EE, I think that the use cases overlap heavily with those of, say, javaws applications. I don't think we can really discuss a module distribution mechanism without considering the security aspects that informed previous distribution mechanisms. We have users who have used security managers not just for untrusted code, but also as a tripwire for trusted code for bug prevention as well as exploitation detection. I'm not sure I'm really the right person to justify the existence of the security manager, but the fact is that people do think they need it, and they do use it, including Java EE permissions.xml files as well as our own per-module permission declaration system. > I've seen little evidence over the years of broad use of the > fine-grained > security model introduced way back in JDK 1.2. Do we really need to > complicate the module system with permission declarations? > > > My instinct would be to make sure a module can provide metadata beyond > just what's in the spec, as you can with module manifests, so if someone > wants that they can implement it and do a tree of per-module > SecurityManagers or whatever; if it proves of broad value, it'd be a > fine subject for its own subsequent JSR. Happy to be convinced otherwise. I don't think it really needs to be more complicated than using the existing ProtectionDomain mechanism to assign permissions to each module/code source/whatever unit of granularity makes the most sense. -- - DML From Tim_Ellison at uk.ibm.com Tue Mar 10 14:55:40 2015 From: Tim_Ellison at uk.ibm.com (Tim Ellison) Date: Tue, 10 Mar 2015 14:55:40 +0000 Subject: Security In-Reply-To: <54FEF3B0.4060903@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, , <54E7AB98.3030005@redhat.com> <20150309210218.0706B51211@eggemoggin.niobe.net> <54FEF3B0.4060903@redhat.com> Message-ID: "jpms-spec-experts" wrote on 10/03/2015 13:37:52: > On 3/9/15 4:02 PM, mark.reinhold at oracle.com wrote: > > 2015/2/20 1:48 -0800, david.lloyd at redhat.com: > >> On 02/16/2015 11:04 AM, Tim Ellison wrote: > >>> ... > >>> > >>> I was going to suggest that modules be considered a CodeSource so that > >>> they can be signed, or form a protection domain with a configurable > >>> security policy, without consideration for their (file system) container - > >>> but I may have talked myself out of it . Perhaps these concepts should > >>> remain with the container, notwithstanding the fact that a linked custom > >>> binary run-time image may well collapse the containers and invalidate any > >>> signing/policies associated with the originals. > >> > >> In JBoss Modules, we did this very thing, with a simple per-module > >> protection domain concept where the permissions are actually established > >> within the module description itself. ... > >> > >> I think there is definitely value in the module knowing what permissions > >> it needs to function, and to be shipped with those permissions. I think > >> that if this is combined with a configuration-specific verification > >> mechanism, this could allow users to express a level of trust the way > >> they do today for signed JARs, and/or perhaps be able to verify (at > >> install time) whether or not they want to go ahead with installing a > >> module with certain permissions. > > > > I can imagine building something like this, but would anyone use it? > > We've seen a reasonably significant usage of security managers and their > associated permissions, especially now that it is a part of Java EE, so > the functionality would at least need to be present to bring Java EE > forward into this world. > > The fact is that many environments (corporate, educational, government) > require the use of a security manager for their Java based applications; > it is my belief that if we each reached out to our associated support > organizations, we'd find that this is generally true. > > Without a convenient way to establish permissions per module, it's > really a step backwards in this area, and (at least at present) it is so > simple to implement that there doesn't seem to be a compelling argument > *not* to do it. > > And I suspect that whether or not we provide a mechanism for > distributors to assign permissions to modules, there still needs to be > some kind of trust decision by the installer to decide whether (s)he > wants to install the module in question based on where it comes from. David, is it not sufficient that the java.policy file can contain a code source url specifying jrt:/$MODULE ? This gives a mechanism for permissions to be granted to code associated with a module irrespective of the artifact that delivers the module. I don't see any mention of the ability to sign a module, but provided there is an extensible metadata space (equivalent to the META-INF directory) associated with a module, then I think we are good for that too, no? > > I've seen little evidence over the years of broad use of the fine-grained > > security model introduced way back in JDK 1.2. Do we really need to > > complicate the module system with permission declarations? > > I think we really should. Should complicate it ;-/ !? I agree that a mechanism for continuing the 'somewhat less than optimal' security model is better than ignoring it. Better yet to use this as an opportunity to improve the code security model, though I see that as beyond the scope of the modularity JSR. Regards, Tim Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU From david.lloyd at redhat.com Tue Mar 10 15:08:39 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 10 Mar 2015 10:08:39 -0500 Subject: Exporting things (Was: Re: Module-system requirements) In-Reply-To: <20150309205917.F049C511F6@eggemoggin.niobe.net> References: <20150211103831.528761@eggemoggin.niobe.net>, <20150215172405.90174@eggemoggin.niobe.net>, <54E1ED8C.8010208@redhat.com> <20150309205917.F049C511F6@eggemoggin.niobe.net> Message-ID: <54FF08F7.8050901@redhat.com> On 3/9/15 3:59 PM, mark.reinhold at oracle.com wrote: > 2015/2/16 5:15 -0800, david.lloyd at redhat.com: >> On 02/15/2015 07:24 PM, mark.reinhold at oracle.com wrote: >>> 2015/2/11 11:27 -0800, david.lloyd at redhat.com: >>>> I notice that under Exports, it is defined that only packages (i.e. >>>> classes within those packages) are considered to be exportable, and not >>>> resources. I'm not certain how I feel about that... >>>> >>>> ... To sum up though, I'd be glad to see (from my perspective) this >>>> behavior go away in favor of the simpler model of restricting exports to >>>> packages (and isolating classes and resources), >>> >>> I tend to agree. >>> >>> Exposing all of a module's resources, as JAR files do today, is wrong >>> from the perspective of encapsulation. ... >>> >>> This suggests an additional requirement: >>> >>> - _Resource encapsulation_ --- The run-time system must ensure that the >>> static resource files within a module are directly accessible only by >>> code within that module. >> >> What you are saying here implies (to me) inspecting the call stack to >> ensure that the caller actually exists within the module being >> referenced. > > Yes. > >> I'm not certain we really want to go this far, if my >> interpretation is accurate. Such a rule is reasonable on the face of >> it, but it does more or less break all the code in existence which >> currently uses resources as a sort of API into a library, and this >> includes very common patterns for loading configuration (among many >> other things). > > True. To what degree should it be a goal to preserve such mechanisms? I really feel that not having this compatibility will strongly diminish adoption of this system, since a substantial body of existing work is unlikely to function properly as modules. > Must Java modules be forever constrained to look like JAR files? No, but I think that isn't really related to the point at hand, which is more related to the solution of specific problems rather than change for its own sake. If solving those problems means that modules have visible resources, does that mean we've tripped an invisible failure criteria by resembling JARs too closely in some specific way? Or put a nicer way, is there a specific problem or set of problems that are solved by strictly encapsulating resources, or is it more of a way to try a new general approach? >>>> but only if we can be >>>> reasonably sure that the use cases solved thereby are somehow sorted >>>> out, which include: >>>> >>>> ??? Supporting this case (or determining that it is not relevant): A >>>> module privately uses services provided by one or more specific peer >>>> module(s) (via ServiceLoader), the arrangement of which is determined >>>> statically by distribution. >>> >>> One way to accomplish this is to make the service interface accessible >>> only by the relevant modules, via qualified exports. Then only those >>> modules can provide or use implementations of the service. >>> >>> Another possibility is to add a requirement that the binding of providers >>> to service interfaces can be controlled manually during the configuration >>> process. Is that more along the lines of what you mean when you say >>> "determined statically by the distribution"? >> >> Yes it *could* be, assuming that configurations end up being >> sufficiently flexible (e.g. I have one distribution configuration, and >> then I have N application configurations for each installed application, >> which is in turn an extension of the distribution configuration). > > Yes, that's exactly the intent of nested configurations. > > To ensure that a distributor of a set of modules can control how service > dependences are resolved amongst those modules here's a new requirement, > for the Fundamentals section: > > - _Selective binding_ --- It must be possible to control the binding > process so that specific services are provided only by specific > providers, and in a specific order. > > Does that make sense? In this requirement, what party is doing the controlling? >> It's a question, I guess, of how the configurations work and how much >> power the application configuration has to customize/mess up the >> distribution configuration. > > The assumption is that a nested configuration, of itself, cannot change > the configuration(s) in which it's nested. That's worth making explicit > in the existing requirement for > > - _Isolated dynamic configurations_ --- An application must be able to > isolate the code in different dynamic configurations at least as well > as is possible today, where this is typically done by using multiple > class loaders. ADD: An application must be able to ensure that code > in a dynamic configuration does not modify other configurations. > >> In a setup like we have, where one module >> can just statically import another's services, it's pretty simple for >> the distributor to set it up and really difficult for an application to >> mess it up. > > So a JBoss module can specify that it depends upon service providers from > one or more specific modules rather than whichever modules happen to > provide implementations of that service? That seems contrary to the > nature of services, if I understand you correctly. Right; this is however functionally similar to having the configuration determine the service bindings without any sub-configurations being able to override it, though at a whole-module granularity rather than a per-module-service granularity. I did consider adding an additional global registry for service providers (because use cases are fairly easy to imagine), however as of yet the feature has never been requested, nor *actual* use cases raised within our domain that might beg for such a feature, so it has always slipped down the list. But functionally I expect that the cases where a module would statically connect to provider implementation(s) would generally not overlap with the ability to do so at the configuration level. In the configuration-level case though, establishment and predictability of order (at least on a per distribution basis) is almost certainly going to be needed I think, so I'd still maintain that it should be worked into the requirements somehow. >>>> ??? Having (generally) the same visibility to .class files as to the >>>> classes themselves, from a given resource-loading entry point (Class or >>>> ClassLoader), i.e. if I have visibility to >>>> classLoader.loadClass("SomeClass"), I also have visibility to >>>> classLoader.getResource("SomeClass.class"), and vice-versa (any security >>>> restrictions aside). >>> >>> Can you describe some use cases that require this property? >> >> Just doing a quick spin through libraries used or referenced by WildFly >> and its dependencies: Libraries such as CGLIB and BCEL use this to >> modify existing classes. Ant apparently uses it to find JavaCC, though >> this odd use is probably better done another way. Hibernate uses it to >> get bytes to instrument in its instrumenting class loader, as does our >> JPA SPI. We also use this trick to generate various fast indexes of >> classes without actually loading the classes themselves for various >> tools and deployment operations. I believe we have some tooling using >> this as well to do various types of analysis on loaded code. > > When an application is assembled by putting JAR files on the class path > then yes, you definitely need CL::getResource("SomeClass.class"), or > something like it, in order to do things like instrumentation. That's > because the class-path mechanism (mostly) hides the identities of the > original JAR files -- there's no way find the JAR file that will contain > a particular class file and then load that file yourself, rewriting its > bytecodes along the way. Well I think this goes beyond the JAR to the class loader itself, which can utilize arbitrary policies for locating classes. Getting the class contents in this way is the only method which is very likely to work across most or all class loaders. But I guess that's a trivial point. > When, in the future, an application is assembled by resolving a set of > actual modules into a configuration, then we won't necessarily have to > use class loaders to mediate resource access, so I'd rather not bake that > into the requirements. A configuration ought to be able to relate > classes, modules, and module artifacts such that, given a module and a > class name, you can read the corresponding class file (or any other kind > of resource) from the module's defining artifact and load it however you > want. > > To capture this as a requirement, in the Development section: > > - _Access to class files and resources_ --- If a module is defined by > an artifact that contains class files and resources then it must be > possible, once that module is added to a configuration, for > suitably-privileged code to access those class files and resources. I like this requirement; I think it hits a good percentage of use cases. > (The "if a module is defined by an artifact" precondition allows for > ahead-of-time-compilation scenarios.) > > This requirement also addresses the problem of how to locate and read EE > deployment descriptors and similar kinds of configuration files. Great. >>>> ??? A consistent ruling on the right behavior of general resources which >>>> are co-packaged with classes that use them (typically loaded via e.g. >>>> getClass().getResourceAsStream("MyResource.properties") or similar). >>> >>> If we take the view that resources are module-internal then the answer >>> is that code within a module can access that module's resources but code >>> outside of the module cannot (with the usual exception for reflection, >>> which would require a security check). >> >> This brings us close to a previous point of contention regarding having >> the implementation require a class loader for each module (or not); the >> problem never came up for us (even in security manager environments) >> because acquiring a class loader already entails a security check. > > As I wrote earlier, I don't think we can mandate one class loader per > module. I hope we can discuss this a bit further (without belaboring it unduly), because the functional overlap between modules and class loaders really rubs me the wrong way. But we can discuss further on the Interoperation sub-thread. >>>> ??? A consistent ruling on whether it makes sense generally to be able to >>>> export resources to dependents. >>> >>> If we take the view that resources are module-internal then the answer is >>> no, this does not make sense. A module can export packages of classes >>> and interfaces, and can publish service providers; it cannot also export >>> files in a string-based hierarchical namespace. >> >> I guess the salient underlying question is, "why not?". I'm being (for >> the most part) a devil's advocate here, but I think this question really >> has to be answered succinctly, because we're really trying to change a >> basic (de facto) principle of Java library interoperation (that they >> often use resources as a sort of API, even if only primarily for service >> loader-style and configuration use cases). >> >> I can easily imagine an "export resources" feature which specifies >> resource directories to make exportable (due in no small part to the >> fact that such a feature has been used extensively by us with good >> results). > > Yes, we could do this, but it doesn't seem necessary. Applications that > need to access a module's class files and resources can do so directly, > as described above. I don't see a need to complicate the module system > itself with a notion of exportable resources. I think a lot of existing applications expect to have a single class loader which "sees" all the resources and configuration it needs. However I'm thinking that what you're describing in the Interoperation sub-thread could provide a means to solve this problem without requiring that resource exporting be a part of the core module system, if class loader based systems can still manufacture class loaders which map to modules but offer additional "legacy" behavior. Let's continue there. > >> Another use case beyond those mentioned above that is >> satisfied by such a capability could be bundling supplementary i18n/l10n >> or time zone data or similar as a module. > > The approach we've taken in the JDK for things like time-zone and locale > data is to define appropriate services and providers thereof. That way > all the information goes through a proper statically-checked interface, > and the layout and format of the actual data can evolve without having to > update its consumers. That seems like a good approach. -- - DML From david.lloyd at redhat.com Tue Mar 10 15:40:28 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 10 Mar 2015 10:40:28 -0500 Subject: Security In-Reply-To: References: <20150211103831.528761@eggemoggin.niobe.net>, , <54E7AB98.3030005@redhat.com> <20150309210218.0706B51211@eggemoggin.niobe.net> <54FEF3B0.4060903@redhat.com> Message-ID: <54FF106C.5090901@redhat.com> On 3/10/15 9:55 AM, Tim Ellison wrote: > "jpms-spec-experts" wrote on > 10/03/2015 13:37:52: >> On 3/9/15 4:02 PM, mark.reinhold at oracle.com wrote: >>> 2015/2/20 1:48 -0800, david.lloyd at redhat.com: >>>> On 02/16/2015 11:04 AM, Tim Ellison wrote: >>>>> ... >>>>> >>>>> I was going to suggest that modules be considered a CodeSource so > that >>>>> they can be signed, or form a protection domain with a configurable >>>>> security policy, without consideration for their (file system) > container - >>>>> but I may have talked myself out of it . Perhaps these concepts > should >>>>> remain with the container, notwithstanding the fact that a linked > custom >>>>> binary run-time image may well collapse the containers and > invalidate any >>>>> signing/policies associated with the originals. >>>> >>>> In JBoss Modules, we did this very thing, with a simple per-module >>>> protection domain concept where the permissions are actually > established >>>> within the module description itself. ... >>>> >>>> I think there is definitely value in the module knowing what > permissions >>>> it needs to function, and to be shipped with those permissions. I > think >>>> that if this is combined with a configuration-specific verification >>>> mechanism, this could allow users to express a level of trust the way >>>> they do today for signed JARs, and/or perhaps be able to verify (at >>>> install time) whether or not they want to go ahead with installing a >>>> module with certain permissions. >>> >>> I can imagine building something like this, but would anyone use it? >> >> We've seen a reasonably significant usage of security managers and their > >> associated permissions, especially now that it is a part of Java EE, so >> the functionality would at least need to be present to bring Java EE >> forward into this world. >> >> The fact is that many environments (corporate, educational, government) >> require the use of a security manager for their Java based applications; > >> it is my belief that if we each reached out to our associated support >> organizations, we'd find that this is generally true. >> >> Without a convenient way to establish permissions per module, it's >> really a step backwards in this area, and (at least at present) it is so > >> simple to implement that there doesn't seem to be a compelling argument >> *not* to do it. >> >> And I suspect that whether or not we provide a mechanism for >> distributors to assign permissions to modules, there still needs to be >> some kind of trust decision by the installer to decide whether (s)he >> wants to install the module in question based on where it comes from. > > David, is it not sufficient that the java.policy file can contain a code > source url specifying jrt:/$MODULE ? This gives a mechanism for > permissions > to be granted to code associated with a module irrespective of the > artifact > that delivers the module. I believe not, for these reasons: ? The policy file is not AFAIK actually standardized in the SE platform, though (of course) the policy mechanism itself is. ? The "jrt" URL scheme, as I understand it, applies specifically to the JDK runtime and not generally to modules. ? The JDK runtime itself effectively has AllPermission as far as I understand. ? Assuming that each configuration (I guess?) gets a unique URL scheme which applies to it, using the global policy exclusively is still non-optimal because it loses a substantial amount of information: the author of the module knows what permissions the module would require, whereas the consumer of the module does not have this knowledge. The consumer would on the other hand want the ability to define a governing or maximal set of permissions, either globally, per-configuration, or per-module, which could well be done by policy (or not). Adding in some kind of interactive process for verifying installation of modules which would violate this maximal constraint falls more under the "gravy" category in my mind. ? Also I think I'll just keep repeating "Java EE 7" over and over again. :-) We likely need, at a minimum, the ability to programmatically define modules with specific permission sets to be able to support EE. > I don't see any mention of the ability to sign a module, but provided > there > is an extensible metadata space (equivalent to the META-INF directory) > associated with a module, then I think we are good for that too, no? I'm not sure. Signing is definitely a distribution/installation concern though, far more than a run time concern in my opinion. >>> I've seen little evidence over the years of broad use of the > fine-grained >>> security model introduced way back in JDK 1.2. Do we really need to >>> complicate the module system with permission declarations? >> >> I think we really should. > > Should complicate it ;-/ !? Yes, I deliberately disregarded the prejudicial language, but addressing it head-on, I firmly believe we are not talking about something that is substantially complex here, so "complicate" is probably far too strong a term. > I agree that a mechanism for continuing the > 'somewhat less than optimal' security model is better than ignoring it. > Better yet to use this as an opportunity to improve the code security > model, > though I see that as beyond the scope of the modularity JSR. I agree, I would love to see improvements in this area. I also agree it's probably out of scope, unless some surprisingly low-hanging fruit would appear. -- - DML From david.lloyd at redhat.com Wed Mar 11 13:14:35 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 11 Mar 2015 08:14:35 -0500 Subject: Interoperation (Was: Re: Module-system requirements) In-Reply-To: <20150309205417.DE06C511E7@eggemoggin.niobe.net> References: <20150211103831.528761@eggemoggin.niobe.net>, <20150215172424.772730@eggemoggin.niobe.net>, <54E1F75C.4070009@redhat.com> <20150309205417.DE06C511E7@eggemoggin.niobe.net> Message-ID: <55003FBB.5050705@redhat.com> On 03/09/2015 03:54 PM, mark.reinhold at oracle.com wrote: > 2015/2/16 5:57 -0800, david.lloyd at redhat.com: >> On 02/15/2015 07:24 PM, mark.reinhold at oracle.com wrote: >>> 2015/2/11 11:56 -0800, david.lloyd at redhat.com: >>>> I'm concerned that the interoperation point doesn't quite go far enough. >>>> It says "It must be possible for another module system, such as OSGi, >>>> to locate Java modules and resolve them using its own resolver, except >>>> possibly for core system modules." >>>> >>>> I agree with this requirement, such as it is, but I would also assert >>>> that it should be possible to retrofit not only OSGi but Java EE, Maven, >>>> and even legacy "modules" like extensions and so on as modules. The >>>> reason I make this assertion is that we have done all of these things >>>> with success and (I dare say) relative ease, even to the extent of >>>> achieving both EE and OSGi certification with such a retrofit. >>> >>> Can you explain what you mean, exactly, by "retrofit"? >> >> We have implemented both a certified OSGi implementation as well as a >> certified Java EE implementation which maps these specifications' module >> concepts onto our own concept of a module. ... >> >>> Are you saying that the JBoss module system can consume Java EE WAR >>> files, OSGi bundles, and Maven projects, as they stand and without >>> change? >> >> Yes*. This is because of implementation choices we've made - each >> module is a class loader, and we can carefully control which classes >> *and* resources are exported from one to another. That said, I do think >> it's still worth exploring a different approach than ours regarding >> conceptually combining resource directories and packages, which is what >> we do today and is not without disadvantage. >> >> ... >> >>>> ... >>>> >>>> To achieve this additional requirement does require a few very specific >>>> implementation considerations but I do not believe that any of these >>>> considerations contraindicate any of the other requirements stated here. >>> >>> Can you be more specific as to what those considerations are? Can they >>> be expressed as module-system requirements? >> >> In the case of both Java EE and OSGi, I think it's safe to say that you >> need to apply a strict 1:1 mapping between modules and class loaders for >> this to work. > > I don't think we can mandate this. To divide the SE platform into > modules in a compatible way requires the ability to load multiple modules > via the same class loader, so as to respect long-standing assumptions > about which platform classes are loaded by the built-in class loaders > (i.e., the bootstrap and extension loaders). In an early Jigsaw > prototype we tried the 1:1 module/loader mapping for platform modules > and found it unworkable from a compatibility perspective. I went through the jigsaw list archives looking for this discussion and didn't turn up anything; was this discussed on-list and I am just failing to find it? > Aside from the SE platform itself, I don't think we can force all > applications to adopt the unique-class-loader-per-module approach. > Oracle's WebLogic application server, e.g., doesn't do that today and > it'd be difficult, if not impossible, to modify it to do so without > breaking existing applications. I guess I can't really comment on what WebLogic does or does not do. I can say however that we were able to move from a flat classpath architecture to a modular architecture like this without breaking things due to the modularization. The Java EE spec is pretty clear about what needs to be visible to what, and by translating this information into module dependency information, we were able to make a smooth transition. I can't really imagine an application that would be broken unless they were relying on some specific non-spec behavior as the EE spec says: "[...] A Java EE application should not assume that all components in the application will be available on the class path of the application at run time. Each component might be loaded into a separate class loader with a separate namespace. [...]" Note that in the EE spec, the term "component" has a finer granularity than "module", i.e. there are 1..N components per module, whereas an "application" can contain 1 or more modules. At the same time though, most of the language around class loaders uses "may" so it's all pretty flexible in terms of implementation requirements. > I suggest the module system be as neutral as possible with respect to how > modules and class loaders are related. A complex application should be > able to spin up a class loader for every module it loads, if needed > (excepting non-upgradeable platform classes), but not all applications > should be forced to do so. > > New requirement, for the Dynamic Configuration section: > > - _Control of class loaders_ --- An application that loads a module > into a dynamic configuration must be able to specify the class loader > to be used to load that module. That loader need not be the same as > the loader used to load any other module in the configuration. This goes part of the way, for sure. I'm not 100% sold yet though. It's not easy to let go of something that (from our perspective) is tried and true. >> Also you need an API to allow run-time systems to create >> their own configurations (we call them "module loaders", akin in purpose >> and function to class loaders) and assemble modules from resources >> (JARs) and dependency specifications, and such APIs have to be able to >> relink modules or deregister modules to allow them to be GC'd (though >> the boot module loader in our case does not expose relink or unload >> functionality for obvious reasons). > > These needs are already captured, I think, by the existing requirements > related to dynamic configurations. > >> OSGi specifically requires that bundle class loaders implement an OSGi >> spec interface, implying that the module class loader class must be >> extensible (as I said a couple of years back though, we make most of our >> methods final on the module class loader to disallow any really bizarre >> behavior, and this works well in practice). > > Which OSGi interface are you thinking of here? In OSGi Core 6.0.0 section 3.9.9: "Class Loader - An OSGi framework must ensure that the class loader of a class that comes from a bundle implements the BundleReference interface." The BundleReference interface has a single getBundle() method on it. >> We also had to implement a >> pre- and/or post-class-define hook per module to allow some OSGi magic >> to happen. > > These are non-final protected methods in your module class loader? > > Could you forward a description of these methods to the list? Yeah, they look like this: protected void preDefine(ClassSpec classSpec, String className) { } protected void postDefine(ClassSpec classSpec, Class definedClass) { } ClassSpec contains the bytes and CodeSource of the class. The hooks don't have opportunity to do anything like rewrite the class bytes, but we also accept a java.lang.instrument.ClassFileTransformer instance in the module definition for that purpose (though we aren't using it at this point). Our OSGi implementation only ended up using one of these hooks (preDefine I think). They were added to support lazy bundle activation. >> Finally there is a per-module fallback resource loader that >> our OSGi implementation used for one particular part of the spec. > > Which part, in particular? I'll have to go back through my notes to say for sure, but I think we used it for a couple of purposes: 1) To support dynamic imports: https://issues.jboss.org/browse/MODULES-30 2) As a general hook to allow for OSGi-style "lazy" packages (the loader would examine the package of the class in question, recalculate the package set, and then retry the class load): https://issues.jboss.org/browse/MODULES-63 > To what extent does OSGi assume that bundles "export" their resources? > If we decide that a module's resources are not available outside the > module itself then will the approach of extending a module class loader > to be an OSGi bundle loader even work? This I will defer to an OSGi expert. My understanding is that OSGi largely operates on a package basis only, but I don't know how resources within the packages fit in to this picture (if at all). >> Java EE's requirements are much simpler and are satisfied by >> module-granularity dependencies and (in certain cases) resource importing. > > Aside from things like deployment descriptors, which are interpreted by > Java EE implementations, does the EE specification require EE "modules" > to export resources to other modules? The Java EE 7 platform specification does seem to do so, with language such as "Components in the web container may have access to the following classes and resources. [...] The classes and resources accessible to any other web modules included in the same ear file, as described above." The spec also seems to use the term "contents" interchangeably with "classes and resources". >> The APIs to do these things are all pretty simple though. >> >>>> Thus I feel this requirement should be amended to something along the >>>> lines of: >>>> >>>> Interoperation - It must be possible for another module system, such as >>>> OSGi, to locate Java modules and resolve them using its own resolver, >>>> except possibly for core system modules. It must be possible for other >>>> module systems to reuse the framework provided by this specification in >>>> order to facilitate a standard, well-defined mechanism by which >>>> different present and future module systems may interoperate. >>> >>> Unless I've misunderstood something, this sounds more like a research >>> project -- a "meta" module system, in fact -- which is well outside the >>> scope of this JSR. >> >> Maybe, but I contend that a lot (enough, insofar as any amount is ever >> enough) of research has been done, and that in any event there will have >> to be an API which implements the functionality of this JSR anyway, and >> it will be used as such no matter our intent so IMO it's best to use >> what has been learned to solve the problem in the best manner possible. > > I agree, but I think your proposed amendment is problematic -- it's too > difficult to test, since it's worded in terms of unspecified "present and > future module systems". Are there other, more-specific requirements we > can identify which would enable this stronger kind of interoperation? That's fair, I suppose. I think I would want to explore the compatibility implications of changing the module to class loader relationship before I come back with something stronger/more specific. -- - DML From david.lloyd at redhat.com Wed Mar 11 13:23:47 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 11 Mar 2015 08:23:47 -0500 Subject: Module-system requirements In-Reply-To: <20150309205617.E54AB511F0@eggemoggin.niobe.net> References: <20150211103831.528761@eggemoggin.niobe.net>, , <20150215172507.827110@eggemoggin.niobe.net>, <20150309205617.E54AB511F0@eggemoggin.niobe.net> Message-ID: <550041E3.4080901@redhat.com> On 03/09/2015 03:56 PM, mark.reinhold at oracle.com wrote: > 2015/2/15 10:12 -0800, Tim Boudreau : >> mark.reinhold at oracle.com: >>> As I wrote in my reply to David, I think resources (in the traditional >>> ClassLoader::getResource sense) should be treated as being strictly >>> internal to modules. ... >> >> There's a common case that I'd like to understand how you solve without at >> least META-INF or its equivalent visible to other modules: It's fairly >> common to see applications that dynamically compose things they find on the >> classpath. Spring and a few other things accomplish this through >> classpath-scanning, which is a fairly horrible pattern (you can wind up >> visiting every class on the classpath, or a large subset, just to look for >> a runtime annotation that is not there for the vast majority). >> >> An alternative approach I've used many times, which I think a few >> frameworks do as well (dagger?) is: Write an annotation processor that >> generates a flat file in META-INF that lists the classes that have the >> annotation in question; runtime code loads all such files on the classpath >> and uses that to initialize things that should be present at runtime. It's >> really the same pattern as ServiceLoader. >> >> Is it safe to assume that, whatever happens, that approach will still work, >> whether by special-casing META-INF/** or some equivalent? > > For JAR files on the legacy class path that approach will, of course, > continue to work. > > I don't expect actual modules to be on the class path (about which more > below), which raises a question: Will a framework such as Spring need to > discover the classes bearing certain annotations in modules that aren't > yet configured and loaded, or in loaded modules, or both? In the former > case it would inspect module artifacts, while in the latter it would > inspect some reflective run-time representation of a loaded module, since > the original module artifact might not be available. > > Either way, this use case suggests a new requirement for the Development > section: > > - _Annotation digests_ --- When packaging a module into an artifact it > must be possible to augment its definition with a summary of all of > the type- and package-level annotations that are present in the > module, together with an indication of the types and packages to > which each annotation applies. This digest must be easily readable > from module artifacts, and at run time those parts of it related to > annotations retained for run time must be available through an > appropriate reflective API. > > Comments? If we're looking to extend this into the domain of Java EE annotations (for example), we'd also have to add method and field annotations to the list. And SE 8 type annotations might add some strange color to this as well. >>> Having said that, I do agree that the module system must provide some >>> kind of escape hatch or "promiscuous mode", as you call it, so here's >>> a stab at an additional requirement for the "Modularizing Java SE" >>> section: >>> >>> - _Overrideable encapsulation_ --- It must be possible to force a >>> module to export one or more of its packages to all other modules, >>> or to some specific modules, even if its definition does not do so. >> >> One sort of low-hanging-fruit approach to this would be: >> - Modules do not use the traditional $CLASSPATH >> - The contents of $CLASSPATH are treated by the system as a single module >> (so everything can see each other, and legacy code behaves as expected) > > That's looking ahead to a design decision, I think, but in fact the > approach I intend to propose treats the legacy class path as the content > of a special "unnamed" module, analogous to the unnamed package, which > implicitly depends upon all other modules so that it works essentially > as you describe. This is more or less what we do as well as I've said before, the difference being that we don't implicitly depend on all other modules (and in general I think we need to exercise great caution before throwing the term "all modules" in to any requirement, as an installed module base might be arbitrarily large). -- - DML From david.lloyd at redhat.com Wed Mar 11 13:38:09 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 11 Mar 2015 08:38:09 -0500 Subject: Configurations... (Was: Re: Module-system requirements) In-Reply-To: <20150309205517.E1578511E9@eggemoggin.niobe.net> References: <20150211103831.528761@eggemoggin.niobe.net>, <20150214133307.760777@eggemoggin.niobe.net>, <54E1FAE2.2000605@redhat.com> <20150309205517.E1578511E9@eggemoggin.niobe.net> Message-ID: <55004541.50508@redhat.com> On 03/09/2015 03:55 PM, mark.reinhold at oracle.com wrote: > 2015/2/16 6:12 -0800, david.lloyd at redhat.com: >> On 02/14/2015 03:33 PM, mark.reinhold at oracle.com wrote: >>> 2015/2/12 7:35 -0800, david.lloyd at redhat.com: >>>> I'm a little concerned that the configurations section implies a strict >>>> hierarchy: >>>> >>>> "Nested dynamic configurations ??? An application must be able to create a >>>> dynamic configuration that relates to an existing dynamic configuration >>>> rather than to the application???s initial configuration." >>>> >>>> Why can a configuration not relate to more than one other configurations? >>> >>> A tree of configurations rather than a simple linear list is more >>> difficult to reason about and trickier to implement (you must, e.g., >>> cope with the diamond-import problem). >>> >>> The feedback we've received so far has not suggested that this >>> limitation will be a problem in practice. Do you have use cases >>> which require this flexibility? >> >> We implement not just a tree, but a general directed graph (even >> allowing cycles) among modules and among their configurations (module >> loaders), > > A configuration, as defined in the requirements document, is a group of > modules related by resolution to each other and, if needed, to modules in > one or more enclosing configurations. If you look just at the graph of > modules in a running system then it's at least a DAG, and possibly even > cyclic, but that doesn't necessarily mean that the configurations > themselves need to be related to each other in a structure any richer > than a tree. OK I guess I need to back up a little. Our module loaders do have an arbitrarily complex logical relationship, but this stems from the fact that they're actually physically fully isolated from one another, and free to delegate module loading decisions arbitrarily. Any module's (programmatic representation of its) dependency declarations may consist of a module loader + identifier combination, and any module loader may choose to delegate part or all of its namespace to any other module loader. >> which enables us to easily support OSGi and Java EE, as well >> as tying these things to filesystem JARs (e.g. for EE deployments which >> have Class-Path declarations that have an absolute path name), and the >> legacy extensions mechanism which is presently also a part of Java EE >> (though this is going/has gone away, I understand), and also allows >> tying in other systems as well. > > Does doing these things require configurations to be related in a > directed graph, or is a tree sufficient? The way we interoperate between extensions, external filesystem class path entries, and EE modules, by using the above mechanism we can (for example) create a dependence relationship from the EE module to an extension module, from the extension module to a filesystem class path module, from a filesystem class path module back to an extension module, and from any of these three module types back to the static platform module set. Really to us a ModuleLoader is just a namespace with some policy connected to it; the actual loading/resolving relationships are simply arrows to specific other modules, optionally in specific other ModuleLoaders. >> Due to the way we support imports, diamonds are really no problem. Each >> module's dependency list is interpreted in order, so the set of imported >> resources is always the union of everything exported by each dependency. >> So if you acquire a different set of items via two different "paths" >> to a module, you get the union of those sets in the end. Duplicated >> paths are removed such that the first occurrence is used. > > Ah, so the order of dependence declarations is significant in your > system. Interesting, but I wonder if it's wise. So far it hasn't caused us any problems; on the contrary it's been proven to be a quite useful trait. Though in retrospect I think it would have been better to spend more time trying to prevent package overlap, as people tend to use this to implement dirty (fragile) tricks (as people do). It was only due to technical reasons that we can't effectively distinguish between a package and a resource directory in all the cases that we need to be able to do so; this may have been resolvable given a bit more perseverance. > Anyway, the diamond-import problem I was thinking of has to do with > configurations rather than modules. It can be solved in a manner similar > to what you describe for resources, i.e., by forcing an application that > wants to do this to place an order on the configurations to which a new > configuration is to be related. When looking for a module the resolver > would consider the configurations in that order, and since only one > module can satisfy any particular dependence it would just stop at the > first one that it finds. > > I'm still concerned, however, that this could lead to some unpleasant > implementation complexities, and also be more difficult to use in > practice, so unless there's a clear need to relate configurations in > graphs then I'd prefer to limit them to trees. Maybe what we need is a clearer definition of "Configuration" as it is only defined as a side-effect of Resolution: "The resulting configuration contains all of the code and data required to compile or run the initial module." And Linking is AFAICT the first section that implies that Configurations actually contain modules in any way. This might help me understand what a tree topology actually means. -- - DML From david.lloyd at redhat.com Wed Mar 11 14:03:32 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 11 Mar 2015 09:03:32 -0500 Subject: Module, Class, and Resource Iteration (Was: Re: Module-system requirements) In-Reply-To: <20150309210418.0DD6D51215@eggemoggin.niobe.net> References: <20150211103831.528761@eggemoggin.niobe.net>, <54EE1C1D.3020400@redhat.com> <20150309210418.0DD6D51215@eggemoggin.niobe.net> Message-ID: <55004B34.2000702@redhat.com> On 03/09/2015 04:04 PM, mark.reinhold at oracle.com wrote: > 2015/2/25 11:01 -0800, david.lloyd at redhat.com: >> One thing that is missing from the requirements is the ability to >> iterate or enumerate modules, classes, and/or resources. This is one >> capability that people seem to be assuming will be magically provided by >> a module solution. >> >> While it seems to me that resource or resource directory iteration is >> (modulo the discussion on the "Exporting things" sub-thread) easily >> accomplished, defining the meaning of module iteration and class >> iteration is a little bit more involved. > > Yes ... > >> To iterate modules implies that the configuration has a complete picture >> of all the modules it can load, and that it can iterate over this >> picture in a relatively inexpensive manner. This is probably >> reasonable, though in my imagination I see that the set of available >> modules may change over time within the life of a single VM process. > > It should be straightforward to iterate through the modules already > present in a configuration. Iterating through the modules available > when creating a new configuration, or extending an existing one, should > be possible but might not be very efficient since it will most likely > require walking the filesystem. > > Do we need a requirement to capture this? It seems pretty low-level. Probably not, I guess, unless someone very feels strongly about having this ability specified. Internally we created an API method to allow both kinds of iteration on hints that certain parties may be requesting this functionality, but it remains unimplemented for all of our module loader types leading me to believe that it might not have a real practical application (that isn't better solved in another way). >> Iterating classes is a stranger expectation though, because while it is >> a reasonable inclination to want to try this, it is possibly actively >> harmful to start loading previously-unloaded classes. So such a >> function would either need to restrict its output set to loaded classes, >> or do something else like just enumerate class names. In addition, >> there are security considerations to take into account; a caller should >> only "see" classes that are accessible to it (though things like >> frameworks may need an elevated capability in order to introspect >> non-public classes). > > It wouldn't be hard to iterate through the class files in a module > artifact. Iterating through the set of classes already loaded for a > module (or by class loader) is possible but trickier. JVM TI already > supports this [1]; does it need to be possible in a direct reflective > API? Are there use cases which require that? For resources, definitely; for classes, none that I'm aware of. We support only resource iteration, though one could extrapolate the set of available classes from that given our simple relationship between those two things. >> Another potentially iterable "thing" is packages, either in the form of >> names or Package objects. No comment on the applicability or lack >> thereof of such a capability though. > > Again, are there use cases for this kind of query? None that I know of. We don't support it at present. -- - DML From forax at univ-mlv.fr Wed Mar 11 14:30:47 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 11 Mar 2015 15:30:47 +0100 Subject: Module-system requirements In-Reply-To: <550041E3.4080901@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, , <20150215172507.827110@eggemoggin.niobe.net>, <20150309205617.E54AB511F0@eggemoggin.niobe.net> <550041E3.4080901@redhat.com> Message-ID: <55005197.9010806@univ-mlv.fr> On 03/11/2015 02:23 PM, David M. Lloyd wrote: > On 03/09/2015 03:56 PM, mark.reinhold at oracle.com wrote: >> 2015/2/15 10:12 -0800, Tim Boudreau : >>> mark.reinhold at oracle.com: >>>> As I wrote in my reply to David, I think resources (in the traditional >>>> ClassLoader::getResource sense) should be treated as being strictly >>>> internal to modules. ... >>> >>> There's a common case that I'd like to understand how you solve >>> without at >>> least META-INF or its equivalent visible to other modules: It's fairly >>> common to see applications that dynamically compose things they find >>> on the >>> classpath. Spring and a few other things accomplish this through >>> classpath-scanning, which is a fairly horrible pattern (you can wind up >>> visiting every class on the classpath, or a large subset, just to >>> look for >>> a runtime annotation that is not there for the vast majority). >>> >>> An alternative approach I've used many times, which I think a few >>> frameworks do as well (dagger?) is: Write an annotation processor that >>> generates a flat file in META-INF that lists the classes that have the >>> annotation in question; runtime code loads all such files on the >>> classpath >>> and uses that to initialize things that should be present at >>> runtime. It's >>> really the same pattern as ServiceLoader. >>> >>> Is it safe to assume that, whatever happens, that approach will >>> still work, >>> whether by special-casing META-INF/** or some equivalent? >> >> For JAR files on the legacy class path that approach will, of course, >> continue to work. >> >> I don't expect actual modules to be on the class path (about which more >> below), which raises a question: Will a framework such as Spring need to >> discover the classes bearing certain annotations in modules that aren't >> yet configured and loaded, or in loaded modules, or both? In the former >> case it would inspect module artifacts, while in the latter it would >> inspect some reflective run-time representation of a loaded module, >> since >> the original module artifact might not be available. >> >> Either way, this use case suggests a new requirement for the Development >> section: >> >> - _Annotation digests_ --- When packaging a module into an >> artifact it >> must be possible to augment its definition with a summary of all of >> the type- and package-level annotations that are present in the >> module, together with an indication of the types and packages to >> which each annotation applies. This digest must be easily readable >> from module artifacts, and at run time those parts of it related to >> annotations retained for run time must be available through an >> appropriate reflective API. >> >> Comments? > > If we're looking to extend this into the domain of Java EE annotations > (for example), we'd also have to add method and field annotations to > the list. And SE 8 type annotations might add some strange color to > this as well. Why do that when packaging a module and not when you install a module ? I really think we need a way to do some actions when modules are installed (and before being transformed on an image), either some global action on all modules like crawling for annotations or on the module that contains the code that does the action (or maybe both). R?mi From niftiness at gmail.com Wed Mar 11 19:53:20 2015 From: niftiness at gmail.com (Tim Boudreau) Date: Wed, 11 Mar 2015 15:53:20 -0400 Subject: Module-system requirements In-Reply-To: <55005197.9010806@univ-mlv.fr> References: <20150211103831.528761@eggemoggin.niobe.net> <20150215172507.827110@eggemoggin.niobe.net> <20150309205617.E54AB511F0@eggemoggin.niobe.net> <550041E3.4080901@redhat.com> <55005197.9010806@univ-mlv.fr> Message-ID: > > If we're looking to extend this into the domain of Java EE annotations >> (for example), we'd also have to add method and field annotations to the >> list. And SE 8 type annotations might add some strange color to this as >> well. >> > > Why do that when packaging a module and not when you install a module ? > > I really think we need a way to do some actions when modules are installed > (and before being transformed on an image), > either some global action on all modules like crawling for annotations > or on the module that contains the code that does the action > (or maybe both). > I think Mark's "annotation digest" proposal covers that. "Crawling for annotations" is kind of pathological - I know people are used to it, but it's also the worst possible way to solve the problem of dynamic discovery. So something that solves the reason people *want* to crawl for annotations, rather than necessarily supporting the crawling, is the way to go. -Tim From david.lloyd at redhat.com Wed Mar 11 21:46:11 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 11 Mar 2015 16:46:11 -0500 Subject: Module-system requirements In-Reply-To: <550041E3.4080901@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, , <20150215172507.827110@eggemoggin.niobe.net>, <20150309205617.E54AB511F0@eggemoggin.niobe.net> <550041E3.4080901@redhat.com> Message-ID: <5500B7A3.5070308@redhat.com> On 03/11/2015 08:23 AM, David M. Lloyd wrote: > On 03/09/2015 03:56 PM, mark.reinhold at oracle.com wrote: >> 2015/2/15 10:12 -0800, Tim Boudreau : >>> mark.reinhold at oracle.com: >>>> As I wrote in my reply to David, I think resources (in the traditional >>>> ClassLoader::getResource sense) should be treated as being strictly >>>> internal to modules. ... >>> >>> There's a common case that I'd like to understand how you solve >>> without at >>> least META-INF or its equivalent visible to other modules: It's fairly >>> common to see applications that dynamically compose things they find >>> on the >>> classpath. Spring and a few other things accomplish this through >>> classpath-scanning, which is a fairly horrible pattern (you can wind up >>> visiting every class on the classpath, or a large subset, just to >>> look for >>> a runtime annotation that is not there for the vast majority). >>> >>> An alternative approach I've used many times, which I think a few >>> frameworks do as well (dagger?) is: Write an annotation processor that >>> generates a flat file in META-INF that lists the classes that have the >>> annotation in question; runtime code loads all such files on the >>> classpath >>> and uses that to initialize things that should be present at >>> runtime. It's >>> really the same pattern as ServiceLoader. >>> >>> Is it safe to assume that, whatever happens, that approach will still >>> work, >>> whether by special-casing META-INF/** or some equivalent? >> >> For JAR files on the legacy class path that approach will, of course, >> continue to work. >> >> I don't expect actual modules to be on the class path (about which more >> below), which raises a question: Will a framework such as Spring need to >> discover the classes bearing certain annotations in modules that aren't >> yet configured and loaded, or in loaded modules, or both? In the former >> case it would inspect module artifacts, while in the latter it would >> inspect some reflective run-time representation of a loaded module, since >> the original module artifact might not be available. >> >> Either way, this use case suggests a new requirement for the Development >> section: >> >> - _Annotation digests_ --- When packaging a module into an artifact it >> must be possible to augment its definition with a summary of all of >> the type- and package-level annotations that are present in the >> module, together with an indication of the types and packages to >> which each annotation applies. This digest must be easily readable >> from module artifacts, and at run time those parts of it related to >> annotations retained for run time must be available through an >> appropriate reflective API. >> >> Comments? > > If we're looking to extend this into the domain of Java EE annotations > (for example), we'd also have to add method and field annotations to the > list. Actually this is not true after all. Other than possibly examining constructor annotations to decide if a class is a CDI managed bean that is. > And SE 8 type annotations might add some strange color to this as > well. > >>>> Having said that, I do agree that the module system must provide some >>>> kind of escape hatch or "promiscuous mode", as you call it, so here's >>>> a stab at an additional requirement for the "Modularizing Java SE" >>>> section: >>>> >>>> - _Overrideable encapsulation_ --- It must be possible to force a >>>> module to export one or more of its packages to all other modules, >>>> or to some specific modules, even if its definition does not do so. >>> >>> One sort of low-hanging-fruit approach to this would be: >>> - Modules do not use the traditional $CLASSPATH >>> - The contents of $CLASSPATH are treated by the system as a single >>> module >>> (so everything can see each other, and legacy code behaves as expected) >> >> That's looking ahead to a design decision, I think, but in fact the >> approach I intend to propose treats the legacy class path as the content >> of a special "unnamed" module, analogous to the unnamed package, which >> implicitly depends upon all other modules so that it works essentially >> as you describe. > > This is more or less what we do as well as I've said before, the > difference being that we don't implicitly depend on all other modules > (and in general I think we need to exercise great caution before > throwing the term "all modules" in to any requirement, as an installed > module base might be arbitrarily large). -- - DML From mark.reinhold at oracle.com Mon Mar 16 20:43:06 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 16 Mar 2015 13:43:06 -0700 (PDT) Subject: Interoperation (Was: Re: Module-system requirements) In-Reply-To: <55003FBB.5050705@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, <20150309205417.DE06C511E7@eggemoggin.niobe.net>, <55003FBB.5050705@redhat.com> Message-ID: <20150316204306.B09C65200B@eggemoggin.niobe.net> 2015/3/11 6:14 -0700, david.lloyd at redhat.com: > On 03/09/2015 03:54 PM, mark.reinhold at oracle.com wrote: >> 2015/2/16 5:57 -0800, david.lloyd at redhat.com: >>> ... >>> >>> In the case of both Java EE and OSGi, I think it's safe to say that you >>> need to apply a strict 1:1 mapping between modules and class loaders for >>> this to work. >> >> I don't think we can mandate this. To divide the SE platform into >> modules in a compatible way requires the ability to load multiple modules >> via the same class loader, so as to respect long-standing assumptions >> about which platform classes are loaded by the built-in class loaders >> (i.e., the bootstrap and extension loaders). In an early Jigsaw >> prototype we tried the 1:1 module/loader mapping for platform modules >> and found it unworkable from a compatibility perspective. > > I went through the jigsaw list archives looking for this discussion and > didn't turn up anything; was this discussed on-list and I am just > failing to find it? I don't recall it being discussed on-list. Perhaps the biggest issue we ran into is that there's lots of code, both in the JDK and out in the wild, that does tests of the form if (ob.getClass().getClassLoader() == null) { ... } which, of course, succeeds for classes loaded by the bootstrap loader. We expect to divide the bootstrap classes into multiple modules (see JEP 200 [1] and the modules.xml file in JDK 9 [2] for the current prototype definition), which would break all such tests if each module had its own class loader. [1] http://openjdk.java.net/jeps/200 [2] http://hg.openjdk.java.net/jdk9/jdk9/file/tip/modules.xml >> Aside from the SE platform itself, I don't think we can force all >> applications to adopt the unique-class-loader-per-module approach. >> Oracle's WebLogic application server, e.g., doesn't do that today and >> it'd be difficult, if not impossible, to modify it to do so without >> breaking existing applications. > > I guess I can't really comment on what WebLogic does or does not do. I > can say however that we were able to move from a flat classpath > architecture to a modular architecture like this without breaking things > due to the modularization. The Java EE spec is pretty clear about what > needs to be visible to what, and by translating this information into > module dependency information, we were able to make a smooth transition. > I can't really imagine an application that would be broken unless they > were relying on some specific non-spec behavior as the EE spec says: > > "[...] A Java EE application should not assume that all components in > the application will be available on the class path of the application > at run time. Each component might be loaded into a separate class loader > with a separate namespace. [...]" The concern with WebLogic is that existing customers rely upon aspects of its particular class-loader hierarchy, beyond what is guaranteed by the Java EE specification. Changing it incompatibly would break those customers' applications. > Note that in the EE spec, the term "component" has a finer granularity > than "module", i.e. there are 1..N components per module, whereas an > "application" can contain 1 or more modules. Yes, and I don't think that the module concept we're discussing here will wind up corresponding exactly to what EE calls "modules". > ... > >> I suggest the module system be as neutral as possible with respect to how >> modules and class loaders are related. A complex application should be >> able to spin up a class loader for every module it loads, if needed >> (excepting non-upgradeable platform classes), but not all applications >> should be forced to do so. >> >> New requirement, for the Dynamic Configuration section: >> >> - _Control of class loaders_ --- An application that loads a module >> into a dynamic configuration must be able to specify the class loader >> to be used to load that module. That loader need not be the same as >> the loader used to load any other module in the configuration. > > This goes part of the way, for sure. I'm not 100% sold yet though. > It's not easy to let go of something that (from our perspective) is > tried and true. This requirement allows you to use the 1:1 model above SE itself, if you want that, but it doesn't force everyone else to do so. >> ... >> >>> OSGi specifically requires that bundle class loaders implement an OSGi >>> spec interface, implying that the module class loader class must be >>> extensible (as I said a couple of years back though, we make most of our >>> methods final on the module class loader to disallow any really bizarre >>> behavior, and this works well in practice). >> >> Which OSGi interface are you thinking of here? > > In OSGi Core 6.0.0 section 3.9.9: > > "Class Loader - An OSGi framework must ensure that the class loader of a > class that comes from > a bundle implements the BundleReference interface." > > The BundleReference interface has a single getBundle() method on it. Ah, got it. >>> We also had to implement a >>> pre- and/or post-class-define hook per module to allow some OSGi magic >>> to happen. >> >> These are non-final protected methods in your module class loader? >> >> Could you forward a description of these methods to the list? > > Yeah, they look like this: > > protected void preDefine(ClassSpec classSpec, String className) { > } > > protected void postDefine(ClassSpec classSpec, Class definedClass) { > } > > ClassSpec contains the bytes and CodeSource of the class. The hooks > don't have opportunity to do anything like rewrite the class bytes, but > we also accept a java.lang.instrument.ClassFileTransformer instance in > the module definition for that purpose (though we aren't using it at > this point). Our OSGi implementation only ended up using one of these > hooks (preDefine I think). They were added to support lazy bundle > activation. Okay. >>> Finally there is a per-module fallback resource loader that >>> our OSGi implementation used for one particular part of the spec. >> >> Which part, in particular? > > I'll have to go back through my notes to say for sure, but I think we > used it for a couple of purposes: > > 1) To support dynamic imports: https://issues.jboss.org/browse/MODULES-30 > 2) As a general hook to allow for OSGi-style "lazy" packages (the loader > would examine the package of the class in question, recalculate the > package set, and then retry the class load): > https://issues.jboss.org/browse/MODULES-63 Interesting. >> To what extent does OSGi assume that bundles "export" their resources? >> If we decide that a module's resources are not available outside the >> module itself then will the approach of extending a module class loader >> to be an OSGi bundle loader even work? > > This I will defer to an OSGi expert. My understanding is that OSGi > largely operates on a package basis only, but I don't know how resources > within the packages fit in to this picture (if at all). ??3.9.6 and ??10.1.5.13 of the OSGi R6 specification imply that all the resources in a bundle are available to anyone with access to the bundle object itself and, if running with a security manager, the appropriate OSGi AdminPermission(s). If we adopt the "Access to class files and resources" requirement which I suggested in the "exporting things" thread then an OSGi implementation built on top of the platform module system could use that to expose the resources of a module artifact as needed. >>> Java EE's requirements are much simpler and are satisfied by >>> module-granularity dependencies and (in certain cases) resource importing. >> >> Aside from things like deployment descriptors, which are interpreted by >> Java EE implementations, does the EE specification require EE "modules" >> to export resources to other modules? > > The Java EE 7 platform specification does seem to do so, with language > such as "Components in the web container may have access to the > following classes and resources. [...] The classes and resources > accessible to any other web modules included in the > same ear file, as described above." > > The spec also seems to use the term "contents" interchangeably with > "classes and resources". The Java EE specification, just like OSGi, is explicitly built on the JAR file format and derivations thereof, so I think the same solution applies for Java EE: An EE implementation can access a module's artifact and expose the artifact's resources as needed. - Mark From mark.reinhold at oracle.com Mon Mar 16 20:44:06 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 16 Mar 2015 13:44:06 -0700 (PDT) Subject: Configurations... (Was: Re: Module-system requirements) In-Reply-To: <55004541.50508@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, <20150309205517.E1578511E9@eggemoggin.niobe.net>, <55004541.50508@redhat.com> Message-ID: <20150316204406.B396C52012@eggemoggin.niobe.net> 2015/3/11 6:38 -0700, david.lloyd at redhat.com: > ... > > Maybe what we need is a clearer definition of "Configuration" as it is > only defined as a side-effect of Resolution: "The resulting > configuration contains all of the code and data required to compile or > run the initial module." And Linking is AFAICT the first section that > implies that Configurations actually contain modules in any way. This > might help me understand what a tree topology actually means. A configuration is the result of resolution and service binding. You can think of it as a graph, with modules for nodes, edges for module dependences, and edges (of a different color) for service-provider bindings. I'm pretty sure that a configuration, in this sense, is not analogous to what you call a module loader in your system. A configuration can be loaded by a single class loader, multiple class loaders, or even one class loader per module, but the module system shouldn't mandate any particular approach. If you build a dynamic configuration on top of the initial configuration, and then another on top of that, as you might do in something like an application server, then the union of their graphs will be a DAG (at least) but the configurations themselves are just in a list, somewhat like a search path. You can imagine having multiple such lists sharing prefixes, i.e., a tree of configurations, but to resolve a dependence in any particular configuration you only need to consult its parent configurations. If configurations can be related in a more general way, i.e., in a DAG, then the resolution algorithm becomes more complex. Do we actually need that? I've yet to see any use cases. - Mark From mark.reinhold at oracle.com Mon Mar 16 20:45:06 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 16 Mar 2015 13:45:06 -0700 (PDT) Subject: Module-system requirements In-Reply-To: <55005197.9010806@univ-mlv.fr> References: <20150211103831.528761@eggemoggin.niobe.net>, <550041E3.4080901@redhat.com>, <55005197.9010806@univ-mlv.fr> Message-ID: <20150316204506.B665052018@eggemoggin.niobe.net> 2015/3/11 7:30 -0700, Remi Forax : > On 03/11/2015 02:23 PM, David M. Lloyd wrote: >> On 03/09/2015 03:56 PM, mark.reinhold at oracle.com wrote: >>> Either way, this use case suggests a new requirement for the Development >>> section: >>> >>> - _Annotation digests_ --- When packaging a module into an artifact >>> it must be possible to augment its definition with a summary of all >>> of the type- and package-level annotations that are present in the >>> module, together with an indication of the types and packages to >>> which each annotation applies. This digest must be easily readable >>> from module artifacts, and at run time those parts of it related to >>> annotations retained for run time must be available through an >>> appropriate reflective API. >>> >>> Comments? >> >> If we're looking to extend this into the domain of Java EE annotations >> (for example), we'd also have to add method and field annotations to >> the list. And SE 8 type annotations might add some strange color to >> this as well. > > Why do that when packaging a module and not when you install a module ? Packaging a module is, generally, not a time-sensitive operation, while configuring and loading a module often is. Doing more work while packaging in order to do less work at run time (or even compile time) is a pretty nice tradeoff to be able to make. > I really think we need a way to do some actions when modules are installed > (and before being transformed on an image), > either some global action on all modules like crawling for annotations > or on the module that contains the code that does the action > (or maybe both). The requirements don't imply or presuppose a notion of "installation", other than the installation of OS-specific packages (e.g., RPMs) that contain Java modules. Modules can be loaded at or after startup, or linked into a static image, but there's no notion of "installation" as such. A sophisticated, module-managing application could certainly pre-process modules in some way before it loads them, if needed, but I think that's outside the scope of the module system. - Mark From mark.reinhold at oracle.com Mon Mar 16 20:46:06 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 16 Mar 2015 13:46:06 -0700 (PDT) Subject: Module-system requirements In-Reply-To: <5500B7A3.5070308@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, <550041E3.4080901@redhat.com>, <5500B7A3.5070308@redhat.com> Message-ID: <20150316204606.B946C5201E@eggemoggin.niobe.net> 2015/3/11 2:46 -0700, david.lloyd at redhat.com: > On 03/11/2015 08:23 AM, David M. Lloyd wrote: >> On 03/09/2015 03:56 PM, mark.reinhold at oracle.com wrote: >>> ... >>> >>> Either way, this use case suggests a new requirement for the Development >>> section: >>> >>> - _Annotation digests_ --- When packaging a module into an artifact it >>> must be possible to augment its definition with a summary of all of >>> the type- and package-level annotations that are present in the >>> module, together with an indication of the types and packages to >>> which each annotation applies. This digest must be easily readable >>> from module artifacts, and at run time those parts of it related to >>> annotations retained for run time must be available through an >>> appropriate reflective API. >>> >>> Comments? >> >> If we're looking to extend this into the domain of Java EE annotations >> (for example), we'd also have to add method and field annotations to the >> list. > > Actually this is not true after all. Other than possibly examining > constructor annotations to decide if a class is a CDI managed bean that is. Do such constructor annotations really need to be in the digest? Or do such classes already carry class-level annotations to indicate that they're to be managed by the CDI framework? >> And SE 8 type annotations might add some strange color to this as >> well. Hrm. I suppose we could summarize all the annotations in a module, regardless of the kinds of elements they're applied to, but is that overkill? - Mark From mark.reinhold at oracle.com Mon Mar 16 20:47:06 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 16 Mar 2015 13:47:06 -0700 (PDT) Subject: Exporting things (Was: Re: Module-system requirements) In-Reply-To: <54FF08F7.8050901@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, <20150309205917.F049C511F6@eggemoggin.niobe.net>, <54FF08F7.8050901@redhat.com> Message-ID: <20150316204706.BC29152023@eggemoggin.niobe.net> 2015/3/10 8:08 -0700, david.lloyd at redhat.com: > On 3/9/15 3:59 PM, mark.reinhold at oracle.com wrote: >> 2015/2/16 5:15 -0800, david.lloyd at redhat.com: >>> On 02/15/2015 07:24 PM, mark.reinhold at oracle.com wrote: >>>> ... >>>> >>>> Exposing all of a module's resources, as JAR files do today, is wrong >>>> from the perspective of encapsulation. ... >>>> >>>> This suggests an additional requirement: >>>> >>>> - _Resource encapsulation_ --- The run-time system must ensure that the >>>> static resource files within a module are directly accessible only by >>>> code within that module. >>> >>> What you are saying here implies (to me) inspecting the call stack to >>> ensure that the caller actually exists within the module being >>> referenced. >> >> Yes. >> >>> I'm not certain we really want to go this far, if my >>> interpretation is accurate. Such a rule is reasonable on the face of >>> it, but it does more or less break all the code in existence which >>> currently uses resources as a sort of API into a library, and this >>> includes very common patterns for loading configuration (among many >>> other things). >> >> True. To what degree should it be a goal to preserve such mechanisms? > > I really feel that not having this compatibility will strongly diminish > adoption of this system, since a substantial body of existing work is > unlikely to function properly as modules. > >> Must Java modules be forever constrained to look like JAR files? > > No, but I think that isn't really related to the point at hand, which is > more related to the solution of specific problems rather than change for > its own sake. If solving those problems means that modules have visible > resources, does that mean we've tripped an invisible failure criteria by > resembling JARs too closely in some specific way? > > Or put a nicer way, is there a specific problem or set of problems that > are solved by strictly encapsulating resources, or is it more of a way > to try a new general approach? Encapsulation of resources is valuable for the same reasons that encapsulation of types is valuable. It allows you to conceal a module's implementation details so that external code does not depend them, giving you greater flexibility to evolve the module going forward. It also allows you to keep adversaries from inspecting the internals of your module to discover information that might be useful in an attack. We'll encapsulate types by default, and also have a way to export them to other modules on a per-package basis. We seemed to agree earlier that a similar style of export control for resources would be overkill, and my suggested requirement to ensure access to a module artifact's resources by suitably-privileged code, when such an artifact exists, appears to cover the use cases so far discussed (as you note later in your message, quoted below). >> ... >> >> To ensure that a distributor of a set of modules can control how service >> dependences are resolved amongst those modules here's a new requirement, >> for the Fundamentals section: >> >> - _Selective binding_ --- It must be possible to control the binding >> process so that specific services are provided only by specific >> providers, and in a specific order. >> >> Does that make sense? > > In this requirement, what party is doing the controlling? Whatever party has invoked the binding process. It might be the user, via command-line flags or build-system configuration settings, or it might be an application server, via the module-system API. >> ... >> >> So a JBoss module can specify that it depends upon service providers from >> one or more specific modules rather than whichever modules happen to >> provide implementations of that service? That seems contrary to the >> nature of services, if I understand you correctly. > > Right; this is however functionally similar to having the configuration > determine the service bindings without any sub-configurations being able > to override it, though at a whole-module granularity rather than a > per-module-service granularity. I can understand having a build system or a run-time framework control service bindings in this way, but it seems odd for a module itself to declare that it requires a specific provider of a service. > ... > > In the configuration-level case though, establishment and predictability > of order (at least on a per distribution basis) is almost certainly > going to be needed I think, so I'd still maintain that it should be > worked into the requirements somehow. Is the proposed "selective binding" requirement (above) sufficient? >> When, in the future, an application is assembled by resolving a set of >> actual modules into a configuration, then we won't necessarily have to >> use class loaders to mediate resource access, so I'd rather not bake that >> into the requirements. A configuration ought to be able to relate >> classes, modules, and module artifacts such that, given a module and a >> class name, you can read the corresponding class file (or any other kind >> of resource) from the module's defining artifact and load it however you >> want. >> >> To capture this as a requirement, in the Development section: >> >> - _Access to class files and resources_ --- If a module is defined by >> an artifact that contains class files and resources then it must be >> possible, once that module is added to a configuration, for >> suitably-privileged code to access those class files and resources. > > I like this requirement; I think it hits a good percentage of use cases. > >> (The "if a module is defined by an artifact" precondition allows for >> ahead-of-time-compilation scenarios.) >> >> This requirement also addresses the problem of how to locate and read EE >> deployment descriptors and similar kinds of configuration files. > > Great. - Mark From mark.reinhold at oracle.com Mon Mar 16 20:48:06 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 16 Mar 2015 13:48:06 -0700 (PDT) Subject: Security In-Reply-To: <54FEF692.5050008@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, , <54E7AB98.3030005@redhat.com>, <20150309210218.0706B51211@eggemoggin.niobe.net>, , <54FEF692.5050008@redhat.com> Message-ID: <20150316204806.BF53E52029@eggemoggin.niobe.net> 2015/3/10 6:50 -0700, david.lloyd at redhat.com: > On 3/9/15 9:51 PM, Tim Boudreau wrote: >> ... >> >> I don't think it works to discuss permissions in the abstract without an >> explicit use case or two to understand what you're after and why. > > In addition to being able to support Java EE, I think that the use cases > overlap heavily with those of, say, javaws applications. I don't think > we can really discuss a module distribution mechanism without > considering the security aspects that informed previous distribution > mechanisms. The Applet and Java Web Start (aka JNLP, JSR 56) security model remains very simple: Code either runs in a highly-restricted sandbox, or it has all permissions. It is possible to grant permissions to specific code bases or code-base origins, or specific code signers, via system-wide or user-specific policy files; so far as I know, however, this is rarely done in practice. There is no way to indicate, in an applet tag or JAR, or a JNLP file, that specific fine-grained permissions are required. OSGi has a sophisticated mechanism for managing the permissions granted to bundles, but unless I've missed something it does not have a way to indicate, in a bundle's definition, that the bundle requires specific permissions. The Java EE 7 use case is somewhere between these two in scope and complexity (see my reply nearby). > We have users who have used security managers not just for untrusted > code, but also as a tripwire for trusted code for bug prevention as well > as exploitation detection. I'm not sure I'm really the right person to > justify the existence of the security manager, but the fact is that > people do think they need it, and they do use it, including Java EE > permissions.xml files as well as our own per-module permission > declaration system. Aside from JBoss modules, are there any other examples of systems in which the definition of a module describes the permissions that the module requires? - Mark From mark.reinhold at oracle.com Mon Mar 16 20:49:06 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 16 Mar 2015 13:49:06 -0700 (PDT) Subject: Security In-Reply-To: <54FEF3B0.4060903@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, <20150309210218.0706B51211@eggemoggin.niobe.net>, <54FEF3B0.4060903@redhat.com> Message-ID: <20150316204906.C2EA05202F@eggemoggin.niobe.net> 2015/3/10 6:37 -0700, david.lloyd at redhat.com: > On 3/9/15 4:02 PM, mark.reinhold at oracle.com wrote: >> 2015/2/20 1:48 -0800, david.lloyd at redhat.com: >>> ... >>> >>> I think there is definitely value in the module knowing what permissions >>> it needs to function, and to be shipped with those permissions. I think >>> that if this is combined with a configuration-specific verification >>> mechanism, this could allow users to express a level of trust the way >>> they do today for signed JARs, and/or perhaps be able to verify (at >>> install time) whether or not they want to go ahead with installing a >>> module with certain permissions. >> >> I can imagine building something like this, but would anyone use it? > > We've seen a reasonably significant usage of security managers and their > associated permissions, especially now that it is a part of Java EE, so > the functionality would at least need to be present to bring Java EE > forward into this world. (See my other reply, nearby, on permissions in EE.) > The fact is that many environments (corporate, educational, government) > require the use of a security manager for their Java based applications; > it is my belief that if we each reached out to our associated support > organizations, we'd find that this is generally true. Would we find that people are granting permissions to specific JAR files and/or code signers, or would we find that they're using coarser-grained mechanisms (the Applet/JNLP sandbox, EE's permissions.xml, etc.)? > Without a convenient way to establish permissions per module, it's > really a step backwards in this area, and (at least at present) it is so > simple to implement that there doesn't seem to be a compelling argument > *not* to do it. Sorry, but in general this kind of reasoning is not persuasive. Many features were included in early versions of Java because they were simple to implement and there weren't compelling reasons not to include them, yet in hindsight we regret having added them. > And I suspect that whether or not we provide a mechanism for > distributors to assign permissions to modules, there still needs to be > some kind of trust decision by the installer to decide whether (s)he > wants to install the module in question based on where it comes from. Isn't that better handled by whatever mechanism is used to install Java modules on the local system (e.g., RPM)? - Mark From mark.reinhold at oracle.com Mon Mar 16 20:50:06 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 16 Mar 2015 13:50:06 -0700 (PDT) Subject: Security In-Reply-To: <54FF106C.5090901@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, , <54FF106C.5090901@redhat.com> Message-ID: <20150316205006.C6D9952036@eggemoggin.niobe.net> 2015/3/10 8:40 -0700, david.lloyd at redhat.com: > On 3/10/15 9:55 AM, Tim Ellison wrote: >> ... >> >> David, is it not sufficient that the java.policy file can contain a >> code source url specifying jrt:/$MODULE ? This gives a mechanism for >> permissions to be granted to code associated with a module >> irrespective of the artifact that delivers the module. > > I believe not, for these reasons: > > ??? The policy file is not AFAIK actually standardized in the SE platform, > though (of course) the policy mechanism itself is. Correct. > ??? The "jrt" URL scheme, as I understand it, applies specifically to the > JDK runtime and not generally to modules. The "jrt" scheme can be used to name any module linked statically into the run-time image, whether an SE/JDK module or a library or application module. It cannot be used to name modules loaded at run time, so from a security perspective (where URLs are needed) it's not a general solution to the module-naming problem. > ??? The JDK runtime itself effectively has AllPermission as far as I > understand. No, the permissions granted to code not loaded by the bootstrap class loader are limited. Have a look at the JDK 9 default java.policy file: http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/tip/src/java.base/share/conf/security/java.policy (This does, however, seem like a side issue.) > ??? Assuming that each configuration (I guess?) gets a unique URL scheme > which applies to it, I don't think each configuration (= set of resolved modules) would get its own URL scheme, or even a URL. I could imagine each module in a system, whether statically linked or dynamically loaded, having a unique URL, but it's yet not clear to me whether that's needed. > using the global policy exclusively is still > non-optimal because it loses a substantial amount of information: the > author of the module knows what permissions the module would require, > whereas the consumer of the module does not have this knowledge. Is this really true, in general? If I ship, say, a JDBC driver then I won't know ahead of time which network hosts and ports it will need to access -- that's up to the application that uses it. > The > consumer would on the other hand want the ability to define a governing > or maximal set of permissions, either globally, per-configuration, or > per-module, which could well be done by policy (or not). Adding in some > kind of interactive process for verifying installation of modules which > would violate this maximal constraint falls more under the "gravy" > category in my mind. I think that kind of functionality is up to specific products to provide; it's well beyond the scope of a module system. > ??? Also I think I'll just keep repeating "Java EE 7" over and over again. > :-) We likely need, at a minimum, the ability to programmatically > define modules with specific permission sets to be able to support EE. Let's dig into this. The EE 7 specification defines a way (??6.2.2.6) to grant permissions to the classes inside an EE "module" -- an artifact, really, i.e., an EAR, WAR, or RAR file -- using the META-INF/permissions.xml resource. What would it mean to support this model in a modular environment? That depends on exactly how a future version of the EE specification adopts the module system. Based on conversations with the EE spec leads, I strongly suspect that it will allow code that's today packaged in JAR files inside EE artifacts (EAR/WAR/RAR) to instead be packaged as modules. I doubt it will involve replacing EAR/WAR/RAR files themselves with modules, so permissions.xml (or its successors) will remain the standard way to grant permissions to code inside EE artifacts. Does this mean that we need a way to grant permissions to specific modules, i.e., do we need to be able to treat modules as code sources? No, I don't think so. Modules will be embedded within an EE artifact, which may have its own permissions.xml file, which grants permissions to all code loaded from the artifact. If we assume that each artifact is loaded by a specific set of class loaders (maybe one, maybe many), then the protection domains can be set up based on class loaders (only) and there's no need to grant permissions on a per-module basis. >> I don't see any mention of the ability to sign a module, but provided >> there is an extensible metadata space (equivalent to the META-INF >> directory) associated with a module, then I think we are good for that >> too, no? > > I'm not sure. Signing is definitely a distribution/installation concern > though, far more than a run time concern in my opinion. I tend to agree. The Java security architecture can grant fine-grained permissions based upon the signers of a code source, but I don't think this ability is widely used. - Mark From mark.reinhold at oracle.com Mon Mar 16 20:51:06 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 16 Mar 2015 13:51:06 -0700 (PDT) Subject: Enhanced ServiceLoader? In-Reply-To: References: , <20150215172631.519378@eggemoggin.niobe.net>, , <20150309210618.1C82051227@eggemoggin.niobe.net>, Message-ID: <20150316205106.CAA005203B@eggemoggin.niobe.net> 2015/3/9 8:23 -0700, Tim Boudreau : > mark.reinhold at oracle.com: >> Tim Boudreau : >>> NetBeans did the ordering hints for years by enhancing META-INF/services >>> files with magic comments, e.g. >>> >>> #position 1000 >>> com.foo.Bar >>> >>> It would be good if ServiceLoader could grow enough heft to handle that >>> role - not necessarily identically - but the feature satisfies a proven >>> need. >> >> How is this any different than an @Ordered annotation? > > It isn't, other than that as a performance optimization, you don't have to > load the class to determine its order. > >> Doesn't it still assume that the author of a module somehow has global >> knowledge of how that module will be used in every possible scenario? > > It assumes the author of a module has at least *partial* knowledge of how > that module will be used in its *intended* scenario: > > ... I understand your argument, but I'm still not convinced that we should enhance ServiceLoader to support a feature like this. It can make sense in a large collection of closely-related modules (e.g., NetBeans), but in a scenario like that an agent with (near-)global knowledge can use selective binding to arrange providers in the correct order. If, by contrast, I'm just publishing a service provider for general use then I'm really not going to have much of a clue as to how to assign an order metric. In the general case this seems like one of those "attractive nuisance" features that will be used correctly by some developers yet incorrectly by most. In any case, as I said earlier I think this is more of a design issue than something we need to consider for the requirements. - Mark From david.lloyd at redhat.com Mon Mar 16 21:43:30 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 16 Mar 2015 16:43:30 -0500 Subject: Exporting things (Was: Re: Module-system requirements) In-Reply-To: <20150316204706.BC29152023@eggemoggin.niobe.net> References: <20150211103831.528761@eggemoggin.niobe.net>, <20150309205917.F049C511F6@eggemoggin.niobe.net>, <54FF08F7.8050901@redhat.com> <20150316204706.BC29152023@eggemoggin.niobe.net> Message-ID: <55074E82.10102@redhat.com> On 03/16/2015 03:47 PM, mark.reinhold at oracle.com wrote: > 2015/3/10 8:08 -0700, david.lloyd at redhat.com: >> On 3/9/15 3:59 PM, mark.reinhold at oracle.com wrote: >>> 2015/2/16 5:15 -0800, david.lloyd at redhat.com: >>>> On 02/15/2015 07:24 PM, mark.reinhold at oracle.com wrote: >>>>> ... >>>>> >>>>> Exposing all of a module's resources, as JAR files do today, is wrong >>>>> from the perspective of encapsulation. ... >>>>> >>>>> This suggests an additional requirement: >>>>> >>>>> - _Resource encapsulation_ --- The run-time system must ensure that the >>>>> static resource files within a module are directly accessible only by >>>>> code within that module. >>>> >>>> What you are saying here implies (to me) inspecting the call stack to >>>> ensure that the caller actually exists within the module being >>>> referenced. >>> >>> Yes. >>> >>>> I'm not certain we really want to go this far, if my >>>> interpretation is accurate. Such a rule is reasonable on the face of >>>> it, but it does more or less break all the code in existence which >>>> currently uses resources as a sort of API into a library, and this >>>> includes very common patterns for loading configuration (among many >>>> other things). >>> >>> True. To what degree should it be a goal to preserve such mechanisms? >> >> I really feel that not having this compatibility will strongly diminish >> adoption of this system, since a substantial body of existing work is >> unlikely to function properly as modules. >> >>> Must Java modules be forever constrained to look like JAR files? >> >> No, but I think that isn't really related to the point at hand, which is >> more related to the solution of specific problems rather than change for >> its own sake. If solving those problems means that modules have visible >> resources, does that mean we've tripped an invisible failure criteria by >> resembling JARs too closely in some specific way? >> >> Or put a nicer way, is there a specific problem or set of problems that >> are solved by strictly encapsulating resources, or is it more of a way >> to try a new general approach? > > Encapsulation of resources is valuable for the same reasons that > encapsulation of types is valuable. It allows you to conceal a module's > implementation details so that external code does not depend them, giving > you greater flexibility to evolve the module going forward. It also > allows you to keep adversaries from inspecting the internals of your > module to discover information that might be useful in an attack. I think you missed the thrust of my question which was "...*strictly* encapsulating...". It seems clear just from our own experience that the user generally wants to keep resources private, but the question is whether we really want to kill off the practice of using specific resources as API (in this way). But the new "Access to class files and resources" that you give should adequately cover this use case for new code at least. > We'll encapsulate types by default, and also have a way to export them to > other modules on a per-package basis. We seemed to agree earlier that a > similar style of export control for resources would be overkill, and my > suggested requirement to ensure access to a module artifact's resources > by suitably-privileged code, when such an artifact exists, appears to > cover the use cases so far discussed (as you note later in your message, > quoted below). I agree that export control for resources adds complexity, however, without it we're definitely going to be looking at a potentially much more onerous conversion process for a broad category of existing code which wants to take advantage of the benefits of Java modularity but presently relies on using class loaders to access resources for the purposes of self-configuration, which does not (in my experience or opinion) violate encapsulation any more than exporting a public class does. My agreement, such as it is, is based purely on the ensuing simplification of the module linkage algorithm, as well as the relative simplicity of *new* code (free of any legacy encumbrances) which isn't really the best of motives. I guess what I'm getting at is, I'd like to be able to see some realistic mechanism to allow such code to work *unchanged* as modules (even if a customized dynamic configuration were required to do so). Whether or not your previously-proposed method of going through the class loader is a practical approach requires a bit more consideration, I guess. It may depend on how difficult it is to control the relationship between modules and class loaders, assuming that we don't end up with a 1:1 relationship, or it may depend on making resource export control a feature of the module architecture which is not utilized by the default platform module configuration but can be used by higher layers. >>> ... >>> >>> To ensure that a distributor of a set of modules can control how service >>> dependences are resolved amongst those modules here's a new requirement, >>> for the Fundamentals section: >>> >>> - _Selective binding_ --- It must be possible to control the binding >>> process so that specific services are provided only by specific >>> providers, and in a specific order. >>> >>> Does that make sense? >> >> In this requirement, what party is doing the controlling? > > Whatever party has invoked the binding process. It might be the user, > via command-line flags or build-system configuration settings, or it > might be an application server, via the module-system API. OK I think this makes sense to me. >>> ... >>> >>> So a JBoss module can specify that it depends upon service providers from >>> one or more specific modules rather than whichever modules happen to >>> provide implementations of that service? That seems contrary to the >>> nature of services, if I understand you correctly. >> >> Right; this is however functionally similar to having the configuration >> determine the service bindings without any sub-configurations being able >> to override it, though at a whole-module granularity rather than a >> per-module-service granularity. > > I can understand having a build system or a run-time framework control > service bindings in this way, but it seems odd for a module itself to > declare that it requires a specific provider of a service. It's not too odd in practice. Many projects (especially OSS frameworks which we consume) split their code into API and implementation aspects for encapsulation reasons (both at run time and at build time), where there are typically a small constant number of implementations (maybe 1, but sometimes one primary and one or two less common/specialized, whose usage is based on run time circumstances). Then the implementations are simply imported into the API in order, the API uses its own class loader to spin up the service loader, and everything is fine. This seems preferable to putting API and implementations into the same module (which is the only practical alternative that we provide), and also frees the user from having to create "obvious" bindings between service and provider (though if there could be a way for modules to influence the binding process by specifying a default or "preferred" set of provider modules for a given service, that could be a practical/satisfactory alternative I think). >> ... >> >> In the configuration-level case though, establishment and predictability >> of order (at least on a per distribution basis) is almost certainly >> going to be needed I think, so I'd still maintain that it should be >> worked into the requirements somehow. > > Is the proposed "selective binding" requirement (above) sufficient? Probably, with the aforementioned caveat. >>> When, in the future, an application is assembled by resolving a set of >>> actual modules into a configuration, then we won't necessarily have to >>> use class loaders to mediate resource access, so I'd rather not bake that >>> into the requirements. A configuration ought to be able to relate >>> classes, modules, and module artifacts such that, given a module and a >>> class name, you can read the corresponding class file (or any other kind >>> of resource) from the module's defining artifact and load it however you >>> want. >>> >>> To capture this as a requirement, in the Development section: >>> >>> - _Access to class files and resources_ --- If a module is defined by >>> an artifact that contains class files and resources then it must be >>> possible, once that module is added to a configuration, for >>> suitably-privileged code to access those class files and resources. >> >> I like this requirement; I think it hits a good percentage of use cases. >> >>> (The "if a module is defined by an artifact" precondition allows for >>> ahead-of-time-compilation scenarios.) >>> >>> This requirement also addresses the problem of how to locate and read EE >>> deployment descriptors and similar kinds of configuration files. >> >> Great. -- - DML From david.lloyd at redhat.com Tue Mar 17 13:02:21 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 17 Mar 2015 08:02:21 -0500 Subject: Configurations... (Was: Re: Module-system requirements) In-Reply-To: <20150316204406.B396C52012@eggemoggin.niobe.net> References: <20150211103831.528761@eggemoggin.niobe.net>, <20150309205517.E1578511E9@eggemoggin.niobe.net>, <55004541.50508@redhat.com> <20150316204406.B396C52012@eggemoggin.niobe.net> Message-ID: <550825DD.9030907@redhat.com> On 03/16/2015 03:44 PM, mark.reinhold at oracle.com wrote: > 2015/3/11 6:38 -0700, david.lloyd at redhat.com: >> ... >> >> Maybe what we need is a clearer definition of "Configuration" as it is >> only defined as a side-effect of Resolution: "The resulting >> configuration contains all of the code and data required to compile or >> run the initial module." And Linking is AFAICT the first section that >> implies that Configurations actually contain modules in any way. This >> might help me understand what a tree topology actually means. > > A configuration is the result of resolution and service binding. You > can think of it as a graph, with modules for nodes, edges for module > dependences, and edges (of a different color) for service-provider > bindings. > > I'm pretty sure that a configuration, in this sense, is not analogous > to what you call a module loader in your system. A configuration can > be loaded by a single class loader, multiple class loaders, or even one > class loader per module, but the module system shouldn't mandate any > particular approach. In our case the module loader is the instigator (not the result) of resolution. Resolution happens lazily, very much like class loading. The result of resolution is thus completely abstract and not represented programmatically. This also means that there is no single resolve step at run time; instead each module is efficiently linked as it is loaded using a relatively simple traversal algorithm, regardless of what module loader contains the dependency being linked in, which allows the system to support an unbounded number of modules. Any expensive or complex resolution processes are part of build or distribution. > If you build a dynamic configuration on top of the initial configuration, > and then another on top of that, as you might do in something like an > application server, then the union of their graphs will be a DAG (at > least) but the configurations themselves are just in a list, somewhat > like a search path. You can imagine having multiple such lists sharing > prefixes, i.e., a tree of configurations, but to resolve a dependence in > any particular configuration you only need to consult its parent > configurations. OK, but if (say) a dynamic configuration can override service-provider bindings, then how does a parent configuration know to use the overridden bindings for services it provides? I can't imagine any scheme at all which would necessitate that the entire configuration be a list. Maybe conflating module dependencies and service provider configuration into one "thing" is an error? > If configurations can be related in a more general way, i.e., in a DAG, > then the resolution algorithm becomes more complex. Do we actually need > that? I've yet to see any use cases. I don't know about framing in terms of configurations, because we don't do that. But I think that modules from one system should be able to express dependencies on other modules from any other system regardless of their relationship, at least on a programmatic basis. This is necessary if you want two completely different module systems to interact, because you cannot always assume that (for example) one namespace can just be polluted with names from another (parent) namespace, nor can you assume that a given configuration would only ever want to interact (link) with one other configuration (this is akin to the logic of moving away from a strict class loader hierarchy). We made this mistake early on, and fixing it resulted in a much better system, which allowed us to have a more logical arrangement between OSGi, EE, the JDK, distribution modules, extension modules, and filesystem JAR modules without worrying about what ended up being a pretty arbitrary constraint. Each module loader interacted with the others in specific and completely separate ways (which also greatly helps in terms of letting the container bind module dependencies with service dependencies, to ensure that things are actually started in the right order). For example if I want to link in a filesystem JAR, I use Class-Path (in the MANIFEST). However, the Extension-List mechanism is totally different (and has a different namespace), our distribution module Dependencies mechanism is different again, and OSGi is *completely* different in every possible way. So, there was really no reason to force all of these resolution strategies to go down the same pipe. It's not really clear to me in any event what makes the resolution algorithm more complex in graph vs. list form; as of yet we don't really have a description of what the algorithm you have in mind actually does, and how and why it does it. So there's probably a more in-depth discussion to be had. But if there's a way for any configuration to contain module dependencies on any other configuration, then maybe the constraint is only practically limited to service resolution and I'm worried about nothing. -- - DML From david.lloyd at redhat.com Tue Mar 17 13:19:36 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 17 Mar 2015 08:19:36 -0500 Subject: Module-system requirements In-Reply-To: <20150316204606.B946C5201E@eggemoggin.niobe.net> References: <20150211103831.528761@eggemoggin.niobe.net>, <550041E3.4080901@redhat.com>, <5500B7A3.5070308@redhat.com> <20150316204606.B946C5201E@eggemoggin.niobe.net> Message-ID: <550829E8.7040900@redhat.com> On 03/16/2015 03:46 PM, mark.reinhold at oracle.com wrote: > 2015/3/11 2:46 -0700, david.lloyd at redhat.com: >> On 03/11/2015 08:23 AM, David M. Lloyd wrote: >>> On 03/09/2015 03:56 PM, mark.reinhold at oracle.com wrote: >>>> ... >>>> >>>> Either way, this use case suggests a new requirement for the Development >>>> section: >>>> >>>> - _Annotation digests_ --- When packaging a module into an artifact it >>>> must be possible to augment its definition with a summary of all of >>>> the type- and package-level annotations that are present in the >>>> module, together with an indication of the types and packages to >>>> which each annotation applies. This digest must be easily readable >>>> from module artifacts, and at run time those parts of it related to >>>> annotations retained for run time must be available through an >>>> appropriate reflective API. >>>> >>>> Comments? >>> >>> If we're looking to extend this into the domain of Java EE annotations >>> (for example), we'd also have to add method and field annotations to the >>> list. >> >> Actually this is not true after all. Other than possibly examining >> constructor annotations to decide if a class is a CDI managed bean that is. > > Do such constructor annotations really need to be in the digest? Or > do such classes already carry class-level annotations to indicate > that they're to be managed by the CDI framework? By my understanding, such classes do not need class-level annotations; however I must say that the interactions between CDI, the EE platform spec, the injection spec, and the common annotations spec are (in my opinion) murky at best (and are often the subject of much internal debate), so this might be a better question for the Java EE 7 EG. >>> And SE 8 type annotations might add some strange color to this as >>> well. > > Hrm. I suppose we could summarize all the annotations in a module, > regardless of the kinds of elements they're applied to, but is that > overkill? Yeah probably; maybe what is needed is an @Indexed meta-annotation... We have a project which specifically creates annotation indexes for JARs for use by our containers, including member and type annotations. Even though our developer spent a great deal of time and effort to minimize the size of the generated indexes, they are still often quite large with respect to the original classes (on the order of 15% uncompressed for a couple JARs that I can see test results for, though by storing the index in the JAR we can shrink the total growth to under 10%). Indexing rt.jar from a recent JDK resulted in a file size of 4.1 MB. If a more optimal-than-JAR format is established, especially one which flattened the constant pool across all classes, the relative percentage would probably be even larger. Having an @Indexed annotation would ease this quite substantially though; most of the annotations in the index aren't actually directly useful or relevant to the consumers of the index. -- - DML From david.lloyd at redhat.com Tue Mar 17 13:46:26 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 17 Mar 2015 08:46:26 -0500 Subject: Interoperation (Was: Re: Module-system requirements) In-Reply-To: <20150316204306.B09C65200B@eggemoggin.niobe.net> References: <20150211103831.528761@eggemoggin.niobe.net>, <20150309205417.DE06C511E7@eggemoggin.niobe.net>, <55003FBB.5050705@redhat.com> <20150316204306.B09C65200B@eggemoggin.niobe.net> Message-ID: <55083032.9060609@redhat.com> On 03/16/2015 03:43 PM, mark.reinhold at oracle.com wrote: > 2015/3/11 6:14 -0700, david.lloyd at redhat.com: >> On 03/09/2015 03:54 PM, mark.reinhold at oracle.com wrote: >>> 2015/2/16 5:57 -0800, david.lloyd at redhat.com: >>>> ... >>>> >>>> In the case of both Java EE and OSGi, I think it's safe to say that you >>>> need to apply a strict 1:1 mapping between modules and class loaders for >>>> this to work. >>> >>> I don't think we can mandate this. To divide the SE platform into >>> modules in a compatible way requires the ability to load multiple modules >>> via the same class loader, so as to respect long-standing assumptions >>> about which platform classes are loaded by the built-in class loaders >>> (i.e., the bootstrap and extension loaders). In an early Jigsaw >>> prototype we tried the 1:1 module/loader mapping for platform modules >>> and found it unworkable from a compatibility perspective. >> >> I went through the jigsaw list archives looking for this discussion and >> didn't turn up anything; was this discussed on-list and I am just >> failing to find it? > > I don't recall it being discussed on-list. > > Perhaps the biggest issue we ran into is that there's lots of code, both > in the JDK and out in the wild, that does tests of the form > > if (ob.getClass().getClassLoader() == null) { ... } > > which, of course, succeeds for classes loaded by the bootstrap loader. > > We expect to divide the bootstrap classes into multiple modules (see JEP > 200 [1] and the modules.xml file in JDK 9 [2] for the current prototype > definition), which would break all such tests if each module had its own > class loader. > > [1] http://openjdk.java.net/jeps/200 > [2] http://hg.openjdk.java.net/jdk9/jdk9/file/tip/modules.xml So it sounds like there is a more concrete underlying requirement here: - Define a set of "bootstrap modules" that must all return null for getClassLoader() - Using ClassLoader.getSystemResource*(*) and Class.forName(*,*,null) must search all "bootstrap modules" - The "java.base" module must be loaded from the bootstrap class loader for implementation reasons - "Bootstrap modules" need not actually be loaded by the bootstrap CL if the above requirements are met Side requirements: - A mechanism is required to get the module for a given class (if module = class loader, this means non-null for bootstrap modules (maybe for java.base as well?)) The way this would play out with a module system like ours is, I'd still have 1:1 class loaders, but I'd have three module loaders at boot: the first only "loads" (wraps really) "java.base"; the second loads all the "bootstrap modules" which have the above special requirements and (uniquely) privileges to carry them out; and the third layer is the "regular" module loader which is used to load the remaining JDK modules (application distribution modules would probably be a layer above, I guess). The module loader implementations would live in "java.base". IOW I don't think that the above constraints alone actually mandate a relationship *or* a lack of relationship between modules and class loaders. >>> Aside from the SE platform itself, I don't think we can force all >>> applications to adopt the unique-class-loader-per-module approach. >>> Oracle's WebLogic application server, e.g., doesn't do that today and >>> it'd be difficult, if not impossible, to modify it to do so without >>> breaking existing applications. >> >> I guess I can't really comment on what WebLogic does or does not do. I >> can say however that we were able to move from a flat classpath >> architecture to a modular architecture like this without breaking things >> due to the modularization. The Java EE spec is pretty clear about what >> needs to be visible to what, and by translating this information into >> module dependency information, we were able to make a smooth transition. >> I can't really imagine an application that would be broken unless they >> were relying on some specific non-spec behavior as the EE spec says: >> >> "[...] A Java EE application should not assume that all components in >> the application will be available on the class path of the application >> at run time. Each component might be loaded into a separate class loader >> with a separate namespace. [...]" > > The concern with WebLogic is that existing customers rely upon aspects of > its particular class-loader hierarchy, beyond what is guaranteed by the > Java EE specification. Changing it incompatibly would break those > customers' applications. > >> Note that in the EE spec, the term "component" has a finer granularity >> than "module", i.e. there are 1..N components per module, whereas an >> "application" can contain 1 or more modules. > > Yes, and I don't think that the module concept we're discussing here will > wind up corresponding exactly to what EE calls "modules". The two above statements, taken together, don't really exclude class loader per module. Think of it instead as "module per class loader" - any place you use a class loader today, you could instead use a module. The parent relationship of a module class loader could easily be set up to reflect any implementation realities that must be dealt with. However if "modular EE" is indeed a completely separate mode of operation then it seems like WebLogic (or any other vendor) would not have anything to worry about. We'd still be free to use SE modules for EE modules in "legacy mode", whereas other containers would be free to continue as they have been. > >> ... >> >>> I suggest the module system be as neutral as possible with respect to how >>> modules and class loaders are related. A complex application should be >>> able to spin up a class loader for every module it loads, if needed >>> (excepting non-upgradeable platform classes), but not all applications >>> should be forced to do so. >>> >>> New requirement, for the Dynamic Configuration section: >>> >>> - _Control of class loaders_ --- An application that loads a module >>> into a dynamic configuration must be able to specify the class loader >>> to be used to load that module. That loader need not be the same as >>> the loader used to load any other module in the configuration. >> >> This goes part of the way, for sure. I'm not 100% sold yet though. >> It's not easy to let go of something that (from our perspective) is >> tried and true. > > This requirement allows you to use the 1:1 model above SE itself, if you > want that, but it doesn't force everyone else to do so. I just wonder how much difference there really is between composing single modules out of multiple collections of classes and resources (as we do), versus composing single class loaders out of modules (as you propose). Given the *high* amount of conceptual overlap between modules and class loaders, I'm still inclined to reject the idea that we need another layer there based on the requirements put forth. While giving the option to define the association sounds good, the fact that the option even *exists* means that everything using class loaders to load classes and resources (read: everything, period) is now broken, but I don't feel like there's a good justification for this that isn't either self-referential or a broadening of the actual minimum requirements (see above). >>> ... >>> >>>> OSGi specifically requires that bundle class loaders implement an OSGi >>>> spec interface, implying that the module class loader class must be >>>> extensible (as I said a couple of years back though, we make most of our >>>> methods final on the module class loader to disallow any really bizarre >>>> behavior, and this works well in practice). >>> >>> Which OSGi interface are you thinking of here? >> >> In OSGi Core 6.0.0 section 3.9.9: >> >> "Class Loader - An OSGi framework must ensure that the class loader of a >> class that comes from >> a bundle implements the BundleReference interface." >> >> The BundleReference interface has a single getBundle() method on it. > > Ah, got it. > >>>> We also had to implement a >>>> pre- and/or post-class-define hook per module to allow some OSGi magic >>>> to happen. >>> >>> These are non-final protected methods in your module class loader? >>> >>> Could you forward a description of these methods to the list? >> >> Yeah, they look like this: >> >> protected void preDefine(ClassSpec classSpec, String className) { >> } >> >> protected void postDefine(ClassSpec classSpec, Class definedClass) { >> } >> >> ClassSpec contains the bytes and CodeSource of the class. The hooks >> don't have opportunity to do anything like rewrite the class bytes, but >> we also accept a java.lang.instrument.ClassFileTransformer instance in >> the module definition for that purpose (though we aren't using it at >> this point). Our OSGi implementation only ended up using one of these >> hooks (preDefine I think). They were added to support lazy bundle >> activation. > > Okay. > >>>> Finally there is a per-module fallback resource loader that >>>> our OSGi implementation used for one particular part of the spec. >>> >>> Which part, in particular? >> >> I'll have to go back through my notes to say for sure, but I think we >> used it for a couple of purposes: >> >> 1) To support dynamic imports: https://issues.jboss.org/browse/MODULES-30 >> 2) As a general hook to allow for OSGi-style "lazy" packages (the loader >> would examine the package of the class in question, recalculate the >> package set, and then retry the class load): >> https://issues.jboss.org/browse/MODULES-63 > > Interesting. > >>> To what extent does OSGi assume that bundles "export" their resources? >>> If we decide that a module's resources are not available outside the >>> module itself then will the approach of extending a module class loader >>> to be an OSGi bundle loader even work? >> >> This I will defer to an OSGi expert. My understanding is that OSGi >> largely operates on a package basis only, but I don't know how resources >> within the packages fit in to this picture (if at all). > > ??3.9.6 and ??10.1.5.13 of the OSGi R6 specification imply that all the > resources in a bundle are available to anyone with access to the bundle > object itself and, if running with a security manager, the appropriate > OSGi AdminPermission(s). > > If we adopt the "Access to class files and resources" requirement which I > suggested in the "exporting things" thread then an OSGi implementation > built on top of the platform module system could use that to expose the > resources of a module artifact as needed. > >>>> Java EE's requirements are much simpler and are satisfied by >>>> module-granularity dependencies and (in certain cases) resource importing. >>> >>> Aside from things like deployment descriptors, which are interpreted by >>> Java EE implementations, does the EE specification require EE "modules" >>> to export resources to other modules? >> >> The Java EE 7 platform specification does seem to do so, with language >> such as "Components in the web container may have access to the >> following classes and resources. [...] The classes and resources >> accessible to any other web modules included in the >> same ear file, as described above." >> >> The spec also seems to use the term "contents" interchangeably with >> "classes and resources". > > The Java EE specification, just like OSGi, is explicitly built on the JAR > file format and derivations thereof, so I think the same solution applies > for Java EE: An EE implementation can access a module's artifact and > expose the artifact's resources as needed. > > - Mark > -- - DML From david.lloyd at redhat.com Tue Mar 17 14:34:17 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 17 Mar 2015 09:34:17 -0500 Subject: Security In-Reply-To: <20150316205006.C6D9952036@eggemoggin.niobe.net> References: <20150211103831.528761@eggemoggin.niobe.net>, , <54FF106C.5090901@redhat.com> <20150316205006.C6D9952036@eggemoggin.niobe.net> Message-ID: <55083B69.9090301@redhat.com> On 03/16/2015 03:50 PM, mark.reinhold at oracle.com wrote: > 2015/3/10 8:40 -0700, david.lloyd at redhat.com: >> On 3/10/15 9:55 AM, Tim Ellison wrote: >>> ... >>> >>> David, is it not sufficient that the java.policy file can contain a >>> code source url specifying jrt:/$MODULE ? This gives a mechanism for >>> permissions to be granted to code associated with a module >>> irrespective of the artifact that delivers the module. >> >> I believe not, for these reasons: >> >> ??? The policy file is not AFAIK actually standardized in the SE platform, >> though (of course) the policy mechanism itself is. > > Correct. > >> ??? The "jrt" URL scheme, as I understand it, applies specifically to the >> JDK runtime and not generally to modules. > > The "jrt" scheme can be used to name any module linked statically into > the run-time image, whether an SE/JDK module or a library or application > module. It cannot be used to name modules loaded at run time, so from a > security perspective (where URLs are needed) it's not a general solution > to the module-naming problem. Indeed I maintain that each module loading configuration should have control over module naming and assignment of code source and URL scheme, as different systems have different requirements. I don't think that "jrt" *must* resolve libraries and application modules, but it certainly *could*; OTOH that job could also be done by another layer. For example I could have my container define a "deployment:/myapp.ear/helloworld.war" kind of URL scheme for modules which correspond to deployments. I could use a URN like "ext:my-extension" to map to legacy extensions, and so on. Each scheme would have a different mapping strategy, and each module configuration may indeed have different naming constraints or rules. >> ??? The JDK runtime itself effectively has AllPermission as far as I >> understand. > > No, the permissions granted to code not loaded by the bootstrap class > loader are limited. Have a look at the JDK 9 default java.policy file: > > http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/tip/src/java.base/share/conf/security/java.policy > > (This does, however, seem like a side issue.) I should have said "the bootstrap class loader portion of the JDK...". But yes this is a side issue for sure. >> ??? Assuming that each configuration (I guess?) gets a unique URL scheme >> which applies to it, > > I don't think each configuration (= set of resolved modules) would get > its own URL scheme, or even a URL. I could imagine each module in a > system, whether statically linked or dynamically loaded, having a unique > URL, but it's yet not clear to me whether that's needed. Another side discussion I guess. >> using the global policy exclusively is still >> non-optimal because it loses a substantial amount of information: the >> author of the module knows what permissions the module would require, >> whereas the consumer of the module does not have this knowledge. > > Is this really true, in general? If I ship, say, a JDBC driver then I > won't know ahead of time which network hosts and ports it will need to > access -- that's up to the application that uses it. True; but the JDBC driver author would definitely know that it needs to be able to access *some* hosts and ports. It would define the broad permission to do so, subject to the consumer's narrowing if they do not trust that code with that privilege for some reason (for example perhaps it is buggy or tries to connect to some other undesired service or something like that). >> The >> consumer would on the other hand want the ability to define a governing >> or maximal set of permissions, either globally, per-configuration, or >> per-module, which could well be done by policy (or not). Adding in some >> kind of interactive process for verifying installation of modules which >> would violate this maximal constraint falls more under the "gravy" >> category in my mind. > > I think that kind of functionality is up to specific products to provide; > it's well beyond the scope of a module system. Sure, it just illustrates the point that the use cases behind the requirements in EE7.6.2.2 which allow the module to define permissions and the container to restrict or enhance them are legitimate and useful, so they should at least be *possible* to implement within the module system. If the module system were to prevent any setting of permission other than the Policy, that would be very unfortunate. >> ??? Also I think I'll just keep repeating "Java EE 7" over and over again. >> :-) We likely need, at a minimum, the ability to programmatically >> define modules with specific permission sets to be able to support EE. > > Let's dig into this. > > The EE 7 specification defines a way (??6.2.2.6) to grant permissions to > the classes inside an EE "module" -- an artifact, really, i.e., an EAR, > WAR, or RAR file -- using the META-INF/permissions.xml resource. > > What would it mean to support this model in a modular environment? That > depends on exactly how a future version of the EE specification adopts > the module system. Based on conversations with the EE spec leads, I > strongly suspect that it will allow code that's today packaged in JAR > files inside EE artifacts (EAR/WAR/RAR) to instead be packaged as > modules. I doubt it will involve replacing EAR/WAR/RAR files themselves > with modules, so permissions.xml (or its successors) will remain the > standard way to grant permissions to code inside EE artifacts. > > Does this mean that we need a way to grant permissions to specific > modules, i.e., do we need to be able to treat modules as code sources? > > No, I don't think so. Modules will be embedded within an EE artifact, > which may have its own permissions.xml file, which grants permissions to > all code loaded from the artifact. If we assume that each artifact is > loaded by a specific set of class loaders (maybe one, maybe many), then > the protection domains can be set up based on class loaders (only) and > there's no need to grant permissions on a per-module basis. I don't think that treating modules as code sources is necessarily the conclusion to draw here. Since JBoss Modules has a high correspondence between artifacts and modules, assigning permissions to modules is easy and convenient, and they inherit automatically to their constituent code sources. But even if the ability were defined in terms of establishing permissions on the code sources directly, that would suffice as well since the former could always be implemented in terms of the latter. Requiring the global Policy to get involved is mostly just useless overhead though, and forces coordination between all module loading parties that want to be able to define permissions. I'd settle for allowing the definition of ProtectionDomains to be customized as they can be today though. >>> I don't see any mention of the ability to sign a module, but provided >>> there is an extensible metadata space (equivalent to the META-INF >>> directory) associated with a module, then I think we are good for that >>> too, no? >> >> I'm not sure. Signing is definitely a distribution/installation concern >> though, far more than a run time concern in my opinion. > > I tend to agree. The Java security architecture can grant fine-grained > permissions based upon the signers of a code source, but I don't think > this ability is widely used. I agree, most of the time it seems that the code source URL is used by itself in security policies. -- - DML From mark.reinhold at oracle.com Wed Mar 25 22:47:12 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 25 Mar 2015 15:47:12 -0700 (PDT) Subject: Exporting things (Was: Re: Module-system requirements) In-Reply-To: <55074E82.10102@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, <20150316204706.BC29152023@eggemoggin.niobe.net>, <55074E82.10102@redhat.com> Message-ID: <20150325224712.A7E57534BC@eggemoggin.niobe.net> 2015/3/16 2:43 -0700, david.lloyd at redhat.com: > On 03/16/2015 03:47 PM, mark.reinhold at oracle.com wrote: >> ... >> >> We'll encapsulate types by default, and also have a way to export them to >> other modules on a per-package basis. We seemed to agree earlier that a >> similar style of export control for resources would be overkill, and my >> suggested requirement to ensure access to a module artifact's resources >> by suitably-privileged code, when such an artifact exists, appears to >> cover the use cases so far discussed (as you note later in your message, >> quoted below). > > I agree that export control for resources adds complexity, however, > without it we're definitely going to be looking at a potentially much > more onerous conversion process for a broad category of existing code > which wants to take advantage of the benefits of Java modularity but > presently relies on using class loaders to access resources for the > purposes of self-configuration, which does not (in my experience or > opinion) violate encapsulation any more than exporting a public class > does. My agreement, such as it is, is based purely on the ensuing > simplification of the module linkage algorithm, as well as the relative > simplicity of *new* code (free of any legacy encumbrances) which isn't > really the best of motives. > > I guess what I'm getting at is, I'd like to be able to see some > realistic mechanism to allow such code to work *unchanged* as modules > (even if a customized dynamic configuration were required to do so). > Whether or not your previously-proposed method of going through the > class loader is a practical approach requires a bit more consideration, > I guess. It may depend on how difficult it is to control the > relationship between modules and class loaders, assuming that we don't > end up with a 1:1 relationship, or it may depend on making resource > export control a feature of the module architecture which is not > utilized by the default platform module configuration but can be used by > higher layers. I agree that we should try to reduce the migration effort for such cases. To capture this, I'll revise the proposed "resource encapsulation" requirement as follows: - _Resource encapsulation_ --- The run-time system must ensure that the static resource files within a module are directly accessible only by code within that module. ADD: The existing resource-access APIs should continue to work as they do today when used to access a module's own resources. >>>> ... >>>> >>>> So a JBoss module can specify that it depends upon service providers from >>>> one or more specific modules rather than whichever modules happen to >>>> provide implementations of that service? That seems contrary to the >>>> nature of services, if I understand you correctly. >>> >>> Right; this is however functionally similar to having the configuration >>> determine the service bindings without any sub-configurations being able >>> to override it, though at a whole-module granularity rather than a >>> per-module-service granularity. >> >> I can understand having a build system or a run-time framework control >> service bindings in this way, but it seems odd for a module itself to >> declare that it requires a specific provider of a service. > > It's not too odd in practice. Many projects (especially OSS frameworks > which we consume) split their code into API and implementation aspects > for encapsulation reasons (both at run time and at build time), where > there are typically a small constant number of implementations (maybe 1, > but sometimes one primary and one or two less common/specialized, whose > usage is based on run time circumstances). Then the implementations are > simply imported into the API in order, the API uses its own class loader > to spin up the service loader, and everything is fine. > > This seems preferable to putting API and implementations into the same > module (which is the only practical alternative that we provide), and > also frees the user from having to create "obvious" bindings between > service and provider (though if there could be a way for modules to > influence the binding process by specifying a default or "preferred" set > of provider modules for a given service, that could be a > practical/satisfactory alternative I think). Interesting. That would essentially be the dual of the provider-declared order we discussed in the "Enhanced ServiceLoader" thread. As with that feature, I think this is more of a design issue than something we need to capture in the requirements. - Mark From mark.reinhold at oracle.com Wed Mar 25 22:48:12 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 25 Mar 2015 15:48:12 -0700 (PDT) Subject: Security In-Reply-To: <55083B69.9090301@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, <20150316205006.C6D9952036@eggemoggin.niobe.net>, <55083B69.9090301@redhat.com> Message-ID: <20150325224812.AAD92534C2@eggemoggin.niobe.net> 2015/3/17 7:34 -0700, david.lloyd at redhat.com: > On 03/16/2015 03:50 PM, mark.reinhold at oracle.com wrote: >> The EE 7 specification defines a way (????6.2.2.6) to grant permissions to >> the classes inside an EE "module" -- an artifact, really, i.e., an EAR, >> WAR, or RAR file -- using the META-INF/permissions.xml resource. >> >> What would it mean to support this model in a modular environment? That >> depends on exactly how a future version of the EE specification adopts >> the module system. Based on conversations with the EE spec leads, I >> strongly suspect that it will allow code that's today packaged in JAR >> files inside EE artifacts (EAR/WAR/RAR) to instead be packaged as >> modules. I doubt it will involve replacing EAR/WAR/RAR files themselves >> with modules, so permissions.xml (or its successors) will remain the >> standard way to grant permissions to code inside EE artifacts. >> >> Does this mean that we need a way to grant permissions to specific >> modules, i.e., do we need to be able to treat modules as code sources? >> >> No, I don't think so. Modules will be embedded within an EE artifact, >> which may have its own permissions.xml file, which grants permissions to >> all code loaded from the artifact. If we assume that each artifact is >> loaded by a specific set of class loaders (maybe one, maybe many), then >> the protection domains can be set up based on class loaders (only) and >> there's no need to grant permissions on a per-module basis. > > I don't think that treating modules as code sources is necessarily the > conclusion to draw here. Since JBoss Modules has a high correspondence > between artifacts and modules, assigning permissions to modules is easy > and convenient, and they inherit automatically to their constituent code > sources. But even if the ability were defined in terms of establishing > permissions on the code sources directly, that would suffice as well > since the former could always be implemented in terms of the latter. > Requiring the global Policy to get involved is mostly just useless > overhead though, and forces coordination between all module loading > parties that want to be able to define permissions. > > I'd settle for allowing the definition of ProtectionDomains to be > customized as they can be today though. Here's a new requirement to capture that: - _Protection domains (NEW)_ --- At run time it must be possible to associate the code in modules with protection domains, for the purpose of granting security permissions. - Mark From mark.reinhold at oracle.com Wed Mar 25 22:49:12 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 25 Mar 2015 15:49:12 -0700 (PDT) Subject: Interoperation (Was: Re: Module-system requirements) In-Reply-To: <55083032.9060609@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, <20150316204306.B09C65200B@eggemoggin.niobe.net>, <55083032.9060609@redhat.com> Message-ID: <20150325224912.AE9CB534C8@eggemoggin.niobe.net> 2015/3/17 6:46 -0700, david.lloyd at redhat.com: > On 03/16/2015 03:43 PM, mark.reinhold at oracle.com wrote: >> 2015/3/11 6:14 -0700, david.lloyd at redhat.com: >>> On 03/09/2015 03:54 PM, mark.reinhold at oracle.com wrote: >>>> 2015/2/16 5:57 -0800, david.lloyd at redhat.com: >>>>> ... >>>>> >>>>> In the case of both Java EE and OSGi, I think it's safe to say that you >>>>> need to apply a strict 1:1 mapping between modules and class loaders for >>>>> this to work. >>>> >>>> I don't think we can mandate this. To divide the SE platform into >>>> modules in a compatible way requires the ability to load multiple modules >>>> via the same class loader, so as to respect long-standing assumptions >>>> about which platform classes are loaded by the built-in class loaders >>>> (i.e., the bootstrap and extension loaders). In an early Jigsaw >>>> prototype we tried the 1:1 module/loader mapping for platform modules >>>> and found it unworkable from a compatibility perspective. >> >> ... >> >> Perhaps the biggest issue we ran into is that there's lots of code, both >> in the JDK and out in the wild, that does tests of the form >> >> if (ob.getClass().getClassLoader() == null) { ... } >> >> which, of course, succeeds for classes loaded by the bootstrap loader. >> >> ... > > So it sounds like there is a more concrete underlying requirement here: > > - Define a set of "bootstrap modules" that must all return null for > getClassLoader() > - Using ClassLoader.getSystemResource*(*) and Class.forName(*,*,null) > must search all "bootstrap modules" > - The "java.base" module must be loaded from the bootstrap class loader > for implementation reasons > - "Bootstrap modules" need not actually be loaded by the bootstrap CL if > the above requirements are met > > ... > > IOW I don't think that the above constraints alone actually mandate a > relationship *or* a lack of relationship between modules and class loaders. Agreed -- they merely suggest the latter approach. There are additional considerations which I could describe here, but I fear we're falling into a morass of design issues when we're still trying to nail down the requirements. We already have a requirement to enable "Compatible Java Platform modularization". Whether that requirement mandates a design in which multiple modules can be loaded by a single class loader is, I suggest, an issue we can discuss later on. >>> ... >>> >>> Note that in the EE spec, the term "component" has a finer granularity >>> than "module", i.e. there are 1..N components per module, whereas an >>> "application" can contain 1 or more modules. >> >> Yes, and I don't think that the module concept we're discussing here will >> wind up corresponding exactly to what EE calls "modules". > > The two above statements, taken together, don't really exclude class > loader per module. Nor do they mandate that approach. > ... > >>>> I suggest the module system be as neutral as possible with respect to how >>>> modules and class loaders are related. A complex application should be >>>> able to spin up a class loader for every module it loads, if needed >>>> (excepting non-upgradeable platform classes), but not all applications >>>> should be forced to do so. >>>> >>>> New requirement, for the Dynamic Configuration section: >>>> >>>> - _Control of class loaders_ --- An application that loads a module >>>> into a dynamic configuration must be able to specify the class loader >>>> to be used to load that module. That loader need not be the same as >>>> the loader used to load any other module in the configuration. >>> >>> This goes part of the way, for sure. I'm not 100% sold yet though. >>> It's not easy to let go of something that (from our perspective) is >>> tried and true. >> >> This requirement allows you to use the 1:1 model above SE itself, if you >> want that, but it doesn't force everyone else to do so. > > I just wonder how much difference there really is between composing > single modules out of multiple collections of classes and resources (as > we do), versus composing single class loaders out of modules (as you > propose). Given the *high* amount of conceptual overlap between modules > and class loaders, I'm still inclined to reject the idea that we need > another layer there based on the requirements put forth. As things stand, we've identified a requirement (quoted above) that sophisticated applications must be able to control how modules are associated with class loaders. I haven't seen a compelling reason to adopt a further requirement that every module must be loaded by a unique class loader, which seems to be what you're suggesting. The fact that some module systems already do that isn't really relevant, from a requirements perspective, though why and how they do that will no doubt be of interest in our design discussions later on. - Mark From mark.reinhold at oracle.com Wed Mar 25 22:50:12 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 25 Mar 2015 15:50:12 -0700 (PDT) Subject: Configurations... (Was: Re: Module-system requirements) In-Reply-To: <550825DD.9030907@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, <20150316204406.B396C52012@eggemoggin.niobe.net>, <550825DD.9030907@redhat.com> Message-ID: <20150325225012.B20F2534CD@eggemoggin.niobe.net> 2015/3/17 6:02 -0700, david.lloyd at redhat.com: > On 03/16/2015 03:44 PM, mark.reinhold at oracle.com wrote: >> 2015/3/11 6:38 -0700, david.lloyd at redhat.com: >>> ... >>> >>> Maybe what we need is a clearer definition of "Configuration" as it is >>> only defined as a side-effect of Resolution: "The resulting >>> configuration contains all of the code and data required to compile or >>> run the initial module." And Linking is AFAICT the first section that >>> implies that Configurations actually contain modules in any way. This >>> might help me understand what a tree topology actually means. >> >> A configuration is the result of resolution and service binding. You >> can think of it as a graph, with modules for nodes, edges for module >> dependences, and edges (of a different color) for service-provider >> bindings. >> >> I'm pretty sure that a configuration, in this sense, is not analogous >> to what you call a module loader in your system. A configuration can >> be loaded by a single class loader, multiple class loaders, or even one >> class loader per module, but the module system shouldn't mandate any >> particular approach. To clarify further: The term "configuration", as used in the requirements document, is meant to be abstract. It's the concept of a set of modules related via the resolution and service-binding processes -- which need not be performed solely at startup, nor necessarily as single steps. The requirements don't mandate a programmatic representation of those relations, though having one can be convenient. > In our case the module loader is the instigator (not the result) of > resolution. Resolution happens lazily, very much like class loading. > The result of resolution is thus completely abstract and not represented > programmatically. This also means that there is no single resolve step > at run time; instead each module is efficiently linked as it is loaded > using a relatively simple traversal algorithm, regardless of what module > loader contains the dependency being linked in, which allows the system > to support an unbounded number of modules. Any expensive or complex > resolution processes are part of build or distribution. Based on what you describe, it sounds like your module loaders can be viewed as agents that incrementally compute configurations, though they don't represent them explicitly. >> If you build a dynamic configuration on top of the initial configuration, >> and then another on top of that, as you might do in something like an >> application server, then the union of their graphs will be a DAG (at >> least) but the configurations themselves are just in a list, somewhat >> like a search path. You can imagine having multiple such lists sharing >> prefixes, i.e., a tree of configurations, but to resolve a dependence in >> any particular configuration you only need to consult its parent >> configurations. > > OK, but if (say) a dynamic configuration can override service-provider > bindings, then how does a parent configuration know to use the > overridden bindings for services it provides? I don't know, but I think that's a design issue we can work out later on. The proposed "selective binding" requirement already gives applications the ability to control how service providers are chosen when using dynamic configurations. >> If configurations can be related in a more general way, i.e., in a DAG, >> then the resolution algorithm becomes more complex. Do we actually need >> that? I've yet to see any use cases. > > I don't know about framing in terms of configurations, because we don't > do that. But I think that modules from one system should be able to > express dependencies on other modules from any other system regardless > of their relationship, at least on a programmatic basis. This is > necessary if you want two completely different module systems to > interact, because you cannot always assume that (for example) one > namespace can just be polluted with names from another (parent) > namespace, nor can you assume that a given configuration would only ever > want to interact (link) with one other configuration (this is akin to > the logic of moving away from a strict class loader hierarchy). Okay, I'll rewrite the "Nested dynamic configurations" requirement as: - _Composable dynamic configurations_ --- An application must be able to create a dynamic configuration that relates to one or more existing configurations rather than solely to the application's initial configuration. - Mark From mark.reinhold at oracle.com Wed Mar 25 22:51:12 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 25 Mar 2015 15:51:12 -0700 (PDT) Subject: Module-system requirements In-Reply-To: <550829E8.7040900@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, <20150316204606.B946C5201E@eggemoggin.niobe.net>, <550829E8.7040900@redhat.com> Message-ID: <20150325225112.B55C0534D2@eggemoggin.niobe.net> 2015/3/17 6:19 -0700, david.lloyd at redhat.com: > On 03/16/2015 03:46 PM, mark.reinhold at oracle.com wrote: >> 2015/3/11 2:46 -0700, david.lloyd at redhat.com: >>> On 03/11/2015 08:23 AM, David M. Lloyd wrote: >>>> On 03/09/2015 03:56 PM, mark.reinhold at oracle.com wrote: >>>>> ... >>>>> >>>>> Either way, this use case suggests a new requirement for the Development >>>>> section: >>>>> >>>>> - _Annotation digests_ --- When packaging a module into an artifact it >>>>> must be possible to augment its definition with a summary of all of >>>>> the type- and package-level annotations that are present in the >>>>> module, together with an indication of the types and packages to >>>>> which each annotation applies. This digest must be easily readable >>>>> from module artifacts, and at run time those parts of it related to >>>>> annotations retained for run time must be available through an >>>>> appropriate reflective API. >>>>> >>>>> Comments? >>>> >>>> If we're looking to extend this into the domain of Java EE annotations >>>> (for example), we'd also have to add method and field annotations to the >>>> list. >>> >>> Actually this is not true after all. Other than possibly examining >>> constructor annotations to decide if a class is a CDI managed bean that is. >> >> Do such constructor annotations really need to be in the digest? Or >> do such classes already carry class-level annotations to indicate >> that they're to be managed by the CDI framework? > > By my understanding, such classes do not need class-level annotations; > however I must say that the interactions between CDI, the EE platform > spec, the injection spec, and the common annotations spec are (in my > opinion) murky at best (and are often the subject of much internal > debate), so this might be a better question for the Java EE 7 EG. > >>>> And SE 8 type annotations might add some strange color to this as >>>> well. >> >> Hrm. I suppose we could summarize all the annotations in a module, >> regardless of the kinds of elements they're applied to, but is that >> overkill? > > Yeah probably; maybe what is needed is an @Indexed meta-annotation... Yes, that could be a good approach. I'll mention it as a potential design in the requirements document. - Mark From mark.reinhold at oracle.com Wed Mar 25 22:52:12 2015 From: mark.reinhold at oracle.com (Mark Reinhold) Date: Wed, 25 Mar 2015 15:52:12 -0700 (PDT) Subject: Module-system requirements, draft 2 Message-ID: <20150325225212.B8AB0534D7@eggemoggin.niobe.net> Thanks for all the comments and suggestions. I've posted draft 2 of the requirements document, here: http://openjdk.java.net/projects/jigsaw/spec/reqs/02 If you'd like to comment on the text in-line, the Markdown source is here: http://openjdk.java.net/projects/jigsaw/spec/reqs/02.md If you do that then please quote just the text upon which you're commenting, rather than the entire document. This draft incorporates the revisions and additional requirements we've discussed. Some of the wording is slightly different, so as to fit better into the overall document. I made three non-trivial changes: - I rewrote the new "Annotation indices" requirement to be more abstract; it's now titled "Efficient annotation detection", but the intent is the same. - I renamed the new "Access to class files and resources" requirement to "Readable artifacts" in order to avoid confusion with the use of the term "access" in other requirements, where it refers to access control. - The "Upgradeable modules" requirement was too restrictive; I rewrote it to no longer require the replacing module to be binary-compatible and from the same platform implementation. (Thanks to Alex Buckley, who suggested these improvements.) I'd like to finalize the requirements by this time next week, so please send any final comments by 23:00 UTC next Wednesday, 1 April (no joke!). Silence will be interpreted as assent. - Mark From Tim_Ellison at uk.ibm.com Tue Mar 31 13:57:50 2015 From: Tim_Ellison at uk.ibm.com (Tim Ellison) Date: Tue, 31 Mar 2015 14:57:50 +0100 Subject: Module-system requirements, draft 2 In-Reply-To: <20150325225212.B8AB0534D7@eggemoggin.niobe.net> References: <20150325225212.B8AB0534D7@eggemoggin.niobe.net> Message-ID: "jpms-spec-experts" wrote on 25/03/2015 22:52:12: > Thanks for all the comments and suggestions. I've posted draft 2 of the > requirements document, here: > > http://openjdk.java.net/projects/jigsaw/spec/reqs/02 > Last few comments: (a) Just to ensure this is not misinterpreted, perhaps consider rephrasing | - _Protection domains_ --- At run time it must be possible to associate | the code in modules with protection domains, for the purpose of | granting security permissions. as - _Protection domains_ --- At run time it must be possible to associate modules with protection domains, for the purpose of granting security permissions to code in those modules. (b) | - _Non-prescriptive version strings_ --- Version strings must have a | precisely-defined syntax and be totally ordered, but otherwise their | format should be as general as possible in order to accommodate | existing popular version-string schemes. I'm trying to reconcile the requirement for version strings to be "non-prescriptive" and have a "precisely-defined syntax". The need for a defined syntax is not reflected elsewhere, so is it sufficient that they are simply totally ordered, and otherwise left to the needs of the configuration management systems that assemble coherent sets of modules? (c) This version doesn't incorporate the "Overrideable encapsulation" requirement described here http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-February/000020.html or the clarification suggested for _Referential integrity_ to read "that each module can be configured only to reference" rather than "each module can only be configured to reference" Regards, Tim Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU