From mark.reinhold at oracle.com Tue Feb 10 19:28:27 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 10 Feb 2015 11:28:27 -0800 Subject: Welcome to the Java Platform Module System (JSR 376) Expert Group Message-ID: <20150210112827.999294@eggemoggin.niobe.net> Welcome! I've set up our primary web page, here: http://openjdk.java.net/projects/jigsaw/spec/ As outlined in the JSR and on the above page I've set up three mailing lists: One for the EG itself, one for observers, and one for incoming comments. I've subscribed every EG member to the EG list; if you wish also to receive incoming comments then let me know and I'll add you to the comments list. Content on our web pages and mailing lists is licensed for evaluation and comment only, per the OpenJDK Terms of Use [1]. - Mark [1] http://openjdk.java.net/legal/tou From mark.reinhold at oracle.com Tue Feb 10 19:30:58 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 10 Feb 2015 11:30:58 -0800 Subject: Java Platform Module System EG Introductions Message-ID: <20150210113058.75724@eggemoggin.niobe.net> I think we EG members are for the most part known to each other, at least by reputation, but for those following along at home it'd be helpful for us to introduce ourselves, providing links to online bios where possible. (This will also verify that the EG list is set up correctly.) I'm Mark Reinhold, Chief Architect of the Java Platform Group at Oracle. My online bio is here: http://mreinhold.org/ . - Mark From david.lloyd at redhat.com Tue Feb 10 20:03:05 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 10 Feb 2015 14:03:05 -0600 Subject: Java Platform Module System EG Introductions In-Reply-To: <20150210113058.75724@eggemoggin.niobe.net> References: <20150210113058.75724@eggemoggin.niobe.net> Message-ID: <54DA63F9.1050109@redhat.com> On 02/10/2015 01:30 PM, mark.reinhold at oracle.com wrote: > I think we EG members are for the most part known to each other, at > least by reputation, but for those following along at home it'd be > helpful for us to introduce ourselves, providing links to online bios > where possible. (This will also verify that the EG list is set up > correctly.) Hi! I'm David Lloyd, a principal engineer at Red Hat working on the JBoss/WildFly family of software, and of particular relevance, the JBoss Modules implementation [1], among other things both related and unrelated. I'm not great at self-promotion so I don't have a cool bio page, but you can get a good idea of my various endeavors, adventures, and mishaps by viewing my GitHub profile [2]. I look forward to a positive experience and outcome working with all of you! [1] https://github.com/jboss-modules/jboss-modules [2] https://github.com/dmlloyd -- - DML From niftiness at gmail.com Tue Feb 10 20:37:41 2015 From: niftiness at gmail.com (Tim Boudreau) Date: Tue, 10 Feb 2015 15:37:41 -0500 Subject: Java Platform Module System EG Introductions In-Reply-To: <54DA63F9.1050109@redhat.com> References: <20150210113058.75724@eggemoggin.niobe.net> <54DA63F9.1050109@redhat.com> Message-ID: Hi! I'm Tim Boudreau. I worked or Sun for 11 years on NetBeans, wrote a couple of books, did a lot of training of developers in how to modularize code, and since then have been doing architecture and performance consulting. More info here: http://timboudreau.com -Tim On Tue, Feb 10, 2015 at 3:03 PM, David M. Lloyd wrote: > On 02/10/2015 01:30 PM, mark.reinhold at oracle.com wrote: > >> I think we EG members are for the most part known to each other, at >> least by reputation, but for those following along at home it'd be >> helpful for us to introduce ourselves, providing links to online bios >> where possible. (This will also verify that the EG list is set up >> correctly.) >> > > Hi! I'm David Lloyd, a principal engineer at Red Hat working on the > JBoss/WildFly family of software, and of particular relevance, the JBoss > Modules implementation [1], among other things both related and unrelated. > I'm not great at self-promotion so I don't have a cool bio page, but you > can get a good idea of my various endeavors, adventures, and mishaps by > viewing my GitHub profile [2]. I look forward to a positive experience and > outcome working with all of you! > > [1] https://github.com/jboss-modules/jboss-modules > [2] https://github.com/dmlloyd > > -- > - DML > -- http://timboudreau.com From peter.kriens at aqute.biz Wed Feb 11 08:29:33 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Wed, 11 Feb 2015 09:29:33 +0100 Subject: Java Platform Module System EG Introductions In-Reply-To: <20150210113058.75724@eggemoggin.niobe.net> References: <20150210113058.75724@eggemoggin.niobe.net> Message-ID: I am Peter Kriens, independent consultant since 1990. I worked with the OSGi Alliance to as participant, editor, gopher, and evangelist. Currently doing OSGi enRoute for about 50% of my time, a project to make it really easy to get started with OSGi. The remaining time I work with Paremus Ltd, a company that provides a large scale computing platform that leverages OSGi to manage and wire together any type of code. I am really excited to participate in this expert and I am looking forward to our collaboration. This EG has a long winded history and it would be great if we could leave that history behind to move Java modularity forward after all those years. The signs so far are very good! Thanks, kind regards, Peter Kriens > On 10 feb. 2015, at 20:30, mark.reinhold at oracle.com wrote: > > I think we EG members are for the most part known to each other, at > least by reputation, but for those following along at home it'd be > helpful for us to introduce ourselves, providing links to online bios > where possible. (This will also verify that the EG list is set up > correctly.) > > I'm Mark Reinhold, Chief Architect of the Java Platform Group at Oracle. > My online bio is here: http://mreinhold.org/ . > > - Mark From Tim_Ellison at uk.ibm.com Wed Feb 11 12:53:52 2015 From: Tim_Ellison at uk.ibm.com (Tim Ellison) Date: Wed, 11 Feb 2015 12:53:52 +0000 Subject: Java Platform Module System EG Introductions In-Reply-To: <20150210113058.75724@eggemoggin.niobe.net> References: <20150210113058.75724@eggemoggin.niobe.net> Message-ID: Hi. I'm Tim Ellison -- currently at IBM and based at Hursley in the UK. I look after a number of runtimes for IBM, most recently getting our Java 8 out to all (yay). In the past I've worked on Smalltalk, some VM designs, Eclipse core platform, Apache Harmony, and dabbled in many more. In the future I'll be implementing the spec we design, so I have a vested interest in keeping things both simple and effective, and I'm interested in OSGi compatibility issues (see Project Penrose). The nearest thing to an on-line bio is my LinkedIn profile (uk.linkedin.com/in/tpellison) IBM's Lotus Notes is notoriously bad for formatting external e-mails, so apologies if you receive html or ascii art-work from me. When you or I get fed up I'll switch to a private ID and sensible client. Regards, Tim "jpms-spec-experts" wrote on 10/02/2015 19:30:58: > From: mark.reinhold at oracle.com > To: jpms-spec-experts at openjdk.java.net, > Date: 10/02/2015 19:31 > Subject: Java Platform Module System EG Introductions > Sent by: "jpms-spec-experts" > > I think we EG members are for the most part known to each other, at > least by reputation, but for those following along at home it'd be > helpful for us to introduce ourselves, providing links to online bios > where possible. (This will also verify that the EG list is set up > correctly.) > > I'm Mark Reinhold, Chief Architect of the Java Platform Group at Oracle. > My online bio is here: http://mreinhold.org/ . > > - Mark > Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU From mark.reinhold at oracle.com Wed Feb 11 18:38:31 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 11 Feb 2015 10:38:31 -0800 Subject: Module-system requirements Message-ID: <20150211103831.528761@eggemoggin.niobe.net> I've posted a first draft of a requirements document, here: http://openjdk.java.net/projects/jigsaw/spec/reqs/01 This is distilled from the draft Goals & Requirements document for the overall modularization effort [1], omitting items that are beyond the scope of the module system itself and rewording others as necessary. As noted in the introduction, the intent of this document is to serve as a set of guideposts for this EG. The specification we produce will, ultimately, satisfy all of these requirements. After we finalize this document it will certainly be possible to revise it, but the bar for doing so will be relatively high. This is the first draft, and I'm sure you'll all have many comments and suggestions. I'd like to finalize the requirements by the end of this month (February 2015), so fire away! - Mark [1]: http://openjdk.java.net/projects/jigsaw/goals-reqs From wayne at eclipse.org Wed Feb 11 14:06:51 2015 From: wayne at eclipse.org (Wayne Beaton) Date: Wed, 11 Feb 2015 09:06:51 -0500 Subject: Java Platform Module System EG Introductions In-Reply-To: <20150210113058.75724@eggemoggin.niobe.net> References: <20150210113058.75724@eggemoggin.niobe.net> Message-ID: <54DB61FB.2000805@eclipse.org> I am Wayne Beaton, Director of Open Source Projects at the Eclipse Foundation. My work involves evangelism which has afforded me the opportunity to become somewhat proficient with OSGi in general and the Eclipse implementation, Equinox, in particular. Before my current job, I worked at IBM WebSphere (AI&M Services) as a services consultant working primarily with Java EE. Prior to that I was a senior member of the technical staff and Old Dude Who Knows Smalltalk at The Object People. Wayne On 10/02/15 02:30 PM, mark.reinhold at oracle.com wrote: > I think we EG members are for the most part known to each other, at > least by reputation, but for those following along at home it'd be > helpful for us to introduce ourselves, providing links to online bios > where possible. (This will also verify that the EG list is set up > correctly.) > > I'm Mark Reinhold, Chief Architect of the Java Platform Group at Oracle. > My online bio is here: http://mreinhold.org/ . > > - Mark -- Wayne Beaton @waynebeaton The Eclipse Foundation EclipseCon 2015 From david.lloyd at redhat.com Wed Feb 11 19:27:06 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 11 Feb 2015 13:27:06 -0600 Subject: Exporting things (Was: Re: Module-system requirements) In-Reply-To: <20150211103831.528761@eggemoggin.niobe.net> References: <20150211103831.528761@eggemoggin.niobe.net> Message-ID: <54DBAD0A.1060005@redhat.com> On 02/11/2015 12:38 PM, mark.reinhold at oracle.com wrote: > This is the first draft, and I'm sure you'll all have many comments and > suggestions. I'd like to finalize the requirements by the end of this > month (February 2015), so fire away! OK, here goes :-) I notice that under Exports, it is defined that only packages (i.e. classes within those packages) are considered to be exportable, and not resources. I'm not certain how I feel about that... in our implementation we flattened the concepts of packages and directories. This was mainly to solve a few specific difficulties of supporting plain JARs as module content; a JAR has a mix of classes and resources, and it is hard to deterministically tell the difference, especially when you can do such (commonly done) things as: classLoader.getResource("org/foo/bar/SomeClass.class"). By leveraging this approach a little more, we were able to have common configuration switches to do things like, export the contents of META-INF/services to modules which elect to import it, which is handy in some specific (but, in our distribution, common) use cases. However this approach is not without its difficulties: it requires the modules' internal index to contain multiple entries for directories which commonly appear in imported modules (especially META-INF and its children, and the root directory). This adds a marginal bit of complexity to the algorithm, but also a bit of complexity in the "human expectation" dimension as well. Also we implicitly allow a module to "see" more than one package with the same name which can lead to various problems with linkage in some cases, especially of one dependency "overlays" another one (in particular if such overlaying is partial or imperfect), though in practice this has proven to only rarely be an issue. To sum up though, I'd be glad to see (from my perspective) this behavior go away in favor of the simpler model of restricting exports to packages (and isolating classes and resources), but only if we can be reasonably sure that the use cases solved thereby are somehow sorted out, which include: ? Supporting this case (or determining that it is not relevant): A module privately uses services provided by one or more specific peer module(s) (via ServiceLoader), the arrangement of which is determined statically by distribution. ? Having (generally) the same visibility to .class files as to the classes themselves, from a given resource-loading entry point (Class or ClassLoader), i.e. if I have visibility to classLoader.loadClass("SomeClass"), I also have visibility to classLoader.getResource("SomeClass.class"), and vice-versa (any security restrictions aside). ? A consistent ruling on the right behavior of general resources which are co-packaged with classes that use them (typically loaded via e.g. getClass().getResourceAsStream("MyResource.properties") or similar). ? A consistent ruling on whether it makes sense generally to be able to export resources to dependents. A minor, somewhat related point that this raises... The term "dependences" (rarely-used pl. of "dependence") is used quite a lot in the document (whether intentional or otherwise), but I think that the term "dependencies" (pl. of "dependency") is probably a better term, and is definitely a more ubiquitous one. The inverse of "dependency" is "dependent", which forms a concise term pair that we use quite a lot internally (i.e. if A is a dependency of B, B is a dependent of A). -- - DML From david.lloyd at redhat.com Wed Feb 11 19:56:22 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 11 Feb 2015 13:56:22 -0600 Subject: Interoperation (Was: Re: Module-system requirements) In-Reply-To: <20150211103831.528761@eggemoggin.niobe.net> References: <20150211103831.528761@eggemoggin.niobe.net> Message-ID: <54DBB3E6.8060202@redhat.com> I'm concerned that the interoperation point doesn't quite go far enough. It says "It must be possible for another module system, such as OSGi, to locate Java modules and resolve them using its own resolver, except possibly for core system modules." I agree with this requirement, such as it is, but I would also assert that it should be possible to retrofit not only OSGi but Java EE, Maven, and even legacy "modules" like extensions and so on as modules. The reason I make this assertion is that we have done all of these things with success and (I dare say) relative ease, even to the extent of achieving both EE and OSGi certification with such a retrofit. The reason I think this is important is that it makes it possible to link modules from different module specifications together in a variety of sane and well-defined ways, and would provide a great deal more flexibility to (for example) Java EE 9 which would presumably build on the work of this EG. To achieve this additional requirement does require a few very specific implementation considerations but I do not believe that any of these considerations contraindicate any of the other requirements stated here. Thus I feel this requirement should be amended to something along the lines of: Interoperation - It must be possible for another module system, such as OSGi, to locate Java modules and resolve them using its own resolver, except possibly for core system modules. It must be possible for other module systems to reuse the framework provided by this specification in order to facilitate a standard, well-defined mechanism by which different present and future module systems may interoperate. -- - DML From niftiness at gmail.com Thu Feb 12 03:05:55 2015 From: niftiness at gmail.com (Tim Boudreau) Date: Wed, 11 Feb 2015 22:05:55 -0500 Subject: Enhanced ServiceLoader? Message-ID: *Binding ? The Java run-time system must bind service implementations to interfaces and make these bindings available via an enhanced version of the existing java.util.ServiceLoader API.* Are there any specifics as to what is meant by "enhanced" here? One thing that has always seemed needed here is ordering. Specifically, it's common to provide an API, and a default implementation of it (which may be a dummy implementation - essentially there to ensure a lookup will never return null, or be an implementation the system should fail over to in tests or even a working but not optimal for every scenario implementation - I've seen all of the above). You want that implementation to appear last in the total order of available implementations, regardless of load-order (ordering on the classpath, or whatever implementation deal generates the order they're discovered in). To play devil's advocate, you *can* do something like define an @Ordered annotation with an integer value, gather all the implementations and sort them according to it (assuming you can enforce that they all have it, or there is something reasonable you can do with ones that don't). But it means instantiating all available instances in order to sort them, which may not be desirable if you're want to lazy-load (I can think of complicated ways you could avoid that, but most are not good ideas). My experience is that whenever you create a API with a set of things that specifies "this has no order", sooner or later someone will write code that, unknowingly, depends on whatever order they see in practice - so when in doubt, allowing explicit ordering leads to more reliable dependent code. -Tim -- http://timboudreau.com From peter.kriens at aqute.biz Thu Feb 12 08:20:47 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Thu, 12 Feb 2015 09:20:47 +0100 Subject: Enhanced ServiceLoader? In-Reply-To: References: Message-ID: <3ED51F23-37D7-4BAE-B13B-0EB06DD4CCD2@aQute.biz> Explicit ordering requires a global view of the runtime, which is imho something you have to avoid in a modular system. Consistent ordering based on an intrinsic aspect is however relevant to prevent spurious failures, e.g. sorted by name. Kind regards, Peter Kriens > On 12 feb. 2015, at 04:05, Tim Boudreau wrote: > > *Binding ? The Java run-time system must bind service implementations to > interfaces and make these bindings available via an enhanced version of the > existing java.util.ServiceLoader API.* > > Are there any specifics as to what is meant by "enhanced" here? > > One thing that has always seemed needed here is ordering. Specifically, > it's common to provide an API, and a default implementation of it (which > may be a dummy implementation - essentially there to ensure a lookup will > never return null, or be an implementation the system should fail over to > in tests or even a working but not optimal for every scenario > implementation - I've seen all of the above). You want that implementation > to appear last in the total order of available implementations, regardless > of load-order (ordering on the classpath, or whatever implementation deal > generates the order they're discovered in). > > To play devil's advocate, you *can* do something like define an @Ordered > annotation with an integer value, gather all the implementations and sort > them according to it (assuming you can enforce that they all have it, or > there is something reasonable you can do with ones that don't). But it > means instantiating all available instances in order to sort them, which > may not be desirable if you're want to lazy-load (I can think of > complicated ways you could avoid that, but most are not good ideas). > > My experience is that whenever you create a API with a set of things that > specifies "this has no order", sooner or later someone will write code > that, unknowingly, depends on whatever order they see in practice - so when > in doubt, allowing explicit ordering leads to more reliable dependent code. > > -Tim > > -- > http://timboudreau.com From david.lloyd at redhat.com Thu Feb 12 13:44:45 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 12 Feb 2015 07:44:45 -0600 Subject: Enhanced ServiceLoader? In-Reply-To: <3ED51F23-37D7-4BAE-B13B-0EB06DD4CCD2@aQute.biz> References: <3ED51F23-37D7-4BAE-B13B-0EB06DD4CCD2@aQute.biz> Message-ID: <54DCAE4D.2050303@redhat.com> I agree that consistent ordering is an important property. I think it's a given that more than one provider will often be available for some particular service; you wouldn't want a situation where (to be specifically pathological) one application could be started twice, each time getting different providers for such services. Explicit ordering *should* be possible as well though, at least within some specific well-defined bounds (e.g. a single service loader provider file should always be reported in order as it is today). I don't think the requirements as stated forbid that at least. And I would hope that explicit ordering at the module level should be achievable through the use of configurations, though that section in general needs some more thought and discussion I suspect. On 02/12/2015 02:20 AM, Peter Kriens wrote: > Explicit ordering requires a global view of the runtime, which is imho something you have to avoid in a modular system. > > Consistent ordering based on an intrinsic aspect is however relevant to prevent spurious failures, e.g. sorted by name. > > Kind regards, > > Peter Kriens > > >> On 12 feb. 2015, at 04:05, Tim Boudreau wrote: >> >> *Binding ? The Java run-time system must bind service implementations to >> interfaces and make these bindings available via an enhanced version of the >> existing java.util.ServiceLoader API.* >> >> Are there any specifics as to what is meant by "enhanced" here? >> >> One thing that has always seemed needed here is ordering. Specifically, >> it's common to provide an API, and a default implementation of it (which >> may be a dummy implementation - essentially there to ensure a lookup will >> never return null, or be an implementation the system should fail over to >> in tests or even a working but not optimal for every scenario >> implementation - I've seen all of the above). You want that implementation >> to appear last in the total order of available implementations, regardless >> of load-order (ordering on the classpath, or whatever implementation deal >> generates the order they're discovered in). >> >> To play devil's advocate, you *can* do something like define an @Ordered >> annotation with an integer value, gather all the implementations and sort >> them according to it (assuming you can enforce that they all have it, or >> there is something reasonable you can do with ones that don't). But it >> means instantiating all available instances in order to sort them, which >> may not be desirable if you're want to lazy-load (I can think of >> complicated ways you could avoid that, but most are not good ideas). >> >> My experience is that whenever you create a API with a set of things that >> specifies "this has no order", sooner or later someone will write code >> that, unknowingly, depends on whatever order they see in practice - so when >> in doubt, allowing explicit ordering leads to more reliable dependent code. >> >> -Tim >> >> -- >> http://timboudreau.com > -- - DML From david.lloyd at redhat.com Thu Feb 12 15:35:08 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 12 Feb 2015 09:35:08 -0600 Subject: Configurations... (Was: Re: Module-system requirements) In-Reply-To: <20150211103831.528761@eggemoggin.niobe.net> References: <20150211103831.528761@eggemoggin.niobe.net> Message-ID: <54DCC82C.4030200@redhat.com> I'm a little concerned that the configurations section implies a strict hierarchy: "Nested dynamic configurations ? An application must be able to create a dynamic configuration that relates to an existing dynamic configuration rather than to the application?s initial configuration." Why can a configuration not relate to more than one other configurations? -- - DML From Tim_Ellison at uk.ibm.com Fri Feb 13 14:06:08 2015 From: Tim_Ellison at uk.ibm.com (Tim Ellison) Date: Fri, 13 Feb 2015 14:06:08 +0000 Subject: Module-system requirements In-Reply-To: <20150211103831.528761@eggemoggin.niobe.net> References: <20150211103831.528761@eggemoggin.niobe.net> Message-ID: "jpms-spec-experts" wrote on 11/02/2015 18:38:31: > From: mark.reinhold at oracle.com > To: jpms-spec-experts at openjdk.java.net, > Date: 11/02/2015 18:38 > Subject: Module-system requirements > Sent by: "jpms-spec-experts" > > I've posted a first draft of a requirements document, here: > > http://openjdk.java.net/projects/jigsaw/spec/reqs/01 > > This is distilled from the draft Goals & Requirements document for the > overall modularization effort [1], omitting items that are beyond the > scope of the module system itself and rewording others as necessary. > > As noted in the introduction, the intent of this document is to serve > as a set of guideposts for this EG. The specification we produce will, > ultimately, satisfy all of these requirements. After we finalize this > document it will certainly be possible to revise it, but the bar for > doing so will be relatively high. > > This is the first draft, and I'm sure you'll all have many comments and > suggestions. I'd like to finalize the requirements by the end of this > month (February 2015), so fire away! > > - Mark > > > [1]: http://openjdk.java.net/projects/jigsaw/goals-reqs > Overall comments: - I have quoted sections below in providing feedback. Hopefully that works well enough for you. You may want to consider sharing the markdown/odf document if you want comments and questions in the full context. - The requirements doc is a mixture of specification and implementation requirements. For example, while I agree with the objectives of the linking and packaging sections, they could be more succinctly written for their requirements on the module system itself. - It would be useful to have a section on Security. While there are aspects throughout other sections of the requirements, such as encapsulation and referential integrity, there may be other considerations where module boundaries can be used to enhance security, e.g. only allowing privilege escalation within a module scope, modules declaring all exported APIs fail when passed a tainted string, etc. I'm not proposing scope-creep but rather ensuring folks have had the chance to think about it and easy opportunities are not missed to make Java 9 even more secure. - I wrote my comments before reading other people's. Apologies to David, who did a better job of explaining the 'Exporting things' issues. Detailed comments follow: Fundamentals > Encapsulation ? The access-control mechanisms of the Java language and > virtual machine must prevent code from accessing classes and interfaces > in packages that are not exported by their containing modules, or in > packages whose containing modules are not required by the module > containing the code. These mechanisms must be operative even when a > security manager is not present. The document explicitly calls out classes and interfaces here as targets for encapsulation. Given a module can also contain native code, static resource files, and user-editable configuration files, are these being explicitly excluded? It would be valuable for the module system to also prevent access to libraries and resources that are not exported by their containing modules. I appreciate this requires extending the Exports to include non-package resources, but the alternative is that native code and resources that are specific to a module's implementation would be accessible to other modules (if the default were to export resources), or that modules cannot access resources managed by other modules (if the default were to not export resources). > Non-interference ? The Java compiler, virtual machine, and run-time > system must ensure that modules that contain packages of the same > name do not interfere with each other. Can you explain "interfere"? Is this referring to ambiguous class and linkage error type problems, or something else here? > Compatible Java Platform modularization ? It must be possible to > divide an existing Java Platform, e.g., Java SE or Java EE, into a > set of modules such that existing libraries and applications can run > without change, so long as they use only standard platform APIs. It is the last nine words of this that I have concerns about. With that caveat I think there is a serious risk that Java 9 will be incapable of running a large body of existing, valuable Java code. There is a longer discussion to be had about the support for 'legacy' Java code, but just as the document provides for 'gradual migration of applications' under Development, I believe there is value in a promiscuous mode for the modular platform or specific code source in being able to run existing libraries and applications without change. > Upgradeable modules ? If a Java SE module exports an endorsed > standard or standalone technology then, given a particular > implementation of the platform, it must be possible to use a > binary-compatible version of that module from a later release of > that implementation. This is the only use of the term 'Java SE module'. What is a Java SE module? Am I correct in assuming this is describing _runtime_ upgradeable modules, as clearly modules can be selected at build, link time in any case? Why the distinction from other modules that may be upgradeable? For example, if I choose to issue a patched version of my java.base that is binary compatible with a particular implementation of the platform, then I would like to be able to upgrade that at runtime (see Referential Integrity below). > Referential integrity ? Given a specific set of modules comprising > an implementation of the Java SE Platform, it must be possible to > ensure that each module can only be configured to reference other > modules in the same set, with the exception of upgradeable modules. (I'm choosing to interpret 'can only be configured to reference' as 'can be configured only to reference') If I am interpreting this correctly it means that it must be possible, but not strictly required, that there is no ad hoc runtime replacement of modules in an SE image. However, the set of permitted modules can be arbitrarily defined to make runtime replacement possible. For example, the VM can identify through naming, code signing, directory location, or whatever, 'modules comprising an implementation of the Java SE Platform' and refuse to load their dependencies if they are not in the definition of that Platform set. However, the 'defined set' is all modules with a given name prefix, or signature, or directory location, etc so I /can/ upgrade any modules comprising an implementation of the Java SE Platform by providing a new binary compatible module that falls within that set. Is that interpretation valid. How do you see this working in practice? Development > Multi-mode artifacts ? The existing JAR-file format must be enhanced > so that a module that does not contain native code can be delivered > in a single artifact that can be used either on the class path or as > a module. Why the restriction on JAR files that contain native code? Of course plain old JAR files would not be able to take advantage of the native code unless the application enabled this. For example, OSGi allows for native code to be delivered via application JAR files. Consider changing to "...so that a module that does not depend upon native code can be delivered..." Dynamic Configuration I am reading this section while making a clear distinction in my mind between the Referential Integrity of the Java SE Platform modules, and the more open nature of the modular application. If this distinction is correct, then I feel we do need a clear 'Fundamentals' definition of a Platform Module. The module system manages the Java Platform modules differently to the modules of a modular Java application. Although described as Basic dynamic configuration, this section does appear to open the door for runtime module dependency resolution. This dynamic configuration is carefully avoided for the Java SE Platform. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU From wayne at eclipse.org Fri Feb 13 14:30:30 2015 From: wayne at eclipse.org (Wayne Beaton) Date: Fri, 13 Feb 2015 09:30:30 -0500 Subject: Module-system requirements In-Reply-To: <20150211103831.528761@eggemoggin.niobe.net> References: <20150211103831.528761@eggemoggin.niobe.net> Message-ID: <54DE0A86.9040207@eclipse.org> Hi Mark. Thanks for this. I have a couple of questions/thoughts: > /Greater platform integrity/, to ensure that code that is internal to > a platform implementation is not accessible from outside the > implementation; In principle, I agree 100% with this. I imagine, however, that it won't take too long to find (many) examples where internal code is accessed. If this is strictly enforced, then we're going to break a lot of code. How strong of a requirement is this? > * > > /Upgradeable modules/? If a Java SE module exports anendorsed > standard or standalone technology > then, > given a particular implementation of the platform, it must be > possible to use abinary-compatible > version > of that module from a later release of that implementation. > I think that "Upgradeable modules" may be at odds with the next statement. Or maybe not. Taken together, an upgrade of a module necessitates the removal of the original (if both are in place, then some sort of resolution to pick one over the other is required). Also, taken in combination, it suggests that the binary-compatibility must be guaranteed by the deployer (i.e the framework has no mechanism to check). > > * /Version selection/? The process of configuring a set of modules > need not consider more than one version of any particular module. > I get that Ivy, Maven, and Gradle all do dependency resolution and I totally get the desire to avoid duplicating behaviour, but my experience with Eclipse/OSGi is such that runtime dependency resolution is different from build-time dependency resolution. Of course, it's probably the case that what we have here is exactly what ~99% of the Java community requires. Wayne On 11/02/15 01:38 PM, mark.reinhold at oracle.com wrote: > I've posted a first draft of a requirements document, here: > > http://openjdk.java.net/projects/jigsaw/spec/reqs/01 > > This is distilled from the draft Goals & Requirements document for the > overall modularization effort [1], omitting items that are beyond the > scope of the module system itself and rewording others as necessary. > > As noted in the introduction, the intent of this document is to serve > as a set of guideposts for this EG. The specification we produce will, > ultimately, satisfy all of these requirements. After we finalize this > document it will certainly be possible to revise it, but the bar for > doing so will be relatively high. > > This is the first draft, and I'm sure you'll all have many comments and > suggestions. I'd like to finalize the requirements by the end of this > month (February 2015), so fire away! > > - Mark > > > [1]: http://openjdk.java.net/projects/jigsaw/goals-reqs -- Wayne Beaton @waynebeaton The Eclipse Foundation EclipseCon 2015 From hans.dockter at gradleware.com Sun Feb 15 17:16:05 2015 From: hans.dockter at gradleware.com (Hans Dockter) Date: Sun, 15 Feb 2015 12:16:05 -0500 Subject: Java Platform Module System EG Introductions In-Reply-To: <20150210113058.75724@eggemoggin.niobe.net> References: <20150210113058.75724@eggemoggin.niobe.net> Message-ID: I am Hans Dockter, Founder of Gradle and CEO of Gradleware. I'm particularly interested in how to provide an excellent build and developer experience when using and migrating to Jigsaw. I'm very excited about the current Jigsaw spec. Hans On Tuesday, February 10, 2015, mark.reinhold at oracle.com < mark.reinhold at oracle.com> wrote: > I think we EG members are for the most part known to each other, at > least by reputation, but for those following along at home it'd be > helpful for us to introduce ourselves, providing links to online bios > where possible. (This will also verify that the EG list is set up > correctly.) > > I'm Mark Reinhold, Chief Architect of the Java Platform Group at Oracle. > My online bio is here: http://mreinhold.org/ . > > - Mark > From mark.reinhold at oracle.com Sat Feb 14 21:33:07 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Sat, 14 Feb 2015 13:33:07 -0800 Subject: Configurations... (Was: Re: Module-system requirements) In-Reply-To: <54DCC82C.4030200@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, <54DCC82C.4030200@redhat.com> Message-ID: <20150214133307.760777@eggemoggin.niobe.net> 2015/2/12 7:35 -0800, david.lloyd at redhat.com: > I'm a little concerned that the configurations section implies a strict > hierarchy: > > "Nested dynamic configurations ? An application must be able to create a > dynamic configuration that relates to an existing dynamic configuration > rather than to the application?s initial configuration." > > Why can a configuration not relate to more than one other configurations? A tree of configurations rather than a simple linear list is more difficult to reason about and trickier to implement (you must, e.g., cope with the diamond-import problem). The feedback we've received so far has not suggested that this limitation will be a problem in practice. Do you have use cases which require this flexibility? - Mark From mark.reinhold at oracle.com Mon Feb 16 01:24:05 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Sun, 15 Feb 2015 17:24:05 -0800 Subject: Exporting things (Was: Re: Module-system requirements) In-Reply-To: <54DBAD0A.1060005@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, <54DBAD0A.1060005@redhat.com> Message-ID: <20150215172405.90174@eggemoggin.niobe.net> 2015/2/11 11:27 -0800, david.lloyd at redhat.com: > I notice that under Exports, it is defined that only packages (i.e. > classes within those packages) are considered to be exportable, and not > resources. I'm not certain how I feel about that... in our > implementation we flattened the concepts of packages and directories. > This was mainly to solve a few specific difficulties of supporting plain > JARs as module content; a JAR has a mix of classes and resources, and it > is hard to deterministically tell the difference, especially when you > can do such (commonly done) things as: > classLoader.getResource("org/foo/bar/SomeClass.class"). > > ... > > However this approach is not without its difficulties: ... > > ... To sum up though, I'd be glad to see (from my perspective) this > behavior go away in favor of the simpler model of restricting exports to > packages (and isolating classes and resources), I tend to agree. Exposing all of a module's resources, as JAR files do today, is wrong from the perspective of encapsulation. Extending the "exports" requirement to allow control over which resources are exported leads to both conceptual and implementation complexity (e.g., what would it mean to export both a class, as a type, as well as the corresponding .class file, or the latter and not the former?) It seems sanest to treat resources as strictly internal to modules. This suggests an additional requirement: - _Resource encapsulation_ --- The run-time system must ensure that the static resource files within a module are directly accessible only by code within that module. > but only if we can be > reasonably sure that the use cases solved thereby are somehow sorted > out, which include: > > ? Supporting this case (or determining that it is not relevant): A > module privately uses services provided by one or more specific peer > module(s) (via ServiceLoader), the arrangement of which is determined > statically by distribution. One way to accomplish this is to make the service interface accessible only by the relevant modules, via qualified exports. Then only those modules can provide or use implementations of the service. Another possibility is to add a requirement that the binding of providers to service interfaces can be controlled manually during the configuration process. Is that more along the lines of what you mean when you say "determined statically by the distribution"? > ? Having (generally) the same visibility to .class files as to the > classes themselves, from a given resource-loading entry point (Class or > ClassLoader), i.e. if I have visibility to > classLoader.loadClass("SomeClass"), I also have visibility to > classLoader.getResource("SomeClass.class"), and vice-versa (any security > restrictions aside). Can you describe some use cases that require this property? What if a module is compiled to native code ahead-of-time, and hence doesn't necessarily contain a .class file for every class? > ? A consistent ruling on the right behavior of general resources which > are co-packaged with classes that use them (typically loaded via e.g. > getClass().getResourceAsStream("MyResource.properties") or similar). If we take the view that resources are module-internal then the answer is that code within a module can access that module's resources but code outside of the module cannot (with the usual exception for reflection, which would require a security check). > ? A consistent ruling on whether it makes sense generally to be able to > export resources to dependents. If we take the view that resources are module-internal then the answer is no, this does not make sense. A module can export packages of classes and interfaces, and can publish service providers; it cannot also export files in a string-based hierarchical namespace. > A minor, somewhat related point that this raises... > > The term "dependences" (rarely-used pl. of "dependence") is used quite a > lot in the document (whether intentional or otherwise), but I think that > the term "dependencies" (pl. of "dependency") is probably a better term, > and is definitely a more ubiquitous one. The inverse of "dependency" is > "dependent", which forms a concise term pair that we use quite a lot > internally (i.e. if A is a dependency of B, B is a dependent of A). You're not the first to ask me about this ... "Dependence" and "dependency" name distinct concepts. A dependence is a relationship, potentially unfulfilled; a dependency is a thing which can fulfill a relationship of dependence. When thinking, speaking, or writing about modules I find it useful to keep this distinction clear. A module, standing alone, can have dependences upon some other, yet-to-be-identified modules. All that is known is that they have specific names, or that they provide specific services. The module's definition, in other words, merely describes relationships and constraints upon how those relationships can be fulfilled; it does not identify the specific modules that will fulfill those relationships. The processes of resolution and service binding identify the actual modules, i.e., the dependencies, that will satisfy a module's dependences. I agree that "dependent" is also a useful term. In ASCII-art form the three concepts are related thus: Dependence Dependent --------------> Dependency - Mark From mark.reinhold at oracle.com Mon Feb 16 01:24:24 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Sun, 15 Feb 2015 17:24:24 -0800 Subject: Interoperation (Was: Re: Module-system requirements) In-Reply-To: <54DBB3E6.8060202@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, <54DBB3E6.8060202@redhat.com> Message-ID: <20150215172424.772730@eggemoggin.niobe.net> 2015/2/11 11:56 -0800, david.lloyd at redhat.com: > I'm concerned that the interoperation point doesn't quite go far enough. > It says "It must be possible for another module system, such as OSGi, > to locate Java modules and resolve them using its own resolver, except > possibly for core system modules." > > I agree with this requirement, such as it is, but I would also assert > that it should be possible to retrofit not only OSGi but Java EE, Maven, > and even legacy "modules" like extensions and so on as modules. The > reason I make this assertion is that we have done all of these things > with success and (I dare say) relative ease, even to the extent of > achieving both EE and OSGi certification with such a retrofit. Can you explain what you mean, exactly, by "retrofit"? Are you saying that the JBoss module system can consume Java EE WAR files, OSGi bundles, and Maven projects, as they stand and without change? > ... > > To achieve this additional requirement does require a few very specific > implementation considerations but I do not believe that any of these > considerations contraindicate any of the other requirements stated here. Can you be more specific as to what those considerations are? Can they be expressed as module-system requirements? > Thus I feel this requirement should be amended to something along the > lines of: > > Interoperation - It must be possible for another module system, such as > OSGi, to locate Java modules and resolve them using its own resolver, > except possibly for core system modules. It must be possible for other > module systems to reuse the framework provided by this specification in > order to facilitate a standard, well-defined mechanism by which > different present and future module systems may interoperate. Unless I've misunderstood something, this sounds more like a research project -- a "meta" module system, in fact -- which is well outside the scope of this JSR. - Mark From mark.reinhold at oracle.com Mon Feb 16 01:25:07 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Sun, 15 Feb 2015 17:25:07 -0800 Subject: Module-system requirements In-Reply-To: References: <20150211103831.528761@eggemoggin.niobe.net>, Message-ID: <20150215172507.827110@eggemoggin.niobe.net> 2015/2/13 6:06 -0800, tim_ellison at uk.ibm.com: > Overall comments: > > - I have quoted sections below in providing feedback. Hopefully that > works well enough for you. You may want to consider sharing the > markdown/odf document if you want comments and questions in the full > context. Good idea -- publishing the Markdown source (not odf, ugh) would be straightforward. I'll look into it. > - The requirements doc is a mixture of specification and implementation > requirements. For example, while I agree with the objectives of the > linking and packaging sections, they could be more succinctly written for > their requirements on the module system itself. Yes, some of the requirements are implementation-oriented but, as such, they do imply requirements upon the specification. I'm not enough of a requirements purist to be bothered by that, but I'm open to adopting more-abstract rewrites of these items if you'd like to draft them. > - It would be useful to have a section on Security. While there are > aspects throughout other sections of the requirements, such as > encapsulation and referential integrity, there may be other considerations > where module boundaries can be used to enhance security, e.g. only > allowing privilege escalation within a module scope, modules declaring all > exported APIs fail when passed a tainted string, etc. I'm not proposing > scope-creep but rather ensuring folks have had the chance to think about > it and easy opportunities are not missed to make Java 9 even more secure. I'm all for improving security (obviously!), but if I've learned one thing about that general topic over the years it's that it's more a state of mind, or a way of life, than a set of features. It runs through everything that we do. Putting every requirement that's somehow related to security into a special "security" section of this document would just make it harder to read, and I'm skeptical that it'd actually make us all think even more about security than we already do. If you'd like to propose additional security-related requirements then by all means please go ahead, and we'll see where they fit into the present structure. > - I wrote my comments before reading other people's. Apologies to David, > who did a better job of explaining the 'Exporting things' issues. > > Detailed comments follow: > >> Fundamentals >> >> Encapsulation ? ... > > The document explicitly calls out classes and interfaces here as targets > for encapsulation. Given a module can also contain native code, static > resource files, and user-editable configuration files, are these being > explicitly excluded? It would be valuable for the module system to also > prevent access to libraries and resources that are not exported by their > containing modules. > > I appreciate this requires extending the Exports to include non-package > resources, but the alternative is that native code and resources that are > specific to a module's implementation would be accessible to other modules > (if the default were to export resources), or that modules cannot access > resources managed by other modules (if the default were to not export > resources). As I wrote in my reply to David, I think resources (in the traditional ClassLoader::getResource sense) should be treated as being strictly internal to modules. If a module needs to make the information in one of its resources available to other modules then it should do so via a properly-exported class, or a service provider. Just because JAR files are miniature filesystems which anybody can dissect doesn't mean that we need to conceive of Java modules in the same way. As to the other categories of information in a module: - I'm not sure it's worth going to great lengths to prevent a native library in one module from being accessed by another module. If a module has the power to load or access native code then it can pretty much do anything it wants. Taking stronger measures here would bring low benefit for high cost. - Configuration files are a pretty special case. They're meant for things like property files which need to wind up in the final run-time image and might be edited by a deployer or end user. They are, in some sense, implicitly exported to all other code, though the only way to access them is by knowing their final location in the file system relative to ${java.home} (arguably there should be an API for that). I don't think it's worth trying to invent a mechanism to prevent one module from reading another's configuration files. (Confession: In some ways I'm a bit queasy about including configuration files in this specification at all, but they're critical to the modularization of both the SE Platform and its RI, the JDK.) >> Non-interference ? The Java compiler, virtual machine, and run-time >> system must ensure that modules that contain packages of the same >> name do not interfere with each other. > > Can you explain "interfere"? Is this referring to ambiguous class and > linkage error type problems, or something else here? The former. If two distinct modules contain packages of the same name then code in that package in one module must not be able to access package-private types or members in that package in the other module. If two distinct modules contain classes of the same name (i.e., with the same binary name) then one must not be able to stand in for the other. >> Compatible Java Platform modularization ? It must be possible to >> divide an existing Java Platform, e.g., Java SE or Java EE, into a >> set of modules such that existing libraries and applications can run >> without change, so long as they use only standard platform APIs. > > It is the last nine words of this that I have concerns about. With that > caveat I think there is a serious risk that Java 9 will be incapable of > running a large body of existing, valuable Java code. There is a longer > discussion to be had about the support for 'legacy' Java code, but just as > the document provides for 'gradual migration of applications' under > Development, I believe there is value in a promiscuous mode for the > modular platform or specific code source in being able to run existing > libraries and applications without change. This is an important point, but remember that the requirements we're discussing here are upon the module system itself rather than upon the modularization of the Java SE Platform which it will enable. The latter will be a topic for the Java SE 9 EG, and I'm sure we'll have vigorous discussions there about the extent to which legacy code that depends upon internal interfaces should continue to work. Having said that, I do agree that the module system must provide some kind of escape hatch or "promiscuous mode", as you call it, so here's a stab at an additional requirement for the "Modularizing Java SE" section: - _Overrideable encapsulation_ --- It must be possible to force a module to export one or more of its packages to all other modules, or to some specific modules, even if its definition does not do so. >> Upgradeable modules ? If a Java SE module exports an endorsed >> standard or standalone technology then, given a particular >> implementation of the platform, it must be possible to use a >> binary-compatible version of that module from a later release of >> that implementation. > > This is the only use of the term 'Java SE module'. What is a Java SE > module? Good catch; the document should define this term. I'll fix that. A "Java SE Module" is one that implements some portion of the Java SE Platform Specification. Such modules will themselves be part of that Specification. A typical platform implementation will contain other modules as well, specific to the implementation. > Am I correct in assuming this is describing _runtime_ upgradeable modules, > as clearly modules can be selected at build, link time in any case? No, this describes modules in any phase, so that the overall model is the same across phases. > Why the distinction from other modules that may be upgradeable? For > example, if I choose to issue a patched version of my java.base that is > binary compatible with a particular implementation of the platform, then I > would like to be able to upgrade that at runtime (see Referential > Integrity below). Upgradeable modules are not meant for patching, and certainly not for patching the base module. They're meant for the small number of modules which implement endorsed standards or standalone technologies and which can, at least theoretically, evolve independently of Java SE itself (though in practice, they rarely do). More below ... >> Referential integrity ? Given a specific set of modules comprising >> an implementation of the Java SE Platform, it must be possible to >> ensure that each module can only be configured to reference other >> modules in the same set, with the exception of upgradeable modules. > > (I'm choosing to interpret 'can only be configured to reference' as 'can > be configured only to reference') Right; I'll fix that. > If I am interpreting this correctly it means that it must be possible, but > not strictly required, that there is no ad hoc runtime replacement of > modules in an SE image. However, the set of permitted modules can be > arbitrarily defined to make runtime replacement possible. Yes, insofar as that's both feasible and legitimate. The modularization of the SE Platform will specify a set of modules, and it will identify which must be upgradeable. I suppose an implementor could choose to make additional modules upgradeable, but that's an implementation choice, it might run afoul of conformance issues, and in some cases (e.g., the base module) it probably won't make much sense. > For example, the VM can identify through naming, code signing, directory > location, or whatever, 'modules comprising an implementation of the Java > SE Platform' and refuse to load their dependencies if they are not in the > definition of that Platform set. However, the 'defined set' is all > modules with a given name prefix, or signature, or directory location, etc > so I /can/ upgrade any modules comprising an implementation of the Java SE > Platform by providing a new binary compatible module that falls within > that set. > > Is that interpretation valid. Yes, I think you've more or less got it. > How do you see this working in practice? If you mean from the user's perspective, I expect the various phases will support an `-upgrademodulepath` option, or its moral equivalent in terms of APIs or configuration files, so that it's easy to specify newer versions of upgradeable modules. > Development > >> Multi-mode artifacts ? The existing JAR-file format must be enhanced >> so that a module that does not contain native code can be delivered >> in a single artifact that can be used either on the class path or as >> a module. > > Why the restriction on JAR files that contain native code? Of course > plain old JAR files would not be able to take advantage of the native code > unless the application enabled this. For example, OSGi allows for native > code to be delivered via application JAR files. Consider changing to > "...so that a module that does not depend upon native code can be > delivered..." I'm not sure what you're getting at here. The JAR-file specification does not describe how to include native code in a JAR file, and the class-path mechanism does not support it. The phrase "does not contain native code" is meant to respect that precedent. (It also, happily, precludes the possibility that we'll extend the class-path mechanism to support native code.) Changing this to read "does not depend upon native code" would, moreover, just make it more confusing, since all Java code everywhere ultimately depends upon some native code. OSGi bundles are, effectively, enhanced JAR files, and they can contain native code, but if they do then they certainly won't work on the regular old class path, so I don't see how they're relevant here. > Dynamic Configuration > > I am reading this section while making a clear distinction in my mind > between the Referential Integrity of the Java SE Platform modules, and the > more open nature of the modular application. > > If this distinction is correct, then I feel we do need a clear > 'Fundamentals' definition of a Platform Module. The module system manages > the Java Platform modules differently to the modules of a modular Java > application. I think it would be unfortunate if the module system were to treat Platform modules any differently than normal modules -- except for the base module, which is special by definition. Reading your comment makes me realize that the "referential integrity" requirement is poorly worded. It is motivated by SE Platform modularization, but it's actually a more-general feature which could well prove useful to other tightly-related collections of modules. I'll reword it along those lines. > Although described as Basic dynamic configuration, this section does > appear to open the door for runtime module dependency resolution. This > dynamic configuration is carefully avoided for the Java SE Platform. Run-time resolution must be possible, even for Platform modules, per the "Run-time augmentation of platform modules" item. - Mark From mark.reinhold at oracle.com Mon Feb 16 01:25:37 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Sun, 15 Feb 2015 17:25:37 -0800 Subject: Module-system requirements In-Reply-To: <54DE0A86.9040207@eclipse.org> References: <20150211103831.528761@eggemoggin.niobe.net>, <54DE0A86.9040207@eclipse.org> Message-ID: <20150215172537.965851@eggemoggin.niobe.net> 2015/2/13 6:30 -0800, wayne at eclipse.org: > Hi Mark. Thanks for this. I have a couple of questions/thoughts: > >> /Greater platform integrity/, to ensure that code that is internal to >> a platform implementation is not accessible from outside the >> implementation; > > In principle, I agree 100% with this. I imagine, however, that it won't > take too long to find (many) examples where internal code is accessed. Oh my, no -- it doesn't take long at all! > If this is strictly enforced, then we're going to break a lot of code. > How strong of a requirement is this? As I wrote in my reply to Tim, this is an important topic but it's more within the realm of the Java SE 9 EG than this EG. The important thing here is that we design a module system that can be used to achieve strong platform integrity but can also, when necessary and appropriate, be used to violate that integrity so as to keep legacy code working. > ... > >> /Upgradeable modules/? If a Java SE module exports an endorsed >> standard or standalone technology ... > > I think that "Upgradeable modules" may be at odds with the next > statement. Or maybe not. Taken together, an upgrade of a module > necessitates the removal of the original (if both are in place, then > some sort of resolution to pick one over the other is required). I think of this more in terms of shadowing than removal. If, e.g., a run-time image already has a particular upgradeable module baked in, and a newer version is provided when the image is invoked, then the run-time resolver will only consider the newer version. So while both versions of the module are present, in some sense, only the upgraded version is actually used. > Also, taken in combination, it suggests that the binary-compatibility > must be guaranteed by the deployer (i.e the framework has no mechanism > to check). Hmm. We could validate binary compatibility mechanically, but I don't know if that would wind up being too restrictive in practice. What if the maintainer of an upgradeable module had a good reason (e.g., to fix a security bug) to violate binary compatibility? >> * /Version selection/? The process of configuring a set of modules >> need not consider more than one version of any particular module. > > I get that Ivy, Maven, and Gradle all do dependency resolution and I > totally get the desire to avoid duplicating behaviour, but my experience > with Eclipse/OSGi is such that runtime dependency resolution is > different from build-time dependency resolution. Remember that there's still a notion of run-time resolution, it's just that it's (intentionally) very simple. It can give a result that's different from build-time resolution, but it's not going to do anything fancy like run a SAT solver or try to apply clever dependency-management heuristics. > Of course, it's probably the case that what we have here is exactly what > ~99% of the Java community requires. Yes, that's the aim. IDEs, app servers, and other sophisticated types of applications will always require something more, so let's make sure that the dynamic-configuration requirements make that possible while keeping it easy for simpler applications to leverage the module system. - Mark From mark.reinhold at oracle.com Mon Feb 16 01:26:31 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Sun, 15 Feb 2015 17:26:31 -0800 Subject: Enhanced ServiceLoader? In-Reply-To: References: Message-ID: <20150215172631.519378@eggemoggin.niobe.net> 2015/2/11 7:05 -0800, Tim Boudreau : > *Binding ? The Java run-time system must bind service implementations to > interfaces and make these bindings available via an enhanced version of the > existing java.util.ServiceLoader API.* > > Are there any specifics as to what is meant by "enhanced" here? Well, at the very least it means, "enhanced so as to integrate it with the module system so that service bindings can be based upon the available modules rather than (or in addition to) whatever META-INF/services files happen to be found on the class path." The ordering of service providers is a tricky topic. I agree with Peter when he says that explicit ordering requires a global view of the run-time system. That's not possible from the perspective of a single module, which is why an @Ordered annotation makes no sense, but sometimes you actually do have a global (or sufficiently-global) view. When assembling an entire standalone application, e.g., it could well make sense, as I think David was trying to suggest, to allow an order to be specified when linking the application together. It could also make sense to allow a sophisticated application that's managing modules on its own, via dynamic configurations, to specify an order in the context of a particular configuration. That way an app server could, e.g., control which providers are seen, and in what order, by the web apps that it hosts. In any case, I think this is a somewhat detailed (and, also, somewhat orthogonal) design issue which we can tackle later on, unless someone thinks it's critical to mention it in the requirements. - Mark From niftiness at gmail.com Mon Feb 16 06:12:18 2015 From: niftiness at gmail.com (Tim Boudreau) Date: Mon, 16 Feb 2015 01:12:18 -0500 Subject: Module-system requirements In-Reply-To: <20150215172507.827110@eggemoggin.niobe.net> References: <20150211103831.528761@eggemoggin.niobe.net> <20150215172507.827110@eggemoggin.niobe.net> Message-ID: > > As I wrote in my reply to David, I think resources (in the traditional > ClassLoader::getResource sense) should be treated as being strictly > internal to modules. If a module needs to make the information in one > of its resources available to other modules then it should do so via a > properly-exported class, or a service provider. Just because JAR files > are miniature filesystems which anybody can dissect doesn't mean that we > need to conceive of Java modules in the same way. > There's a common case that I'd like to understand how you solve without at least META-INF or its equivalent visible to other modules: It's fairly common to see applications that dynamically compose things they find on the classpath. Spring and a few other things accomplish this through classpath-scanning, which is a fairly horrible pattern (you can wind up visiting every class on the classpath, or a large subset, just to look for a runtime annotation that is not there for the vast majority). An alternative approach I've used many times, which I think a few frameworks do as well (dagger?) is: Write an annotation processor that generates a flat file in META-INF that lists the classes that have the annotation in question; runtime code loads all such files on the classpath and uses that to initialize things that should be present at runtime. It's really the same pattern as ServiceLoader. Is it safe to assume that, whatever happens, that approach will still work, whether by special-casing META-INF/** or some equivalent? Having said that, I do agree that the module system must provide some > kind of escape hatch or "promiscuous mode", as you call it, so here's > a stab at an additional requirement for the "Modularizing Java SE" > section: > > - _Overrideable encapsulation_ --- It must be possible to force a > module to export one or more of its packages to all other modules, > or to some specific modules, even if its definition does not do so. > One sort of low-hanging-fruit approach to this would be: - Modules do not use the traditional $CLASSPATH - The contents of $CLASSPATH are treated by the system as a single module (so everything can see each other, and legacy code behaves as expected) -Tim From niftiness at gmail.com Mon Feb 16 06:32:19 2015 From: niftiness at gmail.com (Tim Boudreau) Date: Mon, 16 Feb 2015 01:32:19 -0500 Subject: Enhanced ServiceLoader? In-Reply-To: <20150215172631.519378@eggemoggin.niobe.net> References: <20150215172631.519378@eggemoggin.niobe.net> Message-ID: > > Well, at the very least it means, "enhanced so as to integrate it > with the module system so that service bindings can be based upon > the available modules rather than (or in addition to) whatever > META-INF/services files happen to be found on the class path." > > The ordering of service providers is a tricky topic. > > I agree with Peter when he says that explicit ordering requires a global > view of the run-time system. That's not possible from the perspective of > a single module, which is why an @Ordered annotation makes no sense, but > sometimes you actually do have a global (or sufficiently-global) view. > Happy to hear that. I agree, in principle, in a modular system, the pieces don't know about each other and so ordering seems at odds with the concept. But reality is rarely so pure (I remember someone arguing this exact point with regard to a UI designer insisting on a explicit ordering of menu items where the items were contributed by modules that didn't know about each other - thankfully, UI design won over architectural purism and Exit stayed at the bottom of the File menu). Sometimes the only interesting thing about ordering is the bounds - MIN_VALUE = high-priority/low-performance-cost, 0 = don't care, MAX_VALUE = low-priority/high-performance-cost. Anything using the filter-chain pattern to break a decision tree about what to do with some input into a pluggable list of things that can process the input needs that. The two ways this is usually seen is either comes-before/comes-after constraints + topological sort, or ad-hoc integers. Neither is perfect (both NetBeans and Gnome abandoned the before/after approach because the results are unpredictable if one of the constraints is absent); but the fact that it cannot be done perfectly without a universal view doesn't mean that having nothing is preferable. > When assembling an entire standalone application, e.g., it could well > make sense, as I think David was trying to suggest, to allow an order to > be specified when linking the application together. > > It could also make sense to allow a sophisticated application that's > managing modules on its own, via dynamic configurations, to specify an > order in the context of a particular configuration. An application assembled like that is not necessarily going to have enough knowledge of the pieces it's composed from to impose an order without some hints from the pieces themselves - that's really the catch here. I'll grant that they are hints. > That way an app > server could, e.g., control which providers are seen, and in what order, > by the web apps that it hosts. > > In any case, I think this is a somewhat detailed (and, also, somewhat > orthogonal) design issue which we can tackle later on, unless someone > thinks it's critical to mention it in the requirements. > NetBeans did the ordering hints for years by enhancing META-INF/services files with magic comments, e.g. #position 1000 com.foo.Bar It would be good if ServiceLoader could grow enough heft to handle that role - not necessarily identically - but the feature satisfies a proven need. -Tim From stef at epardaud.fr Mon Feb 16 09:00:19 2015 From: stef at epardaud.fr (Stephane Epardaud) Date: Mon, 16 Feb 2015 10:00:19 +0100 Subject: Module-system requirements In-Reply-To: References: <20150211103831.528761@eggemoggin.niobe.net> <20150215172507.827110@eggemoggin.niobe.net> Message-ID: <54E1B1A3.4040506@epardaud.fr> On 16/02/15 07:12, Tim Boudreau wrote: > There's a common case that I'd like to understand how you solve without at > least META-INF or its equivalent visible to other modules: It's fairly > common to see applications that dynamically compose things they find on the > classpath. Spring and a few other things accomplish this through > classpath-scanning, which is a fairly horrible pattern (you can wind up > visiting every class on the classpath, or a large subset, just to look for > a runtime annotation that is not there for the vast majority). Hi, Stef from the Ceylon programming language team here :) In Ceylon (which is already modular and pre-supports Jigsaw), our reflection library allows the runtime querying of all loaded modules, and from each module we can iterate its packages and from each package we can iterate types, and in particular find types with a certain annotation. I didn't have time yet to read the entire requirements document (though I intend to do so today), so I haven't seen yet if it mentions reflection, but I suggest that that'd be a really good thing to add, especially for frameworks, rather than rely on third-party classpath scanners. Also, if the JRE does the lookup of types by annotation (by scanning modules) then it can very efficiently cache that info or build an index. From david.lloyd at redhat.com Mon Feb 16 13:15:56 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 16 Feb 2015 07:15:56 -0600 Subject: Exporting things (Was: Re: Module-system requirements) In-Reply-To: <20150215172405.90174@eggemoggin.niobe.net> References: <20150211103831.528761@eggemoggin.niobe.net>, <54DBAD0A.1060005@redhat.com> <20150215172405.90174@eggemoggin.niobe.net> Message-ID: <54E1ED8C.8010208@redhat.com> On 02/15/2015 07:24 PM, mark.reinhold at oracle.com wrote: > 2015/2/11 11:27 -0800, david.lloyd at redhat.com: >> I notice that under Exports, it is defined that only packages (i.e. >> classes within those packages) are considered to be exportable, and not >> resources. I'm not certain how I feel about that... in our >> implementation we flattened the concepts of packages and directories. >> This was mainly to solve a few specific difficulties of supporting plain >> JARs as module content; a JAR has a mix of classes and resources, and it >> is hard to deterministically tell the difference, especially when you >> can do such (commonly done) things as: >> classLoader.getResource("org/foo/bar/SomeClass.class"). >> >> ... >> >> However this approach is not without its difficulties: ... >> >> ... To sum up though, I'd be glad to see (from my perspective) this >> behavior go away in favor of the simpler model of restricting exports to >> packages (and isolating classes and resources), > > I tend to agree. > > Exposing all of a module's resources, as JAR files do today, is wrong > from the perspective of encapsulation. Extending the "exports" > requirement to allow control over which resources are exported leads to > both conceptual and implementation complexity (e.g., what would it mean > to export both a class, as a type, as well as the corresponding .class > file, or the latter and not the former?) It seems sanest to treat > resources as strictly internal to modules. > > This suggests an additional requirement: > > - _Resource encapsulation_ --- The run-time system must ensure that the > static resource files within a module are directly accessible only by > code within that module. What you are saying here implies (to me) inspecting the call stack to ensure that the caller actually exists within the module being referenced. I'm not certain we really want to go this far, if my interpretation is accurate. Such a rule is reasonable on the face of it, but it does more or less break all the code in existence which currently uses resources as a sort of API into a library, and this includes very common patterns for loading configuration (among many other things). >> but only if we can be >> reasonably sure that the use cases solved thereby are somehow sorted >> out, which include: >> >> ? Supporting this case (or determining that it is not relevant): A >> module privately uses services provided by one or more specific peer >> module(s) (via ServiceLoader), the arrangement of which is determined >> statically by distribution. > > One way to accomplish this is to make the service interface accessible > only by the relevant modules, via qualified exports. Then only those > modules can provide or use implementations of the service. > > Another possibility is to add a requirement that the binding of providers > to service interfaces can be controlled manually during the configuration > process. Is that more along the lines of what you mean when you say > "determined statically by the distribution"? Yes it *could* be, assuming that configurations end up being sufficiently flexible (e.g. I have one distribution configuration, and then I have N application configurations for each installed application, which is in turn an extension of the distribution configuration). It's a question, I guess, of how the configurations work and how much power the application configuration has to customize/mess up the distribution configuration. In a setup like we have, where one module can just statically import another's services, it's pretty simple for the distributor to set it up and really difficult for an application to mess it up. >> ? Having (generally) the same visibility to .class files as to the >> classes themselves, from a given resource-loading entry point (Class or >> ClassLoader), i.e. if I have visibility to >> classLoader.loadClass("SomeClass"), I also have visibility to >> classLoader.getResource("SomeClass.class"), and vice-versa (any security >> restrictions aside). > > Can you describe some use cases that require this property? Just doing a quick spin through libraries used or referenced by WildFly and its dependencies: Libraries such as CGLIB and BCEL use this to modify existing classes. Ant apparently uses it to find JavaCC, though this odd use is probably better done another way. Hibernate uses it to get bytes to instrument in its instrumenting class loader, as does our JPA SPI. We also use this trick to generate various fast indexes of classes without actually loading the classes themselves for various tools and deployment operations. I believe we have some tooling using this as well to do various types of analysis on loaded code. > What if a module is compiled to native code ahead-of-time, and hence > doesn't necessarily contain a .class file for every class? Then things like instrumentation will fail in what is I guess a self-obviating manner. Things like JPA which may count on instrumenting classes just won't work in such environments. >> ? A consistent ruling on the right behavior of general resources which >> are co-packaged with classes that use them (typically loaded via e.g. >> getClass().getResourceAsStream("MyResource.properties") or similar). > > If we take the view that resources are module-internal then the answer > is that code within a module can access that module's resources but code > outside of the module cannot (with the usual exception for reflection, > which would require a security check). This brings us close to a previous point of contention regarding having the implementation require a class loader for each module (or not); the problem never came up for us (even in security manager environments) because acquiring a class loader already entails a security check. >> ? A consistent ruling on whether it makes sense generally to be able to >> export resources to dependents. > > If we take the view that resources are module-internal then the answer is > no, this does not make sense. A module can export packages of classes > and interfaces, and can publish service providers; it cannot also export > files in a string-based hierarchical namespace. I guess the salient underlying question is, "why not?". I'm being (for the most part) a devil's advocate here, but I think this question really has to be answered succinctly, because we're really trying to change a basic (de facto) principle of Java library interoperation (that they often use resources as a sort of API, even if only primarily for service loader-style and configuration use cases). I can easily imagine an "export resources" feature which specifies resource directories to make exportable (due in no small part to the fact that such a feature has been used extensively by us with good results). Another use case beyond those mentioned above that is satisfied by such a capability could be bundling supplementary i18n/l10n or time zone data or similar as a module. >> A minor, somewhat related point that this raises... >> >> The term "dependences" (rarely-used pl. of "dependence") is used quite a >> lot in the document (whether intentional or otherwise), but I think that >> the term "dependencies" (pl. of "dependency") is probably a better term, >> and is definitely a more ubiquitous one. The inverse of "dependency" is >> "dependent", which forms a concise term pair that we use quite a lot >> internally (i.e. if A is a dependency of B, B is a dependent of A). > > You're not the first to ask me about this ... > > "Dependence" and "dependency" name distinct concepts. A dependence is a > relationship, potentially unfulfilled; a dependency is a thing which can > fulfill a relationship of dependence. When thinking, speaking, or > writing about modules I find it useful to keep this distinction clear. > > A module, standing alone, can have dependences upon some other, > yet-to-be-identified modules. All that is known is that they have > specific names, or that they provide specific services. The module's > definition, in other words, merely describes relationships and > constraints upon how those relationships can be fulfilled; it does not > identify the specific modules that will fulfill those relationships. > > The processes of resolution and service binding identify the actual > modules, i.e., the dependencies, that will satisfy a module's > dependences. OK that makes sense to me. > I agree that "dependent" is also a useful term. In ASCII-art form the > three concepts are related thus: > > Dependence > Dependent --------------> Dependency > > - Mark > -- - DML From david.lloyd at redhat.com Mon Feb 16 13:57:48 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 16 Feb 2015 07:57:48 -0600 Subject: Interoperation (Was: Re: Module-system requirements) In-Reply-To: <20150215172424.772730@eggemoggin.niobe.net> References: <20150211103831.528761@eggemoggin.niobe.net>, <54DBB3E6.8060202@redhat.com> <20150215172424.772730@eggemoggin.niobe.net> Message-ID: <54E1F75C.4070009@redhat.com> On 02/15/2015 07:24 PM, mark.reinhold at oracle.com wrote: > 2015/2/11 11:56 -0800, david.lloyd at redhat.com: >> I'm concerned that the interoperation point doesn't quite go far enough. >> It says "It must be possible for another module system, such as OSGi, >> to locate Java modules and resolve them using its own resolver, except >> possibly for core system modules." >> >> I agree with this requirement, such as it is, but I would also assert >> that it should be possible to retrofit not only OSGi but Java EE, Maven, >> and even legacy "modules" like extensions and so on as modules. The >> reason I make this assertion is that we have done all of these things >> with success and (I dare say) relative ease, even to the extent of >> achieving both EE and OSGi certification with such a retrofit. > > Can you explain what you mean, exactly, by "retrofit"? We have implemented both a certified OSGi implementation as well as a certified Java EE implementation which maps these specifications' module concepts onto our own concept of a module. Also we have multiple projects which consume Maven dependency graphs in various ways to map these graphs onto our module concept, which works for the most part (though Maven dependency graphs are often overly inclusive, since the build system does not really do much to discourage erring on the side of pulling in everything you can "just in case"). > Are you saying that the JBoss module system can consume Java EE WAR > files, OSGi bundles, and Maven projects, as they stand and without > change? Yes*. This is because of implementation choices we've made - each module is a class loader, and we can carefully control which classes *and* resources are exported from one to another. That said, I do think it's still worth exploring a different approach than ours regarding conceptually combining resource directories and packages, which is what we do today and is not without disadvantage. * Occasionally there is a library that makes deep or odd assumptions about the class path or class loader behavior, *but* in practice these are typically older libraries that already don't work in EE and OSGi environments. Overall we found that the vast majority of libraries work just fine in our modular environment. >> ... >> >> To achieve this additional requirement does require a few very specific >> implementation considerations but I do not believe that any of these >> considerations contraindicate any of the other requirements stated here. > > Can you be more specific as to what those considerations are? Can they > be expressed as module-system requirements? In the case of both Java EE and OSGi, I think it's safe to say that you need to apply a strict 1:1 mapping between modules and class loaders for this to work. Also you need an API to allow run-time systems to create their own configurations (we call them "module loaders", akin in purpose and function to class loaders) and assemble modules from resources (JARs) and dependency specifications, and such APIs have to be able to relink modules or deregister modules to allow them to be GC'd (though the boot module loader in our case does not expose relink or unload functionality for obvious reasons). OSGi specifically requires that bundle class loaders implement an OSGi spec interface, implying that the module class loader class must be extensible (as I said a couple of years back though, we make most of our methods final on the module class loader to disallow any really bizarre behavior, and this works well in practice). We also had to implement a pre- and/or post-class-define hook per module to allow some OSGi magic to happen. Finally there is a per-module fallback resource loader that our OSGi implementation used for one particular part of the spec. Java EE's requirements are much simpler and are satisfied by module-granularity dependencies and (in certain cases) resource importing. The APIs to do these things are all pretty simple though. >> Thus I feel this requirement should be amended to something along the >> lines of: >> >> Interoperation - It must be possible for another module system, such as >> OSGi, to locate Java modules and resolve them using its own resolver, >> except possibly for core system modules. It must be possible for other >> module systems to reuse the framework provided by this specification in >> order to facilitate a standard, well-defined mechanism by which >> different present and future module systems may interoperate. > > Unless I've misunderstood something, this sounds more like a research > project -- a "meta" module system, in fact -- which is well outside the > scope of this JSR. Maybe, but I contend that a lot (enough, insofar as any amount is ever enough) of research has been done, and that in any event there will have to be an API which implements the functionality of this JSR anyway, and it will be used as such no matter our intent so IMO it's best to use what has been learned to solve the problem in the best manner possible. -- - DML From david.lloyd at redhat.com Mon Feb 16 14:12:50 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 16 Feb 2015 08:12:50 -0600 Subject: Configurations... (Was: Re: Module-system requirements) In-Reply-To: <20150214133307.760777@eggemoggin.niobe.net> References: <20150211103831.528761@eggemoggin.niobe.net>, <54DCC82C.4030200@redhat.com> <20150214133307.760777@eggemoggin.niobe.net> Message-ID: <54E1FAE2.2000605@redhat.com> On 02/14/2015 03:33 PM, mark.reinhold at oracle.com wrote: > 2015/2/12 7:35 -0800, david.lloyd at redhat.com: >> I'm a little concerned that the configurations section implies a strict >> hierarchy: >> >> "Nested dynamic configurations ? An application must be able to create a >> dynamic configuration that relates to an existing dynamic configuration >> rather than to the application?s initial configuration." >> >> Why can a configuration not relate to more than one other configurations? > > A tree of configurations rather than a simple linear list is more > difficult to reason about and trickier to implement (you must, e.g., > cope with the diamond-import problem). > > The feedback we've received so far has not suggested that this > limitation will be a problem in practice. Do you have use cases > which require this flexibility? We implement not just a tree, but a general directed graph (even allowing cycles) among modules and among their configurations (module loaders), which enables us to easily support OSGi and Java EE, as well as tying these things to filesystem JARs (e.g. for EE deployments which have Class-Path declarations that have an absolute path name), and the legacy extensions mechanism which is presently also a part of Java EE (though this is going/has gone away, I understand), and also allows tying in other systems as well. Due to the way we support imports, diamonds are really no problem. Each module's dependency list is interpreted in order, so the set of imported resources is always the union of everything exported by each dependency. So if you acquire a different set of items via two different "paths" to a module, you get the union of those sets in the end. Duplicated paths are removed such that the first occurrence is used. -- - DML From Tim_Ellison at uk.ibm.com Mon Feb 16 17:04:01 2015 From: Tim_Ellison at uk.ibm.com (Tim Ellison) Date: Mon, 16 Feb 2015 17:04:01 +0000 Subject: Security (was: Re: Module-system requirements) In-Reply-To: <20150215172507.827110@eggemoggin.niobe.net> References: <20150211103831.528761@eggemoggin.niobe.net>, <20150215172507.827110@eggemoggin.niobe.net> Message-ID: mark.reinhold at oracle.com wrote on 16/02/2015 01:25:07: > 2015/2/13 6:06 -0800, tim_ellison at uk.ibm.com: > > > > - It would be useful to have a section on Security. While there are > > aspects throughout other sections of the requirements, such as > > encapsulation and referential integrity, there may be other considerations > > where module boundaries can be used to enhance security, e.g. only > > allowing privilege escalation within a module scope, modules declaring all > > exported APIs fail when passed a tainted string, etc. I'm not proposing > > scope-creep but rather ensuring folks have had the chance to think about > > it and easy opportunities are not missed to make Java 9 even more secure. > > I'm all for improving security (obviously!), but if I've learned one > thing about that general topic over the years it's that it's more a > state of mind, or a way of life, than a set of features. It runs > through everything that we do. > > Putting every requirement that's somehow related to security into a > special "security" section of this document would just make it harder to > read, and I'm skeptical that it'd actually make us all think even more > about security than we already do. If you'd like to propose additional > security-related requirements then by all means please go ahead, and > we'll see where they fit into the present structure. Presently, JAR files fulfil a number of different roles, including being one of the structured container types for resources. As modules are introduced there is an opportunity to use this artifact for some security concepts that are container-based. For example, while it may be implied by the 'Non-interference' requirement, it may be desirable to consider modules to be sealed, thereby guaranteeing that all classes in a package come from the same module. I was going to suggest that modules be considered a CodeSource so that they can be signed, or form a protection domain with a configurable security policy, without consideration for their (file system) container - but I may have talked myself out of it . Perhaps these concepts should remain with the container, notwithstanding the fact that a linked custom binary run-time image may well collapse the containers and invalidate any signing/policies associated with the originals. A security algorithm implementation provider may always have to be part of a separate verifiable container to maintain the current level of integrity. Regards, Tim Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU From Tim_Ellison at uk.ibm.com Mon Feb 16 17:04:01 2015 From: Tim_Ellison at uk.ibm.com (Tim Ellison) Date: Mon, 16 Feb 2015 17:04:01 +0000 Subject: Non-interference (was: Re: Module-system requirements) In-Reply-To: <20150215172507.827110@eggemoggin.niobe.net> References: <20150211103831.528761@eggemoggin.niobe.net>, <20150215172507.827110@eggemoggin.niobe.net> Message-ID: mark.reinhold at oracle.com wrote on 16/02/2015 01:25:07: > 2015/2/13 6:06 -0800, tim_ellison at uk.ibm.com: > > >> Non-interference ? The Java compiler, virtual machine, and run-time > >> system must ensure that modules that contain packages of the same > >> name do not interfere with each other. > > > > Can you explain "interfere"? Is this referring to ambiguous class and > > linkage error type problems, or something else here? > > The former. If two distinct modules contain packages of the same name > then code in that package in one module must not be able to access > package-private types or members in that package in the other module. > If two distinct modules contain classes of the same name (i.e., with the > same binary name) then one must not be able to stand in for the other. Ok, so the module name partitions the package name space. Consider the following change for the Exports fundamental definition: Exports ? A module?s definition must be able to indicate which of its packages are exported for use by other modules, and which are not. Multiple modules may export packages of the same name. Regards, Tim Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU From Tim_Ellison at uk.ibm.com Mon Feb 16 17:21:23 2015 From: Tim_Ellison at uk.ibm.com (Tim Ellison) Date: Mon, 16 Feb 2015 17:21:23 +0000 Subject: Module-system requirements In-Reply-To: References: <20150211103831.528761@eggemoggin.niobe.net> <20150215172507.827110@eggemoggin.niobe.net> Message-ID: Tim Boudreau wrote on 16/02/2015 06:12:18: > As I wrote in my reply to David, I think resources (in the traditional > ClassLoader::getResource sense) should be treated as being strictly > internal to modules. If a module needs to make the information in one > of its resources available to other modules then it should do so via a > properly-exported class, or a service provider. Just because JAR files > are miniature filesystems which anybody can dissect doesn't mean that we > need to conceive of Java modules in the same way. > > There's a common case that I'd like to understand how you solve > without at least META-INF or its equivalent visible to other > modules: It's fairly common to see applications that dynamically > compose things they find on the classpath. Spring and a few other > things accomplish this through classpath-scanning, which is a fairly > horrible pattern (you can wind up visiting every class on the > classpath, or a large subset, just to look for a runtime annotation > that is not there for the vast majority). > > An alternative approach I've used many times, which I think a few > frameworks do as well (dagger?) is: Write an annotation processor > that generates a flat file in META-INF that lists the classes that > have the annotation in question; runtime code loads all such files > on the classpath and uses that to initialize things that should be > present at runtime. It's really the same pattern as ServiceLoader. > > Is it safe to assume that, whatever happens, that approach will > still work, whether by special-casing META-INF/** or some equivalent? > > Having said that, I do agree that the module system must provide some > kind of escape hatch or "promiscuous mode", as you call it, so here's > a stab at an additional requirement for the "Modularizing Java SE" > section: > > - _Overrideable encapsulation_ --- It must be possible to force a > module to export one or more of its packages to all other modules, > or to some specific modules, even if its definition does not do so. > > One sort of low-hanging-fruit approach to this would be: > - Modules do not use the traditional $CLASSPATH > - The contents of $CLASSPATH are treated by the system as a single > module (so everything can see each other, and legacy code behaves asexpected) Mark's new requirement would do it - though I had to swallow hard after reading the definition, as I heard all the safety-catches being clicked off. I'm struggling to see how treating the contents of the classpath as a single module would help here? The problem is with platform modules hiding previously accessible code. Regards, Tim Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU From niftiness at gmail.com Mon Feb 16 20:32:35 2015 From: niftiness at gmail.com (Tim Boudreau) Date: Mon, 16 Feb 2015 15:32:35 -0500 Subject: Module-system requirements In-Reply-To: References: <20150211103831.528761@eggemoggin.niobe.net> <20150215172507.827110@eggemoggin.niobe.net> Message-ID: > > > > One sort of low-hanging-fruit approach to this would be: > > - Modules do not use the traditional $CLASSPATH > > - The contents of $CLASSPATH are treated by the system as a single > > module (so everything can see each other, and legacy code behaves > asexpected) > > Mark's new requirement would do it - though I had to swallow hard after > reading the definition, as I heard all the safety-catches being clicked > off. > > I'm struggling to see how treating the contents of the classpath as a > single module would help here? The problem is with platform modules > hiding previously accessible code. > I'm not sure it does help with that. I'm sure in this JSR there will be lots of different problems that each of us thinks of as The Most Important Problem to Solve. The idea I was trying to get across is that, if you want to partition the world up into modules and do that in a compatible way, then start by modularizing the JDK, and giving *it* a new way to link. At the application layer, keep the default behavior by treating everything linked the traditional way as, for the purposes of the framework, a single large "default" module where all the old rules apply. As libraries modularize, at their own pace, they move out of that "default' module and become modules in their own right, and new linking rules apply. That's not a model for how to access newly hidden code - that's a negotiation between the user of that hidden code and its owner (though it might be facilitated by the JDK and everything else being modularized into more, smaller modules). It's just a way to compatibly evolve toward a modular universe as opposed to trying to carve up the existing universe into modules in a way that doesn't break stuff, which is probably impossible. -Tim From david.lloyd at redhat.com Fri Feb 20 14:33:36 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Fri, 20 Feb 2015 08:33:36 -0600 Subject: Exporting things (Was: Re: Module-system requirements) In-Reply-To: <54E1ED8C.8010208@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net>, <54DBAD0A.1060005@redhat.com> <20150215172405.90174@eggemoggin.niobe.net> <54E1ED8C.8010208@redhat.com> Message-ID: <54E745C0.80603@redhat.com> On 02/16/2015 07:15 AM, David M. Lloyd wrote: > On 02/15/2015 07:24 PM, mark.reinhold at oracle.com wrote: >> 2015/2/11 11:27 -0800, david.lloyd at redhat.com: >>> I notice that under Exports, it is defined that only packages (i.e. >>> classes within those packages) are considered to be exportable, and not >>> resources. I'm not certain how I feel about that... in our >>> implementation we flattened the concepts of packages and directories. >>> This was mainly to solve a few specific difficulties of supporting plain >>> JARs as module content; a JAR has a mix of classes and resources, and it >>> is hard to deterministically tell the difference, especially when you >>> can do such (commonly done) things as: >>> classLoader.getResource("org/foo/bar/SomeClass.class"). >>> >>> ... >>> >>> However this approach is not without its difficulties: ... >>> >>> ... To sum up though, I'd be glad to see (from my perspective) this >>> behavior go away in favor of the simpler model of restricting exports to >>> packages (and isolating classes and resources), >> >> I tend to agree. >> >> Exposing all of a module's resources, as JAR files do today, is wrong >> from the perspective of encapsulation. Extending the "exports" >> requirement to allow control over which resources are exported leads to >> both conceptual and implementation complexity (e.g., what would it mean >> to export both a class, as a type, as well as the corresponding .class >> file, or the latter and not the former?) It seems sanest to treat >> resources as strictly internal to modules. >> >> This suggests an additional requirement: >> >> - _Resource encapsulation_ --- The run-time system must ensure that >> the >> static resource files within a module are directly accessible >> only by >> code within that module. > > What you are saying here implies (to me) inspecting the call stack to > ensure that the caller actually exists within the module being > referenced. I'm not certain we really want to go this far, if my > interpretation is accurate. Such a rule is reasonable on the face of > it, but it does more or less break all the code in existence which > currently uses resources as a sort of API into a library, and this > includes very common patterns for loading configuration (among many > other things). Going into a little more detail on this, I picture a number of scenarios: 1) A resource is fully internal to a module, and there's no reason that any external code ought to need or get access to this data. The module could restrict access to the resource by caller as inferred above. 2) A resource is an internal part of the module, but other things might need access to the resource for specific purposes (like direct .class access for instrumentation). The access restriction of this use case is "solved" by existing containers today by isolating class loaders, so the user has to have access to the target class loader in order to read the resource(s) in question. You could subdivide this into (2a) the .class files themselves and (2b) anything else that might fall under this category, as I believe the use cases for (2a) are fairly specific to run-time analysis and instrumentation tooling and similar uses. 3) A resource is a part of the module's self-configuration, but is not of interest to importers. This would include configuration files (like Java EE deployment descriptors, and other framework-XML-ish things). Access is typically presently solved similarly to #2 above, or by simply accessing the target resource archive directly, bypassing class loading. 4) A resource is a part of a public API intended to be imported and used by consumers. Examples include framework configurations like logging and some other framework-XML-ish things, ServiceLoader, etc. Today this is solved as an unavoidable consequence of flat classpaths, but also (in containers) by implicitly or explicitly exporting and importing some or all of a class loader's resource file/directory structure. While JBoss Modules does in fact solve cases 2-4 well enough, there is no provision for things falling under (1). This is not however due to any fundamental ideal; quite the opposite in fact. I simply could not work out a consistent (and simple) set of rules for treating classes and packages separately from resources and their directories that didn't simply break everything*. The questions are: Are there yet more use cases that I have failed to identify? Is it possible to come up with a consistent set of rules which allows all these primary use cases to continue to function cleanly going forward? Can we do so in a minimally-impacting manner that would allow most existing code to continue to function with (at most) only packaging changes? If not, which use cases can or should be sacrificed, and why? I could easily imagine adding a separate public resources concept which is wholly separate from the module internal content, where importers could import some or all of these resources into their own world, which would easily solve (4), but (2) and (3) are still left "high and dry". An "unexported resources" area of a module could address (3), leaving (2), which could be solved by co-locating those resources with the classes themselves. But by this point there is some additional complexity (maybe not *too* much, but definitely some). Module imports would specify the module they want to import (possibly to package granularity), as well as the resource directories they want to import (which could be re-exported just like module packages could). So from that perspective the additional complexity is manageable. But when it comes to build and packaging of a module itself, users would now have face possibly fairly complex decision regarding which category a resource belongs to (not to mention that packaging tooling would have to support this decision), whereas in today's world they just "throw it in there" and hope for the best. * Though I admit the idea of examining the call stack for this purpose hadn't occurred to me, but I'm not sure I could have done anything concrete with that idea anyway due to the myriad complex paths by which class loader methods are invoked (especially in a multiple-delegation situation like ours), not to mention the performance problems (such an approach would really need [1] to be solved performantly as a prerequisite). [1] https://bugs.openjdk.java.net/browse/JDK-8043814 -- - DML From david.lloyd at redhat.com Fri Feb 20 21:48:08 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Fri, 20 Feb 2015 15:48:08 -0600 Subject: Security In-Reply-To: References: <20150211103831.528761@eggemoggin.niobe.net>, <20150215172507.827110@eggemoggin.niobe.net> Message-ID: <54E7AB98.3030005@redhat.com> On 02/16/2015 11:04 AM, Tim Ellison wrote: > mark.reinhold at oracle.com wrote on 16/02/2015 01:25:07: >> 2015/2/13 6:06 -0800, tim_ellison at uk.ibm.com: >>> >>> - It would be useful to have a section on Security. While there are >>> aspects throughout other sections of the requirements, such as >>> encapsulation and referential integrity, there may be other > considerations >>> where module boundaries can be used to enhance security, e.g. only >>> allowing privilege escalation within a module scope, modules declaring > all >>> exported APIs fail when passed a tainted string, etc. I'm not > proposing >>> scope-creep but rather ensuring folks have had the chance to think > about >>> it and easy opportunities are not missed to make Java 9 even more > secure. >> >> I'm all for improving security (obviously!), but if I've learned one >> thing about that general topic over the years it's that it's more a >> state of mind, or a way of life, than a set of features. It runs >> through everything that we do. >> >> Putting every requirement that's somehow related to security into a >> special "security" section of this document would just make it harder to >> read, and I'm skeptical that it'd actually make us all think even more >> about security than we already do. If you'd like to propose additional >> security-related requirements then by all means please go ahead, and >> we'll see where they fit into the present structure. > > Presently, JAR files fulfil a number of different roles, including being > one of the structured container types for resources. As modules are > introduced there is an opportunity to use this artifact for some security > concepts that are container-based. For example, while it may be implied > by the 'Non-interference' requirement, it may be desirable to consider > modules to be sealed, thereby guaranteeing that all classes in a package > come from the same module. > > I was going to suggest that modules be considered a CodeSource so that > they can be signed, or form a protection domain with a configurable > security policy, without consideration for their (file system) container - > but I may have talked myself out of it . Perhaps these concepts should > remain with the container, notwithstanding the fact that a linked custom > binary run-time image may well collapse the containers and invalidate any > signing/policies associated with the originals. In JBoss Modules, we did this very thing, with a simple per-module protection domain concept where the permissions are actually established within the module description itself. This dovetails nicely into Java EE 7 permission specifications, since we can use the same description API to assign those permissions, where the container configuration defines minimal and maximal permission sets that the deployment permissions are matched against. Other ideas I considered experimenting with were per-code-source permissions (since a JBoss Module can aggregate more than one resource root) as well as per-class/package permissions (maybe via annotations). I think there is definitely value in the module knowing what permissions it needs to function, and to be shipped with those permissions. I think that if this is combined with a configuration-specific verification mechanism, this could allow users to express a level of trust the way they do today for signed JARs, and/or perhaps be able to verify (at install time) whether or not they want to go ahead with installing a module with certain permissions. > A security algorithm implementation provider may always have to be part of > a separate verifiable container to maintain the current level of > integrity. Agreed. -- - DML From david.lloyd at redhat.com Fri Feb 20 22:23:04 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Fri, 20 Feb 2015 16:23:04 -0600 Subject: Security In-Reply-To: References: <20150211103831.528761@eggemoggin.niobe.net>, <20150215172507.827110@eggemoggin.niobe.net> Message-ID: <54E7B3C8.30709@redhat.com> On 02/16/2015 11:04 AM, Tim Ellison wrote: > For example, while it may be implied > by the 'Non-interference' requirement, it may be desirable to consider > modules to be sealed, thereby guaranteeing that all classes in a package > come from the same module. Meant to comment on this too, just to agree that it is important for a package's identity to be connected to the identity of its containing module. I don't think the non-interference requirement quite goes as far as it could to stipulate this (though it's arguably more of an issue for Encapsulation since it pertains mostly to accessibility). Relatedly, the Encapsulation point is somewhat weaker than some of its elder cousins in the old Jigsaw draft requirements, in that it doesn't come out and say that the language needs a per-module class/member access level. I'm not sure if it's just something that we need to visit once we start talking about implementation, or if the concept was abandoned in favor of visibility-style controls as the requirements presently seem to imply, by my reading anyway. -- - DML From david.lloyd at redhat.com Fri Feb 20 22:44:09 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Fri, 20 Feb 2015 16:44:09 -0600 Subject: Module-system requirements In-Reply-To: References: <20150211103831.528761@eggemoggin.niobe.net> <20150215172507.827110@eggemoggin.niobe.net> Message-ID: <54E7B8B9.7040004@redhat.com> On 02/16/2015 02:32 PM, Tim Boudreau wrote: >> >> >>> One sort of low-hanging-fruit approach to this would be: >>> - Modules do not use the traditional $CLASSPATH >>> - The contents of $CLASSPATH are treated by the system as a single >>> module (so everything can see each other, and legacy code behaves >> asexpected) >> >> Mark's new requirement would do it - though I had to swallow hard after >> reading the definition, as I heard all the safety-catches being clicked >> off. >> >> I'm struggling to see how treating the contents of the classpath as a >> single module would help here? The problem is with platform modules >> hiding previously accessible code. >> > > I'm not sure it does help with that. I'm sure in this JSR there will be > lots of different problems that each of us thinks of as The Most Important > Problem to Solve. > > The idea I was trying to get across is that, if you want to partition the > world up into modules and do that in a compatible way, then start by > modularizing the JDK, and giving *it* a new way to link. At the > application layer, keep the default behavior by treating everything linked > the traditional way as, for the purposes of the framework, a single large > "default" module where all the old rules apply. As libraries modularize, > at their own pace, they move out of that "default' module and become > modules in their own right, and new linking rules apply. > > That's not a model for how to access newly hidden code - that's a > negotiation between the user of that hidden code and its owner (though it > might be facilitated by the JDK and everything else being modularized into > more, smaller modules). It's just a way to compatibly evolve toward a > modular universe as opposed to trying to carve up the existing universe > into modules in a way that doesn't break stuff, which is probably > impossible. I don't think this is impossible at all, depending on how much the rules change in the final implementation. Our WildFly environment is a JBoss Modules based modular environment of over 300 modules, many of which are outside projects, and nearly 100% of that code worked fine without any changes (the changes that were required were generally trivial and most were contributed back upstream AFAIK). So having *a* modular environment which is largely compatible with today's world is clearly possible, it's just a question of how much we want/need to change things to meet these requirements. Incidentally, I added on a mechanism to JBoss Modules near the beginning to let you run a "-jar my.jar" file (with Class-Path support) as well as to set up a -classpath and run a main class just like you do today. Both of these mechanisms work, coincidentally, by assembling the "class path" into a module, but also allowing that module to express direct dependencies on modules in the module path, by using a simple -dependencies switch (in the -classpath case) or a "Dependencies:" MANIFEST.MF header (in the -jar case). This works quite well as a bridge between worlds in our case. The way it worked out was quite elegant actually. Since we have a module loader concept (probably roughly equivalent to the "configuration" concept in the current requirements document), and a boot module loader (which equals the flat file system module image), this was achieved by simply adding another module loader for the target JAR or class path, with a resource root per class path item (simulating a localized flat classpath). In this way the isolated classpath module can link against modules from the boot module loader using the dependency constructs, without actually polluting that module space in any way (i.e. dependencies cannot go the other direction to the class path). Setting the TCCL to the application class path module ties up the picture and allows almost all of your normal class loader constructs to function as they always have. -- - DML From david.lloyd at redhat.com Wed Feb 25 19:01:49 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 25 Feb 2015 13:01:49 -0600 Subject: Module, Class, and Resource Iteration (Was: Re: Module-system requirements) In-Reply-To: <20150211103831.528761@eggemoggin.niobe.net> References: <20150211103831.528761@eggemoggin.niobe.net> Message-ID: <54EE1C1D.3020400@redhat.com> One thing that is missing from the requirements is the ability to iterate or enumerate modules, classes, and/or resources. This is one capability that people seem to be assuming will be magically provided by a module solution. While it seems to me that resource or resource directory iteration is (modulo the discussion on the "Exporting things" sub-thread) easily accomplished, defining the meaning of module iteration and class iteration is a little bit more involved. To iterate modules implies that the configuration has a complete picture of all the modules it can load, and that it can iterate over this picture in a relatively inexpensive manner. This is probably reasonable, though in my imagination I see that the set of available modules may change over time within the life of a single VM process. Iterating classes is a stranger expectation though, because while it is a reasonable inclination to want to try this, it is possibly actively harmful to start loading previously-unloaded classes. So such a function would either need to restrict its output set to loaded classes, or do something else like just enumerate class names. In addition, there are security considerations to take into account; a caller should only "see" classes that are accessible to it (though things like frameworks may need an elevated capability in order to introspect non-public classes). Another potentially iterable "thing" is packages, either in the form of names or Package objects. No comment on the applicability or lack thereof of such a capability though. -- - DML From stef at epardaud.fr Thu Feb 26 09:25:03 2015 From: stef at epardaud.fr (Stephane Epardaud) Date: Thu, 26 Feb 2015 10:25:03 +0100 Subject: Module, Class, and Resource Iteration (Was: Re: Module-system requirements) In-Reply-To: <54EE1C1D.3020400@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net> <54EE1C1D.3020400@redhat.com> Message-ID: <54EEE66F.4080509@epardaud.fr> I don't know if it's assumed or not but it's very useful for frameworks. In Ceylon we have that ability to iterate loaded modules, their packages and those packages' types, possibly filtered by annotations or predicates such as supertypes. I don't think it's very useful to add not-yet-loaded modules in the list, though. As for inexpensive listing of classes, if the reflection API provides us with a means to load types given a predicate (annotation, supertype) then it can check those types without loading the classes, pretty much like all frameworks do with third-party scanners nowadays. From niftiness at gmail.com Thu Feb 26 21:11:06 2015 From: niftiness at gmail.com (Tim Boudreau) Date: Thu, 26 Feb 2015 16:11:06 -0500 Subject: Module, Class, and Resource Iteration (Was: Re: Module-system requirements) In-Reply-To: <54EE1C1D.3020400@redhat.com> References: <20150211103831.528761@eggemoggin.niobe.net> <54EE1C1D.3020400@redhat.com> Message-ID: On Wed, Feb 25, 2015 at 2:01 PM, David M. Lloyd wrote: > One thing that is missing from the requirements is the ability to iterate > or enumerate modules, classes, and/or resources. This is one capability > that people seem to be assuming will be magically provided by a module > solution. > > While it seems to me that resource or resource directory iteration is > (modulo the discussion on the "Exporting things" sub-thread) easily > accomplished, defining the meaning of module iteration and class iteration > is a little bit more involved. > > To iterate modules implies that the configuration has a complete picture > of all the modules it can load, and that it can iterate over this picture > in a relatively inexpensive manner. This is probably reasonable, though in > my imagination I see that the set of available modules may change over time > within the life of a single VM process. > Should a module be able to iterate modules it does not depend on (or not in the transitive closure of its dependencies) and could not load? Or is "framework" or bootstrap code logically a module which depends on all other modules in the application? I know that's a very abstract sounding question, but it's cleaner to have a world where it's modules all the way down, than a special "framework code over here" case. So if an "application" is effectively a module that depends, directly or indirectly, on everything else that makes up the application, the module abstraction is far less leaky. > Iterating classes is a stranger expectation though, because while it is a > reasonable inclination to want to try this, it is possibly actively harmful > to start loading prevriously-unloaded classes. I think part of the goal is to make exploits like [1] impossible (using a MethodHandle to call into JMX to get it to load an untrusted byte array as bytecode) by making classes in modules you can't "see" via the closure of your dependencies *really* not visible from call sites that can't see them. That particular exploit was within the JDK, but similar things could easily be done using modules not part of the JDK if it's possible to get things to call each other that shouldn't. It seems desirable for a module system to provide a better firewall than that. For the "iterate all classes" thing mentioned earlier, that is typically used as a nasty hack to accomplish service discovery that can be done with metadata, and doesn't seem a use case worth encouraging. Certain things like profilers and agents might need that, however. -Tim [1] http://timboudreau.com/blog/The_Java_Security_Exploit_in_(Mostly)_Plain_English/read