From mark.reinhold at oracle.com Tue Sep 8 20:01:39 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 08 Sep 2015 13:01:39 -0700 Subject: Module-system requirements, final version Message-ID: <20150908130139.524213@eggemoggin.niobe.net> After I committed the final change to the requirements document back in April, based on Tim Ellison's last comments, I forgot to publish the result. My apologies. It's now available, here: http://openjdk.java.net/projects/jigsaw/spec/reqs/2015-04-01 - Mark From mark.reinhold at oracle.com Wed Sep 9 16:51:20 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 09 Sep 2015 09:51:20 -0700 Subject: Initial module-system design, API, and prototype Message-ID: <20150909095120.687311@eggemoggin.niobe.net> As the promised starting points for this JSR I've just posted an initial set of materials for your review and comment: - "The State of the Module System" [1], an informal overview of the proposed design. - An initial sketch of changes to the Java Language Specification and the Java Virtual Machine Specification [2]. - A preliminary draft API specification [3], along with diffs against a recent JDK 9 build [4]. - For the very brave, an initial prototype in both source [5] and binary forms [6]. (All of these links, and more, are also available on our web page [7].) The initial design covers a lot of ground, yet it is far from complete. There are many open issues, some of which are noted in these materials, and it does not yet satisfy all of our requirements [8]. To get the discussion started I suggest we focus first on the high-level aspects of the design covered in "The State of the Module System" [1]. Thank you for your patience, and I look forward to your feedback! - Mark [1] http://openjdk.java.net/projects/jigsaw/spec/sotms/ [2] http://cr.openjdk.java.net/~mr/jigsaw/spec/lang-vm.html [3] http://cr.openjdk.java.net/~mr/jigsaw/spec/api/ [4] http://cr.openjdk.java.net/~mr/jigsaw/spec/api-diff/ [5] http://hg.openjdk.java.net/jigsaw/jake [6] http://openjdk.java.net/projects/jigsaw/ea [7] http://openjdk.java.net/projects/jigsaw/spec/ [8] http://openjdk.java.net/projects/jigsaw/spec/reqs/ From david.lloyd at redhat.com Wed Sep 9 20:48:12 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 09 Sep 2015 15:48:12 -0500 Subject: Initial module-system design, API, and prototype In-Reply-To: <20150909095120.687311@eggemoggin.niobe.net> References: <20150909095120.687311@eggemoggin.niobe.net> Message-ID: <55F09B0C.7050903@redhat.com> On 09/09/2015 11:51 AM, mark.reinhold at oracle.com wrote: > To get the discussion started I suggest we focus first on the high-level > aspects of the design covered in "The State of the Module System" [1]. > > [1] http://openjdk.java.net/projects/jigsaw/spec/sotms/ 1) The first thing that jumps out at me (again) is making module declarations be a binary construct produced by the actual programming language. It strikes me as arbitrary as well as redundant (given there's a programmatic API to build descriptors, they could simply be parsed from text nearly as well, and a lot more nicely from the user's perspective). As pointed out in the document, it is already expected that additional tooling would synthesize this file at build time, further decreasing the value of such an idea. The module class file just seems pointless to me. 2) The integration of the descriptor into user module artifacts is also concerning for a variety of additional reasons: * Dependency information is non-absolute, and in fact depends very much on the environment and on conditions which can and do change over time. * Any mistake in the description potentially renders the artifact useless, since changing the artifact contents would invalidate any digital signature or hash. * Even versioning is really the conflux of the original source code, the compilation environment, and the module distribution environment; for reference, note how OS distributions nearly always ship customized versions of all nontrivial content. These factors, I think, would unfortunately preclude public code repositories which distribute modules in this format. Such repositories would have to instead ship a bootstrap module which could in turn establish module dependencies in a different way, to ensure that the description information can be handled separately from the code artifacts themselves, fracturing the nascent ecosystem before it can even establish itself. It's not clear how such an ecosystem is expected to function otherwise. 3) Readability and other terminology "Readability" is the term that we were calling "visibility" up until this point, correct? And the term "implied readability" is what I would have erstwhile called a transitive dependency. Also, "exports" in this document appear to apply only to package names and not module dependency names. 4) Layers I am concerned that layers do not go far enough to ensure that different module graphs can interoperate. There is seemingly no provision to establish dependency relationships between modules in different layers, because layers are apparently strictly hierarchical. This would make it much more difficult to (for example) allow different containers to interoperate within an app server (OSGi and Java EE, for example). Today we use this kind of relatability to allow Java EE modules to depend on modules constructed out of the legacy extensions system as well as filesystem-backed JARs which are modularized independently of our Java EE container, allowing them to be reused from other containers as well. -- - DML From david.lloyd at redhat.com Thu Sep 10 11:49:03 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 10 Sep 2015 06:49:03 -0500 Subject: Alternative module system design proposal Message-ID: <55F16E2F.3060908@redhat.com> This email describes an alternative module system design proposal (I'll call it the AMSD for brevity) and serves as a partial rebuttal to, and possible alternative for, the state of the module system doc that Mark just posted (I'll call this the MR approach for simplicity). Included is a link to an example partial implementation (this implementation is not intended (technically or legally) to be a contribution at this time, but merely an example). Principles ---------- The key principle of the alternative design is the reduction of abstraction. While MR's proposal (broadly speaking) includes adding modules as a new layer (with corresponding reflective changes), changes to the language specification, binary descriptors, and a strong hierarchy-based orientation, the AMSD equates modules to class loaders in a stronger way. No new layers of abstraction are introduced (though class loaders are slightly enhanced), the language specification is not changed, and descriptor format remains open. Modules and their configurations (I call them "module loaders") can exist in arbitrary graph relationships and are not bound to a strict hierarchy. A module in the AMSD is a specialization of a class loader, exploiting the high degree of commonality between modules (as specified) and class loaders as they exist today. A 'module loader' fulfills the specification role of "configuration". Behaviorally, it is much like a class loader, but for modules and not classes. Module loaders can be aggregated. This is of specific importance for bootstrapping; when starting the JVM, only the 'java.base' module would initially be available. A second module loader is instantiated to provide access to the remainder of the JDK; to start the user application and load its modules, a third module loader is used. This layering makes bootstrap run smoothly, and also encapsulates the platform from the application's modules. See [1] for a visual example. No additional concepts are introduced into the JVM (beyond what is already in jdk9/jdk9 at this time, specifically JRT support), though some concepts are enhanced in the JDK. The implementation would be almost 100% pure Java with no (or only very minimal) JVM changes necessary to support the run time, with the exception of tooling and the java command itself, which would be enhanced to specify modules as previously discussed. Compatibility ------------- While MR's design hinges upon exactly preserving the existing class loader structure, the AMSD seeks only to emulate the certain behaviors that were given to justify this preservation. The primary compatibility factor was described to be the assumption by certain internal classes that the class loader of certain other internal classes would be 'null'. There exists a provision within the bootstrap module system code to allow internal classes to advertise a class loader of 'null', even if that class was actually loaded by a module. A secondary getModuleClassLoader() method is specified to obtain the actual module that defines the given class. This flavor of design has been shown to be 100% compatible with Java EE (though not all *implementations* of Java EE are necessarily 100% compatible with this design, without replacing their class loader infrastructure, nor would they be compatible with the MR design) as well as with OSGi. More existing behaviors are preserved by this approach, including being able to read .class files as resources (I find it telling that the very first public post about testing the MR implementation already tripped over this). Implementation -------------- A partial example implementation exists at [2] (list of individual commits at [3]). It is (obviously) not complete, but I believe that it does contain all of the groundwork necessary to implement all of the requirements given in the specification. Some requirements are not yet met as of this date. I have a detailed outline of this information that I can share later if there is interest in this implementation alternative. Part of this implementation entails adding a formal Resource concept, which ties an arbitrary resource to its initiating class loader. This is essential to providing a service loading implementation with adequate encapsulation rules. The crux of the success of this implementation revolves around modifying the bootstrap class loading process to only load classes from the java.base section of the JRT. This is not yet done. The linkage algorithm is based heavily on that of JBoss Modules. Each module class loader contains an index of all packages visible to it; a module's class loader can, in constant time, determine whether a class or resource is loadable from that class loader, and if so, where that class or resource comes from. There is no prescribed descriptor format. The module definition API is presently based on a builder pattern; any descriptor methodology can feed into it directly. Supporting textual descriptors is a primary motivator for this; no such format is yet established in the AMSD example code. JBoss Modules used XML (for reasons which are probably not applicable to the JDK); it may be better, from a usability perspective, to use a specific textual language that can be parsed directly. There is no bootstrap support at present. This precludes loading or running modules without external support code; it also precludes things like agents, and the ability to register other early services like loggers and security managers from modules. Objective --------- The goal of this implementation is to spur discussion on certain topics: * Descriptor encoding and module format * Resource handling * Security * Implementation complexity * Forwards compatibility * Programmatic API capabilities [1] https://www.lucidchart.com/publicSegments/view/e5fbad2c-6c3f-402d-93f8-5afad4b90132/image.png [2] https://github.com/dmlloyd/openjdk-modules [3] https://github.com/dmlloyd/openjdk-modules/commits/modules?author=dmlloyd -- - DML From david.lloyd at redhat.com Fri Sep 11 11:07:27 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Fri, 11 Sep 2015 06:07:27 -0500 Subject: Initial module-system design, API, and prototype In-Reply-To: <20150909095120.687311@eggemoggin.niobe.net> References: <20150909095120.687311@eggemoggin.niobe.net> Message-ID: <55F2B5EF.8090005@redhat.com> On 09/09/2015 11:51 AM, mark.reinhold at oracle.com wrote: > As the promised starting points for this JSR I've just posted an initial > set of materials for your review and comment: > > - "The State of the Module System" [1], an informal overview of > the proposed design. > > To get the discussion started I suggest we focus first on the high-level > aspects of the design covered in "The State of the Module System" [1]. It looks like there is already a lot of trouble with the new accessibility rules (or as I'm beginning to think of them, "when is public not really public?") in the Jigsaw prototype. Overall the compatibility story is looking not very compelling at the moment; I wonder if some of these things should be revisited. -- - DML From forax at univ-mlv.fr Fri Sep 11 17:15:35 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Fri, 11 Sep 2015 19:15:35 +0200 (CEST) Subject: AccessibleObject.setAccessible() backward compatibility Message-ID: <1088108454.2110963.1441991735882.JavaMail.zimbra@u-pem.fr> There are two changes to setAccessible that make migration to the module universe hard. The first one is that setAccessible() is now declared final which is a binary incompatible change so it breaks Guava two projects of mine. The second change is that now setAccessible() is now CallerSensitive and throw an exception if the AccessibleObject is not visible from the caller class. While this change may improve the security, it's a backward incompatible change is not strictly required to support modules it's more an enhancement of the current security model and i don't think it's a good idea to mix it with the introduction of the module support. More philosophically, every libraries that propose an abstraction that hide underlying dirts provides an escape hatch, the reflection API is one of such hatch of Java the language allowing to bypass the typechecker and the security sandbox. Trying to close the hatch will just make people to open holes in the nearby wall with hacks that are less secure and that may even compromise the integrity of the plateform. cheers, R?mi From forax at univ-mlv.fr Fri Sep 11 17:31:19 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Fri, 11 Sep 2015 19:31:19 +0200 (CEST) Subject: AccessibleObject.setAccessible() backward compatibility In-Reply-To: <1088108454.2110963.1441991735882.JavaMail.zimbra@u-pem.fr> References: <1088108454.2110963.1441991735882.JavaMail.zimbra@u-pem.fr> Message-ID: <377081125.2114077.1441992679080.JavaMail.zimbra@u-pem.fr> Thinking a little more about the check in setAccessible, i think it is useless because it can be easily bypassed. setAccessible() is implemented by changing a field override, so to avoid the security check, one can change the field directly. private static final Field OVERRIDE; static { try { OVERRIDE = AccessibleObject.class.getDeclaredField("override"); } catch (NoSuchFieldException | SecurityException e) { throw new AssertionError(e); } OVERRIDE.setAccessible(true); // works because AccessibleObject is in java.base } private static void setAccessible(AccessibleObject object) throws AssertionError { try { OVERRIDE.setBoolean(object, true); } catch (IllegalAccessException e) { throw new AssertionError(e); } } I really like the fact that you can use setAccessible() to bypass the check of setAccessible() R?mi ----- Mail original ----- > De: "Remi Forax" > ?: jpms-spec-experts at openjdk.java.net > Envoy?: Vendredi 11 Septembre 2015 19:15:35 > Objet: AccessibleObject.setAccessible() backward compatibility > > There are two changes to setAccessible that make migration to the module > universe hard. > > The first one is that setAccessible() is now declared final which is a binary > incompatible change so it breaks Guava two projects of mine. > > The second change is that now setAccessible() is now CallerSensitive and > throw an exception if the AccessibleObject is not visible from the caller > class. > While this change may improve the security, it's a backward incompatible > change is not strictly required to support modules it's more an enhancement > of the current security model and i don't think it's a good idea to mix it > with the introduction of the module support. More philosophically, every > libraries that propose an abstraction that hide underlying dirts provides an > escape hatch, the reflection API is one of such hatch of Java the language > allowing to bypass the typechecker and the security sandbox. Trying to close > the hatch will just make people to open holes in the nearby wall with hacks > that are less secure and that may even compromise the integrity of the > plateform. > > cheers, > R?mi > From david.lloyd at redhat.com Fri Sep 11 17:44:30 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Fri, 11 Sep 2015 12:44:30 -0500 Subject: AccessibleObject.setAccessible() backward compatibility In-Reply-To: <1088108454.2110963.1441991735882.JavaMail.zimbra@u-pem.fr> References: <1088108454.2110963.1441991735882.JavaMail.zimbra@u-pem.fr> Message-ID: <55F312FE.2040902@redhat.com> On 09/11/2015 12:15 PM, Remi Forax wrote: > There are two changes to setAccessible that make migration to the module universe hard. > > The first one is that setAccessible() is now declared final which is a binary incompatible change so it breaks Guava two projects of mine. > > The second change is that now setAccessible() is now CallerSensitive and throw an exception if the AccessibleObject is not visible from the caller class. > While this change may improve the security, it's a backward incompatible change is not strictly required to support modules it's more an enhancement of the current security model and i don't think it's a good idea to mix it with the introduction of the module support. More philosophically, every libraries that propose an abstraction that hide underlying dirts provides an escape hatch, the reflection API is one of such hatch of Java the language allowing to bypass the typechecker and the security sandbox. Trying to close the hatch will just make people to open holes in the nearby wall with hacks that are less secure and that may even compromise the integrity of the plateform. I'm not sure there are real-world cases where the additional accessibility rules are actually useful or improve security in any real way. Maybe we need some sample use cases that we can pick apart? -- - DML From niftiness at gmail.com Fri Sep 11 18:05:36 2015 From: niftiness at gmail.com (Tim Boudreau) Date: Fri, 11 Sep 2015 14:05:36 -0400 Subject: AccessibleObject.setAccessible() backward compatibility In-Reply-To: <55F312FE.2040902@redhat.com> References: <1088108454.2110963.1441991735882.JavaMail.zimbra@u-pem.fr> <55F312FE.2040902@redhat.com> Message-ID: If the implementation of MethodHandle uses setAccessible() (I don't know its internals), then this Java 0day would qualify: http://pastebin.com/cUG2ayjh which I picked apart here: http://timboudreau.com/blog/The_Java_Security_Exploit_in_(Mostly)_Plain_English/read -Tim On Fri, Sep 11, 2015 at 1:44 PM, David M. Lloyd wrote: > On 09/11/2015 12:15 PM, Remi Forax wrote: > >> There are two changes to setAccessible that make migration to the module >> universe hard. >> >> The first one is that setAccessible() is now declared final which is a >> binary incompatible change so it breaks Guava two projects of mine. >> >> The second change is that now setAccessible() is now CallerSensitive and >> throw an exception if the AccessibleObject is not visible from the caller >> class. >> While this change may improve the security, it's a backward incompatible >> change is not strictly required to support modules it's more an enhancement >> of the current security model and i don't think it's a good idea to mix it >> with the introduction of the module support. More philosophically, every >> libraries that propose an abstraction that hide underlying dirts provides >> an escape hatch, the reflection API is one of such hatch of Java the >> language allowing to bypass the typechecker and the security sandbox. >> Trying to close the hatch will just make people to open holes in the nearby >> wall with hacks that are less secure and that may even compromise the >> integrity of the plateform. >> > > I'm not sure there are real-world cases where the additional accessibility > rules are actually useful or improve security in any real way. Maybe we > need some sample use cases that we can pick apart? > > -- > - DML > -- http://timboudreau.com From forax at univ-mlv.fr Fri Sep 11 18:11:23 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Fri, 11 Sep 2015 20:11:23 +0200 (CEST) Subject: AccessibleObject.setAccessible() backward compatibility In-Reply-To: References: <1088108454.2110963.1441991735882.JavaMail.zimbra@u-pem.fr> <55F312FE.2040902@redhat.com> Message-ID: <763983984.2119237.1441995083631.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "Tim Boudreau" > ?: "David M. Lloyd" > Cc: jpms-spec-experts at openjdk.java.net > Envoy?: Vendredi 11 Septembre 2015 20:05:36 > Objet: Re: AccessibleObject.setAccessible() backward compatibility > > If the implementation of MethodHandle uses setAccessible() (I don't know > its internals), then this Java 0day would qualify: It does not as far as I know. It's the opposite, if you want to bypass the security sandbox with a MethodHandle, you have to use reflection + setAccessible and then use Lookup.unreflect*(). > > http://pastebin.com/cUG2ayjh > > which I picked apart here: > http://timboudreau.com/blog/The_Java_Security_Exploit_in_(Mostly)_Plain_English/read > > -Tim R?mi > > > On Fri, Sep 11, 2015 at 1:44 PM, David M. Lloyd > wrote: > > > On 09/11/2015 12:15 PM, Remi Forax wrote: > > > >> There are two changes to setAccessible that make migration to the module > >> universe hard. > >> > >> The first one is that setAccessible() is now declared final which is a > >> binary incompatible change so it breaks Guava two projects of mine. > >> > >> The second change is that now setAccessible() is now CallerSensitive and > >> throw an exception if the AccessibleObject is not visible from the caller > >> class. > >> While this change may improve the security, it's a backward incompatible > >> change is not strictly required to support modules it's more an > >> enhancement > >> of the current security model and i don't think it's a good idea to mix it > >> with the introduction of the module support. More philosophically, every > >> libraries that propose an abstraction that hide underlying dirts provides > >> an escape hatch, the reflection API is one of such hatch of Java the > >> language allowing to bypass the typechecker and the security sandbox. > >> Trying to close the hatch will just make people to open holes in the > >> nearby > >> wall with hacks that are less secure and that may even compromise the > >> integrity of the plateform. > >> > > > > I'm not sure there are real-world cases where the additional accessibility > > rules are actually useful or improve security in any real way. Maybe we > > need some sample use cases that we can pick apart? > > > > -- > > - DML > > > > > > -- > http://timboudreau.com > From niftiness at gmail.com Fri Sep 11 20:14:56 2015 From: niftiness at gmail.com (Tim Boudreau) Date: Fri, 11 Sep 2015 16:14:56 -0400 Subject: AccessibleObject.setAccessible() backward compatibility In-Reply-To: <763983984.2119237.1441995083631.JavaMail.zimbra@u-pem.fr> References: <1088108454.2110963.1441991735882.JavaMail.zimbra@u-pem.fr> <55F312FE.2040902@redhat.com> <763983984.2119237.1441995083631.JavaMail.zimbra@u-pem.fr> Message-ID: > > >> If the implementation of MethodHandle uses setAccessible() (I don't know > >> its internals), then this Java 0day would qualify: > > > > It does not as far as I know. > > It's the opposite, if you want to bypass the security sandbox with a > MethodHandle, > > you have to use reflection + setAccessible and then use > Lookup.unreflect*(). > Point taken. Regardless, if one of the problems we want to solve here is security related, then having a security sandbox you really can't bypass, even reflectively, is not a bad idea at all. -Tim From niftiness at gmail.com Fri Sep 11 20:16:08 2015 From: niftiness at gmail.com (Tim Boudreau) Date: Fri, 11 Sep 2015 16:16:08 -0400 Subject: Resignation Message-ID: Hi, folks, It is necessary for me to resign my membership in this expert group, since I recently became an Oracle employee, and that would otherwise mean Oracle was over-represented here. It has been a pleasure to contribute to this project, and it appears to be in good hands and moving in good directions. I'll be sorry to miss the ongoing discussion, since a lot is happening right now. Good luck, and I look forward to the fruits of this effort. Best regards, Tim -- http://timboudreau.com From david.lloyd at redhat.com Fri Sep 11 20:42:56 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Fri, 11 Sep 2015 15:42:56 -0500 Subject: AccessibleObject.setAccessible() backward compatibility In-Reply-To: References: <1088108454.2110963.1441991735882.JavaMail.zimbra@u-pem.fr> <55F312FE.2040902@redhat.com> <763983984.2119237.1441995083631.JavaMail.zimbra@u-pem.fr> Message-ID: <55F33CD0.4050601@redhat.com> On 09/11/2015 03:14 PM, Tim Boudreau wrote: > >> If the implementation of MethodHandle uses setAccessible() (I don't know > >> its internals), then this Java 0day would qualify: > > > > It does not as far as I know. > > It's the opposite, if you want to bypass the security sandbox with a MethodHandle, > > you have to use reflection + setAccessible and then use Lookup.unreflect*(). > > > Point taken. > > Regardless, if one of the problems we want to solve here is security > related, then having a security sandbox you really can't bypass, even > reflectively, is not a bad idea at all. In principle, sure. But to paraphrase Schneier, adding complexity is a sucky way to add security. :-) If this is a goal though then the only logical way for it to be done that I can see is by always using AccessController for reflection permission checks even if there is no security manager. It seems out of scope of this JSR to me though... -- - DML From forax at univ-mlv.fr Sat Sep 12 08:33:23 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Sat, 12 Sep 2015 10:33:23 +0200 (CEST) Subject: Resignation In-Reply-To: References: Message-ID: <1774537865.2223765.1442046803860.JavaMail.zimbra@u-pem.fr> Too bad it's happening now ! Good luck in your new job, best, R?mi ----- Mail original ----- > De: "Tim Boudreau" > ?: jpms-spec-experts at openjdk.java.net > Envoy?: Vendredi 11 Septembre 2015 22:16:08 > Objet: Resignation > > Hi, folks, > > It is necessary for me to resign my membership in this expert group, since > I recently became an Oracle employee, and that would otherwise mean Oracle > was over-represented here. > > It has been a pleasure to contribute to this project, and it appears to be > in good hands and moving in good directions. I'll be sorry to miss the > ongoing discussion, since a lot is happening right now. > > Good luck, and I look forward to the fruits of this effort. > > Best regards, > > Tim > > -- > http://timboudreau.com > From forax at univ-mlv.fr Sat Sep 12 08:34:24 2015 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Sat, 12 Sep 2015 10:34:24 +0200 (CEST) Subject: AccessibleObject.setAccessible() backward compatibility In-Reply-To: <55F33CD0.4050601@redhat.com> References: <1088108454.2110963.1441991735882.JavaMail.zimbra@u-pem.fr> <55F312FE.2040902@redhat.com> <763983984.2119237.1441995083631.JavaMail.zimbra@u-pem.fr> <55F33CD0.4050601@redhat.com> Message-ID: <286683614.2223840.1442046864655.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "David M. Lloyd" > ?: "Tim Boudreau" , "Remi Forax" > Cc: jpms-spec-experts at openjdk.java.net > Envoy?: Vendredi 11 Septembre 2015 22:42:56 > Objet: Re: AccessibleObject.setAccessible() backward compatibility > > On 09/11/2015 03:14 PM, Tim Boudreau wrote: > > >> If the implementation of MethodHandle uses setAccessible() (I don't > > >> know > > >> its internals), then this Java 0day would qualify: > > > > > > It does not as far as I know. > > > It's the opposite, if you want to bypass the security sandbox with a > > > MethodHandle, > > > you have to use reflection + setAccessible and then use > > > Lookup.unreflect*(). > > > > > > Point taken. > > > > Regardless, if one of the problems we want to solve here is security > > related, then having a security sandbox you really can't bypass, even > > reflectively, is not a bad idea at all. > > In principle, sure. But to paraphrase Schneier, adding complexity is a > sucky way to add security. :-) > > If this is a goal though then the only logical way for it to be done > that I can see is by always using AccessController for reflection > permission checks even if there is no security manager. It seems out of > scope of this JSR to me though... i agree. > > -- > - DML > R?mi From david.lloyd at redhat.com Tue Sep 15 12:01:07 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 15 Sep 2015 07:01:07 -0500 Subject: Revisiting encapsulation requirement Message-ID: <55F80883.9010603@redhat.com> Experts, I think we should revisit the "Encapsulation" requirement with a view towards removal. The requirement is: > Encapsulation ? The access-control mechanisms of the Java language and virtual machine must prevent code from accessing classes and interfaces in packages that are not exported by their containing modules, or in packages whose containing modules are not required by the module containing the code. These mechanisms must be operative even when a security manager is not present. The implementation amounts to modifying the AccessibleObject.setAccessible() implementation for classes to perform a check of the caller's class loader before determining whether to make the object accessible. As R?mi pointed out, this check is easily bypassed by using reflection, by reflecting on AccessibleObject.class itself - since all modules require "java.base", all modules implicitly can bypass this check. But, even assuming this could somehow be patched over, I think we should consider dropping this requirement; I have two primary reasons for this (though there may be others as well). The first reason is that without a security manager (or maybe even *with* a security manager - but that's another discussion), it is inevitable that any security measure used to protect this mechanism will ultimately be bypassed, rendering its security value useless; adding more complexity to the system to do so will only increase the overall vulnerability of the platform. The second reason is that it is often useful to gain access to and inspect public classes that are not necessarily visible from your module. Frameworks do this commonly for example - framework implementations will almost never import modules that they introspect. I think in the end users will find it more convenient and intuitive (and no less secure in any real sense) if public classes remain accessible. In the absence of any dissent, I would move that we drop this requirement. -- - DML From forax at univ-mlv.fr Tue Sep 15 15:52:48 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Tue, 15 Sep 2015 17:52:48 +0200 (CEST) Subject: Revisiting encapsulation requirement In-Reply-To: <55F80883.9010603@redhat.com> References: <55F80883.9010603@redhat.com> Message-ID: <1307198778.1052434.1442332368236.JavaMail.zimbra@u-pem.fr> Hi David, I don't think we should modify the "encapsulation" requirement if the requirement only apply to regular Java code and not on reflection + setAccessible(true). Currently, when you use reflection, you have the same security checks as in plain Java, and you can bypass those security checks using setAccessible(true). I don't see why we should change that: - Class.forName() should throw a ClassNotFoundException if the module of the class is not declared as dependency. - invoke() should throw a an IllegalAccess exception if the module of the class is not declared as dependency. - setAccessible() should allow to bypass the security check and works as in pre-module world. I don't see the point to either add a new security check on setAccessible or to allow reflection between modules that are not listed in the dependencies. R?mi ----- Mail original ----- > De: "David M. Lloyd" > ?: jpms-spec-experts at openjdk.java.net > Envoy?: Mardi 15 Septembre 2015 14:01:07 > Objet: Revisiting encapsulation requirement > > Experts, I think we should revisit the "Encapsulation" requirement with > a view towards removal. > > The requirement is: > > > Encapsulation ? The access-control mechanisms of the Java language and > > virtual machine must prevent code from accessing classes and interfaces in > > packages that are not exported by their containing modules, or in packages > > whose containing modules are not required by the module containing the > > code. These mechanisms must be operative even when a security manager is > > not present. > > The implementation amounts to modifying the > AccessibleObject.setAccessible() implementation for classes to perform a > check of the caller's class loader before determining whether to make > the object accessible. > > As R?mi pointed out, this check is easily bypassed by using reflection, > by reflecting on AccessibleObject.class itself - since all modules > require "java.base", all modules implicitly can bypass this check. But, > even assuming this could somehow be patched over, I think we should > consider dropping this requirement; I have two primary reasons for this > (though there may be others as well). > > The first reason is that without a security manager (or maybe even > *with* a security manager - but that's another discussion), it is > inevitable that any security measure used to protect this mechanism will > ultimately be bypassed, rendering its security value useless; adding > more complexity to the system to do so will only increase the overall > vulnerability of the platform. > > The second reason is that it is often useful to gain access to and > inspect public classes that are not necessarily visible from your > module. Frameworks do this commonly for example - framework > implementations will almost never import modules that they introspect. > > I think in the end users will find it more convenient and intuitive (and > no less secure in any real sense) if public classes remain accessible. > In the absence of any dissent, I would move that we drop this requirement. > > -- > - DML > From david.lloyd at redhat.com Tue Sep 15 16:12:56 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 15 Sep 2015 11:12:56 -0500 Subject: Revisiting encapsulation requirement In-Reply-To: <1307198778.1052434.1442332368236.JavaMail.zimbra@u-pem.fr> References: <55F80883.9010603@redhat.com> <1307198778.1052434.1442332368236.JavaMail.zimbra@u-pem.fr> Message-ID: <55F84388.7070300@redhat.com> Do you have an example of a real-world problem (security or otherwise) that is solved by adding access checks based on the caller? On 09/15/2015 10:52 AM, Remi Forax wrote: > Hi David, > I don't think we should modify the "encapsulation" requirement if the requirement > only apply to regular Java code and not on reflection + setAccessible(true). > > Currently, when you use reflection, you have the same security checks as in plain Java, > and you can bypass those security checks using setAccessible(true). > > I don't see why we should change that: > - Class.forName() should throw a ClassNotFoundException if the module of the class is not declared as dependency. > - invoke() should throw a an IllegalAccess exception if the module of the class is not declared as dependency. > - setAccessible() should allow to bypass the security check and works as in pre-module world. > > I don't see the point to either add a new security check on setAccessible or to allow reflection between modules that are not listed in the dependencies. > > R?mi > > ----- Mail original ----- >> De: "David M. Lloyd" >> ?: jpms-spec-experts at openjdk.java.net >> Envoy?: Mardi 15 Septembre 2015 14:01:07 >> Objet: Revisiting encapsulation requirement >> >> Experts, I think we should revisit the "Encapsulation" requirement with >> a view towards removal. >> >> The requirement is: >> >>> Encapsulation ? The access-control mechanisms of the Java language and >>> virtual machine must prevent code from accessing classes and interfaces in >>> packages that are not exported by their containing modules, or in packages >>> whose containing modules are not required by the module containing the >>> code. These mechanisms must be operative even when a security manager is >>> not present. >> >> The implementation amounts to modifying the >> AccessibleObject.setAccessible() implementation for classes to perform a >> check of the caller's class loader before determining whether to make >> the object accessible. >> >> As R?mi pointed out, this check is easily bypassed by using reflection, >> by reflecting on AccessibleObject.class itself - since all modules >> require "java.base", all modules implicitly can bypass this check. But, >> even assuming this could somehow be patched over, I think we should >> consider dropping this requirement; I have two primary reasons for this >> (though there may be others as well). >> >> The first reason is that without a security manager (or maybe even >> *with* a security manager - but that's another discussion), it is >> inevitable that any security measure used to protect this mechanism will >> ultimately be bypassed, rendering its security value useless; adding >> more complexity to the system to do so will only increase the overall >> vulnerability of the platform. >> >> The second reason is that it is often useful to gain access to and >> inspect public classes that are not necessarily visible from your >> module. Frameworks do this commonly for example - framework >> implementations will almost never import modules that they introspect. >> >> I think in the end users will find it more convenient and intuitive (and >> no less secure in any real sense) if public classes remain accessible. >> In the absence of any dissent, I would move that we drop this requirement. >> >> -- >> - DML >> -- - DML From forax at univ-mlv.fr Tue Sep 15 19:47:22 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Tue, 15 Sep 2015 21:47:22 +0200 (CEST) Subject: Revisiting encapsulation requirement In-Reply-To: <55F84388.7070300@redhat.com> References: <55F80883.9010603@redhat.com> <1307198778.1052434.1442332368236.JavaMail.zimbra@u-pem.fr> <55F84388.7070300@redhat.com> Message-ID: <864168724.1106362.1442346442276.JavaMail.zimbra@u-pem.fr> Hi David, ----- Mail original ----- > De: "David M. Lloyd" > ?: jpms-spec-experts at openjdk.java.net > Envoy?: Mardi 15 Septembre 2015 18:12:56 > Objet: Re: Revisiting encapsulation requirement > > Do you have an example of a real-world problem (security or otherwise) > that is solved by adding access checks based on the caller ? You take the problem by the wrong side :) Let's suppose you agree that both the notion of visibility (private, package private, etc) and the reflection API are something useful. A simple model is to make the Java visibility rules and the reflection API behavior aligned. This implies that some reflexive calls have to be caller sensitive. Then as i said earlier, you need a way to escape the sandbox, a escape hatch, to program things like meta-protocols as most of the Java EE specs are, hence, the addition of setAccessible(). You may think that the reflection is a bad API, and that something like invokedynamic + the method handle Lookup API is better API, i agree but currently it's not a replacement because there is no way to emit invokedynamic in Java. Yes, this decision is stupid, but this JSR is not about improving the way the introspection/reflection works in Java but how to introduce a simple module API (with the compiler and runtime behavior aligned) in the existing JDK world with the minimal breaking changes. regards, R?mi > > On 09/15/2015 10:52 AM, Remi Forax wrote: > > Hi David, > > I don't think we should modify the "encapsulation" requirement if the > > requirement > > only apply to regular Java code and not on reflection + > > setAccessible(true). > > > > Currently, when you use reflection, you have the same security checks as in > > plain Java, > > and you can bypass those security checks using setAccessible(true). > > > > I don't see why we should change that: > > - Class.forName() should throw a ClassNotFoundException if the module of > > the class is not declared as dependency. > > - invoke() should throw a an IllegalAccess exception if the module of > > the class is not declared as dependency. > > - setAccessible() should allow to bypass the security check and works as > > in pre-module world. > > > > I don't see the point to either add a new security check on setAccessible > > or to allow reflection between modules that are not listed in the > > dependencies. > > > > R?mi > > > > ----- Mail original ----- > >> De: "David M. Lloyd" > >> ?: jpms-spec-experts at openjdk.java.net > >> Envoy?: Mardi 15 Septembre 2015 14:01:07 > >> Objet: Revisiting encapsulation requirement > >> > >> Experts, I think we should revisit the "Encapsulation" requirement with > >> a view towards removal. > >> > >> The requirement is: > >> > >>> Encapsulation ? The access-control mechanisms of the Java language and > >>> virtual machine must prevent code from accessing classes and interfaces > >>> in > >>> packages that are not exported by their containing modules, or in > >>> packages > >>> whose containing modules are not required by the module containing the > >>> code. These mechanisms must be operative even when a security manager is > >>> not present. > >> > >> The implementation amounts to modifying the > >> AccessibleObject.setAccessible() implementation for classes to perform a > >> check of the caller's class loader before determining whether to make > >> the object accessible. > >> > >> As R?mi pointed out, this check is easily bypassed by using reflection, > >> by reflecting on AccessibleObject.class itself - since all modules > >> require "java.base", all modules implicitly can bypass this check. But, > >> even assuming this could somehow be patched over, I think we should > >> consider dropping this requirement; I have two primary reasons for this > >> (though there may be others as well). > >> > >> The first reason is that without a security manager (or maybe even > >> *with* a security manager - but that's another discussion), it is > >> inevitable that any security measure used to protect this mechanism will > >> ultimately be bypassed, rendering its security value useless; adding > >> more complexity to the system to do so will only increase the overall > >> vulnerability of the platform. > >> > >> The second reason is that it is often useful to gain access to and > >> inspect public classes that are not necessarily visible from your > >> module. Frameworks do this commonly for example - framework > >> implementations will almost never import modules that they introspect. > >> > >> I think in the end users will find it more convenient and intuitive (and > >> no less secure in any real sense) if public classes remain accessible. > >> In the absence of any dissent, I would move that we drop this requirement. > >> > >> -- > >> - DML > >> > > -- > - DML > From david.lloyd at redhat.com Tue Sep 15 21:06:35 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 15 Sep 2015 16:06:35 -0500 Subject: Revisiting encapsulation requirement In-Reply-To: <864168724.1106362.1442346442276.JavaMail.zimbra@u-pem.fr> References: <55F80883.9010603@redhat.com> <1307198778.1052434.1442332368236.JavaMail.zimbra@u-pem.fr> <55F84388.7070300@redhat.com> <864168724.1106362.1442346442276.JavaMail.zimbra@u-pem.fr> Message-ID: <55F8885B.3030903@redhat.com> On 09/15/2015 02:47 PM, Remi Forax wrote: > Hi David, > > ----- Mail original ----- >> De: "David M. Lloyd" >> ?: jpms-spec-experts at openjdk.java.net >> Envoy?: Mardi 15 Septembre 2015 18:12:56 >> Objet: Re: Revisiting encapsulation requirement >> >> Do you have an example of a real-world problem (security or otherwise) >> that is solved by adding access checks based on the caller ? > > You take the problem by the wrong side :) Maybe. > Let's suppose you agree that both the notion of visibility (private, package private, etc) and the reflection API are something useful. Sure, we've been calling that "accessibility" up until now. > A simple model is to make the Java visibility rules and the reflection API behavior aligned. > This implies that some reflexive calls have to be caller sensitive. > Then as i said earlier, you need a way to escape the sandbox, a escape hatch, to program things like meta-protocols as most of the Java EE specs are, hence, the addition of setAccessible(). So if I understand correctly, you propose that, in addition to verifying that the target type is accessible to the caller, we should also verify that it is visible when we do the actual access check (i.e. "invoke() should throw...if...the class is not declared as a dependency") and forget doing a special check when we call setAccessible(). If I'm misunderstanding you, stop me here! But assuming all that is true, my concern is that today, I can access any public member of any public class in the system that I can get a Class object for, even if it comes from a different, isolated class loader. Under the new rules and even with your suggestion, a class has to be public *and* visible to me, else setAccessible() is the only choice - even if it's public, and was handed to me! To me this makes no sense and does not add any security- or encapsulation-related benefits that I can come up with - quite the opposite actually, since now you need to be granted sweeping permissions to access it. This is why I'm seeking a counter-example which supports this idea. > You may think that the reflection is a bad API, and that something like invokedynamic + the method handle Lookup API is better API, i agree but currently it's not a replacement because there is no way to emit invokedynamic in Java. > Yes, this decision is stupid, but this JSR is not about improving the way the introspection/reflection works in Java but how to introduce a simple module API (with the compiler and runtime behavior aligned) in the existing JDK world with the minimal breaking changes. > > regards, > R?mi > >> >> On 09/15/2015 10:52 AM, Remi Forax wrote: >>> Hi David, >>> I don't think we should modify the "encapsulation" requirement if the >>> requirement >>> only apply to regular Java code and not on reflection + >>> setAccessible(true). >>> >>> Currently, when you use reflection, you have the same security checks as in >>> plain Java, >>> and you can bypass those security checks using setAccessible(true). >>> >>> I don't see why we should change that: >>> - Class.forName() should throw a ClassNotFoundException if the module of >>> the class is not declared as dependency. >>> - invoke() should throw a an IllegalAccess exception if the module of >>> the class is not declared as dependency. >>> - setAccessible() should allow to bypass the security check and works as >>> in pre-module world. >>> >>> I don't see the point to either add a new security check on setAccessible >>> or to allow reflection between modules that are not listed in the >>> dependencies. >>> >>> R?mi >>> >>> ----- Mail original ----- >>>> De: "David M. Lloyd" >>>> ?: jpms-spec-experts at openjdk.java.net >>>> Envoy?: Mardi 15 Septembre 2015 14:01:07 >>>> Objet: Revisiting encapsulation requirement >>>> >>>> Experts, I think we should revisit the "Encapsulation" requirement with >>>> a view towards removal. >>>> >>>> The requirement is: >>>> >>>>> Encapsulation ? The access-control mechanisms of the Java language and >>>>> virtual machine must prevent code from accessing classes and interfaces >>>>> in >>>>> packages that are not exported by their containing modules, or in >>>>> packages >>>>> whose containing modules are not required by the module containing the >>>>> code. These mechanisms must be operative even when a security manager is >>>>> not present. >>>> >>>> The implementation amounts to modifying the >>>> AccessibleObject.setAccessible() implementation for classes to perform a >>>> check of the caller's class loader before determining whether to make >>>> the object accessible. >>>> >>>> As R?mi pointed out, this check is easily bypassed by using reflection, >>>> by reflecting on AccessibleObject.class itself - since all modules >>>> require "java.base", all modules implicitly can bypass this check. But, >>>> even assuming this could somehow be patched over, I think we should >>>> consider dropping this requirement; I have two primary reasons for this >>>> (though there may be others as well). >>>> >>>> The first reason is that without a security manager (or maybe even >>>> *with* a security manager - but that's another discussion), it is >>>> inevitable that any security measure used to protect this mechanism will >>>> ultimately be bypassed, rendering its security value useless; adding >>>> more complexity to the system to do so will only increase the overall >>>> vulnerability of the platform. >>>> >>>> The second reason is that it is often useful to gain access to and >>>> inspect public classes that are not necessarily visible from your >>>> module. Frameworks do this commonly for example - framework >>>> implementations will almost never import modules that they introspect. >>>> >>>> I think in the end users will find it more convenient and intuitive (and >>>> no less secure in any real sense) if public classes remain accessible. >>>> In the absence of any dissent, I would move that we drop this requirement. >>>> >>>> -- >>>> - DML >>>> >> >> -- >> - DML >> -- - DML From forax at univ-mlv.fr Wed Sep 16 09:00:59 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 16 Sep 2015 11:00:59 +0200 (CEST) Subject: Revisiting encapsulation requirement In-Reply-To: <55F8885B.3030903@redhat.com> References: <55F80883.9010603@redhat.com> <1307198778.1052434.1442332368236.JavaMail.zimbra@u-pem.fr> <55F84388.7070300@redhat.com> <864168724.1106362.1442346442276.JavaMail.zimbra@u-pem.fr> <55F8885B.3030903@redhat.com> Message-ID: <1285323903.1293876.1442394059034.JavaMail.zimbra@u-pem.fr> > De: "David M. Lloyd" > ?: jpms-spec-experts at openjdk.java.net > Envoy?: Mardi 15 Septembre 2015 23:06:35 > Objet: Re: Revisiting encapsulation requirement > > On 09/15/2015 02:47 PM, Remi Forax wrote: > > Hi David, > > > > ----- Mail original ----- > >> De: "David M. Lloyd" > >> ?: jpms-spec-experts at openjdk.java.net > >> Envoy?: Mardi 15 Septembre 2015 18:12:56 > >> Objet: Re: Revisiting encapsulation requirement > >> > >> Do you have an example of a real-world problem (security or otherwise) > >> that is solved by adding access checks based on the caller ? > > > > You take the problem by the wrong side :) > > Maybe. > > > Let's suppose you agree that both the notion of visibility (private, > > package private, etc) and the reflection API are something useful. > > Sure, we've been calling that "accessibility" up until now. > > > A simple model is to make the Java visibility rules and the reflection API > > behavior aligned. > > This implies that some reflexive calls have to be caller sensitive. > > Then as i said earlier, you need a way to escape the sandbox, a escape > > hatch, to program things like meta-protocols as most of the Java EE specs > > are, hence, the addition of setAccessible(). > > So if I understand correctly, you propose that, in addition to verifying > that the target type is accessible to the caller, we should also verify > that it is visible when we do the actual access check (i.e. "invoke() > should throw...if...the class is not declared as a dependency") and > forget doing a special check when we call setAccessible(). If I'm > misunderstanding you, stop me here! yes, > > But assuming all that is true, my concern is that today, I can access > any public member of any public class in the system that I can get a > Class object for, even if it comes from a different, isolated class > loader. Under the new rules and even with your suggestion, a class has > to be public *and* visible to me, else setAccessible() is the only > choice - even if it's public, and was handed to me! To me this makes no > sense and does not add any security- or encapsulation-related benefits > that I can come up with - quite the opposite actually, since now you > need to be granted sweeping permissions to access it. This is why I'm > seeking a counter-example which supports this idea. Let say you have two modules, one that contains an interface and another that contains a class that implements that interface in another package. currently if you write a code like this: o.getClass().getMethods("foo").invoke(o) by accessing to a public method of the implementation class that may be not declared in the interface, you're breaking encapsulation. if you write: Interface o = ... Interface.class.getMethods("foo").invoke(o) there is no problem. but maybe it's not a problem that worth to be fixed ? R?mi > > > You may think that the reflection is a bad API, and that something like > > invokedynamic + the method handle Lookup API is better API, i agree but > > currently it's not a replacement because there is no way to emit > > invokedynamic in Java. > > Yes, this decision is stupid, but this JSR is not about improving the way > > the introspection/reflection works in Java but how to introduce a simple > > module API (with the compiler and runtime behavior aligned) in the > > existing JDK world with the minimal breaking changes. > > > > regards, > > R?mi > > > >> > >> On 09/15/2015 10:52 AM, Remi Forax wrote: > >>> Hi David, > >>> I don't think we should modify the "encapsulation" requirement if the > >>> requirement > >>> only apply to regular Java code and not on reflection + > >>> setAccessible(true). > >>> > >>> Currently, when you use reflection, you have the same security checks as > >>> in > >>> plain Java, > >>> and you can bypass those security checks using setAccessible(true). > >>> > >>> I don't see why we should change that: > >>> - Class.forName() should throw a ClassNotFoundException if the module > >>> of > >>> the class is not declared as dependency. > >>> - invoke() should throw a an IllegalAccess exception if the module of > >>> the class is not declared as dependency. > >>> - setAccessible() should allow to bypass the security check and works > >>> as > >>> in pre-module world. > >>> > >>> I don't see the point to either add a new security check on setAccessible > >>> or to allow reflection between modules that are not listed in the > >>> dependencies. > >>> > >>> R?mi > >>> > >>> ----- Mail original ----- > >>>> De: "David M. Lloyd" > >>>> ?: jpms-spec-experts at openjdk.java.net > >>>> Envoy?: Mardi 15 Septembre 2015 14:01:07 > >>>> Objet: Revisiting encapsulation requirement > >>>> > >>>> Experts, I think we should revisit the "Encapsulation" requirement with > >>>> a view towards removal. > >>>> > >>>> The requirement is: > >>>> > >>>>> Encapsulation ? The access-control mechanisms of the Java language and > >>>>> virtual machine must prevent code from accessing classes and interfaces > >>>>> in > >>>>> packages that are not exported by their containing modules, or in > >>>>> packages > >>>>> whose containing modules are not required by the module containing the > >>>>> code. These mechanisms must be operative even when a security manager > >>>>> is > >>>>> not present. > >>>> > >>>> The implementation amounts to modifying the > >>>> AccessibleObject.setAccessible() implementation for classes to perform a > >>>> check of the caller's class loader before determining whether to make > >>>> the object accessible. > >>>> > >>>> As R?mi pointed out, this check is easily bypassed by using reflection, > >>>> by reflecting on AccessibleObject.class itself - since all modules > >>>> require "java.base", all modules implicitly can bypass this check. But, > >>>> even assuming this could somehow be patched over, I think we should > >>>> consider dropping this requirement; I have two primary reasons for this > >>>> (though there may be others as well). > >>>> > >>>> The first reason is that without a security manager (or maybe even > >>>> *with* a security manager - but that's another discussion), it is > >>>> inevitable that any security measure used to protect this mechanism will > >>>> ultimately be bypassed, rendering its security value useless; adding > >>>> more complexity to the system to do so will only increase the overall > >>>> vulnerability of the platform. > >>>> > >>>> The second reason is that it is often useful to gain access to and > >>>> inspect public classes that are not necessarily visible from your > >>>> module. Frameworks do this commonly for example - framework > >>>> implementations will almost never import modules that they introspect. > >>>> > >>>> I think in the end users will find it more convenient and intuitive (and > >>>> no less secure in any real sense) if public classes remain accessible. > >>>> In the absence of any dissent, I would move that we drop this > >>>> requirement. > >>>> > >>>> -- > >>>> - DML > >>>> > >> > >> -- > >> - DML > >> > > -- > - DML > From mark.reinhold at oracle.com Wed Sep 16 17:00:07 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 16 Sep 2015 10:00:07 -0700 Subject: Resignation In-Reply-To: References: Message-ID: <20150916100007.216437@eggemoggin.niobe.net> Thanks, Tim -- I'm sorry to see you go, but I hope you'll find time to keep up with this work and contribute from outside the EG. - Mark From mark.reinhold at oracle.com Thu Sep 17 14:51:05 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 17 Sep 2015 07:51:05 -0700 Subject: Please welcome Jason van Zyl to JSR 376 Message-ID: <20150917075105.496107@eggemoggin.niobe.net> Jason van Zyl, well-known for his work on Maven (among other things), has asked to join this EG. Ordinarily I'm reluctant to accept new EG members this late in the game, after we've settled the requirements, but Jason has agreed to accept the requirements as-is and understands that the bar for revising them is relatively high. I'm confident that he will bring valuable perspectives to our discussions and help to ensure the overall success of the JSR. (Jason -- if there's anything further you'd like to say by way of introduction, please do so!) - Mark From mark.reinhold at oracle.com Thu Sep 17 14:52:11 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 17 Sep 2015 07:52:11 -0700 (PDT) Subject: Module declarations Message-ID: <20150917145211.06B7371D05@eggemoggin.niobe.net> (splitting your initial reply into separate threads) 2015/9/9 1:48 -0700, david.lloyd at redhat.com: > On 09/09/2015 11:51 AM, mark.reinhold at oracle.com wrote: >> To get the discussion started I suggest we focus first on the high-level >> aspects of the design covered in "The State of the Module System" [1]. >> >> [1] http://openjdk.java.net/projects/jigsaw/spec/sotms/ > > 1) The first thing that jumps out at me (again) is making module > declarations be a binary construct produced by the actual programming > language. It strikes me as arbitrary as well as redundant (given > there's a programmatic API to build descriptors, they could simply be > parsed from text nearly as well, and a lot more nicely from the user's > perspective). As pointed out in the document, it is already expected > that additional tooling would synthesize this file at build time, > further decreasing the value of such an idea. The module class file > just seems pointless to me. One of our requirements is fidelity across all phases. To achieve this, javac (or whatever Java compiler you use) must be able to locate and interpret descriptions of modules. If modules are not described by language constructs that are compiled into class files, then how and where would you suggest that a compiler find such information? (The statement that "build tools, in particular, can synthesize module declarations from information already available in project descriptions" in the SotMS document is, I see, overly terse. A tool could suggest an initial module declaration with `requires` clauses derived from a pom.xml file, or equivalent, but it'd be up to the developer to specify the rest. I'll fix this in the next version.) - Mark From mark.reinhold at oracle.com Thu Sep 17 14:53:11 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 17 Sep 2015 07:53:11 -0700 (PDT) Subject: Module descriptors in artifacts Message-ID: <20150917145311.0A77571D0B@eggemoggin.niobe.net> 2015/9/9 1:48 -0700, david.lloyd at redhat.com: > ... > > 2) The integration of the descriptor into user module artifacts is also > concerning for a variety of additional reasons: > > ... > > These factors, I think, would unfortunately preclude public code > repositories which distribute modules in this format. Such repositories > would have to instead ship a bootstrap module which could in turn > establish module dependencies in a different way, to ensure that the > description information can be handled separately from the code > artifacts themselves, fracturing the nascent ecosystem before it can > even establish itself. It's not clear how such an ecosystem is expected > to function otherwise. We're not trying to establish a new ecosystem of component distribution; we are, rather, trying to fit into existing ones, and in particular the existing Maven-based ecosystem. Success in that space is, I suggest, being able to publish modular JARs (and perhaps even JMODs) to Maven Central, along with their POMs, so that they can be retrieved in the usual ways but placed on the module path rather than the class path. It's up to Maven/Gradle/Ivy/etc., rather than the module system, to deal with version conflicts, dependency management, hashes, signatures, and so forth. Placing module descriptors into module artifacts preserves the simple and familiar "just put your JAR files in the right directory" approach to setting up a build- or run-time environment. To put them somewhere else would add unnecessary complexity. - Mark From mark.reinhold at oracle.com Thu Sep 17 14:54:11 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 17 Sep 2015 07:54:11 -0700 (PDT) Subject: Readability and other terminology Message-ID: <20150917145411.0E1F871D10@eggemoggin.niobe.net> 2015/9/9 1:48 -0700, david.lloyd at redhat.com: > ... > > 3) Readability and other terminology > > "Readability" is the term that we were calling "visibility" up until > this point, correct? No, readability is a new concept, distinct from visibility. Readability is a relation between modules, computed during resolution by interpreting the `requires` clauses in module declarations. Visibility, by contrast, is a relation between types. > And the term "implied readability" is what I would > have erstwhile called a transitive dependency. No, readability and dependence are related but distinct concepts. If module A depends upon module B (`requires B`), and B depends upon C, then A depends transitively upon C. The resolver will construct a module graph in which A reads B, and B reads C. There will be a path in the graph from A to C, but that doesn't mean that A reads C. For that to happen, B must `requires public` C, which during resolution will cause an implied read edge to be added from A directly to C. > Also, "exports" in this document appear to apply only to package names > and not module dependency names. Correct. - Mark From mark.reinhold at oracle.com Thu Sep 17 14:55:11 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 17 Sep 2015 07:55:11 -0700 (PDT) Subject: Layers Message-ID: <20150917145511.11BBB71D16@eggemoggin.niobe.net> 2015/9/9 1:48 -0700, david.lloyd at redhat.com: > ... > > 4) Layers > > I am concerned that layers do not go far enough to ensure that different > module graphs can interoperate. There is seemingly no provision to > establish dependency relationships between modules in different layers, > because layers are apparently strictly hierarchical. True -- a module in one layer can only depend upon modules in that layer or in lower layers. > This would make it > much more difficult to (for example) allow different containers to > interoperate within an app server (OSGi and Java EE, for example). > Today we use this kind of relatability to allow Java EE modules to > depend on modules constructed out of the legacy extensions system as > well as filesystem-backed JARs which are modularized independently of > our Java EE container, allowing them to be reused from other containers > as well. The goal of layers is not to allow completely different containers (or module systems) to interoperate, but rather to satisfy the "dynamic configuration" requirements previously discussed. - Mark From mark.reinhold at oracle.com Thu Sep 17 14:56:11 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 17 Sep 2015 07:56:11 -0700 (PDT) Subject: AccessibleObject.setAccessible() backward compatibility In-Reply-To: <1088108454.2110963.1441991735882.JavaMail.zimbra@u-pem.fr> References: <1088108454.2110963.1441991735882.JavaMail.zimbra@u-pem.fr> Message-ID: <20150917145611.158B371D1B@eggemoggin.niobe.net> 2015/9/11 10:15 -0700, forax at univ-mlv.fr: > There are two changes to setAccessible that make migration to the > module universe hard. > > The first one is that setAccessible() is now declared final which is a > binary incompatible change so it breaks Guava two projects of mine. As Alan Bateman already noted on the jigsaw-dev list, this is a bug (now fixed: http://hg.openjdk.java.net/jigsaw/jake/jdk/rev/9d521fd83608). > The second change is that now setAccessible() is now CallerSensitive > and throw an exception if the AccessibleObject is not visible from the > caller class. No, it throws an exception if the caller's module does not read the module of the target's declaring type, or if the package of the target's declaring type is not exported by its module (either to all modules, or at least to the caller's module), regardless of whether or not the type is public. > While this change may improve the security, it's a > backward incompatible change is not strictly required to support > modules it's more an enhancement of the current security model and i > don't think it's a good idea to mix it with the introduction of the > module support. This is not an enhancement of the security model per se (which would, anyway, be beyond the scope of this JSR). This change is, rather, in service of the strong-encapsulation goal (which does, of course, help ensure security). To truly support strong encapsulation would require taking setAccessible away completely. I think that's desirable in the (very) long term, but it would break too much existing code in the near term. The present behavior of setAccessible is a compromise, one of several possible choices. For types in unnamed modules (e.g., on the class path) it behaves exactly as it does in SE 8, since types in unnamed modules are always in exported packages. What a couple of early testers have run up against is the fact that if a type is in an unexported package in a named module (e.g., in a Java SE or JDK module) then setAccessible fails [1]. Some other consequences of this particular choice: - If you want to use a framework that uses setAccessible upon members of Java SE or JDK modules (e.g., some third-party serializers) then you'll have to use the command-line -XaddExports option to grant them the necessary access. - If you put code into a module, and you want to use a framework that uses setAccessible upon members of that module, then the module must export the packages containing the declaring types of those members, even if those types are not public. (Qualified exports can be used to limit, by name, the modules to which such packages are exported.) - The only way to really hide something in a module is to put it in an unexported package (but at least there's one way to really hide something!). If it's a package-private or private nested member in an exported package then setAccessible can still be used to break into it. While working on the prototype we did consider some alternatives: (a) Limit setAccessible to require the caller to have access to the target member's declaring type, consistent with the rest of the core reflection API. (The problem with this approach is that it does not allow setAccessible to be used on package-private types.) (b) Limit setAccessible so that both the caller and the target must be in the same module. (This would break almost as much existing code as taking setAccessible away completely.) (c) Variants of (a) or (b) which add a way to grant specific modules, via a command-line option, the power to use setAccessible as it exists today. (d) Do not change the behavior of setAccessible in SE 9, but deprecate it with an eye toward removing it in a future release, after supported (and safer!) replacements for known use cases have been defined. >From one of your later messages, I'm guessing that you'd be content with this last option. > More philosophically, every libraries that propose an > abstraction that hide underlying dirts provides an escape hatch, the > reflection API is one of such hatch of Java the language allowing to > bypass the typechecker and the security sandbox. Trying to close the > hatch will just make people to open holes in the nearby wall with > hacks that are less secure and that may even compromise the integrity > of the plateform. Philosophically, I can't agree with this. If encapsulation is to mean anything then it should not be possible to break it solely from within the language itself. Enabling such powers via an external, second-class mechanism such as a command-line option or a debugger is fine, but the history of Java has shown that if it's easy to break encapsulation then people will do so, increasing everyone's maintenance burdens over the long haul. - Mark [1] It throws an exception that does not extend SecurityException, which breaks some existing code in other ways, but that's a separate issue. From mark.reinhold at oracle.com Thu Sep 17 14:57:11 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 17 Sep 2015 07:57:11 -0700 (PDT) Subject: Revisiting encapsulation requirement In-Reply-To: <55F80883.9010603@redhat.com> References: <55F80883.9010603@redhat.com> Message-ID: <20150917145711.195E771D21@eggemoggin.niobe.net> 2015/9/15 5:01 -0700, david.lloyd at redhat.com: > Experts, I think we should revisit the "Encapsulation" requirement with > a view towards removal. > > The requirement is: > >> Encapsulation ??? The access-control mechanisms of the Java language >> and virtual machine must prevent code from accessing classes and >> interfaces in packages that are not exported by their containing >> modules, or in packages whose containing modules are not required by >> the module containing the code. These mechanisms must be operative >> even when a security manager is not present. I think it's premature, less than a week after the publication of the initial design and first early-access builds, to propose that we drop this central requirement. I suggest that we instead focus on the design as proposed, and figure out how to make it better. > The implementation amounts to modifying the > AccessibleObject.setAccessible() implementation This one method is just a small part of the overall encapsulation story, which is described in detail in the SotMS document. > for classes to perform a > check of the caller's class loader before determining whether to make > the object accessible. No, that's not exactly right; please see my detailed reply to R??mi. > As R??mi pointed out, this check is easily bypassed by using reflection, > by reflecting on AccessibleObject.class itself - since all modules > require "java.base", all modules implicitly can bypass this check. A simple bug, easily fixed. > But, > even assuming this could somehow be patched over, I think we should > consider dropping this requirement; I have two primary reasons for this > (though there may be others as well). > > The first reason is that without a security manager (or maybe even > *with* a security manager - but that's another discussion), it is > inevitable that any security measure used to protect this mechanism will > ultimately be bypassed, rendering its security value useless; adding > more complexity to the system to do so will only increase the overall > vulnerability of the platform. Yes, security is hard. That's not a reason to give up on it. > The second reason is that it is often useful to gain access to and > inspect public classes that are not necessarily visible from your > module. Frameworks do this commonly for example - framework > implementations will almost never import modules that they introspect. Frameworks in their current form often work as-is on the class path, because of the way that unnamed modules work, as discussed in my reply to R??mi. See, e.g., Peter Levart's report on the jigsaw-dev list [1]. If you convert a framework (or any other code) into a module then you can use the Module::addReads method [2][3] to add read edges to the module graph at run time. This is, in effect, an API that allows a module to `require` additional modules dynamically. - Mark [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-September/004537.html [2] http://openjdk.java.net/projects/jigsaw/spec/sotms/#increasing-readability [3] http://cr.openjdk.java.net/~mr/jigsaw/spec/api/java/lang/reflect/Module.html#addReads-java.lang.reflect.Module- From david.lloyd at redhat.com Thu Sep 17 15:09:01 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 17 Sep 2015 10:09:01 -0500 Subject: Revisiting encapsulation requirement In-Reply-To: <20150917145711.195E771D21@eggemoggin.niobe.net> References: <55F80883.9010603@redhat.com> <20150917145711.195E771D21@eggemoggin.niobe.net> Message-ID: <55FAD78D.9070306@redhat.com> On 09/17/2015 09:57 AM, mark.reinhold at oracle.com wrote: > 2015/9/15 5:01 -0700, david.lloyd at redhat.com: >> Experts, I think we should revisit the "Encapsulation" requirement with >> a view towards removal. >> >> The requirement is: >> >>> Encapsulation ??? The access-control mechanisms of the Java language >>> and virtual machine must prevent code from accessing classes and >>> interfaces in packages that are not exported by their containing >>> modules, or in packages whose containing modules are not required by >>> the module containing the code. These mechanisms must be operative >>> even when a security manager is not present. > > I think it's premature, less than a week after the publication of the > initial design and first early-access builds, to propose that we drop > this central requirement. > > I suggest that we instead focus on the design as proposed, and figure > out how to make it better. > >> The implementation amounts to modifying the >> AccessibleObject.setAccessible() implementation > > This one method is just a small part of the overall encapsulation story, > which is described in detail in the SotMS document. > >> for classes to perform a >> check of the caller's class loader before determining whether to make >> the object accessible. > > No, that's not exactly right; please see my detailed reply to R??mi. > >> As R??mi pointed out, this check is easily bypassed by using reflection, >> by reflecting on AccessibleObject.class itself - since all modules >> require "java.base", all modules implicitly can bypass this check. > > A simple bug, easily fixed. > >> But, >> even assuming this could somehow be patched over, I think we should >> consider dropping this requirement; I have two primary reasons for this >> (though there may be others as well). >> >> The first reason is that without a security manager (or maybe even >> *with* a security manager - but that's another discussion), it is >> inevitable that any security measure used to protect this mechanism will >> ultimately be bypassed, rendering its security value useless; adding >> more complexity to the system to do so will only increase the overall >> vulnerability of the platform. > > Yes, security is hard. That's not a reason to give up on it. Of course not. But as far as I can tell, it has yet to be shown that this requirement and its corresponding implementation changes actually improve security in any real way. I'm not talking in hand-wavy generalizations like "encapsulation leads to better security", I'm talking about brass-tacks examples. >> The second reason is that it is often useful to gain access to and >> inspect public classes that are not necessarily visible from your >> module. Frameworks do this commonly for example - framework >> implementations will almost never import modules that they introspect. > > Frameworks in their current form often work as-is on the class path, > because of the way that unnamed modules work, as discussed in my reply > to R??mi. See, e.g., Peter Levart's report on the jigsaw-dev list [1]. Correct me if I'm wrong, but you seem to be making an assumption here that no existing artifacts can or should be "graduated" as-is into modules. This is further underscored by this statement: > If you convert a framework (or any other code) into a module then you can > use the Module::addReads method [2][3] to add read edges to the module > graph at run time. This is, in effect, an API that allows a module to > `require` additional modules dynamically. The compatibility story is getting increasingly foggy here. On the one hand, for example, "JDK 9 retains this three-level hierarchy, in order to preserve compatibility" - a hierarchy that as far as I can tell, very, *very* few existing things actually care about - but on the other hand, nothing in the Jigsaw module system prototype is compatible in any way with the class loading world as we know it - which very *very* many existing things rely on. > [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-September/004537.html > [2] http://openjdk.java.net/projects/jigsaw/spec/sotms/#increasing-readability > [3] http://cr.openjdk.java.net/~mr/jigsaw/spec/api/java/lang/reflect/Module.html#addReads-java.lang.reflect.Module- > -- - DML From david.lloyd at redhat.com Thu Sep 17 15:34:37 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 17 Sep 2015 10:34:37 -0500 Subject: Security In-Reply-To: <20150917145611.158B371D1B@eggemoggin.niobe.net> References: <1088108454.2110963.1441991735882.JavaMail.zimbra@u-pem.fr> <20150917145611.158B371D1B@eggemoggin.niobe.net> Message-ID: <55FADD8D.5080807@redhat.com> On 09/17/2015 09:56 AM, mark.reinhold at oracle.com wrote: > 2015/9/11 10:15 -0700, forax at univ-mlv.fr: >> While this change may improve the security, it's a >> backward incompatible change is not strictly required to support >> modules it's more an enhancement of the current security model and i >> don't think it's a good idea to mix it with the introduction of the >> module support. > > This is not an enhancement of the security model per se (which would, > anyway, be beyond the scope of this JSR). This change is, rather, in > service of the strong-encapsulation goal (which does, of course, help > ensure security). I think this assertion should be substantiated with some examples. While strong encapsulation is good in terms of design principles, to me it's only evident that this somehow leads to better security in the most hand-wavy possible way (in particular, as it relates to this specific mechanism). On the other hand, there is strong evidence to support the assertion that enhancing the security model in any way immediately leads to new CVEs. The more behavioral rules there are in place, the more vectors for exploitation will appear. So if you, like me, define "security" as "resistance to exploitation by untrusted code", I fail to see how this kind of change can possibly be an improvement over the status quo (particularly, security between class loaders). If you add in the inevitable breakage in real-world software that will result, the picture looks even worse. > To truly support strong encapsulation would require taking setAccessible > away completely. I think that's desirable in the (very) long term, but > it would break too much existing code in the near term. I don't agree that this is true. Encapsulation is really a social construct; while accessibility can be used to create and enforce encapsulation-justified rules, it doesn't *have* to. You do need some kind of back door, no matter what, or else you severely limit the power and capability of the platform (as evidenced by the widespread usage of the reflection back door by many widely-used frameworks generally considered to be powerful and capable). If encapsulation as a design principle is the goal, then simple isolation rules between modules (as between unrelated class loaders) has already been proven to be highly effective. The relationship between this new encapsulation rule and security, on the other hand, has yet to be shown. In this light, I don't see how these changes cannot be considered a modification of the security model just because they ride in on the coattails of a new abstraction. > The present behavior of setAccessible is a compromise, one of several > possible choices. For types in unnamed modules (e.g., on the class path) > it behaves exactly as it does in SE 8, since types in unnamed modules are > always in exported packages. What a couple of early testers have run up > against is the fact that if a type is in an unexported package in a named > module (e.g., in a Java SE or JDK module) then setAccessible fails [1]. > > Some other consequences of this particular choice: > > - If you want to use a framework that uses setAccessible upon members > of Java SE or JDK modules (e.g., some third-party serializers) then > you'll have to use the command-line -XaddExports option to grant them > the necessary access. > > - If you put code into a module, and you want to use a framework that > uses setAccessible upon members of that module, then the module must > export the packages containing the declaring types of those members, > even if those types are not public. (Qualified exports can be used > to limit, by name, the modules to which such packages are exported.) > > - The only way to really hide something in a module is to put it in > an unexported package (but at least there's one way to really hide > something!). If it's a package-private or private nested member in > an exported package then setAccessible can still be used to break > into it. > > While working on the prototype we did consider some alternatives: > > (a) Limit setAccessible to require the caller to have access to the > target member's declaring type, consistent with the rest of the > core reflection API. (The problem with this approach is that it > does not allow setAccessible to be used on package-private types.) > > (b) Limit setAccessible so that both the caller and the target must be > in the same module. (This would break almost as much existing code > as taking setAccessible away completely.) > > (c) Variants of (a) or (b) which add a way to grant specific modules, > via a command-line option, the power to use setAccessible as it > exists today. > > (d) Do not change the behavior of setAccessible in SE 9, but deprecate > it with an eye toward removing it in a future release, after > supported (and safer!) replacements for known use cases have been > defined. > > From one of your later messages, I'm guessing that you'd be content with > this last option. I would be - if it could be shown that a suitable and effective replacement could be put in place, and if the future release were far enough in the future in order to mitigate the massive amount of breakage in existing reflection-based code, and if it could be shown that there are material benefits to doing so. >> More philosophically, every libraries that propose an >> abstraction that hide underlying dirts provides an escape hatch, the >> reflection API is one of such hatch of Java the language allowing to >> bypass the typechecker and the security sandbox. Trying to close the >> hatch will just make people to open holes in the nearby wall with >> hacks that are less secure and that may even compromise the integrity >> of the plateform. > > Philosophically, I can't agree with this. If encapsulation is to mean > anything then it should not be possible to break it solely from within > the language itself. Enabling such powers via an external, second-class > mechanism such as a command-line option or a debugger is fine, but the > history of Java has shown that if it's easy to break encapsulation then > people will do so, increasing everyone's maintenance burdens over the > long haul. I think this may be a somewhat narrow view. People break encapsulation when they need a capability that cannot be provided another way. Now the JDK, unfortunately, historically contains very many "goodies" and bits of useful functionality that users want access to, so it is a disproportionately popular target for such breakage, but apart from accessing JDK internals, I believe that the use cases for breaking in through the security model are legitimate and should be allowed to adequately privileged code. Using an appropriate security model, it is not difficult to ensure that such powers are not exploited (though while in practice it is more difficult due to a variety of factors, I do not believe those factors are directly related to this mechanism but rather to exceptions, special "holes", and other technical debt that exists in the JDK itself for historical or other reasons). > [1] It throws an exception that does not extend SecurityException, which > breaks some existing code in other ways, but that's a separate issue. Yep I'm personally not too concerned with these issues (which are generally easily addressed); the focus of the discussion lies elsewhere, in my view. -- - DML From david.lloyd at redhat.com Thu Sep 17 15:40:14 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 17 Sep 2015 10:40:14 -0500 Subject: Revisiting encapsulation requirement In-Reply-To: <1285323903.1293876.1442394059034.JavaMail.zimbra@u-pem.fr> References: <55F80883.9010603@redhat.com> <1307198778.1052434.1442332368236.JavaMail.zimbra@u-pem.fr> <55F84388.7070300@redhat.com> <864168724.1106362.1442346442276.JavaMail.zimbra@u-pem.fr> <55F8885B.3030903@redhat.com> <1285323903.1293876.1442394059034.JavaMail.zimbra@u-pem.fr> Message-ID: <55FADEDE.9050002@redhat.com> On 09/16/2015 04:00 AM, Remi Forax wrote: > > >> De: "David M. Lloyd" >> ?: jpms-spec-experts at openjdk.java.net >> Envoy?: Mardi 15 Septembre 2015 23:06:35 >> Objet: Re: Revisiting encapsulation requirement >> >> On 09/15/2015 02:47 PM, Remi Forax wrote: >>> Hi David, >>> >>> ----- Mail original ----- >>>> De: "David M. Lloyd" >>>> ?: jpms-spec-experts at openjdk.java.net >>>> Envoy?: Mardi 15 Septembre 2015 18:12:56 >>>> Objet: Re: Revisiting encapsulation requirement >>>> >>>> Do you have an example of a real-world problem (security or otherwise) >>>> that is solved by adding access checks based on the caller ? >>> >>> You take the problem by the wrong side :) >> >> Maybe. >> >>> Let's suppose you agree that both the notion of visibility (private, >>> package private, etc) and the reflection API are something useful. >> >> Sure, we've been calling that "accessibility" up until now. >> >>> A simple model is to make the Java visibility rules and the reflection API >>> behavior aligned. >>> This implies that some reflexive calls have to be caller sensitive. >>> Then as i said earlier, you need a way to escape the sandbox, a escape >>> hatch, to program things like meta-protocols as most of the Java EE specs >>> are, hence, the addition of setAccessible(). >> >> So if I understand correctly, you propose that, in addition to verifying >> that the target type is accessible to the caller, we should also verify >> that it is visible when we do the actual access check (i.e. "invoke() >> should throw...if...the class is not declared as a dependency") and >> forget doing a special check when we call setAccessible(). If I'm >> misunderstanding you, stop me here! > > yes, > >> >> But assuming all that is true, my concern is that today, I can access >> any public member of any public class in the system that I can get a >> Class object for, even if it comes from a different, isolated class >> loader. Under the new rules and even with your suggestion, a class has >> to be public *and* visible to me, else setAccessible() is the only >> choice - even if it's public, and was handed to me! To me this makes no >> sense and does not add any security- or encapsulation-related benefits >> that I can come up with - quite the opposite actually, since now you >> need to be granted sweeping permissions to access it. This is why I'm >> seeking a counter-example which supports this idea. > > Let say you have two modules, one that contains an interface > and another that contains a class that implements that interface in another package. > > currently if you write a code like this: > o.getClass().getMethods("foo").invoke(o) > by accessing to a public method of the implementation class that may be not declared in the interface, > you're breaking encapsulation. > > if you write: > Interface o = ... > Interface.class.getMethods("foo").invoke(o) > there is no problem. > > but maybe it's not a problem that worth to be fixed ? Yeah you don't always have an interface though, let alone one in a common ancestor module. To me "public" should mean "public". -- - DML From jason at takari.io Thu Sep 17 15:56:08 2015 From: jason at takari.io (Jason van Zyl) Date: Thu, 17 Sep 2015 11:56:08 -0400 Subject: Please welcome Jason van Zyl to JSR 376 In-Reply-To: <20150917075105.496107@eggemoggin.niobe.net> References: <20150917075105.496107@eggemoggin.niobe.net> Message-ID: <8099B41A-6549-47C2-8721-04150F6A0E8B@takari.io> Thanks Mark. Once I?ve finished reading over all the archives I?ll likely have more to say. Again, thanks for letting me join the EG at this late stage. > On Sep 17, 2015, at 10:51 AM, mark.reinhold at oracle.com wrote: > > Jason van Zyl, well-known for his work on Maven (among other things), > has asked to join this EG. > > Ordinarily I'm reluctant to accept new EG members this late in the > game, after we've settled the requirements, but Jason has agreed to > accept the requirements as-is and understands that the bar for revising > them is relatively high. I'm confident that he will bring valuable > perspectives to our discussions and help to ensure the overall success > of the JSR. > > (Jason -- if there's anything further you'd like to say by way of > introduction, please do so!) > > - Mark Thanks, Jason ---------------------------------------------------------- Jason van Zyl Founder, Takari and Apache Maven http://twitter.com/jvanzyl http://twitter.com/takari_io --------------------------------------------------------- the course of true love never did run smooth ... -- Shakespeare From david.lloyd at redhat.com Thu Sep 17 16:17:49 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 17 Sep 2015 11:17:49 -0500 Subject: Please welcome Jason van Zyl to JSR 376 In-Reply-To: <20150917075105.496107@eggemoggin.niobe.net> References: <20150917075105.496107@eggemoggin.niobe.net> Message-ID: <55FAE7AD.6090905@redhat.com> Welcome! On 09/17/2015 09:51 AM, mark.reinhold at oracle.com wrote: > Jason van Zyl, well-known for his work on Maven (among other things), > has asked to join this EG. > > Ordinarily I'm reluctant to accept new EG members this late in the > game, after we've settled the requirements, but Jason has agreed to > accept the requirements as-is and understands that the bar for revising > them is relatively high. I'm confident that he will bring valuable > perspectives to our discussions and help to ensure the overall success > of the JSR. > > (Jason -- if there's anything further you'd like to say by way of > introduction, please do so!) > > - Mark > -- - DML From david.lloyd at redhat.com Thu Sep 17 17:26:07 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 17 Sep 2015 12:26:07 -0500 Subject: Module declarations In-Reply-To: <20150917145211.06B7371D05@eggemoggin.niobe.net> References: <20150917145211.06B7371D05@eggemoggin.niobe.net> Message-ID: <55FAF7AF.6050109@redhat.com> On 09/17/2015 09:52 AM, mark.reinhold at oracle.com wrote: > (splitting your initial reply into separate threads) > > 2015/9/9 1:48 -0700, david.lloyd at redhat.com: >> On 09/09/2015 11:51 AM, mark.reinhold at oracle.com wrote: >>> To get the discussion started I suggest we focus first on the high-level >>> aspects of the design covered in "The State of the Module System" [1]. >>> >>> [1] http://openjdk.java.net/projects/jigsaw/spec/sotms/ >> >> 1) The first thing that jumps out at me (again) is making module >> declarations be a binary construct produced by the actual programming >> language. It strikes me as arbitrary as well as redundant (given >> there's a programmatic API to build descriptors, they could simply be >> parsed from text nearly as well, and a lot more nicely from the user's >> perspective). As pointed out in the document, it is already expected >> that additional tooling would synthesize this file at build time, >> further decreasing the value of such an idea. The module class file >> just seems pointless to me. > > One of our requirements is fidelity across all phases. To achieve this, > javac (or whatever Java compiler you use) must be able to locate and > interpret descriptions of modules. If modules are not described by > language constructs that are compiled into class files, then how and > where would you suggest that a compiler find such information? They could still be bundled with the artifact, but in clear text format perhaps. Or even in binary format, but in either case, the descriptor format should be specific to a specific default module configuration (e.g. an application module loader) rather than be a hard-wired aspect of all modules. To me "fidelity" means the same thing as "integrity": I can build an artifact and expect it to run on any environment which meets its ABI requirements in any way. I don't think that hard-coding module dependency names in the artifact itself is a good way to achieve this though. Doing so hampers the ability to separate building from packaging, since if you need to specify different run time environment parameters in the packaged version, you must crack open and modify the artifact to do so (new SHA1/MD5 checksums, etc.). To me a better way to achieve this is a multi-tiered approach, where each tier depends on the tier before (but not vice-versa). The build artifact is identified and versioned as today (Maven-style), and packaged alongside associated build dependency information. This information is global - i.e. it is the same regardless of where and how the artifact is used, and the artifact implicitly has a clear ABI. Building a new artifact only depends on build dependency information associated with the dependency artifacts. This method of building is well-proven. The next tier is a packaging/distribution tier. This pertains specifically to the environment in which artifacts are utilized. In this tier, modules are defined which consist of the combination of build artifacts (with versions) and run-time, distribution-specific dependency information. The distribution could (and should) be versioned as a whole; individual module versions within the environment may correspond with the artifact(s) that comprise it (but this is not necessary). The distribution information, being versioned, could easily be distributed wholly or in sections on a system like Maven. An important aspect of this tier is testing, and its role in managing a stable module distribution. When a new version of an artifact is produced and made generally available, before a module may be included in a distribution it is necessary to determine whether such inclusion will regress any aspect of the existing infrastructure. While completely retesting an entire distribution every time a component or combination of components is updated is generally infeasible, it is possible to use the distribution's run-time dependency information to attain a level of confidence regarding the impact of an upgrade, by working downwards along the dependent graph incrementally from each changed module until the desired confidence is achieved that consumers (direct or indirect) are unlikely to be regressed. The last tier is the run time tier, in which some or all of the module graph of a distribution is made available for execution, presumably (but not necessarily) local to the running JVM. No version information is used at run time outside of an informational capacity. The reasons that this system is superior to a "flat" system are hopefully obvious: what you run is not the same as what you build, and no system exists which can make it be the same, unless you introduce a central broker to all future Java builds which enforces consistency. A "version" of a module is really a product of the version of the artifact and the environment in which the module is executed. It is highly unlikely that restricting usage of the module system to a single effective distribution will suffice for most, or even many, use cases. But this is exactly what will effectively happen by flattening run time information all the way down to the build time layer: the result will be one massive, headless module ecosystem whose overall security and stability is completely indeterminate, which evolves in a completely unpredictable manner. This is the complete opposite of fidelity in my mind, and solves nothing of practical value. > (The statement that "build tools, in particular, can synthesize module > declarations from information already available in project descriptions" > in the SotMS document is, I see, overly terse. A tool could suggest an > initial module declaration with `requires` clauses derived from a > pom.xml file, or equivalent, but it'd be up to the developer to specify > the rest. I'll fix this in the next version.) Yes I agree that a build tool can, at best, only make suggestions based on information it may have about the target environment (which is conceptually not very much unless the build tool becomes highly aware of packaging considerations). Only something like a packaging tool that has awareness of the distribution as a whole can give really useful dependency hints for run time though. -- - DML From hearn at vinumeris.com Fri Sep 18 19:56:47 2015 From: hearn at vinumeris.com (Mike Hearn) Date: Fri, 18 Sep 2015 20:56:47 +0100 Subject: Module ABI versioning Message-ID: I'd like to appeal for a reconsideration on the stance on module versioning, specifically, I'd like to ask for the simplest/most basic possible versioning scheme (a single number) to be included in module metadata. My use case is a library like Guava. It hits the sweet/sour spot for jar hell: 1. Very widely used by both apps and libraries 2. Iterates rapidly 3. Releases are not backwards compatible This can result in dependency deadlocks where upgrading one library is impossible because it depends on Guava N, yet you're also using another library that uses Guava M, and the two versions are not compatible. Modules would fix this, except for the inability to distinguish the N and M modules. One can argue, perhaps reasonably, that if two versions of a module export classes with the same names but which aren't compatible then the modules should have different names instead, i.e. an ABI version should be included in the module name. Whilst this would work, it seems somehow ad hoc. In the native code world this problem is solved for a long time already. UNIX libraries do indeed include their ABI version in their "soname", but it's done in a standardised way and every DSO built with the normal tools includes an ABI version (libfoo.so.0). From pbenedict at apache.org Fri Sep 18 20:33:38 2015 From: pbenedict at apache.org (Paul Benedict) Date: Fri, 18 Sep 2015 15:33:38 -0500 Subject: Module ABI versioning In-Reply-To: References: Message-ID: Mike, doesn't the semantic versioning scheme support this? There is no ABI compatibility among major version changes. There might be ABI compatiblity changes on minor versions (but I wouldn't rely on that). And patch versions must have ABI compatibility. Guava is using 19 for it's major version. So they had at least 19 ABI incompatible changes. Though anything in the 19.0.X series would be fine to swap, same with 18.0.X, etc. Cheers, Paul On Fri, Sep 18, 2015 at 2:56 PM, Mike Hearn wrote: > I'd like to appeal for a reconsideration on the stance on module > versioning, specifically, I'd like to ask for the simplest/most basic > possible versioning scheme (a single number) to be included in module > metadata. > > My use case is a library like Guava. It hits the sweet/sour spot for jar > hell: > > 1. Very widely used by both apps and libraries > 2. Iterates rapidly > 3. Releases are not backwards compatible > > This can result in dependency deadlocks where upgrading one library is > impossible because it depends on Guava N, yet you're also using another > library that uses Guava M, and the two versions are not compatible. Modules > would fix this, except for the inability to distinguish the N and M > modules. > > One can argue, perhaps reasonably, that if two versions of a module export > classes with the same names but which aren't compatible then the modules > should have different names instead, i.e. an ABI version should be included > in the module name. Whilst this would work, it seems somehow ad hoc. > > In the native code world this problem is solved for a long time already. > UNIX libraries do indeed include their ABI version in their "soname", but > it's done in a standardised way and every DSO built with the normal tools > includes an ABI version (libfoo.so.0). > From hearn at vinumeris.com Fri Sep 18 22:03:17 2015 From: hearn at vinumeris.com (Mike Hearn) Date: Fri, 18 Sep 2015 23:03:17 +0100 Subject: Module ABI versioning In-Reply-To: References: Message-ID: That's nearly how semver works, but not quite. Major.minor.patchlevel means: - Major versions are incompatible - Minor versions are backwards compatible but may be extended - Patchlevels are entirely swappable with each other That is, code may require 18.1+ and not work on 18.0 or 19.0, but it will work on 18.2 I think it'd be great if Java modules understood semantic versioning. However, I would be satisfied with just understanding major versions, as that's all that's required to avoid link conflicts, and it's A-OK by me if that's something as simple as "module names should include the major version as the final dotted component". From forax at univ-mlv.fr Sat Sep 19 16:41:27 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Sat, 19 Sep 2015 18:41:27 +0200 (CEST) Subject: Please welcome Jason van Zyl to JSR 376 In-Reply-To: <8099B41A-6549-47C2-8721-04150F6A0E8B@takari.io> References: <20150917075105.496107@eggemoggin.niobe.net> <8099B41A-6549-47C2-8721-04150F6A0E8B@takari.io> Message-ID: <57536120.2457835.1442680887908.JavaMail.zimbra@u-pem.fr> Welcome too, R?mi ----- Mail original ----- > De: "Jason van Zyl" > ?: jpms-spec-experts at openjdk.java.net > Envoy?: Jeudi 17 Septembre 2015 17:56:08 > Objet: Re: Please welcome Jason van Zyl to JSR 376 > > Thanks Mark. Once I?ve finished reading over all the archives I?ll likely > have more to say. Again, thanks for letting me join the EG at this late > stage. > > > On Sep 17, 2015, at 10:51 AM, mark.reinhold at oracle.com wrote: > > > > Jason van Zyl, well-known for his work on Maven (among other things), > > has asked to join this EG. > > > > Ordinarily I'm reluctant to accept new EG members this late in the > > game, after we've settled the requirements, but Jason has agreed to > > accept the requirements as-is and understands that the bar for revising > > them is relatively high. I'm confident that he will bring valuable > > perspectives to our discussions and help to ensure the overall success > > of the JSR. > > > > (Jason -- if there's anything further you'd like to say by way of > > introduction, please do so!) > > > > - Mark > > Thanks, > > Jason > > ---------------------------------------------------------- > Jason van Zyl > Founder, Takari and Apache Maven > http://twitter.com/jvanzyl > http://twitter.com/takari_io > --------------------------------------------------------- > > the course of true love never did run smooth ... > > -- Shakespeare > > > > > > > > > > > > > From forax at univ-mlv.fr Sat Sep 19 17:02:15 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Sat, 19 Sep 2015 19:02:15 +0200 (CEST) Subject: module name convention In-Reply-To: <1441288605.2461702.1442681029132.JavaMail.zimbra@u-pem.fr> Message-ID: <1052601312.2492216.1442682135412.JavaMail.zimbra@u-pem.fr> Mani from AdoptAJSR has a very good question at the end of this document: https://docs.google.com/document/d/1KlumN74IGt-TU-Md3Fn5h4sXHa75RApWNLszUTVp-DE/edit Quote: "Namespace for modules overlap with that of packages - can be confusing after looking at a lot of module descriptors. When exporting, excluding, using, etc? of modules we do not say this module is dependent on that, but modules dependencies are resolved by the resolving which packages they are dependent on (uses this package, exports that package). And package names look similar to or can look the same as module names, although they are different concepts." Both Eclipse and Maven use package name or at least package prefix as convention to name module and as Mani said it really confusing when you write a module-info file. Given that the name of the JDK modules doesn't respect that convention but still use '.' to separate the different components, I wonder if we should not try to come with a convention for module name that allow to easily disambiguate between package name and module name. By example, java_base, java-base, java~base ... Obviously the other solution is to prefix a name by "package" or "module". R?mi From nipa at codefx.org Sun Sep 20 08:57:02 2015 From: nipa at codefx.org (Nicolai Parlog) Date: Sun, 20 Sep 2015 10:57:02 +0200 Subject: Module ABI versioning In-Reply-To: References: Message-ID: <55FE74DE.3010401@codefx.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi, I can come up with two ugly workarounds (see below) for dependency hell as described by Mike. If other solutions exist, it would be great if someone pointed them out. If no less hacky alternatives arise, it looks like Jigaw would indeed not keep its informal (!) promise to alleviate dependency hell. At build time: The build tool would have to identify the dependency on two conflicting versions. It would then create artificial modules (with 'requires public') that do include the version in their name, e.g. com.google.guava_18_0_0 and com.google.guava_16_0_1. Finally, it would alter the dependent modules' module-info.class files to include these versions. This means that the deployed modules actually differ from the ones available in public repositories. It also breaks when the modules are signed (which, as far as I see, is currently not possible). At run time: A library could offer the ability to start an application in some module constellation and then reflectively alter the module graph (maybe utilizing layers) so that all dependencies on different versions are correctly represented. This would require manual or automatic configuration based on the problematic versions. It also means that the two different versions can be distinguished by the module system which in turn requires the artificial modules described above. so long ... Nicolai On 18.09.2015 21:56, Mike Hearn wrote: > I'd like to appeal for a reconsideration on the stance on module > versioning, specifically, I'd like to ask for the simplest/most > basic possible versioning scheme (a single number) to be included > in module metadata. > > My use case is a library like Guava. It hits the sweet/sour spot > for jar hell: > > 1. Very widely used by both apps and libraries 2. Iterates rapidly > 3. Releases are not backwards compatible > > This can result in dependency deadlocks where upgrading one library > is impossible because it depends on Guava N, yet you're also using > another library that uses Guava M, and the two versions are not > compatible. Modules would fix this, except for the inability to > distinguish the N and M modules. > > One can argue, perhaps reasonably, that if two versions of a module > export classes with the same names but which aren't compatible then > the modules should have different names instead, i.e. an ABI > version should be included in the module name. Whilst this would > work, it seems somehow ad hoc. > > In the native code world this problem is solved for a long time > already. UNIX libraries do indeed include their ABI version in > their "soname", but it's done in a standardised way and every DSO > built with the normal tools includes an ABI version (libfoo.so.0). > - -- PGP Key: http://keys.gnupg.net/pks/lookup?op=vindex&search=0xCA3BAD2E9CCCD509 Web: http://codefx.org a blog about software development http://do-foss.de Free and Open Source Software for the City of Dortmund Twitter: https://twitter.com/nipafx -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJV/nTYAAoJEMo7rS6czNUJ91MP/R4PhtDlL+Hy3zcV3Ql5HuhG aJNrJHODWBIa0TqePmsKNYnMx63+mjvk5a8mc3Wi0mnU8Yw0QSmAqKVfYnJ8Hrp5 i5YQeG57mIAW3jhaPYLDhercMgj9wFFF6RLJWSk2UMNW5S2Ac8VJjteMBdWoGihs IhjS0ZZkBBScNmQvMI7cEEFbeoPuLsDc8S3Lu/Y8EZcpFLKc9KjdjeLYkbrxUeB0 YND3sLvtTbpNosBcdsqDN0k813YJbBX1GVKE5L9lMyImkRdiiR4SQm8+bGzdBq/B RZj06swf8/43ljogjxPipD3uqDxwnxMisqyOJqOwUyO/7oF8IiP9qp9kRUwCr091 w60OGWt+laXN0sg591liS4xvQDFR5FNkxlW5L30UI7h6K4XhbFD3ikWFdhWFdAzu Y7/lFmaYe4en33j7wqRhnevK1mn/BUZtMjMkAxs9c4sALHhblAHADAeCiK5r4qcx TfRspS2lV8H9SgCGKVw1DN50VZ4qfkK4Pkp7hKEF/A4ebRsmDVKLiGNiBRP9iDME N4KjO5RwoXSf69ZYM8LDBCwzFaqvJtTmA4hiZ32McpCZU2nTDMIxwJaTLlDkXY/X aU01sx8ORa92EzDGg0Qa9KzA/7Tvum/GA81upAS3I+5LcEGwNBt3wAyQgVRQWL3w LBaP8WqIJW5EFoWYYHW3 =Rbee -----END PGP SIGNATURE----- From blackdrag at gmx.org Tue Sep 22 07:32:27 2015 From: blackdrag at gmx.org (Jochen Theodorou) Date: Tue, 22 Sep 2015 09:32:27 +0200 Subject: some basic questions about the module system concept Message-ID: <5601040B.80007@gmx.org> Hi all, I am still not sure this is the right list to ask those things, but I guess you guys will tell me if there is a better place. First of all I was looking for a graph or something alike that shows what modules are supposed to be visible to applications. Couldn't easily find that. But I would assume anything in com.sun is not. There are of course "special cases"... I am thinking here about the visibility of the internal javac classes from com.sun.tools.javac and things like "hackish" annotation processors and Project Lombok. They use those internal classes, basically extending javac through annotations. Are those supposed to be possible with jigsaw? It is no official API and as such I would assume the javac classes are not made accessible outside the javac module. The bigger part of course is about a scripting language on the JVM. So let us assume the following scenario... I write an application in Groovy and the application will be a module A, with large parts not accessible to the outside. Let us further assume, that Groovy will make method calls with reflection from its runtime. From what I understood about Jigsaw so far, this can work only, if Groovy has its own module and the application allows Groovy accessing the module. Now, assume there is another application with module B, written in Groovy as well and working on the classes of A. Now, since method calls are done through the runtime and since the Groovy runtime is supposed to have access to module A, doesn't that mean, module B can have full access to module A through the Groovy runtime? Extending this... Groovy can compile and execute scripts at runtime... as far as I understand this, those scripts would normally end up in an unnamed module. And this package has no access rights to internal module classes. If I am right then this means that module A cannot use scripts to operate on internal classes. If I am wrong, any arbitrary script can access any class/method in module A. There are cases when you want access and cases when not. But I have not seen anything yet, explaining how to control this. Is there anything in the concept for this? I think I may have seen that the runtime generated class is added to the module with the same package in the same classloader... Or I should say has access to those. But this does not really matter to use, because in the above scenario we normally have no access to the classloader loading the application and per default a runtime compiled script has its own classloader altogether. bye Jochen -- Jochen "blackdrag" Theodorou blog: http://blackdragsview.blogspot.com/ From Alan.Bateman at oracle.com Tue Sep 22 08:08:39 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 22 Sep 2015 09:08:39 +0100 Subject: some basic questions about the module system concept In-Reply-To: <5601040B.80007@gmx.org> References: <5601040B.80007@gmx.org> Message-ID: <56010C87.7020602@oracle.com> On 22/09/2015 08:32, Jochen Theodorou wrote: > Hi all, > > I am still not sure this is the right list to ask those things, but I > guess you guys will tell me if there is a better place. > > First of all I was looking for a graph or something alike that shows > what modules are supposed to be visible to applications. Couldn't > easily find that. But I would assume anything in com.sun is not. > Do you build the prototype in the jigsaw/jake forest? If so then then the gengraphs directory in the build output will probably help you on this topic. In particular, the build generates a module-summary.html that is a table that summarizes both the proposed standard and JDK-specific modules. That summary has the API packages exported by each module. Some JDK-specific modules don't export any API packages because they are service provider modules or just API-less tools. I expect this table will be useful to the JSR for Java SE 9 once it gets to define the standard modules. There is also a .dot for each module so you can visualize the graphs too. Another easy way to list the observable modules in the prototype is "java -listmods", drill down by specifying the module name as sub-options ("java -listmods:java.se" for example). -Alan. From pbenedict at apache.org Thu Sep 24 01:29:15 2015 From: pbenedict at apache.org (Paul Benedict) Date: Wed, 23 Sep 2015 20:29:15 -0500 Subject: Module metadata doesn't really need to be .class Message-ID: So I was happy to see the .jmod be in zip format. That allows universal tooling to access the contents. I am unhappy to see the module info to be a class file. I can't agree with the justification. I believe David Lloyd brought up a similar issue. There is nothing about the module data requires language support. It's just metadata. This easily could have been put in a MANIFEST.MF or an XML descriptor. What is the benefit for having it be binary ... as a Java class file? It makes no sense and would like this to be reconsidered. Cheers, Paul From david.lloyd at redhat.com Thu Sep 24 15:38:28 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 24 Sep 2015 10:38:28 -0500 Subject: Module use cases versus descriptors, packaging, and configuration policies Message-ID: <560418F4.4060806@redhat.com> Modules as a concept apply to many different use cases: * Standalone applications (class path replacement) using static module path modules * Standalone applications for small/"micro" platforms * Containers like Java EE and OSGi * Various custom dynamic plugin systems on client and server * The JDK itself Every use case has different needs in terms of the way that modules are named, located, utilized, built, and otherwise managed. Furthermore, each use case varies in terms of the different phases of development (build, test, run, package, deploy). Because of this, I feel that a one-size-fits-all approach to module descriptors and loading policies is not in the best interest of most users. There is an alternative strategy that offers superior applicability to most if not all of these use cases. I will describe this strategy in pieces. * Naming Each use case has different naming rules. Static module path modules, it is more or less agreed, are well-suited towards a dot-separated, reverse-domain-name strategy like we use for packages. OSGi and Java EE have looser naming requirements; other systems have other needs. Thus I propose that the core module system support opaque String module identifiers, with each module configuration being solely responsible for any additional name validation rules which apply to it. This allows each use case to be internally consistent, without hampering the overall flexibility of the module system. * Descriptors The modules which are shipped with the JDK have a static, "locked" dependency relationship. Static module path modules have relationships that might not be known until run time. Java EE and OSGi modules may have descriptions that are fully dynamically generated from a variety of inputs, and custom plugin systems may have other strategies for defining modules. Thus mandating a single internal or external descriptor format is not a desirable strategy. Instead this should be made flexible, with each module configuration being solely responsible for the location and description of modules that it contains, be it by name or by globally provided service interface. This enables module systems to use an optimal strategy for that use case. * Packaging Defining a packaging format for the static module path modules is important, but such a packaging format is very unlikely to be applicable to other systems. Dependencies expressed by such modules are necessarily limited to other static modules, or the JDK itself; however (to give just one example) in higher-level configurations like Java EE or OSGi, a corresponding alternative class of dependencies is required to express relative relationships between other modules within that container, and other modules within other peer containers, in addition to the basic static dependencies that refer to the static modules used to boot the application. Module contents may be optimally described in very different formats. It should be possible for configurations to choose to reuse the static module system format, or to utilize their own custom format, as best applies to the configuration. This allows each module configuration to make the choice that is best for that system. * Effect on tooling As I previously described, build tooling is not well-applied to run time module graphs. If the build tooling was instead optimized towards source artifacts, this would free it from being restricted to any one module system as has been previously proposed. The legacy (Java 8 and earlier) JDK build tool chain is actually very nearly already able to support modular building. There are two primary aspects that apply here: Annotation processing and compilation. The API for annotation processing from javax.tools is actually module-ready as it is: the JavaCompiler.CompilationTask can already accept any Iterable of annotation processor instances, which can easily be derived and loaded from installed, runnable static modules. No build system (that I know of) adequately takes advantage of this ability today though. Compilation for a modular world has very similar needs as today's world; Maven is a very good conceptual example of the closeness between the two, in the way that artifacts are transitively resolved and wired up. However, being dependent on the file manager, JavaCompiler is still predicated on a flat class path. While potentially adequate for many modular projects, the inherent lack of transitive isolation can cause a potentially destructive disparity between compilation and run time - in the best case, resulting in missing classes, but in the worst case, resulting in confusing linkage errors and even unexpected behavior. The technical crux of this issue appears to center around the process of location of dependency class data; the file manager only has the selection of locations from the StandardLocation gamut, which limits dependency classes as coming from (essentially) either the class path or the platform. To solve this problem, a more flexible location concept is necessary, which allows the compilation task to (a) know (in an abstract way) what artifact dependencies a given artifact has, (b) know what the package import and export rules are for each dependency specification, and (c) notify the file manager which of these is being examined for the purposes of locating dependency content. Adding this isolation to the compilation phase allows (among other things) more correct and consistent behavior, especially in regards to situations where more than one instance of the same package may occur in the extended dependency graph of the target module, and it also enforces more rigorous and accurate dependency specification (since the tendency for "lucky" transitive dependencies to appear on the compile path will be greatly reduced). Both of these problems have been observed in the real world. * Tying build dependencies and run time dependencies together As I have implied in earlier emails, there is no single easy answer to this problem. Run time and build time dependencies can not be identical in a system that has both long term viability and multiple possible distribution points. This is expressed in a wide variety of operating system and language platforms that have existed in past decades to the present. However these systems have also demonstrated a relatively small number of approaches that have proven to be viable solutions to this disparity while ensuring some degree of fidelity between the build, package, and run phases. I will describe the two that I think are most relevant and likely to apply well to our situation. The first solution is what I'll call "build on the host", which is employed by BSD ports, Gentoo ebuilds, and similar systems. In this system, the shared module repository consists of packages, which in turn contains buildable source code bundles (typically from an upstream source) and environment-specific patches. On installation, packages are downloaded from the repository and built (and possibly unit-tested) locally on the target system before being installed into the local environment. The second solution, binary installation, is employed by most Linux and other OS distributions. The repository packages are (more or less) images of what is to be installed on the local system. These packages are typically built on a per-distribution basis from an upstream source. Both of these solutions generally utilize a distribution-specific version which is usually derived from the upstream version, but with a local modifier which may reflect build number or the disposition of local patches in various ways. This emphasizes the reality that, for a given package, no single version scheme can flow all the way through from source to build to package to run. However, this reality doesn't preclude all forms of fidelity across phases. Maven central, for example, consists of many thousands of already well-versioned artifacts, with clear dependency information which is already often suitable for modular building. Modular packaging systems could easily reuse this build version information to produce sensible distribution- and run-time versioning data. Some systems could, in fact, distribute (license permitting) such artifacts directly; otherwise, the Maven ecosystem provides a fairly complete set of tooling (in the form of Sonatype's Nexus) for managing custom build artifacts, which is another piece of the puzzle. Maven dependency information, combined with introspection tools that are aware of the target distribution contents, could easily be used to set up a very good initial module descriptor in the format appropriate to that distribution's intended installation target. * Summary These concepts allow the JPMS requirements to be met with a far greater flexibility and capability than has previously been proposed. They enable the module system itself to support advanced features which are not necessarily directly exploited by the static module path module configuration, but which then can be consumed and exploited by various containers and other module-oriented systems. Examples include module redefinition and unloading, configurable strategies for native library loading, customized module content loaders, etc. They also allow a variety of useful module ecosystems to form, targeting many use cases. -- - DML From peter.kriens at aqute.biz Mon Sep 28 15:02:13 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 28 Sep 2015 17:02:13 +0200 Subject: Questions Message-ID: <4F941215-86EF-4B9E-A0E3-C43F6A7F5C56@aqute.biz> I?ve read the proposal with great interest. As stated before, I am very happy that packages are now really atomic. This will make any OSGi interoperation easier. Reading the proposal did raise a number of questions. Since they cover a rather wide area I wonder if I should send them as a single mail or per topic? Or should we create issues for them to make sure they are covered? Kind regards, Peter Kriens From mark.reinhold at oracle.com Mon Sep 28 16:12:43 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 28 Sep 2015 09:12:43 -0700 Subject: Questions In-Reply-To: <4F941215-86EF-4B9E-A0E3-C43F6A7F5C56@aqute.biz> References: <4F941215-86EF-4B9E-A0E3-C43F6A7F5C56@aqute.biz> Message-ID: <20150928091243.779716@eggemoggin.niobe.net> 2015/9/28 8:02 -0700, peter.kriens at aqute.biz: > ... > > Reading the proposal did raise a number of questions. Since they cover > a rather wide area I wonder if I should send them as a single mail or > per topic? Or should we create issues for them to make sure they are > covered? I suggest one e-mail thread per topic. JIRA issues are not a very good medium in which to carry on any kind of detailed discussion. - Mark From peter.kriens at aqute.biz Mon Sep 28 16:44:45 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 28 Sep 2015 18:44:45 +0200 Subject: Why can modules not be annotated? Message-ID: <9C2D961B-0072-4C72-9FA6-A63AF65BFC64@aqute.biz> In [lang-vm][1] I found: ?A module declaration may not be annotated.? I was surprised that modules (and their clauses) could not be annotated like package-info.java. Since the source syntax is not extensible, annotations could have been used to provide meta data for other module systems. This would mitigate some of the concerns of using a binary file to encode the module data. From peter.kriens at aqute.biz Mon Sep 28 16:49:53 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 28 Sep 2015 18:49:53 +0200 Subject: Why not use the Manifest? Message-ID: The encoding of the module data in a binary class file remains hard to understand and seems to go against the best practices we learned the hard way over the past decades. (Debugging!) Especially since the current proposal leaves the heavy lifting to build tooling. In the proposal, build tools cannot use the module-info.java since it does not contain sufficient information. (Versions being the most glaring omission for a build too.) Nor is the format extensible to contain build tool specific information. Ergo, the build tool will have to generate the module-info.class file. This make the class file makes these tools unnecessary hard and slower because it must either use the compiler as an intermediate step or use a library like ASM to create the unreadable version of its metadata. This is the first time in a very long time that I see a regression to binary files for meta data. Especially because there is already a good place that all build tools are already using: the manifest. From peter.kriens at aqute.biz Mon Sep 28 16:51:57 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 28 Sep 2015 18:51:57 +0200 Subject: Missing reification of the Module-path? Message-ID: <19F2AB39-D4B3-4D50-8C5A-0043BFDE25DB@aqute.biz> The proposal specifies how to resolve a module from a _module path_, a module path being one or more directories with modular JARs. Since the resolution is without versions, the module-path *must* contain a consistent set of artifacts. That is, the module path cannot be a repository like a maven repository it is closely bound to an executable since for each module only one of its versions can be chosen. For example in maven the module path would consist of the transitive runtime dependencies. This is a unique set per application per version. I do not think it is practical to synchronize this set with other applications so one should assume this is unique and cannot be shared. It is potentially very large. Ergo, the module-path *is* the application. I therefore wonder if there is a need for a reification of the module path + main class? Such an artifact could contain an indirection like a URL to not require creating temporary module path directories for each application with potentially very large duplicated artifacts. Such an artifact could then be made available as an atomic Layer. From peter.kriens at aqute.biz Mon Sep 28 16:54:04 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 28 Sep 2015 18:54:04 +0200 Subject: Do we need to open each file on the module path to find the enclosed name? Message-ID: <5FE37F97-112B-48E5-8784-8D01DB2FF3AB@aqute.biz> The module path is defined as: `A module path is a sequence of directories containing module artifacts which are searched, in order, for the first artifact that defines a suitable module.` This seems to indicate that the VM must iterate over all the directories (I assume not recursive?) and open the JARs to read the module-info.class file so that it knows the module name? This seems quite expensive for large systems? From peter.kriens at aqute.biz Mon Sep 28 16:56:57 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 28 Sep 2015 18:56:57 +0200 Subject: Do we actually need module names? Message-ID: This specification introduces a completely new namespace for modules. One of the lessons of the last decade has been that namespaces are very brittle and error prone. So a valid question seems to be: what do we get for this relatively high cost? The only reason to use a name space is for humans. However, in the current proposal the names will likely not be used by humans since they lack a version. That is, the module-info.class contains too little information to be used by tooling as a source for the compile or runtime paths. It therefore seems inevitable that tools will generate this resource from their native metadata. For example, in maven you define your dependencies in the pom and a maven plugin will then have to assemble a module path. Since the group-id, artifact-id, classifier and version do not always fit in the restricted module name they already must be prepared to mangle these names. And, yes I also expect that tools will hack the version in the module name; and every tool in its proprietary way. It is hard to envision any other way but tools generating the module-info.java. This is not a human editable file ? In the rest of the industry (git, docker) the direction is clearly in using naming based on content (e.g. SHA digests). The advantages of content addressing mechanisms over namespaces are enormous as many can testify. No duplicates, safe to cache, etc. Once you accept that tooling is a given, it seems therefore to make more sense to bind the artifacts into a module path by their digest then to use brittle naming schemes. A nice demonstration is the discussion about modules masquerading as friends. In this description digests were already used limit the scope of the export-to concept. However, this mechanism is hinted at but not part of the proposal. Obviously this leaves the dependency on the jre/jdk; this will be harder to specify as a digest. Well, one could look to .NET for inspiration :-) From peter.kriens at aqute.biz Mon Sep 28 16:57:40 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 28 Sep 2015 18:57:40 +0200 Subject: Has it been considered to apply the export modifier to the package-info.java? Message-ID: <5087F328-84D7-4CC5-89CB-58071B04561F@aqute.biz> The current proposal places the export in a module-info file. It would be much more convenient to place this as a modifier on the package-info.java. public package com.foo; Since it seems inevitable that there is a tooling step this would require little effort and cause no runtime overhead. From peter.kriens at aqute.biz Mon Sep 28 16:58:42 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 28 Sep 2015 18:58:42 +0200 Subject: How to do module refactoring without rippling changes? Message-ID: <974D1CEF-9E53-49F3-A918-07204E9E4458@aqute.biz> According to the requirements specification it should be possible to split a module into two modules and this should not require changing all modules that are dependent on this module. How will this work in the current proposal? From peter.kriens at aqute.biz Mon Sep 28 16:59:22 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 28 Sep 2015 18:59:22 +0200 Subject: How to see the Jigsaw additional classes/methods? Message-ID: <271B2311-1BF0-411F-A222-E13BAC89CA82@aqute.biz> The Jigsaw Javadoc seems to still show versions, main class, and other concepts that are not part of the proposal. How should I see these, will they be removed? From peter.kriens at aqute.biz Mon Sep 28 17:00:28 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 28 Sep 2015 19:00:28 +0200 Subject: Are multiple versions of the same package in the module graph allowed? Message-ID: It is clear that it is illegal for a module to be exposed to 2 or more modules that export the same package. I assume that such an application will not start and throw an exception? That said, there are many graphs were two or more packages are present in the graph, their modules are just never read by any module simultaneously. From reading the proposal I could not deduct if this is flagged as wrong or if this is actually allowed? From peter.kriens at aqute.biz Mon Sep 28 17:01:35 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 28 Sep 2015 19:01:35 +0200 Subject: What is the module scope of Service Loader? Message-ID: <9A6F4558-C48E-42AF-A06D-12C8ED1EF267@aqute.biz> It is not clear to me from the description how Service Loader finds implementation classes. For example DriverManager.getDrivers() returns a list of all _installed_ drivers. It is not clear to me from which modules these drivers are loaded. * Is this system wide? * Just the modules the caller can read? * The modules the java.sql module can read? From peter.kriens at aqute.biz Mon Sep 28 17:02:19 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 28 Sep 2015 19:02:19 +0200 Subject: You can find your own resources, yes? Message-ID: <6844F705-C95B-4DCF-AEB5-4C62A880EBD8@aqute.biz> The javadoc for getResource* says: ?This method does not find resources in named modules?. Literally this implies that a class in a named module cannot read its own resources. I assume that this is an oversight in the description? From peter.kriens at aqute.biz Mon Sep 28 17:03:44 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 28 Sep 2015 19:03:44 +0200 Subject: How is JPA supposed to work? Message-ID: <4D9F195B-48D6-42E8-B3BD-9270FB73E356@aqute.biz> In Java Persistence Architecture (JPA) the persistence.xml resource provides information about the persistence unit. This resource must be read by the Persistence Provider which then does a lot of magic. However, if nobody can read resources from a named module then how can JPA perform its duties? From peter.kriens at aqute.biz Mon Sep 28 17:04:09 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 28 Sep 2015 19:04:09 +0200 Subject: Can you dynamically load classes that are not exported? Message-ID: <15B6EE44-D6D9-4ABA-BBDC-BBD57DAC8979@aqute.biz> Since resource loading is very restrictive I would expect that class loading is also not possible unless a class is exported. Is this true? From peter.kriens at aqute.biz Mon Sep 28 17:05:02 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 28 Sep 2015 19:05:02 +0200 Subject: How will dependency Injection/Annotation scanning work? Message-ID: <71174D99-BB94-4FC8-BA56-9BCED053E7CF@aqute.biz> Dependency injection subsystems (Guice,CDI, Spring) tend to require annotation scanning. This should be able to iterate over implementation classes that are not exported. I could not find how you could iterate over the classes in a module. If there is no such way, how will CDI work with modules? From david.lloyd at redhat.com Mon Sep 28 17:33:38 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 28 Sep 2015 12:33:38 -0500 Subject: How will dependency Injection/Annotation scanning work? In-Reply-To: <71174D99-BB94-4FC8-BA56-9BCED053E7CF@aqute.biz> References: <71174D99-BB94-4FC8-BA56-9BCED053E7CF@aqute.biz> Message-ID: <560979F2.8010307@redhat.com> On 09/28/2015 12:05 PM, Peter Kriens wrote: > Dependency injection subsystems (Guice,CDI, Spring) tend to require annotation scanning. This should be able to iterate over implementation classes that are not exported. > > I could not find how you could iterate over the classes in a module. If there is no such way, how will CDI work with modules? This one I can answer, at least partially: modules are likely to be indexed or contain an index of annotated classes. The index is presumably keyed by any annotation annotated in turn with the @Indexed annotation (package TBD I believe). The following excerpt from the requirements doc (heading "Development") is relevant: > One potential approach is to augment a module?s definition with an index of the annotations that are present in the module, together with an indication of the elements to which each annotation applies. To limit the size of the index, only annotations which themselves are annotated with a new meta-annotation, say @Indexed, would be included. -- - DML From peter.kriens at aqute.biz Wed Sep 30 07:33:36 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Wed, 30 Sep 2015 09:33:36 +0200 Subject: How will dependency Injection/Annotation scanning work? In-Reply-To: <560979F2.8010307@redhat.com> References: <71174D99-BB94-4FC8-BA56-9BCED053E7CF@aqute.biz> <560979F2.8010307@redhat.com> Message-ID: <6D9ACFC7-45C3-46E8-B958-FBCBB4C25109@aqute.biz> Thanks for the answer David about the annotation scanning. I guess this still will have to undergo a more detailed design? Kind regards, Peter Kriens > On 28 sep. 2015, at 19:33, David M. Lloyd wrote: > > On 09/28/2015 12:05 PM, Peter Kriens wrote: >> Dependency injection subsystems (Guice,CDI, Spring) tend to require annotation scanning. This should be able to iterate over implementation classes that are not exported. >> >> I could not find how you could iterate over the classes in a module. If there is no such way, how will CDI work with modules? > > This one I can answer, at least partially: modules are likely to be indexed or contain an index of annotated classes. The index is presumably keyed by any annotation annotated in turn with the @Indexed annotation (package TBD I believe). The following excerpt from the requirements doc (heading "Development") is relevant: > >> One potential approach is to augment a module?s definition with an index of the annotations that are present in the module, together with an indication of the elements to which each annotation applies. To limit the size of the index, only annotations which themselves are annotated with a new meta-annotation, say @Indexed, would be included. > > > -- > - DML