From mark.reinhold at oracle.com Tue Dec 1 17:08:02 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 01 Dec 2015 09:08:02 -0800 Subject: Proposed schedule change for JSR 376 Message-ID: <20151201090802.335771@eggemoggin.niobe.net> The original schedule for this JSR [1] envisioned an Early Draft Review in June 2015, Public Review in October 2015, and a Final Release in March 2016. Given that we have many open discussion threads (which I hope to catch up on soon) and have yet to publish an EDR, I hereby propose a new schedule: Early Draft Review January 2016 Public Review June 2016 Proposed Final Draft December 2016 Final Release March 2017 Please let me know by this time next week (18:00 UTC next Tuesday, 8 December) of any objections to this proposal. (Wearing a different hat, I will propose a similar revision to the JDK 9 schedule in a moment.) - Mark [1] https://www.jcp.org/en/jsr/detail?id=376 From forax at univ-mlv.fr Tue Dec 1 18:51:37 2015 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Tue, 01 Dec 2015 18:51:37 +0000 Subject: Proposed schedule change for JSR 376 In-Reply-To: <20151201090802.335771@eggemoggin.niobe.net> References: <20151201090802.335771@eggemoggin.niobe.net> Message-ID: Fine for me. R?mi Le 1 d?cembre 2015 18:08:02 CET, mark.reinhold at oracle.com a ?crit : >The original schedule for this JSR [1] envisioned an Early Draft Review >in June 2015, Public Review in October 2015, and a Final Release in >March >2016. > >Given that we have many open discussion threads (which I hope to catch >up >on soon) and have yet to publish an EDR, I hereby propose a new >schedule: > > Early Draft Review January 2016 > Public Review June 2016 > Proposed Final Draft December 2016 > Final Release March 2017 > >Please let me know by this time next week (18:00 UTC next Tuesday, 8 >December) of any objections to this proposal. (Wearing a different >hat, >I will propose a similar revision to the JDK 9 schedule in a moment.) > >- Mark > > >[1] https://www.jcp.org/en/jsr/detail?id=376 From david.lloyd at redhat.com Tue Dec 1 21:48:04 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 1 Dec 2015 15:48:04 -0600 Subject: Coupling modules and class loaders In-Reply-To: <745552845.1758274.1445496806235.JavaMail.zimbra@u-pem.fr> References: <56276E91.5060605@redhat.com> <2086237069.1662074.1445447515158.JavaMail.zimbra@u-pem.fr> <5627DC93.2050202@redhat.com> <745552845.1758274.1445496806235.JavaMail.zimbra@u-pem.fr> Message-ID: <565E1594.9050405@redhat.com> I realize now that I didn't notice this reply - my apologies. Responses inline. On 10/22/2015 01:53 AM, forax at univ-mlv.fr wrote: > ----- Mail original ----- >> De: "David M. Lloyd" >> ?: "Remi Forax" >> Cc: jpms-spec-experts at openjdk.java.net >> Envoy?: Mercredi 21 Octobre 2015 20:42:27 >> Objet: Re: Coupling modules and class loaders >> >> On 10/21/2015 12:11 PM, Remi Forax wrote: >>> Let's try to see the big picture, why people uses classloaders, either they >>> want to be able to load/onload an application, have unconventional way to >>> store bytecodes, do bytecode transformations, etc or they want to isolate >>> several parts of the application because those parts use incompatible >>> versions of the same dependency. >> >> I don't think that the reasons users use class loaders really has any >> relevance here; I'm looking at the raw capabilities of class loaders and >> how they fit in to the requirements we have, and as it happens, they fit >> very well. >> >>> For the first reasons, this way to use classloaders is very dependent on >>> the application and these applications tend to be very sensitive to the >>> way class loaders are connected. People that design frameworks have come >>> with very creative way of connecting classloaders and trying to force >>> people that want modules to change the way classloaders work in their >>> application will not work. >> >> Only rare and highly specialized frameworks tend to define their own >> class loaders. It has been proven repeatedly over the past several >> years (by us and by others) that most popular existing artifacts and >> projects already deal with class loaders in a manner which is completely >> compatible with class loader based modules. So I'm not quite sure what >> you're driving at - do you have an example of something that would not >> work with class loader based modules? > > JBoss Module, OSGI implementations, any applications that uses classloader to implement a kind of module-like isolation. All of these applications will not work if we choose a classloader based module implementation because each of them connect classloders in an incompatible way compared to the others. This is actually untrue. The JBoss OSGi implementation was successfully built on top of, and therefore compatible with, JBoss Modules, as is our Java EE class loading implementation. The entire justification of class loader based modules is that they work with everything and with it, you can make everything work together. >>> For the last reason, trying to solve problem of incompatible dependencies. >>> I think it's a bad idea to try to fix a linking time issue with a runtime >>> solution. The way to solve that issue at linktime is to link statically >>> the module with one version of the dependency. This is done in Java by >>> changing the bytecodes of the module and its dependency in order to rename >>> the package and change the code that use this package. If we have an >>> intelligent linker, or at least a pluggable one, this can be done at link >>> time so there is no reason to use classloader for that at runtime. >> >> Are you arguing that there are no cases where you want to run with a >> different version of a dependency than the one you built against? I >> would refer you to just about any real world software deployment for an >> unlimited number of counter-examples. > > no, there are 3 'times'. > The build time, the link time (think jlink but perhaps it should be called the installation time and jlink renamed to jinstall) and the runtime. > You can run with a different version of a dependency than at build time but the modules at installation time should be the same as the modules at runtime. > So at installation time, you can check if you have several incompatible versions of the same module and transform the code to link them (or some of them) statically. I don't really see how this relates to the run time implementation though. Whether you use class loaders or some new construct within class loaders, the link time behavior could be the same. >>> I think that classloaders are the wrong mechanism for implementing modules >>> because as jigsaw current prototype prove, you don't need them to provide >>> stronger encapsulation and as it will slow down the adoption of modules >>> because people will have to rewrite there fancy frameworks to be "module >>> compatible". >> >> Actually I believe your logic is exactly backwards. We (Red Hat) have >> already been able to prove that very many existing frameworks can use >> our existing class loader based module solution without *any* >> modification, *because* they already use class loaders in a reasonable >> way to find classes and resources. On the other hand, the current >> Jigsaw prototype inevitably implies that module adoption *will* be >> slowed down because in order to become module aware, you have to add >> extra code to recognize Modules and all that goes with them, above and >> beyond all the existing (and perfectly adequate and functional) code >> which uses class loaders for that purpose already today. > > If we forget one moment the issue about setAccessible of jigsaw, for most of the applications, it just works. > For applications that already have a notion of modules at runtime, yes, the code need to be touched but as you said there is few frameworks like this so they can be updated by hand. > Maybe JBoss Module requires less code/no code changes to support other applications but here we want to support is applications like JBoss Module. If we used class loader based modules, we wouldn't even need JBoss Modules. The JDK itself would be able to support OSGi, Java EE, and applications all in one interconnected space (even legacy extensions, a class path module, and command-line JARs with Class-Path references), just as the JBoss application servers are capable of doing today, and I think the majority of existing software (at least, that which you'd find on, say, Maven Central) would continue to work largely unchanged. I've opened a discussion about accessibility in another thread. -- - DML From Tim_Ellison at uk.ibm.com Tue Dec 1 22:29:21 2015 From: Tim_Ellison at uk.ibm.com (Tim Ellison) Date: Tue, 1 Dec 2015 22:29:21 +0000 Subject: Proposed schedule change for JSR 376 In-Reply-To: <20151201090802.335771@eggemoggin.niobe.net> References: <20151201090802.335771@eggemoggin.niobe.net> Message-ID: <201512012229.tB1MTSuC021724@d06av07.portsmouth.uk.ibm.com> In the original schedule, the Proposed Final Draft date for this JSR and the Feature Complete date for Java 9 is December 2015. The new proposed schedule has the Public Review (June 2016) one month /after/ the Java 9 proposed Feature Complete date (May 2016) [1]. If Java 9 is Feature Complete before the JSR is ready for Public Review, and well before the Proposed Final Draft, then it seems there will be little chance for the reviews to correct any significant problems in the JSR. [1] http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-December/003149.html Regards, Tim mark.reinhold at oracle.com wrote on 01/12/2015 17:08:02: > The original schedule for this JSR [1] envisioned an Early Draft Review > in June 2015, Public Review in October 2015, and a Final Release in March > 2016. > > Given that we have many open discussion threads (which I hope to catch up > on soon) and have yet to publish an EDR, I hereby propose a new schedule: > > Early Draft Review January 2016 > Public Review June 2016 > Proposed Final Draft December 2016 > Final Release March 2017 > > Please let me know by this time next week (18:00 UTC next Tuesday, 8 > December) of any objections to this proposal. (Wearing a different hat, > I will propose a similar revision to the JDK 9 schedule in a moment.) > > - Mark > > > [1] https://www.jcp.org/en/jsr/detail?id=376 > Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU From mark.reinhold at oracle.com Tue Dec 1 22:39:41 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 01 Dec 2015 14:39:41 -0800 Subject: Proposed schedule change for JSR 376 In-Reply-To: <201512012229.tB1MTRFH012038@d06av03.portsmouth.uk.ibm.com> References: <20151201090802.335771@eggemoggin.niobe.net>, <201512012229.tB1MTRFH012038@d06av03.portsmouth.uk.ibm.com> Message-ID: <20151201143941.329916@eggemoggin.niobe.net> 2015/12/1 2:29 -0800, tim_ellison at uk.ibm.com: > In the original schedule, the Proposed Final Draft date for this JSR and > the Feature Complete date for Java 9 is December 2015. The new proposed > schedule has the Public Review (June 2016) one month /after/ the Java 9 > proposed Feature Complete date (May 2016) [1]. > > If Java 9 is Feature Complete before the JSR is ready for Public Review, > and well before the Proposed Final Draft, then it seems there will be > little chance for the reviews to correct any significant problems in the > JSR. "Feature Complete" does not mean "frozen", it just means "complete". Here's the definition [1]: All features have been implemented and integrated into the master forest, together with unit tests. There are typically several months between FC and the first "rampdown" phase. - Mark [1] http://openjdk.java.net/projects/jdk8/milestones#Feature_Complete From Tim_Ellison at uk.ibm.com Wed Dec 2 12:18:51 2015 From: Tim_Ellison at uk.ibm.com (Tim Ellison) Date: Wed, 2 Dec 2015 12:18:51 +0000 Subject: Proposed schedule change for JSR 376 In-Reply-To: <20151201143941.329916@eggemoggin.niobe.net> References: <20151201090802.335771@eggemoggin.niobe.net>, <201512012229.tB1MTRFH012038@d06av03.portsmouth.uk.ibm.com> <20151201143941.329916@eggemoggin.niobe.net> Message-ID: <201512021219.tB2CIxxP015902@d06av03.portsmouth.uk.ibm.com> mark.reinhold at oracle.com wrote on 01/12/2015 22:39:41: > 2015/12/1 2:29 -0800, tim_ellison at uk.ibm.com: > > In the original schedule, the Proposed Final Draft date for this JSR and > > the Feature Complete date for Java 9 is December 2015. The new proposed > > schedule has the Public Review (June 2016) one month /after/ the Java 9 > > proposed Feature Complete date (May 2016) [1]. > > > > If Java 9 is Feature Complete before the JSR is ready for Public Review, > > and well before the Proposed Final Draft, then it seems there will be > > little chance for the reviews to correct any significant problems in the > > JSR. > > "Feature Complete" does not mean "frozen", it just means "complete". > Here's the definition [1]: > > All features have been implemented and integrated into the master > forest, together with unit tests. > > There are typically several months between FC and the first "rampdown" > phase. > > - Mark > > > [1] http://openjdk.java.net/projects/jdk8/milestones#Feature_Complete So you would not rule out design changes after the modularity feature has been implemented and integrated into the master forest? To be clear, I'm not suggesting we delay progress on the JSR, but it would be good to declare the expected timescales for completing the modularity design and getting the JSR materials out for public review well before the code is feature complete in Java 9 and any subsequent modifications or changes to the approach may be disruptive. Regards, Tim From peter.kriens at aqute.biz Thu Dec 3 15:27:53 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Thu, 3 Dec 2015 16:27:53 +0100 Subject: Proposed schedule change for JSR 376 In-Reply-To: <20151201143941.329916@eggemoggin.niobe.net> References: <20151201090802.335771@eggemoggin.niobe.net> <201512012229.tB1MTRFH012038@d06av03.portsmouth.uk.ibm.com> <20151201143941.329916@eggemoggin.niobe.net> Message-ID: Mark, I am not sure I understand why we need to delay the JSR 376? I think the delay is a great opportunity to deliver the JSR before, or at least aligned with the implementation so the EG can provide its input? Kind regards, Peter Kriens > On 1 dec. 2015, at 23:39, mark.reinhold at oracle.com wrote: > > 2015/12/1 2:29 -0800, tim_ellison at uk.ibm.com: >> In the original schedule, the Proposed Final Draft date for this JSR and >> the Feature Complete date for Java 9 is December 2015. The new proposed >> schedule has the Public Review (June 2016) one month /after/ the Java 9 >> proposed Feature Complete date (May 2016) [1]. >> >> If Java 9 is Feature Complete before the JSR is ready for Public Review, >> and well before the Proposed Final Draft, then it seems there will be >> little chance for the reviews to correct any significant problems in the >> JSR. > > "Feature Complete" does not mean "frozen", it just means "complete". > Here's the definition [1]: > > All features have been implemented and integrated into the master > forest, together with unit tests. > > There are typically several months between FC and the first "rampdown" > phase. > > - Mark > > > [1] http://openjdk.java.net/projects/jdk8/milestones#Feature_Complete From mark.reinhold at oracle.com Fri Dec 4 01:04:14 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 03 Dec 2015 17:04:14 -0800 Subject: Proposed schedule change for JSR 376 In-Reply-To: <201512021219.tB2CIxVi020359@d06av12.portsmouth.uk.ibm.com> References: <20151201090802.335771@eggemoggin.niobe.net>, <20151201143941.329916@eggemoggin.niobe.net>, <201512021219.tB2CIxVi020359@d06av12.portsmouth.uk.ibm.com> Message-ID: <20151203170414.170256@eggemoggin.niobe.net> 2015/12/2 4:18 -0800, tim_ellison at uk.ibm.com: > mark.reinhold at oracle.com wrote on 01/12/2015 22:39:41: >> "Feature Complete" does not mean "frozen", it just means "complete". >> Here's the definition [1]: >> >> All features have been implemented and integrated into the master >> forest, together with unit tests. >> >> There are typically several months between FC and the first "rampdown" >> phase. > > So you would not rule out design changes after the modularity feature > has been implemented and integrated into the master forest? No, though obviously it would be best for the RI team not to merge anything until this EG is reasonably confident that massive design changes will not be required. > To be clear, I'm not suggesting we delay progress on the JSR, but it > would be good to declare the expected timescales for completing the > modularity design and getting the JSR materials out for public review > well before the code is feature complete in Java 9 and any subsequent > modifications or changes to the approach may be disruptive. The JCP defines two mandatory milestones, Early Draft Review and Public Review. I think the most sensible approach is to publish the EDR once this EG has reached rough consensus on a design, hopefully in the next few months, and then publish the PR more or less in sync with the RI's Feature Complete milestone. - Mark From mark.reinhold at oracle.com Fri Dec 4 01:04:22 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 03 Dec 2015 17:04:22 -0800 Subject: Proposed schedule change for JSR 376 In-Reply-To: References: <20151201090802.335771@eggemoggin.niobe.net>, <201512012229.tB1MTRFH012038@d06av03.portsmouth.uk.ibm.com>, <20151201143941.329916@eggemoggin.niobe.net>, Message-ID: <20151203170422.488184@eggemoggin.niobe.net> 2015/12/3 7:27 -0800, peter.kriens at aqute.biz: > I am not sure I understand why we need to delay the JSR 376? I think > the delay is a great opportunity to deliver the JSR before, or at > least aligned with the implementation so the EG can provide its input? I expect the EG to provide input on a continuous basis, ahead of each JCP milestone, not afterward. At least that's how the process is intended to work. In terms of delivery, you can't deliver a JSR without a Reference Implementation, so delivering the JSR earlier than the implementation isn't really an option. - Mark From forax at univ-mlv.fr Fri Dec 4 09:10:07 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Fri, 4 Dec 2015 10:10:07 +0100 (CET) Subject: Non-exported vs restricted packages In-Reply-To: <494377733.2029520.1449219727280.JavaMail.zimbra@u-pem.fr> Message-ID: <1071190767.2041397.1449220207416.JavaMail.zimbra@u-pem.fr> Hi All, I think there is an issue with the current design of jigsaw that we should tackle. The problem have been raised several times of the jigsaw-dev mailing list by different people and I think it's a valid point, currently the way encapsulation is specified doesn't match with the way people use/want to use modules in an application server. Don't get me wrong, I don't not want to have stronger encapsulation that we currently have, i just think we should recognize that the way people currently use modules is a valid use case. So the problem is that most application servers use reflection to create instances of implementation of an interface and people want the implementation on that interface to be in a non-exported package. With Jigsaw only option if you want this configuration is to use a ServiceLoader to load the implementation. But using a ServiceLoader requires coordination between the code that declares and uses the interface and code that implements that interface. Which mean that people will be able to use modules in an application server when the application server itself will use ServiceLoader, so not before JavaEE 9. I think we should provide a way for people to use modules in a JavaEE 7/8 world. The way to do that is to recognize that we should have a way to specify that we want packages that have implementations that are not accessible directly but that are accessible using reflection (with the right handsake). It doesn't mean we don't want stronger encapsulation for package like com.sum.foo, it means we also want to have a way to specify that a package can be non-exported but its implementations can be available by reflection if and only if setAccessbile is used. This will ease the transition by allowing people to modulify there current code without waiting application servers to implement the JavaEE 9 spec because application server vendors can tweak the implementation of their dependency injection mechanism to use setAccessible when necessary*. So I propose that the spec should specify three kind of packages "availability", exported packages that make classes available by module that require the module containing the packages, non-exported packages that make classes non available but that can be used by reflection using setAccessible as an escape hatch and restricted packages that make the classes non-available even by using setAccessible. Note that the question of which one should be the default is another question that can be decided after. R?mi * we also need setAccessible to add a read edge automatically if the package not restricted. From david.lloyd at redhat.com Fri Dec 4 12:53:52 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Fri, 4 Dec 2015 06:53:52 -0600 Subject: Non-exported vs restricted packages In-Reply-To: <1071190767.2041397.1449220207416.JavaMail.zimbra@u-pem.fr> References: <1071190767.2041397.1449220207416.JavaMail.zimbra@u-pem.fr> Message-ID: <56618CE0.3010506@redhat.com> On 12/04/2015 03:10 AM, Remi Forax wrote: > Hi All, > I think there is an issue with the current design of jigsaw that we should tackle. > > The problem have been raised several times of the jigsaw-dev mailing list by different people and I think it's a valid point, currently the way encapsulation is specified doesn't match with the way people use/want to use modules in an application server. Don't get me wrong, I don't not want to have stronger encapsulation that we currently have, i just think we should recognize that the way people currently use modules is a valid use case. > > So the problem is that most application servers use reflection to create instances of implementation of an interface and people want the implementation on that interface to be in a non-exported package. With Jigsaw only option if you want this configuration is to use a ServiceLoader to load the implementation. To me the problem isn't the encapsulation itself - it's the manifold issue that (A) you can't realistically opt out of it, especially only for certain types, (B) it introduces a second mechanism beyond traditional accessibility to control accessibility for no good reason, (C) even if you could somehow opt out of it in a Jigsaw-friendly way (annotations or package lists or something), the default is still to break everything that makes the perfectly reasonable assumption that public means public, (D) these things add up to likely instant incompatibility for hundreds, possibly thousands, of existing public projects and artifacts (let alone private things), etc. > But using a ServiceLoader requires coordination between the code that declares and uses the interface and code that implements that interface. Which mean that people will be able to use modules in an application server when the application server itself will use ServiceLoader, so not before JavaEE 9. > > I think we should provide a way for people to use modules in a JavaEE 7/8 world. > > The way to do that is to recognize that we should have a way to specify that we want packages that have implementations that are not accessible directly but that are accessible using reflection (with the right handsake). It doesn't mean we don't want stronger encapsulation for package like com.sum.foo, it means we also want to have a way to specify that a package can be non-exported but its implementations can be available by reflection if and only if setAccessbile is used. But are packages really the right unit of granularity for this? I think it is reasonable to want to have, in one package, classes that are public and classes that are private to a module. In the wild, I believe that existing usage of package-private access level shows clearly that this would be a common pattern, if it were even possible to do. > This will ease the transition by allowing people to modulify there current code without waiting application servers to implement the JavaEE 9 spec because application server vendors can tweak the implementation of their dependency injection mechanism to use setAccessible when necessary*. I definitely agree that we need a way for current code to be seamlessly modularized. > So I propose that the spec should specify three kind of packages "availability", > exported packages that make classes available by module that require the module containing the packages, non-exported packages that make classes non available but that can be used by reflection using setAccessible as an escape hatch and restricted packages that make the classes non-available even by using setAccessible. This is getting a bit complex, no? > Note that the question of which one should be the default is another question that can be decided after. > > R?mi > > * we also need setAccessible to add a read edge automatically if the package not restricted. > -- - DML From mark.reinhold at oracle.com Thu Dec 10 01:21:45 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 9 Dec 2015 17:21:45 -0800 (PST) Subject: Resource encapsulation In-Reply-To: References: <20151005185923.5984A7B97B@eggemoggin.niobe.net>, Message-ID: <20151210012145.DC38B849C9@eggemoggin.niobe.net> 2015/10/12 8:01 -0700, peter.kriens at aqute.biz: > I think I understand how a module aware JPA can use the Module API to > load resources and classes. This sounds similar to how OSGi works. > > For the service loader scenario I am not clear how you would then > discover and load the persistence.xml resources. Would you then not > still be forced to go through the Module API anyway to read resources > since the javax.persistence.Configuration implementation would not > allow the JPA provider to read the persistence.xml? Isn???t using the > service loader sounds like an unnecessary indirection? The javax.persistence.Configuration service interface would expose whatever methods are needed to provide the required configuration information. It could declare a method that returns an InputStream intended to be the content of a raw persistence.xml file, or returns an XML DOM structure, or returns an instance of some related class which represents, in a statically-typed way, the content of persistence.xml. In the long run persistence.xml could even be abandoned in favor of actual code that initializes the Configuration object explicitly, if so desired. There are many options, but none requires external access to a client module's internal persistence.xml resource, if any. - Mark From mark.reinhold at oracle.com Thu Dec 10 01:22:45 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 9 Dec 2015 17:22:45 -0800 (PST) Subject: Has it been considered to apply the export modifier to the package-info.java? In-Reply-To: References: <5087F328-84D7-4CC5-89CB-58071B04561F@aqute.biz>, <20151005185423.4FD207B95D@eggemoggin.niobe.net>, Message-ID: <20151210012245.E6255849CE@eggemoggin.niobe.net> 2015/10/12 8:30 -0700, peter.kriens at aqute.biz: >> Why would this be more convenient? > > For the same reason it is more convenient to put the public/private > modifier in the class source file instead of the package-info file? > > The subsidiarity principle says that you want to apply this > information at the lowest reasonable level. People that then work in > that package are directly aware of this information. The close this > information is to the source, the better the chance it gets > maintained. It also makes refactoring easier since it does not require > a global overview. And it does not require typing the package name > since this is already implied. I understand the principle, but in this case I think it's trumped by the goal of readability. In practice package-info.java files aren't good for very much, so people tend to ignore them. Putting all of a module's export clauses in one obvious place, morever, makes it easy to tell, at a glance, what it exports for external use. > When I meant zero overhead I included my conclusion that > module-info.java cannot be a human editable file. In any practical > situation it can only be generated from external (build) > information. The module-info.java file just does not contain > sufficient information (versions!) to drive a build so there MUST be > another artifact that contains a superset of the module-info > aspects. To prevent redundancy, the module-info should thus be > generated. This required build step could then easily include the > exported packages. So the only overhead would be in the build. Yes, this information could be gathered at build time and inserted into module-info.java or .class (or whatever) files, so I agree that run-time overhead need not be an issue. - Mark From mark.reinhold at oracle.com Thu Dec 10 01:23:45 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 9 Dec 2015 17:23:45 -0800 (PST) Subject: Do we actually need module names? In-Reply-To: References: , <20151005185323.4DD797B958@eggemoggin.niobe.net>, Message-ID: <20151210012345.E832D849D3@eggemoggin.niobe.net> 2015/10/12 9:48 -0700, peter.kriens at aqute.biz: > If a module could require package names instead of module names it > would not be necessary to introduce a new orthogonal namespace to the > code base. > > If we take an application module path then this will represent a > limited (max a few thousand, on average much less) set of modules. The > current design already implies that in this set modules their exported > packages must be unique, No, there's nothing to prevent artifacts on a module path from defining modules with conflicting packages. What's forbidden is to have two modules with conflicting exported packages related by readability in the module graph, after dependences are resolved. > so there is no need to resolve multiple > versions. It is therefore very almost no overhead to find a module by > its exported packages instead of a module name. > > Since the Java bytecode already contains the reference to the exported > packages it is straightforward to translate these references to a > require package in the module definition. This will actually be much > less work than developers maintaining a module namespace. Developers already maintain what amounts to a module namespace, in pom.xml or its equivalents, they just call it something else. Or, as Bob Martin might say, the unit of reuse is the unit of release [1], so to assemble a large system from such units it's most natural to refer to the units themselves rather than to things inside them. - Mark [1] http://www.butunclebob.com/ArticleS.UncleBob.PrinciplesOfOod From mark.reinhold at oracle.com Thu Dec 10 01:24:45 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 9 Dec 2015 17:24:45 -0800 (PST) Subject: Why not use the Manifest? In-Reply-To: <38A65E32-EECE-44DF-B646-922CF9637C7D@aqute.biz> References: , <20151005185023.443E87B949@eggemoggin.niobe.net>, <81AC5311-7835-4451-8381-CA446A311386@aqute.biz>, <1993489049.1580799.1444830069389.JavaMail.zimbra@u-pem.fr>, <8C724E11-0656-4483-85BD-DFF310F8B463@aqute.biz>, <673841498.1783560.1444850891146.JavaMail.zimbra@u-pem.fr>, <38A65E32-EECE-44DF-B646-922CF9637C7D@aqute.biz> Message-ID: <20151210012445.EA284849D8@eggemoggin.niobe.net> 2015/10/14 11:53 -0700, peter.kriens at aqute.biz: > I think our differences come from the fact that you assume that > module-info.java will be written by developers while I fail how to see > that can work realistically. > > The build system must already have metadata that contains module name > and version (both for the current project and the dependencies), for > example the pom.xml in maven. You think it is realistic to expect > people will write the module name in two places? > > For all practical cases, will the module info not be generated by the > build system? I think that depends upon the build system. With the build systems that we have today it may well be best for the system to generate module-info.java files. That's perfectly fine. (Gradle is, I understand, already doing this.) Some future build system might do something entirely different and use hand-written module-info.java files. That's fine too. The design as proposed allows for both approaches, and it does so by fitting module declarations into the existing source-file to class-file compilation pipeline which all build tools, present and future, must support. - Mark From mark.reinhold at oracle.com Thu Dec 10 01:25:45 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 9 Dec 2015 17:25:45 -0800 (PST) Subject: Why can modules not be annotated? In-Reply-To: <0AC36D44-0539-41DD-9505-76F789D26EE2@aqute.biz> References: <9C2D961B-0072-4C72-9FA6-A63AF65BFC64@aqute.biz>, <20151005184923.394607B943@eggemoggin.niobe.net>, <0AC36D44-0539-41DD-9505-76F789D26EE2@aqute.biz> Message-ID: <20151210012545.EBF49849DD@eggemoggin.niobe.net> 2015/10/12 9:56 -0700, peter.kriens at aqute.biz: > I understand the trade offs. > > That said, I am a bit curious how we are going to handle the > discussing these trade offs in the design? I had actually expected > that more things were set in stone but in this discussion it seems > there are still some open areas. Are we going to have phone > conferences, a face to face? As stated in the original JSR, this EG will work exclusively via e-mail, on this list. I will soon start gathering a list of open issues for us to discuss and work through before publishing the EDR. > Anyway, thanks for all the responses. It did clarify things for me. Glad to hear it. - Mark From mark.reinhold at oracle.com Thu Dec 10 01:26:45 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 9 Dec 2015 17:26:45 -0800 (PST) Subject: Module descriptions versus module descriptors In-Reply-To: <5626CDF7.5060605@redhat.com> References: <5626CDF7.5060605@redhat.com> Message-ID: <20151210012645.EDDB1849E4@eggemoggin.niobe.net> 2015/10/20 4:27 -0700, david.lloyd at redhat.com: > I see no logical path that leads from the requirements as specified > exclusively to the assumption that the descriptor must be bytecode, let > alone part of the JVM and/or language specification. All the reasons > given appear to be self-justifying or based on abstract assumptions, > e.g. "modules are... a new kind of Java program component... therefore > [Jigsaw] treats them as such". I agree that there are other ways to represent module descriptions. I've never stated otherwise. If module boundaries, however they're expressed, are to be enforced by the compiler and the VM -- as you've agreed elsewhere -- then their manner of expression is inevitably going to be a topic for the Java language and VM specifications. This is not avoidable. > ... it seems to me that a significant part of the Jigsaw > design justification for its handling of module metadata hinges around > the conflation of the description of a module, and the descriptor used > by the static module loading implementation. This raises a red flag for > me because it fundamentally locks the capabilities of module > descriptions to whatever makes sense to express in the descriptor, and > then in turn constrains these things to the language and JVM specification. > > In our (JBoss) module system, these concepts are decoupled: a filesystem > module's descriptor is read and parsed into a description which is then > consumed by the module system to create the module. ... If module boundaries are to be enforced by the compiler and the VM then, in such a decoupled system, where and in what form does the compiler locate the information that describes the module being compiled, and any other modules required in order to compile that module? (See also my previous reply on this topic [1].) - Mark [1] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-October/000151.html From mark.reinhold at oracle.com Thu Dec 10 01:27:45 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 9 Dec 2015 17:27:45 -0800 (PST) Subject: Coupling modules and class loaders In-Reply-To: <56276E91.5060605@redhat.com> References: <56276E91.5060605@redhat.com> Message-ID: <20151210012745.EFD0F849EA@eggemoggin.niobe.net> 2015/10/21 3:53 -0700, david.lloyd at redhat.com: > ... > > I propose that modules are best defined as a specialization of class > loaders. Class loaders already have many of the properties that define > a module: encapsulation, isolation, unique identity for packages. In > fact OSGi already uses this unit. The only thing they are really > missing is the ability to use a graph-like dependency structure, hence > the specialization. Class loaders do not provide strong encapsulation. How do you propose to extend them to do so? > ... > > What is the justification for the more complex Jigsaw approach? One developer's complexity is another's simplicity. The primary motivation is, as R??mi noted, to make it easy for existing applications that use class loaders in sophisticated (and sometimes very strange) ways to adopt modules. If the maintainers of such applications must make deep changes to their class-loading architectures in order to support modules then some of them simply won't bother. The fact that OSGi and JBoss have successfully used class loaders to implement module systems that are capable of loading a wide variety of existing artifacts is interesting. It does not, however, imply that all existing applications can be converted to use the class-loader-per-module approach, nor that they should be forced to do so. The proposed design therefore neither mandates nor forbids that approach. (See also my previous reply on this topic [1].) - Mark [1] http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2015-March/000068.html From mark.reinhold at oracle.com Thu Dec 10 01:28:45 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 9 Dec 2015 17:28:45 -0800 (PST) Subject: Modules and build time In-Reply-To: <5627751C.9020702@redhat.com> References: <5627751C.9020702@redhat.com> Message-ID: <20151210012845.F1C30849EF@eggemoggin.niobe.net> 2015/10/21 4:21 -0700, david.lloyd at redhat.com: > I have a problem with the premise that modules should be directly > consumable by the build in the same way they are consumed at run time. > Building a typical Java artifact today generally entails the Maven > approach, which, roughly translated into modularisms, would look > something like this: > > * We build artifacts, not modules > * The build consumes (specific versions of) modules for build tasks such > as resource processing, descriptor generation, annotation processing, etc. > * The build consumes (specific versions of) artifacts for the build > class path; these artifacts generally correspond to a minimum version of > a module that is expected to be mandatorily or optionally present at run > time, but also may correspond only to specific APIs expected to be > present at run time > * The run time environment may consist of a set of modules that does not > have a direct correspondence to the artifacts used for build > * The test time environment may consist of a set of modules that does > not have a direct correspondence to the artifacts used for build > > To me there is no sensible way to reconcile these facts against the > proposed Jigsaw methodology of having javac reference modules directly. > This would mean that all build artifacts have to be packaged as > modules and transported to the user for build, which further means that > every user would need to define a complete, build-specific modular > environment, ... I'm not completely sure what you're getting at here, but it's not the case that all of the modules needed to run a particular module must be present to compile that module. The compiler only needs to resolve the immediate and implied read edges of the module(s) being compiled. If module A requires B, and B requires C, then the compiler can compile A without inspecting C. If B requires public C, however, then the compiler must inspect C also, and therefore resolve it. All of this follows from the definitions of readability and implied readability. - Mark From mark.reinhold at oracle.com Thu Dec 10 01:29:45 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 9 Dec 2015 17:29:45 -0800 (PST) Subject: My position on module security/access control In-Reply-To: <564CA2F6.10406@redhat.com> References: <564CA2F6.10406@redhat.com> Message-ID: <20151210012945.F3B09849F4@eggemoggin.niobe.net> 2015/11/18 8:10 -0800, david.lloyd at redhat.com: > I wanted to summarize my position on module security and access control, > so that all my thoughts are in one place instead of scattered across > various threads. > > The first point I want to make is that I think that the idea of having > public classes that are not public is a red flag. If a class and member > is public, it should always be accessible to everyone regardless of who > imports what module. There should only be one access control mechanism > in the language. Users already seem keen to find ways to break out of > the Jigsaw export-based restrictions and I think this is an indicator > that, while expedient, the Jigsaw approach is far from ideal. In > addition I think this is an essential step towards the ultimate goal of > removing the dreaded setAccessible() method; if frameworks cannot even > access public members then they will never be able break away from using > reflection as a back door. The concept being established here is the > ability to have a public class which is not exported but is still > accessible to anyone. If I understand correctly, you're proposing that a public type in a non-exported package be accessible via reflection, but not via static references in source or class files. (There would still have to be some way to declare whether the public types in a package are exported for use in static references, but how that's done is a different issue.) You argue that "public" should always mean "public", yet your proposal would change the meaning of "public". To tell whether something marked "public" is accessible statically from outside a module, a developer would still have to consult the file (in whatever format) in which the relevant package is declared to be exported (or not). "public" inside a module would still have a different meaning than "public" outside a module, and a different meaning from what "public" meant in Java 8. So, do you want to preserve the meaning of "public", or not? Your proposal would, further, introduce an asymmetry between access control in the language and the VM vs. access control in the reflection APIs. With the exception of escape hatches such as setAccessible, the reflection APIs have always worked in exactly the same way as the corresponding static constructs. Your proposal would address the issue of making it easier for frameworks to migrate to modules, but it is not the only approach to that problem (about which more later). > The second point is around module privacy. It is clear that > (language-wise) we need an access level that is module-private. Given > the relative uselessness of package-private accessibility in the module > world, I propose that any reference to a default-access-level member in > a Java 9 class should be considered to be a module-private access, not a > package-private access, in both javac and the JVM. The protected access > level should be similarly widened. ... If a Java 9 class is not inside a module, i.e., it's on the class path, then what is the default access mode? If it's package-private then it would work differently than from within a module. If it's private to the unnamed module in which all classes found on the class path are defined then it would work differently from any pre-9 classes on the class path. Either outcome is likely to confuse people. If a Java 8 class is inside a module then I assume the default access mode would remain package-private. (We expect this to be common, with modular JAR files intended for use on the module path in 9 and on the class path in earlier releases.) Taken with the proposal in your first point then all public pre-9 classes would be available via the reflection APIs, regardless of whether their packages are exported. The only way to encapsulate such classes, therefore, would be to upgrade them to Java 9, at which point the modular JAR would no longer work on earlier releases. This limitation would add complexity to the migration story. At a higher level, I agree that the package-private access mode has proved to be of limited utility. It has, however, been a fundamental part of the language and the VM from the very beginning. To widen it in such an incompatible fashion now, after twenty years, would make it more difficult to migrate existing code to modules and would risk enabling further security vulnerabilities. The design as proposed, by contrast, narrows rather than widens an existing access mode (public), so security is not an issue, and it narrows that mode in a way that's relatively easy to explain -- just consult the relevant module declaration. > The third point is around "friend" modules. It seems clear that we have > a requirement (which can be extrapolated easily from the document) that > some modules need the ability to provide selective access to nonpublic > members to other modules. ... There's no need to extrapolate, since we captured this requirement explicitly [1]. (I don't see a specific proposal here, so I have no further comments on this point.) > The fourth point relates to ServiceLoader. If ServiceLoader has a > unique and special blessing to bypass the access mechanism, then we've > failed to design an adequately flexible (and secure) access control > mechanism. By implementing the three points above, ServiceLoader no > longer needs to be a special citizen, because all service implementation > classes are public (as they are today, and as they should be). A user > could implement ServiceLoader in unprivileged code and it would work > with no special trapdoors. I agree that, as a proof point, it'd be nice to implement ServiceLoader using only reflective operations that are also available to non-system modules. ServiceLoader is, after all, just a small kind of framework. Services are bound and provisioned by the module system itself, however, so ServiceLoader does, of necessity, have some special privileges, and I don't think that can be avoided. - Mark [1] http://openjdk.java.net/projects/jigsaw/spec/reqs/#qualified-exports From david.lloyd at redhat.com Thu Dec 10 15:02:38 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 10 Dec 2015 09:02:38 -0600 Subject: Module descriptions versus module descriptors In-Reply-To: <20151210012645.EDDB1849E4@eggemoggin.niobe.net> References: <5626CDF7.5060605@redhat.com> <20151210012645.EDDB1849E4@eggemoggin.niobe.net> Message-ID: <5669940E.60902@redhat.com> On 12/09/2015 07:26 PM, mark.reinhold at oracle.com wrote: > 2015/10/20 4:27 -0700, david.lloyd at redhat.com: >> I see no logical path that leads from the requirements as specified >> exclusively to the assumption that the descriptor must be bytecode, let >> alone part of the JVM and/or language specification. All the reasons >> given appear to be self-justifying or based on abstract assumptions, >> e.g. "modules are... a new kind of Java program component... therefore >> [Jigsaw] treats them as such". > > I agree that there are other ways to represent module descriptions. > I've never stated otherwise. > > If module boundaries, however they're expressed, are to be enforced by > the compiler and the VM -- as you've agreed elsewhere -- then their > manner of expression is inevitably going to be a topic for the Java > language and VM specifications. This is not avoidable. Actually that does not logically follow. I could, for example, "express" module boundaries in the VM by creating a hard equivalence between modules and class loaders, and in the compiler via command-line arguments. In this way, no langauge or VM specifications must change; in fact only the specification of the compiler tool itself *must* change. Granted it is *likely* that in such a case, one would go further and maybe make some small modifications to the javax.tools API as well to facilitate the provision of resources on a per-module basis. >> ... it seems to me that a significant part of the Jigsaw >> design justification for its handling of module metadata hinges around >> the conflation of the description of a module, and the descriptor used >> by the static module loading implementation. This raises a red flag for >> me because it fundamentally locks the capabilities of module >> descriptions to whatever makes sense to express in the descriptor, and >> then in turn constrains these things to the language and JVM specification. >> >> In our (JBoss) module system, these concepts are decoupled: a filesystem >> module's descriptor is read and parsed into a description which is then >> consumed by the module system to create the module. ... > > If module boundaries are to be enforced by the compiler and the VM then, > in such a decoupled system, where and in what form does the compiler > locate the information that describes the module being compiled, and any > other modules required in order to compile that module? You (or rather, the build tool) would tell it. In real nontrivial systems, the universe of compiled and packaged (and versioned) artifacts is what you use when you're building another piece of the system. The installed set of modules is necessarily a subset of this total universe. Because of this, if I'm building something I generally cannot say "I have a build dependency upon org.apache.commons.collections". I must say "Build against org.apache.commons.collections, version 3.2.2", and furthermore I might have to also specify whose build of such I use, and where to get it from. And, I commonly must build different pieces of my universe against different versions of the same thing, especially when pieces come from a third party. I will likely not use the same version I built against in the final system, especially in an evolving system, which I believe is a critical use case. Finally, I may use the final built artifact in a variety of systems, whereupon it is necessary to modify the run time name and/or linkage information depending on the requirements of the target system. I will almost certainly not even test the two together until a later integration testing phase, which (in probable addition to an ABI compatibility checking tool) will be the last stage before accepting the new module into the installed set. In other words, it is often necessary to enforce different module boundaries depending on circumstances related to build, install, and run, independently of the rest of the source code. Much of this information may not be available at all when the original artifact is being built. Any bundling of module descriptor information inside of the module contents will have the effect of rendering that module useless in any module environment that is sufficiently unlike the one for/in which the module was built (unless you modify the module itself after the fact). It is not realistic to assume that the version of a module that is compiled against is the version that will be run against, and it's misleading to "newbies" who might expect some kind of global cohesion between modules without any kind of limiting context (e.g. a module distribution, or a specific application). The power that a modular Java gives you is that you can leverage what is arguably Java's best feature - the ability to easily provide perfect ABI stability between versions, and therefore the ability to substitute modules in the run-time environment - in order to maintain a growing and evolving ecosystems, or indeed many such ecosystems. This is not unlike what many OS distributors can do, and for similar reasons in a similar way (though we have the sizable advantage of Java's more powerful linking capabilities at our disposal, which makes some hard things easy, and some impossible things possible). I think that the module system should do what it can to enable this, and to me that means: external module descriptions, i.e. the compiled artifact (which consumes and/or provides specific ABI(s)) should be separate from the install/run linkage (which specifies how to tie modules together in such a way that all necessary ABI requirements are fulfilled). This is because the artifact build dependency information is superficially similar to, but fundamentally not the same as, the run time linkage information - and this will be true no matter what we do or what constraints we place. The only thing we can influence in this regard is, how hard do we make these use cases to achieve? There are other reasons that I prefer this approach as well, but this is the reason that directly answers your question. And there are possibly other approaches that would also solve the same problems - for example, a hybrid approach where exported and non-exported packages are listed and annotated within the module, but the dependence information is external and provided separately at build time and run time (in a manner suitable to each situation). -- - DML From david.lloyd at redhat.com Thu Dec 10 15:39:28 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 10 Dec 2015 09:39:28 -0600 Subject: Coupling modules and class loaders In-Reply-To: <20151210012745.EFD0F849EA@eggemoggin.niobe.net> References: <56276E91.5060605@redhat.com> <20151210012745.EFD0F849EA@eggemoggin.niobe.net> Message-ID: <56699CB0.6020704@redhat.com> On 12/09/2015 07:27 PM, mark.reinhold at oracle.com wrote: > 2015/10/21 3:53 -0700, david.lloyd at redhat.com: >> ... >> >> I propose that modules are best defined as a specialization of class >> loaders. Class loaders already have many of the properties that define >> a module: encapsulation, isolation, unique identity for packages. In >> fact OSGi already uses this unit. The only thing they are really >> missing is the ability to use a graph-like dependency structure, hence >> the specialization. > > Class loaders do not provide strong encapsulation. How do you propose to > extend them to do so? Sure they do - here are some ways: * Package-private and protected members in one class loader are inaccessible to packages in another class loader, even if the package has the same name. * Run-time linkage is strongly encapsulated by class loader boundary; I can declare a public type that nevertheless cannot be linked against by anyone, if the class loader denies such access. There are several use cases for this. If I could extend this in any in any way, I would do two things (whether or not modularity goes forward in the platform): 1) Expand the default access level to be class loader private (and similarly change protected), which would give users a means to define a type and members which are accessible to the module but inaccessible outside of it, while still allowing universal access to public types. Specifically this would apply only when referencing types in a Java 9 class file. References to types in Java 8 (and earlier) class files would still use the package as the protection level (though I doubt this distinction would ever come up in practice as few artifacts have a mix of class file versions within them). 2) Add a single method to the class loader which would allow (at package definition time) a package to be declared to be accessible from specific other parties (probably other class loader+package combinations). This information would theoretically be utilized by the bytecode verifier to restrict access in the same manner that package-private access is restricted today. This solves the old "friend module" case from JSR 294, though I recall that there wasn't agreement as to whether it was necessary (it would certainly eliminate the need for the "shared secrets" mechanism though). The corresponding functionality at build time would amount to telling javac where to find each dependency, and possibly also telling javac which dependency packages are expected to be accessible at run time (if #2 above would be implemented, which would not strictly be necessary IMO but might be a good feature to have). >> ... >> >> What is the justification for the more complex Jigsaw approach? > > One developer's complexity is another's simplicity. Fair enough, that was definitely a subjective assessment. > The primary motivation is, as R?mi noted, to make it easy for existing > applications that use class loaders in sophisticated (and sometimes very > strange) ways to adopt modules. If the maintainers of such applications > must make deep changes to their class-loading architectures in order to > support modules then some of them simply won't bother. I don't think they would though. Even in the complete compatibility case, the "flat classpath" module could and would still just be one module with the whole class path on it, which would be not unlike Ye Olde Application Class Loader, while still allowing the forward migration path of adding real module dependencies to the ugly legacy class path. Running with -jar gives us two options: compile a flat classpath as above, or assemble a module graph (with a variety of transitivity possibilities) based on Class-Path references. Even the legacy extensions thing works surprisingly well as a module system, if a somewhat primitive one, had it not been deprecated (removed?) from the platform in SE 9. If people are doing weird/nasty/ugly stuff with class loaders, they would certainly be free to continue to do so if they want; the per-class loader idea does not necessarily require that the module class loader class be final, nor does it forbid it from being used as a parent or delegate, unless you're thinking of something else? > The fact that OSGi and JBoss have successfully used class loaders to > implement module systems that are capable of loading a wide variety of > existing artifacts is interesting. It does not, however, imply that all > existing applications can be converted to use the class-loader-per-module > approach, nor that they should be forced to do so. The proposed design > therefore neither mandates nor forbids that approach. I think we probably would not be able to put bounds on this without enumerating specific use cases/behaviors (especially those that would fail in this situation but not in the other), as I haven't encountered nor been able to imagine a situation that would fail solely due to the introduction of class loader based modules. -- - DML From david.lloyd at redhat.com Thu Dec 10 15:42:45 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 10 Dec 2015 09:42:45 -0600 Subject: Modules and build time In-Reply-To: <20151210012845.F1C30849EF@eggemoggin.niobe.net> References: <5627751C.9020702@redhat.com> <20151210012845.F1C30849EF@eggemoggin.niobe.net> Message-ID: <56699D75.8040408@redhat.com> On 12/09/2015 07:28 PM, mark.reinhold at oracle.com wrote: > 2015/10/21 4:21 -0700, david.lloyd at redhat.com: >> I have a problem with the premise that modules should be directly >> consumable by the build in the same way they are consumed at run time. >> Building a typical Java artifact today generally entails the Maven >> approach, which, roughly translated into modularisms, would look >> something like this: >> >> * We build artifacts, not modules >> * The build consumes (specific versions of) modules for build tasks such >> as resource processing, descriptor generation, annotation processing, etc. >> * The build consumes (specific versions of) artifacts for the build >> class path; these artifacts generally correspond to a minimum version of >> a module that is expected to be mandatorily or optionally present at run >> time, but also may correspond only to specific APIs expected to be >> present at run time >> * The run time environment may consist of a set of modules that does not >> have a direct correspondence to the artifacts used for build >> * The test time environment may consist of a set of modules that does >> not have a direct correspondence to the artifacts used for build >> >> To me there is no sensible way to reconcile these facts against the >> proposed Jigsaw methodology of having javac reference modules directly. >> This would mean that all build artifacts have to be packaged as >> modules and transported to the user for build, which further means that >> every user would need to define a complete, build-specific modular >> environment, ... > > I'm not completely sure what you're getting at here, but it's not the > case that all of the modules needed to run a particular module must be > present to compile that module. The compiler only needs to resolve the > immediate and implied read edges of the module(s) being compiled. If > module A requires B, and B requires C, then the compiler can compile A > without inspecting C. If B requires public C, however, then the compiler > must inspect C also, and therefore resolve it. All of this follows from > the definitions of readability and implied readability. Sure, I agree this logic is inescapable. The question is, what B and what C do I build against? Once a module is built, it is not "forever", it is only until a different version of the module is built. Every time I build something, I'm using a different subgraph of an overall module universe, and that subgraph extends to all previously built versions of each of those things. At run time I'm using a subgraph of this module universe that is very unlikely (past the first version of something) to match the build subgraph. -- - DML From david.lloyd at redhat.com Thu Dec 10 16:00:18 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 10 Dec 2015 10:00:18 -0600 Subject: My position on module security/access control In-Reply-To: <20151210012945.F3B09849F4@eggemoggin.niobe.net> References: <564CA2F6.10406@redhat.com> <20151210012945.F3B09849F4@eggemoggin.niobe.net> Message-ID: <5669A192.1060509@redhat.com> On 12/09/2015 07:29 PM, mark.reinhold at oracle.com wrote: > 2015/11/18 8:10 -0800, david.lloyd at redhat.com: >> I wanted to summarize my position on module security and access control, >> so that all my thoughts are in one place instead of scattered across >> various threads. >> >> The first point I want to make is that I think that the idea of having >> public classes that are not public is a red flag. If a class and member >> is public, it should always be accessible to everyone regardless of who >> imports what module. There should only be one access control mechanism >> in the language. Users already seem keen to find ways to break out of >> the Jigsaw export-based restrictions and I think this is an indicator >> that, while expedient, the Jigsaw approach is far from ideal. In >> addition I think this is an essential step towards the ultimate goal of >> removing the dreaded setAccessible() method; if frameworks cannot even >> access public members then they will never be able break away from using >> reflection as a back door. The concept being established here is the >> ability to have a public class which is not exported but is still >> accessible to anyone. > > If I understand correctly, you're proposing that a public type in a > non-exported package be accessible via reflection, but not via static > references in source or class files. (There would still have to be some > way to declare whether the public types in a package are exported for > use in static references, but how that's done is a different issue.) > > You argue that "public" should always mean "public", yet your proposal > would change the meaning of "public". To tell whether something marked > "public" is accessible statically from outside a module, a developer > would still have to consult the file (in whatever format) in which the > relevant package is declared to be exported (or not). "public" inside > a module would still have a different meaning than "public" outside a > module, and a different meaning from what "public" meant in Java 8. Not "will", "does". This is the current status quo. Types that are not visible by a class loader cannot be linked against; this doesn't change the definition of "public", which has been well-established since the dawn of Java. Today I can instantiate public types and invoke public methods on them regardless of what class loader they live in, yet I can restrict linkage to such types, simply by arrangement of class loaders. This is all possible and expected behavior today and I propose that these concepts be leveraged, not changed. > So, do you want to preserve the meaning of "public", or not? Yes, I want to preserve it exactly with no changes to the JLS or JVM specification. > Your proposal would, further, introduce an asymmetry between access > control in the language and the VM vs. access control in the reflection > APIs. With the exception of escape hatches such as setAccessible, the > reflection APIs have always worked in exactly the same way as the > corresponding static constructs. It introduces nothing - this is how it works today. Visibility is an existing concept that is useful and used. Accessibility is a distinct existing concept that is also useful and used. > Your proposal would address the issue of making it easier for frameworks > to migrate to modules, but it is not the only approach to that problem > (about which more later). Of course; my position is to make the minimum set of changes (code and concept) that meet the requirements, which I believe is a substantially smaller set of changes than what is presently on the table. This gives us a somewhat objective way to measure the quality of the implementation. >> The second point is around module privacy. It is clear that >> (language-wise) we need an access level that is module-private. Given >> the relative uselessness of package-private accessibility in the module >> world, I propose that any reference to a default-access-level member in >> a Java 9 class should be considered to be a module-private access, not a >> package-private access, in both javac and the JVM. The protected access >> level should be similarly widened. ... > > If a Java 9 class is not inside a module, i.e., it's on the class path, > then what is the default access mode? If it's package-private then it > would work differently than from within a module. If it's private to the > unnamed module in which all classes found on the class path are defined > then it would work differently from any pre-9 classes on the class path. > Either outcome is likely to confuse people. Class path private; see my other mail. > If a Java 8 class is inside a module then I assume the default access > mode would remain package-private. (We expect this to be common, with > modular JAR files intended for use on the module path in 9 and on the > class path in earlier releases.) Taken with the proposal in your first > point then all public pre-9 classes would be available via the reflection > APIs, regardless of whether their packages are exported. The only way to > encapsulate such classes, therefore, would be to upgrade them to Java 9, > at which point the modular JAR would no longer work on earlier releases. > This limitation would add complexity to the migration story. It would add no complexity, because this is the status quo today. Public types are universally accessible today, and I do not propose that to be changed. No application will be worse off security-wise, but they will have a path to improve their encapsulation over time. > At a higher level, I agree that the package-private access mode has > proved to be of limited utility. It has, however, been a fundamental > part of the language and the VM from the very beginning. To widen it in > such an incompatible fashion now, after twenty years, would make it more > difficult to migrate existing code to modules and would risk enabling > further security vulnerabilities. I think this is objectively false. Existing code would continue to have the same encapsulation; only new code would have the expanded capability and only in reference to other new code. > The design as proposed, by contrast, narrows rather than widens an > existing access mode (public), so security is not an issue, and it > narrows that mode in a way that's relatively easy to explain -- just > consult the relevant module declaration. Yes, but this then will break many things. Why not allow application developers to do this themselves, over time? >> The third point is around "friend" modules. It seems clear that we have >> a requirement (which can be extrapolated easily from the document) that >> some modules need the ability to provide selective access to nonpublic >> members to other modules. ... > > There's no need to extrapolate, since we captured this requirement > explicitly [1]. > > (I don't see a specific proposal here, so I have no further comments > on this point.) See other thread. >> The fourth point relates to ServiceLoader. If ServiceLoader has a >> unique and special blessing to bypass the access mechanism, then we've >> failed to design an adequately flexible (and secure) access control >> mechanism. By implementing the three points above, ServiceLoader no >> longer needs to be a special citizen, because all service implementation >> classes are public (as they are today, and as they should be). A user >> could implement ServiceLoader in unprivileged code and it would work >> with no special trapdoors. > > I agree that, as a proof point, it'd be nice to implement ServiceLoader > using only reflective operations that are also available to non-system > modules. ServiceLoader is, after all, just a small kind of framework. > > Services are bound and provisioned by the module system itself, however, > so ServiceLoader does, of necessity, have some special privileges, and I > don't think that can be avoided. To me this is just a convenience feature of the module API: the ability to locate and instantiate a "global" set of services, the accomplishment of which is simply a feature of the thing doing the module loading. One simple realization of this is to use the set of services in modules visible to the started application module, which allows each application to have its own set of global services; using this strategy, global services can be found simply by using the application module class loader. The only real involvement of the service loader with this strategy is the use of privilege to get that class loader; otherwise (using my proposed approach) there is no special case here. It's just a convenience API for users. -- - DML From mark.reinhold at oracle.com Mon Dec 14 19:43:13 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 14 Dec 2015 11:43:13 -0800 (PST) Subject: Module descriptions versus module descriptors In-Reply-To: <5669940E.60902@redhat.com> References: <5626CDF7.5060605@redhat.com>, <20151210012645.EDDB1849E4@eggemoggin.niobe.net>, <5669940E.60902@redhat.com> Message-ID: <20151214194313.C4C8E8530F@eggemoggin.niobe.net> 2015/12/10 7:02 -0800, david.lloyd at redhat.com: > On 12/09/2015 07:26 PM, mark.reinhold at oracle.com wrote: >> If module boundaries, however they're expressed, are to be enforced by >> the compiler and the VM -- as you've agreed elsewhere -- then their >> manner of expression is inevitably going to be a topic for the Java >> language and VM specifications. This is not avoidable. > > Actually that does not logically follow. I could, for example, > "express" module boundaries in the VM by creating a hard equivalence > between modules and class loaders, and in the compiler via command-line > arguments. In this way, no langauge or VM specifications must change; > in fact only the specification of the compiler tool itself *must* > change. If every module is represented at run time by a unique class loader, then would the VM to do anything differently than today? Would it have any role at all to play in enforcing module boundaries, regardless of their means of expression? If we widen the default access mode to mean module-private in some circumstances, as you suggest, then wouldn't that require revisions to the VM and language specifications? If module boundaries are communicated to the compiler via command-line flags, then how do you propose to make all Java compilers work in the same way? Bear in mind that there is no specification for the compiler as a tool -- there is only the Java Language Specification, which to date has never mandated any particular command-line flags, nor even the existence of a command line upon which to place such flags. - Mark From mark.reinhold at oracle.com Mon Dec 14 19:44:13 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 14 Dec 2015 11:44:13 -0800 (PST) Subject: Coupling modules and class loaders In-Reply-To: <56699CB0.6020704@redhat.com> References: <56276E91.5060605@redhat.com>, <20151210012745.EFD0F849EA@eggemoggin.niobe.net>, <56699CB0.6020704@redhat.com> Message-ID: <20151214194413.C698A85311@eggemoggin.niobe.net> 2015/12/10 7:39 -0800, david.lloyd at redhat.com: > On 12/09/2015 07:27 PM, mark.reinhold at oracle.com wrote: >> 2015/10/21 3:53 -0700, david.lloyd at redhat.com: >>> ... >>> >>> I propose that modules are best defined as a specialization of class >>> loaders. Class loaders already have many of the properties that define >>> a module: encapsulation, isolation, unique identity for packages. In >>> fact OSGi already uses this unit. The only thing they are really >>> missing is the ability to use a graph-like dependency structure, hence >>> the specialization. >> >> Class loaders do not provide strong encapsulation. How do you propose to >> extend them to do so? > > Sure they do - here are some ways: > > * Package-private and protected members in one class loader are > inaccessible to packages in another class loader, even if the package > has the same name. (Yes, that's strong encapsulation, but not strong encapsulation of modules or of module-like entities, which is what I meant.) > * Run-time linkage is strongly encapsulated by class loader boundary; I > can declare a public type that nevertheless cannot be linked against by > anyone, if the class loader denies such access. There are several use > cases for this. That is encapsulation of a sort, but it's not strong since it works differently with respect to reflection (which, yes, is the approach that you advocate). >>> ... >>> >>> What is the justification for the more complex Jigsaw approach? >> >> One developer's complexity is another's simplicity. > > Fair enough, that was definitely a subjective assessment. > >> The primary motivation is, as R??mi noted, to make it easy for existing >> applications that use class loaders in sophisticated (and sometimes very >> strange) ways to adopt modules. If the maintainers of such applications >> must make deep changes to their class-loading architectures in order to >> support modules then some of them simply won't bother. > > I don't think they would though. Even in the complete compatibility > case, the "flat classpath" module could and would still just be one > module with the whole class path on it, which would be not unlike Ye > Olde Application Class Loader, while still allowing the forward > migration path of adding real module dependencies to the ugly legacy > class path. ... If people are doing weird/nasty/ugly stuff with class > loaders, they would certainly be free to continue to do so if they want; > the per-class loader idea does not necessarily require that the module > class loader class be final, nor does it forbid it from being used as a > parent or delegate, unless you're thinking of something else? What I'm thinking of is an existing application with its own non-trivial class-loader architecture which, due to compatibility constraints, cannot be changed. If at least one of its class loaders loads classes from logically-distinct components that are later converted into corresponding modules then upgrading the application to load those components as modules would, under your approach, require replacing that one class loader with many class loaders, which would break compatibility. Imagine, e.g., a Java EE application server that loads implementations of all the standard EE components using one class loader. Imagine further that existing users (for good or for ill) depend upon that. If a future version of Java EE defines those components as modules then the only way to arrange for this app server to load them as such would be to change its class-loader architecture in a way that would break existing uses. (This is, roughly, the situation with Oracle's WebLogic app server, as I've mentioned previously.) >> The fact that OSGi and JBoss have successfully used class loaders to >> implement module systems that are capable of loading a wide variety of >> existing artifacts is interesting. It does not, however, imply that all >> existing applications can be converted to use the class-loader-per-module >> approach, nor that they should be forced to do so. The proposed design >> therefore neither mandates nor forbids that approach. > > I think we probably would not be able to put bounds on this without > enumerating specific use cases/behaviors (especially those that would > fail in this situation but not in the other), as I haven't encountered > nor been able to imagine a situation that would fail solely due to the > introduction of class loader based modules. See above. - Mark From mark.reinhold at oracle.com Mon Dec 14 19:45:13 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 14 Dec 2015 11:45:13 -0800 (PST) Subject: My position on module security/access control In-Reply-To: <5669A192.1060509@redhat.com> References: <564CA2F6.10406@redhat.com>, <20151210012945.F3B09849F4@eggemoggin.niobe.net>, <5669A192.1060509@redhat.com> Message-ID: <20151214194513.C87D885313@eggemoggin.niobe.net> 2015/12/10 8:00 -0800, david.lloyd at redhat.com: > On 12/09/2015 07:29 PM, mark.reinhold at oracle.com wrote: >> 2015/11/18 8:10 -0800, david.lloyd at redhat.com: >>> ... The concept being established here is the >>> ability to have a public class which is not exported but is still >>> accessible to anyone. >> >> If I understand correctly, you're proposing that a public type in a >> non-exported package be accessible via reflection, but not via static >> references in source or class files. (There would still have to be some >> way to declare whether the public types in a package are exported for >> use in static references, but how that's done is a different issue.) >> >> You argue that "public" should always mean "public", yet your proposal >> would change the meaning of "public". To tell whether something marked >> "public" is accessible statically from outside a module, a developer >> would still have to consult the file (in whatever format) in which the >> relevant package is declared to be exported (or not). "public" inside >> a module would still have a different meaning than "public" outside a >> module, and a different meaning from what "public" meant in Java 8. > > Not "will", "does". This is the current status quo. Types that are not > visible by a class loader cannot be linked against; this doesn't change > the definition of "public", which has been well-established since the > dawn of Java. Today I can instantiate public types and invoke public > methods on them regardless of what class loader they live in, yet I can > restrict linkage to such types, simply by arrangement of class loaders. > This is all possible and expected behavior today and I propose that > these concepts be leveraged, not changed. > >> So, do you want to preserve the meaning of "public", or not? > > Yes, I want to preserve it exactly with no changes to the JLS or JVM > specification. Okay, I think I'm starting to get it, but in your statement >>> The concept being established here is the >>> ability to have a public class which is not exported but is still >>> accessible to anyone. the clause "is not exported" suggests that some public types are "exported" and some are not. I read this to mean that public types would still be subject to some sort of access control based upon some kind of export declarations, but only for static linkage. If that's not the case, then what's the distinction between "not exported" and "exported"? >>> The second point is around module privacy. It is clear that >>> (language-wise) we need an access level that is module-private. Given >>> the relative uselessness of package-private accessibility in the module >>> world, I propose that any reference to a default-access-level member in >>> a Java 9 class should be considered to be a module-private access, not a >>> package-private access, in both javac and the JVM. The protected access >>> level should be similarly widened. ... >> >> If a Java 9 class is not inside a module, i.e., it's on the class path, >> then what is the default access mode? If it's package-private then it >> would work differently than from within a module. If it's private to the >> unnamed module in which all classes found on the class path are defined >> then it would work differently from any pre-9 classes on the class path. >> Either outcome is likely to confuse people. > > Class path private; see my other mail. Okay. Suppose that I upgrade a pre-9 component into a module using this approach, and ship it as a modular JAR file so that it can be used both as a module and on the class path. If it's used on the class path then any types that formerly were package-private are now accessible to all other code on the class path, regardless of whether that other code was compiled with 9 or with an earlier release, right? >> If a Java 8 class is inside a module then I assume the default access >> mode would remain package-private. (We expect this to be common, with >> modular JAR files intended for use on the module path in 9 and on the >> class path in earlier releases.) Taken with the proposal in your first >> point then all public pre-9 classes would be available via the reflection >> APIs, regardless of whether their packages are exported. The only way to >> encapsulate such classes, therefore, would be to upgrade them to Java 9, >> at which point the modular JAR would no longer work on earlier releases. >> This limitation would add complexity to the migration story. > > It would add no complexity, because this is the status quo today. > > Public types are universally accessible today, and I do not propose that > to be changed. No application will be worse off security-wise, but they > will have a path to improve their encapsulation over time. Yes, they would have a migration path, but a very long one. The only way to encapsulate an existing public type within a module would be to change its access mode to the default and compile it -target 9. As a consequence the resulting class file would only be usable on Java 9 and later releases. The maintainer of an existing library would only be able to convert that library into a module after all existing users upgrade to Java 9. If history is any guide then, for a popular library, that's likely to be many years after Java 9 ships. Is that what we want? >> At a higher level, I agree that the package-private access mode has >> proved to be of limited utility. It has, however, been a fundamental >> part of the language and the VM from the very beginning. To widen it in >> such an incompatible fashion now, after twenty years, would make it more >> difficult to migrate existing code to modules and would risk enabling >> further security vulnerabilities. > > I think this is objectively false. Existing code would continue to have > the same encapsulation; only new code would have the expanded capability > and only in reference to other new code. What about references from outside a module? Would the default access mode on a Java 9 type mean "module private" if the the code attempting to access that type is itself compiled -target 9, and "package private" otherwise? - Mark From david.lloyd at redhat.com Tue Dec 15 16:51:11 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 15 Dec 2015 10:51:11 -0600 Subject: My position on module security/access control In-Reply-To: <20151214194513.C87D885313@eggemoggin.niobe.net> References: <564CA2F6.10406@redhat.com> <20151210012945.F3B09849F4@eggemoggin.niobe.net> <5669A192.1060509@redhat.com> <20151214194513.C87D885313@eggemoggin.niobe.net> Message-ID: <567044FF.5050209@redhat.com> On 12/14/2015 01:45 PM, mark.reinhold at oracle.com wrote: > 2015/12/10 8:00 -0800, david.lloyd at redhat.com: >> On 12/09/2015 07:29 PM, mark.reinhold at oracle.com wrote: >>> 2015/11/18 8:10 -0800, david.lloyd at redhat.com: >>>> ... The concept being established here is the >>>> ability to have a public class which is not exported but is still >>>> accessible to anyone. >>> >>> If I understand correctly, you're proposing that a public type in a >>> non-exported package be accessible via reflection, but not via static >>> references in source or class files. (There would still have to be some >>> way to declare whether the public types in a package are exported for >>> use in static references, but how that's done is a different issue.) >>> >>> You argue that "public" should always mean "public", yet your proposal >>> would change the meaning of "public". To tell whether something marked >>> "public" is accessible statically from outside a module, a developer >>> would still have to consult the file (in whatever format) in which the >>> relevant package is declared to be exported (or not). "public" inside >>> a module would still have a different meaning than "public" outside a >>> module, and a different meaning from what "public" meant in Java 8. >> >> Not "will", "does". This is the current status quo. Types that are not >> visible by a class loader cannot be linked against; this doesn't change >> the definition of "public", which has been well-established since the >> dawn of Java. Today I can instantiate public types and invoke public >> methods on them regardless of what class loader they live in, yet I can >> restrict linkage to such types, simply by arrangement of class loaders. >> This is all possible and expected behavior today and I propose that >> these concepts be leveraged, not changed. >> >>> So, do you want to preserve the meaning of "public", or not? >> >> Yes, I want to preserve it exactly with no changes to the JLS or JVM >> specification. > > Okay, I think I'm starting to get it, but in your statement > >>>> The concept being established here is the >>>> ability to have a public class which is not exported but is still >>>> accessible to anyone. > > the clause "is not exported" suggests that some public types are > "exported" and some are not. I read this to mean that public types would > still be subject to some sort of access control based upon some kind of > export declarations, but only for static linkage. If that's not the > case, then what's the distinction between "not exported" and "exported"? Right though I prefer the existing "visibility" term for this, as this term is well over a decade old and reasonably well-understood at this point. >>>> The second point is around module privacy. It is clear that >>>> (language-wise) we need an access level that is module-private. Given >>>> the relative uselessness of package-private accessibility in the module >>>> world, I propose that any reference to a default-access-level member in >>>> a Java 9 class should be considered to be a module-private access, not a >>>> package-private access, in both javac and the JVM. The protected access >>>> level should be similarly widened. ... >>> >>> If a Java 9 class is not inside a module, i.e., it's on the class path, >>> then what is the default access mode? If it's package-private then it >>> would work differently than from within a module. If it's private to the >>> unnamed module in which all classes found on the class path are defined >>> then it would work differently from any pre-9 classes on the class path. >>> Either outcome is likely to confuse people. >> >> Class path private; see my other mail. > > Okay. Suppose that I upgrade a pre-9 component into a module using this > approach, and ship it as a modular JAR file so that it can be used both > as a module and on the class path. If it's used on the class path then > any types that formerly were package-private are now accessible to all > other code on the class path, regardless of whether that other code was > compiled with 9 or with an earlier release, right? Yes, potentially, if the class path is held as a single module (which it could be, but does not have to be). There are pros and cons to each approach though. >>> If a Java 8 class is inside a module then I assume the default access >>> mode would remain package-private. (We expect this to be common, with >>> modular JAR files intended for use on the module path in 9 and on the >>> class path in earlier releases.) Taken with the proposal in your first >>> point then all public pre-9 classes would be available via the reflection >>> APIs, regardless of whether their packages are exported. The only way to >>> encapsulate such classes, therefore, would be to upgrade them to Java 9, >>> at which point the modular JAR would no longer work on earlier releases. >>> This limitation would add complexity to the migration story. >> >> It would add no complexity, because this is the status quo today. >> >> Public types are universally accessible today, and I do not propose that >> to be changed. No application will be worse off security-wise, but they >> will have a path to improve their encapsulation over time. > > Yes, they would have a migration path, but a very long one. > > The only way to encapsulate an existing public type within a module would > be to change its access mode to the default and compile it -target 9. As > a consequence the resulting class file would only be usable on Java 9 and > later releases. The maintainer of an existing library would only be able > to convert that library into a module after all existing users upgrade to > Java 9. If history is any guide then, for a popular library, that's > likely to be many years after Java 9 ships. > > Is that what we want? I think so. I think users are looking at a long migration path either way: either because everything is broken and in some cases must be redesigned, or because there are new features that they can begin to take advantage of (but are not compelled to do so). I prefer the latter not only because the end user experience is disrupted to a much lesser degree, but also because it is not clear that the Jigsaw access rules can ever be accommodated by existing frameworks like those in Java EE (including CDI and JPA) and other existing and widely used libraries. On the other hand, by maintaining behavioral compatibility but adding new security capabilities, it is far more certain that these things will continue to work in both the short and long term, while providing a useful path to increase security over time. One other part of the appeal of this feature is that it can be done completely independently of the implementation of Jigsaw or any other module system, as the access level could apply to the class loader in the same manner that package-private access does today, though this based on the predicate that any later module implementation use class loaders as its unit of run-time granularity. In terms of the migration concerns you mention, there is no silver bullet that I know of, but users do have a few reasonably good options available: * Use the new JDK multi-version JAR capabilities * Take advantage of Maven's "classifier" functionality to provide multiple artifacts for different JDK versions Also I think it would not be too difficult to create some simple third-party migration aid in the form of an annotation processor which selectively translates certain of the user's Java 8 classes or packages to Java 9 module-private, or vice-versa, using mutli-version JAR output. This is probably what I would do for my various projects to get off the ground quickly. >>> At a higher level, I agree that the package-private access mode has >>> proved to be of limited utility. It has, however, been a fundamental >>> part of the language and the VM from the very beginning. To widen it in >>> such an incompatible fashion now, after twenty years, would make it more >>> difficult to migrate existing code to modules and would risk enabling >>> further security vulnerabilities. >> >> I think this is objectively false. Existing code would continue to have >> the same encapsulation; only new code would have the expanded capability >> and only in reference to other new code. > > What about references from outside a module? If class loaders are used as the representation of a module (for the purposes of this discussion at least), then no code is outside a module, which is highly appealing because it simplifies matters immensely. > Would the default access mode on a Java 9 type mean "module private" if > the code attempting to access that type is itself compiled -target 9, and >"package private" otherwise? I think references to Java 9 class members would have to mean "module private" no matter who is accessing it, as this allows better piece-wise migration to using the new access level (and from my examination and understanding of the OpenJDK bytecode verifier implementation, I suspect this would be simpler to implement as one only has to know what the target class' version is, as opposed to applying a more complex rule on both the source and target versions). In other words, a class of any version accessing a Java 8 class file uses package-private, but a class of any version accessing a Java 9 class file uses module-private. In practice I can think of two places where this would be likely to apply: * In multi-version JAR files, which were, by whatever means, compiled such that a Java 8 class accesses a class which is public for Java 8 but module-private for Java 9 * In the class path, if the class path is implemented as one module and mixed JARs are present In both cases it would seem to be beneficial to migration to allow the legacy Java 8 code to take advantage of the Java 9 access level without recompilation. -- - DML From david.lloyd at redhat.com Tue Dec 15 17:50:35 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 15 Dec 2015 11:50:35 -0600 Subject: Coupling modules and class loaders In-Reply-To: <20151214194413.C698A85311@eggemoggin.niobe.net> References: <56276E91.5060605@redhat.com> <20151210012745.EFD0F849EA@eggemoggin.niobe.net> <56699CB0.6020704@redhat.com> <20151214194413.C698A85311@eggemoggin.niobe.net> Message-ID: <567052EB.4080403@redhat.com> On 12/14/2015 01:44 PM, mark.reinhold at oracle.com wrote: > 2015/12/10 7:39 -0800, david.lloyd at redhat.com: >> On 12/09/2015 07:27 PM, mark.reinhold at oracle.com wrote: >>> 2015/10/21 3:53 -0700, david.lloyd at redhat.com: >>>> ... >>>> >>>> I propose that modules are best defined as a specialization of class >>>> loaders. Class loaders already have many of the properties that define >>>> a module: encapsulation, isolation, unique identity for packages. In >>>> fact OSGi already uses this unit. The only thing they are really >>>> missing is the ability to use a graph-like dependency structure, hence >>>> the specialization. >>> >>> Class loaders do not provide strong encapsulation. How do you propose to >>> extend them to do so? >> >> Sure they do - here are some ways: >> >> * Package-private and protected members in one class loader are >> inaccessible to packages in another class loader, even if the package >> has the same name. > > (Yes, that's strong encapsulation, but not strong encapsulation of modules > or of module-like entities, which is what I meant.) Ah, well I was looking at it from the other perspective: with the exception that this restriction *also* applies by package name, this is exactly per-class-loader encapsulation. But that's all the other thread I guess. >> * Run-time linkage is strongly encapsulated by class loader boundary; I >> can declare a public type that nevertheless cannot be linked against by >> anyone, if the class loader denies such access. There are several use >> cases for this. > > That is encapsulation of a sort, but it's not strong since it works > differently with respect to reflection (which, yes, is the approach that > you advocate). I guess there's a definition of "encapsulation" and "strong encapsulation" implicit here. I am thinking of strength as relative though: we can take the existing level of encapsulation, and build upon it in various ways, but in the end we need to agree what our final compatibility and security requirements are. I think that the answer to both is that we should provide a means of encapsulation which provides the user with the ability to secure things at a module level, while also maintaining existing behaviors to a maximal degree. >>> The primary motivation is, as R?mi noted, to make it easy for existing >>> applications that use class loaders in sophisticated (and sometimes very >>> strange) ways to adopt modules. If the maintainers of such applications >>> must make deep changes to their class-loading architectures in order to >>> support modules then some of them simply won't bother. >> >> I don't think they would though. Even in the complete compatibility >> case, the "flat classpath" module could and would still just be one >> module with the whole class path on it, which would be not unlike Ye >> Olde Application Class Loader, while still allowing the forward >> migration path of adding real module dependencies to the ugly legacy >> class path. ... If people are doing weird/nasty/ugly stuff with class >> loaders, they would certainly be free to continue to do so if they want; >> the per-class loader idea does not necessarily require that the module >> class loader class be final, nor does it forbid it from being used as a >> parent or delegate, unless you're thinking of something else? > > What I'm thinking of is an existing application with its own non-trivial > class-loader architecture which, due to compatibility constraints, cannot > be changed. If at least one of its class loaders loads classes from > logically-distinct components that are later converted into corresponding > modules then upgrading the application to load those components as > modules would, under your approach, require replacing that one class > loader with many class loaders, which would break compatibility. > > Imagine, e.g., a Java EE application server that loads implementations of > all the standard EE components using one class loader. Imagine further > that existing users (for good or for ill) depend upon that. If a future > version of Java EE defines those components as modules then the only way > to arrange for this app server to load them as such would be to change > its class-loader architecture in a way that would break existing uses. > > (This is, roughly, the situation with Oracle's WebLogic app server, as > I've mentioned previously.) With a multi-class-loader arrangement, it should be possible to emulate any single-class-loader arrangement, in two ways and with one caveat that I know of: * You can create modules which provide "cut down" views of another module, such that the module appears as multiple modules but with one initiating class loader for all classes actually loaded from them; * You can create a module which aggregates multiple modules into one, as long as no code relies on the the specific relationship between the initiating class loaders of classes within the module (as the initiating class loader will reflect the module in which the class was defined; this is the caveat I was referring to above). (While these capabilities are not unlike those provided by Jigsaw, I think it is useful to enumerate and reconsider them freshly in the context of class-loader-oriented modules, if you're wondering why I might be saying something already well-understood.) In the JDK itself, there's the special case of the null return of Class.getClassLoader() being used for legacy security checks, for code that depends on this, though there are multiple ways that I know of in which this could potentially be mitigated, if this is a problem that you are thinking of and you want to discuss it. What do you think about these options in terms of WebLogic? Are there additional cases that cannot be covered by fixes in these two areas? Are there other compatibility cases beyond these that you have in mind where using multiple class loaders for either platform components or for Java EE specification modules could break existing behaviors, or does the combination of the change of initiating class loader and behavior of Class.getClassLoader() summarize the potential issues that you see? We made this exact change several years ago, though at the time our decision was that in this case, the internal class loader arrangement of the deployment, application server, and platform classes was not a primary compatibility factor (particularly given the language in the Java EE specification which specifically addresses reliance on the types or arrangements of class loaders), as long as certain of our spec- and non-spec-derived existing behaviors remained the same (e.g. reliance on TCCL to load classes and resources). This admittedly made the process much simpler. >>> The fact that OSGi and JBoss have successfully used class loaders to >>> implement module systems that are capable of loading a wide variety of >>> existing artifacts is interesting. It does not, however, imply that all >>> existing applications can be converted to use the class-loader-per-module >>> approach, nor that they should be forced to do so. The proposed design >>> therefore neither mandates nor forbids that approach. >> >> I think we probably would not be able to put bounds on this without >> enumerating specific use cases/behaviors (especially those that would >> fail in this situation but not in the other), as I haven't encountered >> nor been able to imagine a situation that would fail solely due to the >> introduction of class loader based modules. > > See above. -- - DML From david.lloyd at redhat.com Tue Dec 15 18:10:25 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 15 Dec 2015 12:10:25 -0600 Subject: Module descriptions versus module descriptors In-Reply-To: <20151214194313.C4C8E8530F@eggemoggin.niobe.net> References: <5626CDF7.5060605@redhat.com> <20151210012645.EDDB1849E4@eggemoggin.niobe.net> <5669940E.60902@redhat.com> <20151214194313.C4C8E8530F@eggemoggin.niobe.net> Message-ID: <56705791.2050502@redhat.com> On 12/14/2015 01:43 PM, mark.reinhold at oracle.com wrote: > 2015/12/10 7:02 -0800, david.lloyd at redhat.com: >> On 12/09/2015 07:26 PM, mark.reinhold at oracle.com wrote: >>> If module boundaries, however they're expressed, are to be enforced by >>> the compiler and the VM -- as you've agreed elsewhere -- then their >>> manner of expression is inevitably going to be a topic for the Java >>> language and VM specifications. This is not avoidable. >> >> Actually that does not logically follow. I could, for example, >> "express" module boundaries in the VM by creating a hard equivalence >> between modules and class loaders, and in the compiler via command-line >> arguments. In this way, no langauge or VM specifications must change; >> in fact only the specification of the compiler tool itself *must* >> change. > > If every module is represented at run time by a unique class loader, then > would the VM to do anything differently than today? Would it have any > role at all to play in enforcing module boundaries, regardless of their > means of expression? No, other than if the access mode change I described separately was made. > If we widen the default access mode to mean module-private in some > circumstances, as you suggest, then wouldn't that require revisions to > the VM and language specifications? Yes, it will. To clarify my position though, I don't see this feature as a necessary consequence of classloader-based modularity though, or vice-versa; I see them as independent (but mutually beneficial) ideas, which could already be applied to the existing module systems of today. > If module boundaries are communicated to the compiler via command-line > flags, then how do you propose to make all Java compilers work in the > same way? Bear in mind that there is no specification for the compiler > as a tool -- there is only the Java Language Specification, which to date > has never mandated any particular command-line flags, nor even the > existence of a command line upon which to place such flags. I don't think it becomes any more necessary to do so than it already is today. The compiler is already expected (for example) to have a mechanism by which it can be told where to find classes to link against. I do not think that establishing the grouping of these classes is a large leap from that, nor even establishing "friend" relationships, for that matter, if that kind of thing is ultimately in the cards. This information could even have the capability to be externalized into a structured configuration file - but that configuration would be consumed by the tooling, not the language, and more than likely it would be produced ephemerally at build time by a higher-level build tool like Maven or Gradle in the vast majority of cases (not unlike how these tools handle classpaths today), as they are almost always in the best position to establish this kind of boundary. From discussions I've seen on and off various lists, I think that in this case many projects using such tools would quickly adopt a hybrid approach where some of this metadata comes from build-time/source-retention annotations, some from existing language structures and access directives, some from the build environment, and some from the expected deployment environment. Placing this responsibility on the build tooling is also important to the other issues I've previously mentioned relating to mutability and multiplicity of the installation environment. -- - DML From mark.reinhold at oracle.com Thu Dec 17 16:26:14 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 17 Dec 2015 08:26:14 -0800 (PST) Subject: Coupling modules and class loaders In-Reply-To: <567052EB.4080403@redhat.com> References: <56276E91.5060605@redhat.com>, <20151210012745.EFD0F849EA@eggemoggin.niobe.net>, <56699CB0.6020704@redhat.com>, <20151214194413.C698A85311@eggemoggin.niobe.net>, <567052EB.4080403@redhat.com> Message-ID: <20151217162614.4CA4185A7C@eggemoggin.niobe.net> 2015/12/15 9:50 -0800, david.lloyd at redhat.com: > On 12/14/2015 01:44 PM, mark.reinhold at oracle.com wrote: >> ... >> >> What I'm thinking of is an existing application with its own non-trivial >> class-loader architecture which, due to compatibility constraints, cannot >> be changed. If at least one of its class loaders loads classes from >> logically-distinct components that are later converted into corresponding >> modules then upgrading the application to load those components as >> modules would, under your approach, require replacing that one class >> loader with many class loaders, which would break compatibility. >> >> Imagine, e.g., a Java EE application server that loads implementations of >> all the standard EE components using one class loader. Imagine further >> that existing users (for good or for ill) depend upon that. If a future >> version of Java EE defines those components as modules then the only way >> to arrange for this app server to load them as such would be to change >> its class-loader architecture in a way that would break existing uses. >> >> (This is, roughly, the situation with Oracle's WebLogic app server, as >> I've mentioned previously.) > > With a multi-class-loader arrangement, it should be possible to emulate > any single-class-loader arrangement, in two ways and with one caveat > that I know of: > > * You can create modules which provide "cut down" views of another > module, such that the module appears as multiple modules but with one > initiating class loader for all classes actually loaded from them; I think you mean one defining loader, not one initiating loader. If loader M actually loads a module, and loaders X and Y define "cut down" views of M by delegating to M, then all three loaders can be initiating loaders of classes in M but only M will be the defining loader of such classes. > * You can create a module which aggregates multiple modules into one, as > long as no code relies on the the specific relationship between the > initiating class loaders of classes within the module (as the initiating > class loader will reflect the module in which the class was defined; > this is the caveat I was referring to above). Again, I think you mean defining loader. If loaders M1 and M2 load modules, and loader X aggregates those into a single apparent module, then M1 and M2 will be defining loaders and X will be an initiating loader. > In the JDK itself, there's the special case of the null return of > Class.getClassLoader() being used for legacy security checks, for code > that depends on this, though there are multiple ways that I know of in > which this could potentially be mitigated, if this is a problem that you > are thinking of and you want to discuss it. > > What do you think about these options in terms of WebLogic? Are there > additional cases that cannot be covered by fixes in these two areas? > Are there other compatibility cases beyond these that you have in mind > where using multiple class loaders for either platform components or for > Java EE specification modules could break existing behaviors, or does > the combination of the change of initiating class loader and behavior of > Class.getClassLoader() summarize the potential issues that you see? To make an approach along these lines work for the JDK, for WebLogic, and for any other application with a non-trivial, non-loader-per-module class-loader architecture, I think you're going to have to fundamentally redefine the meaning of the Class::getClassLoader method. It would no longer return the defining loader of a class but, rather, an initiating, aggregating loader as you describe above. Given the roles of class loaders in both VM-level access checking and SecurityManager-level permission checking, this is far from being a simple change. To take just one case, suppose a class C is loaded and defined by a module class loader M1, which itself is aggregated together with some other module class loader M2 by an aggregating loader X, so if I have C's Class object then its getClassLoader method will return X. Suppose further that X is an instance of an end-user-supplied subclass of a ClassLoader subclass provided by the application, long ago, as part of its supported API. When one of the defineClass methods of X is invoked to define some new class D, what will be the defining loader of D? If the defining loader is not X, then X must somehow delegate class definition to either M1 or M2, but which one? If D's package is already defined to just one of those loaders then maybe defining it with that loader would (often) be the right thing. What would X do if D's package is split across M1 and M2, or would you disallow that? What would X do if D's package is not yet defined in either M1 or M2, just define D with X? That might work in some cases but not if some code later on depends upon D having been defined by M1 specifically. If the defining loader is X, rather than M1 or M2, then D would have a different defining loader relative to classes now in M1 or M2 than it did when run on pre-9 releases. If D is meant to have package-private access to some class E in the same package, whose defining loader was X but is now M1 due to modularization, then it will fail, since D's run-time package will be different from that of E, since its defining loader is different. (You could consider an even deeper change and say that X is the defining loader, not just an initiating loader, of all classes in both M1 and M2. Module-private types in those modules would then, however, no longer be strongly encapsulated.) So, that's just one case of one family of methods, defineClass, in the existing ClassLoader API. To change the meaning of Class::getClassLoader would require a deep analysis of all the methods in ClassLoader, not just this one. Maybe there's some super-clever way to make them all work in a manner that's both sensible and compatible, but I don't know what that is and I can't believe it'd be straightforward. This kind of complexity is exactly why we abandoned the module-per-loader approach implemented in the first Jigsaw prototype. - Mark From mark.reinhold at oracle.com Thu Dec 17 16:27:14 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 17 Dec 2015 08:27:14 -0800 (PST) Subject: My position on module security/access control In-Reply-To: <567044FF.5050209@redhat.com> References: <564CA2F6.10406@redhat.com>, <20151210012945.F3B09849F4@eggemoggin.niobe.net>, <5669A192.1060509@redhat.com>, <20151214194513.C87D885313@eggemoggin.niobe.net>, <567044FF.5050209@redhat.com> Message-ID: <20151217162714.4EA3A85A81@eggemoggin.niobe.net> 2015/12/15 8:51 -0800, david.lloyd at redhat.com: > On 12/14/2015 01:45 PM, mark.reinhold at oracle.com wrote: >> 2015/12/10 8:00 -0800, david.lloyd at redhat.com: >>> On 12/09/2015 07:29 PM, mark.reinhold at oracle.com wrote: > ... > >> Okay, I think I'm starting to get it, but in your statement >> >>>>> The concept being established here is the >>>>> ability to have a public class which is not exported but is still >>>>> accessible to anyone. >> >> the clause "is not exported" suggests that some public types are >> "exported" and some are not. I read this to mean that public types would >> still be subject to some sort of access control based upon some kind of >> export declarations, but only for static linkage. If that's not the >> case, then what's the distinction between "not exported" and "exported"? > > Right though I prefer the existing "visibility" term for this, as this > term is well over a decade old and reasonably well-understood at this point. Just to confirm: In your preferred approach an exported public type would be made visible to class loaders other than its own for the purpose of static linkage, but a non-exported public type would not, and whether a public type is exported would have no bearing upon the access-control decisions for that type made by both the VM and the reflection APIs. Is that right? >>>> If a Java 9 class is not inside a module, i.e., it's on the class path, >>>> then what is the default access mode? If it's package-private then it >>>> would work differently than from within a module. If it's private to the >>>> unnamed module in which all classes found on the class path are defined >>>> then it would work differently from any pre-9 classes on the class path. >>>> Either outcome is likely to confuse people. >>> >>> Class path private; see my other mail. >> >> Okay. Suppose that I upgrade a pre-9 component into a module using this >> approach, and ship it as a modular JAR file so that it can be used both >> as a module and on the class path. If it's used on the class path then >> any types that formerly were package-private are now accessible to all >> other code on the class path, regardless of whether that other code was >> compiled with 9 or with an earlier release, right? > > Yes, potentially, if the class path is held as a single module (which it > could be, but does not have to be). There are pros and cons to each > approach though. If all classes on the class path are not loaded into a single module then what other approach do you have in mind? >>>> ... >>>> >>>> At a higher level, I agree that the package-private access mode has >>>> proved to be of limited utility. It has, however, been a fundamental >>>> part of the language and the VM from the very beginning. To widen it in >>>> such an incompatible fashion now, after twenty years, would make it more >>>> difficult to migrate existing code to modules and would risk enabling >>>> further security vulnerabilities. >>> >>> I think this is objectively false. Existing code would continue to have >>> the same encapsulation; only new code would have the expanded capability >>> and only in reference to other new code. >> >> What about references from outside a module? > > If class loaders are used as the representation of a module (for the > purposes of this discussion at least), then no code is outside a module, > which is highly appealing because it simplifies matters immensely. Of course -- I was asking about references from other modules. >> Would the default access mode on a Java 9 type mean "module private" if >> the code attempting to access that type is itself compiled -target 9, and >> "package private" otherwise? > > I think references to Java 9 class members would have to mean "module > private" no matter who is accessing it, as this allows better piece-wise > migration to using the new access level (and from my examination and > understanding of the OpenJDK bytecode verifier implementation, I suspect > this would be simpler to implement as one only has to know what the > target class' version is, as opposed to applying a more complex rule on > both the source and target versions). In other words, a class of any > version accessing a Java 8 class file uses package-private, but a class > of any version accessing a Java 9 class file uses module-private. What about existing source code that's recompiled -target 9? Suppose, e.g., that I maintain a library that dynamically generates classes whose bytecodes depend upon user input (say, an XSLT compiler), and that I load those classes into the same class loader as my library (arguably a bad practice, but I've seen this before). The synthesized classes are in their own special package, rather than in one of my library's packages, so today they cannot access any package-private members of the library itself. If I recompile my library -target 9 then that will no longer be true. Do we really want the simple act of recompiling old code to open the door to new security vulnerabilities? - Mark From david.lloyd at redhat.com Thu Dec 17 17:12:00 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 17 Dec 2015 11:12:00 -0600 Subject: My position on module security/access control In-Reply-To: <20151217162714.4EA3A85A81@eggemoggin.niobe.net> References: <564CA2F6.10406@redhat.com> <20151210012945.F3B09849F4@eggemoggin.niobe.net> <5669A192.1060509@redhat.com> <20151214194513.C87D885313@eggemoggin.niobe.net> <567044FF.5050209@redhat.com> <20151217162714.4EA3A85A81@eggemoggin.niobe.net> Message-ID: <5672ECE0.3030107@redhat.com> On 12/17/2015 10:27 AM, mark.reinhold at oracle.com wrote: > 2015/12/15 8:51 -0800, david.lloyd at redhat.com: >> On 12/14/2015 01:45 PM, mark.reinhold at oracle.com wrote: >>> 2015/12/10 8:00 -0800, david.lloyd at redhat.com: >>>> On 12/09/2015 07:29 PM, mark.reinhold at oracle.com wrote: >> ... >> >>> Okay, I think I'm starting to get it, but in your statement >>> >>>>>> The concept being established here is the >>>>>> ability to have a public class which is not exported but is still >>>>>> accessible to anyone. >>> >>> the clause "is not exported" suggests that some public types are >>> "exported" and some are not. I read this to mean that public types would >>> still be subject to some sort of access control based upon some kind of >>> export declarations, but only for static linkage. If that's not the >>> case, then what's the distinction between "not exported" and "exported"? >> >> Right though I prefer the existing "visibility" term for this, as this >> term is well over a decade old and reasonably well-understood at this point. > > Just to confirm: In your preferred approach an exported public type would > be made visible to class loaders other than its own for the purpose of > static linkage, but a non-exported public type would not, and whether a > public type is exported would have no bearing upon the access-control > decisions for that type made by both the VM and the reflection APIs. > > Is that right? That is correct. >>>>> If a Java 9 class is not inside a module, i.e., it's on the class path, >>>>> then what is the default access mode? If it's package-private then it >>>>> would work differently than from within a module. If it's private to the >>>>> unnamed module in which all classes found on the class path are defined >>>>> then it would work differently from any pre-9 classes on the class path. >>>>> Either outcome is likely to confuse people. >>>> >>>> Class path private; see my other mail. >>> >>> Okay. Suppose that I upgrade a pre-9 component into a module using this >>> approach, and ship it as a modular JAR file so that it can be used both >>> as a module and on the class path. If it's used on the class path then >>> any types that formerly were package-private are now accessible to all >>> other code on the class path, regardless of whether that other code was >>> compiled with 9 or with an earlier release, right? >> >> Yes, potentially, if the class path is held as a single module (which it >> could be, but does not have to be). There are pros and cons to each >> approach though. > > If all classes on the class path are not loaded into a single module then > what other approach do you have in mind? Another option is to load each artifact into its own class loader, tying each one to each other one. Or some hybrid approach wherein some artifacts are joined, and some are not. I prefer the single-module approach though, as that most closely mirrors existing behavior. But see below... >>>>> ... >>>>> >>>>> At a higher level, I agree that the package-private access mode has >>>>> proved to be of limited utility. It has, however, been a fundamental >>>>> part of the language and the VM from the very beginning. To widen it in >>>>> such an incompatible fashion now, after twenty years, would make it more >>>>> difficult to migrate existing code to modules and would risk enabling >>>>> further security vulnerabilities. >>>> >>>> I think this is objectively false. Existing code would continue to have >>>> the same encapsulation; only new code would have the expanded capability >>>> and only in reference to other new code. >>> >>> What about references from outside a module? >> >> If class loaders are used as the representation of a module (for the >> purposes of this discussion at least), then no code is outside a module, >> which is highly appealing because it simplifies matters immensely. > > Of course -- I was asking about references from other modules. Ah, apologies, I interpreted the question differently. I can't envision references from other modules having any access to module private members at all, unless a friend mechanism were introduced and was in use. But, see below... >>> Would the default access mode on a Java 9 type mean "module private" if >>> the code attempting to access that type is itself compiled -target 9, and >>> "package private" otherwise? >> >> I think references to Java 9 class members would have to mean "module >> private" no matter who is accessing it, as this allows better piece-wise >> migration to using the new access level (and from my examination and >> understanding of the OpenJDK bytecode verifier implementation, I suspect >> this would be simpler to implement as one only has to know what the >> target class' version is, as opposed to applying a more complex rule on >> both the source and target versions). In other words, a class of any >> version accessing a Java 8 class file uses package-private, but a class >> of any version accessing a Java 9 class file uses module-private. > > What about existing source code that's recompiled -target 9? > > Suppose, e.g., that I maintain a library that dynamically generates > classes whose bytecodes depend upon user input (say, an XSLT compiler), > and that I load those classes into the same class loader as my library > (arguably a bad practice, but I've seen this before). The synthesized > classes are in their own special package, rather than in one of my > library's packages, so today they cannot access any package-private > members of the library itself. If I recompile my library -target 9 > then that will no longer be true. > > Do we really want the simple act of recompiling old code to open the > door to new security vulnerabilities? I see your point; the recompilation case is definitely problematic, and I know of no way to mitigate this. Any approach which reduces security compared to the status quo is probably going to be ultimately unacceptable, at least barring some very good/clever mitigation strategy. Another variation on the idea that preserves most of the strengths of this plan is to keep access levels as-is, but enable a "friend package" mechanism that operates at the package level. Internally (at run time), each package determines which set of classloader+package names have access, and this is checked by the class loader or bytecode verifier, or by some combination of the two. At compile time friends may to be declared somehow, especially the special/probably common case of internal-to-a-module friends (this is something that would have to be worked out, obviously, but it doesn't seem insurmountable to me), but the important part of the idea is remaining compatible with existing reflective code while at the same time increasing the usefulness of package-level access controls. This seems somewhat similar to the current Jigsaw approach in concept, establishing trust relationships between modules, but without making public types globally unavailable. Overall I'm hoping there is a possible and practical strategy which involves (perhaps optionally) opening up non-public levels, rather than non-optionally restricting public levels, with a view towards compatibility and flexibility - and also does not introduce the second dimension of access checking by visibility, for the same reasons. -- - DML From david.lloyd at redhat.com Thu Dec 17 18:20:22 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 17 Dec 2015 12:20:22 -0600 Subject: Coupling modules and class loaders In-Reply-To: <20151217162614.4CA4185A7C@eggemoggin.niobe.net> References: <56276E91.5060605@redhat.com> <20151210012745.EFD0F849EA@eggemoggin.niobe.net> <56699CB0.6020704@redhat.com> <20151214194413.C698A85311@eggemoggin.niobe.net> <567052EB.4080403@redhat.com> <20151217162614.4CA4185A7C@eggemoggin.niobe.net> Message-ID: <5672FCE6.20501@redhat.com> On 12/17/2015 10:26 AM, mark.reinhold at oracle.com wrote: > 2015/12/15 9:50 -0800, david.lloyd at redhat.com: >> On 12/14/2015 01:44 PM, mark.reinhold at oracle.com wrote: >>> ... >>> >>> What I'm thinking of is an existing application with its own non-trivial >>> class-loader architecture which, due to compatibility constraints, cannot >>> be changed. If at least one of its class loaders loads classes from >>> logically-distinct components that are later converted into corresponding >>> modules then upgrading the application to load those components as >>> modules would, under your approach, require replacing that one class >>> loader with many class loaders, which would break compatibility. >>> >>> Imagine, e.g., a Java EE application server that loads implementations of >>> all the standard EE components using one class loader. Imagine further >>> that existing users (for good or for ill) depend upon that. If a future >>> version of Java EE defines those components as modules then the only way >>> to arrange for this app server to load them as such would be to change >>> its class-loader architecture in a way that would break existing uses. >>> >>> (This is, roughly, the situation with Oracle's WebLogic app server, as >>> I've mentioned previously.) >> >> With a multi-class-loader arrangement, it should be possible to emulate >> any single-class-loader arrangement, in two ways and with one caveat >> that I know of: >> >> * You can create modules which provide "cut down" views of another >> module, such that the module appears as multiple modules but with one >> /defining/ class loader for all classes actually loaded from them; > > I think you mean one defining loader, not one initiating loader. If > loader M actually loads a module, and loaders X and Y define "cut down" > views of M by delegating to M, then all three loaders can be initiating > loaders of classes in M but only M will be the defining loader of such > classes. You're right, I used the wrong term, I don't know what I was thinking about. Sorry about that, I've fixed the text to hopefully reduce confusion... >> * You can create a module which aggregates multiple modules into one, as >> long as no code relies on the the specific relationship between the >> /defining/ class loaders of classes within the module (as the /defining/ >> class loader will reflect the module in which the class was defined; >> this is the caveat I was referring to above). > > Again, I think you mean defining loader. If loaders M1 and M2 load > modules, and loader X aggregates those into a single apparent module, > then M1 and M2 will be defining loaders and X will be an initiating > loader. Fixed... > >> In the JDK itself, there's the special case of the null return of >> Class.getClassLoader() being used for legacy security checks, for code >> that depends on this, though there are multiple ways that I know of in >> which this could potentially be mitigated, if this is a problem that you >> are thinking of and you want to discuss it. >> >> What do you think about these options in terms of WebLogic? Are there >> additional cases that cannot be covered by fixes in these two areas? >> Are there other compatibility cases beyond these that you have in mind >> where using multiple class loaders for either platform components or for >> Java EE specification modules could break existing behaviors, or does >> the combination of the change of /defining/ class loader and behavior of >> Class.getClassLoader() summarize the potential issues that you see? > > To make an approach along these lines work for the JDK, for WebLogic, > and for any other application with a non-trivial, non-loader-per-module > class-loader architecture, I think you're going to have to fundamentally > redefine the meaning of the Class::getClassLoader method. ...because WebLogic depends on the defining class loader being something in particular? Looking at the WebLogic class loading documentation, it seems like it is less important what the defining loader is, and more important what each (EE) module can "see". In fact there are a number of restrictions which actually seem to play in favor of per-classloader modules: limited nesting depth, a clear delegation pattern (because traditional classloader hierarchy trees are just a subclass of the overall directed-graph relationship of modularity), restriction of customization to web and EJB modules, call-by-value in certain situations, etc. It would appear to me (just from reading the documentation) that WebLogic should already be well-positioned to adopt a classloader-oriented modularity strategy for Java EE, unless there is some problem that stems directly from having differing defining class loaders. > It would no > longer return the defining loader of a class but, rather, an initiating, > aggregating loader as you describe above. Given the roles of class > loaders in both VM-level access checking and SecurityManager-level > permission checking, this is far from being a simple change. > > To take just one case, suppose a class C is loaded and defined by a > module class loader M1, which itself is aggregated together with some > other module class loader M2 by an aggregating loader X, so if I have > C's Class object then its getClassLoader method will return X. Suppose > further that X is an instance of an end-user-supplied subclass of a > ClassLoader subclass provided by the application, long ago, as part of > its supported API. When one of the defineClass methods of X is invoked > to define some new class D, what will be the defining loader of D? > > If the defining loader is not X, then X must somehow delegate class > definition to either M1 or M2, but which one? If D's package is already > defined to just one of those loaders then maybe defining it with that > loader would (often) be the right thing. What would X do if D's package > is split across M1 and M2, or would you disallow that? What would X do > if D's package is not yet defined in either M1 or M2, just define D with > X? That might work in some cases but not if some code later on depends > upon D having been defined by M1 specifically. > > If the defining loader is X, rather than M1 or M2, then D would have a > different defining loader relative to classes now in M1 or M2 than it did > when run on pre-9 releases. If D is meant to have package-private access > to some class E in the same package, whose defining loader was X but is > now M1 due to modularization, then it will fail, since D's run-time > package will be different from that of E, since its defining loader is > different. > > (You could consider an even deeper change and say that X is the defining > loader, not just an initiating loader, of all classes in both M1 and M2. > Module-private types in those modules would then, however, no longer be > strongly encapsulated.) > > So, that's just one case of one family of methods, defineClass, in the > existing ClassLoader API. To change the meaning of Class::getClassLoader > would require a deep analysis of all the methods in ClassLoader, not just > this one. Maybe there's some super-clever way to make them all work in a > manner that's both sensible and compatible, but I don't know what that is > and I can't believe it'd be straightforward. This kind of complexity is > exactly why we abandoned the module-per-loader approach implemented in > the first Jigsaw prototype. Yeah I didn't mean to imply that we should make a fundamental shift in what getClassLoader means; however, I would definitely consider the idea of making small changes to it to be reasonable, if it were just for compatibility with things that use e.g. ClassLoader.getSystemResources() or Class.forName(x, y, null) for example, or which use getClassLoader() == null for security checks; this kind of behavior could optionally be deprecated & removed in the long term. The thrust of my question was intended to be regarding platform and application class loaders. As a hopefully better approach, I'll break it into a few (more specific) hypothetical situations which are centered around 1:1 modules/classloaders. Some seem reasonable, some seem more unlikely but might still be worth exploring for the purposes of thought experimentation. 1. Premise: The Java EE 9 specification establishes a rule wherein each Java EE API is required to be available statically to applications by a module name which corresponds to the API in use. Existing application servers currently present all these APIs as one large unit. Question: is it a problem to present each API as a "cut down"/filtered view of one module which consists the of all those APIs in the same large unit? 2. Premise: Same as #1, except the Java EE 9 specification also requires that each API be actually encapsulated in a separate module. Question: is it a problem that the defining class loader of these classes has changed to be per-API rather than a single one? 3. Premise: Java EE 9 requires that each deployment item presently defined in the EE platform spec as a "module" be established as a module, while otherwise maintaining the existing neutrality about the "types and arrangements" of class loaders (meaning for example that it is the app vendor's choice as to whether each transitive Class-Path dependency becomes a (potentially shared) module, or whether those classes are folded in to the EE module, or some other strategy). Question: does this present a compatibility problem that you can foresee? 4. Premise: Same as #3, however it also introduces a means to determine whether other JARs in a deployment should be established as a module, as well as a means to establish dependencies to and from such modules. Question: does this mitigate or exacerbate any problems from #3? 5. Premise: Java EE 9 requires that every single archive within every application or module be established as a module. Question: how big a disaster would this be, and why? -- - DML