From peter.kriens at aqute.biz Thu Oct 1 08:08:10 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Thu, 1 Oct 2015 10:08:10 +0200 Subject: How will dependency Injection/Annotation scanning work? In-Reply-To: References: <71174D99-BB94-4FC8-BA56-9BCED053E7CF@aqute.biz> <560979F2.8010307@redhat.com> <6D9ACFC7-45C3-46E8-B958-FBCBB4C25109@aqute.biz> Message-ID: <11253550-FCDB-42B8-98F2-0713D4F6F5CD@aqute.biz> Paul, Agree. We have experience with this in OSGi with the maven bundle plugin based on bnd. This plugin can generate indexes of classes of packages based on annotations or other indexing criteria. These indexes are generally placed in the manifest. Standardizing such indexes would be very useful. Kind regards, Peter Kriens > On 30 sep. 2015, at 16:02, Paul Benedict wrote: > > If the module-info was not in binary format, it could be possible to list the annotations in the module meta data. This is just another reason to make the "module-info" an open format. A build tool could easily do this index (like Maven) at packaging time. > > > Cheers, > Paul > > On Wed, Sep 30, 2015 at 2:33 AM, Peter Kriens > wrote: > Thanks for the answer David about the annotation scanning. I guess this still will have to undergo a more detailed design? > > Kind regards, > > Peter Kriens > > > On 28 sep. 2015, at 19:33, David M. Lloyd > wrote: > > > > On 09/28/2015 12:05 PM, Peter Kriens wrote: > >> Dependency injection subsystems (Guice,CDI, Spring) tend to require annotation scanning. This should be able to iterate over implementation classes that are not exported. > >> > >> I could not find how you could iterate over the classes in a module. If there is no such way, how will CDI work with modules? > > > > This one I can answer, at least partially: modules are likely to be indexed or contain an index of annotated classes. The index is presumably keyed by any annotation annotated in turn with the @Indexed annotation (package TBD I believe). The following excerpt from the requirements doc (heading "Development") is relevant: > > > >> One potential approach is to augment a module?s definition with an index of the annotations that are present in the module, together with an indication of the elements to which each annotation applies. To limit the size of the index, only annotations which themselves are annotated with a new meta-annotation, say @Indexed, would be included. > > > > > > -- > > - DML > > From peter.kriens at aqute.biz Mon Oct 5 12:54:02 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 5 Oct 2015 14:54:02 +0200 Subject: How will dependency Injection/Annotation scanning work? In-Reply-To: References: <71174D99-BB94-4FC8-BA56-9BCED053E7CF@aqute.biz> <560979F2.8010307@redhat.com> <6D9ACFC7-45C3-46E8-B958-FBCBB4C25109@aqute.biz> Message-ID: <260BDAC1-4704-4091-B6C4-9A3165AAECFE@aqute.biz> It could of course also be stored in the class file format since it is extensible. However, in all those use cases it is not compiled by the java compiler but by (something like) the jar tool. Looking at the state of the design of this aspect I guess this will not make it in Java 9? Kind regards, Peter Kriens > On 30 sep. 2015, at 16:02, Paul Benedict wrote: > > If the module-info was not in binary format, it could be possible to list the annotations in the module meta data. This is just another reason to make the "module-info" an open format. A build tool could easily do this index (like Maven) at packaging time. > > > Cheers, > Paul > > On Wed, Sep 30, 2015 at 2:33 AM, Peter Kriens > wrote: > Thanks for the answer David about the annotation scanning. I guess this still will have to undergo a more detailed design? > > Kind regards, > > Peter Kriens > > > On 28 sep. 2015, at 19:33, David M. Lloyd > wrote: > > > > On 09/28/2015 12:05 PM, Peter Kriens wrote: > >> Dependency injection subsystems (Guice,CDI, Spring) tend to require annotation scanning. This should be able to iterate over implementation classes that are not exported. > >> > >> I could not find how you could iterate over the classes in a module. If there is no such way, how will CDI work with modules? > > > > This one I can answer, at least partially: modules are likely to be indexed or contain an index of annotated classes. The index is presumably keyed by any annotation annotated in turn with the @Indexed annotation (package TBD I believe). The following excerpt from the requirements doc (heading "Development") is relevant: > > > >> One potential approach is to augment a module?s definition with an index of the annotations that are present in the module, together with an indication of the elements to which each annotation applies. To limit the size of the index, only annotations which themselves are annotated with a new meta-annotation, say @Indexed, would be included. > > > > > > -- > > - DML > > From mark.reinhold at oracle.com Mon Oct 5 18:45:23 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 5 Oct 2015 11:45:23 -0700 (PDT) Subject: Security In-Reply-To: <55FADD8D.5080807@redhat.com> References: <1088108454.2110963.1441991735882.JavaMail.zimbra@u-pem.fr>, <20150917145611.158B371D1B@eggemoggin.niobe.net>, <55FADD8D.5080807@redhat.com> Message-ID: <20151005184523.1EDD47B92D@eggemoggin.niobe.net> 2015/9/17 8:34 -0700, david.lloyd at redhat.com: > On 09/17/2015 09:56 AM, mark.reinhold at oracle.com wrote: >> 2015/9/11 10:15 -0700, forax at univ-mlv.fr: >>> While this change may improve the security, it's a >>> backward incompatible change is not strictly required to support >>> modules it's more an enhancement of the current security model and i >>> don't think it's a good idea to mix it with the introduction of the >>> module support. >> >> This is not an enhancement of the security model per se (which would, >> anyway, be beyond the scope of this JSR). This change is, rather, in >> service of the strong-encapsulation goal (which does, of course, help >> ensure security). > > I think this assertion should be substantiated with some examples. > > While strong encapsulation is good in terms of design principles, to me > it's only evident that this somehow leads to better security in the most > hand-wavy possible way (in particular, as it relates to this specific > mechanism). Three of the five zero-day vulnerabilities reported since JDK 7 GA would have been prevented if we'd had the ability to strongly encapsulate JDK-internal packages in the manner proposed, via access-control checks enforced by the VM. Here are public analyses of two of them: https://partners.immunityinc.com/idocs/Java%20MBeanInstantiator.findClass%200day%20Analysis.pdf http://immunityproducts.blogspot.com/2012/08/java-0day-analysis-cve-2012-4681.html The lead of Oracle's Java Vulnerability Team estimates that at least a third of all the vulnerabilities reported since JDK 7 GA would have been prevented if we'd had the ability to strongly encapsulate JDK-internal packages. > On the other hand, there is strong evidence to support the > assertion that enhancing the security model in any way immediately leads > to new CVEs. The more behavioral rules there are in place, the more > vectors for exploitation will appear. In the abstract, I agree with you. The proposed means of encapsulation, however, does not depend upon the existing complex security architecture, so it should be easier to validate. > ... > >> To truly support strong encapsulation would require taking setAccessible >> away completely. I think that's desirable in the (very) long term, but >> it would break too much existing code in the near term. > > I don't agree that this is true. Encapsulation is really a social > construct; while accessibility can be used to create and enforce > encapsulation-justified rules, it doesn't *have* to. You do need some > kind of back door, no matter what, or else you severely limit the power > and capability of the platform (as evidenced by the widespread usage of > the reflection back door by many widely-used frameworks generally > considered to be powerful and capable). > > If encapsulation as a design principle is the goal, then simple > isolation rules between modules (as between unrelated class loaders) has > already been proven to be highly effective. If all you mean by "encapsulation" is a social construct then sure, none of this would be necessary. That's insufficient, however, in the face of actual security threats. Aside from security considerations, a social contract implies a level of understanding and trust that is difficult, if not impossible, to achieve in a broad community of millions of developers. > The relationship between > this new encapsulation rule and security, on the other hand, has yet to > be shown. (See above.) > In this light, I don't see how these changes cannot be considered a > modification of the security model just because they ride in on the > coattails of a new abstraction. It depends upon what you mean by "security model". If you mean some high-level, abstract view of all the various mechanisms in the language, VM, and libraries that help ensure security then yes, the proposed changes are an extension of that model. When I write "security model", however, I usually mean the subsystem which evolved from the simple (and simplistic) java.lang.SecurityManager API in Java 1.0 to the complex policy/permission-based architecture of the 1.2 release. Many run-time operations (in class loaders, reflection, I/O, etc.) are subject to checks done by the security subsystem, and if just one of those checks is missing or incorrect then it may well be "game over". The proposed enforcement of module boundaries via access restrictions expressed in the language, recorded in class files, and enforced by the compiler and the VM do not in any way depend upon that subsystem, nor do they extend it. The VM will never upcall into the security subsystem in order to determine whether one type is permitted to access another. The implementation of access-control checks in a JVM is typically highly localized and much easier to validate than the comparatively large body of code in the (mostly) Java-level security subsystem. >> ... >> >> ... If encapsulation is to mean >> anything then it should not be possible to break it solely from within >> the language itself. Enabling such powers via an external, second-class >> mechanism such as a command-line option or a debugger is fine, but the >> history of Java has shown that if it's easy to break encapsulation then >> people will do so, increasing everyone's maintenance burdens over the >> long haul. > > I think this may be a somewhat narrow view. People break encapsulation > when they need a capability that cannot be provided another way. Now > the JDK, unfortunately, historically contains very many "goodies" and > bits of useful functionality that users want access to, so it is a > disproportionately popular target for such breakage, but apart from > accessing JDK internals, I believe that the use cases for breaking in > through the security model are legitimate and should be allowed to > adequately privileged code. How do you identify "adequately privileged code"? (This is an open issue we need to address.) > Using an appropriate security model, it is > not difficult to ensure that such powers are not exploited (though while > in practice it is more difficult due to a variety of factors, I do not > believe those factors are directly related to this mechanism but rather > to exceptions, special "holes", and other technical debt that exists in > the JDK itself for historical or other reasons). I don't think that the future of the platform would be well-served by a model that works well only for code outside of the JDK, if that's what you mean. - Mark From mark.reinhold at oracle.com Mon Oct 5 18:46:23 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 5 Oct 2015 11:46:23 -0700 (PDT) Subject: Revisiting encapsulation requirement In-Reply-To: <55FAD78D.9070306@redhat.com> References: <55F80883.9010603@redhat.com>, <20150917145711.195E771D21@eggemoggin.niobe.net>, <55FAD78D.9070306@redhat.com> Message-ID: <20151005184623.211A27B932@eggemoggin.niobe.net> 2015/9/17 8:09 -0700, david.lloyd at redhat.com: > On 09/17/2015 09:57 AM, mark.reinhold at oracle.com wrote: >> 2015/9/15 5:01 -0700, david.lloyd at redhat.com: >> ... >> >>> But, >>> even assuming this could somehow be patched over, I think we should >>> consider dropping this requirement; I have two primary reasons for this >>> (though there may be others as well). >>> >>> The first reason is that without a security manager (or maybe even >>> *with* a security manager - but that's another discussion), it is >>> inevitable that any security measure used to protect this mechanism will >>> ultimately be bypassed, rendering its security value useless; adding >>> more complexity to the system to do so will only increase the overall >>> vulnerability of the platform. >> >> Yes, security is hard. That's not a reason to give up on it. > > Of course not. But as far as I can tell, it has yet to be shown that > this requirement and its corresponding implementation changes actually > improve security in any real way. I'm not talking in hand-wavy > generalizations like "encapsulation leads to better security", I'm > talking about brass-tacks examples. Please see my reply to your other message, with the subject "Security". >>> The second reason is that it is often useful to gain access to and >>> inspect public classes that are not necessarily visible from your >>> module. Frameworks do this commonly for example - framework >>> implementations will almost never import modules that they introspect. >> >> Frameworks in their current form often work as-is on the class path, >> because of the way that unnamed modules work, as discussed in my reply >> to R??mi. See, e.g., Peter Levart's report on the jigsaw-dev list [1]. > > Correct me if I'm wrong, but you seem to be making an assumption here > that no existing artifacts can or should be "graduated" as-is into > modules. On the contrary, I think it's critical to have a smooth migration path, as stated in the requirements. "Smooth" does not, however, mean "without any effort". Depending on what the code in an existing artifact does, and how it relates to other artifacts, it might be very simple to upgrade it into a module, or it might require some effort. I have a draft of a longer note about migration, which I'll try to finish up and post soon. > This is further underscored by this statement: > >> If you convert a framework (or any other code) into a module then you can >> use the Module::addReads method [2][3] to add read edges to the module >> graph at run time. This is, in effect, an API that allows a module to >> `require` additional modules dynamically. > > The compatibility story is getting increasingly foggy here. On the one > hand, for example, "JDK 9 retains this three-level hierarchy, in order > to preserve compatibility" - a hierarchy that as far as I can tell, > very, *very* few existing things actually care about - but on the other > hand, nothing in the Jigsaw module system prototype is compatible in any > way with the class loading world as we know it - which very *very* many > existing things rely on. Can you be more specific? What aspects of the design or the prototype are not compatible with "the class loading world as we know it"? Our intent has been to interfere as little possible with all the myriad ways in which class loaders are used today. - Mark From mark.reinhold at oracle.com Mon Oct 5 18:47:23 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 5 Oct 2015 11:47:23 -0700 (PDT) Subject: Module declarations In-Reply-To: <55FAF7AF.6050109@redhat.com> References: <20150917145211.06B7371D05@eggemoggin.niobe.net>, <55FAF7AF.6050109@redhat.com> Message-ID: <20151005184723.2E6ED7B937@eggemoggin.niobe.net> 2015/9/17 10:26 -0700, david.lloyd at redhat.com: > On 09/17/2015 09:52 AM, mark.reinhold at oracle.com wrote: >> (splitting your initial reply into separate threads) >> >> 2015/9/9 1:48 -0700, david.lloyd at redhat.com: >>> ... >>> >>> 1) The first thing that jumps out at me (again) is making module >>> declarations be a binary construct produced by the actual programming >>> language. It strikes me as arbitrary as well as redundant (given >>> there's a programmatic API to build descriptors, they could simply be >>> parsed from text nearly as well, and a lot more nicely from the user's >>> perspective). As pointed out in the document, it is already expected >>> that additional tooling would synthesize this file at build time, >>> further decreasing the value of such an idea. The module class file >>> just seems pointless to me. >> >> One of our requirements is fidelity across all phases. To achieve this, >> javac (or whatever Java compiler you use) must be able to locate and >> interpret descriptions of modules. If modules are not described by >> language constructs that are compiled into class files, then how and >> where would you suggest that a compiler find such information? > > They could still be bundled with the artifact, but in clear text format > perhaps. Or even in binary format, but in either case, the descriptor > format should be specific to a specific default module configuration > (e.g. an application module loader) rather than be a hard-wired aspect > of all modules. > > To me "fidelity" means the same thing as "integrity": I can build an > artifact and expect it to run on any environment which meets its ABI > requirements in any way. I don't think that hard-coding module > dependency names in the artifact itself is a good way to achieve this > though. Doing so hampers the ability to separate building from > packaging, since if you need to specify different run time environment > parameters in the packaged version, you must crack open and modify the > artifact to do so (new SHA1/MD5 checksums, etc.). > > To me a better way to achieve this is a multi-tiered approach, where > each tier depends on the tier before (but not vice-versa). > > ... I don't think our views are all that far apart -- we just mean different things by the term "fidelity". The fidelity requirement [1] says: Resolution, encapsulation, non-interference, services, and all other aspects of the module system should, to the fullest extent possible, work in exactly the same way at compile time, at run time, and in every other phase of development or deployment. That is, given a particular environment, i.e., a particular module path and one or more initial modules, etc., the module system will behave the same way and enforce the same restrictions in every phase. This is critical to providing a good developer experience. Ensuring that the environment actually is the same at multiple points in time, or "the same enough" within the bounds of acceptable change, is what I understand you to mean by "integrity". Integrity is a critical goal in the configuration and distribution of large systems, but it's beyond the scope of the module system. The module system, as proposed, is intentionally limited to the goals of reliable configuration and strong encapsulation; it is not meant to address the higher-level problems that are more appropriately the domain of build tools, packaging systems, container applications, and so forth. At the language and VM level we need to know a module's name, the names of the modules it depends upon, and the names of the packages it exports. Whether the exact same set of modules is provided at compile time vs. run time is a problem for tools to solve, but if the environments are identical in each phase then the module system will behave identically. This is why version selection is an explicit non-requirement [2]; a version string can be recorded in a module's descriptor, but it's for informational purposes only. It's also why we have requirements that the module system work well with OS-specific packaging systems [3]. We're trying to fit in to the existing ecosystems for building, packaging, and distributing Java components, from Maven all the way up to RPM. We are not trying to replace them. There is a small wrinkle in this story, which is perhaps a source of confusion. The section of SotMS that describes qualified exports [4] mentions the idea of recording, in a module's descriptor, the hashes of the modules that are allowed to depend upon it and use its qualified exports. This feature, which is in service of a stated requirement [5], is meant only for situations, such as with the JDK itself, where a set of tightly-related modules are developed and built together with a high level of trust. It may well prove useful in some non-JDK scenarios but it is not intended to address the general integrity story, which is more appropriately addressed by the kind of multi-tier approach you describe. (I'll add some wording in the next version to make this clear.) * * * Now, to get back to the question of the form and placement of module declarations: To achieve fidelity, i.e., the identical behavior of the module system at compile time and run time given the same environment, a Java compiler must be able to locate and interpret descriptions of modules which contain, at least, the information required (i.e., module names, dependences, exports). A JVM must be able to do the same thing, preferably with the same data. If these essential properties of modules are not described by language constructs that are compiled from source files into class files, then where and in what form would you suggest that a compiler or JVM find such information? Bear in mind that a module being compiled is not yet in an artifact, so whatever form this information takes needs to make sense in, or somehow alongside of, an "exploded" directory tree of source files. A module being compiled against, or even being run, might also be an exploded directory tree of class files rather than an artifact, so that case must also be supported. (See also my reply to Peter on this topic, under the subject line "Why not use the Manifest?".) - Mark [1] http://openjdk.java.net/projects/jigsaw/spec/reqs/#fidelity-across-all-phases [2] http://openjdk.java.net/projects/jigsaw/spec/reqs/#version-selection [3] http://openjdk.java.net/projects/jigsaw/spec/reqs/#packaging [4] http://openjdk.java.net/projects/jigsaw/spec/sotms/#qualified-exports [5] http://openjdk.java.net/projects/jigsaw/spec/reqs/#referential-integrity From mark.reinhold at oracle.com Mon Oct 5 18:48:23 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 5 Oct 2015 11:48:23 -0700 (PDT) Subject: module name convention In-Reply-To: <1052601312.2492216.1442682135412.JavaMail.zimbra@u-pem.fr> References: <1441288605.2461702.1442681029132.JavaMail.zimbra@u-pem.fr>, <1052601312.2492216.1442682135412.JavaMail.zimbra@u-pem.fr> Message-ID: <20151005184823.30B397B93E@eggemoggin.niobe.net> 2015/9/19 10:02 -0700, forax at univ-mlv.fr: > Mani from AdoptAJSR has a very good question at the end of this document: > https://docs.google.com/document/d/1KlumN74IGt-TU-Md3Fn5h4sXHa75RApWNLszUTVp-DE/edit > > ... > > Both Eclipse and Maven use package name or at least package prefix as > convention to name module and as Mani said it really confusing when you > write a module-info file. > > Given that the name of the JDK modules doesn't respect that convention > but still use '.' to separate the different components, I wonder if we > should not try to come with a convention for module name that allow to > easily disambiguate between package name and module name. Yes, I think that's worth exploring. > By example, > java_base, > java-base, > java~base ... Quick reactions: java_base is still a Java identifier, which is nice. java-base corresponds usefully to how JAR files are typically named (e.g., java-base-1.2.3.jar). java~base is pretty ugly (especially with longer names, e.g., jdk~scripting~nashorn~shell). I'm sure there are other possibilities ... > Obviously the other solution is to prefix a name by "package" or > "module". As in, `requires module com.foo` and `exports package com.foo`? - Mark From mark.reinhold at oracle.com Mon Oct 5 18:49:23 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 5 Oct 2015 11:49:23 -0700 (PDT) Subject: Why can modules not be annotated? In-Reply-To: <9C2D961B-0072-4C72-9FA6-A63AF65BFC64@aqute.biz> References: <9C2D961B-0072-4C72-9FA6-A63AF65BFC64@aqute.biz> Message-ID: <20151005184923.394607B943@eggemoggin.niobe.net> 2015/9/28 9:44 -0700, peter.kriens at aqute.biz: > In [lang-vm][1] I found: > > ???A module declaration may not be annotated.??? > > I was surprised that modules (and their clauses) could not be > annotated like package-info.java. Since the source syntax is not > extensible, annotations could have been used to provide meta data for > other module systems. This would mitigate some of the concerns of > using a binary file to encode the module data. We haven't seen a compelling use case (so far) for annotations on module declarations, or on clauses in module declarations, and they aren't a documented requirement, so for simplicity we left them out of the initial design and the prototype. Supporting annotations on module declarations is technically feasible but would add significant complexity to both the specification and its implementations. The machinery to support annotations is everywhere, in javac, the VM, core reflection, the language-model API, etc. Annotations in module declarations would also lead to a natural desire for import statements in module declarations, further complicating matters. Annotations are, fundamentally, a way to make extra-linguistic statements in source code that are carried through the tool chain and (usually) into the resulting class file. Beyond the essential linguistic constructs already in source-level module declarations (requires, exports, provides, and uses), all the other examples we've seen of information that could make sense in the resulting module-descriptor class files don't really make sense in source code, so annotations aren't the right tool for the problem. Version strings, copyright/license notices, project URLs, maintainer contact information, and target operating systems and processor architectures, e.g., are more appropriately maintained in build metadata (e.g., pom.xml), or calculated by the build system on-the-fly, than expressed in source code, and they can easily be added as custom attributes to a module-info.class file after it's compiled. (Whether it's actually a good idea to put all these sorts of information in a module descriptor is a different question.) So yes, we could support annotations on module declarations, and that may indeed be useful for other module systems. It would come at significant cost, however, and we'd have to weigh that cost carefully. - Mark From mark.reinhold at oracle.com Mon Oct 5 18:50:23 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 5 Oct 2015 11:50:23 -0700 (PDT) Subject: Why not use the Manifest? In-Reply-To: References: Message-ID: <20151005185023.443E87B949@eggemoggin.niobe.net> 2015/9/28 9:49 -0700, peter.kriens at aqute.biz: > The encoding of the module data in a binary class file remains hard to > understand and seems to go against the best practices we learned the > hard way over the past decades. (Debugging!) > > Especially since the current proposal leaves the heavy lifting to > build tooling. In the proposal, build tools cannot use the > module-info.java since it does not contain sufficient > information. (Versions being the most glaring omission for a build > too.) Nor is the format extensible to contain build tool specific > information. Ergo, the build tool will have to generate the > module-info.class file. > > This make the class file makes these tools unnecessary hard and slower > because it must either use the compiler as an intermediate step or use > a library like ASM to create the unreadable version of its metadata. > > This is the first time in a very long time that I see a regression to > binary files for meta data. Especially because there is already a good > place that all build tools are already using: the manifest. The reasons for expressing module declarations in source files and compiling them into class files are mentioned in SotMS, but here's a longer take on just this topic. The Java programming language, at present, provides for the definition of three kinds of program components: classes, interfaces, and packages. They are defined in Java source files, and compiled into class files. Such files govern, among other things, the mechanisms of symbolic resolution and access control implemented by every Java compiler and JVM. A developer need not reason about any other type of file in order to understand the actions of these mechanisms. A compiler or JVM need not consume or produce any information other than that found in source and class files in order to implement these mechanisms. The module system extends the mechanism of symbolic resolution with the concept of readability in order to provide reliable configuration. It extends the mechanism of access control, in part by relating it to readability, in order to provide strong encapsulation. To support these extensions, every Java compiler and JVM must be able to locate and interpret descriptions of modules which convey, at least, each module's name, dependences, and exported packages. Such descriptions, regardless of their form, will govern symbolic resolution and access control in both compilers and JVMs, so they will have to be specified in both the Java Language Specification (JLS) and the Java Virtual Machine Specification (JVMS). Regardless of how they are described, modules are, fundamentally, a new kind of Java program component. The present proposal therefore treats them as such, in both the language and the JVM: Module descriptions are expressed and encoded in the same way as the other kinds of information that define program components and govern symbolic resolution and access control -- that is, as Java source and class files. This approach is immediately familiar to developers, who already think about program components, symbolic resolution, and access control in terms of the Java programming language. It is easy to retrofit into the JLS and the JVMS, since those specifications are already centered upon source and class files. It is, finally, straightforward for existing Java implementations and tool chains to support, since they need not be revised to handle an entirely new type of file. We could choose another format for module descriptions such as XML, JSON, YAML, or JAR-file manifests. Such formats may be more convenient for tool maintainers but we'd have to bake that format into the JLS and the JVMS, thereby increasing the complexity of those specifications and their implementations. If the format depends upon external standards (XML, JSON, YAML) then these foundational specifications, and their implementations, would become dependent upon those standards. If the format proves unsuitable over time then replacing it with something else would require major revisions to these specifications, and to their implementations. Source and class files are fundamental to the Java platform, to the JLS and the JVMS, and to their implementations and supporting tools. Other formats come and go; these will be with us forever. * * * As to debugging, in the prototype we've already enhanced the jar tool with an option to print the descriptor of a modular JAR file. I expect similar support to show up in related tools such as jmod and javap, and eventually in IDEs and other tools outside the JDK. As to ease of tooling, most tools will only need to read module descriptors, not write them, and the java.lang.module.ModuleDescriptor class already provides convenient static methods for doing that. Adding similar methods to write descriptors would be straightforward, though it would complicate the API a bit. As to extensibility, class files are already in a precisely-specified and extensible format. If a build tool needs to add information to a module descriptor then it can do so via non-standard class-file attributes, using an existing popular library such as ASM. To make it easier for tools to manipulate module descriptors we could consider enhancing the java.lang.module.ModuleDescriptor class, and related tools, to read and write custom, non-standard class-file attributes. This might be a bit tricky depending on how the attributes are defined, but for simple property-style attributes it's likely straightforward and would obviate the need to use ASM directly. - Mark From mark.reinhold at oracle.com Mon Oct 5 18:51:23 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 5 Oct 2015 11:51:23 -0700 (PDT) Subject: Missing reification of the Module-path? In-Reply-To: <19F2AB39-D4B3-4D50-8C5A-0043BFDE25DB@aqute.biz> References: <19F2AB39-D4B3-4D50-8C5A-0043BFDE25DB@aqute.biz> Message-ID: <20151005185123.461CF7B94E@eggemoggin.niobe.net> 2015/9/28 9:51 -0700, peter.kriens at aqute.biz: > The proposal specifies how to resolve a module from a _module path_, a > module path being one or more directories with modular JARs. > > Since the resolution is without versions, the module-path *must* > contain a consistent set of artifacts. That is, the module path cannot > be a repository like a maven repository it is closely bound to an > executable since for each module only one of its versions can be > chosen. > > For example in maven the module path would consist of the transitive > runtime dependencies. This is a unique set per application per > version. I do not think it is practical to synchronize this set with > other applications so one should assume this is unique and cannot be > shared. It is potentially very large. > > Ergo, the module-path *is* the application. Or, almost equivalently, the module graph resolved from the module path on behalf of the main module is the application. > I therefore wonder if there is a need for a reification of the module > path + main class? Such an artifact could contain an indirection like > a URL to not require creating temporary module path directories for > each application with potentially very large duplicated artifacts. > > Such an artifact could then be made available as an atomic Layer. Hmm. Do you mean that such an artifact would contain copies of all of the modules in the module graph, i.e., would it be a "fat" artifact? Or would it just record references to the necessary modules, presumably with content hashes to ensure integrity? We've explored similar notions in earlier prototypes. An artifact in either of these forms could, in principle, be generated by the linker, but would require some new support at install time and/or run time. It's an interesting idea, though outside the scope of this JSR unless we conclude it's important to standardize the format of such artifacts. - Mark From mark.reinhold at oracle.com Mon Oct 5 18:52:23 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 5 Oct 2015 11:52:23 -0700 (PDT) Subject: Do we need to open each file on the module path to find the enclosed name? In-Reply-To: <5FE37F97-112B-48E5-8784-8D01DB2FF3AB@aqute.biz> References: <5FE37F97-112B-48E5-8784-8D01DB2FF3AB@aqute.biz> Message-ID: <20151005185223.481967B953@eggemoggin.niobe.net> 2015/9/28 9:54 -0700, peter.kriens at aqute.biz: > The module path is defined as: > > `A module path is a sequence of directories containing module > artifacts which are searched, in order, for the first artifact that > defines a suitable module.` > > This seems to indicate that the VM must iterate over all the > directories (I assume not recursive?) Correct, not recursive. > and open the JARs to read the > module-info.class file so that it knows the module name? > > This seems quite expensive for large systems? A JVM need not iterate over every directory in the module path when searching for a module of a particular name. It should scan each directory in order, stopping at the first one that contains a module with that name (and, if desired, building a cache of all the other modules that it found). In principle it will have to open every module artifact in each module-path directory that it searches, since there's no specified relationship between the name of a module and the name of an artifact which might define it. Some simple heuristics can be used to avoid that in typical cases, however, since the name of a module will often be a prefix of the name of its artifact. With modern filesystems, even without an SSD, opening a bunch of files in a directory and reading the first couple KB of each one can be done pretty quickly. (We've run some stress tests along these lines, just to be sure.) - Mark From mark.reinhold at oracle.com Mon Oct 5 18:53:23 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 5 Oct 2015 11:53:23 -0700 (PDT) Subject: Do we actually need module names? In-Reply-To: References: Message-ID: <20151005185323.4DD797B958@eggemoggin.niobe.net> 2015/9/28 9:56 -0700, peter.kriens at aqute.biz: > This specification introduces a completely new namespace for > modules. One of the lessons of the last decade has been that > namespaces are very brittle and error prone. So a valid question seems > to be: what do we get for this relatively high cost? > > The only reason to use a name space is for humans. However, in the > current proposal the names will likely not be used by humans since > they lack a version. I don't follow. Humans already use names, without versions, for many things in Java, and in programming environments more generally. > That is, the module-info.class contains too > little information to be used by tooling as a source for the compile > or runtime paths. It therefore seems inevitable that tools will > generate this resource from their native metadata. > > For example, in maven you define your dependencies in the pom and a > maven plugin will then have to assemble a module path. Since the > group-id, artifact-id, classifier and version do not always fit in the > restricted module name they already must be prepared to mangle these > names. ... Module descriptors, i.e., module-info.class files, are not intended to contain all of the information that tools may require to compute compile-time or run-time module paths. They are not intended to replace pom.xml files in Maven, or similar types of files in other build systems. They are, rather, intended to contain just the information required by a Java compiler or JVM to provide reliable configuration and strong encapsulation, i.e., requires, exports, uses, and provides. There is some unavoidable conceptual overlap here in the dependence information (requires), but it should be straightforward for an IDE or other tool to to relate module names to the artifact identifiers of the build system being used, and to keep the dependences expressed in module declarations in sync with those in artifact descriptions. > In the rest of the industry (git, docker) the direction is clearly in > using naming based on content (e.g. SHA digests). The advantages of > content addressing mechanisms over namespaces are enormous as many can > testify. No duplicates, safe to cache, etc. > > Once you accept that tooling is a given, it seems therefore to make > more sense to bind the artifacts into a module path by their digest > then to use brittle naming schemes. Names based on content hashes certainly have their place, but they're not suitable for human consumption. I don't see how it makes sense for Java modules not to have human-understandable names, but perhaps I'm missing something here. > A nice demonstration is the discussion about modules masquerading as > friends. In this description digests were already used limit the scope > of the export-to concept. However, this mechanism is hinted at but not > part of the proposal. The use of hashing to secure the use of qualified exports is part of the proposal, though it's not yet fully implemented in the prototype. > Obviously this leaves the dependency on the jre/jdk; this will be > harder to specify as a digest. Indeed it would. - Mark From mark.reinhold at oracle.com Mon Oct 5 18:54:23 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 5 Oct 2015 11:54:23 -0700 (PDT) Subject: Has it been considered to apply the export modifier to the package-info.java? In-Reply-To: <5087F328-84D7-4CC5-89CB-58071B04561F@aqute.biz> References: <5087F328-84D7-4CC5-89CB-58071B04561F@aqute.biz> Message-ID: <20151005185423.4FD207B95D@eggemoggin.niobe.net> 2015/9/28 9:57 -0700, peter.kriens at aqute.biz: > The current proposal places the export in a module-info file. It would > be much more convenient to place this as a modifier on the > package-info.java. > > public package com.foo; Why would this be more convenient? Placing all of the export clauses in the module descriptor makes it easy to see, by reading just one file, exactly what a module is exposing to the outside world. Sprinkling them across package-info.{java,class} files makes that much more difficult. > Since it seems inevitable that there is a tooling step this would > require little effort and cause no runtime overhead. There would be some additional run-time overhead. In order to enforce access control a JVM would have to search for and, if found, read a separate package-info.class file for every package in a module. This overhead might not be terribly significant, but it would be non-zero. - Mark From mark.reinhold at oracle.com Mon Oct 5 18:55:23 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 5 Oct 2015 11:55:23 -0700 (PDT) Subject: How to do module refactoring without rippling changes? In-Reply-To: <974D1CEF-9E53-49F3-A918-07204E9E4458@aqute.biz> References: <974D1CEF-9E53-49F3-A918-07204E9E4458@aqute.biz> Message-ID: <20151005185523.51B617B963@eggemoggin.niobe.net> 2015/9/28 9:58 -0700, peter.kriens at aqute.biz: > According to the requirements specification it should be possible to > split a module into two modules and this should not require changing > all modules that are dependent on this module. How will this work in > the current proposal? Suppose you start with: module foo { exports foo.bar; exports foo.baz; } Then you split it into two: module foo.bar { exports foo.bar; } module foo.baz { exports foo.bar; } To keep old clients of the original "foo" module working, define a new version of "foo": module foo { requires public foo.bar; requires public foo.baz; } This is a so-called "aggregator" module: It has no actual content of its own, but it brings together the content of two or more other modules and uses implied readability to ensure that any module that reads this module will, in turn, read the other modules. - Mark From mark.reinhold at oracle.com Mon Oct 5 18:56:23 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 5 Oct 2015 11:56:23 -0700 (PDT) Subject: How to see the Jigsaw additional classes/methods? In-Reply-To: <271B2311-1BF0-411F-A222-E13BAC89CA82@aqute.biz> References: <271B2311-1BF0-411F-A222-E13BAC89CA82@aqute.biz> Message-ID: <20151005185623.537EC7B96C@eggemoggin.niobe.net> 2015/9/28 9:59 -0700, peter.kriens at aqute.biz: > The Jigsaw Javadoc seems to still show versions, main class, and other > concepts that are not part of the proposal. How should I see these, > will they be removed? These are part of the proposal, as potential standard documentary attributes for compiled module descriptors, though not source-level module declarations. The exact set of such attributes is an open issue, but it could include some of the kinds of information I mentioned in an earlier reply such as copyright/license notices, project URLs, and maintainer contact information. - Mark From mark.reinhold at oracle.com Mon Oct 5 18:57:23 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 5 Oct 2015 11:57:23 -0700 (PDT) Subject: Are multiple versions of the same package in the module graph allowed? In-Reply-To: References: Message-ID: <20151005185723.555427B971@eggemoggin.niobe.net> 2015/9/28 10:00 -0700, peter.kriens at aqute.biz: > It is clear that it is illegal for a module to be exposed to 2 or more > modules that export the same package. I assume that such an > application will not start and throw an exception? Correct. > That said, there are many graphs were two or more packages are present > in the graph, their modules are just never read by any module > simultaneously. From reading the proposal I could not deduct if this > is flagged as wrong or if this is actually allowed? It may be allowed, or it may not, depending upon how the modules are mapped to class loaders. Within a class loader, any particular package is associated with precisely one module. If the two modules containing the common package are mapped to the same loader then layer creation will fail and an exception will be thrown. If the modules are mapped to different loaders then it will work. Relevant draft API: http://j.mp/1Lq5QVY - Mark From mark.reinhold at oracle.com Mon Oct 5 18:58:23 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 5 Oct 2015 11:58:23 -0700 (PDT) Subject: What is the module scope of Service Loader? In-Reply-To: <9A6F4558-C48E-42AF-A06D-12C8ED1EF267@aqute.biz> References: <9A6F4558-C48E-42AF-A06D-12C8ED1EF267@aqute.biz> Message-ID: <20151005185823.5779F7B976@eggemoggin.niobe.net> 2015/9/28 10:01 -0700, peter.kriens at aqute.biz: > It is not clear to me from the description how Service Loader finds > implementation classes. For example DriverManager.getDrivers() returns > a list of all _installed_ drivers. It is not clear to me from which > modules these drivers are loaded. > > * Is this system wide? > * Just the modules the caller can read? > * The modules the java.sql module can read? No to all three. Readability does not affect service-provider lookup. The existing class-loader-based methods in ServiceLoader will continue to work as before: They will search for providers known to the given (or implied) class loader, and then its parent, etc., regardless of which modules they're in. An existing method such as DriverManager::getDrivers will work as before, except that it will be able to locate providers on the application module path as well as on the class path. - Mark From mark.reinhold at oracle.com Mon Oct 5 18:59:23 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 5 Oct 2015 11:59:23 -0700 (PDT) Subject: Resource encapsulation Message-ID: <20151005185923.5984A7B97B@eggemoggin.niobe.net> (combining several questions into one topic thread) 2015/9/28 10:02 -0700, peter.kriens at aqute.biz: > The javadoc for getResource* says: > > ???This method does not find resources in named modules???. > > Literally this implies that a class in a named module cannot read its > own resources. I assume that this is an oversight in the description? There is a plethora of resource-lookup methods. The current state of affairs is: - A class in a named module can read its own resources via the Class::getResourceAsStream method, which returns an InputStream. It can get a URL to one of its own resources via the Class::getResource method [1]. These methods will not locate resources in other named modules. - The ClassLoader::getResource* methods do not locate resources in any named modules. - All existing resource-lookup methods in Class and ClassLoader work as they do today for resources on the class path. - The new java.lang.reflect.Module::getResourceAsStream method can be used to read the resources of any named module, without restriction. These choices are a first attempt to strike a balance between satisfying the "resource encapsulation" [2] and "readable artifact" requirements [3] while making it reasonably easy to migrate existing components into modules, but I don't think we're quite there yet. At the very least, the Module::getResourceAsStream method needs to be restricted somehow so that only "suitably-privileged" code (for some definition of that term) can access resources in a named module other than its own. 2015/9/28 10:03 -0700, peter.kriens at aqute.biz: > In Java Persistence Architecture (JPA) the persistence.xml resource > provides information about the persistence unit. This resource must be > read by the Persistence Provider which then does a lot of magic. > > However, if nobody can read resources from a named module then how can > JPA perform its duties? JPA will work exactly as it does today for code on the class path. A future version of JPA that supports modules will, I assume, itself be a "suitably-privileged" module. It can use the various reflection APIs (java.lang.reflect.Module, etc.) to locate run-time module objects and read their resources. In the longer term, a better way for a module to publish configuration information for a framework such as JPA would be for it to supply a provider of a service type defined by the framework. JPA could, e.g., export and use a javax.persistence.Configuration service type, and then a client module could provide an instance of that type in order to convey its configuration information. 2015/9/28 10:04 -0700, peter.kriens at aqute.biz: > Since resource loading is very restrictive I would expect that class > loading is also not possible unless a class is exported. Is this true? The module system, as proposed, does not change the run-time visibility mechanism. Methods such as Class::forName and ClassLoader::loadClass will work as they to today. If a class loader queried by these methods finds a definition for the specified class then a Class object will be returned regardless of whether that class's package is exported by its defining module. (Whether you can access members of the resulting class, or instances thereof, is a question of access control and hence is, now, dependent upon the module system.) - Mark [1] This was enabled just last week: http://j.mp/1NgIVg7 [2] http://openjdk.java.net/projects/jigsaw/spec/reqs/#resource-encapsulation [3] http://openjdk.java.net/projects/jigsaw/spec/reqs/#readable-artifacts From mark.reinhold at oracle.com Mon Oct 5 19:00:23 2015 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 5 Oct 2015 12:00:23 -0700 (PDT) Subject: How will dependency Injection/Annotation scanning work? In-Reply-To: <6D9ACFC7-45C3-46E8-B958-FBCBB4C25109@aqute.biz> Message-ID: <20151005190023.5BBB77B980@eggemoggin.niobe.net> 2015/9/30 12:33 -0700, peter.kriens at aqute.biz: > Thanks for the answer David about the annotation scanning. I guess this > still will have to undergo a more detailed design? Yes. The initial design, and the prototype, do not yet implement the "efficient annotation detection" requirement [1]. - Mark [1] http://openjdk.java.net/projects/jigsaw/spec/reqs/#efficient-annotation-detection From peter.kriens at aqute.biz Mon Oct 12 15:01:35 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 12 Oct 2015 17:01:35 +0200 Subject: Resource encapsulation In-Reply-To: <20151005185923.5984A7B97B@eggemoggin.niobe.net> References: <20151005185923.5984A7B97B@eggemoggin.niobe.net> Message-ID: I think I understand how a module aware JPA can use the Module API to load resources and classes. This sounds similar to how OSGi works. For the service loader scenario I am not clear how you would then discover and load the persistence.xml resources. Would you then not still be forced to go through the Module API anyway to read resources since the javax.persistence.Configuration implementation would not allow the JPA provider to read the persistence.xml? Isn?t using the service loader sounds like an unnecessary indirection? Thanks for the reply, kind regards, Peter Kriens > On 5 okt. 2015, at 20:59, mark.reinhold at oracle.com wrote: > > (combining several questions into one topic thread) > > 2015/9/28 10:02 -0700, peter.kriens at aqute.biz: >> The javadoc for getResource* says: >> >> ?This method does not find resources in named modules?. >> >> Literally this implies that a class in a named module cannot read its >> own resources. I assume that this is an oversight in the description? > > There is a plethora of resource-lookup methods. The current state of > affairs is: > > - A class in a named module can read its own resources via the > Class::getResourceAsStream method, which returns an InputStream. It > can get a URL to one of its own resources via the Class::getResource > method [1]. These methods will not locate resources in other named > modules. > > - The ClassLoader::getResource* methods do not locate resources in any > named modules. > > - All existing resource-lookup methods in Class and ClassLoader work as > they do today for resources on the class path. > > - The new java.lang.reflect.Module::getResourceAsStream method can be > used to read the resources of any named module, without restriction. > > These choices are a first attempt to strike a balance between satisfying > the "resource encapsulation" [2] and "readable artifact" requirements [3] > while making it reasonably easy to migrate existing components into > modules, but I don't think we're quite there yet. At the very least, the > Module::getResourceAsStream method needs to be restricted somehow so that > only "suitably-privileged" code (for some definition of that term) can > access resources in a named module other than its own. > > 2015/9/28 10:03 -0700, peter.kriens at aqute.biz: >> In Java Persistence Architecture (JPA) the persistence.xml resource >> provides information about the persistence unit. This resource must be >> read by the Persistence Provider which then does a lot of magic. >> >> However, if nobody can read resources from a named module then how can >> JPA perform its duties? > > JPA will work exactly as it does today for code on the class path. > > A future version of JPA that supports modules will, I assume, itself be > a "suitably-privileged" module. It can use the various reflection APIs > (java.lang.reflect.Module, etc.) to locate run-time module objects and > read their resources. > > In the longer term, a better way for a module to publish configuration > information for a framework such as JPA would be for it to supply a > provider of a service type defined by the framework. JPA could, e.g., > export and use a javax.persistence.Configuration service type, and then a > client module could provide an instance of that type in order to convey > its configuration information. > > 2015/9/28 10:04 -0700, peter.kriens at aqute.biz: >> Since resource loading is very restrictive I would expect that class >> loading is also not possible unless a class is exported. Is this true? > > The module system, as proposed, does not change the run-time visibility > mechanism. Methods such as Class::forName and ClassLoader::loadClass > will work as they to today. If a class loader queried by these methods > finds a definition for the specified class then a Class object will be > returned regardless of whether that class's package is exported by its > defining module. (Whether you can access members of the resulting class, > or instances thereof, is a question of access control and hence is, now, > dependent upon the module system.) > > - Mark > > > [1] This was enabled just last week: http://j.mp/1NgIVg7 > [2] http://openjdk.java.net/projects/jigsaw/spec/reqs/#resource-encapsulation > [3] http://openjdk.java.net/projects/jigsaw/spec/reqs/#readable-artifacts From peter.kriens at aqute.biz Mon Oct 12 15:11:35 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 12 Oct 2015 17:11:35 +0200 Subject: What is the module scope of Service Loader? In-Reply-To: <20151005185823.5779F7B976@eggemoggin.niobe.net> References: <9A6F4558-C48E-42AF-A06D-12C8ED1EF267@aqute.biz> <20151005185823.5779F7B976@eggemoggin.niobe.net> Message-ID: <72FA632B-F282-429E-974D-AEC1D270F9F4@aqute.biz> Yo use the ?application module path? in your answers. I could not find a definition of this in the SOTM nor the API? What happens to getDrivers when there are multiple paths, assuming this is possible? Do I see the drivers of the other path? Thanks, kind regards, Peter Kriens > On 5 okt. 2015, at 20:58, mark.reinhold at oracle.com wrote: > > 2015/9/28 10:01 -0700, peter.kriens at aqute.biz: >> It is not clear to me from the description how Service Loader finds >> implementation classes. For example DriverManager.getDrivers() returns >> a list of all _installed_ drivers. It is not clear to me from which >> modules these drivers are loaded. >> >> * Is this system wide? >> * Just the modules the caller can read? >> * The modules the java.sql module can read? > > No to all three. Readability does not affect service-provider lookup. > > The existing class-loader-based methods in ServiceLoader will continue to > work as before: They will search for providers known to the given (or > implied) class loader, and then its parent, etc., regardless of which > modules they're in. An existing method such as DriverManager::getDrivers > will work as before, except that it will be able to locate providers on > the application module path as well as on the class path. > > - Mark From peter.kriens at aqute.biz Mon Oct 12 15:16:16 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 12 Oct 2015 17:16:16 +0200 Subject: How to do module refactoring without rippling changes? In-Reply-To: <20151005185523.51B617B963@eggemoggin.niobe.net> References: <974D1CEF-9E53-49F3-A918-07204E9E4458@aqute.biz> <20151005185523.51B617B963@eggemoggin.niobe.net> Message-ID: Ah, sorry, I could have figured this out. This was the same solution that Eclipse picked when they had to refactor parts of SWT. It is not a very elegant solution (it creates a lot of extra maintenance work over time) but it does satisfy the requirement. Thanks, kind regards, Peter Kriens > On 5 okt. 2015, at 20:55, mark.reinhold at oracle.com wrote: > > 2015/9/28 9:58 -0700, peter.kriens at aqute.biz: >> According to the requirements specification it should be possible to >> split a module into two modules and this should not require changing >> all modules that are dependent on this module. How will this work in >> the current proposal? > > Suppose you start with: > > module foo { > exports foo.bar; > exports foo.baz; > } > > Then you split it into two: > > module foo.bar { > exports foo.bar; > } > > module foo.baz { > exports foo.bar; > } > > To keep old clients of the original "foo" module working, define a new > version of "foo": > > module foo { > requires public foo.bar; > requires public foo.baz; > } > > This is a so-called "aggregator" module: It has no actual content of its > own, but it brings together the content of two or more other modules and > uses implied readability to ensure that any module that reads this module > will, in turn, read the other modules. > > - Mark From peter.kriens at aqute.biz Mon Oct 12 15:30:43 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 12 Oct 2015 17:30:43 +0200 Subject: Has it been considered to apply the export modifier to the package-info.java? In-Reply-To: <20151005185423.4FD207B95D@eggemoggin.niobe.net> References: <5087F328-84D7-4CC5-89CB-58071B04561F@aqute.biz> <20151005185423.4FD207B95D@eggemoggin.niobe.net> Message-ID: > Why would this be more convenient? For the same reason it is more convenient to put the public/private modifier in the class source file instead of the package-info file? The subsidiarity principle says that you want to apply this information at the lowest reasonable level. People that then work in that package are directly aware of this information. The close this information is to the source, the better the chance it gets maintained. It also makes refactoring easier since it does not require a global overview. And it does not require typing the package name since this is already implied. When I meant zero overhead I included my conclusion that module-info.java cannot be a human editable file. In any practical situation it can only be generated from external (build) information. The module-info.java file just does not contain sufficient information (versions!) to drive a build so there MUST be another artifact that contains a superset of the module-info aspects. To prevent redundancy, the module-info should thus be generated. This required build step could then easily include the exported packages. So the only overhead would be in the build. Thanks, kind regards, Peter Kriens > On 5 okt. 2015, at 20:54, mark.reinhold at oracle.com wrote: > > 2015/9/28 9:57 -0700, peter.kriens at aqute.biz: >> The current proposal places the export in a module-info file. It would >> be much more convenient to place this as a modifier on the >> package-info.java. >> >> public package com.foo; > > Why would this be more convenient? > > Placing all of the export clauses in the module descriptor makes it easy > to see, by reading just one file, exactly what a module is exposing to > the outside world. Sprinkling them across package-info.{java,class} > files makes that much more difficult. > >> Since it seems inevitable that there is a tooling step this would >> require little effort and cause no runtime overhead. > > There would be some additional run-time overhead. In order to enforce > access control a JVM would have to search for and, if found, read a > separate package-info.class file for every package in a module. This > overhead might not be terribly significant, but it would be non-zero. > > - Mark From peter.kriens at aqute.biz Mon Oct 12 16:08:15 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 12 Oct 2015 18:08:15 +0200 Subject: Missing reification of the Module-path? In-Reply-To: <20151005185123.461CF7B94E@eggemoggin.niobe.net> References: <19F2AB39-D4B3-4D50-8C5A-0043BFDE25DB@aqute.biz> <20151005185123.461CF7B94E@eggemoggin.niobe.net> Message-ID: I think some simplifications could be made in the current design if we could standardize an artifact for the application module path at the same time. I think it should be quite straightforward to support fat applications and applications that reference their constituents (or a mix). Thanks, kind regards, Peter Kriens P.S. And I?ve figured out the Application Module path, not sure what I was smoking :-( > On 5 okt. 2015, at 20:51, mark.reinhold at oracle.com wrote: > > 2015/9/28 9:51 -0700, peter.kriens at aqute.biz: >> The proposal specifies how to resolve a module from a _module path_, a >> module path being one or more directories with modular JARs. >> >> Since the resolution is without versions, the module-path *must* >> contain a consistent set of artifacts. That is, the module path cannot >> be a repository like a maven repository it is closely bound to an >> executable since for each module only one of its versions can be >> chosen. >> >> For example in maven the module path would consist of the transitive >> runtime dependencies. This is a unique set per application per >> version. I do not think it is practical to synchronize this set with >> other applications so one should assume this is unique and cannot be >> shared. It is potentially very large. >> >> Ergo, the module-path *is* the application. > > Or, almost equivalently, the module graph resolved from the module path > on behalf of the main module is the application. > >> I therefore wonder if there is a need for a reification of the module >> path + main class? Such an artifact could contain an indirection like >> a URL to not require creating temporary module path directories for >> each application with potentially very large duplicated artifacts. >> >> Such an artifact could then be made available as an atomic Layer. > > Hmm. Do you mean that such an artifact would contain copies of all of > the modules in the module graph, i.e., would it be a "fat" artifact? Or > would it just record references to the necessary modules, presumably with > content hashes to ensure integrity? > > We've explored similar notions in earlier prototypes. An artifact in > either of these forms could, in principle, be generated by the linker, > but would require some new support at install time and/or run time. > > It's an interesting idea, though outside the scope of this JSR unless > we conclude it's important to standardize the format of such artifacts. > > - Mark From peter.kriens at aqute.biz Mon Oct 12 16:48:07 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 12 Oct 2015 18:48:07 +0200 Subject: Do we actually need module names? In-Reply-To: <20151005185323.4DD797B958@eggemoggin.niobe.net> References: <20151005185323.4DD797B958@eggemoggin.niobe.net> Message-ID: If a module could require package names instead of module names it would not be necessary to introduce a new orthogonal namespace to the code base. If we take an application module path then this will represent a limited (max a few thousand, on average much less) set of modules. The current design already implies that in this set modules their exported packages must be unique, so there is no need to resolve multiple versions. It is therefore very almost no overhead to find a module by its exported packages instead of a module name. Since the Java bytecode already contains the reference to the exported packages it is straightforward to translate these references to a require package in the module definition. This will actually be much less work than developers maintaining a module namespace. Just think of the advantages: * No new error prone module namespace * Automatic generation of module requirements * No need for an awkward solution to module refactoring About hashes, they have their place in the application artifact we discussed but I now thinks that using package requirements is a better solution. Thanks, kind regards, Peter Kriens > On 5 okt. 2015, at 20:53, mark.reinhold at oracle.com wrote: > > 2015/9/28 9:56 -0700, peter.kriens at aqute.biz: >> This specification introduces a completely new namespace for >> modules. One of the lessons of the last decade has been that >> namespaces are very brittle and error prone. So a valid question seems >> to be: what do we get for this relatively high cost? >> >> The only reason to use a name space is for humans. However, in the >> current proposal the names will likely not be used by humans since >> they lack a version. > > I don't follow. Humans already use names, without versions, for many > things in Java, and in programming environments more generally. > >> That is, the module-info.class contains too >> little information to be used by tooling as a source for the compile >> or runtime paths. It therefore seems inevitable that tools will >> generate this resource from their native metadata. >> >> For example, in maven you define your dependencies in the pom and a >> maven plugin will then have to assemble a module path. Since the >> group-id, artifact-id, classifier and version do not always fit in the >> restricted module name they already must be prepared to mangle these >> names. ... > > Module descriptors, i.e., module-info.class files, are not intended to > contain all of the information that tools may require to compute > compile-time or run-time module paths. They are not intended to replace > pom.xml files in Maven, or similar types of files in other build systems. > They are, rather, intended to contain just the information required by a > Java compiler or JVM to provide reliable configuration and strong > encapsulation, i.e., requires, exports, uses, and provides. > > There is some unavoidable conceptual overlap here in the dependence > information (requires), but it should be straightforward for an IDE or > other tool to to relate module names to the artifact identifiers of the > build system being used, and to keep the dependences expressed in module > declarations in sync with those in artifact descriptions. > >> In the rest of the industry (git, docker) the direction is clearly in >> using naming based on content (e.g. SHA digests). The advantages of >> content addressing mechanisms over namespaces are enormous as many can >> testify. No duplicates, safe to cache, etc. >> >> Once you accept that tooling is a given, it seems therefore to make >> more sense to bind the artifacts into a module path by their digest >> then to use brittle naming schemes. > > Names based on content hashes certainly have their place, but they're not > suitable for human consumption. I don't see how it makes sense for Java > modules not to have human-understandable names, but perhaps I'm missing > something here. > >> A nice demonstration is the discussion about modules masquerading as >> friends. In this description digests were already used limit the scope >> of the export-to concept. However, this mechanism is hinted at but not >> part of the proposal. > > The use of hashing to secure the use of qualified exports is part of the > proposal, though it's not yet fully implemented in the prototype. > >> Obviously this leaves the dependency on the jre/jdk; this will be >> harder to specify as a digest. > > Indeed it would. > > - Mark From peter.kriens at aqute.biz Mon Oct 12 16:51:52 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 12 Oct 2015 18:51:52 +0200 Subject: Why not use the Manifest? In-Reply-To: <20151005185023.443E87B949@eggemoggin.niobe.net> References: <20151005185023.443E87B949@eggemoggin.niobe.net> Message-ID: <81AC5311-7835-4451-8381-CA446A311386@aqute.biz> Isn?t the manifest already part of the JVM specification? Kind regards, Peter Kriens > On 5 okt. 2015, at 20:50, mark.reinhold at oracle.com wrote: > > 2015/9/28 9:49 -0700, peter.kriens at aqute.biz: >> The encoding of the module data in a binary class file remains hard to >> understand and seems to go against the best practices we learned the >> hard way over the past decades. (Debugging!) >> >> Especially since the current proposal leaves the heavy lifting to >> build tooling. In the proposal, build tools cannot use the >> module-info.java since it does not contain sufficient >> information. (Versions being the most glaring omission for a build >> too.) Nor is the format extensible to contain build tool specific >> information. Ergo, the build tool will have to generate the >> module-info.class file. >> >> This make the class file makes these tools unnecessary hard and slower >> because it must either use the compiler as an intermediate step or use >> a library like ASM to create the unreadable version of its metadata. >> >> This is the first time in a very long time that I see a regression to >> binary files for meta data. Especially because there is already a good >> place that all build tools are already using: the manifest. > > The reasons for expressing module declarations in source files and > compiling them into class files are mentioned in SotMS, but here's a > longer take on just this topic. > > The Java programming language, at present, provides for the definition > of three kinds of program components: classes, interfaces, and packages. > They are defined in Java source files, and compiled into class files. > Such files govern, among other things, the mechanisms of symbolic > resolution and access control implemented by every Java compiler and > JVM. A developer need not reason about any other type of file in order > to understand the actions of these mechanisms. A compiler or JVM need > not consume or produce any information other than that found in source > and class files in order to implement these mechanisms. > > The module system extends the mechanism of symbolic resolution with the > concept of readability in order to provide reliable configuration. It > extends the mechanism of access control, in part by relating it to > readability, in order to provide strong encapsulation. To support these > extensions, every Java compiler and JVM must be able to locate and > interpret descriptions of modules which convey, at least, each module's > name, dependences, and exported packages. Such descriptions, regardless > of their form, will govern symbolic resolution and access control in both > compilers and JVMs, so they will have to be specified in both the Java > Language Specification (JLS) and the Java Virtual Machine Specification > (JVMS). > > Regardless of how they are described, modules are, fundamentally, a new > kind of Java program component. The present proposal therefore treats > them as such, in both the language and the JVM: Module descriptions are > expressed and encoded in the same way as the other kinds of information > that define program components and govern symbolic resolution and access > control -- that is, as Java source and class files. > > This approach is immediately familiar to developers, who already think > about program components, symbolic resolution, and access control in > terms of the Java programming language. It is easy to retrofit into the > JLS and the JVMS, since those specifications are already centered upon > source and class files. It is, finally, straightforward for existing > Java implementations and tool chains to support, since they need not be > revised to handle an entirely new type of file. > > We could choose another format for module descriptions such as XML, JSON, > YAML, or JAR-file manifests. Such formats may be more convenient for > tool maintainers but we'd have to bake that format into the JLS and the > JVMS, thereby increasing the complexity of those specifications and their > implementations. If the format depends upon external standards (XML, > JSON, YAML) then these foundational specifications, and their > implementations, would become dependent upon those standards. If the > format proves unsuitable over time then replacing it with something else > would require major revisions to these specifications, and to their > implementations. > > Source and class files are fundamental to the Java platform, to the JLS > and the JVMS, and to their implementations and supporting tools. Other > formats come and go; these will be with us forever. > > * * * > > As to debugging, in the prototype we've already enhanced the jar tool > with an option to print the descriptor of a modular JAR file. I expect > similar support to show up in related tools such as jmod and javap, and > eventually in IDEs and other tools outside the JDK. > > As to ease of tooling, most tools will only need to read module > descriptors, not write them, and the java.lang.module.ModuleDescriptor > class already provides convenient static methods for doing that. Adding > similar methods to write descriptors would be straightforward, though it > would complicate the API a bit. > > As to extensibility, class files are already in a precisely-specified and > extensible format. If a build tool needs to add information to a module > descriptor then it can do so via non-standard class-file attributes, > using an existing popular library such as ASM. > > To make it easier for tools to manipulate module descriptors we could > consider enhancing the java.lang.module.ModuleDescriptor class, and > related tools, to read and write custom, non-standard class-file > attributes. This might be a bit tricky depending on how the attributes > are defined, but for simple property-style attributes it's likely > straightforward and would obviate the need to use ASM directly. > > - Mark From peter.kriens at aqute.biz Mon Oct 12 16:56:06 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Mon, 12 Oct 2015 18:56:06 +0200 Subject: Why can modules not be annotated? In-Reply-To: <20151005184923.394607B943@eggemoggin.niobe.net> References: <9C2D961B-0072-4C72-9FA6-A63AF65BFC64@aqute.biz> <20151005184923.394607B943@eggemoggin.niobe.net> Message-ID: <0AC36D44-0539-41DD-9505-76F789D26EE2@aqute.biz> I understand the trade offs. That said, I am a bit curious how we are going to handle the discussing these trade offs in the design? I had actually expected that more things were set in stone but in this discussion it seems there are still some open areas. Are we going to have phone conferences, a face to face? Anyway, thanks for all the responses. It did clarify things for me. Kind regards, Peter Kriens > On 5 okt. 2015, at 20:49, mark.reinhold at oracle.com wrote: > > 2015/9/28 9:44 -0700, peter.kriens at aqute.biz: >> In [lang-vm][1] I found: >> >> ?A module declaration may not be annotated.? >> >> I was surprised that modules (and their clauses) could not be >> annotated like package-info.java. Since the source syntax is not >> extensible, annotations could have been used to provide meta data for >> other module systems. This would mitigate some of the concerns of >> using a binary file to encode the module data. > > We haven't seen a compelling use case (so far) for annotations on module > declarations, or on clauses in module declarations, and they aren't a > documented requirement, so for simplicity we left them out of the initial > design and the prototype. > > Supporting annotations on module declarations is technically feasible > but would add significant complexity to both the specification and its > implementations. The machinery to support annotations is everywhere, in > javac, the VM, core reflection, the language-model API, etc. Annotations > in module declarations would also lead to a natural desire for import > statements in module declarations, further complicating matters. > > Annotations are, fundamentally, a way to make extra-linguistic statements > in source code that are carried through the tool chain and (usually) into > the resulting class file. Beyond the essential linguistic constructs > already in source-level module declarations (requires, exports, provides, > and uses), all the other examples we've seen of information that could > make sense in the resulting module-descriptor class files don't really > make sense in source code, so annotations aren't the right tool for the > problem. Version strings, copyright/license notices, project URLs, > maintainer contact information, and target operating systems and > processor architectures, e.g., are more appropriately maintained in build > metadata (e.g., pom.xml), or calculated by the build system on-the-fly, > than expressed in source code, and they can easily be added as custom > attributes to a module-info.class file after it's compiled. > > (Whether it's actually a good idea to put all these sorts of information > in a module descriptor is a different question.) > > So yes, we could support annotations on module declarations, and that may > indeed be useful for other module systems. It would come at significant > cost, however, and we'd have to weigh that cost carefully. > > - Mark From forax at univ-mlv.fr Wed Oct 14 13:36:24 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 14 Oct 2015 15:36:24 +0200 (CEST) Subject: Do we actually need module names? In-Reply-To: References: <20151005185323.4DD797B958@eggemoggin.niobe.net> Message-ID: <752578289.1576366.1444829784960.JavaMail.zimbra@u-pem.fr> The things is that we need a notion of 'meta-package' to not specified dependencies on something that change, packages are added, packages (not exported, i hope) are removed, dependencies changes, module name is the things that are stable here. Bytecodes contains the notion of public re-export, you can derive public required modules from them, but not the notion of exported package, how to make the difference between com.foo.bar.api and com.foo.bar.internal ? regards, R?mi ----- Mail original ----- > De: "Peter Kriens" > ?: "Mark Reinhold" > Cc: jpms-spec-experts at openjdk.java.net > Envoy?: Lundi 12 Octobre 2015 18:48:07 > Objet: Re: Do we actually need module names? > > If a module could require package names instead of module names it would not > be necessary to introduce a new orthogonal namespace to the code base. > > If we take an application module path then this will represent a limited (max > a few thousand, on average much less) set of modules. The current design > already implies that in this set modules their exported packages must be > unique, so there is no need to resolve multiple versions. It is therefore > very almost no overhead to find a module by its exported packages instead of > a module name. > > Since the Java bytecode already contains the reference to the exported > packages it is straightforward to translate these references to a require > package in the module definition. This will actually be much less work than > developers maintaining a module namespace. > > Just think of the advantages: > > * No new error prone module namespace > * Automatic generation of module requirements > * No need for an awkward solution to module refactoring > > About hashes, they have their place in the application artifact we discussed > but I now thinks that using package requirements is a better solution. > > Thanks, kind regards, > > Peter Kriens > > > > > > > > > On 5 okt. 2015, at 20:53, mark.reinhold at oracle.com wrote: > > > > 2015/9/28 9:56 -0700, peter.kriens at aqute.biz: > >> This specification introduces a completely new namespace for > >> modules. One of the lessons of the last decade has been that > >> namespaces are very brittle and error prone. So a valid question seems > >> to be: what do we get for this relatively high cost? > >> > >> The only reason to use a name space is for humans. However, in the > >> current proposal the names will likely not be used by humans since > >> they lack a version. > > > > I don't follow. Humans already use names, without versions, for many > > things in Java, and in programming environments more generally. > > > >> That is, the module-info.class contains too > >> little information to be used by tooling as a source for the compile > >> or runtime paths. It therefore seems inevitable that tools will > >> generate this resource from their native metadata. > >> > >> For example, in maven you define your dependencies in the pom and a > >> maven plugin will then have to assemble a module path. Since the > >> group-id, artifact-id, classifier and version do not always fit in the > >> restricted module name they already must be prepared to mangle these > >> names. ... > > > > Module descriptors, i.e., module-info.class files, are not intended to > > contain all of the information that tools may require to compute > > compile-time or run-time module paths. They are not intended to replace > > pom.xml files in Maven, or similar types of files in other build systems. > > They are, rather, intended to contain just the information required by a > > Java compiler or JVM to provide reliable configuration and strong > > encapsulation, i.e., requires, exports, uses, and provides. > > > > There is some unavoidable conceptual overlap here in the dependence > > information (requires), but it should be straightforward for an IDE or > > other tool to to relate module names to the artifact identifiers of the > > build system being used, and to keep the dependences expressed in module > > declarations in sync with those in artifact descriptions. > > > >> In the rest of the industry (git, docker) the direction is clearly in > >> using naming based on content (e.g. SHA digests). The advantages of > >> content addressing mechanisms over namespaces are enormous as many can > >> testify. No duplicates, safe to cache, etc. > >> > >> Once you accept that tooling is a given, it seems therefore to make > >> more sense to bind the artifacts into a module path by their digest > >> then to use brittle naming schemes. > > > > Names based on content hashes certainly have their place, but they're not > > suitable for human consumption. I don't see how it makes sense for Java > > modules not to have human-understandable names, but perhaps I'm missing > > something here. > > > >> A nice demonstration is the discussion about modules masquerading as > >> friends. In this description digests were already used limit the scope > >> of the export-to concept. However, this mechanism is hinted at but not > >> part of the proposal. > > > > The use of hashing to secure the use of qualified exports is part of the > > proposal, though it's not yet fully implemented in the prototype. > > > >> Obviously this leaves the dependency on the jre/jdk; this will be > >> harder to specify as a digest. > > > > Indeed it would. > > > > - Mark > > From forax at univ-mlv.fr Wed Oct 14 13:41:09 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 14 Oct 2015 15:41:09 +0200 (CEST) Subject: Why not use the Manifest? In-Reply-To: <81AC5311-7835-4451-8381-CA446A311386@aqute.biz> References: <20151005185023.443E87B949@eggemoggin.niobe.net> <81AC5311-7835-4451-8381-CA446A311386@aqute.biz> Message-ID: <1993489049.1580799.1444830069389.JavaMail.zimbra@u-pem.fr> I don't think so, it's part of the tools descriptions, the manifest is described as part of the java/jar commands. Anyway, it's important that the module descriptor is checked at compile time (by javac) and at runtime (by java), thus, it has to be a part of Java the language with a syntax understandable by javac. R?mi ----- Mail original ----- > De: "Peter Kriens" > ?: "Mark Reinhold" > Cc: jpms-spec-experts at openjdk.java.net > Envoy?: Lundi 12 Octobre 2015 18:51:52 > Objet: Re: Why not use the Manifest? > > Isn?t the manifest already part of the JVM specification? > > Kind regards, > > Peter Kriens > > > On 5 okt. 2015, at 20:50, mark.reinhold at oracle.com wrote: > > > > 2015/9/28 9:49 -0700, peter.kriens at aqute.biz: > >> The encoding of the module data in a binary class file remains hard to > >> understand and seems to go against the best practices we learned the > >> hard way over the past decades. (Debugging!) > >> > >> Especially since the current proposal leaves the heavy lifting to > >> build tooling. In the proposal, build tools cannot use the > >> module-info.java since it does not contain sufficient > >> information. (Versions being the most glaring omission for a build > >> too.) Nor is the format extensible to contain build tool specific > >> information. Ergo, the build tool will have to generate the > >> module-info.class file. > >> > >> This make the class file makes these tools unnecessary hard and slower > >> because it must either use the compiler as an intermediate step or use > >> a library like ASM to create the unreadable version of its metadata. > >> > >> This is the first time in a very long time that I see a regression to > >> binary files for meta data. Especially because there is already a good > >> place that all build tools are already using: the manifest. > > > > The reasons for expressing module declarations in source files and > > compiling them into class files are mentioned in SotMS, but here's a > > longer take on just this topic. > > > > The Java programming language, at present, provides for the definition > > of three kinds of program components: classes, interfaces, and packages. > > They are defined in Java source files, and compiled into class files. > > Such files govern, among other things, the mechanisms of symbolic > > resolution and access control implemented by every Java compiler and > > JVM. A developer need not reason about any other type of file in order > > to understand the actions of these mechanisms. A compiler or JVM need > > not consume or produce any information other than that found in source > > and class files in order to implement these mechanisms. > > > > The module system extends the mechanism of symbolic resolution with the > > concept of readability in order to provide reliable configuration. It > > extends the mechanism of access control, in part by relating it to > > readability, in order to provide strong encapsulation. To support these > > extensions, every Java compiler and JVM must be able to locate and > > interpret descriptions of modules which convey, at least, each module's > > name, dependences, and exported packages. Such descriptions, regardless > > of their form, will govern symbolic resolution and access control in both > > compilers and JVMs, so they will have to be specified in both the Java > > Language Specification (JLS) and the Java Virtual Machine Specification > > (JVMS). > > > > Regardless of how they are described, modules are, fundamentally, a new > > kind of Java program component. The present proposal therefore treats > > them as such, in both the language and the JVM: Module descriptions are > > expressed and encoded in the same way as the other kinds of information > > that define program components and govern symbolic resolution and access > > control -- that is, as Java source and class files. > > > > This approach is immediately familiar to developers, who already think > > about program components, symbolic resolution, and access control in > > terms of the Java programming language. It is easy to retrofit into the > > JLS and the JVMS, since those specifications are already centered upon > > source and class files. It is, finally, straightforward for existing > > Java implementations and tool chains to support, since they need not be > > revised to handle an entirely new type of file. > > > > We could choose another format for module descriptions such as XML, JSON, > > YAML, or JAR-file manifests. Such formats may be more convenient for > > tool maintainers but we'd have to bake that format into the JLS and the > > JVMS, thereby increasing the complexity of those specifications and their > > implementations. If the format depends upon external standards (XML, > > JSON, YAML) then these foundational specifications, and their > > implementations, would become dependent upon those standards. If the > > format proves unsuitable over time then replacing it with something else > > would require major revisions to these specifications, and to their > > implementations. > > > > Source and class files are fundamental to the Java platform, to the JLS > > and the JVMS, and to their implementations and supporting tools. Other > > formats come and go; these will be with us forever. > > > > * * * > > > > As to debugging, in the prototype we've already enhanced the jar tool > > with an option to print the descriptor of a modular JAR file. I expect > > similar support to show up in related tools such as jmod and javap, and > > eventually in IDEs and other tools outside the JDK. > > > > As to ease of tooling, most tools will only need to read module > > descriptors, not write them, and the java.lang.module.ModuleDescriptor > > class already provides convenient static methods for doing that. Adding > > similar methods to write descriptors would be straightforward, though it > > would complicate the API a bit. > > > > As to extensibility, class files are already in a precisely-specified and > > extensible format. If a build tool needs to add information to a module > > descriptor then it can do so via non-standard class-file attributes, > > using an existing popular library such as ASM. > > > > To make it easier for tools to manipulate module descriptors we could > > consider enhancing the java.lang.module.ModuleDescriptor class, and > > related tools, to read and write custom, non-standard class-file > > attributes. This might be a bit tricky depending on how the attributes > > are defined, but for simple property-style attributes it's likely > > straightforward and would obviate the need to use ASM directly. > > > > - Mark > > From forax at univ-mlv.fr Wed Oct 14 13:46:09 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 14 Oct 2015 15:46:09 +0200 (CEST) Subject: Has it been considered to apply the export modifier to the package-info.java? In-Reply-To: References: <5087F328-84D7-4CC5-89CB-58071B04561F@aqute.biz> <20151005185423.4FD207B95D@eggemoggin.niobe.net> Message-ID: <1498685221.1583861.1444830369649.JavaMail.zimbra@u-pem.fr> Good question, the main drawback i see, is that it that you can not retrofit a non-module jar to a module jar just by adding one file. R?mi ----- Mail original ----- > De: "Peter Kriens" > ?: "Mark Reinhold" > Cc: jpms-spec-experts at openjdk.java.net > Envoy?: Lundi 12 Octobre 2015 17:30:43 > Objet: Re: Has it been considered to apply the export modifier to the package-info.java? > > > Why would this be more convenient? > > For the same reason it is more convenient to put the public/private modifier > in the class source file instead of the package-info file? > > The subsidiarity principle says that you want to apply this information at > the lowest reasonable level. People that then work in that package are > directly aware of this information. The close this information is to the > source, the better the chance it gets maintained. It also makes refactoring > easier since it does not require a global overview. And it does not require > typing the package name since this is already implied. > > When I meant zero overhead I included my conclusion that module-info.java > cannot be a human editable file. In any practical situation it can only be > generated from external (build) information. The module-info.java file just > does not contain sufficient information (versions!) to drive a build so > there MUST be another artifact that contains a superset of the module-info > aspects. To prevent redundancy, the module-info should thus be generated. > This required build step could then easily include the exported packages. So > the only overhead would be in the build. > > Thanks, kind regards, > > Peter Kriens > > > > On 5 okt. 2015, at 20:54, mark.reinhold at oracle.com wrote: > > > > 2015/9/28 9:57 -0700, peter.kriens at aqute.biz: > >> The current proposal places the export in a module-info file. It would > >> be much more convenient to place this as a modifier on the > >> package-info.java. > >> > >> public package com.foo; > > > > Why would this be more convenient? > > > > Placing all of the export clauses in the module descriptor makes it easy > > to see, by reading just one file, exactly what a module is exposing to > > the outside world. Sprinkling them across package-info.{java,class} > > files makes that much more difficult. > > > >> Since it seems inevitable that there is a tooling step this would > >> require little effort and cause no runtime overhead. > > > > There would be some additional run-time overhead. In order to enforce > > access control a JVM would have to search for and, if found, read a > > separate package-info.class file for every package in a module. This > > overhead might not be terribly significant, but it would be non-zero. > > > > - Mark > > From forax at univ-mlv.fr Wed Oct 14 13:53:31 2015 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Wed, 14 Oct 2015 15:53:31 +0200 (CEST) Subject: module name convention In-Reply-To: <20151005184823.30B397B93E@eggemoggin.niobe.net> References: <1441288605.2461702.1442681029132.JavaMail.zimbra@u-pem.fr> <1052601312.2492216.1442682135412.JavaMail.zimbra@u-pem.fr> <20151005184823.30B397B93E@eggemoggin.niobe.net> Message-ID: <1654087394.1586727.1444830811461.JavaMail.zimbra@u-pem.fr> Just a quick note, I've presented the Java module spec in its current state at JTRES (a realtime Java conf). I've used the '_' notation which clearly make the things more readable, I had no question related to the fact that people were troubled because 'requires' takes a module and 'exports' takes a package. R?mi ----- Mail original ----- > De: "mark reinhold" > ?: "Remi Forax" > Cc: jpms-spec-experts at openjdk.java.net > Envoy?: Lundi 5 Octobre 2015 20:48:23 > Objet: Re: module name convention > > 2015/9/19 10:02 -0700, forax at univ-mlv.fr: > > Mani from AdoptAJSR has a very good question at the end of this document: > > https://docs.google.com/document/d/1KlumN74IGt-TU-Md3Fn5h4sXHa75RApWNLszUTVp-DE/edit > > > > ... > > > > Both Eclipse and Maven use package name or at least package prefix as > > convention to name module and as Mani said it really confusing when you > > write a module-info file. > > > > Given that the name of the JDK modules doesn't respect that convention > > but still use '.' to separate the different components, I wonder if we > > should not try to come with a convention for module name that allow to > > easily disambiguate between package name and module name. > > Yes, I think that's worth exploring. > > > By example, > > java_base, > > java-base, > > java~base ... > > Quick reactions: > > java_base is still a Java identifier, which is nice. > > java-base corresponds usefully to how JAR files are typically named > (e.g., java-base-1.2.3.jar). > > java~base is pretty ugly (especially with longer names, e.g., > jdk~scripting~nashorn~shell). > > I'm sure there are other possibilities ... > > > Obviously the other solution is to prefix a name by "package" or > > "module". > > As in, `requires module com.foo` and `exports package com.foo`? > > - Mark > From peter.kriens at aqute.biz Wed Oct 14 15:00:50 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Wed, 14 Oct 2015 17:00:50 +0200 Subject: Why not use the Manifest? In-Reply-To: <1993489049.1580799.1444830069389.JavaMail.zimbra@u-pem.fr> References: <20151005185023.443E87B949@eggemoggin.niobe.net> <81AC5311-7835-4451-8381-CA446A311386@aqute.biz> <1993489049.1580799.1444830069389.JavaMail.zimbra@u-pem.fr> Message-ID: <8C724E11-0656-4483-85BD-DFF310F8B463@aqute.biz> But packages are sealed by a manifest header, as is the signing structure. So the JVM must already read the manifest today? Kind regards, Peter Kriens > On 14 okt. 2015, at 15:41, Remi Forax wrote: > > I don't think so, it's part of the tools descriptions, > the manifest is described as part of the java/jar commands. > > Anyway, it's important that the module descriptor is checked at compile time (by javac) and at runtime (by java), > thus, it has to be a part of Java the language with a syntax understandable by javac. > > R?mi > > ----- Mail original ----- >> De: "Peter Kriens" >> ?: "Mark Reinhold" >> Cc: jpms-spec-experts at openjdk.java.net >> Envoy?: Lundi 12 Octobre 2015 18:51:52 >> Objet: Re: Why not use the Manifest? >> >> Isn?t the manifest already part of the JVM specification? >> >> Kind regards, >> >> Peter Kriens >> >>> On 5 okt. 2015, at 20:50, mark.reinhold at oracle.com wrote: >>> >>> 2015/9/28 9:49 -0700, peter.kriens at aqute.biz: >>>> The encoding of the module data in a binary class file remains hard to >>>> understand and seems to go against the best practices we learned the >>>> hard way over the past decades. (Debugging!) >>>> >>>> Especially since the current proposal leaves the heavy lifting to >>>> build tooling. In the proposal, build tools cannot use the >>>> module-info.java since it does not contain sufficient >>>> information. (Versions being the most glaring omission for a build >>>> too.) Nor is the format extensible to contain build tool specific >>>> information. Ergo, the build tool will have to generate the >>>> module-info.class file. >>>> >>>> This make the class file makes these tools unnecessary hard and slower >>>> because it must either use the compiler as an intermediate step or use >>>> a library like ASM to create the unreadable version of its metadata. >>>> >>>> This is the first time in a very long time that I see a regression to >>>> binary files for meta data. Especially because there is already a good >>>> place that all build tools are already using: the manifest. >>> >>> The reasons for expressing module declarations in source files and >>> compiling them into class files are mentioned in SotMS, but here's a >>> longer take on just this topic. >>> >>> The Java programming language, at present, provides for the definition >>> of three kinds of program components: classes, interfaces, and packages. >>> They are defined in Java source files, and compiled into class files. >>> Such files govern, among other things, the mechanisms of symbolic >>> resolution and access control implemented by every Java compiler and >>> JVM. A developer need not reason about any other type of file in order >>> to understand the actions of these mechanisms. A compiler or JVM need >>> not consume or produce any information other than that found in source >>> and class files in order to implement these mechanisms. >>> >>> The module system extends the mechanism of symbolic resolution with the >>> concept of readability in order to provide reliable configuration. It >>> extends the mechanism of access control, in part by relating it to >>> readability, in order to provide strong encapsulation. To support these >>> extensions, every Java compiler and JVM must be able to locate and >>> interpret descriptions of modules which convey, at least, each module's >>> name, dependences, and exported packages. Such descriptions, regardless >>> of their form, will govern symbolic resolution and access control in both >>> compilers and JVMs, so they will have to be specified in both the Java >>> Language Specification (JLS) and the Java Virtual Machine Specification >>> (JVMS). >>> >>> Regardless of how they are described, modules are, fundamentally, a new >>> kind of Java program component. The present proposal therefore treats >>> them as such, in both the language and the JVM: Module descriptions are >>> expressed and encoded in the same way as the other kinds of information >>> that define program components and govern symbolic resolution and access >>> control -- that is, as Java source and class files. >>> >>> This approach is immediately familiar to developers, who already think >>> about program components, symbolic resolution, and access control in >>> terms of the Java programming language. It is easy to retrofit into the >>> JLS and the JVMS, since those specifications are already centered upon >>> source and class files. It is, finally, straightforward for existing >>> Java implementations and tool chains to support, since they need not be >>> revised to handle an entirely new type of file. >>> >>> We could choose another format for module descriptions such as XML, JSON, >>> YAML, or JAR-file manifests. Such formats may be more convenient for >>> tool maintainers but we'd have to bake that format into the JLS and the >>> JVMS, thereby increasing the complexity of those specifications and their >>> implementations. If the format depends upon external standards (XML, >>> JSON, YAML) then these foundational specifications, and their >>> implementations, would become dependent upon those standards. If the >>> format proves unsuitable over time then replacing it with something else >>> would require major revisions to these specifications, and to their >>> implementations. >>> >>> Source and class files are fundamental to the Java platform, to the JLS >>> and the JVMS, and to their implementations and supporting tools. Other >>> formats come and go; these will be with us forever. >>> >>> * * * >>> >>> As to debugging, in the prototype we've already enhanced the jar tool >>> with an option to print the descriptor of a modular JAR file. I expect >>> similar support to show up in related tools such as jmod and javap, and >>> eventually in IDEs and other tools outside the JDK. >>> >>> As to ease of tooling, most tools will only need to read module >>> descriptors, not write them, and the java.lang.module.ModuleDescriptor >>> class already provides convenient static methods for doing that. Adding >>> similar methods to write descriptors would be straightforward, though it >>> would complicate the API a bit. >>> >>> As to extensibility, class files are already in a precisely-specified and >>> extensible format. If a build tool needs to add information to a module >>> descriptor then it can do so via non-standard class-file attributes, >>> using an existing popular library such as ASM. >>> >>> To make it easier for tools to manipulate module descriptors we could >>> consider enhancing the java.lang.module.ModuleDescriptor class, and >>> related tools, to read and write custom, non-standard class-file >>> attributes. This might be a bit tricky depending on how the attributes >>> are defined, but for simple property-style attributes it's likely >>> straightforward and would obviate the need to use ASM directly. >>> >>> - Mark >> >> From forax at univ-mlv.fr Wed Oct 14 19:28:11 2015 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Wed, 14 Oct 2015 21:28:11 +0200 (CEST) Subject: Why not use the Manifest? In-Reply-To: <8C724E11-0656-4483-85BD-DFF310F8B463@aqute.biz> References: <20151005185023.443E87B949@eggemoggin.niobe.net> <81AC5311-7835-4451-8381-CA446A311386@aqute.biz> <1993489049.1580799.1444830069389.JavaMail.zimbra@u-pem.fr> <8C724E11-0656-4483-85BD-DFF310F8B463@aqute.biz> Message-ID: <673841498.1783560.1444850891146.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "Peter Kriens" > ?: "Remi Forax" > Cc: "Mark Reinhold" , jpms-spec-experts at openjdk.java.net > Envoy?: Mercredi 14 Octobre 2015 17:00:50 > Objet: Re: Why not use the Manifest? > > But packages are sealed by a manifest header, as is the signing structure. So > the JVM must already read the manifest today? > > Kind regards, > > Peter Kriens Some JVM parse the manifest, but it's not part of the JVM spec. But where are losing ourselves in the details. The compiler should check if the module-info is well formed so the error will be reported at compile time and not at runtime. Now, if the compiler checks the module-info file, the textual representation should be described in the JLS and binary representation described in the JVMS. Using a text file in that context will be weird. regards, R?mi > > > On 14 okt. 2015, at 15:41, Remi Forax wrote: > > > > I don't think so, it's part of the tools descriptions, > > the manifest is described as part of the java/jar commands. > > > > Anyway, it's important that the module descriptor is checked at compile > > time (by javac) and at runtime (by java), > > thus, it has to be a part of Java the language with a syntax understandable > > by javac. > > > > R?mi > > > > ----- Mail original ----- > >> De: "Peter Kriens" > >> ?: "Mark Reinhold" > >> Cc: jpms-spec-experts at openjdk.java.net > >> Envoy?: Lundi 12 Octobre 2015 18:51:52 > >> Objet: Re: Why not use the Manifest? > >> > >> Isn?t the manifest already part of the JVM specification? > >> > >> Kind regards, > >> > >> Peter Kriens > >> > >>> On 5 okt. 2015, at 20:50, mark.reinhold at oracle.com wrote: > >>> > >>> 2015/9/28 9:49 -0700, peter.kriens at aqute.biz: > >>>> The encoding of the module data in a binary class file remains hard to > >>>> understand and seems to go against the best practices we learned the > >>>> hard way over the past decades. (Debugging!) > >>>> > >>>> Especially since the current proposal leaves the heavy lifting to > >>>> build tooling. In the proposal, build tools cannot use the > >>>> module-info.java since it does not contain sufficient > >>>> information. (Versions being the most glaring omission for a build > >>>> too.) Nor is the format extensible to contain build tool specific > >>>> information. Ergo, the build tool will have to generate the > >>>> module-info.class file. > >>>> > >>>> This make the class file makes these tools unnecessary hard and slower > >>>> because it must either use the compiler as an intermediate step or use > >>>> a library like ASM to create the unreadable version of its metadata. > >>>> > >>>> This is the first time in a very long time that I see a regression to > >>>> binary files for meta data. Especially because there is already a good > >>>> place that all build tools are already using: the manifest. > >>> > >>> The reasons for expressing module declarations in source files and > >>> compiling them into class files are mentioned in SotMS, but here's a > >>> longer take on just this topic. > >>> > >>> The Java programming language, at present, provides for the definition > >>> of three kinds of program components: classes, interfaces, and packages. > >>> They are defined in Java source files, and compiled into class files. > >>> Such files govern, among other things, the mechanisms of symbolic > >>> resolution and access control implemented by every Java compiler and > >>> JVM. A developer need not reason about any other type of file in order > >>> to understand the actions of these mechanisms. A compiler or JVM need > >>> not consume or produce any information other than that found in source > >>> and class files in order to implement these mechanisms. > >>> > >>> The module system extends the mechanism of symbolic resolution with the > >>> concept of readability in order to provide reliable configuration. It > >>> extends the mechanism of access control, in part by relating it to > >>> readability, in order to provide strong encapsulation. To support these > >>> extensions, every Java compiler and JVM must be able to locate and > >>> interpret descriptions of modules which convey, at least, each module's > >>> name, dependences, and exported packages. Such descriptions, regardless > >>> of their form, will govern symbolic resolution and access control in both > >>> compilers and JVMs, so they will have to be specified in both the Java > >>> Language Specification (JLS) and the Java Virtual Machine Specification > >>> (JVMS). > >>> > >>> Regardless of how they are described, modules are, fundamentally, a new > >>> kind of Java program component. The present proposal therefore treats > >>> them as such, in both the language and the JVM: Module descriptions are > >>> expressed and encoded in the same way as the other kinds of information > >>> that define program components and govern symbolic resolution and access > >>> control -- that is, as Java source and class files. > >>> > >>> This approach is immediately familiar to developers, who already think > >>> about program components, symbolic resolution, and access control in > >>> terms of the Java programming language. It is easy to retrofit into the > >>> JLS and the JVMS, since those specifications are already centered upon > >>> source and class files. It is, finally, straightforward for existing > >>> Java implementations and tool chains to support, since they need not be > >>> revised to handle an entirely new type of file. > >>> > >>> We could choose another format for module descriptions such as XML, JSON, > >>> YAML, or JAR-file manifests. Such formats may be more convenient for > >>> tool maintainers but we'd have to bake that format into the JLS and the > >>> JVMS, thereby increasing the complexity of those specifications and their > >>> implementations. If the format depends upon external standards (XML, > >>> JSON, YAML) then these foundational specifications, and their > >>> implementations, would become dependent upon those standards. If the > >>> format proves unsuitable over time then replacing it with something else > >>> would require major revisions to these specifications, and to their > >>> implementations. > >>> > >>> Source and class files are fundamental to the Java platform, to the JLS > >>> and the JVMS, and to their implementations and supporting tools. Other > >>> formats come and go; these will be with us forever. > >>> > >>> * * * > >>> > >>> As to debugging, in the prototype we've already enhanced the jar tool > >>> with an option to print the descriptor of a modular JAR file. I expect > >>> similar support to show up in related tools such as jmod and javap, and > >>> eventually in IDEs and other tools outside the JDK. > >>> > >>> As to ease of tooling, most tools will only need to read module > >>> descriptors, not write them, and the java.lang.module.ModuleDescriptor > >>> class already provides convenient static methods for doing that. Adding > >>> similar methods to write descriptors would be straightforward, though it > >>> would complicate the API a bit. > >>> > >>> As to extensibility, class files are already in a precisely-specified and > >>> extensible format. If a build tool needs to add information to a module > >>> descriptor then it can do so via non-standard class-file attributes, > >>> using an existing popular library such as ASM. > >>> > >>> To make it easier for tools to manipulate module descriptors we could > >>> consider enhancing the java.lang.module.ModuleDescriptor class, and > >>> related tools, to read and write custom, non-standard class-file > >>> attributes. This might be a bit tricky depending on how the attributes > >>> are defined, but for simple property-style attributes it's likely > >>> straightforward and would obviate the need to use ASM directly. > >>> > >>> - Mark > >> > >> > > From peter.kriens at aqute.biz Thu Oct 15 06:53:28 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Thu, 15 Oct 2015 08:53:28 +0200 Subject: Why not use the Manifest? In-Reply-To: <673841498.1783560.1444850891146.JavaMail.zimbra@u-pem.fr> References: <20151005185023.443E87B949@eggemoggin.niobe.net> <81AC5311-7835-4451-8381-CA446A311386@aqute.biz> <1993489049.1580799.1444830069389.JavaMail.zimbra@u-pem.fr> <8C724E11-0656-4483-85BD-DFF310F8B463@aqute.biz> <673841498.1783560.1444850891146.JavaMail.zimbra@u-pem.fr> Message-ID: <38A65E32-EECE-44DF-B646-922CF9637C7D@aqute.biz> I think our differences come from the fact that you assume that module-info.java will be written by developers while I fail how to see that can work realistically. The build system must already have metadata that contains module name and version (both for the current project and the dependencies), for example the pom.xml in maven. You think it is realistic to expect people will write the module name in two places? For all practical cases, will the module info not be generated by the build system? Kind regards, Peter Kriens > On 14 okt. 2015, at 21:28, forax at univ-mlv.fr wrote: > > ----- Mail original ----- >> De: "Peter Kriens" >> ?: "Remi Forax" >> Cc: "Mark Reinhold" , jpms-spec-experts at openjdk.java.net >> Envoy?: Mercredi 14 Octobre 2015 17:00:50 >> Objet: Re: Why not use the Manifest? >> >> But packages are sealed by a manifest header, as is the signing structure. So >> the JVM must already read the manifest today? >> >> Kind regards, >> >> Peter Kriens > > Some JVM parse the manifest, but it's not part of the JVM spec. > > But where are losing ourselves in the details. > The compiler should check if the module-info is well formed so the error will be reported at compile time and not at runtime. > Now, if the compiler checks the module-info file, the textual representation should be described in the JLS and > binary representation described in the JVMS. > > Using a text file in that context will be weird. > > regards, > R?mi > >> >>> On 14 okt. 2015, at 15:41, Remi Forax wrote: >>> >>> I don't think so, it's part of the tools descriptions, >>> the manifest is described as part of the java/jar commands. >>> >>> Anyway, it's important that the module descriptor is checked at compile >>> time (by javac) and at runtime (by java), >>> thus, it has to be a part of Java the language with a syntax understandable >>> by javac. >>> >>> R?mi >>> >>> ----- Mail original ----- >>>> De: "Peter Kriens" >>>> ?: "Mark Reinhold" >>>> Cc: jpms-spec-experts at openjdk.java.net >>>> Envoy?: Lundi 12 Octobre 2015 18:51:52 >>>> Objet: Re: Why not use the Manifest? >>>> >>>> Isn?t the manifest already part of the JVM specification? >>>> >>>> Kind regards, >>>> >>>> Peter Kriens >>>> >>>>> On 5 okt. 2015, at 20:50, mark.reinhold at oracle.com wrote: >>>>> >>>>> 2015/9/28 9:49 -0700, peter.kriens at aqute.biz: >>>>>> The encoding of the module data in a binary class file remains hard to >>>>>> understand and seems to go against the best practices we learned the >>>>>> hard way over the past decades. (Debugging!) >>>>>> >>>>>> Especially since the current proposal leaves the heavy lifting to >>>>>> build tooling. In the proposal, build tools cannot use the >>>>>> module-info.java since it does not contain sufficient >>>>>> information. (Versions being the most glaring omission for a build >>>>>> too.) Nor is the format extensible to contain build tool specific >>>>>> information. Ergo, the build tool will have to generate the >>>>>> module-info.class file. >>>>>> >>>>>> This make the class file makes these tools unnecessary hard and slower >>>>>> because it must either use the compiler as an intermediate step or use >>>>>> a library like ASM to create the unreadable version of its metadata. >>>>>> >>>>>> This is the first time in a very long time that I see a regression to >>>>>> binary files for meta data. Especially because there is already a good >>>>>> place that all build tools are already using: the manifest. >>>>> >>>>> The reasons for expressing module declarations in source files and >>>>> compiling them into class files are mentioned in SotMS, but here's a >>>>> longer take on just this topic. >>>>> >>>>> The Java programming language, at present, provides for the definition >>>>> of three kinds of program components: classes, interfaces, and packages. >>>>> They are defined in Java source files, and compiled into class files. >>>>> Such files govern, among other things, the mechanisms of symbolic >>>>> resolution and access control implemented by every Java compiler and >>>>> JVM. A developer need not reason about any other type of file in order >>>>> to understand the actions of these mechanisms. A compiler or JVM need >>>>> not consume or produce any information other than that found in source >>>>> and class files in order to implement these mechanisms. >>>>> >>>>> The module system extends the mechanism of symbolic resolution with the >>>>> concept of readability in order to provide reliable configuration. It >>>>> extends the mechanism of access control, in part by relating it to >>>>> readability, in order to provide strong encapsulation. To support these >>>>> extensions, every Java compiler and JVM must be able to locate and >>>>> interpret descriptions of modules which convey, at least, each module's >>>>> name, dependences, and exported packages. Such descriptions, regardless >>>>> of their form, will govern symbolic resolution and access control in both >>>>> compilers and JVMs, so they will have to be specified in both the Java >>>>> Language Specification (JLS) and the Java Virtual Machine Specification >>>>> (JVMS). >>>>> >>>>> Regardless of how they are described, modules are, fundamentally, a new >>>>> kind of Java program component. The present proposal therefore treats >>>>> them as such, in both the language and the JVM: Module descriptions are >>>>> expressed and encoded in the same way as the other kinds of information >>>>> that define program components and govern symbolic resolution and access >>>>> control -- that is, as Java source and class files. >>>>> >>>>> This approach is immediately familiar to developers, who already think >>>>> about program components, symbolic resolution, and access control in >>>>> terms of the Java programming language. It is easy to retrofit into the >>>>> JLS and the JVMS, since those specifications are already centered upon >>>>> source and class files. It is, finally, straightforward for existing >>>>> Java implementations and tool chains to support, since they need not be >>>>> revised to handle an entirely new type of file. >>>>> >>>>> We could choose another format for module descriptions such as XML, JSON, >>>>> YAML, or JAR-file manifests. Such formats may be more convenient for >>>>> tool maintainers but we'd have to bake that format into the JLS and the >>>>> JVMS, thereby increasing the complexity of those specifications and their >>>>> implementations. If the format depends upon external standards (XML, >>>>> JSON, YAML) then these foundational specifications, and their >>>>> implementations, would become dependent upon those standards. If the >>>>> format proves unsuitable over time then replacing it with something else >>>>> would require major revisions to these specifications, and to their >>>>> implementations. >>>>> >>>>> Source and class files are fundamental to the Java platform, to the JLS >>>>> and the JVMS, and to their implementations and supporting tools. Other >>>>> formats come and go; these will be with us forever. >>>>> >>>>> * * * >>>>> >>>>> As to debugging, in the prototype we've already enhanced the jar tool >>>>> with an option to print the descriptor of a modular JAR file. I expect >>>>> similar support to show up in related tools such as jmod and javap, and >>>>> eventually in IDEs and other tools outside the JDK. >>>>> >>>>> As to ease of tooling, most tools will only need to read module >>>>> descriptors, not write them, and the java.lang.module.ModuleDescriptor >>>>> class already provides convenient static methods for doing that. Adding >>>>> similar methods to write descriptors would be straightforward, though it >>>>> would complicate the API a bit. >>>>> >>>>> As to extensibility, class files are already in a precisely-specified and >>>>> extensible format. If a build tool needs to add information to a module >>>>> descriptor then it can do so via non-standard class-file attributes, >>>>> using an existing popular library such as ASM. >>>>> >>>>> To make it easier for tools to manipulate module descriptors we could >>>>> consider enhancing the java.lang.module.ModuleDescriptor class, and >>>>> related tools, to read and write custom, non-standard class-file >>>>> attributes. This might be a bit tricky depending on how the attributes >>>>> are defined, but for simple property-style attributes it's likely >>>>> straightforward and would obviate the need to use ASM directly. >>>>> >>>>> - Mark >>>> >>>> >> >> From forax at univ-mlv.fr Thu Oct 15 14:26:28 2015 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Thu, 15 Oct 2015 16:26:28 +0200 (CEST) Subject: Why not use the Manifest? In-Reply-To: <38A65E32-EECE-44DF-B646-922CF9637C7D@aqute.biz> References: <20151005185023.443E87B949@eggemoggin.niobe.net> <81AC5311-7835-4451-8381-CA446A311386@aqute.biz> <1993489049.1580799.1444830069389.JavaMail.zimbra@u-pem.fr> <8C724E11-0656-4483-85BD-DFF310F8B463@aqute.biz> <673841498.1783560.1444850891146.JavaMail.zimbra@u-pem.fr> <38A65E32-EECE-44DF-B646-922CF9637C7D@aqute.biz> Message-ID: <685267555.2219378.1444919188588.JavaMail.zimbra@u-pem.fr> I assume that some developpers will want to declare the module-info by hand. Imagine that i want a build system that obey to semver and calculate the version of the modules i create by itself. The version is calculated by checking the bytecodes against the latest produced artefact, and the version is bumped automatically. If there is no conflict, the build tool only needs the major version of the required modules. regards, R?mi ----- Mail original ----- > De: "Peter Kriens" > ?: forax at univ-mlv.fr > Cc: "Mark Reinhold" , jpms-spec-experts at openjdk.java.net > Envoy?: Jeudi 15 Octobre 2015 08:53:28 > Objet: Re: Why not use the Manifest? > > I think our differences come from the fact that you assume that > module-info.java will be written by developers while I fail how to see that > can work realistically. > > The build system must already have metadata that contains module name and > version (both for the current project and the dependencies), for example the > pom.xml in maven. You think it is realistic to expect people will write the > module name in two places? > > For all practical cases, will the module info not be generated by the build > system? > > Kind regards, > > Peter Kriens > > > > > > On 14 okt. 2015, at 21:28, forax at univ-mlv.fr wrote: > > > > ----- Mail original ----- > >> De: "Peter Kriens" > >> ?: "Remi Forax" > >> Cc: "Mark Reinhold" , > >> jpms-spec-experts at openjdk.java.net > >> Envoy?: Mercredi 14 Octobre 2015 17:00:50 > >> Objet: Re: Why not use the Manifest? > >> > >> But packages are sealed by a manifest header, as is the signing structure. > >> So > >> the JVM must already read the manifest today? > >> > >> Kind regards, > >> > >> Peter Kriens > > > > Some JVM parse the manifest, but it's not part of the JVM spec. > > > > But where are losing ourselves in the details. > > The compiler should check if the module-info is well formed so the error > > will be reported at compile time and not at runtime. > > Now, if the compiler checks the module-info file, the textual > > representation should be described in the JLS and > > binary representation described in the JVMS. > > > > Using a text file in that context will be weird. > > > > regards, > > R?mi > > > >> > >>> On 14 okt. 2015, at 15:41, Remi Forax wrote: > >>> > >>> I don't think so, it's part of the tools descriptions, > >>> the manifest is described as part of the java/jar commands. > >>> > >>> Anyway, it's important that the module descriptor is checked at compile > >>> time (by javac) and at runtime (by java), > >>> thus, it has to be a part of Java the language with a syntax > >>> understandable > >>> by javac. > >>> > >>> R?mi > >>> > >>> ----- Mail original ----- > >>>> De: "Peter Kriens" > >>>> ?: "Mark Reinhold" > >>>> Cc: jpms-spec-experts at openjdk.java.net > >>>> Envoy?: Lundi 12 Octobre 2015 18:51:52 > >>>> Objet: Re: Why not use the Manifest? > >>>> > >>>> Isn?t the manifest already part of the JVM specification? > >>>> > >>>> Kind regards, > >>>> > >>>> Peter Kriens > >>>> > >>>>> On 5 okt. 2015, at 20:50, mark.reinhold at oracle.com wrote: > >>>>> > >>>>> 2015/9/28 9:49 -0700, peter.kriens at aqute.biz: > >>>>>> The encoding of the module data in a binary class file remains hard to > >>>>>> understand and seems to go against the best practices we learned the > >>>>>> hard way over the past decades. (Debugging!) > >>>>>> > >>>>>> Especially since the current proposal leaves the heavy lifting to > >>>>>> build tooling. In the proposal, build tools cannot use the > >>>>>> module-info.java since it does not contain sufficient > >>>>>> information. (Versions being the most glaring omission for a build > >>>>>> too.) Nor is the format extensible to contain build tool specific > >>>>>> information. Ergo, the build tool will have to generate the > >>>>>> module-info.class file. > >>>>>> > >>>>>> This make the class file makes these tools unnecessary hard and slower > >>>>>> because it must either use the compiler as an intermediate step or use > >>>>>> a library like ASM to create the unreadable version of its metadata. > >>>>>> > >>>>>> This is the first time in a very long time that I see a regression to > >>>>>> binary files for meta data. Especially because there is already a good > >>>>>> place that all build tools are already using: the manifest. > >>>>> > >>>>> The reasons for expressing module declarations in source files and > >>>>> compiling them into class files are mentioned in SotMS, but here's a > >>>>> longer take on just this topic. > >>>>> > >>>>> The Java programming language, at present, provides for the definition > >>>>> of three kinds of program components: classes, interfaces, and > >>>>> packages. > >>>>> They are defined in Java source files, and compiled into class files. > >>>>> Such files govern, among other things, the mechanisms of symbolic > >>>>> resolution and access control implemented by every Java compiler and > >>>>> JVM. A developer need not reason about any other type of file in order > >>>>> to understand the actions of these mechanisms. A compiler or JVM need > >>>>> not consume or produce any information other than that found in source > >>>>> and class files in order to implement these mechanisms. > >>>>> > >>>>> The module system extends the mechanism of symbolic resolution with the > >>>>> concept of readability in order to provide reliable configuration. It > >>>>> extends the mechanism of access control, in part by relating it to > >>>>> readability, in order to provide strong encapsulation. To support > >>>>> these > >>>>> extensions, every Java compiler and JVM must be able to locate and > >>>>> interpret descriptions of modules which convey, at least, each module's > >>>>> name, dependences, and exported packages. Such descriptions, > >>>>> regardless > >>>>> of their form, will govern symbolic resolution and access control in > >>>>> both > >>>>> compilers and JVMs, so they will have to be specified in both the Java > >>>>> Language Specification (JLS) and the Java Virtual Machine Specification > >>>>> (JVMS). > >>>>> > >>>>> Regardless of how they are described, modules are, fundamentally, a new > >>>>> kind of Java program component. The present proposal therefore treats > >>>>> them as such, in both the language and the JVM: Module descriptions are > >>>>> expressed and encoded in the same way as the other kinds of information > >>>>> that define program components and govern symbolic resolution and > >>>>> access > >>>>> control -- that is, as Java source and class files. > >>>>> > >>>>> This approach is immediately familiar to developers, who already think > >>>>> about program components, symbolic resolution, and access control in > >>>>> terms of the Java programming language. It is easy to retrofit into > >>>>> the > >>>>> JLS and the JVMS, since those specifications are already centered upon > >>>>> source and class files. It is, finally, straightforward for existing > >>>>> Java implementations and tool chains to support, since they need not be > >>>>> revised to handle an entirely new type of file. > >>>>> > >>>>> We could choose another format for module descriptions such as XML, > >>>>> JSON, > >>>>> YAML, or JAR-file manifests. Such formats may be more convenient for > >>>>> tool maintainers but we'd have to bake that format into the JLS and the > >>>>> JVMS, thereby increasing the complexity of those specifications and > >>>>> their > >>>>> implementations. If the format depends upon external standards (XML, > >>>>> JSON, YAML) then these foundational specifications, and their > >>>>> implementations, would become dependent upon those standards. If the > >>>>> format proves unsuitable over time then replacing it with something > >>>>> else > >>>>> would require major revisions to these specifications, and to their > >>>>> implementations. > >>>>> > >>>>> Source and class files are fundamental to the Java platform, to the JLS > >>>>> and the JVMS, and to their implementations and supporting tools. Other > >>>>> formats come and go; these will be with us forever. > >>>>> > >>>>> * * * > >>>>> > >>>>> As to debugging, in the prototype we've already enhanced the jar tool > >>>>> with an option to print the descriptor of a modular JAR file. I expect > >>>>> similar support to show up in related tools such as jmod and javap, and > >>>>> eventually in IDEs and other tools outside the JDK. > >>>>> > >>>>> As to ease of tooling, most tools will only need to read module > >>>>> descriptors, not write them, and the java.lang.module.ModuleDescriptor > >>>>> class already provides convenient static methods for doing that. > >>>>> Adding > >>>>> similar methods to write descriptors would be straightforward, though > >>>>> it > >>>>> would complicate the API a bit. > >>>>> > >>>>> As to extensibility, class files are already in a precisely-specified > >>>>> and > >>>>> extensible format. If a build tool needs to add information to a > >>>>> module > >>>>> descriptor then it can do so via non-standard class-file attributes, > >>>>> using an existing popular library such as ASM. > >>>>> > >>>>> To make it easier for tools to manipulate module descriptors we could > >>>>> consider enhancing the java.lang.module.ModuleDescriptor class, and > >>>>> related tools, to read and write custom, non-standard class-file > >>>>> attributes. This might be a bit tricky depending on how the attributes > >>>>> are defined, but for simple property-style attributes it's likely > >>>>> straightforward and would obviate the need to use ASM directly. > >>>>> > >>>>> - Mark > >>>> > >>>> > >> > >> > > From peter.kriens at aqute.biz Thu Oct 15 14:53:26 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Thu, 15 Oct 2015 16:53:26 +0200 Subject: Why not use the Manifest? In-Reply-To: <685267555.2219378.1444919188588.JavaMail.zimbra@u-pem.fr> References: <20151005185023.443E87B949@eggemoggin.niobe.net> <81AC5311-7835-4451-8381-CA446A311386@aqute.biz> <1993489049.1580799.1444830069389.JavaMail.zimbra@u-pem.fr> <8C724E11-0656-4483-85BD-DFF310F8B463@aqute.biz> <673841498.1783560.1444850891146.JavaMail.zimbra@u-pem.fr> <38A65E32-EECE-44DF-B646-922CF9637C7D@aqute.biz> <685267555.2219378.1444919188588.JavaMail.zimbra@u-pem.fr> Message-ID: <12677039-8FD6-4C96-814F-7D1DFCD62270@aqute.biz> > On 15 okt. 2015, at 16:26, forax at univ-mlv.fr wrote: > I assume that some developpers will want to declare the module-info by hand. Ok, but isn?t this then always redundant? Isn?t the build system forced to have the same (or at least closely corresponding) information with a version? > Imagine that i want a build system that obey to semver and calculate the version of the modules i create by itself. > The version is calculated by checking the bytecodes against the latest produced artefact, and the version is bumped automatically. > If there is no conflict, the build tool only needs the major version of the required modules. It sounds a bit like what bnd does but I am not completely sure I understand what you mean. Could you elaborate? Kind regards, Peter Kriens > > regards, > R?mi > > ----- Mail original ----- >> De: "Peter Kriens" >> ?: forax at univ-mlv.fr >> Cc: "Mark Reinhold" , jpms-spec-experts at openjdk.java.net >> Envoy?: Jeudi 15 Octobre 2015 08:53:28 >> Objet: Re: Why not use the Manifest? >> >> I think our differences come from the fact that you assume that >> module-info.java will be written by developers while I fail how to see that >> can work realistically. >> >> The build system must already have metadata that contains module name and >> version (both for the current project and the dependencies), for example the >> pom.xml in maven. You think it is realistic to expect people will write the >> module name in two places? >> >> For all practical cases, will the module info not be generated by the build >> system? >> >> Kind regards, >> >> Peter Kriens >> >> >> >> >>> On 14 okt. 2015, at 21:28, forax at univ-mlv.fr wrote: >>> >>> ----- Mail original ----- >>>> De: "Peter Kriens" >>>> ?: "Remi Forax" >>>> Cc: "Mark Reinhold" , >>>> jpms-spec-experts at openjdk.java.net >>>> Envoy?: Mercredi 14 Octobre 2015 17:00:50 >>>> Objet: Re: Why not use the Manifest? >>>> >>>> But packages are sealed by a manifest header, as is the signing structure. >>>> So >>>> the JVM must already read the manifest today? >>>> >>>> Kind regards, >>>> >>>> Peter Kriens >>> >>> Some JVM parse the manifest, but it's not part of the JVM spec. >>> >>> But where are losing ourselves in the details. >>> The compiler should check if the module-info is well formed so the error >>> will be reported at compile time and not at runtime. >>> Now, if the compiler checks the module-info file, the textual >>> representation should be described in the JLS and >>> binary representation described in the JVMS. >>> >>> Using a text file in that context will be weird. >>> >>> regards, >>> R?mi >>> >>>> >>>>> On 14 okt. 2015, at 15:41, Remi Forax wrote: >>>>> >>>>> I don't think so, it's part of the tools descriptions, >>>>> the manifest is described as part of the java/jar commands. >>>>> >>>>> Anyway, it's important that the module descriptor is checked at compile >>>>> time (by javac) and at runtime (by java), >>>>> thus, it has to be a part of Java the language with a syntax >>>>> understandable >>>>> by javac. >>>>> >>>>> R?mi >>>>> >>>>> ----- Mail original ----- >>>>>> De: "Peter Kriens" >>>>>> ?: "Mark Reinhold" >>>>>> Cc: jpms-spec-experts at openjdk.java.net >>>>>> Envoy?: Lundi 12 Octobre 2015 18:51:52 >>>>>> Objet: Re: Why not use the Manifest? >>>>>> >>>>>> Isn?t the manifest already part of the JVM specification? >>>>>> >>>>>> Kind regards, >>>>>> >>>>>> Peter Kriens >>>>>> >>>>>>> On 5 okt. 2015, at 20:50, mark.reinhold at oracle.com wrote: >>>>>>> >>>>>>> 2015/9/28 9:49 -0700, peter.kriens at aqute.biz: >>>>>>>> The encoding of the module data in a binary class file remains hard to >>>>>>>> understand and seems to go against the best practices we learned the >>>>>>>> hard way over the past decades. (Debugging!) >>>>>>>> >>>>>>>> Especially since the current proposal leaves the heavy lifting to >>>>>>>> build tooling. In the proposal, build tools cannot use the >>>>>>>> module-info.java since it does not contain sufficient >>>>>>>> information. (Versions being the most glaring omission for a build >>>>>>>> too.) Nor is the format extensible to contain build tool specific >>>>>>>> information. Ergo, the build tool will have to generate the >>>>>>>> module-info.class file. >>>>>>>> >>>>>>>> This make the class file makes these tools unnecessary hard and slower >>>>>>>> because it must either use the compiler as an intermediate step or use >>>>>>>> a library like ASM to create the unreadable version of its metadata. >>>>>>>> >>>>>>>> This is the first time in a very long time that I see a regression to >>>>>>>> binary files for meta data. Especially because there is already a good >>>>>>>> place that all build tools are already using: the manifest. >>>>>>> >>>>>>> The reasons for expressing module declarations in source files and >>>>>>> compiling them into class files are mentioned in SotMS, but here's a >>>>>>> longer take on just this topic. >>>>>>> >>>>>>> The Java programming language, at present, provides for the definition >>>>>>> of three kinds of program components: classes, interfaces, and >>>>>>> packages. >>>>>>> They are defined in Java source files, and compiled into class files. >>>>>>> Such files govern, among other things, the mechanisms of symbolic >>>>>>> resolution and access control implemented by every Java compiler and >>>>>>> JVM. A developer need not reason about any other type of file in order >>>>>>> to understand the actions of these mechanisms. A compiler or JVM need >>>>>>> not consume or produce any information other than that found in source >>>>>>> and class files in order to implement these mechanisms. >>>>>>> >>>>>>> The module system extends the mechanism of symbolic resolution with the >>>>>>> concept of readability in order to provide reliable configuration. It >>>>>>> extends the mechanism of access control, in part by relating it to >>>>>>> readability, in order to provide strong encapsulation. To support >>>>>>> these >>>>>>> extensions, every Java compiler and JVM must be able to locate and >>>>>>> interpret descriptions of modules which convey, at least, each module's >>>>>>> name, dependences, and exported packages. Such descriptions, >>>>>>> regardless >>>>>>> of their form, will govern symbolic resolution and access control in >>>>>>> both >>>>>>> compilers and JVMs, so they will have to be specified in both the Java >>>>>>> Language Specification (JLS) and the Java Virtual Machine Specification >>>>>>> (JVMS). >>>>>>> >>>>>>> Regardless of how they are described, modules are, fundamentally, a new >>>>>>> kind of Java program component. The present proposal therefore treats >>>>>>> them as such, in both the language and the JVM: Module descriptions are >>>>>>> expressed and encoded in the same way as the other kinds of information >>>>>>> that define program components and govern symbolic resolution and >>>>>>> access >>>>>>> control -- that is, as Java source and class files. >>>>>>> >>>>>>> This approach is immediately familiar to developers, who already think >>>>>>> about program components, symbolic resolution, and access control in >>>>>>> terms of the Java programming language. It is easy to retrofit into >>>>>>> the >>>>>>> JLS and the JVMS, since those specifications are already centered upon >>>>>>> source and class files. It is, finally, straightforward for existing >>>>>>> Java implementations and tool chains to support, since they need not be >>>>>>> revised to handle an entirely new type of file. >>>>>>> >>>>>>> We could choose another format for module descriptions such as XML, >>>>>>> JSON, >>>>>>> YAML, or JAR-file manifests. Such formats may be more convenient for >>>>>>> tool maintainers but we'd have to bake that format into the JLS and the >>>>>>> JVMS, thereby increasing the complexity of those specifications and >>>>>>> their >>>>>>> implementations. If the format depends upon external standards (XML, >>>>>>> JSON, YAML) then these foundational specifications, and their >>>>>>> implementations, would become dependent upon those standards. If the >>>>>>> format proves unsuitable over time then replacing it with something >>>>>>> else >>>>>>> would require major revisions to these specifications, and to their >>>>>>> implementations. >>>>>>> >>>>>>> Source and class files are fundamental to the Java platform, to the JLS >>>>>>> and the JVMS, and to their implementations and supporting tools. Other >>>>>>> formats come and go; these will be with us forever. >>>>>>> >>>>>>> * * * >>>>>>> >>>>>>> As to debugging, in the prototype we've already enhanced the jar tool >>>>>>> with an option to print the descriptor of a modular JAR file. I expect >>>>>>> similar support to show up in related tools such as jmod and javap, and >>>>>>> eventually in IDEs and other tools outside the JDK. >>>>>>> >>>>>>> As to ease of tooling, most tools will only need to read module >>>>>>> descriptors, not write them, and the java.lang.module.ModuleDescriptor >>>>>>> class already provides convenient static methods for doing that. >>>>>>> Adding >>>>>>> similar methods to write descriptors would be straightforward, though >>>>>>> it >>>>>>> would complicate the API a bit. >>>>>>> >>>>>>> As to extensibility, class files are already in a precisely-specified >>>>>>> and >>>>>>> extensible format. If a build tool needs to add information to a >>>>>>> module >>>>>>> descriptor then it can do so via non-standard class-file attributes, >>>>>>> using an existing popular library such as ASM. >>>>>>> >>>>>>> To make it easier for tools to manipulate module descriptors we could >>>>>>> consider enhancing the java.lang.module.ModuleDescriptor class, and >>>>>>> related tools, to read and write custom, non-standard class-file >>>>>>> attributes. This might be a bit tricky depending on how the attributes >>>>>>> are defined, but for simple property-style attributes it's likely >>>>>>> straightforward and would obviate the need to use ASM directly. >>>>>>> >>>>>>> - Mark >>>>>> >>>>>> >>>> >>>> >> >> From forax at univ-mlv.fr Thu Oct 15 15:40:33 2015 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Thu, 15 Oct 2015 17:40:33 +0200 (CEST) Subject: Why not use the Manifest? In-Reply-To: <12677039-8FD6-4C96-814F-7D1DFCD62270@aqute.biz> References: <81AC5311-7835-4451-8381-CA446A311386@aqute.biz> <1993489049.1580799.1444830069389.JavaMail.zimbra@u-pem.fr> <8C724E11-0656-4483-85BD-DFF310F8B463@aqute.biz> <673841498.1783560.1444850891146.JavaMail.zimbra@u-pem.fr> <38A65E32-EECE-44DF-B646-922CF9637C7D@aqute.biz> <685267555.2219378.1444919188588.JavaMail.zimbra@u-pem.fr> <12677039-8FD6-4C96-814F-7D1DFCD62270@aqute.biz> Message-ID: <854227844.2262184.1444923633889.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "Peter Kriens" > ?: forax at univ-mlv.fr > Cc: "Mark Reinhold" , jpms-spec-experts at openjdk.java.net > Envoy?: Jeudi 15 Octobre 2015 16:53:26 > Objet: Re: Why not use the Manifest? > > > On 15 okt. 2015, at 16:26, forax at univ-mlv.fr wrote: > > I assume that some developers will want to declare the module-info by > > hand. > Ok, but isn?t this then always redundant? > > Isn?t the build system forced to have the same (or at least closely > corresponding) information with a version? If your build tool is able to increment version automatically, you don't really need versions. Let's take an example, we have JUnit3 and JUnit4. JUnit3 is backward compatible, JUnit4 is backward compatible, JUnit3 is not compatible with JUnit4, that's obvious because they change the package prefix name. In a module world, JUnit3 and JUnit4 will not have the same module name, otherwise, using the most recent module implementation is fine (if you have a tool that doesn't do any auto-update of the required modules). There is one case where you need a version, when you want to blacklist a release because of a bug, for that you will need to tell the build tool to avoid a peculiar module version. Anyway, with this tool, the version is something that tag a module jar, not something the developer as to care of (developers are bad at managing artifact version). R?mi From peter.kriens at aqute.biz Thu Oct 15 17:01:03 2015 From: peter.kriens at aqute.biz (Peter Kriens) Date: Thu, 15 Oct 2015 19:01:03 +0200 Subject: Why not use the Manifest? In-Reply-To: <854227844.2262184.1444923633889.JavaMail.zimbra@u-pem.fr> References: <81AC5311-7835-4451-8381-CA446A311386@aqute.biz> <1993489049.1580799.1444830069389.JavaMail.zimbra@u-pem.fr> <8C724E11-0656-4483-85BD-DFF310F8B463@aqute.biz> <673841498.1783560.1444850891146.JavaMail.zimbra@u-pem.fr> <38A65E32-EECE-44DF-B646-922CF9637C7D@aqute.biz> <685267555.2219378.1444919188588.JavaMail.zimbra@u-pem.fr> <12677039-8FD6-4C96-814F-7D1DFCD62270@aqute.biz> <854227844.2262184.1444923633889.JavaMail.zimbra@u-pem.fr> Message-ID: I am quite puzzled how you even remotely see this work in a real build. Any concrete build on github for example you can let me look at? Kind regards, Peter Kriens > > On 15 okt. 2015, at 17:40, forax at univ-mlv.fr wrote: > > ----- Mail original ----- >> De: "Peter Kriens" >> ?: forax at univ-mlv.fr >> Cc: "Mark Reinhold" , jpms-spec-experts at openjdk.java.net >> Envoy?: Jeudi 15 Octobre 2015 16:53:26 >> Objet: Re: Why not use the Manifest? >> >>> On 15 okt. 2015, at 16:26, forax at univ-mlv.fr wrote: >>> I assume that some developers will want to declare the module-info by >>> hand. >> Ok, but isn?t this then always redundant? >> >> Isn?t the build system forced to have the same (or at least closely >> corresponding) information with a version? > > If your build tool is able to increment version automatically, > you don't really need versions. > > Let's take an example, we have JUnit3 and JUnit4. > JUnit3 is backward compatible, JUnit4 is backward compatible, > JUnit3 is not compatible with JUnit4, that's obvious because they change the package prefix name. > In a module world, JUnit3 and JUnit4 will not have the same module name, > otherwise, using the most recent module implementation is fine > (if you have a tool that doesn't do any auto-update of the required modules). > > There is one case where you need a version, when you want to blacklist a release because of a bug, > for that you will need to tell the build tool to avoid a peculiar module version. > > Anyway, with this tool, the version is something that tag a module jar, not something the developer as to care of > (developers are bad at managing artifact version). > > R?mi From david.lloyd at redhat.com Tue Oct 20 20:38:31 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 20 Oct 2015 15:38:31 -0500 Subject: Why not use the Manifest? In-Reply-To: <20151005185023.443E87B949@eggemoggin.niobe.net> References: <20151005185023.443E87B949@eggemoggin.niobe.net> Message-ID: <5626A647.8060201@redhat.com> On 10/05/2015 01:50 PM, mark.reinhold at oracle.com wrote: > 2015/9/28 9:49 -0700, peter.kriens at aqute.biz: >> The encoding of the module data in a binary class file remains hard to >> understand and seems to go against the best practices we learned the >> hard way over the past decades. (Debugging!) >> [...] >> This is the first time in a very long time that I see a regression to >> binary files for meta data. Especially because there is already a good >> place that all build tools are already using: the manifest. > > The reasons for expressing module declarations in source files and > compiling them into class files are mentioned in SotMS, but here's a > longer take on just this topic. > > The Java programming language, at present, provides for the definition > of three kinds of program components: classes, interfaces, and packages. > They are defined in Java source files, and compiled into class files. > Such files govern, among other things, the mechanisms of symbolic > resolution and access control implemented by every Java compiler and > JVM. > [...] > Regardless of how they are described, modules are, fundamentally, a new > kind of Java program component. The present proposal therefore treats > them as such, in both the language and the JVM: Module descriptions are > expressed and encoded in the same way as the other kinds of information > that define program components and govern symbolic resolution and access > control -- that is, as Java source and class files. As I've described before (I think?), I believe this is a very thin thread to hang a fundamental design principle from. There are lots of other approaches that developers would just as immediately understand, and MANIFEST.MF is just one of those. > [...] > We could choose another format for module descriptions such as XML, JSON, > YAML, or JAR-file manifests. Such formats may be more convenient for > tool maintainers but we'd have to bake that format into the JLS and the > JVMS, thereby increasing the complexity of those specifications and their > implementations. I don't believe this is true at all, but I'll outline that in detail in another email. > [...] > Source and class files are fundamental to the Java platform, to the JLS > and the JVMS, and to their implementations and supporting tools. Other > formats come and go; these will be with us forever. This, I agree with. > As to debugging, in the prototype we've already enhanced the jar tool > with an option to print the descriptor of a modular JAR file. I expect > similar support to show up in related tools such as jmod and javap, and > eventually in IDEs and other tools outside the JDK. > > As to ease of tooling, most tools will only need to read module > descriptors, not write them, and the java.lang.module.ModuleDescriptor > class already provides convenient static methods for doing that. Adding > similar methods to write descriptors would be straightforward, though it > would complicate the API a bit. > > As to extensibility, class files are already in a precisely-specified and > extensible format. If a build tool needs to add information to a module > descriptor then it can do so via non-standard class-file attributes, > using an existing popular library such as ASM. > > To make it easier for tools to manipulate module descriptors we could > consider enhancing the java.lang.module.ModuleDescriptor class, and > related tools, to read and write custom, non-standard class-file > attributes. This might be a bit tricky depending on how the attributes > are defined, but for simple property-style attributes it's likely > straightforward and would obviate the need to use ASM directly. See next email for more on this. -- - DML From david.lloyd at redhat.com Tue Oct 20 23:27:51 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 20 Oct 2015 18:27:51 -0500 Subject: Module descriptions versus module descriptors Message-ID: <5626CDF7.5060605@redhat.com> I see no logical path that leads from the requirements as specified exclusively to the assumption that the descriptor must be bytecode, let alone part of the JVM and/or language specification. All the reasons given appear to be self-justifying or based on abstract assumptions, e.g. "modules are... a new kind of Java program component... therefore [Jigsaw] treats them as such". I propose instead that this and every other aspect of the JSR implementation be as *decoupled* from the JVM and Java specifications as possible, only coupling them when that is the only way to achieve a requirement. Once you get rid of the assumption that modules and their descriptors ought to be a part of the language as a first principle, it becomes obvious to me that there is really no other specific reason or use case for them to be. Additionally, it seems to me that a significant part of the Jigsaw design justification for its handling of module metadata hinges around the conflation of the description of a module, and the descriptor used by the static module loading implementation. This raises a red flag for me because it fundamentally locks the capabilities of module descriptions to whatever makes sense to express in the descriptor, and then in turn constrains these things to the language and JVM specification. In our (JBoss) module system, these concepts are decoupled: a filesystem module's descriptor is read and parsed into a description which is then consumed by the module system to create the module. This enables us to provide additional capabilities - like class transformation, customized filtering, customized dependency ordering rules, customized resource roots, etc. - which do not make sense for the simple static modules used to bootstrap the system. This also allows us to support multiple module descriptor methodologies or formats, including descriptions derived from Java EE or OSGi metadata (or even Maven metadata), and gives us flexibility to completely customize module wiring within our containers without compromising the contract of the module system in any way because simple descriptors are no longer "special" citizens; they consume the same API as everyone else (meaning users have the exact same capabilities as the system). -- - DML From forax at univ-mlv.fr Wed Oct 21 08:48:24 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 21 Oct 2015 10:48:24 +0200 (CEST) Subject: Module descriptions versus module descriptors In-Reply-To: <5626CDF7.5060605@redhat.com> References: <5626CDF7.5060605@redhat.com> Message-ID: <1466000440.1272131.1445417304851.JavaMail.zimbra@u-pem.fr> I may say something stupid because i'm not sure to have fully understood your mail. For me, a module descriptor is defined by the interface java.lang.module.ModuleDescriptor. I think you can create a ModuleDescriptor from any representation using the builder (ModuleDescriptor.Builder). So at runtime, a module descriptor is just anything that implements that interface. At compile time, it's different because you may want a textual representation for the compiler, at least one by default, the compiler API maybe be pluggable enough to provide your own ModuleDescriptor*, this textual representation is the file module-info.java, because as Mark said, the compiler consumes Java source files. With that, a module descriptor can be hand written by a developer or generated by a build tool, exactly like it's not required to have a java source file to create a java class at runtime. regards, R?mi * This is not as easy as it sound because you need a class of Java 9 when you bootstrap your compiler to create the compiler of Java 9. ----- Mail original ----- > De: "David M. Lloyd" > ?: jpms-spec-experts at openjdk.java.net > Envoy?: Mercredi 21 Octobre 2015 01:27:51 > Objet: Module descriptions versus module descriptors > > I see no logical path that leads from the requirements as specified > exclusively to the assumption that the descriptor must be bytecode, let > alone part of the JVM and/or language specification. All the reasons > given appear to be self-justifying or based on abstract assumptions, > e.g. "modules are... a new kind of Java program component... therefore > [Jigsaw] treats them as such". > > I propose instead that this and every other aspect of the JSR > implementation be as *decoupled* from the JVM and Java specifications as > possible, only coupling them when that is the only way to achieve a > requirement. Once you get rid of the assumption that modules and their > descriptors ought to be a part of the language as a first principle, it > becomes obvious to me that there is really no other specific reason or > use case for them to be. > > Additionally, it seems to me that a significant part of the Jigsaw > design justification for its handling of module metadata hinges around > the conflation of the description of a module, and the descriptor used > by the static module loading implementation. This raises a red flag for > me because it fundamentally locks the capabilities of module > descriptions to whatever makes sense to express in the descriptor, and > then in turn constrains these things to the language and JVM specification. > > In our (JBoss) module system, these concepts are decoupled: a filesystem > module's descriptor is read and parsed into a description which is then > consumed by the module system to create the module. This enables us to > provide additional capabilities - like class transformation, customized > filtering, customized dependency ordering rules, customized resource > roots, etc. - which do not make sense for the simple static modules used > to bootstrap the system. This also allows us to support multiple module > descriptor methodologies or formats, including descriptions derived from > Java EE or OSGi metadata (or even Maven metadata), and gives us > flexibility to completely customize module wiring within our containers > without compromising the contract of the module system in any way > because simple descriptors are no longer "special" citizens; they > consume the same API as everyone else (meaning users have the exact same > capabilities as the system). > > -- > - DML > From david.lloyd at redhat.com Wed Oct 21 10:53:05 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 21 Oct 2015 05:53:05 -0500 Subject: Coupling modules and class loaders Message-ID: <56276E91.5060605@redhat.com> There seems to be no user-facing benefit of modules as a separate entity/allowing multiple modules per class loader - in fact it already actively seems to cause trouble for users, due to package naming conflicts - so I'm thinking it's time we discuss this point specifically and directly. I propose that modules are best defined as a specialization of class loaders. Class loaders already have many of the properties that define a module: encapsulation, isolation, unique identity for packages. In fact OSGi already uses this unit. The only thing they are really missing is the ability to use a graph-like dependency structure, hence the specialization. I believe this results in a superior design overall. Risk is lowered because this approach is well-understood and used by multiple containers and vendors. As far as I can ascertain, there are no requirements which cannot be met with this approach (there has been in the past an allusion to security requirements wherein certain modules are expected to appear to have a 'null' class loader; this is easily accomplished however). What is the justification for the more complex Jigsaw approach? -- - DML From david.lloyd at redhat.com Wed Oct 21 11:21:00 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 21 Oct 2015 06:21:00 -0500 Subject: Modules and build time Message-ID: <5627751C.9020702@redhat.com> I have a problem with the premise that modules should be directly consumable by the build in the same way they are consumed at run time. Building a typical Java artifact today generally entails the Maven approach, which, roughly translated into modularisms, would look something like this: * We build artifacts, not modules * The build consumes (specific versions of) modules for build tasks such as resource processing, descriptor generation, annotation processing, etc. * The build consumes (specific versions of) artifacts for the build class path; these artifacts generally correspond to a minimum version of a module that is expected to be mandatorily or optionally present at run time, but also may correspond only to specific APIs expected to be present at run time * The run time environment may consist of a set of modules that does not have a direct correspondence to the artifacts used for build * The test time environment may consist of a set of modules that does not have a direct correspondence to the artifacts used for build To me there is no sensible way to reconcile these facts against the proposed Jigsaw methodology of having javac reference modules directly. This would mean that all build artifacts have to be packaged as modules and transported to the user for build, which further means that every user would need to define a complete, build-specific modular environment, for little or no gain that I can understand. Can we walk through a few cases where people are expected to build real-world applications, and discuss ways that they could do so? -- - DML From david.lloyd at redhat.com Wed Oct 21 11:38:57 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 21 Oct 2015 06:38:57 -0500 Subject: Security In-Reply-To: <20151005184523.1EDD47B92D@eggemoggin.niobe.net> References: <1088108454.2110963.1441991735882.JavaMail.zimbra@u-pem.fr> <20150917145611.158B371D1B@eggemoggin.niobe.net> <55FADD8D.5080807@redhat.com> <20151005184523.1EDD47B92D@eggemoggin.niobe.net> Message-ID: <56277951.6070609@redhat.com> On 10/05/2015 01:45 PM, mark.reinhold at oracle.com wrote: > 2015/9/17 8:34 -0700, david.lloyd at redhat.com: >> On 09/17/2015 09:56 AM, mark.reinhold at oracle.com wrote: >>> 2015/9/11 10:15 -0700, forax at univ-mlv.fr: >>>> While this change may improve the security, it's a >>>> backward incompatible change is not strictly required to support >>>> modules it's more an enhancement of the current security model and i >>>> don't think it's a good idea to mix it with the introduction of the >>>> module support. >>> >>> This is not an enhancement of the security model per se (which would, >>> anyway, be beyond the scope of this JSR). This change is, rather, in >>> service of the strong-encapsulation goal (which does, of course, help >>> ensure security). >> >> I think this assertion should be substantiated with some examples. >> >> While strong encapsulation is good in terms of design principles, to me >> it's only evident that this somehow leads to better security in the most >> hand-wavy possible way (in particular, as it relates to this specific >> mechanism). > > Three of the five zero-day vulnerabilities reported since JDK 7 GA would > have been prevented if we'd had the ability to strongly encapsulate > JDK-internal packages in the manner proposed, via access-control checks > enforced by the VM. > > Here are public analyses of two of them: > > https://partners.immunityinc.com/idocs/Java%20MBeanInstantiator.findClass%200day%20Analysis.pdf > http://immunityproducts.blogspot.com/2012/08/java-0day-analysis-cve-2012-4681.html > > The lead of Oracle's Java Vulnerability Team estimates that at least a > third of all the vulnerabilities reported since JDK 7 GA would have been > prevented if we'd had the ability to strongly encapsulate JDK-internal > packages. Fair enough, though I do want to point out that examining these two analyses doesn't lead me to believe that this would be the only (or best) way to prevent these exploits, which seem to be rooted in coding errors in other parts of the JDK. >> On the other hand, there is strong evidence to support the >> assertion that enhancing the security model in any way immediately leads >> to new CVEs. The more behavioral rules there are in place, the more >> vectors for exploitation will appear. > > In the abstract, I agree with you. The proposed means of encapsulation, > however, does not depend upon the existing complex security architecture, > so it should be easier to validate. The reason it concerns me is that the Java language access modifiers and this new access check solve the same problem in different ways, leading to a sort of NxM growth of possibilities for access checking behavior. Apart from the theoretical danger of the more complex rules, I don't know that the average developer will understand why the (now) two language-level access checks are separated rather than combined. Adding a "traditional" access level - even the simple approach of "weakening" package-private to module-private (even if just for Java 9+ module classes) - seems much safer to me because it's just another point in a well-understood linear spectrum of behavior, and thus I think can be shown to introduce little or no substantial negative impact on the security of existing code, while at the same time definitely improving the security of any code which deescalates the accessibility of otherwise public members. The simplicity of the model ensures that it will be well-understood by both existing and new developers, and also directly implies a lower risk in at least a couple of categories (security and compatibility spring to mind). I think the principle of "Public is public" is important to maintain. >> ... >> >>> To truly support strong encapsulation would require taking setAccessible >>> away completely. I think that's desirable in the (very) long term, but >>> it would break too much existing code in the near term. >> >> I don't agree that this is true. Encapsulation is really a social >> construct; while accessibility can be used to create and enforce >> encapsulation-justified rules, it doesn't *have* to. You do need some >> kind of back door, no matter what, or else you severely limit the power >> and capability of the platform (as evidenced by the widespread usage of >> the reflection back door by many widely-used frameworks generally >> considered to be powerful and capable). >> >> If encapsulation as a design principle is the goal, then simple >> isolation rules between modules (as between unrelated class loaders) has >> already been proven to be highly effective. > > If all you mean by "encapsulation" is a social construct then sure, none > of this would be necessary. That's insufficient, however, in the face > of actual security threats. Aside from security considerations, a social > contract implies a level of understanding and trust that is difficult, > if not impossible, to achieve in a broad community of millions of > developers. I understand that, but consider: of those millions of developers, I think it is very safe to say that the majority of them know what "private" versus "public" means. >> In this light, I don't see how these changes cannot be considered a >> modification of the security model just because they ride in on the >> coattails of a new abstraction. > > It depends upon what you mean by "security model". > > If you mean some high-level, abstract view of all the various mechanisms > in the language, VM, and libraries that help ensure security then yes, > the proposed changes are an extension of that model. > > When I write "security model", however, I usually mean the subsystem > which evolved from the simple (and simplistic) java.lang.SecurityManager > API in Java 1.0 to the complex policy/permission-based architecture of > the 1.2 release. Many run-time operations (in class loaders, reflection, > I/O, etc.) are subject to checks done by the security subsystem, and if > just one of those checks is missing or incorrect then it may well be > "game over". What I'm thinking of here is the compiler/VM security model, which (in terms of specification at least) I believe is presently limited to access level checking on classes and members. > The proposed enforcement of module boundaries via access restrictions > expressed in the language, recorded in class files, and enforced by the > compiler and the VM do not in any way depend upon that subsystem, nor do > they extend it. The VM will never upcall into the security subsystem in > order to determine whether one type is permitted to access another. The > implementation of access-control checks in a JVM is typically highly > localized and much easier to validate than the comparatively large body > of code in the (mostly) Java-level security subsystem. I do agree that enforcing module boundaries via access restrictions in the language and class files should be enforced by the compiler and VM. I'm just hoping we can arrive at something that doesn't create a new sort of Java puzzler: "when is a public class not public?". >>> ... If encapsulation is to mean >>> anything then it should not be possible to break it solely from within >>> the language itself. Enabling such powers via an external, second-class >>> mechanism such as a command-line option or a debugger is fine, but the >>> history of Java has shown that if it's easy to break encapsulation then >>> people will do so, increasing everyone's maintenance burdens over the >>> long haul. >> >> I think this may be a somewhat narrow view. People break encapsulation >> when they need a capability that cannot be provided another way. Now >> the JDK, unfortunately, historically contains very many "goodies" and >> bits of useful functionality that users want access to, so it is a >> disproportionately popular target for such breakage, but apart from >> accessing JDK internals, I believe that the use cases for breaking in >> through the security model are legitimate and should be allowed to >> adequately privileged code. > > How do you identify "adequately privileged code"? (This is an open issue > we need to address.) Good question... today, my understanding is that such code is (a) any code running in a JVM with no security manager, or (b) any class which has a protection domain which is somehow granted the "suppressAccessChecks" RuntimePermission. But I see your point, and I think it is reasonable to want to strengthen this somehow. Specifically I don't like having public classes that aren't actually public - this adds another layer of complexity which I don't think is necessary. Ideally a public class is *always* public. A module with a public class in a non-exported package is (intuitively speaking) not saying "this class is not really public", it's saying "this class is public, but not to link against". If the goal is to eliminate the need for this back door, then the "front door" must suffice for all use cases. This means that frameworks that access modules must be able to do so in a way that is allowable by module authors. If public classes are always public, even when they're hidden, our various EE specs can be changed to say "your EJB implementation (or whatever) must be public, but it MAY be in a non-exported package". This allows anyone who can get access to the Class object free access to its public members *without* a special permission, which actually goes a long way towards getting rid of the desire for a backdoor in the first place. Such classes could be available by name, service loading, or annotation lookup, but not via direct linkage, which is a nice and logical way of saying "you can access this public class indirectly, but it is definitely NOT API". >> Using an appropriate security model, it is >> not difficult to ensure that such powers are not exploited (though while >> in practice it is more difficult due to a variety of factors, I do not >> believe those factors are directly related to this mechanism but rather >> to exceptions, special "holes", and other technical debt that exists in >> the JDK itself for historical or other reasons). > > I don't think that the future of the platform would be well-served by a > model that works well only for code outside of the JDK, if that's what > you mean. Sure, but the reverse is also true. Clearly a solution needs to accommodate both, though at the same time it is generally not good to bake in rules that only exist to accommodate JDK implementation choices, I'm sure you'd agree. At most, you'd want to leave certain things unspecified to allow the JDK flexibility, but even that might go too far in many cases. -- - DML From forax at univ-mlv.fr Wed Oct 21 16:30:53 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 21 Oct 2015 18:30:53 +0200 (CEST) Subject: A small prototype In-Reply-To: <738260766.1649420.1445444413064.JavaMail.zimbra@u-pem.fr> Message-ID: <1323526889.1652772.1445445053050.JavaMail.zimbra@u-pem.fr> Hi guys, I've written a small prototype that load a module-info.java (not the .class) and use the module resolution provided by jigsaw to verify the dependencies. here is the code: https://github.com/forax/jigsaw-builtit/blob/master/src/buildit/com/githuib/forax/buildit/Main.java It basically parses the module-info.java using a hand coded parser, see the result as a ModuleDescriptor and creates a custom ModuleFinder to see the parent directory of the directory that contains a module-info.java a the root of the modules. This is enough to be able to create a Configuration that verifies a module graph. I will send another mail later this night about what should be changed in order to improve the API. cheers, R?mi From forax at univ-mlv.fr Wed Oct 21 17:11:55 2015 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 21 Oct 2015 19:11:55 +0200 (CEST) Subject: Coupling modules and class loaders In-Reply-To: <56276E91.5060605@redhat.com> References: <56276E91.5060605@redhat.com> Message-ID: <2086237069.1662074.1445447515158.JavaMail.zimbra@u-pem.fr> Let's try to see the big picture, why people uses classloaders, either they want to be able to load/onload an application, have unconventional way to store bytecodes, do bytecode transformations, etc or they want to isolate several parts of the application because those parts use incompatible versions of the same dependency. For the first reasons, this way to use classloaders is very dependent on the application and these applications tend to be very sensitive to the way class loaders are connected. People that design frameworks have come with very creative way of connecting classloaders and trying to force people that want modules to change the way classloaders work in their application will not work. For the last reason, trying to solve problem of incompatible dependencies. I think it's a bad idea to try to fix a linking time issue with a runtime solution. The way to solve that issue at linktime is to link statically the module with one version of the dependency. This is done in Java by changing the bytecodes of the module and its dependency in order to rename the package and change the code that use this package. If we have an intelligent linker, or at least a pluggable one, this can be done at link time so there is no reason to use classloader for that at runtime. I think that classloaders are the wrong mechanism for implementing modules because as jigsaw current prototype prove, you don't need them to provide stronger encapsulation and as it will slow down the adoption of modules because people will have to rewrite there fancy frameworks to be "module compatible". regards, R?mi ----- Mail original ----- > De: "David M. Lloyd" > ?: jpms-spec-experts at openjdk.java.net > Envoy?: Mercredi 21 Octobre 2015 12:53:05 > Objet: Coupling modules and class loaders > > There seems to be no user-facing benefit of modules as a separate > entity/allowing multiple modules per class loader - in fact it already > actively seems to cause trouble for users, due to package naming > conflicts - so I'm thinking it's time we discuss this point specifically > and directly. > > I propose that modules are best defined as a specialization of class > loaders. Class loaders already have many of the properties that define > a module: encapsulation, isolation, unique identity for packages. In > fact OSGi already uses this unit. The only thing they are really > missing is the ability to use a graph-like dependency structure, hence > the specialization. > > I believe this results in a superior design overall. Risk is lowered > because this approach is well-understood and used by multiple containers > and vendors. As far as I can ascertain, there are no requirements which > cannot be met with this approach (there has been in the past an allusion > to security requirements wherein certain modules are expected to appear > to have a 'null' class loader; this is easily accomplished however). > > What is the justification for the more complex Jigsaw approach? > -- > - DML > From david.lloyd at redhat.com Wed Oct 21 18:42:27 2015 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 21 Oct 2015 13:42:27 -0500 Subject: Coupling modules and class loaders In-Reply-To: <2086237069.1662074.1445447515158.JavaMail.zimbra@u-pem.fr> References: <56276E91.5060605@redhat.com> <2086237069.1662074.1445447515158.JavaMail.zimbra@u-pem.fr> Message-ID: <5627DC93.2050202@redhat.com> On 10/21/2015 12:11 PM, Remi Forax wrote: > Let's try to see the big picture, why people uses classloaders, either they want to be able to load/onload an application, have unconventional way to store bytecodes, do bytecode transformations, etc or they want to isolate several parts of the application because those parts use incompatible versions of the same dependency. I don't think that the reasons users use class loaders really has any relevance here; I'm looking at the raw capabilities of class loaders and how they fit in to the requirements we have, and as it happens, they fit very well. > For the first reasons, this way to use classloaders is very dependent on the application and these applications tend to be very sensitive to the way class loaders are connected. People that design frameworks have come with very creative way of connecting classloaders and trying to force people that want modules to change the way classloaders work in their application will not work. Only rare and highly specialized frameworks tend to define their own class loaders. It has been proven repeatedly over the past several years (by us and by others) that most popular existing artifacts and projects already deal with class loaders in a manner which is completely compatible with class loader based modules. So I'm not quite sure what you're driving at - do you have an example of something that would not work with class loader based modules? > For the last reason, trying to solve problem of incompatible dependencies. I think it's a bad idea to try to fix a linking time issue with a runtime solution. The way to solve that issue at linktime is to link statically the module with one version of the dependency. This is done in Java by changing the bytecodes of the module and its dependency in order to rename the package and change the code that use this package. If we have an intelligent linker, or at least a pluggable one, this can be done at link time so there is no reason to use classloader for that at runtime. Are you arguing that there are no cases where you want to run with a different version of a dependency than the one you built against? I would refer you to just about any real world software deployment for an unlimited number of counter-examples. > I think that classloaders are the wrong mechanism for implementing modules because as jigsaw current prototype prove, you don't need them to provide stronger encapsulation and as it will slow down the adoption of modules because people will have to rewrite there fancy frameworks to be "module compatible". Actually I believe your logic is exactly backwards. We (Red Hat) have already been able to prove that very many existing frameworks can use our existing class loader based module solution without *any* modification, *because* they already use class loaders in a reasonable way to find classes and resources. On the other hand, the current Jigsaw prototype inevitably implies that module adoption *will* be slowed down because in order to become module aware, you have to add extra code to recognize Modules and all that goes with them, above and beyond all the existing (and perfectly adequate and functional) code which uses class loaders for that purpose already today. > regards, > R?mi > > ----- Mail original ----- >> De: "David M. Lloyd" >> ?: jpms-spec-experts at openjdk.java.net >> Envoy?: Mercredi 21 Octobre 2015 12:53:05 >> Objet: Coupling modules and class loaders >> >> There seems to be no user-facing benefit of modules as a separate >> entity/allowing multiple modules per class loader - in fact it already >> actively seems to cause trouble for users, due to package naming >> conflicts - so I'm thinking it's time we discuss this point specifically >> and directly. >> >> I propose that modules are best defined as a specialization of class >> loaders. Class loaders already have many of the properties that define >> a module: encapsulation, isolation, unique identity for packages. In >> fact OSGi already uses this unit. The only thing they are really >> missing is the ability to use a graph-like dependency structure, hence >> the specialization. >> >> I believe this results in a superior design overall. Risk is lowered >> because this approach is well-understood and used by multiple containers >> and vendors. As far as I can ascertain, there are no requirements which >> cannot be met with this approach (there has been in the past an allusion >> to security requirements wherein certain modules are expected to appear >> to have a 'null' class loader; this is easily accomplished however). >> >> What is the justification for the more complex Jigsaw approach? >> -- >> - DML >> -- - DML From forax at univ-mlv.fr Thu Oct 22 06:53:26 2015 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Thu, 22 Oct 2015 08:53:26 +0200 (CEST) Subject: Coupling modules and class loaders In-Reply-To: <5627DC93.2050202@redhat.com> References: <56276E91.5060605@redhat.com> <2086237069.1662074.1445447515158.JavaMail.zimbra@u-pem.fr> <5627DC93.2050202@redhat.com> Message-ID: <745552845.1758274.1445496806235.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "David M. Lloyd" > ?: "Remi Forax" > Cc: jpms-spec-experts at openjdk.java.net > Envoy?: Mercredi 21 Octobre 2015 20:42:27 > Objet: Re: Coupling modules and class loaders > > On 10/21/2015 12:11 PM, Remi Forax wrote: > > Let's try to see the big picture, why people uses classloaders, either they > > want to be able to load/onload an application, have unconventional way to > > store bytecodes, do bytecode transformations, etc or they want to isolate > > several parts of the application because those parts use incompatible > > versions of the same dependency. > > I don't think that the reasons users use class loaders really has any > relevance here; I'm looking at the raw capabilities of class loaders and > how they fit in to the requirements we have, and as it happens, they fit > very well. > > > For the first reasons, this way to use classloaders is very dependent on > > the application and these applications tend to be very sensitive to the > > way class loaders are connected. People that design frameworks have come > > with very creative way of connecting classloaders and trying to force > > people that want modules to change the way classloaders work in their > > application will not work. > > Only rare and highly specialized frameworks tend to define their own > class loaders. It has been proven repeatedly over the past several > years (by us and by others) that most popular existing artifacts and > projects already deal with class loaders in a manner which is completely > compatible with class loader based modules. So I'm not quite sure what > you're driving at - do you have an example of something that would not > work with class loader based modules? JBoss Module, OSGI implementations, any applications that uses classloader to implement a kind of module-like isolation. All of these applications will not work if we choose a classloader based module implementation because each of them connect classloders in an incompatible way compared to the others. > > > For the last reason, trying to solve problem of incompatible dependencies. > > I think it's a bad idea to try to fix a linking time issue with a runtime > > solution. The way to solve that issue at linktime is to link statically > > the module with one version of the dependency. This is done in Java by > > changing the bytecodes of the module and its dependency in order to rename > > the package and change the code that use this package. If we have an > > intelligent linker, or at least a pluggable one, this can be done at link > > time so there is no reason to use classloader for that at runtime. > > Are you arguing that there are no cases where you want to run with a > different version of a dependency than the one you built against? I > would refer you to just about any real world software deployment for an > unlimited number of counter-examples. no, there are 3 'times'. The build time, the link time (think jlink but perhaps it should be called the installation time and jlink renamed to jinstall) and the runtime. You can run with a different version of a dependency than at build time but the modules at installation time should be the same as the modules at runtime. So at installation time, you can check if you have several incompatible versions of the same module and transform the code to link them (or some of them) statically. > > > I think that classloaders are the wrong mechanism for implementing modules > > because as jigsaw current prototype prove, you don't need them to provide > > stronger encapsulation and as it will slow down the adoption of modules > > because people will have to rewrite there fancy frameworks to be "module > > compatible". > > Actually I believe your logic is exactly backwards. We (Red Hat) have > already been able to prove that very many existing frameworks can use > our existing class loader based module solution without *any* > modification, *because* they already use class loaders in a reasonable > way to find classes and resources. On the other hand, the current > Jigsaw prototype inevitably implies that module adoption *will* be > slowed down because in order to become module aware, you have to add > extra code to recognize Modules and all that goes with them, above and > beyond all the existing (and perfectly adequate and functional) code > which uses class loaders for that purpose already today. If we forget one moment the issue about setAccessible of jigsaw, for most of the applications, it just works. For applications that already have a notion of modules at runtime, yes, the code need to be touched but as you said there is few frameworks like this so they can be updated by hand. Maybe JBoss Module requires less code/no code changes to support other applications but here we want to support is applications like JBoss Module. regards, R?mi > > > regards, > > R?mi > > > > ----- Mail original ----- > >> De: "David M. Lloyd" > >> ?: jpms-spec-experts at openjdk.java.net > >> Envoy?: Mercredi 21 Octobre 2015 12:53:05 > >> Objet: Coupling modules and class loaders > >> > >> There seems to be no user-facing benefit of modules as a separate > >> entity/allowing multiple modules per class loader - in fact it already > >> actively seems to cause trouble for users, due to package naming > >> conflicts - so I'm thinking it's time we discuss this point specifically > >> and directly. > >> > >> I propose that modules are best defined as a specialization of class > >> loaders. Class loaders already have many of the properties that define > >> a module: encapsulation, isolation, unique identity for packages. In > >> fact OSGi already uses this unit. The only thing they are really > >> missing is the ability to use a graph-like dependency structure, hence > >> the specialization. > >> > >> I believe this results in a superior design overall. Risk is lowered > >> because this approach is well-understood and used by multiple containers > >> and vendors. As far as I can ascertain, there are no requirements which > >> cannot be met with this approach (there has been in the past an allusion > >> to security requirements wherein certain modules are expected to appear > >> to have a 'null' class loader; this is easily accomplished however). > >> > >> What is the justification for the more complex Jigsaw approach? > >> -- > >> - DML > >> > > -- > - DML >