From david.lloyd at redhat.com Wed Jun 1 13:55:54 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 01 Jun 2011 15:55:54 -0500 Subject: Project Jigsaw goals and requirements In-Reply-To: <4DE43780.7050002@info9.net> References: <20110525155551.26F572DC4@eggemoggin.niobe.net> <4DE43780.7050002@info9.net> Message-ID: <4DE6A75A.1070606@redhat.com> On 05/30/2011 07:34 PM, Tom Marble wrote: > On 05/25/2011 10:55 AM, mark.reinhold at oracle.com wrote: >> I've posted the latest draft of the requirements document for wider >> review [4]. Comments are most welcome. > Thanks Mark! > > Many of us, as you point out, are eagerly anticipating the > new Java Module System. > > I have a couple of brief comments and then want to dive > into version syntax: an area in need of serious attention. > [...] > I agree with David Lloyd [18] that the wording and interpretation > "Fidelity across all phases" is very important. I also agree that > "[...] dependency version ranges with a closed upper end cannot be > a fixed part of the module's internal metadata". We really don't > want to have to say "requires public foo @ [2.0,99999);" as a surrogate > for "versions 2.0 or later". > > However I disagree that we want to restrict build dependencies to > specific versions (and not version ranges). This will cause build > maintainers endless toil to manage packages when no underlying > change provokes the rebuild maintenance. I think that fixed build > dependencies are a poor substitute for functional testing once > the project has been built in assuring reproducibility. The two aren't mutually exclusive. A fixed build dependency version is essential to build reproducibility, which does not just mean "make sure it still works with new dependency versions" (which is a legitimate thing to worry about, but it pertains to runtime dependencies and testing, not build dependencies); rather "make sure we get the same result that we did the last time we built this". After the artifact is built, testing it in the modular environment is a different problem. Specifically, if a problem crops up in widget-1.2.5, the widget maintainers have to know to a certainty what classes are being used. If build-time version ranges are employed then it's possible that the maintainer of widget could end up testing different classes, and ultimately be unable to reproduce the problem. A versioned module has to be defined as the result of building a specific set of source code against a specific fixed dependency set, else the notion of a version is useless because it doesn't identify an artifact, just a sort of category of possible artifacts with no real way to know what you really have. Put another way, if you build against a different dependency set, you have changed your artifact as surely as if you have changed the source files. It would be very unwise to ignore this as a requirement. Yes, it is possible to make a dependency change which does not affect the final artifact; however the same can be said for source code changes. Now as for runtime "functional" testing, as new dependency versions are released it is important that existing artifacts are tested against the new versions of the dependencies to ensure that they continue to function in the runtime module environment. This is definitely an interesting problem domain in its own right, however it is at best only casually related to build dependencies. -- - DML From tmarble at info9.net Thu Jun 2 07:50:48 2011 From: tmarble at info9.net (Tom Marble) Date: Thu, 02 Jun 2011 09:50:48 -0500 Subject: Project Jigsaw goals and requirements In-Reply-To: <4DE6A75A.1070606@redhat.com> References: <20110525155551.26F572DC4@eggemoggin.niobe.net> <4DE43780.7050002@info9.net> <4DE6A75A.1070606@redhat.com> Message-ID: <4DE7A348.5000805@info9.net> On 06/01/2011 03:55 PM, David M. Lloyd wrote: > The two aren't mutually exclusive. [..] testing it in the modular environment is a different problem. I understand. > [...] If build-time version ranges are employed then it's possible that the maintainer of > widget could end up testing different classes, and ultimately be unable to reproduce the problem. The idea of the range is, IMHO, that the underlying API semantics are reliable. Are you really proposing updating the 'build-depends' graph to *exact* versions for building (possibly with relaxing the versions to become ranges at install and runtime)? I see your rationale, but it (exact build dep versions) does sound like a lot of work to maintain. Are you using that approach now for the Java related dependency graph of *.rpm's in Red Hat? Separately you are probably the expert on RPM version syntax... Can you point us to an authoritative grammar? (A fair answer, of course, would be to use-the-source of rpm). Regards, --Tom From thiebault at artenum.com Thu Jun 2 08:07:09 2011 From: thiebault at artenum.com (=?ISO-8859-1?Q?Beno=EEt_Thi=E9bault?=) Date: Thu, 02 Jun 2011 17:07:09 +0200 Subject: Project Jigsaw goals and requirements In-Reply-To: <4DE43780.7050002@info9.net> References: <20110525155551.26F572DC4@eggemoggin.niobe.net> <4DE43780.7050002@info9.net> Message-ID: <1307027230.2155.10.camel@Mokette> Hi everyone, This discussion about Jigsaw requirements is very interesting and constructive and I am really looking forward to see what Java modules will be capable of. Just to add my own set of remarks, I would like to point that support for native librairies is very important and should not be neglected. In OSGi, it is already feasible, but it is poorly documented and has some very strong constraints (as explained here: http://dev.artenum.com/blog/ben/posts/osgi_vtk_and_macosx). One of the most annoying things is to have to explicitly call System.loadLibrary() for every shared library and its dependencies to tell OSGi framework what .so (.dll or .jnilib) files to extract from the jar bundle. Another requirement that I have not seen up to now is the ability to have a distributed application based on modules. On the OSGi side, there is DOSGi (http://cxf.apache.org/) that is supposed to fulfil this requirement but I have not tested it yet outside prototypes and I have no idea how mature it is. Kind regards, Ben -- Dr Beno?t Thi?bault Project Manager Artenum Toulouse - Science & Groupware http://www.artenum.com From Alan.Bateman at oracle.com Thu Jun 2 08:27:21 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 02 Jun 2011 16:27:21 +0100 Subject: Project Jigsaw goals and requirements In-Reply-To: <1307027230.2155.10.camel@Mokette> References: <20110525155551.26F572DC4@eggemoggin.niobe.net> <4DE43780.7050002@info9.net> <1307027230.2155.10.camel@Mokette> Message-ID: <4DE7ABD9.4040600@oracle.com> Beno?t Thi?bault wrote: > : > > Just to add my own set of remarks, I would like to point that support > for native librairies is very important and should not be neglected. > There is an item on this in the requirements. There is also a page here with a proposed approach: http://openjdk.java.net/projects/jigsaw/doc/topics/nativecode.html -Alan. From richard.s.hall at oracle.com Thu Jun 2 08:30:42 2011 From: richard.s.hall at oracle.com (Richard S. Hall) Date: Thu, 02 Jun 2011 11:30:42 -0400 Subject: Project Jigsaw goals and requirements In-Reply-To: <1307027230.2155.10.camel@Mokette> References: <20110525155551.26F572DC4@eggemoggin.niobe.net> <4DE43780.7050002@info9.net> <1307027230.2155.10.camel@Mokette> Message-ID: <4DE7ACA2.3000708@oracle.com> On 6/2/11 11:07, Beno?t Thi?bault wrote: > Hi everyone, > > This discussion about Jigsaw requirements is very interesting and > constructive and I am really looking forward to see what Java modules > will be capable of. > > Just to add my own set of remarks, I would like to point that support > for native librairies is very important and should not be neglected. > > In OSGi, it is already feasible, but it is poorly documented and has > some very strong constraints (as explained here: > http://dev.artenum.com/blog/ben/posts/osgi_vtk_and_macosx). One of the > most annoying things is to have to explicitly call System.loadLibrary() > for every shared library and its dependencies to tell OSGi framework > what .so (.dll or .jnilib) files to extract from the jar bundle. To be precise, this is not really an OSGi issue, but a more general one about native library dependencies...Java can only intercept native library load requests coming from Java code. Arbitrary native libraries that have dependencies on other native libraries cannot be intercepted by Java and automatically loaded by the JVM. The procedure you describe above is simply a workaround. > Another requirement that I have not seen up to now is the ability to > have a distributed application based on modules. On the OSGi side, there > is DOSGi (http://cxf.apache.org/) that is supposed to fulfil this > requirement but I have not tested it yet outside prototypes and I have > no idea how mature it is. Distribution is largely an orthogonal issue. -> richard From david.lloyd at redhat.com Thu Jun 2 08:39:23 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 02 Jun 2011 10:39:23 -0500 Subject: Project Jigsaw goals and requirements In-Reply-To: <4DE7A348.5000805@info9.net> References: <20110525155551.26F572DC4@eggemoggin.niobe.net> <4DE43780.7050002@info9.net> <4DE6A75A.1070606@redhat.com> <4DE7A348.5000805@info9.net> Message-ID: <4DE7AEAB.1030104@redhat.com> On 06/02/2011 09:50 AM, Tom Marble wrote: > On 06/01/2011 03:55 PM, David M. Lloyd wrote: >> The two aren't mutually exclusive. [..] testing it in the modular environment is a different problem. > I understand. > >> [...] If build-time version ranges are employed then it's possible that the maintainer of >> widget could end up testing different classes, and ultimately be unable to reproduce the problem. > The idea of the range is, IMHO, that the underlying API semantics are > reliable. Yes, that is true. The problem though isn't with the API semantics, it's with the mechanics of class generation, which could cause surprise linkage errors. The problem is further compounded when you consider things like Maven's "shade" plugin, wherein dependency classes are copied into the new artifact (generally under a different package), which could in turn affect API semantics after all. This is a thing that should be supported by a robust build system. This problem even shows up for non-Java things. I remember my days running Gentoo Linux... if you update packages in the wrong order or build things when the wrong packages were present, you could introduce creeping time-bombs into your system. And no Gentoo system is exactly like another, which means that troubleshooting becomes quite a lot more difficult. It's all about reducing the number of variables. The fewer variables, the more reliable the system. > Are you really proposing updating the 'build-depends' graph > to *exact* versions for building (possibly with relaxing the versions to > become ranges at install and runtime)? Yes, that is exactly what I am proposing, though I'd drop the word "possibly" as it would be an inevitable requirement to be able to update a package without a massive cascading dependency graph update. As long as the build versions were current at one time within a version environment though, it is not too hard to maintain since the artifacts are always available at build time. Maven in fact does this very thing, though this (like many other things) it approaches from the wrong end. Artifacts (in a sane build) have a fixed-version dependency graph, which generally come in transitively most of the time, and it is up to the consumer of the dependency to override the version it uses, particularly important at test-time (which is as close a thing as Maven has to a run time). > I see your rationale, but it (exact build dep versions) does sound > like a lot of work to maintain. Are you using that approach now > for the Java related dependency graph of *.rpm's in Red Hat? Yes and no... sometimes... it depends on whether you're talking about Fedora or RHEL, and even then it's sometimes a gray area depending on a few other factors. However fixing this is among my goals for the year, and the fix will probably involve JBoss Modules (our module system) at some level. > Separately you are probably the expert on RPM version syntax... > Can you point us to an authoritative grammar? (A fair answer, > of course, would be to use-the-source of rpm). I'm afraid I am not, but asking around yielded this information: http://fedoraproject.org/wiki/Packaging/NamingGuidelines#Package_Versioning Note that this is the guidelines for Fedora, not for RPM itself. And also note that JPackage packages follow different rules, maddeningly: http://fedoraproject.org/wiki/Packaging/JPackagePolicy -- - DML From thiebault at artenum.com Thu Jun 2 08:40:39 2011 From: thiebault at artenum.com (thiebault at artenum.com) Date: Thu, 2 Jun 2011 17:40:39 +0200 Subject: Project Jigsaw goals and requirements Message-ID: <51768.1307029239@artenum.com> Le jeu 2/06/11 17:30, "Richard S. Hall" richard.s.hall at oracle.com a ?crit: > To be precise, this is not really an OSGi issue, but a more general one > about native library dependencies...Java can only intercept native > library load requests coming from Java code. Arbitrary native libraries > that have dependencies on other native libraries cannot be intercepted > by Java and automatically loaded by the JVM. The procedure you describe > above is simply a workaround. When I use the same native library (i.e. VTK) in a pure Java approach, I only need to call System.loadLibrary() on the top level libraries, not all their dependencies (http://dev.artenum.com/projects/cassandra/forum/how-to-use-eclipse- and-vtk), this is why I thought it was somehow OSGi related, but I guess this is a little off-topic :-). What I wanted to point out is that the use of native libraries in Java modules and the wrapping of such native libs in a module should be as simple as possible. Kind regards, Ben From richard.s.hall at oracle.com Thu Jun 2 08:55:38 2011 From: richard.s.hall at oracle.com (Richard S. Hall) Date: Thu, 02 Jun 2011 11:55:38 -0400 Subject: Project Jigsaw goals and requirements In-Reply-To: <51768.1307029239@artenum.com> References: <51768.1307029239@artenum.com> Message-ID: <4DE7B27A.2010705@oracle.com> On 6/2/11 11:40, thiebault at artenum.com wrote: > Le jeu 2/06/11 17:30, "Richard S. Hall" richard.s.hall at oracle.com a ?crit: >> To be precise, this is not really an OSGi issue, but a more general one >> > about native library dependencies...Java can only intercept native >> library load requests coming from Java code. Arbitrary native libraries >> > that have dependencies on other native libraries cannot be intercepted > by Java and automatically loaded by the JVM. The procedure you describe > above is simply a workaround. > > When I use the same native library (i.e. VTK) in a pure Java approach, I only need to call System.loadLibrary() on the top level libraries, not all their dependencies (http://dev.artenum.com/projects/cassandra/forum/how-to-use-eclipse- > and-vtk), this is why I thought it was somehow OSGi related, but I guess this is a little off-topic :-). > > What I wanted to point out is that the use of native libraries in Java modules and the wrapping of such native libs in a module should be as simple as possible. Agreed. -> richard From gnormington at vmware.com Fri Jun 3 03:19:40 2011 From: gnormington at vmware.com (Glyn Normington) Date: Fri, 3 Jun 2011 11:19:40 +0100 Subject: Project Jigsaw goals and requirements Message-ID: Hi Mark I'm pleased to see the explicit acknowledgement of some basic OSGi interoperation requirements in the requirements document ([1]). I agree with David Bosschaert ([2]), that it would make sense for OSGi to support the Java SE 8 module format and, for modules which can serve equally well as OSGi bundles, I'd like to avoid dual-maintenance of module metadata and OSGi manifest. I'd like to be able to "decorate" the standard metadata. However, the requirement "The syntax must place all extended metadata after all standard metadata, with a clear delineation between them." precludes inline decorations. The result would be duplication and clunkiness. I propose that this requirement be changed so that standard metadata could be decorated inline (the decorations would be ignored by the Java SE module system). What do you think? Regards, Glyn [1] http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12 [2] http://osgithoughts.blogspot.com/2011/05/java-se-8-modularity-requirements.html From chris.hegarty at oracle.com Tue Jun 7 11:06:21 2011 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 07 Jun 2011 19:06:21 +0100 Subject: LZMA compression for jpkg Message-ID: <4DEE689D.2040505@oracle.com> Hi, I have been looking into the possibility of using LZMA compression ( following on from Dalibors work last year ) for the jmod packages. Initial results look positive. I've been using a Java implementation from the latest LZMA 9.20 SDK [1]. I modified jpkg to use LZMA compression for classes, resources, and nativelibs. Here are two of the largest packages: > : du -sk jmod/jdk.boot at 7-ea.jmod jmod/sun.desktop at 7-ea.jmod 12170 jmod/jdk.boot at 7-ea.jmod 8584 jmod/sun.desktop at 7-ea.jmod > : du -sk jmod-lzma/jdk.boot at 7-ea.jmod jmod-lzma/sun.desktop at 7-ea.jmod 8456 jmod-lzma/jdk.boot at 7-ea.jmod 6791 jmod-lzma/sun.desktop at 7-ea.jmod ~30% reduction on jdk.boot and ~20% reduction for sun.desktop. Here is a total for all the JDK modules: > : du -sk jmod jmod-lzma 31171 jmod 23936 jmod-lzma ~23% reduction in size . Q: The results look good, but can we do better? The debian packages that get built during the modules build are still smaller than these lzma-ized jmod files. The main difference seems to be that the deb packages are generated with best compression ( -z9 ). I'm not sure if the Java implementation in the SDK supports this level of compression. It may be that we have to switch to a native implementation distributed with the SDK instead. Q: Is there a License issue with the LZMA SDK code? All the code in the latest LZMA 9.20 SDK is now completely in the public domain so I don't foresee any license issues. The LZMA SDK Java implementation is used in the Grizzly HTTP Web Framework, Apache Ant, and at least two other Java LZMA binding projects. Native tools like 7zip and lzma (on Linux) all appear to be using the native LZMA SDK implementation. Q: What next? I would like to spend some more time trying the native LZMA implementation. I think we can get close to the debian package size with it. I also need to spend some more time analyzing the decompression cost. I believe LZMA would be a good candidate for a public API in OpenJDK, say java.util.[zip|jar].LZMA. I've done a rough prototype, very similar to Pack200 ( static factory methods for creating compressor/decompressor ). I envisage that the OpenJDK LZMA implementation would simply leverage the LZMA SDK implementation. -Chris. [1] http://www.7-zip.org/sdk.html From mandy.chung at oracle.com Tue Jun 7 16:39:26 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Tue, 07 Jun 2011 16:39:26 -0700 Subject: Review request: exports runtime support + jdk modules converted to use exports Message-ID: <4DEEB6AE.9030003@oracle.com> I completed the changes for all jdk modules to use the new "exports" statement. The module-info.java for all jdk modules is at: http://cr.openjdk.java.net/~mchung/jigsaw/moduleinfo-exports/jdk-modules-module-info.txt Full webrev: http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/exports-full-webrev/ Summary of changes: 1. jigsaw runtime support for the new ModuleExports attribute (see [1]) This includes reading the new ModuleExports attribute from module-info.class, updating the context to build a map from a remote class to its suppying context (previously the key was a remote package name), and also the simple library now stores the list of remote classes rather the remote packages. Also, javac's implementation of ModuleInfo is updated to implement the new exports() method. The webrev in [1] has a temporary workaround that assume all public types of the platform modules are exported. In this patch, that workaround is removed. The runtime will read the ModuleExports attribute for all modules including the platform ones. 2. Modify the ClassAnalyzer to generate exports statement. By default, a module exports the supported APIs and it explicitly lists individual package using '*' wildcard. Internal APIs are only exported in the modules that are strictly for implementation use with the base module (see below). Platform modules should only export the platform APIs and any other public supported APIs. '*' wildcard is preferred in this case rather than '**' to export all public types in all subpackages to prevent any internal subpackages from leaking out. For example, jdk.desktop exports individual packages e.g. java.awt.* and java.awt.dnd.* rather that java.awt.** but java.awt.dnd.peer and java.awt.peer are internal packages that are exported in sun.desktop (internal module). It also allows to specify additional exports statement in the input config file. When a module reexports another module, it will export the APIs that are listed in the "exported.packages" property defined in the input modules.properties file. For modules that aggregate multiple modules (e.g. jdk, jdk.jre), it will coalesce all exported packages into a simpler export statement using "**" wildcard to reexport all "exported" types in all subpackages. modules.properties file includes a list of exported packages for the platform that is based on the CORE_PKGS.gmk and NON_CORE_PKG.gmk. http://cr.openjdk.java.net/~mchung/jigsaw/moduleinfo-exports/jdk-modules-module-info.txt shows all module-info.java of the platform modules. jdk.base exports a few types in the sun.reflect.annotation package since javac depends on them for supporting serialization of annotation. These 4 sun.reflect.annotation classes are essentially defacto platform classes. There is some refactoring/clean up in the class analyzer that was something I did some time ago and I'd like to take this chance to get reviewed. 3. This webrev also includes the fix I posted in [2] that the Linker should check if a supplying context and the requesting context are different before adding to the remote suppliers list. 4. The change in java.util.ResourceBundle and sun.security.jca.ProviderConfig are related to the use of system class loader to find classes in ClassPath. In the modular JDK, system class loader is the module class loader for the root module but the system classes/resources are not necessarily visible to it (e.g. sun.security.provider.Sun and other internal classes). The Launcher.getBootLoader method is added as a temporary workaround. We'll examine the use of class loader in the jdk implementation closer in the future. 5. The change in ModuleId.java fixes a memory issue in loading a configuration. Some test failed due to OutOfMemoryError when there are large number of ModuleId stored in a configuration (one in each local and remote class) but there are limited number of unique ones. Instead of returning a new instance of ModuleId, ModuleId.parse() keeps a cache of all unique ModuleId. 6. jmod dump-config is updated to print out the exports statement. $ jmod list -v jdk.javac jdk.javac at 7-ea requires jdk.base@=7-ea requires jdk.compiler@=7-ea requires jdk.logging@=7-ea exports com.sun.source.tree.* exports com.sun.source.util.* exports com.sun.tools.apt.resources.* exports com.sun.tools.javac.* exports com.sun.tools.javac.api.* exports com.sun.tools.javac.code.* exports com.sun.tools.javac.comp.* exports com.sun.tools.javac.file.* exports com.sun.tools.javac.jigsaw.* exports com.sun.tools.javac.jvm.* exports com.sun.tools.javac.main.* exports com.sun.tools.javac.model.* exports com.sun.tools.javac.nio.* exports com.sun.tools.javac.parser.* exports com.sun.tools.javac.processing.* exports com.sun.tools.javac.resources.* exports com.sun.tools.javac.sym.* exports com.sun.tools.javac.tree.* exports com.sun.tools.javac.util.* 7. jmod find-class I added this diagnostic comand to find which contexts a given class can be found to help diagnosing ClassNotFoundException. A class exists but if not visible to a module class loader, CNFE will be thrown. For example, this would be useful to diagnose the CNFE thrown why it can't find the sun.security.provider.Sun security provider. Below is the sample output. $ jmod find-class jdk.base sun.security.provider.Sun configuration roots = [jdk.base at 7-ea] class sun.security.provider.Sun found in: context +jdk.base remote +jdk.boot+sun.charsets+sun.jaxp+sun.localedata+sun.resources context +jdk.jaxp remote +jdk.boot+sun.charsets+sun.jaxp+sun.localedata+sun.resources context +jdk.boot+sun.charsets+sun.jaxp+sun.localedata+sun.resources module jdk.boot at 7-ea The above means that sun.security.provider.Sun is visible to the module loader for the above contexts. It's defined locally from jdk.boot at 7-ea module. $ jmod find-class jdk.base sun.security.pkcs.SunPKCS11 configuration roots = [jdk.base at 7-ea] class sun.security.pkcs.SunPKCS11 found in: none Mandy [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2011-May/001292.html [2] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2011-May/001308.html From tmarble at info9.net Tue Jun 7 17:18:22 2011 From: tmarble at info9.net (Tom Marble) Date: Tue, 07 Jun 2011 19:18:22 -0500 Subject: LZMA compression for jpkg In-Reply-To: <4DEE689D.2040505@oracle.com> References: <4DEE689D.2040505@oracle.com> Message-ID: <4DEEBFCE.5020106@info9.net> On 06/07/2011 01:06 PM, Chris Hegarty wrote: > I have been looking into the possibility of using LZMA compression ( following on from Dalibors work last year ) for the jmod packages. Initial results look positive. It's true that we have an opportunity to do something new here! Note that the Java Module-System Requirements ? DRAFT 12 [0] does list this as a goal (with pack200 for jars of course). While where considering compression techniques we might as well jump to xz (an improvement over lzma) [1]. > I've been using a Java implementation from the latest LZMA 9.20 SDK [1]. I modified jpkg to use LZMA compression [...] > The debian packages that get built during the modules build are still smaller than these lzma-ized jmod files. With our current Jigsaw project in Debian we are looking at how to create policy compliant packages [2]. I highly suspect we will need to rethink the current approach of having jpkg doing the work and, instead, collaborating with native debhelper tools [3]. Point being that we should probably avoid the current *.deb packaging as much of a baseline. In this context it is important to note that classically Debian source packages used to accept only upstream tarballs in gzip but are evolving to include bzip2 and xz in later releases. [4] Regards, --Tom [0] http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#java-native-module-file-format [1] http://tukaani.org/xz/ [2] http://wiki.debian.org/SummerOfCode2011/Jigsaw [3] http://joey.kitenet.net/code/debhelper/ [4] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=556407 From chris.hegarty at oracle.com Wed Jun 8 06:56:06 2011 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 08 Jun 2011 14:56:06 +0100 Subject: LZMA compression for jpkg In-Reply-To: <4DEEBFCE.5020106@info9.net> References: <4DEE689D.2040505@oracle.com> <4DEEBFCE.5020106@info9.net> Message-ID: <4DEF7F76.7050206@oracle.com> On 06/ 8/11 01:18 AM, Tom Marble wrote: > On 06/07/2011 01:06 PM, Chris Hegarty wrote: >> I have been looking into the possibility of using LZMA compression ( following on from Dalibors work last year ) for the jmod packages. Initial results look positive. > It's true that we have an opportunity to do something new here! > > Note that the Java Module-System Requirements ? DRAFT 12 [0] > does list this as a goal (with pack200 for jars of course). > > While where considering compression techniques we might as > well jump to xz (an improvement over lzma) [1]. Yes a very good point. I am planning to evaluate the xz (LZMA2) implementation that is part of the LZMA SDK. It may not help much with compression over LZMA, but certainly has other advantages. >> I've been using a Java implementation from the latest LZMA 9.20 SDK [1]. I modified jpkg to use LZMA compression [...] >> The debian packages that get built during the modules build are still smaller than these lzma-ized jmod files. > > With our current Jigsaw project in Debian we are looking at how > to create policy compliant packages [2]. I highly suspect we > will need to rethink the current approach of having jpkg doing > the work and, instead, collaborating with native debhelper tools [3]. > Point being that we should probably avoid the current *.deb > packaging as much of a baseline. Currently jpkg calls native dpkg to create the debian packages, but it may be that debhelper, or a combination of debhelper and jpkg, is a better approach. I need to spend more time looking at the debhelper tools. Thanks for pointing this out. -Chris. > In this context it is important to note that classically > Debian source packages used to accept only upstream tarballs > in gzip but are evolving to include bzip2 and xz in > later releases. [4] > > Regards, > > --Tom > > [0] http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#java-native-module-file-format > [1] http://tukaani.org/xz/ > [2] http://wiki.debian.org/SummerOfCode2011/Jigsaw > [3] http://joey.kitenet.net/code/debhelper/ > [4] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=556407 From mandy.chung at oracle.com Wed Jun 8 14:37:19 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 08 Jun 2011 14:37:19 -0700 Subject: Optional Module Dependency In-Reply-To: <4DD6D511.7020409@oracle.com> References: <4DD598F2.4010307@oracle.com> <4DD6D511.7020409@oracle.com> Message-ID: <4DEFEB8F.9050301@oracle.com> Hi Roger, On 05/20/11 13:54, Roger Riggs wrote: > The terminology of "optional method" might be improved. > The method itself is not optional; but its behavior is based on the > presence > of classes within a module that may not be present. The behavior depends > on code to check the (class, module or configuration) dependencies and > (in many cases) throw an exception. The @RequireModule annotation only > seems to be informative. Right. @RequireModule annotation is for developer tools such as javadoc to list all optional methods or highlight them in the javadoc if appropriate. The runtime behavior depends on the implementation to check if such dependency can be satisfied. > > The Module.isModulePresent(String mn) method might be renamed to > isModuleAvailable > since it includes the check for availability from the calling module. > I see both the term "present" and "available" don't precisely indicate it's relative to the calling module. In that case, isModulePresent seems preferable. > Does the module name being checked include the version ? I.e. > "jdk.jaxp at 1.4" > Some dependencies will be relative to specific versions of the modules. > I don't see it needs to include the version. If the optional dependency is specific to a particular version, isModulePresent("jdk.jaxp") returns true. You can use Module.getVersion() to check the version or Class.forName to check if a new API exists or not - see below for the Class.forName cost. > There is a potential disconnect between testing for a module name and the > specific dependency on a class. > Developers may not be clear on the relationship between the Class names > that are used in the code and the module packaging. In the modular world, for committed and supported APIs, module and class names relationship should be clear. For uncommitted and non-supported APIs, the vendor can move or remove them in incompatible way. Seems like your concern is for the latter case in which it should use Class.forName in my opinion. If an API is moved from one module "A" to another "B" and not reexported to A, the module that depends on such API would have to be updated to require B rather than A; otherwise, it can't find that class. > As an example, in > the webrev of java.util.Properties, the code that has the dependency is > not in the same class as the test for the module. There may be a > mismatch > between the module name and the understanding of its contents. The platform classes / modules are a special case. We would investigate this when we support aliases. The platform modules are strongly connected (i.e. requires local) and currently loaded by the BootLoader for 2 reasons; 1. compatibility - all system classes are loaded by the null bootstrap class loader 2. split package requirement java.util.Properties is in the jdk.boot module and in the context with jdk.boot+sun.jaxp (+ other platform modules). jdk.boot module shouldn't depend on any modules and it couldn't find jdk.jaxp. I think we should treat the jdk.boot module a special case that it should also find the contexts from jdk.base. Is there other use case you think of? > > In many cases, Class.forName is the right function because it is > specific. > But for classes that are not present and not going to be present, it > is too slow to be repeated often and the method be fail fast since the > contents of the module are/can be known. Class.forName in the modular world doesn't do the linear search on the class path (no more classpath!!). The module class loader will be able to do such check in a couple of lookups (once we implement the fast configuration). > Alternatively, would it be possible to have a (ClassLoader) method to > test if > a class was available but without the side effect of loading it? > I assume you want the class to be loaded if present. So Class.forName can continue to be used. > The embedding of module names in the source code may too rigid and > lead to maintenance issues since changing the dependency will require > changing the code and generating an updated module with a new version > number. > This may be less of problem if the module namespace is very carefully > managed and is generic enough to allow substitution of equivalent > implementation modules. > It depends on how much flexibility can be accomodated in module name > and contents. > Could the argument to isModulePresent be an indirect reference through > the module-info contents? That might help decouple the source code > from the > module relationships. > How? > With respect the @RequireModule annotation on methods, I don't see that > it provides much value. It documents the method as having a dependency > but it will be hard to go beyond that. The method may or may not > directly > refer to the classes in the module. Without analyzing the control flow > of the code it would be difficult for a tool to help the developer > identify > the dependencies that could not be met (at runtime) if the module was > not present. > (All modules have to be present at compile time.) The value I see is for example when an application calls Properties.loadFromXML, an IDE can catch this and ask if the application should require jdk.jaxp and add it to its dependency so that this optional dependency can guarantee to be satisfied at runtime. The developer of the optional methods will be responsible to make sure a ModuleNotPresentException is thrown if the requiring module is not present. > > When refering to types that are not (going to be) present, care must > be taken > to ensure the classfile can be verified without needing to evalate the > missing > types. This can be a bit subtle. In my experience, the coding patterns > that are easy to understand limit where references to optional types > may be > used. Good point. I'm checking with Alex Buckley further on the verification side. We probably have to come up an idiom that prevents a class referencing the optional types being verified unless the optional module is present. > If there is an annotation, it might be most useful if it is placed in the > source at a point that reinforces the way verification works but I > haven't > thought this through. (For now, annotations can only appear in some > places.) > Thanks for the comments. I'll send out an updated webrev some time. Mandy > Thanks for the chance to comment, > Roger > > >> I have implemented the new @RequireAnnotation annotation and a new >> API (Module.isModulePresent method) to test whether a module has been >> resolved and linked. >> >> This note summarizes for the optional module dependency work: >> http://openjdk.java.net/projects/jigsaw/doc/topics/optional.html >> >> Webrev: >> >> http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/require-module-annotation/ >> >> A few questions to discuss: >> 1. There are some APIs that require another module depending on the >> input parameters passed in. Bidi, JNDI, JMX are some examples. The >> optional module is guaranteed to be present when the given input >> parameter depends on it. > The requirement is on the type of the input parameters (not the values). >> >> I wonder what the best practice or recommendation of using >> @RequireAnnotation. For APIs that throws ModuleNotPresentException, >> they clearly need to be annotated so that javadoc and other tools can >> process them when appropriate. >> >> I think it's generally a good idea to annotate the source of an >> optional dependency so that a tool can check if the optional >> dependency is declared in the module-info.java (if there is no static >> reference to types from the optional module; otherwise, the compiler >> can detect that). If a module P didn't declare the optional >> dependency on M and M is present, P will be compiled successfully. >> At runtime, isModulePresent method will return false even if M is >> present but it is not visible to P. P's module class loader will >> fail to find a class in module M. This is the case the new test >> "optional-method.sh" shows. >> >> I currently put this annotation in these APIs (e.g. >> sun.text.bidi.BidiBase, com.sun.jmx.mbeanserver.Introspector, etc). >> Alan mentioned in an offline discussion I had with him if it's more >> appropriate to have a different annotation for this type of >> dependency. Any other thought? > Figure out the semantics of the annotation and then revisit the name. >> >> 2. Class.forName is the existing (legacy) approach to determine if a >> class is present. In a modular world, a modular application can use >> isModulePresent method instead. This leads to another question that >> module P should statically reference types from an optional module M; >> replace the use of Class.forName if it was to eliminate static >> dependency (this is the approach JDK uses). > Directly using references to classes may cause verification failures > because in order to verify class P; > the Class being referenced must be resolved. Check with those most > knowledgable about the verifier. >> >> Mandy > From mandy.chung at oracle.com Wed Jun 8 23:55:20 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 08 Jun 2011 23:55:20 -0700 Subject: Review request: exports runtime support + jdk modules converted to use exports In-Reply-To: <4DEEB6AE.9030003@oracle.com> References: <4DEEB6AE.9030003@oracle.com> Message-ID: <4DF06E58.803@oracle.com> I should also mention that the stored configuration ought be optimized for better performance in the future. The stored configuration was changed to keep a list of remote classes rather than packages so that a module can export some public types of a package rather than the whole package. This approach increases the number of entries in a context's map to look up a supplying context for a given class quite significantly and thus the size of the stored configuration. For example, the number of remote entries of jdk.base module is increased from 224 to 4495 entries. This also impacts all modules requiring jdk.base. With remote classes: context +jdk.base module jdk.base at 7-ea remote (4495) Before (remote packages): context +jdk.base module jdk.base at 7-ea remote (224) The size of jdk.base/7-ea/config is also increased from ~450K to 1.3M. Mandy On 6/7/11 4:39 PM, Mandy Chung wrote: > > > I completed the changes for all jdk modules to use the new "exports" > statement. > The module-info.java for all jdk modules is at: > > http://cr.openjdk.java.net/~mchung/jigsaw/moduleinfo-exports/jdk-modules-module-info.txt > > Full webrev: > http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/exports-full-webrev/ > > Summary of changes: > 1. jigsaw runtime support for the new ModuleExports attribute (see [1]) > This includes reading the new ModuleExports attribute from > module-info.class, > updating the context to build a map from a remote class to its > suppying context > (previously the key was a remote package name), and also the > simple library > now stores the list of remote classes rather the remote packages. > Also, > javac's implementation of ModuleInfo is updated to implement the new > exports() method. > > The webrev in [1] has a temporary workaround that assume all > public types > of the platform modules are exported. In this patch, that > workaround is > removed. The runtime will read the ModuleExports attribute for all > modules including the platform ones. > > 2. Modify the ClassAnalyzer to generate exports statement. > By default, a module exports the supported APIs and it explicitly > lists > individual package using '*' wildcard. Internal APIs are only > exported > in the modules that are strictly for implementation use with the > base module > (see below). Platform modules should only export the platform > APIs and > any other public supported APIs. '*' wildcard is preferred in this > case > rather than '**' to export all public types in all subpackages to > prevent > any internal subpackages from leaking out. For example, jdk.desktop > exports individual packages e.g. java.awt.* and java.awt.dnd.* rather > that java.awt.** but java.awt.dnd.peer and java.awt.peer are > internal packages > that are exported in sun.desktop (internal module). > > It also allows to specify > additional exports statement in the input config file. When a module > reexports another module, it will export the APIs that are listed in > the "exported.packages" property defined in the input > modules.properties > file. For modules that aggregate multiple modules (e.g. jdk, > jdk.jre), > it will coalesce all exported packages into a simpler export > statement > using "**" wildcard to reexport all "exported" types in all > subpackages. > > modules.properties file includes a list of exported packages for > the platform > that is based on the CORE_PKGS.gmk and NON_CORE_PKG.gmk. > > > http://cr.openjdk.java.net/~mchung/jigsaw/moduleinfo-exports/jdk-modules-module-info.txt > shows all module-info.java of the platform modules. > > jdk.base exports a few types in the sun.reflect.annotation package > since javac depends on them for supporting serialization of > annotation. > These 4 sun.reflect.annotation classes are essentially defacto > platform > classes. > > There is some refactoring/clean up in the class analyzer that was > something > I did some time ago and I'd like to take this chance to get > reviewed. > > 3. This webrev also includes the fix I posted in [2] that the Linker > should check > if a supplying context and the requesting context are different > before adding > to the remote suppliers list. > > 4. The change in java.util.ResourceBundle and > sun.security.jca.ProviderConfig > are related to the use of system class loader to find classes in > ClassPath. > In the modular JDK, system class loader is the module class > loader for > the root module but the system classes/resources are not necessarily > visible to it (e.g. sun.security.provider.Sun and other internal > classes). > The Launcher.getBootLoader method is added as a temporary > workaround. > We'll examine the use of class loader in the jdk implementation > closer in the future. > > 5. The change in ModuleId.java fixes a memory issue in loading a > configuration. > Some test failed due to OutOfMemoryError when there are large number > of ModuleId stored in a configuration (one in each local and > remote class) > but there are limited number of unique ones. Instead of > returning a new > instance of ModuleId, ModuleId.parse() keeps a cache of all > unique ModuleId. > > 6. jmod dump-config is updated to print out the exports statement. > > $ jmod list -v jdk.javac > > jdk.javac at 7-ea > requires jdk.base@=7-ea > requires jdk.compiler@=7-ea > requires jdk.logging@=7-ea > exports com.sun.source.tree.* > exports com.sun.source.util.* > exports com.sun.tools.apt.resources.* > exports com.sun.tools.javac.* > exports com.sun.tools.javac.api.* > exports com.sun.tools.javac.code.* > exports com.sun.tools.javac.comp.* > exports com.sun.tools.javac.file.* > exports com.sun.tools.javac.jigsaw.* > exports com.sun.tools.javac.jvm.* > exports com.sun.tools.javac.main.* > exports com.sun.tools.javac.model.* > exports com.sun.tools.javac.nio.* > exports com.sun.tools.javac.parser.* > exports com.sun.tools.javac.processing.* > exports com.sun.tools.javac.resources.* > exports com.sun.tools.javac.sym.* > exports com.sun.tools.javac.tree.* > exports com.sun.tools.javac.util.* > > 7. jmod find-class > > I added this diagnostic comand to find which contexts a given class > can be found > to help diagnosing ClassNotFoundException. A class exists but if not > visible to a > module class loader, CNFE will be thrown. For example, this would be > useful > to diagnose the CNFE thrown why it can't find the > sun.security.provider.Sun > security provider. Below is the sample output. > > $ jmod find-class jdk.base sun.security.provider.Sun > configuration roots = [jdk.base at 7-ea] > class sun.security.provider.Sun found in: > context +jdk.base > remote +jdk.boot+sun.charsets+sun.jaxp+sun.localedata+sun.resources > context +jdk.jaxp > remote +jdk.boot+sun.charsets+sun.jaxp+sun.localedata+sun.resources > context +jdk.boot+sun.charsets+sun.jaxp+sun.localedata+sun.resources > module jdk.boot at 7-ea > > The above means that sun.security.provider.Sun is visible to the > module loader > for the above contexts. It's defined locally from jdk.boot at 7-ea module. > > $ jmod find-class jdk.base sun.security.pkcs.SunPKCS11 > configuration roots = [jdk.base at 7-ea] > class sun.security.pkcs.SunPKCS11 found in: > none > > Mandy > > [1] > http://mail.openjdk.java.net/pipermail/jigsaw-dev/2011-May/001292.html > [2] > http://mail.openjdk.java.net/pipermail/jigsaw-dev/2011-May/001308.html > From Alan.Bateman at oracle.com Thu Jun 9 05:48:05 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 09 Jun 2011 13:48:05 +0100 Subject: Review request: exports runtime support + jdk modules converted to use exports In-Reply-To: <4DF06E58.803@oracle.com> References: <4DEEB6AE.9030003@oracle.com> <4DF06E58.803@oracle.com> Message-ID: <4DF0C105.2020708@oracle.com> Mandy Chung wrote: > > > I should also mention that the stored configuration ought be optimized > for better performance in the future. The stored configuration was > changed to keep a list of remote classes rather than packages so that > a module can export some public types of a package rather than the > whole package. This approach increases the number of entries in a > context's map to look up a supplying context for a given class quite > significantly and thus the size of the stored configuration. > > For example, the number of remote entries of jdk.base module is increased > from 224 to 4495 entries. This also impacts all modules requiring > jdk.base. > > With remote classes: > context +jdk.base > module jdk.base at 7-ea > remote (4495) > > Before (remote packages): > context +jdk.base > module jdk.base at 7-ea > remote (224) > > The size of jdk.base/7-ea/config is also increased from ~450K to 1.3M. I've been looking at the webrev and was wondering about this. I grabbed the patch file from the webrev and did some tests and it looks like the stored configuration for simple tests jumps very significantly, like from about 6.5MB to 90MB. The reason is that java -m spends several seconds loading it. Are you seeing this too? -Alan From Alan.Bateman at oracle.com Thu Jun 9 05:54:05 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 09 Jun 2011 13:54:05 +0100 Subject: Optional Module Dependency In-Reply-To: <4DEFEB8F.9050301@oracle.com> References: <4DD598F2.4010307@oracle.com> <4DD6D511.7020409@oracle.com> <4DEFEB8F.9050301@oracle.com> Message-ID: <4DF0C26D.80307@oracle.com> Mandy Chung wrote: > : > >> >> The Module.isModulePresent(String mn) method might be renamed to >> isModuleAvailable >> since it includes the check for availability from the calling module. >> > > I see both the term "present" and "available" don't precisely indicate > it's > relative to the calling module. In that case, isModulePresent seems > preferable. I agree, and it's somewhat consistent with existing methods such as isAnnotationPresent and exceptions such as TypeNotPresentException. > : > >> As an example, in >> the webrev of java.util.Properties, the code that has the dependency is >> not in the same class as the test for the module. There may be a >> mismatch >> between the module name and the understanding of its contents. > > The platform classes / modules are a special case. We would > investigate this > when we support aliases. The platform modules are strongly connected > (i.e. > requires local) and currently loaded by the BootLoader for 2 reasons; > 1. compatibility - all system classes are loaded by the null bootstrap > class loader > 2. split package requirement > > java.util.Properties is in the jdk.boot module and in the context with > jdk.boot+sun.jaxp > (+ other platform modules). jdk.boot module shouldn't depend on any > modules and it couldn't find jdk.jaxp. I think we should treat the > jdk.boot module > a special case that it should also find the contexts from jdk.base. > > Is there other use case you think of? To Roger's point, I think anyone creating an optional dependency will need to take care. I agree the examples for the platform look confusing and we should probably addressed by providing documentation or examples to show how one might do this in an application or library. >> >> When refering to types that are not (going to be) present, care must >> be taken >> to ensure the classfile can be verified without needing to evalate >> the missing >> types. This can be a bit subtle. In my experience, the coding patterns >> that are easy to understand limit where references to optional types >> may be >> used. > > Good point. I'm checking with Alex Buckley further on the > verification side. > We probably have to come up an idiom that prevents a class referencing > the optional types being verified unless the optional module is present. If there is a static reference to a type then I would assume it is no longer an optional dependency (or maybe I'm not understanding the approach, and static references are actually allowed?). -Alan. From Roger.Riggs at Oracle.com Thu Jun 9 07:59:50 2011 From: Roger.Riggs at Oracle.com (Roger Riggs) Date: Thu, 09 Jun 2011 10:59:50 -0400 Subject: Optional Module Dependency In-Reply-To: <4DEFEB8F.9050301@oracle.com> References: <4DD598F2.4010307@oracle.com> <4DD6D511.7020409@oracle.com> <4DEFEB8F.9050301@oracle.com> Message-ID: <4DF0DFE6.804@Oracle.com> Hi Mandy, On 6/8/11 5:37 PM, Mandy Chung wrote: > Hi Roger, > > On 05/20/11 13:54, Roger Riggs wrote: >> The terminology of "optional method" might be improved. >> The method itself is not optional; but its behavior is based on the >> presence >> of classes within a module that may not be present. The behavior >> depends >> on code to check the (class, module or configuration) dependencies and >> (in many cases) throw an exception. The @RequireModule annotation only >> seems to be informative. > > Right. > > @RequireModule annotation is for developer tools such as javadoc to list > all optional methods or highlight them in the javadoc if appropriate. > The runtime behavior depends on the implementation to check if such > dependency can be satisfied. The semantics need to be strong and useful (or the name of the annotation should be changed to reflect a purely informative function). As is the annotation may even be inconsistent with the declarations in the module-info file. The annotation as presently defined is optional; the behavior of the compiler and runtime is the same whether it is included or not. I don't expect it will be used except where documentation is important. > >> >> The Module.isModulePresent(String mn) method might be renamed to >> isModuleAvailable >> since it includes the check for availability from the calling module. >> > > I see both the term "present" and "available" don't precisely indicate > it's > relative to the calling module. In that case, isModulePresent seems > preferable. > >> Does the module name being checked include the version ? I.e. >> "jdk.jaxp at 1.4" >> Some dependencies will be relative to specific versions of the modules. >> > > I don't see it needs to include the version. If the optional > dependency is > specific to a particular version, isModulePresent("jdk.jaxp") returns > true. > You can use Module.getVersion() to check the version or Class.forName > to check if a new API exists or not - see below for the Class.forName > cost. So the developer will need to code two separate calls in a compound expression to achieve the needed result. It is inefficient and does not adequately expressive the requirement in the code. > >> There is a potential disconnect between testing for a module name and >> the >> specific dependency on a class. >> Developers may not be clear on the relationship between the Class names >> that are used in the code and the module packaging. > > In the modular world, for committed and supported APIs, module and class > names relationship should be clear. Clear from what? Will each module have a separate specification that exhaustively lists the included classes? Will compatibility tests verify that a module contains the public/protected classes that are required? Signature tests need to be at the module level; not the whole platform. > For uncommitted and non-supported APIs, > the vendor can move or remove them in incompatible way. Seems like > your concern is for the latter case in which it should use Class.forName > in my opinion. If an API is moved from one module "A" to another "B" > and > not reexported to A, the module that depends on such API would have to be > updated to require B rather than A; otherwise, it can't find that class. The tools available to vendors and library creators should be as powerful and expressive as those for the system. > >> As an example, in >> the webrev of java.util.Properties, the code that has the dependency is >> not in the same class as the test for the module. There may be a >> mismatch >> between the module name and the understanding of its contents. > > The platform classes / modules are a special case. We would > investigate this > when we support aliases. Where is the alias concept described? > The platform modules are strongly connected (i.e. > requires local) and currently loaded by the BootLoader for 2 reasons; > 1. compatibility - all system classes are loaded by the null bootstrap > class loader > 2. split package requirement > > java.util.Properties is in the jdk.boot module and in the context with > jdk.boot+sun.jaxp > (+ other platform modules). jdk.boot module shouldn't depend on any > modules and it couldn't find jdk.jaxp. I think we should treat the > jdk.boot module > a special case that it should also find the contexts from jdk.base. > > Is there other use case you think of? A notification system (such as Growl) will consist of a library that can operate in many different environments and will want to dynamically take advantage of the available notification mechanisms. In some cases, it will be only non-ui logging, in other cases, there may be a Swing UI present or a JavaFX UI present. Non-system libraries need to able to make the same intelligent choices about which modules are present and which classes are available. > >> >> In many cases, Class.forName is the right function because it is >> specific. >> But for classes that are not present and not going to be present, it >> is too slow to be repeated often and the method be fail fast since the >> contents of the module are/can be known. > > Class.forName in the modular world doesn't do the linear search on > the class path (no more classpath!!). The module class loader will > be able to do such check in a couple of lookups (once we implement > the fast configuration). > >> Alternatively, would it be possible to have a (ClassLoader) method to >> test if >> a class was available but without the side effect of loading it? >> > > I assume you want the class to be loaded if present. So Class.forName > can continue to be used. I was explicit that the test needed to be lightweight and not have the side effect of loading the class. Class.forName does not meet the requirement. The developer may want to test the configuration before selecting which mechanism to use. It will also help startup/first use performance to defer loading of the class until it is used. > >> The embedding of module names in the source code may too rigid and >> lead to maintenance issues since changing the dependency will require >> changing the code and generating an updated module with a new version >> number. >> This may be less of problem if the module namespace is very carefully >> managed and is generic enough to allow substitution of equivalent >> implementation modules. >> It depends on how much flexibility can be accomodated in module name >> and contents. >> Could the argument to isModulePresent be an indirect reference through >> the module-info contents? That might help decouple the source code >> from the >> module relationships. >> > > How? I'll think about it. If the dependencies in the module-info.java file were tagged, that could provide the hook. > >> With respect the @RequireModule annotation on methods, I don't see that >> it provides much value. It documents the method as having a dependency >> but it will be hard to go beyond that. The method may or may not >> directly >> refer to the classes in the module. Without analyzing the control flow >> of the code it would be difficult for a tool to help the developer >> identify >> the dependencies that could not be met (at runtime) if the module was >> not present. >> (All modules have to be present at compile time.) > > The value I see is for example when an application calls > Properties.loadFromXML, > an IDE can catch this and ask if the application should require > jdk.jaxp and add it > to its dependency so that this optional dependency can guarantee to be > satisfied at runtime. I'm not sure that's the best example, Properties has the dependencies on XML, not the application, the requirement for jdk.jaxp should be in the java.util module-info.java file. Unless this is reflecting to the application level that Properties has optional behavior and the application is responsible for resolving whether it is required or not. > > The developer of the optional methods will be responsible to make sure > a ModuleNotPresentException is thrown if the requir*(ed)* module is not > present. I'm looking for compiler support for developer who have to write "optional" methods. Should ModuleNotPresentException should be documented for the method? Strictly speaking it does not need a throws clause because it is a runtime exception but it would useful and consistent documentation. > >> >> When refering to types that are not (going to be) present, care must >> be taken >> to ensure the classfile can be verified without needing to evalate >> the missing >> types. This can be a bit subtle. In my experience, the coding patterns >> that are easy to understand limit where references to optional types >> may be >> used. > > Good point. I'm checking with Alex Buckley further on the > verification side. > We probably have to come up an idiom that prevents a class referencing > the optional types being verified unless the optional module is present. Thanks, this would be most useful. > >> If there is an annotation, it might be most useful if it is placed in >> the >> source at a point that reinforces the way verification works but I >> haven't >> thought this through. (For now, annotations can only appear in some >> places.) >> > > Thanks for the comments. I'll send out an updated webrev some time. > > Mandy Thanks, Roger > >> Thanks for the chance to comment, >> Roger >> >> >>> I have implemented the new @RequireAnnotation annotation and a new >>> API (Module.isModulePresent method) to test whether a module has >>> been resolved and linked. >>> >>> This note summarizes for the optional module dependency work: >>> http://openjdk.java.net/projects/jigsaw/doc/topics/optional.html >>> >>> Webrev: >>> >>> http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/require-module-annotation/ >>> >>> A few questions to discuss: >>> 1. There are some APIs that require another module depending on the >>> input parameters passed in. Bidi, JNDI, JMX are some examples. The >>> optional module is guaranteed to be present when the given input >>> parameter depends on it. >> The requirement is on the type of the input parameters (not the values). >>> >>> I wonder what the best practice or recommendation of using >>> @RequireAnnotation. For APIs that throws ModuleNotPresentException, >>> they clearly need to be annotated so that javadoc and other tools >>> can process them when appropriate. >>> >>> I think it's generally a good idea to annotate the source of an >>> optional dependency so that a tool can check if the optional >>> dependency is declared in the module-info.java (if there is no >>> static reference to types from the optional module; otherwise, the >>> compiler can detect that). If a module P didn't declare the >>> optional dependency on M and M is present, P will be compiled >>> successfully. At runtime, isModulePresent method will return false >>> even if M is present but it is not visible to P. P's module class >>> loader will fail to find a class in module M. This is the case the >>> new test "optional-method.sh" shows. >>> >>> I currently put this annotation in these APIs (e.g. >>> sun.text.bidi.BidiBase, com.sun.jmx.mbeanserver.Introspector, etc). >>> Alan mentioned in an offline discussion I had with him if it's more >>> appropriate to have a different annotation for this type of >>> dependency. Any other thought? >> Figure out the semantics of the annotation and then revisit the name. >>> >>> 2. Class.forName is the existing (legacy) approach to determine if a >>> class is present. In a modular world, a modular application can use >>> isModulePresent method instead. This leads to another question >>> that module P should statically reference types from an optional >>> module M; replace the use of Class.forName if it was to eliminate >>> static dependency (this is the approach JDK uses). >> Directly using references to classes may cause verification failures >> because in order to verify class P; >> the Class being referenced must be resolved. Check with those most >> knowledgable about the verifier. >>> >>> Mandy >> > From mandy.chung at oracle.com Thu Jun 9 11:32:45 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Thu, 09 Jun 2011 11:32:45 -0700 Subject: Review request: exports runtime support + jdk modules converted to use exports In-Reply-To: <4DF0C105.2020708@oracle.com> References: <4DEEB6AE.9030003@oracle.com> <4DF06E58.803@oracle.com> <4DF0C105.2020708@oracle.com> Message-ID: <4DF111CD.8020506@oracle.com> On 6/9/11 5:48 AM, Alan Bateman wrote: > I've been looking at the webrev and was wondering about this. I > grabbed the patch file from the webrev and did some tests and it looks > like the stored configuration for simple tests jumps very > significantly, like from about 6.5MB to 90MB. The reason is that java > -m spends several seconds loading it. Are you seeing this too? I can reproduce this. My test cases require jdk.base and thus I didn't see this. The current default is "requires jdk" if no jdk module is specified in the module-info that causes all jdk modules (contexts and their local/remote maps) be included in the configuration. I am looking into an optimization to this. Mandy From mandy.chung at oracle.com Fri Jun 10 00:27:51 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Fri, 10 Jun 2011 00:27:51 -0700 Subject: Review request: exports runtime support + jdk modules converted to use exports In-Reply-To: <4DF111CD.8020506@oracle.com> References: <4DEEB6AE.9030003@oracle.com> <4DF06E58.803@oracle.com> <4DF0C105.2020708@oracle.com> <4DF111CD.8020506@oracle.com> Message-ID: <4DF1C777.9090206@oracle.com> On 6/9/11 11:32 AM, Mandy Chung wrote: > On 6/9/11 5:48 AM, Alan Bateman wrote: >> I've been looking at the webrev and was wondering about this. I >> grabbed the patch file from the webrev and did some tests and it >> looks like the stored configuration for simple tests jumps very >> significantly, like from about 6.5MB to 90MB. The reason is that java >> -m spends several seconds loading it. Are you seeing this too? > > I can reproduce this. My test cases require jdk.base and thus I > didn't see this. The current default is "requires jdk" if no jdk > module is specified in the module-info that causes all jdk modules > (contexts and their local/remote maps) be included in the > configuration. I am looking into an optimization to this. > An update on the analysis: I instrumented the simple library. This single context's name itself takes up 80% of the config file size (~75M). +jdk.boot+sun.charsets+sun.compat+sun.corba+sun.desktop+sun.ext+sun.instrument+sun.jaas+sun.jaxp+sun.jdbc+sun.jndi+sun.jsse+sun.jta+sun.kerberos+sun.localedata+sun.logging+sun.management+sun.resources+sun.rmi+sun.security.acl+sun.sunec count per-string bytes total bytes 327099 235 76868265 Each context contains a map from a remote class name to a context's name. When the map's key is changed from package name to a class name, the number of context's name is thus significantly increased. A simple fix would be using an integer as the context ID and storing the name only once. Top 15 strings with the highest total of bytes in the config file: count bytes total contexts name 327099 235 76868265 41 +jdk.boot+sun.charsets+sun.compat+sun.corba+sun.desktop+sun.ext+sun.instrument+sun.jaas+sun.jaxp+sun.jdbc+sun.jndi+sun.jsse+sun.jta+sun.kerberos+sun.localedata+sun.logging+sun.management+sun.resources+sun.rmi+sun.security.acl+sun.sunec 5550 16 88800 1 sun.desktop at 7-ea 4121 10 41210 11 +jdk.javac 3722 10 37220 6 +sun.jaxws 3467 13 45071 1 jdk.boot at 7-ea 3325 14 46550 1 jdk.tools at 7-ea 2502 13 32526 1 sun.jaxp at 7-ea 2405 14 33670 1 sun.jaxws at 7-ea 1677 14 23478 1 sun.corba at 7-ea 1441 20 28820 1 jdk.tools.jaxws at 7-ea 801 13 10413 9 +jdk.compiler 768 14 10752 1 jdk.javac at 7-ea 558 19 10602 1 sun.management at 7-ea Total number of strings in the config file is 729694 but 26185 unique strings. Total bytes occupied by these strings is 91839030. The strings created in the configuration includes context's name module's name at version module library path class name There is other opportunity for optimization. For example, for each class from jdk.boot, there are 40 copies in the config as one in each remote map of each context and there are 40 contexts requiring it. Mandy From Alan.Bateman at oracle.com Fri Jun 10 06:34:23 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 10 Jun 2011 14:34:23 +0100 Subject: Review request: exports runtime support + jdk modules converted to use exports In-Reply-To: <4DF1C777.9090206@oracle.com> References: <4DEEB6AE.9030003@oracle.com> <4DF06E58.803@oracle.com> <4DF0C105.2020708@oracle.com> <4DF111CD.8020506@oracle.com> <4DF1C777.9090206@oracle.com> Message-ID: <4DF21D5F.6090902@oracle.com> Mandy Chung wrote: > An update on the analysis: > > I instrumented the simple library. This single context's name itself > takes up 80% of the config file size (~75M). > > +jdk.boot+sun.charsets+sun.compat+sun.corba+sun.desktop+sun.ext+sun.instrument+sun.jaas+sun.jaxp+sun.jdbc+sun.jndi+sun.jsse+sun.jta+sun.kerberos+sun.localedata+sun.logging+sun.management+sun.resources+sun.rmi+sun.security.acl+sun.sunec > > > count per-string bytes total bytes > 327099 235 76868265 > > Each context contains a map from a remote class name to a context's name. > When the map's key is changed from package name to a class name, the > number of context's name is thus significantly increased. A simple > fix would be using an integer as the context ID and storing the name > only once. > > Top 15 strings with the highest total of bytes in the config file: > count bytes total contexts name > 327099 235 76868265 41 > +jdk.boot+sun.charsets+sun.compat+sun.corba+sun.desktop+sun.ext+sun.instrument+sun.jaas+sun.jaxp+sun.jdbc+sun.jndi+sun.jsse+sun.jta+sun.kerberos+sun.localedata+sun.logging+sun.management+sun.resources+sun.rmi+sun.security.acl+sun.sunec > > 5550 16 88800 1 sun.desktop at 7-ea > 4121 10 41210 11 +jdk.javac > 3722 10 37220 6 +sun.jaxws > 3467 13 45071 1 jdk.boot at 7-ea > 3325 14 46550 1 jdk.tools at 7-ea > 2502 13 32526 1 sun.jaxp at 7-ea > 2405 14 33670 1 sun.jaxws at 7-ea > 1677 14 23478 1 sun.corba at 7-ea > 1441 20 28820 1 jdk.tools.jaxws at 7-ea > 801 13 10413 9 +jdk.compiler > 768 14 10752 1 jdk.javac at 7-ea > 558 19 10602 1 sun.management at 7-ea > > Total number of strings in the config file is 729694 but 26185 > unique strings. Total bytes occupied by these strings is 91839030. > > The strings created in the configuration includes > context's name > module's name at version > module library path > class name > > There is other opportunity for optimization. For example, for each class > from jdk.boot, there are 40 copies in the config as one in each remote > map > of each context and there are 40 contexts requiring it. > > Mandy > For the current problem then adding indirection for the context name should do it. If you are anxious to get the exports work completed then this could be separated from other optimization work. -Alan From tmarble at info9.net Fri Jun 10 09:18:34 2011 From: tmarble at info9.net (Tom Marble) Date: Fri, 10 Jun 2011 11:18:34 -0500 Subject: International Solver Competition 2011 Message-ID: <4DF243DA.3070707@info9.net> All: I thought this group would be interested to learn of the International Solver Competition 2011 [0]. Of specific interest is the approach in comparing resolvers [1]. We really have this problem in comparing resolvers between the native Jigsaw algorithm and the resolvers of all the potential downstreams, such as Debian, Red Hat, OSGi, Maven, etc. We need sort of a meta test to see if the full graph of Jigsaw modules can be installed in a set of major resolvers successfully... Over time... continuously *a la* Jenkins. Sounds like a nice afternoon hacking project! :) --Tom [0] http://www.mancoosi.org/misc-2011/ [1] http://www.mancoosi.org/misc-2011/rules/ From Roger.Riggs at Oracle.com Fri Jun 10 10:48:38 2011 From: Roger.Riggs at Oracle.com (Roger Riggs) Date: Fri, 10 Jun 2011 13:48:38 -0400 Subject: Review request: exports runtime support + jdk modules converted to use exports In-Reply-To: <4DF1C777.9090206@oracle.com> References: <4DEEB6AE.9030003@oracle.com> <4DF06E58.803@oracle.com> <4DF0C105.2020708@oracle.com> <4DF111CD.8020506@oracle.com> <4DF1C777.9090206@oracle.com> Message-ID: <4DF258F6.905@Oracle.com> Hi Mandy, The size of the meta-data seems significant. What is the percentage overhead added by the configuration tables? Does the entire configuration occupy the heap? From a small system perspective; the total memory numbers seem out of scale. If it is just an optimization then it can wait. But if the current API implies a huge data structure is needed; then the design should be re-examined. Roger On 6/10/11 3:27 AM, Mandy Chung wrote: > On 6/9/11 11:32 AM, Mandy Chung wrote: >> On 6/9/11 5:48 AM, Alan Bateman wrote: >>> I've been looking at the webrev and was wondering about this. I >>> grabbed the patch file from the webrev and did some tests and it >>> looks like the stored configuration for simple tests jumps very >>> significantly, like from about 6.5MB to 90MB. The reason is that >>> java -m spends several seconds loading it. Are you seeing this too? >> >> I can reproduce this. My test cases require jdk.base and thus I >> didn't see this. The current default is "requires jdk" if no jdk >> module is specified in the module-info that causes all jdk modules >> (contexts and their local/remote maps) be included in the >> configuration. I am looking into an optimization to this. >> > An update on the analysis: > > I instrumented the simple library. This single context's name itself > takes up 80% of the config file size (~75M). > > +jdk.boot+sun.charsets+sun.compat+sun.corba+sun.desktop+sun.ext+sun.instrument+sun.jaas+sun.jaxp+sun.jdbc+sun.jndi+sun.jsse+sun.jta+sun.kerberos+sun.localedata+sun.logging+sun.management+sun.resources+sun.rmi+sun.security.acl+sun.sunec > > > count per-string bytes total bytes > 327099 235 76868265 > > Each context contains a map from a remote class name to a context's name. > When the map's key is changed from package name to a class name, the > number of context's name is thus significantly increased. A simple > fix would be using an integer as the context ID and storing the name > only once. > > Top 15 strings with the highest total of bytes in the config file: > count bytes total contexts name > 327099 235 76868265 41 > +jdk.boot+sun.charsets+sun.compat+sun.corba+sun.desktop+sun.ext+sun.instrument+sun.jaas+sun.jaxp+sun.jdbc+sun.jndi+sun.jsse+sun.jta+sun.kerberos+sun.localedata+sun.logging+sun.management+sun.resources+sun.rmi+sun.security.acl+sun.sunec > 5550 16 88800 1 sun.desktop at 7-ea > 4121 10 41210 11 +jdk.javac > 3722 10 37220 6 +sun.jaxws > 3467 13 45071 1 jdk.boot at 7-ea > 3325 14 46550 1 jdk.tools at 7-ea > 2502 13 32526 1 sun.jaxp at 7-ea > 2405 14 33670 1 sun.jaxws at 7-ea > 1677 14 23478 1 sun.corba at 7-ea > 1441 20 28820 1 jdk.tools.jaxws at 7-ea > 801 13 10413 9 +jdk.compiler > 768 14 10752 1 jdk.javac at 7-ea > 558 19 10602 1 sun.management at 7-ea > > Total number of strings in the config file is 729694 but 26185 > unique strings. Total bytes occupied by these strings is 91839030. > > The strings created in the configuration includes > context's name > module's name at version > module library path > class name > > There is other opportunity for optimization. For example, for each class > from jdk.boot, there are 40 copies in the config as one in each remote > map > of each context and there are 40 contexts requiring it. > > Mandy > From mandy.chung at oracle.com Fri Jun 10 11:20:18 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Fri, 10 Jun 2011 11:20:18 -0700 Subject: Review request: exports runtime support + jdk modules converted to use exports In-Reply-To: <4DF258F6.905@Oracle.com> References: <4DEEB6AE.9030003@oracle.com> <4DF06E58.803@oracle.com> <4DF0C105.2020708@oracle.com> <4DF111CD.8020506@oracle.com> <4DF1C777.9090206@oracle.com> <4DF258F6.905@Oracle.com> Message-ID: <4DF26062.60406@oracle.com> On 6/10/11 10:48 AM, Roger Riggs wrote: > Hi Mandy, > > The size of the meta-data seems significant. What is the percentage > overhead > added by the configuration tables? Does the entire configuration > occupy the heap? > From a small system perspective; the total memory numbers seem out of > scale. > > If it is just an optimization then it can wait. But if the current > API implies > a huge data structure is needed; then the design should be re-examined. > The current implementation is a SimpleLibrary. As it's named, it's a simple prototype for proof-of-concept. There will certainly be performance improvement works e.g. Mike Duigou has started looking at a btree implementation for fast configuration [1]. The size of the config file is just the on-disk storage. I have included a simple fix for the memory issue in the webrev (see [2]) Mandy [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2011-May/001296.html [2] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2011-June/001333.html From david.lloyd at redhat.com Fri Jun 10 11:23:18 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Fri, 10 Jun 2011 13:23:18 -0500 Subject: Review request: exports runtime support + jdk modules converted to use exports In-Reply-To: <4DF258F6.905@Oracle.com> References: <4DEEB6AE.9030003@oracle.com> <4DF06E58.803@oracle.com> <4DF0C105.2020708@oracle.com> <4DF111CD.8020506@oracle.com> <4DF1C777.9090206@oracle.com> <4DF258F6.905@Oracle.com> Message-ID: <4DF26116.4070507@redhat.com> Keeping (additional) maps of every class seems ludicrous to me. In JBoss Modules, we do everything by package, with additional filters which can be (optionally) applied for the non-common case where you want to only partially export a package. It is far more memory-efficient, and empirically seems to be very fast. Note that it is important not to over-promise the contract. If you tell people that they can and should routinely pick and choose classes within a package to export or not export, you're encouraging a usage pattern that may not be easy to implement efficiently. By encouraging users to export whole packages only, but give the option of partial package export (at a possible performance cost) you are encouraging much more performant behaviors without reducing your total functionality. On 06/10/2011 12:48 PM, Roger Riggs wrote: > Hi Mandy, > > The size of the meta-data seems significant. What is the percentage > overhead > added by the configuration tables? Does the entire configuration occupy > the heap? > From a small system perspective; the total memory numbers seem out of > scale. > > If it is just an optimization then it can wait. But if the current API > implies > a huge data structure is needed; then the design should be re-examined. > > Roger > > > On 6/10/11 3:27 AM, Mandy Chung wrote: >> On 6/9/11 11:32 AM, Mandy Chung wrote: >>> On 6/9/11 5:48 AM, Alan Bateman wrote: >>>> I've been looking at the webrev and was wondering about this. I >>>> grabbed the patch file from the webrev and did some tests and it >>>> looks like the stored configuration for simple tests jumps very >>>> significantly, like from about 6.5MB to 90MB. The reason is that >>>> java -m spends several seconds loading it. Are you seeing this too? >>> >>> I can reproduce this. My test cases require jdk.base and thus I >>> didn't see this. The current default is "requires jdk" if no jdk >>> module is specified in the module-info that causes all jdk modules >>> (contexts and their local/remote maps) be included in the >>> configuration. I am looking into an optimization to this. >>> >> An update on the analysis: >> >> I instrumented the simple library. This single context's name itself >> takes up 80% of the config file size (~75M). >> >> +jdk.boot+sun.charsets+sun.compat+sun.corba+sun.desktop+sun.ext+sun.instrument+sun.jaas+sun.jaxp+sun.jdbc+sun.jndi+sun.jsse+sun.jta+sun.kerberos+sun.localedata+sun.logging+sun.management+sun.resources+sun.rmi+sun.security.acl+sun.sunec >> >> >> count per-string bytes total bytes >> 327099 235 76868265 >> >> Each context contains a map from a remote class name to a context's name. >> When the map's key is changed from package name to a class name, the >> number of context's name is thus significantly increased. A simple >> fix would be using an integer as the context ID and storing the name >> only once. >> >> Top 15 strings with the highest total of bytes in the config file: >> count bytes total contexts name >> 327099 235 76868265 41 >> +jdk.boot+sun.charsets+sun.compat+sun.corba+sun.desktop+sun.ext+sun.instrument+sun.jaas+sun.jaxp+sun.jdbc+sun.jndi+sun.jsse+sun.jta+sun.kerberos+sun.localedata+sun.logging+sun.management+sun.resources+sun.rmi+sun.security.acl+sun.sunec >> >> 5550 16 88800 1 sun.desktop at 7-ea >> 4121 10 41210 11 +jdk.javac >> 3722 10 37220 6 +sun.jaxws >> 3467 13 45071 1 jdk.boot at 7-ea >> 3325 14 46550 1 jdk.tools at 7-ea >> 2502 13 32526 1 sun.jaxp at 7-ea >> 2405 14 33670 1 sun.jaxws at 7-ea >> 1677 14 23478 1 sun.corba at 7-ea >> 1441 20 28820 1 jdk.tools.jaxws at 7-ea >> 801 13 10413 9 +jdk.compiler >> 768 14 10752 1 jdk.javac at 7-ea >> 558 19 10602 1 sun.management at 7-ea >> >> Total number of strings in the config file is 729694 but 26185 >> unique strings. Total bytes occupied by these strings is 91839030. >> >> The strings created in the configuration includes >> context's name >> module's name at version >> module library path >> class name >> >> There is other opportunity for optimization. For example, for each class >> from jdk.boot, there are 40 copies in the config as one in each remote >> map >> of each context and there are 40 contexts requiring it. >> >> Mandy >> -- - DML From mandy.chung at oracle.com Fri Jun 10 12:09:47 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Fri, 10 Jun 2011 12:09:47 -0700 Subject: Review request: exports runtime support + jdk modules converted to use exports In-Reply-To: <4DF26116.4070507@redhat.com> References: <4DEEB6AE.9030003@oracle.com> <4DF06E58.803@oracle.com> <4DF0C105.2020708@oracle.com> <4DF111CD.8020506@oracle.com> <4DF1C777.9090206@oracle.com> <4DF258F6.905@Oracle.com> <4DF26116.4070507@redhat.com> Message-ID: <4DF26BFB.9000302@oracle.com> On 6/10/11 11:23 AM, David M. Lloyd wrote: > Keeping (additional) maps of every class seems ludicrous to me. In > JBoss Modules, we do everything by package, with additional filters > which can be (optionally) applied for the non-common case where you > want to only partially export a package. It is far more > memory-efficient, and empirically seems to be very fast. > > Note that it is important not to over-promise the contract. If you > tell people that they can and should routinely pick and choose classes > within a package to export or not export, you're encouraging a usage > pattern that may not be easy to implement efficiently. > > By encouraging users to export whole packages only, but give the > option of partial package export (at a possible performance cost) you > are encouraging much more performant behaviors without reducing your > total functionality. Agree and this is also what I am considering for optimization. Exporting whole packages is the common case in which we would only need to keep maps of package names and handle individual class exports differently. Mandy From mandy.chung at oracle.com Fri Jun 10 13:59:50 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Fri, 10 Jun 2011 13:59:50 -0700 Subject: Review request: exports runtime support + jdk modules converted to use exports In-Reply-To: <4DEEB6AE.9030003@oracle.com> References: <4DEEB6AE.9030003@oracle.com> Message-ID: <4DF285C6.8080907@oracle.com> Updated webrev: http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/exports-full-webrev/ The only change to the previous version is in the SimpleLibrary.StoredConfiguration class that adds indirection for the context name that reduces the config file size bloat from 90M to 16M. There are other opportunities to improve the memory and runtime performance. For example, 1. optimize for the whole package export case 2. another indirection for class names, package names, module names, etc But I'd like to address these performance issues later to push this prototype out for anyone to play with. The default requires is "jdk" which includes the tools modules that is an open issue to be addressed. There was some discussion whether it should be JRE or just the base module. In the meantime, I can change it to JRE that will reduce the config file to ~10M (the number of contexts from 40 down to 20). I'll work with Jon to fix the default in a separate patch. Mandy On 6/7/11 4:39 PM, Mandy Chung wrote: > > > I completed the changes for all jdk modules to use the new "exports" > statement. > The module-info.java for all jdk modules is at: > > http://cr.openjdk.java.net/~mchung/jigsaw/moduleinfo-exports/jdk-modules-module-info.txt > > Full webrev: > http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/exports-full-webrev/ > > Summary of changes: > 1. jigsaw runtime support for the new ModuleExports attribute (see [1]) > This includes reading the new ModuleExports attribute from > module-info.class, > updating the context to build a map from a remote class to its > suppying context > (previously the key was a remote package name), and also the > simple library > now stores the list of remote classes rather the remote packages. > Also, > javac's implementation of ModuleInfo is updated to implement the new > exports() method. > > The webrev in [1] has a temporary workaround that assume all > public types > of the platform modules are exported. In this patch, that > workaround is > removed. The runtime will read the ModuleExports attribute for all > modules including the platform ones. > > 2. Modify the ClassAnalyzer to generate exports statement. > By default, a module exports the supported APIs and it explicitly > lists > individual package using '*' wildcard. Internal APIs are only > exported > in the modules that are strictly for implementation use with the > base module > (see below). Platform modules should only export the platform > APIs and > any other public supported APIs. '*' wildcard is preferred in this > case > rather than '**' to export all public types in all subpackages to > prevent > any internal subpackages from leaking out. For example, jdk.desktop > exports individual packages e.g. java.awt.* and java.awt.dnd.* rather > that java.awt.** but java.awt.dnd.peer and java.awt.peer are > internal packages > that are exported in sun.desktop (internal module). > > It also allows to specify > additional exports statement in the input config file. When a module > reexports another module, it will export the APIs that are listed in > the "exported.packages" property defined in the input > modules.properties > file. For modules that aggregate multiple modules (e.g. jdk, > jdk.jre), > it will coalesce all exported packages into a simpler export > statement > using "**" wildcard to reexport all "exported" types in all > subpackages. > > modules.properties file includes a list of exported packages for > the platform > that is based on the CORE_PKGS.gmk and NON_CORE_PKG.gmk. > > > http://cr.openjdk.java.net/~mchung/jigsaw/moduleinfo-exports/jdk-modules-module-info.txt > shows all module-info.java of the platform modules. > > jdk.base exports a few types in the sun.reflect.annotation package > since javac depends on them for supporting serialization of > annotation. > These 4 sun.reflect.annotation classes are essentially defacto > platform > classes. > > There is some refactoring/clean up in the class analyzer that was > something > I did some time ago and I'd like to take this chance to get > reviewed. > > 3. This webrev also includes the fix I posted in [2] that the Linker > should check > if a supplying context and the requesting context are different > before adding > to the remote suppliers list. > > 4. The change in java.util.ResourceBundle and > sun.security.jca.ProviderConfig > are related to the use of system class loader to find classes in > ClassPath. > In the modular JDK, system class loader is the module class > loader for > the root module but the system classes/resources are not necessarily > visible to it (e.g. sun.security.provider.Sun and other internal > classes). > The Launcher.getBootLoader method is added as a temporary > workaround. > We'll examine the use of class loader in the jdk implementation > closer in the future. > > 5. The change in ModuleId.java fixes a memory issue in loading a > configuration. > Some test failed due to OutOfMemoryError when there are large number > of ModuleId stored in a configuration (one in each local and > remote class) > but there are limited number of unique ones. Instead of > returning a new > instance of ModuleId, ModuleId.parse() keeps a cache of all > unique ModuleId. > > 6. jmod dump-config is updated to print out the exports statement. > > $ jmod list -v jdk.javac > > jdk.javac at 7-ea > requires jdk.base@=7-ea > requires jdk.compiler@=7-ea > requires jdk.logging@=7-ea > exports com.sun.source.tree.* > exports com.sun.source.util.* > exports com.sun.tools.apt.resources.* > exports com.sun.tools.javac.* > exports com.sun.tools.javac.api.* > exports com.sun.tools.javac.code.* > exports com.sun.tools.javac.comp.* > exports com.sun.tools.javac.file.* > exports com.sun.tools.javac.jigsaw.* > exports com.sun.tools.javac.jvm.* > exports com.sun.tools.javac.main.* > exports com.sun.tools.javac.model.* > exports com.sun.tools.javac.nio.* > exports com.sun.tools.javac.parser.* > exports com.sun.tools.javac.processing.* > exports com.sun.tools.javac.resources.* > exports com.sun.tools.javac.sym.* > exports com.sun.tools.javac.tree.* > exports com.sun.tools.javac.util.* > > 7. jmod find-class > > I added this diagnostic comand to find which contexts a given class > can be found > to help diagnosing ClassNotFoundException. A class exists but if not > visible to a > module class loader, CNFE will be thrown. For example, this would be > useful > to diagnose the CNFE thrown why it can't find the > sun.security.provider.Sun > security provider. Below is the sample output. > > $ jmod find-class jdk.base sun.security.provider.Sun > configuration roots = [jdk.base at 7-ea] > class sun.security.provider.Sun found in: > context +jdk.base > remote +jdk.boot+sun.charsets+sun.jaxp+sun.localedata+sun.resources > context +jdk.jaxp > remote +jdk.boot+sun.charsets+sun.jaxp+sun.localedata+sun.resources > context +jdk.boot+sun.charsets+sun.jaxp+sun.localedata+sun.resources > module jdk.boot at 7-ea > > The above means that sun.security.provider.Sun is visible to the > module loader > for the above contexts. It's defined locally from jdk.boot at 7-ea module. > > $ jmod find-class jdk.base sun.security.pkcs.SunPKCS11 > configuration roots = [jdk.base at 7-ea] > class sun.security.pkcs.SunPKCS11 found in: > none > > Mandy > > [1] > http://mail.openjdk.java.net/pipermail/jigsaw-dev/2011-May/001292.html > [2] > http://mail.openjdk.java.net/pipermail/jigsaw-dev/2011-May/001308.html > From mbien at fh-landshut.de Sat Jun 11 16:34:01 2011 From: mbien at fh-landshut.de (Michael Bien) Date: Sun, 12 Jun 2011 01:34:01 +0200 Subject: Project Jigsaw goals and requirements In-Reply-To: <4DE7ACA2.3000708@oracle.com> References: <20110525155551.26F572DC4@eggemoggin.niobe.net> <4DE43780.7050002@info9.net><1307027230.2155.10.camel@Mokette> <4DE7ACA2.3000708@oracle.com> Message-ID: <4DF3FB69.8000903@fh-landshut.de> On 06/02/2011 05:30 PM, Richard S. Hall wrote: > On 6/2/11 11:07, Beno?t Thi?bault wrote: >> Hi everyone, >> >> This discussion about Jigsaw requirements is very interesting and >> constructive and I am really looking forward to see what Java modules >> will be capable of. >> >> Just to add my own set of remarks, I would like to point that support >> for native librairies is very important and should not be neglected. >> >> In OSGi, it is already feasible, but it is poorly documented and has >> some very strong constraints (as explained here: >> http://dev.artenum.com/blog/ben/posts/osgi_vtk_and_macosx). One of the >> most annoying things is to have to explicitly call System.loadLibrary() >> for every shared library and its dependencies to tell OSGi framework >> what .so (.dll or .jnilib) files to extract from the jar bundle. > > To be precise, this is not really an OSGi issue, but a more general > one about native library dependencies...Java can only intercept native > library load requests coming from Java code. Arbitrary native > libraries that have dependencies on other native libraries cannot be > intercepted by Java and automatically loaded by the JVM. The procedure > you describe above is simply a workaround. there is a second problem which native libs: If i remember correctly you can load a lib only once per JVM instance but the scope of the lib is still the classloader. So what everyone did before java 6 got improved jnlp support for native libs was to copy the lib into /tmp, rename it and load it from there. Thats also one of the reasons why there is/was a master-slave jvm when running applets in the new browser plugin as far as I know. Would be nice if jigsaw could handle that properly this time. E.g allowing two versions of the same library in one JVM etc. best regards, michael -- - - - - http://michael-bien.com From richard.s.hall at oracle.com Sat Jun 11 18:20:58 2011 From: richard.s.hall at oracle.com (Richard S. Hall) Date: Sat, 11 Jun 2011 21:20:58 -0400 Subject: Project Jigsaw goals and requirements In-Reply-To: <4DF3FB69.8000903@fh-landshut.de> References: <20110525155551.26F572DC4@eggemoggin.niobe.net> <4DE43780.7050002@info9.net><1307027230.2155.10.camel@Mokette> <4DE7ACA2.3000708@oracle.com> <4DF3FB69.8000903@fh-landshut.de> Message-ID: <4DF4147A.4040203@oracle.com> On 06/11/2011 07:34 PM, Michael Bien wrote: > On 06/02/2011 05:30 PM, Richard S. Hall wrote: >> On 6/2/11 11:07, Beno?t Thi?bault wrote: >>> Hi everyone, >>> >>> This discussion about Jigsaw requirements is very interesting and >>> constructive and I am really looking forward to see what Java modules >>> will be capable of. >>> >>> Just to add my own set of remarks, I would like to point that support >>> for native librairies is very important and should not be neglected. >>> >>> In OSGi, it is already feasible, but it is poorly documented and has >>> some very strong constraints (as explained here: >>> http://dev.artenum.com/blog/ben/posts/osgi_vtk_and_macosx). One of the >>> most annoying things is to have to explicitly call System.loadLibrary() >>> for every shared library and its dependencies to tell OSGi framework >>> what .so (.dll or .jnilib) files to extract from the jar bundle. >> >> To be precise, this is not really an OSGi issue, but a more general >> one about native library dependencies...Java can only intercept >> native library load requests coming from Java code. Arbitrary native >> libraries that have dependencies on other native libraries cannot be >> intercepted by Java and automatically loaded by the JVM. The >> procedure you describe above is simply a workaround. > > there is a second problem which native libs: If i remember correctly > you can load a lib only once per JVM instance but the scope of the lib > is still the classloader. So what everyone did before java 6 got > improved jnlp support for native libs was to copy the lib into /tmp, > rename it and load it from there. Thats also one of the reasons why > there is/was a master-slave jvm when running applets in the new > browser plugin as far as I know. > > Would be nice if jigsaw could handle that properly this time. E.g > allowing two versions of the same library in one JVM etc. That was not my understanding. From what I understand, a given native library can only be associated with a single class loader. Thus, if you want to load the same native library in more than one class loader, it must be loaded from a unique location in the file system (i.e., it doesn't necessarily need to be renamed, but its absolute path must be different). -> richard > > best regards, > michael > From jonathan.gibbons at oracle.com Tue Jun 21 11:12:53 2011 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Tue, 21 Jun 2011 11:12:53 -0700 Subject: Module compilation unit grammar In-Reply-To: <4D965838.2030000@oracle.com> References: <4D7AB596.6060108@oracle.com> <4D965838.2030000@oracle.com> Message-ID: <4E00DF25.7040700@oracle.com> On 04/01/2011 03:56 PM, Jonathan Gibbons wrote: > How does export interact with transitivity of required modules? > > Does export only affect the visibility of types immediately declared > in the module defined by a module compilation unit, or does it affect > types which may be available by virtue of being in required modules? > > -- Jon This question, dated 4/01/11, never got answered, and no, the question was not an April Fools ;-) -- Jon From jonathan.gibbons at oracle.com Tue Jun 21 11:47:06 2011 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Tue, 21 Jun 2011 11:47:06 -0700 Subject: module-info.class spec and attributes Message-ID: <4E00E72A.8020104@oracle.com> Jigsaw folk, We have a reasonably up-to-date proposed simple spec for module-info.java http://openjdk.java.net/projects/jigsaw/doc/topics/grammar.html But, I don't seem to be able to find a corresponding simple spec for module-info.class. I've looked on the Jigsaw group web pages and back (way back) in email. At the time, I believe Alex was giving me marked up JVMS chapters. The most recent ones I have were from June/July 2009, and represent a direction we never took -- a single ModuleDirectives attribute, like this: > ModuleDirectives_attribute { > u2 attribute_name_index; > u4 attribute_length; > u2 directives_length; > { > u2 directive_name_index; > u2 directive_flags; > u4 terms_length; > { u1 term_type; u2 term_index; } terms_table[term_length]; > u2 attributes_count; > attribute_info attributes[attributes_count]; > } directives_table[directives_length]; > } So, what is the format we would like to see for module-info.class files? -- Jon From gnormington at vmware.com Wed Jun 22 01:54:19 2011 From: gnormington at vmware.com (Glyn Normington) Date: Wed, 22 Jun 2011 10:54:19 +0200 Subject: Project Jigsaw goals and requirements In-Reply-To: References: Message-ID: This proposal didn't get a response. What do others think of the requirement? Regards, Glyn On 3 Jun 2011, at 12:19, Glyn Normington wrote: > Hi Mark > > I'm pleased to see the explicit acknowledgement of some basic OSGi interoperation requirements in the requirements document ([1]). > > I agree with David Bosschaert ([2]), that it would make sense for OSGi to support the Java SE 8 module format and, for modules which can serve equally well as OSGi bundles, I'd like to avoid dual-maintenance of module metadata and OSGi manifest. I'd like to be able to "decorate" the standard metadata. > > However, the requirement "The syntax must place all extended metadata after all standard metadata, with a clear delineation between them." precludes inline decorations. The result would be duplication and clunkiness. > > I propose that this requirement be changed so that standard metadata could be decorated inline (the decorations would be ignored by the Java SE module system). > > What do you think? > > Regards, > Glyn > [1] http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12 > [2] http://osgithoughts.blogspot.com/2011/05/java-se-8-modularity-requirements.html From peter.kriens at aqute.biz Wed Jun 22 01:58:32 2011 From: peter.kriens at aqute.biz (Peter Kriens) Date: Wed, 22 Jun 2011 10:58:32 +0200 Subject: Project Jigsaw goals and requirements In-Reply-To: References: Message-ID: <923CB547-9DE4-41B6-9382-2B0B34ED5558@aQute.biz> I agree that this makes sense. It be very inefficient to have to repeat information. Kind regards, Peter Kriens On 22 jun 2011, at 10:54, Glyn Normington wrote: > This proposal didn't get a response. What do others think of the requirement? > > Regards, > Glyn > > On 3 Jun 2011, at 12:19, Glyn Normington wrote: > >> Hi Mark >> >> I'm pleased to see the explicit acknowledgement of some basic OSGi interoperation requirements in the requirements document ([1]). >> >> I agree with David Bosschaert ([2]), that it would make sense for OSGi to support the Java SE 8 module format and, for modules which can serve equally well as OSGi bundles, I'd like to avoid dual-maintenance of module metadata and OSGi manifest. I'd like to be able to "decorate" the standard metadata. >> >> However, the requirement "The syntax must place all extended metadata after all standard metadata, with a clear delineation between them." precludes inline decorations. The result would be duplication and clunkiness. >> >> I propose that this requirement be changed so that standard metadata could be decorated inline (the decorations would be ignored by the Java SE module system). >> >> What do you think? >> >> Regards, >> Glyn >> [1] http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12 >> [2] http://osgithoughts.blogspot.com/2011/05/java-se-8-modularity-requirements.html > From Alan.Bateman at oracle.com Fri Jun 24 09:28:14 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 24 Jun 2011 17:28:14 +0100 Subject: java -modulepath Message-ID: <4E04BB1E.3040602@oracle.com> A while back I started looking into supporting "exploded modules". Lots of things going on (jdk7 mostly) but I've found a few cycles to do a bit more on this so here's a brief status update. The main update since the first prototype is that the startup has improved greatly (as expected) after switching to PathContext and generating the path-base configuration at startup. Native libraries are now supported if deployed into a -bin tree that mirrors the modulepath. So if javac compiles a module "gus" to -d mymodules (mymodules/gus/**) then gus's native libraries are loaded from mymodules-bin/gus/lib. Clearly any convention would need to be agreed but this one is easy to understand. It would be easy to update jpkg and jmod to understand this convention when creating packages or installing from a modules directory. One of the goals with this is to use it in the JDK build so that we can move away from the post-processing approach that we have in the jigsaw build now. To that end, both module and legacy modes have been updated to allow the JDK classes be in a module path structure. This should allow us to create the equivalent of the JDK's "make all" where the classes are compiled into a classes tree today, and a modules tree in the future. A "make images" builds the JDK image including rt.jar today, in the future it could create the modules image, installing the JDK modules into the module library. This will all require coordination with the other build changes going into jdk8 but a reasonable direction to take as it makes incremental building very simple and fast (and incremental builds are really important when working on the JDK). Mandy recently published the runtime changes to support the ModuleExports attribute. I've done an initial merge with her patch in a side forest and added code to check that a type supplied by a remote context is exported. Reexporting is working too although I need to do more testing on this to verify the semantics. Once I've done that and do a bit more clean-up then I'll send our a new webrev for review. -Alan. From tmarble at info9.net Fri Jun 24 12:06:44 2011 From: tmarble at info9.net (Tom Marble) Date: Fri, 24 Jun 2011 14:06:44 -0500 Subject: java -modulepath In-Reply-To: <4E04BB1E.3040602@oracle.com> References: <4E04BB1E.3040602@oracle.com> Message-ID: Alan: This sounds fantastic! I am eagerly awaiting this webrev as this functionality seems critical to experimenting with and verifying intended Jigsaw behavior prior to adding modules to "the system module library". Would it be possible for you to document some accompanying "Hello Modular World!" examples to demonstrate how to think about Jigsaw design? Grokking the new design patterns is complicated by the lack of examples and the existing documention covering many points in development history (many of which may now be out of date). Thanks, --Tom From benjamin.john.evans at gmail.com Fri Jun 24 12:43:25 2011 From: benjamin.john.evans at gmail.com (Ben Evans) Date: Fri, 24 Jun 2011 20:43:25 +0100 Subject: java -modulepath Message-ID: Hi, Completely agree with Tom. On a related note, the LJC are in the early stages of organizing some hack sessions on new JDK features - including Jigsaw and lambda. If the developers involved in implementing the features discussed in this thread could give us as good an idea as possible of when this featureset would be ready then we can try to synchronize our efforts to try it out accordingly. We expect that our hack session participants will be a mix of experienced end-user devs with a strong interest in modularity and in trying out real world scenarios in Jigsaw - plus a core group who are comfortable with OpenJDK and how to build it. That is, broadly working features with potentially some known issues is probably what we're looking for. Suggestions for how best to spend our efforts / maximize usefulness would also be welcome. Thanks, Ben (on behalf of the LJC) On Jun 24, 2011 8:07 PM, "Tom Marble" wrote: From alex.buckley at oracle.com Fri Jun 24 17:17:20 2011 From: alex.buckley at oracle.com (Alex Buckley) Date: Fri, 24 Jun 2011 17:17:20 -0700 Subject: module-info.class spec and attributes In-Reply-To: <4E00E72A.8020104@oracle.com> References: <4E00E72A.8020104@oracle.com> Message-ID: <4E052910.5080001@oracle.com> On 6/21/2011 11:47 AM, Jonathan Gibbons wrote: > So, what is the format we would like to see for module-info.class files? Please see attached for a proposal on ClassFile extensions that would eventually become part of the Java Module System [1]. The proposal refers to the module system requirements document [2] where necessary. Alex [1]http://mail.openjdk.java.net/pipermail/jigsaw-dev/2011-May/001309.html [2]http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12 From alex.buckley at oracle.com Fri Jun 24 18:01:19 2011 From: alex.buckley at oracle.com (Alex Buckley) Date: Fri, 24 Jun 2011 18:01:19 -0700 Subject: module-info.class spec and attributes In-Reply-To: <4E052910.5080001@oracle.com> References: <4E00E72A.8020104@oracle.com> <4E052910.5080001@oracle.com> Message-ID: <4E05335F.6090609@oracle.com> Since attachments seem to be scrubbed, here is the text of the proposal, obviously without links to the requirements document. ClassFile changes for Java Modules Binary form of a module declaration (a.k.a. module-info.class) A compilation unit that contains a module declaration (and potentially additional arbitrary content) is compiled to a ClassFile structure like any other compilation unit. By convention, the name of a compilation unit that contains a module declaration is module-info.java, echoing the package-info.java convention for a compilation unit that contains solely a package declaration. Consequently, by convention, the name for the compiled form of a module declaration is module-info.class. A new flag in the ClassFile.access_flags item, ACC_MODULE (0x0800), indicates that the ClassFile represents a module rather than a class or interface. Note that this flag plays a similar role to ACC_INTERFACE (0x0200), and does not describe accessibility of a class or interface. If ACC_MODULE is set in ClassFile.access_flags, then no other flag in ClassFile.access_flags may be set, and the following rules apply to the rest of the ClassFile structure: * major_version, minor_version: >=52.0 (Java SE 8 and above) * this_class: [Module's name in internal form (JVMS 4.2.1)]/module-info (Traditionally, if this_class indicates "P/Q/R", then the ClassFile can be expected to live in a file R.class in a directory P/Q representing a package. This explains why "/module-info" is a suffix in this_class above: if this_class indicates "P/Q/module-info", then the ClassFile can be expected to live in a file module-info.class in a directory P/Q representing a module. The "real name" of the module, shorn of "/module-info", can be obtained from the Module attribute.) * super_class, interfaces_count, fields_count, methods_count: 0 * attributes: One Module attribute must be present, to record the name and version of the module. At most one of each of the ModuleRequires, ModuleProvides, ModulePermits, ModuleExports, ModuleClass, and ModuleData attributes may be present. Except for these attributes and Synthetic, SourceFile, SourceDebugExtension, and Deprecated, none of the pre-defined attributes in JVMS 4.7 may appear. The CONSTANT_ModuleId_info structure The CONSTANT_ModuleId_info structure in the constant pool is used to represent a pair of a module name and a module version. CONSTANT_ModuleId_info { u1 tag; // 19 u2 name_index; // Points to a CONSTANT_Utf8_info representing module name in internal form (JVMS 4.2.1) u2 version_index; // Optionally points to a CONSTANT_Utf8_info representing module version } If the version_index item is 0 (as opposed to pointing to a CONSTANT_Utf8_info structure which holds the string "0"), then no module version is present. If the version_index item is not 0, then a module version is present. There is no required structure or semantics for the underlying UTF8 string which represents a module version. The Module attribute Module_attribute { u2 attribute_name_index; u4 attribute_length; u2 module_id_index; } The items of the Module_attribute structure are as follows: attribute_name_index The value of the attribute_name_index item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_Utf8_info structure representing the string "Module". attribute_length The value of the attribute_length item must be 2. module_id_index The value of the module_id_index item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_ModuleId_info structure representing the name and version of the module represented by this ClassFile. The name must be equal to the module name indicated by the ClassFile.this_class item without the "/module-info" suffix. The ModuleRequires attribute ModuleRequires_attribute { u2 attribute_name_index; u4 attribute_length; u2 requires_length; { u2 requires_index; u1 flags; } requires_table[requires_length]; } attribute_name_index The value of the attribute_name_index item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_Utf8_info structure representing the string "ModuleRequires". attribute_length The value of the attribute_length item is the length of the attribute excluding the initial six bytes. requires_length The value of the requires_length indicates the number of entries in the requires_table. requires_table Each requires_index must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_ModuleId_info structure representing the target module on which this module depends. The value of the associated flags item is as follows: * 0x01 indicates this dependency on the target module is optional. * 0x02 indictates the target module's types must be loaded by the same defining classloader as the types of the module represented by this ClassFile. A module name may be referenced by at most one entry in a requires_table. The ModulePermits attribute ModulePermits_attribute { u2 attribute_name_index; u4 attribute_length; u2 permits_length; { u2 permits_index } permits_table[permits_length]; } attribute_name_index The value of the attribute_name_index item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_Utf8_info structure representing the string "ModulePermits". attribute_length The value of the attribute_length item is the length of the attribute excluding the initial six bytes. permits_length The value of the permits_length indicates the number of entries in the permits_table. permits_table The value of each permits_index in this item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_ModuleId_info structure representing a module which is permitted to have a dependency on the module represented by this ClassFile. (It is possible that a module system uses only the name of the permitted module, not its version, in determining visibility.) A module name may be referenced by at most one entry in a permits_table. The ModuleProvides attribute ModuleProvides_attribute { u2 attribute_name_index; u4 attribute_length; u2 provides_length; { u2 provides_index; } provides_table[provides_length]; } attribute_name_index The value of the attribute_name_index item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_Utf8_info structure representing the string "ModuleProvides". attribute_length The value of the attribute_length item is the length of the attribute excluding the initial six bytes. provides_length The value of the provides_length indicates the number of entries in the provides_table. provides_table The value of each provides_index in this item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_ModuleId_info structure representing a module that is an alias for the module represented by this ClassFile. A module name may be referenced by at most one entry in a provides_table. The ModuleExports attribute ModuleExports_attribute { u2 attribute_name_index; u4 attribute_length; u2 exports_length; { u2 exports_index; u1 export_kind; u2 source_index; } exports_table[exports_length]; } attribute_name_index The value of the attribute_name_index item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_Utf8_info structure representing the string "ModuleExports". attribute_length The value of the attribute_length item is the length of the attribute excluding the initial six bytes. exports_length The value of the exports_length indicates the number of entries in the exports_table. exports_table The value of each export_index in this item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_Utf8_info structure representing an entity to be exported by the module represented by this ClassFile. The value of the associated export_kind item indicates the kind of entity to be exported, as follows: * 0x01 indicates the entity is a class or interface; it is the entity to be exported. * 0x02 indicates the entity is a class or interface; it and its member types are the entities to be exported. * 0x04 indicates the entity is a package; the member top-level classes and interfaces (and their member types) of this package are the entities to be exported. * 0x08 indicates the entity is a package; the member top-level classes and interfaces (and their member types) of this package and all its subpackages (recursively) are the entities to be exported. The value of the associated source_index item must be a valid index into the constant_pool table where the entry is a CONSTANT_ModuleId_info structure representing the module which declared the exported entity. (In most cases, where the exported entity is declared in the current module, source_index will simply point to the same constant pool entry as the Module attribute. But where the exported entity is re-exported, source_index may assist resolution and diagnostics.) The ModuleClass attribute ModuleClass_attribute { u2 attribute_name_index; u4 attribute_length; u2 main_class_index; } attribute_name_index The value of the attribute_name_index item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_Utf8_info structure representing the string "ModuleClass". attribute_length The value of the attribute_length item is 2. main_class_index The value of the main_class_index item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_Class_info structure representing the name of the class which is the entrypoint to the module represented by this ClassFile. The ModuleData attribute ModuleExtension_attribute { u2 attribute_name_index; u4 attribute_length; u2 data_index; } attribute_name_index The value of the attribute_name_index item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_Utf8_info structure representing the string "ModuleData". attribute_length The value of the attribute_length item is 2. data_index The value of the main_class item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_Utf8_info structure representing the content, other than the module declaration, of the compilation unit that declared the module represented by this ClassFile. From jonathan.gibbons at oracle.com Fri Jun 24 18:22:33 2011 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Fri, 24 Jun 2011 18:22:33 -0700 Subject: module-info.class spec and attributes In-Reply-To: <4E05335F.6090609@oracle.com> References: <4E00E72A.8020104@oracle.com> <4E052910.5080001@oracle.com> <4E05335F.6090609@oracle.com> Message-ID: <4E053859.3060703@oracle.com> Minor not-cut-n-paste typo: > The ModuleData attribute > > ModuleExtension_attribute { > u2 attribute_name_index; > u4 attribute_length; > u2 data_index; > } -- Jon On 06/24/2011 06:01 PM, Alex Buckley wrote: > Since attachments seem to be scrubbed, here is the text of the > proposal, obviously without links to the requirements document. > > > ClassFile changes for Java Modules > > Binary form of a module declaration (a.k.a. module-info.class) > A compilation unit that contains a module declaration (and potentially > additional arbitrary content) is compiled to a ClassFile structure > like any other compilation unit. > > By convention, the name of a compilation unit that contains a module > declaration is module-info.java, echoing the package-info.java > convention for a compilation unit that contains solely a package > declaration. Consequently, by convention, the name for the compiled > form of a module declaration is module-info.class. > > A new flag in the ClassFile.access_flags item, ACC_MODULE (0x0800), > indicates that the ClassFile represents a module rather than a class > or interface. Note that this flag plays a similar role to > ACC_INTERFACE (0x0200), and does not describe accessibility of a class > or interface. > > If ACC_MODULE is set in ClassFile.access_flags, then no other flag in > ClassFile.access_flags may be set, and the following rules apply to > the rest of the ClassFile structure: > > * major_version, minor_version: >=52.0 (Java SE 8 and above) > * this_class: [Module's name in internal form (JVMS > 4.2.1)]/module-info > > (Traditionally, if this_class indicates "P/Q/R", then the > ClassFile can be expected to live in a file R.class in a directory P/Q > representing a package. This explains why "/module-info" is a suffix > in this_class above: if this_class indicates "P/Q/module-info", then > the ClassFile can be expected to live in a file module-info.class in a > directory P/Q representing a module. The "real name" of the module, > shorn of "/module-info", can be obtained from the Module attribute.) > * super_class, interfaces_count, fields_count, methods_count: 0 > * attributes: One Module attribute must be present, to record the > name and version of the module. At most one of each of the > ModuleRequires, ModuleProvides, ModulePermits, ModuleExports, > ModuleClass, and ModuleData attributes may be present. Except for > these attributes and Synthetic, SourceFile, SourceDebugExtension, and > Deprecated, none of the pre-defined attributes in JVMS 4.7 may appear. > > The CONSTANT_ModuleId_info structure > > The CONSTANT_ModuleId_info structure in the constant pool is used to > represent a pair of a module name and a module version. > > CONSTANT_ModuleId_info { > u1 tag; > // 19 > u2 name_index; > // Points to a CONSTANT_Utf8_info representing module name in > internal form (JVMS 4.2.1) > u2 version_index; > // Optionally points to a CONSTANT_Utf8_info representing module > version > } > > If the version_index item is 0 (as opposed to pointing to a > CONSTANT_Utf8_info structure which holds the string "0"), then no > module version is present. > > If the version_index item is not 0, then a module version is present. > There is no required structure or semantics for the underlying UTF8 > string which represents a module version. > > The Module attribute > > Module_attribute { > u2 attribute_name_index; > u4 attribute_length; > u2 module_id_index; > } > > The items of the Module_attribute structure are as follows: > > attribute_name_index > The value of the attribute_name_index item must be a valid index > into the constant_pool table. The constant_pool entry at that index > must be a CONSTANT_Utf8_info structure representing the string "Module". > attribute_length > The value of the attribute_length item must be 2. > module_id_index > The value of the module_id_index item must be a valid index into > the constant_pool table. The constant_pool entry at that index must be > a CONSTANT_ModuleId_info structure representing the name and version > of the module represented by this ClassFile. > The name must be equal to the module name indicated by the > ClassFile.this_class item without the "/module-info" suffix. > > The ModuleRequires attribute > > ModuleRequires_attribute { > u2 attribute_name_index; > u4 attribute_length; > u2 requires_length; > { u2 requires_index; u1 flags; } requires_table[requires_length]; > } > > attribute_name_index > The value of the attribute_name_index item must be a valid index > into the constant_pool table. The constant_pool entry at that index > must be a CONSTANT_Utf8_info structure representing the string > "ModuleRequires". > attribute_length > The value of the attribute_length item is the length of the > attribute excluding the initial six bytes. > requires_length > The value of the requires_length indicates the number of entries > in the requires_table. > requires_table > Each requires_index must be a valid index into the constant_pool > table. The constant_pool entry at that index must be a > CONSTANT_ModuleId_info structure representing the target module on > which this module depends. > The value of the associated flags item is as follows: > > * 0x01 indicates this dependency on the target module is > optional. > * 0x02 indictates the target module's types must be loaded by > the same defining classloader as the types of the module represented > by this ClassFile. > > A module name may be referenced by at most one entry in a > requires_table. > > The ModulePermits attribute > > ModulePermits_attribute { > u2 attribute_name_index; > u4 attribute_length; > u2 permits_length; > { u2 permits_index } permits_table[permits_length]; > } > > attribute_name_index > The value of the attribute_name_index item must be a valid index > into the constant_pool table. The constant_pool entry at that index > must be a CONSTANT_Utf8_info structure representing the string > "ModulePermits". > attribute_length > The value of the attribute_length item is the length of the > attribute excluding the initial six bytes. > permits_length > The value of the permits_length indicates the number of entries in > the permits_table. > permits_table > The value of each permits_index in this item must be a valid index > into the constant_pool table. The constant_pool entry at that index > must be a CONSTANT_ModuleId_info structure representing a module which > is permitted to have a dependency on the module represented by this > ClassFile. > (It is possible that a module system uses only the name of the > permitted module, not its version, in determining visibility.) > A module name may be referenced by at most one entry in a > permits_table. > > The ModuleProvides attribute > > ModuleProvides_attribute { > u2 attribute_name_index; > u4 attribute_length; > u2 provides_length; > { u2 provides_index; } provides_table[provides_length]; > } > > attribute_name_index > The value of the attribute_name_index item must be a valid index > into the constant_pool table. The constant_pool entry at that index > must be a CONSTANT_Utf8_info structure representing the string > "ModuleProvides". > attribute_length > The value of the attribute_length item is the length of the > attribute excluding the initial six bytes. > provides_length > The value of the provides_length indicates the number of entries > in the provides_table. > provides_table > The value of each provides_index in this item must be a valid > index into the constant_pool table. The constant_pool entry at that > index must be a CONSTANT_ModuleId_info structure representing a module > that is an alias for the module represented by this ClassFile. > A module name may be referenced by at most one entry in a > provides_table. > > The ModuleExports attribute > > ModuleExports_attribute { > u2 attribute_name_index; > u4 attribute_length; > u2 exports_length; > { u2 exports_index; u1 export_kind; u2 source_index; } > exports_table[exports_length]; > } > > attribute_name_index > The value of the attribute_name_index item must be a valid index > into the constant_pool table. The constant_pool entry at that index > must be a CONSTANT_Utf8_info structure representing the string > "ModuleExports". > attribute_length > The value of the attribute_length item is the length of the > attribute excluding the initial six bytes. > exports_length > The value of the exports_length indicates the number of entries in > the exports_table. > exports_table > The value of each export_index in this item must be a valid index > into the constant_pool table. The constant_pool entry at that index > must be a CONSTANT_Utf8_info structure representing an entity to be > exported by the module represented by this ClassFile. > The value of the associated export_kind item indicates the kind of > entity to be exported, as follows: > > * 0x01 indicates the entity is a class or interface; it is the > entity to be exported. > * 0x02 indicates the entity is a class or interface; it and > its member types are the entities to be exported. > * 0x04 indicates the entity is a package; the member top-level > classes and interfaces (and their member types) of this package are > the entities to be exported. > * 0x08 indicates the entity is a package; the member top-level > classes and interfaces (and their member types) of this package and > all its subpackages (recursively) are the entities to be exported. > > The value of the associated source_index item must be a valid > index into the constant_pool table where the entry is a > CONSTANT_ModuleId_info structure representing the module which > declared the exported entity. (In most cases, where the exported > entity is declared in the current module, source_index will simply > point to the same constant pool entry as the Module attribute. But > where the exported entity is re-exported, source_index may assist > resolution and diagnostics.) > > The ModuleClass attribute > > ModuleClass_attribute { > u2 attribute_name_index; > u4 attribute_length; > u2 main_class_index; > } > > attribute_name_index > The value of the attribute_name_index item must be a valid index > into the constant_pool table. The constant_pool entry at that index > must be a CONSTANT_Utf8_info structure representing the string > "ModuleClass". > attribute_length > The value of the attribute_length item is 2. > main_class_index > The value of the main_class_index item must be a valid index into > the constant_pool table. The constant_pool entry at that index must be > a CONSTANT_Class_info structure representing the name of the class > which is the entrypoint to the module represented by this ClassFile. > > The ModuleData attribute > > ModuleExtension_attribute { > u2 attribute_name_index; > u4 attribute_length; > u2 data_index; > } > > attribute_name_index > The value of the attribute_name_index item must be a valid index > into the constant_pool table. The constant_pool entry at that index > must be a CONSTANT_Utf8_info structure representing the string > "ModuleData". > attribute_length > The value of the attribute_length item is 2. > data_index > The value of the main_class item must be a valid index into the > constant_pool table. The constant_pool entry at that index must be a > CONSTANT_Utf8_info structure representing the content, other than the > module declaration, of the compilation unit that declared the module > represented by this ClassFile. From weijun.wang at oracle.com Fri Jun 24 18:30:28 2011 From: weijun.wang at oracle.com (Weijun Wang) Date: Sat, 25 Jun 2011 09:30:28 +0800 Subject: java -modulepath In-Reply-To: <4E04BB1E.3040602@oracle.com> References: <4E04BB1E.3040602@oracle.com> Message-ID: On Jun 25, 2011, at 12:28 AM, Alan Bateman wrote: > > One of the goals with this is to use it in the JDK build so that we can move away from the post-processing approach that we have in the jigsaw build now. To that end, both module and legacy modes have been updated to allow the JDK classes be in a module path structure. This should allow us to create the equivalent of the JDK's "make all" where the classes are compiled into a classes tree today, and a modules tree in the future. A "make images" builds the JDK image including rt.jar today, in the future it could create the modules image, installing the JDK modules into the module library. This will all require coordination with the other build changes going into jdk8 but a reasonable direction to take as it makes incremental building very simple and fast (and incremental builds are really important when working on the JDK). I completely agree with this. Every time Eclipse announced a new release, I would grab an installer to see how it feels. It never worked with "exploded" classes directory and complained java.lang.Object missing. Therefore, I've never been able to use any version for more than one hour. -Max > > > > -Alan. From mike.duigou at oracle.com Fri Jun 24 18:31:06 2011 From: mike.duigou at oracle.com (mike.duigou at oracle.com) Date: Fri, 24 Jun 2011 18:31:06 -0700 (PDT) Subject: Auto Reply: jigsaw-dev Digest, Vol 29, Issue 13 Message-ID: <1a3e65be-d461-4f57-9cd0-a0e3a89f44de@default> I am on vacation until July 11, 2011. For urgent matters contact my manager frances.ho at oracle.com or for very urgent matters call my cell phone at five one zero nine one eight five seven two zero. From Alan.Bateman at oracle.com Sun Jun 26 07:07:33 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sun, 26 Jun 2011 15:07:33 +0100 Subject: java -modulepath In-Reply-To: References: <4E04BB1E.3040602@oracle.com> Message-ID: <4E073D25.8000908@oracle.com> Weijun Wang wrote: > : > > Every time Eclipse announced a new release, I would grab an installer to see how it feels. It never worked with "exploded" classes directory and complained java.lang.Object missing. Therefore, I've never been able to use any version for more than one hour. > If the error is that j.l.Object is missing then I will guess that it must be setting the bootclasspath and assuming that rt.jar exists (as otherwise it should at least start). More generally, this is a reminder of the challenge that we will have moving from the existing legacy image to a modules image. There will be code that assumes rt.jar and the existing layout. -Alan. From Alan.Bateman at oracle.com Sun Jun 26 07:37:22 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sun, 26 Jun 2011 15:37:22 +0100 Subject: java -modulepath In-Reply-To: References: <4E04BB1E.3040602@oracle.com> Message-ID: <4E074422.80207@oracle.com> Tom Marble wrote: > > Alan: > > This sounds fantastic! > > I am eagerly awaiting this webrev as this functionality seems critical > to experimenting with and verifying intended Jigsaw behavior prior to > adding modules to "the system module library". > > Would it be possible for you to document some accompanying "Hello > Modular World!" examples to demonstrate how to think about Jigsaw > design? Grokking the new design patterns is complicated by the lack of > examples and the existing documention covering many points in > development history (many of which may now be out of date). > > Thanks, > > --Tom > Thanks Tom. I will send the webrev soon (but not today as I've several things to get cleared away before going on vacation). The tests in jdk/test/org/openjdk/jigsaw/ might be useful to see where things are now. They compile with -modulepath and then install the modules into a module library (named "z.lib" rather than the system module library). The most obvious difference with supporting modulepath at runtime is you can run with the modules that javac generates and so you avoid needing to install them. This is useful for fast edit/build/run cycles. Point taken that we will need to update slides and other material to explain the possible workflows. -Alan. From sean.mullan at oracle.com Mon Jun 27 06:01:27 2011 From: sean.mullan at oracle.com (Sean Mullan) Date: Mon, 27 Jun 2011 09:01:27 -0400 Subject: Bug 4715154 Message-ID: <4E087F27.70202@oracle.com> Several jigsaw unit tests are failing on Windows because of bug 4715154 [1] which was closed a long time ago as "Will Not Fix". Does anyone have any suggestions as the best way to workaround this, short of replacing the call to FileChannel.map with FileChannel.read? Are there any plans to fix this in the future? Thanks, Sean [1] http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4715154 From Alan.Bateman at oracle.com Mon Jun 27 06:19:04 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 27 Jun 2011 14:19:04 +0100 Subject: Bug 4715154 In-Reply-To: <4E087F27.70202@oracle.com> References: <4E087F27.70202@oracle.com> Message-ID: <4E088348.4050409@oracle.com> Sean Mullan wrote: > Several jigsaw unit tests are failing on Windows because of bug > 4715154 [1] which was closed a long time ago as "Will Not Fix". Does > anyone have any suggestions as the best way to workaround this, short > of replacing the call to FileChannel.map with FileChannel.read? Are > there any plans to fix this in the future? > > Thanks, > Sean > > [1] http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4715154 This is a limitation of Windows and I'm not aware of any way to create a file-mapping that would allow the file be deleted while the mapping exists (if there is a way then it would be great to know about it). In a couple of tests we've had to do an ugly System.gc() + short sleep at the end of the test to trigger the unmap. -Alan From sean.mullan at oracle.com Mon Jun 27 06:48:44 2011 From: sean.mullan at oracle.com (Sean Mullan) Date: Mon, 27 Jun 2011 09:48:44 -0400 Subject: Bug 4715154 In-Reply-To: <4E088348.4050409@oracle.com> References: <4E087F27.70202@oracle.com> <4E088348.4050409@oracle.com> Message-ID: <4E088A3C.3010809@oracle.com> On 6/27/11 9:19 AM, Alan Bateman wrote: > Sean Mullan wrote: >> Several jigsaw unit tests are failing on Windows because of bug 4715154 [1] >> which was closed a long time ago as "Will Not Fix". Does anyone have any >> suggestions as the best way to workaround this, short of replacing the call to >> FileChannel.map with FileChannel.read? Are there any plans to fix this in the >> future? >> >> Thanks, >> Sean >> >> [1] http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4715154 > This is a limitation of Windows and I'm not aware of any way to create a > file-mapping that would allow the file be deleted while the mapping exists (if > there is a way then it would be great to know about it). In a couple of tests > we've had to do an ugly System.gc() + short sleep at the end of the test to > trigger the unmap. It isn't only deleting the file that causes problems. If you try to subsequently open the file and do other operations on it, they can fail, and we do this when signing jmod files. We may be able to avoid calling FileChannel.map. I'm looking more into it ... --Sean From gnormington at vmware.com Mon Jun 27 08:01:00 2011 From: gnormington at vmware.com (Glyn Normington) Date: Mon, 27 Jun 2011 16:01:00 +0100 Subject: =?iso-8859-1?Q?D=E9j=E0_vu?= Message-ID: <8BE1580C-5349-44C8-A4AD-79021C9B59BB@vmware.com> For those of you who are not aware of my blog: http://underlap.blogspot.com/2011/06/jigsaw-deja-vu.html Regards, Glyn From sean.mullan at oracle.com Mon Jun 27 08:57:33 2011 From: sean.mullan at oracle.com (Sean Mullan) Date: Mon, 27 Jun 2011 11:57:33 -0400 Subject: Bug 4715154 In-Reply-To: <4E088348.4050409@oracle.com> References: <4E087F27.70202@oracle.com> <4E088348.4050409@oracle.com> Message-ID: <4E08A86D.4010101@oracle.com> Hey Alan, Would it be a good idea if we add this Windows limitation to the NIO javadocs somewhere, say in the FileChannel class? I spent a lot of time chasing this down until I finally figured out it was a known issue. --Sean On 6/27/11 9:19 AM, Alan Bateman wrote: > Sean Mullan wrote: >> Several jigsaw unit tests are failing on Windows because of bug 4715154 [1] >> which was closed a long time ago as "Will Not Fix". Does anyone have any >> suggestions as the best way to workaround this, short of replacing the call to >> FileChannel.map with FileChannel.read? Are there any plans to fix this in the >> future? >> >> Thanks, >> Sean >> >> [1] http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4715154 > This is a limitation of Windows and I'm not aware of any way to create a > file-mapping that would allow the file be deleted while the mapping exists (if > there is a way then it would be great to know about it). In a couple of tests > we've had to do an ugly System.gc() + short sleep at the end of the test to > trigger the unmap. > > -Alan From mandy.chung at oracle.com Mon Jun 27 18:25:58 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Tue, 28 Jun 2011 09:25:58 +0800 Subject: Jigsaw Quick Start guide Message-ID: <4E092DA6.4010606@oracle.com> I put together a document to give a quick demo on how to create a module: http://openjdk.java.net/projects/jigsaw/doc/quickstart.html It's just a simple example and the content will grow over time. It will be updated for the various works we're working on such as exports, exploded module, and the signed module work when we get them in. Mandy From forax at univ-mlv.fr Tue Jun 28 00:20:26 2011 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Tue, 28 Jun 2011 09:20:26 +0200 Subject: Jigsaw Quick Start guide In-Reply-To: <4E092DA6.4010606@oracle.com> References: <4E092DA6.4010606@oracle.com> Message-ID: <4E0980BA.1050702@univ-mlv.fr> On 06/28/2011 03:25 AM, Mandy Chung wrote: > I put together a document to give a quick demo on how to create a > module: > http://openjdk.java.net/projects/jigsaw/doc/quickstart.html > > It's just a simple example and the content will grow over time. It > will be updated for the various works we're working on such as > exports, exploded module, and the signed module work when we get them in. > > Mandy Hi Mandy, There is a typo at the end of section 1: "grammer" -> "grammar". Also in my opinion, there is not enough emphasis about the fact that the way to organize a Java project has changed, src/org.foo.bar/org/foo/bar instead of src/org/foo/bar. cheers, R?mi From Alan.Bateman at oracle.com Tue Jun 28 00:30:00 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 28 Jun 2011 08:30:00 +0100 Subject: Jigsaw Quick Start guide In-Reply-To: <4E0980BA.1050702@univ-mlv.fr> References: <4E092DA6.4010606@oracle.com> <4E0980BA.1050702@univ-mlv.fr> Message-ID: <4E0982F8.5070509@oracle.com> R?mi Forax wrote: > : > > Also in my opinion, there is not enough emphasis about the fact > that the way to organize a Java project has changed, > src/org.foo.bar/org/foo/bar instead of src/org/foo/bar. Mandy - Jon and Alex's slides [1] on making javac module-aware might be useful to reference or maybe a paragraph to explain the module replacement of classpath. -Alan [1] http://openjdk.java.net/projects/jigsaw/doc/ModulesAndJavac.pdf From mandy.chung at oracle.com Tue Jun 28 07:06:01 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Tue, 28 Jun 2011 22:06:01 +0800 Subject: Jigsaw Quick Start guide In-Reply-To: <4E0982F8.5070509@oracle.com> References: <4E092DA6.4010606@oracle.com> <4E0980BA.1050702@univ-mlv.fr> <4E0982F8.5070509@oracle.com> Message-ID: <4E09DFC9.2070703@oracle.com> On 6/28/11 3:30 PM, Alan Bateman wrote: > R?mi Forax wrote: >> There is a typo at the end of section 1: "grammer" -> "grammar". >> >> Also in my opinion, there is not enough emphasis about the fact >> that the way to organize a Java project has changed, >> src/org.foo.bar/org/foo/bar instead of src/org/foo/bar. > Mandy - Jon and Alex's slides [1] on making javac module-aware might > be useful to reference or maybe a paragraph to explain the module > replacement of classpath. > > -Alan > > [1] http://openjdk.java.net/projects/jigsaw/doc/ModulesAndJavac.pdf Thanks, Remi and Alan. I'll add a paragraph to explain the new proposed source organization for modules. Mandy From Roger.Riggs at oracle.com Tue Jun 28 10:41:42 2011 From: Roger.Riggs at oracle.com (Roger Riggs) Date: Tue, 28 Jun 2011 13:41:42 -0400 Subject: Jigsaw Quick Start guide In-Reply-To: <4E092DA6.4010606@oracle.com> References: <4E092DA6.4010606@oracle.com> Message-ID: <4E0A1256.4010608@oracle.com> Hi, Thanks for the writeup of jigsaw modules. This is more of a comment on the module compilation unit grammar and the choice of keywords and semantics. http://openjdk.java.net/projects/jigsaw/doc/topics/grammar.html The "class" keyword in the module-info.java sticks out as an awkward usage. There are plenty of classes in the module. The significance the class is that it is a/the main class. The keyword should identify the usage of the class. "main" would be more meaningful and intuitive. The "class" keyword is being overloaded. While it identifies a main class it is also being used to indicate module is the root of the static context for the module as a whole and the class name is not relevant to that function. A separate keyword defining the module as requiring a static context would clearer to separate the semantics of the entry point from the "classloader" behavior. Can a module, like a jar file could contain multiple classes with main methods? In current usage, for jar files, the main class can be given on the command line allowing multiple applications to share the overhead of packaging. Having a solitary entry point may result in extra overhead because each application must be in a separate module. Roger On 06/27/2011 09:25 PM, Mandy Chung wrote: > I put together a document to give a quick demo on how to create a > module: > http://openjdk.java.net/projects/jigsaw/doc/quickstart.html > > It's just a simple example and the content will grow over time. It > will be updated for the various works we're working on such as > exports, exploded module, and the signed module work when we get them in. > > Mandy From alex.buckley at oracle.com Tue Jun 28 11:10:27 2011 From: alex.buckley at oracle.com (Alex Buckley) Date: Tue, 28 Jun 2011 11:10:27 -0700 Subject: Jigsaw Quick Start guide In-Reply-To: <4E0A1256.4010608@oracle.com> References: <4E092DA6.4010606@oracle.com> <4E0A1256.4010608@oracle.com> Message-ID: <4E0A1913.4020802@oracle.com> I recall discussion of different entrypoints when a module is used in different contexts, e.g. applet v. command-line launcher. However, the Java Module System requirements don't say anything about this, so I stuck with a solitary, unqualified 'class ...' construct in the module declaration. The closest "context-aware" requirement is that a module "can be used both as a JAR on the class path and installed as a module in a module library". Alex On 6/28/2011 10:41 AM, Roger Riggs wrote: > Hi, > > Thanks for the writeup of jigsaw modules. > > This is more of a comment on the module compilation unit grammar > and the choice of keywords and semantics. > http://openjdk.java.net/projects/jigsaw/doc/topics/grammar.html > > The "class" keyword in the module-info.java sticks out as an awkward usage. > There are plenty of classes in the module. The significance the class > is that it is a/the main class. The keyword should identify the usage > of the class. > "main" would be more meaningful and intuitive. > > The "class" keyword is being overloaded. While it identifies a main > class it is > also being used to indicate module is the root of the static context for > the module as a whole and the class name is not relevant to that function. > > A separate keyword defining the module as requiring a static context > would clearer to separate the semantics of the entry point from the > "classloader" > behavior. > > Can a module, like a jar file could contain multiple classes with main > methods? > In current usage, for jar files, the main class can be given on the > command line > allowing multiple applications to share the overhead of packaging. > Having a solitary entry point may result in extra overhead because each > application > must be in a separate module. > > Roger From sean.mullan at oracle.com Tue Jun 28 12:12:51 2011 From: sean.mullan at oracle.com (Sean Mullan) Date: Tue, 28 Jun 2011 15:12:51 -0400 Subject: Code Review request for new "jsign" tool In-Reply-To: <4DCAFFE1.3030700@oracle.com> References: <4DC9719B.6080703@oracle.com> <4DCAFFE1.3030700@oracle.com> Message-ID: <4E0A27B3.8000702@oracle.com> Mandy, I have updated the webrev for the new jsign tool: http://cr.openjdk.java.net/~mullan/jigsaw/webrevs/jsign/webrev.01/ This should address most of your comments (see below). Please let me know if it is ok to push this. On 5/11/11 5:30 PM, Mandy Chung wrote: > On 05/10/11 10:10, Sean Mullan wrote: >> -f, --signedmodulefile File name of signed module file > > This is a question more to the CLI guideline/convention our tools should follow. > > Should an option name with multiple words uses '-' to separate each word? > "--signed-module-file" vs "--signedmodulefile"? > > --output (-o) is the typical option name for specifying the output from a CLI. > Should we use --output for the jsign command? I like the explicit > --signed-module-file option name. It might be good to have a consistent > convention for all of our new tools to follow. For now, I have left the options as they are, but I agree we need to define a more consistent option naming syntax across all of the new jigsaw tools. >> webrev: http://cr.openjdk.java.net/~mullan/jigsaw/webrevs/jsign/webrev.00/ > > Signer.java > L255: what if the input module file is a signed module file? Should it > output an error? Yes, this is fixed. See lines 230-231. > L247, 248: constant 32 and 12 - is it worth defining these constants > in the ModuleFileFormat class? Fixed. There are new constants defined in ModuleFileFormat. > ModuleFileFormat.java > L630-637: I believe if noExtract is true, destination will be null; otherwise, > destination is non-null. Looks like you don't need the noExtract flag and > perhaps replaced with needExtract method? Just a thought. I have not done this. Since there are several public methods that are affected by this flag, I think it is better (and the code is easier to understand) to use a flag to denote this behavior. > L642-643: This assumes the 'sections' field doesn't count the signature > section. I assume you will take out the commented line before you push > the push. In the new webrev, the sections field has been removed from the Module File. It isn't strictly necessary, and avoids the signing issue that I described below. Mark will eventually be sending out an updated Module File Format specification with this change. > > Other than that, looks good. Thanks. I also fixed several unit test failures that were failing on Windows due to the usage of FileChannel.map [1]. I replaced these with FileChannel.read (and non-direct ByteBuffers). I did not see any noticeable performance degradation, but we will probably need to do some tests with larger module files to see if it makes any difference. Also, these calls are only made when packaging the module files, and not while reading. --Sean [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2011-June/001366.html > >> >> I encountered one issue with the module-file format [2] which should be >> addressed. Ideally, when a signature is generated over an existing module >> file, none of the contents of that module file should be modified. However, >> there is one field (the sections field in the module file header) that breaks >> that rule, because the signature itself is a section, and therefore the number >> of sections needs to be incremented by one. It may be possible to do that, but >> it would result in the code being much more complex. Thus, I would like to >> propose that this field be changed to be the number of sections following the >> module-info section (or the signature section if included), i.e. the number of >> sections in the "rest" of the module. This would not affect the Reader >> implementation, as it only uses this field to determine how many sections it >> needs to read in the rest of the module. >> > > Otherwise, you would need to rehash, right? This sounds a good suggestion to me. > > Mandy From david.bosschaert at gmail.com Tue Jun 28 14:24:19 2011 From: david.bosschaert at gmail.com (David Bosschaert) Date: Tue, 28 Jun 2011 22:24:19 +0100 Subject: module-info.class spec and attributes In-Reply-To: <4E053859.3060703@oracle.com> References: <4E00E72A.8020104@oracle.com> <4E052910.5080001@oracle.com> <4E05335F.6090609@oracle.com> <4E053859.3060703@oracle.com> Message-ID: I wonder why such a binary .class file for module declarations is really needed. Why not go for a textual file, for instance META-INF/module-info.jmod (or something like that) and declare the module declarations in there using a textual format of some sort? Like the other files in META-INF, it will become part of the ultimate jar as-is so compilation into a .class file is not needed nor are extensions to the .class file specification. IMO there are multiple benefits associated keeping the textual format. People using the jar can use a simple zip tool to look inside the jar to see what its dependencies are. Additionally tools can easily be written to read the module-info.jmod file without needing to resort to classfile parsing libraries. The module information can be of use to both developer as well as deployer tools. Finally it's probably easier to make the textual file extensible, as required by [1], without resorting to inefficient reflection-style operations. Best regards, David [1] http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#extensible-module-declarations On 25 June 2011 02:22, Jonathan Gibbons wrote: > Minor not-cut-n-paste typo: > >> The ModuleData attribute >> >> ModuleExtension_attribute { >> ?u2 attribute_name_index; >> ?u4 attribute_length; >> ?u2 data_index; >> } > > -- Jon > > > > On 06/24/2011 06:01 PM, Alex Buckley wrote: >> >> Since attachments seem to be scrubbed, here is the text of the proposal, >> obviously without links to the requirements document. >> >> >> ClassFile changes for Java Modules >> >> Binary form of a module declaration (a.k.a. module-info.class) >> A compilation unit that contains a module declaration (and potentially >> additional arbitrary content) is compiled to a ClassFile structure like any >> other compilation unit. >> >> By convention, the name of a compilation unit that contains a module >> declaration is module-info.java, echoing the package-info.java convention >> for a compilation unit that contains solely a package declaration. >> Consequently, by convention, the name for the compiled form of a module >> declaration is module-info.class. >> >> A new flag in the ClassFile.access_flags item, ACC_MODULE (0x0800), >> indicates that the ClassFile represents a module rather than a class or >> interface. Note that this flag plays a similar role to ACC_INTERFACE >> (0x0200), and does not describe accessibility of a class or interface. >> >> If ACC_MODULE is set in ClassFile.access_flags, then no other flag in >> ClassFile.access_flags may be set, and the following rules apply to the rest >> of the ClassFile structure: >> >> ? ?* major_version, minor_version: >=52.0 (Java SE 8 and above) >> ? ?* this_class: [Module's name in internal form (JVMS 4.2.1)]/module-info >> >> ? ? ?(Traditionally, if this_class indicates "P/Q/R", then the ClassFile >> can be expected to live in a file R.class in a directory P/Q representing a >> package. This explains why "/module-info" is a suffix in this_class above: >> if this_class indicates "P/Q/module-info", then the ClassFile can be >> expected to live in a file module-info.class in a directory P/Q representing >> a module. The "real name" of the module, shorn of "/module-info", can be >> obtained from the Module attribute.) >> ? ?* super_class, interfaces_count, fields_count, methods_count: 0 >> ? ?* attributes: One Module attribute must be present, to record the name >> and version of the module. At most one of each of the ModuleRequires, >> ModuleProvides, ModulePermits, ModuleExports, ModuleClass, and ModuleData >> attributes may be present. Except for these attributes and Synthetic, >> SourceFile, SourceDebugExtension, and Deprecated, none of the pre-defined >> attributes in JVMS 4.7 may appear. >> >> The CONSTANT_ModuleId_info structure >> >> The CONSTANT_ModuleId_info structure in the constant pool is used to >> represent a pair of a module name and a module version. >> >> CONSTANT_ModuleId_info { >> ?u1 tag; >> ? ?// 19 >> ?u2 name_index; >> ? ?// Points to a CONSTANT_Utf8_info representing module name in internal >> form (JVMS 4.2.1) >> ?u2 version_index; >> ? ?// Optionally points to a CONSTANT_Utf8_info representing module >> version >> } >> >> If the version_index item is 0 (as opposed to pointing to a >> CONSTANT_Utf8_info structure which holds the string "0"), then no module >> version is present. >> >> If the version_index item is not 0, then a module version is present. >> There is no required structure or semantics for the underlying UTF8 string >> which represents a module version. >> >> The Module attribute >> >> Module_attribute { >> ?u2 attribute_name_index; >> ?u4 attribute_length; >> ?u2 module_id_index; >> } >> >> The items of the Module_attribute structure are as follows: >> >> attribute_name_index >> ? ?The value of the attribute_name_index item must be a valid index into >> the constant_pool table. The constant_pool entry at that index must be a >> CONSTANT_Utf8_info structure representing the string "Module". >> attribute_length >> ? ?The value of the attribute_length item must be 2. >> module_id_index >> ? ?The value of the module_id_index item must be a valid index into the >> constant_pool table. The constant_pool entry at that index must be a >> CONSTANT_ModuleId_info structure representing the name and version of the >> module represented by this ClassFile. >> ? ?The name must be equal to the module name indicated by the >> ClassFile.this_class item without the "/module-info" suffix. >> >> The ModuleRequires attribute >> >> ModuleRequires_attribute { >> ?u2 attribute_name_index; >> ?u4 attribute_length; >> ?u2 requires_length; >> ?{ u2 requires_index; u1 flags; } requires_table[requires_length]; >> } >> >> attribute_name_index >> ? ?The value of the attribute_name_index item must be a valid index into >> the constant_pool table. The constant_pool entry at that index must be a >> CONSTANT_Utf8_info structure representing the string "ModuleRequires". >> attribute_length >> ? ?The value of the attribute_length item is the length of the attribute >> excluding the initial six bytes. >> requires_length >> ? ?The value of the requires_length indicates the number of entries in the >> requires_table. >> requires_table >> ? ?Each requires_index must be a valid index into the constant_pool table. >> The constant_pool entry at that index must be a CONSTANT_ModuleId_info >> structure representing the target module on which this module depends. >> ? ?The value of the associated flags item is as follows: >> >> ? ? ? ?* 0x01 indicates this dependency on the target module is optional. >> ? ? ? ?* 0x02 indictates the target module's types must be loaded by the >> same defining classloader as the types of the module represented by this >> ClassFile. >> >> ? ?A module name may be referenced by at most one entry in a >> requires_table. >> >> The ModulePermits attribute >> >> ModulePermits_attribute { >> ?u2 attribute_name_index; >> ?u4 attribute_length; >> ?u2 permits_length; >> ?{ u2 permits_index } permits_table[permits_length]; >> } >> >> attribute_name_index >> ? ?The value of the attribute_name_index item must be a valid index into >> the constant_pool table. The constant_pool entry at that index must be a >> CONSTANT_Utf8_info structure representing the string "ModulePermits". >> attribute_length >> ? ?The value of the attribute_length item is the length of the attribute >> excluding the initial six bytes. >> permits_length >> ? ?The value of the permits_length indicates the number of entries in the >> permits_table. >> permits_table >> ? ?The value of each permits_index in this item must be a valid index into >> the constant_pool table. The constant_pool entry at that index must be a >> CONSTANT_ModuleId_info structure representing a module which is permitted to >> have a dependency on the module represented by this ClassFile. >> ? ?(It is possible that a module system uses only the name of the >> permitted module, not its version, in determining visibility.) >> ? ?A module name may be referenced by at most one entry in a >> permits_table. >> >> The ModuleProvides attribute >> >> ModuleProvides_attribute { >> ?u2 attribute_name_index; >> ?u4 attribute_length; >> ?u2 provides_length; >> ?{ u2 provides_index; } provides_table[provides_length]; >> } >> >> attribute_name_index >> ? ?The value of the attribute_name_index item must be a valid index into >> the constant_pool table. The constant_pool entry at that index must be a >> CONSTANT_Utf8_info structure representing the string "ModuleProvides". >> attribute_length >> ? ?The value of the attribute_length item is the length of the attribute >> excluding the initial six bytes. >> provides_length >> ? ?The value of the provides_length indicates the number of entries in the >> provides_table. >> provides_table >> ? ?The value of each provides_index in this item must be a valid index >> into the constant_pool table. The constant_pool entry at that index must be >> a CONSTANT_ModuleId_info structure representing a module that is an alias >> for the module represented by this ClassFile. >> ? ?A module name may be referenced by at most one entry in a >> provides_table. >> >> The ModuleExports attribute >> >> ModuleExports_attribute { >> ?u2 attribute_name_index; >> ?u4 attribute_length; >> ?u2 exports_length; >> ?{ u2 exports_index; u1 export_kind; u2 source_index; } >> exports_table[exports_length]; >> } >> >> attribute_name_index >> ? ?The value of the attribute_name_index item must be a valid index into >> the constant_pool table. The constant_pool entry at that index must be a >> CONSTANT_Utf8_info structure representing the string "ModuleExports". >> attribute_length >> ? ?The value of the attribute_length item is the length of the attribute >> excluding the initial six bytes. >> exports_length >> ? ?The value of the exports_length indicates the number of entries in the >> exports_table. >> exports_table >> ? ?The value of each export_index in this item must be a valid index into >> the constant_pool table. The constant_pool entry at that index must be a >> CONSTANT_Utf8_info structure representing an entity to be exported by the >> module represented by this ClassFile. >> ? ?The value of the associated export_kind item indicates the kind of >> entity to be exported, as follows: >> >> ? ? ? ?* 0x01 indicates the entity is a class or interface; it is the >> entity to be exported. >> ? ? ? ?* 0x02 indicates the entity is a class or interface; it and its >> member types are the entities to be exported. >> ? ? ? ?* 0x04 indicates the entity is a package; the member top-level >> classes and interfaces (and their member types) of this package are the >> entities to be exported. >> ? ? ? ?* 0x08 indicates the entity is a package; the member top-level >> classes and interfaces (and their member types) of this package and all its >> subpackages (recursively) are the entities to be exported. >> >> ? ?The value of the associated source_index item must be a valid index >> into the constant_pool table where the entry is a CONSTANT_ModuleId_info >> structure representing the module which declared the exported entity. (In >> most cases, where the exported entity is declared in the current module, >> source_index will simply point to the same constant pool entry as the Module >> attribute. But where the exported entity is re-exported, source_index may >> assist resolution and diagnostics.) >> >> The ModuleClass attribute >> >> ModuleClass_attribute { >> ?u2 attribute_name_index; >> ?u4 attribute_length; >> ?u2 main_class_index; >> } >> >> attribute_name_index >> ? ?The value of the attribute_name_index item must be a valid index into >> the constant_pool table. The constant_pool entry at that index must be a >> CONSTANT_Utf8_info structure representing the string "ModuleClass". >> attribute_length >> ? ?The value of the attribute_length item is 2. >> main_class_index >> ? ?The value of the main_class_index item must be a valid index into the >> constant_pool table. The constant_pool entry at that index must be a >> CONSTANT_Class_info structure representing the name of the class which is >> the entrypoint to the module represented by this ClassFile. >> >> The ModuleData attribute >> >> ModuleExtension_attribute { >> ?u2 attribute_name_index; >> ?u4 attribute_length; >> ?u2 data_index; >> } >> >> attribute_name_index >> ? ?The value of the attribute_name_index item must be a valid index into >> the constant_pool table. The constant_pool entry at that index must be a >> CONSTANT_Utf8_info structure representing the string "ModuleData". >> attribute_length >> ? ?The value of the attribute_length item is 2. >> data_index >> ? ?The value of the main_class item must be a valid index into the >> constant_pool table. The constant_pool entry at that index must be a >> CONSTANT_Utf8_info structure representing the content, other than the module >> declaration, of the compilation unit that declared the module represented by >> this ClassFile. > > From mandy.chung at oracle.com Tue Jun 28 18:07:42 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 29 Jun 2011 09:07:42 +0800 Subject: Multiple entry points requirement In-Reply-To: <4E0A1913.4020802@oracle.com> References: <4E092DA6.4010606@oracle.com> <4E0A1256.4010608@oracle.com> <4E0A1913.4020802@oracle.com> Message-ID: <4E0A7ADE.5080705@oracle.com> I think the multiple entry points should be added to the requirement. There are several tools in the JDK that would be best if they can be packaged in a single module with multiple entry points. For example, jps, jstat, jstack, jmap, etc are the JDK command-line tools that are in tools.jar and each has its own entry point. Each of the tool has only one or few classes and they are diagnosability tools that make sense to be packaged and installed together. The JDK would benefit from the multiple entry points to avoid unnecessary fine-grained modules. Mandy On 6/29/11 2:10 AM, Alex Buckley wrote: > I recall discussion of different entrypoints when a module is used in > different contexts, e.g. applet v. command-line launcher. > > However, the Java Module System requirements don't say anything about > this, so I stuck with a solitary, unqualified 'class ...' construct in > the module declaration. The closest "context-aware" requirement is > that a module "can be used both as a JAR on the class path and > installed as a module in a module library". > > Alex > > On 6/28/2011 10:41 AM, Roger Riggs wrote: >> Hi, >> >> Thanks for the writeup of jigsaw modules. >> >> This is more of a comment on the module compilation unit grammar >> and the choice of keywords and semantics. >> http://openjdk.java.net/projects/jigsaw/doc/topics/grammar.html >> >> The "class" keyword in the module-info.java sticks out as an awkward >> usage. >> There are plenty of classes in the module. The significance the class >> is that it is a/the main class. The keyword should identify the >> usage of the class. >> "main" would be more meaningful and intuitive. >> >> The "class" keyword is being overloaded. While it identifies a main >> class it is >> also being used to indicate module is the root of the static context for >> the module as a whole and the class name is not relevant to that >> function. >> >> A separate keyword defining the module as requiring a static context >> would clearer to separate the semantics of the entry point from the >> "classloader" >> behavior. >> >> Can a module, like a jar file could contain multiple classes with >> main methods? >> In current usage, for jar files, the main class can be given on the >> command line >> allowing multiple applications to share the overhead of packaging. >> Having a solitary entry point may result in extra overhead because >> each application >> must be in a separate module. >> >> Roger From david.lloyd at redhat.com Tue Jun 28 20:43:15 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 28 Jun 2011 22:43:15 -0500 Subject: module-info.class spec and attributes In-Reply-To: References: <4E00E72A.8020104@oracle.com> <4E052910.5080001@oracle.com> <4E05335F.6090609@oracle.com> <4E053859.3060703@oracle.com> Message-ID: <4E0A9F53.3070407@redhat.com> Yeah I agree, following the mold of classes and package-info is a big mistake here. Not sure if it's simple cargo-cultism or what. In JBoss Modules we actually use XML, and performance is very good. If you used a dedicated text format I'm sure that performance would be quite good enough at runtime. On 06/28/2011 04:24 PM, David Bosschaert wrote: > I wonder why such a binary .class file for module declarations is > really needed. Why not go for a textual file, for instance > META-INF/module-info.jmod (or something like that) and declare the > module declarations in there using a textual format of some sort? > > Like the other files in META-INF, it will become part of the ultimate > jar as-is so compilation into a .class file is not needed nor are > extensions to the .class file specification. > > IMO there are multiple benefits associated keeping the textual format. > People using the jar can use a simple zip tool to look inside the jar > to see what its dependencies are. Additionally tools can easily be > written to read the module-info.jmod file without needing to resort to > classfile parsing libraries. The module information can be of use to > both developer as well as deployer tools. > > Finally it's probably easier to make the textual file extensible, as > required by [1], without resorting to inefficient reflection-style > operations. > > Best regards, > > David > > [1] http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#extensible-module-declarations > > On 25 June 2011 02:22, Jonathan Gibbons wrote: >> Minor not-cut-n-paste typo: >> >>> The ModuleData attribute >>> >>> ModuleExtension_attribute { >>> u2 attribute_name_index; >>> u4 attribute_length; >>> u2 data_index; >>> } >> >> -- Jon >> >> >> >> On 06/24/2011 06:01 PM, Alex Buckley wrote: >>> >>> Since attachments seem to be scrubbed, here is the text of the proposal, >>> obviously without links to the requirements document. >>> >>> >>> ClassFile changes for Java Modules >>> >>> Binary form of a module declaration (a.k.a. module-info.class) >>> A compilation unit that contains a module declaration (and potentially >>> additional arbitrary content) is compiled to a ClassFile structure like any >>> other compilation unit. >>> >>> By convention, the name of a compilation unit that contains a module >>> declaration is module-info.java, echoing the package-info.java convention >>> for a compilation unit that contains solely a package declaration. >>> Consequently, by convention, the name for the compiled form of a module >>> declaration is module-info.class. >>> >>> A new flag in the ClassFile.access_flags item, ACC_MODULE (0x0800), >>> indicates that the ClassFile represents a module rather than a class or >>> interface. Note that this flag plays a similar role to ACC_INTERFACE >>> (0x0200), and does not describe accessibility of a class or interface. >>> >>> If ACC_MODULE is set in ClassFile.access_flags, then no other flag in >>> ClassFile.access_flags may be set, and the following rules apply to the rest >>> of the ClassFile structure: >>> >>> * major_version, minor_version:>=52.0 (Java SE 8 and above) >>> * this_class: [Module's name in internal form (JVMS 4.2.1)]/module-info >>> >>> (Traditionally, if this_class indicates "P/Q/R", then the ClassFile >>> can be expected to live in a file R.class in a directory P/Q representing a >>> package. This explains why "/module-info" is a suffix in this_class above: >>> if this_class indicates "P/Q/module-info", then the ClassFile can be >>> expected to live in a file module-info.class in a directory P/Q representing >>> a module. The "real name" of the module, shorn of "/module-info", can be >>> obtained from the Module attribute.) >>> * super_class, interfaces_count, fields_count, methods_count: 0 >>> * attributes: One Module attribute must be present, to record the name >>> and version of the module. At most one of each of the ModuleRequires, >>> ModuleProvides, ModulePermits, ModuleExports, ModuleClass, and ModuleData >>> attributes may be present. Except for these attributes and Synthetic, >>> SourceFile, SourceDebugExtension, and Deprecated, none of the pre-defined >>> attributes in JVMS 4.7 may appear. >>> >>> The CONSTANT_ModuleId_info structure >>> >>> The CONSTANT_ModuleId_info structure in the constant pool is used to >>> represent a pair of a module name and a module version. >>> >>> CONSTANT_ModuleId_info { >>> u1 tag; >>> // 19 >>> u2 name_index; >>> // Points to a CONSTANT_Utf8_info representing module name in internal >>> form (JVMS 4.2.1) >>> u2 version_index; >>> // Optionally points to a CONSTANT_Utf8_info representing module >>> version >>> } >>> >>> If the version_index item is 0 (as opposed to pointing to a >>> CONSTANT_Utf8_info structure which holds the string "0"), then no module >>> version is present. >>> >>> If the version_index item is not 0, then a module version is present. >>> There is no required structure or semantics for the underlying UTF8 string >>> which represents a module version. >>> >>> The Module attribute >>> >>> Module_attribute { >>> u2 attribute_name_index; >>> u4 attribute_length; >>> u2 module_id_index; >>> } >>> >>> The items of the Module_attribute structure are as follows: >>> >>> attribute_name_index >>> The value of the attribute_name_index item must be a valid index into >>> the constant_pool table. The constant_pool entry at that index must be a >>> CONSTANT_Utf8_info structure representing the string "Module". >>> attribute_length >>> The value of the attribute_length item must be 2. >>> module_id_index >>> The value of the module_id_index item must be a valid index into the >>> constant_pool table. The constant_pool entry at that index must be a >>> CONSTANT_ModuleId_info structure representing the name and version of the >>> module represented by this ClassFile. >>> The name must be equal to the module name indicated by the >>> ClassFile.this_class item without the "/module-info" suffix. >>> >>> The ModuleRequires attribute >>> >>> ModuleRequires_attribute { >>> u2 attribute_name_index; >>> u4 attribute_length; >>> u2 requires_length; >>> { u2 requires_index; u1 flags; } requires_table[requires_length]; >>> } >>> >>> attribute_name_index >>> The value of the attribute_name_index item must be a valid index into >>> the constant_pool table. The constant_pool entry at that index must be a >>> CONSTANT_Utf8_info structure representing the string "ModuleRequires". >>> attribute_length >>> The value of the attribute_length item is the length of the attribute >>> excluding the initial six bytes. >>> requires_length >>> The value of the requires_length indicates the number of entries in the >>> requires_table. >>> requires_table >>> Each requires_index must be a valid index into the constant_pool table. >>> The constant_pool entry at that index must be a CONSTANT_ModuleId_info >>> structure representing the target module on which this module depends. >>> The value of the associated flags item is as follows: >>> >>> * 0x01 indicates this dependency on the target module is optional. >>> * 0x02 indictates the target module's types must be loaded by the >>> same defining classloader as the types of the module represented by this >>> ClassFile. >>> >>> A module name may be referenced by at most one entry in a >>> requires_table. >>> >>> The ModulePermits attribute >>> >>> ModulePermits_attribute { >>> u2 attribute_name_index; >>> u4 attribute_length; >>> u2 permits_length; >>> { u2 permits_index } permits_table[permits_length]; >>> } >>> >>> attribute_name_index >>> The value of the attribute_name_index item must be a valid index into >>> the constant_pool table. The constant_pool entry at that index must be a >>> CONSTANT_Utf8_info structure representing the string "ModulePermits". >>> attribute_length >>> The value of the attribute_length item is the length of the attribute >>> excluding the initial six bytes. >>> permits_length >>> The value of the permits_length indicates the number of entries in the >>> permits_table. >>> permits_table >>> The value of each permits_index in this item must be a valid index into >>> the constant_pool table. The constant_pool entry at that index must be a >>> CONSTANT_ModuleId_info structure representing a module which is permitted to >>> have a dependency on the module represented by this ClassFile. >>> (It is possible that a module system uses only the name of the >>> permitted module, not its version, in determining visibility.) >>> A module name may be referenced by at most one entry in a >>> permits_table. >>> >>> The ModuleProvides attribute >>> >>> ModuleProvides_attribute { >>> u2 attribute_name_index; >>> u4 attribute_length; >>> u2 provides_length; >>> { u2 provides_index; } provides_table[provides_length]; >>> } >>> >>> attribute_name_index >>> The value of the attribute_name_index item must be a valid index into >>> the constant_pool table. The constant_pool entry at that index must be a >>> CONSTANT_Utf8_info structure representing the string "ModuleProvides". >>> attribute_length >>> The value of the attribute_length item is the length of the attribute >>> excluding the initial six bytes. >>> provides_length >>> The value of the provides_length indicates the number of entries in the >>> provides_table. >>> provides_table >>> The value of each provides_index in this item must be a valid index >>> into the constant_pool table. The constant_pool entry at that index must be >>> a CONSTANT_ModuleId_info structure representing a module that is an alias >>> for the module represented by this ClassFile. >>> A module name may be referenced by at most one entry in a >>> provides_table. >>> >>> The ModuleExports attribute >>> >>> ModuleExports_attribute { >>> u2 attribute_name_index; >>> u4 attribute_length; >>> u2 exports_length; >>> { u2 exports_index; u1 export_kind; u2 source_index; } >>> exports_table[exports_length]; >>> } >>> >>> attribute_name_index >>> The value of the attribute_name_index item must be a valid index into >>> the constant_pool table. The constant_pool entry at that index must be a >>> CONSTANT_Utf8_info structure representing the string "ModuleExports". >>> attribute_length >>> The value of the attribute_length item is the length of the attribute >>> excluding the initial six bytes. >>> exports_length >>> The value of the exports_length indicates the number of entries in the >>> exports_table. >>> exports_table >>> The value of each export_index in this item must be a valid index into >>> the constant_pool table. The constant_pool entry at that index must be a >>> CONSTANT_Utf8_info structure representing an entity to be exported by the >>> module represented by this ClassFile. >>> The value of the associated export_kind item indicates the kind of >>> entity to be exported, as follows: >>> >>> * 0x01 indicates the entity is a class or interface; it is the >>> entity to be exported. >>> * 0x02 indicates the entity is a class or interface; it and its >>> member types are the entities to be exported. >>> * 0x04 indicates the entity is a package; the member top-level >>> classes and interfaces (and their member types) of this package are the >>> entities to be exported. >>> * 0x08 indicates the entity is a package; the member top-level >>> classes and interfaces (and their member types) of this package and all its >>> subpackages (recursively) are the entities to be exported. >>> >>> The value of the associated source_index item must be a valid index >>> into the constant_pool table where the entry is a CONSTANT_ModuleId_info >>> structure representing the module which declared the exported entity. (In >>> most cases, where the exported entity is declared in the current module, >>> source_index will simply point to the same constant pool entry as the Module >>> attribute. But where the exported entity is re-exported, source_index may >>> assist resolution and diagnostics.) >>> >>> The ModuleClass attribute >>> >>> ModuleClass_attribute { >>> u2 attribute_name_index; >>> u4 attribute_length; >>> u2 main_class_index; >>> } >>> >>> attribute_name_index >>> The value of the attribute_name_index item must be a valid index into >>> the constant_pool table. The constant_pool entry at that index must be a >>> CONSTANT_Utf8_info structure representing the string "ModuleClass". >>> attribute_length >>> The value of the attribute_length item is 2. >>> main_class_index >>> The value of the main_class_index item must be a valid index into the >>> constant_pool table. The constant_pool entry at that index must be a >>> CONSTANT_Class_info structure representing the name of the class which is >>> the entrypoint to the module represented by this ClassFile. >>> >>> The ModuleData attribute >>> >>> ModuleExtension_attribute { >>> u2 attribute_name_index; >>> u4 attribute_length; >>> u2 data_index; >>> } >>> >>> attribute_name_index >>> The value of the attribute_name_index item must be a valid index into >>> the constant_pool table. The constant_pool entry at that index must be a >>> CONSTANT_Utf8_info structure representing the string "ModuleData". >>> attribute_length >>> The value of the attribute_length item is 2. >>> data_index >>> The value of the main_class item must be a valid index into the >>> constant_pool table. The constant_pool entry at that index must be a >>> CONSTANT_Utf8_info structure representing the content, other than the module >>> declaration, of the compilation unit that declared the module represented by >>> this ClassFile. >> >> -- - DML From david.lloyd at redhat.com Wed Jun 29 16:41:36 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 29 Jun 2011 18:41:36 -0500 Subject: Multiple entry points requirement In-Reply-To: <4E0A7ADE.5080705@oracle.com> References: <4E092DA6.4010606@oracle.com> <4E0A1256.4010608@oracle.com> <4E0A1913.4020802@oracle.com> <4E0A7ADE.5080705@oracle.com> Message-ID: <4E0BB830.2070805@redhat.com> On 06/28/2011 08:07 PM, Mandy Chung wrote: > I think the multiple entry points should be added to the requirement. > > There are several tools in the JDK that would be best if they can be > packaged in a single module with multiple entry points. For example, > jps, jstat, jstack, jmap, etc are the JDK command-line tools that are in > tools.jar and each has its own entry point. Each of the tool has only > one or few classes and they are diagnosability tools that make sense to > be packaged and installed together. The JDK would benefit from the > multiple entry points to avoid unnecessary fine-grained modules. > > Mandy > > On 6/29/11 2:10 AM, Alex Buckley wrote: >> I recall discussion of different entrypoints when a module is used in >> different contexts, e.g. applet v. command-line launcher. >> >> However, the Java Module System requirements don't say anything about >> this, so I stuck with a solitary, unqualified 'class ...' construct in >> the module declaration. The closest "context-aware" requirement is >> that a module "can be used both as a JAR on the class path and >> installed as a module in a module library". >> >> Alex >> >> On 6/28/2011 10:41 AM, Roger Riggs wrote: >>> Hi, >>> >>> Thanks for the writeup of jigsaw modules. >>> >>> This is more of a comment on the module compilation unit grammar >>> and the choice of keywords and semantics. >>> http://openjdk.java.net/projects/jigsaw/doc/topics/grammar.html >>> >>> The "class" keyword in the module-info.java sticks out as an awkward >>> usage. >>> There are plenty of classes in the module. The significance the class >>> is that it is a/the main class. The keyword should identify the usage >>> of the class. >>> "main" would be more meaningful and intuitive. >>> >>> The "class" keyword is being overloaded. While it identifies a main >>> class it is >>> also being used to indicate module is the root of the static context for >>> the module as a whole and the class name is not relevant to that >>> function. >>> >>> A separate keyword defining the module as requiring a static context >>> would clearer to separate the semantics of the entry point from the >>> "classloader" >>> behavior. >>> >>> Can a module, like a jar file could contain multiple classes with >>> main methods? >>> In current usage, for jar files, the main class can be given on the >>> command line >>> allowing multiple applications to share the overhead of packaging. >>> Having a solitary entry point may result in extra overhead because >>> each application >>> must be in a separate module. >>> >>> Roger > -- - DML From david.lloyd at redhat.com Wed Jun 29 16:42:36 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 29 Jun 2011 18:42:36 -0500 Subject: Multiple entry points requirement In-Reply-To: <4E0A7ADE.5080705@oracle.com> References: <4E092DA6.4010606@oracle.com> <4E0A1256.4010608@oracle.com> <4E0A1913.4020802@oracle.com> <4E0A7ADE.5080705@oracle.com> Message-ID: <4E0BB86C.7040900@redhat.com> Sorry for the mis-send. I was just going to suggest that alternatively to adding complexity to an already complex system, the wrapper script/program could simply pass in an argument to determine which program to run. On 06/28/2011 08:07 PM, Mandy Chung wrote: > I think the multiple entry points should be added to the requirement. > > There are several tools in the JDK that would be best if they can be > packaged in a single module with multiple entry points. For example, > jps, jstat, jstack, jmap, etc are the JDK command-line tools that are in > tools.jar and each has its own entry point. Each of the tool has only > one or few classes and they are diagnosability tools that make sense to > be packaged and installed together. The JDK would benefit from the > multiple entry points to avoid unnecessary fine-grained modules. > > Mandy > > On 6/29/11 2:10 AM, Alex Buckley wrote: >> I recall discussion of different entrypoints when a module is used in >> different contexts, e.g. applet v. command-line launcher. >> >> However, the Java Module System requirements don't say anything about >> this, so I stuck with a solitary, unqualified 'class ...' construct in >> the module declaration. The closest "context-aware" requirement is >> that a module "can be used both as a JAR on the class path and >> installed as a module in a module library". >> >> Alex >> >> On 6/28/2011 10:41 AM, Roger Riggs wrote: >>> Hi, >>> >>> Thanks for the writeup of jigsaw modules. >>> >>> This is more of a comment on the module compilation unit grammar >>> and the choice of keywords and semantics. >>> http://openjdk.java.net/projects/jigsaw/doc/topics/grammar.html >>> >>> The "class" keyword in the module-info.java sticks out as an awkward >>> usage. >>> There are plenty of classes in the module. The significance the class >>> is that it is a/the main class. The keyword should identify the usage >>> of the class. >>> "main" would be more meaningful and intuitive. >>> >>> The "class" keyword is being overloaded. While it identifies a main >>> class it is >>> also being used to indicate module is the root of the static context for >>> the module as a whole and the class name is not relevant to that >>> function. >>> >>> A separate keyword defining the module as requiring a static context >>> would clearer to separate the semantics of the entry point from the >>> "classloader" >>> behavior. >>> >>> Can a module, like a jar file could contain multiple classes with >>> main methods? >>> In current usage, for jar files, the main class can be given on the >>> command line >>> allowing multiple applications to share the overhead of packaging. >>> Having a solitary entry point may result in extra overhead because >>> each application >>> must be in a separate module. >>> >>> Roger > -- - DML From alex.buckley at oracle.com Wed Jun 29 17:07:07 2011 From: alex.buckley at oracle.com (Alex Buckley) Date: Wed, 29 Jun 2011 17:07:07 -0700 Subject: module-info.class spec and attributes In-Reply-To: References: <4E00E72A.8020104@oracle.com> <4E052910.5080001@oracle.com> <4E05335F.6090609@oracle.com> <4E053859.3060703@oracle.com> Message-ID: <4E0BBE2B.9090907@oracle.com> (Sent this mail yesterday but I think it got lost.) On 6/28/2011 2:24 PM, David Bosschaert wrote: > I wonder why such a binary .class file for module declarations is > really needed. Why not go for a textual file, for instance > META-INF/module-info.jmod (or something like that) and declare the > module declarations in there using a textual format of some sort? The arguments for a binary compiled form are given at: http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#_C > IMO there are multiple benefits associated keeping the textual format. > People using the jar can use a simple zip tool to look inside the jar > to see what its dependencies are. Additionally tools can easily be > written to read the module-info.jmod file without needing to resort to > classfile parsing libraries. The module information can be of use to > both developer as well as deployer tools. > > Finally it's probably easier to make the textual file extensible, as > required by [1], without resorting to inefficient reflection-style > operations. I think these points are also captured in Appendix C. Could not the same points have been made in favor of a textual format for class and interface declarations? Alex From david.lloyd at redhat.com Wed Jun 29 17:24:17 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 29 Jun 2011 19:24:17 -0500 Subject: module-info.class spec and attributes In-Reply-To: <4E0BBE2B.9090907@oracle.com> References: <4E00E72A.8020104@oracle.com> <4E052910.5080001@oracle.com> <4E05335F.6090609@oracle.com> <4E053859.3060703@oracle.com> <4E0BBE2B.9090907@oracle.com> Message-ID: <4E0BC231.7000200@redhat.com> On 06/29/2011 07:07 PM, Alex Buckley wrote: > (Sent this mail yesterday but I think it got lost.) > > On 6/28/2011 2:24 PM, David Bosschaert wrote: >> I wonder why such a binary .class file for module declarations is >> really needed. Why not go for a textual file, for instance >> META-INF/module-info.jmod (or something like that) and declare the >> module declarations in there using a textual format of some sort? > > The arguments for a binary compiled form are given at: > > http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#_C All of which are trivially debunkable. Yes, it is *slightly* more efficient to read structured binary than a text format. It is an insignificant difference though. We actually parse ours as XML using StAX, which should be somewhat more expensive than a dedicated text parser, and it's so fast that I'm not even able to get a significant measurement of the time when booting up a 100+ module project. The performance argument is invalid. And constraining the module metadata to a Java-like syntax with a Java-like file name is just a dumb idea. There's no reason for it. Saying that requiring a tool to modify the metadata for a module is somehow a good thing is pretty off the wall too. I mean just crazy. Come down to Earth guys. >> IMO there are multiple benefits associated keeping the textual format. >> People using the jar can use a simple zip tool to look inside the jar >> to see what its dependencies are. Additionally tools can easily be >> written to read the module-info.jmod file without needing to resort to >> classfile parsing libraries. The module information can be of use to >> both developer as well as deployer tools. >> >> Finally it's probably easier to make the textual file extensible, as >> required by [1], without resorting to inefficient reflection-style >> operations. > > I think these points are also captured in Appendix C. Could not the same > points have been made in favor of a textual format for class and > interface declarations? Yup. But that ship has sailed. A binary format for module meta-information is going to be yet another annoying disaster. -- - DML From alex.buckley at oracle.com Wed Jun 29 17:46:28 2011 From: alex.buckley at oracle.com (Alex Buckley) Date: Wed, 29 Jun 2011 17:46:28 -0700 Subject: module-info.class spec and attributes In-Reply-To: <4E0BC231.7000200@redhat.com> References: <4E00E72A.8020104@oracle.com> <4E052910.5080001@oracle.com> <4E05335F.6090609@oracle.com> <4E053859.3060703@oracle.com> <4E0BBE2B.9090907@oracle.com> <4E0BC231.7000200@redhat.com> Message-ID: <4E0BC764.3030300@oracle.com> In the context of Project Jigsaw, javac parses module declarations every time a source file in a modular directory structure is compiled. That's a powerful driver for Java-like source syntax and a class file form. Projects which don't put module information so close to the compiler may well choose other forms. Alex On 6/29/2011 5:24 PM, David M. Lloyd wrote: > On 06/29/2011 07:07 PM, Alex Buckley wrote: >> (Sent this mail yesterday but I think it got lost.) >> >> On 6/28/2011 2:24 PM, David Bosschaert wrote: >>> I wonder why such a binary .class file for module declarations is >>> really needed. Why not go for a textual file, for instance >>> META-INF/module-info.jmod (or something like that) and declare the >>> module declarations in there using a textual format of some sort? >> >> The arguments for a binary compiled form are given at: >> >> http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#_C >> > > All of which are trivially debunkable. Yes, it is *slightly* more > efficient to read structured binary than a text format. It is an > insignificant difference though. We actually parse ours as XML using > StAX, which should be somewhat more expensive than a dedicated text > parser, and it's so fast that I'm not even able to get a significant > measurement of the time when booting up a 100+ module project. The > performance argument is invalid. > > And constraining the module metadata to a Java-like syntax with a > Java-like file name is just a dumb idea. There's no reason for it. > > Saying that requiring a tool to modify the metadata for a module is > somehow a good thing is pretty off the wall too. I mean just crazy. > > Come down to Earth guys. > >>> IMO there are multiple benefits associated keeping the textual format. >>> People using the jar can use a simple zip tool to look inside the jar >>> to see what its dependencies are. Additionally tools can easily be >>> written to read the module-info.jmod file without needing to resort to >>> classfile parsing libraries. The module information can be of use to >>> both developer as well as deployer tools. >>> >>> Finally it's probably easier to make the textual file extensible, as >>> required by [1], without resorting to inefficient reflection-style >>> operations. >> >> I think these points are also captured in Appendix C. Could not the same >> points have been made in favor of a textual format for class and >> interface declarations? > > Yup. But that ship has sailed. A binary format for module > meta-information is going to be yet another annoying disaster. From david.lloyd at redhat.com Wed Jun 29 18:00:14 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 29 Jun 2011 20:00:14 -0500 Subject: module-info.class spec and attributes In-Reply-To: <4E0BC764.3030300@oracle.com> References: <4E00E72A.8020104@oracle.com> <4E052910.5080001@oracle.com> <4E05335F.6090609@oracle.com> <4E053859.3060703@oracle.com> <4E0BBE2B.9090907@oracle.com> <4E0BC231.7000200@redhat.com> <4E0BC764.3030300@oracle.com> Message-ID: <4E0BCA9E.8020003@redhat.com> Not really. In fact that's basically using the (apparently poor) design choices of the compiler to justify this, which I find to be exceedingly weak. The compiler should never need to parse module information more than once per compilation, regardless of how many source files are compiled. If it's requiring that, fix the real problem, rather than using it as justification for crap decisions elsewhere. On 06/29/2011 07:46 PM, Alex Buckley wrote: > In the context of Project Jigsaw, javac parses module declarations every > time a source file in a modular directory structure is compiled. That's > a powerful driver for Java-like source syntax and a class file form. > > Projects which don't put module information so close to the compiler may > well choose other forms. > > Alex > > On 6/29/2011 5:24 PM, David M. Lloyd wrote: >> On 06/29/2011 07:07 PM, Alex Buckley wrote: >>> (Sent this mail yesterday but I think it got lost.) >>> >>> On 6/28/2011 2:24 PM, David Bosschaert wrote: >>>> I wonder why such a binary .class file for module declarations is >>>> really needed. Why not go for a textual file, for instance >>>> META-INF/module-info.jmod (or something like that) and declare the >>>> module declarations in there using a textual format of some sort? >>> >>> The arguments for a binary compiled form are given at: >>> >>> http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#_C >>> >> >> All of which are trivially debunkable. Yes, it is *slightly* more >> efficient to read structured binary than a text format. It is an >> insignificant difference though. We actually parse ours as XML using >> StAX, which should be somewhat more expensive than a dedicated text >> parser, and it's so fast that I'm not even able to get a significant >> measurement of the time when booting up a 100+ module project. The >> performance argument is invalid. >> >> And constraining the module metadata to a Java-like syntax with a >> Java-like file name is just a dumb idea. There's no reason for it. >> >> Saying that requiring a tool to modify the metadata for a module is >> somehow a good thing is pretty off the wall too. I mean just crazy. >> >> Come down to Earth guys. >> >>>> IMO there are multiple benefits associated keeping the textual format. >>>> People using the jar can use a simple zip tool to look inside the jar >>>> to see what its dependencies are. Additionally tools can easily be >>>> written to read the module-info.jmod file without needing to resort to >>>> classfile parsing libraries. The module information can be of use to >>>> both developer as well as deployer tools. >>>> >>>> Finally it's probably easier to make the textual file extensible, as >>>> required by [1], without resorting to inefficient reflection-style >>>> operations. >>> >>> I think these points are also captured in Appendix C. Could not the same >>> points have been made in favor of a textual format for class and >>> interface declarations? >> >> Yup. But that ship has sailed. A binary format for module >> meta-information is going to be yet another annoying disaster. -- - DML From alex.buckley at oracle.com Tue Jun 28 16:30:38 2011 From: alex.buckley at oracle.com (Alex Buckley) Date: Tue, 28 Jun 2011 16:30:38 -0700 Subject: module-info.class spec and attributes In-Reply-To: References: <4E00E72A.8020104@oracle.com> <4E052910.5080001@oracle.com> <4E05335F.6090609@oracle.com> <4E053859.3060703@oracle.com> Message-ID: <4E0A641E.4000801@oracle.com> On 6/28/2011 2:24 PM, David Bosschaert wrote: > I wonder why such a binary .class file for module declarations is > really needed. Why not go for a textual file, for instance > META-INF/module-info.jmod (or something like that) and declare the > module declarations in there using a textual format of some sort? The arguments for a binary compiled form are given at: http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#_C > IMO there are multiple benefits associated keeping the textual format. > People using the jar can use a simple zip tool to look inside the jar > to see what its dependencies are. Additionally tools can easily be > written to read the module-info.jmod file without needing to resort to > classfile parsing libraries. The module information can be of use to > both developer as well as deployer tools. > > Finally it's probably easier to make the textual file extensible, as > required by [1], without resorting to inefficient reflection-style > operations. I think these points are also captured in Appendix C. Could not the same points have been made in favor of a textual format for class and interface declarations? Alex From peter.kriens at aqute.biz Wed Jun 29 02:33:41 2011 From: peter.kriens at aqute.biz (Peter Kriens) Date: Wed, 29 Jun 2011 11:33:41 +0200 Subject: module-info.class spec and attributes Message-ID: I took a look at the proposal and I have a few comments. But first the picture (cardinalities are allowed by the structure but can be constrained in the text). There seems to be quite a bit of redundancy (and thus complexity) in this structure because it allows * relations between the flags (provide,require,permit,module) that designate a ModuleId_Info struct. I.e. the proposal allows me to require/permit/provide the same module multiple times. Even though this is not allowed with an extraneous constraint this seems inefficient design, why create the opportunity for errors? It seems that provide/require/permit/module could easily be a handled with an enumeration or bitset: { THIS_MODULE, PROVIDE, PERMIT, REQUIRE_OPTIONAL, REQUIRE_FRAGMENT} in the ModuleId_Info struct? The use of attributes to set a bit seems overkill and will require unnecessary constraints on the cardinality of these attributes, making processing this file in tools harder. Similar about the redundancy in the module name. The module name is both the this_class package name as well as defined in the ModuleId_Info struct? Interestingly the model allows package export (though unfortunately not import, one should always wonder why things are not symmetric, but I guess that is another discussion). However, class exports can be top levels only (I assume no nested classes?) and recursively. However, packages are top level classes in the designated package and the all subpackages. This seems to miss the cases for nested classes in the single package and nested classes in the sub-packages? I am a bit confused about the scope of the module as this is not defined. I.e. a class defines all its members, not just the public ones. Shouldn't a module also define its private members? Obviously the extension model is a bit of a joke, all we get is a list of unadorned strings per module? The minimum that is needed is a model that allows the exports, provides, permits, and module attributes to be annotated in an elegant way. For example, ModuleClass is clearly a Jigsaw extension and several of the bit fields are also highly semantic, such semantics should be extensible as it is very unlikely you get it right the first time. Why not model these with an extension model and eat your own dog food? The previous issues were detailed critiques on the design, intended to improve it within the box. However, I believe the premise of using attributes to model modules points to a serious problem with the box. Modularity is hierarchical, a function is modular, a type is modular, a package is modular, and this thing under construction is just the next granular step in modularity. If a module is modeled as a class (as this design implicitly does by using the class file) then I think it should not introduce a completely new spaghetti ontology based on attributes but instead look at what the class file currently does, as a class is also modular. A class consists of a name, class or interface specific information, imports of other types, members, and conceptually nested types. So what is the member of a module? This is clearly the package. And the member of a package is clearly a Type. As members use other members, types import other types symmetry demands that packages import other packages and modules import other modules (as is done in the proposal with ModuleRequire). Treating packages as members of a module and imports as references to other modules would make the class file a very good fit; every Java developer would immediately see the symmetry and have an intuitive understanding. Symmetry is one of the most important indicators that you've understood the problem and found the simplest model because it reuses existing concepts on different levels. For example, module extensions could now easily be modeled as standard annotations on members. It would also support module data/methods (static fields/methods in the module), as well as represent the Module Class implicitly. Falling back to custom class attributes invariably means a huge amount of exceptional cases to handle by the community and makes it surprisingly hard to understand what's going on ... As the current proposal has no affinity to a class file format whatsoever; it seems wrong to abuse the class file format just because it happens to support attributes. There are better formats for this purpose, formats that could also allow some human readability for debugging and easy annotation by third parties without requiring class format changes for every new module concept. Obviously XML is the most powerful and best suited format for this problem. The JLS module specification can just as easy output to XML as attributes in a class file but will be significantly easier to handle in most environments and will not require an upgrade in class version for every minor change. Kind regards, Peter Kriens On Sat, Jun 25, 2011 at 3:01 AM, Alex Buckley wrote: Since attachments seem to be scrubbed, here is the text of the proposal, obviously without links to the requirements document. ClassFile changes for Java Modules Binary form of a module declaration (a.k.a. module-info.class) A compilation unit that contains a module declaration (and potentially additional arbitrary content) is compiled to a ClassFile structure like any other compilation unit. By convention, the name of a compilation unit that contains a module declaration is module-info.java, echoing the package-info.java convention for a compilation unit that contains solely a package declaration. Consequently, by convention, the name for the compiled form of a module declaration is module-info.class. A new flag in the ClassFile.access_flags item, ACC_MODULE (0x0800), indicates that the ClassFile represents a module rather than a class or interface. Note that this flag plays a similar role to ACC_INTERFACE (0x0200), and does not describe accessibility of a class or interface. If ACC_MODULE is set in ClassFile.access_flags, then no other flag in ClassFile.access_flags may be set, and the following rules apply to the rest of the ClassFile structure: * major_version, minor_version: >=52.0 (Java SE 8 and above) * this_class: [Module's name in internal form (JVMS 4.2.1)]/module-info (Traditionally, if this_class indicates "P/Q/R", then the ClassFile can be expected to live in a file R.class in a directory P/Q representing a package. This explains why "/module-info" is a suffix in this_class above: if this_class indicates "P/Q/module-info", then the ClassFile can be expected to live in a file module-info.class in a directory P/Q representing a module. The "real name" of the module, shorn of "/module-info", can be obtained from the Module attribute.) * super_class, interfaces_count, fields_count, methods_count: 0 * attributes: One Module attribute must be present, to record the name and version of the module. At most one of each of the ModuleRequires, ModuleProvides, ModulePermits, ModuleExports, ModuleClass, and ModuleData attributes may be present. Except for these attributes and Synthetic, SourceFile, SourceDebugExtension, and Deprecated, none of the pre-defined attributes in JVMS 4.7 may appear. The CONSTANT_ModuleId_info structure The CONSTANT_ModuleId_info structure in the constant pool is used to represent a pair of a module name and a module version. CONSTANT_ModuleId_info { u1 tag; // 19 u2 name_index; // Points to a CONSTANT_Utf8_info representing module name in internal form (JVMS 4.2.1) u2 version_index; // Optionally points to a CONSTANT_Utf8_info representing module version } If the version_index item is 0 (as opposed to pointing to a CONSTANT_Utf8_info structure which holds the string "0"), then no module version is present. If the version_index item is not 0, then a module version is present. There is no required structure or semantics for the underlying UTF8 string which represents a module version. The Module attribute Module_attribute { u2 attribute_name_index; u4 attribute_length; u2 module_id_index; } The items of the Module_attribute structure are as follows: attribute_name_index The value of the attribute_name_index item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_Utf8_info structure representing the string "Module". attribute_length The value of the attribute_length item must be 2. module_id_index The value of the module_id_index item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_ModuleId_info structure representing the name and version of the module represented by this ClassFile. The name must be equal to the module name indicated by the ClassFile.this_class item without the "/module-info" suffix. The ModuleRequires attribute ModuleRequires_attribute { u2 attribute_name_index; u4 attribute_length; u2 requires_length; { u2 requires_index; u1 flags; } requires_table[requires_length]; } attribute_name_index The value of the attribute_name_index item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_Utf8_info structure representing the string "ModuleRequires". attribute_length The value of the attribute_length item is the length of the attribute excluding the initial six bytes. requires_length The value of the requires_length indicates the number of entries in the requires_table. requires_table Each requires_index must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_ModuleId_info structure representing the target module on which this module depends. The value of the associated flags item is as follows: * 0x01 indicates this dependency on the target module is optional. * 0x02 indictates the target module's types must be loaded by the same defining classloader as the types of the module represented by this ClassFile. A module name may be referenced by at most one entry in a requires_table. The ModulePermits attribute ModulePermits_attribute { u2 attribute_name_index; u4 attribute_length; u2 permits_length; { u2 permits_index } permits_table[permits_length]; } attribute_name_index The value of the attribute_name_index item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_Utf8_info structure representing the string "ModulePermits". attribute_length The value of the attribute_length item is the length of the attribute excluding the initial six bytes. permits_length The value of the permits_length indicates the number of entries in the permits_table. permits_table The value of each permits_index in this item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_ModuleId_info structure representing a module which is permitted to have a dependency on the module represented by this ClassFile. (It is possible that a module system uses only the name of the permitted module, not its version, in determining visibility.) A module name may be referenced by at most one entry in a permits_table. The ModuleProvides attribute ModuleProvides_attribute { u2 attribute_name_index; u4 attribute_length; u2 provides_length; { u2 provides_index; } provides_table[provides_length]; } attribute_name_index The value of the attribute_name_index item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_Utf8_info structure representing the string "ModuleProvides". attribute_length The value of the attribute_length item is the length of the attribute excluding the initial six bytes. provides_length The value of the provides_length indicates the number of entries in the provides_table. provides_table The value of each provides_index in this item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_ModuleId_info structure representing a module that is an alias for the module represented by this ClassFile. A module name may be referenced by at most one entry in a provides_table. The ModuleExports attribute ModuleExports_attribute { u2 attribute_name_index; u4 attribute_length; u2 exports_length; { u2 exports_index; u1 export_kind; u2 source_index; } exports_table[exports_length]; } attribute_name_index The value of the attribute_name_index item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_Utf8_info structure representing the string "ModuleExports". attribute_length The value of the attribute_length item is the length of the attribute excluding the initial six bytes. exports_length The value of the exports_length indicates the number of entries in the exports_table. exports_table The value of each export_index in this item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_Utf8_info structure representing an entity to be exported by the module represented by this ClassFile. The value of the associated export_kind item indicates the kind of entity to be exported, as follows: * 0x01 indicates the entity is a class or interface; it is the entity to be exported. * 0x02 indicates the entity is a class or interface; it and its member types are the entities to be exported. * 0x04 indicates the entity is a package; the member top-level classes and interfaces (and their member types) of this package are the entities to be exported. * 0x08 indicates the entity is a package; the member top-level classes and interfaces (and their member types) of this package and all its subpackages (recursively) are the entities to be exported. The value of the associated source_index item must be a valid index into the constant_pool table where the entry is a CONSTANT_ModuleId_info structure representing the module which declared the exported entity. (In most cases, where the exported entity is declared in the current module, source_index will simply point to the same constant pool entry as the Module attribute. But where the exported entity is re-exported, source_index may assist resolution and diagnostics.) The ModuleClass attribute ModuleClass_attribute { u2 attribute_name_index; u4 attribute_length; u2 main_class_index; } attribute_name_index The value of the attribute_name_index item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_Utf8_info structure representing the string "ModuleClass". attribute_length The value of the attribute_length item is 2. main_class_index The value of the main_class_index item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_Class_info structure representing the name of the class which is the entrypoint to the module represented by this ClassFile. The ModuleData attribute ModuleExtension_attribute { u2 attribute_name_index; u4 attribute_length; u2 data_index; } attribute_name_index The value of the attribute_name_index item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_Utf8_info structure representing the string "ModuleData". attribute_length The value of the attribute_length item is 2. data_index The value of the main_class item must be a valid index into the constant_pool table. The constant_pool entry at that index must be a CONSTANT_Utf8_info structure representing the content, other than the module declaration, of the compilation unit that declared the module represented by this ClassFile. From alex.buckley at oracle.com Wed Jun 29 19:51:59 2011 From: alex.buckley at oracle.com (Alex Buckley) Date: Wed, 29 Jun 2011 19:51:59 -0700 Subject: module-info.class spec and attributes In-Reply-To: <4E0BCA9E.8020003@redhat.com> References: <4E00E72A.8020104@oracle.com> <4E052910.5080001@oracle.com> <4E05335F.6090609@oracle.com> <4E053859.3060703@oracle.com> <4E0BBE2B.9090907@oracle.com> <4E0BC231.7000200@redhat.com> <4E0BC764.3030300@oracle.com> <4E0BCA9E.8020003@redhat.com> Message-ID: <4E0BE4CF.6000502@oracle.com> I think we're slightly missing each other here. I wasn't implying javac is unable to cache module declarations; rather that module declarations are now woven deeply into compilation because determining the visibility of _any_ type (whether available in source or class file form) is dependent on a graph of module declarations (possibly cached by the module system, but not when the world is being built from source). Jigsaw is just keeping the structure of those module declarations in source/binary forms that compilers are familiar with. Alex On 6/29/2011 6:00 PM, David M. Lloyd wrote: > Not really. In fact that's basically using the (apparently poor) design > choices of the compiler to justify this, which I find to be exceedingly > weak. > > The compiler should never need to parse module information more than > once per compilation, regardless of how many source files are compiled. > If it's requiring that, fix the real problem, rather than using it as > justification for crap decisions elsewhere. > > On 06/29/2011 07:46 PM, Alex Buckley wrote: >> In the context of Project Jigsaw, javac parses module declarations every >> time a source file in a modular directory structure is compiled. That's >> a powerful driver for Java-like source syntax and a class file form. >> >> Projects which don't put module information so close to the compiler may >> well choose other forms. >> >> Alex >> >> On 6/29/2011 5:24 PM, David M. Lloyd wrote: >>> On 06/29/2011 07:07 PM, Alex Buckley wrote: >>>> (Sent this mail yesterday but I think it got lost.) >>>> >>>> On 6/28/2011 2:24 PM, David Bosschaert wrote: >>>>> I wonder why such a binary .class file for module declarations is >>>>> really needed. Why not go for a textual file, for instance >>>>> META-INF/module-info.jmod (or something like that) and declare the >>>>> module declarations in there using a textual format of some sort? >>>> >>>> The arguments for a binary compiled form are given at: >>>> >>>> http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#_C >>>> >>>> >>> >>> All of which are trivially debunkable. Yes, it is *slightly* more >>> efficient to read structured binary than a text format. It is an >>> insignificant difference though. We actually parse ours as XML using >>> StAX, which should be somewhat more expensive than a dedicated text >>> parser, and it's so fast that I'm not even able to get a significant >>> measurement of the time when booting up a 100+ module project. The >>> performance argument is invalid. >>> >>> And constraining the module metadata to a Java-like syntax with a >>> Java-like file name is just a dumb idea. There's no reason for it. >>> >>> Saying that requiring a tool to modify the metadata for a module is >>> somehow a good thing is pretty off the wall too. I mean just crazy. >>> >>> Come down to Earth guys. >>> >>>>> IMO there are multiple benefits associated keeping the textual format. >>>>> People using the jar can use a simple zip tool to look inside the jar >>>>> to see what its dependencies are. Additionally tools can easily be >>>>> written to read the module-info.jmod file without needing to resort to >>>>> classfile parsing libraries. The module information can be of use to >>>>> both developer as well as deployer tools. >>>>> >>>>> Finally it's probably easier to make the textual file extensible, as >>>>> required by [1], without resorting to inefficient reflection-style >>>>> operations. >>>> >>>> I think these points are also captured in Appendix C. Could not the >>>> same >>>> points have been made in favor of a textual format for class and >>>> interface declarations? >>> >>> Yup. But that ship has sailed. A binary format for module >>> meta-information is going to be yet another annoying disaster. > > From alex.buckley at oracle.com Wed Jun 29 20:30:36 2011 From: alex.buckley at oracle.com (Alex Buckley) Date: Wed, 29 Jun 2011 20:30:36 -0700 Subject: module-info.class spec and attributes In-Reply-To: References: Message-ID: <4E0BEDDC.3090808@oracle.com> Hi Peter, Thanks for casting your eye over it. On 6/29/2011 2:33 AM, Peter Kriens wrote: > There seems to be quite a bit of redundancy (and thus complexity) in > this structure because it allows * relations between the flags > (provide,require,permit,module) that designate a ModuleId_Info > struct. I.e. the proposal allows me to require/permit/provide the > same module multiple times. Even though this is not allowed with an > extraneous constraint this seems inefficient design, why create the > opportunity for errors? It seems that provide/require/permit/module > could easily be a handled with an enumeration or bitset: { > THIS_MODULE, PROVIDE, PERMIT, REQUIRE_OPTIONAL, REQUIRE_FRAGMENT} in > the ModuleId_Info struct? The use of attributes to set a bit seems > overkill and will require unnecessary constraints on the cardinality > of these attributes, making processing this file in tools harder. You'd still need a constraint that a (module id, enum value) pair must be unique. Would that be extraneous too? The use of multiple attributes is verbose, sure, but it's convenient for prototyping. > Similar about the redundancy in the module name. The module name is > both the this_class package name as well as defined in the > ModuleId_Info struct? The reason for this is noted in the first section: there is conventionally a mapping between pathname and this_class. The tiny duplication is really not a big deal. > Interestingly the model allows package export (though unfortunately > not import, one should always wonder why things are not symmetric, > but I guess that is another discussion). However, class exports can > be top levels only (I assume no nested classes?) and recursively. > However, packages are top level classes in the designated package and > the all subpackages. This seems to miss the cases for nested classes > in the single package and nested classes in the sub-packages? There is nothing limiting an export to a top level class. "If you can name it, you can export it." (I should have said that exported entities are identified by binary names, extended to allow package names, which have canonical names but not binary names today.) Not sure what you mean by "packages are top level classes in the designated package". Both 0x04 and 0x08 for package export include nested classes, in the same package and in subpackages respectively. > I am a bit confused about the scope of the module as this is not > defined. I.e. a class defines all its members, not just the public > ones. Shouldn't a module also define its private members? Since "private" (i.e. non-exported) classes are not visible from outside the module, what difference would enumerating them make? > Obviously the extension model is a bit of a joke, all we get is a > list of unadorned strings per module? The minimum that is needed is a > model that allows the exports, provides, permits, and module > attributes to be annotated in an elegant way. For example, > ModuleClass is clearly a Jigsaw extension and several of the bit > fields are also highly semantic, such semantics should be extensible > as it is very unlikely you get it right the first time. Why not model > these with an extension model and eat your own dog food? The attributes follow the requirements of the Java Module System. ModuleData is for content in a module declaration that is outside those requirements. Regarding ModuleClass, it's a placeholder until there's a requirement that addresses entrypoints. > Treating packages as members of a module and imports as references to > other modules would make the class file a very good fit; every Java > developer would immediately see the symmetry and have an intuitive > understanding. Package imports are an open requirement (http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#package-level-dependences) so are not addressed here. > As the current proposal has no affinity to a class file format > whatsoever; it seems wrong to abuse the class file format just > because it happens to support attributes. There are better formats > for this purpose, formats that could also allow some human > readability for debugging and easy annotation by third parties > without requiring class format changes for every new module concept. > Obviously XML is the most powerful and best suited format for this > problem. The JLS module specification can just as easy output to XML > as attributes in a class file but will be significantly easier to > handle in most environments and will not require an upgrade in class > version for every minor change. Let's not argue about syntax ad infinitum. The big picture is that the Java Module System will define a fixed set of "module concepts", as you put it, whose semantics evolve at the pace of Java SE. New SE, new classfile version, new module system constructs. Tools will evolve to manage those constructs, as they do today for new language, VM, and API features. For example, javadoc will render module dependencies. Alex From forax at univ-mlv.fr Thu Jun 30 00:09:57 2011 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Thu, 30 Jun 2011 09:09:57 +0200 Subject: module-info.class spec and attributes In-Reply-To: <4E0BE4CF.6000502@oracle.com> References: <4E00E72A.8020104@oracle.com> <4E052910.5080001@oracle.com> <4E05335F.6090609@oracle.com> <4E053859.3060703@oracle.com> <4E0BBE2B.9090907@oracle.com> <4E0BC231.7000200@redhat.com> <4E0BC764.3030300@oracle.com> <4E0BCA9E.8020003@redhat.com> <4E0BE4CF.6000502@oracle.com> Message-ID: <4E0C2145.4070004@univ-mlv.fr> David, I think storing the module metadata as a class file is a brilliant idea. First you have to think that what we want is not one text file but two. Some module metadata are generated (the one marked synthetic in the current format) by the compiler and as a user I don't want to see them, edit them, etc. Eclipse plugins use the approach of having one file for the two purposes, be a module descriptor editable by a human and be a module deployment descriptor readable by the module runtime. As a user, the experience is painful, the file is big and a big part of the data are useless and sometimes you have to reconcile the state of the source and the plugin descriptor by hand. To summarize, we need a file which is readable/writable by a human, readable by the compiler and the compiler need to generate a module deployment from it that need to be readable by the module runtime system. Hence, it's logical to have a Java file and a class file. Also a classfile is versioned, compact, has no character encoding problem, is well integrated with the other files and as Alex said is more efficient than an XML file. Another cool point is that because the module descriptor is a Java file you know where to put the module doc :) R?mi On 06/30/2011 04:51 AM, Alex Buckley wrote: > I think we're slightly missing each other here. I wasn't implying > javac is unable to cache module declarations; rather that module > declarations are now woven deeply into compilation because determining > the visibility of _any_ type (whether available in source or class > file form) is dependent on a graph of module declarations (possibly > cached by the module system, but not when the world is being built > from source). Jigsaw is just keeping the structure of those module > declarations in source/binary forms that compilers are familiar with. > > Alex > > On 6/29/2011 6:00 PM, David M. Lloyd wrote: >> Not really. In fact that's basically using the (apparently poor) >> design choices of the compiler to justify this, which I find to be >> exceedingly weak. >> >> The compiler should never need to parse module information more than >> once per compilation, regardless of how many source files are >> compiled. If it's requiring that, fix the real problem, rather than >> using it as justification for crap decisions elsewhere. >> >> On 06/29/2011 07:46 PM, Alex Buckley wrote: >>> In the context of Project Jigsaw, javac parses module declarations >>> every >>> time a source file in a modular directory structure is compiled. That's >>> a powerful driver for Java-like source syntax and a class file form. >>> >>> Projects which don't put module information so close to the compiler >>> may >>> well choose other forms. >>> >>> Alex >>> >>> On 6/29/2011 5:24 PM, David M. Lloyd wrote: >>>> On 06/29/2011 07:07 PM, Alex Buckley wrote: >>>>> (Sent this mail yesterday but I think it got lost.) >>>>> >>>>> On 6/28/2011 2:24 PM, David Bosschaert wrote: >>>>>> I wonder why such a binary .class file for module declarations is >>>>>> really needed. Why not go for a textual file, for instance >>>>>> META-INF/module-info.jmod (or something like that) and declare the >>>>>> module declarations in there using a textual format of some sort? >>>>> >>>>> The arguments for a binary compiled form are given at: >>>>> >>>>> http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#_C >>>>> >>>>> >>>> >>>> All of which are trivially debunkable. Yes, it is *slightly* more >>>> efficient to read structured binary than a text format. It is an >>>> insignificant difference though. We actually parse ours as XML using >>>> StAX, which should be somewhat more expensive than a dedicated text >>>> parser, and it's so fast that I'm not even able to get a significant >>>> measurement of the time when booting up a 100+ module project. The >>>> performance argument is invalid. >>>> >>>> And constraining the module metadata to a Java-like syntax with a >>>> Java-like file name is just a dumb idea. There's no reason for it. >>>> >>>> Saying that requiring a tool to modify the metadata for a module is >>>> somehow a good thing is pretty off the wall too. I mean just crazy. >>>> >>>> Come down to Earth guys. >>>> >>>>>> IMO there are multiple benefits associated keeping the textual >>>>>> format. >>>>>> People using the jar can use a simple zip tool to look inside the >>>>>> jar >>>>>> to see what its dependencies are. Additionally tools can easily be >>>>>> written to read the module-info.jmod file without needing to >>>>>> resort to >>>>>> classfile parsing libraries. The module information can be of use to >>>>>> both developer as well as deployer tools. >>>>>> >>>>>> Finally it's probably easier to make the textual file extensible, as >>>>>> required by [1], without resorting to inefficient reflection-style >>>>>> operations. >>>>> >>>>> I think these points are also captured in Appendix C. Could not >>>>> the same >>>>> points have been made in favor of a textual format for class and >>>>> interface declarations? >>>> >>>> Yup. But that ship has sailed. A binary format for module >>>> meta-information is going to be yet another annoying disaster. >> >> From sean.mullan at oracle.com Thu Jun 30 12:09:55 2011 From: sean.mullan at oracle.com (Sean Mullan) Date: Thu, 30 Jun 2011 15:09:55 -0400 Subject: module-file format and signing Message-ID: <4E0CCA03.40604@oracle.com> Here is a list of issues and suggested changes to the module-file format [1] that I encountered while adding digital signatures to module-files [2]. 1) The csize field of the Module file header cannot be properly calculated when signing a module-file, because the length of the signature cannot be pre-determined before the signature is applied. Thus, once the signature has been applied, the csize field cannot be updated to include the length of the signature without invalidating the signature. Suggested Change: change the csize field to be the size of the "rest" of the file (i.e. the contents after the module-info or signature section (if specified)) 2) (Similar issue as #1) The sections field of the Module file header needs to be incremented by one when signing a module-file. It may be possible to do that, but it would result in the implementation being much more complex. Ideally, when a signature is generated over an existing module-file, none of the contents of that module file should need to be modified. Suggested change: remove the sections field. It is not strictly necessary. 3) The location of the hash field in the Module file header makes it impossible to calculate the hash while you write the contents of the module-file. This is because the Module file header is at the beginning of the file, and some of its fields (ex: csize) cannot be determined until the rest of the file has been written. Although this isn't a showstopper, it does seem wasteful to have to go back and re-read all of the module-file contents in order to calculate the hash. Suggested change: don't include the Module file header in the hash of the entire file. Also, add another field that contains just a hash of the Module file header itself. 4) Do we need the hashLength fields? This seems like redundant information, since this can be determined from the algorithm specified in the hashType field. Suggested change: remove the hashLength fields in the Module file and Section headers. 5) The hash field in the Section header only covers the section content, and not the header itself. This means the integrity of this data cannot be verified until the entire file is processed and the Module file header hash is verified. For example, one could change the compressor field or the number of subsections. These modifications would eventually be detected before the module was installed, but may cause unpredictable behavior before that, since the implementation is acting on the contents of what is in the header before the data can be verified with what is protected by the signature. If you change the hash to cover the Section header as well as the content then you introduce the same problem as in #3 and you will not be able to hash the section content as you write it. Suggested change: Add a hash field that just covers the Section Header. --Sean [1] http://cr.openjdk.java.net/~mr/jigsaw/notes/module-file-format/ [2] http://cr.openjdk.java.net/~mullan/jigsaw/signed-module-file-format