From brian at pontarelli.com Thu Dec 1 08:43:59 2011 From: brian at pontarelli.com (Brian Pontarelli) Date: Thu, 1 Dec 2011 09:43:59 -0700 Subject: Final thoughts Message-ID: Things at Inversoft have been getting extremely busy lately and my free time is becoming less existent each day. While this is good for the company, it is bad for my involvement in other areas. I'm going to have to back burner my involvement with the Jigsaw groups and lists for the foreseeable future, but I wanted to put down some final thoughts before I go. Here's a quick list of topics I wanted to discuss but didn't quite get to: Version compatibility (we started talking about this some) Compiling Java code (I'm not convinced that javac is the best dependency resolution system) Resolving from multiple repositories and security (this is important for corporations with IP stored in internal repositories as well as standard repositories such as the Maven repositories) Dependency meta-data (I feel strongly this should be outside of the module to improve performance. I also feel this should be standardized so that the JDK and other tools can use the format including Maven, Savant, and Ivy) Transitive dependencies (there are some items to discuss such as when transitive dependencies are included in the module/classpath. I feel that at compile time transitive dependencies should not be included) Dependency exclusion (I'm not sure these are a valid use-case if you have version compatibility) Optional dependencies (I think these make more sense than exclusions, but I'm still not convinced about them. I sorta feel that if a project has optional dependencies it really should be multiple projects in a hierarchy) Version ranges (I think these should not be included) Snapshot versions (some form of these needs to be supported but their needs to be rules about them and how they impact releasing. i.e. you should not be able to release a full version of a module that depends on a snapshot version of a dependency. Full versions are things like 1.1) Version numbering (I feel that my EBNF describes the most common version scheme and that should cover 99% of the cases. I also feel that allowing other version schemes isn't a good idea, but is possible) I'm certain that there are other things I'm missing, but those are the things that I've worked on with Savant and other dependency systems over the years. Good luck and feel free to contact me directly at anytime. -bp From mandy.chung at oracle.com Thu Dec 1 11:33:42 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Thu, 01 Dec 2011 11:33:42 -0800 Subject: Compressing class files in the module library In-Reply-To: <4ED3A5ED.1010903@oracle.com> References: <4EBAC493.8080307@oracle.com> <4EBC4E51.3020202@oracle.com> <4EC0DE32.6060202@oracle.com> <4ED3A5ED.1010903@oracle.com> Message-ID: <4ED7D696.5050903@oracle.com> On 11/28/2011 7:17 AM, Alan Bateman wrote: > > I've updated the patch [1] for the compression support to also add an > option to strip the debug attributes from classes when installing a > module. > > [1] http://cr.openjdk.java.net/~alanb/ModuleLibraryCompression/webrev/ > Looks good overall. Some minor comments: SimpleLibrary.java L840: should it clean up and delete classes.pack temp file if IOException is thrown? Defs-modules.gmk L140-143: these flags might be more appropriate to define in make/common/share/Defs-java.gmk. I assume these flags are used in all jmod create/install commands during the jdk build. It might be better and less error-prone to define HOST_JMOD_CREATE_CMD and HOST_JMOD_INSTALL_CMD. Mandy From Alan.Bateman at oracle.com Thu Dec 1 12:41:27 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 01 Dec 2011 20:41:27 +0000 Subject: Compressing class files in the module library In-Reply-To: <4ED7D696.5050903@oracle.com> References: <4EBAC493.8080307@oracle.com> <4EBC4E51.3020202@oracle.com> <4EC0DE32.6060202@oracle.com> <4ED3A5ED.1010903@oracle.com> <4ED7D696.5050903@oracle.com> Message-ID: <4ED7E677.3090301@oracle.com> On 01/12/2011 19:33, Mandy Chung wrote: > > Looks good overall. Some minor comments: > > SimpleLibrary.java > L840: should it clean up and delete classes.pack temp file if > IOException is thrown? If the install fails when it will be deleted (the !complete logic in the install method). We only need to explicitly remove the pack file when the stripping succeeds. > > Defs-modules.gmk > L140-143: these flags might be more appropriate to define in > make/common/share/Defs-java.gmk. I assume these flags are used > in all jmod create/install commands during the jdk build. It might be > better and less error-prone to define HOST_JMOD_CREATE_CMD > and HOST_JMOD_INSTALL_CMD. > Currently it uses the options for all module libraries that the build generates but it should only be for the jre* images as it doesn't make sense to strip debug attributes from modules installed into the module library in jdk* image. For that reason I'd prefer not add to the HOST_JMOD* commands but instead add the options only where needed. -Alan From mandy.chung at oracle.com Thu Dec 1 12:44:57 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Thu, 01 Dec 2011 12:44:57 -0800 Subject: Compressing class files in the module library In-Reply-To: <4ED7E677.3090301@oracle.com> References: <4EBAC493.8080307@oracle.com> <4EBC4E51.3020202@oracle.com> <4EC0DE32.6060202@oracle.com> <4ED3A5ED.1010903@oracle.com> <4ED7D696.5050903@oracle.com> <4ED7E677.3090301@oracle.com> Message-ID: <4ED7E749.2080909@oracle.com> On 12/1/2011 12:41 PM, Alan Bateman wrote: > Currently it uses the options for all module libraries that the build > generates but it should only be for the jre* images as it doesn't make > sense to strip debug attributes from modules installed into the module > library in jdk* image. For that reason I'd prefer not add to the > HOST_JMOD* commands but instead add the options only where needed. Ah, that's right. It's only used in making the images. So your change looks fine. Mandy From martijnverburg at gmail.com Fri Dec 2 04:43:54 2011 From: martijnverburg at gmail.com (Martijn Verburg) Date: Fri, 2 Dec 2011 12:43:54 +0000 Subject: Possible dependency complication arising from a fix to the Manifest class Message-ID: Hi all, I'm cleaning up some warnings in java.util.jar.Manifest and I'm replacing a deprecated String construction with one that takes a Charset. I'm introducing the StandardCharsets import (since it's UTF_8 that is required in this case) to do this. I assume introducing StandardCharsets (a nio package based class) is OK for a fairly low level util class like this? I'm more worried about any potential impact on jigsaw, at the moment Manifest only pulls in java.util and java.io imports - I haven't looked into jigsaw in any way shape or form to know whether adding a nio dependency would cause any headaches. Cheers, Martijn From alan.bateman at oracle.com Fri Dec 2 04:59:48 2011 From: alan.bateman at oracle.com (alan.bateman at oracle.com) Date: Fri, 02 Dec 2011 12:59:48 +0000 Subject: hg: jigsaw/jigsaw/jdk: Allow classes to be stored in compressed form in module library (jmod create -z) Message-ID: <20111202130013.1400047518@hg.openjdk.java.net> Changeset: 22ccf6e7a092 Author: alanb Date: 2011-12-02 12:44 +0000 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/22ccf6e7a092 Allow classes to be stored in compressed form in module library (jmod create -z) Allow debug attributes to be stripped from classes when installing module (jmod install -G) Reviewed-by: mchung, chegar ! make/common/Defs-modules.gmk ! make/common/Modules.gmk ! make/modules/Makefile ! src/share/classes/org/openjdk/jigsaw/ClassInfo.java ! src/share/classes/org/openjdk/jigsaw/Files.java ! src/share/classes/org/openjdk/jigsaw/Launcher.java ! src/share/classes/org/openjdk/jigsaw/ModuleFile.java ! src/share/classes/org/openjdk/jigsaw/SimpleLibrary.java ! src/share/classes/org/openjdk/jigsaw/cli/Librarian.java ! src/share/classes/org/openjdk/jigsaw/cli/Packager.java ! test/org/openjdk/jigsaw/InstallFromRepo.java ! test/org/openjdk/jigsaw/_Library.java ! test/org/openjdk/jigsaw/_RemoteRepositoryList.java ! test/org/openjdk/jigsaw/cli/jmod-basic.sh From Alan.Bateman at oracle.com Fri Dec 2 05:18:04 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 02 Dec 2011 13:18:04 +0000 Subject: Possible dependency complication arising from a fix to the Manifest class In-Reply-To: References: Message-ID: <4ED8D00C.90802@oracle.com> On 02/12/2011 12:43, Martijn Verburg wrote: > Hi all, > > I'm cleaning up some warnings in java.util.jar.Manifest and I'm > replacing a deprecated String construction with one that takes a > Charset. > > I'm introducing the StandardCharsets import (since it's UTF_8 that is > required in this case) to do this. I assume introducing > StandardCharsets (a nio package based class) is OK for a fairly low > level util class like this? > > I'm more worried about any potential impact on jigsaw, at the moment > Manifest only pulls in java.util and java.io imports - I haven't > looked into jigsaw in any way shape or form to know whether adding a > nio dependency would cause any headaches. > > Cheers, > Martijn Hi Martijn, The zip code already uses StandardCharsets so it's likely it is already loaded. In any case, you're not introducing a new dependency in the module graph as nio, zip, and all the core classes are in the (prototype) base module. -Alan. From martijnverburg at gmail.com Fri Dec 2 05:40:53 2011 From: martijnverburg at gmail.com (Martijn Verburg) Date: Fri, 2 Dec 2011 13:40:53 +0000 Subject: Possible dependency complication arising from a fix to the Manifest class In-Reply-To: <4ED8D00C.90802@oracle.com> References: <4ED8D00C.90802@oracle.com> Message-ID: Thanks Alan - will submit patch as is. On 2 December 2011 13:18, Alan Bateman wrote: > On 02/12/2011 12:43, Martijn Verburg wrote: >> >> Hi all, >> >> I'm cleaning up some warnings in java.util.jar.Manifest and I'm >> replacing a deprecated String construction with one that takes a >> Charset. >> >> I'm introducing the StandardCharsets import (since it's UTF_8 that is >> required in this case) to do this. I assume introducing >> StandardCharsets (a nio package based class) is OK for a fairly low >> level util class like this? >> >> I'm more worried about any potential impact on jigsaw, at the moment >> Manifest only pulls in java.util and java.io imports - I haven't >> looked into jigsaw in any way shape or form to know whether adding a >> nio dependency would cause any headaches. >> >> Cheers, >> Martijn > > Hi Martijn, > > The zip code already uses StandardCharsets so it's likely it is already > loaded. In any case, you're not introducing a new dependency in the module > graph as nio, zip, and all the core classes are in the (prototype) base > module. > > -Alan. > From sean.mullan at oracle.com Tue Dec 6 07:05:12 2011 From: sean.mullan at oracle.com (Sean Mullan) Date: Tue, 06 Dec 2011 10:05:12 -0500 Subject: Module library attributes Message-ID: <4EDE2F28.3050406@oracle.com> I'd like to be able to store a persistent attribute with a module when it is installed that could then be retrieved at runtime when all I have is a reference to the Module object. I don't think this is quite the same as metadata, as listed in the jigsaw requirements. It is more like data that we can use to optimize our implementation. It doesn't necessarily have to be exposed via a public API. --Sean From sean.mullan at oracle.com Tue Dec 6 08:01:09 2011 From: sean.mullan at oracle.com (Sean Mullan) Date: Tue, 06 Dec 2011 11:01:09 -0500 Subject: Platform module security Message-ID: <4EDE3C45.2000006@oracle.com> How do we know if a module is part of the JRE platform and can be trusted exclusively? The current code in org.openjdk.jigsaw.Platform uses a fragile mechanism: public static boolean isPlatformModuleName(String mn) { return (mn.equals("jdk") || mn.startsWith("jdk.") || mn.startsWith("sun.")); } // ## We really must do something more secure and robust here! static boolean isPlatformContext(BaseContext cx) { for (ModuleId mid : cx.modules()) { if (!isPlatformModuleName(mid.name())) return false; } return true; } I can think of two potential solutions: 1) Nothing other than JRE platform modules can be installed/stored in the system library. How this is enforced is a separate question. 2) All platform modules are signed with a special certificate that we can identify. Any other thoughts? Thanks, Sean From sean.mullan at oracle.com Tue Dec 6 09:51:35 2011 From: sean.mullan at oracle.com (Sean Mullan) Date: Tue, 06 Dec 2011 12:51:35 -0500 Subject: Module library attributes In-Reply-To: <4EDE2F28.3050406@oracle.com> References: <4EDE2F28.3050406@oracle.com> Message-ID: <4EDE5627.1000403@oracle.com> Just to expand on this a bit - this would be useful to quickly retrieve information associated with a module that has already been checked when it was installed, for example - it is a platform module, or is signed by a specific certificate that gives it special privileges (ex: a JCE provider). Perhaps the module's context would be an appropriate place to store this type of information. Any thoughts? --Sean On 12/6/11 10:05 AM, Sean Mullan wrote: > I'd like to be able to store a persistent attribute with a module when it is > installed that could then be retrieved at runtime when all I have is a reference > to the Module object. I don't think this is quite the same as > metadata, as listed in the jigsaw requirements. It is more like data that we can > use to optimize our implementation. It doesn't necessarily have to be exposed > via a public API. > > --Sean > From hinkmond.wong at oracle.com Tue Dec 6 10:30:19 2011 From: hinkmond.wong at oracle.com (Hinkmond Wong) Date: Tue, 06 Dec 2011 10:30:19 -0800 Subject: Platform module security In-Reply-To: <4EDE3C45.2000006@oracle.com> References: <4EDE3C45.2000006@oracle.com> Message-ID: <4EDE5F3B.80503@oracle.com> On 12/6/2011 8:01 AM, Sean Mullan wrote: > How do we know if a module is part of the JRE platform and can be trusted > exclusively? > > The current code in org.openjdk.jigsaw.Platform uses a fragile mechanism: > > public static boolean isPlatformModuleName(String mn) { > return (mn.equals("jdk") || mn.startsWith("jdk.") || > mn.startsWith("sun.")); > } > > // ## We really must do something more secure and robust here! > static boolean isPlatformContext(BaseContext cx) { > for (ModuleId mid : cx.modules()) { > if (!isPlatformModuleName(mid.name())) > return false; > } > return true; > } > > I can think of two potential solutions: > > 1) Nothing other than JRE platform modules can be installed/stored in the system > library. How this is enforced is a separate question. > > 2) All platform modules are signed with a special certificate that we can identify. If we do go with #2, it would be good to get a measurement of the core platform module loading times before and after the change, especially on a Java Embedded reference target device (such as Linux/ARM) instead of a Java desktop system, to make sure the performance hit due to module code signing is not unreasonable for typical use. Thanks, Hinkmond > > Any other thoughts? > > Thanks, > Sean -- Oracle Hinkmond Wong | Consulting Member of Technical Staff Phone: +1 408.276.7618 | Fax: +1 408.276.7674 Oracle Java Embedded 4210 Network Ci., M/S USCA22-rm2364 | Santa Clara, CA 95054 Green Oracle Oracle is committed to developing practices and products that help protect the environment From Alan.Bateman at oracle.com Wed Dec 7 06:05:42 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 07 Dec 2011 14:05:42 +0000 Subject: Module library attributes In-Reply-To: <4EDE2F28.3050406@oracle.com> References: <4EDE2F28.3050406@oracle.com> Message-ID: <4EDF72B6.30304@oracle.com> On 06/12/2011 15:05, Sean Mullan wrote: > I'd like to be able to store a persistent attribute with a module when it is > installed that could then be retrieved at runtime when all I have is a reference > to the Module object. I don't think this is quite the same as > metadata, as listed in the jigsaw requirements. It is more like data that we can > use to optimize our implementation. It doesn't necessarily have to be exposed > via a public API. > > --Sean > It wouldn't be hard to extend the module library to support additional data, be it on a per-module or per-configuration basis. At some point we'll need to have a database to support faster indexing and potentially this could be used for other data too. -Alan. From karen.kinnear at oracle.com Wed Dec 7 09:33:36 2011 From: karen.kinnear at oracle.com (Karen Kinnear) Date: Wed, 7 Dec 2011 12:33:36 -0500 Subject: Module library attributes In-Reply-To: <4EDF72B6.30304@oracle.com> References: <4EDE2F28.3050406@oracle.com> <4EDF72B6.30304@oracle.com> Message-ID: <0D3894C5-8788-45F1-A044-E7E43745223B@oracle.com> We will want to store the "preverified" state for classes as well. thanks, Karen On Dec 7, 2011, at 9:05 AM, Alan Bateman wrote: > On 06/12/2011 15:05, Sean Mullan wrote: >> I'd like to be able to store a persistent attribute with a module when it is >> installed that could then be retrieved at runtime when all I have is a reference >> to the Module object. I don't think this is quite the same as >> metadata, as listed in the jigsaw requirements. It is more like data that we can >> use to optimize our implementation. It doesn't necessarily have to be exposed >> via a public API. >> >> --Sean >> > It wouldn't be hard to extend the module library to support additional data, be it on a per-module or per-configuration basis. At some point we'll need to have a database to support faster indexing and potentially this could be used for other data too. > > -Alan. From mark.reinhold at oracle.com Wed Dec 7 13:17:02 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 07 Dec 2011 13:17:02 -0800 Subject: Module Views Message-ID: <20111207211702.EFF03809@eggemoggin.niobe.net> A build of the current Jigsaw prototype divides the JDK into 63 separate modules. Roughly a third of these modules are strictly for internal use within the JDK, are useless on their own, and are not intended to be referenced directly by application code. We'd like to keep the total number of modules comprising the JDK as small as is reasonable, so we've been exploring ways to fold shared internal content into other modules. The best technique we've come up with so far is that of _module views_. Herewith a brief introduction. Motivation ---------- About a third of the modules in the current prototype build exist only because they define types used internally across multiple non-internal modules. These modules are named using the prefix "`sun`" rather than "`jdk`", and they're defined with `permits` clauses so that only specific JDK modules can reference them. A typical example is the `sun.tls` module, which contains internal implementation classes for the `javax.net.ssl` package and related APIs defined in the `jdk.tls` module. This module exists separately from `jdk.tls` because the internal classes defined within it are used by several other modules, as documented in the `permits` clause of its declaration: module sun.tls @ 8-ea { permits jdk.crypto, jdk.sunpkcs11, jdk.tls, sun.compat, sun.jndi, sun.kerberos, sun.management, sun.rmi; exports sun.security.internal.interfaces.*; exports sun.security.internal.spec.*; exports sun.security.pkcs.*; exports sun.security.ssl.*; exports sun.security.util.*; exports sun.security.x509.*; } (A summary of the current module-declaration syntax is available [here][1].) The `jdk.tls` module, which is intended for public consumption, is defined simply as: module jdk.tls @ 8-ea { requires sun.tls @ 8-ea; exports com.sun.net.ssl.*; exports java.net.*; exports javax.net.*; exports javax.net.ssl.*; exports javax.security.cert.*; } The `jdk.tls` module does not `requires public` the `sun.tls` module, therefore the internal types defined in `sun.tls` are not available to the outside world. (For illustrative purposes the module declarations shown here differ somewhat from those produced by the current prototype build, which doesn't yet support explicit export declarations.) Now in general we'd like to keep the total number of modules comprising the JDK as small as is reasonable. We're aiming to define a configurable platform built from a set of coarse-grained modules that's easy for developers to learn, straightforward for distro maintainers to package, and simple for end users and system administrators to install and upgrade. Delivering a system in which a third of the modules are strictly for internal use would not achieve that goal. Module views ------------ The essential idea of module views is to separate the definition of the content of a module from the definition of the external view of that content. We can then extend the syntax and semantics of module declarations to allow a module to define more than one view. The _content_ of a module is the set of types defined within it together with those imported from other modules via `requires` clauses. The _view_ of a module is the set of types that it exports, via `exports` clauses, and the set of modules to which those types are available, as determined by any `permits` clauses. Consider the following module declaration: module foo @ 1 { requires bar; exports foo.*; } This defines a module named "`foo`", version 1. It contains types defined locally, e.g., on the module path under the `foo` module directory, as well as all public types exported from the module `bar`. It defines a single view which exports all public types in the `foo` package to any other module. A series of `exports` and `permits` clauses at the top syntactic level of a module declaration defines the module's _default view_. Further views of a module's content can be defined using the new `view` construct, which specifies a view name together with a bracketed list of exports and permits declarations. Suppose, e.g., that our `foo` module defines some private types which need to be made available only to a third module, `baz`. We can do that by defining a `foo.internal` view: module foo @ 1 { requires bar; exports foo.*; view foo.internal { permits baz; exports foo.private.*; } } The `foo` module now defines two views. The default view, available by referencing the module name `foo`, is the same as before---it's as if the declaration also said `view foo { exports foo.*; }`. The new view, named `foo.internal`, is available only to the `baz` module. It exports all public types in the `foo.private` package. It also exports all public types in the `foo` package because the non-default views of a module inherit the exports clauses of that module's default view. A view can also declare an entry point different from that of its containing module's default view, so a single module can now define multiple related commands. For example, the declaration module commands { view cat { class org.foo.commands.Cat; } view find { class org.foo.commands.Find; } view ls { class org.foo.commands.List; } } defines three entry points, `cat`, `find`, and `ls`. A few other details: - A non-default view never has `requires` clauses. - A non-default view cannot declare its version; it inherits the version, if any, of its containing module. - The `permits` clauses, if any, in a module's default view are not inherited by other views in that module. - In addition to declaring exports and entry points, a non-default view may also declare [services][2] and aliases. Application ----------- Returning to the original motivating example, module views allow us to merge the `sun.tls` and `jdk.tls` modules into one: module jdk.tls @ 8-ea { requires local jdk.boot @ 8-ea; // Default view (available to all) exports com.sun.net.ssl.*; exports java.net.*; exports javax.net.*; exports javax.net.ssl.*; exports javax.security.cert.*; // Internal view (available only to permitted modules) view jdk.tls.internal { permits jdk.crypto, jdk.sunpkcs11, sun.compat, sun.jndi, sun.kerberos, sun.management, sun.rmi; exports sun.security.internal.interfaces.*; exports sun.security.internal.spec.*; exports sun.security.pkcs.*; exports sun.security.ssl.*; exports sun.security.util.*; exports sun.security.x509.*; } } Views also allow us to merge together modules that exist primarily to define entry points. The various compiler-like development tools `javah`, `javac`, `javap`, and `javadoc`, e.g., share a lot of code but currently each is defined in its own module since each requires its own entry point. Using views we can define them all in a single module: module jdk.devtools @ 8-ea { requires jdk.base @ 8-ea; requires jdk.compiler @ 8-ea; requires jdk.jaxp @ 8-ea; requires jdk.logging @ 8-ea; // exports external packages exports com.sun.javadoc.*; exports com.sun.source.tree.*; exports com.sun.source.util.*; exports com.sun.tools.doclets.*; view jdk.javah { class com.sun.tools.javah.Main; } view jdk.javac { class com.sun.tools.javac.Main; } view jdk.javadoc { class com.sun.tools.javadoc.Main; } view jdk.javap { class com.sun.tools.javap.Main; } view jdk.devtools.internal { ... } } Applying these two kinds of transformations to the current prototype build reduces the overall module count from 63 to 45. Open issues ----------- - How do views map to native packaging systems such as RPM or Debian? Treating a module view as a virtual package would probably work but might not scale well. Another possibility is to structure the names of non-default views so that they always include the names of their containing modules, but that turns views into second-class entities. History and status ------------------ Module views are not a new idea. The concept proposed here is very similar to that of _structures_ in the module systems of [Scheme 48][3] and [Standard ML][4]. Jon and Mandy are working to extend the Jigsaw prototype to support views, and Alex is drafting new language and class-file specification text. All of this material will be published shortly. - Mark [1]: http://openjdk.java.net/projects/jigsaw/doc/topics/grammar.html [2]: http://openjdk.java.net/projects/jigsaw/doc/topics/services.html [3]: http://s48.org/1.8/manual/manual-Z-H-5.html [4]: http://en.wikipedia.org/wiki/Standard_ML#Module_system From jesse.glick at oracle.com Wed Dec 7 14:32:39 2011 From: jesse.glick at oracle.com (Jesse Glick) Date: Wed, 07 Dec 2011 17:32:39 -0500 Subject: Module Views In-Reply-To: <20111207211702.EFF03809@eggemoggin.niobe.net> References: <20111207211702.EFF03809@eggemoggin.niobe.net> Message-ID: <4EDFE987.20904@oracle.com> On 12/07/2011 04:17 PM, mark.reinhold at oracle.com wrote: > A series of `exports` and `permits` clauses at the top syntactic level of > a module declaration defines the module's _default view_. Further views > of a module's content can be defined using the new `view` construct, > which specifies a view name together with a bracketed list of exports and > permits declarations. This feels confusing. A class may either have an implicit no-arg constructor or an explicit list of constructors, but not both. I think the same should be true of views - either you do not use the view keyword, or all exports in the module must be defined in explicit views (one of which may have the same name as the module). > It also exports all public > types in the `foo` package because the non-default views of a module > inherit the exports clauses of that module's default view. This also seems needlessly confusing, especially in light of the fact that permits are _not_ inherited. Clearer to have each view list its complete contents. > module commands { > view cat { > class org.foo.commands.Cat; > } > view find { > class org.foo.commands.Find; > } > view ls { > class org.foo.commands.List; > } > } > > defines three entry points, `cat`, `find`, and `ls`. Entry points as currently in Jigsaw strike me as very limited; they seem to only apply to main(String[]) classes, meaning they are only useful for command-line programs. A more general approach would be to define package java.lang; // ? public interface Application { void /* or int? */ main(String... arguments) throws Exception; } and permit a module to register it using the 'service' declaration. The command-line launcher in module mode would just look for one (and only one) instance of the Application service and run it, whereas an applet container would look for an Applet, etc. Legacy main classes could trivially be retrofitted. This would eliminate the need for a special 'class' declaration in module metadata. > module jdk.tls @ 8-ea { > requires local jdk.boot @ 8-ea; > exports com.sun.net.ssl.*; > exports java.net.*; > exports javax.net.*; > exports javax.net.ssl.*; > exports javax.security.cert.*; > view jdk.tls.internal { > permits jdk.crypto, jdk.sunpkcs11, sun.compat, > sun.jndi, sun.kerberos, sun.management, sun.rmi; > exports sun.security.internal.interfaces.*; > exports sun.security.internal.spec.*; > exports sun.security.pkcs.*; > exports sun.security.ssl.*; > exports sun.security.util.*; > exports sun.security.x509.*; > } > } With my suggested changes: module jdk.tls @ 8-ea { requires local jdk.boot @ 8-ea; view jdk.tls { exports com.sun.net.ssl.*; exports java.net.*; exports javax.net.*; exports javax.net.ssl.*; exports javax.security.cert.*; } view jdk.tls.internal { permits jdk.crypto, jdk.sunpkcs11, sun.compat, sun.jndi, sun.kerberos, sun.management, sun.rmi; exports sun.security.internal.interfaces.*; exports sun.security.internal.spec.*; exports sun.security.pkcs.*; exports sun.security.ssl.*; exports sun.security.util.*; exports sun.security.x509.*; } } and usage of the internal view: module jdk.crypto @ 8-ea { // assuming it needs the public types (maybe it does not): requires jdk.tls @ 8-ea; // for its special access add: requires jdk.tls.internal @ 8-ea; } > Another possibility is to structure the names of non-default views so that they always include the names of their containing modules, but that turns views into > second-class entities. I suppose you mean that in the above example, the compiled jdk.crypto module's metadata would refer to the name "jdk.tls" only, so that when packaged natively, jdk.crypto.deb would simply depend on jdk.tls.deb. This seems natural enough; the view just controls what is exposed to whom, but does not change the nature of the module-to-module dependency. Virtual dependencies introduce a lot more complexity and points of failure, though perhaps you deal with this anyway as part of the "substitution" requirement. From david.lloyd at redhat.com Wed Dec 7 18:54:28 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 07 Dec 2011 20:54:28 -0600 Subject: Module Views In-Reply-To: <20111207211702.EFF03809@eggemoggin.niobe.net> References: <20111207211702.EFF03809@eggemoggin.niobe.net> Message-ID: <4EE026E4.2010304@redhat.com> On 12/07/2011 03:17 PM, mark.reinhold at oracle.com wrote: > Applying these two kinds of transformations to the current prototype > build reduces the overall module count from 63 to 45. > > Open issues > ----------- > > - How do views map to native packaging systems such as RPM or Debian? > Treating a module view as a virtual package would probably work but > might not scale well. Another possibility is to structure the names > of non-default views so that they always include the names of their > containing modules, but that turns views into second-class entities. These cracks speak to an underlying flaw. It boils down to one question: how do you locate your module views? We can load modules O(1) from a known point on the file system if only we know the name of the module and the filesystem is structured accordingly. If you're using views to define "also known as" then the only way to know what modules are available on the system with this approach is to read every module descriptor. You could optimize this by using a global index, but frankly this would really suck, and should not be necessary; a user should never need to do any more than drop a file or files in the filesystem at the right spot to install a module. Making views always be subordinate to the module name is the only way to make this work reasonably IMO. But anyway at this point you could just as easily use a general alias concept to solve the same problem. It is my opinion that Jigsaw doesn't need more concepts - it arguably has too many already. Even in working with JBoss Modules (which is simpler than Jigsaw in many ways), I've had to do quite a lot of education to bring people up to speed on the modular way of thinking. I can't imagine justifying adding a new concept like this when you could use a more general, simpler to understand concept like module aliases instead. "This module is another name for that one" is about as easy to understand as you can get. Adding a simple override ability to this for the main class is trivial. > History and status > ------------------ > > Module views are not a new idea. The concept proposed here is very > similar to that of _structures_ in the module systems of [Scheme 48][3] > and [Standard ML][4]. > > Jon and Mandy are working to extend the Jigsaw prototype to support > views, and Alex is drafting new language and class-file specification > text. All of this material will be published shortly. -- - DML From mark.reinhold at oracle.com Wed Dec 7 22:08:46 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 07 Dec 2011 22:08:46 -0800 Subject: Module Views In-Reply-To: jesse.glick@oracle.com; Wed, 07 Dec 2011 17:32:39 EST; <4EDFE987.20904@oracle.com> Message-ID: <20111208060846.BF254809@eggemoggin.niobe.net> 2011/12/7 14:32 -0800, jesse.glick at oracle.com: > On 12/07/2011 04:17 PM, mark.reinhold at oracle.com wrote: >> A series of `exports` and `permits` clauses at the top syntactic level of >> a module declaration defines the module's _default view_. Further views >> of a module's content can be defined using the new `view` construct, >> which specifies a view name together with a bracketed list of exports and >> permits declarations. > > This feels confusing. A class may either have an implicit no-arg > constructor or an explicit list of constructors, but not both. I think > the same should be true of views - either you do not use the view > keyword, or all exports in the module must be defined in explicit views > (one of which may have the same name as the module). That would be another way to do it. We'll be in a good position to evaluate alternatives like that after we have some experience using views in anger. (Oh, and for what it's worth I'm not sure I'd rank default no-arg constructors as one of the better design choices in Java.) >> It also exports all public types in the `foo` package because the >> non-default views of a module inherit the exports clauses of that >> module's default view. > > This also seems needlessly confusing, especially in light of the fact > that permits are _not_ inherited. Clearer to have each view list its > complete contents. Implicit export-declaration inheritance is motivated by the observation that non-default views, at least in the modular JDK itself, always export everything that's in the default view. Having to repeat all those export declarations is tedious and error-prone: module jdk.tls @ 8-ea { requires local jdk.boot @ 8-ea; // Default view (available to all) exports com.sun.net.ssl.*; exports java.net.*; exports javax.net.*; exports javax.net.ssl.*; exports javax.security.cert.*; // Internal view (available only to permitted modules) view jdk.tls.internal { permits jdk.crypto, jdk.sunpkcs11, sun.compat, sun.jndi, sun.kerberos, sun.management, sun.rmi; exports com.sun.net.ssl.*; exports java.net.*; exports javax.net.*; exports javax.net.ssl.*; exports javax.security.cert.*; exports sun.security.internal.interfaces.*; exports sun.security.internal.spec.*; exports sun.security.pkcs.*; exports sun.security.ssl.*; exports sun.security.util.*; exports sun.security.x509.*; } } An alternative would be to require clients of jdk.tls.internal to import jdk.tls explicitly if they actually need types from that view, but then that gets tedious on the client side. Another alternative would be to make export-declaration inheritance explicit ("view foo extends bar"), but that seems like overkill. Again, after some actual experience I'm sure we'll come back and consider such variations. For now we needed a plausible initial design so that we could start working it through the rest of the system. > Entry points as currently in Jigsaw strike me as very limited; they seem to > only apply to main(String[]) classes, meaning they are only useful for > command-line programs. A more general approach would be to define > > package java.lang; // ? > public interface Application { > void /* or int? */ main(String... arguments) throws Exception; > } > > and permit a module to register it using the 'service' declaration. The > command-line launcher in module mode would just look for one (and only > one) instance of the Application service and run it, whereas an applet > container would look for an Applet, etc. Legacy main classes could > trivially be retrofitted. This would eliminate the need for a special > 'class' declaration in module metadata. Hmm, interesting idea. We'd still want a simple way to support legacy main methods, but that could be done in tools. Worth exploring further. >> ... >> >> Another possibility is to structure the names of non-default views so >> that they always include the names of their containing modules, but >> that turns views into second-class entities. > > I suppose you mean that in the above example, the compiled jdk.crypto > module's metadata would refer to the name "jdk.tls" only, so that when > packaged natively, jdk.crypto.deb would simply depend on > jdk.tls.deb. This seems natural enough; the view just controls what is > exposed to whom, but does not change the nature of the module-to-module > dependency. Yes, that's what I mean. > Virtual dependencies introduce a lot more complexity and > points of failure, ... Do you mean module aliases ("provides") here, or something else? - Mark From mark.reinhold at oracle.com Wed Dec 7 22:14:27 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 07 Dec 2011 22:14:27 -0800 Subject: Module Views In-Reply-To: david.lloyd@redhat.com; Wed, 07 Dec 2011 20:54:28 CST; <4EE026E4.2010304@redhat.com> Message-ID: <20111208061427.09CB2809@eggemoggin.niobe.net> 2011/12/7 18:54 -0800, david.lloyd at redhat.com: > ... > > Making views always be subordinate to the module name is the only way to make > this work reasonably IMO. But anyway at this point you could just as easily > use a general alias concept to solve the same problem. No, because views are solving an additional problem, namely that of exporting distinct sets of types to different clients of the same module. - Mark From ali.ebrahimi1781 at gmail.com Wed Dec 7 22:28:52 2011 From: ali.ebrahimi1781 at gmail.com (Ali Ebrahimi) Date: Thu, 8 Dec 2011 09:58:52 +0330 Subject: Module Views In-Reply-To: <20111208060846.BF254809@eggemoggin.niobe.net> References: <4EDFE987.20904@oracle.com> <20111208060846.BF254809@eggemoggin.niobe.net> Message-ID: Hi, On Thu, Dec 8, 2011 at 9:38 AM, wrote: > 2011/12/7 14:32 -0800, jesse.glick at oracle.com: > > On 12/07/2011 04:17 PM, mark.reinhold at oracle.com wrote: > > >> It also exports all public types in the `foo` package because the > >> non-default views of a module inherit the exports clauses of that > >> module's default view. > > > > This also seems needlessly confusing, especially in light of the fact > > that permits are _not_ inherited. Clearer to have each view list its > > complete contents. > > Implicit export-declaration inheritance is motivated by the observation > that non-default views, at least in the modular JDK itself, always export > everything that's in the default view. Having to repeat all those export > declarations is tedious and error-prone: > may be you can use view inheritance here. view jdk.tls.internal extends jdk.tls{ ... } view jdk.tls.common_exports{ ... } view jdk.tls.common_imports{ ... } view jdk.tls.nondefault extends common_exports{ ... } Ali From david.lloyd at redhat.com Thu Dec 8 07:33:11 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 08 Dec 2011 09:33:11 -0600 Subject: Module Views In-Reply-To: <20111208061427.09CB2809@eggemoggin.niobe.net> References: <20111208061427.09CB2809@eggemoggin.niobe.net> Message-ID: <4EE0D8B7.8010202@redhat.com> On 12/08/2011 12:14 AM, mark.reinhold at oracle.com wrote: > 2011/12/7 18:54 -0800, david.lloyd at redhat.com: >> ... >> >> Making views always be subordinate to the module name is the only way to make >> this work reasonably IMO. But anyway at this point you could just as easily >> use a general alias concept to solve the same problem. > > No, because views are solving an additional problem, namely that of > exporting distinct sets of types to different clients of the same > module. A module alias could be made to do this as well. To solve this problem, we create modules which simply exist to reexport a subset of another module. This is more flexible because if an API is ever changed to be implemented by someone else, the module basic structure can remain unchanged. Views seem to be too tightly bound to their defining module; to "move" a module view from one module to another (or to make the view its own module) may be significantly more disruptive. -- - DML From mark.reinhold at oracle.com Thu Dec 8 07:53:31 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 08 Dec 2011 07:53:31 -0800 Subject: Module Views In-Reply-To: david.lloyd@redhat.com; Thu, 08 Dec 2011 09:33:11 CST; <4EE0D8B7.8010202@redhat.com> Message-ID: <20111208155331.F1691F68@eggemoggin.niobe.net> 2011/12/8 7:33 -0800, david.lloyd at redhat.com: > On 12/08/2011 12:14 AM, mark.reinhold at oracle.com wrote: >> 2011/12/7 18:54 -0800, david.lloyd at redhat.com: >>> ... >>> >>> Making views always be subordinate to the module name is the only way to make >>> this work reasonably IMO. But anyway at this point you could just as easily >>> use a general alias concept to solve the same problem. >> >> No, because views are solving an additional problem, namely that of >> exporting distinct sets of types to different clients of the same >> module. > > A module alias could be made to do this as well. To solve this problem, we > create modules which simply exist to reexport a subset of another module. Sure, but that's more or less where we were before introducing views. Content-free "filter" modules just clutter up the system, increasing the total number of modules (and hence distro packages) while also making the source code less comprehensible (a point I didn't mention earlier, but which is critical to developers working on large source bases such as the JDK). > This is more flexible because if an API is ever changed to be implemented by > someone else, the module basic structure can remain unchanged. Views seem to > be too tightly bound to their defining module; to "move" a module view from one > module to another (or to make the view its own module) may be significantly > more disruptive. Whether a view is tightly or loosely bound to its defining module is an open issue. If they're loosely bound then converting a non-default view into the default view of an actual module, or vice versa, would be transparent to its original clients. - Mark From david.lloyd at redhat.com Thu Dec 8 08:07:54 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 08 Dec 2011 10:07:54 -0600 Subject: Module Views In-Reply-To: <20111208155331.F1691F68@eggemoggin.niobe.net> References: <20111208155331.F1691F68@eggemoggin.niobe.net> Message-ID: <4EE0E0DA.7070204@redhat.com> On 12/08/2011 09:53 AM, mark.reinhold at oracle.com wrote: > 2011/12/8 7:33 -0800, david.lloyd at redhat.com: >> On 12/08/2011 12:14 AM, mark.reinhold at oracle.com wrote: >>> 2011/12/7 18:54 -0800, david.lloyd at redhat.com: >>>> ... >>>> >>>> Making views always be subordinate to the module name is the only way to make >>>> this work reasonably IMO. But anyway at this point you could just as easily >>>> use a general alias concept to solve the same problem. >>> >>> No, because views are solving an additional problem, namely that of >>> exporting distinct sets of types to different clients of the same >>> module. >> >> A module alias could be made to do this as well. To solve this problem, we >> create modules which simply exist to reexport a subset of another module. > > Sure, but that's more or less where we were before introducing views. > Content-free "filter" modules just clutter up the system, increasing the > total number of modules (and hence distro packages) You /could/ bundle the modules together in one distro package (this is basically no worse than views) if you really want this "feature". But if you want to replace a "view" with another module, if they're in the same distro package you're going to have a conflict. If they're separate, you can do a one-for-one substitution of distro package, which also makes it easy for the user to decide what to install (it might even be impossible to replace the view separately if you combine them). > while also making the > source code less comprehensible (a point I didn't mention earlier, but > which is critical to developers working on large source bases such as the > JDK). I don't really understand the source code argument to be honest. It is relatively rare for a user to refer to a module by name within Java source, yet even if they do, it shouldn't make any difference which approach is taken. If you're talking about the source for the module descriptor, we've not found it to be a problem yet. Our re-exporting modules seem to be easily understandable. >> This is more flexible because if an API is ever changed to be implemented by >> someone else, the module basic structure can remain unchanged. Views seem to >> be too tightly bound to their defining module; to "move" a module view from one >> module to another (or to make the view its own module) may be significantly >> more disruptive. > > Whether a view is tightly or loosely bound to its defining module is an > open issue. If they're loosely bound then converting a non-default view > into the default view of an actual module, or vice versa, would be > transparent to its original clients. If they're that loosely bound then you might as well just go with the more flexible solution. -- - DML From Alan.Bateman at oracle.com Thu Dec 8 11:21:12 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 08 Dec 2011 19:21:12 +0000 Subject: Module Views In-Reply-To: <4EE0E0DA.7070204@redhat.com> References: <20111208155331.F1691F68@eggemoggin.niobe.net> <4EE0E0DA.7070204@redhat.com> Message-ID: <4EE10E28.8020501@oracle.com> On 08/12/2011 16:07, David M. Lloyd wrote: > > I don't really understand the source code argument to be honest. It > is relatively rare for a user to refer to a module by name within Java > source, yet even if they do, it shouldn't make any difference which > approach is taken. > > If you're talking about the source for the module descriptor, we've > not found it to be a problem yet. Our re-exporting modules seem to be > easily understandable. It's the source code layout, as in modulepath [1]. Once we get to restructuring the JDK sources then views should make it cleaner. -Alan. [1] http://openjdk.java.net/projects/jigsaw/doc/ModulesAndJavac.pdf From julien.ponge at gmail.com Fri Dec 9 09:45:32 2011 From: julien.ponge at gmail.com (Julien Ponge) Date: Fri, 9 Dec 2011 18:45:32 +0100 Subject: A report on my Jigsaw experiences Message-ID: <05C6A124EB6345B18F1952FC8794B2FE@gmail.com> Hi there, I have just posted my initial experiences while testing Jigsaw: http://live.julien.ponge.info/84822905 So far it looks good :-) Cheers -- Julien Ponge http://julien.ponge.info/ From chris.hegarty at oracle.com Mon Dec 12 08:46:25 2011 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Mon, 12 Dec 2011 16:46:25 +0000 Subject: native cmds and libs in the module library Message-ID: <4EE62FE1.70006@oracle.com> Hi, I've been looking into where native commands and native libraries get installed in the modules library. Today the files in the NATIVE_LIBS and NATIVE_CMDS sections of jmod files [1] are installed into directories in the the module library. This is not ideal for a few reasons: 1. Native commands hidden down in modules directory are not easy to find, and complicates the PATH. 2. One native library may have a dependency on another, which complicates the LD_LIBRARY_PATH. Again, having to poke around in the modules directory of each module. 3. JDK images cannot be built with simple jmod create/install. This would be nice! Currently, in the build native libs and cmds are copied into place. 4. At some point you could imagine it would be desirable to install native commands into /usr/bin. I've prototyped changes to the SimpleLibrary where, at the point of creation (jmod create), you can optionally specify a path for native libs and cmds. The creator of the library can decide where these get installed. Then, when a jmod file is installed the native cmds and libs are placed in the libraries appropriate directory. If 'jmod create' is called without these optional arguments then the current behavior is maintained, native libs and cmds are installed in each module under its 'lib' and 'bin' directories, respectively. I think this behavior may be desirable in some cases, a module has a single native library that implements some platform specific operation. http://cr.openjdk.java.net/~chegar/jigsaw/libbin_webrev.00/webrev/ Open issues: impact on native deb packages when natlibs/cmds is specified outside of the module library. Thanks, -Chris. [1] http://cr.openjdk.java.net/~mr/jigsaw/notes/module-file-format/ From mandy.chung at oracle.com Tue Dec 13 13:36:41 2011 From: mandy.chung at oracle.com (mandy.chung at oracle.com) Date: Tue, 13 Dec 2011 21:36:41 +0000 Subject: hg: jigsaw/jigsaw/jdk: 3 new changesets Message-ID: <20111213213726.CE2A94769D@hg.openjdk.java.net> Changeset: 3cefacab03df Author: mchung Date: 2011-12-13 10:06 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/3cefacab03df missing sun/text/resources resource files in the base module ! make/modules/modules.config Changeset: 010a0a20e6c5 Author: mchung Date: 2011-12-13 10:36 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/010a0a20e6c5 Mismerge with jdk8 make/sun/jdbc/Makefile ! make/sun/jdbc/Makefile Changeset: 8c6429c279e7 Author: mchung Date: 2011-12-13 13:35 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/8c6429c279e7 Sounds build fails to link with ALSA in multiarch layout Fixed by mr. ! make/javax/sound/jsoundalsa/Makefile From mandy.chung at oracle.com Wed Dec 14 09:27:24 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 14 Dec 2011 09:27:24 -0800 Subject: Review request: module view support Message-ID: <4EE8DC7C.1030809@oracle.com> Jon and I have implemented an initial support for module views [1]. This is the webrev for the jdk change (remaining work listed below): http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/module-views.00/ [2] and [3] are the module-info.java files for the JDK before and after views support. Before the views support, in the current Jigsaw prototype, the JDK is comprised of 63 modules. With views, we fold the internal content (i.e. "sun.*" module) into the "jdk.*" module as a view that gets the JDK down to a total of 45 modules. There are currently 39 tools in the JDK, each has its entry point. Now multiple tools can be present in one single module and define one view with an entry point for each tool. For example, jdk.javac, jdk.javap, jdk.javadoc are different views of the jdk.devtools module. javac can be launched in module mode by specifying the view name: $ java -m jdk.javac The Catalog (and SimpleLibrary) will find a module of a given name. A name can be either a module name and a view name. To avoid the need of reading all module-info.class for each query, the simple library extends the current layout and creates a directory ${name}/${version}, for a view. It currently stores a copy of "info" (i.e. module-info.class) in the view directory and we will explore other alternatives in the future, e.g. simply store a file pointing to the path of its module. There are remaining works to be completed including: o exports o repository to support views o remove the launcher hack for multiple entry points support o add more tests Mandy [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2011-December/001800.html [2] http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/module-views.00/module-info/jdk-modules-no-views.txt [3] http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/module-views.00/module-info/jdk-modules-with-views.txt From alex.buckley at oracle.com Wed Dec 14 15:26:28 2011 From: alex.buckley at oracle.com (Alex Buckley) Date: Wed, 14 Dec 2011 15:26:28 -0800 Subject: Module Views In-Reply-To: <20111207211702.EFF03809@eggemoggin.niobe.net> References: <20111207211702.EFF03809@eggemoggin.niobe.net> Message-ID: <4EE930A4.8070305@oracle.com> On 12/7/2011 1:17 PM, mark.reinhold at oracle.com wrote: > History and status > ------------------ > > Jon and Mandy are working to extend the Jigsaw prototype to support > views, and Alex is drafting new language and class-file specification > text. All of this material will be published shortly. The language and ClassFile spec for module declarations (inc. support for views) is available at: http://openjdk.java.net/projects/jigsaw/doc/lang-vm.html Alex From david.bosschaert at gmail.com Thu Dec 15 05:08:57 2011 From: david.bosschaert at gmail.com (David Bosschaert) Date: Thu, 15 Dec 2011 13:08:57 +0000 Subject: Module Views In-Reply-To: <4EE930A4.8070305@oracle.com> References: <20111207211702.EFF03809@eggemoggin.niobe.net> <4EE930A4.8070305@oracle.com> Message-ID: On 14 December 2011 23:26, Alex Buckley wrote: > The language and ClassFile spec for module declarations (inc. support for > views) is available at: > > http://openjdk.java.net/projects/jigsaw/doc/lang-vm.html > > Alex I still strongly believe that the it's not right to define module metadata in Java source files and have it end up in compiled .class files inside jars. I thought that this decision was going to be made by the modularity JSR so I am worried that the changes like this start appearing in the VM and language design itself without having consulted the JSR about it. The module metadata should be: * extensible. So it can be used and extended by other module systems (such as OSGi) to store additional metadata as well as by tools that may need to store additional information alongside the module definitions. To me the use of java source files and .class binary files seem to give very little room for extensibility. * easily readable by both machines and humans. Why tools and other module systems bust be able to read them is obvious, but experience of working with modules has shown that it can be invaluable for a person to look at the module metadata, esp in troubleshooting situations. .class files are not suitable for reading by humans and will be awkward for tools and other module systems to read too. To me it seems ilke the use of XML would be a good way to represent module metadata. Its highly extensible (you can add custom-namespaced tags anywhere), and quite readable but both humans and machines. But instead of XML another textual format might also do, if properly designed. Best regards, David Bosschaert From Alan.Bateman at oracle.com Thu Dec 15 05:31:20 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 15 Dec 2011 13:31:20 +0000 Subject: Module Views In-Reply-To: References: <20111207211702.EFF03809@eggemoggin.niobe.net> <4EE930A4.8070305@oracle.com> Message-ID: <4EE9F6A8.4010506@oracle.com> On 15/12/2011 13:08, David Bosschaert wrote: > : > I still strongly believe that the it's not right to define module > metadata in Java source files and have it end up in compiled .class > files inside jars. I thought that this decision was going to be made > by the modularity JSR so I am worried that the changes like this start > appearing in the VM and language design itself without having > consulted the JSR about it. I don't know if you've seen the requirements document (which I believe will be a starting point for the JSR) but it captures the divergent opinions on this topic: http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#_C > > : > > To me it seems ilke the use of XML would be a good way to represent > module metadata. Its highly extensible (you can add custom-namespaced > tags anywhere), and quite readable but both humans and machines. > One thing to mention is that with the platform modularization then it's possible that there will not be XML support in the base module (and the module system needs to work with just the base module installed, say in footprint constrained embedded environment for example). -Alan. From david.lloyd at redhat.com Thu Dec 15 08:09:53 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 15 Dec 2011 10:09:53 -0600 Subject: Module Views In-Reply-To: <4EE9F6A8.4010506@oracle.com> References: <20111207211702.EFF03809@eggemoggin.niobe.net> <4EE930A4.8070305@oracle.com> <4EE9F6A8.4010506@oracle.com> Message-ID: <4EEA1BD1.5060505@redhat.com> On 12/15/2011 07:31 AM, Alan Bateman wrote: > On 15/12/2011 13:08, David Bosschaert wrote: >> : >> I still strongly believe that the it's not right to define module >> metadata in Java source files and have it end up in compiled .class >> files inside jars. I thought that this decision was going to be made >> by the modularity JSR so I am worried that the changes like this start >> appearing in the VM and language design itself without having >> consulted the JSR about it. > I don't know if you've seen the requirements document (which I believe > will be a starting point for the JSR) but it captures the divergent > opinions on this topic: > > http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#_C This is really just a political dodge though. Without a JSR, this is all still being shoved down our throats. I'm worried that since outside opinion is being ignored, we're going to end up with another java.util.logging - something developed in isolation which nobody uses, because frankly, it sucks. *Where is the JSR?* -- - DML From mark.reinhold at oracle.com Thu Dec 15 11:15:22 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 15 Dec 2011 11:15:22 -0800 Subject: Module Views In-Reply-To: david.bosschaert@gmail.com; Thu, 15 Dec 2011 13:08:57 GMT; Message-ID: <20111215191522.40E00F21@eggemoggin.niobe.net> 2011/12/15 5:08 -0800, david.bosschaert at gmail.com: > I still strongly believe that the it's not right to define module > metadata in Java source files and have it end up in compiled .class > files inside jars. I thought that this decision was going to be made > by the modularity JSR so I am worried that the changes like this start > appearing in the VM and language design itself without having > consulted the JSR about it. As I've said many times before: The work we're doing in Project Jigsaw is at this point still exploratory. It is not an RI for any JSR, it's a prototype in which we're investigating various ideas in the context of the agreed requirements. Some of these ideas diverge from module systems which have been built on top of Java rather than within it --- and that's a good thing. The issue of metadata formats, along with all other specification issues, will ultimately be settled by the EG of the forthcoming module-system JSR. I expect to submit that JSR early next year. - Mark From Alan.Bateman at oracle.com Fri Dec 16 07:54:00 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 16 Dec 2011 15:54:00 +0000 Subject: services, take 1 Message-ID: <4EEB6998.6070005@oracle.com> One of the areas that I've started to look into is how services might be implemented at installation and runtime. Note that I'm not talking about dynamic services and registration but rather a simple means to install service providers as modules so that they can be found at runtime via the ServiceLoader API. As the subject line suggests, I'm sure that we will go through several iterations on this topic. To get things started, I've created a first implementation [1], intended for discussion rather than code review. With this implementation then a module that declares a dependency on a service (specified by interface or abstract class name, not versioned) is linked with the modules that declare that they provide the service. At runtime the ServiceLoader API can be used to load the implementations of the service. A couple of notes/discussion points: 1. The resolver changes are very simple for now. When a module requires a service then the names of the modules that provide that service are pushed onto the choice stack for the purposes of module selection. Resolution fails if there aren't any modules providing the service and the dependency is not optional. For now permits is ignored. If it were to apply then it would likely act as a filter. Another issue that isn't addressed is the case where several versions of a module are installed and some, but not all, provide the service. I've also ignored remote repositories for now but one could envisage modules that provide the service being candidates to download and install. 2. The changes to the linker are also very simple. Contexts that provide a service becomes service suppliers to modules that require the service. The set of service suppliers is separate to the supplying contexts (for visibility reasons). The additional maps are stored in the configuration to make them available at runtime. There will be a few other changes needed here once we get back to the run-time modulepath support. 3. ServiceLoader is updated so that when we are in module mode it goes to the module class loader to get the service implementation classes that are in both the local and remote contexts supplying services. In legacy mode, or where class loaders other than module class loaders are used, then it continues to look for service configuration files. There are many discussion points around ServiceLoader. The main one that I've been thinking about is the loadInstalled method and the other load methods when presented with the system class loader. In module modes the system class loader is the module loader that loads the class with the entry point. The approach that I've taken for now is to use the class loader of the class of the calling frame for cases where loadInstalled or load is called with the system class loader. This eliminates surprises. ServiceLoader also defines a load method that uses the context class loader and that requires further consideration as to how it might work in module mode. We will need to get more experience using ServiceLoader in module mode before deciding whether additional APIs/changes may be required. We'll also need to see how ResourceBundle will work in module mode. 4. In the current Jigsaw build the module-info classes are generated in the build so I've modified the class analyzer to generate the "requires service" and "provides service" declarations based on the dynamic config file that we have checked into the repository, and the entries in the existing service configuration files. 5. To aid migration the jar command "I" module is updated to add the provides attribute to the generated module-info class when service configuration files are present in the JAR file. That's about it for now. As I said above, this is very much a first implementation and much clean-up and discussion is required. My repository isn't quite up to date with Mandy/Jon's work on views although views providing service is working. I hope to merge/sync up soon. -Alan. [1] http://cr.openjdk.java.net/~alanb/jigsaw-services/webrev/ From david.bosschaert at gmail.com Mon Dec 19 02:30:13 2011 From: david.bosschaert at gmail.com (David Bosschaert) Date: Mon, 19 Dec 2011 10:30:13 +0000 Subject: services, take 1 In-Reply-To: <4EEB6998.6070005@oracle.com> References: <4EEB6998.6070005@oracle.com> Message-ID: I have been looking at using ServiceLoader inside OSGi and one of the biggest problems is (IMHO) that it's completely static. This means that once a service is handed out to a consumer this client has that reference forever (basically as long as that client exists). The consumer may not actually hold on to the reference, but ServiceLoader has no control over that. I think it would be really good to add a mechanism to ServiceLoader that could inform a consumer that a service is now removed. This would mean that in more dynamic environments, such as OSGi, where modules can be removed during the lifetime of the VM, it would be possible to create clients such that they won't hold on to handed out services forever... Best regards, David On 16 December 2011 15:54, Alan Bateman wrote: > > One of the areas that I've started to look into is how services might be > implemented at installation and runtime. Note that I'm not talking about > dynamic services and registration but rather a simple means to install > service providers as modules so that they can be found at runtime via the > ServiceLoader API. As the subject line suggests, I'm sure that we will go > through several iterations on this topic. > > To get things started, I've created a first implementation [1], intended for > discussion rather than code review. With this implementation then a module > that declares a dependency on a service (specified by interface or abstract > class name, not versioned) is linked with the modules that declare that they > provide the service. At runtime the ServiceLoader API can be used to load > the implementations of the service. > > A couple of notes/discussion points: > > 1. The resolver changes are very simple for now. When a module requires a > service then the names of the modules that provide that service are pushed > onto the choice stack for the purposes of module selection. Resolution fails > if there aren't any modules providing the service and the dependency is not > optional. > > For now permits is ignored. If it were to apply then it would likely act as > a filter. Another issue that isn't addressed is the case where several > versions of a module are installed and some, but not all, provide the > service. I've also ignored remote repositories for now but one could > envisage modules that provide the service being candidates to download and > install. > > 2. The changes to the linker are also very simple. Contexts that provide a > service becomes service suppliers to modules that require the service. The > set of ?service suppliers is separate to the supplying contexts (for > visibility reasons). The additional maps are stored in the configuration to > make them available at runtime. There will be a few other changes needed > here once we get back to the run-time modulepath support. > > 3. ?ServiceLoader is updated so that when we are in module mode it goes to > the module class loader to get the service implementation classes that are > in both the local and remote contexts supplying services. In legacy mode, or > where class loaders other than module class loaders are used, then it > continues to look for service configuration files. > > There are many discussion points around ServiceLoader. The main one that > I've been thinking about is the loadInstalled method and the other load > methods when presented with the system class loader. In module modes the > system class loader is the module loader that loads the class with the entry > point. The approach that I've taken for now is to use the class loader of > the class of the calling frame for cases where loadInstalled or load is > called with the system class loader. This eliminates surprises. > ServiceLoader also defines a load method that uses the context class loader > and that requires further consideration as to how it might work in module > mode. We will need to get more experience using ServiceLoader in module mode > before deciding whether additional APIs/changes may be required. We'll also > need to see how ResourceBundle will work in module mode. > > 4. In the current Jigsaw build the module-info classes are generated in the > build so I've modified the class analyzer to generate the "requires service" > and "provides service" declarations based on the dynamic config file that we > have checked into the repository, and the entries in the existing service > configuration files. > > 5. To aid migration the jar command "I" module is updated to add the > provides attribute to the generated module-info class when service > configuration files are present in the JAR file. > > That's about it for now. As I said above, this is very much a first > implementation and much clean-up and discussion is required. My repository > isn't quite up to date with Mandy/Jon's work on views although views > providing service is working. I hope to merge/sync up soon. > > -Alan. > > [1] http://cr.openjdk.java.net/~alanb/jigsaw-services/webrev/ > > From julien.ponge at gmail.com Mon Dec 19 02:41:32 2011 From: julien.ponge at gmail.com (Julien Ponge) Date: Mon, 19 Dec 2011 11:41:32 +0100 Subject: services, take 1 In-Reply-To: <4EEB6998.6070005@oracle.com> References: <4EEB6998.6070005@oracle.com> Message-ID: <61B784DA2E994F92A0E417D0852A05BF@gmail.com> Hi Alan, I have a few questions, but please keep in mind that I have only recently started looking at Jigsaw, and that I may have things wrong there and there. My questions may actually be more general than your initial email? :-) > One of the areas that I've started to look into is how services might be > implemented at installation and runtime. Note that I'm not talking about > dynamic services and registration but rather a simple means to install > service providers as modules so that they can be found at runtime via > the ServiceLoader API. As the subject line suggests, I'm sure that we > will go through several iterations on this topic. > > > Ok so my understanding is that Jigsaw at the moment (aka still an exploratory prototype) is a static modules system where modules have dependencies (names + versions). Some may also be optional. You can also now specify services exports / imports as well. Services get discovered through the the ServiceLoader facility, which is a good thing IMHO. > 2. The changes to the linker are also very simple. Contexts that provide > a service becomes service suppliers to modules that require the service. > The set of service suppliers is separate to the supplying contexts (for > visibility reasons). The additional maps are stored in the configuration > to make them available at runtime. There will be a few other changes > needed here once we get back to the run-time modulepath support. > > > Something which is not clear by looking at the scarce documentation and (empty) javadocs is how modules affect classloading. Do modules form an directed graph of classloaders underneath? Or do modules resolve within their client context classloaders? > 3. ServiceLoader is updated so that when we are in module mode it goes > to the module class loader to get the service implementation classes > that are in both the local and remote contexts supplying services. In > legacy mode, or where class loaders other than module class loaders are > used, then it continues to look for service configuration files. > > > Suppose that I have a module that does a lookup on all services implementing ServiceA. 1. How do you load another module at runtime? 2. Do you see new services via the ServiceLoader? I think a dynamic modules/services API is important to have, not necessarily as an OSGi clone either. My fear is that otherwise people will build such frameworks on top of Jigsaw because there is a need for it in app servers and such, so I'd better have a clean/simple API in Jigsaw. Cheers > -- Julien Ponge http://julien.ponge.info/ From julien.ponge at gmail.com Mon Dec 19 02:43:17 2011 From: julien.ponge at gmail.com (Julien Ponge) Date: Mon, 19 Dec 2011 11:43:17 +0100 Subject: services, take 1 In-Reply-To: References: <4EEB6998.6070005@oracle.com> Message-ID: <5C0F6D5FCA894C7CA1076555791F5E31@gmail.com> > I think it would be really good to add a mechanism to ServiceLoader > that could inform a consumer that a service is now removed. This would > mean that in more dynamic environments, such as OSGi, where modules > can be removed during the lifetime of the VM, it would be possible to > create clients such that they won't hold on to handed out services > forever... > > +1 From Alan.Bateman at oracle.com Mon Dec 19 03:54:43 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 19 Dec 2011 11:54:43 +0000 Subject: services, take 1 In-Reply-To: References: <4EEB6998.6070005@oracle.com> Message-ID: <4EEF2603.70805@oracle.com> On 19/12/2011 10:30, David Bosschaert wrote: > I have been looking at using ServiceLoader inside OSGi and one of the > biggest problems is (IMHO) that it's completely static. This means > that once a service is handed out to a consumer this client has that > reference forever (basically as long as that client exists). The > consumer may not actually hold on to the reference, but ServiceLoader > has no control over that. > > I think it would be really good to add a mechanism to ServiceLoader > that could inform a consumer that a service is now removed. This would > mean that in more dynamic environments, such as OSGi, where modules > can be removed during the lifetime of the VM, it would be possible to > create clients such that they won't hold on to handed out services > forever... > Can you expand a bit on what you mean by "remove"? Do you mean that the module loader for a module providing a service has been GC'ed, or do you mean a module providing a service is removed from a module library? Or maybe you looking to "disable" a module that a provides a service so that ServiceLoader will not return any further references to instances of that type? -Alan. From Alan.Bateman at oracle.com Mon Dec 19 04:41:58 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 19 Dec 2011 12:41:58 +0000 Subject: services, take 1 In-Reply-To: <61B784DA2E994F92A0E417D0852A05BF@gmail.com> References: <4EEB6998.6070005@oracle.com> <61B784DA2E994F92A0E417D0852A05BF@gmail.com> Message-ID: <4EEF3116.8020708@oracle.com> On 19/12/2011 10:41, Julien Ponge wrote: > Ok so my understanding is that Jigsaw at the moment (aka still an > exploratory prototype) is a static modules system where modules have > dependencies (names + versions). Some may also be optional. You can > also now specify services exports / imports as well. Services get > discovered through the the ServiceLoader facility, which is a good > thing IMHO. Conceptually they are discovered but it's relatively efficient at runtime because it is indexed in the configuration. > Something which is not clear by looking at the scarce documentation > and (empty) javadocs is how modules affect classloading. Do modules > form an directed graph of classloaders underneath? Or do modules > resolve within their client context classloaders? In the simple case there will be a module loader per module at runtime (created lazily of course). Where there are local dependencies these these modules become strongly connected and have the same module loader. > Suppose that I have a module that does a lookup on all services > implementing ServiceA. > > 1. How do you load another module at runtime? With the current patch then invoking ServiceLoader.loadInstalled(ServiceA.class) causes the module loaders for the modules that provide an implementation of ServiceA to be created if they haven't been created already. This could be done more lazily, and there's a comment in the patch as a reminder on that. > 2. Do you see new services via the ServiceLoader? > > I think a dynamic modules/services API is important to have, not > necessarily as an OSGi clone either. My fear is that otherwise people > will build such frameworks on top of Jigsaw because there is a need > for it in app servers and such, so I'd better have a clean/simple API > in Jigsaw. If an application is running and someone installs into the module library a new module that provides ServiceA then that application will not see the additional ServiceA implementations until it is re-configured and restarted. Jigsaw doesn't have container support yet but conceptually that involves generating configuration at runtime and this could mean that simultaneous applications running in the same container could observe different sets of modules that provide a particular service. -Alan. From david.bosschaert at gmail.com Mon Dec 19 04:41:56 2011 From: david.bosschaert at gmail.com (David Bosschaert) Date: Mon, 19 Dec 2011 12:41:56 +0000 Subject: services, take 1 In-Reply-To: <4EEF2603.70805@oracle.com> References: <4EEB6998.6070005@oracle.com> <4EEF2603.70805@oracle.com> Message-ID: You can't wait for the loader to be GC'ed because if there are clients that still hold on to the services provided by that module it will never get GCed. It might be easiest to explain with the way OSGi actually does this. In OSGi, modules can be stopped (there is a stop() call on the Bundle). There is a callback associated with this, which allows the module to clean up. One of the things this typically does is un-register any services provided by this module. In OSGi this service unregistration event causes further callbacks on any service consumers which means they will dereference the unregistered service. The whole thing makes it possibly to actually GC the stopped module's classloader in the first place. Maybe we can facilitate something similar with ServiceLoader, for example by allowing it to register a listener which gets called back when certain services need to be dereferenced by clients. But I'm sure other approaches are also possible. There is still the risk of badly coded clients that don't give up the service object. Maybe we could do something for those too, but in general I think that can considered to be a programming error/memory leak. Cheers, David On 19 December 2011 11:54, Alan Bateman wrote: > On 19/12/2011 10:30, David Bosschaert wrote: >> >> I have been looking at using ServiceLoader inside OSGi and one of the >> biggest problems is (IMHO) that it's completely static. This means >> that once a service is handed out to a consumer this client has that >> reference forever (basically as long as that client exists). The >> consumer may not actually hold on to the reference, but ServiceLoader >> has no control over that. >> >> I think it would be really good to add a mechanism to ServiceLoader >> that could inform a consumer that a service is now removed. This would >> mean that in more dynamic environments, such as OSGi, where modules >> can be removed during the lifetime of the VM, it would be possible to >> create clients such that they won't hold on to handed out services >> forever... >> > Can you expand a bit on what you mean by "remove"? Do you mean that the > module loader for a module providing a service has been GC'ed, or do you > mean a module providing a service is removed from a module library? Or maybe > you looking to "disable" a module that a provides a service so that > ServiceLoader will not return any further references to instances of that > type? > > -Alan. From Alan.Bateman at oracle.com Mon Dec 19 07:34:23 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 19 Dec 2011 15:34:23 +0000 Subject: native cmds and libs in the module library In-Reply-To: <4EE62FE1.70006@oracle.com> References: <4EE62FE1.70006@oracle.com> Message-ID: <4EEF597F.3030804@oracle.com> On 12/12/2011 16:46, Chris Hegarty wrote: > Hi, > > I've been looking into where native commands and native libraries get > installed in the modules library. Today the files in the NATIVE_LIBS > and NATIVE_CMDS sections of jmod files [1] are installed into > directories in the the module library. This is not ideal for a few > reasons: > > 1. Native commands hidden down in modules directory are not easy to > find, and complicates the PATH. > > 2. One native library may have a dependency on another, which > complicates the LD_LIBRARY_PATH. Again, having to poke around > in the modules directory of each module. > > 3. JDK images cannot be built with simple jmod create/install. This > would be nice! Currently, in the build native libs and cmds are > copied into place. > > 4. At some point you could imagine it would be desirable to install > native commands into /usr/bin. > > I've prototyped changes to the SimpleLibrary where, at the point of > creation (jmod create), you can optionally specify a path for native > libs and cmds. The creator of the library can decide where these get > installed. Then, when a jmod file is installed the native cmds and > libs are placed in the libraries appropriate directory. > > If 'jmod create' is called without these optional arguments then the > current behavior is maintained, native libs and cmds are installed in > each module under its 'lib' and 'bin' directories, respectively. I > think this behavior may be desirable in some cases, a module has a > single native library that implements some platform specific operation. > > http://cr.openjdk.java.net/~chegar/jigsaw/libbin_webrev.00/webrev/ The approach to optionally specify the location for commands and native libraries make sense to me. I suspect defaulting to directories in each module will need a bit of thought, at least for legacy mode. Once commands and native libraries are installed to locations outside of the per module directory then it does mean handling the case that files already exist. It also means the module library needs somewhere to track these files so that removing a module knows to remove the files in these other locations. The proposed option to jmod create are -natcmd and -natlibs but as they are long names then I assume they should be --. -Alan. From sean.mullan at oracle.com Mon Dec 19 08:23:52 2011 From: sean.mullan at oracle.com (Sean Mullan) Date: Mon, 19 Dec 2011 11:23:52 -0500 Subject: native cmds and libs in the module library In-Reply-To: <4EEF597F.3030804@oracle.com> References: <4EE62FE1.70006@oracle.com> <4EEF597F.3030804@oracle.com> Message-ID: <4EEF6518.4060004@oracle.com> On most systems, you'll probably also run into file permission issues if you aren't superuser or an administrator. Is this something you are going to detect up-front and warn the user or will the command just fail when it tries to install the files? --Sean On 12/19/11 10:34 AM, Alan Bateman wrote: > On 12/12/2011 16:46, Chris Hegarty wrote: >> Hi, >> >> I've been looking into where native commands and native libraries get >> installed in the modules library. Today the files in the NATIVE_LIBS >> and NATIVE_CMDS sections of jmod files [1] are installed into >> directories in the the module library. This is not ideal for a few >> reasons: >> >> 1. Native commands hidden down in modules directory are not easy to >> find, and complicates the PATH. >> >> 2. One native library may have a dependency on another, which >> complicates the LD_LIBRARY_PATH. Again, having to poke around >> in the modules directory of each module. >> >> 3. JDK images cannot be built with simple jmod create/install. This >> would be nice! Currently, in the build native libs and cmds are >> copied into place. >> >> 4. At some point you could imagine it would be desirable to install >> native commands into /usr/bin. >> >> I've prototyped changes to the SimpleLibrary where, at the point of >> creation (jmod create), you can optionally specify a path for native >> libs and cmds. The creator of the library can decide where these get >> installed. Then, when a jmod file is installed the native cmds and >> libs are placed in the libraries appropriate directory. >> >> If 'jmod create' is called without these optional arguments then the >> current behavior is maintained, native libs and cmds are installed in >> each module under its 'lib' and 'bin' directories, respectively. I >> think this behavior may be desirable in some cases, a module has a >> single native library that implements some platform specific operation. >> >> http://cr.openjdk.java.net/~chegar/jigsaw/libbin_webrev.00/webrev/ > The approach to optionally specify the location for commands and native > libraries make sense to me. I suspect defaulting to directories in each > module will need a bit of thought, at least for legacy mode. > > Once commands and native libraries are installed to locations outside of > the per module directory then it does mean handling the case that files > already exist. It also means the module library needs somewhere to track > these files so that removing a module knows to remove the files in these > other locations. > > The proposed option to jmod create are -natcmd and -natlibs but as they > are long names then I assume they should be --. > > -Alan. > > > From njbartlett at gmail.com Tue Dec 20 03:00:40 2011 From: njbartlett at gmail.com (Neil Bartlett) Date: Tue, 20 Dec 2011 11:00:40 +0000 Subject: services, take 1 In-Reply-To: References: <4EEB6998.6070005@oracle.com> <4EEF2603.70805@oracle.com> Message-ID: <406C199E082B4F4EBEC9FE2448ADA45B@gmail.com> I think that before dynamics, the first problem that needs to be solved is to provide hooks that will allow "findability" of services in non-flat classloading models. The lack of visibility into module classloaders was the principal challenge with ServiceLoader usage in OSGi, even more so than the lack of dynamics. Incidentally I do feel that it was a very poor decision to put ServiceLoader into the `java.*` namespace, meaning it cannot be overridden or extended. If there is scope for API modifications in JDK8 then perhaps a ServiceLoaderExtension interface could be added, which would itself be a service published through META-INF/services. Then an OSGi framework or other module system could provide extensions enabling interop between the JRE and OSGi service models. With respect to dynamics, there is an argument for adding API to support registration and unregistration, along with corresponding events and listeners, even if they are never used in Jigsaw. Frankly I don't see dynamics being relevant in Jigsaw or the JDK8 module system so long as modules are static and services are statically declared through META-INF/services/* files. When a dynamic system like OSGi must supply a service to a static system like Jigsaw, we can use the standard fallback of a proxy that internally rebinds to the real service. Regards, Neil On Monday, 19 December 2011 at 12:41, David Bosschaert wrote: > You can't wait for the loader to be GC'ed because if there are clients > that still hold on to the services provided by that module it will > never get GCed. > > It might be easiest to explain with the way OSGi actually does this. > In OSGi, modules can be stopped (there is a stop() call on the > Bundle). There is a callback associated with this, which allows the > module to clean up. One of the things this typically does is > un-register any services provided by this module. In OSGi this service > unregistration event causes further callbacks on any service consumers > which means they will dereference the unregistered service. The whole > thing makes it possibly to actually GC the stopped module's > classloader in the first place. > > Maybe we can facilitate something similar with ServiceLoader, for > example by allowing it to register a listener which gets called back > when certain services need to be dereferenced by clients. But I'm sure > other approaches are also possible. > > There is still the risk of badly coded clients that don't give up the > service object. Maybe we could do something for those too, but in > general I think that can considered to be a programming error/memory > leak. > > Cheers, > > David > > On 19 December 2011 11:54, Alan Bateman wrote: > > On 19/12/2011 10:30, David Bosschaert wrote: > > > > > > I have been looking at using ServiceLoader inside OSGi and one of the > > > biggest problems is (IMHO) that it's completely static. This means > > > that once a service is handed out to a consumer this client has that > > > reference forever (basically as long as that client exists). The > > > consumer may not actually hold on to the reference, but ServiceLoader > > > has no control over that. > > > > > > I think it would be really good to add a mechanism to ServiceLoader > > > that could inform a consumer that a service is now removed. This would > > > mean that in more dynamic environments, such as OSGi, where modules > > > can be removed during the lifetime of the VM, it would be possible to > > > create clients such that they won't hold on to handed out services > > > forever... > > > > > > Can you expand a bit on what you mean by "remove"? Do you mean that the > > module loader for a module providing a service has been GC'ed, or do you > > mean a module providing a service is removed from a module library? Or maybe > > you looking to "disable" a module that a provides a service so that > > ServiceLoader will not return any further references to instances of that > > type? > > > > -Alan. From Alan.Bateman at oracle.com Tue Dec 20 05:04:08 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 20 Dec 2011 13:04:08 +0000 Subject: services, take 1 In-Reply-To: <406C199E082B4F4EBEC9FE2448ADA45B@gmail.com> References: <4EEB6998.6070005@oracle.com> <4EEF2603.70805@oracle.com> <406C199E082B4F4EBEC9FE2448ADA45B@gmail.com> Message-ID: <4EF087C8.8010802@oracle.com> On 20/12/2011 11:00, Neil Bartlett wrote: > I think that before dynamics, the first problem that needs to be solved is to provide hooks that will allow "findability" of services in non-flat classloading models. The lack of visibility into module classloaders was the principal challenge with ServiceLoader usage in OSGi, even more so than the lack of dynamics. Incidentally I do feel that it was a very poor decision to put ServiceLoader into the `java.*` namespace, meaning it cannot be overridden or extended. > > If there is scope for API modifications in JDK8 then perhaps a ServiceLoaderExtension interface could be added, which would itself be a service published through META-INF/services. Then an OSGi framework or other module system could provide extensions enabling interop between the JRE and OSGi service models. > > With respect to dynamics, there is an argument for adding API to support registration and unregistration, along with corresponding events and listeners, even if they are never used in Jigsaw. Frankly I don't see dynamics being relevant in Jigsaw or the JDK8 module system so long as modules are static and services are statically declared through META-INF/services/* files. When a dynamic system like OSGi must supply a service to a static system like Jigsaw, we can use the standard fallback of a proxy that internally rebinds to the real service. > In a container environment with managed services then dynamic registration, life cycle events, etc. are obviously important but maybe a bit beyond (or further up the stack) from what we are doing here. On "findability" then in this first patch the linker has already linked modules that require a service to those that provide implementations of the service and this is what ServiceLoader uses at runtime. Clearly there a bit of an API mismatch and that's what we need to work through. There aren't any additions to ModuleClassLoader or other APIs at this time. -Alan. From Alan.Bateman at oracle.com Tue Dec 20 05:14:08 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 20 Dec 2011 13:14:08 +0000 Subject: native cmds and libs in the module library In-Reply-To: <4EEF6518.4060004@oracle.com> References: <4EE62FE1.70006@oracle.com> <4EEF597F.3030804@oracle.com> <4EEF6518.4060004@oracle.com> Message-ID: <4EF08A20.7030106@oracle.com> On 19/12/2011 16:23, Sean Mullan wrote: > On most systems, you'll probably also run into file permission issues if you > aren't superuser or an administrator. Is this something you are going to detect > up-front and warn the user or will the command just fail when it tries to > install the files? > > --Sean > If the "jmod create" command creates the bin and lib directories and it fails then that gives us the error. If it doesn't create the directories then it could emit a warning if they aren't writable. In any case, if the install fails for any reason then it should rollback all changes. -Alan. From chris.hegarty at oracle.com Tue Dec 20 05:22:29 2011 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 20 Dec 2011 13:22:29 +0000 Subject: native cmds and libs in the module library In-Reply-To: <4EF08A20.7030106@oracle.com> References: <4EE62FE1.70006@oracle.com> <4EEF597F.3030804@oracle.com> <4EEF6518.4060004@oracle.com> <4EF08A20.7030106@oracle.com> Message-ID: <4EF08C15.7060305@oracle.com> On 12/20/11 01:14 PM, Alan Bateman wrote: > On 19/12/2011 16:23, Sean Mullan wrote: >> On most systems, you'll probably also run into file permission issues >> if you >> aren't superuser or an administrator. Is this something you are going >> to detect >> up-front and warn the user or will the command just fail when it tries to >> install the files? >> >> --Sean >> > If the "jmod create" command creates the bin and lib directories and it > fails then that gives us the error. If it doesn't create the directories > then it could emit a warning if they aren't writable. In any case, if > the install fails for any reason then it should rollback all changes. Right, I'm in the process of making this change. Currently we only remove the module directory, we need to track out of module content and remove it if necessary. This will hopefully be useful for a 'jmod remove' command in the future too. I was planning to use the same rollback strategy if a conflict, a native library or command already exists with the same name, occurs. I'm also incorporating your feedback from yesterday. -Chris. > > -Alan. From jesse.glick at oracle.com Tue Dec 20 07:10:00 2011 From: jesse.glick at oracle.com (Jesse Glick) Date: Tue, 20 Dec 2011 10:10:00 -0500 Subject: services, take 1 In-Reply-To: <406C199E082B4F4EBEC9FE2448ADA45B@gmail.com> References: <4EEB6998.6070005@oracle.com> <4EEF2603.70805@oracle.com> <406C199E082B4F4EBEC9FE2448ADA45B@gmail.com> Message-ID: <4EF0A548.6070704@oracle.com> On 12/20/2011 06:00 AM, Neil Bartlett wrote: > provide hooks that will allow "findability" of services in non-flat classloading models. On this topic, I filed an API RFE [1] some time ago that ClassLoader.getResource{,s} returns just URL but this is insufficient for the caller to determine the originating loader, which can lead to problems in the presence of multiply loaded classes. Anyone trying to make something akin to ServiceLoader based on an unspecified ClassLoader (such as Thread.contextClassLoader) might need it. For a loader tied to a particular module system such as Jigsaw I suppose you would rather directly inspect the module hierarchy (especially if an index of available services by module is already available). That reminds me that some libraries, such as Xerces if I recall correctly, have the following curious antipattern: package org.apache.xerces.twiddling; // part of an "advanced users" API: public interface InternalDocumentTwiddler { void twiddle(Document doc); } package org.apache.xerces.twiddling; // registered in META-INF/services/org.apache.xerces.twiddling.InternalDocumentTwiddler public class StandardDocumentTwiddler implements InternalDocumentTwiddler {...} package org.apache.xerces.twiddling; public class TwiddleUtils { public static void twiddle(Document doc) { for (InternalDocumentTwiddler impl : ServiceLoader.load(InternalDocumentTwiddler.class)) { impl.twiddle(doc); } } } Of course they typically reimplement ServiceLoader so as to avoid a JDK 6 dependency, and may have some other configuration mechanism such as system properties etc., and may hardcode the class name of the standard impl rather than using META-INF/services. At any rate, this is a well-meaning attempt to allow components of the library to be replaced (or augmented) with custom services, yet still fall back to the standard implementation by default. And it works fine until you have a module system and try to load both Xerces 1.8.0 and Xerces 1.9.0 in parallel. Suddenly the context loader when asked for "org.apache.xerces.twiddling.StandardDocumentTwiddler" might load the 1.8.0 version, or the 1.9.0 version, or neither. If it loads neither, the library will fail to find its own service registered in the very same JAR. And if it loads the 1.8.0 version of StandardDocumentTwiddler when called from the 1.9.0 version of TwiddleUtils, or vice-versa, you get a mysterious ClassCastException (which ServiceLoader would wrap in a ServiceConfigurationError). The easiest fix in the library is to instantiate the known standard impl directly, reserving reflection for custom implementations; on the caller side, the workaround is ClassLoader old = Thread.currentThread().getContextClassLoader(); Thread.currentThread().setContextClassLoader(TwiddleUtils.class.getClassLoader()); try { TwiddleUtils.twiddle(doc); } finally { Thread.currentThread().setContextClassLoader(old); } The analogous code using org.openide.util.Lookup.getDefault().lookupAll(InternalDocumentTwiddler.class) works inside the NetBeans module system because there is a workaround for #6865375, and the META-INF/services handler knows to ignore service implementations assignable to the wrong copy of the service interface. [1] http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6865375 From hinkmond.wong at oracle.com Tue Dec 20 15:28:27 2011 From: hinkmond.wong at oracle.com (Hinkmond Wong) Date: Tue, 20 Dec 2011 15:28:27 -0800 Subject: services, take 1 In-Reply-To: <4EF087C8.8010802@oracle.com> References: <4EEB6998.6070005@oracle.com> <4EEF2603.70805@oracle.com> <406C199E082B4F4EBEC9FE2448ADA45B@gmail.com> <4EF087C8.8010802@oracle.com> Message-ID: <4EF11A1B.4040204@oracle.com> On 12/20/2011 5:04 AM, Alan Bateman wrote: > On 20/12/2011 11:00, Neil Bartlett wrote: >> ... >> >> With respect to dynamics, there is an argument for adding API to >> support registration and unregistration, along with corresponding >> events and listeners, even if they are never used in Jigsaw. Frankly >> I don't see dynamics being relevant in Jigsaw or the JDK8 module >> system so long as modules are static and services are statically >> declared through META-INF/services/* files. When a dynamic system >> like OSGi must supply a service to a static system like Jigsaw, we >> can use the standard fallback of a proxy that internally rebinds to >> the real service. >> > In a container environment with managed services then dynamic > registration, life cycle events, etc. are obviously important but > maybe a bit beyond (or further up the stack) from what we are doing here. For the Embedded market, it's not just about the registration and life cycle part of a container environment (such as OSGi on an embedded device) that is important, but also the minimal required runtime dynamic memory and static footprint size that is very important in being the smallest size possible (better even if it can dynamically grow and shrink depending on what services are running). When real-world embedded customers mention their desires for Jigsaw, it typically involves the ability to dynamically adjust the size of the underlying Java platform core libraries according to the needs of the runtime environment, possibly requiring the modules to be more dynamic and to minimize the platform's impact to device resources. Thanks, Hinkmond > > On "findability" then in this first patch the linker has already > linked modules that require a service to those that provide > implementations of the service and this is what ServiceLoader uses at > runtime. Clearly there a bit of an API mismatch and that's what we > need to work through. There aren't any additions to ModuleClassLoader > or other APIs at this time. > > -Alan. > > > -- Oracle Hinkmond Wong | Consulting Member of Technical Staff Phone: +1 408.276.7618 | Fax: +1 408.276.7674 Oracle Java Embedded 4210 Network Ci., M/S USCA22-rm2364 | Santa Clara, CA 95054 Green Oracle Oracle is committed to developing practices and products that help protect the environment From mark.reinhold at oracle.com Tue Dec 20 15:28:39 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 20 Dec 2011 15:28:39 -0800 Subject: DRAFT: Project Jigsaw: The Big Picture (part 1) Message-ID: <20111220232839.134977A0@eggemoggin.niobe.net> An embedded and charset-unspecified text was scrubbed... Name: jigsaw-big-picture-01.md Url: http://mail.openjdk.java.net/pipermail/jigsaw-dev/attachments/20111220/d30abc7f/jigsaw-big-picture-01.md From david.lloyd at redhat.com Tue Dec 20 20:17:58 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 20 Dec 2011 22:17:58 -0600 Subject: DRAFT: Project Jigsaw: The Big Picture (part 1) In-Reply-To: <20111220232839.134977A0@eggemoggin.niobe.net> References: <20111220232839.134977A0@eggemoggin.niobe.net> Message-ID: <4EF15DF6.60000@redhat.com> On 12/20/2011 05:28 PM, mark.reinhold at oracle.com wrote: > I've started drafting an overview of the current state of Jigsaw. > The first part is attached below, and also available on the web: > > http://cr.openjdk.java.net/~mr/jigsaw/notes/jigsaw-big-picture-01 > > Comments, questions, and suggestions welcome. Flames will be ignored. The terms 'library' and 'module path' could use a definition in that section. It should maybe be somewhat more clear that the installation of modules into a module repository (library?) requires special tooling, and that modules are not editable in-place; these are both things which will break most developers' assumptions going in. The role of Jigsaw's versioning scheme at the different phases could be clarified somewhat. It could be made more clear that Jigsaw imposes a single version scheme and thus a single module repository format (that's how it appears at least). It is worth clarifying that in the Jigsaw model, nothing is exported by default. There seems to be no provision for filtering the set of packages, classes, and/or resources which are imported from a dependency. The disposition of META-INF and resource (non-package) directories is not clear, especially in terms of export and import behavior. Negative dependencies would add a great deal of complexity; I would advise against it (think of Windows NT ACLs and their mixed allow/deny rule sets; experts knew the best practices but most people just screwed them up). One key guiding principle seems to be missing: keep it simple... -- - DML From julien.ponge at gmail.com Wed Dec 21 01:31:03 2011 From: julien.ponge at gmail.com (Julien Ponge) Date: Wed, 21 Dec 2011 10:31:03 +0100 Subject: DRAFT: Project Jigsaw: The Big Picture (part 1) In-Reply-To: <20111220232839.134977A0@eggemoggin.niobe.net> References: <20111220232839.134977A0@eggemoggin.niobe.net> Message-ID: <67525B47991342EA94BAB6FEDB227101@gmail.com> Mark, I have a bunch of those except flames. So here they are, for what they're worth. > Most applications do not need to add or remove modules dynamically at > run time, nor do they need to use multiple versions of the same > module simultaneously. The module system should be optimized for > common scenarios but also support narrowly-scoped forms of dynamic > multi-version resolution motivated by actual use cases such as, e.g., > application servers, IDEs, and test harnesses. > > Granted, a significant share of applications simply rely on avoiding ClassNotFoundException. Nevertheless there is also a significant share of applications that require some form of side-by-side versioning, dynamic (un)loading and the ability to access functional units through a service locator of some form. This is basically any kind of application that needs to manage some form of lifecycle. This is not just containers: applications deployed to containers often have to deal with dynamic, e.g., a dependent module appearing or disappearing. While I agree that optimizing for static modules matches a reality, you should not be scared of providing a sensible solution to the dynamic cases. Otherwise people will necessarily hack on top of your APIs and come up with incompatible and discussable solutions. People have adopted systems like OSGi, or came up with solutions on their own involving classloader wizardry. They may not be perfect but at the very least they do work now. Narrowly-scoped solutions may be advisable, but at the very least they must not reject prior art that was validated by real needs, especially if you put the long adoption cycles of Java releases into the balance. > The module system does not support general dynamic run-time resolution; > i.e., it is not possible to add or remove dependences or modules after an > application has started running. Sophisticated container-type programs > such as application servers, IDEs, and test harnesses can achieve the > effect of run-time resolution in a limited way by using the module-system > API to install modules into a temporary module library and then run them > from that library. > > > TODO: Finish implementing run-time module-path support. > > > TODO: Design and implement container support. No flame here, but again, this is where Jigsaw will either win or become yet another java.util.logging. Implementing a container within the JDK would be a mistake, but this does not prevent from adding the support for adding/removing modules at runtime. Do you have any public document discussing the possible approaches for your APIs contracts? I guess that you will be able to dynamically load a module like you can load a class from a classloader, but without the ability to unload then people will fall in the traps where others (OSGi?) fell: reliance of the GC to clean up, hoping that no stale references exists, etc. > A module's `exports` declarations govern the [accessibility][acc] of the > public types declared in the named packages. It is thus enforced at both > compile time, by the Java compiler, and at run time, by the virtual > machine. > > Correct me if I'm wrong, but this is the same as OSGi export clauses, right? I always felt like it was weird to have public classes that in reality are not being made visible at the package level. In OSGi this leads to JARs / bundles / modules that have public types not being really public depending on the runtime context (classpath, OSGi, etc). Why not rely on the compilation unit visibility? Like introducing a "module protected visibility" without managing it at the module metadata level? I tend to think that "module protected class Foo { }" is cleaner than "public class Foo { }" only to be made "hidden" in module-info.java by not having a corresponding exports clause. > The `public` modifier makes the types imported into `bar` from `foo` > available to any other module that depends directly upon `bar`. > It may be just me, but I don't find it explicit to have "requires public foo" meaning that the module re-exports from foo. 'reexports foo" as a separate clause may be more readable, although slightly less concise. But isn't the seminal "The Feel of Java" article all about explicit/readable over implicit/concise? :-) > In this case any other module that depends upon either `bar` or `baz` > will be able to use public types exported by `foo` without depending upon > `foo` itself Can't this lead to unexpected types visibility at runtime depending on which module was actually resolved as a dependency? Wouldn't it be useful to be defensive regarding what imports bring you in crappy modules by having the possibility of filtering? Or maybe the "permits" clause could be used just for that? > > ISSUE: Should aliases have version numbers? The syntax currently > allows them. They appear to be necessary to support refactoring by > aggregation. In popular native packaging systems, however, the natural > mapping of a module alias is to a virtual package, and virtual packages > don't have version numbers. I think that they should have version numbers, e.g. "java-base @ 1.8" vs "java-base @ >= 1.9", otherwise aliases will be a second-class citizen kind of naming scheme. > > ISSUE: Should each module in a set of modules related by local > dependence be required explicitly to permit all the other modules? > That is not the case today, but it is arguably safer. > > I think so. > A non-default view can, finally, also declare an entry point different > from that of its containing module's default view, so a single module can > define multiple related entry points. For example, the declaration > > module commands { > view cat { > class org.foo.commands.Cat; > } > view find { > class org.foo.commands.Find; > } > view ls { > class org.foo.commands.List; > } > } > > defines three entry points: `cat`, `find`, and `ls`. You may want to add a sentence and/or example to say how "java -m Foo" can pick one view or the other. > Services What you have here sounds good in principle, especially using ServiceLoader, but we again get to the point of dynamics. On one hand you have a nominal static module system and a mechanism to bind to services provided by modules in a decoupled fashion. Great. On the other hand you seem not to be willing to have full dynamic modules + services + lifecycle notifications although you seem to intend that there will be an API to still load them dynamically? which means that Jigsaw may likely end up being half-baked here? meaning that people will hack on top of that or resort to solutions like OSGi which will most likely not be 100% 2-ways compatible and have their own issues. Today is the chance to provide a clean language + JVM solution to address dynamics instead of letting people come up with hacks. Balkanization is what the Java community would rather avoid I suppose. I'm sure an open discussion such as the ones you are initiating on this list can only lead to pragmatic consensus? and eventually a much needed JSR once it stabilizes. Cheers -- Julien Ponge http://julien.ponge.info/ On Wednesday, December 21, 2011 at 12:28 AM, mark.reinhold at oracle.com wrote: > I've started drafting an overview of the current state of Jigsaw. > The first part is attached below, and also available on the web: > > http://cr.openjdk.java.net/~mr/jigsaw/notes/jigsaw-big-picture-01 > > Comments, questions, and suggestions welcome. Flames will be ignored. > > - Mark > > > Attachments: > - jigsaw-big-picture-01.md > From gnormington at vmware.com Wed Dec 21 02:35:44 2011 From: gnormington at vmware.com (Glyn Normington) Date: Wed, 21 Dec 2011 10:35:44 +0000 Subject: DRAFT: Project Jigsaw: The Big Picture (part 1) References: Message-ID: Here's a reply I didn't manage to send to the list... Regards, Glyn Begin forwarded message: > From: Glyn Normington > Subject: Re: DRAFT: Project Jigsaw: The Big Picture (part 1) > Date: 21 December 2011 09:17:15 GMT > To: mark.reinhold at oracle.com > > On 20 Dec 2011, at 23:28, mark.reinhold at oracle.com wrote: > >> I've started drafting an overview of the current state of Jigsaw. >> The first part is attached below, and also available on the web: >> >> http://cr.openjdk.java.net/~mr/jigsaw/notes/jigsaw-big-picture-01 >> >> Comments, questions, and suggestions welcome. Flames will be ignored. >> >> - Mark >> > > The draft does not mention interoperation ([1]), with other module systems in general or OSGi in particular, and there is no mention of this in the "Still to come" section. If these requirements have been dropped, it would be good to note that explicitly. > > Regards, > Glyn > [1] See section 4 of http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12 > From Alan.Bateman at oracle.com Wed Dec 21 02:51:30 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 21 Dec 2011 10:51:30 +0000 Subject: services, take 1 In-Reply-To: <4EF11A1B.4040204@oracle.com> References: <4EEB6998.6070005@oracle.com> <4EEF2603.70805@oracle.com> <406C199E082B4F4EBEC9FE2448ADA45B@gmail.com> <4EF087C8.8010802@oracle.com> <4EF11A1B.4040204@oracle.com> Message-ID: <4EF1BA32.4050904@oracle.com> On 20/12/2011 23:28, Hinkmond Wong wrote: > > : > > For the Embedded market, it's not just about the registration and life > cycle part of a container environment (such as OSGi on an embedded > device) that is important, but also the minimal required runtime > dynamic memory and static footprint size that is very important in > being the smallest size possible (better even if it can dynamically > grow and shrink depending on what services are running). With Jigsaw then module class loaders will be created and GC'ed when unreachable as you would expect. This includes modules that provide implementations of service types. There is an implementation issue/bug at moment that will prevent module loaders from being GC'ed but we're sort that out. In any case, I think part of the issue here is that the term "service" means different things to different people. When you say "services are running" then it makes me think of examples like a wireless service or http service or something that involves its own threads or resources. With the current effort here then the focus is on getting simple service provider interfaces working with the module system. I've no doubt there is there is a need to have dynamic services, service registries and other things but not clear to me that all of this needs to be the core. > > When real-world embedded customers mention their desires for Jigsaw, > it typically involves the ability to dynamically adjust the size of > the underlying Java platform core libraries according to the needs of > the runtime environment, possibly requiring the modules to be more > dynamic and to minimize the platform's impact to device resources. Clearly having a modularized platform will go a long way to meeting some of these needs. However minimizing footprint at runtime or doing space/performance tradeoffs based the available resources is going require a lot of additional work. -Alan From eric at tibco.com Wed Dec 21 05:01:01 2011 From: eric at tibco.com (Eric Johnson) Date: Wed, 21 Dec 2011 14:01:01 +0100 Subject: DRAFT: Project Jigsaw: The Big Picture (part 1) In-Reply-To: <20111220232839.134977A0@eggemoggin.niobe.net> References: <20111220232839.134977A0@eggemoggin.niobe.net> Message-ID: <4EF1D88D.2090300@tibco.com> Given the amount of discussion about resolution throughout the entire document (which I'm glad to see), it sure seems like it should be mentioned as one of the design principles. Perhaps something like: Requirements for resolution change depending on context - For compile time, resolution should favor the earliest possible version of a module. At installation time, favoring the latest possible version is likely appropriate, and further, persisting this configuration for later reuse and faster load times may be an essential characteristic. For testing scenarios the need to select a variety of combinations for compatibility checks will require recomputing resolution with each execution, possibly with additional constraints added. -Eric. On 12/21/11 12:28 AM, mark.reinhold at oracle.com wrote: > I've started drafting an overview of the current state of Jigsaw. > The first part is attached below, and also available on the web: > > http://cr.openjdk.java.net/~mr/jigsaw/notes/jigsaw-big-picture-01 > > Comments, questions, and suggestions welcome. Flames will be ignored. > > - Mark From mark.reinhold at oracle.com Wed Dec 21 10:07:27 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 21 Dec 2011 10:07:27 -0800 Subject: DRAFT: Project Jigsaw: The Big Picture (part 1) In-Reply-To: david.lloyd@redhat.com; Tue, 20 Dec 2011 22:17:58 CST; <4EF15DF6.60000@redhat.com> Message-ID: <20111221180727.B2C4D23A6@eggemoggin.niobe.net> 2011/12/20 20:17 -0800, david.lloyd at redhat.com: > The terms 'library' and 'module path' could use a definition in that section. Those terms are given brief inline definitions when introduced. They'll be explained more fully later on, in sections still being written. > It should maybe be somewhat more clear that the installation of modules into a > module repository (library?) requires special tooling, and that modules are not > editable in-place; these are both things which will break most developers' > assumptions going in. These points will be made in a forthcoming section. > The role of Jigsaw's versioning scheme at the different phases could be > clarified somewhat. It could be made more clear that Jigsaw imposes a single > version scheme and thus a single module repository format (that's how it > appears at least). > > It is worth clarifying that in the Jigsaw model, nothing is exported by > default. There seems to be no provision for filtering the set of packages, > classes, and/or resources which are imported from a dependency. > > The disposition of META-INF and resource (non-package) directories is not > clear, especially in terms of export and import behavior. I'll make a note to clarify these points. Re. single library format: Yes, there's only one. Re. filtering: You're right, there is at present no way to do that. We haven't yet seen any use cases for it. Re. META-INF and resources: Our current thinking is that these are not subject to import or export; they're just part of a module's private data. If a module needs to export resources then it can do so via the ResourceBundle facility, which will be enhanced to export resources in terms of services (though the details of that remain to be worked out). > Negative dependencies would add a great deal of complexity; I would advise > against it (think of Windows NT ACLs and their mixed allow/deny rule sets; > experts knew the best practices but most people just screwed them up). I agree they'd add complexity, and I'd prefer to avoid them unless there are strong use cases in their favor. > One key guiding principle seems to be missing: keep it simple... Well, we've tried to keep it as simple as possible given our goals and constraints. Suggestions for ways to make it simpler are welcome. Bear in mind that most developers will never need to think about permits clauses, local dependences, or views. I'll see if I can reorganize the next draft to separate the fundamental features from those that are for more advanced scenarios. - Mark From neal at gafter.com Wed Dec 21 10:33:32 2011 From: neal at gafter.com (Neal Gafter) Date: Wed, 21 Dec 2011 10:33:32 -0800 Subject: Comment on state of Project Jigsaw Message-ID: My main high-level comment on project Jigsaw is that is fails its first principle: modularity appears in the draft not to be a language construct, but at best a completely separate language from the Java programming language. This appears to be an example of Conway's law . If you think this is the best approach, then it is not clear why it is described as part of the Java programming language at all. From mark.reinhold at oracle.com Wed Dec 21 10:38:37 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 21 Dec 2011 10:38:37 -0800 Subject: DRAFT: Project Jigsaw: The Big Picture (part 1) In-Reply-To: gnormington@vmware.com; Wed, 21 Dec 2011 10:35:44 GMT; Message-ID: <20111221183837.1591B23A6@eggemoggin.niobe.net> 2011/12/21 1:17 -0800, gnormington at vmware.com: > The draft does not mention interoperation ([1]), with other module > systems in general or OSGi in particular, and there is no mention of > this in the "Still to come" section. If these requirements have been > dropped, it would be good to note that explicitly. Those requirements have not been dropped. The features intended to address them, principally the module-system API, will be covered in forthcoming sections. - Mark From pgt at pgt.de Wed Dec 21 10:55:36 2011 From: pgt at pgt.de (Papick Garcia Taboada) Date: Wed, 21 Dec 2011 19:55:36 +0100 Subject: DRAFT: Project Jigsaw: The Big Picture (part 1) Message-ID: <23A3913D-F47D-4839-B525-0AF769D58310@pgt.de> Hello Mark, hello all others reading here. Sorry for my bad english, I am not a native speaker. I am just a brazilian java developer (living in old germany for 20 years now) that did in fact have to implement some modular applications/ architectures - sometimes at compiletime, sometimes at runtime. I did some OSGi, but mostly I got away with some maven or ivy magic on compile/ build time and spring components/ configurations at runtime. I am trying to follow news on jigsaw for a while now, but I was never involved in building something any close to a jvm. I even managed to build and showcase jigsaw lately here in a conference, but again, I am not a java rock star. Unfortunately, I can't reply directly to the mail because I joined the mailing list too late. It was my understanding, that the jigsaw mailing list was for implementation matters only, not for discussing proposals/ requirements/ documents. So my reply comes out of thread. Reading the "big picture" is see more questions than answers. I simply don't understand what this draft is for. It is titled big picture, but it is draft and part 1. There is no way to address such a topic in so few lines, so my apology if I am not getting it or reading wrongly between the lines. Besides, you have a draft 12 (I did miss the 11 other ones) requirement document online that would better fit the title "big picture". The whole "we don't have a JSR" and we have some bits of information here and there is - IMHO - doing more damage than helping the cause. I share the 2008 visions (wouldn't it be cool(*), there is not a moment to loose, Mark, do you remember?). But meanwhile the world around java shifted surprisingly, and my focus is now on "there is not a moment to loose" and lesser on "wouldn't it be cool". Here some thoughts. Big picture, the design principles: (1) Modularity is a language construct (2) Module boundaries should be strongly enforced (3) Static, single-version module resolution is usually sufficient Sounds good, but (1) is the whole point of adding modularization to the jdk, or we could just stick to OSGi. A modularization that is not enforced is not (read enforcement as explained here(**) ), so we all agree on (2). That we are going to need split packages loaded in a common class loader because of the mess in the jdk (see java.util package) goes against (2). From the requirements document, draft 12: In support of platform modularization, it must be possible to declare that the types defined in a specific set of modules must be loaded by the same class loader. But (3) is something that I really do not get. Again, from the requirements document: Multi-version support should only be enabled when type collisions can be detected and reported as errors. This most likely means that it cannot be enabled for run-time linking since detecting collisions is nontrivial. This most likely means?!? Seriously ? just one version at runtime? The whole point of versioning and modularization is that we HAVE version collisions, and we are witnessing repackaging of all sorts circumventing our version conflicts. This is why we started using something like OSGi, despite of all the doing, tooling and architectural pain. If we can have only one version resolved, we aren?t getting any better than with ivy or maven dependency management and dependency mediation. By the way, the new Big picture is messing around with the terminology we are used to: from maven, spring ? and somehow from common sense ? we know ?scopes? as where to use the artifacts/ components (runtime, compile-time, test-time, etc.). Then we have different life cycles: maven defines a concise project lifecicle. Every single component model defines its own component life cycle, and regarding modules (at the same level of granularity) we have a bundle life cycle defined by OSGi. In this context, take a look at ?Phases? in the big picture? (*) http://mreinhold.org/blog/cool (**) http://www.theserverside.com/feature/OSGi-Because-Sharing-Shouldnt-Be-Painful From mark.reinhold at oracle.com Wed Dec 21 11:18:53 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 21 Dec 2011 11:18:53 -0800 (PST) Subject: DRAFT: Project Jigsaw: The Big Picture (part 1) In-Reply-To: <> <21> <2011> <14:01:01> <+0100; > <4EF1D88D.2090300@tibco.com> Message-ID: <20111221191853.8070B23A6@eggemoggin.niobe.net> 2011/12/21 5:01 -0800, eric at tibco.com: > Given the amount of discussion about resolution throughout the entire document > (which I'm glad to see), it sure seems like it should be mentioned as one of > the design principles. > > Perhaps something like: > Requirements for resolution change depending on context - For compile time, > resolution should favor the earliest possible version of a module. At > installation time, favoring the latest possible version is likely appropriate, > and further, ... I don't think this is worth elevating to the level of a design principle. Phase-dependent resolution behavior is nothing new (e.g., Maven does it), so it's not a distinguishing feature of the Jigsaw module system. - Mark From mark.reinhold at oracle.com Wed Dec 21 13:29:42 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 21 Dec 2011 13:29:42 -0800 Subject: DRAFT: Project Jigsaw: The Big Picture (part 1) In-Reply-To: julien.ponge@gmail.com; Wed, 21 Dec 2011 10:31:03 +0100; <67525B47991342EA94BAB6FEDB227101@gmail.com> Message-ID: <20111221212942.105B323A6@eggemoggin.niobe.net> 2011/12/21 1:31 -0800, julien.ponge at gmail.com: > I have a bunch of those except flames. So here they are, for what they're > worth. Thanks for your comments -- replies below. > ... > > Granted, a significant share of applications simply rely on avoiding > ClassNotFoundException. Nevertheless there is also a significant share of > applications that require some form of side-by-side versioning, dynamic (un) > loading and the ability to access functional units through a service locator of > some form. ... There are definitely important classes of applications that require fully-dynamic multi-version module resolution and service lookup with a rich lifecycle API. That's a pretty complicated programming model, however, and it's not one that most Java developers need, nor is it required in order to modularize the platform itself. In SE 8 we're therefore proposing just to solve the simpler, more-common problem, and to make sure that developers who actually need to use frameworks like OSGi can do so in a way that works well with the base platform. > ... > > Implementing a container within the JDK would be a mistake, but this does not > prevent from adding the support for adding/removing modules at runtime. Do you > have any public document discussing the possible approaches for your APIs > contracts? I completely agree that adding a container to the JDK would be a mistake, and we aren't proposing to do that. You suggest that we could still support the general dynamic loading and unloading of modules at runtime, but that would add significant complexity to both the programming model (i.e., the specification) and the implementation. The Jigsaw design is, so far, very much simpler than OSGi, and that's largely because we chose early on not to try to solve all the big problems that OSGi addresses. As to API documentation, we have some (admittedly sketchy) Javadoc right now; a forthcoming section of the "big picture" document will have more details, and we'll be fleshing out the Javadoc as we go. > A module's `exports` declarations govern the [accessibility][acc] of the > public types declared in the named packages. It is thus enforced at both > compile time, by the Java compiler, and at run time, by the virtual > machine. > > Correct me if I'm wrong, but this is the same as OSGi export clauses, right? No, it's not. Export declarations in Jigsaw are meaningful in all phases, whereas in OSGi they're pretty much just a run-time concept. They're also much stronger than in OSGi, since access to non-exported types is specifically disallowed at run time by the JVM. > I always felt like it was weird to have public classes that in reality are not > being made visible at the package level. In OSGi this leads to JARs / bundles / > modules that have public types not being really public depending on the runtime > context (classpath, OSGi, etc). > > Why not rely on the compilation unit visibility? Like introducing a "module > protected visibility" without managing it at the module metadata level? I tend > to think that "module protected class Foo { }" is cleaner than "public class > Foo { }" only to be made "hidden" in module-info.java by not having a > corresponding exports clause. We've considered that. If we were starting from scratch today, such a general module-accessibility modifier could well be the way to go. With countless lines of existing Java code out in the wild, however, our take is that if developers have to modify all their source code and rebuild their libraries and systems in order to take advantage of modularity (or, equivalently but less robustly, run tools over their existing binaries), then that would be a significant barrier to adoption. > The `public` modifier makes the types imported into `bar` from `foo` > available to any other module that depends directly upon `bar`. > > It may be just me, but I don't find it explicit to have "requires public foo" > meaning that the module re-exports from foo. 'reexports foo" as a separate > clause may be more readable, although slightly less concise. I agree that that's arguably more Java-like. It's really a matter of syntax, so we're going to go with what we have for now; this can easily be revisited later on. > In this case any other module that depends upon either `bar` or `baz` > will be able to use public types exported by `foo` without depending upon > `foo` itself > > Can't this lead to unexpected types visibility at runtime depending on which > module was actually resolved as a dependency? Wouldn't it be useful to be > defensive regarding what imports bring you in crappy modules by having the > possibility of filtering? > > Or maybe the "permits" clause could be used just for that? No, `permits` wouldn't work for that. Whether a module should be able to filter the types that it re-exports is an interesting question; I'll make a note of it. > A non-default view can, finally, also declare an entry point different > from that of its containing module's default view, ... > > You may want to add a sentence and/or example to say how "java -m Foo" can pick > one view or the other. Good point; I'll do that. > Services > > What you have here sounds good in principle, especially using ServiceLoader, > but we again get to the point of dynamics. > > On one hand you have a nominal static module system and a mechanism to bind to > services provided by modules in a decoupled fashion. Great. On the other hand > you seem not to be willing to have full dynamic modules + services + lifecycle > notifications although you seem to intend that there will be an API to still > load them dynamically? which means that Jigsaw may likely end up being > half-baked here? meaning that people will hack on top of that or resort to > solutions like OSGi which will most likely not be 100% 2-ways compatible and > have their own issues. Our aim is that Jigsaw be "baked enough" that container-type applications can be built on top of it, and can load and unload independent modular components or applications. Developers who really need rich dynamism should go use OSGi or a similar framework. - Mark From mark.reinhold at oracle.com Wed Dec 21 13:48:51 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 21 Dec 2011 13:48:51 -0800 Subject: What is a language construct? (was Re: Comment on state of Project Jigsaw) In-Reply-To: neal@gafter.com; Wed, 21 Dec 2011 10:33:32 PST; Message-ID: <20111221214851.6816E23A6@eggemoggin.niobe.net> 2011/12/21 10:33 -0800, neal at gafter.com: > My main high-level comment on project Jigsaw is that is fails its first > principle: modularity appears in the draft not to be a language construct, > but at best a completely separate language from the Java programming > language. Module declarations control the visibility and accessibility of types in all phases of development. If that's not sufficient to make them a language construct, then what is your criterion? - Mark From mark.reinhold at oracle.com Wed Dec 21 13:54:33 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 21 Dec 2011 13:54:33 -0800 Subject: DRAFT: Project Jigsaw: The Big Picture (part 1) In-Reply-To: mark.reinhold@oracle.com; Wed, 21 Dec 2011 11:18:53 PST; <20111221191853.8070B23A6@eggemoggin.niobe.net> Message-ID: <20111221215433.6DEA523A6@eggemoggin.niobe.net> 2011/12/21 11:18 -0800, mark.reinhold at oracle.com: > 2011/12/21 5:01 -0800, eric at tibco.com: >> Given the amount of discussion about resolution throughout the entire document >> (which I'm glad to see), it sure seems like it should be mentioned as one of >> the design principles. >> >> ... > > I don't think this is worth elevating to the level of a design principle. > Phase-dependent resolution behavior is nothing new (e.g., Maven does it), > so it's not a distinguishing feature of the Jigsaw module system. On further thought, and prompted by a hallway conversation with Alex, perhaps the essence of what you're getting at is that modules are relevant to all phases of development rather than just, say, to run time. That's definitely a distinguishing aspect of the Jigsaw module system; I'll work it into the design-principles section in the next draft. Thanks, - Mark From eric at tibco.com Wed Dec 21 14:11:58 2011 From: eric at tibco.com (Eric Johnson) Date: Wed, 21 Dec 2011 23:11:58 +0100 Subject: What is a language construct? (was Re: Comment on state of Project Jigsaw) In-Reply-To: <20111221214851.6816E23A6@eggemoggin.niobe.net> References: <20111221214851.6816E23A6@eggemoggin.niobe.net> Message-ID: <4EF259AE.7020801@tibco.com> Is this really just a comparison with a language like Ruby, wherein a "module" could be done without adding any additional syntax to the language? That is, I take the original point to be that "module" definitions effectively look like a new DSL, and one that requires changes to the Java compiler to accommodate, rather than one that can be accommodated as-is. Keeping true to this principle of using what is already defined for the language might lead one, for example, to leveraging the MANIFEST.MF file already defined for Java, but adding new fields to that file. Alternately, following the principle in question, it could be done as a standard Java class file leveraging static initialization and/or annotations. -Eric. On 12/21/11 10:48 PM, mark.reinhold at oracle.com wrote: > 2011/12/21 10:33 -0800, neal at gafter.com: >> My main high-level comment on project Jigsaw is that is fails its first >> principle: modularity appears in the draft not to be a language construct, >> but at best a completely separate language from the Java programming >> language. > Module declarations control the visibility and accessibility of types in > all phases of development. > > If that's not sufficient to make them a language construct, then what is > your criterion? > > - Mark From neal at gafter.com Wed Dec 21 14:15:22 2011 From: neal at gafter.com (Neal Gafter) Date: Wed, 21 Dec 2011 14:15:22 -0800 Subject: What is a language construct? (was Re: Comment on state of Project Jigsaw) In-Reply-To: <20111221214851.6816E23A6@eggemoggin.niobe.net> References: <20111221214851.6816E23A6@eggemoggin.niobe.net> Message-ID: A language construct is a syntactic construct that appears within source files of the programming language. In this case (if I understand it) the language "addition" is distinct from and never intermixed with syntax of the underlying programming language. It is therefore a distinct language. On Wed, Dec 21, 2011 at 1:48 PM, wrote: > 2011/12/21 10:33 -0800, neal at gafter.com: > > My main high-level comment on project Jigsaw is that is fails its first > > principle: modularity appears in the draft not to be a language > construct, > > but at best a completely separate language from the Java programming > > language. > > Module declarations control the visibility and accessibility of types in > all phases of development. > > If that's not sufficient to make them a language construct, then what is > your criterion? > > - Mark > From jesse.glick at oracle.com Wed Dec 21 14:23:44 2011 From: jesse.glick at oracle.com (Jesse Glick) Date: Wed, 21 Dec 2011 17:23:44 -0500 Subject: How to drain your laptop battery using Jigsaw Message-ID: <4EF25C70.2050401@oracle.com> My colleague Yarda Tulach had investigated module systems supporting version ranges, and wrote up a formal proof [1] that picking one version of each of several modules in a repository such that all dependencies are satisfied is in general NP-complete. I decided to verify that Jigsaw suffers from this problem. The attached program, run against a recent Jigsaw build (8c6429c279e7), starts consuming many seconds of CPU time per round (effectively hanging) somewhere around 9 variables (depending on the random seed), mostly running in Resolver.resolve according to a profiler. The randomized nature of the test and the clause count is designed to stress the system, of course. Possible resolutions to this problem include: 1. Drop the requirement that module dependencies may include version ranges or otherwise permit incompatible changes to be marked, so that a dependency is always on some version or later. This seems unsatisfactory. 2. Drop the requirement that the module system pick a version of each module to load such that all dependencies are satisfied. For example: just try to load the newest available version of each module, and if any dependencies are in fact unsatisfied, report an error and ask the user to override one or more module versions manually. 3. Rewrite Resolver using something like SAT4J, or even leave it as it is, and just document that pathological cases can result in unbounded execution time. 4. Require that a new release of any module which uses an incompatible version of some dependency (even if used only internally) is itself marked incompatible. This seems unsatisfactory since it breaks encapsulation - I should not be forced to label my new widely used graphics toolkit 2.0 rather than 1.1 merely because I started to use a refactored version of some math library after 1.0 was released. Also it is unclear how you would enforce this constraint so long as Jigsaw assigns no semantics to version numbers except that they are totally ordered. 5. Permit multiple versions of a module to be loaded so long as class spaces remain consistent - resolving such configurations when needed - and only require that a new release of a module whose public signature refers to an incompatible version of some dependency is itself marked incompatible. (Whether 'requires public' is used does not matter.) More palatable than #4 but still forces a cascade of incompatibility, even when the incompatible change in the dependency is unrelated to this module's usage of it. Like #4, enforcement would require stronger version semantics. Yarda please step in if I misrepresented any of these options. [1] http://wiki.apidesign.org/wiki/LibraryReExportIsNPComplete From mark.reinhold at oracle.com Wed Dec 21 14:30:27 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 21 Dec 2011 14:30:27 -0800 Subject: What is a language construct? (was Re: Comment on state of Project Jigsaw) In-Reply-To: neal@gafter.com; Wed, 21 Dec 2011 14:15:22 PST; Message-ID: <20111221223027.33AD123A6@eggemoggin.niobe.net> 2011/12/21 14:15 -0800, neal at gafter.com: > A language construct is a syntactic construct that appears within source files > of the programming language. > > In this case (if I understand it) the language "addition" is distinct from and > never intermixed with syntax of the underlying programming language. It is > therefore a distinct language. In purely syntactic terms module declarations are no less intermixed than package, import, or type declarations [1], and I don't think anyone would argue that those are not part of the language. - Mark [1] http://openjdk.java.net/projects/jigsaw/doc/lang-vm.html#jigsaw-1.3.2 From jesse.glick at oracle.com Wed Dec 21 14:37:24 2011 From: jesse.glick at oracle.com (Jesse Glick) Date: Wed, 21 Dec 2011 17:37:24 -0500 Subject: How to drain your laptop battery using Jigsaw In-Reply-To: <4EF25C70.2050401@oracle.com> References: <4EF25C70.2050401@oracle.com> Message-ID: <4EF25FA4.1000800@oracle.com> On 12/21/2011 05:23 PM, Jesse Glick wrote: > The attached program Including inline since I am not sure the original made it through the list manager: import java.io.IOException; import java.lang.annotation.Annotation; import java.lang.module.Dependence; import java.lang.module.ModuleId; import java.lang.module.ModuleIdQuery; import java.lang.module.ModuleInfo; import java.lang.module.ModuleSystem; import java.util.ArrayList; import java.util.Collection; import java.util.Collections; import java.util.EnumSet; import java.util.HashMap; import java.util.HashSet; import java.util.List; import java.util.Map; import java.util.Random; import java.util.Set; import org.openjdk.jigsaw.Catalog; import org.openjdk.jigsaw.Configuration; import org.openjdk.jigsaw.ConfigurationException; import org.openjdk.jigsaw.Configurator; import org.openjdk.jigsaw.PathContext; public class Jigsaw3SAT { public static void main(String[] args) throws Exception { Random rand = new Random(); ModuleSystem ms = ModuleSystem.base(); for (int vars = 3; ; vars++) { for (int round = 0; round < 5; round++) { int clauses = (int) (vars * 4.24); Collection r = new ArrayList<>(); List varNums = new ArrayList<>(); for (int i = 0 ; i < vars; i++) { varNums.add(i); for (int j = 0; j < 2; j++) { r.add(new MockModule(ms.parseModuleId("v" + i + " @ " + j))); } } ModuleIdQuery[] tReqs = new ModuleIdQuery[clauses]; for (int i = 0; i < clauses; i++) { Collections.shuffle(varNums, rand); for (int j = 0; j < 3; j++) { r.add(new MockModule(ms.parseModuleId("f" + i + " @ " + j), ms.parseModuleIdQuery("v" + varNums.get(j) + "@" + rand.nextInt(2)))); } tReqs[i] = ms.parseModuleIdQuery("f" + i); } r.add(new MockModule(ms.parseModuleId("t"), tReqs)); System.out.println("with " + vars + " variables..."); long start = System.currentTimeMillis(); try { Configuration cfg = Configurator.configurePaths(new MockCatalog(r), Collections.singleton(ms.parseModuleIdQuery("t"))); long end = System.currentTimeMillis(); System.out.printf("solved in %.3fs:\n", (0.0 + end - start) / 1000); for (int i = 0 ; i < vars; i++) { System.out.println("v" + i + "=" + cfg.getContextForModuleName("v" + i).modules().iterator().next().version()); } } catch (ConfigurationException x) { long end = System.currentTimeMillis(); System.out.printf("unsatisfiable in %.3fs\n", (0.0 + end - start) / 1000); } System.out.println(); } } } static class MockModule implements ModuleInfo { final ModuleId id; final Set r; MockModule(ModuleId id, ModuleIdQuery... reqs) { this.id = id; r = new HashSet<>(); for (ModuleIdQuery req : reqs) { r.add(new Dependence(EnumSet.noneOf(Dependence.Modifier.class), req)); } } @Override public ModuleId id() { return id; } @Override public Set requires() { return r; } @Override public Set provides() { return Collections.emptySet(); } @Override public Set permits() { return Collections.emptySet(); } @Override public String mainClass() { return null; } @Override public boolean isAnnotationPresent(Class annotationClass) { return false; } @Override public A getAnnotation(Class annotationType) { return null; } } static class MockCatalog extends Catalog { final Map> byName; final Map byId; MockCatalog(Collection modules) { byName = new HashMap<>(); byId = new HashMap<>(); for (ModuleInfo m : modules) { ModuleId id = m.id(); String n = id.name(); Set ids = byName.get(n); if (ids == null) { ids = new HashSet<>(); byName.put(n, ids); } ids.add(id); byId.put(id, m); } } @Override public String name() { return "test"; } @Override public Catalog parent() { return null; } @Override protected void gatherLocalModuleIds(String moduleName, Set mids) throws IOException { mids.addAll(byName.get(moduleName)); } @Override protected ModuleInfo readLocalModuleInfo(ModuleId mid) throws IOException { return byId.get(mid); } } private Jigsaw3SAT() {} } From forax at univ-mlv.fr Wed Dec 21 14:39:37 2011 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Wed, 21 Dec 2011 23:39:37 +0100 Subject: What is a language construct? (was Re: Comment on state of Project Jigsaw) In-Reply-To: <20111221223027.33AD123A6@eggemoggin.niobe.net> References: <20111221223027.33AD123A6@eggemoggin.niobe.net> Message-ID: <4EF26029.1050700@univ-mlv.fr> On 12/21/2011 11:30 PM, mark.reinhold at oracle.com wrote: > 2011/12/21 14:15 -0800, neal at gafter.com: >> A language construct is a syntactic construct that appears within source files >> of the programming language. >> >> In this case (if I understand it) the language "addition" is distinct from and >> never intermixed with syntax of the underlying programming language. It is >> therefore a distinct language. > In purely syntactic terms module declarations are no less intermixed than > package, import, or type declarations [1], and I don't think anyone would > argue that those are not part of the language. > > - Mark > > > [1] http://openjdk.java.net/projects/jigsaw/doc/lang-vm.html#jigsaw-1.3.2 Java is a programming languages with several sources files and jigsaw introduces a rule to find the module declaration from a source file. This rule used a convention instead of a declaration i.e. you don't have to edit all class files to declare that they are part of a module but you have to use a precise source layout. If this layout is defined in the language spec so I don't see the problem. R?mi From hinkmond.wong at oracle.com Wed Dec 21 14:40:16 2011 From: hinkmond.wong at oracle.com (Hinkmond Wong) Date: Wed, 21 Dec 2011 14:40:16 -0800 Subject: services, take 1 In-Reply-To: <4EF1BA32.4050904@oracle.com> References: <4EEB6998.6070005@oracle.com> <4EEF2603.70805@oracle.com> <406C199E082B4F4EBEC9FE2448ADA45B@gmail.com> <4EF087C8.8010802@oracle.com> <4EF11A1B.4040204@oracle.com> <4EF1BA32.4050904@oracle.com> Message-ID: <4EF26050.2060507@oracle.com> On 12/21/2011 2:51 AM, Alan Bateman wrote: > On 20/12/2011 23:28, Hinkmond Wong wrote: >> >> : >> >> For the Embedded market, it's not just about the registration and >> life cycle part of a container environment (such as OSGi on an >> embedded device) that is important, but also the minimal required >> runtime dynamic memory and static footprint size that is very >> important in being the smallest size possible (better even if it can >> dynamically grow and shrink depending on what services are running). > With Jigsaw then module class loaders will be created and GC'ed when > unreachable as you would expect. This includes modules that provide > implementations of service types. There is an implementation issue/bug > at moment that will prevent module loaders from being GC'ed but we're > sort that out. > > In any case, I think part of the issue here is that the term "service" > means different things to different people. When you say "services are > running" then it makes me think of examples like a wireless service or > http service or something that involves its own threads or resources. > With the current effort here then the focus is on getting simple > service provider interfaces working with the module system. I've no > doubt there is there is a need to have dynamic services, service > registries and other things but not clear to me that all of this needs > to be the core. Agreed. I think more feedback from customers and developers (such as comments from here on this alias) will help greatly in shaping what is needed for dynamic services and dynamic service registries, in what is required to utilize the Jigsaw core to its fullest potential (especially on resource constrained embedded devices). >> >> When real-world embedded customers mention their desires for Jigsaw, >> it typically involves the ability to dynamically adjust the size of >> the underlying Java platform core libraries according to the needs of >> the runtime environment, possibly requiring the modules to be more >> dynamic and to minimize the platform's impact to device resources. > Clearly having a modularized platform will go a long way to meeting > some of these needs. However minimizing footprint at runtime or doing > space/performance tradeoffs based the available resources is going > require a lot of additional work. Yes, agreed, and balanced by the added-value that minimizing both dynamic and static footprint helps the Embedded Device market. (As long as those interested in this topic voice up with their feedback that is. ;-) ) Thanks, Hinkmond -- Oracle Hinkmond Wong | Consulting Member of Technical Staff Phone: +1 408.276.7618 | Fax: +1 408.276.7674 Oracle Java Embedded 4210 Network Ci., M/S USCA21-212 | Santa Clara, CA 95054 Green Oracle Oracle is committed to developing practices and products that help protect the environment From david.lloyd at redhat.com Wed Dec 21 14:40:50 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 21 Dec 2011 16:40:50 -0600 Subject: How to drain your laptop battery using Jigsaw In-Reply-To: <4EF25C70.2050401@oracle.com> References: <4EF25C70.2050401@oracle.com> Message-ID: <4EF26072.6070702@redhat.com> We (with JBoss Modules) also came to this conclusion, though not with a formal proof (awesome!)... our solution after analyzing use cases is that at run time it makes the most sense to eliminate any resolver action at all and just load modules directly as an O(1) operation. This basically implies #2 of your solution though we did not add any run-time measures to verify version constraints (we came to the conclusion that this was an install-time concern and verifying at run time was probably redundant, though we haven't completely closed the book on it either). Our linking operation basically amounts to iterating the reachable dependency graph and building a fresh linkage map for each module, with simple cycle elimination via a visited set. This has proven to be fast and effective as a module can be linked on the order of 100s of ?sec on a commodity 1.8GHz Intel system (and there are likely optimizations we haven't yet attempted)... something a resolver-based system can probably never hope to approach. On 12/21/2011 04:23 PM, Jesse Glick wrote: > My colleague Yarda Tulach had investigated module systems supporting > version ranges, and wrote up a formal proof [1] that picking one version > of each of several modules in a repository such that all dependencies > are satisfied is in general NP-complete. I decided to verify that Jigsaw > suffers from this problem. > > The attached program, run against a recent Jigsaw build (8c6429c279e7), > starts consuming many seconds of CPU time per round (effectively > hanging) somewhere around 9 variables (depending on the random seed), > mostly running in Resolver.resolve according to a profiler. The > randomized nature of the test and the clause count is designed to stress > the system, of course. > > Possible resolutions to this problem include: > > 1. Drop the requirement that module dependencies may include version > ranges or otherwise permit incompatible changes to be marked, so that a > dependency is always on some version or later. This seems unsatisfactory. > > 2. Drop the requirement that the module system pick a version of each > module to load such that all dependencies are satisfied. For example: > just try to load the newest available version of each module, and if any > dependencies are in fact unsatisfied, report an error and ask the user > to override one or more module versions manually. > > 3. Rewrite Resolver using something like SAT4J, or even leave it as it > is, and just document that pathological cases can result in unbounded > execution time. > > 4. Require that a new release of any module which uses an incompatible > version of some dependency (even if used only internally) is itself > marked incompatible. This seems unsatisfactory since it breaks > encapsulation - I should not be forced to label my new widely used > graphics toolkit 2.0 rather than 1.1 merely because I started to use a > refactored version of some math library after 1.0 was released. Also it > is unclear how you would enforce this constraint so long as Jigsaw > assigns no semantics to version numbers except that they are totally > ordered. > > 5. Permit multiple versions of a module to be loaded so long as class > spaces remain consistent - resolving such configurations when needed - > and only require that a new release of a module whose public signature > refers to an incompatible version of some dependency is itself marked > incompatible. (Whether 'requires public' is used does not matter.) More > palatable than #4 but still forces a cascade of incompatibility, even > when the incompatible change in the dependency is unrelated to this > module's usage of it. Like #4, enforcement would require stronger > version semantics. > > Yarda please step in if I misrepresented any of these options. > > > [1] http://wiki.apidesign.org/wiki/LibraryReExportIsNPComplete -- - DML From jesse.glick at oracle.com Wed Dec 21 15:07:37 2011 From: jesse.glick at oracle.com (Jesse Glick) Date: Wed, 21 Dec 2011 18:07:37 -0500 Subject: How to drain your laptop battery using Jigsaw In-Reply-To: <4EF26072.6070702@redhat.com> References: <4EF25C70.2050401@oracle.com> <4EF26072.6070702@redhat.com> Message-ID: <4EF266B9.1080207@oracle.com> On 12/21/2011 05:40 PM, David M. Lloyd wrote: > This basically implies #2 of your solution though we did not add > any run-time measures to verify version constraints (we came to the conclusion that this was an install-time concern Well the problem is essentially the same whether you interpret dependency ranges eagerly during installation of a root "application" module, or lazily the first time the application is run and caching thereafter, or simply every time it is run. Obviously a user might accept a twenty-second pause installing a module while rejecting a one-second pause when launching it, but resolution as per Jigsaw's current specification can take time exponential in the number of modules - thus (potentially) intractable even for installation. From alex.buckley at oracle.com Wed Dec 21 15:11:27 2011 From: alex.buckley at oracle.com (Alex Buckley) Date: Wed, 21 Dec 2011 15:11:27 -0800 Subject: What is a language construct? (was Re: Comment on state of Project Jigsaw) In-Reply-To: <4EF26029.1050700@univ-mlv.fr> References: <20111221223027.33AD123A6@eggemoggin.niobe.net> <4EF26029.1050700@univ-mlv.fr> Message-ID: <4EF2679F.9050408@oracle.com> On 12/21/2011 2:39 PM, R?mi Forax wrote: > Java is a programming languages with several sources files > and jigsaw introduces a rule to find the module declaration > from a source file. This rule used a convention instead of a > declaration i.e. you don't have to edit all class files > to declare that they are part of a module but you have to use > a precise source layout. > > If this layout is defined in the language spec so I don't see the problem. As Mark said, a module declaration is just another kind of declaration within a compilation unit. The JLS has never made normative statements about the physical form or layout of compilation units. (Nor, for that matter, has the JVM Spec with regard to class files). A compiler for the Java language is free to make up rules for the placement on a file system of files that represent compilation units. The file system placement of a compilation unit containing a module declaration should no more be specified in the JLS than the placement of a compilation unit containing a type declaration. I am still curious to know how Neal would integrate statements or declarations into the Java language that determine the visibility of types ("observability", in JLS terms) for code within the scope of the statements or declarations. Alex From forax at univ-mlv.fr Wed Dec 21 15:30:49 2011 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Thu, 22 Dec 2011 00:30:49 +0100 Subject: What is a language construct? (was Re: Comment on state of Project Jigsaw) In-Reply-To: <4EF2679F.9050408@oracle.com> References: <20111221223027.33AD123A6@eggemoggin.niobe.net> <4EF26029.1050700@univ-mlv.fr> <4EF2679F.9050408@oracle.com> Message-ID: <4EF26C29.1040109@univ-mlv.fr> On 12/22/2011 12:11 AM, Alex Buckley wrote: > On 12/21/2011 2:39 PM, R?mi Forax wrote: >> Java is a programming languages with several sources files >> and jigsaw introduces a rule to find the module declaration >> from a source file. This rule used a convention instead of a >> declaration i.e. you don't have to edit all class files >> to declare that they are part of a module but you have to use >> a precise source layout. >> >> If this layout is defined in the language spec so I don't see the >> problem. > > As Mark said, a module declaration is just another kind of declaration > within a compilation unit. The JLS has never made normative statements > about the physical form or layout of compilation units. (Nor, for that > matter, has the JVM Spec with regard to class files). > > A compiler for the Java language is free to make up rules for the > placement on a file system of files that represent compilation units. > The file system placement of a compilation unit containing a module > declaration should no more be specified in the JLS than the placement > of a compilation unit containing a type declaration. > > I am still curious to know how Neal would integrate statements or > declarations into the Java language that determine the visibility of > types ("observability", in JLS terms) for code within the scope of the > statements or declarations. > > Alex section 7.2.1 of the JLS3 defines the package layout, while it's presented as an example, it's in the JLS :) and Java had the chance that Visual Age was written before. R?mi From neal at gafter.com Wed Dec 21 15:50:21 2011 From: neal at gafter.com (Neal Gafter) Date: Wed, 21 Dec 2011 15:50:21 -0800 Subject: What is a language construct? (was Re: Comment on state of Project Jigsaw) In-Reply-To: <20111221223027.33AD123A6@eggemoggin.niobe.net> References: <20111221223027.33AD123A6@eggemoggin.niobe.net> Message-ID: I know how to put package (and import) declarations in a Java source file such that they apply to types declared in that source file. There does not appear to be a way to put a module declaration in a Java source file such that the module declaration applies to types declared in that source file. Module declarations are therefore very much less intermixed than package, import, and type declarations. On Wed, Dec 21, 2011 at 2:30 PM, wrote: > 2011/12/21 14:15 -0800, neal at gafter.com: > > A language construct is a syntactic construct that appears within source > files > > of the programming language. > > > > In this case (if I understand it) the language "addition" is distinct > from and > > never intermixed with syntax of the underlying programming language. It > is > > therefore a distinct language. > > In purely syntactic terms module declarations are no less intermixed than > package, import, or type declarations [1], and I don't think anyone would > argue that those are not part of the language. > > - Mark > > > [1] http://openjdk.java.net/projects/jigsaw/doc/lang-vm.html#jigsaw-1.3.2 > From aheritier at gmail.com Wed Dec 21 15:56:55 2011 From: aheritier at gmail.com (=?ISO-8859-1?Q?Arnaud_H=E9ritier?=) Date: Thu, 22 Dec 2011 00:56:55 +0100 Subject: DRAFT: Project Jigsaw: The Big Picture (part 1) In-Reply-To: <20111220232839.134977A0@eggemoggin.niobe.net> References: <20111220232839.134977A0@eggemoggin.niobe.net> Message-ID: Hi, After having participating in Maven for years I think it was a wrong choice to not have a stronger definition/enforcement of versions. It is in the central place of the resolution system and it becomes a nightmare when you have to compare Alpha, beta, RC, CR, ... and many others versions that we may not imagine. Nowadays we have also a lot of issues because of the conflict between OSGi versions and Maven SNAPSHOTs. I agree to relax some rules like having the ability to not use versions as far as you don't share modules with others, but after that when modules have to be shared I think that you need to have something really strict to propose an efficient resolution mechanism. About modules definitions it isn't clear for me if sources are/can be considered as resources and thus shared within modules. We have the case for example to share sources for GWT but in many others cases (from IDEs, ....). Related to that I see nothing maven classifiers. The major part of time this is useful especially at compile/development time to reuse some variations of a same maven artifact (sources, javadocs, ...) but when jigsaw will be out I would like to have the ability in Maven (and others build systems) to reuse jigsaw repositories thus they'll need to be extensible (the repository layout and the identification/resolution system) Thanks. Arnaud On Wed, Dec 21, 2011 at 12:28 AM, wrote: > I've started drafting an overview of the current state of Jigsaw. > The first part is attached below, and also available on the web: > > http://cr.openjdk.java.net/~mr/jigsaw/notes/jigsaw-big-picture-01 > > Comments, questions, and suggestions welcome. Flames will be ignored. > > - Mark > From neal at gafter.com Wed Dec 21 15:57:35 2011 From: neal at gafter.com (Neal Gafter) Date: Wed, 21 Dec 2011 15:57:35 -0800 Subject: What is a language construct? (was Re: Comment on state of Project Jigsaw) In-Reply-To: <4EF2679F.9050408@oracle.com> References: <20111221223027.33AD123A6@eggemoggin.niobe.net> <4EF26029.1050700@univ-mlv.fr> <4EF2679F.9050408@oracle.com> Message-ID: On Wed, Dec 21, 2011 at 3:11 PM, Alex Buckley wrote: > I am still curious to know how Neal would integrate statements or > declarations into the Java language that determine the visibility of types > ("observability", in JLS terms) for code within the scope of the statements > or declarations. > I recommend having the module system define a new form of accessibility, not "observability". The JLS has many examples of language-defined accessibility. From alex.buckley at oracle.com Wed Dec 21 16:13:23 2011 From: alex.buckley at oracle.com (Alex Buckley) Date: Wed, 21 Dec 2011 16:13:23 -0800 Subject: What is a language construct? (was Re: Comment on state of Project Jigsaw) In-Reply-To: References: <20111221223027.33AD123A6@eggemoggin.niobe.net> <4EF26029.1050700@univ-mlv.fr> <4EF2679F.9050408@oracle.com> Message-ID: <4EF27623.2050501@oracle.com> On 12/21/2011 3:57 PM, Neal Gafter wrote: > On Wed, Dec 21, 2011 at 3:11 PM, Alex Buckley > wrote: > > I am still curious to know how Neal would integrate statements or > declarations into the Java language that determine the visibility of > types ("observability", in JLS terms) for code within the scope of > the statements or declarations. > > > I recommend having the module system define a new form of accessibility, > not "observability". The JLS has many examples of language-defined > accessibility. Accessibility is a declaration-site mechanism. A provider (typically a type) defines its accessibility to the world. For the module system, we need a use-site mechanism, whereby a consumer (typically a module) defines its requirements on the rest of the world. Where do you say that? It clearly can't be up to a providing module to say all the places where it'll be observed/visible from. Alex From alex.buckley at oracle.com Wed Dec 21 16:18:01 2011 From: alex.buckley at oracle.com (Alex Buckley) Date: Wed, 21 Dec 2011 16:18:01 -0800 Subject: What is a language construct? (was Re: Comment on state of Project Jigsaw) In-Reply-To: <4EF26C29.1040109@univ-mlv.fr> References: <20111221223027.33AD123A6@eggemoggin.niobe.net> <4EF26029.1050700@univ-mlv.fr> <4EF2679F.9050408@oracle.com> <4EF26C29.1040109@univ-mlv.fr> Message-ID: <4EF27739.40104@oracle.com> On 12/21/2011 3:30 PM, R?mi Forax wrote: > section 7.2.1 of the JLS3 defines the package layout, while it's > presented as an example, it's in the JLS :) > and Java had the chance that Visual Age was written before. JLS3 7.2.1 is non-normative ("As an extremely simple example..."). It is not part of conformance testing and may as well not exist. The intertwining of normative statements and non-normative examples/discussion in the JLS was an unfortunate legacy that I have taken significant steps to fix in JLS7. (See Annex 3 of the JSR 336 Final Release. Before you ask, the PDFs and HTML will be hosted at a friendlier location, and the books will be published, sometime in 2012.) Alex From david.lloyd at redhat.com Wed Dec 21 16:44:17 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 21 Dec 2011 18:44:17 -0600 Subject: How to drain your laptop battery using Jigsaw In-Reply-To: <4EF266B9.1080207@oracle.com> References: <4EF25C70.2050401@oracle.com> <4EF26072.6070702@redhat.com> <4EF266B9.1080207@oracle.com> Message-ID: <4EF27D61.7030303@redhat.com> On 12/21/2011 05:07 PM, Jesse Glick wrote: > On 12/21/2011 05:40 PM, David M. Lloyd wrote: >> This basically implies #2 of your solution though we did not add >> any run-time measures to verify version constraints (we came to the >> conclusion that this was an install-time concern > > Well the problem is essentially the same whether you interpret > dependency ranges eagerly during installation of a root "application" > module, or lazily the first time the application is run and caching > thereafter, or simply every time it is run. Obviously a user might > accept a twenty-second pause installing a module while rejecting a > one-second pause when launching it, but resolution as per Jigsaw's > current specification can take time exponential in the number of modules > - thus (potentially) intractable even for installation. Yeah that's a good point. At most we considered a simple verification type of constraint at install (or run) time which would be a single linear scan of immediate dependencies (to verify consumption of proper versions) and immediate dependents (this would apply to install time only to ensure that the module installation wouldn't introduce an invalid dependency in another module). This keeps the algorithm simple and doesn't introduce scalability problems based upon the number of installed module. -- - DML From david.lloyd at redhat.com Wed Dec 21 16:49:29 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 21 Dec 2011 18:49:29 -0600 Subject: How to drain your laptop battery using Jigsaw In-Reply-To: <4EF27D61.7030303@redhat.com> References: <4EF25C70.2050401@oracle.com> <4EF26072.6070702@redhat.com> <4EF266B9.1080207@oracle.com> <4EF27D61.7030303@redhat.com> Message-ID: <4EF27E99.5030708@redhat.com> On 12/21/2011 06:44 PM, David M. Lloyd wrote: > On 12/21/2011 05:07 PM, Jesse Glick wrote: >> On 12/21/2011 05:40 PM, David M. Lloyd wrote: >>> This basically implies #2 of your solution though we did not add >>> any run-time measures to verify version constraints (we came to the >>> conclusion that this was an install-time concern >> >> Well the problem is essentially the same whether you interpret >> dependency ranges eagerly during installation of a root "application" >> module, or lazily the first time the application is run and caching >> thereafter, or simply every time it is run. Obviously a user might >> accept a twenty-second pause installing a module while rejecting a >> one-second pause when launching it, but resolution as per Jigsaw's >> current specification can take time exponential in the number of modules >> - thus (potentially) intractable even for installation. > > Yeah that's a good point. At most we considered a simple verification > type of constraint at install (or run) time which would be a single > linear scan of immediate dependencies (to verify consumption of proper > versions) and immediate dependents (this would apply to install time > only to ensure that the module installation wouldn't introduce an > invalid dependency in another module). This keeps the algorithm simple > and doesn't introduce scalability problems based upon the number of > installed module. I should clarify that this is all from the standpoint of not reexporting dependencies by default, which is a key facet to the originally linked article. -- - DML From neal at gafter.com Wed Dec 21 17:06:58 2011 From: neal at gafter.com (Neal Gafter) Date: Wed, 21 Dec 2011 17:06:58 -0800 Subject: What is a language construct? (was Re: Comment on state of Project Jigsaw) In-Reply-To: <4EF27623.2050501@oracle.com> References: <20111221223027.33AD123A6@eggemoggin.niobe.net> <4EF26029.1050700@univ-mlv.fr> <4EF2679F.9050408@oracle.com> <4EF27623.2050501@oracle.com> Message-ID: A use-site protection mechanism provides no protection, as one can just say "yes, it's accessible to me" at any place you want to use the thing supposedly protected. But you do want the modularity mechanism to provide a measure of protection, don't you? On Wed, Dec 21, 2011 at 4:13 PM, Alex Buckley wrote: > On 12/21/2011 3:57 PM, Neal Gafter wrote: > >> On Wed, Dec 21, 2011 at 3:11 PM, Alex Buckley > >> wrote: >> >> I am still curious to know how Neal would integrate statements or >> declarations into the Java language that determine the visibility of >> types ("observability", in JLS terms) for code within the scope of >> the statements or declarations. >> >> >> I recommend having the module system define a new form of accessibility, >> not "observability". The JLS has many examples of language-defined >> accessibility. >> > > Accessibility is a declaration-site mechanism. A provider (typically a > type) defines its accessibility to the world. For the module system, we > need a use-site mechanism, whereby a consumer (typically a module) defines > its requirements on the rest of the world. Where do you say that? It > clearly can't be up to a providing module to say all the places where it'll > be observed/visible from. > > Alex > From alex.buckley at oracle.com Wed Dec 21 17:32:12 2011 From: alex.buckley at oracle.com (Alex Buckley) Date: Wed, 21 Dec 2011 17:32:12 -0800 Subject: What is a language construct? (was Re: Comment on state of Project Jigsaw) In-Reply-To: References: <20111221223027.33AD123A6@eggemoggin.niobe.net> <4EF26029.1050700@univ-mlv.fr> <4EF2679F.9050408@oracle.com> <4EF27623.2050501@oracle.com> Message-ID: <4EF2889C.5070707@oracle.com> On 12/21/2011 5:06 PM, Neal Gafter wrote: > A use-site protection mechanism provides no protection, as one can just > say "yes, it's accessible to me" at any place you want to use the thing > supposedly protected. I'm not getting my question across. _How_ do you say this at the place you want to use the thing? If the answer is "Just say the name of the thing!", then I ask: "How do you know which module the thing comes from?" Even with package dependences rather than module dependences, you still need to say at the use-site which version of the package you want. Perhaps you are proposing "import java.util.List @ 8.0;" ? Alex From varradami at gmail.com Wed Dec 21 17:56:52 2011 From: varradami at gmail.com (Daniel Siegmann) Date: Wed, 21 Dec 2011 20:56:52 -0500 Subject: DRAFT: Project Jigsaw: The Big Picture (part 1) In-Reply-To: <20111221212942.105B323A6@eggemoggin.niobe.net> References: <67525B47991342EA94BAB6FEDB227101@gmail.com> <20111221212942.105B323A6@eggemoggin.niobe.net> Message-ID: On Wed, Dec 21, 2011 at 4:29 PM, wrote: > 2011/12/21 1:31 -0800, julien.ponge at gmail.com: >> I always felt like it was weird to have public classes that in reality are not >> being made visible at the package level. In OSGi this leads to JARs / bundles / >> modules that have public types not being really public depending on the runtime >> context (classpath, OSGi, etc). >> >> Why not rely on the compilation unit visibility? Like introducing a "module >> protected visibility" without managing it at the module metadata level? I tend >> to think that "module protected class Foo { }" is cleaner than "public class >> Foo { }" only to be made "hidden" in module-info.java by not having a >> corresponding exports clause. > > We've considered that. ?If we were starting from scratch today, such a > general module-accessibility modifier could well be the way to go. ?With > countless lines of existing Java code out in the wild, however, our take > is that if developers have to modify all their source code and rebuild > their libraries and systems in order to take advantage of modularity (or, > equivalently but less robustly, run tools over their existing binaries), > then that would be a significant barrier to adoption. This doesn't seem a very good justification to me. It is likely this system will be in place for a very long time, and eventually the amount of new code being written for JDK 8 and later will outweigh the legacy code. Why not support both approaches? A "module protected" visibility could be added and could be the recommended approach for new code which targets JDK 8 or later. Meanwhile, the visibility of public types could default to being visible outside the module, but could be restricted by package in module-info.java. Or am I overlooking something? ~Daniel From david.lloyd at redhat.com Wed Dec 21 18:29:36 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Wed, 21 Dec 2011 20:29:36 -0600 Subject: DRAFT: Project Jigsaw: The Big Picture (part 1) In-Reply-To: References: <67525B47991342EA94BAB6FEDB227101@gmail.com> <20111221212942.105B323A6@eggemoggin.niobe.net> Message-ID: <4EF29610.7050904@redhat.com> On 12/21/2011 07:56 PM, Daniel Siegmann wrote: > On Wed, Dec 21, 2011 at 4:29 PM, wrote: >> 2011/12/21 1:31 -0800, julien.ponge at gmail.com: >>> I always felt like it was weird to have public classes that in reality are not >>> being made visible at the package level. In OSGi this leads to JARs / bundles / >>> modules that have public types not being really public depending on the runtime >>> context (classpath, OSGi, etc). >>> >>> Why not rely on the compilation unit visibility? Like introducing a "module >>> protected visibility" without managing it at the module metadata level? I tend >>> to think that "module protected class Foo { }" is cleaner than "public class >>> Foo { }" only to be made "hidden" in module-info.java by not having a >>> corresponding exports clause. >> >> We've considered that. If we were starting from scratch today, such a >> general module-accessibility modifier could well be the way to go. With >> countless lines of existing Java code out in the wild, however, our take >> is that if developers have to modify all their source code and rebuild >> their libraries and systems in order to take advantage of modularity (or, >> equivalently but less robustly, run tools over their existing binaries), >> then that would be a significant barrier to adoption. > > This doesn't seem a very good justification to me. It is likely this > system will be in place for a very long time, and eventually the > amount of new code being written for JDK 8 and later will outweigh the > legacy code. > > Why not support both approaches? A "module protected" visibility could > be added and could be the recommended approach for new code which > targets JDK 8 or later. Meanwhile, the visibility of public types > could default to being visible outside the module, but could be > restricted by package in module-info.java. > > Or am I overlooking something? Be sure you're not confusing visibility with accessibility. I think Mark was talking about accessibility - you seem to be talking about both. All the major module implementations seem to implement visibility in terms of packages primarily, and classes secondarily if at all, so it's not perfectly clear how the accessibility of a class can apply to its visibility. As far as accessibility goes, I still believe that the a simple approach is best: Expand package/default access or protected access (or both) to mean "module-wide" instead of "package-wide" if the source/class file version is Java 8 or later (this can be done a number of ways; it is not my intent to expand on these details at this point though I do have a few ideas). Because the accessibility setting is already confined to a module there's no need to retroactively change source code. The package accessibility limit is not meaningful when a package is confined to a single module (as it very definitely should be) so this becomes conceptual dead weight post Java 8 anyways. Why not clean up the trash while adding a new concept in an intuitive way? This way you don't have to change a single line of code to take advantage of the feature... indeed you don't even have to take advantage of the feature at all if you're just repackaging old class files because they obviously already work with the more constrained visibility that they originally shipped with. -- - DML From jesse.glick at oracle.com Thu Dec 22 03:03:06 2011 From: jesse.glick at oracle.com (Jesse Glick) Date: Thu, 22 Dec 2011 06:03:06 -0500 Subject: Module accessibility (was: Project Jigsaw: The Big Picture (part 1)) In-Reply-To: <4EF29610.7050904@redhat.com> References: <67525B47991342EA94BAB6FEDB227101@gmail.com> <20111221212942.105B323A6@eggemoggin.niobe.net> <4EF29610.7050904@redhat.com> Message-ID: <4EF30E6A.7080801@oracle.com> On 12/21/2011 09:29 PM, David M. Lloyd wrote: > Expand package/default access or protected access (or both) to mean "module-wide" > instead of "package-wide" if the source/class file version is Java 8 or later For code targeted to 8+ and not using any (nondefault) views, I like this a lot: 1. "public" would mean what it says - to javac, to javadoc, to java; and most importantly, to someone reading an individual source file. 2. Refactoring implementation code inside a module into (or out of, or across) nested packages for clarity would become simpler because you would not need to fiddle with modifiers - all the impl code would use default access regardless of package location (keeping the code shorter too), with the obvious exception of overridden public methods. 3. The well-known problem of adding an internal-only method (called from other packages) to an exported class would disappear - just use default access. (In current Jigsaw I think you have to work around this by adding an unexported type in the exported package which delegates to a default-access method; in systems which only allow whole packages to be exported, the workaround is more intricate, involving a callback in a nonexported package and static initializers.) 4. The module-info.java semantics could permit you to omit "exports" clauses, or perhaps explicitly says "exports *", to mean that any class in any package should be accessible iff it is marked public. This avoids the bottleneck of editing module-info.java every time you place a new package into the API (thus risking merge conflicts in an SCM etc.). If you are using views to export certain packages to certain callers only, then I guess module-info.java could use explicit "exports" clauses as in the current proposal, while still taking advantage of #3 and perhaps #2. > you don't even have to take advantage of the feature at all if you're > just repackaging old class files because they obviously already work with the more constrained visibility that they originally shipped with. They would still work, but you may want to ban outside access to implementation packages despite their containing public members. For this case also, an explicit list of "exports" clauses would work if recompiling with -source 8 or using 'jar uvf *.jar module-info.class'. From david.bosschaert at gmail.com Thu Dec 22 03:51:15 2011 From: david.bosschaert at gmail.com (David Bosschaert) Date: Thu, 22 Dec 2011 11:51:15 +0000 Subject: DRAFT: Project Jigsaw: The Big Picture (part 1) In-Reply-To: <20111220232839.134977A0@eggemoggin.niobe.net> References: <20111220232839.134977A0@eggemoggin.niobe.net> Message-ID: The module definition doesn't look like Java at all to me, take this example module bar @ 1.0 { requires foo; exports bar; view bar.internal { permits baz; exports bar.private; } } so I really don't think it should be part of the java language itself. I think it will require a lot of extensions to the language and introduce 'pseudo' keywords - keywords that are only keywords inside the module section, but not elsewhere. Besides, I still like to see how this can be made extensible as required in [1]. Additionally, you say that the module declaration is by convention in a module-info.java file. I think we need something stronger than a convention here as I don't think you'd want 2 files both declaring the module definitions inside a single module. I think we should mandate a single module definition file in a well-known location such as /META-INF/module-info.jmod where the format of that module-info.jmod is either a text-based DSL (maybe similar to the module declaration above, but then extensible) or using XML. I have made this point before, but I don't feel it has been taken into consideration appropriately in the current design. Best regards, David [1] http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#extensible-module-declarations On 20 December 2011 23:28, wrote: > I've started drafting an overview of the current state of Jigsaw. > The first part is attached below, and also available on the web: > > ?http://cr.openjdk.java.net/~mr/jigsaw/notes/jigsaw-big-picture-01 > > Comments, questions, and suggestions welcome. ?Flames will be ignored. > > - Mark From sebastian.sickelmann at gmx.de Thu Dec 22 07:17:23 2011 From: sebastian.sickelmann at gmx.de (Sebastian Sickelmann) Date: Thu, 22 Dec 2011 16:17:23 +0100 Subject: DRAFT: Project Jigsaw: The Big Picture (part 1) In-Reply-To: <20111220232839.134977A0@eggemoggin.niobe.net> References: <20111220232839.134977A0@eggemoggin.niobe.net> Message-ID: <4EF34A03.9050205@gmx.de> Hi, thank you for the good big picture. I know have a somewhat better feeling what jigsaw will/can be. I have a few questions/suggestions: > A series of |exports|, |requires public|, and |permits| clauses at the top syntactic > level of a module declaration defines the module?s /default view/. Further > views of a module?s bindings can be defined using the |view| construct, >which specifies a view name together with a bracketed list of exports >and permits declarations: |>module bar { > requires foo; > exports bar; > view bar.internal { > permits baz; > exports bar.private; > } >} | >The |bar| module now defines two views. The default view, available > by referencing the module name |bar|, is the same as before?it?s > as if the declaration also said |view bar { exports bar; }|. The new view, > named |bar.internal|, is available only to the |baz| module. It exports > all public types in the |bar.private| package. It also exports all public > types in the |bar| package because the non-default views of a module > inherit the |exports| clauses of that module?s default view. >A non-default view never has |requires| clauses. >A non-default view cannot declare its version; it inherits the version, > if any, of its containing module. >A non-default view does not inherit the |permits| clauses, if any, of its > containing module. What happens in the following example? |module bar { requires foo; || permits baz; || exports barpackage; ||| | view bar.publicview { exports barpackage.public; } } Module bar only allows baz to access the module. Through the view bar.publicview every module can access barpackage and barpackage.public. | Isn't here a discrepancy not to inherit the permits clauses of the containing module? |What happens to the local-dependencies? baz can access|package-private types in barpackage. Can baz also localy depend on barpackage.public? My next question goes to reexported and version numbers: |>module foo { > exports foo; >} > >module bar { > requires public foo; >} > >module baz { > requires public bar; >} > >module buz { > requires baz; // Can also use foo's exported types >}| What would happen if definition of buz and bar are the following: |module bar { requires public foo @ 1.0; } | |module buz { requires baz; // Can also use foo's exported types requires foo @>= 2.0; } The use case is: buz want to use functionality of baz. baz also provides some functionality for which foo must be reexported. But buz is not using this functionality. But buz uses some functionlity in foo that is only supported since version 2.0 of foo. Can buz override the reexported dependency of foo at 1.0 to foo@>=2.0? We can get in the the same situation if an old version of bar doen't need foo at all. buz already uses foo @>= 2.0 Now buz bar starts using foo @ 1.0 and unintentionally reexportes foo. ||Can buz override the reexported dependency of foo at 1.0 to foo@>=2.0? | |-- Sebastian | Am 21.12.2011 00:28, schrieb mark.reinhold at oracle.com: > I've started drafting an overview of the current state of Jigsaw. > The first part is attached below, and also available on the web: > > http://cr.openjdk.java.net/~mr/jigsaw/notes/jigsaw-big-picture-01 > > Comments, questions, and suggestions welcome. Flames will be ignored. > > - Mark From david.lloyd at redhat.com Thu Dec 22 08:07:21 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 22 Dec 2011 10:07:21 -0600 Subject: Module accessibility In-Reply-To: <4EF30E6A.7080801@oracle.com> References: <67525B47991342EA94BAB6FEDB227101@gmail.com> <20111221212942.105B323A6@eggemoggin.niobe.net> <4EF29610.7050904@redhat.com> <4EF30E6A.7080801@oracle.com> Message-ID: <4EF355B9.2010804@redhat.com> On 12/22/2011 05:03 AM, Jesse Glick wrote: > They would still work, but you may want to ban outside access to > implementation packages despite their containing public members. For > this case also, an explicit list of "exports" clauses would work if > recompiling with -source 8 or using 'jar uvf *.jar module-info.class'. Just a quick note on this - it's not really clear what banning outside access would entail here. Normal accessibility rules would be a sufficient check at compile-time for sure. But at run-time, if you have a module's class loader, unless you use slow/clunky/possibly error-prone call stack introspection on every class load request you can't really enforce this visibility constraint. So pretty much all you can do to enforce this rule is: 1. Only link exported paths (which is already the case, for us at least) 2. Prevent unauthorized access to a module's class loader (via Permission) when a security manager is present (we use a general RuntimePermission for all modules but I think for the JDK usage we could/should be more fine-grained) I view module ClassLoader access as analogous to having a java.lang.reflect.Method which has its "accessible" attribute set to true. Basically anyone who has the reference the the ClassLoader has full access to the module. So yeah this means that any non-public class in a public and imported package can be *loaded* by, say, Class.forName("org.blah.Name",false,moduleClassLoader) but the normal accessibility checks would apply. The alternative is to add more run-time checks; the performance penalty could be substantial. -- - DML From jesse.glick at oracle.com Thu Dec 22 09:11:52 2011 From: jesse.glick at oracle.com (Jesse Glick) Date: Thu, 22 Dec 2011 12:11:52 -0500 Subject: Module accessibility In-Reply-To: <4EF355B9.2010804@redhat.com> References: <67525B47991342EA94BAB6FEDB227101@gmail.com> <20111221212942.105B323A6@eggemoggin.niobe.net> <4EF29610.7050904@redhat.com> <4EF30E6A.7080801@oracle.com> <4EF355B9.2010804@redhat.com> Message-ID: <4EF364D8.6040603@oracle.com> On 12/22/2011 11:07 AM, David M. Lloyd wrote: > at run-time, if you have a module's class loader [...] you > can't really enforce this visibility constraint. The class in the unexported package might be visible, but the Jigsaw-enhanced VM would ensure that it is not accessible. Just as with reflection generally, Class.forName might return a Class yet Class.newInstance (for example) would fail with an IllegalAccessException. This is in contrast to existing module systems without VM support, where you might not be able to link statically against a class in an unexported package, but you could still call it reflectively without special permissions. > I view module ClassLoader access as analogous to having a java.lang.reflect.Method which has its "accessible" attribute set to true. Again this is a description of current module systems built on older JVMs and only able to control visibility by overriding ClassLoader.loadClass to throw ClassNotFoundException even when a parent loader can load the class. As I understand it, Jigsaw would enforce the same accessibility on reflective calls as in the compiler and linker. From njbartlett at gmail.com Thu Dec 22 09:16:01 2011 From: njbartlett at gmail.com (Neil Bartlett) Date: Thu, 22 Dec 2011 17:16:01 +0000 Subject: Simplicity [was: Re: DRAFT: Project Jigsaw: The Big Picture (part 1)] Message-ID: Mark, I appreciate your effort to document the current status of Jigsaw. Try as I might, I cannot hold back from commenting on your assertion that "the Jigsaw design is, so far, very much simpler than OSGi." Obviously you will disagree, but I feel the exact opposite: the Jigsaw design is already more complicated than OSGi, especially with the recent addition of module views. If Jigsaw's goal is to create a module system that is more easily understandable than OSGi, then it is failing. This is not intended as a flame. I acknowledge that the appearance of complexity is highly subjective, and I would appreciate the opinions of people who are not already inculcated in either of the OSGi or Jigsaw camps. Regards Neil On Wed, Dec 21, 2011 at 9:29 PM, wrote: > 2011/12/21 1:31 -0800, julien.ponge at gmail.com: >> I have a bunch of those except flames. So here they are, for what they're >> worth. > > Thanks for your comments -- replies below. > >> ... >> >> Granted, a significant share of applications simply rely on avoiding >> ClassNotFoundException. Nevertheless there is also a significant share of >> applications that require some form of side-by-side versioning, dynamic (un) >> loading and the ability to access functional units through a service locator of >> some form. ... > > There are definitely important classes of applications that require > fully-dynamic multi-version module resolution and service lookup with > a rich lifecycle API. ?That's a pretty complicated programming model, > however, and it's not one that most Java developers need, nor is it > required in order to modularize the platform itself. ?In SE 8 we're > therefore proposing just to solve the simpler, more-common problem, > and to make sure that developers who actually need to use frameworks > like OSGi can do so in a way that works well with the base platform. > >> ... >> >> Implementing a container within the JDK would be a mistake, but this does not >> prevent from adding the support for adding/removing modules at runtime. Do you >> have any public document discussing the possible approaches for your APIs >> contracts? > > I completely agree that adding a container to the JDK would be a mistake, > and we aren't proposing to do that. ?You suggest that we could still > support the general dynamic loading and unloading of modules at runtime, > but that would add significant complexity to both the programming model > (i.e., the specification) and the implementation. ?The Jigsaw design is, > so far, very much simpler than OSGi, and that's largely because we chose > early on not to try to solve all the big problems that OSGi addresses. > > As to API documentation, we have some (admittedly sketchy) Javadoc right > now; a forthcoming section of the "big picture" document will have more > details, and we'll be fleshing out the Javadoc as we go. > >> ? ? A module's `exports` declarations govern the [accessibility][acc] of the >> ? ? public types declared in the named packages. ?It is thus enforced at both >> ? ? compile time, by the Java compiler, and at run time, by the virtual >> ? ? machine. >> >> Correct me if I'm wrong, but this is the same as OSGi export clauses, right? > > No, it's not. ?Export declarations in Jigsaw are meaningful in all > phases, whereas in OSGi they're pretty much just a run-time concept. > They're also much stronger than in OSGi, since access to non-exported > types is specifically disallowed at run time by the JVM. > >> I always felt like it was weird to have public classes that in reality are not >> being made visible at the package level. In OSGi this leads to JARs / bundles / >> modules that have public types not being really public depending on the runtime >> context (classpath, OSGi, etc). >> >> Why not rely on the compilation unit visibility? Like introducing a "module >> protected visibility" without managing it at the module metadata level? I tend >> to think that "module protected class Foo { }" is cleaner than "public class >> Foo { }" only to be made "hidden" in module-info.java by not having a >> corresponding exports clause. > > We've considered that. ?If we were starting from scratch today, such a > general module-accessibility modifier could well be the way to go. ?With > countless lines of existing Java code out in the wild, however, our take > is that if developers have to modify all their source code and rebuild > their libraries and systems in order to take advantage of modularity (or, > equivalently but less robustly, run tools over their existing binaries), > then that would be a significant barrier to adoption. > >> ? ? The `public` modifier makes the types imported into `bar` from `foo` >> ? ? available to any other module that depends directly upon `bar`. >> >> It may be just me, but I don't find it explicit to have "requires public foo" >> meaning that the module re-exports from foo. 'reexports foo" as a separate >> clause may be more readable, although slightly less concise. > > I agree that that's arguably more Java-like. ?It's really a matter of > syntax, so we're going to go with what we have for now; this can easily > be revisited later on. > >> ? ? In this case any other module that depends upon either `bar` or `baz` >> ? ? will be able to use public types exported by `foo` without depending upon >> ? ? `foo` itself >> >> Can't this lead to unexpected types visibility at runtime depending on which >> module was actually resolved as a dependency? Wouldn't it be useful to be >> defensive regarding what imports bring you in crappy modules by having the >> possibility of filtering? >> >> Or maybe the "permits" clause could be used just for that? > > No, `permits` wouldn't work for that. > > Whether a module should be able to filter the types that it re-exports is > an interesting question; I'll make a note of it. > >> ? ? A non-default view can, finally, also declare an entry point different >> ? ? from that of its containing module's default view, ... >> >> You may want to add a sentence and/or example to say how "java -m Foo" can pick >> one view or the other. > > Good point; I'll do that. > >> ? ? Services >> >> What you have here sounds good in principle, especially using ServiceLoader, >> but we again get to the point of dynamics. >> >> On one hand you have a nominal static module system and a mechanism to bind to >> services provided by modules in a decoupled fashion. Great. On the other hand >> you seem not to be willing to have full dynamic modules + services + lifecycle >> notifications although you seem to intend that there will be an API to still >> load them dynamically? which means that Jigsaw may likely end up being >> half-baked here? meaning that people will hack on top of that or resort to >> solutions like OSGi which will most likely not be 100% 2-ways compatible and >> have their own issues. > > Our aim is that Jigsaw be "baked enough" that container-type applications > can be built on top of it, and can load and unload independent modular > components or applications. ?Developers who really need rich dynamism > should go use OSGi or a similar framework. > > - Mark From julien.ponge at gmail.com Thu Dec 22 09:37:55 2011 From: julien.ponge at gmail.com (Julien Ponge) Date: Thu, 22 Dec 2011 18:37:55 +0100 Subject: DRAFT: Project Jigsaw: The Big Picture (part 1) In-Reply-To: <20111221212942.105B323A6@eggemoggin.niobe.net> References: <67525B47991342EA94BAB6FEDB227101@gmail.com> <20111221212942.105B323A6@eggemoggin.niobe.net> Message-ID: Thanks Mark for your feedback. Minor comments below. Hope they are constructive. > There are definitely important classes of applications that require > fully-dynamic multi-version module resolution and service lookup with > a rich lifecycle API. ?That's a pretty complicated programming model, > however, and it's not one that most Java developers need, nor is it > required in order to modularize the platform itself. ?In SE 8 we're > therefore proposing just to solve the simpler, more-common problem, > and to make sure that developers who actually need to use frameworks > like OSGi can do so in a way that works well with the base platform. Granted, such programming models are inherently complicated and most problems are reminiscent of those encountered in concurrent programming. My point which may be hidden in my previous reply is that while I agree that focusing on the most common problems is absolutely right, there may be a chance to evolve the language+platform to also cope with dynamics in a cleaner way that say, OSGi. I have been using OSGi for some time. I can't blame it for making ways around Java and the platform so as to provide dynamic modules. It's a pragmatic solution for now, but it requires such a rigorous discipline from the developer that it easily and often bites. However if you use OSGi just as a static module system then it's quite easy to use and it already does what Jigsaw promises. > I completely agree that adding a container to the JDK would be a mistake, > and we aren't proposing to do that. ?You suggest that we could still > support the general dynamic loading and unloading of modules at runtime, > but that would add significant complexity to both the programming model > (i.e., the specification) and the implementation. ?The Jigsaw design is, > so far, very much simpler than OSGi, and that's largely because we chose > early on not to try to solve all the big problems that OSGi addresses. I still wonder if we couldn't come up with a clean and simple solution as long as we have some latitude on evolving the specs. This is an open question, really. Jigsaw is indeed quite simple in its current form, but things like views and local requires tend not to be that simple at first... plus they somehow try to solve OSGI-esque problems (breaking isolation for some "friend" packages, etc) ;-) > As to API documentation, we have some (admittedly sketchy) Javadoc right > now; a forthcoming section of the "big picture" document will have more > details, and we'll be fleshing out the Javadoc as we go. Nice > No, it's not. ?Export declarations in Jigsaw are meaningful in all > phases, whereas in OSGi they're pretty much just a run-time concept. +1 > They're also much stronger than in OSGi, since access to non-exported > types is specifically disallowed at run time by the JVM. I don't get it. Do you mean that you cannot pass a reference of a non-exported class instance from a module to another module? In this case it would indeed be stronger than OSGi where you just cannot load such classes, but you can still pass a reference that was allocated from the originating module. Cheers -- Julien Ponge http://julien.ponge.info/ From jesse.glick at oracle.com Thu Dec 22 09:51:05 2011 From: jesse.glick at oracle.com (Jesse Glick) Date: Thu, 22 Dec 2011 12:51:05 -0500 Subject: DRAFT: Project Jigsaw: The Big Picture (part 1) In-Reply-To: <4EF34A03.9050205@gmx.de> References: <20111220232839.134977A0@eggemoggin.niobe.net> <4EF34A03.9050205@gmx.de> Message-ID: <4EF36E09.2070708@oracle.com> On 12/22/2011 10:17 AM, Sebastian Sickelmann wrote: > module baz { // [you wrote bar initially but I think you meant baz] > requires public foo @ 1.0; > } > module buz { > requires baz; // Can also use foo's exported types > requires foo @>= 2.0; > } > > The use case is: buz want to use functionality of baz. > baz also provides some functionality for which foo must be reexported. But buz is not using this functionality. > But buz uses some functionality in foo that is only supported since version 2.0 of foo. > Can buz override the reexported dependency of foo at 1.0 to foo@>=2.0? I guess this cannot work and the module system would just refuse to load (or even install) this version of buz. The module system does not "know" that buz is not using the part of baz referring to foo, so it must assume that it is. And baz has declared that it cannot be used with foo at 2.0, so you are out of luck. This is why the author of baz needed to be as lenient as possible about the acceptable versions of foo - or at least offer to publish a new version of baz based on foo at 2.0. The situation is no different if baz does not use the 'public' modifier on its import; buz still explicitly imports foo, and may legally call those parts of baz referring to foo, with the expectation that the class space is consistent. In other words, the "reexport" feature is a mere convenience, and the underlying problem would exist without it. Since Jigsaw does not normally support loading multiple versions of a single module at once, the situation is no different even if baz's public signature makes no mention of foo's types - if it merely uses foo internally. A module system supporting parallel loading of old and new versions of a single module could fix this if it could ensure that all static linkages between modules passed consistent versions of imported types. (An OSGi expert should discuss whether it can handle this case and how.) Jigsaw could also of course offer a mechanism for manually overriding unsatisfied dependencies, in which case someone - either the author of buz, or the author of the root module (the final app using buz), or the user running the app - has to investigate _why_ the author of baz did not advertise foo at 2.0 compatibility, what the actual compatibility impact will be in the context of this app, and whether it is safe to proceed anyway. From neal at gafter.com Thu Dec 22 10:07:07 2011 From: neal at gafter.com (Neal Gafter) Date: Thu, 22 Dec 2011 10:07:07 -0800 Subject: What is a language construct? (was Re: Comment on state of Project Jigsaw) In-Reply-To: <4EF2889C.5070707@oracle.com> References: <20111221223027.33AD123A6@eggemoggin.niobe.net> <4EF26029.1050700@univ-mlv.fr> <4EF2679F.9050408@oracle.com> <4EF27623.2050501@oracle.com> <4EF2889C.5070707@oracle.com> Message-ID: On Wed, Dec 21, 2011 at 5:32 PM, Alex Buckley wrote: > On 12/21/2011 5:06 PM, Neal Gafter wrote: > >> A use-site protection mechanism provides no protection, as one can just >> say "yes, it's accessible to me" at any place you want to use the thing >> supposedly protected. >> > > I'm not getting my question across. _How_ do you say this at the place you > want to use the thing? > > If the answer is "Just say the name of the thing!", then I ask: "How do > you know which module the thing comes from?" > > Even with package dependences rather than module dependences, you still > need to say at the use-site which version of the package you want. Perhaps > you are proposing "import java.util.List @ 8.0;" ? I don't think a protection mechanism that involves use-site access control (or version control) is an appropriate language extension at all. That is more properly a feature of a build and packaging system. While it may well involve a packaging specification language (e.g. a package specification file *or* a compiler command-line or even the layout and organization of files on the disk drive), that language need have no relationship to the underlying programming languages of the system under construction. Jigsaw appears to have largely taken that approach, with the exception that Jigsaw pretends it is an extension of the Java programming language. From jesse.glick at oracle.com Thu Dec 22 11:07:43 2011 From: jesse.glick at oracle.com (Jesse Glick) Date: Thu, 22 Dec 2011 14:07:43 -0500 Subject: Oddity in ModuleClassLoader.isModulePresent Message-ID: <4EF37FFF.2000805@oracle.com> In java.lang.module.ModuleClassLoader: public boolean isModulePresent(String mn) { return ((org.openjdk.jigsaw.Loader)this).isModulePresent(mn); } And in org.openjdk.jigsaw.Loader this method is indeed overridden. Would it not be more straightforward to just mark isModulePresent abstract? From jesse.glick at oracle.com Thu Dec 22 11:08:32 2011 From: jesse.glick at oracle.com (Jesse Glick) Date: Thu, 22 Dec 2011 14:08:32 -0500 Subject: Module accessibility In-Reply-To: <4EF30E6A.7080801@oracle.com> References: <67525B47991342EA94BAB6FEDB227101@gmail.com> <20111221212942.105B323A6@eggemoggin.niobe.net> <4EF29610.7050904@redhat.com> <4EF30E6A.7080801@oracle.com> Message-ID: <4EF38030.7010904@oracle.com> On 12/22/2011 06:03 AM, Jesse Glick wrote: > 3. The well-known problem of adding an internal-only method (called from other packages) to an exported class would disappear See sun.misc.SharedSecrets as an example. From jaroslav.tulach at oracle.com Thu Dec 22 11:50:13 2011 From: jaroslav.tulach at oracle.com (Jaroslav Tulach) Date: Thu, 22 Dec 2011 20:50:13 +0100 Subject: NP-Completeness Re: How to drain your laptop battery using Jigsaw In-Reply-To: <4EF25C70.2050401@oracle.com> References: <4EF25C70.2050401@oracle.com> Message-ID: <201112222050.13307.jaroslav.tulach@oracle.com> >## 21.?12.?2011 23:23:44 ##< > My colleague Jarda Tulach had investigated module systems supporting > version ranges, and wrote up a formal proof [1] that picking one version > of each of several modules in a repository such that all dependencies are > satisfied is in general NP-complete. I decided to verify that Jigsaw > suffers from this problem. > > The attached program, run against a recent Jigsaw build (8c6429c279e7), > starts consuming many seconds of CPU time per round (effectively hanging) > somewhere around 9 variables (depending on the random seed), mostly > running in Resolver.resolve according to a profiler. The randomized nature > of the test and the clause count is designed to stress the system, of > course. First of all, congrats Jesse! This is a perfect real world demonstration of my theoretical work. I guess for many people it is more important to hear about "draining your batteries", than about facing an NP-complete problem. Anyway, I am glad you brought this problem to the jigsaw-dev mailing list. It is in my opinion interesting topic and if there is a way to avoid pitfalls of range dependencies, I'll be glad to help with that. > Jarda please step in if I misrepresented any of these options. I've been thinking about these issues for last decade while working on NetBeans and its module system (helping you, Jesse). The NP-complete problem exists only when we allow implicit re-export. That means: - using a library internally, without exporting it (and under the assumption that multiple versions of a library can co-exist, if not re-exported[4]) is safe. That is why the NP-complete problem is caused by re-export and actually only re-export that is done on implicit base. I think I have a proof[2] that shows that if an explicit "requires" of all transitive dependencies of a library are enumerated (at compile time), it is polynomial problem to resolve them during runtime. Actually this is my favorite solution. I am not sure if it works with range dependencies, but clearly it works with semantic versioning[5] (different major versiong means incompatible version) as my proof shows[2]. Now few comments to other solutions outlined by Jesse: > 1. Drop the requirement that module dependencies may include version ranges > or otherwise permit incompatible changes to be marked, so that a > dependency is always on some version or later. This seems unsatisfactory. There is clearly a need for an incompatible version of a module. On the other hand, why not rename the name of the module when producing an incompatible version. So there would be: org.w3c.dom (in few revisions like 1.0, 1.1, 1.2, etc.) and then an incompatible change as org.w3c.dom2 (in few revisions) and later also org.w3c.dom3 (with many revisions). I am not saying I like changing the name of a module with every incompatible version, but it is not as unsatisfactory is it might seem. The biggest problem in this issue is to specify a dependency on either org.w3c.dom2 or org.w3c.dom3, if that is ever required. > 2. Drop the requirement that the module system pick a version of each > module to load such that all dependencies are satisfied. For example: just > try to load the newest available version of each module, and if any > dependencies are in fact unsatisfied, report an error and ask the user to > override one or more module versions manually. This is what we do in NetBeans and it works for a single packaged application. However I don't think this will work for Java module system where a single repository may contain modules for multiple applications. If you accept #2, it may happen that after installing new application (and its modules), some previously installed application can no longer be executed. That is not something that would boost the creation of Java module ecosystem. > 3. Rewrite Resolver using something like SAT4J, or even leave it as it is, > and just document that pathological cases can result in unbounded > execution time. E.g. use brute force. Well, the rationalistic approach commonly worshiped on my side of the Ocean is completely against approach like this. If we knew that range dependencies with implicit re-export are the problem, we would not allow them and think about better solution. Less powerful but computable[3] in polynomial time. However I can imagine a pragmatic design (commonly used on the other side of the Ocean) is able to promise something unrealistic and then empower brute force tools like SAT4J (as Equinox does) to deal with "pathological" cases. > 4. Require that a new release of any module which uses an incompatible > version of some dependency (even if used only internally) is itself marked > incompatible. This seems unsatisfactory since it breaks encapsulation - I > should not be forced to label my new widely used graphics toolkit 2.0 > rather than 1.1 merely because I started to use a refactored version of > some math library after 1.0 was released. Remember, this is only about re-exports! If I re-export (by returning it from a signature of my method) from my graphics toolkit java.awt.Rectangle and somebody removes it from JDK, there is no way around: just to produce an incompatible version of my graphical toolkit. > Also it is unclear how you would > enforce this constraint so long as Jigsaw assigns no semantics to version > numbers except that they are totally ordered. Jesse recently pointed out in our other conversation that there is a semantic versioning[5] manifesto. I believe this is much better option than having no semantic and fixing that by allowing range dependencies. > 5. Permit multiple versions of a module to be loaded so long as class > spaces remain consistent This is almost a must. Unless two versions of a class get re-exported for the same loader, this should be allowed (see [4]). Summary. Those are the options me and Jesse see. However the primary question is whether Jigsaw wants to avoid NP-completeness or just play on the pragmatic feelings and pretend the NP-complete problem is just a corner case. I would be willing to invest my knowledge into eliminating the NP-completeness. I believe there are reasonable ways to avoid it. And last but not least: if there should be a module system that avoids NP-complete problems when resolving dependencies, it would be an ultimate reason which rules out OSGi. -jt > [1] http://wiki.apidesign.org/wiki/LibraryReExportIsNPComplete [2] The proof is here http://wiki.apidesign.org/index.php?title=LibraryWithoutImplicitExportIsPolynomial&useskin=monobook btw. it is better to switch to "Black/White" background when reading math statements on my website. [3] http://wiki.apidesign.org/wiki/RangeDependencies [4] Having two modules with same namespace is OK, if the module classes are called, but what if the module exports a service? Is not that going to cause problems? [5] http://semver.org/ From varradami at gmail.com Thu Dec 22 16:08:05 2011 From: varradami at gmail.com (Daniel Siegmann) Date: Thu, 22 Dec 2011 19:08:05 -0500 Subject: Simplicity [was: Re: DRAFT: Project Jigsaw: The Big Picture (part 1)] In-Reply-To: References: Message-ID: On Thu, Dec 22, 2011 at 12:16 PM, Neil Bartlett wrote: > This is not intended as a flame. I acknowledge that the appearance of > complexity is highly subjective, and I would appreciate the opinions > of people who are not already inculcated in either of the OSGi or > Jigsaw camps. OSGi seems more complicated to me, however I only a rudimentary experience with it. The necessity of a container alone makes OSGi a big headache (in my admittedly limited experience). YMMV On the other hand I do have plenty of experience with Maven, and many things in Jigsaw are no more complicated than their equivalents in Maven. So from my perspective Jigsaw seems pretty simple (so far). There are a few complicated bits (such as views), but in practice I doubt these will be needed very often. Of course, draft one of the big picture is pretty light on the details, so there is plenty of room to complicate Jigsaw. ;-) ~Daniel From alex.buckley at oracle.com Thu Dec 22 17:05:20 2011 From: alex.buckley at oracle.com (Alex Buckley) Date: Thu, 22 Dec 2011 17:05:20 -0800 Subject: What is a language construct? (was Re: Comment on state of Project Jigsaw) In-Reply-To: References: <20111221223027.33AD123A6@eggemoggin.niobe.net> <4EF26029.1050700@univ-mlv.fr> <4EF2679F.9050408@oracle.com> <4EF27623.2050501@oracle.com> <4EF2889C.5070707@oracle.com> Message-ID: <4EF3D3D0.50704@oracle.com> On 12/22/2011 10:07 AM, Neal Gafter wrote: > I don't think a protection mechanism that involves use-site access > control (or version control) is an appropriate language extension at > all. That is more properly a feature of a build and packaging system. > While it may well involve a packaging specification language (e.g. a > package specification file /or/ a compiler command-line or even the > layout and organization of files on the disk drive), that language need > have no relationship to the underlying programming languages of the > system under construction. Jigsaw appears to have largely taken that > approach, with the exception that Jigsaw pretends it is an extension of > the Java programming language. OK, now I understand. You do not object to a module system being part of the Java SE platform, or to dependences at the granularity of a module. You object to including a module declaration in the grammar of the Java language, specifically in the CompilationUnit production, and having the effects of a module declaration be noticeable in the language's visibility and accessibility rules. Many presentations on Jigsaw have covered the need for modules to _look_ familiar to Java programmers, leading to the open requirement: "Java-like syntax ? The syntax of module declarations must be similar to that of declarations of a like nature in the Java programming language." From that open requirement flows a module declaration built from the traditional input elements of the Java language, and included under CompilationUnit in the Java grammar. Yes, the requirement is open, but it's the current direction of the Jigsaw prototype. When modules are as much a part of the platform as packages and types, it's not too surprising that their declarations live close together. If the Java SE Platform Specification took it upon itself to define a module declaration in a sibling of CompilationUnit, say ModuleUnit, and mandated that a ModuleUnit be detected and interpreted at compile-time and runtime according to the module system's rules, then I think you would be marginally happier. At least the module declaration would be "outside" the language. You would still object to the language's sense of visibility and accessibility being determined by another part of the platform, namely the module system. But the language has always existed in the context of the overall platform, and the omnipresence of the module system is a fundamental principle in the requirements. Not sure we have a zone of agreement here. Alex From neal at gafter.com Thu Dec 22 17:55:21 2011 From: neal at gafter.com (Neal Gafter) Date: Thu, 22 Dec 2011 17:55:21 -0800 Subject: What is a language construct? (was Re: Comment on state of Project Jigsaw) In-Reply-To: <4EF3D3D0.50704@oracle.com> References: <20111221223027.33AD123A6@eggemoggin.niobe.net> <4EF26029.1050700@univ-mlv.fr> <4EF2679F.9050408@oracle.com> <4EF27623.2050501@oracle.com> <4EF2889C.5070707@oracle.com> <4EF3D3D0.50704@oracle.com> Message-ID: On Thu, Dec 22, 2011 at 5:05 PM, Alex Buckley wrote: > Many presentations on Jigsaw have covered the need for modules to _look_ > familiar to Java programmers, leading to the open requirement: > > "Java-like syntax ? The syntax of module declarations must be similar to > that of declarations of a like nature in the Java programming language." > In what sense are module declarations "of a like nature" to anything currently in the Java programming language? -Neal From alex.buckley at oracle.com Thu Dec 22 18:37:47 2011 From: alex.buckley at oracle.com (Alex Buckley) Date: Thu, 22 Dec 2011 18:37:47 -0800 Subject: What is a language construct? (was Re: Comment on state of Project Jigsaw) In-Reply-To: References: <20111221223027.33AD123A6@eggemoggin.niobe.net> <4EF26029.1050700@univ-mlv.fr> <4EF2679F.9050408@oracle.com> <4EF27623.2050501@oracle.com> <4EF2889C.5070707@oracle.com> <4EF3D3D0.50704@oracle.com> Message-ID: <4EF3E97B.5040904@oracle.com> On 12/22/2011 5:55 PM, Neal Gafter wrote: > On Thu, Dec 22, 2011 at 5:05 PM, Alex Buckley > wrote: > > Many presentations on Jigsaw have covered the need for modules to > _look_ familiar to Java programmers, leading to the open requirement: > > "Java-like syntax ? The syntax of module declarations must be > similar to that of declarations of a like nature in the Java > programming language." > > > In what sense are module declarations "of a like nature" to anything > currently in the Java programming language? I take it the rest of my email accurately summarized your views. You are querying whether the _semantics_ of module declarations are like the semantics of anything currently in the language. Obviously not. The module concept is new, even though the concepts of visibility (a.k.a. observability) and accessibility are old. Precisely because a module is a first-class concept in the language (affecting visibility and accessibility), it is not unreasonable to represent it syntactically using the traditional input elements of the Java programming language. The fact that a type declaration doesn't refer to a module name seems to be causing people to think that modules are not a first-class concept in the language. But they are. Wouldn't it be odd if some declarations of first-class language concepts were made in Java-like syntax while others were made in, say, JSON? After all, a package is a different thing to a field, and yet both exist in the Java grammar. Just because a module is a different thing to a field doesn't mean a module should be excluded from the Java grammar. Alex From neal at gafter.com Thu Dec 22 20:20:35 2011 From: neal at gafter.com (Neal Gafter) Date: Thu, 22 Dec 2011 20:20:35 -0800 Subject: What is a language construct? (was Re: Comment on state of Project Jigsaw) In-Reply-To: <4EF3E97B.5040904@oracle.com> References: <20111221223027.33AD123A6@eggemoggin.niobe.net> <4EF26029.1050700@univ-mlv.fr> <4EF2679F.9050408@oracle.com> <4EF27623.2050501@oracle.com> <4EF2889C.5070707@oracle.com> <4EF3D3D0.50704@oracle.com> <4EF3E97B.5040904@oracle.com> Message-ID: On Thu, Dec 22, 2011 at 6:37 PM, Alex Buckley wrote: > I take it the rest of my email accurately summarized your views. > > You are querying whether the _semantics_ of module declarations are like > the semantics of anything currently in the language. Obviously not. The > module concept is new, even though the concepts of visibility (a.k.a. > observability) and accessibility are old. > The fact that a concept is old is not evidence in favor of supporting it in a programming language. Observability is not a concept currently part of the Java language. The language spec mentions observability explicitly as something essentially outside of its scope (JLS3 7.3). > Precisely because a module is a first-class concept in the language > (affecting visibility and accessibility), it is not unreasonable to > represent it syntactically using the traditional input elements of the Java > programming language. > I don't see module-level accessibility as a language construct in the latest draft (but I may well have missed it). > The fact that a type declaration doesn't refer to a module name seems to > be causing people to think that modules are not a first-class concept in > the language. But they are. > What does it mean for something to be first-class in the language if it cannot be named or referred to in code in any way? Where is the syntax for *anything *in the Java language extended to be module-aware? Adding a new, separate language (and allowing the .java extension for that language) doesn't make that language's concepts first-class in the Java language. Wouldn't it be odd if some declarations of first-class language concepts > were made in Java-like syntax while others were made in, say, JSON? After > all, a package is a different thing to a field, and yet both exist in the > Java grammar. Just because a module is a different thing to a field doesn't > mean a module should be excluded from the Java grammar. > No, I do not find it odd that Observability is defined by the javac command line (e.g. you can't see a class which isn't involved in the compilation at all), even though the javac command line language is something completely different from the Java programming language. I do not consider something to be "first-class" in the language where that thing cannot be referred to by name *or in any other way* in Java code. In short, the rest of your email did not accurately reflect my views. From mandy.chung at oracle.com Fri Dec 23 00:06:17 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Fri, 23 Dec 2011 00:06:17 -0800 Subject: Oddity in ModuleClassLoader.isModulePresent In-Reply-To: <4EF37FFF.2000805@oracle.com> References: <4EF37FFF.2000805@oracle.com> Message-ID: <4EF43679.1020306@oracle.com> On 12/22/2011 11:07 AM, Jesse Glick wrote: > In java.lang.module.ModuleClassLoader: > > public boolean isModulePresent(String mn) { > return ((org.openjdk.jigsaw.Loader)this).isModulePresent(mn); > } > > And in org.openjdk.jigsaw.Loader this method is indeed overridden. > > Would it not be more straightforward to just mark isModulePresent > abstract? I think you're right. This method can be an abstract method. I make a note to clean this up next. Mandy From peter.kriens at aqute.biz Fri Dec 23 00:35:01 2011 From: peter.kriens at aqute.biz (Peter Kriens) Date: Fri, 23 Dec 2011 09:35:01 +0100 Subject: NP-Completeness Re: How to drain your laptop battery using Jigsaw In-Reply-To: <201112222050.13307.jaroslav.tulach@oracle.com> References: <4EF25C70.2050401@oracle.com> <201112222050.13307.jaroslav.tulach@oracle.com> Message-ID: <20B1D2D1-F186-43CC-B4AC-284DA58E7A55@aQute.biz> This issue has been discussed many times inside the OSGi. I just like to point out some experiences. First, Richard S. Hall's resolver in Felix has shown that even large systems resolve in a very short time and using very few electrons on the way. And he resolves with a completely generic require/capabilities model including the so called uses constraints necessary to support side-by-side versioning. If google maps can find a route between Montpellier and Stockholm in less than a second then I fail to see why we could not find a route in the much smaller space of software dependencies. Second, it is necessary to make a distinction between deployment time and build time. During building flexibility and allowing me to specify as little as possible for rapid turnaround. However, after an artifact moves to testing and production it becomes necessary to pin down the configuration of the test artifact to ensure repeatability. For example, one compiles against API only JARs that are properly (package) versioned so that my code remains clean and cannot accidentally pickup unwanted dependencies. If I then test my program I need to combine it with a concrete implementation. When the application goes to production I need to ensure that what was tested is actually moved in production, even further narrowing it down. This model can largely be automated with OSGi. Third, semantic versioning allows us to specify when we break backward compatibility and that is good. However, what the semver people do not seem to get is that API based programming wreaks havoc with this model. With an API you have a consumer of an API and a provider of that API (i.e. JPA, with EclipseLink as provider and me as consumer). Both depend on JPA but virtually any change in the JPA packages breaks EclipseLink while allowing me to happily use a newer version of the JPA API. In discussions I find that few people get the implications of APIs based programming and semantic versioning. Kind regards, Peter Kriens On 22 dec. 2011, at 20:50, Jaroslav Tulach wrote: >> ## 21.?12.?2011 23:23:44 ##< >> My colleague Jarda Tulach had investigated module systems supporting >> version ranges, and wrote up a formal proof [1] that picking one version >> of each of several modules in a repository such that all dependencies are >> satisfied is in general NP-complete. I decided to verify that Jigsaw >> suffers from this problem. >> >> The attached program, run against a recent Jigsaw build (8c6429c279e7), >> starts consuming many seconds of CPU time per round (effectively hanging) >> somewhere around 9 variables (depending on the random seed), mostly >> running in Resolver.resolve according to a profiler. The randomized nature >> of the test and the clause count is designed to stress the system, of >> course. > > First of all, congrats Jesse! This is a perfect real world demonstration of my > theoretical work. I guess for many people it is more important to hear about > "draining your batteries", than about facing an NP-complete problem. > > Anyway, I am glad you brought this problem to the jigsaw-dev mailing list. It > is in my opinion interesting topic and if there is a way to avoid pitfalls of > range dependencies, I'll be glad to help with that. > >> Jarda please step in if I misrepresented any of these options. > > I've been thinking about these issues for last decade while working on > NetBeans and its module system (helping you, Jesse). The NP-complete problem > exists only when we allow implicit re-export. That means: > > - using a library internally, without exporting it (and under the assumption > that multiple versions of a library can co-exist, if not re-exported[4]) is > safe. > > That is why the NP-complete problem is caused by re-export and actually only > re-export that is done on implicit base. I think I have a proof[2] that shows > that if an explicit "requires" of all transitive dependencies of a library are > enumerated (at compile time), it is polynomial problem to resolve them during > runtime. > > Actually this is my favorite solution. I am not sure if it works with range > dependencies, but clearly it works with semantic versioning[5] (different > major versiong means incompatible version) as my proof shows[2]. > > Now few comments to other solutions outlined by Jesse: >> 1. Drop the requirement that module dependencies may include version ranges >> or otherwise permit incompatible changes to be marked, so that a >> dependency is always on some version or later. This seems unsatisfactory. > > There is clearly a need for an incompatible version of a module. On the other > hand, why not rename the name of the module when producing an incompatible > version. So there would be: > > org.w3c.dom (in few revisions like 1.0, 1.1, 1.2, etc.) > > and then an incompatible change as > > org.w3c.dom2 (in few revisions) > > and later also > > org.w3c.dom3 (with many revisions). > > I am not saying I like changing the name of a module with every incompatible > version, but it is not as unsatisfactory is it might seem. The biggest problem > in this issue is to specify a dependency on either org.w3c.dom2 or > org.w3c.dom3, if that is ever required. > >> 2. Drop the requirement that the module system pick a version of each >> module to load such that all dependencies are satisfied. For example: just >> try to load the newest available version of each module, and if any >> dependencies are in fact unsatisfied, report an error and ask the user to >> override one or more module versions manually. > > This is what we do in NetBeans and it works for a single packaged application. > However I don't think this will work for Java module system where a single > repository may contain modules for multiple applications. If you accept #2, it > may happen that after installing new application (and its modules), some > previously installed application can no longer be executed. That is not > something that would boost the creation of Java module ecosystem. > >> 3. Rewrite Resolver using something like SAT4J, or even leave it as it is, >> and just document that pathological cases can result in unbounded >> execution time. > > E.g. use brute force. Well, the rationalistic approach commonly worshiped on > my side of the Ocean is completely against approach like this. If we knew that > range dependencies with implicit re-export are the problem, we would not allow > them and think about better solution. Less powerful but computable[3] in > polynomial time. > > However I can imagine a pragmatic design (commonly used on the other side of > the Ocean) is able to promise something unrealistic and then empower brute > force tools like SAT4J (as Equinox does) to deal with "pathological" cases. > >> 4. Require that a new release of any module which uses an incompatible >> version of some dependency (even if used only internally) is itself marked >> incompatible. This seems unsatisfactory since it breaks encapsulation - I >> should not be forced to label my new widely used graphics toolkit 2.0 >> rather than 1.1 merely because I started to use a refactored version of >> some math library after 1.0 was released. > > Remember, this is only about re-exports! If I re-export (by returning it from > a signature of my method) from my graphics toolkit java.awt.Rectangle and > somebody removes it from JDK, there is no way around: just to produce an > incompatible version of my graphical toolkit. > >> Also it is unclear how you would >> enforce this constraint so long as Jigsaw assigns no semantics to version >> numbers except that they are totally ordered. > > Jesse recently pointed out in our other conversation that there is a semantic > versioning[5] manifesto. I believe this is much better option than having no > semantic and fixing that by allowing range dependencies. > >> 5. Permit multiple versions of a module to be loaded so long as class >> spaces remain consistent > > This is almost a must. Unless two versions of a class get re-exported for the > same loader, this should be allowed (see [4]). > > Summary. Those are the options me and Jesse see. However the primary question > is whether Jigsaw wants to avoid NP-completeness or just play on the pragmatic > feelings and pretend the NP-complete problem is just a corner case. I would be > willing to invest my knowledge into eliminating the NP-completeness. I believe > there are reasonable ways to avoid it. And last but not least: if there should > be a module system that avoids NP-complete problems when resolving > dependencies, it would be an ultimate reason which rules out OSGi. > > -jt > >> [1] http://wiki.apidesign.org/wiki/LibraryReExportIsNPComplete > > [2] The proof is here > http://wiki.apidesign.org/index.php?title=LibraryWithoutImplicitExportIsPolynomial&useskin=monobook > btw. it is better to switch to "Black/White" background when reading math > statements on my website. > > [3] http://wiki.apidesign.org/wiki/RangeDependencies > > [4] Having two modules with same namespace is OK, if the module classes are > called, but what if the module exports a service? Is not that going to cause > problems? > > [5] http://semver.org/ From jaroslav.tulach at oracle.com Fri Dec 23 01:54:49 2011 From: jaroslav.tulach at oracle.com (Jaroslav Tulach) Date: Fri, 23 Dec 2011 10:54:49 +0100 Subject: What is (Re:) NP-Completeness? In-Reply-To: <20B1D2D1-F186-43CC-B4AC-284DA58E7A55@aQute.biz> References: <4EF25C70.2050401@oracle.com> <201112222050.13307.jaroslav.tulach@oracle.com> <20B1D2D1-F186-43CC-B4AC-284DA58E7A55@aQute.biz> Message-ID: <201112231054.49496.jaroslav.tulach@oracle.com> >## 23.?12.?2011 09:35:01 ##< > First, Richard S. Hall's resolver in Felix has shown that even large > systems resolve in a very short time and using very few electrons on the > way. I am sure Jesse can modify his simple sample to generate few OSGi bundles and make Felix spend all its time resolving dependencies until its computer turns into dust. > And he resolves with a completely generic require/capabilities model > including the so called uses constraints necessary to support side-by-side > versioning. If google maps can find a route between Montpellier and > Stockholm in less than a second then I fail to see why we could not find a > route in the much smaller space of software dependencies. I don't want to occupy this mailing list with computer science topic, however I have to point out this faux pas for the benefit of everyone who will try to understand why having systems with inherent NP-complete problems is no good. However complicated the task of finding a route between two towns may seem, it is a simple graph algorithm which can be solved in O((n+m)*log(n)) steps. Thus trying to deduce from the performance of google maps anything with respect to module systems with range dependencies is meaningless. When talking about maps I can give you another task which would be NP- complete: Find a route from Montpelier to Stockholm that visits every town larger than 10000 citizens in France, Germany, Denmark and Sweden exactly once. If you try this, you'll find out it is much more complicated tasks then finding a shortest path between two towns. I am sure even google maps would not handle problem of this size in 1s. -jt PS: If anyone knew how to effectively find a route through passing once every town, then we would also have effective algorithm for resolving range dependencies. From peter.kriens at aqute.biz Fri Dec 23 01:58:41 2011 From: peter.kriens at aqute.biz (Peter Kriens) Date: Fri, 23 Dec 2011 10:58:41 +0100 Subject: Comment on state of Project Jigsaw In-Reply-To: References: Message-ID: I think at JavaOne 2006 Gilad Bracha claimed that he did not trust the deployment engineers to touch his precious language and spun off JSR 294 from JSR 277 for exactly this reason :-) And I actually now do agree with him after watching the story of Java modularity story for the last 6 years ... :-( The purity of the separation between JLS and implementation of the environment has tremendous value in reducing conceptual complexity. Purity seems to have gotten a bad name but too many people are doing shortcuts only to get bitten later. Using a dummy class file's attributes to store Jigsaw data could be the archetypical example of Conway's law. That said, I think there are two issues that are not always obvious to observers. First, Jigsaw attempts to address the class path assembly problem (maven, unix package dependency managers) and not the software complexity problem. It fights for split packages and module friends, clearly the antithesis of modularity as a tool to reduce software complexity. With Jigsaw it will be easy to call your existing code modular but it will provide few of its benefits since these benefits require unfettered encapsulation. Second, confusingly the JLS already has modules, they're called packages ... the only complaint I hear is about the granularity of packages, it is often too small. This could easily be fixed with a module concept that leverages the package/type namespace. I.e. it encapsulates a set of packages that are below it in the namespace. The normal access keywords plus an additional module keyword can then be used to encapsulate packages and types. module org.foo; package bar; class Baz {} org.foo.bar.Baz One could call this concept super packages ... :-) Anyway, maybe in another life. Kind regards, Peter Kriens On 21 dec. 2011, at 19:33, Neal Gafter wrote: > My main high-level comment on project Jigsaw is that is fails its first > principle: modularity appears in the draft not to be a language construct, > but at best a completely separate language from the Java programming > language. This appears to be an example of Conway's > law > . > > If you think this is the best approach, then it is not clear why it is > described as part of the Java programming language at all. From peter.kriens at aqute.biz Fri Dec 23 02:27:19 2011 From: peter.kriens at aqute.biz (Peter Kriens) Date: Fri, 23 Dec 2011 11:27:19 +0100 Subject: What is (Re:) NP-Completeness? In-Reply-To: <201112231054.49496.jaroslav.tulach@oracle.com> References: <4EF25C70.2050401@oracle.com> <201112222050.13307.jaroslav.tulach@oracle.com> <20B1D2D1-F186-43CC-B4AC-284DA58E7A55@aQute.biz> <201112231054.49496.jaroslav.tulach@oracle.com> Message-ID: In theory practice and theory are the same, in practice they tend to differ ... The OSGi has its fair share of complaints but resolving time is definitely not one of them. OSGi is used by some very, very large applications with thousands of bundles ... Am I missing something here? Don't get me wrong, the theoretical work is crucial and we're not doing enough of it in our industry. However, the fact that you prove that a problem is translatable into an NP complete problem does not mean it cannot perform in reasonable time since the real world problem usually has lots of pruning capabilities, heuristics, and require "a" solution not an optimal solution. And I do not think the problem is going once through every town since this set is based on who you visit. The resolving problem is more that you should not pass through any town that is unacceptable by any of the other towns you visit/visited. Kind regards, Peter Kriens P.S. Nav systems optimize for highways, toll roads, so it has similar problems. On 23 dec. 2011, at 10:54, Jaroslav Tulach wrote: >> ## 23.?12.?2011 09:35:01 ##< >> First, Richard S. Hall's resolver in Felix has shown that even large >> systems resolve in a very short time and using very few electrons on the >> way. > I am sure Jesse can modify his simple sample to generate few OSGi bundles and > make Felix spend all its time resolving dependencies until its computer turns > into dust. > >> And he resolves with a completely generic require/capabilities model >> including the so called uses constraints necessary to support side-by-side >> versioning. If google maps can find a route between Montpellier and >> Stockholm in less than a second then I fail to see why we could not find a >> route in the much smaller space of software dependencies. > > I don't want to occupy this mailing list with computer science topic, however > I have to point out this faux pas for the benefit of everyone who will try to > understand why having systems with inherent NP-complete problems is no good. > > However complicated the task of finding a route between two towns may seem, it > is a simple graph algorithm which can be solved in O((n+m)*log(n)) steps. Thus > trying to deduce from the performance of google maps anything with respect to > module systems with range dependencies is meaningless. > > When talking about maps I can give you another task which would be NP- > complete: Find a route from Montpelier to Stockholm that visits every town > larger than 10000 citizens in France, Germany, Denmark and Sweden exactly > once. > > If you try this, you'll find out it is much more complicated tasks then > finding a shortest path between two towns. I am sure even google maps would > not handle problem of this size in 1s. > -jt > > PS: If anyone knew how to effectively find a route through passing once every > town, then we would also have effective algorithm for resolving range > dependencies. > From Alan.Bateman at oracle.com Fri Dec 23 03:48:22 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 23 Dec 2011 11:48:22 +0000 Subject: DRAFT: Project Jigsaw: The Big Picture (part 1) In-Reply-To: References: <20111220232839.134977A0@eggemoggin.niobe.net> Message-ID: <4EF46A86.7060302@oracle.com> On 22/12/2011 11:51, David Bosschaert wrote: > : > Besides, I still like to see how this can be made extensible as required in [1]. > : > > [1] http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#extensible-module-declarations > The compiler just passes it through as the ModuleData attribute [1]. -Alan. [1] http://openjdk.java.net/projects/jigsaw/doc/lang-vm.html#jigsaw-2.5 From jesse.glick at oracle.com Fri Dec 23 06:33:09 2011 From: jesse.glick at oracle.com (Jesse Glick) Date: Fri, 23 Dec 2011 09:33:09 -0500 Subject: Semantic versioning (was: NP-Completeness...) In-Reply-To: <20B1D2D1-F186-43CC-B4AC-284DA58E7A55@aQute.biz> References: <4EF25C70.2050401@oracle.com> <201112222050.13307.jaroslav.tulach@oracle.com> <20B1D2D1-F186-43CC-B4AC-284DA58E7A55@aQute.biz> Message-ID: <4EF49125.70203@oracle.com> On 12/23/2011 03:35 AM, Peter Kriens wrote: > With an API you have a consumer of an API and a provider of that API [...] > few people get the implications of APIs based programming and semantic versioning. I guess you are referring to semver [1]. I agree that, while well-intentioned, that system does not adequately deal with modules which expose both a widely-used API and an SPI with a few implementations, or (more generally) multiple categories of interface intended for different clients with varying tolerance for incompatibility. If all these interfaces are presented indiscriminately as the "exports" of a versioned module, you get into a catch-22 when trying to introduce an incompatible SPI change while keeping the API fully compatible (or, generally, changing only those interfaces with a few well-known clients): you must mark the module version incompatible, to ensure that SPI implementors are updated to match the changes; but if you do so then the universe of API users will be gratuitously shut out of what would otherwise be a simple upgrade. An unsatisfactory resolution is to version all SPI changes like compatible API changes, and just hope for the best. This seems to have been the policy of Java SE, with the result that the occasional implementor of DOM or JDBC interfaces (proxies, unit test mocks, ...) routinely becomes uncompilable with a new JDK (*). Such a policy becomes intolerable when pieces of the platform are downloaded according to version constraints. Another resolution is to separate the SPI into its own package structure (as is common anyway) and version individual packages, as might be done in OSGi bundles. This is impossible in the current design of Jigsaw, of course, or generally in any module system aspiring to have a 1-1 mapping between module versions (and dependencies) and native package versions (resp. dependencies). An arguably simpler resolution is to use a single module version but treat the semantics of its dotted components differently depending on the category of import. The OSGi "Semantic Versioning" section (3.7.3 in r4 4.3) advocates this, though it assumes that _any_ change to the SPI is automatically incompatible which seems overly pessimistic. The Jigsaw draft does not offer any real advice on version policy, or even describe exactly how version ranges may be expressed, but it seems to support (manual and unenforced) use of this policy. (**) In my draft proposal for updated versioning policy in the NetBeans module system [2] I take this approach, with the added twist of using the build system to enforce the correct kind of version range for an importer based upon its actual imports (under the condition that import categories map to package boundaries). (*) Yes I have seen the proposal for interfaces with default method bodies in JDK 8. This provides a workaround for one class of incompatible changes - adding interface methods. Conscientious SPI designers could already avoid the problem in various ways, such as optional extension interfaces or abstract classes, but obviously many chose not to. (Nor are incompatible changes limited to this, of course - you may want a static factory method to take a list of something rather than a single object, ad nauseam.) I would urge the JLS [3] to be amended to say that adding an interface method (without a proposed default body) _does_ break binary compatibility, and be treated accordingly for purposes of versioning. (**) For better or worse, Jigsaw also supports "permits" clauses by which only enumerated friend modules could use the SPI. The NetBeans module system has had a similar feature since 2005 but it has proven unworkable, and my proposal [2] essentially deprecates it. My understanding is that Eclipse continues to declare "friends", via package import constraints with partial tool enforcement. [1] http://semver.org/ [2] http://wiki.netbeans.org/NbmPackageStability#Marking_stability_of_import [3] http://java.sun.com/docs/books/jls/third_edition/html/binaryComp.html#13.5.3 From richard.s.hall at oracle.com Fri Dec 23 07:07:16 2011 From: richard.s.hall at oracle.com (Richard S. Hall) Date: Fri, 23 Dec 2011 10:07:16 -0500 Subject: What is (Re:) NP-Completeness? In-Reply-To: <201112231054.49496.jaroslav.tulach@oracle.com> References: <4EF25C70.2050401@oracle.com> <201112222050.13307.jaroslav.tulach@oracle.com> <20B1D2D1-F186-43CC-B4AC-284DA58E7A55@aQute.biz> <201112231054.49496.jaroslav.tulach@oracle.com> Message-ID: <4EF49924.9030200@oracle.com> On 12/23/11 04:54 , Jaroslav Tulach wrote: >> ## 23.?12.?2011 09:35:01 ##< >> First, Richard S. Hall's resolver in Felix has shown that even large >> systems resolve in a very short time and using very few electrons on the >> way. > I am sure Jesse can modify his simple sample to generate few OSGi bundles and > make Felix spend all its time resolving dependencies until its computer turns > into dust. Given that it is an NP-complete problem, then there is likely no avoiding the fact that some scenarios will blow it up. I don't think that's the point. The Felix 2 resolver was pretty stupid and would blow up occasionally on scenarios we really saw in the wild. The Felix 3 resolver was rewritten to be smarter and every scenario we ever discovered in the wild that caused the Felix 2 resolver to blow up was handled very nicely by the Felix 3 resolver. This is still the case to my knowledge. Regardless, it still can blow up. However, if Jesse can modify his sample to create an reasonably realistic example (i.e., something you are likely to see in the wild) that blows up the newer Felix resolver, then we over at Apache Felix would certainly be interested in looking at it to see if there is a heuristic lurking in there somewhere. We don't need to solve every scenario, just the ones that actually exist... -> richard From jesse.glick at oracle.com Fri Dec 23 07:45:57 2011 From: jesse.glick at oracle.com (Jesse Glick) Date: Fri, 23 Dec 2011 10:45:57 -0500 Subject: What is (Re:) NP-Completeness? In-Reply-To: <4EF49924.9030200@oracle.com> References: <4EF25C70.2050401@oracle.com> <201112222050.13307.jaroslav.tulach@oracle.com> <20B1D2D1-F186-43CC-B4AC-284DA58E7A55@aQute.biz> <201112231054.49496.jaroslav.tulach@oracle.com> <4EF49924.9030200@oracle.com> Message-ID: <4EF4A235.2060401@oracle.com> On 12/23/2011 10:07 AM, Richard S. Hall wrote: > if Jesse can modify his sample to create an reasonably realistic example (i.e., something you are likely to see in the wild) I doubt it. The crux of the translation to 3-SAT, and of the choice of hard 3-SAT problems, is that the "clause" modules fi appear in multiple (three) releases, yet show a pathological usage of the "variable" modules vi: f5 @ 1 might require v9 @ 1 while f5 @ 2 requires v9 @ 0. To decide whether to use f5 @ 1 or @ 2 you need to figure out how v9 is used by other fi, but picking one of these can screw up the choice of a different vi used by f5, leading to near-endless backtracking. In practice you would rarely start to use an _older_ version of some library in a new release of your own module. It can happen - e.g. due to regressions discovered in the library and not yet fixed in a patch update - but perhaps not often enough to break heuristic solvers. When dependencies are only ever upgraded, I am guessing that a polynomial resolution algorithm exists (proof or contradiction welcome!), though enforcing such a policy would probably be unacceptable anyway. From alex.buckley at oracle.com Fri Dec 23 08:59:14 2011 From: alex.buckley at oracle.com (Alex Buckley) Date: Fri, 23 Dec 2011 08:59:14 -0800 Subject: What is a language construct? (was Re: Comment on state of Project Jigsaw) In-Reply-To: References: <20111221223027.33AD123A6@eggemoggin.niobe.net> <4EF26029.1050700@univ-mlv.fr> <4EF2679F.9050408@oracle.com> <4EF27623.2050501@oracle.com> <4EF2889C.5070707@oracle.com> <4EF3D3D0.50704@oracle.com> <4EF3E97B.5040904@oracle.com> Message-ID: <4EF4B362.3030603@oracle.com> One of the fundamental principles of the module system is fidelity between compile-time and runtime. Conceptually, to align the language with the module system, the scope of observability will grow to encompass module dependences and the scope of accessibility will grow to encompass module exports. Practically, a developer will know which module his code belongs to and which modules he depends on, so there is no need for module names within type declarations. But since you are unwilling to accept this growth in the platform and its constituent parts, I don't think there's any point in continuing with this thread. Alex On 12/22/2011 8:20 PM, Neal Gafter wrote: > On Thu, Dec 22, 2011 at 6:37 PM, Alex Buckley > wrote: > > I take it the rest of my email accurately summarized your views. > > You are querying whether the _semantics_ of module declarations are > like the semantics of anything currently in the language. Obviously > not. The module concept is new, even though the concepts of > visibility (a.k.a. observability) and accessibility are old. > > > The fact that a concept is old is not evidence in favor of supporting it > in a programming language. > > Observability is not a concept currently part of the Java language. The > language spec mentions observability explicitly as something essentially > outside of its scope (JLS3 7.3). > > Precisely because a module is a first-class concept in the language > (affecting visibility and accessibility), it is not unreasonable to > represent it syntactically using the traditional input elements of > the Java programming language. > > > I don't see module-level accessibility as a language construct in the > latest draft (but I may well have missed it). > > The fact that a type declaration doesn't refer to a module name > seems to be causing people to think that modules are not a > first-class concept in the language. But they are. > > > What does it mean for something to be first-class in the language if it > cannot be named or referred to in code in any way? Where is the syntax > for /anything /in the Java language extended to be module-aware? Adding > a new, separate language (and allowing the .java extension for that > language) doesn't make that language's concepts first-class in the Java > language. > > Wouldn't it be odd if some declarations of first-class language > concepts were made in Java-like syntax while others were made in, > say, JSON? After all, a package is a different thing to a field, and > yet both exist in the Java grammar. Just because a module is a > different thing to a field doesn't mean a module should be excluded > from the Java grammar. > > > No, I do not find it odd that Observability is defined by the javac > command line (e.g. you can't see a class which isn't involved in the > compilation at all), even though the javac command line language is > something completely different from the Java programming language. > > I do not consider something to be "first-class" in the language where > that thing cannot be referred to by name /or in any other way/ in Java code. > > In short, the rest of your email did not accurately reflect my views. From peter.kriens at aqute.biz Fri Dec 23 10:38:54 2011 From: peter.kriens at aqute.biz (Peter Kriens) Date: Fri, 23 Dec 2011 19:38:54 +0100 Subject: What is a language construct? (was Re: Comment on state of Project Jigsaw) In-Reply-To: <4EF4B362.3030603@oracle.com> References: <20111221223027.33AD123A6@eggemoggin.niobe.net> <4EF26029.1050700@univ-mlv.fr> <4EF2679F.9050408@oracle.com> <4EF27623.2050501@oracle.com> <4EF2889C.5070707@oracle.com> <4EF3D3D0.50704@oracle.com> <4EF3E97B.5040904@oracle.com> <4EF4B362.3030603@oracle.com> Message-ID: <69F5C3D4-2CB1-4D98-A750-E5382ECFEB47@aqute.biz> I think that fidelity between build and run time is red herring. When I build a house I need scaffolding and lots of stuff I do not need when I live in the house. The build time should help me build robust components that work in different places and do not require me to change metadata all the time. I usually do not know where my code ends so I fail to see how I can get fidelity ... The trick is to manage your dependencies and then minimize them. Kind regards, Peter kriens ----- Sent while on the road, mobile +33698332260 Op 23 dec. 2011 om 17:59 heeft Alex Buckley het volgende geschreven: > One of the fundamental principles of the module system is fidelity between compile-time and runtime. Conceptually, to align the language with the module system, the scope of observability will grow to encompass module dependences and the scope of accessibility will grow to encompass module exports. Practically, a developer will know which module his code belongs to and which modules he depends on, so there is no need for module names within type declarations. But since you are unwilling to accept this growth in the platform and its constituent parts, I don't think there's any point in continuing with this thread. > > Alex > > On 12/22/2011 8:20 PM, Neal Gafter wrote: >> On Thu, Dec 22, 2011 at 6:37 PM, Alex Buckley > > wrote: >> >> I take it the rest of my email accurately summarized your views. >> >> You are querying whether the _semantics_ of module declarations are >> like the semantics of anything currently in the language. Obviously >> not. The module concept is new, even though the concepts of >> visibility (a.k.a. observability) and accessibility are old. >> >> >> The fact that a concept is old is not evidence in favor of supporting it >> in a programming language. >> >> Observability is not a concept currently part of the Java language. The >> language spec mentions observability explicitly as something essentially >> outside of its scope (JLS3 7.3). >> >> Precisely because a module is a first-class concept in the language >> (affecting visibility and accessibility), it is not unreasonable to >> represent it syntactically using the traditional input elements of >> the Java programming language. >> >> >> I don't see module-level accessibility as a language construct in the >> latest draft (but I may well have missed it). >> >> The fact that a type declaration doesn't refer to a module name >> seems to be causing people to think that modules are not a >> first-class concept in the language. But they are. >> >> >> What does it mean for something to be first-class in the language if it >> cannot be named or referred to in code in any way? Where is the syntax >> for /anything /in the Java language extended to be module-aware? Adding >> a new, separate language (and allowing the .java extension for that >> language) doesn't make that language's concepts first-class in the Java >> language. >> >> Wouldn't it be odd if some declarations of first-class language >> concepts were made in Java-like syntax while others were made in, >> say, JSON? After all, a package is a different thing to a field, and >> yet both exist in the Java grammar. Just because a module is a >> different thing to a field doesn't mean a module should be excluded >> from the Java grammar. >> >> >> No, I do not find it odd that Observability is defined by the javac >> command line (e.g. you can't see a class which isn't involved in the >> compilation at all), even though the javac command line language is >> something completely different from the Java programming language. >> >> I do not consider something to be "first-class" in the language where >> that thing cannot be referred to by name /or in any other way/ in Java code. >> >> In short, the rest of your email did not accurately reflect my views. From jesse.glick at oracle.com Fri Dec 23 10:52:14 2011 From: jesse.glick at oracle.com (Jesse Glick) Date: Fri, 23 Dec 2011 13:52:14 -0500 Subject: What is a language construct? In-Reply-To: <4EF4B362.3030603@oracle.com> References: <20111221223027.33AD123A6@eggemoggin.niobe.net> <4EF26029.1050700@univ-mlv.fr> <4EF2679F.9050408@oracle.com> <4EF27623.2050501@oracle.com> <4EF2889C.5070707@oracle.com> <4EF3D3D0.50704@oracle.com> <4EF3E97B.5040904@oracle.com> <4EF4B362.3030603@oracle.com> Message-ID: <4EF4CDDE.500@oracle.com> On 12/23/2011 11:59 AM, Alex Buckley wrote: > to align the language with the module system, the scope > of observability will grow to encompass module dependences > and the scope of accessibility will grow to encompass module exports. I fail to see how the tradition of the JLS forces such a system to represent modules directly in Java source files. Following the style of chapter 7, the JLS could simply say something to the effect of: - A host system may group compilation units [or types?] into sets called "modules". The host system must then determine which other modules are observable from a given module, and may specify that some types [or packages? or individual elements?] within those modules are to be considered inaccessible despite the other provisions in the section "Access Control". JSR 199's JavaFileManager would need to provide a way to specify module identity, observability, and accessibility. At runtime, java.lang.module would define opaque module identifiers with no details such as version, just informational toString() and a boolean hook for the VM to conduct its access checks; and permit ClassLoader.defineClass to pass a module identifier. Then it would be up to the javac command line and the module-aware Java launcher to interpret a physical format of modules - which could, for example, be based on JARs with special manifest attributes defining Jigsaw semantics. Alternate 199-based build tools and java.lang.module-based runtime containers could just as easily interpret OSGi or various other existing formats, or some hybrid system capable of hosting and cross-wiring multiple containers; unlike with older JDKs, they could benefit from compiler- and runtime-enforced module accessibility checks. For that matter, a Maven plugin could easily produce a Jigsaw-compatible binary using a natural POM to determine observability (from ), as well as other module metadata (from , , etc.), with either plugin or source annotations to determine other information such as exports. Making a Jigsaw plugin would be awkward under the current proposal, as it would need to either create target/generated-sources/jigsaw/module-info.java in the generate-sources phase; or (like Tycho?) inject dependencies dynamically by parsing src/main/java/module-info.java, breaking many popular workflows and complicating IDE support. Of course you could argue that module-info.java as currently proposed is simply more readable than manifests and is what Java developers want, especially those who use no IDE or build framework or module system currently. That does not preclude the above approach - it just means that the syntax of module-info.java would be specified by Jigsaw, not the JLS, and could just as easily be module.yaml. From david.lloyd at redhat.com Fri Dec 23 11:14:15 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Fri, 23 Dec 2011 13:14:15 -0600 Subject: What is a language construct? In-Reply-To: <4EF4CDDE.500@oracle.com> References: <20111221223027.33AD123A6@eggemoggin.niobe.net> <4EF26029.1050700@univ-mlv.fr> <4EF2679F.9050408@oracle.com> <4EF27623.2050501@oracle.com> <4EF2889C.5070707@oracle.com> <4EF3D3D0.50704@oracle.com> <4EF3E97B.5040904@oracle.com> <4EF4B362.3030603@oracle.com> <4EF4CDDE.500@oracle.com> Message-ID: <4EF4D307.6070803@redhat.com> On 12/23/2011 12:52 PM, Jesse Glick wrote: > On 12/23/2011 11:59 AM, Alex Buckley wrote: >> to align the language with the module system, the scope >> of observability will grow to encompass module dependences >> and the scope of accessibility will grow to encompass module exports. > > I fail to see how the tradition of the JLS forces such a system to > represent modules directly in Java source files. Following the style of > chapter 7, the JLS could simply say something to the effect of: > > - A host system may group compilation units [or types?] into sets called > "modules". The host system must then determine which other modules are > observable from a given module, and may specify that some types [or > packages? or individual elements?] within those modules are to be > considered inaccessible despite the other provisions in the section > "Access Control". It would be even better if this could be refined to draw a 1:0..1 equivalence between a class loader and a module, and define the access control in terms of the class loader. This also expands the visibility rule to non-modular code which could be quite useful, and closely matches existing package accessibility rules as well. The only reason given to allow multiple modules to share a class loader is for the purposes of code which expects JDK classes to have a "null" class loader. But I don't see this as a serious hurdle - there are many reasons that code may need updating after Java 8 and singling out this one as the one unacceptable change seems a bit unrealistic. -- - DML From neal at gafter.com Fri Dec 23 11:49:04 2011 From: neal at gafter.com (Neal Gafter) Date: Fri, 23 Dec 2011 11:49:04 -0800 Subject: What is a language construct? (was Re: Comment on state of Project Jigsaw) In-Reply-To: <4EF4B362.3030603@oracle.com> References: <20111221223027.33AD123A6@eggemoggin.niobe.net> <4EF26029.1050700@univ-mlv.fr> <4EF2679F.9050408@oracle.com> <4EF27623.2050501@oracle.com> <4EF2889C.5070707@oracle.com> <4EF3D3D0.50704@oracle.com> <4EF3E97B.5040904@oracle.com> <4EF4B362.3030603@oracle.com> Message-ID: On Fri, Dec 23, 2011 at 8:59 AM, Alex Buckley wrote: > One of the fundamental principles of the module system is fidelity between > compile-time and runtime. Conceptually, to align the language with the > module system, the scope of observability will grow to encompass module > dependences and the scope of accessibility will grow to encompass module > exports. Practically, a developer will know which module his code belongs > to and which modules he depends on, so there is no need for module names > within type declarations. I agree that module names need not be mentioned in Java code. That was precisely my point! But since you are unwilling to accept this growth in the platform and its > constituent parts, I don't think there's any point in continuing with this > thread. > Actually, I *want* the *platform* to grow to add support for modules. Since we agree that there is no place for the mention of module names or versions in Java code (i.e. the domain of the language specification), there isn't any point in continuing with this thread. From mark.reinhold at oracle.com Fri Dec 23 12:27:34 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Fri, 23 Dec 2011 12:27:34 -0800 Subject: NP-Completeness Re: How to drain your laptop battery using Jigsaw In-Reply-To: jaroslav.tulach@oracle.com; Thu, 22 Dec 2011 20:50:13 +0100; <201112222050.13307.jaroslav.tulach@oracle.com> Message-ID: <20111223202734.4467CF7F@eggemoggin.niobe.net> Thanks (to both you and Jesse) for raising this issue. I'm not at all surprised that our current resolver can be pushed into pathological run-time behavior. It's critical that we understand the complexity of what we're trying to build, even if we ultimately decide to live with it rather than reduce it. I've skimmed your proof (I won't have time to study it in detail until after the holidays), and one question I have so far is this: Is saying "supports implicit re-export on an implicit base" just another way of saying "only one version of any particular module of a given name is allowed"? You use the term "implicit re-export", but your proof of course doesn't actually consider what types are or are not re-exported, so I'm a bit confused by this terminology. On your LibraryWithoutImplicitExportIsPolynomial [1] page you define the concept of a "complete" repository, in which the transitive dependencies of all modules are explicitly enumerated. You suggest that the set of a module's transitive dependencies could be generated at compile time, but doesn't that mean that you'd still have to run a general -- and possibly NP-complete -- resolution algorithm at compile time in order to compute that set? As to the potential solutions, renaming a module in the case of an incompatible change is a bit ugly, but it's worth exploring further. It basically amounts to moving the major version number into the module name, which is already a common practice in native packaging systems. I agree that the module system must pick a version of each module to load such that all dependences are satisfied for the reason you state, i.e., a single repository (= library) can contain modules for many applications. Finally, to the big question of whether Jigsaw should try to avoid NP-completeness: I'd prefer to avoid it, but I'm willing to accept it. Plenty of existing systems seem to work just fine in the face of this theoretical complexity (e.g., OSGi, Debian, RPM), and in many scenarios Jigsaw will never do resolution at run time anyway. (I suppose this places me firmly on the Western side of your Ocean, but with a keen eye toward the East.) - Mark [1] http://wiki.apidesign.org/index.php?title=LibraryWithoutImplicitExportIsPolynomial&useskin=monobook From alex.buckley at oracle.com Fri Dec 23 12:46:57 2011 From: alex.buckley at oracle.com (Alex Buckley) Date: Fri, 23 Dec 2011 12:46:57 -0800 Subject: What is a language construct? In-Reply-To: <4EF4CDDE.500@oracle.com> References: <20111221223027.33AD123A6@eggemoggin.niobe.net> <4EF26029.1050700@univ-mlv.fr> <4EF2679F.9050408@oracle.com> <4EF27623.2050501@oracle.com> <4EF2889C.5070707@oracle.com> <4EF3D3D0.50704@oracle.com> <4EF3E97B.5040904@oracle.com> <4EF4B362.3030603@oracle.com> <4EF4CDDE.500@oracle.com> Message-ID: <4EF4E8C1.8030904@oracle.com> On 12/23/2011 10:52 AM, Jesse Glick wrote: > I fail to see how the tradition of the JLS forces such a system to > represent modules directly in Java source files. Following the style of > chapter 7, the JLS could simply say something to the effect of: > > - A host system may group compilation units [or types?] into sets called > "modules". The host system must then determine which other modules are > observable from a given module, and may specify that some types [or > packages? or individual elements?] within those modules are to be > considered inaccessible despite the other provisions in the section > "Access Control". Yes, this is pretty much what the JLS needs to say. The host system _must_ attempt to identify a module declaration among its observable compilation units, and then based on the module system's interpretation of that declaration, make other compilation units observable. (The JLS only knows about Java code in compilation units, so there may be some finessing here to cover code in class files.) It doesn't even have to be in the JLS. The package-info javadoc for the java.lang.module package would do fine. Both are part of the overall Java SE Platform Specification. The accessibility rules embedded in the JLS can refer to module system constructs (e.g. exports) whether those constructs are in the JLS or in the SE API. (This is how JSR 292 is specified. The JVM Spec's text for the invokedynamic instruction is deeply dependent on concepts and rules defined in the spec of the java.lang.invoke package, because the instruction is fundamentally a front-end to operations on intrinsic types like MethodHandle.) Note we are not yet specifying the syntax of a module declaration, or a physical placement of a module declaration on a file system. > JSR 199's JavaFileManager would need to provide a way to specify module > identity, observability, and accessibility. At runtime, java.lang.module > would define opaque module identifiers with no details such as version, > just informational toString() and a boolean hook for the VM to conduct > its access checks; and permit ClassLoader.defineClass to pass a module > identifier. Basically yes. (I don't want to get into versioning here.) > Then it would be up to the javac command line and the module-aware Java > launcher to interpret a physical format of modules - which could, for > example, be based on JARs with special manifest attributes defining > Jigsaw semantics. ... > Of course you could argue that module-info.java Stop right there. A file called module-info.java is an implementation detail, not something for the spec. You should say: "Of course you could argue that a module declaration having Java-like syntax" ... OK, let's proceed. > as currently proposed is > simply more readable than manifests and is what Java developers want, > especially those who use no IDE or build framework or module system > currently. That does not preclude the above approach - it just means > that the syntax of module-info.java would be specified by Jigsaw, not > the JLS, and could just as easily be module.yaml. Again, to be fully correct: "it just means that the syntax of a module declaration would be specified by Jigsaw, not the JLS". Sure, the module system could define a YAML-based grammar for a module declaration, and the host system would still be mandated (by text _somewhere_ in the Platform Specification) to find and interpret a module declaration at compile-time. This is where we remember there's another requirement, currently open, that says the grammar should use Java-like syntax. Alex From mark.reinhold at oracle.com Fri Dec 23 13:16:51 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Fri, 23 Dec 2011 13:16:51 -0800 Subject: Semantic versioning (was: NP-Completeness...) In-Reply-To: jesse.glick@oracle.com; Fri, 23 Dec 2011 09:33:09 EST; <4EF49125.70203@oracle.com> Message-ID: <20111223211651.770FBF7F@eggemoggin.niobe.net> 2011/12/23 6:33 -0800, jesse.glick at oracle.com: > On 12/23/2011 03:35 AM, Peter Kriens wrote: >> With an API you have a consumer of an API and a provider of that API [...] >> few people get the implications of APIs based programming and semantic versioning. > > I guess you are referring to semver [1]. I agree that, while well-intentioned, > that system does not adequately deal with modules which expose both a > widely-used API and an SPI with a few implementations, or (more generally) > multiple categories of interface intended for different clients with varying > tolerance for incompatibility. If all these interfaces are presented > indiscriminately as the "exports" of a versioned module, you get into a > catch-22 when trying to introduce an incompatible SPI change while keeping the > API fully compatible (or, generally, changing only those interfaces with a few > well-known clients): you must mark the module version incompatible, to ensure > that SPI implementors are updated to match the changes; but if you do so then > the universe of API users will be gratuitously shut out of what would otherwise > be a simple upgrade. This is an important use case -- we hadn't considered it before, so thanks for bringing it up. > An unsatisfactory resolution is to version all SPI changes like compatible API > changes, and just hope for the best. This seems to have been the policy of Java > SE, with the result that the occasional implementor of DOM or JDBC interfaces > (proxies, unit test mocks, ...) routinely becomes uncompilable with a new JDK > (*). Such a policy becomes intolerable when pieces of the platform are > downloaded according to version constraints. Agreed. > Another resolution is to separate the SPI into its own package structure (as is > common anyway) and version individual packages, as might be done in OSGi > bundles. This is impossible in the current design of Jigsaw, of course, or > generally in any module system aspiring to have a 1-1 mapping between module > versions (and dependencies) and native package versions (resp. dependencies). Another possibility for Jigsaw is to remove the constraint that a non-default module view inherits the version of its containing module. If a view could declare its own version then an SPI view could evolve independently from the API view of the same module. This change would, however, complicate the module-to-native-package mapping even further. I'll take a close look at your proposed new NetBeans versioning policy after the holidays. - Mark From mark.reinhold at oracle.com Fri Dec 23 13:26:22 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Fri, 23 Dec 2011 13:26:22 -0800 Subject: What is a language construct? In-Reply-To: david.lloyd@redhat.com; Fri, 23 Dec 2011 13:14:15 CST; <4EF4D307.6070803@redhat.com> Message-ID: <20111223212622.7FA95F7F@eggemoggin.niobe.net> 2011/12/23 11:14 -0800, david.lloyd at redhat.com: > ... > > The only reason given to allow multiple modules to share a class loader is for > the purposes of code which expects JDK classes to have a "null" class loader. That's not the only reason. The other is to allow existing API packages to be split across two or more modules, which is necessary in order to modularize the platform while maintaining compatibility. > But I don't see this as a serious hurdle - there are many reasons that code may > need updating after Java 8 and singling out this one as the one unacceptable > change seems a bit unrealistic. There is lots and lots of code out there that assumes platform classes to have the null class loader. I'm not willing to break it. - Mark From mark.reinhold at oracle.com Fri Dec 23 13:34:25 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Fri, 23 Dec 2011 13:34:25 -0800 Subject: DRAFT: Project Jigsaw: The Big Picture (part 1) In-Reply-To: david.bosschaert@gmail.com; Thu, 22 Dec 2011 11:51:15 GMT; Message-ID: <20111223213425.9F35FF7F@eggemoggin.niobe.net> 2011/12/22 3:51 -0800, david.bosschaert at gmail.com: > The module definition doesn't look like Java at all to me, take this example > ... > > I have made this point before, but I don't feel it has been taken into > consideration appropriately in the current design. You're right, it hasn't. That's intentional, because we've chosen for now to explore the option of treating module declarations as part of the language. The ultimate design and specification could well turn out differently, but going into the JSR process I want to make sure that we have a good understanding of the linguistic approach. - Mark From mark.reinhold at oracle.com Fri Dec 23 15:36:54 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Fri, 23 Dec 2011 15:36:54 -0800 Subject: DRAFT: Project Jigsaw: The Big Picture (part 1) In-Reply-To: julien.ponge@gmail.com; Thu, 22 Dec 2011 18:37:55 +0100; Message-ID: <20111223233654.8E8BD10B2@eggemoggin.niobe.net> 2011/12/22 9:37 -0800, julien.ponge at gmail.com: > 2011/12/21 13:29 -0800, mark.reinhold at oracle.com: >> There are definitely important classes of applications that require >> fully-dynamic multi-version module resolution and service lookup with >> a rich lifecycle API. That's a pretty complicated programming model, >> however, and it's not one that most Java developers need, nor is it >> required in order to modularize the platform itself. ... > > Granted, such programming models are inherently complicated and most > problems are reminiscent of those encountered in concurrent > programming. > > My point which may be hidden in my previous reply is that while I > agree that focusing on the most common problems is absolutely right, > there may be a chance to evolve the language+platform to also cope > with dynamics in a cleaner way that say, OSGi. As you say later on, this is an open question. I'm highly skeptical that a "clean and simple" design supporting fully-dynamic resolution is possible, but I'd be happy to be proved wrong. > I have been using OSGi for some time. I can't blame it for making ways > around Java and the platform so as to provide dynamic modules. Me neither -- in fact it's impressive what OSGi can do given that it's not deeply integrated with the underlying platform. > It's a > pragmatic solution for now, but it requires such a rigorous discipline > from the developer that it easily and often bites. However if you use > OSGi just as a static module system then it's quite easy to use and it > already does what Jigsaw promises. Perhaps, but only for the simplest of scenarios. >> 2011/12/21 1:31 -0800, julien.ponge at gmail.com: >>> Correct me if I'm wrong, but this is the same as OSGi export clauses, >>> right? >> >> No, it's not. Export declarations in Jigsaw are meaningful in all >> phases, whereas in OSGi they're pretty much just a run-time concept. > > +1 > >> They're also much stronger than in OSGi, since access to non-exported >> types is specifically disallowed at run time by the JVM. > > I don't get it. Do you mean that you cannot pass a reference of a > non-exported class instance from a module to another module? In this > case it would indeed be stronger than OSGi where you just cannot load > such classes, but you can still pass a reference that was allocated > from the originating module. No, I mean that if you somehow do get a reference to such an object you won't be able to do anything with it. The JVM's access-checking machinery won't let you, in the same way and for the same reason that it won't let you invoke a private method. - Mark From mark.reinhold at oracle.com Fri Dec 23 15:42:15 2011 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Fri, 23 Dec 2011 15:42:15 -0800 Subject: Simplicity [was: Re: DRAFT: Project Jigsaw: The Big Picture (part 1)] In-Reply-To: njbartlett@gmail.com; Thu, 22 Dec 2011 17:16:01 GMT; Message-ID: <20111223234215.8943B10B2@eggemoggin.niobe.net> 2011/12/22 9:16 -0800, njbartlett at gmail.com: > I appreciate your effort to document the current status of Jigsaw. Try > as I might, I cannot hold back from commenting on your assertion that > "the Jigsaw design is, so far, very much simpler than OSGi." Obviously > you will disagree, but I feel the exact opposite: the Jigsaw design is > already more complicated than OSGi, especially with the recent > addition of module views. If Jigsaw's goal is to create a module > system that is more easily understandable than OSGi, then it is > failing. Yes, naturally I disagree. For one thing, Jigsaw neither has nor needs a concept of module lifecycles, so that's a pretty big simplification right there. I could list more, but I don't expect to convince you to change your mind. > This is not intended as a flame. I acknowledge that the appearance of > complexity is highly subjective, and I would appreciate the opinions > of people who are not already inculcated in either of the OSGi or > Jigsaw camps. Informed opinions from such people would definitely be valuable. - Mark From david.lloyd at redhat.com Fri Dec 23 15:54:43 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Fri, 23 Dec 2011 17:54:43 -0600 Subject: DRAFT: Project Jigsaw: The Big Picture (part 1) In-Reply-To: <20111223233654.8E8BD10B2@eggemoggin.niobe.net> References: <20111223233654.8E8BD10B2@eggemoggin.niobe.net> Message-ID: <4EF514C3.8090407@redhat.com> On 12/23/2011 05:36 PM, mark.reinhold at oracle.com wrote: > 2011/12/22 9:37 -0800, julien.ponge at gmail.com: >> 2011/12/21 13:29 -0800, mark.reinhold at oracle.com: >>> There are definitely important classes of applications that require >>> fully-dynamic multi-version module resolution and service lookup with >>> a rich lifecycle API. That's a pretty complicated programming model, >>> however, and it's not one that most Java developers need, nor is it >>> required in order to modularize the platform itself. ... >> >> Granted, such programming models are inherently complicated and most >> problems are reminiscent of those encountered in concurrent >> programming. >> >> My point which may be hidden in my previous reply is that while I >> agree that focusing on the most common problems is absolutely right, >> there may be a chance to evolve the language+platform to also cope >> with dynamics in a cleaner way that say, OSGi. > > As you say later on, this is an open question. I'm highly skeptical > that a "clean and simple" design supporting fully-dynamic resolution > is possible, but I'd be happy to be proved wrong. > >> I have been using OSGi for some time. I can't blame it for making ways >> around Java and the platform so as to provide dynamic modules. > > Me neither -- in fact it's impressive what OSGi can do given that > it's not deeply integrated with the underlying platform. > >> It's a >> pragmatic solution for now, but it requires such a rigorous discipline >> from the developer that it easily and often bites. However if you use >> OSGi just as a static module system then it's quite easy to use and it >> already does what Jigsaw promises. > > Perhaps, but only for the simplest of scenarios. > >>> 2011/12/21 1:31 -0800, julien.ponge at gmail.com: >>>> Correct me if I'm wrong, but this is the same as OSGi export clauses, >>>> right? >>> >>> No, it's not. Export declarations in Jigsaw are meaningful in all >>> phases, whereas in OSGi they're pretty much just a run-time concept. >> >> +1 >> >>> They're also much stronger than in OSGi, since access to non-exported >>> types is specifically disallowed at run time by the JVM. >> >> I don't get it. Do you mean that you cannot pass a reference of a >> non-exported class instance from a module to another module? In this >> case it would indeed be stronger than OSGi where you just cannot load >> such classes, but you can still pass a reference that was allocated >> from the originating module. > > No, I mean that if you somehow do get a reference to such an object > you won't be able to do anything with it. The JVM's access-checking > machinery won't let you, in the same way and for the same reason that > it won't let you invoke a private method. There needs to be a release valve for this equivalent to setAccessible(), else frameworks will be much more difficult to implement. Anyway I'm not really sure there is a good reason to introduce this level of access checking. It seems like feature creep to me - there isn't an obvious use case which is being fulfilled here which isn't adequately met today. -- - DML From jaroslav.tulach at oracle.com Sat Dec 24 02:41:51 2011 From: jaroslav.tulach at oracle.com (Jaroslav Tulach) Date: Sat, 24 Dec 2011 11:41:51 +0100 Subject: Semantic versioning (was: NP-Completeness...) In-Reply-To: <20111223211651.770FBF7F@eggemoggin.niobe.net> References: <20111223211651.770FBF7F@eggemoggin.niobe.net> Message-ID: <201112241141.52042.jaroslav.tulach@oracle.com> >## 23.?12.?2011 22:16:51 ##< > 2011/12/23 6:33 -0800, jesse.glick at oracle.com: > > On 12/23/2011 03:35 AM, Peter Kriens wrote: > >> With an API you have a consumer of an API and a provider of that API > >> [...] few people get the implications of APIs based programming and > >> semantic versioning. > > > > I guess you are referring to semver [1]. I agree that, while > > well-intentioned, that system does not adequately deal with modules > > which expose both a widely-used API and an SPI with a few > > implementations, or (more generally) multiple categories of interface Yes, having API and SPI inside of a single module tights them together. It takes a bit of self-dedication to keep backward compatibility of the SPI (implemented by few guys) on par with compatibility of API used by millions. However it is not impossible. In the NetBeans modules where we properly separated the types in API from the types in SPI, we have not usually had problems keeping compatibility for both. > > few well-known clients): you must mark the module version incompatible, > > to ensure that SPI implementors are updated to match the changes; but if > > you do so then the universe of API users will be gratuitously shut out > > of what would otherwise be a simple upgrade. Jesse is right that sometimes (when there is not enough will to design both API and SPI for evolution and separate them), we get into problems and then proper versioning (e.g. marking the new version as incompatible due to change in SPI only) is more hurting regular users (of the API) than helping them. Btw. JDK8 extender methods may mitigate this problem a lot. With them one can extend an interface used both as API and SPI with new methods and provide good enough default implementations so even the old implementation of the SPI can link and execute with new API... > This is an important use case -- we hadn't considered it before, so > thanks for bringing it up. > > Another possibility for Jigsaw is to remove the constraint that a > non-default module view inherits the version of its containing module. > If a view could declare its own version then an SPI view could evolve > independently from the API view of the same module. This change would, > however, complicate the module-to-native-package mapping even further. What about defining new SPI view in case of an incompatible change in the SPI? If it was possible to keep the old view around while introducing the new one, it would be a smooth change. Removing old View1 and creating new View2 may be a complication, but my feeling is that if (in terms of semantic versioning[1]), it is clearly stated that a view can only be removed in case a micro version is incremented[2], there still may be a way avoid the NP-completeness of the dependency resolution algorithm. -jt [1] http://semver.org/ [2] That is how I understand Jesse's proposal for change of NetBeans versioning http://wiki.netbeans.org/NbmPackageStability From sebastian.sickelmann at gmx.de Sun Dec 25 02:16:57 2011 From: sebastian.sickelmann at gmx.de (Sebastian Sickelmann) Date: Sun, 25 Dec 2011 11:16:57 +0100 Subject: Multi-version was [DRAFT: Project Jigsaw: The Big Picture (part 1)] In-Reply-To: <4EF36E09.2070708@oracle.com> References: <20111220232839.134977A0@eggemoggin.niobe.net> <4EF34A03.9050205@gmx.de> <4EF36E09.2070708@oracle.com> Message-ID: <4EF6F819.5090405@gmx.de> Am 22.12.2011 18:51, schrieb Jesse Glick: > On 12/22/2011 10:17 AM, Sebastian Sickelmann wrote: >> module baz { // [you wrote bar initially but I think you meant baz] >> requires public foo @ 1.0; >> } >> module buz { >> requires baz; // Can also use foo's exported types >> requires foo @>= 2.0; >> } >> >> The use case is: buz want to use functionality of baz. >> baz also provides some functionality for which foo must be >> reexported. But buz is not using this functionality. >> But buz uses some functionality in foo that is only supported since >> version 2.0 of foo. >> Can buz override the reexported dependency of foo at 1.0 to foo@>=2.0? > > I guess this cannot work and the module system would just refuse to > load (or even install) this version of buz. The module system does not > "know" that buz is not using the part of baz referring to foo, so it > must assume that it is. And baz has declared that it cannot be used > with foo at 2.0, so you are out of luck. This is why the author of baz > needed to be as lenient as possible about the acceptable versions of > foo - or at least offer to publish a new version of baz based on foo at 2.0. I'am knowing that trying to resolve that buz doesn't not need foo at 1.0 cannot be made automatically. I see the reexport more than a "service" of the implementor/publisher/programmer of buz. If buz uses types of foo at 1.0 in it's public API than an reexport is a nice feature, so that the users of buz are not forced to require foo explicit. > > The situation is no different if baz does not use the 'public' > modifier on its import; buz still explicitly imports foo, and may > legally call those parts of baz referring to foo, with the expectation > that the class space is consistent. In other words, the "reexport" > feature is a mere convenience, and the underlying problem would exist > without it. > > Since Jigsaw does not normally support loading multiple versions of a > single module at once, the situation is no different even if baz's > public signature makes no mention of foo's types - if it merely uses > foo internally. A module system supporting parallel loading of old and > new versions of a single module could fix this if it could ensure that > all static linkages between modules passed consistent versions of > imported types. (An OSGi expert should discuss whether it can handle > this case and how.) But what is the following design principle mean? />Static, single-version module resolution is usually sufficient/ ? Most >applications do not need to add or remove modules dynamically >at run time, nor do they need to use multiple versions of the same >module simultaneously. The module system should be optimized >for common scenarios but also support narrowly-scoped forms of >dynamic multi-version resolution motivated by actual use cases >such as, /e.g./, application servers, IDEs, and test harnesses. How can i achieve multi-version resolution? > > Jigsaw could also of course offer a mechanism for manually overriding > unsatisfied dependencies, in which case someone - either the author of > buz, or the author of the root module (the final app using buz), or > the user running the app - has to investigate _why_ the author of baz > did not advertise foo at 2.0 compatibility, what the actual compatibility > impact will be in the context of this app, and whether it is safe to > proceed anyway. This is the minimum i would expect. As a user i need a way to override wrong decisions of the module author. -- Sebastian From jaroslav.tulach at oracle.com Sun Dec 25 06:30:59 2011 From: jaroslav.tulach at oracle.com (Jaroslav Tulach) Date: Sun, 25 Dec 2011 15:30:59 +0100 Subject: Comments on the proof In-Reply-To: <20111223202734.4467CF7F@eggemoggin.niobe.net> References: <20111223202734.4467CF7F@eggemoggin.niobe.net> Message-ID: <201112251530.59525.jaroslav.tulach@oracle.com> Hello Mark, thanks for your reply. >## 23.?12.?2011 21:27:34 ##< > run-time behavior. It's critical that we understand the complexity of > what we're trying to build, even if we ultimately decide to live with > it rather than reduce it. OK, that is fair approach. > I've skimmed your proof (I won't have time to study it in detail until > after the holidays), What I had in mind was: Every symbol (e.g. class name) has exactly one meaning in a each classloader space. It must not happen that a single fully qualified class name could be resolved to two different classes in any classloader in the system. The rationale for the above is: during compilation classes are referred by FQNs. Each FQN refers to a single class[2]. I believe it is essential to enforce that during runtime for each classloader as well. How can a single class name be resolved to two classes? Not by a direct dependency. That would require my module to have dependency on org.w3c.dom at 1.0 and org.w3c.dom at 3.0 at once which is a non-sense and compiler would reject that. During compilation there is always one exact dependency on a module to compile against. > You use the term "implicit re-export", but your proof of > course doesn't actually consider what types are or are not re-exported, > so I'm a bit confused by this terminology. However some modules re-export types. In the jigsaw terminology use "requires public". Still, each classloader should have just a single class for a FQN in its namespace. > and one question I have so far is this: Is saying > "supports implicit re-export on an implicit base" just another way of > saying "only one version of any particular module of a given name is > allowed"? Not necessarily. If there is a "master" classloader in each application that sees all classes of all available modules (which is sort of true in NetBeans Runtime Container, but not at all in OSGi), then the requirement that each FQN in this "master" classloader represents at most one class naturally leads to requirement that "only one version of any particular module of a given name is allowed". On the other hand, there does not need to be such "master" classloader. If one module requires (without "requires public", e.g. without re-export) org.w3c.dom at 1.0 and another module requires org.w3c.dom at 2.0, then it should be OK to load the org.w3c.dom in two versions as no classloader will see the duplicated classes from these modules at once[3]. > On your LibraryWithoutImplicitExportIsPolynomial [1] page you define the > concept of a "complete" repository, in which the transitive dependencies > of all modules are explicitly enumerated. You suggest that the set of a > module's transitive dependencies could be generated at compile time, but > doesn't that mean that you'd still have to run a general -- and possibly > NP-complete -- resolution algorithm at compile time in order to compute > that set? I have a feeling that during compilation the NP-completeness is avoided. Here is a sketch of the proof using mathematical induction: 1. java.base can be resolved in finite time. 2. when compiling a module M with (direct) dependencies on M1, ..., Mn we can assume that modules M1,...,Mn have all their transitive dependencies enumerated. Let's now collect all transitive dependencies of M1, ..., Mn into a set and verify whether they are all consistent (as described in [1] section "Consistency"). If not, reject the compilation (per arguments in [2]). If they are consistent, select highest requested version of each module in the transitive set and use it as dependencies of M. Seems polynomial to me. > Plenty of existing systems seem to work just fine in the face of this > theoretical complexity (e.g., OSGi, Debian, RPM), and in many scenarios > Jigsaw will never do resolution at run time anyway. Debian and RPM workaround this by having control over whole distribution. When you have it you usually avoid many incompatibilities before you ship. Also each distribution tests the upgrade scenario. Thus it is generally OK to use repositories for current+1 version and mix the modules. However once I tried to update a 2006 Linux RPM distribution with 2009 version. It failed miserably - it has not even started. I am not claiming this was due to the NP-compete problem described above, but it seems to me that unless you control and verify state of the whole repository (as Glassfish, Eclipse, Fedora, Ubuntu and NetBeans do), you'll face incompatibilities far often and having polynomial time resolution algorithm will pay off. > (I suppose this places me firmly on the Western side of your Ocean, but > with a keen eye toward the East.) One of the important minds among European rationalistic computer scientists - Niklaus Wirth (the inventor of Pascal, Oberon, etc.) once noted: Simple, elegant solutions are more effective, but they are harder to find and require more time. JDK has to ship on time and unless we find simple and elegant solution by then, we'll have to accept the NP-complete one. That is completely acceptable decision from my point of view. > Finally, to the big question of whether Jigsaw should try to avoid > NP-completeness: I'd prefer to avoid it, but I'm willing to accept it. I am glad avoiding NP-completeness is preferred. It increases my enthusiasm to seek for a dependency system that would be powerful enough and avoid it. -jt [1] http://wiki.apidesign.org/index.php?title=LibraryWithoutImplicitExportIsPolynomial&useskin=monobook [2] In current classpath system this is not fully true, there can be multiple classes with the same FQN and depending on the order of classpath roots, the first one is selected. I believe in the "modulepath" mode, this is not desirable, there is no order as transitive dependencies are just partially ordered, thus I believe compiler should detect if two classes representing the same FQN are available and emit a compilation error in such case. Correct? [3] Btw. the module system should check that types are not re-exported without using "requires public". E.g. if there is an exported class that extends non- public type or returns non-public type from a visible method, the compiler should emit an error. Correct? From jesse.glick at oracle.com Tue Dec 27 07:22:48 2011 From: jesse.glick at oracle.com (Jesse Glick) Date: Tue, 27 Dec 2011 10:22:48 -0500 Subject: Accessibility of nonexported types (was: Project Jigsaw: The Big Picture (part 1)) In-Reply-To: <4EF514C3.8090407@redhat.com> References: <20111223233654.8E8BD10B2@eggemoggin.niobe.net> <4EF514C3.8090407@redhat.com> Message-ID: <4EF9E2C8.1060207@oracle.com> On 12/23/2011 06:54 PM, David M. Lloyd wrote: >>>> access to non-exported types is specifically disallowed at run time by the JVM. >>> >>> Do you mean that you cannot pass a reference of a >>> non-exported class instance from a module to another module? >> >> No, I mean that if you somehow do get a reference to such an object >> you won't be able to do anything with it. The JVM's access-checking >> machinery won't let you > > There needs to be a release valve for this equivalent to setAccessible() To make a potentially confusing subject more concrete, let us say you have module m1 {exports m1;} package m1; public abstract class SomeAPI { public abstract void run(); public static SomeAPI instance() { return new m1.internal.SomeImpl(); } } package m1.internal; // not exported! public class SomeImpl extends SomeAPI { @Override public void run() {...} public void other() {...} } module m2 {requires m1;} package m2; public class APIClient { void use() { m1.SomeAPI obj = m1.SomeAPI.instance(); // ...now see below } } Obviously APIClient.use can call "obj.run()". Almost as obviously, it cannot refer statically to m1.internal.SomeImpl: using modular javac that would not even compile; and if you used nonmodular javac or an assembler to force it to do so, the module system would throw some sort of linkage error when loading or running APIClient. (A module system based on a current JRE would throw NoClassDefFoundError, wrapping a ClassNotFoundException intentionally thrown from a ClassLoader.loadClass override on m2's loader.) The subtlety is whether APIClient can use reflection to bypass the intent of package exports. In the analogous situation in the NetBeans module system, and I guess in OSGi as well, the following will run: obj.getClass().getMethod("other").invoke(obj); since this bypasses the trigger - attempting to load "m1.internal.SomeImpl" from m2's ClassLoader. Under Jigsaw I would expect the call to Method.invoke to throw IllegalAccessException, unless you rewrite to Method m = obj.getClass().getMethod("other"); m.setAccessible(true); m.invoke(obj); and the SecurityManager does not complain. By the way - since Class does not extend AccessibleObject, it is unclear whether obj.getClass().newInstance(); should also throw IllegalAccessException, and if so, how you could suppress that (given adequate permissions). Since Class.newInstance seems to essentially be a convenience method, I suppose you could rewrite as obj.getClass().getConstructor().newInstance(); and if necessary make it accessible: Constructor c = obj.getClass().getConstructor(); c.setAccessible(true); c.newInstance(); Here I am assuming that for a "public" class in a package not exported to the caller, every member defined in that class is considered inaccessible, just as if the class had default access and the caller was in another package. On that topic, a minor nit: the IllegalAccessException detail message needs to explain that the entire class, not just one member of it, is inaccessible; the following in JDK 7: package p1; class Private implements Runnable { public Private() {} @Override public void run() { System.out.println("run"); } public void other() { System.out.println("other"); } } package p1; public class Access { public static Runnable r() { return new Private(); } } package p2; public class Main { public static void main(String[] args) throws Exception { Runnable r = p1.Access.r(); r.run(); r.getClass().getMethod("other").invoke(r); } } prints run Exception in thread "main" java.lang.IllegalAccessException: Class p2.Main can not access a member of class p1.Private with modifiers "public" at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:95) at java.lang.reflect.AccessibleObject.slowCheckMemberAccess(AccessibleObject.java:261) at java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:253) at java.lang.reflect.Method.invoke(Method.java:594) at p2.Main.main(Main.java:6) which is a little misleading - the modifiers of the member are irrelevant. From jesse.glick at oracle.com Tue Dec 27 09:30:52 2011 From: jesse.glick at oracle.com (Jesse Glick) Date: Tue, 27 Dec 2011 12:30:52 -0500 Subject: Comments on the proof In-Reply-To: <201112251530.59525.jaroslav.tulach@oracle.com> References: <20111223202734.4467CF7F@eggemoggin.niobe.net> <201112251530.59525.jaroslav.tulach@oracle.com> Message-ID: <4EFA00CC.4000904@oracle.com> On 12/25/2011 09:30 AM, Jaroslav Tulach wrote: > If there is a "master" classloader in each application that > sees all classes of all available modules (which is sort of true in NetBeans > Runtime Container It is worth explaining that this is useful for two reasons: 1. As a default value for Thread.getContextClassLoader(), which many legacy libraries assume can "see" every class in the app (unlike Class.forName(String) which only works to find classes in dependencies of the caller). A safer but awkward workaround is temporarily setting the CCL before making any call into such a library, then resetting it in a finally block. 2. For loading an implementation class referred to by name from a static resource, where locating the (module) ClassLoader defining that resource is impractical. Bug #6865375 [1] gives a typical case. Jigsaw proposes to bypass the problem in the limited case of ServiceLoader by using some kind of index to directly find modules providing a given service, but other service locator frameworks are left to find their own solution. > the requirement that each FQN > in this "master" classloader represents at most one class naturally leads to > requirement that "only one version of any particular module of a given name is > allowed" Rather, that only one module is allowed to define a given class - i.e. org.w3c.dom.v2 @ 2.0 plus org.w3c.dom.v3 @ 3.0 is just as bad, but having multiple versions of a module with the same name is fine so long as all the packages in it are renamed. The NetBeans master loader simply punts in this case, refusing to load any of the alternatives. > unless you control and verify state of the whole repository (as Glassfish, > Eclipse, Fedora, Ubuntu and NetBeans do), you'll face incompatibilities far > [more] often Of course a repository of Java modules might be as tightly controlled as the ones mentioned above - but not if it, say, mirrors Maven Central. > I am glad avoiding NP-completeness is preferred. It increases my enthusiasm to > seek for a dependency system that would be powerful enough and avoid it. An apparent precondition of your proof about complete repositories which was not adequately emphasized is that a module must declare itself incompatible if anything it reexports is incompatibly changed. As I have written before, this is not a restriction to be taken lightly! Consider that java.beans.PropertyEditor.getCustomEditor() returns a java.awt.Component, i.e. module java.beans @ 1.8 requires public module java.awt @ [1.8,2), so anyone with a dep on java.beans [1.8,2) gets an implicit dep on java.awt @ [1.8,2) as well. Now suppose that later it is decided to finally just delete java.awt.List.delItems(int,int) which probably no one calls anyway, and to be on the safe side java.awt is bumped up to 2.0 accordingly. But now the next release of java.beans must reexport java.awt @ [2.0,3), meaning by the postulated rule it must also be marked 2.0. And that means that a module with a property editor based on Swing, never using java.awt.List at all and otherwise happy with java.awt @ [1.8,3), is forced to drop support for java.beans 1.x. Indeed a module using java.beans.Introspector for server configuration from properties files and never using PropertyEditor or loading any AWT classes whatsoever will also be forced to drop java.beans 1.x. Making the java.beans -> java.awt dependency "optional" does not seem to solve anything, since you would still have to mark the new java.beans version incompatible for the benefit of clients which do use PropertyEditor.getCustomEditor. The situation would be more robust if reexports were dropped, modules were required to explicitly specify all their direct dependencies but given the option of permitting more lax ranges according to their actual usage, and the module system permitted multiple versions of a module to be loaded so long as class linkage remained consistent - which I think comes down to merely forbidding a module to explicitly import two versions of another module at once (ignoring transitive dependencies), which is uncontroversial. Whether such a system has a polynomial resolver, I am not sure. My proposal for per-package stability in the NetBeans module system optimistically assumes that most incompatible changes are really compatible for most clients and checks each affected dependency for actual linkage problems [2]. (Complete rewrites of an API would merit a new module name.) Transitive propagation of potential incompatibility is less onerous in such a case, since clients can just keep declaring dependencies on old versions even after unrelated incompatibilities appear, but would still feel unintuitive to developers (the author of java.beans in the above example). The NetBeans system is unaffected by the NP-completeness issue (since there is no multiversion module repository and hence no version resolution), so propagating incompatibilities is not required to make resolution tractable; a repository-based system with a linkage check and no propagation requirement could still be hit with intractable problems in case actual linkage errors appear, but since _inferred_ dependency ranges would often be more forgiving than _declared_ ranges, resolution would on average be much simpler. > in the "modulepath" mode [...the] compiler should detect if two classes representing the > same FQN are available and emit a compilation error in such case. Does not seem to be explicitly mentioned as a requirement anywhere, but I agree this is a good idea. > the module system should check that types are not re-exported without > using "requires public". My impression is that the currently proposed "required public" does _not_ work this way - that it is entirely optional, a convenience for the importer. The module-aware compiler could try to enforce the restriction you propose... but see above for the problems this can cause for importers using only unrelated subsets of an API. > if there is an exported class that extends non- > public type or returns non-public type from a visible method, the compiler > should emit an error. This is a good idea, and indeed the language could enforce this sensible restriction even without modules; currently you need to use third-party tools to check for such mistakes. Anyway, this is different than what you suggested above - that an exported class should be prohibited from referring in its public signature to an exported type from another module unless that dependency were marked "public". [1] http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6865375 [2] http://wiki.netbeans.org/NbmPackageStability#Handling_possibly_broken_dependencies From david.lloyd at redhat.com Tue Dec 27 09:48:41 2011 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 27 Dec 2011 11:48:41 -0600 Subject: Accessibility of nonexported types In-Reply-To: <4EF9E2C8.1060207@oracle.com> References: <20111223233654.8E8BD10B2@eggemoggin.niobe.net> <4EF514C3.8090407@redhat.com> <4EF9E2C8.1060207@oracle.com> Message-ID: <4EFA04F9.2060506@redhat.com> On 12/27/2011 09:22 AM, Jesse Glick wrote: > On 12/23/2011 06:54 PM, David M. Lloyd wrote: >>>>> access to non-exported types is specifically disallowed at run time >>>>> by the JVM. >>>> >>>> Do you mean that you cannot pass a reference of a >>>> non-exported class instance from a module to another module? >>> >>> No, I mean that if you somehow do get a reference to such an object >>> you won't be able to do anything with it. The JVM's access-checking >>> machinery won't let you >> >> There needs to be a release valve for this equivalent to setAccessible() > > To make a potentially confusing subject more concrete, let us say you have > > module m1 {exports m1;} > package m1; > public abstract class SomeAPI { > public abstract void run(); > public static SomeAPI instance() { > return new m1.internal.SomeImpl(); > } > } > package m1.internal; // not exported! > public class SomeImpl extends SomeAPI { > @Override public void run() {...} > public void other() {...} > } > module m2 {requires m1;} > package m2; > public class APIClient { > void use() { > m1.SomeAPI obj = m1.SomeAPI.instance(); > // ...now see below > } > } > > Obviously APIClient.use can call "obj.run()". Almost as obviously, it > cannot refer statically to m1.internal.SomeImpl: using modular javac > that would not even compile; and if you used nonmodular javac or an > assembler to force it to do so, the module system would throw some sort > of linkage error when loading or running APIClient. (A module system > based on a current JRE would throw NoClassDefFoundError, wrapping a > ClassNotFoundException intentionally thrown from a ClassLoader.loadClass > override on m2's loader.) > > The subtlety is whether APIClient can use reflection to bypass the > intent of package exports. In the analogous situation in the NetBeans > module system, and I guess in OSGi as well, the following will run: > > obj.getClass().getMethod("other").invoke(obj); > > since this bypasses the trigger - attempting to load > "m1.internal.SomeImpl" from m2's ClassLoader. > > Under Jigsaw I would expect the call to Method.invoke to throw > IllegalAccessException, unless you rewrite to > > Method m = obj.getClass().getMethod("other"); > m.setAccessible(true); > m.invoke(obj); > > and the SecurityManager does not complain. I disagree. I don't see a practical application for this kind of accessibility constraint. It is redundant with respect to accessibility modifiers on the class. If we have a module-level accessibility modifier, then providing an additional check for non-exported public classes is redundant at best. At worst it makes it impossible for users to provide a public class under a non-exported package (think again of frameworks, JavaBeans spec etc.). I say let visibility be controlled by exports, and let accessibility be controlled by modifiers, and we sidestep this whole business. It's cleaner, and frankly less surprising in my opinion. -- - DML From jesse.glick at oracle.com Tue Dec 27 11:31:40 2011 From: jesse.glick at oracle.com (Jesse Glick) Date: Tue, 27 Dec 2011 14:31:40 -0500 Subject: Accessibility of nonexported types In-Reply-To: <4EFA04F9.2060506@redhat.com> References: <20111223233654.8E8BD10B2@eggemoggin.niobe.net> <4EF514C3.8090407@redhat.com> <4EF9E2C8.1060207@oracle.com> <4EFA04F9.2060506@redhat.com> Message-ID: <4EFA1D1C.80802@oracle.com> On 12/27/2011 12:48 PM, David M. Lloyd wrote: > If we have a module-level accessibility modifier, then providing an additional check for non-exported public classes If we have a module-level accessibility modifier - or just treat default access this way (cf. thread I renamed "Module accessibility") - then you could argue that a non-exported public class should be an oxymoron: public should really mean public. > impossible for users to provide a public class under a non-exported package (think again of frameworks, JavaBeans spec etc.) I guess the use case you are thinking of is something like Introspector.getBeanInfo(Class) where the bean is in an exported package but the BeanInfo is in a nonexported package in the search path. The case of a framework which loads a class reflectively from another module's ClassLoader, say to interpret some XML configuration file it has found in that module, is similar; in that case the module may be providing some sort of service but not intending to expose any direct API. Here the framework is trusted to only perform reflection which the module's author expected; you would expect it to not access unrelated elements in your module, and you would not want other modules to reflectively access your classes (even those mentioned in the configuration file). Trusting the framework means it can use setAccessible(true) and assume corresponding permissions from a security manager, though there is the compatibility issue of existing frameworks which do not yet use setAccessible (many do). The question is whether a module system ought to have an accessibility level which basically means "inaccessible at compile time but accessible at runtime", contrary to the requirement of fidelity between phases. A broader question is how to indicate in the language that a class or class member is expected to be used reflectively. This not only affects security; such members are also unsafe to refactor in certain ways, whether by IDEs or obfuscators. [1] discusses this in detail and proposes an @Opaque metaannotation. In conjunction with a module system, accessibility rules could be relaxed so that an opaquely annotated member would be considered accessible to the module defining the opaque annotation (i.e. the framework) without needing to call setAccessible. Thus ---%<--- org.runtasks/src/module-info.java module org.runtasks { exports org.runtasks; provides service javax.annotation.processing.Processor with org.runtasks.internal.TaskProc; } ---%<--- org.runtasks/src/org/runtasks/Task.java package org.runtasks; @Opaque(Opacity.SIGNATURE) @Retention(RetentionPolicy.RUNTIME) public @interface Task {} ---%<--- org.runtasks/src/org/runtasks/Tasks.java package org.runtasks; public class Tasks { public static void runAll() { for (Module m : ...find loaded modules...) { URL xml = m.getClassLoader().findResource("META-INF/tasks.xml"); if (xml == null) continue; for (String clazz : ...parse XML...) { ((Runnable) m.getClassLoader().loadClass(clazz).newInstance()).run(); } } } } ---%<--- org.runtasks/src/org/runtasks/internal/TaskProc.java package org.runtasks.internal; @SupportedAnnotationTypes("org.runtasks.Task") public class TaskProc extends AbstractProcessor {...create tasks.xml...} ---%<--- com.stuff/src/module-info.java module com.stuff { requires org.runtasks; // no exports } ---%<--- com.stuff/src/com/stuff/MyTask.java package com.stuff; @org.runtasks.Task /* not public */class MyTask implements Runnable { @Override public void run() {...} } ---%<--- com.stuff/build/META-INF/tasks.xml ---%<--- could work under a security manager without giving org.runtasks nondefault permissions, without letting other modules load MyTask directly, or even letting org.runtasks load unrelated classes in com.stuff. Furthermore, com.stuff.jmod could be safely passed through a compliant obfuscator without any manual configuration, since @Task would protect MyTask.class from being renamed or stripped. [1] http://blogs.oracle.com/jglick/entry/many_annotations_are_referentially_opaque From jesse.glick at oracle.com Tue Dec 27 16:43:04 2011 From: jesse.glick at oracle.com (Jesse Glick) Date: Tue, 27 Dec 2011 19:43:04 -0500 Subject: Problems with JavacTool & -L Message-ID: <4EFA6618.2010005@oracle.com> Is there some reason that (as of 1e829edfb340) RecognizedOptions.javacOptions includes OptionName.L but javacToolOptions does not? Passing -L among the 'options' to JavaCompiler.getTask throws an exception: java.lang.IllegalArgumentException: invalid flag: -L at com.sun.tools.javac.api.JavacTool.processOptions(JavacTool.java:231) at com.sun.tools.javac.api.JavacTool.getTask(JavacTool.java:199) at com.sun.tools.javac.api.JavacTool.getTask(JavacTool.java:68) (I am trying to run Jigsaw langtools on top of JRE 7, so far without success: I get "Unable to find package java.lang in classpath or bootclasspath".) From mandy.chung at oracle.com Tue Dec 27 18:37:17 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Tue, 27 Dec 2011 18:37:17 -0800 Subject: Problems with JavacTool & -L In-Reply-To: <4EFA6618.2010005@oracle.com> References: <4EFA6618.2010005@oracle.com> Message-ID: <4EFA80DD.9000602@oracle.com> On 12/27/2011 4:43 PM, Jesse Glick wrote: > Is there some reason that (as of 1e829edfb340) > RecognizedOptions.javacOptions includes OptionName.L but > javacToolOptions does not? Passing -L among the 'options' to > JavaCompiler.getTask throws an exception: > > java.lang.IllegalArgumentException: invalid flag: -L > at > com.sun.tools.javac.api.JavacTool.processOptions(JavacTool.java:231) > at com.sun.tools.javac.api.JavacTool.getTask(JavacTool.java:199) > at com.sun.tools.javac.api.JavacTool.getTask(JavacTool.java:68) > Possibly just a bug. Jon is on vacation and will wait for him to confirm. > (I am trying to run Jigsaw langtools on top of JRE 7, so far without > success: I get "Unable to find package java.lang in classpath or > bootclasspath".) Jigsaw JDK is built using jigsaw langtools on top of JDK 7 as the boot JDK. So I expect it should work to run on top of JRE 7. How do you run it? Mandy From jesse.glick at oracle.com Tue Dec 27 20:02:37 2011 From: jesse.glick at oracle.com (Jesse Glick) Date: Tue, 27 Dec 2011 23:02:37 -0500 Subject: Problems with JavacTool & -L In-Reply-To: <4EFA80DD.9000602@oracle.com> References: <4EFA6618.2010005@oracle.com> <4EFA80DD.9000602@oracle.com> Message-ID: <4EFA94DD.1070708@oracle.com> On 12/27/2011 09:37 PM, Mandy Chung wrote: > Jigsaw JDK is built using jigsaw langtools on top of JDK 7 as the boot JDK. So I expect it should work to run on top of JRE 7. That is what I hoped. > How do you run it? Using JSR 199, in a URLClassLoader which makes sure all of com.sun.source, com.sun.tools, javax.annotation.processing, javax.lang, and javax.tools are loaded from Jigsaw langtools rather than the host JRE/JDK 7. More or less like: File moduleInfoJava = ...; JavaCompiler jc = (JavaCompiler) Class.forName("com.sun.tools.javac.api.JavacTool").newInstance(); StringWriter out = new StringWriter(); CompilationTask task = jc.getTask(out, null, null, Arrays.asList("-source", "8"), null, jc.getStandardFileManager(null, null, null).getJavaFileObjects(moduleInfoJava)); task.setProcessors(...); task.call(); which returns false and prints the error about java.lang to the output writer. Loading JavacTool by name seems to be necessary as ToolProvider.getSystemJavaCompiler gives me a CCE (investigating). From jonathan.gibbons at oracle.com Wed Dec 28 02:10:58 2011 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Wed, 28 Dec 2011 10:10:58 +0000 Subject: Problems with JavacTool & -L In-Reply-To: <4EFA80DD.9000602@oracle.com> References: <4EFA6618.2010005@oracle.com> <4EFA80DD.9000602@oracle.com> Message-ID: <4EFAEB32.5000000@oracle.com> On 12/28/2011 02:37 AM, Mandy Chung wrote: > On 12/27/2011 4:43 PM, Jesse Glick wrote: >> Is there some reason that (as of 1e829edfb340) >> RecognizedOptions.javacOptions includes OptionName.L but >> javacToolOptions does not? Passing -L among the 'options' to >> JavaCompiler.getTask throws an exception: >> >> java.lang.IllegalArgumentException: invalid flag: -L >> at >> com.sun.tools.javac.api.JavacTool.processOptions(JavacTool.java:231) >> at com.sun.tools.javac.api.JavacTool.getTask(JavacTool.java:199) >> at com.sun.tools.javac.api.JavacTool.getTask(JavacTool.java:68) >> > > Possibly just a bug. Jon is on vacation and will wait for him to > confirm. Sorry, minor bug and victim of a recent refactoring. Will fix when I get back to California. Note that RecognizedOptions, JavacOption and OptionName have been merged into a single new enum, Option. > >> (I am trying to run Jigsaw langtools on top of JRE 7, so far without >> success: I get "Unable to find package java.lang in classpath or >> bootclasspath".) > > Jigsaw JDK is built using jigsaw langtools on top of JDK 7 as the boot > JDK. So I expect it should work to run on top of JRE 7. How do you > run it? This *may* be a side-effect of the above bug. Mandy is right in what she says, but that applies to javac as used to build and rebuild JDK. The -L flag is the module equivalent of -bootclasspath, and so the message you describe might occur when javac is used in module mode to compile client code, if the -L flag has not been processed correctly. -- Jon > > Mandy From jonathan.gibbons at oracle.com Wed Dec 28 02:13:51 2011 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Wed, 28 Dec 2011 10:13:51 +0000 Subject: Problems with JavacTool & -L In-Reply-To: <4EFA94DD.1070708@oracle.com> References: <4EFA6618.2010005@oracle.com> <4EFA80DD.9000602@oracle.com> <4EFA94DD.1070708@oracle.com> Message-ID: <4EFAEBDF.3050601@oracle.com> On 12/28/2011 04:02 AM, Jesse Glick wrote: > On 12/27/2011 09:37 PM, Mandy Chung wrote: >> Jigsaw JDK is built using jigsaw langtools on top of JDK 7 as the >> boot JDK. So I expect it should work to run on top of JRE 7. > > That is what I hoped. > >> How do you run it? > > Using JSR 199, in a URLClassLoader which makes sure all of > com.sun.source, com.sun.tools, javax.annotation.processing, > javax.lang, and javax.tools are loaded from Jigsaw langtools rather > than the host JRE/JDK 7. More or less like: > > File moduleInfoJava = ...; > JavaCompiler jc = (JavaCompiler) > Class.forName("com.sun.tools.javac.api.JavacTool").newInstance(); > StringWriter out = new StringWriter(); > CompilationTask task = jc.getTask(out, null, null, > Arrays.asList("-source", "8"), null, jc.getStandardFileManager(null, > null, null).getJavaFileObjects(moduleInfoJava)); > task.setProcessors(...); > task.call(); > > which returns false and prints the error about java.lang to the output > writer. Loading JavacTool by name seems to be necessary as > ToolProvider.getSystemJavaCompiler gives me a CCE (investigating). We need to revisit the impl of ToolProvider in the Jigsaw world, since the JDK 7 code creates a URLClassLoader for tools.jar is the classes are not found on the bootclasspath. tools.jar is going away, so we should probably rewrite ToolProvider to use the module services mechanism. -- Jon From mandy.chung at oracle.com Fri Dec 30 09:42:16 2011 From: mandy.chung at oracle.com (Mandy Chung) Date: Fri, 30 Dec 2011 09:42:16 -0800 Subject: Module views with exports support Message-ID: <4EFDF7F8.7040900@oracle.com> A new webrev for the module views [1] with the exports support: http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/module-views.01/ High-level summary: - Introduced a new ModuleView class and Service.Dependence Service.ProviderInfo classes. Each ModuleInfo has one or more views. - Renamed ModuleIdQuery to ModuleViewQuery - Renamed methods in the Catalog class to return ModuleId for module views. E.g. gatherLocalModuleIds -> gatherLocalModuleViewIds listModuleIds -> listModuleViewIds findModuleIds -> findModuleViewIds findLatestModuleId -> findLatestModuleViewId - Linker is updated to link with types that are exported by its dependence and only exported set of remote packages are stored in the configuration. I also considered renaming ModuleId to ModuleViewId (not done in the above webrev) but another thought would be to keep ModuleId to represents a general identifier. I'd like to get your opinion for this refactoring/renaming before moving forward. With exports, a module can access types exported from a module view that it requires. A configuration has a set of contexts and each context has a list of local classes and also a map from a package name to a remote context. Before exports, all public types are exported. For a simple application that requires the "jdk" module, the configuration size is reduced by 73% (6.3) 8.6M mlib-tip/com.greetings/0.1/config 2.3M mlib-views/com.greetings/0.1/config If it requires jdk.base, the configuration size is much smaller as jdk.base has fewer public classes than jdk. With exports, the size is reduced by 8.7% (72K). 824K mlib-base-tip/com.greetings/0.1/config 752K mlib-base-views/com.greetings/0.1/config This is the module-info.java requiring the jdk module (the current default platform module). ---------------------- module com.greetings @ 0.1 { requires org.astro; class com.greetings.Hello; } module org.astro @ 2.0 { exports org.astro; } ---------------------------- For the JDK modules with exports [2], the system module library size is reduced by 8M (7.6%): 105M jigsaw-repo/jdk/build/linux-i586/lib/modules 97M jigsaw-views/jdk/build/linux-i586/lib/modules It is still big in the current prototype and we expect that the fast configuration work will further reduce the footprint (this is work in progress). The number here is just to compare the size before and after with exports support. Thanks Mandy [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2011-December/001813.html [2] http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/module-views.01/module-info/jdk-modules-with-views.txt [3] http://cr.openjdk.java.net/~mchung/jigsaw/views-api/