From richard.warburton at gmail.com Fri Oct 5 11:52:22 2012 From: richard.warburton at gmail.com (Richard Warburton) Date: Fri, 5 Oct 2012 19:52:22 +0100 Subject: Splitting of packages between modules Message-ID: Hey all, I'm still a bit uncertain of mail aliases, but I'm assuming its acceptable to post questions/design to this alias. If this is just openjdk stuff I'm sorry and could you send me in the right direction. Suppose I have an API that I wish to split into smaller and larger versions. I have a package called com.foo.bar and which is split into some set of classes that I wish to put into the smaller API and some into the larger API. My natural instinct would be to have a module say called "Foo-Minimal" and another module called "Foo-Extras". Foo-Extras would depend on Foo-Minimal. Is it possible for me to split the com.foo.bar package between Foo-Minimal and Foo-Extras or am I required to introduce a com.foo.bar.extras package? regards, Richard From Alan.Bateman at oracle.com Sun Oct 7 07:14:10 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sun, 07 Oct 2012 15:14:10 +0100 Subject: Splitting of packages between modules In-Reply-To: References: Message-ID: <50718E32.9060903@oracle.com> On 05/10/2012 19:52, Richard Warburton wrote: > Hey all, > > I'm still a bit uncertain of mail aliases, but I'm assuming its > acceptable to post questions/design to this alias. If this is just > openjdk stuff I'm sorry and could you send me in the right direction. > > Suppose I have an API that I wish to split into smaller and larger > versions. I have a package called com.foo.bar and which is split into > some set of classes that I wish to put into the smaller API and some > into the larger API. My natural instinct would be to have a module > say called "Foo-Minimal" and another module called "Foo-Extras". > Foo-Extras would depend on Foo-Minimal. Is it possible for me to > split the com.foo.bar package between Foo-Minimal and Foo-Extras or am > I required to introduce a com.foo.bar.extras package? > > regards, > > Richard The current Jigsaw prototype supports this, using "requires local". In this case Foo_Minimal and Foo_Extras could be separate physical modules and with requires local they will be brought together at runtime so classes from both modules are loaded by the same module class loader. There's a lot issues that stem from this and it creates a lot of challenges, particularly with figuring out and specifying how certain cases should work. That plus having it easily understood by the average developer. It remains to be seen where this ultimately goes. One of benefits of moving out Jigsaw to beyond JDK 8 is that it provides an opportunity to re-examine some of the cases in JDK where this has been needed. That said, I don't know if you have a specific Foo in mind, maybe it could be refactored without breaking too many existing consumers? Are you thinking about a well established API? Does it rely heavily on package-private access? -Alan From mark.reinhold at oracle.com Tue Oct 9 15:33:39 2012 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 09 Oct 2012 22:33:39 +0000 Subject: hg: jigsaw/jigsaw/jdk: 3 new changesets Message-ID: <20121009223448.DCCFD4725A@hg.openjdk.java.net> Changeset: c0461b5e31d5 Author: mr Date: 2012-10-09 15:22 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/c0461b5e31d5 Fix whitespace ! src/share/native/common/jigsaw.c Changeset: 6769f2120b40 Author: mr Date: 2012-10-09 15:22 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/6769f2120b40 Fix Jigsaw native-library initialization logic ! src/share/native/common/jigsaw.c Changeset: 0657f2013d9c Author: mr Date: 2012-10-09 15:22 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/0657f2013d9c Don't open the module library when just getting the path ! src/share/native/common/jigsaw.c From mark.reinhold at oracle.com Wed Oct 10 14:55:46 2012 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 10 Oct 2012 14:55:46 -0700 Subject: Bug in ClassPathContext? Message-ID: <20121010215546.0D68017E6@eggemoggin.niobe.net> Chasing down a couple of failing unit tests, I ran across this code in org.openjdk.jigsaw.ClassPathContext: private static String[] toolsModules() { // jdk.tools.jre is not in this list because its classes are // included in rt.jar in the legacy image. String[] modules = new String[] { "jdk.tools.base", "jdk.tools.jaxws", "jdk.tools", "jdk.devtools" }; // ## modulepath // JDK development build - all classes are loaded by bootstrap loader return file("classes") != null ? modules : new String[0]; } I suspect the last comment and the last statement to be incorrect. Now that the VM is loading classes directly from module libraries, even in a development build, the $BUILD/classes directory is irrelevant to a running VM even in class-path mode, right? There's similar code in the nearby extModules() method. FYI the failing tests try to use com.sun.tools.javac.Main. If I move the classes directory, or use the jdk module image, then the test passes. The jdk.classpath context does, correctly, have an optional dependence upon the jdk.devtools module, which is where javac lives. - Mark From mandy.chung at oracle.com Wed Oct 10 15:53:47 2012 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 10 Oct 2012 15:53:47 -0700 Subject: Bug in ClassPathContext? In-Reply-To: <20121010215546.0D68017E6@eggemoggin.niobe.net> References: <20121010215546.0D68017E6@eggemoggin.niobe.net> Message-ID: <5075FC7B.9070803@oracle.com> On 10/10/2012 2:55 PM, mark.reinhold at oracle.com wrote: > Chasing down a couple of failing unit tests, I ran across this code in > org.openjdk.jigsaw.ClassPathContext: > > private static String[] toolsModules() { > // jdk.tools.jre is not in this list because its classes are > // included in rt.jar in the legacy image. > String[] modules = new String[] { > "jdk.tools.base", "jdk.tools.jaxws", "jdk.tools", "jdk.devtools" > }; > // ## modulepath > // JDK development build - all classes are loaded by bootstrap loader > return file("classes") != null ? modules : new String[0]; > } > > I suspect the last comment and the last statement to be incorrect. Now > that the VM is loading classes directly from module libraries, even in > a development build, the $BUILD/classes directory is irrelevant to a > running VM even in class-path mode, right? You're right - the last statement doesn't match the comment. I haven't been able to understand why the unit tests have been passing with jdk-module-image (no $BUILD/classes directory and also JPRT build) in my testing before the push. Hmm.. all modules are loaded by the bootstrap null loader except and so that works. Are the unit tests failing with jigsaw/jigsaw tip? I'm going to fix it. > There's similar code in the nearby extModules() method. yes. > FYI the failing tests try to use com.sun.tools.javac.Main. If I move > the classes directory, or use the jdk module image, then the test > passes. The jdk.classpath context does, correctly, have an optional > dependence upon the jdk.devtools module, which is where javac lives. I ran into those failing tests during the development of the classpath mode and thought that fixed properly. Sorry - one of those mistakes. Mandy From mandy.chung at oracle.com Wed Oct 10 15:58:55 2012 From: mandy.chung at oracle.com (mandy.chung at oracle.com) Date: Wed, 10 Oct 2012 22:58:55 +0000 Subject: hg: jigsaw/jigsaw/jdk: Fix windows build failure Message-ID: <20121010225940.66051472A2@hg.openjdk.java.net> Changeset: 78e4af1d8e1a Author: mchung Date: 2012-10-10 15:58 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/78e4af1d8e1a Fix windows build failure ! src/share/native/common/jigsaw.c From mark.reinhold at oracle.com Wed Oct 10 16:38:11 2012 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 10 Oct 2012 16:38:11 -0700 Subject: Bug in ClassPathContext? In-Reply-To: mandy.chung@oracle.com; Wed, 10 Oct 2012 15:53:47 PDT; <5075FC7B.9070803@oracle.com> Message-ID: <20121010233811.05ED417E6@eggemoggin.niobe.net> 2012/10/10 15:53 -0700, mandy.chung at oracle.com: > You're right - the last statement doesn't match the comment. I haven't > been able to understand why the unit tests have been passing > with jdk-module-image (no $BUILD/classes directory and also JPRT build) > in my testing before the push. Hmm.. all modules are loaded by the > bootstrap null loader except and so that works. Now that I've studied your recent URLClassPath changes, I think what's happening is that if a $BUILD/classes directory doesn't exist then the tools and ext modules are marked as bootstrap modules, so they're loaded by the bootstrap loader rather than a URLClassPath.ModuleLibraryLoader as you probably intended. What I don't yet understand is why things don't work in development mode, when these modules aren't marked as bootstrap and so should be loaded by an appropriate ModuleLibraryLoader. > Are the unit tests failing with jigsaw/jigsaw tip? Yes: test/org/openjdk/jigsaw/cli/SignedModuleFileTest.sh test/org/openjdk/jigsaw/classpath/classpath.sh test/org/openjdk/jigsaw/hello-native.sh - Mark From mandy.chung at oracle.com Wed Oct 10 20:13:34 2012 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 10 Oct 2012 20:13:34 -0700 Subject: Bug in ClassPathContext? In-Reply-To: <20121010233811.05ED417E6@eggemoggin.niobe.net> References: <20121010233811.05ED417E6@eggemoggin.niobe.net> Message-ID: <5076395E.9010400@oracle.com> Sorry for the delay in getting back to this (distracted and working on another bug). On 10/10/2012 4:38 PM, mark.reinhold at oracle.com wrote: > 2012/10/10 15:53 -0700, mandy.chung at oracle.com: >> You're right - the last statement doesn't match the comment. I haven't >> been able to understand why the unit tests have been passing >> with jdk-module-image (no $BUILD/classes directory and also JPRT build) >> in my testing before the push. Hmm.. all modules are loaded by the >> bootstrap null loader except and so that works. > Now that I've studied your recent URLClassPath changes, I think what's > happening is that if a $BUILD/classes directory doesn't exist then the > tools and ext modules are marked as bootstrap modules, so they're loaded > by the bootstrap loader rather than a URLClassPath.ModuleLibraryLoader > as you probably intended. No that's not my intent. If there is no $BUILD/classes directory (not a development build), tools/ext modules should not be loaded by the bootstrap loader. The comment is correct and the check should be (in both toolsModule() and extModules() methods): - return file("classes") != null ? modules : new String[0]; + return file("classes") == null ? modules : new String[0]; This tries to retain the classpath mode behavior as with a legacy development build. In a legacy jdk build, all classes including tools are loaded by the bootstrap loader. Perhaps this isn't highly necessary with module library. The module-info for jdk.classpath requires all jdk modules and thus we need to filter out the tools/ext modules from the list to be loaded by the bootstrap loader during system initialization. See the comment in ClassPathContext.loadClassPathConfiguration method What I don't yet understand is why things don't work in development mode, when these modules aren't marked as bootstrap and so should be loaded by an appropriate ModuleLibraryLoader. URLClassPath.ModuleLibraryLoader is only used by non-bootstrap loaders (i.e. system and extension class loader) and also used to search resource files from the bootclasspath. >> Are the unit tests failing with jigsaw/jigsaw tip? > Yes: > > test/org/openjdk/jigsaw/cli/SignedModuleFileTest.sh > test/org/openjdk/jigsaw/classpath/classpath.sh > test/org/openjdk/jigsaw/hello-native.sh Sorry for missing this. I validated my patch with the development build and jdk-module-image. I will wait for the JPRT job to finish before I push the changeset. Mandy From mark.reinhold at oracle.com Thu Oct 11 08:25:07 2012 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 11 Oct 2012 08:25:07 -0700 Subject: Bug in ClassPathContext? In-Reply-To: mandy.chung@oracle.com; Wed, 10 Oct 2012 20:13:34 PDT; <5076395E.9010400@oracle.com> Message-ID: <20121011152507.376F9C23@eggemoggin.niobe.net> 2012/10/10 20:13 -0700, mandy.chung at oracle.com: > On 10/10/2012 4:38 PM, mark.reinhold at oracle.com wrote: >> Now that I've studied your recent URLClassPath changes, I think what's >> happening is that if a $BUILD/classes directory doesn't exist then the >> tools and ext modules are marked as bootstrap modules, so they're loaded >> by the bootstrap loader rather than a URLClassPath.ModuleLibraryLoader >> as you probably intended. > > No that's not my intent. If there is no $BUILD/classes directory > (not a development build), tools/ext modules should not be loaded by > the bootstrap loader. > > The comment is correct and the check should be (in both toolsModule() > and extModules() methods): > > - return file("classes") != null ? modules : new String[0]; > + return file("classes") == null ? modules : new String[0]; > > This tries to retain the classpath mode behavior as with a legacy > development build. In a legacy jdk build, all classes including tools > are loaded by the bootstrap loader. Perhaps this isn't highly > necessary with module library. In the legacy build it was an expedient hack to load tool classes via the bootstrap class loader in a development build. I don't see any reason to preserve that behavior, especially when we now have a chance to reduce the difference between development and image builds. FYI, I tried changing the lines in question to return the array of module names unconditionally, but that didn't work. Haven't yet figured out why. - Mark From chris.hegarty at oracle.com Thu Oct 11 09:37:56 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 11 Oct 2012 17:37:56 +0100 Subject: RFR: Module constraints on targets Message-ID: <5076F5E4.8050006@oracle.com> This is a proposal to fulfill the 'module constraints on targets' requirement [1]. Essentially, support setting OS and Arch values on module-files and libraries. It meets the common use-cases, like installing platform specific modules into the wrong module library, when cross compiling, etc.. 1) Extend the Jigsaw module-file format [2] (jmod) to accommodate Operating System and Architecture values in the module header. ModuleFileHeader { u4 magic; // FileConstants.MAGIC u2 type; // FileConstants.Type.MODULE_FILE u2 major; // FileConstants.ModuleFile.MAJOR_VERSION u2 minor; // FileConstants.ModuleFile.MINOR_VERSION u8 csize; // Size of entire file, compressed u8 usize; // Space required for uncompressed contents // (upper bound; need not be exact) u2 sections; // Number of following sections u2 hashType; // One of FileConstants.ModuleFile.HashType // (applies to all hashes in this file) u2 hashLength; // Length of following hash b* hash; // Hash of entire file (except this hash) u2 osLength; // Length of following Operating System, in bytes <<< ADDED b* os; // Operating System, in Java-modified UTF-8 <<< ADDED u2 archLength; // Length of following Architecture, in bytes <<< ADDED b* arch; // Architecture, in Java-modified UTF-8 <<< ADDED } The os and arch values are Java-modified UTF-8 strings, representing the Operating System and Architecture where the module-file/library is intended to be deployed. A value of 0 is a valid value for both osLength and archLength. This effectively means that these values are unset, and the module-file/library is agnostic of Operating System and Architecture. It should be considered a candidate for any module library. 2) Update CLIs Both jmod and jpkg need to be updated to support passing of os and arch values during creation. Define two new arguments -os, -arch for both jmod and jpkg. These arguments are optional. If specified, can only be specified once. For example: ./bin/jmod create -L m.lib -os linux -arch x64 ./bin/jmod id -L m.lib path /export/chris/repos/jigsaw/constraints/test/m.lib version 0.1 os: linux arch: x64 ./bin/jmod install -L m.lib jigsaw-pkgs/jmod/jdk.logging at 8-ea.jmod java.io.IOException: Module architecture [linux-i586] does not match library [linux-x64] org.openjdk.jigsaw.cli.Command$Exception: java.io.IOException: Module architecture [linux-i586] does not match library [linux-x64] at org.openjdk.jigsaw.cli.Librarian$Install.go(Librarian.java:230) at org.openjdk.jigsaw.cli.Librarian$Install.go(Librarian.java:189) at org.openjdk.jigsaw.cli.Command.run(Command.java:100) ..... 3) Update the build to produce images and module-files that are $PLATFORM and $ARCH specific ( matching the bundles ). Webrev: http://cr.openjdk.java.net/~chegar/jigsaw/constraints.00/webrev/ -Chris. [1] http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#module-constraints-on-targets [2] http://cr.openjdk.java.net/~mr/jigsaw/notes/module-file-format/ From mandy.chung at oracle.com Thu Oct 11 12:17:36 2012 From: mandy.chung at oracle.com (Mandy Chung) Date: Thu, 11 Oct 2012 12:17:36 -0700 Subject: Bug in ClassPathContext? In-Reply-To: <20121011152507.376F9C23@eggemoggin.niobe.net> References: <20121011152507.376F9C23@eggemoggin.niobe.net> Message-ID: <50771B50.2020300@oracle.com> On 10/11/2012 8:25 AM, mark.reinhold at oracle.com wrote: > 2012/10/10 20:13 -0700, mandy.chung at oracle.com: >> On 10/10/2012 4:38 PM, mark.reinhold at oracle.com wrote: >>> Now that I've studied your recent URLClassPath changes, I think what's >>> happening is that if a $BUILD/classes directory doesn't exist then the >>> tools and ext modules are marked as bootstrap modules, so they're loaded >>> by the bootstrap loader rather than a URLClassPath.ModuleLibraryLoader >>> as you probably intended. >> No that's not my intent. If there is no $BUILD/classes directory >> (not a development build), tools/ext modules should not be loaded by >> the bootstrap loader. >> >> The comment is correct and the check should be (in both toolsModule() >> and extModules() methods): >> >> - return file("classes") != null ? modules : new String[0]; >> + return file("classes") == null ? modules : new String[0]; >> >> This tries to retain the classpath mode behavior as with a legacy >> development build. In a legacy jdk build, all classes including tools >> are loaded by the bootstrap loader. Perhaps this isn't highly >> necessary with module library. > In the legacy build it was an expedient hack to load tool classes via the > bootstrap class loader in a development build. I don't see any reason to > preserve that behavior, especially when we now have a chance to reduce > the difference between development and image builds. > > FYI, I tried changing the lines in question to return the array of module > names unconditionally, but that didn't work. Haven't yet figured out why. It doesn't work because the development build doesn't have tools.jar which is another piece of the puzzle. In a legacy image, the tools classes are loaded only if tools.jar is on the classpath. If we always load the tools classes in a module image even if -cp tools.jar is not specified, it will be a change in behavior. In addition, there are also existing applications that checks if tools.jar exists and loads the javac main class using a URLClassLoader. For compatibility, we decided to create an empty tools.jar in the module image to retain the same behavior when running in classpath mode. There is no tools.jar in the development build and thus it doesn't work. ClasssPathContext.LoaderType.isDefaultPath method tests if tools.jar exists (that's the default path for system class loader). URLClassPath.getLoader(URL) will determine if the given URL is the default path for the classpath, extension, or bootclasspath and then searches classes/resources from the appropriate modules. I agree it's a good chance to reduce the difference between development and image builds. I think a good time to do that is when we add the runtime modulepath (aka exploded module) support (Alan and I briefly talked about the exploded modules during the development of classpath mode support). With the exploded modules, the development build would not need to have a module library. We'll follow up on this. The JPRT job almost completes and tests passed on most builds except solaris-sparcv9 and solaris-x64 since there is a bug in the makefile that didn't create tools.jar (solaris 64-bit build logic is different than linux/windows/macosx 64-bit). I'll fix the makefile and push the changeset. Mandy From mark.reinhold at oracle.com Thu Oct 11 14:54:44 2012 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 11 Oct 2012 14:54:44 -0700 Subject: Bug in ClassPathContext? In-Reply-To: mandy.chung@oracle.com; Thu, 11 Oct 2012 12:17:36 PDT; <50771B50.2020300@oracle.com> Message-ID: <20121011215444.70E2EC23@eggemoggin.niobe.net> 2012/10/11 12:17 -0700, mandy.chung at oracle.com: > On 10/11/2012 8:25 AM, mark.reinhold at oracle.com wrote: >> FYI, I tried changing the lines in question to return the array of module >> names unconditionally, but that didn't work. Haven't yet figured out why. > > It doesn't work because the development build doesn't have tools.jar > which is another piece of the puzzle. In a legacy image, the tools > classes are loaded only if tools.jar is on the classpath. > If we always load the tools classes in a module image even if > -cp tools.jar is not specified, it will be a change in behavior. > In addition, there are also existing applications that checks if > tools.jar exists and loads the javac main class using a URLClassLoader. > For compatibility, we decided to create an empty tools.jar in the > module image to retain the same behavior when running in classpath mode. > There is no tools.jar in the development build and thus it doesn't work. Got it. > ClasssPathContext.LoaderType.isDefaultPath method tests if > tools.jar exists (that's the default path for system class loader). > > URLClassPath.getLoader(URL) will determine if the given URL > is the default path for the classpath, extension, or bootclasspath > and then searches classes/resources from the appropriate modules. > > I agree it's a good chance to reduce the difference between development > and image builds. I think a good time to do that is when we add > the runtime modulepath (aka exploded module) support (Alan and I > briefly talked about the exploded modules during the development > of classpath mode support). With the exploded modules, the development > build would not need to have a module library. We'll follow up on > this. Sounds good. > The JPRT job almost completes and tests passed on most builds > except solaris-sparcv9 and solaris-x64 since there is a bug > in the makefile that didn't create tools.jar (solaris 64-bit > build logic is different than linux/windows/macosx 64-bit). > I'll fix the makefile and push the changeset. Okay, thanks. - Mark From mandy.chung at oracle.com Thu Oct 11 15:49:01 2012 From: mandy.chung at oracle.com (mandy.chung at oracle.com) Date: Thu, 11 Oct 2012 22:49:01 +0000 Subject: hg: jigsaw/jigsaw/jdk: Correct the test to determine if it's a development build vs image build Message-ID: <20121011224946.35DBE472FB@hg.openjdk.java.net> Changeset: e3ad33822fa3 Author: mchung Date: 2012-10-11 15:48 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/e3ad33822fa3 Correct the test to determine if it's a development build vs image build ! make/common/Modules.gmk ! src/share/classes/org/openjdk/jigsaw/ClassPathContext.java ! test/ModulesProblemList.txt From paul.sandoz at oracle.com Tue Oct 16 01:24:07 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 16 Oct 2012 10:24:07 +0200 Subject: RFR: Module constraints on targets In-Reply-To: <5076F5E4.8050006@oracle.com> References: <5076F5E4.8050006@oracle.com> Message-ID: <64B911F5-5227-4F0D-A2A9-125DC61C3002@oracle.com> Hi Chris, Looks good. Paul. On Oct 11, 2012, at 6:37 PM, Chris Hegarty wrote: > > This is a proposal to fulfill the 'module constraints on targets' requirement [1]. Essentially, support setting OS and Arch values on module-files and libraries. It meets the common use-cases, like installing platform specific modules into the wrong module library, when cross compiling, etc.. > > 1) Extend the Jigsaw module-file format [2] (jmod) to accommodate Operating System and Architecture values in the module header. > > ModuleFileHeader { > u4 magic; // FileConstants.MAGIC > u2 type; // FileConstants.Type.MODULE_FILE > u2 major; // FileConstants.ModuleFile.MAJOR_VERSION > u2 minor; // FileConstants.ModuleFile.MINOR_VERSION > u8 csize; // Size of entire file, compressed > u8 usize; // Space required for uncompressed contents > // (upper bound; need not be exact) > u2 sections; // Number of following sections > u2 hashType; // One of FileConstants.ModuleFile.HashType > // (applies to all hashes in this file) > u2 hashLength; // Length of following hash > b* hash; // Hash of entire file (except this hash) > u2 osLength; // Length of following Operating System, in bytes <<< ADDED > b* os; // Operating System, in Java-modified UTF-8 <<< ADDED > u2 archLength; // Length of following Architecture, in bytes <<< ADDED > b* arch; // Architecture, in Java-modified UTF-8 <<< ADDED > } > > The os and arch values are Java-modified UTF-8 strings, representing the Operating System and Architecture where the module-file/library is intended to be deployed. > > A value of 0 is a valid value for both osLength and archLength. This effectively means that these values are unset, and the module-file/library is agnostic of Operating System and Architecture. It should be considered a candidate for any module library. > > 2) Update CLIs > > Both jmod and jpkg need to be updated to support passing of os and arch values during creation. > > Define two new arguments -os, -arch for both jmod and jpkg. These arguments are optional. If specified, can only be specified once. For example: > > ./bin/jmod create -L m.lib -os linux -arch x64 > > ./bin/jmod id -L m.lib > path /export/chris/repos/jigsaw/constraints/test/m.lib > version 0.1 > os: linux > arch: x64 > > ./bin/jmod install -L m.lib jigsaw-pkgs/jmod/jdk.logging at 8-ea.jmod > java.io.IOException: Module architecture [linux-i586] does not match library [linux-x64] > org.openjdk.jigsaw.cli.Command$Exception: java.io.IOException: Module architecture [linux-i586] does not match library [linux-x64] > at org.openjdk.jigsaw.cli.Librarian$Install.go(Librarian.java:230) > at org.openjdk.jigsaw.cli.Librarian$Install.go(Librarian.java:189) > at org.openjdk.jigsaw.cli.Command.run(Command.java:100) > ..... > > > 3) Update the build to produce images and module-files that are > $PLATFORM and $ARCH specific ( matching the bundles ). > > Webrev: > http://cr.openjdk.java.net/~chegar/jigsaw/constraints.00/webrev/ > > > -Chris. > > [1] http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#module-constraints-on-targets > [2] http://cr.openjdk.java.net/~mr/jigsaw/notes/module-file-format/ From henri.gomez at gmail.com Tue Oct 16 02:54:26 2012 From: henri.gomez at gmail.com (Henri Gomez) Date: Tue, 16 Oct 2012 11:54:26 +0200 Subject: build error on Jigsaw Message-ID: I get this error while building on Fedora 17 with jigsaw and new build system. /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/utilities/hashtable.hpp: In instantiation of 'int Hashtable::index_for(Symbol*) [with T = oopDesc*; short unsigned int F = 2304u]': /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/utilities/hashtable.cpp:324:16: required from here /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/utilities/hashtable.hpp:263:44: error: 'hash_to_index' was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive] /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/utilities/hashtable.hpp:263:44: note: declarations in dependent base 'BasicHashtable<2304u>' are not found by unqualified lookup /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/utilities/hashtable.hpp:263:44: note: use 'this->hash_to_index' instead /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/utilities/hashtable.cpp: In instantiation of 'void Hashtable::move_to(Hashtable*) [with T = oopDesc*; short unsigned int F = 2304u]': /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/utilities/hashtable.cpp:324:16: required from here /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/utilities/hashtable.cpp:138:7: error: 'unlink_entry' was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive] /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/utilities/hashtable.cpp:138:7: note: declarations in dependent base 'BasicHashtable<2304u>' are not found by unqualified lookup /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/utilities/hashtable.cpp:138:7: note: use 'this->unlink_entry' instead In file included from /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/runtime/sharedRuntime.hpp:34:0, from /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/gc_interface/collectedHeap.inline.hpp:33, from /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/oops/oop.inline.hpp:30, from /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/classfile/classFileParser.hpp:30, from /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/classfile/classLoader.hpp:28, from /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/classfile/systemDictionary.hpp:29, from /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/ci/ciEnv.hpp:30, from /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/ci/ciUtilities.hpp:28, from /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/ci/ciNullObject.hpp:30, from /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/ci/ciConstant.hpp:29, from /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/precompiled/precompiled.hpp:36, from /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/utilities/hashtable.cpp:25: /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/utilities/hashtable.hpp: In instantiation of 'int Hashtable::index_for(Symbol*) [with T = Symbol*; short unsigned int F = 256u]': /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/utilities/hashtable.cpp:325:16: required from here /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/utilities/hashtable.hpp:263:44: error: 'hash_to_index' was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive] /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/utilities/hashtable.hpp:263:44: note: declarations in dependent base 'BasicHashtable<256u>' are not found by unqualified lookup /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/utilities/hashtable.hpp:263:44: note: use 'this->hash_to_index' instead /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/utilities/hashtable.cpp: In instantiation of 'void Hashtable::move_to(Hashtable*) [with T = Symbol*; short unsigned int F = 256u]': /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/utilities/hashtable.cpp:325:16: required from here /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/utilities/hashtable.cpp:138:7: error: 'unlink_entry' was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive] /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/utilities/hashtable.cpp:138:7: note: declarations in dependent base 'BasicHashtable<256u>' are not found by unqualified lookup /home/builder/workspace/openjdk8-jigsaw/linux/fedora17-x86_64-builder/sources/hotspot/src/share/vm/utilities/hashtable.cpp:138:7: note: use 'this->unlink_entry' instead make[6]: *** [hashtable.o] Error 1 make[6]: *** Waiting for unfinished jobs.... From Alan.Bateman at oracle.com Tue Oct 16 02:58:41 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 16 Oct 2012 10:58:41 +0100 Subject: build error on Jigsaw In-Reply-To: References: Message-ID: <507D2FD1.7050205@oracle.com> On 16/10/2012 10:54, Henri Gomez wrote: > I get this error while building on Fedora 17 with jigsaw and new build system. > Jigsaw doesn't build with the new build system yet, hopefully this will be resolved over the next few months. jigsaw/jigsaw isn't sync'ed up with jdk8 latest and so doesn't have the patches required to build with gcc 4.7. Hopefully we will get to this soon (there is a bit of work required to sync up langtools and hotspot due to significant changes in jdk8). -Alan From henri.gomez at gmail.com Tue Oct 16 06:26:31 2012 From: henri.gomez at gmail.com (Henri Gomez) Date: Tue, 16 Oct 2012 15:26:31 +0200 Subject: build error on Jigsaw In-Reply-To: <507D2FD1.7050205@oracle.com> References: <507D2FD1.7050205@oracle.com> Message-ID: Oups, my bad, I'm using old build system for jigsaw. This error appears for Fedora 17 and not some distros like opens use 2012/10/16 Alan Bateman : > On 16/10/2012 10:54, Henri Gomez wrote: >> >> I get this error while building on Fedora 17 with jigsaw and new build >> system. >> > Jigsaw doesn't build with the new build system yet, hopefully this will be > resolved over the next few months. > > jigsaw/jigsaw isn't sync'ed up with jdk8 latest and so doesn't have the > patches required to build with gcc 4.7. Hopefully we will get to this soon > (there is a bit of work required to sync up langtools and hotspot due to > significant changes in jdk8). > > -Alan From Alan.Bateman at oracle.com Tue Oct 16 07:25:29 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 16 Oct 2012 15:25:29 +0100 Subject: RFR: Module constraints on targets In-Reply-To: <5076F5E4.8050006@oracle.com> References: <5076F5E4.8050006@oracle.com> Message-ID: <507D6E59.6070905@oracle.com> On 11/10/2012 17:37, Chris Hegarty wrote: > > This is a proposal to fulfill the 'module constraints on targets' > requirement [1]. Essentially, support setting OS and Arch values on > module-files and libraries. It meets the common use-cases, like > installing platform specific modules into the wrong module library, > when cross compiling, etc.. > : > > Webrev: > http://cr.openjdk.java.net/~chegar/jigsaw/constraints.00/webrev/ This looks very good. The filtering in RepositoryCatalog means the changes aren't intrusive to the resolver which is nice. Also the options to jmod and jpkg are simple and consistent and good to see the identify command updated too. I see you've got the comment "## find a better name" in ModuleArchitecture. The name you have works for module files, probably less so for module libraries, particularly the Library.architecture() method. I would be tempted to change to something like Architecture or Platform until we get a better name but what you have is okay too. One small comment on ModuleArchitecture is that I'm not sure about that createFromOS and createFromArch are useful and I would be tempted to leave them out unless we need them. Otherwise I think this is a good work. -Alan. From Alan.Bateman at oracle.com Tue Oct 16 07:44:25 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 16 Oct 2012 15:44:25 +0100 Subject: RFR: Module constraints on targets In-Reply-To: <507D6E59.6070905@oracle.com> References: <5076F5E4.8050006@oracle.com> <507D6E59.6070905@oracle.com> Message-ID: <507D72C9.1060708@oracle.com> On 16/10/2012 15:25, Alan Bateman wrote: > : > > I see you've got the comment "## find a better name" in > ModuleArchitecture. The name you have works for module files, probably > less so for module libraries, particularly the Library.architecture() > method. I would be tempted to change to something like Architecture or > Platform until we get a better name but what you have is okay too. A possible better name TargetPlatform unless we think of something better. From mark.reinhold at oracle.com Tue Oct 16 08:13:32 2012 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 16 Oct 2012 15:13:32 +0000 Subject: hg: jigsaw/jigsaw/jdk: 2 new changesets Message-ID: <20121016151413.E8C86473A4@hg.openjdk.java.net> Changeset: c640b6feaf8a Author: mr Date: 2012-10-15 14:17 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/c640b6feaf8a Remove tabs and trailing whitespace ! make/tools/classanalyzer/src/com/sun/classanalyzer/AnnotatedDependency.java ! make/tools/classanalyzer/src/com/sun/classanalyzer/BootAnalyzer.java ! make/tools/classanalyzer/src/com/sun/classanalyzer/ConstantPoolParser.java ! make/tools/classanalyzer/src/com/sun/classanalyzer/Dependence.java ! make/tools/classanalyzer/src/com/sun/classanalyzer/JigsawModuleBuilder.java ! make/tools/classanalyzer/src/com/sun/classanalyzer/Module.java ! make/tools/classanalyzer/src/com/sun/classanalyzer/ModuleBuilder.java ! make/tools/classanalyzer/src/com/sun/classanalyzer/ModuleInfo.java ! src/share/bin/java.c ! src/share/classes/java/lang/Package.java ! src/share/classes/java/lang/module/ModuleClassLoader.java ! src/share/classes/java/lang/module/ModuleInfo.java ! src/share/classes/java/lang/module/ModuleInfoReader.java ! src/share/classes/java/lang/module/ModuleNotPresentException.java ! src/share/classes/java/lang/module/ModuleView.java ! src/share/classes/java/lang/module/RequireOptionalModule.java ! src/share/classes/java/lang/module/ServiceDependence.java ! src/share/classes/java/net/URLClassLoader.java ! src/share/classes/java/util/ResourceBundle.java ! src/share/classes/java/util/ServiceLoader.java ! src/share/classes/java/util/jar/JarFile.java ! src/share/classes/org/openjdk/jigsaw/BaseContext.java ! src/share/classes/org/openjdk/jigsaw/BootLoader.java ! src/share/classes/org/openjdk/jigsaw/Catalog.java ! src/share/classes/org/openjdk/jigsaw/ClassPathContext.java ! src/share/classes/org/openjdk/jigsaw/Configuration.java ! src/share/classes/org/openjdk/jigsaw/Context.java ! src/share/classes/org/openjdk/jigsaw/ContextBuilder.java ! src/share/classes/org/openjdk/jigsaw/ContextSet.java ! src/share/classes/org/openjdk/jigsaw/Linker.java ! src/share/classes/org/openjdk/jigsaw/LinkingContext.java ! src/share/classes/org/openjdk/jigsaw/Loader.java ! src/share/classes/org/openjdk/jigsaw/LoaderPool.java ! src/share/classes/org/openjdk/jigsaw/Manifest.java ! src/share/classes/org/openjdk/jigsaw/ModuleFile.java ! src/share/classes/org/openjdk/jigsaw/ModuleFileParserException.java ! src/share/classes/org/openjdk/jigsaw/ModuleServiceLoader.java ! src/share/classes/org/openjdk/jigsaw/PathContext.java ! src/share/classes/org/openjdk/jigsaw/PathLinker.java ! src/share/classes/org/openjdk/jigsaw/PersistentTreeMap.java ! src/share/classes/org/openjdk/jigsaw/Platform.java ! src/share/classes/org/openjdk/jigsaw/PublishedRepository.java ! src/share/classes/org/openjdk/jigsaw/RemoteRepository.java ! src/share/classes/org/openjdk/jigsaw/Repository.java ! src/share/classes/org/openjdk/jigsaw/RepositoryCatalog.java ! src/share/classes/org/openjdk/jigsaw/Resolution.java ! src/share/classes/org/openjdk/jigsaw/Resolver.java ! src/share/classes/org/openjdk/jigsaw/SimpleLibrary.java ! src/share/classes/org/openjdk/jigsaw/Trace.java ! src/share/classes/org/openjdk/jigsaw/cli/Commands.java ! src/share/classes/sun/jvmstat/monitor/MonitoredVmUtil.java ! src/share/classes/sun/misc/Launcher.java ! src/share/classes/sun/misc/URLClassPath.java ! src/share/classes/sun/reflect/LangReflectAccess.java ! src/share/classes/sun/reflect/misc/ReflectUtil.java ! src/share/classes/sun/security/jca/Providers.java ! src/share/classes/sun/tools/jar/Main.java ! src/share/classes/sun/tools/jar/ModuleInfo.java ! src/share/native/common/jigsaw.c ! src/share/native/common/jigsaw.h ! src/share/native/org/openjdk/jigsaw/ClassPathContext.c ! src/solaris/demo/jigsaw/basic/src/org/hello/swing/Main.java ! test/java/lang/module/_ModuleInfoReader.java ! test/org/openjdk/jigsaw/ContextBuilder.java ! test/org/openjdk/jigsaw/ModuleInfoBuilder.java ! test/org/openjdk/jigsaw/TrivialWebServer.java ! test/org/openjdk/jigsaw/_Configurator.java ! test/org/openjdk/jigsaw/_Library.java ! test/org/openjdk/jigsaw/_PublishedRepository.java ! test/org/openjdk/jigsaw/_RemoteRepository.java ! test/org/openjdk/jigsaw/classpath/ClassPathLoader.java ! test/org/openjdk/jigsaw/cli/ImportPrivateKey.java Changeset: 10c39db40644 Author: mr Date: 2012-10-16 08:12 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/10c39db40644 Re-enable jcheck, laxly + .jcheck/conf From mark.reinhold at oracle.com Tue Oct 16 09:12:53 2012 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 16 Oct 2012 09:12:53 -0700 Subject: hg: jigsaw/jigsaw/jdk: 2 new changesets In-Reply-To: mark.reinhold@oracle.com; Tue, 16 Oct 2012 15:13:32 -0000; <20121016151413.E8C86473A4@hg.openjdk.java.net> Message-ID: <20121016161253.B3B907BA@eggemoggin.niobe.net> 2012/10/16 8:13 -0700, mark.reinhold at oracle.com: > Changeset: c640b6feaf8a > Author: mr > Date: 2012-10-15 14:17 -0700 > URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/c640b6feaf8a > > Remove tabs and trailing whitespace > > ... > > Changeset: 10c39db40644 > Author: mr > Date: 2012-10-16 08:12 -0700 > URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/10c39db40644 > > Re-enable jcheck, laxly FYI, a bunch of gratuitous whitespace crept into our code recently, which makes it annoying to edit in (at least) Emacs. I eradicated it and then re-enabled jcheck, but with appropriate lax options so it won't complain about missing bug numbers or odd mq tags. - Mark From mark.reinhold at oracle.com Tue Oct 16 09:38:15 2012 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Tue, 16 Oct 2012 09:38:15 -0700 Subject: RFR: Module constraints on targets In-Reply-To: chris.hegarty@oracle.com; Thu, 11 Oct 2012 17:37:56 BST; <5076F5E4.8050006@oracle.com> Message-ID: <20121016163815.6CBD87BA@eggemoggin.niobe.net> 2012/10/11 9:37 -0700, chris.hegarty at oracle.com: > 1) Extend the Jigsaw module-file format [2] (jmod) to accommodate Operating > System and Architecture values in the module header. > > ModuleFileHeader { > ... > u2 osLength; // Length of following Operating System, in bytes <<< ADDED > b* os; // Operating System, in Java-modified UTF-8 <<< ADDED > u2 archLength; // Length of following Architecture, in bytes <<< ADDED > b* arch; // Architecture, in Java-modified UTF-8 <<< ADDED > } > > The os and arch values are Java-modified UTF-8 strings, representing the > Operating System and Architecture where the module-file/library is intended to > be deployed. > > A value of 0 is a valid value for both osLength and archLength. This > effectively means that these values are unset, and the module-file/library is > agnostic of Operating System and Architecture. It should be considered a > candidate for any module library. Is it valid to specify an OS but not an architecture, or the other way around? You can write OS-dependent code in otherwise pure Java, so we should allow osLength > 0 && archLength == 0. If you specify an architecture, however, then you must specify an OS, so archLength > 0 implies osLength > 0. > 2) Update CLIs > > Both jmod and jpkg need to be updated to support passing of os and arch values > during creation. > > Define two new arguments -os, -arch for both jmod and jpkg. These arguments are > optional. If specified, can only be specified once. For example: > > ./bin/jmod create -L m.lib -os linux -arch x64 Please stick to the existing GNU-style option syntax: These should be --os and --arch. > ./bin/jmod id -L m.lib > path /export/chris/repos/jigsaw/constraints/test/m.lib > version 0.1 > os: linux > arch: x64 s/os:/os/ s/arch:/arch/ > ... > > Webrev: > http://cr.openjdk.java.net/~chegar/jigsaw/constraints.00/webrev/ I'll have a closer look at this shortly. - Mark From mandy.chung at oracle.com Tue Oct 16 09:45:20 2012 From: mandy.chung at oracle.com (mandy.chung at oracle.com) Date: Tue, 16 Oct 2012 16:45:20 +0000 Subject: hg: jigsaw/jigsaw/jdk: 2 new changesets Message-ID: <20121016164556.BFA26473A9@hg.openjdk.java.net> Changeset: 153d78d78f13 Author: mchung Date: 2012-10-16 09:36 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/153d78d78f13 Missing jaxws packages when imported from JDK ! make/common/internal/Defs-jaxws.gmk Changeset: 9258ce765f2b Author: mchung Date: 2012-10-16 09:37 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/9258ce765f2b Add jdk_jigsaw to the default testset ! make/jprt.properties From mandy.chung at oracle.com Tue Oct 16 09:46:06 2012 From: mandy.chung at oracle.com (mandy.chung at oracle.com) Date: Tue, 16 Oct 2012 16:46:06 +0000 Subject: hg: jigsaw/jigsaw: Add jdk_jigsaw to the default testset Message-ID: <20121016164606.7E5DE473AA@hg.openjdk.java.net> Changeset: 87800c69975c Author: mchung Date: 2012-10-16 09:37 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/rev/87800c69975c Add jdk_jigsaw to the default testset ! make/jprt.properties From chris.hegarty at oracle.com Tue Oct 16 10:08:25 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 16 Oct 2012 18:08:25 +0100 Subject: RFR: Module constraints on targets In-Reply-To: <20121016163815.6CBD87BA@eggemoggin.niobe.net> References: <20121016163815.6CBD87BA@eggemoggin.niobe.net> Message-ID: <507D9489.3090007@oracle.com> On 10/16/2012 05:38 PM, mark.reinhold at oracle.com wrote: > 2012/10/11 9:37 -0700, chris.hegarty at oracle.com: >> 1) Extend the Jigsaw module-file format [2] (jmod) to accommodate Operating >> System and Architecture values in the module header. >> >> ModuleFileHeader { >> ... >> u2 osLength; // Length of following Operating System, in bytes <<< ADDED >> b* os; // Operating System, in Java-modified UTF-8 <<< ADDED >> u2 archLength; // Length of following Architecture, in bytes <<< ADDED >> b* arch; // Architecture, in Java-modified UTF-8 <<< ADDED >> } >> >> The os and arch values are Java-modified UTF-8 strings, representing the >> Operating System and Architecture where the module-file/library is intended to >> be deployed. >> >> A value of 0 is a valid value for both osLength and archLength. This >> effectively means that these values are unset, and the module-file/library is >> agnostic of Operating System and Architecture. It should be considered a >> candidate for any module library. > > Is it valid to specify an OS but not an architecture, or the other way > around? The current implementation/webrev supports optionally specifying an OS or Architecture. > You can write OS-dependent code in otherwise pure Java, so we should > allow osLength > 0 && archLength == 0. > > If you specify an architecture, however, then you must specify an OS, > so archLength > 0 implies osLength > 0. Makes sense, I'll update the tools to enforce this restriction. > >> 2) Update CLIs >> >> Both jmod and jpkg need to be updated to support passing of os and arch values >> during creation. >> >> Define two new arguments -os, -arch for both jmod and jpkg. These arguments are >> optional. If specified, can only be specified once. For example: >> >> ./bin/jmod create -L m.lib -os linux -arch x64 > > Please stick to the existing GNU-style option syntax: These should be > --os and --arch. Yes, will do. > >> ./bin/jmod id -L m.lib >> path /export/chris/repos/jigsaw/constraints/test/m.lib >> version 0.1 >> os: linux >> arch: x64 > > s/os:/os/ > s/arch:/arch/ Got it. > >> ... >> >> Webrev: >> http://cr.openjdk.java.net/~chegar/jigsaw/constraints.00/webrev/ > > I'll have a closer look at this shortly. Thanks, I won't update the webrev yet, with the above comments, until you get back on this. -Chris. > > - Mark > From henri.gomez at gmail.com Wed Oct 17 06:38:02 2012 From: henri.gomez at gmail.com (Henri Gomez) Date: Wed, 17 Oct 2012 15:38:02 +0200 Subject: build error on Jigsaw In-Reply-To: References: <507D2FD1.7050205@oracle.com> Message-ID: Any news about this error ? I retried with latest code and problem is still here for Fedora 17 (64bits - gcc 4.7.0) No problem to build it with openSuse 12.1 (64bits - gcc 4.6.2) 2012/10/16 Henri Gomez : > Oups, my bad, I'm using old build system for jigsaw. > > This error appears for Fedora 17 and not some distros like opens use > > 2012/10/16 Alan Bateman : >> On 16/10/2012 10:54, Henri Gomez wrote: >>> >>> I get this error while building on Fedora 17 with jigsaw and new build >>> system. >>> >> Jigsaw doesn't build with the new build system yet, hopefully this will be >> resolved over the next few months. >> >> jigsaw/jigsaw isn't sync'ed up with jdk8 latest and so doesn't have the >> patches required to build with gcc 4.7. Hopefully we will get to this soon >> (there is a bit of work required to sync up langtools and hotspot due to >> significant changes in jdk8). >> >> -Alan From Alan.Bateman at oracle.com Wed Oct 17 06:44:34 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 17 Oct 2012 14:44:34 +0100 Subject: build error on Jigsaw In-Reply-To: References: <507D2FD1.7050205@oracle.com> Message-ID: <507EB642.4090302@oracle.com> On 17/10/2012 14:38, Henri Gomez wrote: > Any news about this error ? > > I retried with latest code and problem is still here for Fedora 17 > (64bits - gcc 4.7.0) > No problem to build it with openSuse 12.1 (64bits - gcc 4.6.2) > As I said, we haven't sync'ed up jigsaw/jigsaw for a few builds so don't have the hotspot changes to allow the code be compiled with gcc 4.7. There is a complex merge to do but hopefully we will be up to date in a few weeks. In the mean-time then you could try the merge yourself, alternatively downgrade gcc. There might also be gcc options that you could use that would ignore these warnings. -Alan From henri.gomez at gmail.com Wed Oct 17 07:46:53 2012 From: henri.gomez at gmail.com (Henri Gomez) Date: Wed, 17 Oct 2012 16:46:53 +0200 Subject: build error on Jigsaw In-Reply-To: <507EB642.4090302@oracle.com> References: <507D2FD1.7050205@oracle.com> <507EB642.4090302@oracle.com> Message-ID: > As I said, we haven't sync'ed up jigsaw/jigsaw for a few builds so don't > have the hotspot changes to allow the code be compiled with gcc 4.7. There > is a complex merge to do but hopefully we will be up to date in a few weeks. > In the mean-time then you could try the merge yourself, alternatively > downgrade gcc. There might also be gcc options that you could use that would > ignore these warnings. Ok, no problem, Fedora 17 will wait a big longer for Jigsaw. Thanks From hendy at soluvas.com Wed Oct 17 09:31:12 2012 From: hendy at soluvas.com (Hendy Irawan) Date: Wed, 17 Oct 2012 23:31:12 +0700 Subject: modules vs module "interfaces"? Message-ID: Hi, I'm new in Jigsaw. >From reading the specs it seems there is no distinction between module name and a module "specification/interface/API" name. Basically a similar concept to the distinction between classes and interfaces. A real world usecase is in logging. SLF4J API 1.7 is implemented by Logback, SLF4J-JDK14, and also Pax Logging. During build time, an application JAR would depend to e.g. org.slf4j @ 1.7.0.0; then org.slf4j @ [1.7.0.0,2.0.0.0) or any other module which conforms to slf4j 1.7 can be provided for building. For testing and runtime, both a module conforming to org.slf4j @ 1.7.0.0 and an implementation, such as logback or others is required. The point here is that even though the JAR was originally built with org.slf4j @ 1.7.0.0, in other cases (testing/runtime/etc.) the concrete org.slf4j module is not explicitly required, because it can be replaced with another module, like org.ops4j.pax.logging @ 2.0.0, which happens to implement org.slf4j @ 1.7.0.0, and also org.apache.commons.logging @ 1.1.1, and also log4j packages, and also JUL packages, and also OSGi logging service. (so Pax Logging is actually a 5-module-in-1 module) In OSGi this is handled using Import-Package mechanism, but I haven't seen similar to this in Jigsaw. Thank you. -- Hendy Irawan - on Twitter - on LinkedIn Web Developer | Bippo Indonesia | Akselerator Bisnis | Bandung From mark.reinhold at oracle.com Wed Oct 17 09:33:07 2012 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Wed, 17 Oct 2012 16:33:07 +0000 Subject: hg: jigsaw/jigsaw/jdk: 3 new changesets Message-ID: <20121017163408.CFEB6473EA@hg.openjdk.java.net> Changeset: 264e44427ac1 Author: mr Date: 2012-10-17 09:32 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/264e44427ac1 No executable files Changeset: b62337dc7f1c Author: mr Date: 2012-10-17 09:32 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/b62337dc7f1c Retain db files in PersistentTreeMap tests ! test/org/openjdk/jigsaw/PersistentTreeMap/Basher.java ! test/org/openjdk/jigsaw/PersistentTreeMap/Basic.java Changeset: 7c902084942a Author: mr Date: 2012-10-17 09:32 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/7c902084942a Remove annoying INFO: messages from build ! make/common/Defs-linux.gmk ! make/common/Defs-solaris.gmk ! make/common/Defs-windows.gmk From Alan.Bateman at oracle.com Wed Oct 17 09:48:51 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 17 Oct 2012 17:48:51 +0100 Subject: modules vs module "interfaces"? In-Reply-To: References: Message-ID: <507EE173.8020907@oracle.com> On 17/10/2012 17:31, Hendy Irawan wrote: > Hi, > > I'm new in Jigsaw. > > > From reading the specs it seems there is no distinction between module name > and a module "specification/interface/API" name. > Basically a similar concept to the distinction between classes and > interfaces. > > A real world usecase is in logging. SLF4J API 1.7 is implemented by > Logback, SLF4J-JDK14, and also Pax Logging. > > During build time, an application JAR would depend to e.g. org.slf4j @ > 1.7.0.0; then org.slf4j @ [1.7.0.0,2.0.0.0) or any other module which > conforms to slf4j 1.7 can be provided for building. > For testing and runtime, both a module conforming to org.slf4j @ 1.7.0.0 > and an implementation, such as logback or others is required. > > The point here is that even though the JAR was originally built with > org.slf4j @ 1.7.0.0, in other cases (testing/runtime/etc.) the concrete > org.slf4j module is not explicitly required, because it can be replaced > with another module, like org.ops4j.pax.logging @ 2.0.0, which happens to > implement org.slf4j @ 1.7.0.0, and also org.apache.commons.logging @ 1.1.1, > and also log4j packages, and also JUL packages, and also OSGi logging > service. (so Pax Logging is actually a 5-module-in-1 module) > > In OSGi this is handled using Import-Package mechanism, but I haven't seen > similar to this in Jigsaw. > > Thank you. Have a look at services: http://openjdk.java.net/projects/jigsaw/doc/openjdk-jigsaw-modular-services.pdf -Alan From hendy at soluvas.com Wed Oct 17 10:06:33 2012 From: hendy at soluvas.com (Hendy Irawan) Date: Thu, 18 Oct 2012 00:06:33 +0700 Subject: ServiceLoader.load() alternative Message-ID: Hi Jigsaw, After reading openjdk-jigsaw-modular-services.pdf. Given the issues with static methods like ServiceLoader.load(), particular in dynamic environments (OSGi) and even more in scoped frameworks (like OSGi regions), is it planned to offer a new recommended approach for locating services? Instead of ServiceLoader.load(), the mechanism to locate a service would be in a ServiceLookup interface, that can be provided to the consumer via injection or other means... (or perhaps via static calls for simple cases). This will be useful for testing, mocking, reusability, and also dynamic frameworks like OSGi. ServiceLookup can be scoped to a specific region, and the implementation can even change during runtime (e.g. by proxying). Even better if an async version of ServiceLookup is provided, or a sync version with a timeout. This will allow expanded use of ServiceLookup for distributed environments, or stricter timing requirements. ServiceLoader.load() will then be deprecated, or at least given a note that it only supports a subset of the Jigsaw service capabilities. New implementors should use ServiceLookup interface. Is this feasible? -- Hendy Irawan - on Twitter - on LinkedIn Web Developer | Bippo Indonesia | Akselerator Bisnis | Bandung From Alan.Bateman at oracle.com Wed Oct 17 14:06:15 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 17 Oct 2012 22:06:15 +0100 Subject: ServiceLoader.load() alternative In-Reply-To: References: Message-ID: <507F1DC7.1030106@oracle.com> On 17/10/2012 18:06, Hendy Irawan wrote: > Hi Jigsaw, > > After reading openjdk-jigsaw-modular-services.pdf. Given the issues with > static methods like ServiceLoader.load(), particular in dynamic > environments (OSGi) and even more in scoped frameworks (like OSGi regions), > is it planned to offer a new recommended approach for locating services? > > Instead of ServiceLoader.load(), the mechanism to locate a service would be > in a ServiceLookup interface, that can be provided to the consumer via > injection or other means... (or perhaps via static calls for simple cases). > This will be useful for testing, mocking, reusability, and also dynamic > frameworks like OSGi. ServiceLookup can be scoped to a specific region, and > the implementation can even change during runtime (e.g. by proxying). > > Even better if an async version of ServiceLookup is provided, or a sync > version with a timeout. This will allow expanded use of ServiceLookup for > distributed environments, or stricter timing requirements. > > ServiceLoader.load() will then be deprecated, or at least given a note that > it only supports a subset of the Jigsaw service capabilities. New > implementors should use ServiceLookup interface. > > Is this feasible? > I would suggest coming back to this when the container support [1] is in place. It may be that the container support will require updates to ServiceLoader to allow the container arrange for additional service providers to be included in the relative configuration for the application. -Alan [1] http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12#_3 From jaroslav.tulach at oracle.com Thu Oct 18 06:42:07 2012 From: jaroslav.tulach at oracle.com (Jaroslav Tulach) Date: Thu, 18 Oct 2012 15:42:07 +0200 Subject: ServiceLoader.load() alternative In-Reply-To: References: Message-ID: <4223276.LbHTG2a5xO@logoutik> Dne ?t 18. ??jna 2012 00:06:33, Hendy Irawan napsal(a): > Hi Jigsaw, > > After reading openjdk-jigsaw-modular-services.pdf. Given the issues with > static methods like ServiceLoader.load(), particular in dynamic > environments (OSGi) and even more in scoped frameworks (like OSGi regions), > is it planned to offer a new recommended approach for locating services? > > Instead of ServiceLoader.load(), the mechanism to locate a service would be > in a ServiceLookup interface, that can be provided to the consumer via > injection or other means... (or perhaps via static calls for simple cases). > This will be useful for testing, mocking, reusability, and also dynamic > frameworks like OSGi. ServiceLookup can be scoped to a specific region, and > the implementation can even change during runtime (e.g. by proxying). > > Even better if an async version of ServiceLookup is provided, or a sync > version with a timeout. This will allow expanded use of ServiceLookup for > distributed environments, or stricter timing requirements. > > ServiceLoader.load() will then be deprecated, or at least given a note that > it only supports a subset of the Jigsaw service capabilities. New > implementors should use ServiceLookup interface. > > Is this feasible? It is hard to resist a temptation to share the NetBeans experience: http://bits.netbeans.org/dev/javadoc/org-openide-util-lookup/ The essential difference to ServiceLoader seems to be the fact that Lookup.getDefault is pluggable, e.g. people can implement its own behavior to suite their needs - we have implementation that bridges to OSGi's service registry, for example. -jt From mark.reinhold at oracle.com Fri Oct 19 11:31:26 2012 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Fri, 19 Oct 2012 11:31:26 -0700 Subject: RFR: Module constraints on targets In-Reply-To: chris.hegarty@oracle.com; Thu, 11 Oct 2012 17:37:56 BST; <5076F5E4.8050006@oracle.com> Message-ID: <20121019183126.40861F68@eggemoggin.niobe.net> 2012/10/11 9:37 -0700, chris.hegarty at oracle.com: > ... > > Webrev: > http://cr.openjdk.java.net/~chegar/jigsaw/constraints.00/webrev/ Overall this looks pretty good; detailed comments below. After this change goes in we need to think further about repository layout and module-file naming conventions. The filtering mechanism you've put into the repository code is fine, but we need a story for how to create a repository that contains multiple instances of the same module built for different os/arch combinations. ModuleArchitecture.java I think one reason this class is hard to name is that it's serving two purposes. It can be a concrete os/arch pair or it can be a pattern against which a concrete pair can be matched. I suggest splitting this into two classes, say Host and HostPattern. The first is just a simple wrapper around os and arch strings, and has a static method to return a Host object describing the os/arch on which we're running. The second has the same instance methods as the first -- but it's not a subtype -- and also includes a boolean matches(Host h) method. Then a Library can have a Host host() method, and a ModuleFileHeader can have a HostPattern getHostPattern() method. The string representations of these classes should be unambiguous, and always in lower case. E.g., "linux-x86", "linux-any", or "any". As I suggested previously, patterns like "any-x86" should be disallowed, at least until someone comes up with a compelling use case. (At some point we need to rationalize all our cpu-architecture names, but that's a different topic.) SimpleLibrary A Library will always have a Host, so wherever you're passing around separate os/arch string pairs I'd pass a Host object instead. [204-211] There's no need to distinguish null from the empty string, since neither can occur, so you can just use writeUTF(String) here, and readUTF() when reading [234-238]. [1260] Throw a ConfigurationException for now, not an IOException. We should consider later whether to have a separate exception type for this kind of failure. Likewise on [1567]. Librarian [74-75] If the os isn't specified on the command line then the host os should be used, and likewise for arch. The right place to arrange for that is probably in the Host.create method. [137-141] You can just out.format("host %s%n", lib.host()) here. [357] When creating a new library, if it has a parent then we should ensure that the parent's os/arch is identical. ModuleFile [662-702] Having a builder for ModuleFileHeader seems like overkill, but maybe that's just me. ModuleFileWriter [323-339] How much slower is this going to be than generating the hash directly, as we do now? Why is using a non-validating module-file parser here a better approach? (If you retained the current code then you wouldn't need the dynamic dispatching in ModuleFile.readFileHash [573-578].) RepositoryCatalog [129] Shouldn't this be using .matches rather than .equals ? RemoteRepository [340] s/findlocalModuleIds/findModuleIds/ [351-357] Why a separate method here? I don't see it used anywhere else. I'd just fold line 356 into the find(Local)ModuleIds method. Packager If native commands or libraries are specified on the command line but os or arch aren't given then we should default them to the values for the local system. hello-repo.sh This test is meant to be very basic, like the other hello tests. You've added essentially two new tests at the end; I'd break these out into separate files. - Mark From SEK at VIAUC.DK Mon Oct 22 06:37:54 2012 From: SEK at VIAUC.DK (Stephan Erbs Korsholm (SEK)) Date: Mon, 22 Oct 2012 15:37:54 +0200 Subject: Dependency leaks and embedded Java Message-ID: To jigsaw-dev, Just want to let you know about our research project called the HVM, which is an effort to (re)enable the Java SE Platform for low-end embedded devices. The targets we have in mind are micro-controllers with approx. 8kB (or more) RAM and approx 64 kB (or more) ROM (see http://icelab.dk). On such devices the monolithic nature of most Java execution platforms is obviously a challenge. Using some tricks we are able to execute code like e.g the following on low-end embedded devices: ArrayList list = new ArrayList(); list.add("foo"); list.add("horse"); list.add("fish"); list.add("London"); list.add("Jack"); Object[] array = list.toArray(); Arrays.sort(array); Most other embedded Java environments seek to get around the monolithic JDK problem by developing their own JDKs. The goal of the HVM is to actually allow developers to use e.g. Java 6 or 7 to run code like above on low-end embedded devices. We use static program analysis to calculate a conservative approximation to the set of classes and methods that may be required to execute the code above. This set we call the 'dependency extent' of the program. The method applied to do program specialization (the act of shrinking a given application to contain only actual dependencies) is an instance of the k-CFA algorithm and also known as variable-type analysis (VTA). Code like above can be analyzed and results in a manageable dependency extent, but code like e.g. System.out.println("Hello") cannot. For 'HelloWorld' the dependency extent becomes impractical for low-end embedded devices. For the 'HelloWorld' scenario we say that a 'dependency leak' has been introduced. An interesting observation that we have made is that for Java 6, many of the collection classes and other utility class like e.g. String and StringBuffer, the dependency extent is reasonable, but for Java 7 the same classes leak a lot more, and classes that we could previously handle can no longer be part of our embedded programs :-( I find the jigsaw project very interesting in relation to our efforts, and look forward to eventually explore the dependency extents of our programs based on a modularized JDK. To ease the use of Java for embedded devices a modularized JDK (and VM) would definitely be attractive. Since we base the HVM program specialization on static program analysis we have to preclude the use of dynamic class-loading. Other similar efforts exists e.g. ProGuard. Kind regards, - Stephan Korsholm, VIAUC, DK From jaroslav.tulach at oracle.com Tue Oct 23 00:10:41 2012 From: jaroslav.tulach at oracle.com (Jaroslav Tulach) Date: Tue, 23 Oct 2012 09:10:41 +0200 Subject: java.mini? was: Dependency leaks and embedded Java In-Reply-To: References: Message-ID: <1679143.Nj5tQ95O0u@logoutik> I am also trying to play with Java in a very small environment (but probably with more memory than 72kb) and it would help me if the smallest profile would not be as bloated as currently proposed java.base. > Most other embedded Java environments seek to get around the > monolithic JDK problem by developing their own JDKs. Right. That is sort of what I am doing. But I don't consider forking good from a long term point of view. > ....the dependency extent is reasonable, but for Java 7 the > same classes leak a lot more, and classes that we could previously > handle can no longer be part of our embedded programs :-( This is my observation as well. A Throwable has now a direct dependency on ArrayList (as of JDK7)! That really does not make the transitive closure of Object smaller. A modularization has to be driven by a need, I know. So far the only need jigsaw team tried to achieve was to fit Hotspot on mobile devices and for that java.base is enough (as demonstrated @ J1). However for certain environments it is clearly to big. Can't jigsaw also define the smallest possible profile (e.g. with classes that are directly needed during compilation by Javac like String, StringBuilder, Enum, Throwable)? Having such "java.mini" profile would help porting Java to unusual small environments. It would ultimately lead to smaller fragmentation of Java - as it is better if the smallest profile is standardized than created by independent individuals. Just my 2 K? to provide something to think about during sleepless nights. -jt Dne Po 22. ??jna 2012 15:37:54, Stephan Erbs Korsholm napsal(a): > To jigsaw-dev, > > Just want to let you know about our research project called the HVM, > which is an effort to (re)enable the Java SE Platform for low-end > embedded devices. The targets we have in mind are micro-controllers > with approx. 8kB (or more) RAM and approx 64 kB (or more) ROM (see > http://icelab.dk). > > On such devices the monolithic nature of most Java execution platforms > is obviously a challenge. Using some tricks we are able to execute > code like e.g the following on low-end embedded devices: > > ArrayList list = new ArrayList(); > list.add("foo"); > list.add("horse"); > list.add("fish"); > list.add("London"); > list.add("Jack"); > Object[] array = list.toArray(); > Arrays.sort(array); > > Most other embedded Java environments seek to get around the > monolithic JDK problem by developing their own JDKs. The goal of the > HVM is to actually allow developers to use e.g. Java 6 or 7 to run > code like above on low-end embedded devices. > > We use static program analysis to calculate a conservative > approximation to the set of classes and methods that may be required > to execute the code above. This set we call the 'dependency extent' of > the program. The method applied to do program specialization (the act > of shrinking a given application to contain only actual dependencies) > is an instance of the k-CFA algorithm and also known as variable-type > analysis (VTA). > > Code like above can be analyzed and results in a manageable dependency > extent, but code like e.g. System.out.println("Hello") cannot. For > 'HelloWorld' the dependency extent becomes impractical for low-end > embedded devices. For the 'HelloWorld' scenario we say that a > 'dependency leak' has been introduced. > > An interesting observation that we have made is that for Java 6, many > of the collection classes and other utility class like e.g. String and > StringBuffer, the dependency extent is reasonable, but for Java 7 the > same classes leak a lot more, and classes that we could previously > handle can no longer be part of our embedded programs :-( > > I find the jigsaw project very interesting in relation to our efforts, and > look forward to eventually explore the dependency extents of our programs > based on a modularized JDK. > > To ease the use of Java for embedded devices a modularized JDK (and > VM) would definitely be attractive. > > Since we base the HVM program specialization on static program > analysis we have to preclude the use of dynamic class-loading. Other > similar efforts exists e.g. ProGuard. > > Kind regards, > > - Stephan Korsholm, VIAUC, DK From Alan.Bateman at oracle.com Tue Oct 23 02:01:37 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 23 Oct 2012 10:01:37 +0100 Subject: java.mini? was: Dependency leaks and embedded Java In-Reply-To: <1679143.Nj5tQ95O0u@logoutik> References: <1679143.Nj5tQ95O0u@logoutik> Message-ID: <50865CF1.2040604@oracle.com> On 23/10/2012 08:10, Jaroslav Tulach wrote: > : > > A modularization has to be driven by a need, I know. So far the only need > jigsaw team tried to achieve was to fit Hotspot on mobile devices and for that > java.base is enough (as demonstrated @ J1). > > However for certain environments it is clearly to big. Can't jigsaw also define > the smallest possible profile (e.g. with classes that are directly needed > during compilation by Javac like String, StringBuilder, Enum, Throwable)? > > Having such "java.mini" profile would help porting Java to unusual small > environments. It would ultimately lead to smaller fragmentation of Java - as > it is better if the smallest profile is standardized than created by > independent individuals. > Right, the convergence effort is focused at Java SE / Java ME CDC rather than the much smaller environments that have been the focus of the CLDC and JavaCard specifications. Bob Vandette's slides from JavaOne 2012 provides up to date information on the current efforts and the footprint goals for JDK 8: https://oracleus.activeevents.com/connect/fileDownload/session/CDC887FAEAD8ABE54064406AC304AD59/CON4538_Vandette.pdf -Alan. From djorm at redhat.com Wed Oct 24 22:03:40 2012 From: djorm at redhat.com (David Jorm) Date: Thu, 25 Oct 2012 01:03:40 -0400 (EDT) Subject: Proposal: mandatory versioning metadata for modules In-Reply-To: <1689245716.21803756.1351140684273.JavaMail.root@redhat.com> Message-ID: <804739505.21804202.1351141420894.JavaMail.root@redhat.com> Hi All I am currently working on a system for tracking JAR files that expose known security flaws, identifying them by checksums or metadata. In short, both of these mechanisms have shortcomings. Checksums of JAR files provide zero false positives, but have huge scope for false negatives. Whenever a JAR is re-compiled, bits in the JAR change and the checksum changes. Metadata is unreliable, as META-INF/MANIFEST.MF does not require the inclusion of title and version elements. A more detailed overview is available here [0]. To give an example of the problem I'm attempting to solve, Spring 2.5.6 exposes a remote code execution flaw. It is fixed in Spring 2.5.6.SEC01. I want to be able to distinguish between 2.5.6 and 2.5.6.SEC01 to identify whether a system is deploying the vulnerable JAR. I can't just use a checksum of the file, because if anyone recompiles it, bits change. I also can't rely on the MANIFEST.MF, because it may or may not include any version data; it may not even identify the title of the component as "Spring". The Jigsaw project offers a great opportunity to solve this problem. However, I note in the documentation [1]: "A /module/ is a collection of Java types (/i.e./, classes and interfaces) with a name, an optional version number, and a formal description of its relationships to other modules." The problem here is "optional" version number. What I'm trying to achieve is mandatory minimal version metadata. What do people think about making version number a requirement for Jigsaw modules? Would that be feasible and if so would it be desirable? Thanks -- David Jorm / Red Hat Security Response Team [0] http://www.slideshare.net/davidjorm/tracking-vulnerable-jars [1] http://cr.openjdk.java.net/~mr/jigsaw/notes/jigsaw-big-picture-01 From hendy at soluvas.com Wed Oct 24 22:18:42 2012 From: hendy at soluvas.com (Hendy Irawan) Date: Thu, 25 Oct 2012 12:18:42 +0700 Subject: Proposal: mandatory versioning metadata for modules In-Reply-To: <804739505.21804202.1351141420894.JavaMail.root@redhat.com> References: <1689245716.21803756.1351140684273.JavaMail.root@redhat.com> <804739505.21804202.1351141420894.JavaMail.root@redhat.com> Message-ID: I'm not sure what OSGi says about this, but if omitting the version number implicitly means version 0.0.0, then whether it's optional or mandatory *technically* doesn't matter to the module consumer. However, practically due to human concerns, I truly support this mandatory version requirement. Even if tooling will lax the version support (and generate 0.0.0 anyway) there should be clear WARNING message to the module developer when this is the case. In your case, your modules are already versioned (Spring 2.5.6 and Spring 2.5.6.SEC01 respectively) so there shouldn't be a problem in identifying which one. Hendy On Thu, Oct 25, 2012 at 12:03 PM, David Jorm wrote: > Hi All > > I am currently working on a system for tracking JAR files that expose > known security flaws, identifying them by checksums or metadata. In short, > both of these mechanisms have shortcomings. Checksums of JAR files provide > zero false positives, but have huge scope for false negatives. Whenever a > JAR is re-compiled, bits in the JAR change and the checksum changes. > Metadata is unreliable, as META-INF/MANIFEST.MF does not require the > inclusion of title and version elements. A more detailed overview is > available here [0]. To give an example of the problem I'm attempting to > solve, Spring 2.5.6 exposes a remote code execution flaw. It is fixed in > Spring 2.5.6.SEC01. I want to be able to distinguish between 2.5.6 and > 2.5.6.SEC01 to identify whether a system is deploying the vulnerable JAR. I > can't > just use a checksum of the file, because if anyone recompiles it, bits > change. I also can't rely on the MANIFEST.MF, because it may or may not > include any version data; it may not even identify the title of the > component as "Spring". > > The Jigsaw project offers a great opportunity to solve this problem. > However, I note in the documentation [1]: > > "A /module/ is a collection of Java types (/i.e./, classes and interfaces) > with a name, an optional version number, and a formal description of its > relationships to other modules." > > The problem here is "optional" version number. What I'm trying to achieve > is mandatory minimal version metadata. What do people think about making > version number a requirement for Jigsaw modules? Would that be feasible and > if so would it be desirable? > > Thanks > -- > David Jorm / Red Hat Security Response Team > > [0] http://www.slideshare.net/davidjorm/tracking-vulnerable-jars > [1] http://cr.openjdk.java.net/~mr/jigsaw/notes/jigsaw-big-picture-01 > -- Hendy Irawan - on Twitter - on LinkedIn Web Developer | Bippo Indonesia | Akselerator Bisnis | Bandung From eric at tibco.com Wed Oct 24 22:32:33 2012 From: eric at tibco.com (Eric Johnson) Date: Wed, 24 Oct 2012 22:32:33 -0700 Subject: Proposal: mandatory versioning metadata for modules In-Reply-To: <804739505.21804202.1351141420894.JavaMail.root@redhat.com> References: <804739505.21804202.1351141420894.JavaMail.root@redhat.com> Message-ID: <5088CEF1.5090003@tibco.com> Is this a security system, or a JAR management system? If it is merely a management system, I can see how your proposal to require version numbers might work (although it seems to be assuming that people will remember to update version numbers). If it is a security system, then I think you need to investigate the threat model a little more, because it might not be sufficient for a JAR to self-identify as a specific version, since a rogue actor will simply do that to defeat your code. In that case, to derail the rogue actor, don't you need to do a SHA-512 or other signature of the interesting bits of a JAR file (some sort of c14n process that eliminates the spurious bit twiddling that otherwise occurs upon recompile)? Eric. On 10/24/12 10:03 PM, David Jorm wrote: > Hi All > > I am currently working on a system for tracking JAR files that expose known security flaws, identifying them by checksums or metadata. In short, both of these mechanisms have shortcomings. Checksums of JAR files provide zero false positives, but have huge scope for false negatives. Whenever a JAR is re-compiled, bits in the JAR change and the checksum changes. Metadata is unreliable, as META-INF/MANIFEST.MF does not require the inclusion of title and version elements. A more detailed overview is available here [0]. To give an example of the problem I'm attempting to solve, Spring 2.5.6 exposes a remote code execution flaw. It is fixed in Spring 2.5.6.SEC01. I want to be able to distinguish between 2.5.6 and 2.5.6.SEC01 to identify whether a system is deploying the vulnerable JAR. I can't > just use a checksum of the file, because if anyone recompiles it, bits change. I also can't rely on the MANIFEST.MF, because it may or may not include any version data; it may not even identify the title of the component as "Spring". > > The Jigsaw project offers a great opportunity to solve this problem. However, I note in the documentation [1]: > > "A /module/ is a collection of Java types (/i.e./, classes and interfaces) with a name, an optional version number, and a formal description of its relationships to other modules." > > The problem here is "optional" version number. What I'm trying to achieve is mandatory minimal version metadata. What do people think about making version number a requirement for Jigsaw modules? Would that be feasible and if so would it be desirable? > > Thanks From djorm at redhat.com Wed Oct 24 22:41:26 2012 From: djorm at redhat.com (David Jorm) Date: Thu, 25 Oct 2012 15:41:26 +1000 Subject: Proposal: mandatory versioning metadata for modules In-Reply-To: <5088CEF1.5090003@tibco.com> References: <804739505.21804202.1351141420894.JavaMail.root@redhat.com> <5088CEF1.5090003@tibco.com> Message-ID: <5088D106.8010604@redhat.com> The system is aimed primarily at security, but a similar approach can be used for other aspects of component management, e.g. license auditing. You're absolutely right that even with mandatory version metadata, an attacker could create a modified version of the JAR that intentionally does not identify as being vulnerable. That's a problem with no ultimate solution, though. In the malware world you have all kinds of polymorphic, obfuscated code designed to defy any kind of identification. Anti virus companies are fighting this arms race with malware authors, but I have no intention of getting involved in that battle. Furthermore, if you're consuming components supplied by an attacker, they're likely to include backdoors or other things, and you've got bigger problems to contend with. I'm more interested in the case where a person consumes a component from a trusted source, for example the maven central repo, but is not aware that it exposes a known security flaw. Doing some partial contextual hashing of the JAR bits is another approach I am currently investigating. This will probably work for most simple re-compiles, but where people maintain their own branches with backported patches, it would run into problems, and simply identifying the version by metadata would be preferable. All that is to say: yes, this isn't a fool-proof mechanism, but it would be an improvement. Thanks David On 10/25/2012 03:32 PM, Eric Johnson wrote: > Is this a security system, or a JAR management system? > > If it is merely a management system, I can see how your proposal to > require version numbers might work (although it seems to be assuming > that people will remember to update version numbers). > > If it is a security system, then I think you need to investigate the > threat model a little more, because it might not be sufficient for a > JAR to self-identify as a specific version, since a rogue actor will > simply do that to defeat your code. In that case, to derail the rogue > actor, don't you need to do a SHA-512 or other signature of the > interesting bits of a JAR file (some sort of c14n process that > eliminates the spurious bit twiddling that otherwise occurs upon > recompile)? > > Eric. > > On 10/24/12 10:03 PM, David Jorm wrote: >> Hi All >> >> I am currently working on a system for tracking JAR files that expose >> known security flaws, identifying them by checksums or metadata. In >> short, both of these mechanisms have shortcomings. Checksums of JAR >> files provide zero false positives, but have huge scope for false >> negatives. Whenever a JAR is re-compiled, bits in the JAR change and >> the checksum changes. Metadata is unreliable, as META-INF/MANIFEST.MF >> does not require the inclusion of title and version elements. A more >> detailed overview is available here [0]. To give an example of the >> problem I'm attempting to solve, Spring 2.5.6 exposes a remote code >> execution flaw. It is fixed in Spring 2.5.6.SEC01. I want to be able >> to distinguish between 2.5.6 and 2.5.6.SEC01 to identify whether a >> system is deploying the vulnerable JAR. I can't >> just use a checksum of the file, because if anyone recompiles it, >> bits change. I also can't rely on the MANIFEST.MF, because it may or >> may not include any version data; it may not even identify the title >> of the component as "Spring". >> >> The Jigsaw project offers a great opportunity to solve this problem. >> However, I note in the documentation [1]: >> >> "A /module/ is a collection of Java types (/i.e./, classes and >> interfaces) with a name, an optional version number, and a formal >> description of its relationships to other modules." >> >> The problem here is "optional" version number. What I'm trying to >> achieve is mandatory minimal version metadata. What do people think >> about making version number a requirement for Jigsaw modules? Would >> that be feasible and if so would it be desirable? >> >> Thanks > From chris.hegarty at oracle.com Thu Oct 25 02:42:00 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 25 Oct 2012 10:42:00 +0100 Subject: RFR: Module constraints on targets In-Reply-To: <20121019183126.40861F68@eggemoggin.niobe.net> References: <20121019183126.40861F68@eggemoggin.niobe.net> Message-ID: <50890968.20908@oracle.com> Mark, Updated webrev, and comments inline. http://cr.openjdk.java.net/~chegar/jigsaw/constraints.01/webrev/ On 10/19/2012 07:31 PM, mark.reinhold at oracle.com wrote: > 2012/10/11 9:37 -0700, chris.hegarty at oracle.com: >> ... >> >> Webrev: >> http://cr.openjdk.java.net/~chegar/jigsaw/constraints.00/webrev/ > > Overall this looks pretty good; detailed comments below. > > After this change goes in we need to think further about repository > layout and module-file naming conventions. The filtering mechanism > you've put into the repository code is fine, but we need a story for > how to create a repository that contains multiple instances of the > same module built for different os/arch combinations. Right, and if os/arch are to be part of this convention, then maybe this is something that jpkg could do. It will now know if a module-file is platform dependent during creation. Anyway, your right further thought is required. > ModuleArchitecture.java > > I think one reason this class is hard to name is that it's serving two > purposes. It can be a concrete os/arch pair or it can be a pattern > against which a concrete pair can be matched. I suggest splitting this > into two classes, say Host and HostPattern. The first is just a simple > wrapper around os and arch strings, and has a static method to return a > Host object describing the os/arch on which we're running. The second > has the same instance methods as the first -- but it's not a subtype -- > and also includes a boolean matches(Host h) method. > > Then a Library can have a Host host() method, and a ModuleFileHeader > can have a HostPattern getHostPattern() method. > > The string representations of these classes should be unambiguous, and > always in lower case. E.g., "linux-x86", "linux-any", or "any". As I > suggested previously, patterns like "any-x86" should be disallowed, at > least until someone comes up with a compelling use case. I have taken this suggestion and added Host and HostPattern, and enforced this sensible restriction within jmod and jpkg. > (At some point we need to rationalize all our cpu-architecture names, > but that's a different topic.) > > SimpleLibrary > > A Library will always have a Host, so wherever you're passing around > separate os/arch string pairs I'd pass a Host object instead. Got it. > [204-211] There's no need to distinguish null from the empty string, > since neither can occur, so you can just use writeUTF(String) here, and > readUTF() when reading [234-238]. Yes. Still stored as separate os/arch strings so as not to have to parse a host string representation. > [1260] Throw a ConfigurationException for now, not an IOException. We > should consider later whether to have a separate exception type for > this kind of failure. Likewise on [1567]. Thank you. Changed to throw ConfigurationException, for now. > Librarian > > [74-75] If the os isn't specified on the command line then the host os > should be used, and likewise for arch. The right place to arrange for > that is probably in the Host.create method. Good idea. Changed. > [137-141] You can just out.format("host %s%n", lib.host()) here. Done. > [357] When creating a new library, if it has a parent then we should > ensure that the parent's os/arch is identical. Ah yes, good catch. latest webrev implements this. > ModuleFile > > [662-702] Having a builder for ModuleFileHeader seems like overkill, > but maybe that's just me. Yes, I flip flopped on this myself. For now, the builder has been removed. > ModuleFileWriter > > [323-339] How much slower is this going to be than generating the hash > directly, as we do now? Why is using a non-validating module-file > parser here a better approach? (If you retained the current code then > you wouldn't need the dynamic dispatching in ModuleFile.readFileHash > [573-578].) I did not do any perf measurements on this. The main issue is that the file header is now variable length. I have reverted the changes in this area ( dynamic dispatching in ModuleFile.readFileHash too ), and you can see that the writer can workaround this issue once it can locate the hash bytes for exclusion. Should be fine, and the most optimal. > RepositoryCatalog > > [129] Shouldn't this be using .matches rather than .equals ? D'oh! Yes, you are of course correct. Fixed. > RemoteRepository > > [340] s/findlocalModuleIds/findModuleIds/ Changed to findModuleIds. > [351-357] Why a separate method here? I don't see it used anywhere > else. I'd just fold line 356 into the find(Local)ModuleIds method. Yes, Done > Packager > > If native commands or libraries are specified on the command line but > os or arch aren't given then we should default them to the values for > the local system. Good idea. Implemented in the latest webrev. > > hello-repo.sh > > This test is meant to be very basic, like the other hello tests. > You've added essentially two new tests at the end; I'd break these out > into separate files. Moved into new repofilter.sh -Chris. > > - Mark > From pascal at rapicault.net Thu Oct 25 03:53:29 2012 From: pascal at rapicault.net (Pascal Rapicault) Date: Thu, 25 Oct 2012 06:53:29 -0400 Subject: Proposal: mandatory versioning metadata for modules In-Reply-To: <804739505.21804202.1351141420894.JavaMail.root@redhat.com> References: <804739505.21804202.1351141420894.JavaMail.root@redhat.com> Message-ID: <8cf2b5fb982c9135983de395ed2fd727.squirrel@webmail.rapicault.net> Like you I'm a bit surprised by the optional nature of the version number. However even if the version was made mandatory I don't think this would suffice to identify jars (btw, another case of jar modification is shading (the jar is copied into another one, and sometimes the package renamed)). For example people could forget to update the jar version or do it deliberately (because they don't want to update dependent libs), or simply strip it when aggregating jars. There is a lot of research papers talking about finding similarities in jars, but the one that I found to be the most robust to do this kind of work is http://dl.acm.org/citation.cfm?doid=1985441.1985468 HTH Pascal On Thu, October 25, 2012 01:03, David Jorm wrote: > Hi All > > I am currently working on a system for tracking JAR files that expose > known security flaws, identifying them by checksums or metadata. In short, > both of these mechanisms have shortcomings. Checksums of JAR files provide > zero false positives, but have huge scope for false negatives. Whenever a > JAR is re-compiled, bits in the JAR change and the checksum changes. > Metadata is unreliable, as META-INF/MANIFEST.MF does not require the > inclusion of title and version elements. A more detailed overview is > available here [0]. To give an example of the problem I'm attempting to > solve, Spring 2.5.6 exposes a remote code execution flaw. It is fixed in > Spring 2.5.6.SEC01. I want to be able to distinguish between 2.5.6 and > 2.5.6.SEC01 to identify whether a system is deploying the vulnerable JAR. > I can't > just use a checksum of the file, because if anyone recompiles it, bits > change. I also can't rely on the MANIFEST.MF, because it may or may not > include any version data; it may not even identify the title of the > component as "Spring". > > The Jigsaw project offers a great opportunity to solve this problem. > However, I note in the documentation [1]: > > "A /module/ is a collection of Java types (/i.e./, classes and interfaces) > with a name, an optional version number, and a formal description of its > relationships to other modules." > > The problem here is "optional" version number. What I'm trying to achieve > is mandatory minimal version metadata. What do people think about making > version number a requirement for Jigsaw modules? Would that be feasible > and if so would it be desirable? > > Thanks > -- > David Jorm / Red Hat Security Response Team > > [0] http://www.slideshare.net/davidjorm/tracking-vulnerable-jars > [1] http://cr.openjdk.java.net/~mr/jigsaw/notes/jigsaw-big-picture-01 > From Alan.Bateman at oracle.com Thu Oct 25 04:29:10 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 25 Oct 2012 12:29:10 +0100 Subject: Proposal: mandatory versioning metadata for modules In-Reply-To: <804739505.21804202.1351141420894.JavaMail.root@redhat.com> References: <804739505.21804202.1351141420894.JavaMail.root@redhat.com> Message-ID: <50892286.40208@oracle.com> On 25/10/2012 06:03, David Jorm wrote: > Hi All > > I am currently working on a system for tracking JAR files that expose known security flaws, identifying them by checksums or metadata. In short, both of these mechanisms have shortcomings. Checksums of JAR files provide zero false positives, but have huge scope for false negatives. Whenever a JAR is re-compiled, bits in the JAR change and the checksum changes. Metadata is unreliable, as META-INF/MANIFEST.MF does not require the inclusion of title and version elements. A more detailed overview is available here [0]. To give an example of the problem I'm attempting to solve, Spring 2.5.6 exposes a remote code execution flaw. It is fixed in Spring 2.5.6.SEC01. I want to be able to distinguish between 2.5.6 and 2.5.6.SEC01 to identify whether a system is deploying the vulnerable JAR. I can't > just use a checksum of the file, because if anyone recompiles it, bits change. I also can't rely on the MANIFEST.MF, because it may or may not include any version data; it may not even identify the title of the component as "Spring". > > The Jigsaw project offers a great opportunity to solve this problem. However, I note in the documentation [1]: > > "A /module/ is a collection of Java types (/i.e./, classes and interfaces) with a name, an optional version number, and a formal description of its relationships to other modules." > > The problem here is "optional" version number. What I'm trying to achieve is mandatory minimal version metadata. What do people think about making version number a requirement for Jigsaw modules? Would that be feasible and if so would it be desirable? > > Thanks I could imagine module repositories having policies to require all modules hosted in the repository to have a version number. On blacklisting JAR files then just an FYI that Sun/Oracle's JDK has had support for blacklisting of signed JAR files [1] for some time (this is pluging/webstart so I can't point to you something in OpenJDK). Jigsaw already has signed modules, I could imagine in time having blacklisting of signed modules too. -Alan. [1] http://docs.oracle.com/javase/7/docs/technotes/guides/security/blacklistfeature.html From hendy at soluvas.com Thu Oct 25 04:50:55 2012 From: hendy at soluvas.com (Hendy Irawan) Date: Thu, 25 Oct 2012 18:50:55 +0700 Subject: Proposal: mandatory versioning metadata for modules In-Reply-To: <50892286.40208@oracle.com> References: <804739505.21804202.1351141420894.JavaMail.root@redhat.com> <50892286.40208@oracle.com> Message-ID: Hi all, Regardless of security concerns (of which versioning is only little help), I support making versions mandatory. Is there a formal process for accepting this proposal ? Hendy On Thu, Oct 25, 2012 at 6:29 PM, Alan Bateman wrote: > On 25/10/2012 06:03, David Jorm wrote: > >> Hi All >> >> I am currently working on a system for tracking JAR files that expose >> known security flaws, identifying them by checksums or metadata. In short, >> both of these mechanisms have shortcomings. Checksums of JAR files provide >> zero false positives, but have huge scope for false negatives. Whenever a >> JAR is re-compiled, bits in the JAR change and the checksum changes. >> Metadata is unreliable, as META-INF/MANIFEST.MF does not require the >> inclusion of title and version elements. A more detailed overview is >> available here [0]. To give an example of the problem I'm attempting to >> solve, Spring 2.5.6 exposes a remote code execution flaw. It is fixed in >> Spring 2.5.6.SEC01. I want to be able to distinguish between 2.5.6 and >> 2.5.6.SEC01 to identify whether a system is deploying the vulnerable JAR. I >> can't >> just use a checksum of the file, because if anyone recompiles it, bits >> change. I also can't rely on the MANIFEST.MF, because it may or may not >> include any version data; it may not even identify the title of the >> component as "Spring". >> >> The Jigsaw project offers a great opportunity to solve this problem. >> However, I note in the documentation [1]: >> >> "A /module/ is a collection of Java types (/i.e./, classes and >> interfaces) with a name, an optional version number, and a formal >> description of its relationships to other modules." >> >> The problem here is "optional" version number. What I'm trying to achieve >> is mandatory minimal version metadata. What do people think about making >> version number a requirement for Jigsaw modules? Would that be feasible and >> if so would it be desirable? >> >> Thanks >> > I could imagine module repositories having policies to require all modules > hosted in the repository to have a version number. > > On blacklisting JAR files then just an FYI that Sun/Oracle's JDK has had > support for blacklisting of signed JAR files [1] for some time (this is > pluging/webstart so I can't point to you something in OpenJDK). Jigsaw > already has signed modules, I could imagine in time having blacklisting of > signed modules too. > > -Alan. > > [1] http://docs.oracle.com/javase/**7/docs/technotes/guides/** > security/blacklistfeature.html > -- Hendy Irawan - on Twitter - on LinkedIn Web Developer | Bippo Indonesia | Akselerator Bisnis | Bandung From sean.mullan at oracle.com Thu Oct 25 12:39:38 2012 From: sean.mullan at oracle.com (Sean Mullan) Date: Thu, 25 Oct 2012 15:39:38 -0400 Subject: Proposal: mandatory versioning metadata for modules In-Reply-To: <50892286.40208@oracle.com> References: <804739505.21804202.1351141420894.JavaMail.root@redhat.com> <50892286.40208@oracle.com> Message-ID: <5089957A.7020402@oracle.com> On 10/25/12 7:29 AM, Alan Bateman wrote: > On 25/10/2012 06:03, David Jorm wrote: >> Hi All >> >> I am currently working on a system for tracking JAR files that expose known security flaws, identifying them by checksums or metadata. In short, both of these mechanisms have shortcomings. Checksums of JAR files provide zero false positives, but have huge scope for false negatives. Whenever a JAR is re-compiled, bits in the JAR change and the checksum changes. Metadata is unreliable, as META-INF/MANIFEST.MF does not require the inclusion of title and version elements. A more detailed overview is available here [0]. To give an example of the problem I'm attempting to solve, Spring 2.5.6 exposes a remote code execution flaw. It is fixed in Spring 2.5.6.SEC01. I want to be able to distinguish between 2.5.6 and 2.5.6.SEC01 to identify whether a system is deploying the vulnerable JAR. I can't >> just use a checksum of the file, because if anyone recompiles it, bits change. I also can't rely on the MANIFEST.MF, because it may or may not include any version data; it may not even identify the title of the component as "Spring". >> >> The Jigsaw project offers a great opportunity to solve this problem. However, I note in the documentation [1]: >> >> "A /module/ is a collection of Java types (/i.e./, classes and interfaces) with a name, an optional version number, and a formal description of its relationships to other modules." >> >> The problem here is "optional" version number. What I'm trying to achieve is mandatory minimal version metadata. What do people think about making version number a requirement for Jigsaw modules? Would that be feasible and if so would it be desirable? >> >> Thanks > I could imagine module repositories having policies to require all > modules hosted in the repository to have a version number. > > On blacklisting JAR files then just an FYI that Sun/Oracle's JDK has had > support for blacklisting of signed JAR files [1] for some time (this is > pluging/webstart so I can't point to you something in OpenJDK). Jigsaw > already has signed modules, I could imagine in time having blacklisting > of signed modules too. Yes, and a Jigsaw blacklisting feature would likely be more flexible than the current signed JAR blacklisting mechanism Alan mentioned, which is based on the hash of the Manifest, so each JAR needs to be individually blacklisted. I could imagine providing the capability to blacklist several modules at once based on version ranges, etc. --Sean > > -Alan. > > [1] > http://docs.oracle.com/javase/7/docs/technotes/guides/security/blacklistfeature.html > From djorm at redhat.com Thu Oct 25 20:02:24 2012 From: djorm at redhat.com (David Jorm) Date: Fri, 26 Oct 2012 13:02:24 +1000 Subject: Proposal: mandatory versioning metadata for modules In-Reply-To: <8cf2b5fb982c9135983de395ed2fd727.squirrel@webmail.rapicault.net> References: <804739505.21804202.1351141420894.JavaMail.root@redhat.com> <8cf2b5fb982c9135983de395ed2fd727.squirrel@webmail.rapicault.net> Message-ID: <5089FD40.5020900@redhat.com> Hi Pascal After various discussions on both this list and elsewhere, I've investigated several options to improve the bad-jar matching, including the technique outlined in that paper. Unfortunately I don't think the anchored signature matching technique will work, since the anchored signature of a vulnerable jar may not differ from that of a patched jar. Many security patches don't modify any method signatures, just the logic inside a method. Cases where other patches have been applied that do change method signatures, but don't address the security flaw, would also result in false negatives with this technique. So far the best candidate solution I have is to: 1) Explode the jar 2) Remove META-INF elements 3) Re-package jar 4) Checksum re-packaged jar 5) [Phase 2: somehow handle one jar being a superset of another] I intend to try this in the next version of the tools I'm developing. That said, mandatory version metadata is still helpful for both this and other use cases. Thanks David On 10/25/2012 08:53 PM, Pascal Rapicault wrote: > Like you I'm a bit surprised by the optional nature of the version number. > However even if the version was made mandatory I don't think this would > suffice to identify jars (btw, another case of jar modification is shading > (the jar is copied into another one, and sometimes the package renamed)). > For example people could forget to update the jar version or do it > deliberately (because they don't want to update dependent libs), or simply > strip it when aggregating jars. > > There is a lot of research papers talking about finding similarities in > jars, but the one that I found to be the most robust to do this kind of > work is http://dl.acm.org/citation.cfm?doid=1985441.1985468 > > HTH > > Pascal > > > On Thu, October 25, 2012 01:03, David Jorm wrote: >> Hi All >> >> I am currently working on a system for tracking JAR files that expose >> known security flaws, identifying them by checksums or metadata. In short, >> both of these mechanisms have shortcomings. Checksums of JAR files provide >> zero false positives, but have huge scope for false negatives. Whenever a >> JAR is re-compiled, bits in the JAR change and the checksum changes. >> Metadata is unreliable, as META-INF/MANIFEST.MF does not require the >> inclusion of title and version elements. A more detailed overview is >> available here [0]. To give an example of the problem I'm attempting to >> solve, Spring 2.5.6 exposes a remote code execution flaw. It is fixed in >> Spring 2.5.6.SEC01. I want to be able to distinguish between 2.5.6 and >> 2.5.6.SEC01 to identify whether a system is deploying the vulnerable JAR. >> I can't >> just use a checksum of the file, because if anyone recompiles it, bits >> change. I also can't rely on the MANIFEST.MF, because it may or may not >> include any version data; it may not even identify the title of the >> component as "Spring". >> >> The Jigsaw project offers a great opportunity to solve this problem. >> However, I note in the documentation [1]: >> >> "A /module/ is a collection of Java types (/i.e./, classes and interfaces) >> with a name, an optional version number, and a formal description of its >> relationships to other modules." >> >> The problem here is "optional" version number. What I'm trying to achieve >> is mandatory minimal version metadata. What do people think about making >> version number a requirement for Jigsaw modules? Would that be feasible >> and if so would it be desirable? >> >> Thanks >> -- >> David Jorm / Red Hat Security Response Team >> >> [0] http://www.slideshare.net/davidjorm/tracking-vulnerable-jars >> [1] http://cr.openjdk.java.net/~mr/jigsaw/notes/jigsaw-big-picture-01 >> > From dalibor.topic at oracle.com Tue Oct 30 11:35:02 2012 From: dalibor.topic at oracle.com (Dalibor Topic) Date: Tue, 30 Oct 2012 19:35:02 +0100 Subject: Proposal: mandatory versioning metadata for modules In-Reply-To: <5088D106.8010604@redhat.com> References: <804739505.21804202.1351141420894.JavaMail.root@redhat.com> <5088CEF1.5090003@tibco.com> <5088D106.8010604@redhat.com> Message-ID: <50901DD6.30209@oracle.com> On 10/25/12 7:41 AM, David Jorm wrote: > I'm more interested in the case where a person consumes a component from a trusted source, for example the maven central repo For a different view, see http://www.slideshare.net/SanderMak/crossbuild-injection-attacks-how-safe-is-your-java-build -- Oracle Dalibor Topic | Principal Product Manager Phone: +494089091214 | Mobile: +491737185961 Oracle Java Platform Group ORACLE Deutschland B.V. & Co. KG | K?hneh?fe 5 | 22761 Hamburg ORACLE Deutschland B.V. & Co. KG Hauptverwaltung: Riesstr. 25, D-80992 M?nchen Registergericht: Amtsgericht M?nchen, HRA 95603 Gesch?ftsf?hrer: J?rgen Kunz Komplement?rin: ORACLE Deutschland Verwaltung B.V. Hertogswetering 163/167, 3543 AS Utrecht, Niederlande Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697 Gesch?ftsf?hrer: Alexander van der Ven, Astrid Kepper, Val Maher Green Oracle Oracle is committed to developing practices and products that help protect the environment From dalibor.topic at oracle.com Wed Oct 31 05:05:19 2012 From: dalibor.topic at oracle.com (Dalibor Topic) Date: Wed, 31 Oct 2012 13:05:19 +0100 Subject: java.mini? was: Dependency leaks and embedded Java In-Reply-To: <1679143.Nj5tQ95O0u@logoutik> References: <1679143.Nj5tQ95O0u@logoutik> Message-ID: <509113FF.5080708@oracle.com> On 10/23/12 9:10 AM, Jaroslav Tulach wrote: > This is my observation as well. A Throwable has now a direct dependency on > ArrayList (as of JDK7)! That really does not make the transitive closure of > Object smaller. Not necessarily. While the JDK 7 implementation uses ArrayList internally, it does not occur in the public API of Throwable. So you could write Throwable without it in one way or another if you had to. > However for certain environments it is clearly to big. Can't jigsaw also define > the smallest possible profile (e.g. with classes that are directly needed > during compilation by Javac like String, StringBuilder, Enum, Throwable)? That would be implementation-specific by definition, as per ArrayList example above. cheers, dalibor topic -- Oracle Dalibor Topic | Principal Product Manager Phone: +494089091214 | Mobile: +491737185961 Oracle Java Platform Group ORACLE Deutschland B.V. & Co. KG | K?hneh?fe 5 | 22761 Hamburg ORACLE Deutschland B.V. & Co. KG Hauptverwaltung: Riesstr. 25, D-80992 M?nchen Registergericht: Amtsgericht M?nchen, HRA 95603 Gesch?ftsf?hrer: J?rgen Kunz Komplement?rin: ORACLE Deutschland Verwaltung B.V. Hertogswetering 163/167, 3543 AS Utrecht, Niederlande Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697 Gesch?ftsf?hrer: Alexander van der Ven, Astrid Kepper, Val Maher Green Oracle Oracle is committed to developing practices and products that help protect the environment From jaroslav.tulach at oracle.com Wed Oct 31 06:22:37 2012 From: jaroslav.tulach at oracle.com (Jaroslav Tulach) Date: Wed, 31 Oct 2012 14:22:37 +0100 Subject: java.mini In-Reply-To: <509113FF.5080708@oracle.com> References: <1679143.Nj5tQ95O0u@logoutik> <509113FF.5080708@oracle.com> Message-ID: <2468327.n2g2Dr2Zd1@logoutik> Hello Dalibor, thanks for your reply. Dne St 31. ??jna 2012 13:05:19, Dalibor Topic napsal(a): > On 10/23/12 9:10 AM, Jaroslav Tulach wrote: > > This is my observation as well. A Throwable has now a direct dependency on > > ArrayList (as of JDK7)! That really does not make the transitive closure > > of > > Object smaller. > > Not necessarily. While the JDK 7 implementation uses ArrayList internally, > it does not occur in the public API of Throwable. So you could write > Throwable without it in one way or another if you had to. Right. The class is not exposed in public signature, so I can re-implement the internal implementation (which is what I did). On the other hand, that means I have to fork the code base and that is not good in a long term. > > However for certain environments it is clearly to big. Can't jigsaw also > > define the smallest possible profile (e.g. with classes that are directly > > needed during compilation by Javac like String, StringBuilder, Enum, > > Throwable)? > That would be implementation-specific by definition, as per ArrayList > example above. I don't think so. JavaC really cares only about some aspects of the above mentioned classes. It needs a Throwable, but does not need its addSuppressed method. I know the Jigsaw team is considering to have "methods that require other module to be present to work" and there is even a special annotation to document such requirement. I'd be still interested in defining java.mini - e.g. set of classes directly need by JavaC for compilation and marking their methods that require java.base (for example because of usage of ArrayList) with the annotation. This would be cleaner solution than forking the codebase. -jt