From Jonathan.Gibbons at Sun.COM Tue Jan 5 10:00:17 2010 From: Jonathan.Gibbons at Sun.COM (Jonathan Gibbons) Date: Tue, 05 Jan 2010 10:00:17 -0800 Subject: jigsaw/hotspot fails to build on JPRT In function 'void JVM_ExtendBootClassPath(JNIEnv*, const char*) Message-ID: <4B437E31.7020803@sun.com> For full details, see http://jprt-web.sfbay.sun.com/archive/2010/01/2010-01-04-234151.jjg.jigsaw/JobStatus.txt The failure occurs on all platforms. -- Jon rm -f jvm_linux.o g++ -DLINUX -D_GNU_SOURCE -DIA32 -DASSERT -DFASTDEBUG -I. -I../generated/adfiles -I../generated/jvmtifiles -I/tmp/jprt/P1/B/234151.jjg/source/hotspot/src/share/vm/asm -I/tmp/jprt/P1/B/234151.jjg/source/hotspot/src/share/vm/c1 -I/tmp/jprt/P1/B/234151.jjg/source/hotspot/src/share/vm/ci -I/tmp/jprt/P1/B/234151.jjg/source/hotspot/src/share/vm/classfile -I/tmp/jprt/P1/B/234151.jjg/source/hotspot/src/share/vm/code -I/tmp/jprt/P1/B/234151.jjg/source/hotspot/src/share/vm/compiler -I/tmp/jprt/P1/B/234151.jjg/source/hotspot/src/share/vm/gc_implementation -I/tmp/jprt/P1/B/234151.jjg/source/hotspot/src/share/vm/gc_implementation/shared -I/tmp/jprt/P1/B/234151.jjg/source/hotspot/src/share/vm/gc_implementation/parNew -I/tmp/jprt/P1/B/234151.jjg/source/hotspot/src/share/vm/gc_implementation/concurrentMarkSweep -I/tmp/jprt/P1/B/234151.jjg/source/hotspot/src/share/vm/gc_implementation/parallelScavenge -I/tmp/jprt/P1/B/234151.jjg/source/hotspot/src/share/vm/gc_implementation/g1 -I/tmp/jprt/P1/B/234151.jjg/source/hotspot/src/share/vm/gc_interface -I/tmp/jprt/P1/B/234151.jjg/source/hotspot/src/share/vm/interpreter -I/tmp/jprt/P1/B/234151.jjg/source/hotspot/src/share/vm/libadt -I/tmp/jprt/P1/B/234151.jjg/source/hotspot/src/share/vm/memory -I/tmp/jprt/P1/B/234151.jjg/source/hotspot/src/share/vm/oops -I/tmp/jprt/P1/B/234151.jjg/source/hotspot/src/share/vm/opto -I/tmp/jprt/P1/B/234151.jjg/source/hotspot/src/share/vm/prims -I/tmp/jprt/P1/B/234151.jjg/source/hotspot/src/share/vm/runtime -I/tmp/jprt/P1/B/234151.jjg/source/hotspot/src/share/vm/services -I/tmp/jprt/P1/B/234151.jjg/source/hotspot/src/share/vm/utilities -I/tmp/jprt/P1/B/234151.jjg/source/hotspot/src/cpu/x86/vm -I/tmp/jprt/P1/B/234151.jjg/source/hotspot/src/os/linux/vm -I/tmp/jprt/P1/B/234151.jjg/source/hotspot/src/os_cpu/linux_x86/vm -I../generated -DHOTSPOT_RELEASE_VERSION="\"17.0-b05-2010-01-04-234151.jjg.jigsaw\"" -DHOTSPOT_BUILD_TARGET="\"fastdebug\"" -DHOTSPOT_BUILD_USER="\"jprtadm\"" -DHOTSPOT_LIB_ARCH=\"i386\" -DJRE_RELEASE_VERSION="\"1.7.0-internal-fastdebug-jprtadm_2010_01_04_16_42-b00\"" -DHOTSPOT_VM_DISTRO="\"OpenJDK\"" -DCOMPILER2 -DCOMPILER1 -fPIC -fno-rtti -fno-exceptions -D_REENTRANT -fcheck-new -m32 -march=i586 -pipe -O3 -fno-strict-aliasing -DVM_LITTLE_ENDIAN -Werror -Wpointer-arith -Wsign-compare -c -o jvm_linux.o /tmp/jprt/P1/B/234151.jjg/source/hotspot/src/os/linux/vm/jvm_linux.cpp /tmp/jprt/P1/B/234151.jjg/source/hotspot/src/share/vm/prims/jvm.cpp: In function 'void JVM_ExtendBootClassPath(JNIEnv*, const char*)': /tmp/jprt/P1/B/234151.jjg/source/hotspot/src/share/vm/prims/jvm.cpp:883: error: expected `;' before '{' token make[8]: *** [jvm.o] Error 1 make[8]: *** Waiting for unfinished jobs.... make[8]: Leaving directory `/tmp/jprt/P1/B/234151.jjg/source/build/linux-i586/bootjdk-fastdebug/hotspot/outputdir/linux_i486_compiler2/fastdebug' make[7]: *** [the_vm] Error 2 make[7]: Leaving directory `/tmp/jprt/P1/B/234151.jjg/source/build/linux-i586/bootjdk-fastdebug/hotspot/outputdir/linux_i486_compiler2/fastdebug' make[6]: *** [fastdebug] Error 2 make[6]: Leaving directory `/tmp/jprt/P1/B/234151.jjg/source/build/linux-i586/bootjdk-fastdebug/hotspot/outputdir' make[5]: *** [generic_build2] Error 2 make[5]: Leaving directory `/tmp/jprt/P1/B/234151.jjg/source/hotspot/make' make[4]: *** [fastdebug] Error 2 make[4]: Leaving directory `/tmp/jprt/P1/B/234151.jjg/source/hotspot/make' make[3]: *** [hotspot-build] Error 2 make[3]: Leaving directory `/tmp/jprt/P1/B/234151.jjg/source' make[2]: *** [generic_debug_build] Error 2 make[2]: Leaving directory `/tmp/jprt/P1/B/234151.jjg/source' make[1]: *** [build_fastdebug_image] Error 2 make[1]: Leaving directory `/tmp/jprt/P1/B/234151.jjg/source' make: *** [create_fresh_fastdebug_bootdir] Error 2 ########################################################### From Jonathan.Gibbons at Sun.COM Mon Jan 11 18:36:53 2010 From: Jonathan.Gibbons at Sun.COM (Jonathan Gibbons) Date: Mon, 11 Jan 2010 18:36:53 -0800 Subject: serialized annotations use sun.reflect.* Message-ID: <4B4BE045.3050008@sun.com> Mandy, Alan, In the modularization effort, is it going to be an issue that javac (and apt) has an explicit dependence on sun.reflect.annotation.*? Since JDK 5, the serialized form of annotations has used these otherwise internal classes. This is true both for annotations created by the runtime, and by javac as part of JSR 269. For full details, if you really want to, see 6538914 and 6270734. Joe and I are trying to figure out the disposition of these bugs -- whether to use new public classes (and break compatibility) or to accept the status quo that these classes should be part of the public J2SE spec. -- Jon From mr at sun.com Tue Jan 12 10:06:24 2010 From: mr at sun.com (Mark Reinhold) Date: Tue, 12 Jan 2010 10:06:24 -0800 Subject: TL make/modules/Makefile requires ALT_JDK_IMPORT_PATH Message-ID: <20100112180624.C922059E@eggemoggin.niobe.net> I tried building the modules target in the current TL forest with this command: $ make JDK_BUILD_TARGETS=modules jdk_only but ran into the following error: >>>Making gen-classlist @ Thu Jan 7 09:57:24 PST 2010 ... /NOT-SET/re/jdk/1.7.0/promoted/latest/binaries/linux-i586/bin/java -XX:-PrintVMOptions -XX:+UnlockDiagnosticVMOptions -XX:-LogVMOutput -client -Xmx896m -Xms128m -XX:PermSize=32m -XX:MaxPermSize=160m -Xbootclasspath:/w/tl/build/classes \ -Dclassanalyzer.debug \ -jar /w/tl/build/btjars/classanalyzer.jar \ -jdkhome /w/tl/build \ -config modules.config \ -config modules.group \ -depconfig jdk7.depconfig \ -depconfig optional.depconfig \ -showdynamic \ -output /w/tl/build/tmp/modules/classlist make[4]: /NOT-SET/re/jdk/1.7.0/promoted/latest/binaries/linux-i586/bin/java: Command not found make[4]: *** [gen-classlist] Error 127 It looks like your use of JAVA_TOOLS_DIR in the gen-classlist target winds up requiring that ALT_JDK_IMPORT_PATH be set, but it shouldn't be necessary to set that variable when doing a full build [1]. This isn't a critical issue -- the workaround is simple enough -- but it would be good to fix it at some point. - Mark [1] http://mail.openjdk.java.net/pipermail/build-dev/2009-October/002457.html From Mandy.Chung at Sun.COM Tue Jan 12 10:42:24 2010 From: Mandy.Chung at Sun.COM (Mandy Chung) Date: Tue, 12 Jan 2010 10:42:24 -0800 Subject: TL make/modules/Makefile requires ALT_JDK_IMPORT_PATH In-Reply-To: <20100112180624.C922059E@eggemoggin.niobe.net> References: <20100112180624.C922059E@eggemoggin.niobe.net> Message-ID: <4B4CC290.4010709@sun.com> Thanks for the pointer to [1]. I will fix it (cr# 6916217). Mandy Mark Reinhold wrote: > I tried building the modules target in the current TL forest with this > command: > > $ make JDK_BUILD_TARGETS=modules jdk_only > > but ran into the following error: > > >>>Making gen-classlist @ Thu Jan 7 09:57:24 PST 2010 ... > /NOT-SET/re/jdk/1.7.0/promoted/latest/binaries/linux-i586/bin/java -XX:-PrintVMOptions -XX:+UnlockDiagnosticVMOptions -XX:-LogVMOutput -client -Xmx896m -Xms128m -XX:PermSize=32m -XX:MaxPermSize=160m -Xbootclasspath:/w/tl/build/classes \ > -Dclassanalyzer.debug \ > -jar /w/tl/build/btjars/classanalyzer.jar \ > -jdkhome /w/tl/build \ > -config modules.config \ > -config modules.group \ > -depconfig jdk7.depconfig \ > -depconfig optional.depconfig \ > -showdynamic \ > -output /w/tl/build/tmp/modules/classlist > make[4]: /NOT-SET/re/jdk/1.7.0/promoted/latest/binaries/linux-i586/bin/java: Command not found > make[4]: *** [gen-classlist] Error 127 > > It looks like your use of JAVA_TOOLS_DIR in the gen-classlist target > winds up requiring that ALT_JDK_IMPORT_PATH be set, but it shouldn't > be necessary to set that variable when doing a full build [1]. > > This isn't a critical issue -- the workaround is simple enough -- but it > would be good to fix it at some point. > > - Mark > > > [1] http://mail.openjdk.java.net/pipermail/build-dev/2009-October/002457.html > From mr at sun.com Tue Jan 12 11:56:55 2010 From: mr at sun.com (Mark Reinhold) Date: Tue, 12 Jan 2010 11:56:55 -0800 Subject: MODULE variable in makefiles limits module-definition refactoring Message-ID: <20100112195655.22D0C59E@eggemoggin.niobe.net> (This is about Mandy's most recent changes in the TL forest, which are not (yet) in the Jigsaw forest.) The use of the MODULE variable in many makefiles seems to tie those files to the contents of modules.config and modules.group; i.e., if one of the latter files changes then some MODULE definitions may also need to be adjusted. This will make it more difficult to experiment with, and refactor, the module definitions. To support changes in the modules.group file, at least, would it make sense to enhance the class analyzer to generate another type of output file, say foo.members, which for a grouped module lists its lowest-level member modules? Then the modularize target in the modules makefile could copy the non-class contents of all the low-level modules into the corresponding grouped module. - Mark From mr at sun.com Tue Jan 12 11:59:06 2010 From: mr at sun.com (Mark Reinhold) Date: Tue, 12 Jan 2010 11:59:06 -0800 Subject: Meaning of [dynamic] in modules.summary Message-ID: <20100112195906.AE2C959E@eggemoggin.niobe.net> In the modules.summary file generated during the build, are all dependences marked as "[dynamic]" effectively optional? That is, if the target of the dependence is not there then will anything break? - Mark From Alan.Bateman at Sun.COM Tue Jan 12 12:33:55 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Tue, 12 Jan 2010 20:33:55 +0000 Subject: Meaning of [dynamic] in modules.summary In-Reply-To: <20100112195906.AE2C959E@eggemoggin.niobe.net> References: <20100112195906.AE2C959E@eggemoggin.niobe.net> Message-ID: <4B4CDCB3.5080704@sun.com> Mark Reinhold wrote: > In the modules.summary file generated during the build, are all > dependences marked as "[dynamic]" effectively optional? That is, > if the target of the dependence is not there then will anything > break? > > - Mark > I didn't realize this was going through to the modules.summary as "dynamic" (Mandy, I think that has implications for CheckDeps as it understands "optional but not "dynamic"). The dependencies marked as "dynamic" are Class.forName usages and are optional. For example, the base module has the mandatory charsets; all others are (currently) in "charsets". Many of the others cases in modules.summary are also because of service provider interfaces - for example the base dependency on "net-compat" is because of legacy URL protocol handlers, the base dependency on the security-* modules is because the security API is mostly a framework with the implementations in various providers. A few of them are not SPI related. JSSE can optionally negotiate to use Kerberos ciphers if present, and there are a few more like this. In all these cases, if the module is not present, then it is handled "gracefully". This could break something of course - for example, if an application requires a specific provider to be present then it would need to specify the dependency to ensure that the provider is installed. -Alan. From Mandy.Chung at Sun.COM Tue Jan 12 13:05:31 2010 From: Mandy.Chung at Sun.COM (Mandy Chung) Date: Tue, 12 Jan 2010 13:05:31 -0800 Subject: MODULE variable in makefiles limits module-definition refactoring In-Reply-To: <20100112195655.22D0C59E@eggemoggin.niobe.net> References: <20100112195655.22D0C59E@eggemoggin.niobe.net> Message-ID: <4B4CE41B.8020909@sun.com> Mark Reinhold wrote: > (This is about Mandy's most recent changes in the TL forest, which are > not (yet) in the Jigsaw forest.) > > The use of the MODULE variable in many makefiles seems to tie those files > to the contents of modules.config and modules.group; i.e., if one of the > latter files changes then some MODULE definitions may also need to be > adjusted. This will make it more difficult to experiment with, and > refactor, the module definitions. > > To support changes in the modules.group file, at least, would it make > sense to enhance the class analyzer to generate another type of output > file, say foo.members, which for a grouped module lists its lowest-level > member modules? Then the modularize target in the modules makefile could > copy the non-class contents of all the low-level modules into the > corresponding grouped module. > > It's currently done that way if I understand correctly what you are suggesting. The member list for all modules is generated at build/linux-i586/tmp/modules/modules.list (one output file for all module groups instead of one group). I added several lowest-level modules such as awt, java2d, etc that are grouped under the client module specified in the modules.config so that we can experiment with the modules.group without requiring modification to the makefiles. The non-class contents are copied from /tmp/modules in its low-level module during the normal build process (i.e. all target). The modularize target merges the low-level modules into the corresponding grouped modules in the /modules directory. Does that match what you're suggesting? Mandy From Mandy.Chung at Sun.COM Tue Jan 12 13:07:53 2010 From: Mandy.Chung at Sun.COM (Mandy Chung) Date: Tue, 12 Jan 2010 13:07:53 -0800 Subject: MODULE variable in makefiles limits module-definition refactoring In-Reply-To: <4B4CE41B.8020909@sun.com> References: <20100112195655.22D0C59E@eggemoggin.niobe.net> <4B4CE41B.8020909@sun.com> Message-ID: <4B4CE4A9.8080502@sun.com> Mandy Chung wrote: > > It's currently done that way if I understand correctly what you are > suggesting. The member list for all modules is generated at > build/linux-i586/tmp/modules/modules.list (one output file for all > module groups instead of one group). > Correct location of this generated file should be: build/linux-i586/tmp/modules/classlist/modules.list Mandy From gnu_andrew at member.fsf.org Tue Jan 12 13:24:42 2010 From: gnu_andrew at member.fsf.org (Andrew John Hughes) Date: Tue, 12 Jan 2010 21:24:42 +0000 Subject: serialized annotations use sun.reflect.* In-Reply-To: <4B4BE045.3050008@sun.com> References: <4B4BE045.3050008@sun.com> Message-ID: <17c6771e1001121324j279f19f4w3965669c475b46b8@mail.gmail.com> 2010/1/12 Jonathan Gibbons : > Mandy, Alan, > > In the modularization effort, is it going to be an issue that javac (and > apt) has an explicit dependence on sun.reflect.annotation.*? Since JDK 5, > the serialized form of annotations has used these otherwise internal > classes. This is true both for annotations created by the runtime, and by > javac as part of JSR 269. For full details, if you really want to, see > 6538914 and 6270734. > Joe and I are trying to figure out the disposition of these bugs -- whether > to use new public classes (and break compatibility) or to accept the status > quo that these classes should be part of the public J2SE spec. > > -- Jon > I think it's already become implictly part of the specification as it is now. We had to implement it for GNU Classpath: http://cvs.savannah.gnu.org/viewvc/classpath/sun/reflect/annotation/AnnotationInvocationHandler.java?root=classpath&view=markup so making it explicit and documenting it seems a very good idea! -- Andrew :-) Free Java Software Engineer Red Hat, Inc. (http://www.redhat.com) Support Free Java! Contribute to GNU Classpath and the OpenJDK http://www.gnu.org/software/classpath http://openjdk.java.net PGP Key: 94EFD9D8 (http://subkeys.pgp.net) Fingerprint: F8EF F1EA 401E 2E60 15FA 7927 142C 2591 94EF D9D8 From mr at sun.com Tue Jan 12 13:35:37 2010 From: mr at sun.com (Mark Reinhold) Date: Tue, 12 Jan 2010 13:35:37 -0800 Subject: MODULE variable in makefiles limits module-definition refactoring In-Reply-To: mandy.chung@sun.com; Tue, 12 Jan 2010 13:05:31 PST; <4B4CE41B.8020909@sun.com> Message-ID: <20100112213537.751DE59E@eggemoggin.niobe.net> > Date: Tue, 12 Jan 2010 13:05:31 -0800 > From: mandy.chung at sun.com > It's currently done that way if I understand correctly what you are suggesting. > The member list for all modules is generated at > build/linux-i586/tmp/modules/modules.list (one output file for all module > groups instead of one group). > > I added several lowest-level modules such as awt, java2d, etc that are grouped > under the client module specified in the modules.config so that we can > experiment with the modules.group without requiring modification to the > makefiles. > > The non-class contents are copied from /tmp/modules in its low-level > module during the normal build process (i.e. all target). The modularize > target merges the low-level modules into the corresponding grouped modules in > the /modules directory. > > Does that match what you're suggesting? Yes, exactly -- my mistake for not reading the makefiles more carefully. It'd be good to have some explanatory text at the top of modules/Makefile to explain that the value of the MODULE variable in another makefile might or might not be the name of the module into which that makefile's results are copied. Thanks, - Mark From Mandy.Chung at Sun.COM Tue Jan 12 13:53:56 2010 From: Mandy.Chung at Sun.COM (Mandy Chung) Date: Tue, 12 Jan 2010 13:53:56 -0800 Subject: MODULE variable in makefiles limits module-definition refactoring In-Reply-To: <20100112213537.751DE59E@eggemoggin.niobe.net> References: <20100112213537.751DE59E@eggemoggin.niobe.net> Message-ID: <4B4CEF74.4040308@sun.com> Mark Reinhold wrote: > Yes, exactly -- my mistake for not reading the makefiles more carefully. > > It'd be good to have some explanatory text at the top of modules/Makefile > to explain that the value of the MODULE variable in another makefile > might or might not be the name of the module into which that makefile's > results are copied. > > Will do. Mandy From Mandy.Chung at Sun.COM Wed Jan 13 16:04:53 2010 From: Mandy.Chung at Sun.COM (Mandy Chung) Date: Wed, 13 Jan 2010 16:04:53 -0800 Subject: serialized annotations use sun.reflect.* In-Reply-To: <4B4BE045.3050008@sun.com> References: <4B4BE045.3050008@sun.com> Message-ID: <4B4E5FA5.7090408@Sun.com> Jon, Joe, There should be no issue for jdk classes (such as javac/apt) to have a dependency on sun.reflect.annotation.* classes and they're currently in the base module. At some point in time, we would make private API (e.g sun.*) only accessible by jdk. There are two ways to do it: (1) make the private API to be module-private; or (2) include the private API as a separate private module that only permits by the jdk. There are private sun.* or com.sun.* APIs that applications might depend on. One idea to support the legacy mode is to put these APIs in a private-legacy module so that existing applications can continue to run. If any sun.reflect.annotation.* become part of the public SE spec, we would keep those classes in the base module and remain public then. As the sun.* API is used in the serialized form, I don't know enough if there is any other implication if they are made inaccessible other than the jdk. I need to verify this case. Mandy On 01/11/10 18:36, Jonathan Gibbons wrote: > Mandy, Alan, > > In the modularization effort, is it going to be an issue that javac (and > apt) has an explicit dependence on sun.reflect.annotation.*? Since JDK > 5, the serialized form of annotations has used these otherwise internal > classes. This is true both for annotations created by the runtime, and > by javac as part of JSR 269. For full details, if you really want to, > see 6538914 and 6270734. > Joe and I are trying to figure out the disposition of these bugs -- > whether to use new public classes (and break compatibility) or to accept > the status quo that these classes should be part of the public J2SE spec. > > -- Jon From Mandy.Chung at Sun.COM Thu Jan 14 20:51:48 2010 From: Mandy.Chung at Sun.COM (Mandy Chung) Date: Thu, 14 Jan 2010 20:51:48 -0800 Subject: Review request for 6916217: make/modules/Makefile requires ALT_JDK_IMPORT_PATH Message-ID: <4B4FF464.2010509@sun.com> Fix for 6916217: make/modules/Makefile requires ALT_JDK_IMPORT_PATH Webrev at: http://cr.openjdk.java.net/~mchung/6916217/webrev.00/ This fix addresses two problems: 1) A full jdk build would never need a ALT_JDK_IMPORT_PATH. Fix the make/modules/Makefile to build modules if ALT_JDK_IMPORT_PATH not set. 2) ClassAnalyzer is using the com.sun.tools.classfile library from the langtools repository. We want to always use the latest version from the langtools repository, if exists, as the tool should be synchronized with any classfile API change. For partial forest build where langtools doesn't exist, it will use the build outputdir that imports the tools.jar from JDK_IMPORT_PATH. Thanks Mandy From jonathan.gibbons at sun.com Fri Jan 15 11:15:57 2010 From: jonathan.gibbons at sun.com (jonathan.gibbons at sun.com) Date: Fri, 15 Jan 2010 19:15:57 +0000 Subject: hg: jigsaw/jigsaw/langtools: 25 new changesets Message-ID: <20100115191704.C4A3141755@hg.openjdk.java.net> Changeset: fbeb560f39e7 Author: jjg Date: 2009-12-12 09:28 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/fbeb560f39e7 6907575: [classfile] add support for classfile dependency analysis Reviewed-by: ksrini + src/share/classes/com/sun/tools/classfile/Dependencies.java + src/share/classes/com/sun/tools/classfile/Dependency.java + test/tools/javap/classfile/deps/GetDeps.java + test/tools/javap/classfile/deps/T6907575.java + test/tools/javap/classfile/deps/T6907575.out + test/tools/javap/classfile/deps/p/C1.java Changeset: 0666a8f87661 Author: jjg Date: 2009-12-15 13:26 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/0666a8f87661 6910317: [classfile] typo and other issues in Dependency classes Reviewed-by: ksrini ! src/share/classes/com/sun/tools/classfile/Dependencies.java ! src/share/classes/com/sun/tools/classfile/Dependency.java ! test/tools/javap/classfile/deps/GetDeps.java Changeset: 96c71cbc544b Author: darcy Date: 2009-12-18 11:15 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/96c71cbc544b 6911854: Update TestElementsAnnotatedWith.java to use @compile/proc Reviewed-by: jjg ! test/tools/javac/processing/environment/round/TestElementsAnnotatedWith.java Changeset: 45bd41dcb614 Author: mikejwre Date: 2009-12-03 12:53 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/45bd41dcb614 Added tag jdk7-b77 for changeset 0398ae15b90a ! .hgtags Changeset: ceb2857fce7d Author: tbell Date: 2009-12-08 09:16 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/ceb2857fce7d Merge Changeset: 381399872958 Author: ohair Date: 2009-12-16 12:52 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/381399872958 6909462: Fix nbproject/private references in .hgignore Summary: See bugzilla issue 100097 Reviewed-by: tbell Contributed-by: Jesse Glick ! .hgignore Changeset: acc1e40a5874 Author: mikejwre Date: 2009-12-16 23:39 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/acc1e40a5874 Merge Changeset: 44022ba69c2f Author: mikejwre Date: 2009-12-17 14:10 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/44022ba69c2f Added tag jdk7-b78 for changeset acc1e40a5874 ! .hgtags Changeset: ac5b4c5644ce Author: tbell Date: 2009-12-19 10:26 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/ac5b4c5644ce Merge - src/share/classes/com/sun/tools/javac/file/CloseableURLClassLoader.java Changeset: 0220a3ab1a40 Author: jjg Date: 2010-01-06 13:09 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/0220a3ab1a40 6307206: missing lint control for pkg-info Reviewed-by: darcy ! src/share/classes/com/sun/tools/javac/resources/compiler.properties Changeset: d4e0ae9b4ecb Author: jjg Date: 2010-01-06 13:16 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/d4e0ae9b4ecb 6855236: Compiler Tree API TreePath class generates NullPointerException from Iterator Reviewed-by: darcy + test/tools/javac/T6855236.java Changeset: c315df443ff2 Author: jjg Date: 2010-01-08 11:11 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/c315df443ff2 6878147: Keywords.log is declared and initialized but unused Reviewed-by: darcy ! src/share/classes/com/sun/tools/javac/parser/Keywords.java Changeset: 2d15bf467aea Author: jjg Date: 2010-01-08 11:16 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/2d15bf467aea 6878146: incorrect unused value should be deleted Reviewed-by: darcy ! src/share/classes/com/sun/tools/javac/util/LayoutCharacters.java Changeset: 0e75f9f6d1d4 Author: jjg Date: 2010-01-08 11:28 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/0e75f9f6d1d4 6665791: com.sun.source.tree.MethodTree.toString() does not output default values Reviewed-by: darcy ! src/share/classes/com/sun/tools/javac/tree/Pretty.java + test/tools/javac/T6665791.java Changeset: aa06467be3a2 Author: jjg Date: 2010-01-08 11:32 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/aa06467be3a2 6915078: ALT_JDK_IMPORT_PATH typo in langtools/make/Makefile Reviewed-by: tbell ! make/Makefile Changeset: 96c56220dcc2 Author: jjg Date: 2010-01-08 13:14 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/96c56220dcc2 6915152: langtools build failures with import.jdk on Windows Reviewed-by: ohair ! make/build.xml Changeset: d02e99d31cc0 Author: jjg Date: 2010-01-11 14:05 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/d02e99d31cc0 6326754: Compiler will fail to handle -Xmaxerrs with -ve numbers Reviewed-by: ksrini ! src/share/classes/com/sun/tools/javac/util/Log.java + test/tools/javac/T6326754.java + test/tools/javac/T6326754.out Changeset: f983c1dca202 Author: jjg Date: 2010-01-11 14:09 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/f983c1dca202 6764569: [PATCH] Fix unused imports in list resource bundles Reviewed-by: ksrini Contributed-by: jesse.glick at sun.com ! make/tools/CompileProperties/CompileProperties.java ! make/tools/CompileProperties/CompilePropertiesTask.java Changeset: ca6bc36b2305 Author: jjg Date: 2010-01-11 14:12 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/ca6bc36b2305 6915476: java.util.regex.PatternSyntaxException in com.sun.tools.javac.nio.PathFileObject Reviewed-by: darcy ! src/share/classes/com/sun/tools/javac/nio/PathFileObject.java ! test/tools/javac/nio/compileTest/CompileTest.java Changeset: 14a4c45ef734 Author: jjg Date: 2010-01-11 14:17 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/14a4c45ef734 6915497: test test/tools/javac/nio/compileTest/CompileTest.java fails under Hudson Reviewed-by: darcy ! test/tools/javac/nio/compileTest/CompileTest.java Changeset: 51011e02c02f Author: jjg Date: 2010-01-11 16:18 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/51011e02c02f 6909470: langtools stub generator should prune unnecessary imports Reviewed-by: darcy ! make/tools/GenStubs/GenStubs.java Changeset: ccd51af119b4 Author: jjg Date: 2010-01-13 17:39 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/ccd51af119b4 6472751: SourcePositions.getStartPos returns incorrect value for enum constants 6567414: javac compiler reports no source file or line on enum constant declaration error Reviewed-by: darcy ! src/share/classes/com/sun/tools/javac/parser/JavacParser.java + test/tools/javac/T6472751.java + test/tools/javac/T6567414.java + test/tools/javac/T6567414.out Changeset: b96ad32c004a Author: jjg Date: 2010-01-14 17:18 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/b96ad32c004a 6917122: provide utility method to find the inner most type of a type tree Reviewed-by: darcy, jjg Contributed-by: mali at csail.mit.edu, mernst at cs.washington.edu ! src/share/classes/com/sun/tools/javac/tree/Pretty.java ! src/share/classes/com/sun/tools/javac/tree/TreeInfo.java Changeset: 2d0f4e7b44b2 Author: jjg Date: 2010-01-14 17:23 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/2d0f4e7b44b2 6916986: handle spaces in langtools launcher path Reviewed-by: darcy, jjg Contributed-by: mali at csail.mit.edu, mernst at cs.washington.edu ! src/share/bin/launcher.sh-template Changeset: c74948444cba Author: jjg Date: 2010-01-15 11:14 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/c74948444cba Merge ! .hgtags ! make/build.xml ! make/tools/GenStubs/GenStubs.java + src/share/classes/com/sun/tools/classfile/Dependencies.java ! src/share/classes/com/sun/tools/javac/parser/JavacParser.java ! src/share/classes/com/sun/tools/javac/parser/Keywords.java ! src/share/classes/com/sun/tools/javac/resources/compiler.properties ! src/share/classes/com/sun/tools/javac/tree/Pretty.java ! src/share/classes/com/sun/tools/javac/tree/TreeInfo.java From mr at sun.com Fri Jan 15 14:11:06 2010 From: mr at sun.com (Mark Reinhold) Date: Fri, 15 Jan 2010 14:11:06 -0800 Subject: Module-file format (DRAFT) Message-ID: <20100115221106.25DC91C7F@eggemoggin.niobe.net> Here's a first cut at a simple module-file format. Comments welcome! - Mark ---- Jigsaw module-file format DRAFT =============================== Mark Reinhold 15 January 2010 Goals ----- - Optimized for streamed reading, from beginning to end. Random access should not be required when reading a module file, though it can be required when writing. - Content-specific compression: Pack200+gzip/lzma for classes, bzip2 for native code, etc. - Independent of any specific installed-module format or target-filesystem capability. Layout ------ A module file has the following sections, in the order listed. The first two sections are required: - Header -- Magic number, format version, sizes - module-info.class (never compressed, so as to be easily readable by tools) These sections are optional: - Classes (excluding module-info.class) - Resources - Native libraries - Native launchers - Configuration (i.e., properties) files The final section is required: - Secure hash of the entire file (except for this section) Module file header ------------------ ModuleFileHeader { u4 magic; // FileConstants.MAGIC u2 type; // FileConstants.Type.MODULE_FILE u2 major; // FileConstants.ModuleFile.MAJOR_VERSION u2 minor; // FileConstants.ModuleFile.MINOR_VERSION u4 csize; // Size of entire file, compressed u4 usize; // Space required for uncompressed contents // (upper bound; need not be exact) } All integers are in network byte order. Section headers --------------- Each section has a header: SectionHeader { u2 type; // One of FileConstants.ModuleFile.SectionType u2 compressor; // One of FileConstants.ModuleFile.Compressor } Compression is only relevant to section content; it never applies to the headers defined here. For sections that contain only one entity (i.e., module-info, classes, hash) the section header is followed by a size pair, which is then followed by the content: SectionSize { u4 csize; // Size of section content, compressed u4 usize; // Size of section content, uncompressed } If a section contains named files (i.e., resources, native code, config) then each file within that section has a header, which is then followed by the content: SectionFileHeader { u4 csize; // Size of file, compressed u4 usize; // Size of file, uncompressed u2 nameLength; // Length of file name b* name; // File name, in UTF-8 } File names may include directory separators, though they should not be used for native-code files. There is no mode field on files within a section. The module-installation code must set the local mode appropriately; e.g., it must make native-code files executable as appropriate. A hash section has the content: HashSection { u2 type; // One of FileConstants.ModuleFile.HashType u2 size; // Never compressed b* hash; } Open issues ----------- - Signing: Should module files carry their own digital-signature information, as jar files do today, or should such information be provided elsewhere? From Jonathan.Gibbons at Sun.COM Fri Jan 15 14:17:18 2010 From: Jonathan.Gibbons at Sun.COM (Jonathan Gibbons) Date: Fri, 15 Jan 2010 14:17:18 -0800 Subject: Module-file format (DRAFT) In-Reply-To: <20100115221106.25DC91C7F@eggemoggin.niobe.net> References: <20100115221106.25DC91C7F@eggemoggin.niobe.net> Message-ID: <4B50E96E.7080505@sun.com> I assume that UTF8 always means Java's version of UTF8, as used in classfiles and DataInputStream/DataOutputStream. -- Jon Mark Reinhold wrote: > Here's a first cut at a simple module-file format. > > Comments welcome! > > - Mark > > ---- > > Jigsaw module-file format DRAFT > =============================== > > Mark Reinhold > 15 January 2010 > > > Goals > ----- > > - Optimized for streamed reading, from beginning to end. Random access > should not be required when reading a module file, though it can be > required when writing. > > - Content-specific compression: Pack200+gzip/lzma for classes, bzip2 for > native code, etc. > > - Independent of any specific installed-module format or target-filesystem > capability. > > > Layout > ------ > > A module file has the following sections, in the order listed. > > The first two sections are required: > > - Header -- Magic number, format version, sizes > - module-info.class (never compressed, so as to be easily readable by tools) > > These sections are optional: > > - Classes (excluding module-info.class) > - Resources > - Native libraries > - Native launchers > - Configuration (i.e., properties) files > > The final section is required: > > - Secure hash of the entire file (except for this section) > > > Module file header > ------------------ > > ModuleFileHeader { > u4 magic; // FileConstants.MAGIC > u2 type; // FileConstants.Type.MODULE_FILE > u2 major; // FileConstants.ModuleFile.MAJOR_VERSION > u2 minor; // FileConstants.ModuleFile.MINOR_VERSION > u4 csize; // Size of entire file, compressed > u4 usize; // Space required for uncompressed contents > // (upper bound; need not be exact) > } > > All integers are in network byte order. > > > Section headers > --------------- > > Each section has a header: > > SectionHeader { > u2 type; // One of FileConstants.ModuleFile.SectionType > u2 compressor; // One of FileConstants.ModuleFile.Compressor > } > > Compression is only relevant to section content; it never applies to the > headers defined here. > > For sections that contain only one entity (i.e., module-info, classes, hash) > the section header is followed by a size pair, which is then followed by the > content: > > SectionSize { > u4 csize; // Size of section content, compressed > u4 usize; // Size of section content, uncompressed > } > > If a section contains named files (i.e., resources, native code, config) then > each file within that section has a header, which is then followed by the > content: > > SectionFileHeader { > u4 csize; // Size of file, compressed > u4 usize; // Size of file, uncompressed > u2 nameLength; // Length of file name > b* name; // File name, in UTF-8 > } > > File names may include directory separators, though they should not be used for > native-code files. > > There is no mode field on files within a section. The module-installation code > must set the local mode appropriately; e.g., it must make native-code files > executable as appropriate. > > A hash section has the content: > > HashSection { > u2 type; // One of FileConstants.ModuleFile.HashType > u2 size; // Never compressed > b* hash; > } > > > Open issues > ----------- > > - Signing: Should module files carry their own digital-signature information, > as jar files do today, or should such information be provided elsewhere? > From mr at sun.com Fri Jan 15 14:29:28 2010 From: mr at sun.com (mr at sun.com) Date: Fri, 15 Jan 2010 22:29:28 +0000 Subject: hg: jigsaw/jigsaw/jdk: Module-file constants Message-ID: <20100115222949.5A79241788@hg.openjdk.java.net> Changeset: 908a82c7f43e Author: mr Date: 2010-01-15 14:28 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/908a82c7f43e Module-file constants ! src/share/classes/org/openjdk/jigsaw/FileConstants.java From mr at sun.com Fri Jan 15 14:30:55 2010 From: mr at sun.com (Mark Reinhold) Date: Fri, 15 Jan 2010 14:30:55 -0800 Subject: Module-file format (DRAFT) In-Reply-To: jonathan.gibbons@sun.com; Fri, 15 Jan 2010 14:17:18 PST; <4B50E96E.7080505@sun.com> Message-ID: <20100115223055.A2FF81C7F@eggemoggin.niobe.net> > I assume that UTF8 always means Java's version of UTF8, as used in classfiles > and DataInputStream/DataOutputStream. Yep. - Mark From Alex.Buckley at Sun.COM Fri Jan 15 14:52:01 2010 From: Alex.Buckley at Sun.COM (Alex Buckley) Date: Fri, 15 Jan 2010 14:52:01 -0800 Subject: Module-file format (DRAFT) In-Reply-To: <20100115221106.25DC91C7F@eggemoggin.niobe.net> References: <20100115221106.25DC91C7F@eggemoggin.niobe.net> Message-ID: <4B50F191.90405@sun.com> - Does "length of filename" mean a) number of bytes in the 'name' item, or b) the number of Unicode code points in the 'name' item ? - Filenames can include path separators, implying relative filenames. What are they relative to, or does it not matter? - A top-level ModuleFile structure would contain ModuleFileHeader + ModuleInfoSection + an array of section objects + HashSection. A count of the number of section objects is useful as a minimal well-formedness check. - In ModuleFileHeader, it would be super-smart to add a 'u2 meta_version' item that indicates the version scheme used for 'major' and 'minor'. This allows corrections to the semantics of major/minor over time. - Is a u4 really enough for every size item? Alex Mark Reinhold wrote: > Here's a first cut at a simple module-file format. > > Comments welcome! > > - Mark > > ---- > > Jigsaw module-file format DRAFT > =============================== > > Mark Reinhold > 15 January 2010 > > > Goals > ----- > > - Optimized for streamed reading, from beginning to end. Random access > should not be required when reading a module file, though it can be > required when writing. > > - Content-specific compression: Pack200+gzip/lzma for classes, bzip2 for > native code, etc. > > - Independent of any specific installed-module format or target-filesystem > capability. > > > Layout > ------ > > A module file has the following sections, in the order listed. > > The first two sections are required: > > - Header -- Magic number, format version, sizes > - module-info.class (never compressed, so as to be easily readable by tools) > > These sections are optional: > > - Classes (excluding module-info.class) > - Resources > - Native libraries > - Native launchers > - Configuration (i.e., properties) files > > The final section is required: > > - Secure hash of the entire file (except for this section) > > > Module file header > ------------------ > > ModuleFileHeader { > u4 magic; // FileConstants.MAGIC > u2 type; // FileConstants.Type.MODULE_FILE > u2 major; // FileConstants.ModuleFile.MAJOR_VERSION > u2 minor; // FileConstants.ModuleFile.MINOR_VERSION > u4 csize; // Size of entire file, compressed > u4 usize; // Space required for uncompressed contents > // (upper bound; need not be exact) > } > > All integers are in network byte order. > > > Section headers > --------------- > > Each section has a header: > > SectionHeader { > u2 type; // One of FileConstants.ModuleFile.SectionType > u2 compressor; // One of FileConstants.ModuleFile.Compressor > } > > Compression is only relevant to section content; it never applies to the > headers defined here. > > For sections that contain only one entity (i.e., module-info, classes, hash) > the section header is followed by a size pair, which is then followed by the > content: > > SectionSize { > u4 csize; // Size of section content, compressed > u4 usize; // Size of section content, uncompressed > } > > If a section contains named files (i.e., resources, native code, config) then > each file within that section has a header, which is then followed by the > content: > > SectionFileHeader { > u4 csize; // Size of file, compressed > u4 usize; // Size of file, uncompressed > u2 nameLength; // Length of file name > b* name; // File name, in UTF-8 > } > > File names may include directory separators, though they should not be used for > native-code files. > > There is no mode field on files within a section. The module-installation code > must set the local mode appropriately; e.g., it must make native-code files > executable as appropriate. > > A hash section has the content: > > HashSection { > u2 type; // One of FileConstants.ModuleFile.HashType > u2 size; // Never compressed > b* hash; > } > > > Open issues > ----------- > > - Signing: Should module files carry their own digital-signature information, > as jar files do today, or should such information be provided elsewhere? From Alan.Bateman at Sun.COM Mon Jan 18 10:06:28 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Mon, 18 Jan 2010 18:06:28 +0000 Subject: Review request for 6916217: make/modules/Makefile requires ALT_JDK_IMPORT_PATH In-Reply-To: <4B4FF464.2010509@sun.com> References: <4B4FF464.2010509@sun.com> Message-ID: <4B54A324.1060905@sun.com> Mandy Chung wrote: > Fix for 6916217: make/modules/Makefile requires ALT_JDK_IMPORT_PATH > > Webrev at: > http://cr.openjdk.java.net/~mchung/6916217/webrev.00/ > > This fix addresses two problems: > 1) A full jdk build would never need a ALT_JDK_IMPORT_PATH. Fix the > make/modules/Makefile to build modules if ALT_JDK_IMPORT_PATH not set. > > 2) ClassAnalyzer is using the com.sun.tools.classfile library from the > langtools repository. We want to always use the latest version from > the langtools repository, if exists, as the tool should be > synchronized with any classfile API change. For partial forest build > where langtools doesn't exist, it will use the build outputdir that > imports the tools.jar from JDK_IMPORT_PATH. > > Thanks > Mandy I worked through the make file changes and it looks OK to me. -Alan. From Mandy.Chung at Sun.COM Mon Jan 18 11:35:26 2010 From: Mandy.Chung at Sun.COM (Mandy Chung) Date: Mon, 18 Jan 2010 11:35:26 -0800 Subject: Review request for 6916217: make/modules/Makefile requires ALT_JDK_IMPORT_PATH In-Reply-To: <4B54A324.1060905@sun.com> References: <4B4FF464.2010509@sun.com> <4B54A324.1060905@sun.com> Message-ID: <4B54B7FE.6070602@sun.com> Thanks Alan. Mandy Alan Bateman wrote: > Mandy Chung wrote: >> Fix for 6916217: make/modules/Makefile requires ALT_JDK_IMPORT_PATH >> >> Webrev at: >> http://cr.openjdk.java.net/~mchung/6916217/webrev.00/ >> >> This fix addresses two problems: >> 1) A full jdk build would never need a ALT_JDK_IMPORT_PATH. Fix the >> make/modules/Makefile to build modules if ALT_JDK_IMPORT_PATH not set. >> >> 2) ClassAnalyzer is using the com.sun.tools.classfile library from >> the langtools repository. We want to always use the latest version >> from the langtools repository, if exists, as the tool should be >> synchronized with any classfile API change. For partial forest build >> where langtools doesn't exist, it will use the build outputdir that >> imports the tools.jar from JDK_IMPORT_PATH. >> >> Thanks >> Mandy > I worked through the make file changes and it looks OK to me. > > -Alan. From Alan.Bateman at Sun.COM Tue Jan 19 10:07:04 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Tue, 19 Jan 2010 18:07:04 +0000 Subject: Module-file format (DRAFT) In-Reply-To: <20100115221106.25DC91C7F@eggemoggin.niobe.net> References: <20100115221106.25DC91C7F@eggemoggin.niobe.net> Message-ID: <4B55F4C8.5020302@sun.com> Mark Reinhold wrote: > Here's a first cut at a simple module-file format. > The format seem straight-forward. One thing that isn't completely clear (to me anyway) is how does one recognize the end of a section that contains several named files? In the following, for example, how do I know distinguish a new section header from another SectionFileHeader? SectionHeader type = NATIVE_LIBS SectionFileHeader usize = 50k, name = libfoo.so [libfoo.so] SectionSize usize = 70k, name = libbar.so [libbar.so] SectionHeader type = CLASSES : I agree with Alex's comment that u4 might not be sufficient for the compressed or uncompressed size. Is the intention that a module can be launched/started before it is fully transferred/"installed" and so lend itself to placement optimizations to improve first-time startup? If so, I wonder if the end of the entire file is the right place for the secure hash. I realize signing is an open issue I assume that the streaming means that the signature verification would be required per section to allow classes or resources to be loaded before transfer has completed. -Alan. From mr at sun.com Tue Jan 19 10:30:04 2010 From: mr at sun.com (Mark Reinhold) Date: Tue, 19 Jan 2010 10:30:04 -0800 Subject: Module-file format (DRAFT) In-Reply-To: alex.buckley@sun.com; Fri, 15 Jan 2010 14:52:01 PST; <4B50F191.90405@sun.com> Message-ID: <20100119183004.44E51293@eggemoggin.niobe.net> > Date: Fri, 15 Jan 2010 14:52:01 -0800 > From: alex.buckley at sun.com > - Does "length of filename" mean a) number of bytes in the 'name' item, or b) > the number of Unicode code points in the 'name' item ? The former. > - Filenames can include path separators, implying relative filenames. What are > they relative to, or does it not matter? It doesn't matter. > - A top-level ModuleFile structure would contain ModuleFileHeader + > ModuleInfoSection + an array of section objects + HashSection. A count of the > number of section objects is useful as a minimal well-formedness check. Okay; added. > - In ModuleFileHeader, it would be super-smart to add a 'u2 meta_version' item > that indicates the version scheme used for 'major' and 'minor'. This allows > corrections to the semantics of major/minor over time. That seems like overkill -- it's really just another version-number element. If we someday have to make such a deep change that new major/minor semantics are required then we can just define a new file type. > - Is a u4 really enough for every size item? Probably not. Upgraded to u8. - Mark From mr at sun.com Tue Jan 19 10:50:30 2010 From: mr at sun.com (Mark Reinhold) Date: Tue, 19 Jan 2010 10:50:30 -0800 Subject: Module-file format (DRAFT) In-Reply-To: alan.bateman@sun.com; Tue, 19 Jan 2010 18:07:04 GMT; <4B55F4C8.5020302@sun.com> Message-ID: <20100119185030.F27D2293@eggemoggin.niobe.net> > Date: Tue, 19 Jan 2010 18:07:04 +0000 > From: alan.bateman at sun.com > The format seem straight-forward. One thing that isn't completely clear (to me > anyway) is how does one recognize the end of a section that contains several > named files? In the following, for example, how do I know distinguish a new > section header from another SectionFileHeader? > > SectionHeader > type = NATIVE_LIBS > SectionFileHeader > usize = 50k, name = libfoo.so > [libfoo.so] > SectionSize (I think you mean SectionFileHeader here.) > usize = 70k, name = libbar.so > [libbar.so] > SectionHeader > type = CLASSES > : Good question -- you can't. I've added a u2 type field to SectionFileHeader so that this is possible. > I agree with Alex's comment that u4 might not be sufficient for the compressed > or uncompressed size. Fixed. > Is the intention that a module can be launched/started before it is fully > transferred/"installed" No. An application that needs to (appear to) start quickly should be structured as a small launch module with optional dependences upon its other modules. (This is just as in JNLP today.) We'll eventually come up with a way to label such dependences so that they'll be downloaded and installed right after the launch module. - Mark From mr at sun.com Tue Jan 19 10:54:37 2010 From: mr at sun.com (Mark Reinhold) Date: Tue, 19 Jan 2010 10:54:37 -0800 Subject: Module-file format (DRAFT) In-Reply-To: mr@sun.com; Fri, 15 Jan 2010 14:11:06 PST; <20100115221106.25DC91C7F@eggemoggin.niobe.net> Message-ID: <20100119185437.5D72A293@eggemoggin.niobe.net> Thanks for all the comments. I've posted an update version here: http://cr.openjdk.java.net/~mr/jigsaw/notes/module-file-format - Mark From mr at sun.com Tue Jan 19 11:16:33 2010 From: mr at sun.com (mr at sun.com) Date: Tue, 19 Jan 2010 19:16:33 +0000 Subject: hg: jigsaw/jigsaw/jdk: Module-file constants update Message-ID: <20100119191726.207E241D81@hg.openjdk.java.net> Changeset: 9b0c93dee7a7 Author: mr Date: 2010-01-19 11:16 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/9b0c93dee7a7 Module-file constants update ! src/share/classes/org/openjdk/jigsaw/FileConstants.java From reinier at zwitserloot.com Tue Jan 19 12:09:32 2010 From: reinier at zwitserloot.com (Reinier Zwitserloot) Date: Tue, 19 Jan 2010 21:09:32 +0100 Subject: Module-file format (DRAFT) Message-ID: <560fb5ed1001191209i74ed9499oe96bb8d30425eaf5@mail.gmail.com> * Should SectionSize/SectionFileHeader.usize/csize also join ModuleFileHeader.u/csize and become u8 instead of the current u4? * Should SectionHeader include a new u2 'version' field? analysis for this issue: If an individual section type gets an update, then either the entire module file format needs a version rev, which means all dependent tools immediately stop working with it until they also upgrade their code. This is a bad thing if the section's type identifies it as something that is irrelevant for a certain module file processor. Alternatively, the section can rev itself by picking a new FileConstants.ModuleFile.SectionType, but this will surely lead to haphazard type ids or a risk of running out of type ids if each id claims lets say 100 slots for potential future versioning. Even if this is not a problem, a dependent tool that is looking for, say, an "Author Information" section will get confused and fail with an error 'module file is missing author information', even though the actual problem is that the author information block is present as a version too new for the tool to understand. If the version of a section was separate, then the tool could correctly report that the relevant section is stored in a version that's too new for the tool to understand. * What's the point of having multiple different hash algorithms? I presume the hash is simply a mechanism to assert that the file has not been tampered with or corrupted during transit, and possibly as a quick way to identify a given module file as having remained unchanged, and that it isn't intended to fulfill the rule of a signature like signed jars (if that does get added to the module file format, I'd presume this will occur via a new section, and not via the hash section). analysis for this issue: Every so often a hash algorithm gets cryptographically compromised. SHA-1 is basically in the process of getting compromised in this fashion, and I would certainly suggest SHA256 or SHA512 is used for the module file format, which so far haven't been compromised in any practical manner. It is of course feasible to believe SHA512 gets compromised at some point. There's a fix for that, though: Rev the version of the module file format itself, and dictate a new hash algorithm in the new version. The hash is something everyone basically needs to adhere to; if it is to have any significant security impact, _ALL_ tools that read module files should, with strong preference, check it and refuse to process module files with a bad hash. However, if there's a smattering of different hash algorithms available, this is going to make writing these tools far more difficult. After all, you'd have to carefully write, test, and maintain each and every legal hash algorithm. I can easily imagine this scenario occurring: java itself understands all of lets say 5 legal hash algorithms. However, all tools that ship with the JDK in practice only ever generate 4 of those. Some fairly obscure tool for some reason decides to hash with rarely used hasher algorithm 5, and it is released to the world as it gets tested with the JVM itself, which understands it. Some other fairly obscure tool has a bug in its hasher for hash type 5, but it, too, passes internal testing and is released to the world because it gets tested with the output from the JDK tools, which never generate this hash. Then some poor soul gets confronted with the fact that obscure tool number one (which generates hash #5) crashes when used with obscure tool number two (which has a bug in its hash #5 algorithm reader). Seems simpler to me to just dictate hash algorithm with the understanding that if it is ever compromised, the module file format itself will rev up a version to fix it. * Without knowing every possible section type, how can a tool know if a certain section is based around SectionFileHeaders, or SectionSize? It cannot check the type of the next chunk - if it is indeed FileConstants.ModuleFile.SectionType.FILE it's most likely SectionFileHeader based, but it could just be a coincidence and part of the SectionSize.csize field. What happens if a SectionFileHeader based section has 0 files in it? As it stands, HashSection's type numbers cannot conflict with FileConstants.ModuleFile.SectionType.FILE lest tools are forced to keep a running count of the file's csize to realize it must have reached the end, which sounds rather cumbersome. This can be fixed relatively cheaply: Reserve 1 bit in the SectionHeader.type field as indicating whether the section is single-entity or FileHeaders based, then let a FileHeaders-based section be followed by a u4: number of files that follow. A tool that neither knows nor cares about a FileHeaders based section can now easily skip it by looping through each file, reading the csize and pathLength from it, and skipping the appropriate number of bytes. Alternatively, move the SectionSize block into the SectionHeader block (in other words, make it mandatory even for files-based sections). --Reinier Zwitserloot From Dalibor.Topic at Sun.COM Tue Jan 19 12:43:35 2010 From: Dalibor.Topic at Sun.COM (Dalibor Topic) Date: Tue, 19 Jan 2010 21:43:35 +0100 Subject: Module-file format (DRAFT) In-Reply-To: <20100119185437.5D72A293@eggemoggin.niobe.net> References: <20100119185437.5D72A293@eggemoggin.niobe.net> Message-ID: <4B561977.8010100@sun.com> Mark Reinhold wrote: > Thanks for all the comments. I've posted an update version here: > > http://cr.openjdk.java.net/~mr/jigsaw/notes/module-file-format > Thanks for the updated version. Given that we know the rough size of the total uncompressed files from the module file header, do we need to spend a u4 for each file's uncompressed size? cheers, dalibor topic -- ******************************************************************* Dalibor Topic Tel: (+49 40) 23 646 738 Java F/OSS Ambassador AIM: robiladonaim Sun Microsystems GmbH Mobile: (+49 177) 2664 192 Nagelsweg 55 http://openjdk.java.net D-20097 Hamburg mailto:Dalibor.Topic at sun.com Sitz der Gesellschaft: Sonnenallee 1, D-85551 Kirchheim-Heimstetten Amtsgericht M?nchen: HRB 161028 Gesch?ftsf?hrer: Thomas Schr?der, Wolfgang Engels, Wolf Frenkel Vorsitzender des Aufsichtsrates: Martin H?ring From Jonathan.Gibbons at Sun.COM Tue Jan 19 13:52:03 2010 From: Jonathan.Gibbons at Sun.COM (Jonathan Gibbons) Date: Tue, 19 Jan 2010 13:52:03 -0800 Subject: issues with jdk/make/modules/modularize.sh Message-ID: <4B562983.8010008@sun.com> The current jigsaw forest does not build with the latest javac that I would like to put back. The build fails like this: > make[3]: Entering directory > `/mnt/w/jjg/work/jigsaw/jigsaw.testbuild/jdk/make/modules' > SRC=../../src/share/modules > BIN=/mnt/w/jjg/work/jigsaw/jigsaw.testbuild/build/linux-amd64/bin > CLASSES=/mnt/w/jjg/work/jigsaw/jigsaw.testbuild/build/linux-amd64/classes > TMP=/mnt/w/jjg/work/jigsaw/jigsaw.testbuild/build/linux-amd64/tmp/jigsaw > MLIB=/mnt/w/jjg/work/jigsaw/jigsaw.testbuild/build/linux-amd64/lib/modules > \ > JIGSAW_TRACE=3 sh modularize.sh boot base awt swing tools > -- jdk.boot > error: module library not found: > /mnt/w/jjg/work/jigsaw/jigsaw.testbuild/build/linux-amd64/lib/modules > 1 error In modularize.sh I see some lines of the form: $BIN/javac -source 7 -d $MCLS -modulepath $SRC $SRC/$m/module-info.java This line looks suspect, if only for putting what looks like a source directory on the module path. It should probably be: $BIN/javac -source 7 -d $MCLS -modulepath $MCLS -sourcepath $SRC $SRC/$m/module-info.java It's also not the best way to invoke javac -- once per source file, using sourcepath, as compared to a single invocation for all relevant module-info.java files. Finally, this is the beginning of the iceberg of how to use javac to bootstrap the module system into existence. The error message from javac (module library not found) indicates that javac is trying to do a normal "end user" compilation, involving the default system module library. This is new functionality in this version of javac. The question is, how should we be handling this situation here? -- Should we simply create (mkdir) an empty modules directory to keep javac happy? -- Or, should javac have an option to disable looking for the system module library? -- Or should javac not be looking for the system module library if it can satisfy all references via the command line. -- Or should javac be running in a special bootstrap mode that is used for internal cases like this? -- Jon From Mandy.Chung at Sun.COM Tue Jan 19 14:38:34 2010 From: Mandy.Chung at Sun.COM (Mandy Chung) Date: Tue, 19 Jan 2010 14:38:34 -0800 Subject: issues with jdk/make/modules/modularize.sh In-Reply-To: <4B562983.8010008@sun.com> References: <4B562983.8010008@sun.com> Message-ID: <4B56346A.30206@sun.com> Jonathan Gibbons wrote: > The current jigsaw forest does not build with the latest javac that I > would like to put back. > > The build fails like this: > >> make[3]: Entering directory >> `/mnt/w/jjg/work/jigsaw/jigsaw.testbuild/jdk/make/modules' >> SRC=../../src/share/modules >> BIN=/mnt/w/jjg/work/jigsaw/jigsaw.testbuild/build/linux-amd64/bin >> CLASSES=/mnt/w/jjg/work/jigsaw/jigsaw.testbuild/build/linux-amd64/classes >> TMP=/mnt/w/jjg/work/jigsaw/jigsaw.testbuild/build/linux-amd64/tmp/jigsaw >> MLIB=/mnt/w/jjg/work/jigsaw/jigsaw.testbuild/build/linux-amd64/lib/modules >> \ >> JIGSAW_TRACE=3 sh modularize.sh boot base awt swing tools >> -- jdk.boot >> error: module library not found: >> /mnt/w/jjg/work/jigsaw/jigsaw.testbuild/build/linux-amd64/lib/modules >> 1 error > > In modularize.sh I see some lines of the form: > > $BIN/javac -source 7 -d $MCLS -modulepath $SRC $SRC/$m/module-info.java > > This line looks suspect, if only for putting what looks like a source > directory on the module path. It should probably be: > > $BIN/javac -source 7 -d $MCLS -modulepath $MCLS -sourcepath $SRC > $SRC/$m/module-info.java > > It's also not the best way to invoke javac -- once per source file, > using sourcepath, as compared to a single invocation for all relevant > module-info.java files. > Jon, I'm working on merging the modules build makefile changes in TL with the jigsaw modules build. modularize.sh and imagine.sh will go away. I have changed the makefile to compile all module-info.java files in one javac invocation in my repo. As this is new javac functionality, I didn't run into the problem you described above in my repo. modularize.sh invokes jmod to install the module library. I wonder if this issue is related to the invocaton to jmod in creating the module library. Mandy > Finally, this is the beginning of the iceberg of how to use javac to > bootstrap the module system into existence. > > The error message from javac (module library not found) indicates that > javac is trying to do a normal "end user" compilation, involving the > default system module library. This is new functionality in this > version of javac. > > The question is, how should we be handling this situation here? > -- Should we simply create (mkdir) an empty modules directory to keep > javac happy? > -- Or, should javac have an option to disable looking for the system > module library? > -- Or should javac not be looking for the system module library if it > can satisfy all references via the command line. > -- Or should javac be running in a special bootstrap mode that is used > for internal cases like this? > > -- Jon From Jonathan.Gibbons at Sun.COM Tue Jan 19 14:50:56 2010 From: Jonathan.Gibbons at Sun.COM (Jonathan Gibbons) Date: Tue, 19 Jan 2010 14:50:56 -0800 Subject: issues with jdk/make/modules/modularize.sh In-Reply-To: <4B56346A.30206@sun.com> References: <4B562983.8010008@sun.com> <4B56346A.30206@sun.com> Message-ID: <4B563750.907@sun.com> On 01/19/2010 02:38 PM, Mandy Chung wrote: > > Jonathan Gibbons wrote: >> The current jigsaw forest does not build with the latest javac that >> I would like to put back. >> >> The build fails like this: >> >>> make[3]: Entering directory >>> `/mnt/w/jjg/work/jigsaw/jigsaw.testbuild/jdk/make/modules' >>> SRC=../../src/share/modules >>> BIN=/mnt/w/jjg/work/jigsaw/jigsaw.testbuild/build/linux-amd64/bin >>> CLASSES=/mnt/w/jjg/work/jigsaw/jigsaw.testbuild/build/linux-amd64/classes >>> TMP=/mnt/w/jjg/work/jigsaw/jigsaw.testbuild/build/linux-amd64/tmp/jigsaw >>> MLIB=/mnt/w/jjg/work/jigsaw/jigsaw.testbuild/build/linux-amd64/lib/modules >>> \ >>> JIGSAW_TRACE=3 sh modularize.sh boot base awt swing tools >>> -- jdk.boot >>> error: module library not found: >>> /mnt/w/jjg/work/jigsaw/jigsaw.testbuild/build/linux-amd64/lib/modules >>> 1 error >> >> In modularize.sh I see some lines of the form: >> >> $BIN/javac -source 7 -d $MCLS -modulepath $SRC >> $SRC/$m/module-info.java >> >> This line looks suspect, if only for putting what looks like a source >> directory on the module path. It should probably be: >> >> $BIN/javac -source 7 -d $MCLS -modulepath $MCLS -sourcepath $SRC >> $SRC/$m/module-info.java >> >> It's also not the best way to invoke javac -- once per source file, >> using sourcepath, as compared to a single invocation for all relevant >> module-info.java files. >> > Jon, > > I'm working on merging the modules build makefile changes in TL with > the jigsaw modules build. modularize.sh and imagine.sh will go away. > > I have changed the makefile to compile all module-info.java files in > one javac invocation in my repo. As this is new javac functionality, > I didn't run into the problem you described above in my repo. > modularize.sh invokes jmod to install the module library. I wonder if > this issue is related to the invocaton to jmod in creating the module > library. > > Mandy > >> Finally, this is the beginning of the iceberg of how to use javac to >> bootstrap the module system into existence. >> >> The error message from javac (module library not found) indicates >> that javac is trying to do a normal "end user" compilation, involving >> the default system module library. This is new functionality in this >> version of javac. >> >> The question is, how should we be handling this situation here? >> -- Should we simply create (mkdir) an empty modules directory to keep >> javac happy? >> -- Or, should javac have an option to disable looking for the system >> module library? >> -- Or should javac not be looking for the system module library if it >> can satisfy all references via the command line. >> -- Or should javac be running in a special bootstrap mode that is >> used for internal cases like this? >> >> -- Jon > Mandy, I'm referring to my latest version of javac, that I have yet to push to a public forest. However, if modularize.sh will go away soon, I won't worry about it too much. -- Jon From Roger.Riggs at Sun.COM Tue Jan 19 14:51:15 2010 From: Roger.Riggs at Sun.COM (Roger Riggs) Date: Tue, 19 Jan 2010 17:51:15 -0500 Subject: Module-file format (DRAFT) In-Reply-To: <20100119185437.5D72A293@eggemoggin.niobe.net> References: <20100119185437.5D72A293@eggemoggin.niobe.net> Message-ID: <4B563763.5080609@Sun.COM> Hi Mark, A couple of comments: Do I read this correctly that compression is only file-by-file (SectionFileHeader)? Frequently compression algorithms work better with more content to compress. But perhaps I don't understand what goes into a "CLASSES" section. Is it assumed to be a sequence of classfiles without names? A description of the allowed ordering or nesting would be useful (BNF). It would be useful if the HashSection was allowed to be in each section as well as the entire file and was required to be after the ModuleFileHeader or SectionHeader. Then an implementation could verify that the module or section had the expected data before reading it. The would be useful when verifying signatures. There is no benefit to putting the hash at the end of the file because the format requires all of the compression to be done before any of the sections can be written. Roger On 1/19/10 1:54 PM, Mark Reinhold wrote: > Thanks for all the comments. I've posted an update version here: > > http://cr.openjdk.java.net/~mr/jigsaw/notes/module-file-format > > - Mark > From Roger.Riggs at Sun.COM Tue Jan 19 14:54:12 2010 From: Roger.Riggs at Sun.COM (Roger Riggs) Date: Tue, 19 Jan 2010 17:54:12 -0500 Subject: Module-file format (DRAFT) In-Reply-To: <20100119185030.F27D2293@eggemoggin.niobe.net> References: <20100119185030.F27D2293@eggemoggin.niobe.net> Message-ID: <4B563814.7070205@Sun.COM> Hi Mark, It is a known problem with existing formats that breaking a download into multiple transfers introduces additional latency while the client has to make multiple requests to the server especially when the "first" section has to be downloaded and examined before additional segments can fetched. An encoding that enables streaming would be useful. If the signatures were put into sections also they should precede the contents so that the certificates can be checked for validity before reading through the data. Roger >> Is the intention that a module can be launched/started before it is fully >> transferred/"installed" >> > No. An application that needs to (appear to) start quickly should be > structured as a small launch module with optional dependences upon its > other modules. (This is just as in JNLP today.) We'll eventually come > up with a way to label such dependences so that they'll be downloaded > and installed right after the launch module. > > - Mark > From Dalibor.Topic at Sun.COM Tue Jan 19 18:02:05 2010 From: Dalibor.Topic at Sun.COM (Dalibor Topic) Date: Wed, 20 Jan 2010 03:02:05 +0100 Subject: Module-file format (DRAFT) In-Reply-To: <4B563763.5080609@Sun.COM> References: <20100119185437.5D72A293@eggemoggin.niobe.net> <4B563763.5080609@Sun.COM> Message-ID: <4B56641D.1050300@sun.com> Roger Riggs wrote: > Hi Mark, > > But perhaps I don't understand what goes into a "CLASSES" section. Is > it assumed > to be a sequence of classfiles without names? The classes section contains only one entity: pack.gz of the module's classes without the module-info.class. > It would be useful if the HashSection was allowed to be in each section > as well as > the entire file and was required to be after the ModuleFileHeader or > SectionHeader. > Then an implementation could verify that the module or section had the > expected > data before reading it. The would be useful when verifying signatures. That could be useful for sections that are used as input for further processing, rather then just dumped (un)compressed to disk. In the current draft, that would mean module-info.class & classes sections. cheers, dalibor topic -- ******************************************************************* Dalibor Topic Tel: (+49 40) 23 646 738 Java F/OSS Ambassador AIM: robiladonaim Sun Microsystems GmbH Mobile: (+49 177) 2664 192 Nagelsweg 55 http://openjdk.java.net D-20097 Hamburg mailto:Dalibor.Topic at sun.com Sitz der Gesellschaft: Sonnenallee 1, D-85551 Kirchheim-Heimstetten Amtsgericht M?nchen: HRB 161028 Gesch?ftsf?hrer: Thomas Schr?der, Wolfgang Engels, Wolf Frenkel Vorsitzender des Aufsichtsrates: Martin H?ring From mr at sun.com Tue Jan 19 20:54:11 2010 From: mr at sun.com (Mark Reinhold) Date: Tue, 19 Jan 2010 20:54:11 -0800 Subject: Module-file format (DRAFT) In-Reply-To: roger.riggs@sun.com; Tue, 19 Jan 2010 17:54:12 EST; <4B563814.7070205@Sun.COM> Message-ID: <20100120045411.B9F1F409@eggemoggin.niobe.net> > Date: Tue, 19 Jan 2010 17:54:12 -0500 > From: roger.riggs at sun.com > It is a known problem with existing formats that breaking a download > into multiple transfers introduces additional latency while the client > has to make multiple requests to the server especially when the "first" > section has to be downloaded and examined before additional segments > can fetched. An encoding that enables streaming would be useful. That's one of the primary goals of this format. It's not necessary to read the first section (or two) before reading the rest -- you can just read the entire thing in one pass. > If the signatures were put into sections also they should precede the > contents so that the certificates can be checked for validity before > reading through the data. Good point. Or perhaps they should all just be at the front of the file, right after the module-info section. - Mark From mr at sun.com Tue Jan 19 20:56:45 2010 From: mr at sun.com (Mark Reinhold) Date: Tue, 19 Jan 2010 20:56:45 -0800 Subject: Module-file format (DRAFT) In-Reply-To: dalibor.topic@sun.com; Tue, 19 Jan 2010 21:43:35 +0100; <4B561977.8010100@sun.com> Message-ID: <20100120045645.E6AD3409@eggemoggin.niobe.net> > Date: Tue, 19 Jan 2010 21:43:35 +0100 > From: dalibor.topic at sun.com > Given that we know the rough size of the total uncompressed files > from the module file header, do we need to spend a u4 for each > file's uncompressed size? Hmm, maybe not, but let's leave it in for now. - Mark From mr at sun.com Tue Jan 19 21:07:06 2010 From: mr at sun.com (Mark Reinhold) Date: Tue, 19 Jan 2010 21:07:06 -0800 Subject: Module-file format (DRAFT) In-Reply-To: roger.riggs@sun.com; Tue, 19 Jan 2010 17:51:15 EST; <4B563763.5080609@Sun.COM> Message-ID: <20100120050706.9B157409@eggemoggin.niobe.net> > Date: Tue, 19 Jan 2010 17:51:15 -0500 > From: roger.riggs at sun.com > Do I read this correctly that compression is only file-by-file > (SectionFileHeader)? Frequently compression algorithms work better > with more content to compress. But perhaps I don't understand what > goes into a "CLASSES" section. Is it assumed to be a sequence of > classfiles without names? As Dalibor said, it's a pack.gz stream of the module's classes. > A description of the allowed ordering or nesting would be useful (BNF). I'll consider that (though it's pretty trivial). > It would be useful if the HashSection was allowed to be in each section > as well as the entire file and was required to be after the > ModuleFileHeader or SectionHeader. Then an implementation could verify > that the module or section had the expected data before reading it. You really mean, "had the expected data before using that data", right? I agree, especially for the module-info section since that will often be read and interpreted independently of the rest of the stream. > The would be useful when verifying signatures. Right. > There is no benefit to putting the hash at the end of the file because > the format requires all of the compression to be done before any of the > sections can be written. Not sure I see your point. The end of the stream is the most convenient location for both writers and readers, though obviously the overall hash could be placed earlier in the stream. - Mark From mr at sun.com Tue Jan 19 21:57:27 2010 From: mr at sun.com (Mark Reinhold) Date: Tue, 19 Jan 2010 21:57:27 -0800 Subject: Module-file format (DRAFT) In-Reply-To: reinier@zwitserloot.com; Tue, 19 Jan 2010 21:09:32 +0100; <560fb5ed1001191209i74ed9499oe96bb8d30425eaf5@mail.gmail.com> Message-ID: <20100120055727.187DB409@eggemoggin.niobe.net> > Date: Tue, 19 Jan 2010 21:09:32 +0100 > From: Reinier Zwitserloot Thanks for your thorough comments -- replies below. > * Should SectionSize/SectionFileHeader.usize/csize also join > ModuleFileHeader.u/csize and become u8 instead of the current u4? It's difficult to imagine a module-file section being larger than 4GB. I upgraded the ModuleFileHeader size fields in order to accomodate the theoretical maximum, not because I think that anyone will actually go and create multi-gigabyte module files in practice. > * Should SectionHeader include a new u2 'version' field? analysis for this > issue: > > If an individual section type gets an update, then either the entire module > file format needs a version rev, which means all dependent tools immediately > stop working with it until they also upgrade their code. This is a bad thing > if the section's type identifies it as something that is irrelevant for a > certain module file processor. Alternatively, the section can rev itself by > picking a new FileConstants.ModuleFile.SectionType, but this will surely > lead to haphazard type ids or a risk of running out of type ids if each id > claims lets say 100 slots for potential future versioning. Even if this is > not a problem, a dependent tool that is looking for, say, an "Author > Information" section will get confused and fail with an error 'module file > is missing author information', even though the actual problem is that the > author information block is present as a version too new for the tool to > understand. If the version of a section was separate, then the tool could > correctly report that the relevant section is stored in a version that's too > new for the tool to understand. I'd like to see more use cases before adding this level of complexity. A better place for the data in an "author information" section is likely to be the module-info file itself, in the form of annotations. > * What's the point of having multiple different hash algorithms? I presume > the hash is simply a mechanism to assert that the file has not been tampered > with or corrupted during transit, and possibly as a quick way to identify a > given module file as having remained unchanged, and that it isn't intended > to fulfill the rule of a signature like signed jars (if that does get added > to the module file format, I'd presume this will occur via a new section, > and not via the hash section). Most digital-signature information would go into a new section, assuming we store it in the file. Signatures could, however, leverage the hashes already specified. > analysis for this issue: > > Every so often a hash algorithm gets cryptographically compromised. ... > ... There's a fix for > that, though: Rev the version of the module file format itself, and dictate > a new hash algorithm in the new version. > > The hash is something everyone basically needs to adhere to; if it is to > have any significant security impact, _ALL_ tools that read module files > should, with strong preference, check it and refuse to process module files > with a bad hash. However, if there's a smattering of different hash > algorithms available, this is going to make writing these tools far more > difficult. ... > > Seems simpler to me to just dictate hash algorithm with the understanding > that if it is ever compromised, the module file format itself will rev up a > version to fix it. It's already the case that the module-file format version must be bumped if a new HashType is defined, since existing readers won't understand the new type. In other words, the HashType enum in the FileConstants class already constrains the supported hash algorithms. The benefit of allowing multiple hash algorithms is that in some applications (e.g., embedded devices) a smaller hash might be preferred, due to computational constraints, and also acceptable if access to the device is suitably constrained. I don't think that supporting a few different hash algorithms is all that problematic, assuming that any such algorithm is part of the standard JRE. > * Without knowing every possible section type, how can a tool know if a > certain section is based around SectionFileHeaders, or SectionSize? The FileConstants.ModuleFile.SectionType.hasFiles() method answers this question, though it may be better for the format itself to record it. I'll consider that. > It > cannot check the type of the next chunk - if it is > indeed FileConstants.ModuleFile.SectionType.FILE it's most likely > SectionFileHeader based, but it could just be a coincidence and part of the > SectionSize.csize field. That could be, but invoking the hasFiles() method on the section's type field will tell you which interpretation to take. > What happens if a SectionFileHeader based section > has 0 files in it? As it stands, HashSection's type numbers cannot conflict > with FileConstants.ModuleFile.SectionType.FILE lest tools are forced to keep > a running count of the file's csize to realize it must have reached the end, > which sounds rather cumbersome. No, a HashSection is preceded by its own SectionHeader and SectionSize structures. (Hmm, HashSection should really be named HashContent; I'll fix that.) > This can be fixed relatively cheaply: Reserve 1 bit in the > SectionHeader.type field as indicating whether the section is single-entity > or FileHeaders based, then let a FileHeaders-based section be followed by a > u4: number of files that follow. A tool that neither knows nor cares about a > FileHeaders based section can now easily skip it by looping through each > file, reading the csize and pathLength from it, and skipping the appropriate > number of bytes. Alternatively, move the SectionSize block into the > SectionHeader block (in other words, make it mandatory even for files-based > sections). Moving the section size up into the overall section header does make sense for the purpose of skipping whole sections; I'll do that. - Mark From Dalibor.Topic at Sun.COM Wed Jan 20 02:29:09 2010 From: Dalibor.Topic at Sun.COM (Dalibor Topic) Date: Wed, 20 Jan 2010 11:29:09 +0100 Subject: Module-file format (DRAFT) In-Reply-To: <20100119185437.5D72A293@eggemoggin.niobe.net> References: <20100119185437.5D72A293@eggemoggin.niobe.net> Message-ID: <4B56DAF5.8090803@sun.com> Mark Reinhold wrote: > Thanks for all the comments. I've posted an update version here: > > http://cr.openjdk.java.net/~mr/jigsaw/notes/module-file-format It'd be nice to have the hash type in the module file header rather then at the end of file - that way the reader could wrap the input stream in a digest stream of appropriate type and compute the digest trivially at the end, without having to store the contents of the stream for later hashing. cheers, dalibor topic -- ******************************************************************* Dalibor Topic Tel: (+49 40) 23 646 738 Java F/OSS Ambassador AIM: robiladonaim Sun Microsystems GmbH Mobile: (+49 177) 2664 192 Nagelsweg 55 http://openjdk.java.net D-20097 Hamburg mailto:Dalibor.Topic at sun.com Sitz der Gesellschaft: Sonnenallee 1, D-85551 Kirchheim-Heimstetten Amtsgericht M?nchen: HRB 161028 Gesch?ftsf?hrer: Thomas Schr?der, Wolfgang Engels, Wolf Frenkel Vorsitzender des Aufsichtsrates: Martin H?ring From mr at sun.com Wed Jan 20 08:36:18 2010 From: mr at sun.com (Mark Reinhold) Date: Wed, 20 Jan 2010 08:36:18 -0800 Subject: Module-file format (DRAFT) In-Reply-To: dalibor.topic@sun.com; Wed, 20 Jan 2010 11:29:09 +0100; <4B56DAF5.8090803@sun.com> Message-ID: <20100120163618.7E4C1622@eggemoggin.niobe.net> > Date: Wed, 20 Jan 2010 11:29:09 +0100 > From: dalibor.topic at sun.com > It'd be nice to have the hash type in the module file header rather > then at the end of file - that way the reader could wrap the input > stream in a digest stream of appropriate type and compute the digest > trivially at the end, without having to store the contents of the > stream for later hashing. Duh, yes, of course -- I'll fix that. - Mark From Dalibor.Topic at Sun.COM Wed Jan 20 09:08:21 2010 From: Dalibor.Topic at Sun.COM (Dalibor Topic) Date: Wed, 20 Jan 2010 18:08:21 +0100 Subject: Module-file format (DRAFT) In-Reply-To: <20100120163618.7E4C1622@eggemoggin.niobe.net> References: <20100120163618.7E4C1622@eggemoggin.niobe.net> Message-ID: <4B573885.9030409@sun.com> Mark Reinhold wrote: >> Date: Wed, 20 Jan 2010 11:29:09 +0100 >> From: dalibor.topic at sun.com > >> It'd be nice to have the hash type in the module file header rather >> then at the end of file - that way the reader could wrap the input >> stream in a digest stream of appropriate type and compute the digest >> trivially at the end, without having to store the contents of the >> stream for later hashing. > > Duh, yes, of course -- I'll fix that. > Thanks! cheers, dalibor topic -- ******************************************************************* Dalibor Topic Tel: (+49 40) 23 646 738 Java F/OSS Ambassador AIM: robiladonaim Sun Microsystems GmbH Mobile: (+49 177) 2664 192 Nagelsweg 55 http://openjdk.java.net D-20097 Hamburg mailto:Dalibor.Topic at sun.com Sitz der Gesellschaft: Sonnenallee 1, D-85551 Kirchheim-Heimstetten Amtsgericht M?nchen: HRB 161028 Gesch?ftsf?hrer: Thomas Schr?der, Wolfgang Engels, Wolf Frenkel Vorsitzender des Aufsichtsrates: Martin H?ring From mr at sun.com Wed Jan 20 09:35:08 2010 From: mr at sun.com (Mark Reinhold) Date: Wed, 20 Jan 2010 09:35:08 -0800 Subject: Module-file format (DRAFT) In-Reply-To: mr@sun.com; Fri, 15 Jan 2010 14:11:06 PST; <20100115221106.25DC91C7F@eggemoggin.niobe.net> Message-ID: <20100120173508.7AB1C622@eggemoggin.niobe.net> Latest update (v4): http://cr.openjdk.java.net/~mr/jigsaw/notes/module-file-format There's now a change summary at the end of the document. - Mark From mr at sun.com Wed Jan 20 09:41:18 2010 From: mr at sun.com (mr at sun.com) Date: Wed, 20 Jan 2010 17:41:18 +0000 Subject: hg: jigsaw/jigsaw/jdk: Module-file constants update Message-ID: <20100120174144.5381941EF3@hg.openjdk.java.net> Changeset: 96cb90d73a92 Author: mr Date: 2010-01-20 09:40 -0800 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/96cb90d73a92 Module-file constants update ! src/share/classes/org/openjdk/jigsaw/FileConstants.java From reinier at zwitserloot.com Wed Jan 20 10:06:21 2010 From: reinier at zwitserloot.com (Reinier Zwitserloot) Date: Wed, 20 Jan 2010 19:06:21 +0100 Subject: Module-file format (DRAFT) Message-ID: <560fb5ed1001201006l405178b4l3ec7c1839acc740f@mail.gmail.com> Looks good, Mark. One more comment: > Moving the section size up into the overall section header does make > sense for the purpose of skipping whole sections; I'll do that. Right, but if file sections get their own sizes, then it doesn't make much sense anymore to let the entire file size be stored in u8, but have section sizes be constrained by u4; I imagine that if ever a module file does need to be made that is >4gb in size, then there's one chunk in the module containing, lets say, resources which includes a giant video resource, that is itself also >4gb in size. Limiting to 4gb now seems like a pretty big mistake; you're already seeing video resources that are over 4gb (a DVD dump, for example). Lets say someone wants to ship a standalone video player app that includes video. In a few years, any 2 hour stream of HD video data will definitely be larger than 4gb in size. It would seem unfortunate if the author of such a tool is enforced to either ship the video data separately or somehow manage to encode it into multiple chunks. Second point to this: does the 'u' stand for unsigned? u8 in java isn't exactly convenient. Shaving the top bit off isn't going to have any significant impact on this format's future-proof-ness. --Reinier Zwitserloot From reinier at zwitserloot.com Wed Jan 20 10:13:45 2010 From: reinier at zwitserloot.com (Reinier Zwitserloot) Date: Wed, 20 Jan 2010 19:13:45 +0100 Subject: Module-file format: Hash type at beginning, result at end? Message-ID: <560fb5ed1001201013s63e80f1dq4f2deb9784a10d83@mail.gmail.com> By moving the hash itself to the top of the module file, a tool that creates module files either needs to be 2-pass, or needs to store the entire content to be written in memory. It seems a bit awkward, but storing the _type_ of hash at the top and the _hash_ itself at the bottom is the most convenient for both writers and readers. Same applies to per-section hashes. --Roel Spilker and Reinier Zwitserloot From Roger.Riggs at Sun.COM Wed Jan 20 13:14:48 2010 From: Roger.Riggs at Sun.COM (Roger Riggs) Date: Wed, 20 Jan 2010 16:14:48 -0500 Subject: Module-file format (DRAFT) In-Reply-To: <20100120173508.7AB1C622@eggemoggin.niobe.net> References: <20100120173508.7AB1C622@eggemoggin.niobe.net> Message-ID: <4B577248.20802@Sun.COM> Hi Mark, Where is the content of the config section defined? Does it contain information about which OS and which hardware architecture the native code is appropriate for? Or is that in the module-info.class? Can I have a "fat" package with multiple native code implementations for different platforms? If the sections are strictly in order then having the config section earlier would be useful to setup how to handle the rest of the sections. Can you remind me why multiple sections of each type are not useful/allowed. Thanks, Roger On 1/20/10 12:35 PM, Mark Reinhold wrote: > Latest update (v4): > > http://cr.openjdk.java.net/~mr/jigsaw/notes/module-file-format > > There's now a change summary at the end of the document. > > - Mark > From cjyxiaodi1 at yahoo.com Thu Jan 21 01:57:04 2010 From: cjyxiaodi1 at yahoo.com (cjy xiaodi) Date: Thu, 21 Jan 2010 01:57:04 -0800 (PST) Subject: Training Jigsaw problem facing Message-ID: <339136.47991.qm@web38306.mail.mud.yahoo.com> Hi, Regarding the jigsaw tutorial, I type the following command: train_jigsaw.pl -l my_dir_list.txt -e my_evidence_list.txt -f seq -t It gives the following error message: write: ../42492/42492.my_evidence_list.txt [jigsaw -f ../42492/42492.seq -e ../42492/42492.my_evidence_list.txt -t ./42492.ev_vec -d . -q 50] unable to open [./42492.ev_vec.start] Does anybody facing the same problem? I'm believe I follow all the instruction to install JIGSAW. Can I know what is the reason causing this happen? How to solve this error message? I'm using ia64 linux. Thanks a lot for any advice and suggestion. best regards Patrick From Alex.Buckley at Sun.COM Thu Jan 21 11:09:17 2010 From: Alex.Buckley at Sun.COM (Alex Buckley) Date: Thu, 21 Jan 2010 11:09:17 -0800 Subject: Training Jigsaw problem facing In-Reply-To: <339136.47991.qm@web38306.mail.mud.yahoo.com> References: <339136.47991.qm@web38306.mail.mud.yahoo.com> Message-ID: <4B58A65D.8070805@sun.com> You seem to have confused JIGSAW from http://www.cbcb.umd.edu/software/jigsaw/README_tutorial.shtml with the Jigsaw module system developed by Sun for JDK7. This mailing list is for the latter. Alex cjy xiaodi wrote: > Hi, > > Regarding the jigsaw tutorial, I type the following command: > train_jigsaw.pl -l my_dir_list.txt -e my_evidence_list.txt -f seq -t > > It gives the following error message: > write: ../42492/42492.my_evidence_list.txt > [jigsaw -f ../42492/42492.seq -e ../42492/42492.my_evidence_list.txt -t ./42492.ev_vec -d . -q 50] > unable to open [./42492.ev_vec.start] > > Does anybody facing the same problem? > I'm believe I follow all the instruction to install JIGSAW. Can I know what is the reason causing this happen? How to solve this error message? > I'm using ia64 linux. Thanks a lot for any advice and suggestion. > > best regards > Patrick > > > > From mr at sun.com Fri Jan 22 16:05:10 2010 From: mr at sun.com (Mark Reinhold) Date: Fri, 22 Jan 2010 16:05:10 -0800 Subject: Module-file format (DRAFT) In-Reply-To: roger.riggs@sun.com; Wed, 20 Jan 2010 16:14:48 EST; <4B577248.20802@Sun.COM> Message-ID: <20100123000510.BE53466F@eggemoggin.niobe.net> > Date: Wed, 20 Jan 2010 16:14:48 -0500 > From: roger.riggs at sun.com > Where is the content of the config section defined? It's just a collection of named files. It's meant to contain things like the properties files currently found in $JAVA_HOME/lib, nothing more. > Does it contain information about which OS and which hardware architecture > the native code is appropriate for? No. > Or is that in the module-info.class? No. > Can I have a "fat" package with multiple native code implementations for > different platforms? No, and I'm skeptical that "fat" packages are the way to go. When modules are published on a server we can use simple URL conventions to distinguish "pure" modules from those containing native code for a specific target os/arch combination. In the long run we should probably record that information in the format itself, as a sanity check. > If the sections are strictly in order then having the config section > earlier would be useful to setup how to handle the rest of the > sections. (N/A, see above) > Can you remind me why multiple sections of each type are not useful/allowed. Simplicity. - Mark From Roger.Riggs at Sun.COM Sat Jan 23 04:07:45 2010 From: Roger.Riggs at Sun.COM (Roger Riggs) Date: Sat, 23 Jan 2010 07:07:45 -0500 Subject: Module-file format (DRAFT) In-Reply-To: <20100123000510.BE53466F@eggemoggin.niobe.net> References: <20100123000510.BE53466F@eggemoggin.niobe.net> Message-ID: <4B5AE691.8070308@Sun.COM> Hi Mark, Current development environments and runtimes are expected to support automated discovery, distribution and verification. This is made possible when the artifacts completely describe themselves and their dependencies. This information is used to establish and verify assumptions and invariants needed to support robust operation. It is no longer acceptable to require a user to establish or fix a valid configuration. The kinds of information are needed are: * End user identification of the module, icon, languages supported, etc. * Dependencies of native code - instruction set/versions, OS name/version, specific drivers or other native code packages. * Hardware dependencies - is there a keyboard, screen size, type of networking, etc. * Ownership and rights - vendor, license term either for user or automated verification * Etc. In the JAR format, the manifest was available as a full extensible place to record additional information. Its main drawback was having to search the JAR to find it and the requirement to encode as text. This meta information is needed for communication between the IDE, distribution server, and runtime at least. Jigsaw itself does not need to define the meta data semantics. Incorporating an extensible placeholder into the distribution format can enable a single artifact to keep the information consistent and available. Roger On 1/22/10 7:05 PM, Mark Reinhold wrote: >> Date: Wed, 20 Jan 2010 16:14:48 -0500 >> From: roger.riggs at sun.com >> > >> Where is the content of the config section defined? >> > It's just a collection of named files. It's meant to contain things like > the properties files currently found in $JAVA_HOME/lib, nothing more. > > >> Does it contain information about which OS and which hardware architecture >> the native code is appropriate for? >> > No. > > >> Or is that in the module-info.class? >> > No. > > >> Can I have a "fat" package with multiple native code implementations for >> different platforms? >> > No, and I'm skeptical that "fat" packages are the way to go. > > When modules are published on a server we can use simple URL conventions > to distinguish "pure" modules from those containing native code for a > specific target os/arch combination. In the long run we should probably > record that information in the format itself, as a sanity check. > > >> If the sections are strictly in order then having the config section >> earlier would be useful to setup how to handle the rest of the >> sections. >> > (N/A, see above) > > >> Can you remind me why multiple sections of each type are not useful/allowed. >> > Simplicity. > > - Mark > From Mandy.Chung at Sun.COM Sat Jan 23 12:34:16 2010 From: Mandy.Chung at Sun.COM (Mandy Chung) Date: Sat, 23 Jan 2010 12:34:16 -0800 Subject: Fail to look for a resource bundle that is in jdk.boot module Message-ID: <4B5B5D48.7000007@sun.com> Hi Mark, I have made some good progress in creating the jdk base image and also the entire jdk image. All jdk modules are installed successfully on jigsaw on all platforms. Running org.openjdk.jigsaw.Hi works! > jdk-module-image/bin/java -m jdk.base The JDK boot module is brought to you by the letter 'J' and the number 7. > jre-base-image/bin/java -m jdk.base The JDK boot module is brought to you by the letter 'J' and the number 7. I now run into two different issues. In the module mode, I run into a missing resource exception where it fails to find a resource bundle that exists. sun.launcher.resources.launcher is a compiled class installed in the jdk.boot module. > ls jdk-module-image/lib/modules/jdk.boot/7-ea/classes/sun/launcher/resources/launcher.class > jdk-module-image/bin/java -m jdk.javac Exception in thread "main" java.lang.ExceptionInInitializerError at sun.launcher.LauncherHelper.getLocalizedMessage(LauncherHelper.java:71) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:211) Caused by: java.util.MissingResourceException: Can't find bundle for base name sun.launcher.resources.launcher, locale en at java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:1539) at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1278) at java.util.ResourceBundle.getBundle(ResourceBundle.java:733) at sun.launcher.LauncherHelper$ResourceBundleHolder.(LauncherHelper.java:59) ... 2 more Is there anything look suspicious to you? ResourceBundle uses its own RBClassLoader to find resources. Is it considered as legacy mode? Even it's legacy mode, it should be able to find it since it's in the jdk.boot module. How does the module loader find a resource file different than the legacy class loader? I just suspect this is class loader issue. The module-info for jdk.javac module is: > jdk.javac/module-info.java module jdk.javac @ 7-ea { requires local public jdk.langtools @ 7-ea; requires local public jdk.logging @ 7-ea; permits jdk, jdk.tools; class org.openjdk.jigsaw.Hi; // ## Testing } In the legacy mode, it loads sun.launcher.LauncherHelper and finds the resource as the jdk.boot classes/resources are added in the bootclasspath. But I got a NoClassDefFoundError instead that the app classloader can't find com.sun.tools.javac.Main from the jdk.javac module. I know that the hotspot bootclasspath is just temporary workaround for the boot classes but not for all jdk modules. But looks like we need a solution soon. > jdk-module-image/bin/javac -version Error: Could not find main class com.sun.tools.javac.Main Exception in thread "main" java.lang.NoClassDefFoundError: com.sun.tools.javac.Main at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:213) Caused by: java.lang.ClassNotFoundException: com.sun.tools.javac.Main at java.net.URLClassLoader$1.run(URLClassLoader.java:299) at java.net.URLClassLoader$1.run(URLClassLoader.java:288) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:287) at java.lang.ClassLoader.loadClass(ClassLoader.java:422) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:325) at java.lang.ClassLoader.loadClass(ClassLoader.java:355) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:209) > ls jdk-module-image/lib/modules/jdk.javac/7-ea/classes/com/sun/tools/javac/Main.class I'll continue to dig into this further. Any thoughts or hints to diagnose these issues would be appreciated. Thanks Mandy From Mandy.Chung at Sun.COM Sat Jan 23 14:10:15 2010 From: Mandy.Chung at Sun.COM (Mandy Chung) Date: Sat, 23 Jan 2010 14:10:15 -0800 Subject: Fail to look for a resource bundle that is in jdk.boot module In-Reply-To: <4B5B5D48.7000007@sun.com> References: <4B5B5D48.7000007@sun.com> Message-ID: <4B5B73C7.80908@sun.com> I think I find the cause of the issue (when learning the jmod show option) The context of jdk.javac module is: > jdk-module-image/bin/jmod show jdk.javac configuration roots = [jdk.javac at 7-ea] context +jdk.javac+jdk.langtools+jdk.logging [jdk.javac at 7-ea, jdk.logging at 7-ea, jdk.langtools at 7-ea] I guess it's considered as boot context only when it requires jdk.boot. I will workaround it and see if it resolves the problem. Mandy Mandy Chung wrote: > Hi Mark, > > I have made some good progress in creating the jdk base > image and also the entire jdk image. All jdk modules > are installed successfully on jigsaw on all platforms. > Running org.openjdk.jigsaw.Hi works! > >> jdk-module-image/bin/java -m jdk.base > The JDK boot module is brought to you by the letter 'J' and the number 7. > >> jre-base-image/bin/java -m jdk.base > The JDK boot module is brought to you by the letter 'J' and the number 7. > > I now run into two different issues. > In the module mode, I run into a missing resource exception where it > fails to find a resource bundle that exists. > sun.launcher.resources.launcher is a compiled class installed in the > jdk.boot module. > >> ls >> jdk-module-image/lib/modules/jdk.boot/7-ea/classes/sun/launcher/resources/launcher.class >> > >> jdk-module-image/bin/java -m jdk.javac > Exception in thread "main" java.lang.ExceptionInInitializerError > at > sun.launcher.LauncherHelper.getLocalizedMessage(LauncherHelper.java:71) > at > sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:211) > Caused by: java.util.MissingResourceException: Can't find bundle for > base name sun.launcher.resources.launcher, locale en > at > java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:1539) > > at > java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1278) > at java.util.ResourceBundle.getBundle(ResourceBundle.java:733) > at > sun.launcher.LauncherHelper$ResourceBundleHolder.(LauncherHelper.java:59) > > ... 2 more > > Is there anything look suspicious to you? ResourceBundle uses > its own RBClassLoader to find resources. Is it considered > as legacy mode? Even it's legacy mode, it should be able to > find it since it's in the jdk.boot module. > > How does the module loader find a resource file different > than the legacy class loader? I just suspect this is class > loader issue. > > The module-info for jdk.javac module is: > >> jdk.javac/module-info.java > module jdk.javac @ 7-ea { > requires local public jdk.langtools @ 7-ea; > requires local public jdk.logging @ 7-ea; > permits jdk, jdk.tools; > class org.openjdk.jigsaw.Hi; // ## Testing > } > > In the legacy mode, it loads sun.launcher.LauncherHelper > and finds the resource as the jdk.boot classes/resources > are added in the bootclasspath. But I got a NoClassDefFoundError > instead that the app classloader can't find com.sun.tools.javac.Main > from the jdk.javac module. I know that the hotspot bootclasspath > is just temporary workaround for the boot classes but not for all jdk > modules. But looks like we need a solution > soon. > >> jdk-module-image/bin/javac -version > > Error: Could not find main class com.sun.tools.javac.Main > Exception in thread "main" java.lang.NoClassDefFoundError: > com.sun.tools.javac.Main > at > sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:213) > Caused by: java.lang.ClassNotFoundException: com.sun.tools.javac.Main > at java.net.URLClassLoader$1.run(URLClassLoader.java:299) > at java.net.URLClassLoader$1.run(URLClassLoader.java:288) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:287) > at java.lang.ClassLoader.loadClass(ClassLoader.java:422) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:325) > at java.lang.ClassLoader.loadClass(ClassLoader.java:355) > at > sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:209) > >> ls >> jdk-module-image/lib/modules/jdk.javac/7-ea/classes/com/sun/tools/javac/Main.class >> > > I'll continue to dig into this further. Any thoughts > or hints to diagnose these issues would be appreciated. > > Thanks > Mandy > From mr at sun.com Wed Jan 27 14:51:32 2010 From: mr at sun.com (Mark Reinhold) Date: Wed, 27 Jan 2010 14:51:32 -0800 Subject: Module-file format (DRAFT) In-Reply-To: roger.riggs@sun.com; Sat, 23 Jan 2010 07:07:45 EST; <4B5AE691.8070308@Sun.COM> Message-ID: <20100127225132.C99B810B4@eggemoggin.niobe.net> > Date: Sat, 23 Jan 2010 07:07:45 -0500 > From: roger.riggs at sun.com > Current development environments and runtimes are expected to support > automated discovery, distribution and verification. This is made > possible when the artifacts completely describe themselves and their > dependencies. This information is used to establish and verify > assumptions and invariants needed to support robust operation. It is > no longer acceptable to require a user to establish or fix a valid > configuration. Completely agree. > The kinds of information are needed are: > > * End user identification of the module, icon, languages supported, etc. > * Dependencies of native code - instruction set/versions, > OS name/version, specific drivers or other native code packages. > * Hardware dependencies - is there a keyboard, screen size, type of > networking, etc. > * Ownership and rights - vendor, license term either for user or > automated verification > * Etc. > > In the JAR format, the manifest was available as a full extensible > place to record additional information. Its main drawback was having > to search the JAR to find it and the requirement to encode as text. Yep. > This meta information is needed for communication between the IDE, > distribution server, and runtime at least. Jigsaw itself does not need > to define the meta data semantics. Incorporating an extensible > placeholder into the distribution format can enable a single artifact > to keep the information consistent and available. Much (though not all) of the above information can be expressed as annotations in a module's module-info file. I suspect the rest can be provided as part of the packaging process, but we'd have to work through the use cases. - Mark From Mandy.Chung at Sun.COM Wed Jan 27 17:36:15 2010 From: Mandy.Chung at Sun.COM (Mandy Chung) Date: Wed, 27 Jan 2010 17:36:15 -0800 Subject: jdk modules running on jigsaw! Message-ID: <4B60EA0F.7090801@Sun.com> Status update: I now have the jdk modules installed and running. 9 of the 10 jigsaw jtreg tests passed on linux. I'm lookiing into the library.sh failure. I need to fix the jigsaw jtreg tests to run on other platforms. Also, I'm running the jdk jtreg tests (in legacy mode) with the full jdk module image and so far so good. Here is the summary of what I have done. 1. jdk.boot at 7-ea is hardcoded in the VM bootclasspath as the current workaround for bootstrapping. I added a jdk.boot module for now. 2. modules.config and modules.group define how the "private" jdk modules are defined. The resulting modules are private and they only permit the public jdk modules to use. The public jdk modules don't have any permits rule. When a module has a local dependency on another module, the requesting module would need to be explicitly permitted in its dependence which would become a private module since any other module not specified in its permits rule can't depend on it. I currently name the private jdk modules to start with "sun." prefix whereas the names of the public jdk modules start with the "jdk." prefix. E.g. sun.logging is the private module and jdk.logging is the public module. 2. The "modules" make target will create 4 new images: jdk-base-image, jre-base-image, jdk-module-image and jre-module-image I currently put them under the directory. jre-base-image contains the base module. jdk-base-image contains the base module and the javac modules + its dependencies). jdk-module-image is the full jdk image with all jdk modules installed whereas jre-module-image is the full jre image minus the tools installed. 3. The jdk tools except javac are converted to run in the module mode. I added a temporary solution in java.c to determine if the java home is a module image and set it to run module mode. When javac running in module mode, it doesn't find the jdk modules other than the ones in its context and so compilation will fail if the source file imports classes in other modules. I believe Jon already has the change ready in javac to use the jigsaw module resolver. Just waiting for my change to go in. 4. Legacy mode - I also modified the launcher to set the bootclasspath to the jigsaw module library (i.e. lib/modules/*/7-ea/classes + lib/modules/*/7-ea/resources) if the java home is a module image. 5. One open question is what the default platform module should be. jdk at 7-ea is the current default platform module and I assume it's full jdk. The jdk-base-image only has jdk.base module installed. Unless the module-info of an application requires jdk.base at 7-ea, the default platform module will be added to its dependence. It means that an application whose module-info compiled with the default platform module can't be deployed on the jdk-base-image since jdk at 7-ea is not installed even if it doesn't depend on any other jdk module. I think making the default to the jdk.base module would help the developers be cautious of what jdk modules their module should require and think through this kind of deployment issue during development. I have a whole day meeting tomorrow. I hope that I can clean up the changes and get the webrev ready by next Monday. Mandy