From jesse.glick at oracle.com Tue Jun 5 16:56:42 2012 From: jesse.glick at oracle.com (Jesse Glick) Date: Tue, 05 Jun 2012 19:56:42 -0400 Subject: Separate compilation with modular JARs? Message-ID: <4FCE9CBA.3040303@oracle.com> Unless I am missing something, I am finding it impossible to make Jigsaw's javac run sanely with modular JARs or unpacked class trees, i.e. without having to use jmod or multi-module layout. Basically I just want to run javac exactly like you would with JDK 7 for a project with multiple physically separated components and acyclic dependencies: with an explicit classpath for each component, and with module-info.java compiled to module-info.class with error checking, while enforcing module-based accessibility (i.e. honoring 'exports' and similar). But it does not work: mkdir -p modone/src/pkgone modone/bin modtwo/src/pkgtwo modtwo/bin echo 'module modone at 1 {exports pkgone;}' > modone/src/module-info.java echo 'package pkgone; public class C1 {}' > modone/src/pkgone/C1.java echo 'module modtwo at 1 {requires modone at 1; class pkgtwo.C2;}' > modtwo/src/module-info.java echo 'package pkgtwo; public class C2 extends pkgone.C1 {public static void main(String... args) {System.out.println(new C2());}}' > modtwo/src/pkgtwo/C2.java javac -d modone/bin modone/src/module-info.java modone/src/pkgone/C1.java javac -classpath modone/bin -d modtwo/bin modtwo/src/module-info.java modtwo/src/pkgtwo/C2.java fails with: error: Cannot resolve module dependencies using Jigsaw module resolver modtwo@=1: Cannot resolve 1 error with no further explanation. Same if I use -modulepath instead of -classpath. Even more worrisome, if I remove the 'requires' clause then compilation succeeds, whereas I would expect classes from modone to be considered inaccessible in this case. Also if I try to run the result: java -modulepath modone/bin:modtwo/bin -m two I get another error: Error: -classpath and -cp cannot be used with -m and -modulepath is not recognized by this command at all. Is it planned for javac and java to work nicely with unpacked modules but just not yet implemented? From paul.sandoz at oracle.com Wed Jun 6 00:25:04 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 6 Jun 2012 09:25:04 +0200 Subject: Strange errors when making changes to the configuration Message-ID: <5B9D761D-DDA1-41A5-AB3A-15F29FCE2290@oracle.com> Hi, I am experimenting with some changes to the serialized configuration related to services (see SimpleLibrary.StoredConfiguration), specifically removing the service supplier data. However, when i remove the reading/writing of the service supplier data for each context the execution of my compiled and installed sample module either fails with output like the following: java(17661,0x10ed89000) malloc: *** mmap(size=18446743966130372608) failed (error code=12) *** error: can't allocate region *** set a breakpoint in malloc_error_break to debug Error occurred during initialization of VM java/lang/NoClassDefFoundError: java/lang/Object Or just spins at 100% eating up as much heap as possible. If i modify the reading/writing of the service supplier data to explicitly writeInt(0)/readInt(), the equivalent of 0 service suppliers it works. I am building Jigsaw on a Mac, i have yet to verify if i can reproduce on another architecture. It's too early in the VM initialization process to attach a debugger. Is there something else, perhaps VM specific, that depends on the serialized configuration? I searched but cannot find such a second dependency. Any tips/hints on how to get more information from the VM? Paul. From chris.hegarty at oracle.com Wed Jun 6 00:55:12 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 06 Jun 2012 08:55:12 +0100 Subject: Strange errors when making changes to the configuration In-Reply-To: <5B9D761D-DDA1-41A5-AB3A-15F29FCE2290@oracle.com> References: <5B9D761D-DDA1-41A5-AB3A-15F29FCE2290@oracle.com> Message-ID: <4FCF0CE0.1050504@oracle.com> The jigsaw native interface is now reading the configuration directly http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/file/7b282c826118/src/share/native/common/jigsaw.c -Chris. On 06/06/2012 08:25, Paul Sandoz wrote: > Hi, > > I am experimenting with some changes to the serialized configuration related to services (see SimpleLibrary.StoredConfiguration), specifically removing the service supplier data. > > However, when i remove the reading/writing of the service supplier data for each context the execution of my compiled and installed sample module either fails with output like the following: > > java(17661,0x10ed89000) malloc: *** mmap(size=18446743966130372608) failed (error code=12) > *** error: can't allocate region > *** set a breakpoint in malloc_error_break to debug > Error occurred during initialization of VM > java/lang/NoClassDefFoundError: java/lang/Object > > Or just spins at 100% eating up as much heap as possible. > > If i modify the reading/writing of the service supplier data to explicitly writeInt(0)/readInt(), the equivalent of 0 service suppliers it works. > > I am building Jigsaw on a Mac, i have yet to verify if i can reproduce on another architecture. > > It's too early in the VM initialization process to attach a debugger. > > Is there something else, perhaps VM specific, that depends on the serialized configuration? I searched but cannot find such a second dependency. > > Any tips/hints on how to get more information from the VM? > > Paul. From paul.sandoz at oracle.com Wed Jun 6 01:05:05 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 6 Jun 2012 10:05:05 +0200 Subject: Strange errors when making changes to the configuration In-Reply-To: <4FCF0CE0.1050504@oracle.com> References: <5B9D761D-DDA1-41A5-AB3A-15F29FCE2290@oracle.com> <4FCF0CE0.1050504@oracle.com> Message-ID: <5943BACF-B567-48BB-B75D-7C3958C1A399@oracle.com> Hi Chris, Many thanks! With some help of process sampling (using dtrace) i was so close to finding that file :-) This is a particularly fragile area. Paul. 2426 Thread_6858398 + 2426 thread_start (in libsystem_c.dylib) + 13 [0x7fff96059b75] + 2426 _pthread_start (in libsystem_c.dylib) + 335 [0x7fff960568bf] + 2426 JavaMain (in java) + 308 [0x106af1eb9] + 2426 JNI_CreateJavaVM (in libjvm.dylib) + 98 [0x106f9bf76] + 2426 Threads::create_vm(JavaVMInitArgs*, bool*) (in libjvm.dylib) + 455 [0x107171f53] + 2426 init_globals() (in libjvm.dylib) + 124 [0x106f4c8bd] + 2426 universe2_init() (in libjvm.dylib) + 31 [0x107181233] + 2426 Universe::genesis(Thread*) (in libjvm.dylib) + 1125 [0x107180cbf] + 2426 SystemDictionary::initialize_preloaded_classes(Thread*) (in libjvm.dylib) + 36 [0x107145964] + 2426 SystemDictionary::initialize_wk_klasses_until(SystemDictionary::WKID, SystemDictionary::WKID&, Thread*) (in libjvm.dylib) + 72 [0x1071458de] + 2426 SystemDictionary::initialize_wk_klass(SystemDictionary::WKID, int, Thread*) (in libjvm.dylib) + 90 [0x107145864] + 2426 SystemDictionary::resolve_or_fail(Symbol*, Handle, Handle, bool, Thread*) (in libjvm.dylib) + 49 [0x10714578f] + 2426 SystemDictionary::resolve_or_null(Symbol*, Handle, Handle, Thread*) (in libjvm.dylib) + 248 [0x10714520e] + 2426 SystemDictionary::resolve_instance_class_or_null(Symbol*, Handle, Handle, Thread*) (in libjvm.dylib) + 1083 [0x107144abd] + 2426 SystemDictionary::load_instance_class(Symbol*, Handle, Thread*) (in libjvm.dylib) + 281 [0x107144f11] + 2426 SystemDictionary::load_local_instance_class(Symbol*, Handle, Thread*) (in libjvm.dylib) + 286 [0x1071456f6] + 2426 ClassLoader::load_class_from_module_library(Symbol*, Thread*) (in libjvm.dylib) + 325 [0x106e60a69] + 2426 JDK_LoadContexts (in libjava.dylib) + 473 [0x106bcf63e] + 2338 load_config (in libjava.dylib) + 919 [0x106bcf2d5] On Jun 6, 2012, at 9:55 AM, Chris Hegarty wrote: > The jigsaw native interface is now reading the configuration directly > > http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/file/7b282c826118/src/share/native/common/jigsaw.c > > -Chris. > > On 06/06/2012 08:25, Paul Sandoz wrote: >> Hi, >> >> I am experimenting with some changes to the serialized configuration related to services (see SimpleLibrary.StoredConfiguration), specifically removing the service supplier data. >> >> However, when i remove the reading/writing of the service supplier data for each context the execution of my compiled and installed sample module either fails with output like the following: >> >> java(17661,0x10ed89000) malloc: *** mmap(size=18446743966130372608) failed (error code=12) >> *** error: can't allocate region >> *** set a breakpoint in malloc_error_break to debug >> Error occurred during initialization of VM >> java/lang/NoClassDefFoundError: java/lang/Object >> >> Or just spins at 100% eating up as much heap as possible. >> >> If i modify the reading/writing of the service supplier data to explicitly writeInt(0)/readInt(), the equivalent of 0 service suppliers it works. >> >> I am building Jigsaw on a Mac, i have yet to verify if i can reproduce on another architecture. >> >> It's too early in the VM initialization process to attach a debugger. >> >> Is there something else, perhaps VM specific, that depends on the serialized configuration? I searched but cannot find such a second dependency. >> >> Any tips/hints on how to get more information from the VM? >> >> Paul. From paul.sandoz at oracle.com Wed Jun 6 01:45:14 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 6 Jun 2012 10:45:14 +0200 Subject: Separate compilation with modular JARs? In-Reply-To: <4FCE9CBA.3040303@oracle.com> References: <4FCE9CBA.3040303@oracle.com> Message-ID: <220A6336-F067-4BBD-A720-A3534B864557@oracle.com> Hi Jesse, I can reproduce the same behavior. I had to install modone into a library so that i could compile modtwo separately. Execution in exploded/unpacked mode is currently not implemented. It is planned. Alan has a patch that is currently bit rotting... i dunno if that patch also addresses the compilation case, which i think is important too. Paul. On Jun 6, 2012, at 1:56 AM, Jesse Glick wrote: > Unless I am missing something, I am finding it impossible to make Jigsaw's javac run sanely with modular JARs or unpacked class trees, i.e. without having to use jmod or multi-module layout. Basically I just want to run javac exactly like you would with JDK 7 for a project with multiple physically separated components and acyclic dependencies: with an explicit classpath for each component, and with module-info.java compiled to module-info.class with error checking, while enforcing module-based accessibility (i.e. honoring 'exports' and similar). But it does not work: > > mkdir -p modone/src/pkgone modone/bin modtwo/src/pkgtwo modtwo/bin > echo 'module modone at 1 {exports pkgone;}' > modone/src/module-info.java > echo 'package pkgone; public class C1 {}' > modone/src/pkgone/C1.java > echo 'module modtwo at 1 {requires modone at 1; class pkgtwo.C2;}' > modtwo/src/module-info.java > echo 'package pkgtwo; public class C2 extends pkgone.C1 {public static void main(String... args) {System.out.println(new C2());}}' > modtwo/src/pkgtwo/C2.java > javac -d modone/bin modone/src/module-info.java modone/src/pkgone/C1.java > javac -classpath modone/bin -d modtwo/bin modtwo/src/module-info.java modtwo/src/pkgtwo/C2.java > > fails with: > > error: Cannot resolve module dependencies using Jigsaw module resolver > modtwo@=1: Cannot resolve > 1 error > > with no further explanation. Same if I use -modulepath instead of -classpath. Even more worrisome, if I remove the 'requires' clause then compilation succeeds, whereas I would expect classes from modone to be considered inaccessible in this case. > > Also if I try to run the result: > > java -modulepath modone/bin:modtwo/bin -m two > > I get another error: > > Error: -classpath and -cp cannot be used with -m > > and -modulepath is not recognized by this command at all. > > Is it planned for javac and java to work nicely with unpacked modules but just not yet implemented? From paul.sandoz at oracle.com Wed Jun 6 01:46:35 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 6 Jun 2012 10:46:35 +0200 Subject: Strange errors when making changes to the configuration In-Reply-To: <4FCF0CE0.1050504@oracle.com> References: <5B9D761D-DDA1-41A5-AB3A-15F29FCE2290@oracle.com> <4FCF0CE0.1050504@oracle.com> Message-ID: <7A2A4D48-7255-41CA-BAC0-D22690DC06EA@oracle.com> Fixed. Muchas gracias, Paul. On Jun 6, 2012, at 9:55 AM, Chris Hegarty wrote: > The jigsaw native interface is now reading the configuration directly > > http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/file/7b282c826118/src/share/native/common/jigsaw.c > > -Chris. From Alan.Bateman at oracle.com Wed Jun 6 02:47:09 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 06 Jun 2012 10:47:09 +0100 Subject: Separate compilation with modular JARs? In-Reply-To: <4FCE9CBA.3040303@oracle.com> References: <4FCE9CBA.3040303@oracle.com> Message-ID: <4FCF271D.3080809@oracle.com> On 06/06/2012 00:56, Jesse Glick wrote: > Unless I am missing something, I am finding it impossible to make > Jigsaw's javac run sanely with modular JARs or unpacked class trees, > i.e. without having to use jmod or multi-module layout. Basically I > just want to run javac exactly like you would with JDK 7 for a project > with multiple physically separated components and acyclic > dependencies: with an explicit classpath for each component, and with > module-info.java compiled to module-info.class with error checking, > while enforcing module-based accessibility (i.e. honoring 'exports' > and similar). But it does not work: > > mkdir -p modone/src/pkgone modone/bin modtwo/src/pkgtwo modtwo/bin > echo 'module modone at 1 {exports pkgone;}' > modone/src/module-info.java > echo 'package pkgone; public class C1 {}' > modone/src/pkgone/C1.java > echo 'module modtwo at 1 {requires modone at 1; class pkgtwo.C2;}' > > modtwo/src/module-info.java > echo 'package pkgtwo; public class C2 extends pkgone.C1 {public static > void main(String... args) {System.out.println(new C2());}}' > > modtwo/src/pkgtwo/C2.java > javac -d modone/bin modone/src/module-info.java modone/src/pkgone/C1.java > javac -classpath modone/bin -d modtwo/bin modtwo/src/module-info.java > modtwo/src/pkgtwo/C2.java > > fails with: > > error: Cannot resolve module dependencies using Jigsaw module resolver > modtwo@=1: Cannot resolve > 1 error > > with no further explanation. Same if I use -modulepath instead of > -classpath. Even more worrisome, if I remove the 'requires' clause > then compilation succeeds, whereas I would expect classes from modone > to be considered inaccessible in this case. > > Also if I try to run the result: > > java -modulepath modone/bin:modtwo/bin -m two > > I get another error: > > Error: -classpath and -cp cannot be used with -m > > and -modulepath is not recognized by this command at all. > > Is it planned for javac and java to work nicely with unpacked modules > but just not yet implemented? There are a couple of things here. One is that Jon hasn't pushed the javac changes to support exports yet, I expect he will soon. Secondly, we don't have the runtime changes for module path in the runtime yet so you can't run with -mp or -modulepath yet. Thirdly, the resolver diagnostics need work so that we get good diagnostic messages at compilation or install time (or any time that the resolver is run). In the mean-time you need to get JIGSAW_TRACE to see what the resolver is going. On using classpath vs. modulepath with separate compilations then I'll defer to Jon as there may be a javac issue there. -Alan. From jesse.glick at oracle.com Wed Jun 6 06:37:44 2012 From: jesse.glick at oracle.com (Jesse Glick) Date: Wed, 06 Jun 2012 09:37:44 -0400 Subject: Separate compilation with modular JARs? In-Reply-To: <4FCF271D.3080809@oracle.com> References: <4FCE9CBA.3040303@oracle.com> <4FCF271D.3080809@oracle.com> Message-ID: <4FCF5D28.7060409@oracle.com> On 06/06/2012 05:47 AM, Alan Bateman wrote: > There are a couple of things here. OK, thanks for explanation. This is another case where it would be very handy to have a wiki page (or versioned TODO file) listing things which are believed to be implemented vs. things which are known not to work yet. By the way doc/topics/modularjar.html talks about a "module-info in the META-INF directory" but this is inconsistent with the exploded class structure - I would expect a modular JAR to be one in which module-info.class is present at top level (i.e. jar:file:/m.jar!/module-info.class). From Alan.Bateman at oracle.com Wed Jun 6 07:35:11 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 06 Jun 2012 15:35:11 +0100 Subject: Separate compilation with modular JARs? In-Reply-To: <4FCF5D28.7060409@oracle.com> References: <4FCE9CBA.3040303@oracle.com> <4FCF271D.3080809@oracle.com> <4FCF5D28.7060409@oracle.com> Message-ID: <4FCF6A9F.2030700@oracle.com> On 06/06/2012 14:37, Jesse Glick wrote: > > OK, thanks for explanation. This is another case where it would be > very handy to have a wiki page (or versioned TODO file) listing things > which are believed to be implemented vs. things which are known not to > work yet. > > By the way doc/topics/modularjar.html talks about a "module-info in > the META-INF directory" but this is inconsistent with the exploded > class structure - I would expect a modular JAR to be one in which > module-info.class is present at top level (i.e. > jar:file:/m.jar!/module-info.class). I fixed up the modularjar.html page. We don't maintain a TODO list but it is something that we should do. -Alan From mandy.chung at oracle.com Wed Jun 6 09:13:06 2012 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 06 Jun 2012 09:13:06 -0700 Subject: Strange errors when making changes to the configuration In-Reply-To: <4FCF0CE0.1050504@oracle.com> References: <5B9D761D-DDA1-41A5-AB3A-15F29FCE2290@oracle.com> <4FCF0CE0.1050504@oracle.com> Message-ID: <4FCF8192.9070209@oracle.com> On 6/6/2012 12:55 AM, Chris Hegarty wrote: > The jigsaw native interface is now reading the configuration directly > > http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/file/7b282c826118/src/share/native/common/jigsaw.c I will add a comment in SimpleLibrary.StoredConfiguration that changes to the configuration have to be changed in load_config in jigsaw.c (sorry I should have done that in my previous changeset). > This is a particularly fragile area. Yes indeed. We will need to revisit what's the best way to keep the native and Java implementation in sync with minimal maintainance overhead [1]. Mandy [1] http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/file/7b282c826118/src/share/native/common/jigsaw.h From paul.sandoz at oracle.com Thu Jun 7 00:23:31 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 7 Jun 2012 09:23:31 +0200 Subject: Strange errors when making changes to the configuration In-Reply-To: <4FCF8192.9070209@oracle.com> References: <5B9D761D-DDA1-41A5-AB3A-15F29FCE2290@oracle.com> <4FCF0CE0.1050504@oracle.com> <4FCF8192.9070209@oracle.com> Message-ID: <37C32E4F-DE5C-4AEE-9938-5B17DF5489A6@oracle.com> On Jun 6, 2012, at 6:13 PM, Mandy Chung wrote: > On 6/6/2012 12:55 AM, Chris Hegarty wrote: >> The jigsaw native interface is now reading the configuration directly >> >> http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/file/7b282c826118/src/share/native/common/jigsaw.c > > I will add a comment in SimpleLibrary.StoredConfiguration that changes to the configuration have to be changed in load_config in jigsaw.c (sorry I should have done that in my previous change set). > >> This is a particularly fragile area. > > Yes indeed. We will need to revisit what's the best way to keep the native and Java implementation in sync with minimal maintainance overhead [1]. > I guess the BDB work will affect this area. Paul. From paul.sandoz at oracle.com Thu Jun 7 00:48:31 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 7 Jun 2012 09:48:31 +0200 Subject: Compiling just the classes in the JDK source Message-ID: <303FC08A-2716-4A14-956A-B65CBDDE4F14@oracle.com> Hi, How can i compile just the classes in the JDK rather than everything including re-building all the modules? Say i built Jigsaw already then i made a tweak to one source file e.g. org.openjdk.jigsaw.SimpleLibrary. I want to compile that source file then stuff that Java class into the appropriate area in the existing JDK library. I know it is risky :-) but for certain cases one knows nothing else has changed. Paul. From chris.hegarty at oracle.com Thu Jun 7 02:01:30 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 07 Jun 2012 10:01:30 +0100 Subject: Compiling just the classes in the JDK source In-Reply-To: <303FC08A-2716-4A14-956A-B65CBDDE4F14@oracle.com> References: <303FC08A-2716-4A14-956A-B65CBDDE4F14@oracle.com> Message-ID: <4FD06DEA.1030008@oracle.com> On 07/06/2012 08:48, Paul Sandoz wrote: > Hi, > > How can i compile just the classes in the JDK rather than everything including re-building all the modules? > > Say i built Jigsaw already then i made a tweak to one source file e.g. org.openjdk.jigsaw.SimpleLibrary. I want to compile that source file then stuff that Java class into the appropriate area in the existing JDK library. > > I know it is risky :-) but for certain cases one knows nothing else has changed. > > Paul. WARNING... WARNING... NOT FOR THE FAINT-HEARTED!! :> cd myrepo/jdk/src/share/classes/org/openjdk/jigsaw :> javac -d . `find . -name "*.java"` :> jar -uvf myrepo//build/solaris-i586/lib/modules/jdk.base/8-ea/classes org (Or update the image, solaris-i586/jdk-module-image/lib/modules/jdk.base/8-ea/classes) This only works because the classes are currently stored in a zip archive. If this changes the above will obviously not work. -Chris. From weijun.wang at oracle.com Thu Jun 7 02:23:00 2012 From: weijun.wang at oracle.com (Weijun Wang) Date: Thu, 07 Jun 2012 17:23:00 +0800 Subject: Compiling just the classes in the JDK source In-Reply-To: <4FD06DEA.1030008@oracle.com> References: <303FC08A-2716-4A14-956A-B65CBDDE4F14@oracle.com> <4FD06DEA.1030008@oracle.com> Message-ID: <4FD072F4.7000004@oracle.com> I remember once saw a mail from Alan talking about "exploded" modules, i.e. class files in a directory with the name of a module. Can't you just javac -d to that diretory? Thanks Max On 06/07/2012 05:01 PM, Chris Hegarty wrote: > On 07/06/2012 08:48, Paul Sandoz wrote: >> Hi, >> >> How can i compile just the classes in the JDK rather than everything >> including re-building all the modules? >> >> Say i built Jigsaw already then i made a tweak to one source file e.g. >> org.openjdk.jigsaw.SimpleLibrary. I want to compile that source file >> then stuff that Java class into the appropriate area in the existing >> JDK library. >> >> I know it is risky :-) but for certain cases one knows nothing else >> has changed. >> >> Paul. > > WARNING... WARNING... NOT FOR THE FAINT-HEARTED!! > > :> cd myrepo/jdk/src/share/classes/org/openjdk/jigsaw > :> javac -d . `find . -name "*.java"` > :> jar -uvf myrepo//build/solaris-i586/lib/modules/jdk.base/8-ea/classes > org > (Or update the image, > solaris-i586/jdk-module-image/lib/modules/jdk.base/8-ea/classes) > > This only works because the classes are currently stored in a zip > archive. If this changes the above will obviously not work. > > > > -Chris. From chris.hegarty at oracle.com Thu Jun 7 02:24:55 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 07 Jun 2012 10:24:55 +0100 Subject: Compiling just the classes in the JDK source In-Reply-To: <4FD072F4.7000004@oracle.com> References: <303FC08A-2716-4A14-956A-B65CBDDE4F14@oracle.com> <4FD06DEA.1030008@oracle.com> <4FD072F4.7000004@oracle.com> Message-ID: <4FD07367.1000103@oracle.com> On 07/06/2012 10:23, Weijun Wang wrote: > I remember once saw a mail from Alan talking about "exploded" modules, > i.e. class files in a directory with the name of a module. Can't you > just javac -d to that diretory? Exploded modules support is not yet in. When it is, it will make this type of quick development cycle possible. -Chris. > > Thanks > Max > > On 06/07/2012 05:01 PM, Chris Hegarty wrote: >> On 07/06/2012 08:48, Paul Sandoz wrote: >>> Hi, >>> >>> How can i compile just the classes in the JDK rather than everything >>> including re-building all the modules? >>> >>> Say i built Jigsaw already then i made a tweak to one source file e.g. >>> org.openjdk.jigsaw.SimpleLibrary. I want to compile that source file >>> then stuff that Java class into the appropriate area in the existing >>> JDK library. >>> >>> I know it is risky :-) but for certain cases one knows nothing else >>> has changed. >>> >>> Paul. >> >> WARNING... WARNING... NOT FOR THE FAINT-HEARTED!! >> >> :> cd myrepo/jdk/src/share/classes/org/openjdk/jigsaw >> :> javac -d . `find . -name "*.java"` >> :> jar -uvf myrepo//build/solaris-i586/lib/modules/jdk.base/8-ea/classes >> org >> (Or update the image, >> solaris-i586/jdk-module-image/lib/modules/jdk.base/8-ea/classes) >> >> This only works because the classes are currently stored in a zip >> archive. If this changes the above will obviously not work. >> >> >> >> -Chris. From Alan.Bateman at oracle.com Thu Jun 7 02:39:06 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 07 Jun 2012 10:39:06 +0100 Subject: Compiling just the classes in the JDK source In-Reply-To: <4FD07367.1000103@oracle.com> References: <303FC08A-2716-4A14-956A-B65CBDDE4F14@oracle.com> <4FD06DEA.1030008@oracle.com> <4FD072F4.7000004@oracle.com> <4FD07367.1000103@oracle.com> Message-ID: <4FD076BA.5090207@oracle.com> The "modularize" target is helpful when trying to do incremental builds. It re-runs the class analyzer, modularizer and and re-installs the modules, eg: make -C make/modules ALT_OUTPUTDIR= modularize Once the new build system is working with Jigsaw then I expect it will become a lot better. -Alan From alan.bateman at oracle.com Thu Jun 7 06:47:12 2012 From: alan.bateman at oracle.com (alan.bateman at oracle.com) Date: Thu, 07 Jun 2012 13:47:12 +0000 Subject: hg: jigsaw/jigsaw/jdk: Export supported/documented Sun/Oracle specific APIs Message-ID: <20120607134734.08DD14778E@hg.openjdk.java.net> Changeset: 23f6722e22c5 Author: alanb Date: 2012-06-07 14:46 +0100 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/23f6722e22c5 Export supported/documented Sun/Oracle specific APIs ! make/modules/modules.group From michael.x.mcmahon at oracle.com Thu Jun 7 08:07:04 2012 From: michael.x.mcmahon at oracle.com (Michael McMahon) Date: Thu, 07 Jun 2012 16:07:04 +0100 Subject: source code re-organization Message-ID: <4FD0C398.1050300@oracle.com> Hi, The webrev below contains a utility which re-organizes the jdk8 source code into one possible modularized layout suitable for jigsaw. The way it works is that first a meta script called 'makeshufflescripts' is run which looks at the current layout and generates two further sets of scripts (so-called 'shuffle' and 'unshuffle' scripts). These scripts contain a set of 'rsync' commands that copy changed files between the old and the new layouts. 'shuffle' copies from the old to the new, and 'unshuffle' copies any modified files from the new layout back to the old/present jdk source layout. The script operates on the jdk, langtools corba, jaxws and jaxp repositories copying from the 'src' sub-tree in each to/from a 'newsrc' sub-tree containing the new layout. The purpose of the utility is for people to experiment with the new layout, editing source files etc. Changed files can be 'unshuffled' back to the old layout, for building. There is a README which explains how exactly to use it. Currently, it has only been tested on Linux. http://cr.openjdk.java.net/~michaelm/shuffle/1/webrev/ As mentioned above, the main script in the utility is 'makeshufflescripts'. Most of the other files in the webrev contain other information needed by the script. 'makeshufflescripts' generates the shuffle/unshuffle scripts into a directory called shufflescripts off the root of the jigsaw forest. To do a shuffle, execute "sh shufflescripts/shuffle.sh" and to do an unshuffle "sh shufflescripts/unshuffle.sh" All comments/questions welcome. - Michael. From david.holmes at oracle.com Thu Jun 7 18:46:16 2012 From: david.holmes at oracle.com (David Holmes) Date: Fri, 08 Jun 2012 11:46:16 +1000 Subject: Spurious .trash folder ? Message-ID: <4FD15968.60207@oracle.com> In my jdk-module-image I'm finding lib/modules/.trash ? David From mandy.chung at oracle.com Thu Jun 7 19:09:09 2012 From: mandy.chung at oracle.com (Mandy Chung) Date: Thu, 07 Jun 2012 19:09:09 -0700 Subject: Spurious .trash folder ? In-Reply-To: <4FD15968.60207@oracle.com> References: <4FD15968.60207@oracle.com> Message-ID: <4FD15EC5.8000402@oracle.com> On 6/7/2012 6:46 PM, David Holmes wrote: > In my jdk-module-image I'm finding > > lib/modules/.trash > This is used during the process of module removal (jmod rm) [1]. Looks like this is created at the library creation time rather than when needed (first time when jmod rm is called). Mandy [1] http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/e38d4d5251d5 From david.holmes at oracle.com Thu Jun 7 20:17:27 2012 From: david.holmes at oracle.com (David Holmes) Date: Fri, 08 Jun 2012 13:17:27 +1000 Subject: Spurious .trash folder ? In-Reply-To: <4FD15EC5.8000402@oracle.com> References: <4FD15968.60207@oracle.com> <4FD15EC5.8000402@oracle.com> Message-ID: <4FD16EC7.1020300@oracle.com> On 8/06/2012 12:09 PM, Mandy Chung wrote: > On 6/7/2012 6:46 PM, David Holmes wrote: >> In my jdk-module-image I'm finding >> >> lib/modules/.trash >> > > This is used during the process of module removal (jmod rm) [1]. Looks > like this is created at the library creation time rather than when > needed (first time when jmod rm is called). Thanks Mandy. I would have thought a tmp directory would be used. David > Mandy > [1] http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/e38d4d5251d5 From chris.hegarty at oracle.com Fri Jun 8 01:08:16 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Fri, 08 Jun 2012 09:08:16 +0100 Subject: Spurious .trash folder ? In-Reply-To: <4FD16EC7.1020300@oracle.com> References: <4FD15968.60207@oracle.com> <4FD15EC5.8000402@oracle.com> <4FD16EC7.1020300@oracle.com> Message-ID: <4FD1B2F0.3020104@oracle.com> On 08/06/12 04:17, David Holmes wrote: > On 8/06/2012 12:09 PM, Mandy Chung wrote: >> On 6/7/2012 6:46 PM, David Holmes wrote: >>> In my jdk-module-image I'm finding >>> >>> lib/modules/.trash >>> >> >> This is used during the process of module removal (jmod rm) [1]. Looks >> like this is created at the library creation time rather than when >> needed (first time when jmod rm is called). Right, there are already enough potential issues that can arise during removal, so I thought the creation of the trash directory was best done at library creation time. Not a big deal to revisit if we want this done lazily. > Thanks Mandy. I would have thought a tmp directory would be used. tmp|trash|.|%. There were not specific comments during review time on the name so I stuck with '.trash'. Again trivial to change, but this is internal to the library layout. -Chris. > > David > >> Mandy >> [1] http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/e38d4d5251d5 From paul.sandoz at oracle.com Fri Jun 8 01:20:36 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Fri, 8 Jun 2012 10:20:36 +0200 Subject: Compiling just the classes in the JDK source In-Reply-To: <4FD076BA.5090207@oracle.com> References: <303FC08A-2716-4A14-956A-B65CBDDE4F14@oracle.com> <4FD06DEA.1030008@oracle.com> <4FD072F4.7000004@oracle.com> <4FD07367.1000103@oracle.com> <4FD076BA.5090207@oracle.com> Message-ID: <8DB34CC1-9468-4B66-A720-F401502FC9C2@oracle.com> On Jun 7, 2012, at 11:39 AM, Alan Bateman wrote: > > The "modularize" target is helpful when trying to do incremental builds. It re-runs the class analyzer, modularizer and and re-installs the modules, eg: > > make -C make/modules ALT_OUTPUTDIR= modularize > Is there in equivalent make target for just compiling all the sources? Paul. > Once the new build system is working with Jigsaw then I expect it will become a lot better. > > -Alan > > From erik.joelsson at oracle.com Fri Jun 8 05:46:37 2012 From: erik.joelsson at oracle.com (Erik Joelsson) Date: Fri, 08 Jun 2012 14:46:37 +0200 Subject: source code re-organization In-Reply-To: <4FD0C398.1050300@oracle.com> References: <4FD0C398.1050300@oracle.com> Message-ID: <4FD1F42D.5040900@oracle.com> Great to see this getting ready! I'm trying it out in preparation for working on the build-infra-jigsaw build. Noticed a couple of things: * Why the need to set MAKESHUFFLE_SRCDIR? I put the following in makeshufflescripts and it seems to work: MS=$(cd "$(dirname "$0")" && pwd) * In non-standard.includes, the line for jdwpTransport.h has an extra jdk/newsrc in the target path. * zipfs.shuffle, zipfs.unshuffle and non-standard.includes contains target paths that look like "jdk/newsrc/$platform/module/...". I assume these should to be changed to mclasses and etc. /Erik On 2012-06-07 17:07, Michael McMahon wrote: > Hi, > > The webrev below contains a utility which re-organizes the jdk8 source > code into one possible > modularized layout suitable for jigsaw. The way it works is that first > a meta script called 'makeshufflescripts' > is run which looks at the current layout and generates two further > sets of scripts (so-called 'shuffle' and > 'unshuffle' scripts). These scripts contain a set of 'rsync' commands > that copy changed files between the old > and the new layouts. 'shuffle' copies from the old to the new, and > 'unshuffle' copies any modified files > from the new layout back to the old/present jdk source layout. The > script operates on the jdk, langtools > corba, jaxws and jaxp repositories copying from the 'src' sub-tree in > each to/from a 'newsrc' sub-tree > containing the new layout. > > The purpose of the utility is for people to experiment with the new > layout, editing source files etc. > Changed files can be 'unshuffled' back to the old layout, for > building. There is a README which explains > how exactly to use it. Currently, it has only been tested on Linux. > > http://cr.openjdk.java.net/~michaelm/shuffle/1/webrev/ > > As mentioned above, the main script in the utility is > 'makeshufflescripts'. Most of the other files > in the webrev contain other information needed by the script. > 'makeshufflescripts' generates the shuffle/unshuffle > scripts into a directory called shufflescripts off the root of the > jigsaw forest. > To do a shuffle, execute "sh shufflescripts/shuffle.sh" and to do an > unshuffle "sh shufflescripts/unshuffle.sh" > > All comments/questions welcome. > > - Michael. From michael.x.mcmahon at oracle.com Fri Jun 8 06:47:24 2012 From: michael.x.mcmahon at oracle.com (Michael McMahon) Date: Fri, 08 Jun 2012 14:47:24 +0100 Subject: source code re-organization In-Reply-To: <4FD1F42D.5040900@oracle.com> References: <4FD0C398.1050300@oracle.com> <4FD1F42D.5040900@oracle.com> Message-ID: <4FD2026C.5030901@oracle.com> Thanks Erik. All good points. I'll make these updates as suggested. I moved the script into a fixed location late in the day, so the environment variable is not required any more. - Michael On 08/06/12 13:46, Erik Joelsson wrote: > Great to see this getting ready! I'm trying it out in preparation for > working on the build-infra-jigsaw build. Noticed a couple of things: > > * Why the need to set MAKESHUFFLE_SRCDIR? I put the following in > makeshufflescripts and it seems to work: > MS=$(cd "$(dirname "$0")" && pwd) > > * In non-standard.includes, the line for jdwpTransport.h has an extra > jdk/newsrc in the target path. > > * zipfs.shuffle, zipfs.unshuffle and non-standard.includes contains > target paths that look like "jdk/newsrc/$platform/module/...". I > assume these should to be changed to mclasses and etc. > > /Erik > > On 2012-06-07 17:07, Michael McMahon wrote: >> Hi, >> >> The webrev below contains a utility which re-organizes the jdk8 >> source code into one possible >> modularized layout suitable for jigsaw. The way it works is that >> first a meta script called 'makeshufflescripts' >> is run which looks at the current layout and generates two further >> sets of scripts (so-called 'shuffle' and >> 'unshuffle' scripts). These scripts contain a set of 'rsync' commands >> that copy changed files between the old >> and the new layouts. 'shuffle' copies from the old to the new, and >> 'unshuffle' copies any modified files >> from the new layout back to the old/present jdk source layout. The >> script operates on the jdk, langtools >> corba, jaxws and jaxp repositories copying from the 'src' sub-tree in >> each to/from a 'newsrc' sub-tree >> containing the new layout. >> >> The purpose of the utility is for people to experiment with the new >> layout, editing source files etc. >> Changed files can be 'unshuffled' back to the old layout, for >> building. There is a README which explains >> how exactly to use it. Currently, it has only been tested on Linux. >> >> http://cr.openjdk.java.net/~michaelm/shuffle/1/webrev/ >> >> As mentioned above, the main script in the utility is >> 'makeshufflescripts'. Most of the other files >> in the webrev contain other information needed by the script. >> 'makeshufflescripts' generates the shuffle/unshuffle >> scripts into a directory called shufflescripts off the root of the >> jigsaw forest. >> To do a shuffle, execute "sh shufflescripts/shuffle.sh" and to do an >> unshuffle "sh shufflescripts/unshuffle.sh" >> >> All comments/questions welcome. >> >> - Michael. From alan.bateman at oracle.com Mon Jun 11 02:36:03 2012 From: alan.bateman at oracle.com (alan.bateman at oracle.com) Date: Mon, 11 Jun 2012 09:36:03 +0000 Subject: hg: jigsaw/jigsaw: 3 new changesets Message-ID: <20120611093604.1CEE547848@hg.openjdk.java.net> Changeset: 1ce5dc164166 Author: cl Date: 2012-06-01 14:11 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/rev/1ce5dc164166 Added tag jdk8-b41 for changeset 1a8c7c530f8a ! .hgtags Changeset: dac58047c19f Author: cl Date: 2012-06-07 12:10 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/rev/dac58047c19f Added tag jdk8-b42 for changeset 1ce5dc164166 ! .hgtags Changeset: f6d91d9a9ae1 Author: alanb Date: 2012-06-09 13:12 +0100 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/rev/f6d91d9a9ae1 Merge ! .hgtags From alan.bateman at oracle.com Mon Jun 11 02:36:21 2012 From: alan.bateman at oracle.com (alan.bateman at oracle.com) Date: Mon, 11 Jun 2012 09:36:21 +0000 Subject: hg: jigsaw/jigsaw/corba: 3 new changesets Message-ID: <20120611093625.4E86047849@hg.openjdk.java.net> Changeset: 79cc42c9c71b Author: cl Date: 2012-06-01 14:12 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/corba/rev/79cc42c9c71b Added tag jdk8-b41 for changeset 113f0d5f0a08 ! .hgtags Changeset: 4def0723b0b0 Author: cl Date: 2012-06-07 12:10 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/corba/rev/4def0723b0b0 Added tag jdk8-b42 for changeset 79cc42c9c71b ! .hgtags Changeset: ad39737f4189 Author: alanb Date: 2012-06-09 13:13 +0100 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/corba/rev/ad39737f4189 Merge ! .hgtags From alan.bateman at oracle.com Mon Jun 11 02:37:01 2012 From: alan.bateman at oracle.com (alan.bateman at oracle.com) Date: Mon, 11 Jun 2012 09:37:01 +0000 Subject: hg: jigsaw/jigsaw/hotspot: 46 new changesets Message-ID: <20120611093846.AAFB84784A@hg.openjdk.java.net> Changeset: 03d61caacd1e Author: amurillo Date: 2012-05-18 14:57 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/03d61caacd1e 7170006: new hotspot build - hs24-b12 Reviewed-by: jcoomes ! make/hotspot_version Changeset: 0251d217257f Author: sla Date: 2012-05-08 20:28 +0200 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/0251d217257f 7162726: Wrong filter predicate of visible locals in SA JSJavaFrame Reviewed-by: sla, dcubed Contributed-by: Krystal Mok ! agent/src/share/classes/sun/jvm/hotspot/utilities/soql/JSJavaFrame.java Changeset: 7f410b6ea66c Author: dholmes Date: 2012-05-09 00:28 -0400 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/7f410b6ea66c 7167406: (Zero) Fix for InvokeDynamic needed Reviewed-by: chrisphi, dholmes Contributed-by: Andrew Dinn ! src/cpu/zero/vm/cppInterpreter_zero.cpp Changeset: d506b2cf2ad0 Author: dholmes Date: 2012-05-09 04:32 -0400 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/d506b2cf2ad0 Merge Changeset: 78d2ae5ab35b Author: nloodin Date: 2012-05-09 16:24 +0200 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/78d2ae5ab35b 7163117: Agent can't connect to process on Mac OSX Reviewed-by: dholmes, coleenp, sla, minqi, kvn ! agent/src/share/classes/sun/jvm/hotspot/HotSpotAgent.java ! agent/src/share/classes/sun/jvm/hotspot/bugspot/BugSpotAgent.java ! agent/src/share/classes/sun/jvm/hotspot/debugger/bsd/BsdCDebugger.java ! agent/src/share/classes/sun/jvm/hotspot/debugger/bsd/BsdThreadContextFactory.java ! agent/src/share/classes/sun/jvm/hotspot/debugger/proc/ProcDebuggerLocal.java ! agent/src/share/classes/sun/jvm/hotspot/debugger/remote/RemoteDebuggerClient.java ! agent/src/share/classes/sun/jvm/hotspot/runtime/Threads.java ! agent/src/share/classes/sun/jvm/hotspot/ui/classbrowser/HTMLGenerator.java Changeset: 037973617842 Author: kevinw Date: 2012-05-11 17:24 +0100 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/037973617842 7157734: hotspot test scripts not testing 64-bit JVM under JPRT/JTREG. Reviewed-by: kvn ! test/compiler/6894807/Test6894807.sh ! test/gc/6941923/test6941923.sh ! test/runtime/6626217/Test6626217.sh ! test/runtime/6878713/Test6878713.sh ! test/runtime/6929067/Test6929067.sh ! test/runtime/7020373/Test7020373.sh ! test/runtime/7051189/Xchecksig.sh ! test/runtime/7158988/TestFieldMonitor.sh Changeset: 2f4819f92dc7 Author: zgu Date: 2012-05-10 18:19 -0400 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/2f4819f92dc7 Merge - src/share/vm/gc_implementation/concurrentMarkSweep/binaryTreeDictionary.cpp - src/share/vm/gc_implementation/concurrentMarkSweep/binaryTreeDictionary.hpp - src/share/vm/gc_implementation/concurrentMarkSweep/freeBlockDictionary.cpp - src/share/vm/gc_implementation/concurrentMarkSweep/freeBlockDictionary.hpp - src/share/vm/gc_implementation/concurrentMarkSweep/freeList.cpp - src/share/vm/gc_implementation/concurrentMarkSweep/freeList.hpp Changeset: b4f7c4315c36 Author: zgu Date: 2012-05-12 06:50 -0400 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/b4f7c4315c36 Merge Changeset: 7d4e6dabc6bf Author: dcubed Date: 2012-05-15 10:52 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/7d4e6dabc6bf 7165060: dtrace tests fail with FDS debug info files Summary: Work around 'gobjcopy --add-gnu-debuglink' failure by adding a temporary tool that adds the '.gnu_debuglink' section and nothing more. Reviewed-by: sspitsyn, acorn + make/solaris/makefiles/add_gnu_debuglink.make ! make/solaris/makefiles/vm.make + src/os/solaris/add_gnu_debuglink/add_gnu_debuglink.c Changeset: 80b9cc90b643 Author: dcubed Date: 2012-05-15 11:27 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/80b9cc90b643 Merge ! make/solaris/makefiles/defs.make ! make/solaris/makefiles/vm.make Changeset: 9793f47cdebc Author: dcubed Date: 2012-05-15 15:16 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/9793f47cdebc 7169102: 7165060 merge lost changes to make/windows/makefiles/defs.make Reviewed-by: sspitsyn, acorn ! make/windows/makefiles/defs.make Changeset: 7432b9db36ff Author: nloodin Date: 2012-05-10 15:44 +0200 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/7432b9db36ff 7165755: OS Information much longer on linux than other platforms Reviewed-by: sla, dholmes ! src/os/bsd/vm/os_bsd.cpp ! src/os/linux/vm/os_linux.cpp ! src/os/linux/vm/os_linux.hpp ! src/os/posix/vm/os_posix.cpp + src/os/posix/vm/os_posix.hpp ! src/os/solaris/vm/os_solaris.cpp ! src/os/solaris/vm/os_solaris.hpp ! src/os/windows/vm/os_windows.cpp ! src/os/windows/vm/os_windows.hpp ! src/share/vm/runtime/os.hpp Changeset: 198dcc84088c Author: dcubed Date: 2012-05-16 12:47 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/198dcc84088c 7169409: enabling ZIP_DEBUGINFO_FILES causes unexpected test failures on Windows X86 Summary: Disable ZIP_DEBUGINFO_FILES by default on Windows. Reviewed-by: acorn ! make/windows/makefiles/defs.make Changeset: 4b37c0dafe3a Author: dcubed Date: 2012-05-18 09:15 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/4b37c0dafe3a 7170010: conditional "ZIP_DEBUGINFO_FILES ?= 0" setting is not reliable on Windows Summary: Always disable ZIP_DEBUGINFO_FILES on Windows. Reviewed-by: acorn ! make/windows/makefiles/defs.make Changeset: cee14a6fc5ac Author: zgu Date: 2012-05-22 20:29 -0400 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/cee14a6fc5ac Merge Changeset: 6759698e3140 Author: roland Date: 2012-05-15 10:10 +0200 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/6759698e3140 7133857: exp() and pow() should use the x87 ISA on x86 Summary: use x87 instructions to implement exp() and pow() in interpreter/c1/c2. Reviewed-by: kvn, never, twisti ! src/cpu/sparc/vm/c1_LIRGenerator_sparc.cpp ! src/cpu/sparc/vm/interpreter_sparc.cpp ! src/cpu/x86/vm/assembler_x86.cpp ! src/cpu/x86/vm/assembler_x86.hpp ! src/cpu/x86/vm/c1_LIRAssembler_x86.cpp ! src/cpu/x86/vm/c1_LIRGenerator_x86.cpp ! src/cpu/x86/vm/c1_LinearScan_x86.cpp ! src/cpu/x86/vm/interpreter_x86_32.cpp ! src/cpu/x86/vm/interpreter_x86_64.cpp ! src/cpu/x86/vm/stubGenerator_x86_32.cpp ! src/cpu/x86/vm/stubGenerator_x86_64.cpp ! src/cpu/x86/vm/templateInterpreter_x86_32.cpp ! src/cpu/x86/vm/templateInterpreter_x86_64.cpp ! src/cpu/x86/vm/x86_32.ad ! src/cpu/x86/vm/x86_64.ad ! src/share/vm/c1/c1_GraphBuilder.cpp ! src/share/vm/c1/c1_LIR.cpp ! src/share/vm/c1/c1_LIR.hpp ! src/share/vm/c1/c1_LIRAssembler.cpp ! src/share/vm/c1/c1_LIRGenerator.cpp ! src/share/vm/c1/c1_LinearScan.cpp ! src/share/vm/interpreter/abstractInterpreter.hpp ! src/share/vm/interpreter/interpreter.cpp ! src/share/vm/interpreter/templateInterpreter.cpp ! src/share/vm/opto/library_call.cpp ! src/share/vm/opto/subnode.cpp Changeset: 4073d9478a90 Author: roland Date: 2012-05-18 15:50 +0200 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/4073d9478a90 7167254: Crash on OSX in Enumerator.nextElement() with compressed oops Summary: null checks in "compressed oops with base" mode may trigger a SIGBUS rather than a SIGSEGV. Reviewed-by: dsamersoff, dcubed, rbackman, kvn ! src/os_cpu/bsd_x86/vm/os_bsd_x86.cpp Changeset: cdd249497b34 Author: twisti Date: 2012-05-18 12:20 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/cdd249497b34 7170053: crash in C2 when using -XX:+CountCompiledCalls Reviewed-by: kvn, twisti Contributed-by: Krystal Mok ! src/share/vm/opto/doCall.cpp Changeset: e2961d14584b Author: roland Date: 2012-05-21 09:46 +0200 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/e2961d14584b 7169934: pow(x,y) or x64 computes incorrect result when x<0 and y is an odd integer Summary: bad test for parity of y in pow(x,y) (c1, interpreter) Reviewed-by: kvn, twisti ! src/cpu/x86/vm/assembler_x86.cpp Changeset: 7089278210e2 Author: kvn Date: 2012-05-24 18:39 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/7089278210e2 Merge Changeset: 785573170238 Author: amurillo Date: 2012-05-25 14:45 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/785573170238 Merge Changeset: 37add4fa0296 Author: amurillo Date: 2012-05-25 14:45 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/37add4fa0296 Added tag hs24-b12 for changeset 785573170238 ! .hgtags Changeset: 2040997cba56 Author: cl Date: 2012-06-01 14:12 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/2040997cba56 Added tag jdk8-b41 for changeset 37add4fa0296 ! .hgtags Changeset: c80d42d78a01 Author: amurillo Date: 2012-05-25 14:56 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/c80d42d78a01 7171853: new hotspot build - hs24-b13 Reviewed-by: jcoomes ! make/hotspot_version Changeset: 960a442eae91 Author: rbackman Date: 2012-05-22 10:11 +0200 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/960a442eae91 7161732: Improve handling of thread_id in OSThread Reviewed-by: dholmes, kamg ! src/os/bsd/vm/osThread_bsd.hpp ! src/os/linux/vm/osThread_linux.hpp ! src/os/solaris/vm/osThread_solaris.hpp ! src/os/windows/vm/osThread_windows.hpp ! src/os_cpu/bsd_x86/vm/vmStructs_bsd_x86.hpp ! src/os_cpu/linux_sparc/vm/vmStructs_linux_sparc.hpp ! src/os_cpu/linux_x86/vm/vmStructs_linux_x86.hpp ! src/os_cpu/solaris_sparc/vm/vmStructs_solaris_sparc.hpp ! src/os_cpu/solaris_x86/vm/vmStructs_solaris_x86.hpp ! src/os_cpu/windows_x86/vm/vmStructs_windows_x86.hpp ! src/share/vm/runtime/osThread.hpp Changeset: df84b4a3ebcb Author: rbackman Date: 2012-05-24 13:37 +0200 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/df84b4a3ebcb 7171422: Change 7161732 breaks SA on Windows Reviewed-by: dholmes, sla ! src/os_cpu/windows_x86/vm/vmStructs_windows_x86.hpp Changeset: 9c1709c4c80c Author: dcubed Date: 2012-05-24 12:03 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/9c1709c4c80c 7165598: enable FDS on Solaris X64 when 7165593 is fixed Summary: Work around 'gobjcopy' failure on Solaris X64 by adding a temporary tool that removes the SHF_ALLOC flag from "empty" section headers. Reviewed-by: sspitsyn, acorn ! make/solaris/makefiles/defs.make ! make/solaris/makefiles/dtrace.make + make/solaris/makefiles/fix_empty_sec_hdr_flags.make ! make/solaris/makefiles/jsig.make ! make/solaris/makefiles/saproc.make ! make/solaris/makefiles/vm.make + src/os/solaris/fix_empty_sec_hdr_flags/fix_empty_sec_hdr_flags.c Changeset: fb8f9ab27c14 Author: dcubed Date: 2012-05-24 12:57 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/fb8f9ab27c14 Merge ! make/solaris/makefiles/defs.make ! make/solaris/makefiles/vm.make ! make/windows/makefiles/defs.make Changeset: cd3a8ab0352b Author: zgu Date: 2012-05-24 20:04 -0400 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/cd3a8ab0352b Merge Changeset: 17be2328b50b Author: zgu Date: 2012-05-26 06:07 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/17be2328b50b Merge Changeset: 5be76dc5304d Author: zgu Date: 2012-05-29 20:06 -0400 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/5be76dc5304d Merge Changeset: 71afdabfd05b Author: jiangli Date: 2012-05-21 14:10 -0400 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/71afdabfd05b 7168280: Eliminate the generic signature index slot from field array for field without generic signature. Summary: Only allocate the generic signature index slot in the field array for field with generic signature attribute. Reviewed-by: coleenp, dlong ! agent/src/share/classes/sun/jvm/hotspot/oops/AccessFlags.java ! agent/src/share/classes/sun/jvm/hotspot/oops/InstanceKlass.java ! agent/src/share/classes/sun/jvm/hotspot/runtime/ClassConstants.java ! src/share/vm/classfile/classFileParser.cpp ! src/share/vm/classfile/systemDictionary.cpp ! src/share/vm/oops/fieldInfo.hpp ! src/share/vm/oops/fieldStreams.hpp ! src/share/vm/oops/instanceKlass.hpp ! src/share/vm/runtime/fieldDescriptor.cpp ! src/share/vm/runtime/fieldDescriptor.hpp ! src/share/vm/runtime/vmStructs.cpp ! src/share/vm/utilities/accessFlags.hpp Changeset: 48df98e03639 Author: dholmes Date: 2012-05-23 20:09 -0400 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/48df98e03639 7170197: Update JPRT default build targets to support embedded builds Reviewed-by: jcoomes, kvn ! make/jprt.properties Changeset: 4e64a590066e Author: jprovino Date: 2012-05-26 08:49 -0400 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/4e64a590066e Merge Changeset: dd45f26b4282 Author: dholmes Date: 2012-05-30 17:45 -0400 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/dd45f26b4282 Merge Changeset: c92a79900986 Author: brutisso Date: 2012-05-17 21:53 +0200 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/c92a79900986 7169062: CMS: Assertion failed with -XX:+ObjectAlignmentInBytes=64 Summary: Removed the assert in CompactibleFreeListSpace::check_free_list_consistency() since it was too strict. Simplified CompactibleFreeListSpace::set_cms_values() to reducde the need for asserts. Reviewed-by: jcoomes, stefank ! src/share/vm/gc_implementation/concurrentMarkSweep/compactibleFreeListSpace.cpp Changeset: 9a344d88dc22 Author: mnunez Date: 2012-05-21 14:59 +0200 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/9a344d88dc22 7168848: Add test to check that humongous object allocation path also checks the heap occupancy. Summary: Added test that checks humongous object allocation path also check the heap occupancy and initiate a marking cycle when / if needed. Reviewed-by: brutisso, jwilhelm + test/gc/7168848/HumongousAlloc.java Changeset: 1d478c993020 Author: johnc Date: 2012-05-29 10:18 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/1d478c993020 7143858: G1: Back to back young GCs with the second GC having a minimally sized eden Summary: Before the last thread to leave a JNI critical region was able to schedule a GCLocker Initiated GC, another thread was attempting an allocation and saw that the GCLocker region was no longer active and successfully scheduled a GC. Stall allocating threads until the GCLocker Initiated GC is performed and then retry the allocation. Reviewed-by: brutisso, huntch ! src/share/vm/gc_implementation/g1/g1CollectedHeap.cpp Changeset: 5c8bd7c16119 Author: brutisso Date: 2012-05-25 22:35 +0200 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/5c8bd7c16119 7171936: LOG_G incorrectly defined in globalDefinitions.hpp Summary: Removed LOG_G and LOG_K. Moved LOG_M to where it is being used. Reviewed-by: twisti, johnc ! src/share/vm/gc_implementation/g1/heapRegionRemSet.cpp ! src/share/vm/utilities/globalDefinitions.hpp Changeset: c52a6a39546c Author: johnc Date: 2012-05-30 10:26 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/c52a6a39546c 7158682: G1: Handle leak when running nsk.sysdict tests Summary: Place HandleMarks in the code that allocates handles for the pending list lock so that the handles are freed and multiple, unsuccessful, attempts to schedule a GC do not cause an OOM. Reviewed-by: brutisso ! src/share/vm/oops/instanceRefKlass.cpp Changeset: bbc900c2482a Author: brutisso Date: 2012-05-31 21:10 +0200 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/bbc900c2482a 7172279: G1: Clean up TraceGen0Time and TraceGen1Time data gathering Summary: Simplify code, remove unused code, remove ExitAfterGCNum Reviewed-by: huntch, johnc ! src/share/vm/gc_implementation/g1/g1CollectedHeap.cpp ! src/share/vm/gc_implementation/g1/g1CollectorPolicy.cpp ! src/share/vm/gc_implementation/g1/g1CollectorPolicy.hpp ! src/share/vm/memory/genCollectedHeap.cpp ! src/share/vm/runtime/globals.hpp ! src/share/vm/utilities/numberSeq.cpp ! src/share/vm/utilities/numberSeq.hpp Changeset: 7121cd2c58b5 Author: jcoomes Date: 2012-06-01 10:29 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/7121cd2c58b5 Merge Changeset: bd568544be7f Author: amurillo Date: 2012-06-01 15:24 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/bd568544be7f Merge Changeset: 55954061c6e8 Author: amurillo Date: 2012-06-01 15:24 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/55954061c6e8 Added tag hs24-b13 for changeset bd568544be7f ! .hgtags Changeset: e77b8e0ed1f8 Author: cl Date: 2012-06-07 12:10 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/e77b8e0ed1f8 Added tag jdk8-b42 for changeset 55954061c6e8 ! .hgtags Changeset: 65b16592424b Author: alanb Date: 2012-06-09 13:54 +0100 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/hotspot/rev/65b16592424b Merge ! .hgtags ! make/jprt.properties ! src/os/linux/vm/os_linux.cpp ! src/os/solaris/vm/os_solaris.cpp ! src/share/vm/classfile/classFileParser.cpp ! src/share/vm/classfile/systemDictionary.cpp ! src/share/vm/oops/instanceKlass.hpp ! src/share/vm/runtime/fieldDescriptor.hpp ! src/share/vm/runtime/globals.hpp ! src/share/vm/runtime/vmStructs.cpp ! src/share/vm/utilities/accessFlags.hpp From alan.bateman at oracle.com Mon Jun 11 02:40:38 2012 From: alan.bateman at oracle.com (alan.bateman at oracle.com) Date: Mon, 11 Jun 2012 09:40:38 +0000 Subject: hg: jigsaw/jigsaw/jaxp: 3 new changesets Message-ID: <20120611094050.120074784B@hg.openjdk.java.net> Changeset: 39ee03c16021 Author: cl Date: 2012-06-01 14:12 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jaxp/rev/39ee03c16021 Added tag jdk8-b41 for changeset 6f5c0e17415d ! .hgtags Changeset: c73aaf2b0d14 Author: cl Date: 2012-06-07 12:10 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jaxp/rev/c73aaf2b0d14 Added tag jdk8-b42 for changeset 39ee03c16021 ! .hgtags Changeset: 46a4ac83f199 Author: alanb Date: 2012-06-09 13:38 +0100 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jaxp/rev/46a4ac83f199 Merge ! .hgtags From alan.bateman at oracle.com Mon Jun 11 02:41:04 2012 From: alan.bateman at oracle.com (alan.bateman at oracle.com) Date: Mon, 11 Jun 2012 09:41:04 +0000 Subject: hg: jigsaw/jigsaw/jaxws: 3 new changesets Message-ID: <20120611094113.AF1E44784C@hg.openjdk.java.net> Changeset: 1f20f37818a9 Author: cl Date: 2012-06-01 14:12 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jaxws/rev/1f20f37818a9 Added tag jdk8-b41 for changeset f2072b164b05 ! .hgtags Changeset: 35e0f6001816 Author: cl Date: 2012-06-07 12:10 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jaxws/rev/35e0f6001816 Added tag jdk8-b42 for changeset 1f20f37818a9 ! .hgtags Changeset: 319f81ca93f5 Author: alanb Date: 2012-06-09 13:39 +0100 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jaxws/rev/319f81ca93f5 Merge ! .hgtags From alan.bateman at oracle.com Mon Jun 11 02:41:25 2012 From: alan.bateman at oracle.com (alan.bateman at oracle.com) Date: Mon, 11 Jun 2012 09:41:25 +0000 Subject: hg: jigsaw/jigsaw/jdk: 7 new changesets Message-ID: <20120611094341.853194784D@hg.openjdk.java.net> Changeset: edb02bee325e Author: ihse Date: 2012-05-25 18:55 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/edb02bee325e 7172014: Remove empty and unused javah include files Reviewed-by: ohair, erikj ! src/share/native/sun/awt/image/cvutils/img_dcm.h ! src/share/native/sun/awt/image/cvutils/img_dcm8.h ! src/share/native/sun/java2d/pipe/SpanClipRenderer.c ! src/solaris/native/sun/awt/initIDs.c Changeset: 8b4dd321b8a2 Author: dholmes Date: 2012-05-30 00:37 -0400 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/8b4dd321b8a2 7171653: 32-bit cross-compile on 64-bit build host generates 64-bit data for awt/X11 leading to crash Reviewed-by: ohair, anthony ! make/sun/xawt/Makefile ! makefiles/sun/xawt/Makefile Changeset: 4eac56f073ea Author: katleman Date: 2012-05-30 15:59 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/4eac56f073ea Merge Changeset: cf5c1f6fbc5b Author: cl Date: 2012-06-01 14:12 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/cf5c1f6fbc5b Added tag jdk8-b41 for changeset 4eac56f073ea ! .hgtags Changeset: 8e8fb6500a87 Author: cl Date: 2012-06-07 12:10 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/8e8fb6500a87 Added tag jdk8-b42 for changeset cf5c1f6fbc5b ! .hgtags Changeset: e7922372ee52 Author: alanb Date: 2012-06-09 13:30 +0100 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/e7922372ee52 Merge ! .hgtags ! make/sun/xawt/Makefile Changeset: 127011c2e629 Author: alanb Date: 2012-06-11 10:19 +0100 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/127011c2e629 Increase -Xmx used by tools to avoid OOME with jpkg ! make/common/shared/Platform.gmk ! make/docs/Makefile From alan.bateman at oracle.com Mon Jun 11 02:48:41 2012 From: alan.bateman at oracle.com (alan.bateman at oracle.com) Date: Mon, 11 Jun 2012 09:48:41 +0000 Subject: hg: jigsaw/jigsaw/langtools: 3 new changesets Message-ID: <20120611094855.7E94B4784E@hg.openjdk.java.net> Changeset: 02c5a3575539 Author: cl Date: 2012-06-01 14:12 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/02c5a3575539 Added tag jdk8-b41 for changeset 179fa85aeefa ! .hgtags Changeset: 8280c4f9f619 Author: cl Date: 2012-06-07 12:10 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/8280c4f9f619 Added tag jdk8-b42 for changeset 02c5a3575539 ! .hgtags Changeset: 8ca432de0c30 Author: alanb Date: 2012-06-09 13:59 +0100 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/langtools/rev/8ca432de0c30 Merge ! .hgtags From paul.sandoz at oracle.com Tue Jun 12 08:03:08 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 12 Jun 2012 17:03:08 +0200 Subject: ServiceLoader.load* take 2 Message-ID: Hi, A set of slides has been created [1] which presents a set of terms, an overview of modular services, describes the current state of affairs with service instance creation and the use of ServiceLoader.load*, then proposes an alternative solution for ServiceLoader, the first part of which is implemented by the following webrev, which changes service instance creation so that it is scoped to the configuration rather than the class loader: http://cr.openjdk.java.net/~psandoz/jigsaw/global-services/webrev/ Summary: in module mode any module class loader can be used with ServiceLoader to create the service instances for a service interface. Previously the class loader of the service consumer module, that consumes the service interface, was required to be used. Next steps: - Modify the configuration so that service information is stored directly with the configuration rather than with each context (that is stored with the configuration). Currently service provider classes are looked up by iterating through all the contexts. This is not very efficient. The configuration is the place where to optimize. This requires some refactoring of org.openjdk.jigsaw.Configuration because for instances of Configuration (the configuration produced when compiling) such state related to services is not relevant. This is also a potential opportunity to improve the immutability of Configuration using a builder pattern for construction. - Modify the resolver so that the permits clause does not affect resolution of a service provider module (and it's dependencies). This area is tricky. Rather than choices of service provider module optional dependencies being pushed on to the choice stack of each service consumer module this potentially could be done once per service interface (although will likely change what solutions are produced), but might be very tricky to implement given that service provider modules may also be service consumer modules. Good diagnostics are required to help track down why a service provider module is not being resolved. Paul. [1] http://openjdk.java.net/projects/jigsaw/doc/openjdk-jigsaw-modular-services.pdf From david.lloyd at redhat.com Tue Jun 12 13:25:11 2012 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 12 Jun 2012 15:25:11 -0500 Subject: ServiceLoader.load* take 2 In-Reply-To: References: Message-ID: <4FD7A5A7.4030401@redhat.com> I have a high level of concern about "tactical and temporary solutions" and things that "might often" work. Here's what this looks like to me. 1. We have an existing solution that does the job. 2. It isn't Jigsaw-ish enough; let's instead create a new service thing that is Jigsaw-ish. 3. New service thing is conceptually divergent from original thing; now let's put in a lot of nasty hacks to make it kinda sorta work. Especially if we can use heuristics - heuristics are neat! 4. (and this is the next step) Yowee, we really screwed things up - now new apps are screwy *and* old apps don't work with any semblance of predictability. A formerly reliable and predictable core API now is unusable for both old and new apps. Guys the existing service loader mechanism works well with multiple class loaders as is. I don't understand what the thinking is here. It isn't broken, and yet it is being fixed. I think that the classpath vs module mode discrepancy is the ultimate realization of this sort of thinking. There is a basic assumption that such a split is needed; the technology is then implemented to realize this, and then becomes its own justification. But such a discrepancy should not be necessary. If an old-style classpath application cannot run in module mode then not enough effort was put forth in terms of compatibility. Note that so much of this service loader solution is predicated upon two modes! Red flag? On 06/12/2012 10:03 AM, Paul Sandoz wrote: > Hi, > > A set of slides has been created [1] which presents a set of terms, an overview of modular services, describes the current state of affairs with service instance creation and the use of ServiceLoader.load*, then proposes an alternative solution for ServiceLoader, the first part of which is implemented by the following webrev, which changes service instance creation so that it is scoped to the configuration rather than the class loader: > > http://cr.openjdk.java.net/~psandoz/jigsaw/global-services/webrev/ > > Summary: in module mode any module class loader can be used with ServiceLoader to create the service instances for a service interface. Previously the class loader of the service consumer module, that consumes the service interface, was required to be used. > > Next steps: > > - Modify the configuration so that service information is stored directly with the configuration rather than with each context (that is stored with the configuration). Currently service provider classes are looked up by iterating through all the contexts. This is not very efficient. The configuration is the place where to optimize. > This requires some refactoring of org.openjdk.jigsaw.Configuration because for instances of Configuration (the configuration produced when compiling) such state related to services is not relevant. This is also a potential opportunity to improve the immutability of Configuration using a builder pattern for construction. > > - Modify the resolver so that the permits clause does not affect resolution of a service provider module (and it's dependencies). This area is tricky. Rather than choices of service provider module optional dependencies being pushed on to the choice stack of each service consumer module this potentially could be done once per service interface (although will likely change what solutions are produced), but might be very tricky to implement given that service provider modules may also be service consumer modules. Good diagnostics are required to help track down why a service provider module is not being resolved. > > Paul. > > [1] http://openjdk.java.net/projects/jigsaw/doc/openjdk-jigsaw-modular-services.pdf -- - DML From jesse.glick at oracle.com Tue Jun 12 13:46:49 2012 From: jesse.glick at oracle.com (Jesse Glick) Date: Tue, 12 Jun 2012 16:46:49 -0400 Subject: ServiceLoader.load* take 2 In-Reply-To: <4FD7A5A7.4030401@redhat.com> References: <4FD7A5A7.4030401@redhat.com> Message-ID: <4FD7AAB9.8030704@oracle.com> On 06/12/2012 04:25 PM, David M. Lloyd wrote: > the existing service loader mechanism works well with multiple class loaders as is *If* the TCCL is set to a special value which can "see into" all modules in the configuration, rather than just being the module loader for the "main" module. That is possible, given a ClassLoader.getResources variant mentioned by me earlier. But not necessarily less hackish, and probably trickier to optimize than what Paul's webrev has. Anyway I think the observed behavior would be identical for most cases. From Alan.Bateman at oracle.com Tue Jun 12 13:57:02 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 12 Jun 2012 21:57:02 +0100 Subject: ServiceLoader.load* take 2 In-Reply-To: <4FD7A5A7.4030401@redhat.com> References: <4FD7A5A7.4030401@redhat.com> Message-ID: <4FD7AD1E.7080703@oracle.com> On 12/06/2012 21:25, David M. Lloyd wrote: > I have a high level of concern about "tactical and temporary > solutions" and things that "might often" work. Hang on, the point there is that we had to put in a few hacks to keep things working, and with the proposal and the changes that Paul has prepared, then those hacks come out. -Alan. From david.lloyd at redhat.com Tue Jun 12 14:01:24 2012 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 12 Jun 2012 16:01:24 -0500 Subject: ServiceLoader.load* take 2 In-Reply-To: <4FD7AAB9.8030704@oracle.com> References: <4FD7A5A7.4030401@redhat.com> <4FD7AAB9.8030704@oracle.com> Message-ID: <4FD7AE24.1040707@redhat.com> On 06/12/2012 03:46 PM, Jesse Glick wrote: > On 06/12/2012 04:25 PM, David M. Lloyd wrote: >> the existing service loader mechanism works well with multiple class >> loaders as is > > *If* the TCCL is set to a special value which can "see into" all modules > in the configuration, rather than just being the module loader for the > "main" module. Well this is just a difference of expectation. My expectation is that TCCL won't have global visibility. In fact my expectation is that *nothing* will have global visibility. I don't know what this "seeing into all modules" business is supposed to mean or how it's supposed to work. If you're talking about full dependency tree traversal, it's not clear to me why this is desirable; it does seem to me like undesired (private) implementations might pop in though. > That is possible, given a ClassLoader.getResources > variant mentioned by me earlier. But not necessarily less hackish, and > probably trickier to optimize than what Paul's webrev has. Anyway I > think the observed behavior would be identical for most cases. I just don't understand what problem is being solved here. It seems like a specific behavior is the goal, but what is the purpose of the behavior? How does it relate to current real-world libraries and applications (including third-party stuff)? -- - DML From jesse.glick at oracle.com Tue Jun 12 14:16:33 2012 From: jesse.glick at oracle.com (Jesse Glick) Date: Tue, 12 Jun 2012 17:16:33 -0400 Subject: ServiceLoader.load* take 2 In-Reply-To: <4FD7AE24.1040707@redhat.com> References: <4FD7A5A7.4030401@redhat.com> <4FD7AAB9.8030704@oracle.com> <4FD7AE24.1040707@redhat.com> Message-ID: <4FD7B1B1.3030501@oracle.com> On 06/12/2012 05:01 PM, David M. Lloyd wrote: > It seems like a specific behavior is the goal, but what is the purpose of the behavior? That the customary invocation of ServiceLoader load service provider classes from all eligible modules in the current configuration, whether or not the caller's module and/or the "main module" happen to have a direct or indirect dependency on that provider module. If you want this behavior and want to use the original SL behavior based on TCCL, then this TCCL has to return every possible result from getResources("META-INF/services/svc.Name") and be able to load every class thus named - or, given my proposed new API in ClassLoader, has to return every possible (URL,ClassLoader) pair from all loaded modules. A regular module class loader would not have such behavior - it would only "see" resources & classes from the associated module and exported packages from its declared dependencies. From david.lloyd at redhat.com Tue Jun 12 14:34:38 2012 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 12 Jun 2012 16:34:38 -0500 Subject: ServiceLoader.load* take 2 In-Reply-To: <4FD7B1B1.3030501@oracle.com> References: <4FD7A5A7.4030401@redhat.com> <4FD7AAB9.8030704@oracle.com> <4FD7AE24.1040707@redhat.com> <4FD7B1B1.3030501@oracle.com> Message-ID: <4FD7B5EE.9020302@redhat.com> On 06/12/2012 04:16 PM, Jesse Glick wrote: > On 06/12/2012 05:01 PM, David M. Lloyd wrote: >> It seems like a specific behavior is the goal, but what is the purpose >> of the behavior? > > That the customary invocation of ServiceLoader load service provider > classes from all eligible modules in the current configuration, whether > or not the caller's module and/or the "main module" happen to have a > direct or indirect dependency on that provider module. What defines "eligible"? Is there an expectation that a configuration will exist per-application, so that applications do not tend to have overlapping modules? Does this mean that the opinion of Oracle is that service implementations should generally be global per installation? -- - DML From jesse.glick at oracle.com Tue Jun 12 14:42:51 2012 From: jesse.glick at oracle.com (Jesse Glick) Date: Tue, 12 Jun 2012 17:42:51 -0400 Subject: ServiceLoader.load* take 2 In-Reply-To: <4FD7B5EE.9020302@redhat.com> References: <4FD7A5A7.4030401@redhat.com> <4FD7AAB9.8030704@oracle.com> <4FD7AE24.1040707@redhat.com> <4FD7B1B1.3030501@oracle.com> <4FD7B5EE.9020302@redhat.com> Message-ID: <4FD7B7DB.3090207@oracle.com> On 06/12/2012 05:34 PM, David M. Lloyd wrote: > Is there an expectation that a configuration will exist per-application Yes, I think that is the intent. > so that applications do not tend to have overlapping modules? They could certainly overlap in contained modules, but a given JVM (or isolate??) would be running one configuration with a particular set of modules in it, according to the application's needs. > Does this mean that [...] service implementations should generally be global per installation? No, I think just global per _application_. In other words, given a service package api; public interface ImageFileReader { boolean supports(Path image); Image read(Path image) throws IOException; } you would expect that any module in a given application calling for (ImageFileReader r : ServiceLoader.load(ImageFileReader.class)) {...} would get the same list of services. This would be true in classpath mode simply because the TCCL is always going to be ClassLoader.getSystemClassLoader(), i.e. JARs dumped in the -classpath. From chris.hegarty at oracle.com Wed Jun 13 03:42:20 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 13 Jun 2012 11:42:20 +0100 Subject: Module file parse API Message-ID: <4FD86E8C.3080208@oracle.com> Hi, A while back it was suggested that it may be useful to define a pull parser type API for reading from a module file. I gives separation of the parsing of the module file from the writing of it (either installing into the module library, or extraction elsewhere ). I've prototyped such an API and updated the ModuleFile.Reader to use it. The first four files in the webrev contain the API, its implementation, and a test that demonstrates some sample usage ( extraction of one or more classes, listing sections, verifying hashes ). The remainder of the changes simply updates some of the tools for installing and handling signed modules. http://cr.openjdk.java.net/~chegar/jigsaw/moduleParser_webrev.00/webrev/ Thanks, -Chris. From erik.joelsson at oracle.com Wed Jun 13 04:33:24 2012 From: erik.joelsson at oracle.com (Erik Joelsson) Date: Wed, 13 Jun 2012 13:33:24 +0200 Subject: source code re-organization In-Reply-To: <4FD2026C.5030901@oracle.com> References: <4FD0C398.1050300@oracle.com> <4FD1F42D.5040900@oracle.com> <4FD2026C.5030901@oracle.com> Message-ID: <4FD87A84.7050908@oracle.com> Hello, It would be great if this could go in anytime soon. For my needs it doesn't need to be perfect from the start, we will just tweak it as we go. I'm currently working with the patch applied locally and have started adding to it, but would like to not diverge too far. /Erik On 2012-06-08 15:47, Michael McMahon wrote: > Thanks Erik. All good points. I'll make these updates as suggested. I > moved the script into a fixed location > late in the day, so the environment variable is not required any more. > > - Michael > > On 08/06/12 13:46, Erik Joelsson wrote: >> Great to see this getting ready! I'm trying it out in preparation for >> working on the build-infra-jigsaw build. Noticed a couple of things: >> >> * Why the need to set MAKESHUFFLE_SRCDIR? I put the following in >> makeshufflescripts and it seems to work: >> MS=$(cd "$(dirname "$0")" && pwd) >> >> * In non-standard.includes, the line for jdwpTransport.h has an extra >> jdk/newsrc in the target path. >> >> * zipfs.shuffle, zipfs.unshuffle and non-standard.includes contains >> target paths that look like "jdk/newsrc/$platform/module/...". I >> assume these should to be changed to mclasses and etc. >> >> /Erik >> >> On 2012-06-07 17:07, Michael McMahon wrote: >>> Hi, >>> >>> The webrev below contains a utility which re-organizes the jdk8 >>> source code into one possible >>> modularized layout suitable for jigsaw. The way it works is that >>> first a meta script called 'makeshufflescripts' >>> is run which looks at the current layout and generates two further >>> sets of scripts (so-called 'shuffle' and >>> 'unshuffle' scripts). These scripts contain a set of 'rsync' >>> commands that copy changed files between the old >>> and the new layouts. 'shuffle' copies from the old to the new, and >>> 'unshuffle' copies any modified files >>> from the new layout back to the old/present jdk source layout. The >>> script operates on the jdk, langtools >>> corba, jaxws and jaxp repositories copying from the 'src' sub-tree >>> in each to/from a 'newsrc' sub-tree >>> containing the new layout. >>> >>> The purpose of the utility is for people to experiment with the new >>> layout, editing source files etc. >>> Changed files can be 'unshuffled' back to the old layout, for >>> building. There is a README which explains >>> how exactly to use it. Currently, it has only been tested on Linux. >>> >>> http://cr.openjdk.java.net/~michaelm/shuffle/1/webrev/ >>> >>> As mentioned above, the main script in the utility is >>> 'makeshufflescripts'. Most of the other files >>> in the webrev contain other information needed by the script. >>> 'makeshufflescripts' generates the shuffle/unshuffle >>> scripts into a directory called shufflescripts off the root of the >>> jigsaw forest. >>> To do a shuffle, execute "sh shufflescripts/shuffle.sh" and to do an >>> unshuffle "sh shufflescripts/unshuffle.sh" >>> >>> All comments/questions welcome. >>> >>> - Michael. > From Alan.Bateman at oracle.com Wed Jun 13 04:55:05 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 13 Jun 2012 12:55:05 +0100 Subject: source code re-organization In-Reply-To: <4FD87A84.7050908@oracle.com> References: <4FD0C398.1050300@oracle.com> <4FD1F42D.5040900@oracle.com> <4FD2026C.5030901@oracle.com> <4FD87A84.7050908@oracle.com> Message-ID: <4FD87F99.8080105@oracle.com> On 13/06/2012 12:33, Erik Joelsson wrote: > Hello, > > It would be great if this could go in anytime soon. For my needs it > doesn't need to be perfect from the start, we will just tweak it as we > go. I'm currently working with the patch applied locally and have > started adding to it, but would like to not diverge too far. > > /Erik I've also been using it and don't have any objection to it going in. I did find a few issues with the original version that Michael published but he has since fixed them. I don't know if that is the reason for your patch. So assume I'll sponsor this (as Michael doesn't isn't a commtitter on this project) and get it in soon. -Alan From sean.mullan at oracle.com Wed Jun 13 06:00:46 2012 From: sean.mullan at oracle.com (Sean Mullan) Date: Wed, 13 Jun 2012 09:00:46 -0400 Subject: Class loading error when looking up security providers Message-ID: <4FD88EFE.8000406@oracle.com> I'm seeing several ClassNotFoundException errors when some of the security providers are loaded, ex: (using java.security.debug trace): ProviderConfig: Error loading provider com.sun.crypto.provider.SunJCE java.lang.ClassNotFoundException: com.sun.crypto.provider.SunJCE : requested by +jdk.auth+jdk.base+jdk.desktop+jdk.jndi+jdk.prefs+jdk.tls+sun.charsets+sun.localedata+sun.resources at org.openjdk.jigsaw.Loader.loadClass(Loader.java:116) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:231) at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:207) at java.security.AccessController.doPrivileged(Native Method) at sun.security.jca.ProviderConfig.doLoadProvider(ProviderConfig.java:207) at sun.security.jca.ProviderConfig.getProvider(ProviderConfig.java:188) at sun.security.jca.ProviderList.getProvider(ProviderList.java:233) at sun.security.jca.ProviderList.getIndex(ProviderList.java:263) at sun.security.jca.ProviderList.getProviderConfig(ProviderList.java:247) at sun.security.jca.ProviderList.getProvider(ProviderList.java:253) at java.security.Security.getProvider(Security.java:472) at javax.crypto.Cipher.getInstance(Cipher.java:587) at test.CipherTest.main(CipherTest.java:10) I'm looking into this right now, but if anyone has any tips on how to debug further let me know. Thanks, Sean From paul.sandoz at oracle.com Wed Jun 13 06:58:03 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 13 Jun 2012 15:58:03 +0200 Subject: Class loading error when looking up security providers In-Reply-To: <4FD88EFE.8000406@oracle.com> References: <4FD88EFE.8000406@oracle.com> Message-ID: Hi Sean, Looking at the code the problem is the class "com.sun.crypto.provider.SunJCE" is in the jdk.crypto module and the code to load crypto providers does: ClassLoader cl; if (Platform.isModuleMode()) { cl = Platform.getBaseModuleLoader(); } else { cl = ClassLoader.getSystemClassLoader(); } Class provClass; if (cl != null) { provClass = cl.loadClass(className); } else { provClass = Class.forName(className); } So it is using the CL of the jdk.base module, which cannot see classes from the jdk.crypto module. As the comment in the source that occurs above the above snippet states, this area needs to start using services. For a build of Jigsaw you can find which class belongs to which module by grep'ing the files in build//moduleinfo/classlist for the class name. Paul. On Jun 13, 2012, at 3:00 PM, Sean Mullan wrote: > I'm seeing several ClassNotFoundException errors when some of the security > providers are loaded, ex: (using java.security.debug trace): > > ProviderConfig: Error loading provider com.sun.crypto.provider.SunJCE > java.lang.ClassNotFoundException: com.sun.crypto.provider.SunJCE : requested by > +jdk.auth+jdk.base+jdk.desktop+jdk.jndi+jdk.prefs+jdk.tls+sun.charsets+sun.localedata+sun.resources > at org.openjdk.jigsaw.Loader.loadClass(Loader.java:116) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:231) > at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:207) > at java.security.AccessController.doPrivileged(Native Method) > at sun.security.jca.ProviderConfig.doLoadProvider(ProviderConfig.java:207) > at sun.security.jca.ProviderConfig.getProvider(ProviderConfig.java:188) > at sun.security.jca.ProviderList.getProvider(ProviderList.java:233) > at sun.security.jca.ProviderList.getIndex(ProviderList.java:263) > at sun.security.jca.ProviderList.getProviderConfig(ProviderList.java:247) > at sun.security.jca.ProviderList.getProvider(ProviderList.java:253) > at java.security.Security.getProvider(Security.java:472) > at javax.crypto.Cipher.getInstance(Cipher.java:587) > at test.CipherTest.main(CipherTest.java:10) > > I'm looking into this right now, but if anyone has any tips on how to debug > further let me know. > > Thanks, > Sean From paul.sandoz at oracle.com Wed Jun 13 07:00:45 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 13 Jun 2012 16:00:45 +0200 Subject: Additional JAX-WS/JAXB services, issue with declaring service provider class Message-ID: Hi, There are three more internal services that need to be declared for JAX-WS/JAXB: --- jdk $ hg diff diff -r 5185aec6cc3d make/modules/jdk.depconfig --- a/make/modules/jdk.depconfig Mon Jun 11 15:28:01 2012 +0200 +++ b/make/modules/jdk.depconfig Wed Jun 13 15:59:18 2012 +0200 @@ -543,6 +543,15 @@ javax.xml.soap.SOAPFactory -> META-INF/services/javax.xml.soap.SOAPFactory @Provider(optional) + com.sun.tools.internal.ws.wscompile.WsimportOptions -> META-INF/services/com.sun.tools.internal.ws.wscompile.Plugin + + at Provider(optional) + com.sun.tools.internal.xjc.Options -> META-INF/services/com.sun.tools.internal.xjc.Plugin + + at Provider(optional) + com.sun.xml.internal.ws.spi.db.BindingContextFactory -> META-INF/services/com.sun.xml.internal.ws.spi.db.BindingContextFactory + + at Provider(optional) javax.sound.midi.spi.MidiDeviceProvider -> META-INF/services/javax.sound.midi.spi.MidiDeviceProvider @Provider(optional) --- However com.sun.xml.internal.ws.spi.db.BindingContextFactory is problematic. The following is generated in the jdk.jaxws module: module jdk.jaxws @ 8-ea { ... requires optional service com.sun.xml.internal.ws.spi.db.BindingContextFactory; ... view jdk.jaxws.internal { ... exports com.sun.xml.internal.ws.spi.db; ... } } But there is no service provider class declaration in the jdk.jaxws.internal view: providers service com.sun.xml.internal.ws.spi.db.BindingContextFactory with com.sun.xml.internal.ws.db.glassfish.JAXBRIContextFactory; Is this an issue with the generation of module-info files or do i need to declare some additional information in one of the files in jdk/make/modules? Paul. From Alan.Bateman at oracle.com Wed Jun 13 07:06:34 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 13 Jun 2012 15:06:34 +0100 Subject: Class loading error when looking up security providers In-Reply-To: <4FD88EFE.8000406@oracle.com> References: <4FD88EFE.8000406@oracle.com> Message-ID: <4FD89E6A.5040308@oracle.com> On 13/06/2012 14:00, Sean Mullan wrote: > I'm seeing several ClassNotFoundException errors when some of the security > providers are loaded, ex: (using java.security.debug trace): > > ProviderConfig: Error loading provider com.sun.crypto.provider.SunJCE > java.lang.ClassNotFoundException: com.sun.crypto.provider.SunJCE : requested by > +jdk.auth+jdk.base+jdk.desktop+jdk.jndi+jdk.prefs+jdk.tls+sun.charsets+sun.localedata+sun.resources > at org.openjdk.jigsaw.Loader.loadClass(Loader.java:116) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:231) > at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:207) > at java.security.AccessController.doPrivileged(Native Method) > at sun.security.jca.ProviderConfig.doLoadProvider(ProviderConfig.java:207) > at sun.security.jca.ProviderConfig.getProvider(ProviderConfig.java:188) > at sun.security.jca.ProviderList.getProvider(ProviderList.java:233) > at sun.security.jca.ProviderList.getIndex(ProviderList.java:263) > at sun.security.jca.ProviderList.getProviderConfig(ProviderList.java:247) > at sun.security.jca.ProviderList.getProvider(ProviderList.java:253) > at java.security.Security.getProvider(Security.java:472) > at javax.crypto.Cipher.getInstance(Cipher.java:587) > at test.CipherTest.main(CipherTest.java:10) > > I'm looking into this right now, but if anyone has any tips on how to debug > further let me know. > The JCA code is using the system class loader so it doesn't make sense when running with modules. I think this security code will need to changed to load the security provider as services. -Alan. From sean.mullan at oracle.com Wed Jun 13 08:10:39 2012 From: sean.mullan at oracle.com (Sean Mullan) Date: Wed, 13 Jun 2012 11:10:39 -0400 Subject: Class loading error when looking up security providers In-Reply-To: References: <4FD88EFE.8000406@oracle.com> Message-ID: <4FD8AD6F.1080605@oracle.com> On 6/13/12 9:58 AM, Paul Sandoz wrote: > Hi Sean, > > Looking at the code the problem is the class "com.sun.crypto.provider.SunJCE" is in the jdk.crypto module and the code to load crypto providers does: > > ClassLoader cl; > if (Platform.isModuleMode()) { > cl = Platform.getBaseModuleLoader(); > } else { > cl = ClassLoader.getSystemClassLoader(); > } > Class provClass; > if (cl != null) { > provClass = cl.loadClass(className); > } else { > provClass = Class.forName(className); > } > > So it is using the CL of the jdk.base module, which cannot see classes from the jdk.crypto module. > > As the comment in the source that occurs above the above snippet states, this area needs to start using services. Ok, but I need a workaround for the time-being. If you think of anything, let me know. --Sean From Alan.Bateman at oracle.com Wed Jun 13 08:12:38 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 13 Jun 2012 16:12:38 +0100 Subject: Additional JAX-WS/JAXB services, issue with declaring service provider class In-Reply-To: References: Message-ID: <4FD8ADE6.3000009@oracle.com> On 13/06/2012 15:00, Paul Sandoz wrote: > : > > However com.sun.xml.internal.ws.spi.db.BindingContextFactory is problematic. > > The following is generated in the jdk.jaxws module: > > module jdk.jaxws @ 8-ea { > ... > requires optional service com.sun.xml.internal.ws.spi.db.BindingContextFactory; > ... > > view jdk.jaxws.internal { > ... > exports com.sun.xml.internal.ws.spi.db; > ... > } > } > > But there is no service provider class declaration in the jdk.jaxws.internal view: > > providers service com.sun.xml.internal.ws.spi.db.BindingContextFactory with > com.sun.xml.internal.ws.db.glassfish.JAXBRIContextFactory; > > > Is this an issue with the generation of module-info files or do i need to declare some additional information in one of the files in jdk/make/modules? > > Paul. I'm happy to help get the jdk.depconfig changes in. Is there a META-INF/services/com.sun.xml.internal.ws.spi.db.BindingContextFactory that lists com.sun.xml.internal.ws.db.glassfish.JAXBRIContextFactory? That probably explains why it's not in the generated module-info. As an aside, if permits doesn't make sense with services, then I assume this eliminates the needs a non-default view to provide a service, the provides service declaration only needs to be in the default view. -Alan. From paul.sandoz at oracle.com Wed Jun 13 08:34:33 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 13 Jun 2012 17:34:33 +0200 Subject: Additional JAX-WS/JAXB services, issue with declaring service provider class In-Reply-To: <4FD8ADE6.3000009@oracle.com> References: <4FD8ADE6.3000009@oracle.com> Message-ID: On Jun 13, 2012, at 5:12 PM, Alan Bateman wrote: > On 13/06/2012 15:00, Paul Sandoz wrote: >> : >> >> However com.sun.xml.internal.ws.spi.db.BindingContextFactory is problematic. >> >> The following is generated in the jdk.jaxws module: >> >> module jdk.jaxws @ 8-ea { >> ... >> requires optional service com.sun.xml.internal.ws.spi.db.BindingContextFactory; >> ... >> >> view jdk.jaxws.internal { >> ... >> exports com.sun.xml.internal.ws.spi.db; >> ... >> } >> } >> >> But there is no service provider class declaration in the jdk.jaxws.internal view: >> >> providers service com.sun.xml.internal.ws.spi.db.BindingContextFactory with >> com.sun.xml.internal.ws.db.glassfish.JAXBRIContextFactory; >> >> >> Is this an issue with the generation of module-info files or do i need to declare some additional information in one of the files in jdk/make/modules? >> >> Paul. > I'm happy to help get the jdk.depconfig changes in. > > Is there a META-INF/services/com.sun.xml.internal.ws.spi.db.BindingContextFactory that lists com.sun.xml.internal.ws.db.glassfish.JAXBRIContextFactory? Yes there is, see: jaxws/src/share/jaxws_classes/com/sun/tools/etc/META-INF/services > That probably explains why it's not in the generated module-info. > > As an aside, if permits doesn't make sense with services, then I assume this eliminates the needs a non-default view to provide a service, the provides service declaration only needs to be in the default view. > Yes. I wonder if the view is confusing the module file generation code? Paul. From paul.sandoz at oracle.com Wed Jun 13 08:55:18 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 13 Jun 2012 17:55:18 +0200 Subject: Class loading error when looking up security providers In-Reply-To: <4FD8AD6F.1080605@oracle.com> References: <4FD88EFE.8000406@oracle.com> <4FD8AD6F.1080605@oracle.com> Message-ID: On Jun 13, 2012, at 5:10 PM, Sean Mullan wrote: > > On 6/13/12 9:58 AM, Paul Sandoz wrote: >> Hi Sean, >> >> Looking at the code the problem is the class "com.sun.crypto.provider.SunJCE" is in the jdk.crypto module and the code to load crypto providers does: >> >> ClassLoader cl; >> if (Platform.isModuleMode()) { >> cl = Platform.getBaseModuleLoader(); >> } else { >> cl = ClassLoader.getSystemClassLoader(); >> } >> Class provClass; >> if (cl != null) { >> provClass = cl.loadClass(className); >> } else { >> provClass = Class.forName(className); >> } >> >> So it is using the CL of the jdk.base module, which cannot see classes from the jdk.crypto module. >> >> As the comment in the source that occurs above the above snippet states, this area needs to start using services. > > Ok, but I need a workaround for the time-being. If you think of anything, let me > know. > Hmm... you might be able to muck around with the platform library and copy the classes from the jdk.crypto location into the jdk.base location then refresh the library. Paul. From Alan.Bateman at oracle.com Wed Jun 13 08:55:21 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 13 Jun 2012 16:55:21 +0100 Subject: Additional JAX-WS/JAXB services, issue with declaring service provider class In-Reply-To: References: <4FD8ADE6.3000009@oracle.com> Message-ID: <4FD8B7E9.1090802@oracle.com> On 13/06/2012 16:34, Paul Sandoz wrote: > On Jun 13, 2012, at 5:12 PM, Alan Bateman wrote: > >> : >> >> Is there a META-INF/services/com.sun.xml.internal.ws.spi.db.BindingContextFactory that lists com.sun.xml.internal.ws.db.glassfish.JAXBRIContextFactory? > Yes there is, see: > > jaxws/src/share/jaxws_classes/com/sun/tools/etc/META-INF/services I think this is a build issue and com.sun.xml.internal.ws.spi.db.BindingContextFactory is not being copied into the $OUTPUTDIR/META-INF/services during the build. I just checked rt.jar from a regular build and I don't see it in META-INF/services either so this is specific to jigsaw/jigsaw. It might be worth checking with the JAX-WS folks as to whether this should even be there, especially given the "internal" in the package name. -Alan. From sean.mullan at oracle.com Wed Jun 13 09:26:52 2012 From: sean.mullan at oracle.com (Sean Mullan) Date: Wed, 13 Jun 2012 12:26:52 -0400 Subject: Class loading error when looking up security providers In-Reply-To: References: <4FD88EFE.8000406@oracle.com> <4FD8AD6F.1080605@oracle.com> Message-ID: <4FD8BF4C.9040403@oracle.com> On 6/13/12 11:55 AM, Paul Sandoz wrote: > > On Jun 13, 2012, at 5:10 PM, Sean Mullan wrote: > >> >> On 6/13/12 9:58 AM, Paul Sandoz wrote: >>> Hi Sean, >>> >>> Looking at the code the problem is the class "com.sun.crypto.provider.SunJCE" is in the jdk.crypto module and the code to load crypto providers does: >>> >>> ClassLoader cl; >>> if (Platform.isModuleMode()) { >>> cl = Platform.getBaseModuleLoader(); >>> } else { >>> cl = ClassLoader.getSystemClassLoader(); >>> } >>> Class provClass; >>> if (cl != null) { >>> provClass = cl.loadClass(className); >>> } else { >>> provClass = Class.forName(className); >>> } >>> >>> So it is using the CL of the jdk.base module, which cannot see classes from the jdk.crypto module. >>> >>> As the comment in the source that occurs above the above snippet states, this area needs to start using services. >> >> Ok, but I need a workaround for the time-being. If you think of anything, let me >> know. >> > > Hmm... you might be able to muck around with the platform library and copy the classes from the jdk.crypto location into the jdk.base location then refresh the library. One workaround is to instantiate and specify the provider class directly, ex: Cipher.getInstance("AES", new com.sun.crypto.provider.SunJCE()); and make sure you add a "requires jdk.crypto.internal" to module-info.java, since SunJCE is now exported via the jdk.crypto.internal view: jdk.crypto at 8-ea requires jdk.base.internal@=8-ea requires jdk.tls.internal@=8-ea requires synthesized java.base@>=8 view jdk.crypto.internal exports com.sun.crypto.provider --Sean From sean.mullan at oracle.com Wed Jun 13 12:51:35 2012 From: sean.mullan at oracle.com (Sean Mullan) Date: Wed, 13 Jun 2012 15:51:35 -0400 Subject: Classpath mode class loading question Message-ID: <4FD8EF47.7000206@oracle.com> In classpath mode, I need to find out if a class has been loaded from the system library, as opposed to from the CLASSPATH. Is there an easy way to find that out? It's probably a simple question for someone, so thought I would ask it while I continue experimenting with different solutions. Thanks, Sean From mandy.chung at oracle.com Wed Jun 13 13:51:29 2012 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 13 Jun 2012 13:51:29 -0700 Subject: Classpath mode class loading question In-Reply-To: <4FD8EF47.7000206@oracle.com> References: <4FD8EF47.7000206@oracle.com> Message-ID: <4FD8FD51.1040106@oracle.com> On 6/13/2012 12:51 PM, Sean Mullan wrote: > In classpath mode, I need to find out if a class has been loaded from the system > library, as opposed to from the CLASSPATH. Is there an easy way to find that out? Would the classloader be adequate for your use (null vs non-null)? A class on CLASSPATH is loaded by the system class loader. What do you want to do with this check? Just to be clear: The current prototype puts all modules in the bootclasspath in classpath mode as a temporary hack. Karen and I are working on the classpath mode support to replace the -Xbootclasspath hack. Mandy > It's probably a simple question for someone, so thought I would ask it while I > continue experimenting with different solutions. > > Thanks, > Sean From sean.mullan at oracle.com Wed Jun 13 14:13:23 2012 From: sean.mullan at oracle.com (Sean Mullan) Date: Wed, 13 Jun 2012 17:13:23 -0400 Subject: Classpath mode class loading question In-Reply-To: <4FD8FD51.1040106@oracle.com> References: <4FD8EF47.7000206@oracle.com> <4FD8FD51.1040106@oracle.com> Message-ID: <4FD90273.9040301@oracle.com> On 6/13/12 4:51 PM, Mandy Chung wrote: > On 6/13/2012 12:51 PM, Sean Mullan wrote: >> In classpath mode, I need to find out if a class has been loaded from the system >> library, as opposed to from the CLASSPATH. Is there an easy way to find that out? > > Would the classloader be adequate for your use (null vs non-null)? A > class on CLASSPATH is loaded by the system class loader. What do you > want to do with this check? I'm working on adjusting our JCE framework code to verify a JCE provider. I need to know if the provider is loaded from the system module library (i.e. one of the JCE providers bundled with the JDK), as opposed to say, a 3rd party provider jar. The JCE framework currently rejects any providers loaded from the bootclasspath. > Just to be clear: The current prototype puts all modules in the > bootclasspath in classpath mode as a temporary hack. Karen and I are > working on the classpath mode support to replace the -Xbootclasspath hack. Ah, maybe I will wait for that then. Thanks, Sean From paul.sandoz at oracle.com Thu Jun 14 07:31:38 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 14 Jun 2012 16:31:38 +0200 Subject: Patch for additional JAX-WS/JAXB services Message-ID: Hi, Please see below for a patch to declare some additional JAX-WS/JAXB services. I checked with the JAX-WS team that the BindingContextFactory is not relevant and the services file should not really be distributed with the source for the JDK. Paul. diff -r 5185aec6cc3d make/modules/jdk.depconfig --- a/make/modules/jdk.depconfig Mon Jun 11 15:28:01 2012 +0200 +++ b/make/modules/jdk.depconfig Thu Jun 14 16:27:29 2012 +0200 @@ -543,6 +543,12 @@ javax.xml.soap.SOAPFactory -> META-INF/services/javax.xml.soap.SOAPFactory @Provider(optional) + com.sun.tools.internal.ws.wscompile.WsimportOptions -> META-INF/services/com.sun.tools.internal.ws.wscompile.Plugin + + at Provider(optional) + com.sun.tools.internal.xjc.Options -> META-INF/services/com.sun.tools.internal.xjc.Plugin + + at Provider(optional) javax.sound.midi.spi.MidiDeviceProvider -> META-INF/services/javax.sound.midi.spi.MidiDeviceProvider @Provider(optional) From Alan.Bateman at oracle.com Thu Jun 14 07:49:07 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 14 Jun 2012 15:49:07 +0100 Subject: Patch for additional JAX-WS/JAXB services In-Reply-To: References: Message-ID: <4FD9F9E3.4000806@oracle.com> On 14/06/2012 15:31, Paul Sandoz wrote: > Hi, > > Please see below for a patch to declare some additional JAX-WS/JAXB services. > > I checked with the JAX-WS team that the BindingContextFactory is not relevant and the services file should not really be distributed with the source for the JDK. > > Paul. Thanks for checking, the "internal" in the name was suspicious. I'll push the change to jdk.depconfig now. -Alan From alan.bateman at oracle.com Thu Jun 14 08:57:14 2012 From: alan.bateman at oracle.com (alan.bateman at oracle.com) Date: Thu, 14 Jun 2012 15:57:14 +0000 Subject: hg: jigsaw/jigsaw: Initial load of scripts to support source code shuffle/unshuffle Message-ID: <20120614155715.061534790F@hg.openjdk.java.net> Changeset: c5f8b970c983 Author: michaelm Date: 2012-06-14 16:55 +0100 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/rev/c5f8b970c983 Initial load of scripts to support source code shuffle/unshuffle + make/scripts/shuffle/CheckNative.java + make/scripts/shuffle/Compare.java + make/scripts/shuffle/README + make/scripts/shuffle/Reverse.java + make/scripts/shuffle/docompare + make/scripts/shuffle/getdifferences + make/scripts/shuffle/jdk.base.includes + make/scripts/shuffle/jdk.corba.includes + make/scripts/shuffle/jdk.desktop.includes + make/scripts/shuffle/jdk.jaxp.includes + make/scripts/shuffle/jdk.jaxws.includes + make/scripts/shuffle/jdk.jaxws.tools.includes + make/scripts/shuffle/jdk.jdbc.includes + make/scripts/shuffle/jdk.jx.annotations.includes + make/scripts/shuffle/jdk.scripting.includes + make/scripts/shuffle/jdk.tools.includes + make/scripts/shuffle/makeshufflescripts + make/scripts/shuffle/moduleinfo.others/classlist/jdk.auth.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.base.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.compat.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.compiler.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.corba.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.cosnaming.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.crypto.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.deploy.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.desktop.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.devtools.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.httpserver.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.instrument.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.jaxp.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.jaxws.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.jdbc.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.jdbc.rowset.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.jndi.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.jta.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.jx.annotations.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.kerberos.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.logging.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.management.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.management.iiop.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.prefs.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.rmi.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.scripting.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.sctp.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.security.acl.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.smartcardio.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.snmp.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.sunec.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.sunmscapi.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.sunpkcs11.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.tls.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.tools.base.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.tools.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.tools.jaxws.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.tools.jre.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.ucrypto.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.xmldsig.classlist + make/scripts/shuffle/moduleinfo.others/classlist/jdk.zipfs.classlist + make/scripts/shuffle/moduleinfo.others/classlist/sun.charsets.classlist + make/scripts/shuffle/moduleinfo.others/classlist/sun.localedata.classlist + make/scripts/shuffle/moduleinfo.others/classlist/sun.resources.classlist + make/scripts/shuffle/moduleinfo.others/shuffledata/native.dirs + make/scripts/shuffle/non-standard.includes + make/scripts/shuffle/shuffle.sh + make/scripts/shuffle/sun.localedata.includes + make/scripts/shuffle/unshuffle.sh + make/scripts/shuffle/zipfs.shuffle + make/scripts/shuffle/zipfs.unshuffle From alan.bateman at oracle.com Thu Jun 14 09:11:21 2012 From: alan.bateman at oracle.com (alan.bateman at oracle.com) Date: Thu, 14 Jun 2012 16:11:21 +0000 Subject: hg: jigsaw/jigsaw/jdk: jdk.tools.jaxws missing requires service com.sun.tools.internal.xjc.Plugin Message-ID: <20120614161239.379F147914@hg.openjdk.java.net> Changeset: 2bb67319c588 Author: psandoz Date: 2012-06-14 17:10 +0100 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/2bb67319c588 jdk.tools.jaxws missing requires service com.sun.tools.internal.xjc.Plugin ! make/modules/jdk.depconfig From sean.mullan at oracle.com Thu Jun 14 11:09:25 2012 From: sean.mullan at oracle.com (Sean Mullan) Date: Thu, 14 Jun 2012 14:09:25 -0400 Subject: Module file parse API In-Reply-To: <4FD86E8C.3080208@oracle.com> References: <4FD86E8C.3080208@oracle.com> Message-ID: <4FDA28D5.9040204@oracle.com> Hi Chris, I am going to review this, at least the changes to the security parts, but I may need a few days. --Sean On 06/13/2012 06:42 AM, Chris Hegarty wrote: > Hi, > > A while back it was suggested that it may be useful to define a pull > parser type API for reading from a module file. I gives separation of > the parsing of the module file from the writing of it (either installing > into the module library, or extraction elsewhere ). > > I've prototyped such an API and updated the ModuleFile.Reader to use it. > The first four files in the webrev contain the API, its implementation, > and a test that demonstrates some sample usage ( extraction of one or > more classes, listing sections, verifying hashes ). The remainder of the > changes simply updates some of the tools for installing and handling > signed modules. > > http://cr.openjdk.java.net/~chegar/jigsaw/moduleParser_webrev.00/webrev/ > > Thanks, > -Chris. From Alan.Bateman at oracle.com Fri Jun 15 04:09:18 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 15 Jun 2012 12:09:18 +0100 Subject: ServiceLoader.load* take 2 In-Reply-To: References: Message-ID: <4FDB17DE.3070205@oracle.com> On 12/06/2012 16:03, Paul Sandoz wrote: > Hi, > > A set of slides has been created [1] which presents a set of terms, an overview of modular services, describes the current state of affairs with service instance creation and the use of ServiceLoader.load*, then proposes an alternative solution for ServiceLoader, the first part of which is implemented by the following webrev, which changes service instance creation so that it is scoped to the configuration rather than the class loader: > > http://cr.openjdk.java.net/~psandoz/jigsaw/global-services/webrev/ > > Summary: in module mode any module class loader can be used with ServiceLoader to create the service instances for a service interface. Previously the class loader of the service consumer module, that consumes the service interface, was required to be used. > I skimmed through the webrev and the changes look reasonable to me. It's good to have the hacks removed from ServiceLoader, as you know these were needed to avoid a pit stop to change the gear box. I'm also happy with that the proposal addresses the permits issue, that was a left-over open issue from the first installation. With the proposal then it's much simpler and is conceptually equivalent to have all the service providers on the class path. One thing I didn't quite get was the reason for moving LazyIterator1 out of ServiceLoader to a new org.openjdk.jigsaw.ModuleServiceLoader. My comment is really just wondering about the iteration/instantiation in two places rather than one. You listed in the "next steps" that the configuration needs to be updated to include a map from the service interface to the list of service provider modules that supply implementations and it would be good to do that (as it's regressing to have to check every context for service implementations). On the new test then it reminds me that we may be able to consolidate the tests in the services directory into a more complete set of unit tests (just thinking that there is a bit of duplication between the new test the existing hello.sh, many.sh and optional.sh). -Alan. From paul.sandoz at oracle.com Fri Jun 15 08:23:15 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Fri, 15 Jun 2012 17:23:15 +0200 Subject: ServiceLoader.load* take 2 In-Reply-To: <4FDB17DE.3070205@oracle.com> References: <4FDB17DE.3070205@oracle.com> Message-ID: On Jun 15, 2012, at 1:09 PM, Alan Bateman wrote: > On 12/06/2012 16:03, Paul Sandoz wrote: >> Hi, >> >> A set of slides has been created [1] which presents a set of terms, an overview of modular services, describes the current state of affairs with service instance creation and the use of ServiceLoader.load*, then proposes an alternative solution for ServiceLoader, the first part of which is implemented by the following webrev, which changes service instance creation so that it is scoped to the configuration rather than the class loader: >> >> http://cr.openjdk.java.net/~psandoz/jigsaw/global-services/webrev/ >> >> Summary: in module mode any module class loader can be used with ServiceLoader to create the service instances for a service interface. Previously the class loader of the service consumer module, that consumes the service interface, was required to be used. >> > I skimmed through the webrev and the changes look reasonable to me. It's good to have the hacks removed from ServiceLoader, as you know these were needed to avoid a pit stop to change the gear box. I'm also happy with that the proposal addresses the permits issue, that was a left-over open issue from the first installation. With the proposal then it's much simpler and is conceptually equivalent to have all the service providers on the class path. > Exactly. > One thing I didn't quite get was the reason for moving LazyIterator1 out of ServiceLoader to a new org.openjdk.jigsaw.ModuleServiceLoader. My comment is really just wondering about the iteration/instantiation in two places rather than one. > I wanted to separate concerns so that ServiceLoader did not know any specifics of modular services (and vice versa FWIW). It should hopefully keep any future changes located in the org.openjdk.jigsaw package. I forgot to do so but it means, for example, we can make Loader.findServices protected or package private. > You listed in the "next steps" that the configuration needs to be updated to include a map from the service interface to the list of service provider modules that supply implementations and it would be good to do that (as it's regressing to have to check every context for service implementations). > Yes, i hope you don't mind me doing that as second commit. My preference is to do the next step properly with a refactoring of Configuration/Context instead of one commit that may have created a lot of ancillary noise around the core the issue. > On the new test then it reminds me that we may be able to consolidate the tests in the services directory into a more complete set of unit tests (just thinking that there is a bit of duplication between the new test the existing hello.sh, many.sh and optional.sh). > What do you propose? They definitely need to be fleshed out. Personally i prefer things to be in exploded form, i find it easier to grok. Paul. From paul.sandoz at oracle.com Fri Jun 15 09:17:52 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Fri, 15 Jun 2012 18:17:52 +0200 Subject: ServiceLoader.load* take 2 In-Reply-To: <4FD7B7DB.3090207@oracle.com> References: <4FD7A5A7.4030401@redhat.com> <4FD7AAB9.8030704@oracle.com> <4FD7AE24.1040707@redhat.com> <4FD7B1B1.3030501@oracle.com> <4FD7B5EE.9020302@redhat.com> <4FD7B7DB.3090207@oracle.com> Message-ID: On Jun 12, 2012, at 11:42 PM, Jesse Glick wrote: > On 06/12/2012 05:34 PM, David M. Lloyd wrote: >> Is there an expectation that a configuration will exist per-application > > Yes, I think that is the intent. > Yes, one app == one config I don't know how familiar people are with the code, but if one is I suspect it is reasonably easy to grok the change in scope to service instance creation. >> so that applications do not tend to have overlapping modules? > > They could certainly overlap in contained modules, but a given JVM (or isolate??) would be running one configuration with a particular set of modules in it, according to the application's needs. > There is definitely a connection between an app/config and a container "hosting" the app that is isolated from other containers within the same JVM e.g. an analogy would be an app server running multiple web apps, where there are platform modules supplied by the app server and modules that are specific to the app. >> Does this mean that [...] service implementations should generally be global per installation? > > No, I think just global per _application_. In other words, given a service > > package api; > public interface ImageFileReader { > boolean supports(Path image); > Image read(Path image) throws IOException; > } > > you would expect that any module in a given application calling > > for (ImageFileReader r : ServiceLoader.load(ImageFileReader.class)) {...} > > would get the same list of services. This would be true in classpath mode simply because the TCCL is always going to be ClassLoader.getSystemClassLoader(), i.e. JARs dumped in the -classpath. +1 Paul. From jesse.glick at oracle.com Fri Jun 15 14:17:32 2012 From: jesse.glick at oracle.com (Jesse Glick) Date: Fri, 15 Jun 2012 17:17:32 -0400 Subject: Modularizing URLStreamHandlerFactory Message-ID: <4FDBA66C.1090703@oracle.com> Something to consider - in a big application it is not uncommon to have multiple URL handlers implemented in various places. Yet the JRE only lets you set a single URLStreamHandlerFactory for the whole JVM. So for the application to be modular, you need to write a proxy factory which accepts modular registrations, but that impedes interoperability since you must define a proprietary interface for this. It would be nicer for a modular JRE to allow URL handler factories to be registered as services. OSGi and the NetBeans module system both do something like this. Similar arguments could perhaps be made for other pluggable singletons in the current Java platform API (UncaughtExceptionHandler, FileNameMap, ProxySelector, HostnameVerifier, SSLSocketFactory, ...) but URLStreamHandlerFactory is the one which I know matters. From mark.reinhold at oracle.com Fri Jun 15 16:38:55 2012 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Fri, 15 Jun 2012 16:38:55 -0700 Subject: Modularizing URLStreamHandlerFactory In-Reply-To: jesse.glick@oracle.com; Fri, 15 Jun 2012 17:17:32 EDT; <4FDBA66C.1090703@oracle.com> Message-ID: <20120615233855.07E9FEBA@eggemoggin.niobe.net> 2012/6/15 14:17 -0700, jesse.glick at oracle.com: > ... > > It would be nicer for a modular JRE to allow URL handler factories to > be registered as services. OSGi and the NetBeans module system both do > something like this. > > Similar arguments could perhaps be made for other pluggable singletons in the > current Java platform API (UncaughtExceptionHandler, FileNameMap, > ProxySelector, HostnameVerifier, SSLSocketFactory, ...) but > URLStreamHandlerFactory is the one which I know matters. Agreed. We should evaluate all such extension points for potential retrofitting with services. - Mark From Alan.Bateman at oracle.com Sat Jun 16 08:56:14 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 16 Jun 2012 16:56:14 +0100 Subject: Modularizing URLStreamHandlerFactory In-Reply-To: <4FDBA66C.1090703@oracle.com> References: <4FDBA66C.1090703@oracle.com> Message-ID: <4FDCAC9E.5060700@oracle.com> On 15/06/2012 22:17, Jesse Glick wrote: > Something to consider - in a big application it is not uncommon to > have multiple URL handlers implemented in various places. Yet the JRE > only lets you set a single URLStreamHandlerFactory for the whole JVM. > So for the application to be modular, you need to write a proxy > factory which accepts modular registrations, but that impedes > interoperability since you must define a proprietary interface for this. > > It would be nicer for a modular JRE to allow URL handler factories to > be registered as services. OSGi and the NetBeans module system both do > something like this. > > Similar arguments could perhaps be made for other pluggable singletons > in the current Java platform API (UncaughtExceptionHandler, > FileNameMap, ProxySelector, HostnameVerifier, SSLSocketFactory, ...) > but URLStreamHandlerFactory is the one which I know matters. I also agree with should look re-visit some of these areas for opportunities to use services. I'm less sure about dynamic services, which I think you might be suggesting too. A bit of history on URLStreamHandlerFactory is that there was an attempt to allow URL protocol handlers be registered back in the JDK1.4 time frame. If I recall correctly it had to be backed out due to security concerns. There are also a few land mines in this area, the main one being that URLs are used early in the VM startup, something that can lead to recursive initialization issues if someone attempts to override the core protocol handlers. -Alan. From david.holmes at oracle.com Sun Jun 17 18:20:06 2012 From: david.holmes at oracle.com (David Holmes) Date: Mon, 18 Jun 2012 11:20:06 +1000 Subject: Modularizing URLStreamHandlerFactory In-Reply-To: <4FDBA66C.1090703@oracle.com> References: <4FDBA66C.1090703@oracle.com> Message-ID: <4FDE8246.9090300@oracle.com> On 16/06/2012 7:17 AM, Jesse Glick wrote: > Something to consider - in a big application it is not uncommon to have > multiple URL handlers implemented in various places. Yet the JRE only > lets you set a single URLStreamHandlerFactory for the whole JVM. So for > the application to be modular, you need to write a proxy factory which > accepts modular registrations, but that impedes interoperability since > you must define a proprietary interface for this. > > It would be nicer for a modular JRE to allow URL handler factories to be > registered as services. OSGi and the NetBeans module system both do > something like this. > > Similar arguments could perhaps be made for other pluggable singletons > in the current Java platform API (UncaughtExceptionHandler, FileNameMap, When we added UncaughtExceptionHandler we recognized the desirability to have a "per app" default rather than only having the global default, but there was/is no "application context" to define-in or extract-from the UEH. I don't see how a modular JRE has changed that at all? Even if you say the UEH is a service, what is the scope/context of that service? Further, what additional class support do services drag into the runtime? Are these classes that are already in the base module? David ----- > ProxySelector, HostnameVerifier, SSLSocketFactory, ...) but > URLStreamHandlerFactory is the one which I know matters. From paul.sandoz at oracle.com Mon Jun 18 01:49:40 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Mon, 18 Jun 2012 10:49:40 +0200 Subject: Modularizing URLStreamHandlerFactory In-Reply-To: <4FDE8246.9090300@oracle.com> References: <4FDBA66C.1090703@oracle.com> <4FDE8246.9090300@oracle.com> Message-ID: <855CB3B6-E1DE-4C08-B57E-349C4A74F2D5@oracle.com> On Jun 18, 2012, at 3:20 AM, David Holmes wrote: > On 16/06/2012 7:17 AM, Jesse Glick wrote: >> Something to consider - in a big application it is not uncommon to have >> multiple URL handlers implemented in various places. Yet the JRE only >> lets you set a single URLStreamHandlerFactory for the whole JVM. So for >> the application to be modular, you need to write a proxy factory which >> accepts modular registrations, but that impedes interoperability since >> you must define a proprietary interface for this. >> >> It would be nicer for a modular JRE to allow URL handler factories to be >> registered as services. OSGi and the NetBeans module system both do >> something like this. >> >> Similar arguments could perhaps be made for other pluggable singletons >> in the current Java platform API (UncaughtExceptionHandler, FileNameMap, > > When we added UncaughtExceptionHandler we recognized the desirability to have a "per app" default rather than only having the global default, but there was/is no "application context" to define-in or extract-from the UEH. I don't see how a modular JRE has changed that at all? A modular JRE might not change that but a module *system* can. The "application context" is the configuration, the thing that is pre-calculated (currently at install time) for an application module (one that uses the "class" clause e.g. "class foo.Main"). > Even if you say the UEH is a service, what is the scope/context of that service? > Per configuration. The module dependencies of the application are resolved into a list of resolved modules, then the resolved modules are linked to produce a configuration. If a module declared "requires service ", i.e. is a service consumer module, then any modules that declare "provides service with " that are present in the library, i.e. service provider modules, get pulled in as dependencies when resolving. > Further, what additional class support do services drag into the runtime? Are these classes that are already in the base module? > It would depend on the service provider modules that are installed in the library. IIUC UncaughtExceptionHandler this is a use-case of where only one service instance should be selected as the UncaughtExceptionHandler. Arguably if there are two or more such service instances it could be considered an application configuration error. Paul. From Alan.Bateman at oracle.com Mon Jun 18 01:58:46 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 18 Jun 2012 09:58:46 +0100 Subject: Modularizing URLStreamHandlerFactory In-Reply-To: <4FDE8246.9090300@oracle.com> References: <4FDBA66C.1090703@oracle.com> <4FDE8246.9090300@oracle.com> Message-ID: <4FDEEDC6.2020603@oracle.com> On 18/06/2012 02:20, David Holmes wrote: > > When we added UncaughtExceptionHandler we recognized the desirability > to have a "per app" default rather than only having the global > default, but there was/is no "application context" to define-in or > extract-from the UEH. I don't see how a modular JRE has changed that > at all? It hasn't, but at some point there will be support added to allow containers arrange things so it can load modules that may have a different version to that loaded by the container itself. So there may be notions "per app" further up the stack but that doesn't make it any clearer as to whether UHE is a good candidate on using services. > Even if you say the UEH is a service, what is the scope/context of > that service? > > Further, what additional class support do services drag into the > runtime? Are these classes that are already in the base module? The services support is part of the proposal language and module system. ServiceLoader is in the base module. -Alan From paul.sandoz at oracle.com Mon Jun 18 02:00:14 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Mon, 18 Jun 2012 11:00:14 +0200 Subject: Resolving and services Re: ServiceLoader.load* take 2 In-Reply-To: References: Message-ID: <310C3C04-4D10-4D2F-9711-D73E1799A050@oracle.com> On Jun 12, 2012, at 5:03 PM, Paul Sandoz wrote: > - Modify the resolver so that the permits clause does not affect resolution of a service provider module (and it's dependencies). This area is tricky. Rather than choices of service provider module optional dependencies being pushed on to the choice stack of each service consumer module this potentially could be done once per service interface (although will likely change what solutions are produced), but might be very tricky to implement given that service provider modules may also be service consumer modules. Good diagnostics are required to help track down why a service provider module is not being resolved. Here is my latest thoughts on this: 1) maintain a separate stack of Choice for service provider modules. When the resolver comes across a service consume module add Choice instances to that stack for all known corresponding service provider modules. 2) when the main choice stack is null and is about to return true move all the choices on the service stack to the current stack and keep on processing. To be experimented with soon... Paul. From Alan.Bateman at oracle.com Mon Jun 18 03:01:42 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 18 Jun 2012 11:01:42 +0100 Subject: Resolving and services Re: ServiceLoader.load* take 2 In-Reply-To: <310C3C04-4D10-4D2F-9711-D73E1799A050@oracle.com> References: <310C3C04-4D10-4D2F-9711-D73E1799A050@oracle.com> Message-ID: <4FDEFC86.5060101@oracle.com> On 18/06/2012 10:00, Paul Sandoz wrote: > : > Here is my latest thoughts on this: > > 1) maintain a separate stack of Choice for service provider modules. When the resolver comes across a service consume module add Choice instances to that stack for all known corresponding service provider modules. > > 2) when the main choice stack is null and is about to return true move all the choices on the service stack to the current stack and keep on processing. > > To be experimented with soon... > > Paul. I think this comes down as to whether a service provider module's dependencies (and transitive dependencies) are allowed to change the solution that would otherwise be arrived at if the service provider module were not present. At the moment it can change the solution, meaning that the dependencies are treated equally as it tries to find a solution. Assuming they are to be treated equally then, and with permits out of the equation for service modules, then a possible route is to change the Choice stack so that it is requesting module -> Dependence rather than a ViewDependence. That way it knows whether to check permits or not. The result should be the same as today, assuming that service modules don't have a permits clause. -Alan. From jaroslav.tulach at oracle.com Mon Jun 18 03:43:53 2012 From: jaroslav.tulach at oracle.com (Jaroslav Tulach) Date: Mon, 18 Jun 2012 12:43:53 +0200 Subject: Using Compile Time Annotations for Modularizing URLStreamHandlerFactory In-Reply-To: <4FDBA66C.1090703@oracle.com> References: <4FDBA66C.1090703@oracle.com> Message-ID: <10985988.FBv3kuCx4y@logoutik> Dne P? 15. ?ervna 2012 17:17:32, Jesse Glick napsal(a): > Something to consider - in a big application it is not uncommon to have > multiple URL handlers implemented in various places. Yet the JRE only lets > you set a single URLStreamHandlerFactory for the whole JVM. What a coincidence! On Friday I gave a talk at GeekOut, Tallinn, Estonia about annotation processors and my main example was creation of effective, modular, scalable, robust, condense, no-boiler plate and lazy support for registration of URLStreamHandlers. > So for the > application to be modular, you need to write a proxy factory which accepts > modular registrations, Indeed I had to start with Proxy factory. Then I demonstrated classical ServiceLoader approach (which you probably have in mind when thinking about modularity & URLStreamHandlers). I continued showing, that it is not scalable. To fix that I provided annotation and its processor. Still the code was too verbose. I fixed that by allowing direct registration of URLConnection - and generating the rest on a fly. At the end I eliminated almost all boiler plate code in my final example[1] by generating wrapper URLConnection registration by the annotation processor. Source code: https://github.com/jtulach/Annotations Presentation: http://wiki.apidesign.org/images/0/08/HackIntoYourCompiler.pdf I know many of the ideas of using annotations go beyond the scope of Jigsaw and you have your own problems, but (as I think this is the way effective modularity should be built) I'd like to be able to build this on top of Jigsaw. Won't Jigsaw try to prevent us doing this? -jt [1] https://github.com/jtulach/Annotations/blob/master/UniversalResourceLocator/test/org/netbeans/geekout/demo/impl/Geeks.java > I also agree with should look re-visit some of these areas for > opportunities to use services. I'm less sure about dynamic services, > which I think you might be suggesting too. PS: I am not sure people really need dynamic services when it comes to URLStreamHandlers. I have never seen a use case requiring addition of new protocol support dynamically (without enabling new module). From paul.sandoz at oracle.com Mon Jun 18 03:59:22 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Mon, 18 Jun 2012 12:59:22 +0200 Subject: Resolving and services Re: ServiceLoader.load* take 2 In-Reply-To: <4FDEFC86.5060101@oracle.com> References: <310C3C04-4D10-4D2F-9711-D73E1799A050@oracle.com> <4FDEFC86.5060101@oracle.com> Message-ID: On Jun 18, 2012, at 12:01 PM, Alan Bateman wrote: > On 18/06/2012 10:00, Paul Sandoz wrote: >> : >> Here is my latest thoughts on this: >> >> 1) maintain a separate stack of Choice for service provider modules. When the resolver comes across a service consume module add Choice instances to that stack for all known corresponding service provider modules. >> >> 2) when the main choice stack is null and is about to return true move all the choices on the service stack to the current stack and keep on processing. >> >> To be experimented with soon... >> >> Paul. > I think this comes down as to whether a service provider module's dependencies (and transitive dependencies) are allowed to change the solution that would otherwise be arrived at if the service provider module were not present. At the moment it can change the solution, meaning that the dependencies are treated equally as it tries to find a solution. > Yes, the key difference is *when* such Choice instances for service provider modules are processed. Currently every service consumer module, for the same service interface, adds Choice instances for service provider modules to the stack of the consumer module. That bugs me. I am trying to work out whether such Choice instances for service provider modules can be stuffed on at the end of what would otherwise be a successful resolution if no "requires service " declarations were present. That might result in a more consistent solution. It can potentially allow us to separate out resolving errors. > Assuming they are to be treated equally then, and with permits out of the equation for service modules, then a possible route is to change the Choice stack so that it is requesting module -> Dependence rather than a ViewDependence. That way it knows whether to check permits or not. The result should be the same as today, assuming that service modules don't have a permits cla A quick solution to ignore permits is to do: new Choice(null, vd, ch); IIUC part of what you are saying is we no longer need to add a Choice for each view of the module, we just need to add one Choice for the default view if that view or any other view declares "provides service ...". Does "provides service ..." make sense in the non-default view anymore? I suppose it is harmless but IMHO it would be better to state: A non-default view never has a "provides service ..." clauses. Paul. From Alan.Bateman at oracle.com Mon Jun 18 04:18:47 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 18 Jun 2012 12:18:47 +0100 Subject: Resolving and services Re: ServiceLoader.load* take 2 In-Reply-To: References: <310C3C04-4D10-4D2F-9711-D73E1799A050@oracle.com> <4FDEFC86.5060101@oracle.com> Message-ID: <4FDF0E97.9070704@oracle.com> On 18/06/2012 11:59, Paul Sandoz wrote: > : > > > Does "provides service ..." make sense in the non-default view anymore? I suppose it is harmless but IMHO it would be better to state: > > A non-default view never has a "provides service ..." clauses. > > I mentioned that in one of the recent mails too as it is no longer necessary (but harmless as you say). -Alan From david.holmes at oracle.com Mon Jun 18 05:29:48 2012 From: david.holmes at oracle.com (David Holmes) Date: Mon, 18 Jun 2012 22:29:48 +1000 Subject: Modularizing URLStreamHandlerFactory In-Reply-To: <855CB3B6-E1DE-4C08-B57E-349C4A74F2D5@oracle.com> References: <4FDBA66C.1090703@oracle.com> <4FDE8246.9090300@oracle.com> <855CB3B6-E1DE-4C08-B57E-349C4A74F2D5@oracle.com> Message-ID: <4FDF1F3C.1010608@oracle.com> On 18/06/2012 6:49 PM, Paul Sandoz wrote: > On Jun 18, 2012, at 3:20 AM, David Holmes wrote: >> On 16/06/2012 7:17 AM, Jesse Glick wrote: >>> Something to consider - in a big application it is not uncommon to have >>> multiple URL handlers implemented in various places. Yet the JRE only >>> lets you set a single URLStreamHandlerFactory for the whole JVM. So for >>> the application to be modular, you need to write a proxy factory which >>> accepts modular registrations, but that impedes interoperability since >>> you must define a proprietary interface for this. >>> >>> It would be nicer for a modular JRE to allow URL handler factories to be >>> registered as services. OSGi and the NetBeans module system both do >>> something like this. >>> >>> Similar arguments could perhaps be made for other pluggable singletons >>> in the current Java platform API (UncaughtExceptionHandler, FileNameMap, >> >> When we added UncaughtExceptionHandler we recognized the desirability to have a "per app" default rather than only having the global default, but there was/is no "application context" to define-in or extract-from the UEH. I don't see how a modular JRE has changed that at all? > > A modular JRE might not change that but a module *system* can. The "application context" is the configuration, the thing that is pre-calculated (currently at install time) for an application module (one that uses the "class" clause e.g. "class foo.Main"). If a configuration represents a single running instance of the VM then it isn't adding anything to the existing global default UEH situation - other than perhaps a more flexible way of establishing that global default. The "app context" I was referring to is more like the AppContext as used for Applets - "sub-contexts" within a single VM instance. Anyway these kinds of conversions can be done down the track. No need to debate the intricate details at this stage. Cheers, David ----- > >> Even if you say the UEH is a service, what is the scope/context of that service? >> > > Per configuration. The module dependencies of the application are resolved into a list of resolved modules, then the resolved modules are linked to produce a configuration. If a module declared "requires service", i.e. is a service consumer module, then any modules that declare "provides service with" that are present in the library, i.e. service provider modules, get pulled in as dependencies when resolving. > > >> Further, what additional class support do services drag into the runtime? Are these classes that are already in the base module? >> > > It would depend on the service provider modules that are installed in the library. > > IIUC UncaughtExceptionHandler this is a use-case of where only one service instance should be selected as the UncaughtExceptionHandler. Arguably if there are two or more such service instances it could be considered an application configuration error. > > Paul. From chris.hegarty at oracle.com Mon Jun 18 05:49:12 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Mon, 18 Jun 2012 13:49:12 +0100 Subject: Module file parse API In-Reply-To: <4FD86E8C.3080208@oracle.com> References: <4FD86E8C.3080208@oracle.com> Message-ID: <4FDF23C8.3070402@oracle.com> Paul gave me some feedback off list. Updated the webrev, same location: > http://cr.openjdk.java.net/~chegar/jigsaw/moduleParser_webrev.00/webrev/ * ModuleFileParserException is now a RuntimeException * Fixed up javadoc example code * Some minor bug fixes * Method naming consistency I see this as step one in a phased approach to the reader/writer API's. There is still work to be done to clean up the Reader ( now really an Installed/Extractor ), the *Header types, and obviously the writer. What I would like is get this first set of changes in ( if others agree with the API ), and then address the above outstanding issues with further commits. Thanks, -Chris. On 13/06/2012 11:42, Chris Hegarty wrote: > Hi, > > A while back it was suggested that it may be useful to define a pull > parser type API for reading from a module file. I gives separation of > the parsing of the module file from the writing of it (either installing > into the module library, or extraction elsewhere ). > > I've prototyped such an API and updated the ModuleFile.Reader to use it. > The first four files in the webrev contain the API, its implementation, > and a test that demonstrates some sample usage ( extraction of one or > more classes, listing sections, verifying hashes ). The remainder of the > changes simply updates some of the tools for installing and handling > signed modules. > > http://cr.openjdk.java.net/~chegar/jigsaw/moduleParser_webrev.00/webrev/ > > Thanks, > -Chris. From paul.sandoz at oracle.com Mon Jun 18 06:15:03 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Mon, 18 Jun 2012 15:15:03 +0200 Subject: Modularizing URLStreamHandlerFactory In-Reply-To: <4FDF1F3C.1010608@oracle.com> References: <4FDBA66C.1090703@oracle.com> <4FDE8246.9090300@oracle.com> <855CB3B6-E1DE-4C08-B57E-349C4A74F2D5@oracle.com> <4FDF1F3C.1010608@oracle.com> Message-ID: <757DDF20-22EB-404B-8EC7-1AFB599E9F18@oracle.com> On Jun 18, 2012, at 2:29 PM, David Holmes wrote: > On 18/06/2012 6:49 PM, Paul Sandoz wrote: >> On Jun 18, 2012, at 3:20 AM, David Holmes wrote: >>> On 16/06/2012 7:17 AM, Jesse Glick wrote: >>>> Something to consider - in a big application it is not uncommon to have >>>> multiple URL handlers implemented in various places. Yet the JRE only >>>> lets you set a single URLStreamHandlerFactory for the whole JVM. So for >>>> the application to be modular, you need to write a proxy factory which >>>> accepts modular registrations, but that impedes interoperability since >>>> you must define a proprietary interface for this. >>>> >>>> It would be nicer for a modular JRE to allow URL handler factories to be >>>> registered as services. OSGi and the NetBeans module system both do >>>> something like this. >>>> >>>> Similar arguments could perhaps be made for other pluggable singletons >>>> in the current Java platform API (UncaughtExceptionHandler, FileNameMap, >>> >>> When we added UncaughtExceptionHandler we recognized the desirability to have a "per app" default rather than only having the global default, but there was/is no "application context" to define-in or extract-from the UEH. I don't see how a modular JRE has changed that at all? >> >> A modular JRE might not change that but a module *system* can. The "application context" is the configuration, the thing that is pre-calculated (currently at install time) for an application module (one that uses the "class" clause e.g. "class foo.Main"). > > If a configuration represents a single running instance of the VM then it isn't adding anything to the existing global default UEH situation - other than perhaps a more flexible way of establishing that global default. > Sorry, i am being a little obtuse with the above. A configuration is associated with an application. Currently the Jigsaw module system supports one application running per VM. For multiple applications, that are to some degree isolated from each other, the Jigsaw module system requires (the yet to be defined and implemented) container support, previously mentioned by Alan. Paul. From paul.sandoz at oracle.com Mon Jun 18 06:50:59 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Mon, 18 Jun 2012 15:50:59 +0200 Subject: Module file parse API In-Reply-To: <4FDF23C8.3070402@oracle.com> References: <4FD86E8C.3080208@oracle.com> <4FDF23C8.3070402@oracle.com> Message-ID: <679B7267-55C7-4E1E-82BE-78CEDBBECD27@oracle.com> Hi Chris, This is an excellent start. Looks good to me. -- Some potential things we discussed off list for further commits: - s/ModuleFileParser/ModuleFileReader. I can think of two alternative implementations; one that wraps a module installed into a library so the reader can be connected to a writer for extraction. Another that wraps a modular jar file so the same installer code can be reused, i am not sure exactly if that is possible but sounds nice in theory :-) - IIRC the complete size of the jmod file is encoded in the file itself, thus after the file header has been read we can wrap everything around a CountingInputStream. - An adapting implementation of ModuleFileReader which accumulates hashes (ignoring the hash for a section of the signature type) and validates hashes. It can be reused by the installer/extractor and the signer. Paul. On Jun 18, 2012, at 2:49 PM, Chris Hegarty wrote: > Paul gave me some feedback off list. Updated the webrev, same location: > > > http://cr.openjdk.java.net/~chegar/jigsaw/moduleParser_webrev.00/webrev/ > > * ModuleFileParserException is now a RuntimeException > * Fixed up javadoc example code > * Some minor bug fixes > * Method naming consistency > > I see this as step one in a phased approach to the reader/writer API's. There is still work to be done to clean up the Reader ( now really an Installed/Extractor ), the *Header types, and obviously the writer. > > What I would like is get this first set of changes in ( if others agree with the API ), and then address the above outstanding issues with further commits. > > Thanks, > -Chris. > > On 13/06/2012 11:42, Chris Hegarty wrote: >> Hi, >> >> A while back it was suggested that it may be useful to define a pull >> parser type API for reading from a module file. I gives separation of >> the parsing of the module file from the writing of it (either installing >> into the module library, or extraction elsewhere ). >> >> I've prototyped such an API and updated the ModuleFile.Reader to use it. >> The first four files in the webrev contain the API, its implementation, >> and a test that demonstrates some sample usage ( extraction of one or >> more classes, listing sections, verifying hashes ). The remainder of the >> changes simply updates some of the tools for installing and handling >> signed modules. >> >> http://cr.openjdk.java.net/~chegar/jigsaw/moduleParser_webrev.00/webrev/ >> >> Thanks, >> -Chris. From chris.hegarty at oracle.com Mon Jun 18 07:16:21 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Mon, 18 Jun 2012 15:16:21 +0100 Subject: Module file parse API In-Reply-To: <679B7267-55C7-4E1E-82BE-78CEDBBECD27@oracle.com> References: <4FD86E8C.3080208@oracle.com> <4FDF23C8.3070402@oracle.com> <679B7267-55C7-4E1E-82BE-78CEDBBECD27@oracle.com> Message-ID: <4FDF3835.8070201@oracle.com> On 18/06/2012 14:50, Paul Sandoz wrote: > Hi Chris, > > This is an excellent start. Looks good to me. > > -- > > Some potential things we discussed off list for further commits: > > - s/ModuleFileParser/ModuleFileReader. I can think of two alternative implementations; one that wraps a module installed into a library so the reader can be connected to a writer for extraction. Another that wraps a modular jar file so the same installer code can be reused, i am not sure exactly if that is possible but sounds nice in theory :-) Right, I think we can look at this as part of future commits. As you say it could be nice. > - IIRC the complete size of the jmod file is encoded in the file itself, thus after the file header has been read we can wrap everything around a CountingInputStream. I really like this idea too, but I need to think carefully about the impact of concatenating some rogue module to another module file. Again, something that could be nice for a future commit. > - An adapting implementation of ModuleFileReader which accumulates hashes (ignoring the hash for a section of the signature type) and validates hashes. It can be reused by the installer/extractor and the signer. Yes, this would seems to fit nicely with the signer and verifying installer. -Chris. > > Paul. > > > On Jun 18, 2012, at 2:49 PM, Chris Hegarty wrote: > >> Paul gave me some feedback off list. Updated the webrev, same location: >> >>> http://cr.openjdk.java.net/~chegar/jigsaw/moduleParser_webrev.00/webrev/ >> >> * ModuleFileParserException is now a RuntimeException >> * Fixed up javadoc example code >> * Some minor bug fixes >> * Method naming consistency >> >> I see this as step one in a phased approach to the reader/writer API's. There is still work to be done to clean up the Reader ( now really an Installed/Extractor ), the *Header types, and obviously the writer. >> >> What I would like is get this first set of changes in ( if others agree with the API ), and then address the above outstanding issues with further commits. >> >> Thanks, >> -Chris. >> >> On 13/06/2012 11:42, Chris Hegarty wrote: >>> Hi, >>> >>> A while back it was suggested that it may be useful to define a pull >>> parser type API for reading from a module file. I gives separation of >>> the parsing of the module file from the writing of it (either installing >>> into the module library, or extraction elsewhere ). >>> >>> I've prototyped such an API and updated the ModuleFile.Reader to use it. >>> The first four files in the webrev contain the API, its implementation, >>> and a test that demonstrates some sample usage ( extraction of one or >>> more classes, listing sections, verifying hashes ). The remainder of the >>> changes simply updates some of the tools for installing and handling >>> signed modules. >>> >>> http://cr.openjdk.java.net/~chegar/jigsaw/moduleParser_webrev.00/webrev/ >>> >>> Thanks, >>> -Chris. > From paul.sandoz at oracle.com Mon Jun 18 09:37:37 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Mon, 18 Jun 2012 18:37:37 +0200 Subject: Module file parse API In-Reply-To: <4FDF3835.8070201@oracle.com> References: <4FD86E8C.3080208@oracle.com> <4FDF23C8.3070402@oracle.com> <679B7267-55C7-4E1E-82BE-78CEDBBECD27@oracle.com> <4FDF3835.8070201@oracle.com> Message-ID: <415CB6CB-78F1-4FAD-9B71-B421B0F7AB27@oracle.com> On Jun 18, 2012, at 4:16 PM, Chris Hegarty wrote: >> - IIRC the complete size of the jmod file is encoded in the file itself, thus after the file header has been read we can wrap everything around a CountingInputStream. > > I really like this idea too, but I need to think carefully about the impact of concatenating some rogue module to another module file. Signed modules? Why does using a CountingInputStream over the content introduce a security issue? Paul. From chris.hegarty at oracle.com Mon Jun 18 09:49:32 2012 From: chris.hegarty at oracle.com (chris hegarty) Date: Mon, 18 Jun 2012 17:49:32 +0100 Subject: Module file parse API In-Reply-To: <415CB6CB-78F1-4FAD-9B71-B421B0F7AB27@oracle.com> References: <4FD86E8C.3080208@oracle.com> <4FDF23C8.3070402@oracle.com> <679B7267-55C7-4E1E-82BE-78CEDBBECD27@oracle.com> <4FDF3835.8070201@oracle.com> <415CB6CB-78F1-4FAD-9B71-B421B0F7AB27@oracle.com> Message-ID: <4FDF5C1C.2030202@oracle.com> On 18/06/2012 17:37, Paul Sandoz wrote: > On Jun 18, 2012, at 4:16 PM, Chris Hegarty wrote: >>> - IIRC the complete size of the jmod file is encoded in the file itself, thus after the file header has been read we can wrap everything around a CountingInputStream. >> >> I really like this idea too, but I need to think carefully about the impact of concatenating some rogue module to another module file. > > Signed modules? > > Why does using a CountingInputStream over the content introduce a security issue? I was thinking that if the reader/installer was able to parse multiple module files from a single input stream something like this may be a problem... cat foo.jmod bar.jmod > foo.jmod jmod install foo.jmod jmod ls foo bar It just means that the installer is responsible for handling this situation rather than the parser. -Chris. > > Paul. From tdiekman at tibco.com Mon Jun 18 12:06:19 2012 From: tdiekman at tibco.com (Tim Diekmann) Date: Mon, 18 Jun 2012 19:06:19 +0000 Subject: Resolving and services Re: ServiceLoader.load* take 2 In-Reply-To: <310C3C04-4D10-4D2F-9711-D73E1799A050@oracle.com> Message-ID: This kind of problem looks very familiar. Has anyone taken a look at the OSGi Resolver lately? It has a solution for the resolution of resources with capabilities and requirements and the latest OSGi Spec R5 has made that a separate service. Just a thought, trying to prevent the wheel being re-invented. Tim. On 6/18/12 5:00 AM, "Paul Sandoz" wrote: >On Jun 12, 2012, at 5:03 PM, Paul Sandoz wrote: >> - Modify the resolver so that the permits clause does not affect >>resolution of a service provider module (and it's dependencies). This >>area is tricky. Rather than choices of service provider module optional >>dependencies being pushed on to the choice stack of each service >>consumer module this potentially could be done once per service >>interface (although will likely change what solutions are produced), but >>might be very tricky to implement given that service provider modules >>may also be service consumer modules. Good diagnostics are required to >>help track down why a service provider module is not being resolved. > >Here is my latest thoughts on this: > >1) maintain a separate stack of Choice for service provider modules. When >the resolver comes across a service consume module add Choice instances >to that stack for all known corresponding service provider modules. > >2) when the main choice stack is null and is about to return true move >all the choices on the service stack to the current stack and keep on >processing. > >To be experimented with soon... > >Paul. From david.holmes at oracle.com Mon Jun 18 19:20:17 2012 From: david.holmes at oracle.com (David Holmes) Date: Tue, 19 Jun 2012 12:20:17 +1000 Subject: exports directive Message-ID: <4FDFE1E1.4030300@oracle.com> Does export work like a prefix such that exports foo; means the module exports all types in package foo, plus all types in all subpackages of foo? Or do I need to use multiple exports directives, or wildcards? And is this documented somewhere? Thanks, David From mandy.chung at oracle.com Mon Jun 18 20:10:36 2012 From: mandy.chung at oracle.com (Mandy Chung) Date: Mon, 18 Jun 2012 20:10:36 -0700 Subject: exports directive In-Reply-To: <4FDFE1E1.4030300@oracle.com> References: <4FDFE1E1.4030300@oracle.com> Message-ID: <4FDFEDAC.2020007@oracle.com> On 6/18/2012 7:20 PM, David Holmes wrote: > Does export work like a prefix such that > > exports foo; > > means the module exports all types in package foo, plus all types in > all subpackages of foo? Or do I need to use multiple exports > directives, or wildcards? > |exports| takes a qualified identifier indicating a package name. You need to use multiple |exports| directives to list its subpackages: |module foo { exports foo; exports foo.spi; exports foo.util; } | > And is this documented somewhere? > http://openjdk.java.net/projects/jigsaw/doc/lang-vm.html http://cr.openjdk.java.net/~mr/jigsaw/notes/jigsaw-big-picture-01 Mandy From david.holmes at oracle.com Mon Jun 18 20:17:24 2012 From: david.holmes at oracle.com (David Holmes) Date: Tue, 19 Jun 2012 13:17:24 +1000 Subject: exports directive In-Reply-To: <4FDFEDAC.2020007@oracle.com> References: <4FDFE1E1.4030300@oracle.com> <4FDFEDAC.2020007@oracle.com> Message-ID: <4FDFEF44.2010108@oracle.com> On 19/06/2012 1:10 PM, Mandy Chung wrote: > On 6/18/2012 7:20 PM, David Holmes wrote: >> Does export work like a prefix such that >> >> exports foo; >> >> means the module exports all types in package foo, plus all types in >> all subpackages of foo? Or do I need to use multiple exports >> directives, or wildcards? >> > > |exports| takes a qualified identifier indicating a package name. You > need to use multiple |exports| directives to list its subpackages: > > |module foo { > exports foo; > exports foo.spi; > exports foo.util; > } > | > > >> And is this documented somewhere? >> > > http://openjdk.java.net/projects/jigsaw/doc/lang-vm.html > http://cr.openjdk.java.net/~mr/jigsaw/notes/jigsaw-big-picture-01 Thanks Mandy! Annoyingly tedious to have to list all the subpackages directly though. Libraries that make extensive use of subpackages for grouping things will need to use long exports lists. Some kind of wildcard support would be quite useful I think. David ----- > Mandy > From david.holmes at oracle.com Mon Jun 18 21:24:11 2012 From: david.holmes at oracle.com (David Holmes) Date: Tue, 19 Jun 2012 14:24:11 +1000 Subject: RFR: jpkg contents/show commands In-Reply-To: <4FB52FA1.1070305@oracle.com> References: <4FB52FA1.1070305@oracle.com> Message-ID: <4FDFFEEB.7000301@oracle.com> Hi Chris, Did this change not get pushed? I just did a clean clone and build and jpkg doesn't have these new commands. David On 18/05/2012 3:04 AM, Chris Hegarty wrote: > This change adds two new commands to the packaging tool. 'jpkg show' and > 'jpkg contents', which came up on this list a few weeks ago. > > http://cr.openjdk.java.net/~chegar/jigsaw/jpkg_contents.00/webrev/ > > Also, included is various cleanup and restructuring, especially to > ModuleFile, to put in place a structure to more easily support new > compression algorithms. Specifically LZMA. I have a prototype from last > year I need to rebase. > > For now, 'contents' and 'show' are only added jpkg, but I could see that > these may be useful before installing a module into the module library, > so could be applicable to the jmod command also? Or just jmod? > > If these new commands are to stay in jpkg long term we should probably > change the command that builds the jmod/deb files to create/build, so > 'jpkg create ... jmod'. > > Two specific implementation changes to draw attention to in the webrev: > 1) ModuleFile.java (L535 new file). Resources were never being > deflated in the module library. They are now. > 2) Sean, > Can you verify the changes to changes to Singer and > ModuleFile.SignatureSection? All jigsaw signing related tests pass. > > -Chris. From paul.sandoz at oracle.com Tue Jun 19 01:01:01 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 19 Jun 2012 10:01:01 +0200 Subject: Module file parse API In-Reply-To: <4FDF5C1C.2030202@oracle.com> References: <4FD86E8C.3080208@oracle.com> <4FDF23C8.3070402@oracle.com> <679B7267-55C7-4E1E-82BE-78CEDBBECD27@oracle.com> <4FDF3835.8070201@oracle.com> <415CB6CB-78F1-4FAD-9B71-B421B0F7AB27@oracle.com> <4FDF5C1C.2030202@oracle.com> Message-ID: On Jun 18, 2012, at 6:49 PM, chris hegarty wrote: > On 18/06/2012 17:37, Paul Sandoz wrote: >> On Jun 18, 2012, at 4:16 PM, Chris Hegarty wrote: >>>> - IIRC the complete size of the jmod file is encoded in the file itself, thus after the file header has been read we can wrap everything around a CountingInputStream. >>> >>> I really like this idea too, but I need to think carefully about the impact of concatenating some rogue module to another module file. >> >> Signed modules? >> >> Why does using a CountingInputStream over the content introduce a security issue? > > I was thinking that if the reader/installer was able to parse multiple module files from a single input stream something like this may be a problem... > cat foo.jmod bar.jmod > foo.jmod > jmod install foo.jmod > jmod ls > foo > bar > > It just means that the installer is responsible for handling this situation rather than the parser. > OK, i understand what you are getting at. I was not suggesting that the tooling or the parser directly support what you say above. Just suggesting something simpler: not to rule out the following capability: InputStream s = // some stream where multiple jmods are streamed while (s.available() != 0) { ModuleFileParser mfp = ModuleFile.newParser(s); while (parser.hasNext()) { parser.next(); } } Paul. From chris.hegarty at oracle.com Tue Jun 19 01:52:14 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 19 Jun 2012 09:52:14 +0100 Subject: RFR: jpkg contents/show commands In-Reply-To: <4FDFFEEB.7000301@oracle.com> References: <4FB52FA1.1070305@oracle.com> <4FDFFEEB.7000301@oracle.com> Message-ID: <4FE03DBE.50108@oracle.com> On 19/06/12 05:24, David Holmes wrote: > Hi Chris, > > Did this change not get pushed? I just did a clean clone and build and > jpkg doesn't have these new commands. This never got pushed. I got distracted with refactoring and the new Module file parser API. When this API is pushed I'll be able to update this proposed change to use it ( should be much cleaner ).[ Note to self, do not mix refactoring changes with others, just slows everything down! ] Most importantly, I think we now have agreement that the jpkg tool should be extended to provide all commands related to the packaging of the module file, similar to what the jar tool provides for jar files ( create, list, extract, etc... ) Once I push the parser changes I'll update this webrev and try to make progress on it. -Chris. > > David > > On 18/05/2012 3:04 AM, Chris Hegarty wrote: >> This change adds two new commands to the packaging tool. 'jpkg show' and >> 'jpkg contents', which came up on this list a few weeks ago. >> >> http://cr.openjdk.java.net/~chegar/jigsaw/jpkg_contents.00/webrev/ >> >> Also, included is various cleanup and restructuring, especially to >> ModuleFile, to put in place a structure to more easily support new >> compression algorithms. Specifically LZMA. I have a prototype from last >> year I need to rebase. >> >> For now, 'contents' and 'show' are only added jpkg, but I could see that >> these may be useful before installing a module into the module library, >> so could be applicable to the jmod command also? Or just jmod? >> >> If these new commands are to stay in jpkg long term we should probably >> change the command that builds the jmod/deb files to create/build, so >> 'jpkg create ... jmod'. >> >> Two specific implementation changes to draw attention to in the webrev: >> 1) ModuleFile.java (L535 new file). Resources were never being >> deflated in the module library. They are now. >> 2) Sean, >> Can you verify the changes to changes to Singer and >> ModuleFile.SignatureSection? All jigsaw signing related tests pass. >> >> -Chris. From paul.sandoz at oracle.com Tue Jun 19 02:20:19 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 19 Jun 2012 11:20:19 +0200 Subject: Resolving and services Re: ServiceLoader.load* take 2 In-Reply-To: References: Message-ID: <3C81B0C1-0574-49A4-BD5D-E747DECD8EF7@oracle.com> Hi Tim, Many thanks for the hints, it got me thinking about this more :-) On Jun 18, 2012, at 9:06 PM, Tim Diekmann wrote: > This kind of problem looks very familiar. > > Has anyone taken a look at the OSGi Resolver lately? It has a solution for > the resolution of resources with capabilities and requirements and the > latest OSGi Spec R5 has made that a separate service. > Do you mean look at an implementation of the Resolver Hook Service? The OSGI core R5 spec is deliberately vague on resolving, specifying constraints, but not restricting the potential solutions that implementations may produce. I looked at bit at section 6 "Resource API Specification" and there are indeed similarities between a requirement (a consumer of a service [*]) and a capability (a provider of a service), although i don't know enough about OSGi to ascertain if the requirements and capabilities can support an equivalent of JDK services. In terms of resolving it is not specified when such resolution of requirements be performed [1]. It's hard to say when the JSR spins up what will be specified for resolution but i suspect it will walk a fine line as has the OSGI spec to not unduly handcuff implementations. In this regard when Jigsaw services are resolved is potentially an implementation detail of the resolver. I can see advantages to why it may be useful specify that service dependencies are resolved after all other dependencies are resolved, but can also see why this may unduly restrict implementations. Paul. [*] Service in the sense of JDK service using ServiceLoader, not to be confused with the OSGi service layer. [1] Section 6.4.2 Resolution Requirements can be optional or mandatory, as specified in the resolution directive, which is only available on the Requirement. Optional Requirements do not have to be satisfied by the Resolver. Environments can be eager or relaxed in finding Resources to resolve optional Requirements. All mandatory Requirements of a Resource must be satisfied before that Resource?s Capabilities can be provided to the Environment. The syntax of the resolution directive is therefore: resolution ::= ?optional? | ?mandatory? The default is mandatory. > Just a thought, trying to prevent the wheel being re-invented. > > Tim. > > > > > > On 6/18/12 5:00 AM, "Paul Sandoz" wrote: > >> On Jun 12, 2012, at 5:03 PM, Paul Sandoz wrote: >>> - Modify the resolver so that the permits clause does not affect >>> resolution of a service provider module (and it's dependencies). This >>> area is tricky. Rather than choices of service provider module optional >>> dependencies being pushed on to the choice stack of each service >>> consumer module this potentially could be done once per service >>> interface (although will likely change what solutions are produced), but >>> might be very tricky to implement given that service provider modules >>> may also be service consumer modules. Good diagnostics are required to >>> help track down why a service provider module is not being resolved. >> >> Here is my latest thoughts on this: >> >> 1) maintain a separate stack of Choice for service provider modules. When >> the resolver comes across a service consume module add Choice instances >> to that stack for all known corresponding service provider modules. >> >> 2) when the main choice stack is null and is about to return true move >> all the choices on the service stack to the current stack and keep on >> processing. >> >> To be experimented with soon... >> >> Paul. > From jaroslav.tulach at oracle.com Tue Jun 19 03:06:04 2012 From: jaroslav.tulach at oracle.com (Jaroslav Tulach) Date: Tue, 19 Jun 2012 12:06:04 +0200 Subject: exports directive In-Reply-To: <4FDFEF44.2010108@oracle.com> References: <4FDFE1E1.4030300@oracle.com> <4FDFEDAC.2020007@oracle.com> <4FDFEF44.2010108@oracle.com> Message-ID: <1884412.Me21NKFpK3@logoutik> Dne ?t 19. ?ervna 2012 13:17:24, David Holmes napsal(a): > On 19/06/2012 1:10 PM, Mandy Chung wrote: > > On 6/18/2012 7:20 PM, David Holmes wrote: > >> Does export work like a prefix such that > >> > >> exports foo; > >> > >> means the module exports all types in package foo, plus all types in > >> all subpackages of foo? Or do I need to use multiple exports > >> directives, or wildcards? > >> > > |exports| takes a qualified identifier indicating a package name. You > > > > need to use multiple |exports| directives to list its subpackages: > > |module foo { > > > > exports foo; > > exports foo.spi; > > exports foo.util; > > } > > > >> And is this documented somewhere? > > > > http://openjdk.java.net/projects/jigsaw/doc/lang-vm.html > > http://cr.openjdk.java.net/~mr/jigsaw/notes/jigsaw-big-picture-01 > > Thanks Mandy! > > Annoyingly tedious to have to list all the subpackages directly though. > Libraries that make extensive use of subpackages for grouping things > will need to use long exports lists. Some kind of wildcard support would > be quite useful I think. A tools perspective: Wildcards are pain to support. Example: just by looking at module-info.java one cannot deduce list of exported packages. One needs to examine all module source files. Tedious. A user perspective: While wildcards save some time initially, they usually bite you back after few years of maintenance. Also they don't work well when somebody adds org.myorg.myexportedapi.internal package, which is a common pattern used by some open source projects. -jt From njbartlett at gmail.com Tue Jun 19 04:12:32 2012 From: njbartlett at gmail.com (Neil Bartlett) Date: Tue, 19 Jun 2012 12:12:32 +0100 Subject: exports directive In-Reply-To: <1884412.Me21NKFpK3@logoutik> References: <4FDFE1E1.4030300@oracle.com> <4FDFEDAC.2020007@oracle.com> <4FDFEF44.2010108@oracle.com> <1884412.Me21NKFpK3@logoutik> Message-ID: +1 Amongst OSGi developers using the "bnd" tool (which generates MANIFEST.MF from a descriptor file), the use of wildcards for exports is widely held to be an anti-pattern. It obscures the actual exports and leads to the module author exposing more implementation details than intended. On Tue, Jun 19, 2012 at 11:06 AM, Jaroslav Tulach wrote: > Dne ?t 19. ?ervna 2012 13:17:24, David Holmes napsal(a): >> On 19/06/2012 1:10 PM, Mandy Chung wrote: >> > On 6/18/2012 7:20 PM, David Holmes wrote: >> >> Does export work like a prefix such that >> >> >> >> exports foo; >> >> >> >> means the module exports all types in package foo, plus all types in >> >> all subpackages of foo? Or do I need to use multiple exports >> >> directives, or wildcards? >> >> >> > |exports| takes a qualified identifier indicating a package name. You >> > >> > need to use multiple |exports| directives to list its subpackages: >> > |module foo { >> > >> > exports foo; >> > exports foo.spi; >> > exports foo.util; >> > } >> > >> >> And is this documented somewhere? >> > >> > http://openjdk.java.net/projects/jigsaw/doc/lang-vm.html >> > http://cr.openjdk.java.net/~mr/jigsaw/notes/jigsaw-big-picture-01 >> >> Thanks Mandy! >> >> Annoyingly tedious to have to list all the subpackages directly though. >> Libraries that make extensive use of subpackages for grouping things >> will need to use long exports lists. Some kind of wildcard support would >> be quite useful I think. > > A tools perspective: Wildcards are pain to support. Example: just by looking > at module-info.java one cannot deduce list of exported packages. One needs to > examine all module source files. Tedious. > > A user perspective: While wildcards save some time initially, they usually > bite you back after few years of maintenance. Also they don't work well when > somebody adds org.myorg.myexportedapi.internal package, which is a common > pattern used by some open source projects. > > -jt > From david.holmes at oracle.com Tue Jun 19 04:22:42 2012 From: david.holmes at oracle.com (David Holmes) Date: Tue, 19 Jun 2012 21:22:42 +1000 Subject: exports directive In-Reply-To: <1884412.Me21NKFpK3@logoutik> References: <4FDFE1E1.4030300@oracle.com> <4FDFEDAC.2020007@oracle.com> <4FDFEF44.2010108@oracle.com> <1884412.Me21NKFpK3@logoutik> Message-ID: <4FE06102.70509@oracle.com> On 19/06/2012 8:06 PM, Jaroslav Tulach wrote: > Dne ?t 19. ?ervna 2012 13:17:24, David Holmes napsal(a): >> On 19/06/2012 1:10 PM, Mandy Chung wrote: >>> On 6/18/2012 7:20 PM, David Holmes wrote: >>>> Does export work like a prefix such that >>>> >>>> exports foo; >>>> >>>> means the module exports all types in package foo, plus all types in >>>> all subpackages of foo? Or do I need to use multiple exports >>>> directives, or wildcards? >>>> >>> |exports| takes a qualified identifier indicating a package name. You >>> >>> need to use multiple |exports| directives to list its subpackages: >>> |module foo { >>> >>> exports foo; >>> exports foo.spi; >>> exports foo.util; >>> } >>> >>>> And is this documented somewhere? >>> >>> http://openjdk.java.net/projects/jigsaw/doc/lang-vm.html >>> http://cr.openjdk.java.net/~mr/jigsaw/notes/jigsaw-big-picture-01 >> >> Thanks Mandy! >> >> Annoyingly tedious to have to list all the subpackages directly though. >> Libraries that make extensive use of subpackages for grouping things >> will need to use long exports lists. Some kind of wildcard support would >> be quite useful I think. > > A tools perspective: Wildcards are pain to support. Example: just by looking > at module-info.java one cannot deduce list of exported packages. One needs to > examine all module source files. Tedious. > > A user perspective: While wildcards save some time initially, they usually > bite you back after few years of maintenance. Also they don't work well when > somebody adds org.myorg.myexportedapi.internal package, which is a common > pattern used by some open source projects. I suppose once we have tools to assist in generating the module-info the point will be moot. But here today I would have liked to have had a wildcard. ;-) YMMV. David > -jt > From david.holmes at oracle.com Tue Jun 19 04:46:18 2012 From: david.holmes at oracle.com (David Holmes) Date: Tue, 19 Jun 2012 21:46:18 +1000 Subject: exports directive In-Reply-To: References: <4FDFE1E1.4030300@oracle.com> <4FDFEDAC.2020007@oracle.com> <4FDFEF44.2010108@oracle.com> <1884412.Me21NKFpK3@logoutik> Message-ID: <4FE0668A.7030501@oracle.com> On 19/06/2012 9:12 PM, Neil Bartlett wrote: > +1 > > Amongst OSGi developers using the "bnd" tool (which generates > MANIFEST.MF from a descriptor file), the use of wildcards for exports > is widely held to be an anti-pattern. It obscures the actual exports > and leads to the module author exposing more implementation details > than intended. I think whether you have wildcards or use explicit lists, the basic problem being described is the inability to readily validate that what you wrote is what you intended - whether you left something out, or added something extra in. I would have thought, perhaps naively, that the majority of subpackages would be for exporting, and so ease of use should focus on making it easy to declare those exports. David ----- > > On Tue, Jun 19, 2012 at 11:06 AM, Jaroslav Tulach > wrote: >> Dne ?t 19. ?ervna 2012 13:17:24, David Holmes napsal(a): >>> On 19/06/2012 1:10 PM, Mandy Chung wrote: >>>> On 6/18/2012 7:20 PM, David Holmes wrote: >>>>> Does export work like a prefix such that >>>>> >>>>> exports foo; >>>>> >>>>> means the module exports all types in package foo, plus all types in >>>>> all subpackages of foo? Or do I need to use multiple exports >>>>> directives, or wildcards? >>>>> >>>> |exports| takes a qualified identifier indicating a package name. You >>>> >>>> need to use multiple |exports| directives to list its subpackages: >>>> |module foo { >>>> >>>> exports foo; >>>> exports foo.spi; >>>> exports foo.util; >>>> } >>>> >>>>> And is this documented somewhere? >>>> >>>> http://openjdk.java.net/projects/jigsaw/doc/lang-vm.html >>>> http://cr.openjdk.java.net/~mr/jigsaw/notes/jigsaw-big-picture-01 >>> >>> Thanks Mandy! >>> >>> Annoyingly tedious to have to list all the subpackages directly though. >>> Libraries that make extensive use of subpackages for grouping things >>> will need to use long exports lists. Some kind of wildcard support would >>> be quite useful I think. >> >> A tools perspective: Wildcards are pain to support. Example: just by looking >> at module-info.java one cannot deduce list of exported packages. One needs to >> examine all module source files. Tedious. >> >> A user perspective: While wildcards save some time initially, they usually >> bite you back after few years of maintenance. Also they don't work well when >> somebody adds org.myorg.myexportedapi.internal package, which is a common >> pattern used by some open source projects. >> >> -jt >> From david.lloyd at redhat.com Tue Jun 19 06:43:09 2012 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 19 Jun 2012 08:43:09 -0500 Subject: exports directive In-Reply-To: <1884412.Me21NKFpK3@logoutik> References: <4FDFE1E1.4030300@oracle.com> <4FDFEDAC.2020007@oracle.com> <4FDFEF44.2010108@oracle.com> <1884412.Me21NKFpK3@logoutik> Message-ID: <4FE081ED.4060209@redhat.com> On 06/19/2012 05:06 AM, Jaroslav Tulach wrote: > Dne ?t 19. ?ervna 2012 13:17:24, David Holmes napsal(a): >> On 19/06/2012 1:10 PM, Mandy Chung wrote: >>> On 6/18/2012 7:20 PM, David Holmes wrote: >>>> Does export work like a prefix such that >>>> >>>> exports foo; >>>> >>>> means the module exports all types in package foo, plus all types in >>>> all subpackages of foo? Or do I need to use multiple exports >>>> directives, or wildcards? >>>> >>> |exports| takes a qualified identifier indicating a package name. You >>> >>> need to use multiple |exports| directives to list its subpackages: >>> |module foo { >>> >>> exports foo; >>> exports foo.spi; >>> exports foo.util; >>> } >>> >>>> And is this documented somewhere? >>> >>> http://openjdk.java.net/projects/jigsaw/doc/lang-vm.html >>> http://cr.openjdk.java.net/~mr/jigsaw/notes/jigsaw-big-picture-01 >> >> Thanks Mandy! >> >> Annoyingly tedious to have to list all the subpackages directly though. >> Libraries that make extensive use of subpackages for grouping things >> will need to use long exports lists. Some kind of wildcard support would >> be quite useful I think. > > A tools perspective: Wildcards are pain to support. Example: just by looking > at module-info.java one cannot deduce list of exported packages. One needs to > examine all module source files. Tedious. > > A user perspective: While wildcards save some time initially, they usually > bite you back after few years of maintenance. Also they don't work well when > somebody adds org.myorg.myexportedapi.internal package, which is a common > pattern used by some open source projects. I have to agree. I think that supporting wildcards is only a moderate convenience; it's not worth the inevitable performance and complexity cost. Our own experiments and experiences support this argument. -- - DML From Alan.Bateman at oracle.com Tue Jun 19 07:19:55 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 19 Jun 2012 15:19:55 +0100 Subject: Module file parse API In-Reply-To: <4FD86E8C.3080208@oracle.com> References: <4FD86E8C.3080208@oracle.com> Message-ID: <4FE08A8B.9010001@oracle.com> On 13/06/2012 11:42, Chris Hegarty wrote: > Hi, > > A while back it was suggested that it may be useful to define a pull > parser type API for reading from a module file. I gives separation of > the parsing of the module file from the writing of it (either > installing into the module library, or extraction elsewhere ). > > I've prototyped such an API and updated the ModuleFile.Reader to use > it. The first four files in the webrev contain the API, its > implementation, and a test that demonstrates some sample usage ( > extraction of one or more classes, listing sections, verifying hashes > ). The remainder of the changes simply updates some of the tools for > installing and handling signed modules. > > http://cr.openjdk.java.net/~chegar/jigsaw/moduleParser_webrev.00/webrev/ > > Thanks, > -Chris. I skimmed through this (not a detailed review) and the approach looks very reasonable and nicely done. I agree with Paul's comment that it would be good to think about the case where a tool needs to read/extract a module from a module library too. In that usage it would be more of a stream reader rather than a parser. A minor comment is that fileHeader() should be clear that it will return the header at any time, irrespective of which section the cursor is at. One method that stands out a bit is getClasses where an alternative would be to specify how it is laid out in the content stream and then provide a helper that decodes the stream into map entries. I'm not saying should need to change to this, just something to think about in the context of providing more support for specific sections / content type. Minor nit noticed in passing is that ClassesEntry.setValue should probably throw UOE. -Alan. From paul.sandoz at oracle.com Tue Jun 19 07:33:42 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 19 Jun 2012 16:33:42 +0200 Subject: Service provider module dependency resolving webrev Message-ID: <2FD90826-88FD-4FA8-B59A-4DF558EB8D6C@oracle.com> Hi, http://cr.openjdk.java.net/~psandoz/jigsaw/resolver-services/webrev/ - "permits" does not affect the resolving of service provider modules dependencies - service provider modules dependencies are resolved after the explicit dependencies have been successfully resolved. - if service provider modules or their dependencies are service consumer modules then the correspond service provide modules are resolved after the explicit dependences and so on. See below for an example trace output. All service related tests pass, but they are so few... Paul. [exec] | Configuring [mapp@=1.0] using library build/mlib [exec] | resolving ROOT requires mapp@=1.0 [exec] | - trying mapp at 1.0 [exec] | - pushing service provider module dependency stringer.StringTransformer -> requires optional mhasher [exec] | - pushing service provider module dependency stringer.StringTransformer -> requires optional mrotter [exec] | -- resolving mapp at 1.0 requires mstringer [exec] | --- trying mstringer at 1.0 [exec] | ---- resolving mstringer at 1.0 requires synthesized java.base@>=8 [exec] | ----- trying java.base [exec] | ----- pushing service provider module dependency java.nio.charset.spi.CharsetProvider -> requires optional sun.charsets [exec] | ----- pushing service provider module dependency java.nio.file.spi.FileSystemProvider -> requires optional jdk.zipfs [exec] | ----- pushing service provider module dependency sun.net.spi.nameservice.NameServiceDescriptor -> requires optional jdk.jndi [exec] | ------ resolving jdk.base at 8-ea requires local optional jdk.desktop.internal@=8-ea [exec] | ------- trying jdk.desktop.internal at 8-ea [exec] | ------- pushing service provider module dependency javax.print.PrintServiceLookup -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.print.StreamPrintServiceFactory -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.sound.midi.spi.MidiDeviceProvider -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.sound.midi.spi.MidiFileReader -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.sound.midi.spi.MidiFileWriter -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.sound.midi.spi.SoundbankReader -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.sound.sampled.spi.AudioFileReader -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.sound.sampled.spi.AudioFileWriter -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.sound.sampled.spi.FormatConversionProvider -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.sound.sampled.spi.MixerProvider -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency sun.java2d.cmm.PCMM -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency sun.java2d.pipe.RenderingEngine -> requires optional jdk.desktop [exec] | -------- resolving jdk.desktop at 8-ea requires local jdk.base.internal@=8-ea [exec] | --------- resolving jdk.desktop at 8-ea requires jdk.jaxp@=8-ea [exec] | ---------- trying jdk.jaxp at 8-ea [exec] | ----------- resolving jdk.jaxp at 8-ea requires jdk.base@=8-ea [exec] | ------------ resolving jdk.jaxp at 8-ea requires synthesized java.base@>=8 [exec] | ------------- resolving jdk.desktop at 8-ea requires jdk.logging@=8-ea [exec] | -------------- trying jdk.logging at 8-ea [exec] | --------------- resolving jdk.logging at 8-ea requires jdk.base.internal@=8-ea [exec] | ---------------- resolving jdk.logging at 8-ea requires synthesized java.base@>=8 [exec] | ----------------- resolving jdk.desktop at 8-ea requires jdk.prefs@=8-ea [exec] | ------------------ trying jdk.prefs at 8-ea [exec] | ------------------- resolving jdk.prefs at 8-ea requires local jdk.base.internal@=8-ea [exec] | -------------------- resolving jdk.prefs at 8-ea requires jdk.jaxp@=8-ea [exec] | --------------------- resolving jdk.prefs at 8-ea requires synthesized java.base@>=8 [exec] | ---------------------- resolving jdk.desktop at 8-ea requires sun.charsets.internal@=8-ea [exec] | ----------------------- trying sun.charsets.internal at 8-ea [exec] | ------------------------ resolving sun.charsets at 8-ea requires local jdk.base.internal@=8-ea [exec] | ------------------------- resolving sun.charsets at 8-ea requires synthesized java.base@>=8 [exec] | -------------------------- resolving jdk.desktop at 8-ea requires synthesized java.base@>=8 [exec] | --------------------------- resolving jdk.base at 8-ea requires optional jdk.jaxp.internal@=8-ea [exec] | ---------------------------- resolving jdk.base at 8-ea requires local optional jdk.tls.internal@=8-ea [exec] | ----------------------------- trying jdk.tls.internal at 8-ea [exec] | ------------------------------ resolving jdk.tls at 8-ea requires local jdk.base.internal@=8-ea [exec] | ------------------------------- resolving jdk.tls at 8-ea requires synthesized java.base@>=8 [exec] | -------------------------------- resolving jdk.base at 8-ea requires local optional sun.localedata@=8-ea [exec] | --------------------------------- trying sun.localedata at 8-ea [exec] | ---------------------------------- resolving sun.localedata at 8-ea requires local jdk.base.internal@=8-ea [exec] | ----------------------------------- resolving sun.localedata at 8-ea requires synthesized java.base@>=8 [exec] | ------------------------------------ resolving jdk.base at 8-ea requires local optional sun.resources@=8-ea [exec] | ------------------------------------- trying sun.resources at 8-ea [exec] | -------------------------------------- resolving sun.resources at 8-ea requires local jdk.base.internal@=8-ea [exec] | --------------------------------------- resolving sun.resources at 8-ea requires synthesized java.base@>=8 [exec] | ---------------------------------------- resolving mapp at 1.0 requires synthesized java.base@>=8 [exec] | ----------------------------------------- resolving SERVICE PROVIDER requires optional mhasher [exec] | ------------------------------------------ trying mhasher at 1.0 [exec] | ------------------------------------------- resolving mhasher at 1.0 requires mstringer [exec] | -------------------------------------------- resolving mhasher at 1.0 requires synthesized java.base@>=8 [exec] | --------------------------------------------- resolving SERVICE PROVIDER requires optional mrotter [exec] | ---------------------------------------------- trying mrotter at 1.0 [exec] | ----------------------------------------------- resolving mrotter at 1.0 requires mstringer [exec] | ------------------------------------------------ resolving mrotter at 1.0 requires synthesized java.base@>=8 [exec] | ------------------------------------------------- resolving SERVICE PROVIDER requires optional sun.charsets [exec] | -------------------------------------------------- resolving SERVICE PROVIDER requires optional jdk.zipfs [exec] | --------------------------------------------------- trying jdk.zipfs at 8-ea [exec] | ---------------------------------------------------- resolving jdk.zipfs at 8-ea requires jdk.base@=8-ea [exec] | ----------------------------------------------------- resolving jdk.zipfs at 8-ea requires synthesized java.base@>=8 [exec] | ------------------------------------------------------ resolving SERVICE PROVIDER requires optional jdk.jndi [exec] | ------------------------------------------------------- trying jdk.jndi at 8-ea [exec] | -------------------------------------------------------- resolving jdk.jndi at 8-ea requires local jdk.auth.internal@=8-ea [exec] | --------------------------------------------------------- trying jdk.auth.internal at 8-ea [exec] | ---------------------------------------------------------- resolving jdk.auth at 8-ea requires local jdk.base.internal@=8-ea [exec] | ----------------------------------------------------------- resolving jdk.auth at 8-ea requires jdk.logging@=8-ea [exec] | ------------------------------------------------------------ resolving jdk.auth at 8-ea requires synthesized java.base@>=8 [exec] | ------------------------------------------------------------- resolving jdk.jndi at 8-ea requires local jdk.base.internal@=8-ea [exec] | -------------------------------------------------------------- resolving jdk.jndi at 8-ea requires optional jdk.desktop@=8-ea [exec] | --------------------------------------------------------------- resolving jdk.jndi at 8-ea requires jdk.rmi@=8-ea [exec] | ---------------------------------------------------------------- trying jdk.rmi at 8-ea [exec] | ----------------------------------------------------------------- resolving jdk.rmi at 8-ea requires jdk.base.internal@=8-ea [exec] | ------------------------------------------------------------------ resolving jdk.rmi at 8-ea requires jdk.logging@=8-ea [exec] | ------------------------------------------------------------------- resolving jdk.rmi at 8-ea requires jdk.tls@=8-ea [exec] | -------------------------------------------------------------------- resolving jdk.rmi at 8-ea requires synthesized java.base@>=8 [exec] | --------------------------------------------------------------------- resolving jdk.jndi at 8-ea requires jdk.tls.internal@=8-ea [exec] | ---------------------------------------------------------------------- resolving jdk.jndi at 8-ea requires synthesized java.base@>=8 [exec] | ----------------------------------------------------------------------- resolving SERVICE PROVIDER requires optional jdk.desktop From Alan.Bateman at oracle.com Tue Jun 19 08:43:05 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 19 Jun 2012 16:43:05 +0100 Subject: Service provider module dependency resolving webrev In-Reply-To: <2FD90826-88FD-4FA8-B59A-4DF558EB8D6C@oracle.com> References: <2FD90826-88FD-4FA8-B59A-4DF558EB8D6C@oracle.com> Message-ID: <4FE09E09.50807@oracle.com> On 19/06/2012 15:33, Paul Sandoz wrote: > Hi, > > http://cr.openjdk.java.net/~psandoz/jigsaw/resolver-services/webrev/ > > - "permits" does not affect the resolving of service provider modules dependencies > > - service provider modules dependencies are resolved after the explicit dependencies have been successfully resolved. > > - if service provider modules or their dependencies are service consumer modules then the correspond service provide modules > are resolved after the explicit dependences and so on. > I haven't tried your patch but I wonder about cases where the service provide modules have transient dependencies on modules that are in the initial chosen set, something like: M requires X; requires service S; P requires X @ 1.0 provides service S with S1; X @ 1.0 X @ 2.0 With today's implementation then M, P, and X at 1.0 will be chosen. If I understand the proposal then M and X at 2.0 will be initially chosen and then P and its dependencies will need to be worked out. With the proposal then does this result in the same answer or will it fail because X at 1.0 and X2.0 cannot be the same configuration? -Alan. From chris.hegarty at oracle.com Tue Jun 19 08:46:03 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 19 Jun 2012 16:46:03 +0100 Subject: Module file parse API In-Reply-To: References: <4FD86E8C.3080208@oracle.com> <4FDF23C8.3070402@oracle.com> <679B7267-55C7-4E1E-82BE-78CEDBBECD27@oracle.com> <4FDF3835.8070201@oracle.com> <415CB6CB-78F1-4FAD-9B71-B421B0F7AB27@oracle.com> <4FDF5C1C.2030202@oracle.com> Message-ID: <4FE09EBB.70509@oracle.com> On 19/06/12 09:01, Paul Sandoz wrote: > > On Jun 18, 2012, at 6:49 PM, chris hegarty wrote: > >> On 18/06/2012 17:37, Paul Sandoz wrote: >>> On Jun 18, 2012, at 4:16 PM, Chris Hegarty wrote: >>>>> - IIRC the complete size of the jmod file is encoded in the file itself, thus after the file header has been read we can wrap everything around a CountingInputStream. >>>> >>>> I really like this idea too, but I need to think carefully about the impact of concatenating some rogue module to another module file. >>> >>> Signed modules? >>> >>> Why does using a CountingInputStream over the content introduce a security issue? >> >> I was thinking that if the reader/installer was able to parse multiple module files from a single input stream something like this may be a problem... >> cat foo.jmod bar.jmod> foo.jmod >> jmod install foo.jmod >> jmod ls >> foo >> bar >> >> It just means that the installer is responsible for handling this situation rather than the parser. >> > > OK, i understand what you are getting at. > > I was not suggesting that the tooling or the parser directly support what you say above. > > Just suggesting something simpler: not to rule out the following capability: > > InputStream s = // some stream where multiple jmods are streamed > while (s.available() != 0) { > ModuleFileParser mfp = ModuleFile.newParser(s); > while (parser.hasNext()) { parser.next(); } > } OK, got it. I'll change the parser to support this. The Reader/Installer will then ensure that all bytes are read from the stream (rather than the parser). CountingInputStream is our friend ;-) -Chris. > > Paul. From paul.sandoz at oracle.com Tue Jun 19 09:53:51 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 19 Jun 2012 18:53:51 +0200 Subject: Service provider module dependency resolving webrev In-Reply-To: <4FE09E09.50807@oracle.com> References: <2FD90826-88FD-4FA8-B59A-4DF558EB8D6C@oracle.com> <4FE09E09.50807@oracle.com> Message-ID: On Jun 19, 2012, at 5:43 PM, Alan Bateman wrote: > On 19/06/2012 15:33, Paul Sandoz wrote: >> Hi, >> >> http://cr.openjdk.java.net/~psandoz/jigsaw/resolver-services/webrev/ >> >> - "permits" does not affect the resolving of service provider modules dependencies >> >> - service provider modules dependencies are resolved after the explicit dependencies have been successfully resolved. >> >> - if service provider modules or their dependencies are service consumer modules then the correspond service provide modules >> are resolved after the explicit dependences and so on. >> > I haven't tried your patch but I wonder about cases where the service provide modules have transient dependencies on modules that are in the initial chosen set, something like: > > M > requires X; > requires service S; > > P > requires X @ 1.0 > provides service S with S1; > > X @ 1.0 > > X @ 2.0 > > With today's implementation then M, P, and X at 1.0 will be chosen. > Yes, since the service provider dependencies are pushed after the explicit dependencies. > If I understand the proposal then M and X at 2.0 will be initially chosen and then P and its dependencies will need to be worked out. With the proposal then does this result in the same answer or will it fail because X at 1.0 and X2.0 cannot be the same configuration? > I would need to verify but IIUC it will fail, since the service provider dependency is optional module P will not be included in the configuration, the resolver will calculate a successful solution, but will barf when verifying the optional constraint. It's swings and roundabouts :-) i can come up with situations where the current approach may produce odd results or barf. For example another that happens to be processed before M: M' requires X @ > 1.0; I am leaning towards a solution that may present a more consistent results to developers. With the current solution it can be harder to predict or understand how the system is affected by the presence of service provider modules since service consumer modules are decoupled from service provider modules. Paul. From sean.mullan at oracle.com Tue Jun 19 10:48:36 2012 From: sean.mullan at oracle.com (Sean Mullan) Date: Tue, 19 Jun 2012 13:48:36 -0400 Subject: JCE providers as services Message-ID: <4FE0BB74.1010905@oracle.com> I'm working on trying to convert the JCE providers to modular services. In the JDK, more than one module will need to export/provide the same JCE service interface. Is this a problem? For example: 1. jdk.crypto at 8-ea ... provides service java.security.Provider with com.sun.crypto.provider.SunJCE 2. jdk.xmldsig at 8-ea ... provides service java.security.Provider with org.jcp.xmldsig.internal.dom.XMLDSigRI Will a service consumer be able to discover all of the installed JCE services just by adding a "requires service java.security.Provider"? In other words, I'm assuming it doesn't need to also add the modules exporting that service as a dependency. --Sean From Alan.Bateman at oracle.com Tue Jun 19 11:33:34 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 19 Jun 2012 19:33:34 +0100 Subject: JCE providers as services In-Reply-To: <4FE0BB74.1010905@oracle.com> References: <4FE0BB74.1010905@oracle.com> Message-ID: <4FE0C5FE.6010002@oracle.com> On 19/06/2012 18:48, Sean Mullan wrote: > I'm working on trying to convert the JCE providers to modular services. > > In the JDK, more than one module will need to export/provide the same > JCE service interface. Is this a problem? > > For example: > > 1. jdk.crypto at 8-ea > ... > provides service java.security.Provider > with com.sun.crypto.provider.SunJCE > > 2. jdk.xmldsig at 8-ea > ... > provides service java.security.Provider > with org.jcp.xmldsig.internal.dom.XMLDSigRI > > Will a service consumer be able to discover all of the installed JCE > services just by adding a "requires service java.security.Provider"? > In other words, I'm assuming it doesn't need to also add the modules > exporting that service as a dependency. That's right, and you use java.util.ServiceLoader to iterate over the service providers at runtime. -Alan. From paul.sandoz at oracle.com Tue Jun 19 11:37:29 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 19 Jun 2012 20:37:29 +0200 Subject: JCE providers as services In-Reply-To: <4FE0BB74.1010905@oracle.com> References: <4FE0BB74.1010905@oracle.com> Message-ID: On Jun 19, 2012, at 7:48 PM, Sean Mullan wrote: > I'm working on trying to convert the JCE providers to modular services. > > In the JDK, more than one module will need to export/provide the same JCE service interface. Is this a problem? > No. The "provide service S with I" clause does not export the service interface S (in this case java.security.Provider) nor does it export the service provider class I (in this case com.sun.crypto.provider.SunJCE or org.jcp.xmldsig.internal.dom.XMLDSigRI). In your example there is an implicit requires of the jdk.base module that exports the service interface java.security.Provider. > For example: > > 1. jdk.crypto at 8-ea > ... > provides service java.security.Provider > with com.sun.crypto.provider.SunJCE > > 2. jdk.xmldsig at 8-ea > ... > provides service java.security.Provider > with org.jcp.xmldsig.internal.dom.XMLDSigRI > > Will a service consumer be able to discover all of the installed JCE services just by adding a "requires service java.security.Provider"? Yes. Paul. > In other words, I'm assuming it doesn't need to also add the modules exporting that service as a dependency. > From sean.mullan at oracle.com Tue Jun 19 13:19:13 2012 From: sean.mullan at oracle.com (Sean Mullan) Date: Tue, 19 Jun 2012 16:19:13 -0400 Subject: JCE providers as services In-Reply-To: References: <4FE0BB74.1010905@oracle.com> Message-ID: <4FE0DEC1.1010503@oracle.com> Thanks. Here are a couple of issues I have run into so far with converting JCE Providers to modular services: 1. Some Providers take an optional String argument, i.e. they can define a Provider(String) constructor. Not sure how to workaround that one yet. 2. The order of the returned providers is important and is based on the java.security configuration file. This one I can probably workaround by changing the internal provider lookup code to reorder the Providers returned by ServiceLoader.iterator(). --Sean From Alan.Bateman at oracle.com Tue Jun 19 14:11:59 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 19 Jun 2012 22:11:59 +0100 Subject: JCE providers as services In-Reply-To: <4FE0DEC1.1010503@oracle.com> References: <4FE0BB74.1010905@oracle.com> <4FE0DEC1.1010503@oracle.com> Message-ID: <4FE0EB1F.7070304@oracle.com> On 19/06/2012 21:19, Sean Mullan wrote: > Thanks. Here are a couple of issues I have run into so far with > converting JCE Providers to modular services: > > 1. Some Providers take an optional String argument, i.e. they can > define a Provider(String) constructor. Not sure how to workaround that > one yet. Is this the provider name or the description? Maybe you can add a no-arg constructor that invokes super with the name. > > 2. The order of the returned providers is important and is based on > the java.security configuration file. This one I can probably > workaround by changing the internal provider lookup code to reorder > the Providers returned by ServiceLoader.iterator(). > There isn't any way to order the providers at this time. It's come up a few times and one idea floated was to make this configurable. For now, and to get this working, then using the existing configuration file to re-order them would be a good start. -Alan. From sean.mullan at oracle.com Tue Jun 19 14:18:36 2012 From: sean.mullan at oracle.com (Sean Mullan) Date: Tue, 19 Jun 2012 17:18:36 -0400 Subject: JCE providers as services In-Reply-To: <4FE0EB1F.7070304@oracle.com> References: <4FE0BB74.1010905@oracle.com> <4FE0DEC1.1010503@oracle.com> <4FE0EB1F.7070304@oracle.com> Message-ID: <4FE0ECAC.7060800@oracle.com> On 06/19/2012 05:11 PM, Alan Bateman wrote: > On 19/06/2012 21:19, Sean Mullan wrote: >> Thanks. Here are a couple of issues I have run into so far with >> converting JCE Providers to modular services: >> >> 1. Some Providers take an optional String argument, i.e. they can >> define a Provider(String) constructor. Not sure how to workaround that >> one yet. > Is this the provider name or the description? Maybe you can add a no-arg > constructor that invokes super with the name. It's used internally to pass a configuration file to the SunPKCS11 and Ucrypto providers. I suppose I could add an init method and pass it in that way after it is instantiated and returned by the ServiceLoader. --Sean From Alan.Bateman at oracle.com Tue Jun 19 14:28:35 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 19 Jun 2012 22:28:35 +0100 Subject: JCE providers as services In-Reply-To: <4FE0ECAC.7060800@oracle.com> References: <4FE0BB74.1010905@oracle.com> <4FE0DEC1.1010503@oracle.com> <4FE0EB1F.7070304@oracle.com> <4FE0ECAC.7060800@oracle.com> Message-ID: <4FE0EF03.7050405@oracle.com> On 19/06/2012 22:18, Sean Mullan wrote: > > It's used internally to pass a configuration file to the SunPKCS11 and > Ucrypto providers. I suppose I could add an init method and pass it in > that way after it is instantiated and returned by the ServiceLoader. Is this something that the provider could pick up itself? Also if I'm using these providers today then how to I specify the configuration file if I'm not doing it through the API? -Alan From david.lloyd at redhat.com Tue Jun 19 14:39:11 2012 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 19 Jun 2012 16:39:11 -0500 Subject: JCE providers as services In-Reply-To: <4FE0DEC1.1010503@oracle.com> References: <4FE0BB74.1010905@oracle.com> <4FE0DEC1.1010503@oracle.com> Message-ID: <4FE0F17F.5040202@redhat.com> On 06/19/2012 03:19 PM, Sean Mullan wrote: > Thanks. Here are a couple of issues I have run into so far with > converting JCE Providers to modular services: > > 1. Some Providers take an optional String argument, i.e. they can define > a Provider(String) constructor. Not sure how to workaround that one yet. This doesn't seem to be in the JCryptoA specification, or have I just missed it? > 2. The order of the returned providers is important and is based on the > java.security configuration file. This one I can probably workaround by > changing the internal provider lookup code to reorder the Providers > returned by ServiceLoader.iterator(). Since these services are essentially global to a configuration, surely it's easy enough to specify, in that global configuration, what order to prefer services to load from in a general sense (i.e. not limited to Provider impls)? If not... why not? -- - DML From sean.mullan at oracle.com Tue Jun 19 14:39:21 2012 From: sean.mullan at oracle.com (Sean Mullan) Date: Tue, 19 Jun 2012 17:39:21 -0400 Subject: JCE providers as services In-Reply-To: <4FE0EF03.7050405@oracle.com> References: <4FE0BB74.1010905@oracle.com> <4FE0DEC1.1010503@oracle.com> <4FE0EB1F.7070304@oracle.com> <4FE0ECAC.7060800@oracle.com> <4FE0EF03.7050405@oracle.com> Message-ID: <4FE0F189.8020707@oracle.com> On 06/19/2012 05:28 PM, Alan Bateman wrote: > On 19/06/2012 22:18, Sean Mullan wrote: >> >> It's used internally to pass a configuration file to the SunPKCS11 and >> Ucrypto providers. I suppose I could add an init method and pass it in >> that way after it is instantiated and returned by the ServiceLoader. > Is this something that the provider could pick up itself? I suppose it could but the path may vary depending on the platform, or someone could edit the java.security file and specify their own path. > Also if I'm > using these providers today then how to I specify the configuration file > if I'm not doing it through the API? It's specified in the java.security file, ex, for Solaris: security.provider.2=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/sunpkcs11-solaris.cfg --Sean From sean.mullan at oracle.com Tue Jun 19 14:45:50 2012 From: sean.mullan at oracle.com (Sean Mullan) Date: Tue, 19 Jun 2012 17:45:50 -0400 Subject: JCE providers as services In-Reply-To: <4FE0F17F.5040202@redhat.com> References: <4FE0BB74.1010905@oracle.com> <4FE0DEC1.1010503@oracle.com> <4FE0F17F.5040202@redhat.com> Message-ID: <4FE0F30E.3070809@oracle.com> On 06/19/2012 05:39 PM, David M. Lloyd wrote: > On 06/19/2012 03:19 PM, Sean Mullan wrote: >> Thanks. Here are a couple of issues I have run into so far with >> converting JCE Providers to modular services: >> >> 1. Some Providers take an optional String argument, i.e. they can define >> a Provider(String) constructor. Not sure how to workaround that one yet. > > This doesn't seem to be in the JCryptoA specification, or have I just > missed it? It's not part of the public API, it's an internal mechanism used by our provider lookup/registration code to pass an additional config file to some of our providers. --Sean >> 2. The order of the returned providers is important and is based on the >> java.security configuration file. This one I can probably workaround by >> changing the internal provider lookup code to reorder the Providers >> returned by ServiceLoader.iterator(). > > Since these services are essentially global to a configuration, surely > it's easy enough to specify, in that global configuration, what order to > prefer services to load from in a general sense (i.e. not limited to > Provider impls)? If not... why not? > From weijun.wang at oracle.com Tue Jun 19 18:47:35 2012 From: weijun.wang at oracle.com (Weijun Wang) Date: Wed, 20 Jun 2012 09:47:35 +0800 Subject: JCE providers as services In-Reply-To: <4FE0F189.8020707@oracle.com> References: <4FE0BB74.1010905@oracle.com> <4FE0DEC1.1010503@oracle.com> <4FE0EB1F.7070304@oracle.com> <4FE0ECAC.7060800@oracle.com> <4FE0EF03.7050405@oracle.com> <4FE0F189.8020707@oracle.com> Message-ID: <4FE12BB7.4070407@oracle.com> On 06/20/2012 05:39 AM, Sean Mullan wrote: > On 06/19/2012 05:28 PM, Alan Bateman wrote: >> On 19/06/2012 22:18, Sean Mullan wrote: >>> >>> It's used internally to pass a configuration file to the SunPKCS11 and >>> Ucrypto providers. I suppose I could add an init method and pass it in >>> that way after it is instantiated and returned by the ServiceLoader. >> Is this something that the provider could pick up itself? > > I suppose it could but the path may vary depending on the platform, or > someone could edit the java.security file and specify their own path. Or someone might configure 2 SunPKCS11 providers, one Solaris and one NSS. -Max > >> Also if I'm >> using these providers today then how to I specify the configuration file >> if I'm not doing it through the API? > > It's specified in the java.security file, ex, for Solaris: > > security.provider.2=sun.security.pkcs11.SunPKCS11 > ${java.home}/lib/security/sunpkcs11-solaris.cfg > > --Sean > > > From erik.joelsson at oracle.com Wed Jun 20 03:04:05 2012 From: erik.joelsson at oracle.com (Erik Joelsson) Date: Wed, 20 Jun 2012 12:04:05 +0200 Subject: Build-infra with jigsaw, progress update Message-ID: <4FE1A015.9090702@oracle.com> Hello, A progress update on the build-infra with jigsaw work. In short: I've managed to bootstrap build a minimal workjdk and using this I have compiled all java classes in langtools, corba, jaxp, jaxws and jdk in module mode in one go, using the code layout from the shuffle-scripts. Details follows. The source is just in my local repo. It's dependant on the changes we pushed into jdk/build last week. It also needs the moduleinfo files which still haven't been pushed afaik. I would also like to clean it up some before being ready for consumption by the jigsaw forest. I can publish a webrev if anyone wants to try it out. Only tested on linux_x64 so far, but probably works on 32bit too. Here is a description of the currently implemented bootstrap order. I will experiment with some other approaches too. Feedback is welcome. 1. langtools-bootstrap In langtools: * Compile buildtools (stripprop and genstubs, this also includes an almost full compile of the langtools source since genstubs depends on it), target bootjdk * Process properties and generate sources * Generate stubs for module and jigsaw classes in jdk repo * Compile langtools using the stubs, target bootjdk 2. hotspot * Build hotspot (nothing new here) 3. jdk-base * Generate sources in the jdk.base module * Compile classes in jdk.base in legacy mode * Import libraries and jars from hotspot * Compile the native libraries in jdk.base (libzip, libjava, libjli, libverify, libfdlibm, libnio, libnet) * Generate jvm.cfg and currency.data * Compile launchers (not all are needed at this point, but no dependencies block compiling all of them) * Construct the workjdk image (A separate image might not be needed, but it keeps things cleaner) Copies the libs above+libjvm, the needed launchers (java, javac, jmod and javadoc), the classes from jdk.base and langtools bootstrap into workjdk/classes, and finally a jigsaw library is created using jmod in workjdk/lib/modules. 4. corba-gensrc * Generate sources in corba (IDL, properties...) 5. jdk-modules * Generate the rest of the sources and data files in the jdk repo * Use the workjdk javac to compile the newsrc and gensrc dirs of all repos except hotspot. "Compiling 15795 files in batch BUILD_JDK_ALL" Still missing: * Compiling the rest of the native libs (not hard, just a lot of redirecting the source to newsrc layout and fixing the shuffle scripts to include all those native files) * Demos and Samples? * Images creation. Next I will look at compiling the other repos separately as I suspect his will be preferable. /Erik From Alan.Bateman at oracle.com Wed Jun 20 03:15:33 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 20 Jun 2012 11:15:33 +0100 Subject: Build-infra with jigsaw, progress update In-Reply-To: <4FE1A015.9090702@oracle.com> References: <4FE1A015.9090702@oracle.com> Message-ID: <4FE1A2C5.8050804@oracle.com> On 20/06/2012 11:04, Erik Joelsson wrote: > : > The source is just in my local repo. It's dependant on the changes we > pushed into jdk/build last week. It also needs the moduleinfo files > which still haven't been pushed afaik. Thanks for the update and great to hear about the progress. We're not at the point yet where we can check-in the module-info sources but one thing that we could do for the short term is to have the shuffle scripts copy the module-info.java sources into place. -Alan From erik.joelsson at oracle.com Wed Jun 20 03:49:01 2012 From: erik.joelsson at oracle.com (Erik Joelsson) Date: Wed, 20 Jun 2012 12:49:01 +0200 Subject: Build-infra with jigsaw, progress update In-Reply-To: <4FE1A2C5.8050804@oracle.com> References: <4FE1A015.9090702@oracle.com> <4FE1A2C5.8050804@oracle.com> Message-ID: <4FE1AA9D.2070504@oracle.com> On 2012-06-20 12:15, Alan Bateman wrote: > We're not at the point yet where we can check-in the module-info > sources but one thing that we could do for the short term is to have > the shuffle scripts copy the module-info.java sources into place. > Shuffling them into place works for me. /Erik From paul.sandoz at oracle.com Wed Jun 20 04:13:42 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 20 Jun 2012 13:13:42 +0200 Subject: JCE providers as services In-Reply-To: <4FE0F17F.5040202@redhat.com> References: <4FE0BB74.1010905@oracle.com> <4FE0DEC1.1010503@oracle.com> <4FE0F17F.5040202@redhat.com> Message-ID: <13C30A8F-8459-47E2-A34B-5A950CF1B7E0@oracle.com> On Jun 19, 2012, at 11:39 PM, David M. Lloyd wrote: >> 2. The order of the returned providers is important and is based on the >> java.security configuration file. This one I can probably workaround by >> changing the internal provider lookup code to reorder the Providers >> returned by ServiceLoader.iterator(). > > Since these services are essentially global to a configuration, surely it's easy enough to specify, in that global configuration, what order to prefer services to load from in a general sense (i.e. not limited to Provider impls)? What do you mean by "in a general sense"? (because "services" is such an overloaded term i am not sure which aspects of services you are referring to). We will require iteration order of service instances to work with ServiceLoader in both classpath mode and module mode. E.g.: // list of FQCN of a service provider class to control service instance iteration order // for those classes that are present List l = ... ServiceLoader.orderWith(l).load(ServiceInterface.class); Paul. > If not... why not? > From paul.sandoz at oracle.com Wed Jun 20 06:06:51 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 20 Jun 2012 15:06:51 +0200 Subject: Service configuration/context Re: ServiceLoader.load* take 2 In-Reply-To: References: Message-ID: <7197ED2D-769C-493D-A73E-AB105375CAF9@oracle.com> On Jun 12, 2012, at 5:03 PM, Paul Sandoz wrote: > - Modify the configuration so that service information is stored directly with the configuration rather than with each context (that is stored with the configuration). Currently service provider classes are looked up by iterating through all the contexts. This is not very efficient. The configuration is the place where to optimize. > This requires some refactoring of org.openjdk.jigsaw.Configuration because for instances of Configuration (the configuration produced when compiling) such state related to services is not relevant. This is also a potential opportunity to improve the immutability of Configuration using a builder pattern for construction. Some details on improving this area: - Configuration has two constructors. One is used by the Linker/PathLinker to pass in all the configuration state. The other, that takes a subset of the complete state, is used by SimpleLibrary when parsing the serialized configuration and constructing the rest of the state while parsing. The methods Configuration.add/put are solely there for the purpose of SimpleLibrary to mutate state on Configuration. A Configuration builder could be used for both purposes. - BaseContext.freeze. This transforms mutable state into immutable state, but only for state held by BaseContext and not for sub types (because it is not overridden). This is a sure sign that a builder pattern can be used to construct mutable state, then built into an immutable object. - On Context there is code such as: public final Map moduleForLocalClassMap() { return Collections.unmodifiableMap(moduleForLocalClass); } and see the more complex case for Context.services(), which will any way move to a subtype of Configuration. This class is really trying very hard to be immutable :-) - In ContextBuilder there is casting, from line 132: if (cx instanceof LinkingContext) { ((LinkingContext) cx).addModule(mi); } if (cx instanceof Context) { URI lp = res.locationForName.get(mi.id().name()); if (lp != null) { String s = lp.getScheme(); if (s == null || !s.equals("file")) { throw new AssertionError(s); } ((Context) cx).putLibraryPathForModule(mi.id(), new File(lp)); } } this is a sign that the abstraction constructing a context is broken. Notice that a ContextFactory instance is passed to the constructor of ContextBuilder. ContextBuilder is really a ContextSetBuilder, and ContextFactory is acting a limited builder with just the build() method implemented. - Linker.LinkedContext is an internal detail thus could be a special builder that creates instances of Context. All the additional state held by Linker.LinkedContext is not relevant once the Configuration has been created, as shown by the method on Linker to return the Configuration: static Configuration run(Library lib, ContextSet cxs) - When i see LinkingContext i am inclined to think that there should be the following: interface Context // was class BaseContext interface LinkingContext extends Context abstract class AbstractContext implements Context class InstallContext extends AbstractContext // was Context class PathContext extends AbstractContext implements LinkingContext class Linker.LinkerContext extends InstallContext implements LinkingContext // was Linker.Context class InstallConfiguration extends Configuration -- service state belongs here not on InstallContext Paul. From Alan.Bateman at oracle.com Wed Jun 20 06:37:32 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 20 Jun 2012 14:37:32 +0100 Subject: Service provider module dependency resolving webrev In-Reply-To: References: <2FD90826-88FD-4FA8-B59A-4DF558EB8D6C@oracle.com> <4FE09E09.50807@oracle.com> Message-ID: <4FE1D21C.4010909@oracle.com> On 19/06/2012 17:53, Paul Sandoz wrote: > : > I would need to verify but IIUC it will fail, since the service provider dependency is optional module P will not be included in the configuration, the resolver will calculate a successful solution, but will barf when verifying the optional constraint. > > It's swings and roundabouts :-) i can come up with situations where the current approach may produce odd results or barf. For example another that happens to be processed before M: > > M' > requires X @> 1.0; > > > I am leaning towards a solution that may present a more consistent results to developers. With the current solution it can be harder to predict or understand how the system is affected by the presence of service provider modules since service consumer modules are decoupled from service provider modules. > > Paul. For this one example (M') then I assume today that it just means that the service module P is be left out of the configuration (assuming requires optional service) as that the result is M, M1, X at 2.0. In any case, i think it is critical bit is that the result can be "explained" to the user as optional dependencies and services means that the solution isn't always obvious, and in some cases it's possible to have several solutions too. As to the proposed approach then I think it will continue to some predictability issues, at least when there are multiple service provider modules involved (maybe implementing different service interfaces) with transitive dependencies on the same module, but different versions. -Alan From jaroslav.tulach at oracle.com Wed Jun 20 06:55:43 2012 From: jaroslav.tulach at oracle.com (Jaroslav Tulach) Date: Wed, 20 Jun 2012 15:55:43 +0200 Subject: Service provider module dependency resolving webrev In-Reply-To: <2FD90826-88FD-4FA8-B59A-4DF558EB8D6C@oracle.com> References: <2FD90826-88FD-4FA8-B59A-4DF558EB8D6C@oracle.com> Message-ID: <4014825.bFAaAlK6ZU@logoutik> Dne ?t 19. ?ervna 2012 16:33:42, Paul Sandoz napsal(a): > Hi, > > http://cr.openjdk.java.net/~psandoz/jigsaw/resolver-services/webrev/ This is exactly the order I was recently thinking about when trying to extend the concept of "complete repository[1]" to deal with services, but it is so NP-completenish! > - service provider modules dependencies are resolved after the explicit > dependencies have been successfully resolved. This part can be made easy with complete repository[1]. > - if service provider modules or their dependencies are service consumer > modules then the correspond service provide modules are resolved after the > explicit dependences and so on. In this step you have to select one of the modules that "provides a service" which has been requested by some module in the first step. This is the part when the algorithm turns into 3-SAT solver. No surprise, things may get unclear and complicated (which is how I understood your further discussion with Alan). -jt [1] http://wiki.apidesign.org/wiki/RangeDependenciesAnalysed#Complete_Module_Repository_with_Ranges > > [exec] | Configuring [mapp@=1.0] using library build/mlib > [exec] | resolving ROOT requires mapp@=1.0 > [exec] | - trying mapp at 1.0 > [exec] | - pushing service provider module dependency > stringer.StringTransformer -> requires optional mhasher [exec] | - > pushing service provider module dependency stringer.StringTransformer -> > requires optional mrotter [exec] | -- resolving mapp at 1.0 requires > mstringer > [exec] | --- trying mstringer at 1.0 > [exec] | ---- resolving mstringer at 1.0 requires synthesized > java.base@>=8 [exec] | ----- trying java.base > [exec] | ----- pushing service provider module dependency > java.nio.charset.spi.CharsetProvider -> requires optional sun.charsets > [exec] | ----- pushing service provider module dependency > java.nio.file.spi.FileSystemProvider -> requires optional jdk.zipfs [exec] > | ----- pushing service provider module dependency > sun.net.spi.nameservice.NameServiceDescriptor -> requires optional jdk.jndi > [exec] | ------ resolving jdk.base at 8-ea requires local optional > jdk.desktop.internal@=8-ea [exec] | ------- trying > jdk.desktop.internal at 8-ea > [exec] | ------- pushing service provider module dependency > javax.print.PrintServiceLookup -> requires optional jdk.desktop [exec] | > ------- pushing service provider module dependency > javax.print.StreamPrintServiceFactory -> requires optional jdk.desktop > [exec] | ------- pushing service provider module dependency > javax.sound.midi.spi.MidiDeviceProvider -> requires optional jdk.desktop > [exec] | ------- pushing service provider module dependency > javax.sound.midi.spi.MidiFileReader -> requires optional jdk.desktop [exec] > | ------- pushing service provider module dependency > javax.sound.midi.spi.MidiFileWriter -> requires optional jdk.desktop [exec] > | ------- pushing service provider module dependency > javax.sound.midi.spi.SoundbankReader -> requires optional jdk.desktop > [exec] | ------- pushing service provider module dependency > javax.sound.sampled.spi.AudioFileReader -> requires optional jdk.desktop > [exec] | ------- pushing service provider module dependency > javax.sound.sampled.spi.AudioFileWriter -> requires optional jdk.desktop > [exec] | ------- pushing service provider module dependency > javax.sound.sampled.spi.FormatConversionProvider -> requires optional > jdk.desktop [exec] | ------- pushing service provider module dependency > javax.sound.sampled.spi.MixerProvider -> requires optional jdk.desktop > [exec] | ------- pushing service provider module dependency > sun.java2d.cmm.PCMM -> requires optional jdk.desktop [exec] | ------- > pushing service provider module dependency sun.java2d.pipe.RenderingEngine > -> requires optional jdk.desktop [exec] | -------- resolving > jdk.desktop at 8-ea requires local jdk.base.internal@=8-ea [exec] | > --------- resolving jdk.desktop at 8-ea requires jdk.jaxp@=8-ea [exec] | > ---------- trying jdk.jaxp at 8-ea > [exec] | ----------- resolving jdk.jaxp at 8-ea requires jdk.base@=8-ea > [exec] | ------------ resolving jdk.jaxp at 8-ea requires synthesized > java.base@>=8 [exec] | ------------- resolving jdk.desktop at 8-ea requires > jdk.logging@=8-ea [exec] | -------------- trying jdk.logging at 8-ea > [exec] | --------------- resolving jdk.logging at 8-ea requires > jdk.base.internal@=8-ea [exec] | ---------------- resolving > jdk.logging at 8-ea requires synthesized java.base@>=8 [exec] | > ----------------- resolving jdk.desktop at 8-ea requires jdk.prefs@=8-ea > [exec] | ------------------ trying jdk.prefs at 8-ea > [exec] | ------------------- resolving jdk.prefs at 8-ea requires local > jdk.base.internal@=8-ea [exec] | -------------------- resolving > jdk.prefs at 8-ea requires jdk.jaxp@=8-ea [exec] | --------------------- > resolving jdk.prefs at 8-ea requires synthesized java.base@>=8 [exec] | > ---------------------- resolving jdk.desktop at 8-ea requires > sun.charsets.internal@=8-ea [exec] | ----------------------- trying > sun.charsets.internal at 8-ea [exec] | ------------------------ resolving > sun.charsets at 8-ea requires local jdk.base.internal@=8-ea [exec] | > ------------------------- resolving sun.charsets at 8-ea requires synthesized > java.base@>=8 [exec] | -------------------------- resolving > jdk.desktop at 8-ea requires synthesized java.base@>=8 [exec] | > --------------------------- resolving jdk.base at 8-ea requires optional > jdk.jaxp.internal@=8-ea [exec] | ---------------------------- resolving > jdk.base at 8-ea requires local optional jdk.tls.internal@=8-ea [exec] | > ----------------------------- trying jdk.tls.internal at 8-ea [exec] | > ------------------------------ resolving jdk.tls at 8-ea requires local > jdk.base.internal@=8-ea [exec] | ------------------------------- > resolving jdk.tls at 8-ea requires synthesized java.base@>=8 [exec] | > -------------------------------- resolving jdk.base at 8-ea requires local > optional sun.localedata@=8-ea [exec] | --------------------------------- > trying sun.localedata at 8-ea [exec] | ---------------------------------- > resolving sun.localedata at 8-ea requires local jdk.base.internal@=8-ea [exec] > | ----------------------------------- resolving sun.localedata at 8-ea > requires synthesized java.base@>=8 [exec] | > ------------------------------------ resolving jdk.base at 8-ea requires local > optional sun.resources@=8-ea [exec] | > ------------------------------------- trying sun.resources at 8-ea [exec] | > -------------------------------------- resolving sun.resources at 8-ea > requires local jdk.base.internal@=8-ea [exec] | > --------------------------------------- resolving sun.resources at 8-ea > requires synthesized java.base@>=8 [exec] | > ---------------------------------------- resolving mapp at 1.0 requires > synthesized java.base@>=8 [exec] | > ----------------------------------------- resolving SERVICE PROVIDER > requires optional mhasher [exec] | > ------------------------------------------ trying mhasher at 1.0 [exec] | > ------------------------------------------- resolving mhasher at 1.0 requires > mstringer [exec] | -------------------------------------------- resolving > mhasher at 1.0 requires synthesized java.base@>=8 [exec] | > --------------------------------------------- resolving SERVICE PROVIDER > requires optional mrotter [exec] | > ---------------------------------------------- trying mrotter at 1.0 [exec] | > ----------------------------------------------- resolving mrotter at 1.0 > requires mstringer [exec] | > ------------------------------------------------ resolving mrotter at 1.0 > requires synthesized java.base@>=8 [exec] | > ------------------------------------------------- resolving SERVICE > PROVIDER requires optional sun.charsets [exec] | > -------------------------------------------------- resolving SERVICE > PROVIDER requires optional jdk.zipfs [exec] | > --------------------------------------------------- trying jdk.zipfs at 8-ea > [exec] | ---------------------------------------------------- resolving > jdk.zipfs at 8-ea requires jdk.base@=8-ea [exec] | > ----------------------------------------------------- resolving > jdk.zipfs at 8-ea requires synthesized java.base@>=8 [exec] | > ------------------------------------------------------ resolving SERVICE > PROVIDER requires optional jdk.jndi [exec] | > ------------------------------------------------------- trying > jdk.jndi at 8-ea [exec] | > -------------------------------------------------------- resolving > jdk.jndi at 8-ea requires local jdk.auth.internal@=8-ea [exec] | > --------------------------------------------------------- trying > jdk.auth.internal at 8-ea [exec] | > ---------------------------------------------------------- resolving > jdk.auth at 8-ea requires local jdk.base.internal@=8-ea [exec] | > ----------------------------------------------------------- resolving > jdk.auth at 8-ea requires jdk.logging@=8-ea [exec] | > ------------------------------------------------------------ resolving > jdk.auth at 8-ea requires synthesized java.base@>=8 [exec] | > ------------------------------------------------------------- resolving > jdk.jndi at 8-ea requires local jdk.base.internal@=8-ea [exec] | > -------------------------------------------------------------- resolving > jdk.jndi at 8-ea requires optional jdk.desktop@=8-ea [exec] | > --------------------------------------------------------------- resolving > jdk.jndi at 8-ea requires jdk.rmi@=8-ea [exec] | > ---------------------------------------------------------------- trying > jdk.rmi at 8-ea [exec] | > ----------------------------------------------------------------- resolving > jdk.rmi at 8-ea requires jdk.base.internal@=8-ea [exec] | > ------------------------------------------------------------------ > resolving jdk.rmi at 8-ea requires jdk.logging@=8-ea [exec] | > ------------------------------------------------------------------- > resolving jdk.rmi at 8-ea requires jdk.tls@=8-ea [exec] | > -------------------------------------------------------------------- > resolving jdk.rmi at 8-ea requires synthesized java.base@>=8 [exec] | > --------------------------------------------------------------------- > resolving jdk.jndi at 8-ea requires jdk.tls.internal@=8-ea [exec] | > ---------------------------------------------------------------------- > resolving jdk.jndi at 8-ea requires synthesized java.base@>=8 [exec] | > ----------------------------------------------------------------------- > resolving SERVICE PROVIDER requires optional jdk.desktop From paul.sandoz at oracle.com Wed Jun 20 07:39:58 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 20 Jun 2012 16:39:58 +0200 Subject: Service provider module dependency resolving webrev In-Reply-To: <4FE1D21C.4010909@oracle.com> References: <2FD90826-88FD-4FA8-B59A-4DF558EB8D6C@oracle.com> <4FE09E09.50807@oracle.com> <4FE1D21C.4010909@oracle.com> Message-ID: On Jun 20, 2012, at 3:37 PM, Alan Bateman wrote: > On 19/06/2012 17:53, Paul Sandoz wrote: >> : >> I would need to verify but IIUC it will fail, since the service provider dependency is optional module P will not be included in the configuration, the resolver will calculate a successful solution, but will barf when verifying the optional constraint. >> >> It's swings and roundabouts :-) i can come up with situations where the current approach may produce odd results or barf. For example another that happens to be processed before M: >> >> M' >> requires X @> 1.0; >> >> >> I am leaning towards a solution that may present a more consistent results to developers. With the current solution it can be harder to predict or understand how the system is affected by the presence of service provider modules since service consumer modules are decoupled from service provider modules. >> >> Paul. > For this one example (M') then I assume today that it just means that the service module P is be left out of the configuration (assuming requires optional service) as that the result is M, M1, X at 2.0. > Yes. > In any case, i think it is critical bit is that the result can be "explained" to the user as optional dependencies and services means that the solution isn't always obvious, and in some cases it's possible to have several solutions too. > Yes, that was a motivation to separate out the resolution. > As to the proposed approach then I think it will continue to some predictability issues, at least when there are multiple service provider modules involved (maybe implementing different service interfaces) with transitive dependencies on the same module, but different versions. > Indeed. I wanted to bucket things into phases as i think that will make a little easier to grok: Phase 0: application module (explicit) dependency resolution Phase 1: service provider dependency resolution as a result of traversing service consumer modules in Phase 0 Phase 2: service provider dependency resolution as a result of traversing service consumer modules in Phase 1 ... Phase N: service provider dependency resolution as a result of traversing service consumer modules in Phase N - 1 Paul. From michael.x.mcmahon at oracle.com Wed Jun 20 08:38:08 2012 From: michael.x.mcmahon at oracle.com (Michael McMahon) Date: Wed, 20 Jun 2012 16:38:08 +0100 Subject: Build-infra with jigsaw, progress update In-Reply-To: <4FE1AA9D.2070504@oracle.com> References: <4FE1A015.9090702@oracle.com> <4FE1A2C5.8050804@oracle.com> <4FE1AA9D.2070504@oracle.com> Message-ID: <4FE1EE60.6020208@oracle.com> On 20/06/12 11:49, Erik Joelsson wrote: > > > On 2012-06-20 12:15, Alan Bateman wrote: >> We're not at the point yet where we can check-in the module-info >> sources but one thing that we could do for the short term is to have >> the shuffle scripts copy the module-info.java sources into place. >> > Shuffling them into place works for me. > > /Erik Okay, I'll look at updating the script to do that. - Michael From alan.bateman at oracle.com Wed Jun 20 08:41:11 2012 From: alan.bateman at oracle.com (alan.bateman at oracle.com) Date: Wed, 20 Jun 2012 15:41:11 +0000 Subject: hg: jigsaw/jigsaw/jdk: Swing Look and Feels should be exported Message-ID: <20120620154132.35EFD47A30@hg.openjdk.java.net> Changeset: 29e9f2da3338 Author: alanb Date: 2012-06-20 16:31 +0100 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/29e9f2da3338 Swing Look and Feels should be exported ! make/modules/modules.group From paul.sandoz at oracle.com Wed Jun 20 08:42:44 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 20 Jun 2012 17:42:44 +0200 Subject: Service provider module dependency resolving webrev In-Reply-To: <4014825.bFAaAlK6ZU@logoutik> References: <2FD90826-88FD-4FA8-B59A-4DF558EB8D6C@oracle.com> <4014825.bFAaAlK6ZU@logoutik> Message-ID: On Jun 20, 2012, at 3:55 PM, Jaroslav Tulach wrote: > Dne ?t 19. ?ervna 2012 16:33:42, Paul Sandoz napsal(a): >> Hi, >> >> http://cr.openjdk.java.net/~psandoz/jigsaw/resolver-services/webrev/ > > This is exactly the order I was recently thinking about when trying to extend > the concept of "complete repository[1]" to deal with services, but it is so > NP-completenish! > I think you have invented a new term :-) >> - service provider modules dependencies are resolved after the explicit >> dependencies have been successfully resolved. > > This part can be made easy with complete repository[1]. > Would you be able to elaborate in layman's terms? I think i might grok it: we essentially treat each service provider module dependency as a direct (or top-level) dependency. >> - if service provider modules or their dependencies are service consumer >> modules then the correspond service provide modules are resolved after the >> explicit dependences and so on. > > In this step you have to select one of the modules that "provides a service" > which has been requested by some module in the first step. This is the part > when the algorithm turns into 3-SAT solver. > I would very like to do a mind meld with you on this! It's interesting in that, with the current Resolver implementation, when you push service provider module dependency resolution to the end then there is no longer any backtracking above those dependencies, since they are optional. Which i think means one can break things down into ordered sub-phases resolving the dependencies of each service provider module (which IIUC is still NP-Complete as you point out). That may make it easier to report service provider module dependency resolution issues to the developer. > No surprise, things may get unclear and complicated (which is how I understood > your further discussion with Alan). Yes, i am trying to run the resolver in my head. It's concerns me if developers have to do the same. Paul. From sean.mullan at oracle.com Wed Jun 20 09:28:33 2012 From: sean.mullan at oracle.com (Sean Mullan) Date: Wed, 20 Jun 2012 12:28:33 -0400 Subject: Module file parse API In-Reply-To: <4FD86E8C.3080208@oracle.com> References: <4FD86E8C.3080208@oracle.com> Message-ID: <4FE1FA31.9090306@oracle.com> Hi Chris, I've just been reviewing the changes to Signer.java. With the new code, it looks like the caller is now responsible for verifying the hashes whereas in the old code, the Reader would automatically take care of it for you. Can you explain why you changed that? Thanks, Sean On 06/13/2012 06:42 AM, Chris Hegarty wrote: > Hi, > > A while back it was suggested that it may be useful to define a pull > parser type API for reading from a module file. I gives separation of > the parsing of the module file from the writing of it (either installing > into the module library, or extraction elsewhere ). > > I've prototyped such an API and updated the ModuleFile.Reader to use it. > The first four files in the webrev contain the API, its implementation, > and a test that demonstrates some sample usage ( extraction of one or > more classes, listing sections, verifying hashes ). The remainder of the > changes simply updates some of the tools for installing and handling > signed modules. > > http://cr.openjdk.java.net/~chegar/jigsaw/moduleParser_webrev.00/webrev/ > > Thanks, > -Chris. From chris.hegarty at oracle.com Wed Jun 20 09:49:55 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 20 Jun 2012 17:49:55 +0100 Subject: Module file parse API In-Reply-To: <4FE1FA31.9090306@oracle.com> References: <4FD86E8C.3080208@oracle.com> <4FE1FA31.9090306@oracle.com> Message-ID: <4FE1FF33.3090505@oracle.com> On 20/06/2012 17:28, Sean Mullan wrote: > Hi Chris, > > I've just been reviewing the changes to Signer.java. With the new code, > it looks like the caller is now responsible for verifying the hashes > whereas in the old code, the Reader would automatically take care of it > for you. Can you explain why you changed that? The parser is just that, a parser. It generates the hashes of section data and exposes the section headers hashes, giving flexibility to the caller on how to handle them ( you may decide to have a non validating implementation! ). The previous way of doing this was just a hack that preventing the Reader from actually writing any data, 'extract = false', when the reader is actually an installer. Now that Signer can invoke the parser directly rather than creating a Reader/Installer we can actually simplify the Reader/Installer so that it always extracts the content. I found this to be a more flexible and cleaner solution. Sorry if I've missed the point of your question. Was this what you were asking? -Chris. > > Thanks, > Sean > > On 06/13/2012 06:42 AM, Chris Hegarty wrote: >> Hi, >> >> A while back it was suggested that it may be useful to define a pull >> parser type API for reading from a module file. I gives separation of >> the parsing of the module file from the writing of it (either installing >> into the module library, or extraction elsewhere ). >> >> I've prototyped such an API and updated the ModuleFile.Reader to use it. >> The first four files in the webrev contain the API, its implementation, >> and a test that demonstrates some sample usage ( extraction of one or >> more classes, listing sections, verifying hashes ). The remainder of the >> changes simply updates some of the tools for installing and handling >> signed modules. >> >> http://cr.openjdk.java.net/~chegar/jigsaw/moduleParser_webrev.00/webrev/ >> >> Thanks, >> -Chris. > From sean.mullan at oracle.com Wed Jun 20 11:07:16 2012 From: sean.mullan at oracle.com (Sean Mullan) Date: Wed, 20 Jun 2012 14:07:16 -0400 Subject: Module file parse API In-Reply-To: <4FE1FF33.3090505@oracle.com> References: <4FD86E8C.3080208@oracle.com> <4FE1FA31.9090306@oracle.com> <4FE1FF33.3090505@oracle.com> Message-ID: <4FE21154.2020604@oracle.com> On 06/20/2012 12:49 PM, Chris Hegarty wrote: > On 20/06/2012 17:28, Sean Mullan wrote: >> Hi Chris, >> >> I've just been reviewing the changes to Signer.java. With the new code, >> it looks like the caller is now responsible for verifying the hashes >> whereas in the old code, the Reader would automatically take care of it >> for you. Can you explain why you changed that? > > The parser is just that, a parser. It generates the hashes of section > data and exposes the section headers hashes, giving flexibility to the > caller on how to handle them ( you may decide to have a non validating > implementation! ). The previous way of doing this was just a hack that > preventing the Reader from actually writing any data, 'extract = false', > when the reader is actually an installer. Should we rename it to Installer then? > Now that Signer can invoke the parser directly rather than creating a > Reader/Installer we can actually simplify the Reader/Installer so that > it always extracts the content. I found this to be a more flexible and > cleaner solution. > > Sorry if I've missed the point of your question. Was this what you were > asking? Signer is an unusual case in that it is reading the module file without extracting its contents in order to calculate the hashes that it will then sign. I think my concern is that if the hashes inside the module file don't actually match what is being read, you would want to know about that regardless of whether you are extracting the contents. Having said that, I've been somewhat dubious on the overall value/purpose of the hashes inside the module file. They don't provide any security without something additional such as a signature, but when generating the signature, it has to recalculate all of the hashes again to be sure they are still correct since the module file was created. So their only value is as a checksum, but my experience with checksums is that they are usually stored separately from what they are computed over. --Sean From mandy.chung at oracle.com Wed Jun 20 18:01:03 2012 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 20 Jun 2012 18:01:03 -0700 Subject: ServiceLoader.load* take 2 In-Reply-To: References: Message-ID: <4FE2724F.3010305@oracle.com> Paul, On 6/12/2012 8:03 AM, Paul Sandoz wrote: > the first part of which is implemented by the following webrev, which changes service instance creation so that it is scoped to the configuration rather than the class loader: > > http://cr.openjdk.java.net/~psandoz/jigsaw/global-services/webrev/ I believe you want to discuss and push the service loader changes separately. I reviewed this first part which looks good overall. I also wonder if the new ModuleServiceLoader could just be an inner class in ServiceLoader and I understand your thought behind and change Loader.findServices to package-private. I'm fine if you prefer to keep it as is. LoaderPool.java L119 - Is 'servceInterfaceName' a typo (missing letter 'i' in "service") ModuleServiceLoader.java L124: does javadoc parse '' correctly? classloader.sh test - looks like you can just extend hello.sh, many.sh, and possibly optional.sh to test the different ServiceLoader.load* methods that could also be launched in a new VM if appropriate. Mandy From paul.sandoz at oracle.com Thu Jun 21 00:54:33 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 21 Jun 2012 09:54:33 +0200 Subject: ServiceLoader.load* take 2 In-Reply-To: <4FE2724F.3010305@oracle.com> References: <4FE2724F.3010305@oracle.com> Message-ID: <63E8F31D-62D8-4C7D-8DAB-26C4DFB79053@oracle.com> Hi Mandy, Thanks for reviewing. On Jun 21, 2012, at 3:01 AM, Mandy Chung wrote: > Paul, > > On 6/12/2012 8:03 AM, Paul Sandoz wrote: >> the first part of which is implemented by the following webrev, which changes service instance creation so that it is scoped to the configuration rather than the class loader: >> >> http://cr.openjdk.java.net/~psandoz/jigsaw/global-services/webrev/ > > I believe you want to discuss and push the service loader changes separately. I reviewed this first part which looks good overall. > > I also wonder if the new ModuleServiceLoader could just be an inner class in ServiceLoader and I understand your thought behind and change Loader.findServices to package-private. I'm fine if you prefer to keep it as is. > OK. > LoaderPool.java L119 - Is 'servceInterfaceName' a typo (missing letter 'i' in "service") > Grrr... from day one the 'i' key on my MacBook Pro keyboard has been dodgy. > ModuleServiceLoader.java L124: does javadoc parse '' correctly? > Yes, JavaDoc will display it in the "Type Parameters" section of the documented method. FWIW NetBeans auto-generated it for me :-) -- I will send out another webrev today with an update for the two changes. > classloader.sh test - looks like you can just extend hello.sh, many.sh, and possibly optional.sh to test the different ServiceLoader.load* methods that could also be launched in a new VM if appropriate. > Yes, there is lots of duplication here that can be fixed, plus we also need to expand the tests to include such things as transitive dependencies and second level service consumer modules. Do you mind if we tackle that as a separate set of commits? I am not too familiar with the test infrastructure and the reuse of such shall scripts (it seems like open season and we can do what we want as long as there is appropriate meta-data that jtreg understands). Any more advice on how to improve this would be greatly appreciated? Paul. From jaroslav.tulach at oracle.com Thu Jun 21 01:05:27 2012 From: jaroslav.tulach at oracle.com (Jaroslav Tulach) Date: Thu, 21 Jun 2012 10:05:27 +0200 Subject: Direct dependency on service provider modules? was: Service provider module dependency resolving webrev In-Reply-To: References: <2FD90826-88FD-4FA8-B59A-4DF558EB8D6C@oracle.com> <4014825.bFAaAlK6ZU@logoutik> Message-ID: <1580889.kgaz451cTH@logoutik> Dne St 20. ?ervna 2012 17:42:44, Paul Sandoz napsal(a): > > >> - service provider modules dependencies are resolved after the explicit > >> dependencies have been successfully resolved. > > > > This part can be made easy with complete repository[1]. > > Would you be able to elaborate in layman's terms? Paul, please excuse me for not expressing myself clearly: I wanted to say that "resolving explicit dependencies" can be made easy with complete repository[1]. It is the easy/easier part. Resolving later the service provider modules will then be the hard part. > I think i might grok it: we essentially treat each service provider module > dependency as a direct (or top-level) dependency. Do you say that "require service xyz" declaration would mean to enable all modules which "provide service xyz with something"? E.g. like saying that a module that "require service xyz" has a direct dependency on all modules that "provide service xyz". That is radical approach. It can work if the set of modules is properly selected at beginning (e.g. NetBeans IDE version 7.2 without external modules, or Ubuntu 12.04 release) - and we know all modules that "provide service xyz" can in fact be linked under (almost) all situations. But if you apply the same rule to a collection of modules with history, provided by various parties (e.g. Maven Central), than it is very limiting - there will always be at least one module that "provides service xyz" and it cannot be enabled. Then the whole resolution should collapse, right? The approach with "direct dependency on each service provider" certainly eliminates the NP-complete problem. But it suffers from the fact that it is not very usable. Unless the set of modules is carefully pre-selected by humans (e.g. humans solve the NP-completeness first) it almost certainly leads to "can't resolve" results all over. An interesting property of "treating each service provider module dependency as direct" is that one can break the system by installing new module! E.g. your module which "requires service xyz" can be resolved. By installing new module which "provide service xyz" and is unresolvable (for any reasons) and the original module won't resolve any more either. How do you like this (mis)behavior? > >> - if service provider modules or their dependencies are service consumer > >> > >> modules then the correspond service provide modules are resolved after > >> the > >> explicit dependences and so on. > > > > In this step you have to select one of the modules that "provides a > > service" which has been requested by some module in the first step. This > > is the part when the algorithm turns into 3-SAT solver. > > I would very like to do a mind meld with you on this! > > It's interesting in that, with the current Resolver implementation, when you > push service provider module dependency resolution to the end then there is > no longer any backtracking above those dependencies, since they are > optional. Are they really optional? What is optional on "requires service xyz"? But... Heur?ka! I think I know how to overcome the NP-complete problem with services. We need to make sure we make the dependencies optional. E.g. there are no NP- complete problems with "requires service optional xyz". Thus, if we allow only optional services, we simplify everything a lot. Of course, some "requires service" are not that optional. So the question for me is: how to make the dependency system usable and still pretend all services are in fact optional? > Which i think means one can break things down into ordered sub-phases > resolving the dependencies of each service provider module (which IIUC is > still NP-Complete as you point out). That may make it easier to report > service provider module dependency resolution issues to the developer. > > No surprise, things may get unclear and complicated (which is how I > > understood your further discussion with Alan). > > Yes, i am trying to run the resolver in my head. It's concerns me if > developers have to do the same. ;-) Well, people can do this on a small system (e.g. distribution). They have their ideal vision and see the whole picture. They just need a language (e.g. requires/provides/service/optional) to describe such vision. The problem arises when few such small systems are combined together (e.g. Maven central). No human is able to see the whole picture then, still we want computers to operate on such mess effectively. -jt From erik.joelsson at oracle.com Thu Jun 21 01:14:37 2012 From: erik.joelsson at oracle.com (Erik Joelsson) Date: Thu, 21 Jun 2012 10:14:37 +0200 Subject: Build-infra with jigsaw, progress update In-Reply-To: <4FE1EE60.6020208@oracle.com> References: <4FE1A015.9090702@oracle.com> <4FE1A2C5.8050804@oracle.com> <4FE1AA9D.2070504@oracle.com> <4FE1EE60.6020208@oracle.com> Message-ID: <4FE2D7ED.7080104@oracle.com> On 2012-06-20 17:38, Michael McMahon wrote: > Okay, I'll look at updating the script to do that. > > - Michael That would be great! In my setup I've put them in a separate directory (jdk/newsrc/share/moduleinfo). The reason for this is that javac won't let me compile the classes in jdk.base in legacy mode if it sees a module-info.java anywhere. I'm not sure if this is intended or not, but that's how it currently works. /Erik From jaroslav.tulach at oracle.com Thu Jun 21 02:35:10 2012 From: jaroslav.tulach at oracle.com (Jaroslav Tulach) Date: Thu, 21 Jun 2012 11:35:10 +0200 Subject: Optional service dependencies only was: Direct dependency on service provider modules? In-Reply-To: <1580889.kgaz451cTH@logoutik> References: <2FD90826-88FD-4FA8-B59A-4DF558EB8D6C@oracle.com> <1580889.kgaz451cTH@logoutik> Message-ID: <1394015.gmxIYdubyC@logoutik> Hello Paul, here is my initial proposal for improving the dependencies mechanism to eliminate the NP-Completeness: [1] http://wiki.apidesign.org/wiki/JigsawServices Dne ?t 21. ?ervna 2012 10:05:27, Jaroslav Tulach napsal(a): > > I would very like to do a mind meld with you on this! > > > > It's interesting in that, with the current Resolver implementation, when > > you push service provider module dependency resolution to the end then > > there is no longer any backtracking above those dependencies, since they > > are optional. > > Are they really optional? What is optional on "requires service xyz"? But... As the first part of [1] argues, with "requires service" you always need backtracking. > Heur?ka! I think I know how to overcome the NP-complete problem with > services. We need to make sure we make the dependencies optional. E.g. > there are no NP- complete problems with "requires service optional xyz". > Thus, if we allow only optional services, we simplify everything a lot. > > Of course, some "requires service" are not that optional. So the question > for me is: how to make the dependency system usable and still pretend all > services are in fact optional? [1] talks about a "hint". I'd rather leave the exact language construct for providing a hint up to language experts, but I can imagine it makes a logical sense for the language. When I am JAXP module at 1.33 I probably want to recommend Xerces from JDK 1.33 as well. I should have a way to provide such hint to the Jigsaw resolution system. However there should be a way to reconfigure these hints at the deployment time. I've noticed a discussion about security providers. You were in need of changing their order and that could be also considered a "hint". Just probably a hint provided externally, at the time the application is started (e.g. not compiled). -jt From chris.hegarty at oracle.com Thu Jun 21 03:03:39 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 21 Jun 2012 11:03:39 +0100 Subject: Module file parse API In-Reply-To: <4FE21154.2020604@oracle.com> References: <4FD86E8C.3080208@oracle.com> <4FE1FA31.9090306@oracle.com> <4FE1FF33.3090505@oracle.com> <4FE21154.2020604@oracle.com> Message-ID: <4FE2F17B.3060906@oracle.com> On 20/06/2012 19:07, Sean Mullan wrote: > On 06/20/2012 12:49 PM, Chris Hegarty wrote: >> On 20/06/2012 17:28, Sean Mullan wrote: >>> Hi Chris, >>> >>> I've just been reviewing the changes to Signer.java. With the new code, >>> it looks like the caller is now responsible for verifying the hashes >>> whereas in the old code, the Reader would automatically take care of it >>> for you. Can you explain why you changed that? >> >> The parser is just that, a parser. It generates the hashes of section >> data and exposes the section headers hashes, giving flexibility to the >> caller on how to handle them ( you may decide to have a non validating >> implementation! ). The previous way of doing this was just a hack that >> preventing the Reader from actually writing any data, 'extract = false', >> when the reader is actually an installer. > > Should we rename it to Installer then? I think it should be renamed, but I'd prefer to do that in a separate commit/review. It can be difficult to review code that has been refactored and then moved to a new file. My intent here was to keep the change focused on the parser itself, and push further cleanup in subsequent changesets. >> Now that Signer can invoke the parser directly rather than creating a >> Reader/Installer we can actually simplify the Reader/Installer so that >> it always extracts the content. I found this to be a more flexible and >> cleaner solution. >> >> Sorry if I've missed the point of your question. Was this what you were >> asking? > > Signer is an unusual case in that it is reading the module file without > extracting its contents in order to calculate the hashes that it will > then sign. Right, but it is a very good use-case. The new parser API makes the code cleaner and more readable. > I think my concern is that if the hashes inside the module file don't > actually match what is being read, you would want to know about that > regardless of whether you are extracting the contents. Right, but this API is positioned at a low level. It felt very restricting to force this requirement at this level. I agree that most tools will want to fail if the hashes don't match, but I didn't want to preclude the possibility of building a non validating parser. It seems overly restrictive for the API to specify this. > Having said that, I've been somewhat dubious on the overall > value/purpose of the hashes inside the module file. They don't provide > any security without something additional such as a signature, but when > generating the signature, it has to recalculate all of the hashes again > to be sure they are still correct since the module file was created. So > their only value is as a checksum, but my experience with checksums is > that they are usually stored separately from what they are computed over. I view the hashes in an unsigned module file as a checksum also, but I think they are very useful. For example, if a tool is only interested in the classes of a module file, it can skip to the classes section, extract/process it, validate the hash, and exit without having to finish reading the remainder of the module file. This is nice, especially if the module file is being read from a remote stream, and only possible with per section hashes. -Chris. > > --Sean From paul.sandoz at oracle.com Thu Jun 21 03:36:52 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 21 Jun 2012 12:36:52 +0200 Subject: Module file parse API In-Reply-To: <4FE2F17B.3060906@oracle.com> References: <4FD86E8C.3080208@oracle.com> <4FE1FA31.9090306@oracle.com> <4FE1FF33.3090505@oracle.com> <4FE21154.2020604@oracle.com> <4FE2F17B.3060906@oracle.com> Message-ID: <11A151E3-3E3C-471F-BF2E-890FDB4928A5@oracle.com> On Jun 21, 2012, at 12:03 PM, Chris Hegarty wrote: > >>> Now that Signer can invoke the parser directly rather than creating a >>> Reader/Installer we can actually simplify the Reader/Installer so that >>> it always extracts the content. I found this to be a more flexible and >>> cleaner solution. >>> >>> Sorry if I've missed the point of your question. Was this what you were >>> asking? >> >> Signer is an unusual case in that it is reading the module file without >> extracting its contents in order to calculate the hashes that it will >> then sign. > > Right, but it is a very good use-case. The new parser API makes the code cleaner and more readable. > There is a shared layer that can be reused by the signer and installer (see below). >> I think my concern is that if the hashes inside the module file don't >> actually match what is being read, you would want to know about that >> regardless of whether you are extracting the contents. > > Right, but this API is positioned at a low level. It felt very restricting to force this requirement at this level. I agree that most tools will want to fail if the hashes don't match, but I didn't want to preclude the possibility of building a non validating parser. It seems overly restrictive for the API to specify this. > It's trivial to layer a hash accumulation and hash validation parser/reader on top of the current parser/reader. The signer can use this to skip through and obtain the accumulation if the hashes validate, in addition to its layer of checking if the module is already signed and obtaining the a length (to do its shuffle to stuff in the signed section, this should be re-factored to use a writer so the signer never should know about bits/bytes of the format). >> Having said that, I've been somewhat dubious on the overall >> value/purpose of the hashes inside the module file. They don't provide >> any security without something additional such as a signature, but when >> generating the signature, it has to recalculate all of the hashes again >> to be sure they are still correct since the module file was created. So >> their only value is as a checksum, but my experience with checksums is >> that they are usually stored separately from what they are computed over. > > I view the hashes in an unsigned module file as a checksum also, but I think they are very useful. For example, if a tool is only interested in the classes of a module file, it can skip to the classes section, extract/process it, validate the hash, and exit without having to finish reading the remainder of the module file. This is nice, especially if the module file is being read from a remote stream, and only possible with per section hashes. > Also one does not have to upload/download separate data to/from a repository for the content and the hash of the content. This also has the advantage that the repository can verify the hashes before accepting deployment of a module (instead of say uploading using multipart MIME). Paul. From chris.hegarty at oracle.com Thu Jun 21 03:43:26 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 21 Jun 2012 11:43:26 +0100 Subject: Module file parse API In-Reply-To: <11A151E3-3E3C-471F-BF2E-890FDB4928A5@oracle.com> References: <4FD86E8C.3080208@oracle.com> <4FE1FA31.9090306@oracle.com> <4FE1FF33.3090505@oracle.com> <4FE21154.2020604@oracle.com> <4FE2F17B.3060906@oracle.com> <11A151E3-3E3C-471F-BF2E-890FDB4928A5@oracle.com> Message-ID: <4FE2FACE.9070309@oracle.com> On 21/06/2012 11:36, Paul Sandoz wrote: > On Jun 21, 2012, at 12:03 PM, Chris Hegarty wrote: >> >>>> Now that Signer can invoke the parser directly rather than creating a >>>> Reader/Installer we can actually simplify the Reader/Installer so that >>>> it always extracts the content. I found this to be a more flexible and >>>> cleaner solution. >>>> >>>> Sorry if I've missed the point of your question. Was this what you were >>>> asking? >>> >>> Signer is an unusual case in that it is reading the module file without >>> extracting its contents in order to calculate the hashes that it will >>> then sign. >> >> Right, but it is a very good use-case. The new parser API makes the code cleaner and more readable. >> > > There is a shared layer that can be reused by the signer and installer (see below). Thanks Paul, you did mention this in a previous mail and I will include such a layer in the next update. >>> I think my concern is that if the hashes inside the module file don't >>> actually match what is being read, you would want to know about that >>> regardless of whether you are extracting the contents. >> >> Right, but this API is positioned at a low level. It felt very restricting to force this requirement at this level. I agree that most tools will want to fail if the hashes don't match, but I didn't want to preclude the possibility of building a non validating parser. It seems overly restrictive for the API to specify this. >> > > It's trivial to layer a hash accumulation and hash validation parser/reader on top of the current parser/reader. > > The signer can use this to skip through and obtain the accumulation if the hashes validate, in addition to its layer of checking if the module is already signed and obtaining the a length (to do its shuffle to stuff in the signed section, this should be re-factored to use a writer so the signer never should know about bits/bytes of the format). > > >>> Having said that, I've been somewhat dubious on the overall >>> value/purpose of the hashes inside the module file. They don't provide >>> any security without something additional such as a signature, but when >>> generating the signature, it has to recalculate all of the hashes again >>> to be sure they are still correct since the module file was created. So >>> their only value is as a checksum, but my experience with checksums is >>> that they are usually stored separately from what they are computed over. >> >> I view the hashes in an unsigned module file as a checksum also, but I think they are very useful. For example, if a tool is only interested in the classes of a module file, it can skip to the classes section, extract/process it, validate the hash, and exit without having to finish reading the remainder of the module file. This is nice, especially if the module file is being read from a remote stream, and only possible with per section hashes. >> > > Also one does not have to upload/download separate data to/from a repository for the content and the hash of the content. This also has the advantage that the repository can verify the hashes before accepting deployment of a module (instead of say uploading using multipart MIME). Yes, this makes the sharing/deployment environment much cleaner. -Chris. > > Paul. From paul.sandoz at oracle.com Thu Jun 21 04:16:54 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 21 Jun 2012 13:16:54 +0200 Subject: Direct dependency on service provider modules? was: Service provider module dependency resolving webrev In-Reply-To: <1580889.kgaz451cTH@logoutik> References: <2FD90826-88FD-4FA8-B59A-4DF558EB8D6C@oracle.com> <4014825.bFAaAlK6ZU@logoutik> <1580889.kgaz451cTH@logoutik> Message-ID: Hi, On Jun 21, 2012, at 10:05 AM, Jaroslav Tulach wrote: > Dne St 20. ?ervna 2012 17:42:44, Paul Sandoz napsal(a): >> >>>> - service provider modules dependencies are resolved after the explicit >>>> dependencies have been successfully resolved. >>> >>> This part can be made easy with complete repository[1]. >> >> Would you be able to elaborate in layman's terms? > > Paul, please excuse me for not expressing myself clearly: I think it is more the case of me struggling to understand. > I wanted to say that > "resolving explicit dependencies" can be made easy with complete > repository[1]. It is the easy/easier part. > > Resolving later the service provider modules will then be the hard part. > OK. >> I think i might grok it: we essentially treat each service provider module >> dependency as a direct (or top-level) dependency. > > Do you say that "require service xyz" declaration would mean to enable all > modules which "provide service xyz with something"? E.g. like saying that a > module that "require service xyz" has a direct dependency on all modules that > "provide service xyz". No, each service provider module dependency becomes an equivalent of a root dependency, like an application module. Such dependencies are processed *after* the application module dependencies have been resolved. Previously they were processed as direct dependencies of the service consumer module. BUT in either either case such dependencies are *optional*. I think that might help clear up some of the discussion. I am attaching tracing output from the resolver. Notice the "pushing service provider" and that all "resolving SERVICE PROVIDER" are at the end and they are all optional even though the "mapp" module declares "requires service ..."; > > That is radical approach. It can work if the set of modules is properly > selected at beginning (e.g. NetBeans IDE version 7.2 without external modules, > or Ubuntu 12.04 release) - and we know all modules that "provide service xyz" > can in fact be linked under (almost) all situations. > > But if you apply the same rule to a collection of modules with history, > provided by various parties (e.g. Maven Central), than it is very limiting - > there will always be at least one module that "provides service xyz" and it > cannot be enabled. Then the whole resolution should collapse, right? > > The approach with "direct dependency on each service provider" certainly > eliminates the NP-complete problem. But it suffers from the fact that it is > not very usable. Unless the set of modules is carefully pre-selected by humans > (e.g. humans solve the NP-completeness first) it almost certainly leads to > "can't resolve" results all over. > > An interesting property of "treating each service provider module dependency > as direct" is that one can break the system by installing new module! E.g. > your module which "requires service xyz" can be resolved. By installing new > module which "provide service xyz" and is unresolvable (for any reasons) and > the original module won't resolve any more either. How do you like this > (mis)behavior? > >>>> - if service provider modules or their dependencies are service consumer >>>> >>>> modules then the correspond service provide modules are resolved after >>>> the >>>> explicit dependences and so on. >>> >>> In this step you have to select one of the modules that "provides a >>> service" which has been requested by some module in the first step. This >>> is the part when the algorithm turns into 3-SAT solver. >> >> I would very like to do a mind meld with you on this! >> >> It's interesting in that, with the current Resolver implementation, when you >> push service provider module dependency resolution to the end then there is >> no longer any backtracking above those dependencies, since they are >> optional. > > Are they really optional? They are synthesized as such. > What is optional on "requires service xyz"? But... > > Heur?ka! I think I know how to overcome the NP-complete problem with services. > We need to make sure we make the dependencies optional. E.g. there are no NP- > complete problems with "requires service optional xyz". Thus, if we allow only > optional services, we simplify everything a lot. > > Of course, some "requires service" are not that optional. So the question for > me is: how to make the dependency system usable and still pretend all services > are in fact optional? > I think we may be on the same page! The Jigsaw resolver does a post processing step on successful resolution: private boolean run() throws ConfigurationException, IOException { Choice ch = null; for (ModuleIdQuery midq : rootQueries) { ViewDependence dep = new ViewDependence(EnumSet.noneOf(Modifier.class), midq); ch = new Choice(null, dep, ch); } boolean resolved = resolve(0, ch); if (resolved) ensureServicesPresent(); // <--- check mandatory requires service constraints return resolved; } That step iterates through for all resolved modules and checks for each mandatory "requires service ..." that there is at least one "provides service ...": private void ensureServicesPresent() throws ConfigurationException { Set serviceTypes = new HashSet<>(); for (ModuleInfo mi: modules) { for (ModuleView view: mi.views()) { Map> services = view.services(); serviceTypes.addAll(services.keySet()); } } for (ModuleInfo mi: modules) { for (ServiceDependence sd: mi.requiresServices()) { if (!sd.modifiers().contains((Modifier.OPTIONAL))) { String sn = sd.service(); if (!serviceTypes.contains(sn)) { fail("No implementations of service %s, required by %s", sn, mi.id()); } } } } } >> Which i think means one can break things down into ordered sub-phases >> resolving the dependencies of each service provider module (which IIUC is >> still NP-Complete as you point out). That may make it easier to report >> service provider module dependency resolution issues to the developer. >>> No surprise, things may get unclear and complicated (which is how I >>> understood your further discussion with Alan). >> >> Yes, i am trying to run the resolver in my head. It's concerns me if >> developers have to do the same. > > ;-) > > Well, people can do this on a small system (e.g. distribution). They have > their ideal vision and see the whole picture. They just need a language (e.g. > requires/provides/service/optional) to describe such vision. > > The problem arises when few such small systems are combined together (e.g. > Maven central). No human is able to see the whole picture then, still we want > computers to operate on such mess effectively. > Agreed. Paul. [exec] | Configuring [mapp@=1.0] using library build/mlib [exec] | resolving ROOT requires mapp@=1.0 [exec] | - trying mapp at 1.0 [exec] | - pushing service provider module dependency stringer.StringTransformer -> requires optional mhasher [exec] | - pushing service provider module dependency stringer.StringTransformer -> requires optional mrotter [exec] | -- resolving mapp at 1.0 requires mstringer [exec] | --- trying mstringer at 1.0 [exec] | ---- resolving mstringer at 1.0 requires synthesized java.base@>=8 [exec] | ----- trying java.base [exec] | ----- pushing service provider module dependency java.nio.charset.spi.CharsetProvider -> requires optional sun.charsets [exec] | ----- pushing service provider module dependency java.nio.file.spi.FileSystemProvider -> requires optional jdk.zipfs [exec] | ----- pushing service provider module dependency sun.net.spi.nameservice.NameServiceDescriptor -> requires optional jdk.jndi [exec] | ------ resolving jdk.base at 8-ea requires local optional jdk.desktop.internal@=8-ea [exec] | ------- trying jdk.desktop.internal at 8-ea [exec] | ------- pushing service provider module dependency javax.print.PrintServiceLookup -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.print.StreamPrintServiceFactory -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.sound.midi.spi.MidiDeviceProvider -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.sound.midi.spi.MidiFileReader -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.sound.midi.spi.MidiFileWriter -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.sound.midi.spi.SoundbankReader -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.sound.sampled.spi.AudioFileReader -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.sound.sampled.spi.AudioFileWriter -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.sound.sampled.spi.FormatConversionProvider -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.sound.sampled.spi.MixerProvider -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency sun.java2d.cmm.PCMM -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency sun.java2d.pipe.RenderingEngine -> requires optional jdk.desktop [exec] | -------- resolving jdk.desktop at 8-ea requires local jdk.base.internal@=8-ea [exec] | --------- resolving jdk.desktop at 8-ea requires jdk.jaxp@=8-ea [exec] | ---------- trying jdk.jaxp at 8-ea [exec] | ----------- resolving jdk.jaxp at 8-ea requires jdk.base@=8-ea [exec] | ------------ resolving jdk.jaxp at 8-ea requires synthesized java.base@>=8 [exec] | ------------- resolving jdk.desktop at 8-ea requires jdk.logging@=8-ea [exec] | -------------- trying jdk.logging at 8-ea [exec] | --------------- resolving jdk.logging at 8-ea requires jdk.base.internal@=8-ea [exec] | ---------------- resolving jdk.logging at 8-ea requires synthesized java.base@>=8 [exec] | ----------------- resolving jdk.desktop at 8-ea requires jdk.prefs@=8-ea [exec] | ------------------ trying jdk.prefs at 8-ea [exec] | ------------------- resolving jdk.prefs at 8-ea requires local jdk.base.internal@=8-ea [exec] | -------------------- resolving jdk.prefs at 8-ea requires jdk.jaxp@=8-ea [exec] | --------------------- resolving jdk.prefs at 8-ea requires synthesized java.base@>=8 [exec] | ---------------------- resolving jdk.desktop at 8-ea requires sun.charsets.internal@=8-ea [exec] | ----------------------- trying sun.charsets.internal at 8-ea [exec] | ------------------------ resolving sun.charsets at 8-ea requires local jdk.base.internal@=8-ea [exec] | ------------------------- resolving sun.charsets at 8-ea requires synthesized java.base@>=8 [exec] | -------------------------- resolving jdk.desktop at 8-ea requires synthesized java.base@>=8 [exec] | --------------------------- resolving jdk.base at 8-ea requires optional jdk.jaxp.internal@=8-ea [exec] | ---------------------------- resolving jdk.base at 8-ea requires local optional jdk.tls.internal@=8-ea [exec] | ----------------------------- trying jdk.tls.internal at 8-ea [exec] | ------------------------------ resolving jdk.tls at 8-ea requires local jdk.base.internal@=8-ea [exec] | ------------------------------- resolving jdk.tls at 8-ea requires synthesized java.base@>=8 [exec] | -------------------------------- resolving jdk.base at 8-ea requires local optional sun.localedata@=8-ea [exec] | --------------------------------- trying sun.localedata at 8-ea [exec] | ---------------------------------- resolving sun.localedata at 8-ea requires local jdk.base.internal@=8-ea [exec] | ----------------------------------- resolving sun.localedata at 8-ea requires synthesized java.base@>=8 [exec] | ------------------------------------ resolving jdk.base at 8-ea requires local optional sun.resources@=8-ea [exec] | ------------------------------------- trying sun.resources at 8-ea [exec] | -------------------------------------- resolving sun.resources at 8-ea requires local jdk.base.internal@=8-ea [exec] | --------------------------------------- resolving sun.resources at 8-ea requires synthesized java.base@>=8 [exec] | ---------------------------------------- resolving mapp at 1.0 requires synthesized java.base@>=8 [exec] | ----------------------------------------- resolving SERVICE PROVIDER requires optional mhasher [exec] | ------------------------------------------ trying mhasher at 1.0 [exec] | ------------------------------------------- resolving mhasher at 1.0 requires mstringer [exec] | -------------------------------------------- resolving mhasher at 1.0 requires synthesized java.base@>=8 [exec] | --------------------------------------------- resolving SERVICE PROVIDER requires optional mrotter [exec] | ---------------------------------------------- trying mrotter at 1.0 [exec] | ----------------------------------------------- resolving mrotter at 1.0 requires mstringer [exec] | ------------------------------------------------ resolving mrotter at 1.0 requires synthesized java.base@>=8 [exec] | ------------------------------------------------- resolving SERVICE PROVIDER requires optional sun.charsets [exec] | -------------------------------------------------- resolving SERVICE PROVIDER requires optional jdk.zipfs [exec] | --------------------------------------------------- trying jdk.zipfs at 8-ea [exec] | ---------------------------------------------------- resolving jdk.zipfs at 8-ea requires jdk.base@=8-ea [exec] | ----------------------------------------------------- resolving jdk.zipfs at 8-ea requires synthesized java.base@>=8 [exec] | ------------------------------------------------------ resolving SERVICE PROVIDER requires optional jdk.jndi [exec] | ------------------------------------------------------- trying jdk.jndi at 8-ea [exec] | -------------------------------------------------------- resolving jdk.jndi at 8-ea requires local jdk.auth.internal@=8-ea [exec] | --------------------------------------------------------- trying jdk.auth.internal at 8-ea [exec] | ---------------------------------------------------------- resolving jdk.auth at 8-ea requires local jdk.base.internal@=8-ea [exec] | ----------------------------------------------------------- resolving jdk.auth at 8-ea requires jdk.logging@=8-ea [exec] | ------------------------------------------------------------ resolving jdk.auth at 8-ea requires synthesized java.base@>=8 [exec] | ------------------------------------------------------------- resolving jdk.jndi at 8-ea requires local jdk.base.internal@=8-ea [exec] | -------------------------------------------------------------- resolving jdk.jndi at 8-ea requires optional jdk.desktop@=8-ea [exec] | --------------------------------------------------------------- resolving jdk.jndi at 8-ea requires jdk.rmi@=8-ea [exec] | ---------------------------------------------------------------- trying jdk.rmi at 8-ea [exec] | ----------------------------------------------------------------- resolving jdk.rmi at 8-ea requires jdk.base.internal@=8-ea [exec] | ------------------------------------------------------------------ resolving jdk.rmi at 8-ea requires jdk.logging@=8-ea [exec] | ------------------------------------------------------------------- resolving jdk.rmi at 8-ea requires jdk.tls@=8-ea [exec] | -------------------------------------------------------------------- resolving jdk.rmi at 8-ea requires synthesized java.base@>=8 [exec] | --------------------------------------------------------------------- resolving jdk.jndi at 8-ea requires jdk.tls.internal@=8-ea [exec] | ---------------------------------------------------------------------- resolving jdk.jndi at 8-ea requires synthesized java.base@>=8 [exec] | ----------------------------------------------------------------------- resolving SERVICE PROVIDER requires optional jdk.desktop From sean.mullan at oracle.com Thu Jun 21 04:59:58 2012 From: sean.mullan at oracle.com (Sean Mullan) Date: Thu, 21 Jun 2012 07:59:58 -0400 Subject: Module file parse API In-Reply-To: <11A151E3-3E3C-471F-BF2E-890FDB4928A5@oracle.com> References: <4FD86E8C.3080208@oracle.com> <4FE1FA31.9090306@oracle.com> <4FE1FF33.3090505@oracle.com> <4FE21154.2020604@oracle.com> <4FE2F17B.3060906@oracle.com> <11A151E3-3E3C-471F-BF2E-890FDB4928A5@oracle.com> Message-ID: <4FE30CBE.6070502@oracle.com> On 6/21/12 6:36 AM, Paul Sandoz wrote: >>> Having said that, I've been somewhat dubious on the overall >>> value/purpose of the hashes inside the module file. They don't >>> provide any security without something additional such as a >>> signature, but when generating the signature, it has to >>> recalculate all of the hashes again to be sure they are still >>> correct since the module file was created. So their only value is >>> as a checksum, but my experience with checksums is that they are >>> usually stored separately from what they are computed over. >> >> I view the hashes in an unsigned module file as a checksum also, >> but I think they are very useful. For example, if a tool is only >> interested in the classes of a module file, it can skip to the >> classes section, extract/process it, validate the hash, and exit >> without having to finish reading the remainder of the module file. >> This is nice, especially if the module file is being read from a >> remote stream, and only possible with per section hashes. >> > > Also one does not have to upload/download separate data to/from a > repository for the content and the hash of the content. This also has > the advantage that the repository can verify the hashes before > accepting deployment of a module (instead of say uploading using > multipart MIME). Ok, but I want to be sure I understand what you think the value of the hashes provides. Hashes without a signature don't provide any security since an attacker can change both the data and the hash without detection. How are they providing value as a checksum? Do you think it will be common for someone to change the internal sections of the module file without updating the corresponding section hash? --Sean From paul.sandoz at oracle.com Thu Jun 21 05:10:13 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 21 Jun 2012 14:10:13 +0200 Subject: Module file parse API In-Reply-To: <4FE30CBE.6070502@oracle.com> References: <4FD86E8C.3080208@oracle.com> <4FE1FA31.9090306@oracle.com> <4FE1FF33.3090505@oracle.com> <4FE21154.2020604@oracle.com> <4FE2F17B.3060906@oracle.com> <11A151E3-3E3C-471F-BF2E-890FDB4928A5@oracle.com> <4FE30CBE.6070502@oracle.com> Message-ID: <5EE4B25C-7890-4487-A812-B22A679BAA58@oracle.com> On Jun 21, 2012, at 1:59 PM, Sean Mullan wrote: > On 6/21/12 6:36 AM, Paul Sandoz wrote: > >>>> Having said that, I've been somewhat dubious on the overall >>>> value/purpose of the hashes inside the module file. They don't >>>> provide any security without something additional such as a >>>> signature, but when generating the signature, it has to >>>> recalculate all of the hashes again to be sure they are still >>>> correct since the module file was created. So their only value is >>>> as a checksum, but my experience with checksums is that they are >>>> usually stored separately from what they are computed over. >>> >>> I view the hashes in an unsigned module file as a checksum also, >>> but I think they are very useful. For example, if a tool is only >>> interested in the classes of a module file, it can skip to the >>> classes section, extract/process it, validate the hash, and exit >>> without having to finish reading the remainder of the module file. >>> This is nice, especially if the module file is being read from a >>> remote stream, and only possible with per section hashes. >>> >> >> Also one does not have to upload/download separate data to/from a >> repository for the content and the hash of the content. This also has >> the advantage that the repository can verify the hashes before >> accepting deployment of a module (instead of say uploading using >> multipart MIME). > > Ok, but I want to be sure I understand what you think the value of the hashes provides. Hashes without a signature don't provide any security since an attacker can change both the data and the hash without detection. Yes. > How are they providing value as a checksum? Do you think it will be common for someone to change the internal sections of the module file without updating the corresponding section hash? > No. Not someone but something by non-nefarious means :-) Bit rot on network transmission or on disk. Paul. From paul.sandoz at oracle.com Thu Jun 21 05:14:29 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 21 Jun 2012 14:14:29 +0200 Subject: Optional service dependencies only was: Direct dependency on service provider modules? In-Reply-To: <1394015.gmxIYdubyC@logoutik> References: <2FD90826-88FD-4FA8-B59A-4DF558EB8D6C@oracle.com> <1580889.kgaz451cTH@logoutik> <1394015.gmxIYdubyC@logoutik> Message-ID: <78FA2DE4-33EC-405F-A80A-72014A15DDA9@oracle.com> Hi, On Jun 21, 2012, at 11:35 AM, Jaroslav Tulach wrote: > Hello Paul, > here is my initial proposal for improving the dependencies mechanism to > eliminate the NP-Completeness: > > [1] http://wiki.apidesign.org/wiki/JigsawServices > > Dne ?t 21. ?ervna 2012 10:05:27, Jaroslav Tulach napsal(a): >>> I would very like to do a mind meld with you on this! >>> >>> It's interesting in that, with the current Resolver implementation, when >>> you push service provider module dependency resolution to the end then >>> there is no longer any backtracking above those dependencies, since they >>> are optional. >> >> Are they really optional? What is optional on "requires service xyz"? But... > > As the first part of [1] argues, with "requires service" you always need > backtracking. > Nice write up. We are definitely on the same page w.r.t. optionality. What i was trying to do with my patch is change when those optional dependencies are processed i.e. at the end, thus first the application module dependencies are resolved as if no service provider modules are present. My hunch is that because service consumers are decoupled from service providers it can be harder to predict or understand the result when the resolver mixes in the resolving of service provider dependencies. >> Heur?ka! I think I know how to overcome the NP-complete problem with >> services. We need to make sure we make the dependencies optional. E.g. >> there are no NP- complete problems with "requires service optional xyz". >> Thus, if we allow only optional services, we simplify everything a lot. >> >> Of course, some "requires service" are not that optional. So the question >> for me is: how to make the dependency system usable and still pretend all >> services are in fact optional? > > [1] talks about a "hint". I'd rather leave the exact language construct for > providing a hint up to language experts, but I can imagine it makes a logical > sense for the language. When I am JAXP module at 1.33 I probably want to > recommend Xerces from JDK 1.33 as well. I should have a way to provide such > hint to the Jigsaw resolution system. > > However there should be a way to reconfigure these hints at the deployment > time. You raise a very interesting point! I think this suggests control of versions of transitive dependencies, much like what can be done with maven. Although the JAXP module may hint it strongly wants a version of Xerces another module may strongly hint it wants a different version. Perhaps the conflict could be mediated by changing module processing order, or perhaps by declaring what version should be used. > I've noticed a discussion about security providers. You were in need of > changing their order and that could be also considered a "hint". Just probably > a hint provided externally, at the time the application is started (e.g. not > compiled). Yes, although in this case the order is associated with a service interface not the module. Modules can be service provider modules for multiple service interfaces. So hints on the order of service provider classes may need to be something separate to hints on the order modules are resolved. Paul. From paul.sandoz at oracle.com Thu Jun 21 05:49:05 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 21 Jun 2012 14:49:05 +0200 Subject: ServiceLoader.load* take 2 In-Reply-To: <63E8F31D-62D8-4C7D-8DAB-26C4DFB79053@oracle.com> References: <4FE2724F.3010305@oracle.com> <63E8F31D-62D8-4C7D-8DAB-26C4DFB79053@oracle.com> Message-ID: Hi Mandy, Updated webrev: http://cr.openjdk.java.net/~psandoz/jigsaw/global-services/webrev.1/ Paul. On Jun 21, 2012, at 9:54 AM, Paul Sandoz wrote: > Hi Mandy, > > Thanks for reviewing. > > On Jun 21, 2012, at 3:01 AM, Mandy Chung wrote: > >> Paul, >> >> On 6/12/2012 8:03 AM, Paul Sandoz wrote: >>> the first part of which is implemented by the following webrev, which changes service instance creation so that it is scoped to the configuration rather than the class loader: >>> >>> http://cr.openjdk.java.net/~psandoz/jigsaw/global-services/webrev/ >> >> I believe you want to discuss and push the service loader changes separately. I reviewed this first part which looks good overall. >> >> I also wonder if the new ModuleServiceLoader could just be an inner class in ServiceLoader and I understand your thought behind and change Loader.findServices to package-private. I'm fine if you prefer to keep it as is. >> > > OK. > > >> LoaderPool.java L119 - Is 'servceInterfaceName' a typo (missing letter 'i' in "service") >> > > Grrr... from day one the 'i' key on my MacBook Pro keyboard has been dodgy. > > >> ModuleServiceLoader.java L124: does javadoc parse '' correctly? >> > > Yes, JavaDoc will display it in the "Type Parameters" section of the documented method. FWIW NetBeans auto-generated it for me :-) > > -- > > I will send out another webrev today with an update for the two changes. > > >> classloader.sh test - looks like you can just extend hello.sh, many.sh, and possibly optional.sh to test the different ServiceLoader.load* methods that could also be launched in a new VM if appropriate. >> > > Yes, there is lots of duplication here that can be fixed, plus we also need to expand the tests to include such things as transitive dependencies and second level service consumer modules. Do you mind if we tackle that as a separate set of commits? > > I am not too familiar with the test infrastructure and the reuse of such shall scripts (it seems like open season and we can do what we want as long as there is appropriate meta-data that jtreg understands). Any more advice on how to improve this would be greatly appreciated? > > Paul. From sean.mullan at oracle.com Thu Jun 21 06:00:55 2012 From: sean.mullan at oracle.com (Sean Mullan) Date: Thu, 21 Jun 2012 09:00:55 -0400 Subject: Module file parse API In-Reply-To: <5EE4B25C-7890-4487-A812-B22A679BAA58@oracle.com> References: <4FD86E8C.3080208@oracle.com> <4FE1FA31.9090306@oracle.com> <4FE1FF33.3090505@oracle.com> <4FE21154.2020604@oracle.com> <4FE2F17B.3060906@oracle.com> <11A151E3-3E3C-471F-BF2E-890FDB4928A5@oracle.com> <4FE30CBE.6070502@oracle.com> <5EE4B25C-7890-4487-A812-B22A679BAA58@oracle.com> Message-ID: <4FE31B07.5040108@oracle.com> On 6/21/12 8:10 AM, Paul Sandoz wrote: > > On Jun 21, 2012, at 1:59 PM, Sean Mullan wrote: > >> On 6/21/12 6:36 AM, Paul Sandoz wrote: >> >>>>> Having said that, I've been somewhat dubious on the overall >>>>> value/purpose of the hashes inside the module file. They don't >>>>> provide any security without something additional such as a >>>>> signature, but when generating the signature, it has to >>>>> recalculate all of the hashes again to be sure they are still >>>>> correct since the module file was created. So their only value is >>>>> as a checksum, but my experience with checksums is that they are >>>>> usually stored separately from what they are computed over. >>>> >>>> I view the hashes in an unsigned module file as a checksum also, >>>> but I think they are very useful. For example, if a tool is only >>>> interested in the classes of a module file, it can skip to the >>>> classes section, extract/process it, validate the hash, and exit >>>> without having to finish reading the remainder of the module file. >>>> This is nice, especially if the module file is being read from a >>>> remote stream, and only possible with per section hashes. >>>> >>> >>> Also one does not have to upload/download separate data to/from a >>> repository for the content and the hash of the content. This also has >>> the advantage that the repository can verify the hashes before >>> accepting deployment of a module (instead of say uploading using >>> multipart MIME). >> >> Ok, but I want to be sure I understand what you think the value of the hashes provides. Hashes without a signature don't provide any security since an attacker can change both the data and the hash without detection. > > Yes. > > >> How are they providing value as a checksum? Do you think it will be common for someone to change the internal sections of the module file without updating the corresponding section hash? >> > > > No. Not someone but something by non-nefarious means :-) Bit rot on network transmission or on disk. I don't know it still seems like unnecessary overhead for an extremely small likelihood. And wouldn't the entire file including the hashes be suspect then? The hashes aren't even useful for signatures. This is because the signer *must* generate the hashes itself, and then generate a signature over them inside a PKCS#7 SignedData blob. It doesn't even use the existing hashes, so they are just extra duplication. --Sean From paul.sandoz at oracle.com Thu Jun 21 06:44:48 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 21 Jun 2012 15:44:48 +0200 Subject: Module file parse API In-Reply-To: <4FE31B07.5040108@oracle.com> References: <4FD86E8C.3080208@oracle.com> <4FE1FA31.9090306@oracle.com> <4FE1FF33.3090505@oracle.com> <4FE21154.2020604@oracle.com> <4FE2F17B.3060906@oracle.com> <11A151E3-3E3C-471F-BF2E-890FDB4928A5@oracle.com> <4FE30CBE.6070502@oracle.com> <5EE4B25C-7890-4487-A812-B22A679BAA58@oracle.com> <4FE31B07.5040108@oracle.com> Message-ID: On Jun 21, 2012, at 3:00 PM, Sean Mullan wrote: >> >> No. Not someone but something by non-nefarious means :-) Bit rot on network transmission or on disk. > > I don't know it still seems like unnecessary overhead for an extremely small likelihood. http://en.wikipedia.org/wiki/ZFS#Data_Integrity Given the potentially large number of copies of stuff obtained from a central repository, that is probably replicated around the world and has to keep stuff for a reasonable amount of time (10 years or more) and is backed up many times, perhaps that likelihood increases? I would prefer to use ZFS to host a file-based repository :-) > And wouldn't the entire file including the hashes be suspect then? > As Chris said a consumer may only want to look at some section, e.g. the module declaration, or the classes. > The hashes aren't even useful for signatures. This is because the signer *must* generate the hashes itself, and then generate a signature over them inside a PKCS#7 SignedData blob. It doesn't even use the existing hashes, so they are just extra duplication. > Agreed. They are not relevant for signing. Paul. From chris.hegarty at oracle.com Thu Jun 21 06:47:37 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 21 Jun 2012 14:47:37 +0100 Subject: Module file parse API In-Reply-To: <4FE31B07.5040108@oracle.com> References: <4FD86E8C.3080208@oracle.com> <4FE1FA31.9090306@oracle.com> <4FE1FF33.3090505@oracle.com> <4FE21154.2020604@oracle.com> <4FE2F17B.3060906@oracle.com> <11A151E3-3E3C-471F-BF2E-890FDB4928A5@oracle.com> <4FE30CBE.6070502@oracle.com> <5EE4B25C-7890-4487-A812-B22A679BAA58@oracle.com> <4FE31B07.5040108@oracle.com> Message-ID: <4FE325F9.7060008@oracle.com> On 21/06/2012 14:00, Sean Mullan wrote: > ... >> No. Not someone but something by non-nefarious means :-) Bit rot on >> network transmission or on disk. > > I don't know it still seems like unnecessary overhead for an extremely > small likelihood. And wouldn't the entire file including the hashes be > suspect then? > > The hashes aren't even useful for signatures. This is because the signer > *must* generate the hashes itself, and then generate a signature over > them inside a PKCS#7 SignedData blob. It doesn't even use the existing > hashes, so they are just extra duplication. I guess I'd ask the question another way. Without the hashes would we need to circulate a separate checksum file to verify data integrity? If so, we wouldn't be able to verify the integrated on a per section basis ( unless the checksum file contained multiple hashes ). Verifying data integrity on a per section basis appears to be very useful. -Chris. > > --Sean From Alan.Bateman at oracle.com Thu Jun 21 06:57:54 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 21 Jun 2012 14:57:54 +0100 Subject: ServiceLoader.load* take 2 In-Reply-To: References: <4FDB17DE.3070205@oracle.com> Message-ID: <4FE32862.4090007@oracle.com> On 15/06/2012 16:23, Paul Sandoz wrote: > : > > >> One thing I didn't quite get was the reason for moving LazyIterator1 out of ServiceLoader to a new org.openjdk.jigsaw.ModuleServiceLoader. My comment is really just wondering about the iteration/instantiation in two places rather than one. >> > I wanted to separate concerns so that ServiceLoader did not know any specifics of modular services (and vice versa FWIW). It should hopefully keep any future changes located in the org.openjdk.jigsaw package. I forgot to do so but it means, for example, we can make Loader.findServices protected or package private. I see Mandy also commented on this too. Personally I thought it was marginally better to have the lazy loading iterators in one place rather than two but it does require that org.openjdk.jigsaw export a means to get the map without SL getting too tied into the implementation details. Part of the comment is probably the name too as MSL does not extend SL. > : >> On the new test then it reminds me that we may be able to consolidate the tests in the services directory into a more complete set of unit tests (just thinking that there is a bit of duplication between the new test the existing hello.sh, many.sh and optional.sh). >> > What do you propose? > > They definitely need to be fleshed out. Personally i prefer things to be in exploded form, i find it easier to grok. > When I added these tests then the idea was that they be standalone and easy to understand. I guess I'm just concerned that if they are copied/edited that they will come back to bite us as any change will require a change to every one of them. So just something to thing about, not important for this patch. In any case, I'm happy to push these changes for you. One other thing for ServiceLoader, a future round, is that the spec/javadoc will need to be updated to describe how it works with modules. -Alan. From sean.mullan at oracle.com Thu Jun 21 07:11:16 2012 From: sean.mullan at oracle.com (Sean Mullan) Date: Thu, 21 Jun 2012 10:11:16 -0400 Subject: Module file parse API In-Reply-To: <4FE325F9.7060008@oracle.com> References: <4FD86E8C.3080208@oracle.com> <4FE1FA31.9090306@oracle.com> <4FE1FF33.3090505@oracle.com> <4FE21154.2020604@oracle.com> <4FE2F17B.3060906@oracle.com> <11A151E3-3E3C-471F-BF2E-890FDB4928A5@oracle.com> <4FE30CBE.6070502@oracle.com> <5EE4B25C-7890-4487-A812-B22A679BAA58@oracle.com> <4FE31B07.5040108@oracle.com> <4FE325F9.7060008@oracle.com> Message-ID: <4FE32B84.3060502@oracle.com> On 06/21/2012 09:47 AM, Chris Hegarty wrote: > On 21/06/2012 14:00, Sean Mullan wrote: >> ... >>> No. Not someone but something by non-nefarious means :-) Bit rot on >>> network transmission or on disk. >> >> I don't know it still seems like unnecessary overhead for an extremely >> small likelihood. And wouldn't the entire file including the hashes be >> suspect then? >> >> The hashes aren't even useful for signatures. This is because the signer >> *must* generate the hashes itself, and then generate a signature over >> them inside a PKCS#7 SignedData blob. It doesn't even use the existing >> hashes, so they are just extra duplication. > > I guess I'd ask the question another way. Without the hashes would we > need to circulate a separate checksum file to verify data integrity? If > so, we wouldn't be able to verify the integrated on a per section basis > ( unless the checksum file contained multiple hashes ). Verifying data > integrity on a per section basis appears to be very useful. Right, I see your point. In fact, I'm glad we revisited this and had this discussion. I guess the only downside is that there is no "golden" checksum file that I can double-check against when things go wrong. So it may be a little less clear what may have caused the problem. I've been a little bothered by the fact that the hashes aren't reusable when generating the signatures, and maybe there is a better design, but if nobody else is concerned for now, I'm ok with it. --Sean From jaroslav.tulach at oracle.com Thu Jun 21 07:12:55 2012 From: jaroslav.tulach at oracle.com (Jaroslav Tulach) Date: Thu, 21 Jun 2012 16:12:55 +0200 Subject: Optional service dependencies only In-Reply-To: <78FA2DE4-33EC-405F-A80A-72014A15DDA9@oracle.com> References: <2FD90826-88FD-4FA8-B59A-4DF558EB8D6C@oracle.com> <1394015.gmxIYdubyC@logoutik> <78FA2DE4-33EC-405F-A80A-72014A15DDA9@oracle.com> Message-ID: <28519014.AsdedVrNik@logoutik> Dne ?t 21. ?ervna 2012 14:14:29, Paul Sandoz napsal(a): > On Jun 21, 2012, at 11:35 AM, Jaroslav Tulach wrote: > > [1] http://wiki.apidesign.org/wiki/JigsawServices > Nice write up. We are definitely on the same page w.r.t. optionality. Right. Although, I would not call it optional. Each individual service implementation is optional, but the resolution fails if none can be enabled. > What i was trying to do with my patch is change when those optional > dependencies are processed i.e. at the end, thus first the application > module dependencies are resolved as if no service provider modules are > present. This is what should happen, with following clarification: I am still operating in context of "complete repository" - e.g. I assume that the whole set of transitive dependencies of a module has been computed and recorded at compile time[1]. Only under such assumption range dependencies are acceptable, as far as I understand the problem. [1] http://wiki.apidesign.org/wiki/RangeDependenciesAnalysed#No_Longer_NP- Complete > My hunch is that because service consumers are decoupled from service > providers it can be harder to predict or understand the result when the > resolver mixes in the resolving of service provider dependencies. Right. "Classpath" dependencies first, services later is going to be more understandable to humans. > > However there should be a way to reconfigure these hints at the deployment > > time. > > You raise a very interesting point! > > I think this suggests control of versions of transitive dependencies, much > like what can be done with maven. Although the JAXP module may hint it > strongly wants a version of Xerces another module may strongly hint it > wants a different version. Perhaps the conflict could be mediated by > changing module processing order, or perhaps by declaring what version > should be used. The general principle should be: the one that knows more should make the decision. That is why: #1 - a JAXP API knows it needs an implementation and knows Xerces module provides one that is good enough #2 - the application vendor (providing the application module) knows all the modules in its application and all the services and can decide certain services should have different default (this is what Maven tried to do with its "shortest" path dependency). Also when compiling the whole application, it should be easy to decide whether different services are compatible with the rest of modules or not as one knows the transitive closure of the whole application. #3 - a deployer should have the option to override the application defaults without coding in Java, imho. E.g. something like: $ java \ -module com.myapp.xsltproc \ -service javax.xml.parser.DocumentBuilderFactory \ -default org.apache.crimson.CrimsonBuilderFactory ...of course one needs to install com.myapp.xsltproc module into the library (is it the right term, I am still learning the terminology) as well as org.apache.crimson module, so the resolver can enable both. #3a - the same using some configuration file which format is beyond the scope of my current thinking > > I've noticed a discussion about security providers. You were in need of > > changing their order and that could be also considered a "hint". Just > > probably a hint provided externally, at the time the application is > > started (e.g. not compiled). > > Yes, although in this case the order is associated with a service interface > not the module. Good point. The order needs to be associated with service interface types. Btw. NetBeans have an order of modules defined by topological sort on top of graph with "requires" edges (not "requires service"). This graph is used for startup and shutdown order. But services are orthogonal to this concept, they have their own ordering (per interface type, in general). -jt > Modules can be service provider modules for multiple > service interfaces. So hints on the order of service provider classes may > need to be something separate to hints on the order modules are resolved. PS: What exactly order of modules is good for in Jigsaw? PPS: In NetBeans it is used for order of activators and also for order of URLs returned from systemWideClassLoader.getResources(...). It is guaranteed resources are returned in topological order with respect to order of modules. From Alan.Bateman at oracle.com Thu Jun 21 07:44:29 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 21 Jun 2012 15:44:29 +0100 Subject: Module file parse API In-Reply-To: <4FE32B84.3060502@oracle.com> References: <4FD86E8C.3080208@oracle.com> <4FE1FA31.9090306@oracle.com> <4FE1FF33.3090505@oracle.com> <4FE21154.2020604@oracle.com> <4FE2F17B.3060906@oracle.com> <11A151E3-3E3C-471F-BF2E-890FDB4928A5@oracle.com> <4FE30CBE.6070502@oracle.com> <5EE4B25C-7890-4487-A812-B22A679BAA58@oracle.com> <4FE31B07.5040108@oracle.com> <4FE325F9.7060008@oracle.com> <4FE32B84.3060502@oracle.com> Message-ID: <4FE3334D.1050608@oracle.com> On 21/06/2012 15:11, Sean Mullan wrote: > : > > I've been a little bothered by the fact that the hashes aren't > reusable when generating the signatures, and maybe there is a better > design, but if nobody else is concerned for now, I'm ok with it. I agree with your previous comments that the current hash isn't providing any real security, but it does help with basic integrity, say where it gets corrupted during transfer ("corrupted" might something like legacy ftp transfer in ascii rather than binary mode for example). As I understand it, the rational for per-section hashes was to provid this basic integrity check when streaming. In any case, it might worth looking briefly at other formats as this is now a new problem. Debian packages usually include a MD5 control file for example. -Alan From mandy.chung at oracle.com Thu Jun 21 07:55:50 2012 From: mandy.chung at oracle.com (Mandy Chung) Date: Thu, 21 Jun 2012 07:55:50 -0700 Subject: ServiceLoader.load* take 2 In-Reply-To: References: <4FE2724F.3010305@oracle.com> <63E8F31D-62D8-4C7D-8DAB-26C4DFB79053@oracle.com> Message-ID: <4FE335F6.6090307@oracle.com> On 6/21/2012 5:49 AM, Paul Sandoz wrote: > > http://cr.openjdk.java.net/~psandoz/jigsaw/global-services/webrev.1/ > > Thanks for the update that looks okay. It'd be good to consolidate the tests in the next round as Alan suggests. Mandy From paul.sandoz at oracle.com Thu Jun 21 08:16:12 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 21 Jun 2012 17:16:12 +0200 Subject: ServiceLoader.load* take 2 In-Reply-To: <4FE335F6.6090307@oracle.com> References: <4FE2724F.3010305@oracle.com> <63E8F31D-62D8-4C7D-8DAB-26C4DFB79053@oracle.com> <4FE335F6.6090307@oracle.com> Message-ID: <712298CA-11BA-46DC-B89A-F2C4A5E05049@oracle.com> On Jun 21, 2012, at 4:55 PM, Mandy Chung wrote: > On 6/21/2012 5:49 AM, Paul Sandoz wrote: >> >> http://cr.openjdk.java.net/~psandoz/jigsaw/global-services/webrev.1/ >> > > Thanks for the update that looks okay. Thanks. > It'd be good to consolidate the tests in the next round as Alan suggests. > OK. Certain tests may/should be sensitive to resolution, thus we just need to be careful that those are isolated either by an entry point or in a separate library. I think i will experiment exploding out the source files. It makes it so hard to edit correctly and maintain, i wanna load 'em in NetBeans! Paul. From jonathan.gibbons at oracle.com Thu Jun 21 08:17:00 2012 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Thu, 21 Jun 2012 08:17:00 -0700 Subject: Build-infra with jigsaw, progress update In-Reply-To: <4FE2D7ED.7080104@oracle.com> References: <4FE1A015.9090702@oracle.com> <4FE1A2C5.8050804@oracle.com> <4FE1AA9D.2070504@oracle.com> <4FE1EE60.6020208@oracle.com> <4FE2D7ED.7080104@oracle.com> Message-ID: <4FE33AEC.6080902@oracle.com> On 06/21/2012 01:14 AM, Erik Joelsson wrote: > > > On 2012-06-20 17:38, Michael McMahon wrote: >> Okay, I'll look at updating the script to do that. >> >> - Michael > > That would be great! In my setup I've put them in a separate directory > (jdk/newsrc/share/moduleinfo). The reason for this is that javac won't > let me compile the classes in jdk.base in legacy mode if it sees a > module-info.java anywhere. I'm not sure if this is intended or not, > but that's how it currently works. > > /Erik Erik, I'm not sure what you mean by "legacy mode" in this context. javac has different rules for 0 or 1 module, and many modules. If you are trying to compile many modules together, you have to set up the options appropriately. -- Jon From chris.hegarty at oracle.com Thu Jun 21 08:25:23 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 21 Jun 2012 16:25:23 +0100 Subject: Module file parse API In-Reply-To: <4FE3334D.1050608@oracle.com> References: <4FD86E8C.3080208@oracle.com> <4FE1FA31.9090306@oracle.com> <4FE1FF33.3090505@oracle.com> <4FE21154.2020604@oracle.com> <4FE2F17B.3060906@oracle.com> <11A151E3-3E3C-471F-BF2E-890FDB4928A5@oracle.com> <4FE30CBE.6070502@oracle.com> <5EE4B25C-7890-4487-A812-B22A679BAA58@oracle.com> <4FE31B07.5040108@oracle.com> <4FE325F9.7060008@oracle.com> <4FE32B84.3060502@oracle.com> <4FE3334D.1050608@oracle.com> Message-ID: <4FE33CE3.7030804@oracle.com> On 21/06/2012 15:44, Alan Bateman wrote: > On 21/06/2012 15:11, Sean Mullan wrote: >> : >> >> I've been a little bothered by the fact that the hashes aren't >> reusable when generating the signatures, and maybe there is a better >> design, but if nobody else is concerned for now, I'm ok with it. > I agree with your previous comments that the current hash isn't > providing any real security, but it does help with basic integrity, say > where it gets corrupted during transfer ("corrupted" might something > like legacy ftp transfer in ascii rather than binary mode for example). > As I understand it, the rational for per-section hashes was to provid > this basic integrity check when streaming. In any case, it might worth > looking briefly at other formats as this is now a new problem. Debian > packages usually include a MD5 control file for example. To the best of my knowledge deb files are not streamable. You need the complete archive before you can extract the control information. The archive itself contains the md5 sums to verify the integrity of the extracted contents, rather than the archive itself. I'm not sure if there is a way to verify the integrity of the actual archive, or maybe this is just a function of whether the archive actually extracts or not. -Chris. > > -Alan From paul.sandoz at oracle.com Thu Jun 21 09:03:24 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 21 Jun 2012 18:03:24 +0200 Subject: ClassLoader.getParent to throw exception for module classloader Message-ID: <1E9A3E19-E61C-4852-A9AA-2367624A3578@oracle.com> Hi, http://cr.openjdk.java.net/~psandoz/jigsaw/classloader-getparent/webrev/ In a off-list discussion there was a side comment about making ClassLoader.getParent throwing an exception for a module class loader. This type of thing is easy to loose so marking this down in email. This patch results in the failure of: jdk/test/org/openjdk/jigsaw/optional-jaxp.sh See below for the exception in Xerces. The code below is incorrect for module mode because it assumes a hierarchical delegation model. In com.sun.org.apache.xerces.internal.utils.ObjectFactory.findClassLoader: ClassLoader context = SecuritySupport.getContextClassLoader(); ClassLoader system = SecuritySupport.getSystemClassLoader(); ClassLoader chain = system; while (true) { if (context == chain) { // Assert: we are on JDK 1.1 or we have no Context ClassLoader // or any Context ClassLoader in chain of system classloader // (including extension ClassLoader) so extend to widest // ClassLoader (always look in system ClassLoader if Xerces // is in boot/extension/system classpath and in current // ClassLoader otherwise); normal classloaders delegate // back to system ClassLoader first so this widening doesn't // change the fact that context ClassLoader will be consulted ClassLoader current = ObjectFactory.class.getClassLoader(); chain = system; while (true) { if (current == chain) { // Assert: Current ClassLoader in chain of // boot/extension/system ClassLoaders return system; } if (chain == null) { break; } chain = SecuritySupport.getParentClassLoader(chain); } // Assert: Current ClassLoader not in chain of // boot/extension/system ClassLoaders return current; } When ClassLoader.getParent() returns null in module mode it luckily works for the case of loading a class from within the JAXP module itself. This area of Xerces may need to be cleaned up to use ServiceLoader. In this respect the exception actually signaled the potential location of the issue that would of otherwise been hidden and trickier to find. However, the approach of throwing an exception does seem brutal. If an XML parser library is barfing, what else will? Paul. Exception in thread "main" java.lang.IllegalStateException: A module class loader does not support the hierarchical delegation mode and has no parent. at java.lang.ClassLoader.getParent(ClassLoader.java:1429) at com.sun.org.apache.xerces.internal.utils.SecuritySupport$3.run(SecuritySupport.java:82) at java.security.AccessController.doPrivileged(Native Method) at com.sun.org.apache.xerces.internal.utils.SecuritySupport.getParentClassLoader(SecuritySupport.java:78) at com.sun.org.apache.xerces.internal.utils.ObjectFactory.findClassLoader(ObjectFactory.java:221) at com.sun.org.apache.xerces.internal.utils.ObjectFactory.newInstance(ObjectFactory.java:255) at com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory.getInstance(DTDDVFactory.java:64) at com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory.getInstance(DTDDVFactory.java:49) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.(XML11Configuration.java:565) at com.sun.org.apache.xerces.internal.parsers.XIncludeAwareParserConfiguration.(XIncludeAwareParserConfiguration.java:130) at com.sun.org.apache.xerces.internal.parsers.XIncludeAwareParserConfiguration.(XIncludeAwareParserConfiguration.java:91) at com.sun.org.apache.xerces.internal.parsers.DOMParser.(DOMParser.java:138) at com.sun.org.apache.xerces.internal.parsers.DOMParser.(DOMParser.java:122) at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.(DocumentBuilderImpl.java:120) at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(DocumentBuilderFactoryImpl.java:76) at sun.util.xml.XMLUtils.getLoadingDoc(XMLUtils.java:99) at sun.util.xml.XMLUtils.load(XMLUtils.java:75) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:474) at java.util.Properties$XMLUtils.invoke(Properties.java:1157) at java.util.Properties$XMLUtils.load(Properties.java:1178) at java.util.Properties.loadFromXML(Properties.java:875) at org.astro.World.name(World.java:11) at com.greetings.Hello.main(Hello.java:5) From Alan.Bateman at oracle.com Thu Jun 21 09:10:49 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 21 Jun 2012 17:10:49 +0100 Subject: ClassLoader.getParent to throw exception for module classloader In-Reply-To: <1E9A3E19-E61C-4852-A9AA-2367624A3578@oracle.com> References: <1E9A3E19-E61C-4852-A9AA-2367624A3578@oracle.com> Message-ID: <4FE34789.1020103@oracle.com> On 21/06/2012 17:03, Paul Sandoz wrote: > : > > In a off-list discussion there was a side comment about making ClassLoader.getParent throwing an exception for a module class loader. This type of thing is easy to loose so marking this down in email. > There is commented out implementation of getParent in org.openjdk.jigsaw.Loader that we should probably remove too (commented out because getParent is final). On the exception then wouldn't UOE be more suitable than IllegalStateException? -Alan From david.lloyd at redhat.com Thu Jun 21 09:19:42 2012 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 21 Jun 2012 11:19:42 -0500 Subject: ClassLoader.getParent to throw exception for module classloader In-Reply-To: <1E9A3E19-E61C-4852-A9AA-2367624A3578@oracle.com> References: <1E9A3E19-E61C-4852-A9AA-2367624A3578@oracle.com> Message-ID: <4FE3499E.90305@redhat.com> If "null" is to no longer mean "the JDK", then why not make it into a "nothing" class loader which loads nothing? If it is continuing to mean the system class loader for base "java." stuff, why not just continue to let it be the parent? Every module will need visibility to "java.lang" at the least, right? On 06/21/2012 11:03 AM, Paul Sandoz wrote: > Hi, > > http://cr.openjdk.java.net/~psandoz/jigsaw/classloader-getparent/webrev/ > > In a off-list discussion there was a side comment about making ClassLoader.getParent throwing an exception for a module class loader. This type of thing is easy to loose so marking this down in email. > > This patch results in the failure of: > > jdk/test/org/openjdk/jigsaw/optional-jaxp.sh > > See below for the exception in Xerces. The code below is incorrect for module mode because it assumes a hierarchical delegation model. In com.sun.org.apache.xerces.internal.utils.ObjectFactory.findClassLoader: > > ClassLoader context = SecuritySupport.getContextClassLoader(); > ClassLoader system = SecuritySupport.getSystemClassLoader(); > > ClassLoader chain = system; > while (true) { > if (context == chain) { > // Assert: we are on JDK 1.1 or we have no Context ClassLoader > // or any Context ClassLoader in chain of system classloader > // (including extension ClassLoader) so extend to widest > // ClassLoader (always look in system ClassLoader if Xerces > // is in boot/extension/system classpath and in current > // ClassLoader otherwise); normal classloaders delegate > // back to system ClassLoader first so this widening doesn't > // change the fact that context ClassLoader will be consulted > ClassLoader current = ObjectFactory.class.getClassLoader(); > > chain = system; > while (true) { > if (current == chain) { > // Assert: Current ClassLoader in chain of > // boot/extension/system ClassLoaders > return system; > } > if (chain == null) { > break; > } > chain = SecuritySupport.getParentClassLoader(chain); > } > > // Assert: Current ClassLoader not in chain of > // boot/extension/system ClassLoaders > return current; > } > > When ClassLoader.getParent() returns null in module mode it luckily works for the case of loading a class from within the JAXP module itself. > > This area of Xerces may need to be cleaned up to use ServiceLoader. > > In this respect the exception actually signaled the potential location of the issue that would of otherwise been hidden and trickier to find. However, the approach of throwing an exception does seem brutal. If an XML parser library is barfing, what else will? > > Paul. > > Exception in thread "main" java.lang.IllegalStateException: A module class loader does not support the hierarchical delegation mode and has no parent. > at java.lang.ClassLoader.getParent(ClassLoader.java:1429) > at com.sun.org.apache.xerces.internal.utils.SecuritySupport$3.run(SecuritySupport.java:82) > at java.security.AccessController.doPrivileged(Native Method) > at com.sun.org.apache.xerces.internal.utils.SecuritySupport.getParentClassLoader(SecuritySupport.java:78) > at com.sun.org.apache.xerces.internal.utils.ObjectFactory.findClassLoader(ObjectFactory.java:221) > at com.sun.org.apache.xerces.internal.utils.ObjectFactory.newInstance(ObjectFactory.java:255) > at com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory.getInstance(DTDDVFactory.java:64) > at com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory.getInstance(DTDDVFactory.java:49) > at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.(XML11Configuration.java:565) > at com.sun.org.apache.xerces.internal.parsers.XIncludeAwareParserConfiguration.(XIncludeAwareParserConfiguration.java:130) > at com.sun.org.apache.xerces.internal.parsers.XIncludeAwareParserConfiguration.(XIncludeAwareParserConfiguration.java:91) > at com.sun.org.apache.xerces.internal.parsers.DOMParser.(DOMParser.java:138) > at com.sun.org.apache.xerces.internal.parsers.DOMParser.(DOMParser.java:122) > at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.(DocumentBuilderImpl.java:120) > at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(DocumentBuilderFactoryImpl.java:76) > at sun.util.xml.XMLUtils.getLoadingDoc(XMLUtils.java:99) > at sun.util.xml.XMLUtils.load(XMLUtils.java:75) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:474) > at java.util.Properties$XMLUtils.invoke(Properties.java:1157) > at java.util.Properties$XMLUtils.load(Properties.java:1178) > at java.util.Properties.loadFromXML(Properties.java:875) > at org.astro.World.name(World.java:11) > at com.greetings.Hello.main(Hello.java:5) > -- - DML From mandy.chung at oracle.com Thu Jun 21 12:02:29 2012 From: mandy.chung at oracle.com (Mandy Chung) Date: Thu, 21 Jun 2012 12:02:29 -0700 Subject: Review request for the incorrect check for "getClassLoader" permission Message-ID: <4FE36FC5.9020702@oracle.com> David, Paul, I have a fix for the incorrect check w.r.t. "getClassLoader" permission [1] and also update j.u.c.atomic for module mode. Webrev at: http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/getclassloader-permission-fix/ The security.sh test demonstrates what can be accessed in module mode and lists the open issue. This patch is intended to fix the bug introduced in this changeset: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/7b282c826118 Since there is no parent-child delegation relationship and the existing security check applies, it can only access its own class loader in module mode. This remains an open issue what security checks would be appropriate in module mode and how well it plays with existing java security policy file etc. As for testing, I ran test/java/util/concurrent/atomic tests in hybrid mode and they passed (in fact I verified all java/lang and java/util tests). Since AtomicUpdaters is marked to run in othervm mode, I manually converted it as a module and it passes. Thanks Mandy [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2012-May/002614.html From alan.bateman at oracle.com Thu Jun 21 14:00:59 2012 From: alan.bateman at oracle.com (alan.bateman at oracle.com) Date: Thu, 21 Jun 2012 21:00:59 +0000 Subject: hg: jigsaw/jigsaw/jdk: Service declarations for JAXP Message-ID: <20120621210145.73D4247A73@hg.openjdk.java.net> Changeset: c2c4b5425238 Author: alanb Date: 2012-06-21 21:27 +0100 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/c2c4b5425238 Service declarations for JAXP ! make/modules/jdk.depconfig From david.holmes at oracle.com Thu Jun 21 17:00:11 2012 From: david.holmes at oracle.com (David Holmes) Date: Fri, 22 Jun 2012 10:00:11 +1000 Subject: ClassLoader.getParent to throw exception for module classloader In-Reply-To: <1E9A3E19-E61C-4852-A9AA-2367624A3578@oracle.com> References: <1E9A3E19-E61C-4852-A9AA-2367624A3578@oracle.com> Message-ID: <4FE3B58B.7000005@oracle.com> On 22/06/2012 2:03 AM, Paul Sandoz wrote: > Hi, > > http://cr.openjdk.java.net/~psandoz/jigsaw/classloader-getparent/webrev/ > > In a off-list discussion there was a side comment about making ClassLoader.getParent throwing an exception for a module class loader. This type of thing is easy to loose so marking this down in email. So that would seem to have far-reaching consequences on various bits of code that check the classloader hierarchy to determine when they need to do security manager checks. This leaves me wondering how custom classloaders fit into this world - do they have a parent that is a module loader? David ----- > This patch results in the failure of: > > jdk/test/org/openjdk/jigsaw/optional-jaxp.sh > > See below for the exception in Xerces. The code below is incorrect for module mode because it assumes a hierarchical delegation model. In com.sun.org.apache.xerces.internal.utils.ObjectFactory.findClassLoader: > > ClassLoader context = SecuritySupport.getContextClassLoader(); > ClassLoader system = SecuritySupport.getSystemClassLoader(); > > ClassLoader chain = system; > while (true) { > if (context == chain) { > // Assert: we are on JDK 1.1 or we have no Context ClassLoader > // or any Context ClassLoader in chain of system classloader > // (including extension ClassLoader) so extend to widest > // ClassLoader (always look in system ClassLoader if Xerces > // is in boot/extension/system classpath and in current > // ClassLoader otherwise); normal classloaders delegate > // back to system ClassLoader first so this widening doesn't > // change the fact that context ClassLoader will be consulted > ClassLoader current = ObjectFactory.class.getClassLoader(); > > chain = system; > while (true) { > if (current == chain) { > // Assert: Current ClassLoader in chain of > // boot/extension/system ClassLoaders > return system; > } > if (chain == null) { > break; > } > chain = SecuritySupport.getParentClassLoader(chain); > } > > // Assert: Current ClassLoader not in chain of > // boot/extension/system ClassLoaders > return current; > } > > When ClassLoader.getParent() returns null in module mode it luckily works for the case of loading a class from within the JAXP module itself. > > This area of Xerces may need to be cleaned up to use ServiceLoader. > > In this respect the exception actually signaled the potential location of the issue that would of otherwise been hidden and trickier to find. However, the approach of throwing an exception does seem brutal. If an XML parser library is barfing, what else will? > > Paul. > > Exception in thread "main" java.lang.IllegalStateException: A module class loader does not support the hierarchical delegation mode and has no parent. > at java.lang.ClassLoader.getParent(ClassLoader.java:1429) > at com.sun.org.apache.xerces.internal.utils.SecuritySupport$3.run(SecuritySupport.java:82) > at java.security.AccessController.doPrivileged(Native Method) > at com.sun.org.apache.xerces.internal.utils.SecuritySupport.getParentClassLoader(SecuritySupport.java:78) > at com.sun.org.apache.xerces.internal.utils.ObjectFactory.findClassLoader(ObjectFactory.java:221) > at com.sun.org.apache.xerces.internal.utils.ObjectFactory.newInstance(ObjectFactory.java:255) > at com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory.getInstance(DTDDVFactory.java:64) > at com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory.getInstance(DTDDVFactory.java:49) > at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.(XML11Configuration.java:565) > at com.sun.org.apache.xerces.internal.parsers.XIncludeAwareParserConfiguration.(XIncludeAwareParserConfiguration.java:130) > at com.sun.org.apache.xerces.internal.parsers.XIncludeAwareParserConfiguration.(XIncludeAwareParserConfiguration.java:91) > at com.sun.org.apache.xerces.internal.parsers.DOMParser.(DOMParser.java:138) > at com.sun.org.apache.xerces.internal.parsers.DOMParser.(DOMParser.java:122) > at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.(DocumentBuilderImpl.java:120) > at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(DocumentBuilderFactoryImpl.java:76) > at sun.util.xml.XMLUtils.getLoadingDoc(XMLUtils.java:99) > at sun.util.xml.XMLUtils.load(XMLUtils.java:75) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:474) > at java.util.Properties$XMLUtils.invoke(Properties.java:1157) > at java.util.Properties$XMLUtils.load(Properties.java:1178) > at java.util.Properties.loadFromXML(Properties.java:875) > at org.astro.World.name(World.java:11) > at com.greetings.Hello.main(Hello.java:5) > From david.holmes at oracle.com Thu Jun 21 17:08:46 2012 From: david.holmes at oracle.com (David Holmes) Date: Fri, 22 Jun 2012 10:08:46 +1000 Subject: Review request for the incorrect check for "getClassLoader" permission In-Reply-To: <4FE36FC5.9020702@oracle.com> References: <4FE36FC5.9020702@oracle.com> Message-ID: <4FE3B78E.70405@oracle.com> Hi Mandy, On 22/06/2012 5:02 AM, Mandy Chung wrote: > David, Paul, > > I have a fix for the incorrect check w.r.t. "getClassLoader" permission > [1] and also update j.u.c.atomic for module mode. > > Webrev at: > http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/getclassloader-permission-fix/ I think ClassLoader.doClassLoaderPermissionCheck should be renamed ClassLoader.needsClassLoaderPermissionCheck. When I see "do" I expect it to actually check the permission which it doesn't. I don't understand the Atomic changes if the parent is going away. The whole isAncestor check becomes undefined. I also find it odd that a java.* class relies on an org.openjdk.jigsaw class. Will Platform eventually move into a core package like java.lang.modules ? > The security.sh test demonstrates what can be accessed in module mode > and lists the open issue. This patch is intended to fix the bug > introduced in this changeset: > http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/7b282c826118 > > Since there is no parent-child delegation relationship and the existing > security check applies, it can only access its own class loader in > module mode. This remains an open issue what security checks would be > appropriate in module mode and how well it plays with existing java > security policy file etc. And how a mix of custom and module loaders gets handled. David > As for testing, I ran test/java/util/concurrent/atomic tests in hybrid > mode and they passed (in fact I verified all java/lang and java/util > tests). Since AtomicUpdaters is marked to run in othervm mode, I > manually converted it as a module and it passes. > > Thanks > Mandy > [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2012-May/002614.html From mandy.chung at oracle.com Thu Jun 21 19:24:10 2012 From: mandy.chung at oracle.com (Mandy Chung) Date: Thu, 21 Jun 2012 19:24:10 -0700 Subject: Review request for the incorrect check for "getClassLoader" permission In-Reply-To: <4FE3B78E.70405@oracle.com> References: <4FE36FC5.9020702@oracle.com> <4FE3B78E.70405@oracle.com> Message-ID: <4FE3D74A.3060202@oracle.com> On 6/21/2012 5:08 PM, David Holmes wrote: > Hi Mandy, > > On 22/06/2012 5:02 AM, Mandy Chung wrote: >> David, Paul, >> >> I have a fix for the incorrect check w.r.t. "getClassLoader" permission >> [1] and also update j.u.c.atomic for module mode. >> >> Webrev at: >> http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/getclassloader-permission-fix/ >> > > I think ClassLoader.doClassLoaderPermissionCheck should be renamed > ClassLoader.needsClassLoaderPermissionCheck. When I see "do" I expect > it to actually check the permission which it doesn't. > Okay and will rename it - "needsClassLoaderPermissionCheck" was what I had initially :) > I don't understand the Atomic changes if the parent is going away. The > whole isAncestor check becomes undefined. > The main motivation of this fix is to change the check "cl == null" to determine if it's trusted i.e. a loader for loading system classes in module mode that can be non-null. The isAncestor check will be dealt with separately. > I also find it odd that a java.* class relies on an org.openjdk.jigsaw > class. Will Platform eventually move into a core package like > java.lang.modules ? > Yes, these static methods, if not the Platform class, will eventually move into an appropriate package. Paul, Alan and I have also discussed that we want to integrate jdk modularization changes to jdk8 (those not tied with jigsaw) periodically. We expect that some jdk changes will depend on these utility methods that may likely be moved to sun.misc. Thanks Mandy From paul.sandoz at oracle.com Fri Jun 22 01:55:45 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Fri, 22 Jun 2012 10:55:45 +0200 Subject: ClassLoader.getParent to throw exception for module classloader In-Reply-To: <4FE34789.1020103@oracle.com> References: <1E9A3E19-E61C-4852-A9AA-2367624A3578@oracle.com> <4FE34789.1020103@oracle.com> Message-ID: <7150408F-FEB2-4BCC-B703-DCEB04A2302F@oracle.com> On Jun 21, 2012, at 6:10 PM, Alan Bateman wrote: > On 21/06/2012 17:03, Paul Sandoz wrote: >> : >> >> In a off-list discussion there was a side comment about making ClassLoader.getParent throwing an exception for a module class loader. This type of thing is easy to loose so marking this down in email. >> > There is commented out implementation of getParent in org.openjdk.jigsaw.Loader that we should probably remove too (commented out because getParent is final). > Ah, well spotted. > On the exception then wouldn't UOE be more suitable than IllegalStateException? > Yes. I updated the webrev for future reference. Paul. From paul.sandoz at oracle.com Fri Jun 22 02:13:57 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Fri, 22 Jun 2012 11:13:57 +0200 Subject: ClassLoader.getParent to throw exception for module classloader In-Reply-To: <4FE3499E.90305@redhat.com> References: <1E9A3E19-E61C-4852-A9AA-2367624A3578@oracle.com> <4FE3499E.90305@redhat.com> Message-ID: On Jun 21, 2012, at 6:19 PM, David M. Lloyd wrote: > If "null" is to no longer mean "the JDK", then why not make it into a "nothing" class loader which loads nothing? What will be the parent CL of a "nothing" CL? > If it is continuing to mean the system class loader for base "java." stuff, why not just continue to let it be the parent? Every module will need visibility to "java.lang" at the least, right? > Note that the system CL is equal to the application module CL. What will be the parent CL of the base module CL? Paul. > On 06/21/2012 11:03 AM, Paul Sandoz wrote: >> Hi, >> >> http://cr.openjdk.java.net/~psandoz/jigsaw/classloader-getparent/webrev/ >> >> In a off-list discussion there was a side comment about making ClassLoader.getParent throwing an exception for a module class loader. This type of thing is easy to loose so marking this down in email. >> >> This patch results in the failure of: >> >> jdk/test/org/openjdk/jigsaw/optional-jaxp.sh >> >> See below for the exception in Xerces. The code below is incorrect for module mode because it assumes a hierarchical delegation model. In com.sun.org.apache.xerces.internal.utils.ObjectFactory.findClassLoader: >> >> ClassLoader context = SecuritySupport.getContextClassLoader(); >> ClassLoader system = SecuritySupport.getSystemClassLoader(); >> >> ClassLoader chain = system; >> while (true) { >> if (context == chain) { >> // Assert: we are on JDK 1.1 or we have no Context ClassLoader >> // or any Context ClassLoader in chain of system classloader >> // (including extension ClassLoader) so extend to widest >> // ClassLoader (always look in system ClassLoader if Xerces >> // is in boot/extension/system classpath and in current >> // ClassLoader otherwise); normal classloaders delegate >> // back to system ClassLoader first so this widening doesn't >> // change the fact that context ClassLoader will be consulted >> ClassLoader current = ObjectFactory.class.getClassLoader(); >> >> chain = system; >> while (true) { >> if (current == chain) { >> // Assert: Current ClassLoader in chain of >> // boot/extension/system ClassLoaders >> return system; >> } >> if (chain == null) { >> break; >> } >> chain = SecuritySupport.getParentClassLoader(chain); >> } >> >> // Assert: Current ClassLoader not in chain of >> // boot/extension/system ClassLoaders >> return current; >> } >> >> When ClassLoader.getParent() returns null in module mode it luckily works for the case of loading a class from within the JAXP module itself. >> >> This area of Xerces may need to be cleaned up to use ServiceLoader. >> >> In this respect the exception actually signaled the potential location of the issue that would of otherwise been hidden and trickier to find. However, the approach of throwing an exception does seem brutal. If an XML parser library is barfing, what else will? >> >> Paul. >> >> Exception in thread "main" java.lang.IllegalStateException: A module class loader does not support the hierarchical delegation mode and has no parent. >> at java.lang.ClassLoader.getParent(ClassLoader.java:1429) >> at com.sun.org.apache.xerces.internal.utils.SecuritySupport$3.run(SecuritySupport.java:82) >> at java.security.AccessController.doPrivileged(Native Method) >> at com.sun.org.apache.xerces.internal.utils.SecuritySupport.getParentClassLoader(SecuritySupport.java:78) >> at com.sun.org.apache.xerces.internal.utils.ObjectFactory.findClassLoader(ObjectFactory.java:221) >> at com.sun.org.apache.xerces.internal.utils.ObjectFactory.newInstance(ObjectFactory.java:255) >> at com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory.getInstance(DTDDVFactory.java:64) >> at com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory.getInstance(DTDDVFactory.java:49) >> at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.(XML11Configuration.java:565) >> at com.sun.org.apache.xerces.internal.parsers.XIncludeAwareParserConfiguration.(XIncludeAwareParserConfiguration.java:130) >> at com.sun.org.apache.xerces.internal.parsers.XIncludeAwareParserConfiguration.(XIncludeAwareParserConfiguration.java:91) >> at com.sun.org.apache.xerces.internal.parsers.DOMParser.(DOMParser.java:138) >> at com.sun.org.apache.xerces.internal.parsers.DOMParser.(DOMParser.java:122) >> at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.(DocumentBuilderImpl.java:120) >> at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(DocumentBuilderFactoryImpl.java:76) >> at sun.util.xml.XMLUtils.getLoadingDoc(XMLUtils.java:99) >> at sun.util.xml.XMLUtils.load(XMLUtils.java:75) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:474) >> at java.util.Properties$XMLUtils.invoke(Properties.java:1157) >> at java.util.Properties$XMLUtils.load(Properties.java:1178) >> at java.util.Properties.loadFromXML(Properties.java:875) >> at org.astro.World.name(World.java:11) >> at com.greetings.Hello.main(Hello.java:5) >> > > > -- > - DML From paul.sandoz at oracle.com Fri Jun 22 02:26:36 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Fri, 22 Jun 2012 11:26:36 +0200 Subject: ClassLoader.getParent to throw exception for module classloader In-Reply-To: <4FE3B58B.7000005@oracle.com> References: <1E9A3E19-E61C-4852-A9AA-2367624A3578@oracle.com> <4FE3B58B.7000005@oracle.com> Message-ID: <859A39DD-46A2-4DAA-8CF5-04C68587E49D@oracle.com> On Jun 22, 2012, at 2:00 AM, David Holmes wrote: > On 22/06/2012 2:03 AM, Paul Sandoz wrote: >> Hi, >> >> http://cr.openjdk.java.net/~psandoz/jigsaw/classloader-getparent/webrev/ >> >> In a off-list discussion there was a side comment about making ClassLoader.getParent throwing an exception for a module class loader. This type of thing is easy to loose so marking this down in email. > > So that would seem to have far-reaching consequences on various bits of code that check the classloader hierarchy to determine when they need to do security manager checks. > Indeed. All such JDK related code needs to be reviewed. > This leaves me wondering how custom classloaders fit into this world - do they have a parent that is a module loader? > Yes. Paul. From Alan.Bateman at oracle.com Fri Jun 22 02:35:29 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 22 Jun 2012 10:35:29 +0100 Subject: ClassLoader.getParent to throw exception for module classloader In-Reply-To: <4FE3B58B.7000005@oracle.com> References: <1E9A3E19-E61C-4852-A9AA-2367624A3578@oracle.com> <4FE3B58B.7000005@oracle.com> Message-ID: <4FE43C61.3090206@oracle.com> On 22/06/2012 01:00, David Holmes wrote: > > So that would seem to have far-reaching consequences on various bits > of code that check the classloader hierarchy to determine when they > need to do security manager checks. > > This leaves me wondering how custom classloaders fit into this world - > do they have a parent that is a module loader? The parent is determined when creating a class loader and I suspect it will be common that a module class loader will be a parent. There are lots of other issues that arise when there are module and non-module class loaders in the same fish tank and these will have to be worked out. I think the notion of a single parent class loader is somewhat dated now. For a long time there has been enterprise apps and frameworks that delegate to multiple parents and lots of configurations where there are graphs of class loaders. In any case, if there is code that is assuming a hierarchy and is using getParent to walk the hierarchy, maybe to find a specific loader, then it may not work as expected with modules. For that reason having getParent throw UOE would help identify the code that needs to be changed. As regards the JDK and the usages that are there for security reasons then these can be changed. I don't know if there is much of this kind of thing outside of the core platform. -Alan. From Alan.Bateman at oracle.com Fri Jun 22 02:42:35 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 22 Jun 2012 10:42:35 +0100 Subject: ClassLoader.getParent to throw exception for module classloader In-Reply-To: <7150408F-FEB2-4BCC-B703-DCEB04A2302F@oracle.com> References: <1E9A3E19-E61C-4852-A9AA-2367624A3578@oracle.com> <4FE34789.1020103@oracle.com> <7150408F-FEB2-4BCC-B703-DCEB04A2302F@oracle.com> Message-ID: <4FE43E0B.3090805@oracle.com> On 22/06/2012 09:55, Paul Sandoz wrote: > : > > > I updated the webrev for future reference. > BTW: What is the reason for Platform.isModuleClassLoader when you can simply check if this is an instanceof ModuleClassLoader? -Alan. From forax at univ-mlv.fr Fri Jun 22 03:20:45 2012 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Fri, 22 Jun 2012 12:20:45 +0200 Subject: ClassLoader.getParent to throw exception for module classloader In-Reply-To: <4FE43E0B.3090805@oracle.com> References: <1E9A3E19-E61C-4852-A9AA-2367624A3578@oracle.com> <4FE34789.1020103@oracle.com> <7150408F-FEB2-4BCC-B703-DCEB04A2302F@oracle.com> <4FE43E0B.3090805@oracle.com> Message-ID: <4FE446FD.3080501@univ-mlv.fr> On 06/22/2012 11:42 AM, Alan Bateman wrote: > On 22/06/2012 09:55, Paul Sandoz wrote: >> : >> >> >> I updated the webrev for future reference. >> > BTW: What is the reason for Platform.isModuleClassLoader when you can > simply check if this is an instanceof ModuleClassLoader? instanceof is evil :) and having a method that document that a module classloader has not the same behavior as a 'raw' classloader is a good idea (but maybe the method should be moved in ClassLoader). > > -Alan. R?mi From paul.sandoz at oracle.com Fri Jun 22 03:36:08 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Fri, 22 Jun 2012 12:36:08 +0200 Subject: ClassLoader.getParent to throw exception for module classloader In-Reply-To: <4FE43E0B.3090805@oracle.com> References: <1E9A3E19-E61C-4852-A9AA-2367624A3578@oracle.com> <4FE34789.1020103@oracle.com> <7150408F-FEB2-4BCC-B703-DCEB04A2302F@oracle.com> <4FE43E0B.3090805@oracle.com> Message-ID: <1FDD0B44-8D4C-4BAD-8701-6BF9307A61D2@oracle.com> On Jun 22, 2012, at 11:42 AM, Alan Bateman wrote: > On 22/06/2012 09:55, Paul Sandoz wrote: >> : >> >> >> I updated the webrev for future reference. >> > BTW: What is the reason for Platform.isModuleClassLoader when you can simply check if this is an instanceof ModuleClassLoader? > Me being pedantic. But i did think of some valid reasons :-) Anyone can currently extend ModuleClassLoader: ModuleClassLoader mcl = new ModuleClassLoader(ModuleSystem.base()) {}; Which i guess is a separate problem. Should we be checking for an instance of Loader? We can make Loader package private and then in ServiceLoader also do: if (Platform.isModuleClassLoader(loader)) { ... } else { ... } The general point is we can just change the implementation of Platform.isModuleClassLoader() if our assumptions on what a module class loader is changes. Paul. From Alan.Bateman at oracle.com Fri Jun 22 03:40:41 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 22 Jun 2012 11:40:41 +0100 Subject: ClassLoader.getParent to throw exception for module classloader In-Reply-To: <4FE446FD.3080501@univ-mlv.fr> References: <1E9A3E19-E61C-4852-A9AA-2367624A3578@oracle.com> <4FE34789.1020103@oracle.com> <7150408F-FEB2-4BCC-B703-DCEB04A2302F@oracle.com> <4FE43E0B.3090805@oracle.com> <4FE446FD.3080501@univ-mlv.fr> Message-ID: <4FE44BA9.6020804@oracle.com> On 22/06/2012 11:20, R?mi Forax wrote: > > instanceof is evil :) Sure but it is necessary because getParent is final (and can't be made non-final without open a big hole). Once ModuleClassLoader is final then an alternative would of course be to just check the class. > and having a method that document that a module classloader has not > the same behavior > as a 'raw' classloader is a good idea (but maybe the method should be > moved in ClassLoader). Yes, a big update to the javadoc will be required, including ClassLoader and java.lang.module.ModuleClassLoader. -Alan. From paul.sandoz at oracle.com Fri Jun 22 05:46:05 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Fri, 22 Jun 2012 14:46:05 +0200 Subject: Optional service dependencies only In-Reply-To: <28519014.AsdedVrNik@logoutik> References: <2FD90826-88FD-4FA8-B59A-4DF558EB8D6C@oracle.com> <1394015.gmxIYdubyC@logoutik> <78FA2DE4-33EC-405F-A80A-72014A15DDA9@oracle.com> <28519014.AsdedVrNik@logoutik> Message-ID: <08C0FF8C-FF38-4E13-8F55-984B9C19DDD7@oracle.com> On Jun 21, 2012, at 4:12 PM, Jaroslav Tulach wrote: > Dne ?t 21. ?ervna 2012 14:14:29, Paul Sandoz napsal(a): >> On Jun 21, 2012, at 11:35 AM, Jaroslav Tulach wrote: >>> [1] http://wiki.apidesign.org/wiki/JigsawServices >> Nice write up. We are definitely on the same page w.r.t. optionality. > > Right. Although, I would not call it optional. Each individual service > implementation is optional, but the resolution fails if none can be enabled. > >> What i was trying to do with my patch is change when those optional >> dependencies are processed i.e. at the end, thus first the application >> module dependencies are resolved as if no service provider modules are >> present. > > This is what should happen, with following clarification: I am still operating > in context of "complete repository" - e.g. I assume that the whole set of > transitive dependencies of a module has been computed and recorded at compile > time[1]. Only under such assumption range dependencies are acceptable, as far > as I understand the problem. > > [1] http://wiki.apidesign.org/wiki/RangeDependenciesAnalysed#No_Longer_NP- > Complete > For the moment i am gently sweeping this aspect under my mental rug :-) Yes there is room, but it is getting rather lumpy. >> My hunch is that because service consumers are decoupled from service >> providers it can be harder to predict or understand the result when the >> resolver mixes in the resolving of service provider dependencies. > > Right. "Classpath" dependencies first, services later is going to be more > understandable to humans. > OK. I am gonna refine my patch to make this more explicit. For a start we can unwind the call stack for those "Classpath" dependencies, then start again for services and essentially do: if (resolve(o, ch)) { // "Classpath" dependencies // Service dependencies while(/has service dependencies/) { Set serviceDependencies = ... // copy // Reset service dependency state for (ViewDependency sd: serviceDependencies) { resolve(0, new Choice(null, vd, null)); // Did the optional dependency get resolved? if not log warning as to why } } ensureServicesPresent(); } >>> However there should be a way to reconfigure these hints at the deployment >>> time. >> >> You raise a very interesting point! >> >> I think this suggests control of versions of transitive dependencies, much >> like what can be done with maven. Although the JAXP module may hint it >> strongly wants a version of Xerces another module may strongly hint it >> wants a different version. Perhaps the conflict could be mediated by >> changing module processing order, or perhaps by declaring what version >> should be used. > > The general principle should be: the one that knows more should make the > decision. That is why: > > #1 - a JAXP API knows it needs an implementation and knows Xerces module > provides one that is good enough > Every module declares what it needs with "requires ...". Perhaps it is a lack of imagination on my part but i don't currently see how we can strengthen that need in a decentralized environment. It is not when it all comes together that such needs can be compared? > #2 - the application vendor (providing the application module) knows all the > modules in its application and all the services and can decide certain > services should have different default (this is what Maven tried to do with > its "shortest" path dependency). Also when compiling the whole application, it > should be easy to decide whether different services are compatible with the > rest of modules or not as one knows the transitive closure of the whole > application. > > #3 - a deployer should have the option to override the application defaults > without coding in Java, imho. E.g. something like: > > $ java \ > -module com.myapp.xsltproc \ > -service javax.xml.parser.DocumentBuilderFactory \ > -default org.apache.crimson.CrimsonBuilderFactory > > ...of course one needs to install com.myapp.xsltproc module into the library > (is it the right term, I am still learning the terminology) as well as > org.apache.crimson module, so the resolver can enable both. > Yes, "install module M into library L". > #3a - the same using some configuration file which format is beyond the scope > of my current thinking > Or part of a module-info that declares entry points. > >>> I've noticed a discussion about security providers. You were in need of >>> changing their order and that could be also considered a "hint". Just >>> probably a hint provided externally, at the time the application is >>> started (e.g. not compiled). >> >> Yes, although in this case the order is associated with a service interface >> not the module. > > Good point. The order needs to be associated with service interface types. > Btw. NetBeans have an order of modules defined by topological sort on top of > graph with "requires" edges (not "requires service"). This graph is used for > startup and shutdown order. But services are orthogonal to this concept, they > have their own ordering (per interface type, in general). > > -jt > >> Modules can be service provider modules for multiple >> service interfaces. So hints on the order of service provider classes may >> need to be something separate to hints on the order modules are resolved. > > PS: What exactly order of modules is good for in Jigsaw? > I was not referring to any topological sort of the module graph. Order can affect the solution produced by the resolver. e.g. if module A depends on C, D, E in that order as per the declaration order of "requires ..." clauses in A's module declaration then perhaps one can override and say try E, C and D. I was assuming that this was one approach you were proposing. I don't think the Jigsaw resolver will attempt to preserve a topological order (e.g. depth first traversal order) for the list of resolved modules, mainly because it does not use LinkedHashSet and LinkedHashMap and when building the set of context HashSet is also used so that set would not be preserved either. This means there is no consistent order, based on module graph traversal, for the service provider classes in the configuration. It can also unnecessarily change each time the configuration is recalculated! (e.g. because of the hash code changes in SE8). We need to fix that. > PPS: In NetBeans it is used for order of activators and also for order of URLs > returned from systemWideClassLoader.getResources(...). It is guaranteed > resources are returned in topological order with respect to order of modules. > There is currently no concept of activators in Jigsaw and of course there is no system wide (or global) class loader. However, i think we need to preserve a topological order for modules throughout the process of producing a configuration so that iteration of service instances is consistent with that order. Paul. From alan.bateman at oracle.com Fri Jun 22 06:18:08 2012 From: alan.bateman at oracle.com (alan.bateman at oracle.com) Date: Fri, 22 Jun 2012 13:18:08 +0000 Subject: hg: jigsaw/jigsaw: Source code shuffle should copy module-info sources into place Message-ID: <20120622131809.2776247AA9@hg.openjdk.java.net> Changeset: fe2cf8de1297 Author: michaelm Date: 2012-06-22 14:08 +0100 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/rev/fe2cf8de1297 Source code shuffle should copy module-info sources into place ! make/scripts/shuffle/makeshufflescripts From Alan.Bateman at oracle.com Fri Jun 22 06:24:52 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 22 Jun 2012 14:24:52 +0100 Subject: Build-infra with jigsaw, progress update In-Reply-To: <4FE2D7ED.7080104@oracle.com> References: <4FE1A015.9090702@oracle.com> <4FE1A2C5.8050804@oracle.com> <4FE1AA9D.2070504@oracle.com> <4FE1EE60.6020208@oracle.com> <4FE2D7ED.7080104@oracle.com> Message-ID: <4FE47224.4070706@oracle.com> On 21/06/2012 09:14, Erik Joelsson wrote: > : > > That would be great! In my setup I've put them in a separate directory > (jdk/newsrc/share/moduleinfo). The reason for this is that javac won't > let me compile the classes in jdk.base in legacy mode if it sees a > module-info.java anywhere. I'm not sure if this is intended or not, > but that's how it currently works. > > /Erik I've just pushed a patch for Michael that puts the module-info.java source files in the expected location, newsrc/share/mclasses/. Jon - did you go through the sequence in Erik's mail, I can't quite tell if the compilation of the base module at step 3 is with hybrid javac. -Alan From paul.sandoz at oracle.com Fri Jun 22 07:32:54 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Fri, 22 Jun 2012 16:32:54 +0200 Subject: Service provider module dependency resolving webrev In-Reply-To: <2FD90826-88FD-4FA8-B59A-4DF558EB8D6C@oracle.com> References: <2FD90826-88FD-4FA8-B59A-4DF558EB8D6C@oracle.com> Message-ID: Hi, http://cr.openjdk.java.net/~psandoz/jigsaw/resolver-services/webrev.1/ Here is an alternative implementation, inspired from the discussions with Yarda, that should produce the same results as previous webrev. Each service provider module dependency is resolved separately. Paul. On Jun 19, 2012, at 4:33 PM, Paul Sandoz wrote: > Hi, > > http://cr.openjdk.java.net/~psandoz/jigsaw/resolver-services/webrev/ > [exec] | Configuring [mapp@=1.0] using library build/mlib [exec] | resolving ROOT requires mapp@=1.0 [exec] | - trying mapp at 1.0 [exec] | - pushing service provider module dependency stringer.StringTransformer -> requires optional mhasher [exec] | - pushing service provider module dependency stringer.StringTransformer -> requires optional mrotter [exec] | -- resolving mapp at 1.0 requires mstringer [exec] | --- trying mstringer at 1.0 [exec] | ---- resolving mstringer at 1.0 requires synthesized java.base@>=8 [exec] | ----- trying java.base [exec] | ----- pushing service provider module dependency java.nio.charset.spi.CharsetProvider -> requires optional sun.charsets [exec] | ----- pushing service provider module dependency java.nio.file.spi.FileSystemProvider -> requires optional jdk.zipfs [exec] | ----- pushing service provider module dependency sun.net.spi.nameservice.NameServiceDescriptor -> requires optional jdk.jndi [exec] | ------ resolving jdk.base at 8-ea requires local optional jdk.desktop.internal@=8-ea [exec] | ------- trying jdk.desktop.internal at 8-ea [exec] | ------- pushing service provider module dependency javax.print.PrintServiceLookup -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.print.StreamPrintServiceFactory -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.sound.midi.spi.MidiDeviceProvider -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.sound.midi.spi.MidiFileReader -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.sound.midi.spi.MidiFileWriter -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.sound.midi.spi.SoundbankReader -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.sound.sampled.spi.AudioFileReader -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.sound.sampled.spi.AudioFileWriter -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.sound.sampled.spi.FormatConversionProvider -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency javax.sound.sampled.spi.MixerProvider -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency sun.java2d.cmm.PCMM -> requires optional jdk.desktop [exec] | ------- pushing service provider module dependency sun.java2d.pipe.RenderingEngine -> requires optional jdk.desktop [exec] | -------- resolving jdk.desktop at 8-ea requires local jdk.base.internal@=8-ea [exec] | --------- resolving jdk.desktop at 8-ea requires jdk.jaxp@=8-ea [exec] | ---------- trying jdk.jaxp at 8-ea [exec] | ----------- resolving jdk.jaxp at 8-ea requires jdk.base@=8-ea [exec] | ------------ resolving jdk.jaxp at 8-ea requires synthesized java.base@>=8 [exec] | ------------- resolving jdk.desktop at 8-ea requires jdk.logging@=8-ea [exec] | -------------- trying jdk.logging at 8-ea [exec] | --------------- resolving jdk.logging at 8-ea requires jdk.base.internal@=8-ea [exec] | ---------------- resolving jdk.logging at 8-ea requires synthesized java.base@>=8 [exec] | ----------------- resolving jdk.desktop at 8-ea requires jdk.prefs@=8-ea [exec] | ------------------ trying jdk.prefs at 8-ea [exec] | ------------------- resolving jdk.prefs at 8-ea requires local jdk.base.internal@=8-ea [exec] | -------------------- resolving jdk.prefs at 8-ea requires jdk.jaxp@=8-ea [exec] | --------------------- resolving jdk.prefs at 8-ea requires synthesized java.base@>=8 [exec] | ---------------------- resolving jdk.desktop at 8-ea requires sun.charsets.internal@=8-ea [exec] | ----------------------- trying sun.charsets.internal at 8-ea [exec] | ------------------------ resolving sun.charsets at 8-ea requires local jdk.base.internal@=8-ea [exec] | ------------------------- resolving sun.charsets at 8-ea requires synthesized java.base@>=8 [exec] | -------------------------- resolving jdk.desktop at 8-ea requires synthesized java.base@>=8 [exec] | --------------------------- resolving jdk.base at 8-ea requires optional jdk.jaxp.internal@=8-ea [exec] | ---------------------------- resolving jdk.base at 8-ea requires local optional jdk.tls.internal@=8-ea [exec] | ----------------------------- trying jdk.tls.internal at 8-ea [exec] | ------------------------------ resolving jdk.tls at 8-ea requires local jdk.base.internal@=8-ea [exec] | ------------------------------- resolving jdk.tls at 8-ea requires synthesized java.base@>=8 [exec] | -------------------------------- resolving jdk.base at 8-ea requires local optional sun.localedata@=8-ea [exec] | --------------------------------- trying sun.localedata at 8-ea [exec] | ---------------------------------- resolving sun.localedata at 8-ea requires local jdk.base.internal@=8-ea [exec] | ----------------------------------- resolving sun.localedata at 8-ea requires synthesized java.base@>=8 [exec] | ------------------------------------ resolving jdk.base at 8-ea requires local optional sun.resources@=8-ea [exec] | ------------------------------------- trying sun.resources at 8-ea [exec] | -------------------------------------- resolving sun.resources at 8-ea requires local jdk.base.internal@=8-ea [exec] | --------------------------------------- resolving sun.resources at 8-ea requires synthesized java.base@>=8 [exec] | ---------------------------------------- resolving mapp at 1.0 requires synthesized java.base@>=8 [exec] | resolving SERVICE PROVIDER requires optional mhasher [exec] | - trying mhasher at 1.0 [exec] | -- resolving mhasher at 1.0 requires mstringer [exec] | --- resolving mhasher at 1.0 requires synthesized java.base@>=8 [exec] | resolving SERVICE PROVIDER requires optional mrotter [exec] | - trying mrotter at 1.0 [exec] | -- resolving mrotter at 1.0 requires mstringer [exec] | --- resolving mrotter at 1.0 requires synthesized java.base@>=8 [exec] | resolving SERVICE PROVIDER requires optional sun.charsets [exec] | resolving SERVICE PROVIDER requires optional jdk.zipfs [exec] | - trying jdk.zipfs at 8-ea [exec] | -- resolving jdk.zipfs at 8-ea requires jdk.base@=8-ea [exec] | --- resolving jdk.zipfs at 8-ea requires synthesized java.base@>=8 [exec] | resolving SERVICE PROVIDER requires optional jdk.jndi [exec] | - trying jdk.jndi at 8-ea [exec] | -- resolving jdk.jndi at 8-ea requires local jdk.auth.internal@=8-ea [exec] | --- trying jdk.auth.internal at 8-ea [exec] | ---- resolving jdk.auth at 8-ea requires local jdk.base.internal@=8-ea [exec] | ----- resolving jdk.auth at 8-ea requires jdk.logging@=8-ea [exec] | ------ resolving jdk.auth at 8-ea requires synthesized java.base@>=8 [exec] | ------- resolving jdk.jndi at 8-ea requires local jdk.base.internal@=8-ea [exec] | -------- resolving jdk.jndi at 8-ea requires optional jdk.desktop@=8-ea [exec] | --------- resolving jdk.jndi at 8-ea requires jdk.rmi@=8-ea [exec] | ---------- trying jdk.rmi at 8-ea [exec] | ----------- resolving jdk.rmi at 8-ea requires jdk.base.internal@=8-ea [exec] | ------------ resolving jdk.rmi at 8-ea requires jdk.logging@=8-ea [exec] | ------------- resolving jdk.rmi at 8-ea requires jdk.tls@=8-ea [exec] | -------------- resolving jdk.rmi at 8-ea requires synthesized java.base@>=8 [exec] | --------------- resolving jdk.jndi at 8-ea requires jdk.tls.internal@=8-ea [exec] | ---------------- resolving jdk.jndi at 8-ea requires synthesized java.base@>=8 [exec] | resolving SERVICE PROVIDER requires optional jdk.desktop From jonathan.gibbons at oracle.com Fri Jun 22 07:53:30 2012 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Fri, 22 Jun 2012 07:53:30 -0700 Subject: Build-infra with jigsaw, progress update In-Reply-To: <4FE47224.4070706@oracle.com> References: <4FE1A015.9090702@oracle.com> <4FE1A2C5.8050804@oracle.com> <4FE1AA9D.2070504@oracle.com> <4FE1EE60.6020208@oracle.com> <4FE2D7ED.7080104@oracle.com> <4FE47224.4070706@oracle.com> Message-ID: <4FE486EA.3010800@oracle.com> On 06/22/2012 06:24 AM, Alan Bateman wrote: > On 21/06/2012 09:14, Erik Joelsson wrote: >> : >> >> That would be great! In my setup I've put them in a separate >> directory (jdk/newsrc/share/moduleinfo). The reason for this is that >> javac won't let me compile the classes in jdk.base in legacy mode if >> it sees a module-info.java anywhere. I'm not sure if this is intended >> or not, but that's how it currently works. >> >> /Erik > I've just pushed a patch for Michael that puts the module-info.java > source files in the expected location, newsrc/share/mclasses/. > > Jon - did you go through the sequence in Erik's mail, I can't quite > tell if the compilation of the base module at step 3 is with hybrid > javac. > > -Alan > The sequence looks mostly fine, to the level of detail that is given. I agree that once you're past langtools, the build should use hybrid javac. I was surprised to see "langtools" and "stripprop" mentioned in close proximity. langtools uses compileproperties, not strip properties. But that is not a Jigsaw issue, it is either a typo, or an artifact of new build that needs to be fixed. -- Jon From chris.hegarty at oracle.com Fri Jun 22 08:24:48 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Fri, 22 Jun 2012 16:24:48 +0100 Subject: ModuleFileParser reasoning Message-ID: <4FE48E40.7020305@oracle.com> I realize I sent out a webrev [1] for this proposed change without giving any previous context about the decision making to arrive at a pull parser API. So, here it is... We have use cases that require a module file to be read without decompressing or extracting one or more sections, to support more than on disk extraction, and quickly terminate when you get what you want. * Signer tool The signer tool needs to extract and validate the module file hashes. For performance reasons it is desirable to skip decompression of any section content. * IDE's An IDE will need to be able to easily extract the class files for analysis, possibly into memory. Again, for performance reasons it is desirable to skip decompressing any other section content, and also finish processing directly after the class files have been extracted. * Installer Extract module file content and install it into the module library. Separating out the internal library format from the parsing of the module file. * Tools for listing contents and/or display module information. It is desirable to support a tool for listing the contents of a module file, say 'jpkg contents ', and showing the module info. Again, it is desirable to skip decompression of any section content. The same interface for reading a jmod file can potentially be used with corresponding implementations for wrapping a modular jar and also wrapping a module installed into a library. The former enables uniform installation into a library, the latter enables easy extraction when hooked up to a module file writer (which is also what the signer needs to do when signing). Given that, it is clear we need to separate out the parsing of the module file from the other operations. There are two obvious streaming parser choices, pull or push. Pull was chosen mainly for simplicity of the API, and placing complete control in the hands of the developer. While scoping, here are some of the reasons that lead to this conclusion: * Supporting both raw and decompressed section content in a push parser is a little confusing. The callback could always be supplied with the raw data, but then it forces the developer to deal with decompressing. * A content handler interface could return a value to indicate that further processing of the module file is not required, but we would still need the ability to retrieve/verify the section hash. Doesn't seem to fit well to build a validating parse on top of a non validating parser. * Some compression formats can be time consuming to extract, handing of raw section data to be extracted seems reasonable, in some cases. Again, doesn't seem to fit well with push. * Simple tests to iterate over section and subsection content, validate the module file, and extract a single section, using a pull parser show that they are readable and easily understandable. * While you can possibly layer a push parser over a pull, it is much more difficult to do the opposite. * Anonymous inner classes can be a PITA for developers. While lambda will improve this for SAM-type classes, such classes for any push model may have more than one method and/or implementations will likely mutate state. Along with the usual comments that are made when comparing SAX and StAX ;-) Thanks, -Chris. [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2012-June/002794.html From jaroslav.tulach at oracle.com Fri Jun 22 09:41:34 2012 From: jaroslav.tulach at oracle.com (Jaroslav Tulach) Date: Fri, 22 Jun 2012 09:41:34 -0700 (PDT) Subject: Optional service dependencies only Message-ID: <67a9b3c3-98cd-4b74-a939-50d285523ed3@default> Hello Paul, I smell a little misunderstanding. When I titled the message "optinal service dependencies only" I really ment it! If you want to get rid of the NP-Complete problem, remove "requires service xyz" concept and allow only "requires service optional xyz". In your algorithm it would be equal to commenting out call to "ensureServicesPresent();" unless I am mistaken. Would you accept that? -jt From jaroslav.tulach at oracle.com Fri Jun 22 09:59:29 2012 From: jaroslav.tulach at oracle.com (Jaroslav Tulach) Date: Fri, 22 Jun 2012 09:59:29 -0700 (PDT) Subject: Offer default was: Optional service dependencies only Message-ID: Hello. I've just send a message asking whether having "Optional service dependencies only" is acceptable. Now I will continue this doublethink and assume the answer to that question is "No way!". For example the JAXP API. While it can always throw FactoryConfigurationException when the required service is not present, it is not very polite for the API users. The only reason to have dependency on the JAXP API is to parse something, so the implementation is in fact not "optional" service at all. Can we eliminate the NP-Complete nature of the resolution algorithm and still support "requires service"? Executive summary is "Yes, we can". Explanation of what would have to be done follows... >>> However there should be a way to reconfigure these hints at the deployment >>> time. >> >> You raise a very interesting point! >> >> I think this suggests control of versions of transitive dependencies, much >> like what can be done with maven. Although the JAXP module may hint it >> strongly wants a version of Xerces another module may strongly hint it >> wants a different version. Perhaps the conflict could be mediated by >> changing module processing order, or perhaps by declaring what version >> should be used. > > The general principle should be: the one that knows more should make the > decision. That is why: > > #1 - a JAXP API knows it needs an implementation and knows Xerces module > provides one that is good enough > Paul wrote: > Every module declares what it needs with "requires ...". Perhaps it is a lack of imagination on > my part but i don't currently see how we can strengthen that need in a decentralized > environment. It is not when it all comes together that such needs can be compared? The way to eliminate NP-Completeness is to require a hint. So a JAXP module can either say: requires service optional javax.xml.parsers.DocumentBuilderFactory; and that is fine. Such definition does not need hints, as it is trivially satisfied with nothing, so the resolution algorithm is simple. On the other hand the JAXP module can require real implementation: requires service javax.xml.parsers.DocumentBuilderFactory default from com.oracle.apache.xerces at 1.8 By giving hint - e.g. by directly specifying that the default implementation of XML parser is known (at the time of compilation) to be provided by com.oracle.apache.xerces at 1.8 module, we can avoid NP-Completeness from the system. I can formalize the concept, if there is an interest. Can the Jigsaw gang consider this little change in "requires service" grammar? -jt > #3a - the same using some configuration file which format is beyond the scope > of my current thinking > Paul wrote: > Or part of a module-info that declares entry points. PS: Well, the whole point of #3 and #3a is to do this configuration change without any compilation. So it cannot be in in module-info.java that would be the case #2. From paul.sandoz at oracle.com Fri Jun 22 10:39:27 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Fri, 22 Jun 2012 19:39:27 +0200 Subject: Optional service dependencies only In-Reply-To: <67a9b3c3-98cd-4b74-a939-50d285523ed3@default> References: <67a9b3c3-98cd-4b74-a939-50d285523ed3@default> Message-ID: Hi, On Jun 22, 2012, at 6:41 PM, Jaroslav Tulach wrote: > Hello Paul, > I smell a little misunderstanding. When I titled the message "optinal service dependencies only" I really ment it! > > If you want to get rid of the NP-Complete problem, remove "requires service xyz" concept and allow only "requires service optional xyz". As i said i am shoving the NP-Complete problem under the rug for now. Forgive me! IIUC the resolving of the root module dependency and the service provider module dependencies although performed separately, in the webrev, is still NP-Complete. > In your algorithm it would be equal to commenting out call to "ensureServicesPresent();" unless I am mistaken. > "ensureServicesPresent" is validating constraints, post resolving, on the set of resolved modules so it does not affect the resolving algorithm. > Would you accept that? Not sure :-) it depends on your response. I think a source of misunderstand is the getting rid of the NP-Complete problem, which i am currently ignoring. Paul. From alan.bateman at oracle.com Fri Jun 22 11:42:07 2012 From: alan.bateman at oracle.com (alan.bateman at oracle.com) Date: Fri, 22 Jun 2012 18:42:07 +0000 Subject: hg: jigsaw/jigsaw/jdk: ServiceLoader updates Message-ID: <20120622184236.9150947AB6@hg.openjdk.java.net> Changeset: 99f164e91346 Author: psandoz Date: 2012-06-22 19:32 +0100 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/99f164e91346 ServiceLoader updates - list of service provider modules is now global in configuration - restore ServiceLoaer.load methods (remove hacks) ! make/java/java/FILES_java.gmk ! src/share/classes/java/util/ServiceLoader.java ! src/share/classes/org/openjdk/jigsaw/Configuration.java ! src/share/classes/org/openjdk/jigsaw/Context.java ! src/share/classes/org/openjdk/jigsaw/Linker.java ! src/share/classes/org/openjdk/jigsaw/Loader.java ! src/share/classes/org/openjdk/jigsaw/LoaderPool.java + src/share/classes/org/openjdk/jigsaw/ModuleServiceLoader.java ! src/share/classes/org/openjdk/jigsaw/SimpleLibrary.java ! src/share/native/common/jigsaw.c + test/org/openjdk/jigsaw/services/classloader.sh From jaroslav.tulach at oracle.com Sat Jun 23 05:00:11 2012 From: jaroslav.tulach at oracle.com (Jaroslav Tulach) Date: Sat, 23 Jun 2012 14:00:11 +0200 Subject: Unit testing the resolver was: Optional service dependencies only In-Reply-To: References: <67a9b3c3-98cd-4b74-a939-50d285523ed3@default> Message-ID: <3483956.tVrVunNkeV@logoutik> Dne P? 22. ?ervna 2012 19:39:27, Paul Sandoz napsal(a): > Hi, > > If you want to get rid of the NP-Complete problem, remove "requires > > service xyz" concept and allow only "requires service optional xyz". > As i said i am shoving the NP-Complete problem under the rug for now. > Forgive me! Paul, are you looking only for comments to your webrev? If so, then I am sorry for taking much larger perspective. I don't want to stop you integrating your change at all. The more low level comment I can provide: Is there a way to write simple unit test? For example what would be the simplest way to simulate the erroneous resolution described at [1] when the b service provider can't be found easily? -jt [1] http://wiki.apidesign.org/wiki/JigsawServices From Alan.Bateman at oracle.com Sat Jun 23 13:05:27 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 23 Jun 2012 21:05:27 +0100 Subject: Review request for the incorrect check for "getClassLoader" permission In-Reply-To: <4FE36FC5.9020702@oracle.com> References: <4FE36FC5.9020702@oracle.com> Message-ID: <4FE62187.2040208@oracle.com> On 21/06/2012 20:02, Mandy Chung wrote: > David, Paul, > > I have a fix for the incorrect check w.r.t. "getClassLoader" > permission [1] and also update j.u.c.atomic for module mode. > > Webrev at: > > http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/getclassloader-permission-fix/ I agree with David that"doClassLoaderPermissionCheck" is a bit odd given that it returns a boolean. In Class.checkMemberAccess L2187 it might be better to swap the expressions so that it reads: if (!Platform.isPlatformLoader(ccl) && (ccl != cl) && It doesn't change anything of course but some of these checks are subtle so this gives you the minimum change against jdk8. Same comment on Atomic* classes where it would be very easy to break something in a merge. Just on thought on the Atomic* changes: given that this is maintained in Doug's CVS and then pulled into OpenJDK periodically then maybe we should just create a package private class in j,u.c with one static method like Platform.isPlatformLoader. This could make life easier for a few people. -Alan. From erik.joelsson at oracle.com Mon Jun 25 01:36:17 2012 From: erik.joelsson at oracle.com (Erik Joelsson) Date: Mon, 25 Jun 2012 10:36:17 +0200 Subject: Build-infra with jigsaw, progress update In-Reply-To: <4FE486EA.3010800@oracle.com> References: <4FE1A015.9090702@oracle.com> <4FE1A2C5.8050804@oracle.com> <4FE1AA9D.2070504@oracle.com> <4FE1EE60.6020208@oracle.com> <4FE2D7ED.7080104@oracle.com> <4FE47224.4070706@oracle.com> <4FE486EA.3010800@oracle.com> Message-ID: <4FE82301.8040309@oracle.com> I will try to clear this up. On 2012-06-22 16:53, Jonathan Gibbons wrote: > On 06/22/2012 06:24 AM, Alan Bateman wrote: >> I've just pushed a patch for Michael that puts the module-info.java >> source files in the expected location, newsrc/share/mclasses/. >> >> Jon - did you go through the sequence in Erik's mail, I can't quite >> tell if the compilation of the base module at step 3 is with hybrid >> javac. >> >> -Alan >> > I'm not sure what you mean by "legacy mode" in this context. > > javac has different rules for 0 or 1 module, and many modules. If > you are trying to compile many modules together, you have to set up > the options appropriately. > In step 3, I'm compiling the classes in jdk.base using the hybrid javac running on the bootjdk. I used "legacy mode" to mean compiling using hybrid javac without modules. The command line used looks something like this: bootjdk/bin/java "-Xbootclasspath/p:$(LANGTOOLS_OUTPUTDIR)/dist/bootstrap/lib/javac.jar" -jar $(LANGTOOLS_OUTPUTDIR)/dist/bootstrap/lib/javac.jar -implicit:none -sourcepath "/newsrc/share/mclasses/jdk.base:/jdk.base:/jdk.base" -d @ The above line works if there is no module-info.java in the sourcepath. If there is a module-info.java (and it doesn't matter if it's listed in the java-file list or not), then a bunch of errors like this are given: /localhome/mercurial/jigsaw-build-infra/jdk/newsrc/share/mclasses/jdk.base/module-info.java:2: error: no version available for module jdk.desktop.internal at 8-ea requires optional local jdk.desktop.internal at 8-ea; Is there a way I can force javac to ignore modules and module-info.java? If there isn't, I will have to copy all java-files in jdk.base to a temp directory in order to manage step 3 in the bootstrap sequence. Another question on javac and modules. I have successfully compiled all modules together in one compile in step 5. I'm not succeeding in compiling just a subset of them however. What I would like to be able to do is compiling the bare minimum set of modules in the jdk repo and then jump over to jaxp, jaxws, corba and langtools and compile them properly in a modular way. Are there known issues concerning this at this point? > The sequence looks mostly fine, to the level of detail that is given. > I agree that once you're past langtools, the build should use hybrid > javac. Yes, I can't see it working any other way. > > I was surprised to see "langtools" and "stripprop" mentioned in close > proximity. langtools uses compileproperties, not strip properties. > But that is not a Jigsaw issue, it is either a typo, or an artifact > of new build that needs to be fixed. > This was a typo, I meant compileproperties. /Erik From Alan.Bateman at oracle.com Mon Jun 25 07:15:06 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 25 Jun 2012 15:15:06 +0100 Subject: Build-infra with jigsaw, progress update In-Reply-To: <4FE82301.8040309@oracle.com> References: <4FE1A015.9090702@oracle.com> <4FE1A2C5.8050804@oracle.com> <4FE1AA9D.2070504@oracle.com> <4FE1EE60.6020208@oracle.com> <4FE2D7ED.7080104@oracle.com> <4FE47224.4070706@oracle.com> <4FE486EA.3010800@oracle.com> <4FE82301.8040309@oracle.com> Message-ID: <4FE8726A.7060906@oracle.com> On 25/06/2012 09:36, Erik Joelsson wrote: > : > > Another question on javac and modules. I have successfully compiled > all modules together in one compile in step 5. I'm not succeeding in > compiling just a subset of them however. What I would like to be able > to do is compiling the bare minimum set of modules in the jdk repo and > then jump over to jaxp, jaxws, corba and langtools and compile them > properly in a modular way. Are there known issues concerning this at > this point? There are a few warts. One is java.util.Properties due to a dependency on types in the JAXP module. This one hasn't leaked in the API and so is fixable, just hasn't been done yet. The GenerateNativeHeader annotation that came up on build-dev recently is in the langtools repository but as per the agreement there, it has been removed from classes in the base module. Otherwise I think the only issue is optional dependencies that involve static references as those types need to be available at compile time. The base module still has a couple of optional dependencies that mean the classes from a few other modules are needed, including the desktop module. -Alan. From jonathan.gibbons at oracle.com Mon Jun 25 07:56:27 2012 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Mon, 25 Jun 2012 07:56:27 -0700 Subject: Build-infra with jigsaw, progress update In-Reply-To: <4FE82301.8040309@oracle.com> References: <4FE1A015.9090702@oracle.com> <4FE1A2C5.8050804@oracle.com> <4FE1AA9D.2070504@oracle.com> <4FE1EE60.6020208@oracle.com> <4FE2D7ED.7080104@oracle.com> <4FE47224.4070706@oracle.com> <4FE486EA.3010800@oracle.com> <4FE82301.8040309@oracle.com> Message-ID: <4FE87C1B.2080907@oracle.com> On 06/25/2012 01:36 AM, Erik Joelsson wrote: > > bootjdk/bin/java > "-Xbootclasspath/p:$(LANGTOOLS_OUTPUTDIR)/dist/bootstrap/lib/javac.jar" -jar > $(LANGTOOLS_OUTPUTDIR)/dist/bootstrap/lib/javac.jar -implicit:none > -sourcepath > "/newsrc/share/mclasses/jdk.base:/jdk.base:/jdk.base" > -d @ Try adding -modulepath -- Jon From mandy.chung at oracle.com Mon Jun 25 10:00:04 2012 From: mandy.chung at oracle.com (Mandy Chung) Date: Mon, 25 Jun 2012 10:00:04 -0700 Subject: Review request for the incorrect check for "getClassLoader" permission In-Reply-To: <4FE62187.2040208@oracle.com> References: <4FE36FC5.9020702@oracle.com> <4FE62187.2040208@oracle.com> Message-ID: <4FE89914.1020207@oracle.com> On 6/23/2012 1:05 PM, Alan Bateman wrote: > On 21/06/2012 20:02, Mandy Chung wrote: >> David, Paul, >> >> I have a fix for the incorrect check w.r.t. "getClassLoader" >> permission [1] and also update j.u.c.atomic for module mode. >> >> Webrev at: >> http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/getclassloader-permission-fix/ > I agree with David that"doClassLoaderPermissionCheck" is a bit odd > given that it returns a boolean. > I rename it to "needsClassLoaderPermissionCheck" as David suggests. > In Class.checkMemberAccess L2187 it might be better to swap the > expressions so that it reads: > > if (!Platform.isPlatformLoader(ccl) && (ccl != cl) && > > It doesn't change anything of course but some of these checks are > subtle so this gives you the minimum change against jdk8. > > Same comment on Atomic* classes where it would be very easy to break > something in a merge. I see your concern of the potential mistake in merging with jdk8. As we plan to push some jdk changes in jigsaw to jdk8 regularly to keep jigsaw repo with minimum change against jdk8, to prepare for it - what do you think if I add a new method sun.misc.VM.isSystemDomain(ClassLoader) something like that? The jdk8 version of this method will just check cl == null. > Just on thought on the Atomic* changes: given that this is maintained > in Doug's CVS and then pulled into OpenJDK periodically then maybe we > should just create a package private class in j,u.c with one static > method like Platform.isPlatformLoader. This could make life easier for > a few people. > Would the new sun.misc.VM.isSystemDomain method help? That would avoid having j.u.c. to reference org.openjdk.jigsaw code. Mandy From paul.sandoz at oracle.com Mon Jun 25 10:55:23 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Mon, 25 Jun 2012 19:55:23 +0200 Subject: Review request for the incorrect check for "getClassLoader" permission In-Reply-To: <4FE89914.1020207@oracle.com> References: <4FE36FC5.9020702@oracle.com> <4FE62187.2040208@oracle.com> <4FE89914.1020207@oracle.com> Message-ID: <401226DF-D833-426E-A97B-79CE6ECF55CA@oracle.com> On Jun 25, 2012, at 7:00 PM, Mandy Chung wrote: > > > On 6/23/2012 1:05 PM, Alan Bateman wrote: >> On 21/06/2012 20:02, Mandy Chung wrote: >>> David, Paul, >>> >>> I have a fix for the incorrect check w.r.t. "getClassLoader" permission [1] and also update j.u.c.atomic for module mode. >>> >>> Webrev at: >>> http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/getclassloader-permission-fix/ >> I agree with David that"doClassLoaderPermissionCheck" is a bit odd given that it returns a boolean. >> > > I rename it to "needsClassLoaderPermissionCheck" as David suggests. > >> In Class.checkMemberAccess L2187 it might be better to swap the expressions so that it reads: >> >> if (!Platform.isPlatformLoader(ccl) && (ccl != cl) && >> >> It doesn't change anything of course but some of these checks are subtle so this gives you the minimum change against jdk8. >> >> Same comment on Atomic* classes where it would be very easy to break something in a merge. > > I see your concern of the potential mistake in merging with jdk8. As we plan to push some jdk changes in jigsaw to jdk8 regularly to keep jigsaw repo with minimum change against jdk8, to prepare for it - what do you think if I add a new method sun.misc.VM.isSystemDomain(ClassLoader) something like that? The jdk8 version of this method will just check cl == null. > >> Just on thought on the Atomic* changes: given that this is maintained in Doug's CVS and then pulled into OpenJDK periodically then maybe we should just create a package private class in j,u.c with one static method like Platform.isPlatformLoader. This could make life easier for a few people. >> > > Would the new sun.misc.VM.isSystemDomain method help? That would avoid having j.u.c. to reference org.openjdk.jigsaw code. > Should we encapsulate the following into a method? + if (ccl != cl && !Platform.isPlatformLoader(ccl) && + (Platform.isPlatformLoader(cl) || !isAncestor(cl, ccl))) if (Platform.needsPackageAccessCheck(ccl, cl)) Because it is rather subtle i think it useful to stuff the implementation (and document) in one place, plus we could decide to change the impl e.g. if both CLs are module CLs then it always returns false. Paul. From paul.sandoz at oracle.com Mon Jun 25 11:54:24 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Mon, 25 Jun 2012 20:54:24 +0200 Subject: Unit testing the resolver was: Optional service dependencies only In-Reply-To: <3483956.tVrVunNkeV@logoutik> References: <67a9b3c3-98cd-4b74-a939-50d285523ed3@default> <3483956.tVrVunNkeV@logoutik> Message-ID: <1CBB7F73-3D1D-479E-B3F4-3F7BD2312850@oracle.com> On Jun 23, 2012, at 2:00 PM, Jaroslav Tulach wrote: > Dne P? 22. ?ervna 2012 19:39:27, Paul Sandoz napsal(a): >> Hi, >>> If you want to get rid of the NP-Complete problem, remove "requires >>> service xyz" concept and allow only "requires service optional xyz". >> As i said i am shoving the NP-Complete problem under the rug for now. >> Forgive me! > > Paul, are you looking only for comments to your webrev? If so, then I am sorry > for taking much larger perspective. No need to apologize! I think that larger perspective has been quite informative and useful (in addition to also helping find a bug in configuration ordering :-) ). > I don't want to stop you integrating your > change at all. > No problem. Mostly what i am looking for is agreement that service provider module dependencies should be resolved after the "Classpath" dependences since it is: - easier to report errors; - easier for developers to understand; and - has wider implications to help avoid, in part, NP-Complete problems. Would you consider the above a fair assessment? > The more low level comment I can provide: Is there a way to write simple unit > test? For example what would be the simplest way to simulate the erroneous > resolution described at [1] when the b service provider can't be found easily? > The closest thing in the current tests is the _Configurator.java located in jdk/test/org/openjdk/jigsaw e.g.: new Test("diamond", true, "x at 1") { void init(MockLibrary mlib) { mlib.add(module("x at 1").requires("y at 2").requires("w at 4")) .add(module("y at 2").requires("z@>=3")) .add(module("z at 9")) .add(module("z at 4")) .add(module("z at 3")) .add(module("w at 4").requires("z@<=4")); } void ref(ConfigurationBuilder cfbd) { cfbd.add(context("x at 1").remote("+w", "+y")) .add(context("y at 2").remote("+z")) .add(context("z at 4")) .add(context("w at 4").remote("+z")); } }; new Test("diamond-fail", false, "x at 1") { void init(MockLibrary mlib) { mlib.add(module("x at 1").requires("y at 2").requires("w at 4")) .add(module("y at 2").requires("z@<=3")) .add(module("z at 4")) .add(module("z at 3")) .add(module("z at 9")) .add(module("w at 4").requires("z@>=4")); } }; It encompasses the process of resolving and linking. IMHO it looks quite reasonable, although i wish we could use TestNG to help out a bit :-) anyway.... the module and configuration builders need to be updated to support services. It could easily be adapted to perform just resolving and check the resolution result. Paul. > -jt > > [1] http://wiki.apidesign.org/wiki/JigsawServices > From Alan.Bateman at oracle.com Mon Jun 25 12:11:18 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 25 Jun 2012 20:11:18 +0100 Subject: Review request for the incorrect check for "getClassLoader" permission In-Reply-To: <4FE89914.1020207@oracle.com> References: <4FE36FC5.9020702@oracle.com> <4FE62187.2040208@oracle.com> <4FE89914.1020207@oracle.com> Message-ID: <4FE8B7D6.7020009@oracle.com> On 25/06/2012 18:00, Mandy Chung wrote: > > : > > I see your concern of the potential mistake in merging with jdk8. As > we plan to push some jdk changes in jigsaw to jdk8 regularly to keep > jigsaw repo with minimum change against jdk8, to prepare for it - what > do you think if I add a new method > sun.misc.VM.isSystemDomain(ClassLoader) something like that? The jdk8 > version of this method will just check cl == null. VM.isSystemDomain would be fine, except perhaps for the j.u.c classes where we'd need to check with Doug (David or Chris might do that). The reason is that we try to avoid changes to the j.u.c classes in OpenJDK so that it makes it easier to sync up. Also the j.u.c classes are released standalone too and this makes it a bit awkward to have a dependency on a new method in sun.misc.VM. A package-private class in j.u.c might help there. -Alan From paul.sandoz at oracle.com Mon Jun 25 12:29:12 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Mon, 25 Jun 2012 21:29:12 +0200 Subject: Offer default was: Optional service dependencies only In-Reply-To: References: Message-ID: <18B4FDB3-78BA-4153-93EB-21862658B290@oracle.com> Hi, On Jun 22, 2012, at 6:59 PM, Jaroslav Tulach wrote: > > Paul wrote: >> Every module declares what it needs with "requires ...". Perhaps it is a lack of imagination on >> my part but i don't currently see how we can strengthen that need in a decentralized >> environment. It is not when it all comes together that such needs can be compared? > > The way to eliminate NP-Completeness is to require a hint. So a JAXP module can either say: > > requires service optional javax.xml.parsers.DocumentBuilderFactory; > > and that is fine. Such definition does not need hints, as it is trivially satisfied with nothing, so the resolution algorithm is simple. On the other hand the JAXP module can require real implementation: > > requires service javax.xml.parsers.DocumentBuilderFactory default from com.oracle.apache.xerces at 1.8 > > By giving hint - e.g. by directly specifying that the default implementation of XML parser is known (at the time of compilation) to be provided by com.oracle.apache.xerces at 1.8 module, we can avoid NP-Completeness from the system. I can formalize the concept, if there is an interest. > Some questions: 1) when would the dependency "com.oracle.apache.xerces at 1.8" get resolved? 2) what if other service provider modules for javax.xml.parsers.DocumentBuilderFactory are present in the library? My attempt to answer: 1) it would be resolved as non-optional service provider module dependency iff no other service provider modules are present 2) it would be resolved after the other service provider module dependencies, thus would appear last when iterating over the service instances. > Can the Jigsaw gang consider this little change in "requires service" grammar? > -jt > >> #3a - the same using some configuration file which format is beyond the scope >> of my current thinking >> > > Paul wrote: >> Or part of a module-info that declares entry points. > > PS: Well, the whole point of #3 and #3a is to do this configuration change without any compilation. So it cannot be in in module-info.java that would be the case #2. > Doh. Paul. From mandy.chung at oracle.com Mon Jun 25 13:12:20 2012 From: mandy.chung at oracle.com (Mandy Chung) Date: Mon, 25 Jun 2012 13:12:20 -0700 Subject: Review request for the incorrect check for "getClassLoader" permission In-Reply-To: <4FE36FC5.9020702@oracle.com> References: <4FE36FC5.9020702@oracle.com> Message-ID: <4FE8C624.8020209@oracle.com> Alan, David, Paul, Given that j.u.c is pulled down from Doug's CVS into OpenJDK, I am having a second thought to leave the j.u.c.atomic change out from this patch. I have added a static needsPackalgeAccessCheck method in sun.reflect.misc.ReflectUtil class that could be used by j.u.c.atomic updater once it's integrated to jdk8. Since this j.u.c. change isn't critical for jigsaw at this time, we'll check with Doug (probably with David or Chris' help) and get appropriate change made in JSR166e source that will subsequently get pulled down in JDK 8 and jigsaw. Updated webrev: http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/getclassloader-permission-fix.01/ Thanks Mandy On 6/21/2012 12:02 PM, Mandy Chung wrote: > David, Paul, > > I have a fix for the incorrect check w.r.t. "getClassLoader" > permission [1] and also update j.u.c.atomic for module mode. > > Webrev at: > > http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/getclassloader-permission-fix/ > > The security.sh test demonstrates what can be accessed in module mode > and lists the open issue. This patch is intended to fix the bug > introduced in this changeset: > http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/7b282c826118 > > Since there is no parent-child delegation relationship and the > existing security check applies, it can only access its own class > loader in module mode. This remains an open issue what security > checks would be appropriate in module mode and how well it plays with > existing java security policy file etc. > > As for testing, I ran test/java/util/concurrent/atomic tests in hybrid > mode and they passed (in fact I verified all java/lang and java/util > tests). Since AtomicUpdaters is marked to run in othervm mode, I > manually converted it as a module and it passes. > > Thanks > Mandy > [1] > http://mail.openjdk.java.net/pipermail/jigsaw-dev/2012-May/002614.html From david.holmes at oracle.com Mon Jun 25 22:02:05 2012 From: david.holmes at oracle.com (David Holmes) Date: Tue, 26 Jun 2012 15:02:05 +1000 Subject: Review request for the incorrect check for "getClassLoader" permission In-Reply-To: <4FE8C624.8020209@oracle.com> References: <4FE36FC5.9020702@oracle.com> <4FE8C624.8020209@oracle.com> Message-ID: <4FE9424D.7050907@oracle.com> On 26/06/2012 6:12 AM, Mandy Chung wrote: > Alan, David, Paul, > > Given that j.u.c is pulled down from Doug's CVS into OpenJDK, I am > having a second thought to leave the j.u.c.atomic change out from this > patch. > > I have added a static needsPackalgeAccessCheck method in > sun.reflect.misc.ReflectUtil class that could be used by j.u.c.atomic > updater once it's integrated to jdk8. Since this j.u.c. change isn't > critical for jigsaw at this time, we'll check with Doug (probably with > David or Chris' help) and get appropriate change made in JSR166e source > that will subsequently get pulled down in JDK 8 and jigsaw. I think it is not unreasonable that our jigsaw sources and jsr166e sources will naturally diverge for a while. This shouldn't be seen as a "bad thing". We want to modularize as much as we can to gain experience with the new architecture for classloading and the access control aspects of that. Conversely Doug wants to continue with functional updates targeted for 8 without needing to work around a potentially fluctuating modularity API - and the need to work in modular and non-modular versions of 8. So my take is that we modify the Atomic classes as needed and worry about sync'ing with Doug at a much later date. David ----- > Updated webrev: > http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/getclassloader-permission-fix.01/ > > > Thanks > Mandy > > On 6/21/2012 12:02 PM, Mandy Chung wrote: >> David, Paul, >> >> I have a fix for the incorrect check w.r.t. "getClassLoader" >> permission [1] and also update j.u.c.atomic for module mode. >> >> Webrev at: >> http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/getclassloader-permission-fix/ >> >> >> The security.sh test demonstrates what can be accessed in module mode >> and lists the open issue. This patch is intended to fix the bug >> introduced in this changeset: >> http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/7b282c826118 >> >> Since there is no parent-child delegation relationship and the >> existing security check applies, it can only access its own class >> loader in module mode. This remains an open issue what security checks >> would be appropriate in module mode and how well it plays with >> existing java security policy file etc. >> >> As for testing, I ran test/java/util/concurrent/atomic tests in hybrid >> mode and they passed (in fact I verified all java/lang and java/util >> tests). Since AtomicUpdaters is marked to run in othervm mode, I >> manually converted it as a module and it passes. >> >> Thanks >> Mandy >> [1] >> http://mail.openjdk.java.net/pipermail/jigsaw-dev/2012-May/002614.html From paul.sandoz at oracle.com Mon Jun 25 22:07:17 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 26 Jun 2012 07:07:17 +0200 Subject: Review request for the incorrect check for "getClassLoader" permission In-Reply-To: <4FE8C624.8020209@oracle.com> References: <4FE36FC5.9020702@oracle.com> <4FE8C624.8020209@oracle.com> Message-ID: <0E8B5DB6-8E45-47A2-8DA4-7FD226391CF3@oracle.com> On Jun 25, 2012, at 10:12 PM, Mandy Chung wrote: > Alan, David, Paul, > > Given that j.u.c is pulled down from Doug's CVS into OpenJDK, I am having a second thought to leave the j.u.c.atomic change out from this patch. > > I have added a static needsPackalgeAccessCheck method in sun.reflect.misc.ReflectUtil class that could be used by j.u.c.atomic updater once it's integrated to jdk8. Since this j.u.c. change isn't critical for jigsaw at this time, we'll check with Doug (probably with David or Chris' help) and get appropriate change made in JSR166e source that will subsequently get pulled down in JDK 8 and jigsaw. > Good idea. > Updated webrev: > http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/getclassloader-permission-fix.01/ > Looks good to me. Paul. From jaroslav.tulach at oracle.com Mon Jun 25 23:42:58 2012 From: jaroslav.tulach at oracle.com (Jaroslav Tulach) Date: Tue, 26 Jun 2012 08:42:58 +0200 Subject: Provider dependencies later was: Unit testing the resolver In-Reply-To: <1CBB7F73-3D1D-479E-B3F4-3F7BD2312850@oracle.com> References: <67a9b3c3-98cd-4b74-a939-50d285523ed3@default> <3483956.tVrVunNkeV@logoutik> <1CBB7F73-3D1D-479E-B3F4-3F7BD2312850@oracle.com> Message-ID: <5167195.aJinfXxLY7@logoutik> Dne Po 25. ?ervna 2012 20:54:24, Paul Sandoz napsal(a): > No problem. Mostly what i am looking for is agreement that service provider > module dependencies should be resolved after the "Classpath" dependences > since it is: > > - easier to report errors; I'd bet this is true. > - easier for developers to understand; and Right, it follows the way I think about the resolution. > - has wider implications to help avoid, in part, NP-Complete problems. I have doubts about this statement in general. It goes in the right direction, but unless the whole system[1] is in place, I don't see NP-Completeness being prevented. Maybe P2 and Sat4J guys could jump in and share their experience with optimizing the NP-problem space... -jt [1] http://wiki.apidesign.org/wiki/JigsawServices From paul.sandoz at oracle.com Tue Jun 26 01:11:00 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 26 Jun 2012 10:11:00 +0200 Subject: Provider dependencies later was: Unit testing the resolver In-Reply-To: <5167195.aJinfXxLY7@logoutik> References: <67a9b3c3-98cd-4b74-a939-50d285523ed3@default> <3483956.tVrVunNkeV@logoutik> <1CBB7F73-3D1D-479E-B3F4-3F7BD2312850@oracle.com> <5167195.aJinfXxLY7@logoutik> Message-ID: <1349C441-3776-4189-BC54-D8E83722C99D@oracle.com> On Jun 26, 2012, at 8:42 AM, Jaroslav Tulach wrote: > Dne Po 25. ?ervna 2012 20:54:24, Paul Sandoz napsal(a): >> No problem. Mostly what i am looking for is agreement that service provider >> module dependencies should be resolved after the "Classpath" dependences >> since it is: >> >> - easier to report errors; > > I'd bet this is true. > >> - easier for developers to understand; and > > Right, it follows the way I think about the resolution. > >> - has wider implications to help avoid, in part, NP-Complete problems. > > I have doubts about this statement in general. It goes in the right direction, > but unless the whole system[1] is in place, I don't see NP-Completeness being > prevented. Thanks, i suspected as much, it's just a small piece of the larger puzzle. > Maybe P2 and Sat4J guys could jump in and share their experience > with optimizing the NP-problem space... > Yes, that would be most interesting. Paul. > -jt > > > [1] http://wiki.apidesign.org/wiki/JigsawServices From jaroslav.tulach at oracle.com Tue Jun 26 02:14:37 2012 From: jaroslav.tulach at oracle.com (Jaroslav Tulach) Date: Tue, 26 Jun 2012 11:14:37 +0200 Subject: Self-describing sources: the JavaC patch Message-ID: <10091150.uPWDTv3VrT@logoutik> Hello guys. We had a discussion with Paul recently trying to find a way to have a form of self-describing sources and not to compromise ''module-info.java'' purity. Here is a proposal http://wiki.netbeans.org/JavacOptionsInSourceFiles that modifies JavaC a bit and seems to fulfil many of the needs IDEs have[1]. What can stop us from changing JavaC to perform $ javac @source-info.cfg by default? Using this format for OpenJDK sources itself? Changing other CLI tools jar, jmod to extract information from the same configuration file? -jt [1] http://wiki.netbeans.org/Jigsaw From chris.hegarty at oracle.com Tue Jun 26 06:32:59 2012 From: chris.hegarty at oracle.com (chris.hegarty at oracle.com) Date: Tue, 26 Jun 2012 13:32:59 +0000 Subject: hg: jigsaw/jigsaw/bdb: Support building BDB on ppc Message-ID: <20120626133259.7D8FF47B1F@hg.openjdk.java.net> Changeset: 8ff6e91a0455 Author: dholmes Date: 2012-06-26 14:32 +0100 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/bdb/rev/8ff6e91a0455 Support building BDB on ppc Reviewed-by: chegar ! make/Makefile From jaroslav.tulach at oracle.com Tue Jun 26 06:42:48 2012 From: jaroslav.tulach at oracle.com (Jaroslav Tulach) Date: Tue, 26 Jun 2012 15:42:48 +0200 Subject: Offer default In-Reply-To: <18B4FDB3-78BA-4153-93EB-21862658B290@oracle.com> References: <18B4FDB3-78BA-4153-93EB-21862658B290@oracle.com> Message-ID: <23119567.ULkq6DqdqJ@logoutik> Hello Paul, > On Jun 22, 2012, at 6:59 PM, Jaroslav Tulach wrote: > > Paul wrote: > >> Every module declares what it needs with "requires ...". Perhaps it is a > >> lack of imagination on my part but i don't currently see how we can > >> strengthen that need in a decentralized environment. It is not when it > >> all comes together that such needs can be compared?> > > The way to eliminate NP-Completeness is to require a hint. So a JAXP > > module can either say: > > > > requires service optional javax.xml.parsers.DocumentBuilderFactory; > > > > and that is fine. Such definition does not need hints, as it is trivially > > satisfied with nothing, so the resolution algorithm is simple. On the > > other hand the JAXP module can require real implementation: > > > > requires service javax.xml.parsers.DocumentBuilderFactory default from > > com.oracle.apache.xerces at 1.8 > > > > By giving hint - e.g. by directly specifying that the default > > implementation of XML parser is known (at the time of compilation) to be > > provided by com.oracle.apache.xerces at 1.8 module, we can avoid > > NP-Completeness from the system. I can formalize the concept, if there is > > an interest. > Some questions: There are important questions! I can either reply from the point of view of someone trying to eliminate NP-Completeness, or from a point of view of user of such dependency system. > 1) when would the dependency "com.oracle.apache.xerces at 1.8" get resolved? > > 2) what if other service provider modules for > javax.xml.parsers.DocumentBuilderFactory are present in the library? The NP-Aware guy would suggest to follow the hint first. E.g. try the service that has been hinted - either by "default from" definition or by any other hint made by the assembler or deployer of the whole application. Only later fallback to trying all the services available and enabling those that can be enabled. This prevents the system to be trapped in insolvable situation due to trying some weird service provider first and then preventing all others to be found incompatible. > My attempt to answer: > > 1) it would be resolved as non-optional service provider module dependency > iff no other service provider modules are present > > 2) it would be resolved after the other service provider module > dependencies, thus would appear last when iterating over the service > instances. Possible. But this does not guarantee success. As I mentioned above, you may get to situation when for soem requested services a provider can't be enabled. If that happens you can either do the NP-Complete algorithm or throw away all your computation and try again following the hinted service at first. There is one thing that is not very nice on my proposal. It seems to me (a rigorous proof would clarify it) that when composing the transitive dependencies of a module, one needs to take into account the "default with". E.g. in the above case: requires service javax.xml.parsers.DocumentBuilderFactory default from com.oracle.apache.xerces at 1.8 When compiling a module against JAXP one would have to have the Xerces at 1.8 module around and the transitive dependencies of the new module would include not only all dependencies of JAXP, but also of Xerces at 1.8. Sounds strange, but it may not be as bad: #1 - when compiling against standard JavaSE at 1.8 the JAXP API as well as Xerces at 1.8 are already present. It is not restrictive at all. #2 - when compling against smaller Java (without Xerces at 1.8) one would need to have different hint. #2a - the module itself could use requires service javax.xml.parsers.DocumentBuilderFactory default from org.w3c.lightweightparser at 3.7 #2b - among the dependencies of the new module, there could be some which specifies the different default. E.g. JavaME at 1.8 could requires service javax.xml.parsers.DocumentBuilderFactory default from org.w3c.lightweightparser at 3.7 and then whoever compiles against JavaME would get the right one. So the "default from" restriction seems to be sort of acceptable in practise, imho. -jt From Alan.Bateman at oracle.com Tue Jun 26 14:24:18 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 26 Jun 2012 22:24:18 +0100 Subject: Preview builds of Jigsaw updated Message-ID: <4FEA2882.1010603@oracle.com> The download page on java.net to host preview builds has been updated: http://jdk8.java.net/jigsaw/ Mac OS X and Solaris 64-bit builds are still not there but that issue should get sorted out soon so that all platforms are available. -Alan. From Alan.Bateman at oracle.com Tue Jun 26 23:44:40 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 27 Jun 2012 07:44:40 +0100 Subject: Review request for the incorrect check for "getClassLoader" permission In-Reply-To: <4FE8C624.8020209@oracle.com> References: <4FE36FC5.9020702@oracle.com> <4FE8C624.8020209@oracle.com> Message-ID: <4FEAABD8.9060002@oracle.com> On 25/06/2012 21:12, Mandy Chung wrote: > Alan, David, Paul, > > Given that j.u.c is pulled down from Doug's CVS into OpenJDK, I am > having a second thought to leave the j.u.c.atomic change out from this > patch. > > I have added a static needsPackalgeAccessCheck method in > sun.reflect.misc.ReflectUtil class that could be used by j.u.c.atomic > updater once it's integrated to jdk8. Since this j.u.c. change isn't > critical for jigsaw at this time, we'll check with Doug (probably with > David or Chris' help) and get appropriate change made in JSR166e > source that will subsequently get pulled down in JDK 8 and jigsaw. > > Updated webrev: > > http://cr.openjdk.java.net/~mchung/jigsaw/webrevs/getclassloader-permission-fix.01/ The ReflectUtil addition looks good. I agree that the j.u.c. changes aren't really important here. I see David's reply suggesting that we could manage the difference so it may be best to have a complete fix rather than trying to remember the issues later. -Alan. From mandy.chung at oracle.com Wed Jun 27 03:15:53 2012 From: mandy.chung at oracle.com (Mandy Chung) Date: Wed, 27 Jun 2012 03:15:53 -0700 Subject: Review request for the incorrect check for "getClassLoader" permission In-Reply-To: <4FE9424D.7050907@oracle.com> References: <4FE36FC5.9020702@oracle.com> <4FE8C624.8020209@oracle.com> <4FE9424D.7050907@oracle.com> Message-ID: <4FEADD59.1050202@oracle.com> David, On 6/25/2012 10:02 PM, David Holmes wrote: > > I think it is not unreasonable that our jigsaw sources and jsr166e > sources will naturally diverge for a while. This shouldn't be seen as > a "bad thing". We want to modularize as much as we can to gain > experience with the new architecture for classloading and the access > control aspects of that. Conversely Doug wants to continue with > functional updates targeted for 8 without needing to work around a > potentially fluctuating modularity API - and the need to work in > modular and non-modular versions of 8. > > So my take is that we modify the Atomic classes as needed and worry > about sync'ing with Doug at a much later date. Sorry for the late reply as I took a day off yesterday. I don't see the jigsaw source and jsr166e diverging a bad thing. As we want to keep jdk changes minimal in jigsaw repo and integrate jdk change to jdk8 regularly, I was thinking of the steps to get the Atomic classes change into jigsaw. Since this is not time critical, it may be best to follow the normal way to get the change in jsr166e first and then pulled down to openjdk. The new needsPackageAccessCheck method in jigsaw repo will be the version to work in modular and non-modular versions. The jdk8 version will be slightly different and that will not call any jigsaw API (just check class loader == null). So I hope it shouldn't be an issue making such change in jsr166e. BTW with the new needsPackageAccessCheck method you can remove the isAncestor method in all 3 atomic updater classes. I'd like to push this changeset first and separate the j.u.c. change. If it's determined that jsr166e won't make such change for some reason, I can modify the Atomic classes and fix that in jigsaw repo. Mandy From david.holmes at oracle.com Wed Jun 27 04:26:23 2012 From: david.holmes at oracle.com (David Holmes) Date: Wed, 27 Jun 2012 21:26:23 +1000 Subject: Review request for the incorrect check for "getClassLoader" permission In-Reply-To: <4FEADD59.1050202@oracle.com> References: <4FE36FC5.9020702@oracle.com> <4FE8C624.8020209@oracle.com> <4FE9424D.7050907@oracle.com> <4FEADD59.1050202@oracle.com> Message-ID: <4FEAEDDF.9010303@oracle.com> On 27/06/2012 8:15 PM, Mandy Chung wrote: > David, > > On 6/25/2012 10:02 PM, David Holmes wrote: >> >> I think it is not unreasonable that our jigsaw sources and jsr166e >> sources will naturally diverge for a while. This shouldn't be seen as >> a "bad thing". We want to modularize as much as we can to gain >> experience with the new architecture for classloading and the access >> control aspects of that. Conversely Doug wants to continue with >> functional updates targeted for 8 without needing to work around a >> potentially fluctuating modularity API - and the need to work in >> modular and non-modular versions of 8. >> >> So my take is that we modify the Atomic classes as needed and worry >> about sync'ing with Doug at a much later date. > > Sorry for the late reply as I took a day off yesterday. I don't see the > jigsaw source and jsr166e diverging a bad thing. As we want to keep jdk > changes minimal in jigsaw repo and integrate jdk change to jdk8 > regularly, I was thinking of the steps to get the Atomic classes change > into jigsaw. Since this is not time critical, it may be best to follow > the normal way to get the change in jsr166e first and then pulled down > to openjdk. Feel free to proceed but I don't see jsr166e adopting anything module specific until well after module stuff becomes mainline. I would think we need to make modularization changes ahead of time to "bake" them and then feed them back to jsr166e. But this is new ground so ... David ------ > The new needsPackageAccessCheck method in jigsaw repo will be the > version to work in modular and non-modular versions. The jdk8 version > will be slightly different and that will not call any jigsaw API (just > check class loader == null). So I hope it shouldn't be an issue making > such change in jsr166e. BTW with the new needsPackageAccessCheck method > you can remove the isAncestor method in all 3 atomic updater classes. > > I'd like to push this changeset first and separate the j.u.c. change. If > it's determined that jsr166e won't make such change for some reason, I > can modify the Atomic classes and fix that in jigsaw repo. > > Mandy From paul.sandoz at oracle.com Wed Jun 27 07:52:33 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 27 Jun 2012 16:52:33 +0200 Subject: Service configuration/context Re: ServiceLoader.load* take 2 In-Reply-To: <7197ED2D-769C-493D-A73E-AB105375CAF9@oracle.com> References: <7197ED2D-769C-493D-A73E-AB105375CAF9@oracle.com> Message-ID: Hi, Here is a webrev of configuration changes for a more immutable approach to the contexts, just to show where this is heading: http://cr.openjdk.java.net/~psandoz/jigsaw/configuration/webrev/ The modular JDK builds OK and all jigsaw-related tests pass. BaseContext, Context and PathContext are now interfaces. ContextBuilder and PathContextBuilder build up state to create immutable instances of Context and PathContext respectively. Linker also contains a builder LinkerContextBuilder that creates an immutable instance of Context. ContextBuilder becomes ContextBuilderSetBuilder. There is no longer any need to check if the context instance is an instance of Context or LinkingContext, implementations of the method BaseContextBuilder.add(ModuleInfo mi) can encapsulate the different behavior. Thus LinkingContext is deleted. The general steps are the same as before: 1) resolve 2) construct the context *builders*, using ContextBuilderSetBuilder 3, 4) link local and remote supplies, then build contexts from the context builders Two awkward areas: a) The method on PathContext: Set remoteContexts(); requires that a second pass be performed after construction of PathContext to make the instance immutable. I believe things could be simplified if that method was Set remoteContexts() like on Context, then that method it could be moved to BaseContext. I left it alone for now since it may affect the java compiler (have not checked). b) Building contexts from context builders requires the map of context builder for module views is transformed into the corresponding map of context for module views. It's easy to do by maintaining a map of context for context builder, which anyway has to be done for a). However, it would be nice to remove this aspect if this information could be calculated by Configuration just from the contexts. I suspect it can given the way a stored configuration is loaded [*]. Next steps: - Make Configuration immutable using a corresponding ConfigurationBuilder. - Shift the state of services from Context to InstalledConfiguration (extends Configuration) - Ensure that the set of contexts retains a topological order as traversed by the resolver. This will ensure service instances can be iterated over in a consistent order. Paul. [*] I notice that ContextBuilderSetBuilder is adding aliases to the map of context for module view: 96 private void addContextForModuleView(CB cx, ModuleView mv) { 97 contextForModuleView.put(mv.id().name(), cx); 98 for (ModuleId alias : mv.aliases()) { 99 contextForModuleView.put(alias.name(), cx); 100 } 101 } but that information is never re-generated when a stored configuration is loaded in SimpleLibrary: 598 Context c = cx.build(); 599 assert c.name().equals(cxn); 600 cf.add(c); 601 602 for (ModuleId m: c.modules()) { 603 for (ModuleId v: c.views(m)) { 604 cf.put(v.name(), c); 605 } 606 } Bug? On Jun 20, 2012, at 3:06 PM, Paul Sandoz wrote: > On Jun 12, 2012, at 5:03 PM, Paul Sandoz wrote: >> - Modify the configuration so that service information is stored directly with the configuration rather than with each context (that is stored with the configuration). Currently service provider classes are looked up by iterating through all the contexts. This is not very efficient. The configuration is the place where to optimize. >> This requires some refactoring of org.openjdk.jigsaw.Configuration because for instances of Configuration (the configuration produced when compiling) such state related to services is not relevant. This is also a potential opportunity to improve the immutability of Configuration using a builder pattern for construction. > > > Some details on improving this area: > > - Configuration has two constructors. One is used by the Linker/PathLinker to pass in all the configuration state. The other, that takes a subset of the complete state, is used by SimpleLibrary when parsing the serialized configuration and constructing the rest of the state while parsing. The methods Configuration.add/put are solely there for the purpose of SimpleLibrary to mutate state on Configuration. A Configuration builder could be used for both purposes. > > - BaseContext.freeze. This transforms mutable state into immutable state, but only for state held by BaseContext and not for sub types (because it is not overridden). This is a sure sign that a builder pattern can be used to construct mutable state, then built into an immutable object. > > - On Context there is code such as: > > public final Map moduleForLocalClassMap() { > return Collections.unmodifiableMap(moduleForLocalClass); > } > > and see the more complex case for Context.services(), which will any way move to a subtype of Configuration. > > This class is really trying very hard to be immutable :-) > > - In ContextBuilder there is casting, from line 132: > > if (cx instanceof LinkingContext) { > ((LinkingContext) cx).addModule(mi); > } > if (cx instanceof Context) { > URI lp = res.locationForName.get(mi.id().name()); > if (lp != null) { > String s = lp.getScheme(); > if (s == null || !s.equals("file")) { > throw new AssertionError(s); > } > ((Context) cx).putLibraryPathForModule(mi.id(), new File(lp)); > } > } > > this is a sign that the abstraction constructing a context is broken. Notice that a ContextFactory > instance is passed to the constructor of ContextBuilder. ContextBuilder is really a ContextSetBuilder, and > ContextFactory is acting a limited builder with just the build() method implemented. > > - Linker.LinkedContext is an internal detail thus could be a special builder that creates instances of Context. > All the additional state held by Linker.LinkedContext is not relevant once the Configuration has been created, > as shown by the method on Linker to return the Configuration: > > static Configuration > run(Library lib, ContextSet cxs) > > - When i see LinkingContext i am inclined to think that there should be the following: > > interface Context // was class BaseContext > interface LinkingContext extends Context > abstract class AbstractContext implements Context > class InstallContext extends AbstractContext // was Context > class PathContext extends AbstractContext implements LinkingContext > class Linker.LinkerContext extends InstallContext implements LinkingContext // was Linker.Context > > class InstallConfiguration extends Configuration > -- service state belongs here not on InstallContext > > > Paul. From chris.hegarty at oracle.com Wed Jun 27 10:13:59 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 27 Jun 2012 18:13:59 +0100 Subject: Module file parse API In-Reply-To: <679B7267-55C7-4E1E-82BE-78CEDBBECD27@oracle.com> References: <4FD86E8C.3080208@oracle.com> <4FDF23C8.3070402@oracle.com> <679B7267-55C7-4E1E-82BE-78CEDBBECD27@oracle.com> Message-ID: <4FEB3F57.7060507@oracle.com> Paul, Here is an updated version of the parser API. It includes a validating and hash accumulating parser. Also, there are some issues around using the csize in the parser as the signer does not (yet) update this field. This is a bug, but it can't be easily fixed until we clean up the writer API ( phase 2 ), the file hash needs to be updated too. Anyway, best done in a separate commit. http://cr.openjdk.java.net/~chegar/jigsaw/moduleParser_webrev.01/webrev/ -Chris. On 18/06/2012 14:50, Paul Sandoz wrote: > Hi Chris, > > This is an excellent start. Looks good to me. > > -- > > Some potential things we discussed off list for further commits: > > - s/ModuleFileParser/ModuleFileReader. I can think of two alternative implementations; one that wraps a module installed into a library so the reader can be connected to a writer for extraction. Another that wraps a modular jar file so the same installer code can be reused, i am not sure exactly if that is possible but sounds nice in theory :-) > > - IIRC the complete size of the jmod file is encoded in the file itself, thus after the file header has been read we can wrap everything around a CountingInputStream. > > - An adapting implementation of ModuleFileReader which accumulates hashes (ignoring the hash for a section of the signature type) and validates hashes. It can be reused by the installer/extractor and the signer. > > Paul. > > > On Jun 18, 2012, at 2:49 PM, Chris Hegarty wrote: > >> Paul gave me some feedback off list. Updated the webrev, same location: >> >>> http://cr.openjdk.java.net/~chegar/jigsaw/moduleParser_webrev.00/webrev/ >> >> * ModuleFileParserException is now a RuntimeException >> * Fixed up javadoc example code >> * Some minor bug fixes >> * Method naming consistency >> >> I see this as step one in a phased approach to the reader/writer API's. There is still work to be done to clean up the Reader ( now really an Installed/Extractor ), the *Header types, and obviously the writer. >> >> What I would like is get this first set of changes in ( if others agree with the API ), and then address the above outstanding issues with further commits. >> >> Thanks, >> -Chris. >> >> On 13/06/2012 11:42, Chris Hegarty wrote: >>> Hi, >>> >>> A while back it was suggested that it may be useful to define a pull >>> parser type API for reading from a module file. I gives separation of >>> the parsing of the module file from the writing of it (either installing >>> into the module library, or extraction elsewhere ). >>> >>> I've prototyped such an API and updated the ModuleFile.Reader to use it. >>> The first four files in the webrev contain the API, its implementation, >>> and a test that demonstrates some sample usage ( extraction of one or >>> more classes, listing sections, verifying hashes ). The remainder of the >>> changes simply updates some of the tools for installing and handling >>> signed modules. >>> >>> http://cr.openjdk.java.net/~chegar/jigsaw/moduleParser_webrev.00/webrev/ >>> >>> Thanks, >>> -Chris. > From mandy.chung at oracle.com Wed Jun 27 18:13:48 2012 From: mandy.chung at oracle.com (mandy.chung at oracle.com) Date: Thu, 28 Jun 2012 01:13:48 +0000 Subject: hg: jigsaw/jigsaw/jdk: Implement a proper check to determine if getClassLoader permission is required Message-ID: <20120628011419.1E13947B7F@hg.openjdk.java.net> Changeset: 3ade388252f8 Author: mchung Date: 2012-06-27 18:12 -0700 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/3ade388252f8 Implement a proper check to determine if getClassLoader permission is required Reviewed-by: alanb, dholmes, psandoz ! src/share/classes/java/lang/Class.java ! src/share/classes/java/lang/ClassLoader.java ! src/share/classes/java/lang/Thread.java ! src/share/classes/sun/reflect/misc/ReflectUtil.java ! test/org/openjdk/jigsaw/security.sh From mandy.chung at oracle.com Thu Jun 28 00:37:21 2012 From: mandy.chung at oracle.com (Mandy Chung) Date: Thu, 28 Jun 2012 00:37:21 -0700 Subject: Service configuration/context Re: ServiceLoader.load* take 2 In-Reply-To: References: <7197ED2D-769C-493D-A73E-AB105375CAF9@oracle.com> Message-ID: <4FEC09B1.9040007@oracle.com> On 6/27/2012 7:52 AM, Paul Sandoz wrote: > [*] I notice that ContextBuilderSetBuilder is adding aliases to the map of context for module view: > > 96 private void addContextForModuleView(CB cx, ModuleView mv) { > 97 contextForModuleView.put(mv.id().name(), cx); > 98 for (ModuleId alias : mv.aliases()) { > 99 contextForModuleView.put(alias.name(), cx); > 100 } > 101 } > > but that information is never re-generated when a stored configuration is loaded in SimpleLibrary: > > 598 Context c = cx.build(); > 599 assert c.name().equals(cxn); > 600 cf.add(c); > 601 > 602 for (ModuleId m: c.modules()) { > 603 for (ModuleId v: c.views(m)) { > 604 cf.put(v.name(), c); > 605 } > 606 } > > Bug? In fact, the aliases information are not stored in the configuration. It's a bug - good catch. Currently the jdk implementation doesn't depend on it. This only affects ModuleClassLoader.isModulePresent("java.xml") if the parameter is an alias name. Mandy P.S. I'll try to look at your webrev and understand your proposed approach next week. From paul.sandoz at oracle.com Thu Jun 28 05:46:15 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 28 Jun 2012 14:46:15 +0200 Subject: Service configuration/context Re: ServiceLoader.load* take 2 In-Reply-To: <4FEC09B1.9040007@oracle.com> References: <7197ED2D-769C-493D-A73E-AB105375CAF9@oracle.com> <4FEC09B1.9040007@oracle.com> Message-ID: <6EE8C2BE-C47A-4F55-8887-A2E703E47101@oracle.com> On Jun 28, 2012, at 9:37 AM, Mandy Chung wrote: > On 6/27/2012 7:52 AM, Paul Sandoz wrote: >> [*] I notice that ContextBuilderSetBuilder is adding aliases to the map of context for module view: >> >> 96 private void addContextForModuleView(CB cx, ModuleView mv) { >> 97 contextForModuleView.put(mv.id().name(), cx); >> 98 for (ModuleId alias : mv.aliases()) { >> 99 contextForModuleView.put(alias.name(), cx); >> 100 } >> 101 } >> >> but that information is never re-generated when a stored configuration is loaded in SimpleLibrary: >> >> 598 Context c = cx.build(); >> 599 assert c.name().equals(cxn); >> 600 cf.add(c); >> 601 >> 602 for (ModuleId m: c.modules()) { >> 603 for (ModuleId v: c.views(m)) { >> 604 cf.put(v.name(), c); >> 605 } >> 606 } >> >> Bug? > > > In fact, the aliases information are not stored in the configuration. It's a bug - good catch. Currently the jdk implementation doesn't depend on it. This only affects ModuleClassLoader.isModulePresent("java.xml") if the parameter is an alias name. > OK, i just fixed it to support aliases, the simplest thing to do is ensure that aliases+views are returned by BaseContext.views(ModuleId ), thus they become stored as part of the configuration. I now see stuff like "java." in the dump config. I think this sort of confirms we can simplify Configuration to always calculate the map of context for module view on construction in one place. Thus there is no need to have explicit methods to update that map state or pass in that map state on construction and more importantly there is no need to trust that the map state is in sync with the contexts themselves. It will also simplify Context*Set and the process of building Context instances. > Mandy > P.S. I'll try to look at your webrev and understand your proposed approach next week. OK thanks, i expect by then i will have updated to also include the 2 next steps in the previous email. I am a bit nervous about size of the changes, it's tricky to unravel this piece by piece. Paul. From paul.sandoz at oracle.com Thu Jun 28 05:47:36 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 28 Jun 2012 14:47:36 +0200 Subject: Module file parse API In-Reply-To: <4FEB3F57.7060507@oracle.com> References: <4FD86E8C.3080208@oracle.com> <4FDF23C8.3070402@oracle.com> <679B7267-55C7-4E1E-82BE-78CEDBBECD27@oracle.com> <4FEB3F57.7060507@oracle.com> Message-ID: Hi Chris, On Jun 27, 2012, at 7:13 PM, Chris Hegarty wrote: > Paul, > > Here is an updated version of the parser API. > > It includes a validating and hash accumulating parser. Also, there are some issues around using the csize in the parser as the signer does not (yet) update this field. This is a bug, but it can't be easily fixed until we clean up the writer API ( phase 2 ), the file hash needs to be updated too. Anyway, best done in a separate commit. > > http://cr.openjdk.java.net/~chegar/jigsaw/moduleParser_webrev.01/webrev/ > Looks good to me. Paul. > -Chris. > > On 18/06/2012 14:50, Paul Sandoz wrote: >> Hi Chris, >> >> This is an excellent start. Looks good to me. >> >> -- >> >> Some potential things we discussed off list for further commits: >> >> - s/ModuleFileParser/ModuleFileReader. I can think of two alternative implementations; one that wraps a module installed into a library so the reader can be connected to a writer for extraction. Another that wraps a modular jar file so the same installer code can be reused, i am not sure exactly if that is possible but sounds nice in theory :-) >> >> - IIRC the complete size of the jmod file is encoded in the file itself, thus after the file header has been read we can wrap everything around a CountingInputStream. >> >> - An adapting implementation of ModuleFileReader which accumulates hashes (ignoring the hash for a section of the signature type) and validates hashes. It can be reused by the installer/extractor and the signer. >> >> Paul. >> >> >> On Jun 18, 2012, at 2:49 PM, Chris Hegarty wrote: >> >>> Paul gave me some feedback off list. Updated the webrev, same location: >>> >>>> http://cr.openjdk.java.net/~chegar/jigsaw/moduleParser_webrev.00/webrev/ >>> >>> * ModuleFileParserException is now a RuntimeException >>> * Fixed up javadoc example code >>> * Some minor bug fixes >>> * Method naming consistency >>> >>> I see this as step one in a phased approach to the reader/writer API's. There is still work to be done to clean up the Reader ( now really an Installed/Extractor ), the *Header types, and obviously the writer. >>> >>> What I would like is get this first set of changes in ( if others agree with the API ), and then address the above outstanding issues with further commits. >>> >>> Thanks, >>> -Chris. >>> >>> On 13/06/2012 11:42, Chris Hegarty wrote: >>>> Hi, >>>> >>>> A while back it was suggested that it may be useful to define a pull >>>> parser type API for reading from a module file. I gives separation of >>>> the parsing of the module file from the writing of it (either installing >>>> into the module library, or extraction elsewhere ). >>>> >>>> I've prototyped such an API and updated the ModuleFile.Reader to use it. >>>> The first four files in the webrev contain the API, its implementation, >>>> and a test that demonstrates some sample usage ( extraction of one or >>>> more classes, listing sections, verifying hashes ). The remainder of the >>>> changes simply updates some of the tools for installing and handling >>>> signed modules. >>>> >>>> http://cr.openjdk.java.net/~chegar/jigsaw/moduleParser_webrev.00/webrev/ >>>> >>>> Thanks, >>>> -Chris. >> From paul.sandoz at oracle.com Thu Jun 28 07:47:19 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 28 Jun 2012 16:47:19 +0200 Subject: Service configuration/context Re: ServiceLoader.load* take 2 In-Reply-To: <6EE8C2BE-C47A-4F55-8887-A2E703E47101@oracle.com> References: <7197ED2D-769C-493D-A73E-AB105375CAF9@oracle.com> <4FEC09B1.9040007@oracle.com> <6EE8C2BE-C47A-4F55-8887-A2E703E47101@oracle.com> Message-ID: <6D3640C7-4048-4B64-9EE2-75FDCE96184E@oracle.com> Hi Mandy, Here is an updated webrev that also fixes aliases and cleans up the map of context for module view/alias: http://cr.openjdk.java.net/~psandoz/jigsaw/configuration/webrev.1/ The configuration does not distinguish between a view and an alias (like ContextBuilderSetBuilder.contextForModuleView does not distinguish), so when you dump the config it will be listed as a view , i dunno if this a big deal, it's easy to fix it so there is a method BaseContext.aliases(ModuleId mid), and aliases are stored separately to views. Paul. On Jun 28, 2012, at 2:46 PM, Paul Sandoz wrote: > On Jun 28, 2012, at 9:37 AM, Mandy Chung wrote: >> On 6/27/2012 7:52 AM, Paul Sandoz wrote: >>> [*] I notice that ContextBuilderSetBuilder is adding aliases to the map of context for module view: >>> >>> 96 private void addContextForModuleView(CB cx, ModuleView mv) { >>> 97 contextForModuleView.put(mv.id().name(), cx); >>> 98 for (ModuleId alias : mv.aliases()) { >>> 99 contextForModuleView.put(alias.name(), cx); >>> 100 } >>> 101 } >>> >>> but that information is never re-generated when a stored configuration is loaded in SimpleLibrary: >>> >>> 598 Context c = cx.build(); >>> 599 assert c.name().equals(cxn); >>> 600 cf.add(c); >>> 601 >>> 602 for (ModuleId m: c.modules()) { >>> 603 for (ModuleId v: c.views(m)) { >>> 604 cf.put(v.name(), c); >>> 605 } >>> 606 } >>> >>> Bug? >> >> >> In fact, the aliases information are not stored in the configuration. It's a bug - good catch. Currently the jdk implementation doesn't depend on it. This only affects ModuleClassLoader.isModulePresent("java.xml") if the parameter is an alias name. >> > > OK, i just fixed it to support aliases, the simplest thing to do is ensure that aliases+views are returned by BaseContext.views(ModuleId ), thus they become stored as part of the configuration. I now see stuff like "java." in the dump config. > > I think this sort of confirms we can simplify Configuration to always calculate the map of context for module view on construction in one place. Thus there is no need to have explicit methods to update that map state or pass in that map state on construction and more importantly there is no need to trust that the map state is in sync with the contexts themselves. It will also simplify Context*Set and the process of building Context instances. > > >> Mandy >> P.S. I'll try to look at your webrev and understand your proposed approach next week. > > OK thanks, i expect by then i will have updated to also include the 2 next steps in the previous email. > > I am a bit nervous about size of the changes, it's tricky to unravel this piece by piece. > > Paul. From chris.hegarty at oracle.com Thu Jun 28 10:57:41 2012 From: chris.hegarty at oracle.com (chris.hegarty at oracle.com) Date: Thu, 28 Jun 2012 17:57:41 +0000 Subject: hg: jigsaw/jigsaw/jdk: Add module file parser API Message-ID: <20120628175754.25EF847B98@hg.openjdk.java.net> Changeset: d11cc59f4daf Author: chegar Date: 2012-06-28 18:05 +0100 URL: http://hg.openjdk.java.net/jigsaw/jigsaw/jdk/rev/d11cc59f4daf Add module file parser API Reviewed-by: psandoz, alanb, mullan ! make/java/java/FILES_java.gmk ! src/share/classes/org/openjdk/jigsaw/Library.java ! src/share/classes/org/openjdk/jigsaw/ModuleFile.java + src/share/classes/org/openjdk/jigsaw/ModuleFileParser.java + src/share/classes/org/openjdk/jigsaw/ModuleFileParserDelegate.java + src/share/classes/org/openjdk/jigsaw/ModuleFileParserException.java + src/share/classes/org/openjdk/jigsaw/ModuleFileParserImpl.java ! src/share/classes/org/openjdk/jigsaw/ModuleFileWriter.java ! src/share/classes/org/openjdk/jigsaw/SignedModule.java ! src/share/classes/org/openjdk/jigsaw/SimpleLibrary.java + src/share/classes/org/openjdk/jigsaw/ValidatingModuleFileParser.java + src/share/classes/org/openjdk/jigsaw/ValidatingModuleFileParserImpl.java ! src/share/classes/org/openjdk/jigsaw/cli/Librarian.java ! src/share/classes/org/openjdk/jigsaw/cli/Signer.java ! test/org/openjdk/jigsaw/cli/ModuleFormatTestLeftOverBytes.java From mandy.chung at oracle.com Thu Jun 28 18:33:08 2012 From: mandy.chung at oracle.com (Mandy Chung) Date: Thu, 28 Jun 2012 18:33:08 -0700 Subject: Preview builds of Jigsaw for all platforms available Message-ID: <4FED05D4.5060804@oracle.com> Alan Bateman wrote: > The download page on java.net to host preview builds has been updated: > > http://jdk8.java.net/jigsaw/ > > Mac OS X and Solaris 64-bit builds are still not there but that issue > should get sorted out soon so that all platforms are available. The preview jigsaw builds for ALL platforms including Mac OS X and Solaris 64-bit are now available at: http://jdk8.java.net/jigsaw/ Mandy From paul.sandoz at oracle.com Fri Jun 29 09:59:13 2012 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Fri, 29 Jun 2012 18:59:13 +0200 Subject: Service configuration/context Re: ServiceLoader.load* take 2 In-Reply-To: <6D3640C7-4048-4B64-9EE2-75FDCE96184E@oracle.com> References: <7197ED2D-769C-493D-A73E-AB105375CAF9@oracle.com> <4FEC09B1.9040007@oracle.com> <6EE8C2BE-C47A-4F55-8887-A2E703E47101@oracle.com> <6D3640C7-4048-4B64-9EE2-75FDCE96184E@oracle.com> Message-ID: Hi Mandy, Here is the webrev for the context/configuration refactoring and services: http://cr.openjdk.java.net/~psandoz/jigsaw/configuration/webrev.2/ I chose to retain the existing approach to storing service information in the stored configuration. There is no point storing that information separately from the context itself, since it would just result in duplication of context names. The InstalledConfiguration class creates an optimized data structure for iterating globally over service provider class names for a given service interface. I also took the opportunity to further simplify BaseContextBuilder and ContextBuilderSetBuilder: - BaseContextBuilder does not implement equals/hashCode; - ContextBuilderSetBuilder no longer needs to use an instance of IdentityHashSet to store context builder instances and rehash when creating the ContextBuilderSet; and - I retained the freeze approach in BaseContextBuilder as it seems handy to catch potential bugs if the linking process attempts to monkey around with the set of modules IDs of a context builder. There are potentially further improvements to better isolate the building of contexts by SimpleLibrary and Linker/PathLinker respectively, but i think at this point it is time to stop. FWIW i understand the core Jigsaw code much better now :-) Paul. On Jun 28, 2012, at 4:47 PM, Paul Sandoz wrote: > Hi Mandy, > > Here is an updated webrev that also fixes aliases and cleans up the map of context for module view/alias: > > http://cr.openjdk.java.net/~psandoz/jigsaw/configuration/webrev.1/ > > The configuration does not distinguish between a view and an alias (like ContextBuilderSetBuilder.contextForModuleView does not distinguish), so when you dump the config it will be listed as a view , i dunno if this a big deal, it's easy to fix it so there is a method BaseContext.aliases(ModuleId mid), and aliases are stored separately to views. > > Paul. > > > On Jun 28, 2012, at 2:46 PM, Paul Sandoz wrote: > >> On Jun 28, 2012, at 9:37 AM, Mandy Chung wrote: >>> On 6/27/2012 7:52 AM, Paul Sandoz wrote: >>>> [*] I notice that ContextBuilderSetBuilder is adding aliases to the map of context for module view: >>>> >>>> 96 private void addContextForModuleView(CB cx, ModuleView mv) { >>>> 97 contextForModuleView.put(mv.id().name(), cx); >>>> 98 for (ModuleId alias : mv.aliases()) { >>>> 99 contextForModuleView.put(alias.name(), cx); >>>> 100 } >>>> 101 } >>>> >>>> but that information is never re-generated when a stored configuration is loaded in SimpleLibrary: >>>> >>>> 598 Context c = cx.build(); >>>> 599 assert c.name().equals(cxn); >>>> 600 cf.add(c); >>>> 601 >>>> 602 for (ModuleId m: c.modules()) { >>>> 603 for (ModuleId v: c.views(m)) { >>>> 604 cf.put(v.name(), c); >>>> 605 } >>>> 606 } >>>> >>>> Bug? >>> >>> >>> In fact, the aliases information are not stored in the configuration. It's a bug - good catch. Currently the jdk implementation doesn't depend on it. This only affects ModuleClassLoader.isModulePresent("java.xml") if the parameter is an alias name. >>> >> >> OK, i just fixed it to support aliases, the simplest thing to do is ensure that aliases+views are returned by BaseContext.views(ModuleId ), thus they become stored as part of the configuration. I now see stuff like "java." in the dump config. >> >> I think this sort of confirms we can simplify Configuration to always calculate the map of context for module view on construction in one place. Thus there is no need to have explicit methods to update that map state or pass in that map state on construction and more importantly there is no need to trust that the map state is in sync with the contexts themselves. It will also simplify Context*Set and the process of building Context instances. >> >> >>> Mandy >>> P.S. I'll try to look at your webrev and understand your proposed approach next week. >> >> OK thanks, i expect by then i will have updated to also include the 2 next steps in the previous email. >> >> I am a bit nervous about size of the changes, it's tricky to unravel this piece by piece. >> >> Paul. >