-Xbootclasspath/p: (or similar) please

[Jonathan Gibbons]

Mandy Chung wrote:
> Jonathan Gibbons wrote:
>> What are the limitations of the bootclasspath options in the jigsaw 
>> module images?
>>
>
> bootclasspath continues to work as it is today in legacy mode.  
> bootclasspath is not supported in module mode.
>> "java -X" says they cannot be used when executing a module.  Does 
>> that also apply when the module is implicit, as in tools invoked by 
>> the launcher, like javac, javadoc, etc.
>
> I converted most of the jdk tools to run in module mode and 
> -J-Xbootclasspath can no longer be passed to those tools running in 
> module mode.
>
> If a tool remains in legacy mode, it will continue to accept 
> -J-Xbootclasspath option.
>
>>
>> If so, can I put it a BIG request that -Xbootclasspath/p: or an 
>> equivalent should be supported.  I think it is CRITICALLY important 
>> for JDK developers to have a way of substituting new classes for 
>> existing ones when working on bug fixes and similar work.
>
> Completely agree.

How about if I suggest that all Jigsaw developers have to fix one or 
more bugs in jdk-module-image to share the pain ;-)


>>   In the jdk/ repo I know that folk tend to do a partial build and 
>> execute out of the "build/classes/" directory.  This will not work in 
>> the modules world. 
>
> We talked about implementing the "zero mod" facility (currently in 
> javac) in the Java launcher so that we can say something like:
>
>   $ java -L $OUTPUTDIR/modules MyUnitTest

We need another name here.  "ZeroMod" in javac refers to something else. 
It is a private minimal alternate module system provided by javac for 
bootstrapping and for testing javac functionality independently of any 
external module system, such as Jigsaw. In time, ZeroMod should never be 
used outside the langtools/ repository.   

Separate from ZeroMod, javac can handle -L, -Xbootclasspath etc, and can 
handle "module directories", meaning directory hierarchies of classes 
that have not been converted into "proper modules" with jmod.  This all 
works in conjunction with the Jigsaw module system, and does not require 
the ZeroMod module system.

As a historical note, I had intended/expected ZeroMod to be used to 
bootstrap compile parts of the jdk/ repo up to the point where the 
Jigsaw module system was available. In practice, you have worked around 
the bootstrapping problem a different way, by compiling most of the jdk 
classes the same old way as ever (without any module info) and then 
compiling the **/module-info files separately, as a much later step. 
This allows us to use the Jigsaw code to handle these **/module-info 
files, at a cost of providing and using javac support for 
-Xbootclasspath.  

>
>> In the langtools/ repo, we routinely compile just the langtools 
>> classes and execute them with a existing JDK using 
>> -Xbootclasspath/p.   The limitation on -Xbootclasspath means that 
>> this too will not work in the modules world.
>>
> I would imagine that we could do something like this for langtools:
>
>   $ java -L langtools/dist/modules JavacTest
>
> I haven't put much thought about this though and there are possibly 
> issues to be resolved.  Hmm.... javac and other jdk tools take 
> -J<argument> and pass the argument to the launcher invoking the tool.  
> I don't think we can prepend  "-L <module-lib>" with -J.

While -L might be the basis of a direction to take, note that we would 
still need some effective way of incrementally updating a module 
library. Currently all the make rules for building modules and for 
knowing what goes where are all embodied in jdk/make/modules/ which is 
all-or-nothing and not incremental-friendly at all.  Perhaps it would 
help if one of the by-products of a build was data in a form that could 
be given to a tool like jmod to say, "build me a new module Foo with 
these classes...".  

>
>> Development will be severely impacted if we have to build new modules 
>> images every time we want to try out a fix.
>>
>> From a testing point of view, I know that most of the langtools tests 
>> exercise functionality that is independent of whether JDK is running 
>> in legacy mode or modules mode. But, currently 31 tests fail in 
>> modules mode. These tests do not fail in legacy mode.  This means the 
>> fixes for these tests will need to be tested on a full modules 
>> build.  Starting from scratch, a full modules build takes me about an 
>> hour, whereas building langtools and using -Xbootclasspath typically 
>> takes under half a minute. (That's a factor of 100 or more faster.)
>>
>
> My bad.  I ought to speed up the modules build to help our 
> development.  It's on my list to work on.  Mark also has a patch to 
> replace cpio with rsync to improve the subsequent modules build time.

Having a faster build would be nice, but it will not be an effective 
solution to the problem. As an analogy, look how JDK developers tend to 
start off with a full build, then when they are fixing a bug, they go 
into a particular make/XYZ directory,  rebuild a few classes, then run 
tests using build/PLATFORM_ARCH/classes, without rebuilding images, 
because building images is just too slow and inconvenient. That 
technique will no longer work -- you'll have to do an modules build 
because there is no other way to update the modules that you want to 
test. There is no way of saying "I've rebuilt java.lang.Integer: please 
incrementally rebuild whatever modules it lives in".   Even if I know 
what module it gives in, there is no way I can use the build to say 
"just remake the XYZ module".  It's even worse for langtools developers 
(!!) because the langtools classes live in one repository and the make 
files for modules live in another repository!  And no, that is not a 
suggestion that langtools/ should merge back into jdk/ !!!

Back before the days of OpenJDK, there was a way of doing the 
incremental development by making a shadow copy of a JDK image using 
symbolic links, and replacing/adding in new classes as necessary. (And 
who remembers the eponymous "gafter builds"!) When we went to OpenJDK, 
we put effort into using -Xbootclasspath/p: and making that work as a 
slicker more convenient way of testing incremental builds.

I guess I can see a way going forward of hacks of the form ... here's a 
full copy of JDK, here's a bunch of classes, please update the classes 
in the Foo module with any that you find in my classes directory.

>
>> In javac, it proved necessary to implement -Xbootclasspath options 
>> even in module mode (Note, I mean the javac options here, not the 
>> underlying JVM options). In practice, it was reasonably easy to 
>> augment the search for platform classes with the extra locations 
>> being provided by -Xbootclasspath, and -Xbootclasspath/p: in 
>> particular.  Surely it is possible to do something similar in the 
>> JVM. It might be idealistic to say we should not to subvert the 
>> classes provided by the module system, but pragmatically, there are 
>> times when it will be necessary.
>>
>
> I would hope that there is a way to address this requirement in the 
> modules world while not needed to keep bootclasspath :)
>
> Mandy
>

Yes. In fact, we have reached a notable milestone to be having this 
discussion. Up to now, we've been focussed on building the tools and 
infrastructure necessary to modularize the JDK. And we've achieved 
that!   You can get a copy of the jigsaw repo, go "make modules", and a 
while and a few (ahem) cups of coffee later, you can have your very own 
jdk-module-image. Woohoo! No lack of congratulations to the team on that 
aspect.

But, cynic that I am, this JDK is not perfect :-( and we need to be able 
to work in this new modular world, so that by the time we deliver JDK7 
it is somewhat closer to perfection. In particular, we need to be able 
to fix bugs that only show up this modular world.  Thus it doesn't 
matter that I can use -Xbootclasspath in a legacy mode. That's not what 
needs fixing.

And *all* the old techniques have gone out the window.  jdk/ developers 
can't use build/PLATFORM_ARCH/classes to avoid building images. 
langtools/ developers can't use "gafter builds", or -Xbootclasspath/p:

We need a way of testing incremental updates. It needn't be specifically 
-Xbootclasspath/p: but that does provide an excellent model of what we 
should strive to achieve -- it's light weight, it's non-destructive, it 
doesn't require me to make a complete copy of JDK into which I can edit 
my classes.

-- Jon