-Xbootclasspath/p: (or similar) please

[Jonathan Gibbons]

Reinier,

That's an interesting and different aspect of the [problem. You'll only 
get the problem on a Mac if you use the current Mac JVM as the baseline. 
The problems I'm putting forward are for when the baseline is JDK7, so 
we won't see these problems on a Mac until a JDK7 Mac port is available, 
and then, hopefully, it'll support Jigsaw, and have classes loaded the 
same way as on other Jigsaw impls.

The -L switch should allow the user to (create and) use a new module 
library instead of the standard system module library. That can be the 
basis of the "module version of bootclasspath".  Then, the problems I've 
been describing boil down to being able to create a new library to use. 
Mandy also alluded to the minor issue of making sure tools like javac 
can support -J-L so that you can set -L for the JVM running tools 
started by the launcher.  The problem with creating a new library is 
that currently the rules to build a library are buried in some 
complicated makefiles and are not currently capable of incremental 
building.  I suspect the way forward is to figure out how to solve that 
problem -- by making it possible to update a JDK library without running 
the full set of makefiles.

Note that apps running on top of JDK should not have all the same 
problems we're seeing with JDK right now.   JDK has its own special 
problems caused by the complexity of modularizing the existing JDK code 
and by the need to be able to bootstrap-build the JDK itself.  The 
bootstrap problem means that the classes for all the modules are 
currently still in a single src/*/classes hierarchy, and the build uses 
custom config files and class file dependency analysis to figure out 
which classes go in which module and which modules depend on other 
modules.   Hopefully, apps running on top of JDK should not have these 
same problems.

-- Jon




Reinier Zwitserloot wrote:
> On apple JVMs, tools.jar is part of the *boot* classpath, and as a result
> you get the system javac. In projects where you want your own copy of javac,
> for example because you need specific features from javac7 but you don't
> want to demand a jvm7, then the only way to solve the problem is to add this
> javac.jar to the bootclasspath. Adding it to the plain classpath doesn't
> help because the system javac, as it is part of the boot classpath, wins out
> over it.
>
> This and a gazillion other situations require -Xbootclasspath(/p) to work
> around the issues.
>
> I get that 'classpath' isn't really a concept that remains intact in a
> module world, but there is an obvious analogy, isn't there? "Load these
> modules before any others". Also, I take it there's some sort of ordering
> that occurs when 2 or more modules all say they provide the same service.
> The analogy to bootclasspath for many many cases where its used today would
> be to say: *THIS* module should be used to satisfy a requirement for "a.b.c"
> in preference to whatever the system module registry has.
>
>
>
> --Reinier Zwitserloot
>
>
>
> On Sat, Mar 27, 2010 at 8:05 PM, Jonathan Gibbons
> <Jonathan.Gibbons at sun.com>wrote:
>
>   
>> Mandy Chung wrote:
>>
>>     
>>> Jonathan Gibbons wrote:
>>>
>>>       
>>>> What are the limitations of the bootclasspath options in the jigsaw
>>>> module images?
>>>>
>>>>
>>>>         
>>> bootclasspath continues to work as it is today in legacy mode.
>>>  bootclasspath is not supported in module mode.
>>>
>>>       
>>>> "java -X" says they cannot be used when executing a module.  Does that
>>>> also apply when the module is implicit, as in tools invoked by the launcher,
>>>> like javac, javadoc, etc.
>>>>
>>>>         
>>> I converted most of the jdk tools to run in module mode and
>>> -J-Xbootclasspath can no longer be passed to those tools running in module
>>> mode.
>>>
>>> If a tool remains in legacy mode, it will continue to accept
>>> -J-Xbootclasspath option.
>>>
>>>
>>>       
>>>> If so, can I put it a BIG request that -Xbootclasspath/p: or an
>>>> equivalent should be supported.  I think it is CRITICALLY important for JDK
>>>> developers to have a way of substituting new classes for existing ones when
>>>> working on bug fixes and similar work.
>>>>
>>>>         
>>> Completely agree.
>>>
>>>       
>> How about if I suggest that all Jigsaw developers have to fix one or more
>> bugs in jdk-module-image to share the pain ;-)
>>
>>
>>
>>   In the jdk/ repo I know that folk tend to do a partial build and execute
>>     
>>>> out of the "build/classes/" directory.  This will not work in the modules
>>>> world.
>>>>
>>>>         
>>> We talked about implementing the "zero mod" facility (currently in javac)
>>> in the Java launcher so that we can say something like:
>>>
>>>  $ java -L $OUTPUTDIR/modules MyUnitTest
>>>
>>>       
>> We need another name here.  "ZeroMod" in javac refers to something else. It
>> is a private minimal alternate module system provided by javac for
>> bootstrapping and for testing javac functionality independently of any
>> external module system, such as Jigsaw. In time, ZeroMod should never be
>> used outside the langtools/ repository.
>> Separate from ZeroMod, javac can handle -L, -Xbootclasspath etc, and can
>> handle "module directories", meaning directory hierarchies of classes that
>> have not been converted into "proper modules" with jmod.  This all works in
>> conjunction with the Jigsaw module system, and does not require the ZeroMod
>> module system.
>>
>> As a historical note, I had intended/expected ZeroMod to be used to
>> bootstrap compile parts of the jdk/ repo up to the point where the Jigsaw
>> module system was available. In practice, you have worked around the
>> bootstrapping problem a different way, by compiling most of the jdk classes
>> the same old way as ever (without any module info) and then compiling the
>> **/module-info files separately, as a much later step. This allows us to use
>> the Jigsaw code to handle these **/module-info files, at a cost of providing
>> and using javac support for -Xbootclasspath.
>>
>>
>>     
>>>  In the langtools/ repo, we routinely compile just the langtools classes
>>>       
>>>> and execute them with a existing JDK using -Xbootclasspath/p.   The
>>>> limitation on -Xbootclasspath means that this too will not work in the
>>>> modules world.
>>>>
>>>>  I would imagine that we could do something like this for langtools:
>>>>         
>>>  $ java -L langtools/dist/modules JavacTest
>>>
>>> I haven't put much thought about this though and there are possibly issues
>>> to be resolved.  Hmm.... javac and other jdk tools take -J<argument> and
>>> pass the argument to the launcher invoking the tool.  I don't think we can
>>> prepend  "-L <module-lib>" with -J.
>>>
>>>       
>> While -L might be the basis of a direction to take, note that we would
>> still need some effective way of incrementally updating a module library.
>> Currently all the make rules for building modules and for knowing what goes
>> where are all embodied in jdk/make/modules/ which is all-or-nothing and not
>> incremental-friendly at all.  Perhaps it would help if one of the
>> by-products of a build was data in a form that could be given to a tool like
>> jmod to say, "build me a new module Foo with these classes...".
>>
>>
>>     
>>>  Development will be severely impacted if we have to build new modules
>>>       
>>>> images every time we want to try out a fix.
>>>>
>>>> From a testing point of view, I know that most of the langtools tests
>>>> exercise functionality that is independent of whether JDK is running in
>>>> legacy mode or modules mode. But, currently 31 tests fail in modules mode.
>>>> These tests do not fail in legacy mode.  This means the fixes for these
>>>> tests will need to be tested on a full modules build.  Starting from
>>>> scratch, a full modules build takes me about an hour, whereas building
>>>> langtools and using -Xbootclasspath typically takes under half a minute.
>>>> (That's a factor of 100 or more faster.)
>>>>
>>>>
>>>>         
>>> My bad.  I ought to speed up the modules build to help our development.
>>>  It's on my list to work on.  Mark also has a patch to replace cpio with
>>> rsync to improve the subsequent modules build time.
>>>
>>>       
>> Having a faster build would be nice, but it will not be an effective
>> solution to the problem. As an analogy, look how JDK developers tend to
>> start off with a full build, then when they are fixing a bug, they go into a
>> particular make/XYZ directory,  rebuild a few classes, then run tests using
>> build/PLATFORM_ARCH/classes, without rebuilding images, because building
>> images is just too slow and inconvenient. That technique will no longer work
>> -- you'll have to do an modules build because there is no other way to
>> update the modules that you want to test. There is no way of saying "I've
>> rebuilt java.lang.Integer: please incrementally rebuild whatever modules it
>> lives in".   Even if I know what module it gives in, there is no way I can
>> use the build to say "just remake the XYZ module".  It's even worse for
>> langtools developers (!!) because the langtools classes live in one
>> repository and the make files for modules live in another repository!  And
>> no, that is not a suggestion that langtools/ should merge back into jdk/ !!!
>>
>> Back before the days of OpenJDK, there was a way of doing the incremental
>> development by making a shadow copy of a JDK image using symbolic links, and
>> replacing/adding in new classes as necessary. (And who remembers the
>> eponymous "gafter builds"!) When we went to OpenJDK, we put effort into
>> using -Xbootclasspath/p: and making that work as a slicker more convenient
>> way of testing incremental builds.
>>
>> I guess I can see a way going forward of hacks of the form ... here's a
>> full copy of JDK, here's a bunch of classes, please update the classes in
>> the Foo module with any that you find in my classes directory.
>>
>>
>>
>>     
>>>  In javac, it proved necessary to implement -Xbootclasspath options even
>>>       
>>>> in module mode (Note, I mean the javac options here, not the underlying JVM
>>>> options). In practice, it was reasonably easy to augment the search for
>>>> platform classes with the extra locations being provided by -Xbootclasspath,
>>>> and -Xbootclasspath/p: in particular.  Surely it is possible to do something
>>>> similar in the JVM. It might be idealistic to say we should not to subvert
>>>> the classes provided by the module system, but pragmatically, there are
>>>> times when it will be necessary.
>>>>
>>>>
>>>>         
>>> I would hope that there is a way to address this requirement in the
>>> modules world while not needed to keep bootclasspath :)
>>>
>>> Mandy
>>>
>>>
>>>       
>> Yes. In fact, we have reached a notable milestone to be having this
>> discussion. Up to now, we've been focussed on building the tools and
>> infrastructure necessary to modularize the JDK. And we've achieved that!
>> You can get a copy of the jigsaw repo, go "make modules", and a while and a
>> few (ahem) cups of coffee later, you can have your very own
>> jdk-module-image. Woohoo! No lack of congratulations to the team on that
>> aspect.
>>
>> But, cynic that I am, this JDK is not perfect :-( and we need to be able to
>> work in this new modular world, so that by the time we deliver JDK7 it is
>> somewhat closer to perfection. In particular, we need to be able to fix bugs
>> that only show up this modular world.  Thus it doesn't matter that I can use
>> -Xbootclasspath in a legacy mode. That's not what needs fixing.
>>
>> And *all* the old techniques have gone out the window.  jdk/ developers
>> can't use build/PLATFORM_ARCH/classes to avoid building images. langtools/
>> developers can't use "gafter builds", or -Xbootclasspath/p:
>>
>> We need a way of testing incremental updates. It needn't be specifically
>> -Xbootclasspath/p: but that does provide an excellent model of what we
>> should strive to achieve -- it's light weight, it's non-destructive, it
>> doesn't require me to make a complete copy of JDK into which I can edit my
>> classes.
>>
>> -- Jon
>>
>>
>>
>>