bootstrapping javac and JDK 7

[Jonathan Gibbons]

Although I'm not in a position to push any bits yet, I've just reached a 
notable milestone for being able to compile langtools against (selected) 
JDK7 API.

Here's the problem we are dealing with:  we have to be able to compile 
(most of) langtools using the boot JDK. (Currently, JDK 6)   This is 
because we will be using the langtools javac to compile the rest of the 
JDK 7 source code, which means the we don't have a JDK 7 binary to run 
on or even compile against at this point.  Until recently, the boot JDK 
was sufficient for all of langtools, and life was (relatively) simple.

However, now there are new features in JDK 7 we would like to be able to 
use if the compiler is running on JDK 7. So far, there are two such 
features: new NIO, and the module system. My blog entry [1] describes 
how javac can provide a file manager based on the new NIO API. This 
means that a JSR 199 client can wrap any new NIO FileSystem with a 
JavaFileManager for use by javac and related tools.  And separately, 
javac must also be able to interact with the module system when it is 
available. (i.e. when javac is actually running on a fully built JDK 7.)

So, how do we get to build javac such that it can use JDK 7 API, which 
for the most part should be allowed to use JDK 7 language features?

For a start, we have to restrict the use of JDK 7 features within javac 
as much as possible.  That means limiting the use of JDK 7 features to a 
restricted set of javac classes, such that javac can function 
sufficiently well when those classes are not available (i.e. during 
bootstrapping.) For all the other classes, they can be built as before, 
using the boot JDK. The problem is how to compile the remaining classes 
that depend on JDK 7 API.

There are two solutions, each with their respective advantages and 
disadvantages.

The first approach is to restructure the JDK build, so that we can build 
the bootstrap compiler first, then the other repos, like corba, jaxp, 
jaxws, and jdk, and then go back to langtools a second time to build the 
remaining classes, now that we have class files for the rest of JDK that 
we can compile against. This approach is conceptually simple, but would 
be complicated to achieve in practice.

The second approach is to retain the structure of the current JDK build, 
and to compile javac against the source files of the as-yet-uncompiled 
JDK. Here's how this works: at least in theory.   Use the boot JDK to 
compile a bootstrap compiler that can run on the boot JDK. This 
bootstrap compiler must not use any JDK 7 features.  Even though the 
bootstrap compiler must run on the boot JDK, it will be able to read and 
compile JDK 7 code. So we can use it to compile a new copy of javac, 
including those parts that depend on JDK 7 API.  To do that we need to 
put the JDK 7 source code on the source path for the compilation.   Or 
so you would think.   There are two major problems.  The first is that 
when javac reads files from the source path, it will end up trying to 
compile the transitive closure of all the dependencies of those files. 
So even though javac may initially depend on just the new NIO API and 
the module system API, you eventually end up trying to compile files in 
AWT (really!) because of the dependencies that exist. As bad as that is, 
the second problem is worse. At the time we want to compile javac, not 
all of the JDK source code is available.  Some may be platform specific, 
and some will need to be generated automatically by rules inside the JDK 
makefiles that have yet to run.

The problem is that by placing the JDK source tree on the source path, 
we end up having to analyze the implementation details of those classes, 
and from there to all their dependencies. The solution is not to use the 
source files directly, but to compile against stubs files that contain 
the publicly visible details of the class, as needed for the 
compilation.  It would be a maintenance nightmare to create and manually 
maintain the necessary stub files, so I have written a utility which 
strips out enough of the details of the classes we need, such they can 
used to compile langtools. We don't need stub files for all of JDK, just 
for those parts of JDK which we need to be able to use from javac, plus, 
of course, enough of their dependencies to be able to compile the stubs 
themselves.

As a result, langtools can now be built in the following ways:

-- if you don't supply a path for JDK 7 classes, it will build langtools 
excluding any classes that depend on JDK 7 API.  This is done 
automatically to build a bootstrap compiler.

-- if you supply a path for the main JDK 7 source code (i.e. 
jdk/src/share/classes), it will create those stubs necessary to build 
langtools, and then build all of langtools.  This would be the normal 
mode used as part of a full JDK build. The list of stubs is specified 
explicitly, but maintaining a list of the files for which stubs are 
required is a lot simpler than maintaining the stubs themselves.

-- if you supply a path for a previously built version of JDK (i.e. 
jre/lib/rt.jar), it will simply compile langtools in conjunction with 
that jar file. This saves having to create and compile the stub files, 
and is for the convenience of langtools developers who don't need to do 
a full JDK build using the boot JDK.



--------------------
[1] http://blogs.sun.com/jjg/entry/jsr_199_meets_jsr_203