The future of partial builds

[Fredrik Öhrström]

Oookay, long story, this was when we used jdk6 to build jdk7, same kind of problem could occur when using jdk7 to build jdk8:

jdk7javac used new features and classes in jdk7, for example:
  java/util/Objects.java java/util/ConcurrentModificationException.java java/io/File.java java/nio/file/*
  java/nio/channels/AsynchronousChannel.java java/nio/channels/AsynchronousFileChannel.java
  java/nio/channels/CompletionHandler.java java/nio/channels/SeekableByteChannel.java

However first a bootstrapjdk7javac (that can be run using jdk6) was compiled, which was easy since it only needed to avoid com/sun/tools/javac/nio/....

Then we want to use bootstrapjdk7javac to compile a full jdk7javac, at this moment, the classes above had to be found by the bootstrapjdk7javac. 
To avoid the risk of recursing into far to many jdk classes, we use GenStubs to create stripped down source files for the above classes.
Thus -sourcepath points to the generated stubs for the above classes.

Now the GenStubs tool itself depends a bit on jdk7javac! Really? Yes, really. Thus to compile the GenStubs tool
using jdk6javac you needed -sourcepath jdk7javacsource so that GenStubs could find the odd parts it needed.
(And yes, those parts did of course not refer to the new jdk7 classes above. :-) )

In particular, one of those needed parts was RESOURCE_VARIABLE inside javax.lang.model.element.ElementKind
which was new for jdk7.

However javax.lang.model.element.ElementKind was already available to jdk6javac since it is part of the tools.jar
that is used to run jdk6javac and depending on the timestamps inside the jdk6 rt.jar file, jdk6javac could pickup either
the new ElementKind that had RESOURCE_VARIABLE from the source path, or the old ElementKind from jdk6 rt.jar 
that did not.

Cleaning out the bootclasspath does not work here, since we need access to Object,String etc  from jdk6 to be
able to compile/run GenStubs using jdk6. So the only remaining option was to add -Xprefer:source.

But this is the only place where -Xprefer:source is necessary. In fact, for the moment it is not necessary, since
there is no real difference in dependencies between jdk7javac and jdk8javac, GenStubs is still exercised on
a dummy class, because we expect it to be used in the future.

With this in mind, I just committed fixes to the Makefiles that do not use -Xprefer:source anywhere, except for GenStubs.
The main jdk build, already used "-bootclasspath outputdir/classes". 

Other than that, corba/jaxp/jaxws and hotspot, can safely link against the boot jdk rt.jar, since they must not (can't)
use any new features. Thus no need for bootclasspath or -Xprefer:source here. Corba does however use classes
inside the tools.jar from the boot jdk. We could get rid of the dependency on the boot jdk tools.jar, if we compiled 
Corba as part of the big jdk build, or after the jdk build. Same thing with jaxp/jaxws/corba and the boot rt.jar.

//Fredrik

Here is Jims original mail that solved the first problem.
--8<------------------------------------------------------------------------------------------------
Well, this is the most fun I've had in a long time :-) At least partly because there is no regression in JDK 6u27 as we had feared.

One of the errors I got doing your compile command was an undefined symbol - RESOURCE_VARIABLE.  This was added to javac file
   javax/lang/model/element/ElementKind.java/ElementKind.java

in JDK 7, so it is in the JDK 7 version of ElementKind but is not in the JDK 6 version.  ElementKind.class is located in the rt.jar file of a JDK.   It as if when the 6u26 javac is used, the jdk 7 version of ElementKind is used whereas when the 6u27 javac is used, the jdk 6 version of Element Kind is used.


To resolve a ref to a type, javac searches (not counting extension dirs):
- bootclasspath  for .class
- sourcepath      for .java
- classpath        for either

Then javac looks at the result of the search:
> A successful type search may produce a class file, a source file, or both. Here is how javac handles each situation:
>    * Search produces a class file but no source file: javac uses the class file.
>    * Search produces a source file but no class file: javac compiles the source file and uses the resulting class file.
>    * Search produces both a source file and a class file: javac determines whether the class file is out of date. If the class file is out of date, javac recompiles the source file and uses the updated class file. Otherwise, javac just uses the class file.
> 
>      javac considers a class file out of date only if it is older than the source file.

So, in this case, javac finds
- jdk 6 ElementKind.class in the  rt.jar of the jdk
- jdk 7 ElementKind.java under the -sourcepath

And, javac looks at the creation times of the two to decide which to use.
You guessed it:

jar tfv  j:/re/jdk/6u26/latest/binaries/windows-i586/jre/lib/rt.jar  | fgrep ElementKind.class
 1984 Wed May 04 01:04:44 PDT 2011 javax/lang/model/element/ElementKind.class

jar tfv  j:/re/jdk/6u27/latest/binaries/windows-i586/jre/lib/rt.jar  | fgrep ElementKind.class
 1984 Tue Jul 19 01:19:08 PDT 2011 javax/lang/model/element/ElementKind.class

find src -name ElementKind.java -ls
2 -rwxr-xr-x 1 jjh None 3691 May 12 16:52 src/share/classes/javax/lang/model/element/ElementKind.java


So, javac compiles the jdk 7 version of ElementKind.java when javac from 6u26 is used, but not when 6u27 is used :-)

The solution (thanks to Jon) is to add  -Xprefer:source  to your javac command line.  I suppose you could touch all the files in your repo, or re-clone and reapply your patches, but that is ugly and would just leave the door open for this sort of shenanigan to re-occur.

- jjh
--8<------------------------------------------------------------------------------------------------