Improved build system

[Fredrik Öhrström]

I pushed the current state of the improved build system 
for the OpenJDK.

It currently only builds successfully on 64bit GNU/Linux.

The first thing you should do after you fetched the update.

cd common/config 
wget http://cvs.savannah.gnu.org/viewvc/*checkout*/config/config/config.guess
wget http://cvs.savannah.gnu.org/viewvc/*checkout*/config/config/config.sub
wget http://www.opensource.apple.com/source/libdispatch/libdispatch-187.5/m4/pkg.m4?txt
mv pkg.m4?txt pkg.m4

The paperwork has not yet cleared on these files, but as soon as it does, I will
commit them to the repository.

Now run:
./configure

This will setup the build. If you are missing something it complains
and tries to give a helpful hint on what rpm/deb package to install.

(For developers within Oracle, you can do:
  wget ftp://buildtools.se.oracle.com/buildtools/openjdk/builddeps.conf
  ./configure --with-builddeps-server=ftp://buildtools.se.oracle.com/buildtools/openjdk --with-builddeps-dir=/localhome/builddeps
  you can change /localhome/builddeps to your own preference.
)

You can also run:
./configure --help
to see the options if you need to set the paths exactly.

If configure succeed, type make, or perhaps rather:
make 2>&1 | ./common/bin/hide_important_warnings_from_javac.sh

The jdk sources are unfortunately stained with deprecated and
unchecked warnings. Previously these warnings were hidden
in the voluminous output from the build process. But now
the stand out, as they should. But since we are working on
the build process, using the above filter, we can hide them.

If that succeed you can run openjdk like this:
./build/linux-amd64-server-release/jdk/bin/java -version

You can now do "make images" this will create the jdk/jre
images in ./build/linux-amd64-server-release/images

And you can do "make install" which will try to install
into your prefix of choice, typically /usr/local/jvm/openjdk-1.7.0-internal
and create links from /usr/local/bin

Now, assuming that your machine has a lot of RAM and a lot of CPUs,
the build will move be reasonably fast. So what has happened.

The new build system offers:
  1) fast dependency tracking from class/obj to source
     a re-make after the first successful make does nothing,
     and it takes a short time for make to figure this out.
     It does not need to start a jvm. Pure makefiles deal with
     both C-code and Java-code.

  2) incremental build, when you change a java source file. 
     Only the necessary package is recompiled
     and the resulting jar is incrementally updated with new,changed
     or deleted classes.

  3) working dependency propagation from java source to 
     java class to jar to javah-headers to c-library.
     E.g. if you add a new native method
     in java/util/zip/CRC32.java and type make.
     It will recompile the class files in the package.
     Regenerate the javah-headers and recompile CRC32.c
     then relink libzip.so.

  4) working dependency propagation between java packages.
     If you add a public void foo() method to 
     com/sun/naming/internal/ResourceManager.java and remake.
     It will detect that the public api of that package had changed
     and also recompile: javax/naming/ldap since the latter package
     imports features from the first package. This dependency tracking
     is currently grouped on packages, lest the meta data would be too large.

     Yes, it is true. Add public void foo() to Object.java, and watch
     it recompile the whole jdk. Well, at least the jdk repository, it cant
     do anything about the already compiled repos lt,corba,jaxp,jaxws and hotspot.

  5) It uses all available cores by default, if you have enough RAM.
     It uses a background javac server to avoid throwing away optimizations
     made by the JVM running the javac server.

I have not changed the hotspot makefiles. Only the minimum necessary to make
use of ccache and allow configure to the the compiler properly.

The long pause when the hotspot makefiles tries to figure out that there is
nothing to be done, is a good reminder that the hotspot makefiles need to be
cleaned up as well. :-) 

To support these features, javac had to be improved. I added the options:
-XDserver..     spawns a background javac server, or reuses the existing one  
-XDdeps..       generate a file with package dependencies (like -MF in gcc)
-XDpubapi...    generate a file with the public api of the package
-XDnativeapi..  generate a file with the native api of the package

Since javac is inherently serial, the multi core problem was solved
by creating (inside the javac server) one JavaCompiler for each core.
When a request for compilation reach the javac server, it is distributed
randomly to any of the JavaCompilers. By making sure that the
JavaCompiler is reusable between client calls and any compiled classes
are available to the next client call. As we continue to improve the 
sharing of data between JavaCompilers in the javac server, ie essentially
making javac more and more parallell, we can reduce the number of time
a class is compiled (now some classes get compiled as many
times as you have cores). 

If we look at the bad boy, of the repositories, the corba repo.
It had 144 makefiles, now it has one (1).
It took 2 minutes and 20 seconds to compile on my workstation,
now it takes 15 seconds.

This improvement was of course possible since its makefiles were
not good at all. For langtools,jaxp and jaxws, the new makefiles are 
faster than the ant makefiles, but not that much. This is of course
due to the fact that the javac server still does not share much
of the data between the JavaCompilers. Thus we should be moving
towards getting almost linear speedup from the cores, since we
have soo many packages that can be distributed on the different cores.

In the jdk, we have not yet had time to convert all the makefiles.
But several makefiles have been removed already. 

As you can see in the jdk/make/Makefile, there is a call to the legacy 
makefile system. Thus there are still many calls to javac, the legacy way.
And then this is followed by a compile of all the classes again, using
the new system. Thus there is a lot of speedups left to do.

Unfortunately almost every jdk makefile has some quirk in it. 
It is sometimes platform specific code that is part of src/share/classes.
It is sometimes generated code that should be ignored.
It sometimes compile classes in the jdk using -target 6 to be run but
not be part of the final jdk. etc etc.

So to convert a jdk makefile, you need to figure out what java files
does it create and move java source creation to GenerateJavaSources.gmk
and into a suitable Gensrc....gmk file, then the actual classes are
compiled already in CompileJavaClasses.gmk. If the makefile also generates
a c-library you have to update CompileNativeLibraries.gmk.

(Notice, how I cleverly changed the pronoun from I/we to you in
the previous paragraph? :-) )

On windows, you need to install cygwin and Visual Studio 10.
./configure will actively go looking for VS10 and setup the compiler
correctly for the build. Yay! Then, the build probably does not work
right now, for other reasons, most likely you have to increase
the timeout in JavacServer.java to wait longer before it shuts down. 
This is probably necessary when building on slower computers. We will
have to figure out a better way to decide when to shutdown the javac server.
But we have successfully compiled langtools/corba/jaxp/jaxws and hotspot on windows. 

For solaris, there should be no significant hurdles either, just a lot of
quirks to deal with. Again langtools/corba/jaxp/jaxws and hotspot have
been compiled on Solaris.

The ./configure runs on MacOSX with xcode installed. But fails quite early.
Well, no wonder, since neither the bsd-port nor the MacOSX native port is
part of build-infra, yet.

I am very grateful for the enormous work made by the icedtea team! 
Your configure script was very helpful! I tried to name options in the 
same way as you did. 

Thanks to Erik,Kelly,Magnus,Maurizio and Robert for their extremely valuable 
work and support!

Many bugs fixed, many more bugs left to fix! 

//Fredrik