Improved build system

[Weijun Wang]

I just tried this on my Ubuntu 11.04 64-bit system:

1. Clone http://hg.openjdk.java.net/build-infra/jdk7
2. get_source.sh
3. Download ccache 3.1.6 source and build it
4. Download the 3 files in common/config and mv one of them
5. ./configure --with-boot-jdk=$J6
6. make

and soon see an error

Compiling 10 files in package com/sun/source/util
/space/repos/build-infra/jdk7/langtools/src/share/classes/com/sun/mirror/util/TypeVisitor.java:52: 
error: duplicate class: com.sun.mirror.util.TypeVisitor
public interface TypeVisitor {
        ^

-Max


On 09/30/2011 08:33 AM, Fredrik Öhrström wrote:
> I pushed the current state of the improved build system
> for the OpenJDK.
>
> It currently only builds successfully on 64bit GNU/Linux.
>
> The first thing you should do after you fetched the update.
>
> cd common/config
> wget http://cvs.savannah.gnu.org/viewvc/*checkout*/config/config/config.guess
> wget http://cvs.savannah.gnu.org/viewvc/*checkout*/config/config/config.sub
> wget http://www.opensource.apple.com/source/libdispatch/libdispatch-187.5/m4/pkg.m4?txt
> mv pkg.m4?txt pkg.m4
>
> The paperwork has not yet cleared on these files, but as soon as it does, I will
> commit them to the repository.
>
> Now run:
> ./configure
>
> This will setup the build. If you are missing something it complains
> and tries to give a helpful hint on what rpm/deb package to install.
>
> (For developers within Oracle, you can do:
>    wget ftp://buildtools.se.oracle.com/buildtools/openjdk/builddeps.conf
>    ./configure --with-builddeps-server=ftp://buildtools.se.oracle.com/buildtools/openjdk --with-builddeps-dir=/localhome/builddeps
>    you can change /localhome/builddeps to your own preference.
> )
>
> You can also run:
> ./configure --help
> to see the options if you need to set the paths exactly.
>
> If configure succeed, type make, or perhaps rather:
> make 2>&1 | ./common/bin/hide_important_warnings_from_javac.sh
>
> The jdk sources are unfortunately stained with deprecated and
> unchecked warnings. Previously these warnings were hidden
> in the voluminous output from the build process. But now
> the stand out, as they should. But since we are working on
> the build process, using the above filter, we can hide them.
>
> If that succeed you can run openjdk like this:
> ./build/linux-amd64-server-release/jdk/bin/java -version
>
> You can now do "make images" this will create the jdk/jre
> images in ./build/linux-amd64-server-release/images
>
> And you can do "make install" which will try to install
> into your prefix of choice, typically /usr/local/jvm/openjdk-1.7.0-internal
> and create links from /usr/local/bin
>
> Now, assuming that your machine has a lot of RAM and a lot of CPUs,
> the build will move be reasonably fast. So what has happened.
>
> The new build system offers:
>    1) fast dependency tracking from class/obj to source
>       a re-make after the first successful make does nothing,
>       and it takes a short time for make to figure this out.
>       It does not need to start a jvm. Pure makefiles deal with
>       both C-code and Java-code.
>
>    2) incremental build, when you change a java source file.
>       Only the necessary package is recompiled
>       and the resulting jar is incrementally updated with new,changed
>       or deleted classes.
>
>    3) working dependency propagation from java source to
>       java class to jar to javah-headers to c-library.
>       E.g. if you add a new native method
>       in java/util/zip/CRC32.java and type make.
>       It will recompile the class files in the package.
>       Regenerate the javah-headers and recompile CRC32.c
>       then relink libzip.so.
>
>    4) working dependency propagation between java packages.
>       If you add a public void foo() method to
>       com/sun/naming/internal/ResourceManager.java and remake.
>       It will detect that the public api of that package had changed
>       and also recompile: javax/naming/ldap since the latter package
>       imports features from the first package. This dependency tracking
>       is currently grouped on packages, lest the meta data would be too large.
>
>       Yes, it is true. Add public void foo() to Object.java, and watch
>       it recompile the whole jdk. Well, at least the jdk repository, it cant
>       do anything about the already compiled repos lt,corba,jaxp,jaxws and hotspot.
>
>    5) It uses all available cores by default, if you have enough RAM.
>       It uses a background javac server to avoid throwing away optimizations
>       made by the JVM running the javac server.
>
> I have not changed the hotspot makefiles. Only the minimum necessary to make
> use of ccache and allow configure to the the compiler properly.
>
> The long pause when the hotspot makefiles tries to figure out that there is
> nothing to be done, is a good reminder that the hotspot makefiles need to be
> cleaned up as well. :-)
>
> To support these features, javac had to be improved. I added the options:
> -XDserver..     spawns a background javac server, or reuses the existing one
> -XDdeps..       generate a file with package dependencies (like -MF in gcc)
> -XDpubapi...    generate a file with the public api of the package
> -XDnativeapi..  generate a file with the native api of the package
>
> Since javac is inherently serial, the multi core problem was solved
> by creating (inside the javac server) one JavaCompiler for each core.
> When a request for compilation reach the javac server, it is distributed
> randomly to any of the JavaCompilers. By making sure that the
> JavaCompiler is reusable between client calls and any compiled classes
> are available to the next client call. As we continue to improve the
> sharing of data between JavaCompilers in the javac server, ie essentially
> making javac more and more parallell, we can reduce the number of time
> a class is compiled (now some classes get compiled as many
> times as you have cores).
>
> If we look at the bad boy, of the repositories, the corba repo.
> It had 144 makefiles, now it has one (1).
> It took 2 minutes and 20 seconds to compile on my workstation,
> now it takes 15 seconds.
>
> This improvement was of course possible since its makefiles were
> not good at all. For langtools,jaxp and jaxws, the new makefiles are
> faster than the ant makefiles, but not that much. This is of course
> due to the fact that the javac server still does not share much
> of the data between the JavaCompilers. Thus we should be moving
> towards getting almost linear speedup from the cores, since we
> have soo many packages that can be distributed on the different cores.
>
> In the jdk, we have not yet had time to convert all the makefiles.
> But several makefiles have been removed already.
>
> As you can see in the jdk/make/Makefile, there is a call to the legacy
> makefile system. Thus there are still many calls to javac, the legacy way.
> And then this is followed by a compile of all the classes again, using
> the new system. Thus there is a lot of speedups left to do.
>
> Unfortunately almost every jdk makefile has some quirk in it.
> It is sometimes platform specific code that is part of src/share/classes.
> It is sometimes generated code that should be ignored.
> It sometimes compile classes in the jdk using -target 6 to be run but
> not be part of the final jdk. etc etc.
>
> So to convert a jdk makefile, you need to figure out what java files
> does it create and move java source creation to GenerateJavaSources.gmk
> and into a suitable Gensrc....gmk file, then the actual classes are
> compiled already in CompileJavaClasses.gmk. If the makefile also generates
> a c-library you have to update CompileNativeLibraries.gmk.
>
> (Notice, how I cleverly changed the pronoun from I/we to you in
> the previous paragraph? :-) )
>
> On windows, you need to install cygwin and Visual Studio 10.
> ./configure will actively go looking for VS10 and setup the compiler
> correctly for the build. Yay! Then, the build probably does not work
> right now, for other reasons, most likely you have to increase
> the timeout in JavacServer.java to wait longer before it shuts down.
> This is probably necessary when building on slower computers. We will
> have to figure out a better way to decide when to shutdown the javac server.
> But we have successfully compiled langtools/corba/jaxp/jaxws and hotspot on windows.
>
> For solaris, there should be no significant hurdles either, just a lot of
> quirks to deal with. Again langtools/corba/jaxp/jaxws and hotspot have
> been compiled on Solaris.
>
> The ./configure runs on MacOSX with xcode installed. But fails quite early.
> Well, no wonder, since neither the bsd-port nor the MacOSX native port is
> part of build-infra, yet.
>
> I am very grateful for the enormous work made by the icedtea team!
> Your configure script was very helpful! I tried to name options in the
> same way as you did.
>
> Thanks to Erik,Kelly,Magnus,Maurizio and Robert for their extremely valuable
> work and support!
>
> Many bugs fixed, many more bugs left to fix!
>
> //Fredrik