The future of partial builds

[Kelly O'Hair]

Some stats on incremental builds. Not partial builds..

This is an older Solaris machine svc6.us.oracle.com, building the complete openjdk forest from scratch
for 64bit including images took less than 14 minutes (parallel build setting was 8) and images took 2mins of this 14mins:

-- Build times ----------
Target 
Start 2012-09-12 10:56:17
End   2012-09-12 11:09:55
00:00:30 corba
00:04:51 hotspot
00:00:23 jaxp
00:00:33 jaxws
00:04:43 jdk
00:01:59 jdk-images
00:00:39 langtools
00:13:38 TOTAL
-------------------------

A repeated 'gmake images" with no changes took 31 seconds:

-- Build times ----------
Target 
Start 2012-09-12 11:13:30
End   2012-09-12 11:14:01
00:00:02 corba
00:00:07 hotspot
00:00:02 jaxp
00:00:04 jaxws
00:00:08 jdk
00:00:07 jdk-images
00:00:01 langtools
00:00:31 TOTAL
-------------------------

I touched a few awt java sources  (touch jdk/src/share/classes/java/awt/*.java) and did just a 'gmake all' (no images)
and it took 2 minutes (this will improve significantly with smartjavac):

-- Build times ----------
Target 
Start 2012-09-12 11:17:34
End   2012-09-12 11:19:32
00:00:00 corba
00:00:00 hotspot
00:00:00 jaxp
00:00:01 jaxws
00:01:56 jdk
00:00:01 langtools
00:01:58 TOTAL
-------------------------

I touched a few awt native sources (touch jdk/src/share/native/sun/awt/*/*.c) and did the 'gmake all' (no images)
and it took 37 seconds):

-- Build times ----------
Target 
Start 2012-09-12 11:22:24
End   2012-09-12 11:23:01
00:00:00 corba
00:00:00 hotspot
00:00:00 jaxp
00:00:01 jaxws
00:00:35 jdk
00:00:01 langtools
00:00:37 TOTAL
-------------------------

So as far as incremental builds go, I think this is a vast improvement. And the incremental build should be accurate.

I'm not taking a position on partial builds here, just providing some hard data on full openjdk builds with the
new build-infra makefiles.

The above builds are complete openjdk builds, with no import jdk needed, and no concerns about
sync issues between jdk components (hotspot<->jdk).

-kto


On Sep 11, 2012, at 6:37 AM, Anthony Petrov wrote:

> Magnus,
> 
> You've only explained how incremental builds could work for Java classes in the new build-infra. What about incremental builds of native code? E.g. in AWT we often do the following:
> 
> $ cd make/sun/awt (or make/java/awt, or make/sun/lwawt)
> $ make
> 
> And this re-builds both AWT classes and AWT native code (and some 2D stuff too, btw). What has build-infra got to offer for incremental builds of native code?
> 
> Also, how fast are those incremental solutions (both the temporary, and the expected final one) in build-infra? Performing the above commands with the current build system takes literally just a few seconds on any platform (given you've built the JDK repo once before, and it might took like 10-30 minutes for the first build).
> 
> --
> best regards,
> Anthony
> 
> On 9/10/2012 6:00 PM, Magnus Ihse Bursie wrote:
>> On 2012-09-10 14:13, Alan Bateman wrote:
>>> When you say "sub-directory builds" then I think you mean incremental builds, or "poor-man increment builds" as I call it. I think the majority of people working in the jdk repository, at least in Oracle, do this because they know the area and know which make files that re-build the files that they have changed. Incremental builds are generally not very reliable but seem to be "good enough" for most people needs, at least those that aren't doing significant refactoring. The main motive of course is to be able to re-build in a few seconds and the #1 goal of any new build system has to be fast incremental builds.
>> In the new build system, fast incremental builds of Java code is dependent on the new "smart javac", which unfortunately has still not proven stable enough to be enabled by default, even in the experimental build-infra forest. It is still our hope that it will turn out to be good enough to be used as default, but I don't want to trust hope alone.
>> As an interim solution, I have just created the following two shortcuts. Both of these bypass the proper dependency detection. This means that make will cut to the chase and recompile faster, but it also means that you need to take over the responsibility of making sure dependencies are correct. This is very similar to the "*-only" (e.g. hotspot-only) targets that are current available in the new build system.
>> First shortcut is to skip parts of the JDK makefile. By typing e.g. "make jdk JDK_TARGET=classes", the JDK build will stop after building the "classes" target (which compiles the majority of the JDK java files). An even faster, but less safe, version is "make jdk-only JDK_TARGET=classes-only", which will jump directly to the JDK Makefile, and only execute the "classes" target. In this way, absolutely no time is spent before starting to compile Java classes.
>> Second shortcut is to limit the classes that are recompiled by the JDK "classes" target. Currently, *all* (more or less) Java classes in the JDK is recompiled at once. While this speeds up things considerably at a first time compilation, it makes all subsequent builds take the same time. By specifying e.g. JDK_FILTER=java/nio, only files in the java/nio package will get compiled. This is, I think, similar (but not strictly not identical) to the effect of "sub-directory builds". It is possible to use an even more fine-grained filter, separated by comma, e.g. JDK_FILTER=java/nio/channels,java/nio/charset -- the smaller number of Java files your filter includes, the faster the build will go (more or less).
>> These two can of course be combined.
>> I would like to point out that this is hopefully a temporary solution -- if proper dependency checking and incremental builds get fast enough, this kind of trickery should not be needed. If this happens, we will remove the support for these shortcuts since they do carry a risk of improper builds.
>> /Magnus