The future of partial builds

[Chris Hegarty]

On 11/09/12 14:37, Anthony Petrov wrote:
> Magnus,
>
> You've only explained how incremental builds could work for Java classes
> in the new build-infra. What about incremental builds of native code?
> E.g. in AWT we often do the following:
>
> $ cd make/sun/awt (or make/java/awt, or make/sun/lwawt)
> $ make

We do the same for java/net and sun/net.

This greatly improves productivity when working platform specific issues 
( and the only hardware configured to build is not lightning fast, I'm 
thinking of older sparc machines for example ).

-Chris.

>
> And this re-builds both AWT classes and AWT native code (and some 2D
> stuff too, btw). What has build-infra got to offer for incremental
> builds of native code?
>
> Also, how fast are those incremental solutions (both the temporary, and
> the expected final one) in build-infra? Performing the above commands
> with the current build system takes literally just a few seconds on any
> platform (given you've built the JDK repo once before, and it might took
> like 10-30 minutes for the first build).
>
> --
> best regards,
> Anthony
>
> On 9/10/2012 6:00 PM, Magnus Ihse Bursie wrote:
>> On 2012-09-10 14:13, Alan Bateman wrote:
>>> When you say "sub-directory builds" then I think you mean incremental
>>> builds, or "poor-man increment builds" as I call it. I think the
>>> majority of people working in the jdk repository, at least in Oracle,
>>> do this because they know the area and know which make files that
>>> re-build the files that they have changed. Incremental builds are
>>> generally not very reliable but seem to be "good enough" for most
>>> people needs, at least those that aren't doing significant
>>> refactoring. The main motive of course is to be able to re-build in a
>>> few seconds and the #1 goal of any new build system has to be fast
>>> incremental builds.
>>
>> In the new build system, fast incremental builds of Java code is
>> dependent on the new "smart javac", which unfortunately has still not
>> proven stable enough to be enabled by default, even in the
>> experimental build-infra forest. It is still our hope that it will
>> turn out to be good enough to be used as default, but I don't want to
>> trust hope alone.
>>
>> As an interim solution, I have just created the following two
>> shortcuts. Both of these bypass the proper dependency detection. This
>> means that make will cut to the chase and recompile faster, but it
>> also means that you need to take over the responsibility of making
>> sure dependencies are correct. This is very similar to the "*-only"
>> (e.g. hotspot-only) targets that are current available in the new
>> build system.
>>
>> First shortcut is to skip parts of the JDK makefile. By typing e.g.
>> "make jdk JDK_TARGET=classes", the JDK build will stop after building
>> the "classes" target (which compiles the majority of the JDK java
>> files). An even faster, but less safe, version is "make jdk-only
>> JDK_TARGET=classes-only", which will jump directly to the JDK
>> Makefile, and only execute the "classes" target. In this way,
>> absolutely no time is spent before starting to compile Java classes.
>>
>> Second shortcut is to limit the classes that are recompiled by the JDK
>> "classes" target. Currently, *all* (more or less) Java classes in the
>> JDK is recompiled at once. While this speeds up things considerably at
>> a first time compilation, it makes all subsequent builds take the same
>> time. By specifying e.g. JDK_FILTER=java/nio, only files in the
>> java/nio package will get compiled. This is, I think, similar (but not
>> strictly not identical) to the effect of "sub-directory builds". It is
>> possible to use an even more fine-grained filter, separated by comma,
>> e.g. JDK_FILTER=java/nio/channels,java/nio/charset -- the smaller
>> number of Java files your filter includes, the faster the build will
>> go (more or less).
>>
>> These two can of course be combined.
>>
>> I would like to point out that this is hopefully a temporary solution
>> -- if proper dependency checking and incremental builds get fast
>> enough, this kind of trickery should not be needed. If this happens,
>> we will remove the support for these shortcuts since they do carry a
>> risk of improper builds.
>>
>> /Magnus