Faster incremental OpenJDK compilation

[Louis Wasserman]

(cc:+cushon at google.com, responsible for most of this work)

We do something like this at Google for all our Java compilations (in
production!) with https://github.com/google/turbine, and it is built into
our open-source Bazel build system.

When A depends on B, we compile B with normal javac in parallel with a
Turbine compilation of B, which emits a "header" jar file, which e.g. omits
method bodies, skipping type inference and most other things that make
javac expensive.  Turbine does some amount of canonicalization, but keeps
things like static constants that affect downstream compiles.  A then
compiles against B's headers as soon as they are available, which is
usually much faster than waiting for the full compiled jar.  Our build
system's native caching features only recompile A if B's header jar has
changed -- that's how we detect an "interface change," as you put it -- but
since the output is an actual jar that A can compile against, we get the
added benefit of concurrency in compiling A and B.  Turbine has evolved to
support the annotation processing API, so the approach works even for
complex builds.  Before Turbine, we postprocessed the compiled jar to get a
header jar and emitted that as a build graph node, which is more directly
isomorphic to the system you describe.

I don't know if the JDK would want to use Turbine directly, but we can
certainly confirm that the concept is sound.  The specific granularity of
per-file analysis you have in mind might require breaking the module down
into groups of files whose dependencies are acyclic, though.

On Wed, May 6, 2020 at 8:35 AM Jan Lahoda <jan.lahoda at oracle.com> wrote:

> Hi,
>
> Triggered by Magnus' recent e-mail on adjusting the location of the IDE
> files, I looked at possibilities to improve speed of incremental
> compilation using make. About 3 years ago, we have sped up incremental
> build by only rebuilding modules when API of modules they depend on
> changed. But the module which contains modified sources is always
> compiled in full. So, for changes in java.base, this change improved the
> incremental build time significantly (by not recompiling e.g.
> java.desktop), but it can still take many seconds to build java.base
> after a trivial change. So, this time, I am thinking of speeding up
> module builds by not rebuilding all the source if possible.
>
> What I am thinking of is a relatively simple approach: detect changed
> files in a module and check if their "interface" changed. If it did,
> recompile the whole module. If it didn't, only compile the modified
> files. As a consequence, a small change inside a method body should lead
> to a fast build. Changes outside of method bodies may trigger longer
> build, but my hope would be that these would be less common.
>
> So far, I unfortunately don't know how to efficiently do this as nicely
> as the API digests used for inter-module dependencies. The approach that
> seems might work is this: the Depend plugin hooks itself into javac
> internals, and filters the incoming files - it first parses the modified
> ones, and if it cannot find a significant change, it will throw away the
> unmodified files, and only compile the modified ones. (In principle, it
> could also do dependency analysis, if we at some point decide it is
> important.)
>
> For a simple "touch Object.java && make", the wall-clock time is less
> then 5s, which sounds interesting:
> ---
> $ touch src/java.base/share/classes/java/lang/Object.java && time make
> Building target 'default (exploded-image)' in configuration
> 'linux-x86_64-server-release'
> Compiling 3028 files for java.base
> Stopping sjavac server
> Finished building target 'default (exploded-image)' in configuration
> 'linux-x86_64-server-release'
>
> real    0m3,104s
> user    0m5,731s
> sys     0m1,086s
> ---
>
> My current prototype is in the jdk/sandbox repository, branch
> "jlahoda-depend-in-module":
> http://hg.openjdk.java.net/jdk/sandbox/shortlog/jlahoda-depend-in-module
>
> I wonder if this would sound interesting to developers working on base
> modules, like java.base.
>
> What do you think? Any ideas/feedback?
>
> Thanks,
>       Jan
>
>

-- 
Louis Wasserman (he/him)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.java.net/pipermail/compiler-dev/attachments/20200506/6130c3b6/attachment.htm>