Multi-threaded javac

[David Schlosnagle]

Hello friendly compiler-devs!

One of the projects I work on has a large number of Java files (around
25,000 total), and I've been thinking of ways to decrease the overall
compilation time, especially in the case of clean builds. We have
recently undergone an effort to modularize a previous monolithic code
base into smaller more maintainable modules, very similar to the
modularization work done for Jigsaw. This modularization does allow us
to build a proper module dependency graph and build some modules in
parallel; however, we still have several large modules (in the 2,000
to 7,000 compilation unit range) with a couple being a serial
bottleneck in the overall project compilation until we can refactor
these into smaller modules we're stuck with long compilations that
don't fully utilize available resources. Due to the dependency
structure, several of these modules can only be compiled one at a
time. I'm looking for ways to improve the throughput of javac,
especially on multi-core machines, possibly using finer grained
parallelism than what is currently offered by JSR-199 and
javax.tools.JavaCompiler.

I've read through Jonathan Gibbons' blog post "Towards a
multi-threaded javac" [1] and related enhancement requests [2] [3]
[4], and I was wondering if there is any additional work along these
lines targeted for JDK7. I saw some changes in the repository already
that seem to be related such as the changes to
com.sun.tools.javac.util.Log (6724071: refactor Log into a front end
and back end [5] [6] and 6720185: DiagnosticFormatter refactoring [7]
[8]) and com.sun.tools.javac.parser.JavacParser (6724551: Use Queues
instead of Lists to link compiler phases [9]); however, I don't see
the changes mentioned related to parallel parsing in the current JDK7
langtools repository or further work on making the class generation
asynchronous.

I've also read through the OpenJDK Compilation Overview [10] and gone
through a fair amount of com.sun.tools.javac.** code, and I'd like to
take a shot at trying to parallelize some of the process, but before I
spend too much time on this, is this something reasonable to work on
and submit a patch for review and possible inclusion? Are there any
known issues blocking progress on these changes to javac? Does anyone
have any advice or warnings before I start hacking on a patch? I'm
aware that many developers may not want to incur the CPU and memory
overhead this would entail, so I'd assume this functionality would not
be enabled by default. I was thinking an option similar to `make -j`
would be a reasonable starting point for defining the max number of
concurrent tasks or desired parallelism level. I'd love to hear
others' thoughts on the subject.

[1]: http://blogs.sun.com/jjg/entry/towards_a_multi_threaded_javac
[2]: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4229449
[3]: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6629150
[4]: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6713663
[5]: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6724071
[6]: http://hg.openjdk.java.net/jdk7/jdk7/langtools/rev/1cf29847eb6e
[7]: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6720185
[8]: http://hg.openjdk.java.net/jdk7/jdk7/langtools/rev/37470f5ea179
[9]: http://hg.openjdk.java.net/jdk7/jdk7/langtools/rev/62fcf8d73dc5
[10]: http://openjdk.java.net/groups/compiler/doc/compilation-overview/index.html

Thanks,
Dave