Faster incremental OpenJDK compilation

[Jan Lahoda]

Hi Jon,

Good question. I was first experimenting with hashing Elements, as we do 
currently for the module API hashes, but it was too slow to enter the 
sources (although it is not completely impossible). So I did a new 
hashing based on the AST - basically, it is a TreeScanner sending Tree 
kinds, names and sub-trees into a MessageDigest. It ignores things like 
method bodies, class initializers, and private class members. Does not 
currently skip imports, although that may be seen as too conservative. 
But tweaks to the AST hashing are surely still needed, so we can tweak 
the exact meaning of a "significant change".

Jan

On 06. 05. 20 18:07, Jonathan Gibbons wrote:
> Jan,
> 
> This seems like an interesting approach.
> 
> How are you determining "significant change"?  I could imagine trying to 
> do this by looking at the changed lines, to see if they only within 
> method bodies and comments (for example), or by doing some sort of 
> lexical hash on the signatures, assuming that folk aren't messing with 
> imports to change the resolution of simple type names in the signature.
> 
> -- Jon
> 
> On 5/6/20 8:30 AM, Jan Lahoda wrote:
>> Hi,
>>
>> Triggered by Magnus' recent e-mail on adjusting the location of the 
>> IDE files, I looked at possibilities to improve speed of incremental 
>> compilation using make. About 3 years ago, we have sped up incremental 
>> build by only rebuilding modules when API of modules they depend on 
>> changed. But the module which contains modified sources is always 
>> compiled in full. So, for changes in java.base, this change improved 
>> the incremental build time significantly (by not recompiling e.g. 
>> java.desktop), but it can still take many seconds to build java.base 
>> after a trivial change. So, this time, I am thinking of speeding up 
>> module builds by not rebuilding all the source if possible.
>>
>> What I am thinking of is a relatively simple approach: detect changed 
>> files in a module and check if their "interface" changed. If it did, 
>> recompile the whole module. If it didn't, only compile the modified 
>> files. As a consequence, a small change inside a method body should 
>> lead to a fast build. Changes outside of method bodies may trigger 
>> longer build, but my hope would be that these would be less common.
>>
>> So far, I unfortunately don't know how to efficiently do this as 
>> nicely as the API digests used for inter-module dependencies. The 
>> approach that seems might work is this: the Depend plugin hooks itself 
>> into javac internals, and filters the incoming files - it first parses 
>> the modified ones, and if it cannot find a significant change, it will 
>> throw away the unmodified files, and only compile the modified ones. 
>> (In principle, it could also do dependency analysis, if we at some 
>> point decide it is important.)
>>
>> For a simple "touch Object.java && make", the wall-clock time is less 
>> then 5s, which sounds interesting:
>> ---
>> $ touch src/java.base/share/classes/java/lang/Object.java && time make
>> Building target 'default (exploded-image)' in configuration 
>> 'linux-x86_64-server-release'
>> Compiling 3028 files for java.base
>> Stopping sjavac server
>> Finished building target 'default (exploded-image)' in configuration 
>> 'linux-x86_64-server-release'
>>
>> real    0m3,104s
>> user    0m5,731s
>> sys     0m1,086s
>> ---
>>
>> My current prototype is in the jdk/sandbox repository, branch 
>> "jlahoda-depend-in-module":
>> http://hg.openjdk.java.net/jdk/sandbox/shortlog/jlahoda-depend-in-module
>>
>> I wonder if this would sound interesting to developers working on base 
>> modules, like java.base.
>>
>> What do you think? Any ideas/feedback?
>>
>> Thanks,
>>      Jan
>>