Faster incremental OpenJDK compilation

[Jan Lahoda]

On 06. 05. 20 19:23, Jonathan Gibbons wrote:
> You could refine the imports by filtering them according to whether they 
> are 'star-imports' or the simple name matches a simple name that has 
> been seen by the TreeScanner.  In other words, try and filter out 
> imports that are definitely not relevant.

Good idea. I've tried to do that, and also fixed some bugs in the AST 
hashing, and enhanced tests.

All is still on "jlahoda-depend-in-module" branch in jdk/sandbox.

Jan

> 
> -- Jon
> 
> 
> On 5/6/20 9:49 AM, Jan Lahoda wrote:
>> Hi Jon,
>>
>> Good question. I was first experimenting with hashing Elements, as we 
>> do currently for the module API hashes, but it was too slow to enter 
>> the sources (although it is not completely impossible). So I did a new 
>> hashing based on the AST - basically, it is a TreeScanner sending Tree 
>> kinds, names and sub-trees into a MessageDigest. It ignores things 
>> like method bodies, class initializers, and private class members. 
>> Does not currently skip imports, although that may be seen as too 
>> conservative. But tweaks to the AST hashing are surely still needed, 
>> so we can tweak the exact meaning of a "significant change".
>>
>> Jan
>>
>> On 06. 05. 20 18:07, Jonathan Gibbons wrote:
>>> Jan,
>>>
>>> This seems like an interesting approach.
>>>
>>> How are you determining "significant change"?  I could imagine trying 
>>> to do this by looking at the changed lines, to see if they only 
>>> within method bodies and comments (for example), or by doing some 
>>> sort of lexical hash on the signatures, assuming that folk aren't 
>>> messing with imports to change the resolution of simple type names in 
>>> the signature.
>>>
>>> -- Jon
>>>
>>> On 5/6/20 8:30 AM, Jan Lahoda wrote:
>>>> Hi,
>>>>
>>>> Triggered by Magnus' recent e-mail on adjusting the location of the 
>>>> IDE files, I looked at possibilities to improve speed of incremental 
>>>> compilation using make. About 3 years ago, we have sped up 
>>>> incremental build by only rebuilding modules when API of modules 
>>>> they depend on changed. But the module which contains modified 
>>>> sources is always compiled in full. So, for changes in java.base, 
>>>> this change improved the incremental build time significantly (by 
>>>> not recompiling e.g. java.desktop), but it can still take many 
>>>> seconds to build java.base after a trivial change. So, this time, I 
>>>> am thinking of speeding up module builds by not rebuilding all the 
>>>> source if possible.
>>>>
>>>> What I am thinking of is a relatively simple approach: detect 
>>>> changed files in a module and check if their "interface" changed. If 
>>>> it did, recompile the whole module. If it didn't, only compile the 
>>>> modified files. As a consequence, a small change inside a method 
>>>> body should lead to a fast build. Changes outside of method bodies 
>>>> may trigger longer build, but my hope would be that these would be 
>>>> less common.
>>>>
>>>> So far, I unfortunately don't know how to efficiently do this as 
>>>> nicely as the API digests used for inter-module dependencies. The 
>>>> approach that seems might work is this: the Depend plugin hooks 
>>>> itself into javac internals, and filters the incoming files - it 
>>>> first parses the modified ones, and if it cannot find a significant 
>>>> change, it will throw away the unmodified files, and only compile 
>>>> the modified ones. (In principle, it could also do dependency 
>>>> analysis, if we at some point decide it is important.)
>>>>
>>>> For a simple "touch Object.java && make", the wall-clock time is 
>>>> less then 5s, which sounds interesting:
>>>> ---
>>>> $ touch src/java.base/share/classes/java/lang/Object.java && time make
>>>> Building target 'default (exploded-image)' in configuration 
>>>> 'linux-x86_64-server-release'
>>>> Compiling 3028 files for java.base
>>>> Stopping sjavac server
>>>> Finished building target 'default (exploded-image)' in configuration 
>>>> 'linux-x86_64-server-release'
>>>>
>>>> real    0m3,104s
>>>> user    0m5,731s
>>>> sys     0m1,086s
>>>> ---
>>>>
>>>> My current prototype is in the jdk/sandbox repository, branch 
>>>> "jlahoda-depend-in-module":
>>>> http://hg.openjdk.java.net/jdk/sandbox/shortlog/jlahoda-depend-in-module 
>>>>
>>>>
>>>> I wonder if this would sound interesting to developers working on 
>>>> base modules, like java.base.
>>>>
>>>> What do you think? Any ideas/feedback?
>>>>
>>>> Thanks,
>>>>      Jan
>>>>