Faster incremental OpenJDK compilation

[Jonathan Gibbons]

You could refine the imports by filtering them according to whether they 
are 'star-imports' or the simple name matches a simple name that has 
been seen by the TreeScanner.  In other words, try and filter out 
imports that are definitely not relevant.

-- Jon


On 5/6/20 9:49 AM, Jan Lahoda wrote:
> Hi Jon,
>
> Good question. I was first experimenting with hashing Elements, as we 
> do currently for the module API hashes, but it was too slow to enter 
> the sources (although it is not completely impossible). So I did a new 
> hashing based on the AST - basically, it is a TreeScanner sending Tree 
> kinds, names and sub-trees into a MessageDigest. It ignores things 
> like method bodies, class initializers, and private class members. 
> Does not currently skip imports, although that may be seen as too 
> conservative. But tweaks to the AST hashing are surely still needed, 
> so we can tweak the exact meaning of a "significant change".
>
> Jan
>
> On 06. 05. 20 18:07, Jonathan Gibbons wrote:
>> Jan,
>>
>> This seems like an interesting approach.
>>
>> How are you determining "significant change"?  I could imagine trying 
>> to do this by looking at the changed lines, to see if they only 
>> within method bodies and comments (for example), or by doing some 
>> sort of lexical hash on the signatures, assuming that folk aren't 
>> messing with imports to change the resolution of simple type names in 
>> the signature.
>>
>> -- Jon
>>
>> On 5/6/20 8:30 AM, Jan Lahoda wrote:
>>> Hi,
>>>
>>> Triggered by Magnus' recent e-mail on adjusting the location of the 
>>> IDE files, I looked at possibilities to improve speed of incremental 
>>> compilation using make. About 3 years ago, we have sped up 
>>> incremental build by only rebuilding modules when API of modules 
>>> they depend on changed. But the module which contains modified 
>>> sources is always compiled in full. So, for changes in java.base, 
>>> this change improved the incremental build time significantly (by 
>>> not recompiling e.g. java.desktop), but it can still take many 
>>> seconds to build java.base after a trivial change. So, this time, I 
>>> am thinking of speeding up module builds by not rebuilding all the 
>>> source if possible.
>>>
>>> What I am thinking of is a relatively simple approach: detect 
>>> changed files in a module and check if their "interface" changed. If 
>>> it did, recompile the whole module. If it didn't, only compile the 
>>> modified files. As a consequence, a small change inside a method 
>>> body should lead to a fast build. Changes outside of method bodies 
>>> may trigger longer build, but my hope would be that these would be 
>>> less common.
>>>
>>> So far, I unfortunately don't know how to efficiently do this as 
>>> nicely as the API digests used for inter-module dependencies. The 
>>> approach that seems might work is this: the Depend plugin hooks 
>>> itself into javac internals, and filters the incoming files - it 
>>> first parses the modified ones, and if it cannot find a significant 
>>> change, it will throw away the unmodified files, and only compile 
>>> the modified ones. (In principle, it could also do dependency 
>>> analysis, if we at some point decide it is important.)
>>>
>>> For a simple "touch Object.java && make", the wall-clock time is 
>>> less then 5s, which sounds interesting:
>>> ---
>>> $ touch src/java.base/share/classes/java/lang/Object.java && time make
>>> Building target 'default (exploded-image)' in configuration 
>>> 'linux-x86_64-server-release'
>>> Compiling 3028 files for java.base
>>> Stopping sjavac server
>>> Finished building target 'default (exploded-image)' in configuration 
>>> 'linux-x86_64-server-release'
>>>
>>> real    0m3,104s
>>> user    0m5,731s
>>> sys     0m1,086s
>>> ---
>>>
>>> My current prototype is in the jdk/sandbox repository, branch 
>>> "jlahoda-depend-in-module":
>>> http://hg.openjdk.java.net/jdk/sandbox/shortlog/jlahoda-depend-in-module 
>>>
>>>
>>> I wonder if this would sound interesting to developers working on 
>>> base modules, like java.base.
>>>
>>> What do you think? Any ideas/feedback?
>>>
>>> Thanks,
>>>      Jan
>>>