Faster incremental OpenJDK compilation

Wed May 6 18:24:35 UTC 2020

On 06. 05. 20 19:39, forax at univ-mlv.fr wrote:
> ----- Mail original -----
>> De: "jan lahoda" <jan.lahoda at oracle.com>
>> À: "Remi Forax" <forax at univ-mlv.fr>
>> Cc: "ide-support-dev" <ide-support-dev at openjdk.java.net>, "compiler-dev" <compiler-dev at openjdk.java.net>
>> Envoyé: Mercredi 6 Mai 2020 18:52:54
>> Objet: Re: Faster incremental OpenJDK compilation
> 
>> FWIW, I should explain this only relates to the OpenJDK build. This is
>> not a proposal to do any kind of tweaks like this in javac for other
>> compilations. I realized I only wrote that into the subject, but not the
>> body. Sorry for that.
> 
> What i saying is just that your incremental mode is not safe to use, checking the interface of the classfile is not enough.

I definitely would not recommend to use outcomes of incremental builds 
in production, but my goal is to have it safe enough so that it can be 
enabled, and developers would only notice having faster incremental 
builds (mostly for testing).

Note the existing module API hashing exists, and is enabled by default, 
for some time, and I am not aware of any cases where the hashing as such 
would not trigger a rebuild of module even if it should (but it is 
possible such cases exist, I just don't know about them).

Please note the hashing is not based on classfiles. The existing hashing 
is using javax.lang.model to hash APIs in exported packages. See:
http://hg.openjdk.java.net/jdk/jdk/file/7223c6d61034/make/jdk/src/classes/build/tools/depend/Depend.java#l122
http://hg.openjdk.java.net/jdk/jdk/file/7223c6d61034/make/jdk/src/classes/build/tools/depend/Depend.java#l167

This produces a hash, and if the hash differs from the last recorded 
hash, a recompilation of all modules that depend on the current module 
is triggered (this is done using make dependencies, and existed even 
before the API hashing).

This new proposal attempts to take a similar approach inside modules 
(esp. java.base and jdk.compiler). Basically, what I tried is to let the 
javac's AST for the modified files go through a visitor that will hash 
the AST nodes (ignoring method bodies, etc.), and if the hash is not the 
same as the last time the given file was compiled, all the sources of 
the given module are compiled:
http://hg.openjdk.java.net/jdk/sandbox/file/4b454fa5c057/make/jdk/src/classes/build/tools/depend/Depend.java#l639

There may be bugs in the hashing, of course.

Jan

> 
>>
>> Jan
> 
> Rémi
> 
>>
>> On 06. 05. 20 18:37, Jan Lahoda wrote:
>>> On 06. 05. 20 17:56, Remi Forax wrote:
>>>>
>>>>
>>>> ----- Mail original -----
>>>>> De: "jan lahoda" <jan.lahoda at oracle.com>
>>>>> À: ide-support-dev at openjdk.java.net, "compiler-dev"
>>>>> <compiler-dev at openjdk.java.net>
>>>>> Envoyé: Mercredi 6 Mai 2020 17:30:52
>>>>> Objet: Faster incremental OpenJDK compilation
>>>>
>>>>> Hi,
>>>>>
>>>>> Triggered by Magnus' recent e-mail on adjusting the location of the IDE
>>>>> files, I looked at possibilities to improve speed of incremental
>>>>> compilation using make. About 3 years ago, we have sped up incremental
>>>>> build by only rebuilding modules when API of modules they depend on
>>>>> changed. But the module which contains modified sources is always
>>>>> compiled in full. So, for changes in java.base, this change improved the
>>>>> incremental build time significantly (by not recompiling e.g.
>>>>> java.desktop), but it can still take many seconds to build java.base
>>>>> after a trivial change. So, this time, I am thinking of speeding up
>>>>> module builds by not rebuilding all the source if possible.
>>>>>
>>>>> What I am thinking of is a relatively simple approach: detect changed
>>>>> files in a module and check if their "interface" changed. If it did,
>>>>> recompile the whole module. If it didn't, only compile the modified
>>>>> files. As a consequence, a small change inside a method body should lead
>>>>> to a fast build. Changes outside of method bodies may trigger longer
>>>>> build, but my hope would be that these would be less common.
>>>>>
>>>>> So far, I unfortunately don't know how to efficiently do this as nicely
>>>>> as the API digests used for inter-module dependencies. The approach that
>>>>> seems might work is this: the Depend plugin hooks itself into javac
>>>>> internals, and filters the incoming files - it first parses the modified
>>>>> ones, and if it cannot find a significant change, it will throw away the
>>>>> unmodified files, and only compile the modified ones. (In principle, it
>>>>> could also do dependency analysis, if we at some point decide it is
>>>>> important.)
>>>>
>>>>
>>>> I think you are missing the dependencies that are changing the
>>>> generated code,
>>>
>>> Do you mean for the current hashing, or for the new hashing?
>>>
>>> The new hashing is more sketchy now (I just realized modifiers are not
>>> reflected in the hash properly). But in both cases, I believe the intent
>>> is to make the hashing conservative (and easy), potentially causing more
>>> rebuilds than necessary, although I could see some arguments for being
>>> more optimistic. (This is basically only for people developing OpenJDK,
>>> after all.)
>>>
>>>> - in a switch, javac generates a different code depending on the order
>>>> of the enum constants,
>>>
>>> I believe that any change to order of significant members will change
>>> the API hashes. (Which, admittedly, might be overly conservative.)
>>>
>>>> - static final are inlined in the code,
>>>
>>> For the existing API hash, changes to compile-time constants should
>>> change the hash. For the new intra-module hash, currently, (significant)
>>> fields initializers are included in full (this is probably overly
>>> pessimistic.)
>>>
>>>> - for valhalla, changing a class to an inline class or vice versa
>>>> change all the method/field descriptors
>>>
>>> As long as that is reflected in the modifiers, we should be able to
>>> handle fine. If not, then the hashing may need some more tweaks.
>>>
>>>>
>>>> and i'm sure there are more ..
>>>
>>> Possibly - and it is entirely possible the current hashes are way too
>>> conservative. But what I wonder about is if people developing e.g.
>>> java.base would find such faster incremental builds useful.
>>>
>>> Jan
>>>
>>>>
>>>>>
>>>>> For a simple "touch Object.java && make", the wall-clock time is less
>>>>> then 5s, which sounds interesting:
>>>>> ---
>>>>> $ touch src/java.base/share/classes/java/lang/Object.java && time make
>>>>> Building target 'default (exploded-image)' in configuration
>>>>> 'linux-x86_64-server-release'
>>>>> Compiling 3028 files for java.base
>>>>> Stopping sjavac server
>>>>> Finished building target 'default (exploded-image)' in configuration
>>>>> 'linux-x86_64-server-release'
>>>>>
>>>>> real    0m3,104s
>>>>> user    0m5,731s
>>>>> sys     0m1,086s
>>>>> ---
>>>>>
>>>>> My current prototype is in the jdk/sandbox repository, branch
>>>>> "jlahoda-depend-in-module":
>>>>> http://hg.openjdk.java.net/jdk/sandbox/shortlog/jlahoda-depend-in-module
>>>>>
>>>>> I wonder if this would sound interesting to developers working on base
>>>>> modules, like java.base.
>>>>>
>>>>> What do you think? Any ideas/feedback?
>>>>>
>>>>> Thanks,
>>>>>         Jan
>>>>
>>>> Rémi