Faster incremental OpenJDK compilation

[forax at univ-mlv.fr]

----- Mail original -----
> De: "jan lahoda" <jan.lahoda at oracle.com>
> À: "Remi Forax" <forax at univ-mlv.fr>
> Cc: "ide-support-dev" <ide-support-dev at openjdk.java.net>, "compiler-dev" <compiler-dev at openjdk.java.net>
> Envoyé: Mercredi 6 Mai 2020 18:52:54
> Objet: Re: Faster incremental OpenJDK compilation

> FWIW, I should explain this only relates to the OpenJDK build. This is
> not a proposal to do any kind of tweaks like this in javac for other
> compilations. I realized I only wrote that into the subject, but not the
> body. Sorry for that.

What i saying is just that your incremental mode is not safe to use, checking the interface of the classfile is not enough.

> 
> Jan

Rémi

> 
> On 06. 05. 20 18:37, Jan Lahoda wrote:
>> On 06. 05. 20 17:56, Remi Forax wrote:
>>>
>>>
>>> ----- Mail original -----
>>>> De: "jan lahoda" <jan.lahoda at oracle.com>
>>>> À: ide-support-dev at openjdk.java.net, "compiler-dev"
>>>> <compiler-dev at openjdk.java.net>
>>>> Envoyé: Mercredi 6 Mai 2020 17:30:52
>>>> Objet: Faster incremental OpenJDK compilation
>>>
>>>> Hi,
>>>>
>>>> Triggered by Magnus' recent e-mail on adjusting the location of the IDE
>>>> files, I looked at possibilities to improve speed of incremental
>>>> compilation using make. About 3 years ago, we have sped up incremental
>>>> build by only rebuilding modules when API of modules they depend on
>>>> changed. But the module which contains modified sources is always
>>>> compiled in full. So, for changes in java.base, this change improved the
>>>> incremental build time significantly (by not recompiling e.g.
>>>> java.desktop), but it can still take many seconds to build java.base
>>>> after a trivial change. So, this time, I am thinking of speeding up
>>>> module builds by not rebuilding all the source if possible.
>>>>
>>>> What I am thinking of is a relatively simple approach: detect changed
>>>> files in a module and check if their "interface" changed. If it did,
>>>> recompile the whole module. If it didn't, only compile the modified
>>>> files. As a consequence, a small change inside a method body should lead
>>>> to a fast build. Changes outside of method bodies may trigger longer
>>>> build, but my hope would be that these would be less common.
>>>>
>>>> So far, I unfortunately don't know how to efficiently do this as nicely
>>>> as the API digests used for inter-module dependencies. The approach that
>>>> seems might work is this: the Depend plugin hooks itself into javac
>>>> internals, and filters the incoming files - it first parses the modified
>>>> ones, and if it cannot find a significant change, it will throw away the
>>>> unmodified files, and only compile the modified ones. (In principle, it
>>>> could also do dependency analysis, if we at some point decide it is
>>>> important.)
>>>
>>>
>>> I think you are missing the dependencies that are changing the
>>> generated code,
>> 
>> Do you mean for the current hashing, or for the new hashing?
>> 
>> The new hashing is more sketchy now (I just realized modifiers are not
>> reflected in the hash properly). But in both cases, I believe the intent
>> is to make the hashing conservative (and easy), potentially causing more
>> rebuilds than necessary, although I could see some arguments for being
>> more optimistic. (This is basically only for people developing OpenJDK,
>> after all.)
>> 
>>> - in a switch, javac generates a different code depending on the order
>>> of the enum constants,
>> 
>> I believe that any change to order of significant members will change
>> the API hashes. (Which, admittedly, might be overly conservative.)
>> 
>>> - static final are inlined in the code,
>> 
>> For the existing API hash, changes to compile-time constants should
>> change the hash. For the new intra-module hash, currently, (significant)
>> fields initializers are included in full (this is probably overly
>> pessimistic.)
>> 
>>> - for valhalla, changing a class to an inline class or vice versa
>>> change all the method/field descriptors
>> 
>> As long as that is reflected in the modifiers, we should be able to
>> handle fine. If not, then the hashing may need some more tweaks.
>> 
>>>
>>> and i'm sure there are more ..
>> 
>> Possibly - and it is entirely possible the current hashes are way too
>> conservative. But what I wonder about is if people developing e.g.
>> java.base would find such faster incremental builds useful.
>> 
>> Jan
>> 
>>>
>>>>
>>>> For a simple "touch Object.java && make", the wall-clock time is less
>>>> then 5s, which sounds interesting:
>>>> ---
>>>> $ touch src/java.base/share/classes/java/lang/Object.java && time make
>>>> Building target 'default (exploded-image)' in configuration
>>>> 'linux-x86_64-server-release'
>>>> Compiling 3028 files for java.base
>>>> Stopping sjavac server
>>>> Finished building target 'default (exploded-image)' in configuration
>>>> 'linux-x86_64-server-release'
>>>>
>>>> real    0m3,104s
>>>> user    0m5,731s
>>>> sys     0m1,086s
>>>> ---
>>>>
>>>> My current prototype is in the jdk/sandbox repository, branch
>>>> "jlahoda-depend-in-module":
>>>> http://hg.openjdk.java.net/jdk/sandbox/shortlog/jlahoda-depend-in-module
>>>>
>>>> I wonder if this would sound interesting to developers working on base
>>>> modules, like java.base.
>>>>
>>>> What do you think? Any ideas/feedback?
>>>>
>>>> Thanks,
>>>>        Jan
>>>
>>> Rémi