Faster incremental OpenJDK compilation

[Jan Lahoda]

On 06. 05. 20 17:56, Remi Forax wrote:
> 
> 
> ----- Mail original -----
>> De: "jan lahoda" <jan.lahoda at oracle.com>
>> À: ide-support-dev at openjdk.java.net, "compiler-dev" <compiler-dev at openjdk.java.net>
>> Envoyé: Mercredi 6 Mai 2020 17:30:52
>> Objet: Faster incremental OpenJDK compilation
> 
>> Hi,
>>
>> Triggered by Magnus' recent e-mail on adjusting the location of the IDE
>> files, I looked at possibilities to improve speed of incremental
>> compilation using make. About 3 years ago, we have sped up incremental
>> build by only rebuilding modules when API of modules they depend on
>> changed. But the module which contains modified sources is always
>> compiled in full. So, for changes in java.base, this change improved the
>> incremental build time significantly (by not recompiling e.g.
>> java.desktop), but it can still take many seconds to build java.base
>> after a trivial change. So, this time, I am thinking of speeding up
>> module builds by not rebuilding all the source if possible.
>>
>> What I am thinking of is a relatively simple approach: detect changed
>> files in a module and check if their "interface" changed. If it did,
>> recompile the whole module. If it didn't, only compile the modified
>> files. As a consequence, a small change inside a method body should lead
>> to a fast build. Changes outside of method bodies may trigger longer
>> build, but my hope would be that these would be less common.
>>
>> So far, I unfortunately don't know how to efficiently do this as nicely
>> as the API digests used for inter-module dependencies. The approach that
>> seems might work is this: the Depend plugin hooks itself into javac
>> internals, and filters the incoming files - it first parses the modified
>> ones, and if it cannot find a significant change, it will throw away the
>> unmodified files, and only compile the modified ones. (In principle, it
>> could also do dependency analysis, if we at some point decide it is
>> important.)
> 
> 
> I think you are missing the dependencies that are changing the generated code,

Do you mean for the current hashing, or for the new hashing?

The new hashing is more sketchy now (I just realized modifiers are not 
reflected in the hash properly). But in both cases, I believe the intent 
is to make the hashing conservative (and easy), potentially causing more 
rebuilds than necessary, although I could see some arguments for being 
more optimistic. (This is basically only for people developing OpenJDK, 
after all.)

> - in a switch, javac generates a different code depending on the order of the enum constants,

I believe that any change to order of significant members will change 
the API hashes. (Which, admittedly, might be overly conservative.)

> - static final are inlined in the code,

For the existing API hash, changes to compile-time constants should 
change the hash. For the new intra-module hash, currently, (significant) 
fields initializers are included in full (this is probably overly 
pessimistic.)

> - for valhalla, changing a class to an inline class or vice versa change all the method/field descriptors

As long as that is reflected in the modifiers, we should be able to 
handle fine. If not, then the hashing may need some more tweaks.

> 
> and i'm sure there are more ..

Possibly - and it is entirely possible the current hashes are way too 
conservative. But what I wonder about is if people developing e.g. 
java.base would find such faster incremental builds useful.

Jan

> 
>>
>> For a simple "touch Object.java && make", the wall-clock time is less
>> then 5s, which sounds interesting:
>> ---
>> $ touch src/java.base/share/classes/java/lang/Object.java && time make
>> Building target 'default (exploded-image)' in configuration
>> 'linux-x86_64-server-release'
>> Compiling 3028 files for java.base
>> Stopping sjavac server
>> Finished building target 'default (exploded-image)' in configuration
>> 'linux-x86_64-server-release'
>>
>> real    0m3,104s
>> user    0m5,731s
>> sys     0m1,086s
>> ---
>>
>> My current prototype is in the jdk/sandbox repository, branch
>> "jlahoda-depend-in-module":
>> http://hg.openjdk.java.net/jdk/sandbox/shortlog/jlahoda-depend-in-module
>>
>> I wonder if this would sound interesting to developers working on base
>> modules, like java.base.
>>
>> What do you think? Any ideas/feedback?
>>
>> Thanks,
>>        Jan
> 
> Rémi
>