Faster incremental OpenJDK compilation

[Jan Lahoda]

FWIW, I should explain this only relates to the OpenJDK build. This is 
not a proposal to do any kind of tweaks like this in javac for other 
compilations. I realized I only wrote that into the subject, but not the 
body. Sorry for that.

Jan

On 06. 05. 20 18:37, Jan Lahoda wrote:
> On 06. 05. 20 17:56, Remi Forax wrote:
>>
>>
>> ----- Mail original -----
>>> De: "jan lahoda" <jan.lahoda at oracle.com>
>>> À: ide-support-dev at openjdk.java.net, "compiler-dev" 
>>> <compiler-dev at openjdk.java.net>
>>> Envoyé: Mercredi 6 Mai 2020 17:30:52
>>> Objet: Faster incremental OpenJDK compilation
>>
>>> Hi,
>>>
>>> Triggered by Magnus' recent e-mail on adjusting the location of the IDE
>>> files, I looked at possibilities to improve speed of incremental
>>> compilation using make. About 3 years ago, we have sped up incremental
>>> build by only rebuilding modules when API of modules they depend on
>>> changed. But the module which contains modified sources is always
>>> compiled in full. So, for changes in java.base, this change improved the
>>> incremental build time significantly (by not recompiling e.g.
>>> java.desktop), but it can still take many seconds to build java.base
>>> after a trivial change. So, this time, I am thinking of speeding up
>>> module builds by not rebuilding all the source if possible.
>>>
>>> What I am thinking of is a relatively simple approach: detect changed
>>> files in a module and check if their "interface" changed. If it did,
>>> recompile the whole module. If it didn't, only compile the modified
>>> files. As a consequence, a small change inside a method body should lead
>>> to a fast build. Changes outside of method bodies may trigger longer
>>> build, but my hope would be that these would be less common.
>>>
>>> So far, I unfortunately don't know how to efficiently do this as nicely
>>> as the API digests used for inter-module dependencies. The approach that
>>> seems might work is this: the Depend plugin hooks itself into javac
>>> internals, and filters the incoming files - it first parses the modified
>>> ones, and if it cannot find a significant change, it will throw away the
>>> unmodified files, and only compile the modified ones. (In principle, it
>>> could also do dependency analysis, if we at some point decide it is
>>> important.)
>>
>>
>> I think you are missing the dependencies that are changing the 
>> generated code,
> 
> Do you mean for the current hashing, or for the new hashing?
> 
> The new hashing is more sketchy now (I just realized modifiers are not 
> reflected in the hash properly). But in both cases, I believe the intent 
> is to make the hashing conservative (and easy), potentially causing more 
> rebuilds than necessary, although I could see some arguments for being 
> more optimistic. (This is basically only for people developing OpenJDK, 
> after all.)
> 
>> - in a switch, javac generates a different code depending on the order 
>> of the enum constants,
> 
> I believe that any change to order of significant members will change 
> the API hashes. (Which, admittedly, might be overly conservative.)
> 
>> - static final are inlined in the code,
> 
> For the existing API hash, changes to compile-time constants should 
> change the hash. For the new intra-module hash, currently, (significant) 
> fields initializers are included in full (this is probably overly 
> pessimistic.)
> 
>> - for valhalla, changing a class to an inline class or vice versa 
>> change all the method/field descriptors
> 
> As long as that is reflected in the modifiers, we should be able to 
> handle fine. If not, then the hashing may need some more tweaks.
> 
>>
>> and i'm sure there are more ..
> 
> Possibly - and it is entirely possible the current hashes are way too 
> conservative. But what I wonder about is if people developing e.g. 
> java.base would find such faster incremental builds useful.
> 
> Jan
> 
>>
>>>
>>> For a simple "touch Object.java && make", the wall-clock time is less
>>> then 5s, which sounds interesting:
>>> ---
>>> $ touch src/java.base/share/classes/java/lang/Object.java && time make
>>> Building target 'default (exploded-image)' in configuration
>>> 'linux-x86_64-server-release'
>>> Compiling 3028 files for java.base
>>> Stopping sjavac server
>>> Finished building target 'default (exploded-image)' in configuration
>>> 'linux-x86_64-server-release'
>>>
>>> real    0m3,104s
>>> user    0m5,731s
>>> sys     0m1,086s
>>> ---
>>>
>>> My current prototype is in the jdk/sandbox repository, branch
>>> "jlahoda-depend-in-module":
>>> http://hg.openjdk.java.net/jdk/sandbox/shortlog/jlahoda-depend-in-module
>>>
>>> I wonder if this would sound interesting to developers working on base
>>> modules, like java.base.
>>>
>>> What do you think? Any ideas/feedback?
>>>
>>> Thanks,
>>>        Jan
>>
>> Rémi
>>