Hall of shame: Header compilation times

[Ioi Lam]

I mainly look at non-PCH compilation time. I want to optimize the build 
time to HotSpot developers.

Here are some recent fixes

https://bugs.openjdk.org/issues/?jql=labels%20%3D%20include%20AND%20Status%20in%20(Resolved)%20ORDER%20BY%20resolved%20DESC

I usually avoid doing large changes. A lot of times you can remove 
superfluous dependencies by making small tweaks, like moving inline 
function definitions from .hpp to .cpp files.

Thanks

- Ioi

On 9/19/2022 7:51 PM, David Holmes wrote:
> FYI Ioi (cc'd) has already been doing a lot of work with reorganising 
> include files and their contents.
>
> David
> -----
>
> On 6/09/2022 8:31 pm, Magnus Ihse Bursie wrote:
>> I recently discovered the clang -ftime-trace flag, combined with the 
>> nifty tool ClangBuildAnalyzer[1] which analyses the output files from 
>> -ftime-trace, and presents the "top offenders" for a whole project. 
>> Naturally, I ran this on hotspot. :-)
>>
>> I think the most important information gathered was how costly 
>> different header files are. It is hard to understand which files are 
>> included many times, and if that is a problem or if they are trivial. 
>> But now we can get this in black and white. So, without further ado, 
>> here is the top list of costly header files in Hotspot:
>>
>> src/hotspot/share/asm/assembler.hpp (total 30284 ms, included 541 
>> times, avg 55 ms)
>> src/hotspot/share/runtime/frame.inline.hpp (total 29234 ms, included 
>> 344 times, avg 84 ms)
>> src/hotspot/share/runtime/javaThread.inline.hpp (total 22297 ms, 
>> included 226 times, avg 98 ms)
>> src/hotspot/share/oops/access.inline.hpp (total 19675 ms, included 
>> 514 times, avg 38 ms)
>> src/hotspot/share/ci/ciEnv.hpp (total 18703 ms, included 231 times, 
>> avg 80 ms)
>> src/hotspot/share/runtime/continuationEntry.inline.hpp (total 17318 
>> ms, included 230 times, avg 75 ms)
>> src/hotspot/share/oops/access.hpp (total 17123 ms, included 594 
>> times, avg 28 ms)
>> src/hotspot/share/code/nativeInst.hpp (total 14769 ms, included 389 
>> times, avg 37 ms)
>> src/hotspot/cpu/aarch64/nativeInst_aarch64.hpp (total 14757 ms, 
>> included 393 times, avg 37 ms)
>> src/hotspot/share/code/debugInfoRec.hpp (total 13809 ms, included 257 
>> times, avg 53 ms)
>> src/hotspot/share/oops/stackChunkOop.inline.hpp (total 13549 ms, 
>> included 347 times, avg 39 ms)
>> src/hotspot/share/code/debugInfo.hpp (total 13411 ms, included 503 
>> times, avg 26 ms)
>> src/hotspot/share/gc/g1/g1BarrierSet.inline.hpp (total 13393 ms, 
>> included 512 times, avg 26 ms)
>> src/hotspot/share/code/compiledMethod.inline.hpp (total 13337 ms, 
>> included 330 times, avg 40 ms)
>> src/hotspot/share/gc/g1/g1CollectedHeap.inline.hpp (total 13295 ms, 
>> included 60 times, avg 221 ms)
>> src/hotspot/share/code/location.hpp (total 12095 ms, included 316 
>> times, avg 38 ms)
>> gensrc/adfiles/ad_aarch64.hpp (total 11987 ms, included 44 times, avg 
>> 272 ms)
>> src/hotspot/share/gc/g1/g1ConcurrentMark.inline.hpp (total 11588 ms, 
>> included 60 times, avg 193 ms)
>> src/hotspot/share/jfr/recorder/checkpoint/types/traceid/jfrTraceId.inline.hpp 
>> (total 11406 ms, included 97 times, avg 117 ms)
>> gensrc/jfrfiles/jfrEventClasses.hpp (total 11147 ms, included 72 
>> times, avg 154 ms)
>> src/hotspot/share/oops/oop.inline.hpp (total 11128 ms, included 510 
>> times, avg 21 ms)
>> src/hotspot/share/jfr/jfrEvents.hpp (total 10968 ms, included 71 
>> times, avg 154 ms)
>> src/hotspot/share/memory/iterator.inline.hpp (total 10512 ms, 
>> included 83 times, avg 126 ms)
>> src/hotspot/share/opto/compile.hpp (total 10107 ms, included 117 
>> times, avg 86 ms)
>> src/hotspot/share/gc/g1/g1OopClosures.inline.hpp (total 10018 ms, 
>> included 60 times, avg 166 ms)
>> src/hotspot/share/opto/node.hpp (total 8731 ms, included 93 times, 
>> avg 93 ms)
>> src/hotspot/share/code/codeCache.hpp (total 8574 ms, included 587 
>> times, avg 14 ms)
>> src/hotspot/share/gc/z/zBarrier.inline.hpp (total 8446 ms, included 
>> 509 times, avg 16 ms)
>> src/hotspot/share/asm/macroAssembler.hpp (total 8419 ms, included 433 
>> times, avg 19 ms)
>> src/hotspot/share/jfr/recorder/service/jfrEvent.hpp (total 8390 ms, 
>> included 73 times, avg 114 ms)
>> src/hotspot/share/classfile/vmSymbols.hpp (total 8346 ms, included 
>> 297 times, avg 28 ms)
>> src/hotspot/share/opto/ad.hpp (total 8258 ms, included 34 times, avg 
>> 242 ms)
>> src/hotspot/share/classfile/javaClasses.inline.hpp (total 8174 ms, 
>> included 185 times, avg 44 ms)
>> src/hotspot/share/runtime/interfaceSupport.inline.hpp (total 8053 ms, 
>> included 135 times, avg 59 ms)
>> src/hotspot/share/asm/codeBuffer.hpp (total 7117 ms, included 692 
>> times, avg 10 ms)
>> src/hotspot/share/interpreter/interpreter.hpp (total 6891 ms, 
>> included 380 times, avg 18 ms)
>> src/hotspot/share/ci/ciUtilities.hpp (total 6759 ms, included 196 
>> times, avg 34 ms)
>> src/hotspot/share/memory/allocation.inline.hpp (total 6557 ms, 
>> included 266 times, avg 24 ms)
>> src/hotspot/share/oops/instanceKlass.inline.hpp (total 6395 ms, 
>> included 193 times, avg 33 ms)
>> src/hotspot/share/code/codeBlob.hpp (total 6108 ms, included 685 
>> times, avg 8 ms)
>>
>> So, given this list, it seems like it would have a huge potential 
>> payoff to start looking more closely on assembler.hpp. Is it strictly 
>> needed everywhere it is included? Can it be split into multiple 
>> parts, so most users only need to include some smaller subsets? Etc. 
>> And then continuing doing the same with ciEnv.hpp and the top 
>> inline.hpp files...
>>
>> Caveats: This is from compiling with clang on a mac. There is 
>> probably variations due to OS/compiler, etc. But I think this is a 
>> very good approximation on what is problematic even for gcc and msvc.
>>
>> /Magnus
>>
>> [1] https://github.com/aras-p/ClangBuildAnalyzer
>>