Hall of shame: Header compilation times

[David Holmes]

FYI Ioi (cc'd) has already been doing a lot of work with reorganising 
include files and their contents.

David
-----

On 6/09/2022 8:31 pm, Magnus Ihse Bursie wrote:
> I recently discovered the clang -ftime-trace flag, combined with the 
> nifty tool ClangBuildAnalyzer[1] which analyses the output files from 
> -ftime-trace, and presents the "top offenders" for a whole project. 
> Naturally, I ran this on hotspot. :-)
> 
> I think the most important information gathered was how costly different 
> header files are. It is hard to understand which files are included many 
> times, and if that is a problem or if they are trivial. But now we can 
> get this in black and white. So, without further ado, here is the top 
> list of costly header files in Hotspot:
> 
> src/hotspot/share/asm/assembler.hpp (total 30284 ms, included 541 times, 
> avg 55 ms)
> src/hotspot/share/runtime/frame.inline.hpp (total 29234 ms, included 344 
> times, avg 84 ms)
> src/hotspot/share/runtime/javaThread.inline.hpp (total 22297 ms, 
> included 226 times, avg 98 ms)
> src/hotspot/share/oops/access.inline.hpp (total 19675 ms, included 514 
> times, avg 38 ms)
> src/hotspot/share/ci/ciEnv.hpp (total 18703 ms, included 231 times, avg 
> 80 ms)
> src/hotspot/share/runtime/continuationEntry.inline.hpp (total 17318 ms, 
> included 230 times, avg 75 ms)
> src/hotspot/share/oops/access.hpp (total 17123 ms, included 594 times, 
> avg 28 ms)
> src/hotspot/share/code/nativeInst.hpp (total 14769 ms, included 389 
> times, avg 37 ms)
> src/hotspot/cpu/aarch64/nativeInst_aarch64.hpp (total 14757 ms, included 
> 393 times, avg 37 ms)
> src/hotspot/share/code/debugInfoRec.hpp (total 13809 ms, included 257 
> times, avg 53 ms)
> src/hotspot/share/oops/stackChunkOop.inline.hpp (total 13549 ms, 
> included 347 times, avg 39 ms)
> src/hotspot/share/code/debugInfo.hpp (total 13411 ms, included 503 
> times, avg 26 ms)
> src/hotspot/share/gc/g1/g1BarrierSet.inline.hpp (total 13393 ms, 
> included 512 times, avg 26 ms)
> src/hotspot/share/code/compiledMethod.inline.hpp (total 13337 ms, 
> included 330 times, avg 40 ms)
> src/hotspot/share/gc/g1/g1CollectedHeap.inline.hpp (total 13295 ms, 
> included 60 times, avg 221 ms)
> src/hotspot/share/code/location.hpp (total 12095 ms, included 316 times, 
> avg 38 ms)
> gensrc/adfiles/ad_aarch64.hpp (total 11987 ms, included 44 times, avg 
> 272 ms)
> src/hotspot/share/gc/g1/g1ConcurrentMark.inline.hpp (total 11588 ms, 
> included 60 times, avg 193 ms)
> src/hotspot/share/jfr/recorder/checkpoint/types/traceid/jfrTraceId.inline.hpp (total 11406 ms, included 97 times, avg 117 ms)
> gensrc/jfrfiles/jfrEventClasses.hpp (total 11147 ms, included 72 times, 
> avg 154 ms)
> src/hotspot/share/oops/oop.inline.hpp (total 11128 ms, included 510 
> times, avg 21 ms)
> src/hotspot/share/jfr/jfrEvents.hpp (total 10968 ms, included 71 times, 
> avg 154 ms)
> src/hotspot/share/memory/iterator.inline.hpp (total 10512 ms, included 
> 83 times, avg 126 ms)
> src/hotspot/share/opto/compile.hpp (total 10107 ms, included 117 times, 
> avg 86 ms)
> src/hotspot/share/gc/g1/g1OopClosures.inline.hpp (total 10018 ms, 
> included 60 times, avg 166 ms)
> src/hotspot/share/opto/node.hpp (total 8731 ms, included 93 times, avg 
> 93 ms)
> src/hotspot/share/code/codeCache.hpp (total 8574 ms, included 587 times, 
> avg 14 ms)
> src/hotspot/share/gc/z/zBarrier.inline.hpp (total 8446 ms, included 509 
> times, avg 16 ms)
> src/hotspot/share/asm/macroAssembler.hpp (total 8419 ms, included 433 
> times, avg 19 ms)
> src/hotspot/share/jfr/recorder/service/jfrEvent.hpp (total 8390 ms, 
> included 73 times, avg 114 ms)
> src/hotspot/share/classfile/vmSymbols.hpp (total 8346 ms, included 297 
> times, avg 28 ms)
> src/hotspot/share/opto/ad.hpp (total 8258 ms, included 34 times, avg 242 
> ms)
> src/hotspot/share/classfile/javaClasses.inline.hpp (total 8174 ms, 
> included 185 times, avg 44 ms)
> src/hotspot/share/runtime/interfaceSupport.inline.hpp (total 8053 ms, 
> included 135 times, avg 59 ms)
> src/hotspot/share/asm/codeBuffer.hpp (total 7117 ms, included 692 times, 
> avg 10 ms)
> src/hotspot/share/interpreter/interpreter.hpp (total 6891 ms, included 
> 380 times, avg 18 ms)
> src/hotspot/share/ci/ciUtilities.hpp (total 6759 ms, included 196 times, 
> avg 34 ms)
> src/hotspot/share/memory/allocation.inline.hpp (total 6557 ms, included 
> 266 times, avg 24 ms)
> src/hotspot/share/oops/instanceKlass.inline.hpp (total 6395 ms, included 
> 193 times, avg 33 ms)
> src/hotspot/share/code/codeBlob.hpp (total 6108 ms, included 685 times, 
> avg 8 ms)
> 
> So, given this list, it seems like it would have a huge potential payoff 
> to start looking more closely on assembler.hpp. Is it strictly needed 
> everywhere it is included? Can it be split into multiple parts, so most 
> users only need to include some smaller subsets? Etc. And then 
> continuing doing the same with ciEnv.hpp and the top inline.hpp files...
> 
> Caveats: This is from compiling with clang on a mac. There is probably 
> variations due to OS/compiler, etc. But I think this is a very good 
> approximation on what is problematic even for gcc and msvc.
> 
> /Magnus
> 
> [1] https://github.com/aras-p/ClangBuildAnalyzer
>