Hall of shame: Header compilation times

Tue Sep 6 10:31:58 UTC 2022

I recently discovered the clang -ftime-trace flag, combined with the 
nifty tool ClangBuildAnalyzer[1] which analyses the output files from 
-ftime-trace, and presents the "top offenders" for a whole project. 
Naturally, I ran this on hotspot. :-)

I think the most important information gathered was how costly different 
header files are. It is hard to understand which files are included many 
times, and if that is a problem or if they are trivial. But now we can 
get this in black and white. So, without further ado, here is the top 
list of costly header files in Hotspot:

src/hotspot/share/asm/assembler.hpp (total 30284 ms, included 541 times, 
avg 55 ms)
src/hotspot/share/runtime/frame.inline.hpp (total 29234 ms, included 344 
times, avg 84 ms)
src/hotspot/share/runtime/javaThread.inline.hpp (total 22297 ms, 
included 226 times, avg 98 ms)
src/hotspot/share/oops/access.inline.hpp (total 19675 ms, included 514 
times, avg 38 ms)
src/hotspot/share/ci/ciEnv.hpp (total 18703 ms, included 231 times, avg 
80 ms)
src/hotspot/share/runtime/continuationEntry.inline.hpp (total 17318 ms, 
included 230 times, avg 75 ms)
src/hotspot/share/oops/access.hpp (total 17123 ms, included 594 times, 
avg 28 ms)
src/hotspot/share/code/nativeInst.hpp (total 14769 ms, included 389 
times, avg 37 ms)
src/hotspot/cpu/aarch64/nativeInst_aarch64.hpp (total 14757 ms, included 
393 times, avg 37 ms)
src/hotspot/share/code/debugInfoRec.hpp (total 13809 ms, included 257 
times, avg 53 ms)
src/hotspot/share/oops/stackChunkOop.inline.hpp (total 13549 ms, 
included 347 times, avg 39 ms)
src/hotspot/share/code/debugInfo.hpp (total 13411 ms, included 503 
times, avg 26 ms)
src/hotspot/share/gc/g1/g1BarrierSet.inline.hpp (total 13393 ms, 
included 512 times, avg 26 ms)
src/hotspot/share/code/compiledMethod.inline.hpp (total 13337 ms, 
included 330 times, avg 40 ms)
src/hotspot/share/gc/g1/g1CollectedHeap.inline.hpp (total 13295 ms, 
included 60 times, avg 221 ms)
src/hotspot/share/code/location.hpp (total 12095 ms, included 316 times, 
avg 38 ms)
gensrc/adfiles/ad_aarch64.hpp (total 11987 ms, included 44 times, avg 
272 ms)
src/hotspot/share/gc/g1/g1ConcurrentMark.inline.hpp (total 11588 ms, 
included 60 times, avg 193 ms)
src/hotspot/share/jfr/recorder/checkpoint/types/traceid/jfrTraceId.inline.hpp 
(total 11406 ms, included 97 times, avg 117 ms)
gensrc/jfrfiles/jfrEventClasses.hpp (total 11147 ms, included 72 times, 
avg 154 ms)
src/hotspot/share/oops/oop.inline.hpp (total 11128 ms, included 510 
times, avg 21 ms)
src/hotspot/share/jfr/jfrEvents.hpp (total 10968 ms, included 71 times, 
avg 154 ms)
src/hotspot/share/memory/iterator.inline.hpp (total 10512 ms, included 
83 times, avg 126 ms)
src/hotspot/share/opto/compile.hpp (total 10107 ms, included 117 times, 
avg 86 ms)
src/hotspot/share/gc/g1/g1OopClosures.inline.hpp (total 10018 ms, 
included 60 times, avg 166 ms)
src/hotspot/share/opto/node.hpp (total 8731 ms, included 93 times, avg 
93 ms)
src/hotspot/share/code/codeCache.hpp (total 8574 ms, included 587 times, 
avg 14 ms)
src/hotspot/share/gc/z/zBarrier.inline.hpp (total 8446 ms, included 509 
times, avg 16 ms)
src/hotspot/share/asm/macroAssembler.hpp (total 8419 ms, included 433 
times, avg 19 ms)
src/hotspot/share/jfr/recorder/service/jfrEvent.hpp (total 8390 ms, 
included 73 times, avg 114 ms)
src/hotspot/share/classfile/vmSymbols.hpp (total 8346 ms, included 297 
times, avg 28 ms)
src/hotspot/share/opto/ad.hpp (total 8258 ms, included 34 times, avg 242 ms)
src/hotspot/share/classfile/javaClasses.inline.hpp (total 8174 ms, 
included 185 times, avg 44 ms)
src/hotspot/share/runtime/interfaceSupport.inline.hpp (total 8053 ms, 
included 135 times, avg 59 ms)
src/hotspot/share/asm/codeBuffer.hpp (total 7117 ms, included 692 times, 
avg 10 ms)
src/hotspot/share/interpreter/interpreter.hpp (total 6891 ms, included 
380 times, avg 18 ms)
src/hotspot/share/ci/ciUtilities.hpp (total 6759 ms, included 196 times, 
avg 34 ms)
src/hotspot/share/memory/allocation.inline.hpp (total 6557 ms, included 
266 times, avg 24 ms)
src/hotspot/share/oops/instanceKlass.inline.hpp (total 6395 ms, included 
193 times, avg 33 ms)
src/hotspot/share/code/codeBlob.hpp (total 6108 ms, included 685 times, 
avg 8 ms)

So, given this list, it seems like it would have a huge potential payoff 
to start looking more closely on assembler.hpp. Is it strictly needed 
everywhere it is included? Can it be split into multiple parts, so most 
users only need to include some smaller subsets? Etc. And then 
continuing doing the same with ciEnv.hpp and the top inline.hpp files...

Caveats: This is from compiling with clang on a mac. There is probably 
variations due to OS/compiler, etc. But I think this is a very good 
approximation on what is problematic even for gcc and msvc.

/Magnus

[1] https://github.com/aras-p/ClangBuildAnalyzer