How to assess impact of specializations on warmup

[Stefan Marr]

Hi:

I was wondering whether you guys have a way of assessing the impact of optimizations and/or specializations on warmup.

I was thinking of a simple approach based on the observed peak performance.
So, assuming that each benchmark iteration does a constant amount of work (minus optimizations), perhaps taking the index of the first iteration that is within an error margin of the peak performance would be a good proxy for the time it takes to warmup.

However, I also see on a number of benchmarks that there are later compilations happening, that do not necessarily contribute to the peak performance.

Some benchmarks even have one fast run very early after some compilation. Then however further compilation happens that degrades performance again and it takes a while to reach the peak performance again.
So, well, I am not sure whether just taking the first iteration which is close to peak is a good idea.
On the other hand, saying peak is reached when it was observed for 20 or so iterations seems also to be a little arbitrary.

Is there perhaps literature on the topic? So far I have mostly seen people reporting startup time by measuring a first iteration. However, that isn’t really what I am interested in, since I want to know what the impact of certain specializations is on the warmup behavior.

Thanks
Stefan



-- 
Stefan Marr
INRIA Lille - Nord Europe
http://stefan-marr.de/research/