RFR 8221623 : Add StackWalker micro benchmarks to jdk repo

[Brent Christian]

Hi, Claes

On 9/19/19 1:40 PM, Claes Redestad wrote:
> 
> FYI, you can control the number of forks etc run by make test:
> 
> make test TEST="micro:java.lang.StackWalkBench" 
> MICRO="FORK=1;WARMUP_ITER=2;OPTIONS=-p depth=128"
> 
> (documented in doc/testing.md|html)

Neat, thanks!

> Still, I agree it might be a good idea to have make test TEST=... run
> fewer forks/iterations by default, since this shortcut was primarily
> intended as a means for devs to quickly verify specific microbenchmarks.

OK, I've submitted:
https://bugs.openjdk.java.net/browse/JDK-8231315

> However, reducing forks and iterations only cuts down one factor of the
> total runtime: others being the number of methods and the number of
> chosen parameters (as controlled by @Param-annotated fields).
> 
> So IMHO, what should be thought through carefully when adding micros is
> whether all methods are necessary or at least useful and whether we can
> reduce the number of parameter chosen.
> 
> For these two, I think picking 2 or 3 values for depth (lowest, highest,
> maybe a midpoint) might be sufficient. This would cut total runtime
> budget by 60% or 40%, respectively.

Mandy makes a good point that configurations would be different between 
doing a quick performance verification/sanity check, versus a more 
comprehensive run to measure tuning work or larger scale work.

In the interest of reducing the run time when using 'make', I'd be OK 
with having fewer values in the 'depth' @Param in the source.  It's easy 
enough to add values back from the commandline for more thorough 
testing.  (In the case of StackWalker, it's good to exercise its various 
batch sizes.)

Thanks,
-Brent

P.S.
As you alluded to, there is also a "quick-and-efficient versus thorough 
coverage" tension WRT what @Benchmark methods get included/run.  Worth 
giving thought to sometime...

> On 2019-09-19 20:09, Brent Christian wrote:
>> Hi, Mandy
>>
>> Yes, that 'make' job would take ~7 hours on my machine.
>> I believe this is typical for running micros using 'make'.  AFAICT, 
>> the jmh defaults are used, so it forks (running all the warmups + 
>> iterations) 5 times.  Trying another (>100 loc) benchmark  at random, 
>> the java.lang.ArrayCopy one would take 25 hours.
>>
>> There might be a way to pass jmh args (e.g. '-f 1') via 'make test 
>> TEST=...', though I really only ever run benchmarks using 'java -jar 
>> path/to/benchmark.jar'.
>>
>> It may be worth looking into updating the make target to setup more 
>> practical JMH runtimes by default.  But I don't think this an issue 
>> particular the new benchmarks.
>>
>> Thanks,
>> -Brent
>> On 9/18/19 7:33 PM, Mandy Chung wrote:
>>> Hi Brent,
>>>
>>> $ make test TEST="micro:java.lang.StackWalkBench"
>>>
>>> It took very long that I killed the job.  Does this happen to you?
>>>
>>> Mandy
>>>
>>> On 9/13/19 3:07 PM, Brent Christian wrote:
>>>> Hi,
>>>>
>>>> Please review these StackWalker and Throwable benchmarks for 
>>>> addition into the JDK microbenchmarks.
>>>>
>>>> Bug:
>>>> https://bugs.openjdk.java.net/browse/JDK-8221623
>>>> Webrev:
>>>> http://cr.openjdk.java.net/~bchristi/8221623/webrev07/
>>>>
>>>> The StackWalker benchmarks use StackWalker's forEach(), walk(), and 
>>>> Options to measure retrieval of various types of information from 
>>>> the call stack.
>>>>
>>>> The Throwable benchmarks do corresponding exercises; there are also 
>>>> a couple of Logging benchmarks.
>>>>
>>>> A JMH @Param is used to test a variety of call stack depths.
>>>>
>>>>
>>>> In the future, we might consider a benchmark for 
>>>> Reflection.getCallerClass().  (It is more involved today to 
>>>> benchmark that method than at the time these benchmarks were 
>>>> originally written, so that one's commented out.) See JDK-8230976.
>>>>
>>>> Thanks,
>>>> -Brent
>>>>
>>>> 1. https://bugs.openjdk.java.net/browse/JDK-8230976
>>>
>>