JMH and continuos integration

[Zoltan Farkas]

We have JMH “benchmark projects” which we run nightly with jenkins to validate the performance of our core components.
we use the jmh-jenkins plugin you mention to identify performance degradations during development.

Here is an example of jmh-jenkins report, (something I recently investigated) (build numbers on x axis):




On top of this we run the benchmarks with java flight recorder/stack sampling profiler, so that when we see something like above we have something to look at….

An example of a benchmark project that I can share you can see at: https://github.com/zolyfarkas/spf4j/tree/master/spf4j-benchmarks <https://github.com/zolyfarkas/spf4j/tree/master/spf4j-benchmarks> 
All one needs to do is “build” a project like this in jenkins with jmh-jenkins plugin…

This approach not only helped us keep our components performing well, it also helped us identify bugs, help us better understand how things work (reviewing the perf profiles before release)…

hope it helps.

—Z



> On Nov 10, 2016, at 7:12 PM, Leonardo Gomes <leonardo.f.gomes at gmail.com> wrote:
> 
> Hi,
> 
> I'm looking for feedback on how people use JMH as part of their development
> process.
> 
> I've played a bit with https://github.com/blackboard/jmh-jenkins and am
> wondering if people in this discussion list do use any sort of automation
> (Jenkins or other) to detect performance regression on their software, on a
> pull-request basis.
> 
> I've seen JMH being used on different open-source projects, but the
> benchmarks seem to have been introduced mostly to compare different
> implementations (when introducing / validating changes) and not as part of
> CI.
> 
> Any feedback would be appreciated.
> 
> Thank you,
> Leonardo.