JMH and continuos integration

[Radim Vansa]

I think you are looking for Alerting [1] - yes, you can check the test 
result against average of last X runs (and much more), and get warning 
mail if the condition is broken.

Radim

[1] https://github.com/PerfCake/PerfRepo/wiki/Alerting

On 11/19/2016 08:07 PM, Sergey Melnikov wrote:
> Hi Radim,
>
> I've never heard about PerfRapo. Thank you for hint.
>
> Does PerfRepo provide any functionality for automatic performance anomalies checking?
> Is it possible to calculate geometric mean for selected group if scores?
>
> --Sergey
>
> On Wed, Nov 16, 2016 at 09:09:25AM +0100, Radim Vansa wrote:
>> On 11/15/2016 04:09 PM, Mark Price wrote:
>>> Hi Leonardo,
>>> at LMAX, we run a number of JMH benchmarks continuously.
>>>
>>> We also try to make sure that there is minimal OS jitter in the results [1], using cpu isolation, sched_setaffinity, and making sure that the benchmark thread gets its own CPU, with no contention from child/spawned threads.
>>>
>>> We record the JSON output in a database, and generate charts based on these results. We have tried (unsuccessfully) to implement a pass/fail CI job that will automatically fail if there are regressions in performance. Unfortunately, we haven't got around to open-sourcing the component that persists results & generates charts.
>> FYI exactly that kind of tool (opensource) is PerfRepo [1]. It does not
>> have a JMH integration, but it has a Java client [2]/RESTful API to
>> upload results.
>>
>> Radim
>>
>> [1] https://github.com/PerfCake/PerfRepo
>> [2] https://github.com/PerfCake/PerfRepo/wiki/PerfRepo-Java-client
>>
>>> We found that even with a very highly tuned system, there was enough inter-run noise that we would get false positives frequently. This led to broken-windows syndrome, so these days we just regularly eyeball the charts for any obvious signal. We record the timestamp and revisions associated with each test-run, so finding a culprit isn't too hard.
>>>
>>> With warm-up iterations, measurement iterations and multiple forks, the feedback-loop can become longer than we'd like. Our micro-benchmarks currently take over an hour to run, though with more hardware we could run them in parallel to improve this. That's still not bad, but for comparison, our suite of ~11k acceptance tests only takes ~25mins...
>>>
>>> I'd be interested to hear of any headway you make in this area.
>>>
>>> Mark
>>>
>>>
>>>
>>> 1] https://epickrram.blogspot.co.uk/2015/08/jvm-guaranteed-safepoints.html
>>>
>>> ----- On 11 Nov, 2016, at 00:12, Leonardo Gomes leonardo.f.gomes at gmail.com wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm looking for feedback on how people use JMH as part of their development
>>>> process.
>>>>
>>>> I've played a bit with https://github.com/blackboard/jmh-jenkins and am
>>>> wondering if people in this discussion list do use any sort of automation
>>>> (Jenkins or other) to detect performance regression on their software, on a
>>>> pull-request basis.
>>>>
>>>> I've seen JMH being used on different open-source projects, but the
>>>> benchmarks seem to have been introduced mostly to compare different
>>>> implementations (when introducing / validating changes) and not as part of
>>>> CI.
>>>>
>>>> Any feedback would be appreciated.
>>>>
>>>> Thank you,
>>>> Leonardo.
>>> ---
>>>
>>> LMAX Exchange, Yellow Building, 1A Nicholas Road, London W11 4AN
>>> http://www.LMAX.com/
>>>
>>> Recognised by the most prestigious business and technology awards
>>>    
>>> 2016 Best Trading & Execution, HFM US Technology Awards
>>> 2016, 2015, 2014, 2013 Best FX Trading Venue - ECN/MTF, WSL Institutional Trading Awards
>>>
>>> 2015 Winner, Deloitte UK Technology Fast 50
>>> 2015, 2014, 2013, One of the UK's fastest growing technology firms, The Sunday Times Tech Track 100
>>> 2015 Winner, Deloitte EMEA Technology Fast 500
>>> 2015, 2014, 2013 Best Margin Sector Platform, Profit & Loss Readers' Choice Awards
>>>
>>> ---
>>>
>>> FX and CFDs are leveraged products that can result in losses exceeding your deposit. They are not suitable for everyone so please ensure you fully understand the risks involved.
>>>
>>> This message and its attachments are confidential, may not be disclosed or used by any person other than the addressee and are intended only for the named recipient(s). This message is not intended for any recipient(s) who based on their nationality, place of business, domicile or for any other reason, is/are subject to local laws or regulations which prohibit the provision of such products and services. This message is subject to the following terms (http://lmax.com/pdf/general-disclaimers.pdf), if you cannot access these, please notify us by replying to this email and we will send you the terms. If you are not the intended recipient, please notify the sender immediately and delete any copies of this message.
>>>
>>> LMAX Exchange is the trading name of LMAX Limited. LMAX Limited operates a multilateral trading facility. LMAX Limited is authorised and regulated by the Financial Conduct Authority (firm registration number 509778) and is a company registered in England and Wales (number 6505809).
>>>
>>> LMAX Hong Kong Limited is a wholly-owned subsidiary of LMAX Limited. LMAX Hong Kong is licensed by the Securities and Futures Commission in Hong Kong to conduct Type 3 (leveraged foreign exchange trading) regulated activity with CE Number BDV088.
>>>
>>
>> -- 
>> Radim Vansa <rvansa at redhat.com>
>> JBoss Performance Team
>>


-- 
Radim Vansa <rvansa at redhat.com>
JBoss Performance Team