JMH runner should fail on error

[Aleksey Shipilev]

Hi Thomas,

On 02/11/2015 04:28 PM, Thomas Johnson wrote:

> Is there a reason why the runner (org.openjdk.jmh.Main) doesn't exit with a nonzero return value when it's detected what's clearly an error?

No reason, that's a usability bug:
 https://bugs.openjdk.java.net/browse/CODETOOLS-7901294

However, it would seem that you need a better wrapper for JMH tests
rather than dumb API/CLI call if you want to record the failures for the
individual tests.

Also, there is an option, which is cryptically called "failOnError()" in
API and "-foe" in the CLI, that may terminate the run early when
benchmark experiences an error. It sometimes makes sense to do a shorter
"sanity" check with -foe as a part of build process.

Thanks,
-Aleksey.

P.S. Oh boy:

> LMAX Exchange, Yellow Building, 1A Nicholas Road, London W11 4AN
> http://www.LMAX.com/
> 
> 2014 #1 Fastest Growing Tech Company in the UK - Sunday Times Tech Track 100
> 2014 Best Margin Sector Platform - Profit & Loss Readers' Choice Awards
> 2014 Best FX Trading Venue - ECN/MTF - WSL Institutional Trading Awards
> 2014 Best Infrastructure/Technology Initiative - WSL Institutional Trading Awards
> 2013 #15 Fastest Growing Tech Company in the UK - Sunday Times Tech Track 100
> 2013 Best Overall Testing Project - The European Software Testing Awards
> 2013 Best Margin Sector Platform - Profit & Loss Readers' Choice Awards
> 2013 Best FX Trading Platform - ECN/MTF - WSL Institutional Trading Awards
> 2013 Best Executing Venue - Forex Magnates Awards
> 
> ---
> 
> FX and CFDs are leveraged products that can result in losses exceeding
> your deposit. They are not suitable for everyone so please ensure you fully
> understand the risks involved.
> 
> The information on this email is not directed at residents of the United States of America,
> Australia (we will only deal with Australian clients who are "wholesale clients" as defined
> under the Corporations Act 2001), Canada (although we may deal with Canadian residents
> who meet the "Permitted Client" criteria), Singapore or any other jurisdiction where
> FX trading and/or CFD trading is restricted or prohibited by local laws or regulations. 
> 
> The information in this email and any attachment is confidential and is
> intended only for the named recipient(s). The email may not be disclosed
> or used by any person other than the addressee, nor may it be copied
> in any way. If you are not the intended recipient please notify the sender
> immediately and delete any copies of this message. Any unauthorised
> copying, disclosure or distribution of the material in this e-mail
> is strictly forbidden.
> 
> LMAX Limited is regulated by the Financial Conduct Authority under
> the UK laws, which differ from Australian law. We are exempt from the
> requirement to hold an Australian financial services licence under the
> Corporations Act 2001 (Cth) (Act) in respect of the financial services
> we offer to you. We only offer our services to Australian clients who are
> "wholesale clients" as defined under the Act. We may provide services in
> Canada as an exempt international advisor. Consequently we may only
> provide such services to clients who meet the "Permitted Client" criteria.
> We are not a dealer in Canada.
> 
> LMAX Limited operates a multilateral trading facility. LMAX Limited is
> authorised and regulated by the Financial Conduct Authority (firm
> registration number 509778) and is a company registered in England
> and Wales (number 6505809). Our registered address is Yellow
> Building, 1A Nicholas Road, London, W11 4AN.
>