Runner API: standard output vs custom output

[Aleksey Shipilev]

Hi,

On 11/19/2013 04:47 PM, Sergey Kuksenko wrote:
> I suggest to eliminate Silent output format. Make examples: if someone
> wants to avoid interfere  between standard and custom output -> they may
> redirect standard output to a file (for later examination):
>     Options opts = new OptionsBuilder()   ...
>     .output("filename")
>     .build();
> It looks like nobody knows about ability to write jmh output to a file.

I don't think eliminating the OutputFormat.Silent and forcing users to
redirect output to the auxiliary file has anything to do with the fact
people are blindly trusting the numbers. In the same way, users will
ignore the file contents they have just redirected to. So, I think the
suggestion to remove OF.Silent is unwarranted, sorry.

> The second - we have a lack of statistics examples and APIs.
> May be it worth to add to standard Result API something like this:

...

> /**
>   * Returns whether the two closed intervals are disjoint.
>   */
> public static boolean isSignificant(double base_score, double base_err,
>             double opt_score, double opt_err) {
>         double base_min = base_score - base_err;
>         double base_max = base_score + base_err;
>         double opt_min = opt_score - opt_err;
>         double opt_max = opt_score + opt_err;
>         return !((base_min <= opt_min && opt_min <= base_max) ||
>                 (base_min <= opt_max && opt_max <= base_max) ||
>                 (opt_min <= base_min && base_min <= opt_max) ||
>                 (opt_min <= base_max && base_max <= opt_max));
>     }

This abuses the notion of confidence interval... we need to do the
proper inference, but that will only be available when we have the
Commons Math to do the inference for us. This is in the roadmap. You can
use the trick above in the meantime.

-Aleksey.