Runner API: standard output vs custom output

[Sergey Kuksenko]

Hi Aleksey,

I'd like share an issue which I've seen TWICE in micros created by others:

- People want to get their custom result output
- They implement benchmarks with Runner API
- Standard jmh output interfere with custom output
- jmh users decided to suppress standard output; do it:
    Options opts = new OptionsBuilder()   ...
    .outputFormat(OutputFormatType.Silent)
    .build();
- they lost standard jmh output => they are looking into results score
which is an average through several forks => they lost the fact that
results have high run-to-run variance what is an issue itself and should
be tracked. I've seed examples with noticeable bimodal distribution,
which may be easily detected if original output would be preserved.

I suggest to eliminate Silent output format. Make examples: if someone
wants to avoid interfere  between standard and custom output -> they may
redirect standard output to a file (for later examination):
    Options opts = new OptionsBuilder()   ...
    .output("filename")
    .build();
It looks like nobody knows about ability to write jmh output to a file.

The second - we have a lack of statistics examples and APIs.
May be it worth to add to standard Result API something like this:

public static double ci99(Result res) {
  double[] interval99 = res.getStatistics().getConfidenceInterval(0.01);
  return (interval99[1] - interval99[0]) / 2;
}

It the method which I use my benchmarks.

and like this (Brian B. did it):

/**
  * Returns whether the two closed intervals are disjoint.
  */
public static boolean isSignificant(double base_score, double base_err,
            double opt_score, double opt_err) {
        double base_min = base_score - base_err;
        double base_max = base_score + base_err;
        double opt_min = opt_score - opt_err;
        double opt_max = opt_score + opt_err;
        return !((base_min <= opt_min && opt_min <= base_max) ||
                (base_min <= opt_max && opt_max <= base_max) ||
                (opt_min <= base_min && base_min <= opt_max) ||
                (opt_min <= base_max && base_max <= opt_max));
    }


-- 
Best regards,
Sergey Kuksenko