Value returned by benchmark method.

[Aleksey Shipilev]

On 08/09/2016 04:33 PM, Artem Barger wrote:
> Is there a way to enable printing out or showing the computation results
> from the benchmarking method? For example:
> 
> @Benchmark
> public double bechComputation(SomeState state) {
>      return computeResult(state);
> }
> 
> And assuming that based on state "computeResult" runs different algorithms,
> therefore besides of getting performance wise results, it would be also
> interesting to compare the actual results of the computation to
> validate/compare the accuracy.
> 
> Is that possible to achieve it using current JMH version? 

No. Current infrastructure generates the synthetic code that runs
@Benchmark in a specially-constructed loop, and consumes the results
into the Blackhole. This code is supposed to be fast, and we cannot
afford taking detours for logging/asserts there.

> Does it look reasonable to ask for such feature?

Somewhat. However, we would need to explore are there less painful ways
to achieve this result already, or not. For example, if you want to
assert that @Benchmark methods return the same result, you may construct
a custom @Setup method that will call the methods of interest directly,
compare their results, and throw exceptions when results disagree.

I think this is less painful than turning JMH into full-fledged assert
machine. Some workloads would indeed benefit from checking that results
returned from @Benchmark are the same across different @Benchmark-s, but
many would require scripting to spell out the invariants to test. This
seems like something @Setup/@TearDown methods are already providing.

See e.g.:
 http://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_05_StateFixtures.java

Alternatively, you may want to structure your project is such a way that
there is a shared implementation, separate benchmarks that benchmark
those implementations, into those implementations, and separate unit
tests that assert implementation correctness.

(To the extreme point, JMH-annotated tests and JUnit/TestNg tests are
known to coexist well, which makes it possible to run the functional and
performance tests off the same compilation unit -- JMH's own tests
exploit that all the time)

Thanks,
-Aleksey