Raw data output

[Jens Wilke]

On Tuesday 28 June 2016 13:19:54 Petr Stefan wrote:
> Hi.
> 
> > Depending on the scenario I need to benchmark, I choose the benchmark mode.
> > I would find it very limiting to be restricted to one, by some additional
> > tooling.
> >
> > Maybe you want to share some benchmarks and discuss about that?
> 
> We are developing a tool, that collects benchmarking data, processes 
> them and saves them depending on version (like git commit). Then you can 
> specify some SPL formula (if you're interested, look at papers here 
> http://d3s.mff.cuni.cz/research/development_awareness/#performance-testing) 
> and evaluate it on top of the data. Typical use case is that you don't 
> want your code to slow down, so your formula will reflect this 
> requirement and our tool will find regressions in your code between 
> commits. This is too big piece of code (and probably little 
> out-of-topic) to nicely fit into JMH, so we want to get raw data out of 
> JMH and process them ourselves.

Yes, I looked at your site. BTW: All the papers are hidden behind pay walls, so I cannot
really asses the scientific approach in a glimpse.

That wasn't the direction what I am aiming at. It's clear that this is out of the scope
of JMH.

Let's say I have two benchmarks:

Throughput: 
https://github.com/cache2k/cache2k-benchmark/blob/master/jmh-suite/src/main/java/org/cache2k/benchmark/jmh/suite/noEviction/symmetrical/ReadOnlyBenchmark.java

Single shot:
https://github.com/cache2k/cache2k-benchmark/blob/master/jmh-suite/src/main/java/org/cache2k/benchmark/jmh/suite/noEviction/symmetrical/PopulateParallelOnceBenchmark.java

The reason why I choose single shot or through put, is because this results in the most accurate result
for the type of operation I want to measure.

How can I use your tools to some comparison? If it is fixed to sample mode I can't. But why this limitation?

> > Just for the record: I am not the author of JMH, that is Aleksey Shipilev.
> > I am a humble user, just as you.
> Sure, thanks for your opinion. We're looking forward to hear some 
> comments from author, too.
> 
> > In my opinion, the current JSON contains lots of different data, so I would be tempted
> > to say it is actually meant to include "everything". Extending it in a compatible
> > way should be possible.
> >
> > My personal preference would be to have an output which contains every peace of gathered
> > information, so I can archive it and get back later and extract more things as I need it.
> >
> > Maybe you can show an example JSON output that you currently get from the existing
> > output and make a proposal what additional values should be included?
> > It is easier to discuss some data output or format with some examples.
> It's a big question how to represent the raw data. If you print all the 
> values one by one into current JSON file, you'll get huge amount of 
> numbers per iteration (like tens of thousands). In our patch we're 
> printing the values as pairs of value and quantity, which seems to be 
> more space saving. Also, our JSON format has no preatty indentation, 
> which adds a lot of spaces per line. However we don't want custom binary 
> format, because it's hard to check it by hand.

My bad! Now I understand what you are trying to do.

For example the ReadOnlyBenchmark yields 123378830 operations per seconds on latest hardware with four threads 
(with through put mode).  For 30 seconds iterations time that would be 3701 million data points. It's a mental example, of course it couldn't be gathered with sample mode that way because it would heavily skew the result. But if it would work, why I should want to analyze the runtime of each operation?

To be honest, the sample mode is the only mode I wonder how it can ever be of any use. That's
why I am confused about the fact that you say it's the only useful one.

Can you give an example?

Cheers,

Jens

-- 
"Everything superfluous is wrong!"

   // Jens Wilke - headissue GmbH - Germany
 \//  https://headissue.com