Why I am getting weird benchmark results between 0.99 and 100 percentile

[Michal Cervenansky]

Hi,

from the code I am not sure what precisely is going on, but during
benchmarking speed of the code might differ drastically. That is why I feel
like the median(p50) is the best metric and can be a good idea to combine
it with average duration or throughput. The maximum(p1.00) is just a single
execution and the p0.9999 is also affected by a small number of executions.
It can be affected by JVM warming up the code, garbage collector, network
latency if you use network or just something going on in the operating
system. It is a good idea to have some kind of notion of what would the
histogram of test results look like, but for me the median is the best
metric.

Michal

On Thu, May 6, 2021 at 7:19 AM Sarath Babu <sarath.chalasani46 at gmail.com>
wrote:

> Hi Team,
>
> When I run the JMH benchmark results using simpletime mode getting weird
> benchmark results between 0.99 and 100 percentile, could you please provide
> me a reasons behind.
>
> *Results* :
> Benchmark                                                            Mode
>      Cnt     Score   Error  Units
> CacheTest.measureCacheGet                                          sample
>  152939771     0.050 ± 0.001  ms/op
> CacheTest.measureCacheGet:measureCacheGet·p0.00                    sample
>              0.001          ms/op
> CacheTest.measureCacheGet:measureCacheGet·p0.50                    sample
>              0.001          ms/op
> CacheTest.measureCacheGet:measureCacheGet·p0.90                    sample
>              0.002          ms/op
> CacheTest.measureCacheGet:measureCacheGet·p0.95                    sample
>              0.003          ms/op
> CacheTest.measureCacheGet:measureCacheGet·p0.99                    sample
>              0.003          ms/op
> CacheTest.measureCacheGet:measureCacheGet·p0.999                   sample
>              0.006          ms/op
> CacheTest.measureCacheGet:measureCacheGet·p0.9999                  sample
>            136.839          ms/op
> CacheTest.measureCacheGet:measureCacheGet·p1.00                    sample
>           4487.905          ms/op
>
> *Code snippet:*
>
> @Benchmark
>
> @BenchmarkMode({ Mode.*SampleTime* })
>
> @OutputTimeUnit(TimeUnit.*MILLISECONDS*)
>
> @Warmup(iterations = 3)
>
> @Measurement(iterations = 10)
>
> *public* *void* measureCacheGet(CachePut cPut, RandomNumberBelowLakh rNum)
> *throws* InterruptedException {
>
> cPut.cache.get(rNum.key);
>
> }
>
>
>         @State(Scope.*Thread*)
>
> *public* *static* *class* RandomNumberBelowLakh {
>
>
> *public* *long* key;
>
>
> @Setup(Level.*Invocation*)
>
> *public* *void* doSetup() {
>
> key = getRandomNumber();
>
> }
>
>
> @TearDown(Level.*Invocation*)
>
> *public* *void* doTearDown() {
>
> key=0;
>
> }
>
> *public* *long* getRandomNumber() {
>
> *return* (*long*) ((Math.*random*() * (100000 - 1)) + 1);
>
> }
>
> }
>
>
> @State(Scope.*Benchmark*)
>
> *public* *static* *class* CachePut {
>
>
> *public* CacheManagerImpl impl;
>
> *public* Cache cache;
>
>
> @Setup
>
> *public* *void* setup() {
>
> impl = *new* CacheManagerImpl();
>
> cache = impl.getCache("account");
>
> cachePutAccountObjects();
>
> }
>
>
> *public* *void* cachePutAccountObjects() {
>
> *for* (*long* i = 1; i <= 100000; i++) {
>
> cache.put(i, AccountUtil.*getAccount*(i));
>
> }
>
> }
>
>
> @TearDown
>
> *public* *void* doTearDown() {
>
> impl = *null*;
>
> cache = *null*;
>
> }
>
> }
>
>
>
>
> Thanks and Regards
> C.Sarathbabu
> Cont No: 9652511597
>
>