From rengels at ix.netcom.com Thu Sep 6 16:57:41 2018 From: rengels at ix.netcom.com (robert engels) Date: Thu, 6 Sep 2018 11:57:41 -0500 Subject: jmh avoid test? Message-ID: <016950F1-2310-4AA4-B61E-483E46532DB0@ix.netcom.com> Hello, jmh has the @Param but there doesn?t seem to be a way to restrict a benchmark to only certain param values. The simplest case would be to do a test in the benchmark method, and throw an UnsupportedStateException, but when you do that all of the rests of the tests abort? Maybe another annotation like, @Valid(argname=?list??) so that jmh can know which tests should be run ? Doesn?t seem to be an easy way to file an issue with OpenJDK on this. Regards. From ecki at zusammenkunft.net Fri Sep 7 10:34:24 2018 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Fri, 7 Sep 2018 10:34:24 +0000 Subject: jmh avoid test? In-Reply-To: <016950F1-2310-4AA4-B61E-483E46532DB0@ix.netcom.com> References: <016950F1-2310-4AA4-B61E-483E46532DB0@ix.netcom.com> Message-ID: Hello, If you specify the parameters on the command line the JMH run will only use the specified values. -p param1=10,20 If you want to restrict parameter combinations you would most likely have to start multiple runs. Gruss Bernd -- http://bernd.eckenfels.net ________________________________ Von: -993459040m Auftrag von Gesendet: Freitag, September 7, 2018 12:24 PM An: jmh-dev at openjdk.java.net Betreff: jmh avoid test? Hello, jmh has the @Param but there doesn?t seem to be a way to restrict a benchmark to only certain param values. The simplest case would be to do a test in the benchmark method, and throw an UnsupportedStateException, but when you do that all of the rests of the tests abort? Maybe another annotation like, @Valid(argname=?list??) so that jmh can know which tests should be run ? Doesn?t seem to be an easy way to file an issue with OpenJDK on this. Regards. From Alan.Burlison at oracle.com Wed Sep 12 16:48:57 2018 From: Alan.Burlison at oracle.com (Alan Burlison) Date: Wed, 12 Sep 2018 17:48:57 +0100 Subject: Setting classpath with benchmark.java Message-ID: <4dd2952a-4f47-d857-7f51-1d882dec9e83@oracle.com> I'm trying to run a benchmark which needs a specific classpath setting, to include a library that isn't available via Maven. Unfortunately it seems to be impossible to set the classpath with jmh. I'm using "-jvmArgs '-Djava.class.path=/opt/instantclient/ojdbc8.jar" to try to do so but here's the resulting command-line that's used to run the target VM by benchmark.java: Forking 10 times using command: [/usr/lib/jvm/oracle-java8-jdk-amd64/jre/bin/java, -cp, /opt/instantclient/ojdbc8.jar, -Djava.library.path=/opt/instantclient, -Dargs.mode=thin, -XX:CompileCommandFile=/tmp/jmh8572122209057614698compilecommand, -cp, target/benchmarks.jar, org.openjdk.jmh.runner.ForkedMain, 127.0.0.1, 43795] Because benchmark.java is appending "-cp target/benchmarks.jar" the second "-cp" wins and overrides the one specified with "-jvmArgs". I can't imagine I'm the first person to have needed to set the classpath for a jmh benchmark but I haven't been able to find any way around the problem, and it is a showstopper for me. Thanks, -- Alan Burlison -- From Alan.Burlison at oracle.com Wed Sep 12 16:51:36 2018 From: Alan.Burlison at oracle.com (Alan Burlison) Date: Wed, 12 Sep 2018 17:51:36 +0100 Subject: Setting classpath with benchmark.java In-Reply-To: <4dd2952a-4f47-d857-7f51-1d882dec9e83@oracle.com> References: <4dd2952a-4f47-d857-7f51-1d882dec9e83@oracle.com> Message-ID: benchmark.jar rather, duh... -- Alan Burlison -- From dmitry.timofeiev at gmail.com Fri Sep 14 05:44:39 2018 From: dmitry.timofeiev at gmail.com (Dmitry Timofeiev) Date: Fri, 14 Sep 2018 08:44:39 +0300 Subject: Setting classpath with benchmark.java In-Reply-To: <4dd2952a-4f47-d857-7f51-1d882dec9e83@oracle.com> References: <4dd2952a-4f47-d857-7f51-1d882dec9e83@oracle.com> Message-ID: Hi Alan, I don't have a solution to setting a classpath at runtime ? and it also seems the shade plugin does not support including non-maven artifacts [1]. Would it be possible in your case to install the library in your local (or company) artefact repository to have the shade plugin include it into benchmark.jar as a dependency of your SuT/benchmark project? [1] https://maven.apache.org/plugins/maven-shade-plugin/examples/includes-excludes.html [2] https://stackoverflow.com/a/27646585/1792188 Cheers, Dmitry On 12.09.18 19:48, Alan Burlison wrote: > I'm trying to run a benchmark which needs a specific classpath > setting, to include a library that isn't available via Maven. > > Unfortunately it seems to be impossible to set the classpath with jmh. > I'm using "-jvmArgs '-Djava.class.path=/opt/instantclient/ojdbc8.jar" > to try to do so but here's the resulting command-line that's used to > run the target VM by benchmark.java: > > Forking 10 times using command: > [/usr/lib/jvm/oracle-java8-jdk-amd64/jre/bin/java, -cp, > /opt/instantclient/ojdbc8.jar, -Djava.library.path=/opt/instantclient, > -Dargs.mode=thin, > -XX:CompileCommandFile=/tmp/jmh8572122209057614698compilecommand, -cp, > target/benchmarks.jar, org.openjdk.jmh.runner.ForkedMain, 127.0.0.1, > 43795] > > Because benchmark.java is appending "-cp target/benchmarks.jar" the > second "-cp" wins and overrides the one specified with "-jvmArgs". I > can't imagine I'm the first person to have needed to set the classpath > for a jmh benchmark but I haven't been able to find any way around the > problem, and it is a showstopper for me. > > Thanks, > From erich at debian.org Wed Sep 26 14:08:58 2018 From: erich at debian.org (Erich Schubert) Date: Wed, 26 Sep 2018 16:08:58 +0200 Subject: JMH numeric precision of auxiliary counters only 8 digits Message-ID: <72b235b7-cd02-cff8-136d-225acc6d8689@debian.org> The Aux Counters - even when long counters - are only written to the JSON with float precision, with 8 digits of precision. For large values, this loses some precision. Given that the aux counter API only allows integer and long, printing the raw values as float does not make sense. We are currently doing some memory measurements using aux counters in @TearDown, and these need more than 8 digits... Regards, Erich P.S. I also provided a patch in July to not print useless quantiles: http://mail.openjdk.java.net/pipermail/jmh-dev/2018-July/002755.html but I never got an answer... Furthermore, I send an email to report that Aux counters should be enabled for single-shot benchmarks, it wasn't even moderated yet? Is there any actual issue tracker (github?) for JMH? From shade at redhat.com Wed Sep 26 14:17:06 2018 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 26 Sep 2018 16:17:06 +0200 Subject: JMH numeric precision of auxiliary counters only 8 digits In-Reply-To: <72b235b7-cd02-cff8-136d-225acc6d8689@debian.org> References: <72b235b7-cd02-cff8-136d-225acc6d8689@debian.org> Message-ID: <25a5e30f-852e-c10f-0699-e3c5ed2f5926@redhat.com> Hi, On 09/26/2018 04:08 PM, Erich Schubert wrote: > The Aux Counters - even when long counters - are only written to the JSON with float precision, > with 8 digits of precision. For large values, this loses some precision. Given that the aux > counter API only allows integer and long, printing the raw values as float does not make sense. Not sure I understand without trying the examples. Are you saying that large long values are getting cut? Can you show the sample value that is erroneously printed in JSON? > We are currently doing some memory measurements using aux counters in @TearDown, and these need more > than 8 digits... Any particular reason why JMH profilers are not usable for memory measurements like these? We are not paying much attention to AuxCounters (as mentioned in its Javadoc, it is experimental API), thinking that most data should be acquirable by other means. > I also provided a patch in July to not print useless quantiles: > http://mail.openjdk.java.net/pipermail/jmh-dev/2018-July/002755.html > > but I never got an answer... I'll reply later. > Furthermore, I send an email to report that Aux counters should be enabled for single-shot > benchmarks, it wasn't even moderated yet? > > Is there any actual issue tracker (github?) for JMH? There is codetools/jmh at bugs.openjdk.java.net, but it is unfortunately only writable by OpenJDK people, starting from Authors. -Aleksey From shade at redhat.com Wed Sep 26 14:33:33 2018 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 26 Sep 2018 16:33:33 +0200 Subject: Patch: do not report percentiles that are just the maximum In-Reply-To: <6c36df06-05cb-8dc7-d9f5-8fc4e9ce1da8@gmail.com> References: <6c36df06-05cb-8dc7-d9f5-8fc4e9ce1da8@gmail.com> Message-ID: <29344f1a-991e-0ee4-575c-438d6e5a64cc@redhat.com> On 07/04/2018 05:32 PM, Erich Schubert wrote: > The attached patch adds a threshold for the reported quantiles. > > For example, for 10 measurements, it will report the 50% and 90% quantiles, but it will no longer > report the 95%, 99%, 99.9%, 99.99%, 99.999% and 99.9999% quantiles as before, which were simply > equal to the maximum, so completely redundant information & misleading. > > The logic is straightforward. getMaxPercentile() returns the maximum percentile supported by the > data, assumed to be the location of the last measurement on the empirical CDF. For anything larger > than this, the getPercentile() function will simply return the maximum. This makes sense. Let me see if that works. Contribution wrinkle: OpenJDK patch acceptance rules need your OCA, as per http://openjdk.java.net/contribute. Once you appear on this list: https://www.oracle.com/technetwork/community/oca-486395.html -- we can accept the patch as is. -Aleksey From shade at redhat.com Wed Sep 26 14:33:53 2018 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 26 Sep 2018 16:33:53 +0200 Subject: Setting classpath with benchmark.java In-Reply-To: <4dd2952a-4f47-d857-7f51-1d882dec9e83@oracle.com> References: <4dd2952a-4f47-d857-7f51-1d882dec9e83@oracle.com> Message-ID: <74e613b4-f65e-c674-885a-cf3741feb946@redhat.com> On 09/12/2018 06:48 PM, Alan Burlison wrote: > I'm trying to run a benchmark which needs a specific classpath setting, to include a library that > isn't available via Maven. > > Unfortunately it seems to be impossible to set the classpath with jmh. I'm using "-jvmArgs > '-Djava.class.path=/opt/instantclient/ojdbc8.jar" to try to do so but here's the resulting > command-line that's used to run the target VM by benchmark.java: It is not possible to combine runnable JAR and classpath settings. It is possible to improve JMH to accept classpath parameter, but the usual workflow is to make every library available via the build system (whether it is Maven or Gradle) and shaded into the final benchmark. This also explains why this is not a showstopper for most users. -Aleksey From shade at redhat.com Wed Sep 26 14:58:36 2018 From: shade at redhat.com (shade at redhat.com) Date: Wed, 26 Sep 2018 14:58:36 +0000 Subject: hg: code-tools/jmh: 7902329: perf profiler should accept events Message-ID: <201809261458.w8QEwadl025795@aojmv0008.oracle.com> Changeset: 66fb723292d4 Author: shade Date: 2018-09-26 16:57 +0200 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/66fb723292d4 7902329: perf profiler should accept events Contributed-by: Eric Caspole ! jmh-core/src/main/java/org/openjdk/jmh/profile/LinuxPerfProfiler.java From shade at redhat.com Wed Sep 26 14:59:41 2018 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 26 Sep 2018 16:59:41 +0200 Subject: [Patch] Pass counter values to -prof perf In-Reply-To: <07b69cdf-b810-ce11-8eff-d485c999487b@oracle.com> References: <07b69cdf-b810-ce11-8eff-d485c999487b@oracle.com> Message-ID: <53d7ebc4-3636-3d3d-3f9f-822fc6923fbc@redhat.com> On 08/22/2018 10:49 PM, Eric Caspole wrote: > I have a use case with JMH on new hardware where I need to pass the counters I want to see to -prof > perf, I would like to contribute a patch for this below: > > ?http://cr.openjdk.java.net/~ecaspole/jmh_stat_counters/webrev/ Thanks, pushed with some corrections under: https://bugs.openjdk.java.net/browse/CODETOOLS-7902329 Note that perfnorm is probably more useful than just perf in most cases. -Aleksey From eric.caspole at oracle.com Wed Sep 26 15:24:37 2018 From: eric.caspole at oracle.com (Eric Caspole) Date: Wed, 26 Sep 2018 11:24:37 -0400 Subject: [Patch] Pass counter values to -prof perf In-Reply-To: <53d7ebc4-3636-3d3d-3f9f-822fc6923fbc@redhat.com> References: <07b69cdf-b810-ce11-8eff-d485c999487b@oracle.com> <53d7ebc4-3636-3d3d-3f9f-822fc6923fbc@redhat.com> Message-ID: <4b2e9862-3db1-ef5d-8b0b-483c29b6a195@oracle.com> On 9/26/2018 10:59 AM, Aleksey Shipilev wrote: > On 08/22/2018 10:49 PM, Eric Caspole wrote: >> I have a use case with JMH on new hardware where I need to pass the counters I want to see to -prof >> perf, I would like to contribute a patch for this below: >> >> ?http://cr.openjdk.java.net/~ecaspole/jmh_stat_counters/webrev/ > Thanks, pushed with some corrections under: > https://bugs.openjdk.java.net/browse/CODETOOLS-7902329 > > Note that perfnorm is probably more useful than just perf in most cases. Thanks, I hadn't noticed that, I will try it. Eric > -Aleksey > From erich at debian.org Wed Sep 26 15:42:45 2018 From: erich at debian.org (Erich Schubert) Date: Wed, 26 Sep 2018 17:42:45 +0200 Subject: Patch: do not report percentiles that are just the maximum In-Reply-To: <29344f1a-991e-0ee4-575c-438d6e5a64cc@redhat.com> References: <6c36df06-05cb-8dc7-d9f5-8fc4e9ce1da8@gmail.com> <29344f1a-991e-0ee4-575c-438d6e5a64cc@redhat.com> Message-ID: Hi, Quick follow up to my previous email today (numeric resolution of aux counters): I probably was wrong - the counters may simply have tailing 0s, so the precision should be fine within double precision bounds (good enough for me). > This makes sense. Let me see if that works. > > Contribution wrinkle: OpenJDK patch acceptance rules need your OCA, as per > http://openjdk.java.net/contribute. Once you appear on this list: > https://www.oracle.com/technetwork/community/oca-486395.html -- we can accept the patch as is. Sorry, I can't sign this general waiver. I cannot trust Oracle - they've ruined too many projects already (OpenOffice, OpenSolaris, MySQL, ...). But the actual changes are trivial enough to not require a contributors agreement. public double getMaxPercentile() { return 100 - 100. / (count + 1); } clearly is a trivial contribution, and does not need an CA. And you may want to refactor the redundancies in the percentile loops anyway, and probably want to refresh the unit tests yourself anyway. If there is only a single sample (e.g. in single shot mode), it may also be desirable to omit the entire percentiles section. Regards, Erich From davidmichaelkarr at gmail.com Sat Sep 29 15:40:51 2018 From: davidmichaelkarr at gmail.com (David Karr) Date: Sat, 29 Sep 2018 08:40:51 -0700 Subject: Average time computation just prints out "10??" Message-ID: I was able to get throughput numbers for benchmarks, but now I want to see average times for them. They are pretty short blocks, so it would be nice if I could get them rendered in microseconds, but I don't see an obvious way to do that. However, even before trying to configure the units, I'm trying to understand the output I'm getting. The results I get all say "10?? s/op". I thought perhaps this was trying to render unicode, but when I stored the output in a file and inspected it, I saw that those really are "?" characters. What can I do about this? From ecki at zusammenkunft.net Sat Sep 29 18:28:31 2018 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Sat, 29 Sep 2018 18:28:31 +0000 Subject: Average time computation just prints out "10??" In-Reply-To: References: Message-ID: It does use unicode characters (mu and plusminus), you need to make sure your console encoding which is reflected in the platform encoding actually supports this. Besides that can you show some samples, the unit annotation works in my experience for the aggregate tables at the end of the run. Gruss Bernd Gruss Bernd -- http://bernd.eckenfels.net ________________________________ Von: -2083622320m Auftrag von Gesendet: Samstag, September 29, 2018 6:21 PM An: jmh-dev at openjdk.java.net Betreff: Average time computation just prints out "10??" I was able to get throughput numbers for benchmarks, but now I want to see average times for them. They are pretty short blocks, so it would be nice if I could get them rendered in microseconds, but I don't see an obvious way to do that. However, even before trying to configure the units, I'm trying to understand the output I'm getting. The results I get all say "10?? s/op". I thought perhaps this was trying to render unicode, but when I stored the output in a file and inspected it, I saw that those really are "?" characters. What can I do about this? From ecki at zusammenkunft.net Sat Sep 29 23:57:05 2018 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Sat, 29 Sep 2018 23:57:05 +0000 Subject: Average time computation just prints out "10??" In-Reply-To: References: <5bafe795.1c69fb81.376c4.f343@mx.google.com>, Message-ID: I think on Windows the encoding depends on the non-Unicode setting which controls then OEM codepage. JMH tries to read that from the console class, maybe you use a modular Java where this does not work? Or maybe it?s a cygwin Problem, not sure. You can try to set -Dfile.encoding=utf-8 Gruss Bernd -- http://bernd.eckenfels.net ________________________________ Von: David Karr Gesendet: Sonntag, September 30, 2018 1:13 AM An: ecki at zusammenkunft.net Betreff: Re: Average time computation just prints out "10??" On Sat, Sep 29, 2018 at 1:59 PM Bernd Eckenfels > wrote: On reasonalbe current and complete Linux it should work with LANG=?en_US.UTF8? Java -jar target/microbenchmarks.jar Unfortunately, my desktop is Windows 7. I do run with Cygwin, however. Unfortunately, this command line makes no difference. Same result. Greetings Bernd -- http://bernd.eckenfels.net Von: David Karr Gesendet: Samstag, 29. September 2018 20:37 An: ecki at zusammenkunft.net Betreff: Re: Average time computation just prints out "10??" On Sat, Sep 29, 2018 at 11:28 AM Bernd Eckenfels > wrote: It does use unicode characters (mu and plusminus), you need to make sure your console encoding which is reflected in the platform encoding actually supports this. Yeah, that appears to be the key. I have no idea how to do that. Besides that can you show some samples, the unit annotation works in my experience for the aggregate tables at the end of the run. There are no surprises here. --------------- Benchmark Mode Cnt Score Error Units MyBenchmark.stuff1 avgt 25 ? 10?? s/op MyBenchmark.stuff2 avgt 25 ? 10?? s/op -------------------- Gruss Bernd Gruss Bernd -- http://bernd.eckenfels.net Von: -2083622320m Auftrag von Gesendet: Samstag, September 29, 2018 6:21 PM An: jmh-dev at openjdk.java.net Betreff: Average time computation just prints out "10??" I was able to get throughput numbers for benchmarks, but now I want to see average times for them. They are pretty short blocks, so it would be nice if I could get them rendered in microseconds, but I don't see an obvious way to do that. However, even before trying to configure the units, I'm trying to understand the output I'm getting. The results I get all say "10?? s/op". I thought perhaps this was trying to render unicode, but when I stored the output in a file and inspected it, I saw that those really are "?" characters. What can I do about this? From davidmichaelkarr at gmail.com Sun Sep 30 01:02:40 2018 From: davidmichaelkarr at gmail.com (David Karr) Date: Sat, 29 Sep 2018 18:02:40 -0700 Subject: Average time computation just prints out "10??" In-Reply-To: References: <5bafe795.1c69fb81.376c4.f343@mx.google.com> Message-ID: On Sat, Sep 29, 2018 at 4:57 PM Bernd Eckenfels wrote: > I think on Windows the encoding depends on the non-Unicode setting which > controls then OEM codepage. JMH tries to read that from the console class, > maybe you use a modular Java where this does not work? Or maybe it?s a > cygwin Problem, not sure. You can try to set -Dfile.encoding=utf-8 > I guess that's an improvement. I still can't read it in my text editor, but I can paste it into my browser and see what it was supposed to be printing. This produces a text line like this: ----------------- # Warmup Iteration 1: ? 10?? s/op ----------------- Why did the JMH implementors decide to introduce the complexity of unicode into this? This should be simple textual information. I shouldn't have to work this hard to get this. > Gruss > Bernd > -- > http://bernd.eckenfels.net > > ------------------------------ > *Von:* David Karr > *Gesendet:* Sonntag, September 30, 2018 1:13 AM > *An:* ecki at zusammenkunft.net > *Betreff:* Re: Average time computation just prints out "10??" > > On Sat, Sep 29, 2018 at 1:59 PM Bernd Eckenfels > wrote: > >> On reasonalbe current and complete Linux it should work with >> >> >> >> LANG=?en_US.UTF8? Java -jar target/microbenchmarks.jar >> > > Unfortunately, my desktop is Windows 7. I do run with Cygwin, however. > Unfortunately, this command line makes no difference. Same result. > > >> >> Greetings >> >> Bernd >> >> -- >> http://bernd.eckenfels.net >> >> >> >> *Von: *David Karr >> *Gesendet: *Samstag, 29. September 2018 20:37 >> *An: *ecki at zusammenkunft.net >> *Betreff: *Re: Average time computation just prints out "10??" >> >> >> >> >> >> On Sat, Sep 29, 2018 at 11:28 AM Bernd Eckenfels >> wrote: >> >> It does use unicode characters (mu and plusminus), you need to make sure >> your console encoding which is reflected in the platform encoding actually >> supports this. >> >> >> >> Yeah, that appears to be the key. I have no idea how to do that. >> >> >> >> >> >> Besides that can you show some samples, the unit annotation works in my >> experience for the aggregate tables at the end of the run. >> >> >> >> There are no surprises here. >> >> --------------- >> >> Benchmark Mode Cnt Score >> Error Units >> MyBenchmark.stuff1 avgt 25 ? 10?? s/op >> MyBenchmark.stuff2 avgt 25 ? 10?? s/op >> >> -------------------- >> >> >> >> Gruss >> >> Bernd >> >> >> >> Gruss >> >> Bernd >> >> -- >> >> http://bernd.eckenfels.net >> >> >> >> *Von:* -2083622320m Auftrag von >> *Gesendet:* Samstag, September 29, 2018 6:21 PM >> *An:* jmh-dev at openjdk.java.net >> *Betreff:* Average time computation just prints out "10??" >> >> >> >> I was able to get throughput numbers for benchmarks, but now I want to >> see >> average times for them. They are pretty short blocks, so it would be nice >> if I could get them rendered in microseconds, but I don't see an obvious >> way to do that. >> >> However, even before trying to configure the units, I'm trying to >> understand the output I'm getting. The results I get all say "10?? s/op". >> I thought perhaps this was trying to render unicode, but when I stored >> the >> output in a file and inspected it, I saw that those really are "?" >> characters. >> >> What can I do about this? >> >> >> > From shade at redhat.com Sun Sep 30 11:33:59 2018 From: shade at redhat.com (Aleksey Shipilev) Date: Sun, 30 Sep 2018 13:33:59 +0200 Subject: Average time computation just prints out "10??" In-Reply-To: References: <5bafe795.1c69fb81.376c4.f343@mx.google.com> Message-ID: <8519bb49-4ea6-5a5e-ca75-eadf52db2949@redhat.com> On 09/30/2018 03:02 AM, David Karr wrote: > On Sat, Sep 29, 2018 at 4:57 PM Bernd Eckenfels > wrote: > I guess that's an improvement. I still can't read it in my text editor, but > I can paste it into my browser and see what it was supposed to be > printing. This produces a text line like this: > ----------------- > # Warmup Iteration 1: ? 10?? s/op > ----------------- > > Why did the JMH implementors decide to introduce the complexity of unicode > into this? This should be simple textual information. I shouldn't have to > work this hard to get this. Maintainer's perspective: in the year 2018, JMH or any other power user tool should be not be held hostage by environments that do not properly support Unicode. Notice, for example, how browsers and mail clients show this correctly. Anyway, "? 10?? s/op" is the sign that the units selected are very far off default precision. The order of magnitude gives you the hint by how much, without letting you use the low-precision data. Or, as in this case, raise the question what is printed exactly -- think about this as a glorified Not-A-Number value. In this example, time units should be microseconds, not seconds. -Aleksey From schubert at informatik.uni-heidelberg.de Fri Sep 14 12:08:50 2018 From: schubert at informatik.uni-heidelberg.de (Erich Schubert) Date: Fri, 14 Sep 2018 12:08:50 -0000 Subject: AuxCounter support missing in Single Shot benchmarks Message-ID: AFAICT, AuxCounters do not work in single shot benchmarks. Why? IMHO there are useful scenarios, for example when benchmarking the amount of memory used by particular data structures: ??? @AuxCounters(AuxCounters.Type.EVENTS) ??? @State(Scope.Thread) ??? public static class Memory { ??? ??? public long totalsize; ??? } ??? /** ??? ?* Memory measurement at teardown. ??? ?* ??? ?* @param memory ??? ?*??????????? Memory counter ??? ?*/ ??? @TearDown(Level.Iteration) ??? public void memory(Memory memory) { ??? ??? memory.totalsize = GraphLayout.parseInstance(datastructure).totalSize(); ??? } this seems to work fine with throughput benchmarks, but not with single shot. The teardown method is called, but the counters are not reported. But clearly for measuring deterministic things like memory of different data structures (which tend to not be micro-benchmarks) a single shot measurement may be fine.