From aleksey.shipilev at oracle.com Wed Dec 3 12:13:17 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Wed, 03 Dec 2014 15:13:17 +0300 Subject: Passing current fork number to profiler In-Reply-To: References: Message-ID: <547EFE5D.9010602@oracle.com> On 11/28/2014 03:12 AM, Aggelos Biboudis wrote: > What do you think of the attached patch? Note, that the added field is not > final. I don't like BenchmarkParams becoming mutable. By construction, BenchmarkParams is the immutable runtime configuration for the benchmark. It should not hold the transient data, e.g. current fork ID. If anything, another (new) object class should encapsulate all the transient run info. A great refactoring lies that way: this object class should also contain the current iteration ids, which forces us to reconsider how the control flows through the harness. In current code, iterations are handled not in the Runner, but in BenchmarkHandler, which can be run solo in the forked VM. Thanks, -Aleksey. From weijun.wang at oracle.com Thu Dec 4 04:09:49 2014 From: weijun.wang at oracle.com (Wang Weijun) Date: Thu, 4 Dec 2014 12:09:49 +0800 Subject: Is it reasonable to compare outputs between JMH and hprof? Message-ID: <0823DBD6-4049-4BA3-8219-13FF72427805@oracle.com> Hi All I am comparing the difference of SHA-1 and SHA-256. First I wrote a JMH benchmark: @Benchmark public void sig1(Blackhole bh) throws Exception { bh.consume(sig("SHA-1")); } @Benchmark public void sig2(Blackhole bh) throws Exception { bh.consume(sig("SHA-256")); } byte[] sig(String alg) throws Exception { MessageDigest md = MessageDigest.getInstance(alg); md.update(new byte[10000]); return md.digest(); } The output is Benchmark Mode Samples Score Error Units o.o.b.Weird.sig1 thrpt 5 20984.435 ? 3356.455 ops/s o.o.b.Weird.sig2 thrpt 5 13130.330 ? 976.824 ops/s so the difference is there but not huge. Then I wrote a simple app with public static void main(String args[]) throws Exception { int i = Arrays.hashCode(sig("SHA-1")); i += Arrays.hashCode(sig("SHA-256")); System.out.println(i); } static byte[] sig(String alg) throws Exception { MessageDigest md = MessageDigest.getInstance(alg); md.update(new byte[10000]); return md.digest(); } and then profile it with -agentlib:hprof=cpu=times, and get SHA2 1 10.16% 10.16% 156 303276 sun.security.provider.SHA2.implCompress SHA2 2 6.91% 17.07% 9984 303274 sun.security.provider.SHA2.lf_sigma0 SHA2 3 5.28% 22.36% 9984 303271 sun.security.provider.SHA2.lf_sigma1 SHA2 4 4.61% 26.96% 7488 303269 sun.security.provider.SHA2.lf_delta0 SHA2 5 4.20% 31.17% 29952 303273 sun.security.provider.SHA2.lf_S SHA2 7 3.79% 39.16% 7488 303266 sun.security.provider.SHA2.lf_delta1 SHA2 9 2.85% 44.99% 29952 303270 sun.security.provider.SHA2.lf_S SHA2 13 1.90% 54.47% 14976 303267 sun.security.provider.SHA2.lf_S SHA2 17 1.49% 61.25% 14976 303264 sun.security.provider.SHA2.lf_S SHA2 22 0.81% 66.12% 7488 303265 sun.security.provider.SHA2.lf_R SHA2 23 0.81% 66.94% 9984 303275 sun.security.provider.SHA2.lf_maj SHA2 25 0.81% 68.56% 156 303263 sun.security.provider.ByteArrayAccess.b2iBig64 SHA2 27 0.68% 70.05% 9984 303272 sun.security.provider.SHA2.lf_ch SHA2 31 0.54% 72.63% 7488 303268 sun.security.provider.SHA2.lf_R SHA1 34 0.54% 74.25% 156 303224 sun.security.provider.SHA.implCompress SHA1 43 0.41% 78.05% 156 303223 sun.security.provider.ByteArrayAccess.b2iBig64 SHA2 60 0.27% 82.66% 2496 303262 java.lang.Integer.reverseBytes SHA2 61 0.27% 82.93% 64 303290 sun.security.provider.SHA2.lf_sigma1 SHA1 116 0.14% 91.06% 2496 303222 java.lang.Integer.reverseBytes These are calls with SHA (i.e. SHA1) or SHA2 in the stack (depth=4), and time for SHA2 vs SHA1 is 45.38% vs 1.09%. With such a small app I don't think SHA or SHA2 is called anywhere else. This is jdk9 b40. Why is the output so different from JMH? Is it reasonable comparing them? Thanks Max From ecki at zusammenkunft.net Thu Dec 4 05:38:20 2014 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Thu, 4 Dec 2014 06:38:20 +0100 Subject: Is it reasonable to compare outputs between JMH and hprof? In-Reply-To: <0823DBD6-4049-4BA3-8219-13FF72427805@oracle.com> References: <0823DBD6-4049-4BA3-8219-13FF72427805@oracle.com> Message-ID: <20141204063820.000063ce.ecki@zusammenkunft.net> Hello, I would use the power of JMH to test this with different array sizes and with and without provider instantiation. I did that a while back for MDC/MAC, and SHA256 is not that attractive anymore as SHA512 is faster (and with SHA512-256 not bigger) (Java7Windows7:) https://gist.github.com/ecki/a4d598c4231a0165d0a1 Benchmark (bufsize) Mode Samples Score Score error Units n.e.t.j.HashBenchmark.testHmacSHA1 1 avgt 3 0,005 0,001 ms/op n.e.t.j.HashBenchmark.testHmacSHA1 1024 avgt 3 0,015 0,005 ms/op n.e.t.j.HashBenchmark.testHmacSHA1 1048576 avgt 3 9,479 1,836 ms/op n.e.t.j.HashBenchmark.testHmacSHA256 1 avgt 3 0,006 0,003 ms/op n.e.t.j.HashBenchmark.testHmacSHA256 1024 avgt 3 0,021 0,010 ms/op n.e.t.j.HashBenchmark.testHmacSHA256 1048576 avgt 3 14,211 2,847 ms/op n.e.t.j.HashBenchmark.testHmacSHA512 1 avgt 3 0,009 0,002 ms/op n.e.t.j.HashBenchmark.testHmacSHA512 1024 avgt 3 0,021 0,011 ms/op n.e.t.j.HashBenchmark.testHmacSHA512 1048576 avgt 3 11,727 11,595 ms/op n.e.t.j.HashBenchmark.testMD5 1 avgt 3 0,001 0,001 ms/op n.e.t.j.HashBenchmark.testMD5 1024 avgt 3 0,007 0,001 ms/op n.e.t.j.HashBenchmark.testMD5 1048576 avgt 3 6,102 0,360 ms/op n.e.t.j.HashBenchmark.testMD5Reuse 1 avgt 3 0,000 0,000 ms/op n.e.t.j.HashBenchmark.testMD5Reuse 1024 avgt 3 0,007 0,004 ms/op n.e.t.j.HashBenchmark.testMD5Reuse 1048576 avgt 3 6,210 2,319 ms/op n.e.t.j.HashBenchmark.testSHA1 1 avgt 3 0,001 0,003 ms/op n.e.t.j.HashBenchmark.testSHA1 1024 avgt 3 0,011 0,018 ms/op n.e.t.j.HashBenchmark.testSHA1 1048576 avgt 3 9,446 0,750 ms/op n.e.t.j.HashBenchmark.testSHA1Reuse 1 avgt 3 0,001 0,000 ms/op n.e.t.j.HashBenchmark.testSHA1Reuse 1024 avgt 3 0,010 0,002 ms/op n.e.t.j.HashBenchmark.testSHA1Reuse 1048576 avgt 3 9,455 3,341 ms/op n.e.t.j.HashBenchmark.testSHA256 1 avgt 3 0,002 0,000 ms/op n.e.t.j.HashBenchmark.testSHA256 1024 avgt 3 0,016 0,008 ms/op n.e.t.j.HashBenchmark.testSHA256 1048576 avgt 3 13,938 2,390 ms/op n.e.t.j.HashBenchmark.testSHA256Reuse 1 avgt 3 0,001 0,001 ms/op n.e.t.j.HashBenchmark.testSHA256Reuse 1024 avgt 3 0,017 0,007 ms/op n.e.t.j.HashBenchmark.testSHA256Reuse 1048576 avgt 3 15,431 14,422 ms/op n.e.t.j.HashBenchmark.testSHA512 1 avgt 3 0,003 0,001 ms/op n.e.t.j.HashBenchmark.testSHA512 1024 avgt 3 0,015 0,010 ms/op n.e.t.j.HashBenchmark.testSHA512 1048576 avgt 3 11,532 1,528 ms/op n.e.t.j.HashBenchmark.testSHA512Reuse 1 avgt 3 0,002 0,000 ms/op n.e.t.j.HashBenchmark.testSHA512Reuse 1024 avgt 3 0,013 0,009 ms/op n.e.t.j.HashBenchmark.testSHA512Reuse 1048576 avgt 3 10,801 2,124 ms/op But to your question. They way the Profiler samples the stack and the fact that SHA1 and SHA2 are implementations with different depths, they are hard to compare. You would actually have to compare the time spent in high level methods (main/sig/update) not the low level code. And you need to run the test longer otherwise you see only startup/shutdown samples. Besides that Profilers greatly influence the runtime and behavior, its not at all useable for micro benchmarkings (at least not unless you have less intrusive profilers like cpu counter assisted kernel stuff (Linux perf)). Somewhat related a recent article: http://psy-lob-saw.blogspot.de/2014/12/the-escape-of-arraylistiterator.html I see this: public static void main(String args[]) throws Exception { int i=0; MessageDigest md256 = MessageDigest.getInstance("SHA-256"); MessageDigest md1 = MessageDigest.getInstance("SHA-1"); for(int t=0;t<100;t++) { i = Arrays.hashCode(sig1(md1)); i += Arrays.hashCode(sig256(md256));} System.out.println(i); } static byte[] sig1(MessageDigest md) throws Exception { md.update(new byte[10000]); return md.digest(); } static byte[] sig256(MessageDigest md) throws Exception { md.update(new byte[10000]); return md.digest(); } rank self accum count trace method 23 0.05% 99.25% 100 303130 sun.security.provider.DigestBase.engineUpdate ... 27 0.05% 99.45% 100 303096 sun.security.provider.DigestBase.engineUpdate TRACE 303130: sun.security.provider.DigestBase.engineUpdate(DigestBase.java:Unknown line) java.security.MessageDigest$Delegate.engineUpdate(MessageDigest.java:Unknown line) java.security.MessageDigest.update(MessageDigest.java:Unknown line) net.eckenfels.test.jmh.HashMain.sig256(HashMain.java:Unknown line) net.eckenfels.test.jmh.HashMain.main(HashMain.java:Unknown line) TRACE 303096: sun.security.provider.DigestBase.engineUpdate(DigestBase.java:Unknown line) java.security.MessageDigest$Delegate.engineUpdate(MessageDigest.java:Unknown line) java.security.MessageDigest.update(MessageDigest.java:Unknown line) net.eckenfels.test.jmh.HashMain.sig1(HashMain.java:Unknown line) net.eckenfels.test.jmh.HashMain.main(HashMain.java:Unknown line) The test runs 1s without and 40s with profiler. That does tell all IMHO. Gruss Bernd Am Thu, 4 Dec 2014 12:09:49 +0800 schrieb Wang Weijun : > Hi All > > I am comparing the difference of SHA-1 and SHA-256. First I wrote a > JMH benchmark: > > @Benchmark > public void sig1(Blackhole bh) throws Exception { > bh.consume(sig("SHA-1")); > } > > @Benchmark > public void sig2(Blackhole bh) throws Exception { > bh.consume(sig("SHA-256")); > } > > byte[] sig(String alg) throws Exception { > MessageDigest md = MessageDigest.getInstance(alg); > md.update(new byte[10000]); > return md.digest(); > } > > The output is > > Benchmark Mode Samples Score Error Units > o.o.b.Weird.sig1 thrpt 5 20984.435 ? 3356.455 ops/s > o.o.b.Weird.sig2 thrpt 5 13130.330 ? 976.824 ops/s > > so the difference is there but not huge. > > Then I wrote a simple app with > > public static void main(String args[]) throws Exception { > int i = Arrays.hashCode(sig("SHA-1")); > i += Arrays.hashCode(sig("SHA-256")); > System.out.println(i); > } > > static byte[] sig(String alg) throws Exception { > MessageDigest md = MessageDigest.getInstance(alg); > md.update(new byte[10000]); > return md.digest(); > } > > and then profile it with -agentlib:hprof=cpu=times, and get > > SHA2 1 10.16% 10.16% 156 303276 > sun.security.provider.SHA2.implCompress SHA2 2 6.91% 17.07% > 9984 303274 sun.security.provider.SHA2.lf_sigma0 SHA2 3 5.28% > 22.36% 9984 303271 sun.security.provider.SHA2.lf_sigma1 SHA2 4 > 4.61% 26.96% 7488 303269 sun.security.provider.SHA2.lf_delta0 > SHA2 5 4.20% 31.17% 29952 303273 > sun.security.provider.SHA2.lf_S SHA2 7 3.79% 39.16% 7488 > 303266 sun.security.provider.SHA2.lf_delta1 SHA2 9 2.85% 44.99% > 29952 303270 sun.security.provider.SHA2.lf_S SHA2 13 1.90% > 54.47% 14976 303267 sun.security.provider.SHA2.lf_S SHA2 17 > 1.49% 61.25% 14976 303264 sun.security.provider.SHA2.lf_S SHA2 > 22 0.81% 66.12% 7488 303265 sun.security.provider.SHA2.lf_R > SHA2 23 0.81% 66.94% 9984 303275 > sun.security.provider.SHA2.lf_maj SHA2 25 0.81% 68.56% 156 > 303263 sun.security.provider.ByteArrayAccess.b2iBig64 SHA2 27 > 0.68% 70.05% 9984 303272 sun.security.provider.SHA2.lf_ch SHA2 > 31 0.54% 72.63% 7488 303268 sun.security.provider.SHA2.lf_R > SHA1 34 0.54% 74.25% 156 303224 > sun.security.provider.SHA.implCompress SHA1 43 0.41% 78.05% > 156 303223 sun.security.provider.ByteArrayAccess.b2iBig64 SHA2 60 > 0.27% 82.66% 2496 303262 java.lang.Integer.reverseBytes SHA2 61 > 0.27% 82.93% 64 303290 sun.security.provider.SHA2.lf_sigma1 > SHA1 116 0.14% 91.06% 2496 303222 java.lang.Integer.reverseBytes > > These are calls with SHA (i.e. SHA1) or SHA2 in the stack (depth=4), > and time for SHA2 vs SHA1 is 45.38% vs 1.09%. With such a small app I > don't think SHA or SHA2 is called anywhere else. This is jdk9 b40. > > Why is the output so different from JMH? Is it reasonable comparing > them? > > Thanks > Max > From aleksey.shipilev at oracle.com Thu Dec 4 16:46:08 2014 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 04 Dec 2014 16:46:08 +0000 Subject: hg: code-tools/jmh: 7901118: Internal sleeps limit the lower bound for time spent doing an iteration by 100ms Message-ID: <201412041646.sB4Gk9Zr006917@aojmv0008> Changeset: adb6047266d8 Author: shade Date: 2014-12-04 19:45 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/adb6047266d8 7901118: Internal sleeps limit the lower bound for time spent doing an iteration by 100ms Summary: Use the blocking flags to communicate the end of sync-iteration catchup loops. ! jmh-core/src/main/java/org/openjdk/jmh/runner/InfraControl.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/LoopBenchmarkHandler.java From aleksey.shipilev at oracle.com Thu Dec 4 20:39:02 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 04 Dec 2014 23:39:02 +0300 Subject: Is it reasonable to compare outputs between JMH and hprof? In-Reply-To: <0823DBD6-4049-4BA3-8219-13FF72427805@oracle.com> References: <0823DBD6-4049-4BA3-8219-13FF72427805@oracle.com> Message-ID: <5480C666.1010504@oracle.com> Hi Wang, On 04.12.2014 07:09, Wang Weijun wrote: > I am comparing the difference of SHA-1 and SHA-256. First I wrote a JMH benchmark: > > @Benchmark > public void sig1(Blackhole bh) throws Exception { > bh.consume(sig("SHA-1")); > } > > @Benchmark > public void sig2(Blackhole bh) throws Exception { > bh.consume(sig("SHA-256")); > } You can just return byte[] from the @Benchmark methods here for brevity. > public static void main(String args[]) throws Exception { > int i = Arrays.hashCode(sig("SHA-1")); > i += Arrays.hashCode(sig("SHA-256")); > System.out.println(i); > } > > static byte[] sig(String alg) throws Exception { > MessageDigest md = MessageDigest.getInstance(alg); > md.update(new byte[10000]); > return md.digest(); > } > Why is the output so different from JMH? Step back from profiler for a bit. What you just did is a custom benchmark, which you then tried to "measure" with profiler. If you "just" measure the time spent in different parts of your custom main() with System.nanoTime(), you will probably see the same "weird" distribution. That is because you have committed a number of benchmark sins JMH was trying to protect you from: 1. The absence of warmup. 2. Mixing the profiles in the same run. 3. High-overhead result consumption. Once you realize that, your question transforms into: "Is it reasonable to compare the JMH-driven benchmark and this custom benchmark?" The answer should be obvious once you digest the JMH samples. > Is it reasonable comparing them? In addition to what Bernd said, what are you trying to compare? Profilers are not the tools for measuring time. Profilers are the tools for assessing where the time is spent. In other words, they tell you how the time is distributed around the code, not the absolute timings. This is because profilers normally incur overhead and try to recover from even greater overhead by being probabilistic. We are just crossing fingers in the belief the profiler overhead is uniform across all methods, lines of code, etc. If anything, the saner idea would be attaching the profiler to a JMH benchmark to explain the benchmark numbers. Thanks, -Aleksey. From aleksey.shipilev at oracle.com Thu Dec 4 21:41:12 2014 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 04 Dec 2014 21:41:12 +0000 Subject: hg: code-tools/jmh: 7901119: Sync iterations are broken by CODETOOLS-7901118, and SyncIterNotOnlyMeasurementTest is @Ignore-d Message-ID: <201412042141.sB4LfDuF029496@aojmv0008> Changeset: 2f9c7421ec8e Author: shade Date: 2014-12-05 00:39 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/2f9c7421ec8e 7901119: Sync iterations are broken by CODETOOLS-7901118, and SyncIterNotOnlyMeasurementTest is @Ignore-d ! jmh-core-it/src/test/java/org/openjdk/jmh/it/synciter/SyncIterNotOnlyMeasurementTest.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/InfraControl.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/LoopBenchmarkHandler.java From aleksey.shipilev at oracle.com Thu Dec 4 23:13:55 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Fri, 05 Dec 2014 02:13:55 +0300 Subject: JMH 1.3.3 Message-ID: <5480EAB3.5070508@oracle.com> Hi, JMH 1.3.3 is released and available at Maven Central (thanks to Evgeny Mandrikov, as usual). This is a minor patch release, and it includes two bugfixes: * The control thread in JMH was waiting on worker threads catching up with each other in sync-iterations, by doing a sleep-loop with 100 ms delay. This did not affect the measurement, but it made the benchmark iterations to always take >100ms, even if user requested, say, 10ms iterations. Fixed: https://bugs.openjdk.java.net/browse/CODETOOLS-7901118 * The previous change discovered the slippage in our sync-iteration tests, as it had failed to catch the obvious regression. Don't @Ignore tests, kids! Fixed: https://bugs.openjdk.java.net/browse/CODETOOLS-7901119 Enjoy! Thanks, -Aleksey. From aleksey.shipilev at oracle.com Thu Dec 4 23:15:45 2014 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 04 Dec 2014 23:15:45 +0000 Subject: hg: code-tools/jmh: 3 new changesets Message-ID: <201412042315.sB4NFkEZ024370@aojmv0008> Changeset: 23dfeb5ab926 Author: shade Date: 2014-12-05 01:09 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/23dfeb5ab926 JMH v1.3.3. ! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml ! jmh-archetypes/pom.xml ! jmh-core-benchmarks/pom.xml ! jmh-core-ct/pom.xml ! jmh-core-it/pom.xml ! jmh-core/pom.xml ! jmh-generator-annprocess/pom.xml ! jmh-generator-asm/pom.xml ! jmh-generator-bytecode/pom.xml ! jmh-generator-reflection/pom.xml ! jmh-samples/pom.xml ! pom.xml Changeset: 13908664eb68 Author: shade Date: 2014-12-05 01:09 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/13908664eb68 Added tag 1.3.3 for changeset 23dfeb5ab926 ! .hgtags Changeset: 448ef82f0360 Author: shade Date: 2014-12-05 01:10 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/448ef82f0360 Continue in 1.4-SNAPSHOT. ! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml ! jmh-archetypes/pom.xml ! jmh-core-benchmarks/pom.xml ! jmh-core-ct/pom.xml ! jmh-core-it/pom.xml ! jmh-core/pom.xml ! jmh-generator-annprocess/pom.xml ! jmh-generator-asm/pom.xml ! jmh-generator-bytecode/pom.xml ! jmh-generator-reflection/pom.xml ! jmh-samples/pom.xml ! pom.xml From kon.dmitriev at gmail.com Sat Dec 6 20:27:22 2014 From: kon.dmitriev at gmail.com (Konstantin Dmitriev) Date: Sat, 6 Dec 2014 20:27:22 +0000 Subject: Why is SampleTime has a limit of 20000 measurements and the rest are thrown away? Message-ID: int targetSamples = (int) (control.getDuration(TimeUnit.MILLISECONDS) * 20); // at max, 20 timestamps per millisecond Regards Konstantin From aleksey.shipilev at oracle.com Tue Dec 9 12:13:58 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Tue, 09 Dec 2014 15:13:58 +0300 Subject: Why is SampleTime has a limit of 20000 measurements and the rest are thrown away? In-Reply-To: References: Message-ID: <5486E786.2050502@oracle.com> On 12/06/2014 11:27 PM, Konstantin Dmitriev wrote: > int targetSamples = (int) > (control.getDuration(TimeUnit.MILLISECONDS) * 20); // at max, 20 > timestamps per millisecond This is why [1]: "Looking at this graph, calling nanoTime quicker than 32 us across all threads will hit the scalability bottleneck on this particular machine. The entire white spot under the "scalability front" is a forbidden zone. You can?t measure anything there. NOTE: This is why JMH-ish SampleTime mode will backoff aggressively instead of measuring every single call." Thanks, -Aleksey. [1] http://shipilev.net/blog/2014/nanotrusting-nanotime/ From julien.nicoulaud at gmail.com Tue Dec 9 17:31:02 2014 From: julien.nicoulaud at gmail.com (Julien Nicoulaud) Date: Tue, 9 Dec 2014 18:31:02 +0100 Subject: Binary link server binding address Message-ID: Hi, A long time ago (0.2), BinaryLinkServer binding address was the default one (InetAddress.getLocalHost()). In the meanwhile, this has been changed to the loopback interface address. I used to rely on this to execute the forked VM on a remote compute node. I know this is a weird use case, but it would be nice if there was an option (or even an undocumented system property) to get this back. Regards, Julien From aleksey.shipilev at oracle.com Tue Dec 9 20:01:00 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Tue, 09 Dec 2014 23:01:00 +0300 Subject: Binary link server binding address In-Reply-To: References: Message-ID: <548754FC.5070205@oracle.com> Hi Julien, On 09.12.2014 20:31, Julien Nicoulaud wrote: > A long time ago (0.2), BinaryLinkServer binding address was the default one > (InetAddress.getLocalHost()). In the meanwhile, this has been changed to > the loopback interface address. Yes, getLocalHost() is odd, since sometimes it can resolve the external address, that might be filtered. These are the fixes that brought us a proper loopback: http://hg.openjdk.java.net/code-tools/jmh/rev/1846fe42b40b http://hg.openjdk.java.net/code-tools/jmh/rev/77ac1de4a238 > I used to rely on this to execute the forked VM on a remote compute node. I > know this is a weird use case, but it would be nice if there was an option > (or even an undocumented system property) to get this back. Oh... Please tell me more about this scenario. I agree, some sort of manual override for binding addresses would be nice, filed: https://bugs.openjdk.java.net/browse/CODETOOLS-7901123 Thanks, -Aleksey. From aleksey.shipilev at oracle.com Tue Dec 9 20:29:56 2014 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Tue, 09 Dec 2014 20:29:56 +0000 Subject: hg: code-tools/jmh: 7901123: Manual override for binary link server/client addresses. Message-ID: <201412092029.sB9KTuok028161@aojmv0008> Changeset: 9d3ce623fb33 Author: shade Date: 2014-12-09 23:29 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/9d3ce623fb33 7901123: Manual override for binary link server/client addresses. Summary: introduce "jmh.link.address" and "jmh.link.port" override options. ! jmh-core/src/main/java/org/openjdk/jmh/runner/link/BinaryLinkServer.java From aleksey.shipilev at oracle.com Tue Dec 9 20:32:04 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Tue, 09 Dec 2014 23:32:04 +0300 Subject: Binary link server binding address In-Reply-To: <548754FC.5070205@oracle.com> References: <548754FC.5070205@oracle.com> Message-ID: <54875C44.4090506@oracle.com> On 09.12.2014 23:01, Aleksey Shipilev wrote: > I agree, some sort of manual override for binding addresses would be > nice, filed: > https://bugs.openjdk.java.net/browse/CODETOOLS-7901123 This should do the trick: http://hg.openjdk.java.net/code-tools/jmh/rev/9d3ce623fb33 Doing the autodetection with getLocalHost() is not a way to go. Power users like you can now bind to some other address, bypassing the autodetection. Please test. Thanks, -Aleksey. From aleksey.shipilev at oracle.com Wed Dec 10 13:12:48 2014 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Wed, 10 Dec 2014 13:12:48 +0000 Subject: hg: code-tools/jmh: Additional @CompilerControl tests, probing the actual compiler output. Message-ID: <201412101312.sBADCmxi007899@aojmv0008> Changeset: 73451bc77ee0 Author: shade Date: 2014-12-10 16:08 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/73451bc77ee0 Additional @CompilerControl tests, probing the actual compiler output. + jmh-core-it/src/test/java/org/openjdk/jmh/it/ccontrol/CompilerControlDontInlineActualTest.java + jmh-core-it/src/test/java/org/openjdk/jmh/it/ccontrol/CompilerControlExcludeActualTest.java + jmh-core-it/src/test/java/org/openjdk/jmh/it/ccontrol/CompilerControlInlineActualTest.java + jmh-core-it/src/test/java/org/openjdk/jmh/it/ccontrol/CompilerControlUtils.java + jmh-core-it/src/test/java/org/openjdk/jmh/it/ccontrol/LogConsumeProfiler.java From aleksey.shipilev at oracle.com Thu Dec 11 13:42:36 2014 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 11 Dec 2014 13:42:36 +0000 Subject: hg: code-tools/jmh: 7901126: Print JMH version in the run output Message-ID: <201412111342.sBBDga9E025103@aojmv0008> Changeset: 87eea88f9819 Author: shade Date: 2014-12-11 16:42 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/87eea88f9819 7901126: Print JMH version in the run output ! jmh-core/pom.xml ! jmh-core/src/main/java/org/openjdk/jmh/runner/BaseRunner.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java + jmh-core/src/main/java/org/openjdk/jmh/util/Version.java + jmh-core/src/main/resources/jmh.properties From julien.nicoulaud at gmail.com Thu Dec 11 15:55:44 2014 From: julien.nicoulaud at gmail.com (Julien Nicoulaud) Date: Thu, 11 Dec 2014 16:55:44 +0100 Subject: Binary link server binding address In-Reply-To: <54875C44.4090506@oracle.com> References: <548754FC.5070205@oracle.com> <54875C44.4090506@oracle.com> Message-ID: It's perfect, thank you. About my use case, to make simple: I use this to run the forked VM on a remote compute node grid managed by a batch system, I have my own runner that handles parameterization, runs the N parameter combinations in parallel (each thread running one JMH Runner instance), and aggregates results into charts. JMH Runners are "tricked" into launching their client VM remotely by setting the JVM command option to a wrapper script that does the forwarding. This has many advantages: faster, parallelized, benchmarks running on the target hardware and software stack, can launch benchmarks right from within the IDE without any deployment step, etc. This would not be possible if JMH was not so embeddable, so I hope it stays this way in the future :) Cheers, Julien 2014-12-09 21:32 GMT+01:00 Aleksey Shipilev : > On 09.12.2014 23:01, Aleksey Shipilev wrote: > > I agree, some sort of manual override for binding addresses would be > > nice, filed: > > https://bugs.openjdk.java.net/browse/CODETOOLS-7901123 > > This should do the trick: > http://hg.openjdk.java.net/code-tools/jmh/rev/9d3ce623fb33 > > Doing the autodetection with getLocalHost() is not a way to go. Power > users like you can now bind to some other address, bypassing the > autodetection. Please test. > > Thanks, > -Aleksey. > > > From aleksey.shipilev at oracle.com Thu Dec 11 16:07:30 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 11 Dec 2014 19:07:30 +0300 Subject: Binary link server binding address In-Reply-To: References: <548754FC.5070205@oracle.com> <54875C44.4090506@oracle.com> Message-ID: <5489C142.2080606@oracle.com> On 11.12.2014 18:55, Julien Nicoulaud wrote: > It's perfect, thank you. Excellent! Will be the part of the next patch release. > About my use case, to make simple: I use this to run the forked VM on a > remote compute node grid managed by a batch system, I have my own runner > that handles parameterization, runs the N parameter combinations in > parallel (each thread running one JMH Runner instance), and aggregates > results into charts. JMH Runners are "tricked" into launching their > client VM remotely by setting the JVM command option to a wrapper script > that does the forwarding. Okay, I see, thanks. This sounds deceptively like Japex: https://japex.java.net/ > This would not be possible if JMH was not so embeddable, so I hope it > stays this way in the future :) Yes, that's the plan. -Aleksey From maximk at twitter.com Thu Dec 11 20:07:42 2014 From: maximk at twitter.com (Maxim Khutornenko) Date: Thu, 11 Dec 2014 12:07:42 -0800 Subject: warning: Supported source version 'RELEASE_6' Message-ID: <52038F04-FD79-4C84-8A0F-51C1270F70B8@twitter.com> Hi, Could not find anything in the archive. We are getting the following warning with running jmh with jdk 1.7: warning: Supported source version 'RELEASE_6' from annotation processor 'org.openjdk.jmh.generators.BenchmarkProcessor' less than -source '1.7' I recon it's benign but are there any plans/options to get rid of it? Thanks, Maxim From aleksey.shipilev at oracle.com Thu Dec 11 20:39:26 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 11 Dec 2014 23:39:26 +0300 Subject: warning: Supported source version 'RELEASE_6' In-Reply-To: <52038F04-FD79-4C84-8A0F-51C1270F70B8@twitter.com> References: <52038F04-FD79-4C84-8A0F-51C1270F70B8@twitter.com> Message-ID: <548A00FE.8060108@oracle.com> Hi Maxim, On 11.12.2014 23:07, Maxim Khutornenko wrote: > warning: Supported source version 'RELEASE_6' from annotation > processor 'org.openjdk.jmh.generators.BenchmarkProcessor' less than > -source '1.7' > > I recon it's benign but are there any plans/options to get rid of > it? Yes, it is benign. We might want to shun it with more careful coding: https://bugs.openjdk.java.net/browse/CODETOOLS-7901127 Thanks, -Aleksey. From mingqin1 at yahoo.com Fri Dec 12 05:33:33 2014 From: mingqin1 at yahoo.com (Ming Qin) Date: Fri, 12 Dec 2014 05:33:33 +0000 (UTC) Subject: Using JMH to benchmark Java 8 Stream API ( parellel vs sequential) to sum up integer in list Message-ID: <1581909584.119564.1418362413976.JavaMail.yahoo@jws10755.mail.gq1.yahoo.com> Hi:? I wrote a blog (? http://wp.me/p1FzYe-9k ) to use JMH 1.3.3 to benchmark throughput of summing up integers hosted? in a List object through? different Java 8 API ( array, iterator , Stream sequential and Stream Parallel) .The result of my JMH benachmark codes indicated that Stream Parallel was the slowest one to sum up all integers .? I run this command to get results -java -jar target/benchmarks.jar JMHSample_01 -wi 5 -t 1 -i 5 -f 1 ?Before seeing the results, I anticipated that Steam Parallel approach should get better throughput number that Stream Sequential.?? ? I? doubted that the method of parallelSumIntegers() was not implemented correctly . Below are my codes. ?import java.util.ArrayList;import java.util.Iterator;import java.util.List;import java.util.concurrent.TimeUnit;import org.openjdk.jmh.annotations.Benchmark;import org.openjdk.jmh.annotations.BenchmarkMode;import org.openjdk.jmh.annotations.Mode;import org.openjdk.jmh.annotations.OutputTimeUnit;import org.openjdk.jmh.annotations.Scope;import org.openjdk.jmh.annotations.Setup;import org.openjdk.jmh.annotations.State;import org.openjdk.jmh.runner.Runner;import org.openjdk.jmh.runner.RunnerException;import org.openjdk.jmh.runner.options.Options;import org.openjdk.jmh.runner.options.OptionsBuilder;?@State(Scope.Benchmark)public class JMHSample_01_HelloWorld {volatile int counts = 9999999;volatile List values = new ArrayList<>(counts);volatile int processors = Runtime.getRuntime().availableProcessors();?@Setuppublic void setup() {populate(values);}?public? void populate ( List? list){for ( int i=0; ii).sum();?return result;?}?@Benchmark at BenchmarkMode(Mode.Throughput)@OutputTimeUnit(TimeUnit.SECONDS)public int sequentialSumIntegers( ){int result = values.stream().mapToInt(i->i).sum();?return result;?}?public static void main(String[] args) throws RunnerException {?Options opt = new OptionsBuilder().include(JMHSample_01_HelloWorld.class.getSimpleName()).forks(1).build();?new Runner(opt).run();}?} From aleksey.shipilev at oracle.com Fri Dec 12 12:39:39 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Fri, 12 Dec 2014 15:39:39 +0300 Subject: JMH 1.3.4 Message-ID: <548AE20B.3050405@oracle.com> Hi, JMH 1.3.4 is released and available at Maven Central (thanks to Evgeny Mandrikov, as usual). This is a minor patch release, and it includes a few bugfixes: * Binary link that is used to communicate between host and forked VM has a loopback address autodetection scheme, which gets in the way in some weird corner cases. Added the "jmh.link.address" and "jmh.link.port" properties to override: https://bugs.openjdk.java.net/browse/CODETOOLS-7901123 * JMH now prints the JMH version in the output log, and also begs for upgrade if the outdated version is detected. Please make sure you have updated to 1.3.4 manually first. Here's the change: https://bugs.openjdk.java.net/browse/CODETOOLS-7901126 * Additional @CompilerControl tests are added to figure if the underlying VM operates properly. If you suspect your VM is misbehaving, it is a good idea to run all tests from JMH workspace -- this would be done if you just build JMH. Enjoy! Thanks, -Aleksey. From aleksey.shipilev at oracle.com Fri Dec 12 13:28:12 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Fri, 12 Dec 2014 16:28:12 +0300 Subject: Using JMH to benchmark Java 8 Stream API ( parellel vs sequential) to sum up integer in list In-Reply-To: <1581909584.119564.1418362413976.JavaMail.yahoo@jws10755.mail.gq1.yahoo.com> References: <1581909584.119564.1418362413976.JavaMail.yahoo@jws10755.mail.gq1.yahoo.com> Message-ID: <548AED6C.6060206@oracle.com> Hi Ming Qin, On 12.12.2014 08:33, Ming Qin wrote: > Hi: I wrote a blog ( http://wp.me/p1FzYe-9k ) to use JMH 1.3.3 to > benchmark throughput of summing up integers hosted in a List object > through different Java 8 API ( array, iterator , Stream sequential > and Stream Parallel) .The result of my JMH benachmark codes indicated > that Stream Parallel was the slowest one to sum up all integers . I > run this command to get results -java -jar target/benchmarks.jar > JMHSample_01 -wi 5 -t 1 -i 5 -f 1 Before seeing the results, I > anticipated that Steam Parallel approach should get better throughput > number that Stream Sequential. I doubted that the method of > parallelSumIntegers() was not implemented correctly . Below are my > codes. Three things: 1. This is the list for JMH development, not general benchmark reviews. Do you suspect a bug in JMH? I think not; therefore, your question belongs on StackOverflow. 2. The benchmark code is badly formatted [1], and it requires additional steps to even read it, let alone build and run it. Why would anyone abuse a simple JMH sample like that? Bootstrap a separate benchmark project, set the source/target 1.8, and publish the entire reproducer e.g. on GitHub. 3. Stopping at the very first and obvious inconsistency while running and publishing the benchmark results is an appalling practice, contributing almost nothing to the world. The word "scrutinize" you mention in your blog is key here. What you did does not count as scrutinizing at all, it is just producing numbers, not figuring out anything useful from them. Try to find the explanations yourself before asking for help. Here are the examples of actual scrutinizing [2][3][4]. Thanks, -Aleksey. [1] http://mail.openjdk.java.net/pipermail/jmh-dev/2014-December/001516.html [2] http://shipilev.net/blog/2014/java-scala-divided-we-fail/ [3] http://biboudis.github.io/clashofthelambdas/ [4] http://stackoverflow.com/questions/25847397/erratic-performance-of-arrays-stream-map-sum [5] http://stackoverflow.com/questions/23756966/why-is-stringbuilderappendint-faster-in-java-7-than-in-java-8 From maximk at twitter.com Fri Dec 12 20:36:03 2014 From: maximk at twitter.com (Maxim Khutornenko) Date: Fri, 12 Dec 2014 12:36:03 -0800 Subject: warning: Supported source version 'RELEASE_6' In-Reply-To: <52038F04-FD79-4C84-8A0F-51C1270F70B8@twitter.com> References: <52038F04-FD79-4C84-8A0F-51C1270F70B8@twitter.com> Message-ID: <85A560A2-DC31-4D68-A6B0-C76C316854E9@twitter.com> Thanks Aleksey! Will keep an eye on CODETOOLS-7901127. Maxim > Date: Thu, 11 Dec 2014 23:39:26 +0300 > From: Aleksey Shipilev > To: jmh-dev at openjdk.java.net > Subject: Re: warning: Supported source version 'RELEASE_6' > Message-ID: <548A00FE.8060108 at oracle.com> > Content-Type: text/plain; charset=windows-1252 > > Hi Maxim, > > On 11.12.2014 23:07, Maxim Khutornenko wrote: >> warning: Supported source version 'RELEASE_6' from annotation >> processor 'org.openjdk.jmh.generators.BenchmarkProcessor' less than >> -source '1.7' >> >> I recon it's benign but are there any plans/options to get rid of >> it? > > Yes, it is benign. > > We might want to shun it with more careful coding: > https://bugs.openjdk.java.net/browse/CODETOOLS-7901127 > > Thanks, > -Aleksey. > > > > On Dec 11, 2014, at 12:07 PM, Maxim Khutornenko wrote: > >> Hi, >> >> Could not find anything in the archive. We are getting the following warning with running jmh with jdk 1.7: >> >> warning: Supported source version 'RELEASE_6' from annotation processor 'org.openjdk.jmh.generators.BenchmarkProcessor' less than -source '1.7' >> >> I recon it's benign but are there any plans/options to get rid of it? >> >> Thanks, >> Maxim >> >> >> > From ppozerov at gmail.com Sat Dec 13 23:19:58 2014 From: ppozerov at gmail.com (Vladimir Ozerov) Date: Sun, 14 Dec 2014 02:19:58 +0300 Subject: Getting forked JVM details on the server side. Message-ID: Hi, I'm trying to implement perfasm counterpart for Windows which employs "xperf" utility. While actual profiler implementation is trivial and requires only minor changes comparing to LinuxPerfAsmProfiler, there is an infrastructure problem - I need pid of the spawned fork. Reason for this - xperf cannot log events for particular process, it logs events for all of them, so I need pid to filter out unnecessary events. What do you think about making a change to BinaryLinkClient, so that method "requestOptions()" will also send information about it's environment to the server? I.e. it will be not "requestOptions", but a kind of handshake where both participants share some information with each other? Vladimir. From aleksey.shipilev at oracle.com Mon Dec 15 08:11:23 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Mon, 15 Dec 2014 11:11:23 +0300 Subject: Getting forked JVM details on the server side. In-Reply-To: References: Message-ID: <548E97AB.3050309@oracle.com> Hi Vladimir, On 12/14/2014 02:19 AM, Vladimir Ozerov wrote: > I'm trying to implement perfasm counterpart for Windows which employs > "xperf" utility. Great! > While actual profiler implementation is trivial and requires only minor > changes comparing to LinuxPerfAsmProfiler, there is an infrastructure > problem - I need pid of the spawned fork. Reason for this - xperf cannot > log events for particular process, it logs events for all of them, so I > need pid to filter out unnecessary events. Getting PID in a cross-platform manner is hard in Java, I think. Current perfasm code avoids this by calling the forked JVM under perf, maybe xperf can do the same? I can see "-pidnewprocess" in the docs. > What do you think about making a change to BinaryLinkClient, so that method > "requestOptions()" will also send information about it's environment to the > server? I.e. it will be not "requestOptions", but a kind of handshake where > both participants share some information with each other? Sure, hack it together, let's see what happens. -Aleksey. From matt.warren at live.co.uk Thu Dec 18 12:42:14 2014 From: matt.warren at live.co.uk (Matt Warren) Date: Thu, 18 Dec 2014 12:42:14 +0000 Subject: Unable to run validation.jar benchmarks Message-ID: Hi If I try to run the validation benchmarks, I get a RunnerException message, see below: C:\dev\jmh\jmh-core-benchmarks>java -jar target\validation.jar JMH Infrastructure Validation Tests ------------------------------------------------------------------------- Basic benchmark tests: These tests show the basic overheads of calling the benchmark method. Additionally, the overheads of calling empty Fixture methods are measured. running empty benchmark: 5.95 +- 7.52 ns > Multiple matching benchmarks (expected only 1): > org.openjdk.jmh.benchmarks.LevelTrialBench.*benchmark_setup* > org.openjdk.jmh.benchmarks.LevelTrialBench. *benchmark_setup_teardown* Exception in thread "main" org.openjdk.jmh.runner.RunnerException: More than single benchmark is matching the options at org.openjdk.jmh.runner.Runner.runSingle(Runner.java:169) at org.openjdk.jmh.MainValidation.basic(MainValidation.java:85) at org.openjdk.jmh.MainValidation.main(MainValidation.java:55) C:\dev\jmh\jmh-core-benchmarks> The lines starting ">" are my own debugging code and they show the problem. The code is calling runSingle(), but there are multiple benchmarks that match the regex. One fix is to change the regex (in jmh\jmh-core-benchmarks\src\main\java\org\openjdk\jmh\MainValidation.java) so that it includes the anchor character '$', i.e. from Options opts = new OptionsBuilder() .include(LevelTrialBench.class.getCanonicalName() + ".*benchmark_setup") .verbosity(VerboseMode.SILENT) .build(); to Options opts = new OptionsBuilder() .include(LevelTrialBench.class.getCanonicalName() + ".*benchmark_setup*$*") .verbosity(VerboseMode.SILENT) .build(); This fix would have to be replicated in a few other places in MainValidation.java, as other tests also fail. But I don't know if this fits with what the code is supposed to be doing? Cheers Matt From matt.warren at live.co.uk Thu Dec 18 12:53:14 2014 From: matt.warren at live.co.uk (Matt Warren) Date: Thu, 18 Dec 2014 12:53:14 +0000 Subject: =?UTF-8?Q?Encoding_issues_with_=27=C2=B1=27_on_Windows?= Message-ID: Hi When I run any JMH Benchmark within the Windows Console, the '?' is replaced with a '?'. This makes the benchmarks a bit confusing when you first run them as '?' is used quite a bit. I've written up a gist[1] that shows the problem, it's based on a StackOverflow answer[2]. The problem is due to default encodings and there seems to be a fix/workaround. I don't know if Windows is a priority or not, but it would be nice of OOTB the JHM results were a bit more readable? Cheers Matt [1] https://gist.github.com/mattwarren/b560a8e1de010cfdac00 [2] http://stackoverflow.com/questions/4005378/console-writeline-and-system-out-println/4005717#4005717 From aleksey.shipilev at oracle.com Thu Dec 18 14:10:01 2014 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 18 Dec 2014 14:10:01 +0000 Subject: hg: code-tools/jmh: 3 new changesets Message-ID: <201412181410.sBIEA6RB017906@aojmv0008> Changeset: 86754261b8bb Author: shade Date: 2014-12-11 17:52 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/86754261b8bb JMH v1.3.4. ! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml ! jmh-archetypes/pom.xml ! jmh-core-benchmarks/pom.xml ! jmh-core-ct/pom.xml ! jmh-core-it/pom.xml ! jmh-core/pom.xml ! jmh-generator-annprocess/pom.xml ! jmh-generator-asm/pom.xml ! jmh-generator-bytecode/pom.xml ! jmh-generator-reflection/pom.xml ! jmh-samples/pom.xml ! pom.xml Changeset: 516fa8a1c384 Author: shade Date: 2014-12-11 17:54 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/516fa8a1c384 Added tag 1.3.4 for changeset 86754261b8bb ! .hgtags Changeset: d6d10326e70f Author: shade Date: 2014-12-11 17:54 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/d6d10326e70f Continue in 1.4-SNAPSHOT. ! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml ! jmh-archetypes/pom.xml ! jmh-core-benchmarks/pom.xml ! jmh-core-ct/pom.xml ! jmh-core-it/pom.xml ! jmh-core/pom.xml ! jmh-generator-annprocess/pom.xml ! jmh-generator-asm/pom.xml ! jmh-generator-bytecode/pom.xml ! jmh-generator-reflection/pom.xml ! jmh-samples/pom.xml ! pom.xml From aleksey.shipilev at oracle.com Thu Dec 18 14:15:36 2014 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 18 Dec 2014 14:15:36 +0000 Subject: hg: code-tools/jmh: 7901127: BenchmarkProcessor should report proper SourceVersion Message-ID: <201412181415.sBIEFa4r018741@aojmv0008> Changeset: df162425ce6f Author: shade Date: 2014-12-18 17:15 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/df162425ce6f 7901127: BenchmarkProcessor should report proper SourceVersion ! jmh-generator-annprocess/src/main/java/org/openjdk/jmh/generators/BenchmarkProcessor.java From aleksey.shipilev at oracle.com Thu Dec 18 14:31:53 2014 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 18 Dec 2014 14:31:53 +0000 Subject: hg: code-tools/jmh: 7901218: jmh-core-benchmarks/validation.jar is failing Message-ID: <201412181431.sBIEVsIs021929@aojmv0008> Changeset: cfbc4996b875 Author: shade Date: 2014-12-18 17:31 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/cfbc4996b875 7901218: jmh-core-benchmarks/validation.jar is failing ! jmh-core-benchmarks/src/main/java/org/openjdk/jmh/MainValidation.java From aleksey.shipilev at oracle.com Thu Dec 18 14:32:53 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 18 Dec 2014 17:32:53 +0300 Subject: Unable to run validation.jar benchmarks In-Reply-To: References: Message-ID: <5492E595.8060407@oracle.com> Hi Matt, On 12/18/2014 03:42 PM, Matt Warren wrote: > This fix would have to be replicated in a few other places in > MainValidation.java, as other tests also fail. But I don't know if this > fits with what the code is supposed to be doing? The jmh-core-benchmarks/validation.jar is one of those hidden non-advertised gems, and we run it only occasionally. In this particular case, the last time we have run the validation.jar was before the change in benchmark matching policy. You are right we may anchor the benchmark names to overcome this. Also, the code is missing the reasonable benchmark defaults. Fixed both now: https://bugs.openjdk.java.net/browse/CODETOOLS-7901218 Thanks, -Aleksey. From aleksey.shipilev at oracle.com Thu Dec 18 14:34:41 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 18 Dec 2014 17:34:41 +0300 Subject: Getting forked JVM details on the server side. In-Reply-To: <548E97AB.3050309@oracle.com> References: <548E97AB.3050309@oracle.com> Message-ID: <5492E601.9010901@oracle.com> On 12/15/2014 11:11 AM, Aleksey Shipilev wrote: >> What do you think about making a change to BinaryLinkClient, so that method >> "requestOptions()" will also send information about it's environment to the >> server? I.e. it will be not "requestOptions", but a kind of handshake where >> both participants share some information with each other? > > Sure, hack it together, let's see what happens. I completely forgot that if you want to contribute the patch back to JMH, it is a good idea to sign the OCA right now. See http://openjdk.java.net/contribute/, step 0. Thanks, -Aleksey. From aleksey.shipilev at oracle.com Thu Dec 18 14:42:37 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 18 Dec 2014 17:42:37 +0300 Subject: Encoding issues with =?UTF-8?B?J8KxJyBvbiBXaW5kb3dz?= In-Reply-To: References: Message-ID: <5492E7DD.4030407@oracle.com> On 12/18/2014 03:53 PM, Matt Warren wrote: > Hi > > When I run any JMH Benchmark within the Windows Console, the '?' is > replaced with a '?'. This makes the benchmarks a bit confusing when you > first run them as '?' is used quite a bit. > > I've written up a gist[1] that shows the problem, it's based on > a StackOverflow answer[2]. The problem is due to default encodings and > there seems to be a fix/workaround. Thanks, I have reproduced this on my (dusty) Windows dev machine. Filed: https://bugs.openjdk.java.net/browse/CODETOOLS-7901217 -Aleksey From matt.warren at live.co.uk Thu Dec 18 14:42:05 2014 From: matt.warren at live.co.uk (Matt Warren) Date: Thu, 18 Dec 2014 14:42:05 +0000 Subject: Unable to run validation.jar benchmarks In-Reply-To: <5492E595.8060407@oracle.com> References: <5492E595.8060407@oracle.com> Message-ID: Thanks for fixing that so quickly. I guessed that validation.jar isn't being run that much ;-) On 18 December 2014 at 14:32, Aleksey Shipilev wrote: > > Hi Matt, > > On 12/18/2014 03:42 PM, Matt Warren wrote: > > This fix would have to be replicated in a few other places in > > MainValidation.java, as other tests also fail. But I don't know if this > > fits with what the code is supposed to be doing? > > The jmh-core-benchmarks/validation.jar is one of those hidden > non-advertised gems, and we run it only occasionally. In this particular > case, the last time we have run the validation.jar was before the change > in benchmark matching policy. > > You are right we may anchor the benchmark names to overcome this. Also, > the code is missing the reasonable benchmark defaults. Fixed both now: > https://bugs.openjdk.java.net/browse/CODETOOLS-7901218 > > Thanks, > -Aleksey. > > > From nilskp at gmail.com Thu Dec 18 15:55:22 2014 From: nilskp at gmail.com (Nils Kilden-Pedersen) Date: Thu, 18 Dec 2014 09:55:22 -0600 Subject: Charts Message-ID: I spent the last two nights whipping up a way to chart the results from JMH. It's still pretty rough around the edges, but is useful enough that I wanted to publish it. http://nilskp.github.io/jmh-charts/ I plan on adding more features, so feel free to file an issue here: https://github.com/nilskp/jmh-charts/issues From aleksey.shipilev at oracle.com Thu Dec 18 18:35:35 2014 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 18 Dec 2014 18:35:35 +0000 Subject: hg: code-tools/jmh: 7901217: \plusminus sign is garbled in JMH output on Windows Message-ID: <201412181835.sBIIZaWY011907@aojmv0008> Changeset: 9453f3e72b6e Author: shade Date: 2014-12-18 21:35 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/9453f3e72b6e 7901217: \plusminus sign is garbled in JMH output on Windows Summary: Guess the console encoding and use it. ! jmh-core/src/main/java/org/openjdk/jmh/results/format/JSONResultFormat.java ! jmh-core/src/main/java/org/openjdk/jmh/results/format/LaTeXResultFormat.java ! jmh-core/src/main/java/org/openjdk/jmh/results/format/ResultFormatFactory.java ! jmh-core/src/main/java/org/openjdk/jmh/results/format/TextResultFormat.java ! jmh-core/src/main/java/org/openjdk/jmh/results/format/XSVResultFormat.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/format/TextReportFormat.java ! jmh-core/src/main/java/org/openjdk/jmh/util/UnCloseablePrintStream.java ! jmh-core/src/main/java/org/openjdk/jmh/util/Utils.java ! jmh-core/src/test/java/org/openjdk/jmh/results/format/ResultFormatTest.java From aleksey.shipilev at oracle.com Thu Dec 18 18:41:03 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 18 Dec 2014 21:41:03 +0300 Subject: Encoding issues with =?UTF-8?B?J8KxJyBvbiBXaW5kb3dz?= In-Reply-To: <5492E7DD.4030407@oracle.com> References: <5492E7DD.4030407@oracle.com> Message-ID: <54931FBF.3050000@oracle.com> On 12/18/2014 05:42 PM, Aleksey Shipilev wrote: > On 12/18/2014 03:53 PM, Matt Warren wrote: >> When I run any JMH Benchmark within the Windows Console, the '?' is >> replaced with a '?'. This makes the benchmarks a bit confusing when you >> first run them as '?' is used quite a bit. > > Thanks, I have reproduced this on my (dusty) Windows dev machine. UTF support in Windows is still an abomination, I see. > Filed: > https://bugs.openjdk.java.net/browse/CODETOOLS-7901217 So, using System.console() directly is not an option for us, since we also need to push the "already encoded" stdout/stderr from the forked VM. After trying the alternatives, I can conclude that the one of the least messier ways to deal with this is to poke the Charset out of Console, and use it: http://hg.openjdk.java.net/code-tools/jmh/rev/9453f3e72b6e It works on my Linux/Windows machines well. I would appreciate more testing ASAP. Thanks, -Aleksey. From aleksey.shipilev at oracle.com Thu Dec 18 18:47:23 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 18 Dec 2014 21:47:23 +0300 Subject: Charts In-Reply-To: References: Message-ID: <5493213B.3040601@oracle.com> On 12/18/2014 06:55 PM, Nils Kilden-Pedersen wrote: > http://nilskp.github.io/jmh-charts/ Interesting, and it seems to work. Thanks, -Aleksey. From zolyfarkas at yahoo.com Thu Dec 18 19:03:48 2014 From: zolyfarkas at yahoo.com (Zoltan Farkas) Date: Thu, 18 Dec 2014 14:03:48 -0500 Subject: Charts In-Reply-To: <5493213B.3040601@oracle.com> References: <5493213B.3040601@oracle.com> Message-ID: <06A12781-02BC-4652-BBC8-B8B82451B3BD@yahoo.com> FYI, there is a JMH plugin for jenkins at: https://github.com/blackboard/jmh-jenkins It allows me to trend by benchmarks with every build, and immediately notice any performance degradation. ?Z > On Dec 18, 2014, at 1:47 PM, Aleksey Shipilev wrote: > > On 12/18/2014 06:55 PM, Nils Kilden-Pedersen wrote: >> http://nilskp.github.io/jmh-charts/ > > Interesting, and it seems to work. > > Thanks, > -Aleksey. > From staffan.friberg at oracle.com Thu Dec 18 19:05:50 2014 From: staffan.friberg at oracle.com (Staffan Friberg) Date: Thu, 18 Dec 2014 11:05:50 -0800 Subject: Rawtypes in JMH and generated classes Message-ID: <5493258E.1040501@oracle.com> Hi, Saw that we got unchecked / rawtypes warnings when compiling benchmarks so updated the generator to avoid that. Since I was already updating this type of warnings I fixed some other files as well. http://cr.openjdk.java.net/~sfriberg/jmh_rawtypes/webrev.00 The main part I'm after is to avoid warnings when compiling the microbenchmarks rather than JMH, so would be fine if we limit the patch to that. //Staffan From nilskp at gmail.com Thu Dec 18 19:14:44 2014 From: nilskp at gmail.com (Nils Kilden-Pedersen) Date: Thu, 18 Dec 2014 13:14:44 -0600 Subject: Charts In-Reply-To: <5493213B.3040601@oracle.com> References: <5493213B.3040601@oracle.com> Message-ID: Any chance we can get the JVM information into the JSON also? On Thu, Dec 18, 2014 at 12:47 PM, Aleksey Shipilev < aleksey.shipilev at oracle.com> wrote: > > On 12/18/2014 06:55 PM, Nils Kilden-Pedersen wrote: > > http://nilskp.github.io/jmh-charts/ > > Interesting, and it seems to work. > > Thanks, > -Aleksey. > > From aleksey.shipilev at oracle.com Thu Dec 18 19:53:31 2014 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 18 Dec 2014 19:53:31 +0000 Subject: hg: code-tools/jmh: 7901219: Clean up raw types in harness and generated classes Message-ID: <201412181953.sBIJrVlp025861@aojmv0008> Changeset: 9686a533ffe6 Author: shade Date: 2014-12-18 22:53 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/9686a533ffe6 7901219: Clean up raw types in harness and generated classes Contributed-by: Staffan Friberg ! jmh-core-it/src/test/java/org/openjdk/jmh/it/ccontrol/LogConsumeProfiler.java ! jmh-core/src/main/java/org/openjdk/jmh/generators/core/BenchmarkGenerator.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/LinuxPerfAsmProfiler.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/LinuxPerfProfiler.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/ProfilerResultAggregator.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/StackProfiler.java ! jmh-core/src/main/java/org/openjdk/jmh/results/Aggregator.java ! jmh-core/src/main/java/org/openjdk/jmh/results/AverageTimeResult.java ! jmh-core/src/main/java/org/openjdk/jmh/results/SampleTimeResult.java ! jmh-core/src/main/java/org/openjdk/jmh/results/SingleShotResult.java ! jmh-core/src/main/java/org/openjdk/jmh/results/ThroughputResult.java From aleksey.shipilev at oracle.com Thu Dec 18 19:54:58 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 18 Dec 2014 22:54:58 +0300 Subject: Rawtypes in JMH and generated classes In-Reply-To: <5493258E.1040501@oracle.com> References: <5493258E.1040501@oracle.com> Message-ID: <54933112.1040608@oracle.com> Thanks, Staffan! On 12/18/2014 10:05 PM, Staffan Friberg wrote: > Saw that we got unchecked / rawtypes warnings when compiling > benchmarks so updated the generator to avoid that. Since I was > already updating this type of warnings I fixed some other files as > well. > > http://cr.openjdk.java.net/~sfriberg/jmh_rawtypes/webrev.00 This is submitted, fixed and pushed as: https://bugs.openjdk.java.net/browse/CODETOOLS-7901219 (You also missed a LogConsumeResult in one of the newer integration tests, fixed that up as well). Thanks, -Aleksey. From staffan.friberg at oracle.com Thu Dec 18 20:54:06 2014 From: staffan.friberg at oracle.com (Staffan Friberg) Date: Thu, 18 Dec 2014 12:54:06 -0800 Subject: Rawtypes in JMH and generated classes In-Reply-To: <54933112.1040608@oracle.com> References: <5493258E.1040501@oracle.com> <54933112.1040608@oracle.com> Message-ID: <54933EEE.5000808@oracle.com> Thanks! On 12/18/2014 11:54 AM, Aleksey Shipilev wrote: > Thanks, Staffan! > > On 12/18/2014 10:05 PM, Staffan Friberg wrote: >> Saw that we got unchecked / rawtypes warnings when compiling >> benchmarks so updated the generator to avoid that. Since I was >> already updating this type of warnings I fixed some other files as >> well. >> >> http://cr.openjdk.java.net/~sfriberg/jmh_rawtypes/webrev.00 > This is submitted, fixed and pushed as: > https://bugs.openjdk.java.net/browse/CODETOOLS-7901219 > > (You also missed a LogConsumeResult in one of the newer integration > tests, fixed that up as well). > > Thanks, > -Aleksey. > From staffan.friberg at oracle.com Thu Dec 18 22:30:36 2014 From: staffan.friberg at oracle.com (Staffan Friberg) Date: Thu, 18 Dec 2014 14:30:36 -0800 Subject: One more warning removal Message-ID: <5493558C.5090405@oracle.com> Another small rawtype update. After this one I can compile with -Xlint on my current set of benchmarks without any warnings. //Staffan diff --git a/jmh-core/src/main/java/org/openjdk/jmh/generators/core/StateObjectHandler.java b/jmh-core/src/main/java/org/openjdk/jmh/generators/core/StateObjectHandler.java --- a/jmh-core/src/main/java/org/openjdk/jmh/generators/core/StateObjectHandler.java +++ b/jmh-core/src/main/java/org/openjdk/jmh/generators/core/StateObjectHandler.java @@ -776,9 +776,9 @@ for (Level level : Level.values()) { pw.println(" public volatile int setup" + level + "Mutex;"); pw.println(" public volatile int tear" + level + "Mutex;"); - pw.println(" public final static AtomicIntegerFieldUpdater setup" + level + "MutexUpdater = " + + pw.println(" public final static AtomicIntegerFieldUpdater<" + so.type + "_B2> setup" + level + "MutexUpdater = " + "AtomicIntegerFieldUpdater.newUpdater(" + so.type + "_B2.class, \"setup" + level + "Mutex\");"); - pw.println(" public final static AtomicIntegerFieldUpdater tear" + level + "MutexUpdater = " + + pw.println(" public final static AtomicIntegerFieldUpdater<" + so.type + "_B2> tear" + level + "MutexUpdater = " + "AtomicIntegerFieldUpdater.newUpdater(" + so.type + "_B2.class, \"tear" + level + "Mutex\");"); pw.println(""); } From aleksey.shipilev at oracle.com Thu Dec 18 22:37:08 2014 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 18 Dec 2014 22:37:08 +0000 Subject: hg: code-tools/jmh: Amend 7901219: Clean up raw types in harness and generated classes Message-ID: <201412182237.sBIMb84W025904@aojmv0008> Changeset: de75b9914a57 Author: shade Date: 2014-12-19 01:36 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/de75b9914a57 Amend 7901219: Clean up raw types in harness and generated classes Contributed-by: Staffan Friberg ! jmh-core/src/main/java/org/openjdk/jmh/generators/core/StateObjectHandler.java From aleksey.shipilev at oracle.com Thu Dec 18 22:37:19 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Fri, 19 Dec 2014 01:37:19 +0300 Subject: One more warning removal In-Reply-To: <5493558C.5090405@oracle.com> References: <5493558C.5090405@oracle.com> Message-ID: <5493571F.2030301@oracle.com> Done: http://hg.openjdk.java.net/code-tools/jmh/rev/de75b9914a57 Thanks, -Aleksey. On 12/19/2014 01:30 AM, Staffan Friberg wrote: > Another small rawtype update. After this one I can compile with -Xlint > on my current set of benchmarks without any warnings. > > //Staffan > > diff --git > a/jmh-core/src/main/java/org/openjdk/jmh/generators/core/StateObjectHandler.java > b/jmh-core/src/main/java/org/openjdk/jmh/generators/core/StateObjectHandler.java > > --- > a/jmh-core/src/main/java/org/openjdk/jmh/generators/core/StateObjectHandler.java > > +++ > b/jmh-core/src/main/java/org/openjdk/jmh/generators/core/StateObjectHandler.java > > @@ -776,9 +776,9 @@ > for (Level level : Level.values()) { > pw.println(" public volatile int setup" + level > + "Mutex;"); > pw.println(" public volatile int tear" + level + > "Mutex;"); > - pw.println(" public final static > AtomicIntegerFieldUpdater setup" + level + "MutexUpdater = " + > + pw.println(" public final static > AtomicIntegerFieldUpdater<" + so.type + "_B2> setup" + level + > "MutexUpdater = " + > "AtomicIntegerFieldUpdater.newUpdater(" + > so.type + "_B2.class, \"setup" + level + "Mutex\");"); > - pw.println(" public final static > AtomicIntegerFieldUpdater tear" + level + "MutexUpdater = " + > + pw.println(" public final static > AtomicIntegerFieldUpdater<" + so.type + "_B2> tear" + level + > "MutexUpdater = " + > "AtomicIntegerFieldUpdater.newUpdater(" + > so.type + "_B2.class, \"tear" + level + "Mutex\");"); > pw.println(""); > } > > > From matt.warren at live.co.uk Thu Dec 18 23:39:33 2014 From: matt.warren at live.co.uk (Matt Warren) Date: Thu, 18 Dec 2014 23:39:33 +0000 Subject: =?UTF-8?Q?Re=3A_Encoding_issues_with_=27=C2=B1=27_on_Windows?= In-Reply-To: <54931FBF.3050000@oracle.com> References: <5492E7DD.4030407@oracle.com> <54931FBF.3050000@oracle.com> Message-ID: I got the latest changes and it works for me (Java 1.7, Windows 2008), lots of '?' in the results nice fix reflection FTW, http://hg.openjdk.java.net/code-tools/jmh/rev/9453f3e72b6e#l9.22 Cheers Matt On 18 December 2014 at 18:41, Aleksey Shipilev wrote: > > On 12/18/2014 05:42 PM, Aleksey Shipilev wrote: > > On 12/18/2014 03:53 PM, Matt Warren wrote: > >> When I run any JMH Benchmark within the Windows Console, the '?' is > >> replaced with a '?'. This makes the benchmarks a bit confusing when you > >> first run them as '?' is used quite a bit. > > > > Thanks, I have reproduced this on my (dusty) Windows dev machine. > > UTF support in Windows is still an abomination, I see. > > > Filed: > > https://bugs.openjdk.java.net/browse/CODETOOLS-7901217 > > So, using System.console() directly is not an option for us, since we > also need to push the "already encoded" stdout/stderr from the forked > VM. After trying the alternatives, I can conclude that the one of the > least messier ways to deal with this is to poke the Charset out of > Console, and use it: > http://hg.openjdk.java.net/code-tools/jmh/rev/9453f3e72b6e > > It works on my Linux/Windows machines well. > > I would appreciate more testing ASAP. > > Thanks, > -Aleksey. > > > From aleksey.shipilev at oracle.com Thu Dec 18 23:41:32 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Fri, 19 Dec 2014 02:41:32 +0300 Subject: Encoding issues with =?UTF-8?B?J8KxJyBvbiBXaW5kb3dz?= In-Reply-To: References: <5492E7DD.4030407@oracle.com> <54931FBF.3050000@oracle.com> Message-ID: <5493662C.6010707@oracle.com> Excellent, thanks for testing! -Aleksey. On 12/19/2014 02:39 AM, Matt Warren wrote: > I got the latest changes and it works for me (Java 1.7, Windows 2008), > lots of '?' in the results > > nice fix reflection > FTW, http://hg.openjdk.java.net/code-tools/jmh/rev/9453f3e72b6e#l9.22 > > Cheers > > Matt > > On 18 December 2014 at 18:41, Aleksey Shipilev > > wrote: > > On 12/18/2014 05:42 PM, Aleksey Shipilev wrote: > > On 12/18/2014 03:53 PM, Matt Warren wrote: > >> When I run any JMH Benchmark within the Windows Console, the '?' is > >> replaced with a '?'. This makes the benchmarks a bit confusing when you > >> first run them as '?' is used quite a bit. > > > > Thanks, I have reproduced this on my (dusty) Windows dev machine. > > UTF support in Windows is still an abomination, I see. > > > Filed: > > https://bugs.openjdk.java.net/browse/CODETOOLS-7901217 > > So, using System.console() directly is not an option for us, since we > also need to push the "already encoded" stdout/stderr from the forked > VM. After trying the alternatives, I can conclude that the one of the > least messier ways to deal with this is to poke the Charset out of > Console, and use it: > http://hg.openjdk.java.net/code-tools/jmh/rev/9453f3e72b6e > > It works on my Linux/Windows machines well. > > I would appreciate more testing ASAP. > > Thanks, > -Aleksey. > > From aleksey.shipilev at oracle.com Thu Dec 18 23:52:28 2014 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 18 Dec 2014 23:52:28 +0000 Subject: hg: code-tools/jmh: ResultFormatTest fixes and cleanup. Message-ID: <201412182352.sBINqSxG011532@aojmv0008> Changeset: 3ed73e6b009c Author: shade Date: 2014-12-19 02:52 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/3ed73e6b009c ResultFormatTest fixes and cleanup. ! jmh-core/src/test/java/org/openjdk/jmh/results/format/ResultFormatTest.java From aleksey.shipilev at oracle.com Fri Dec 19 10:03:35 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Fri, 19 Dec 2014 13:03:35 +0300 Subject: JMH 1.4 Message-ID: <5493F7F7.5030409@oracle.com> Hi, JMH 1.4 is released and available at Maven Central (thanks to... hold it... yes, that's Evgeny Mandrikov, as usual). This is a patch release, but it has some intrusive bugfixes that change the internal API on at least one occasion. Therefore, we are bumping the minor version. Please test if 1.4 works good for you! Summary of changes: * "?" sign was garbled on Windows. The fix required to reconsider what we do with the PrintStreams, and notably it changed the argument type in public ResultFormatFactory.getInstance method. Therefore, some code that uses this JMH API may break and require a trivial fix in user code. Fix: https://bugs.openjdk.java.net/browse/CODETOOLS-7901217 * JMH generates some code that uses raw types, leading to excess compilation warnings in benchmarks. Staffan Friberg fixed that for us: https://bugs.openjdk.java.net/browse/CODETOOLS-7901219 * Annotation processors now "accept" the latest() source code available to javac. This should be a safe change for up to JDK 9, and we save a few warning messages in the build: https://bugs.openjdk.java.net/browse/CODETOOLS-7901127 * There is a validation.jar in jmh-core-benchmarks that is an experimental workload to estimate basic things about the benchmarking environment. We do not yet fully support it, but it needed some fixes to run properly: https://bugs.openjdk.java.net/browse/CODETOOLS-7901218 * Polished some tests. Enjoy! Thanks, -Aleksey. From aleksey.shipilev at oracle.com Fri Dec 19 10:05:58 2014 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Fri, 19 Dec 2014 10:05:58 +0000 Subject: hg: code-tools/jmh: 3 new changesets Message-ID: <201412191005.sBJA5xl7005963@aojmv0008> Changeset: 6af8778e554e Author: shade Date: 2014-12-19 02:53 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/6af8778e554e JMH v1.4. ! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml ! jmh-archetypes/pom.xml ! jmh-core-benchmarks/pom.xml ! jmh-core-ct/pom.xml ! jmh-core-it/pom.xml ! jmh-core/pom.xml ! jmh-generator-annprocess/pom.xml ! jmh-generator-asm/pom.xml ! jmh-generator-bytecode/pom.xml ! jmh-generator-reflection/pom.xml ! jmh-samples/pom.xml ! pom.xml Changeset: 9230a92ce2c8 Author: shade Date: 2014-12-19 02:53 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/9230a92ce2c8 Added tag 1.4 for changeset 6af8778e554e ! .hgtags Changeset: 56fd40275263 Author: shade Date: 2014-12-19 02:54 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/56fd40275263 Continue in 1.5-SNAPSHOT. ! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml ! jmh-archetypes/pom.xml ! jmh-core-benchmarks/pom.xml ! jmh-core-ct/pom.xml ! jmh-core-it/pom.xml ! jmh-core/pom.xml ! jmh-generator-annprocess/pom.xml ! jmh-generator-asm/pom.xml ! jmh-generator-bytecode/pom.xml ! jmh-generator-reflection/pom.xml ! jmh-samples/pom.xml ! pom.xml From staffan.friberg at oracle.com Fri Dec 19 19:42:05 2014 From: staffan.friberg at oracle.com (Staffan Friberg) Date: Fri, 19 Dec 2014 11:42:05 -0800 Subject: Discussion: CODETOOLS-7901222 - Enable benchmarks to have configurable Operation unit Message-ID: <54947F8D.1060200@oracle.com> Hi, Wanted to discuss the RFE I just created and think could be useful. https://bugs.openjdk.java.net/browse/CODETOOLS-7901222 Today all benchmarks will have ops/TimeUnit or TimeUnit/op. For some benchmarks it would be a nice feature to be able to express the Operation to be more specific to what is actually measured. An example would be a benchmark working on a byte[] and to better understand the performance of the benchmark the output could be bytes/TimeUnit or TimeUnit/byte, which would be more inline in what the benchmark actually tries to measure. A suggested approach would to have an annotation allow the user to set the Operation unit. Something similar to this. @Benchmark @OperationUnit(singular = "byte") @OperationsPerInvocation(64 * 1024) public void measureBytes() { random.nextBytes(bytes); } @Benchmark @OperationUnit(singular = "foot", plural = "feet") @OperationsPerInvocation(64 * 1024) public void measureDistance() { random.nextBytes(bytes); } Here is a proof of concept patch, far from done, so nothing to really review for code quality etc yet. http://cr.openjdk.java.net/~sfriberg/CODETOOLS-7901222/webrev.00 An option could also be to not look up the unit dynamically during runtime, but simply hardcode it as part of the benchmark generation as changing the operations unit probably doesn't make much sense from the command line. This would probably allow for a much smaller patch as well. Would JMH users find this useful? //Staffan From matt.warren at live.co.uk Mon Dec 22 11:21:55 2014 From: matt.warren at live.co.uk (Matt Warren) Date: Mon, 22 Dec 2014 11:21:55 +0000 Subject: Understanding the code-generation strategy Message-ID: I'm thinking about writing a .NET version of JMH or at the very least using several of the (applicable) ideas in JMH to write a .NET benchmarking framework. Fortunately (or unfortunately) depending on how you look at it, in .NET things are a bit simpler, i.e. one-time JITting, no HotSpot, etc One thing that I've not been able to understand is the technical reasoning behind the code-generation strategy used in JMH. I.e. emitting java code that is then compiled into a single .jar, versus calling the compiled benchmark code directly via reflection? I've read the canonical "JMH vs Caliper: reference thread"[1] and in-particular the section: ------------------------------------------------------------------------- A. Dynamic selection of benchmarks. Since you don't know at "harness" compile time what benchmarks it would run, the obvious choice would be calling the benchmark methods via Reflection. Back in the days, this pushed us to accept the same "repetition" counter in the method to amortize the reflective costs. This already introduces the major pitfall about looping, see below. But infrastructure-wise, harness then should intelligently choose the repetition count. This almost always leads to calibrating mechanics, which is almost always broken when loop optimizations are in effect. If one benchmark is "slower" and autobalances with lower reps count, and another benchmark is "faster" and autobalances with higher reps count, then optimizer have more opportunity to optimize "faster" benchmark even further. Which departs us from seeing how exactly the benchmark performs and introduces another (hidden! and uncontrollable!) degree of freedom. In retrospect, the early days decision of JMH to generate synthetic benchmark code around the method, which contains the loop (carefully chosen by us to avoid the optimizations in current VMs -- separation on concerns, basically), is paying off *very* nicely. We can then call that synthetic stub via Reflection without even bothering about the costs. ...That is not to mention users can actually review the generated benchmark code looking for the explanations for the weird effects. We do that frequently as the additional control. ------------------------------------------------------------------------- My question is, what are the *technical* reasons for doing code generation this way? I understand the nice benefits you get (can inspect the generated code), but I see those as side-benefits. Is it purely related to the overhead of call a method via a reflection call? I wrote a simple benchmark[2] (using JMH of course!) and assuming I didn't mess it up, got the following numbers[3]: # Run complete. Total time: 00:06:47 Benchmark Mode Samples Score Error Units o.s.MyBenchmark.baseline avgt 50 0.283 ? 0.004 ns/op o.s.MyBenchmark.reflectionMethodCall avgt 50 80.220 ? 1.082 ns/op o.s.MyBenchmark.reflectionMethodCallNoCasting avgt 50 79.908 ? 1.266 ns/op o.s.MyBenchmark.regularMethodCall avgt 50 77.749 ? 1.342 ns/op Is it this difference alone that makes the Reflection call per invocation not possible? I'm wondering if I'd need to use this technique in a .NET benchmark harness as .NET gives you the ability to compile a method call (obtained via reflection) into a regular delegate, so you don't pay the same penalty when calling it, see [4] and [5]. Or have I missed the point entirely? Cheers Matt [1] https://groups.google.com/d/msg/mechanical-sympathy/m4opvy4xq3U/7lY8x8SvHgwJ [2] https://gist.github.com/mattwarren/75c969ce58e72883a9a0 (Benchmark) [3] https://gist.github.com/mattwarren/15d88da37c57a222dfd7 (Full Results) [4] http://msdn.microsoft.com/en-us/library/ms228976(v=vs.110).aspx [5] http://msdn.microsoft.com/en-us/library/system.delegate.createdelegate(v=vs.110).aspx From jozefhartinger at gmail.com Mon Dec 22 13:01:24 2014 From: jozefhartinger at gmail.com (Jozef Hartinger) Date: Mon, 22 Dec 2014 14:01:24 +0100 Subject: Lowercase result file name Message-ID: <54981624.4080306@gmail.com> Hi, I noticed that JMH converts the result file name to lowercase before writing the result. This clashes with our versioning strategy as we get e.g: 3.0.0.alpha3.csv instead of 3.0.0.Alpha3 or 3.0.0-snapshot instead of 3.0.0-SNAPSHOT I'm wondering what is the reason for doing this. It would make our life (and scripts) a bit easier if JMH could preserve the original file name. Thanks! Jozef From erich at debian.org Mon Dec 22 12:54:21 2014 From: erich at debian.org (Erich Schubert) Date: Mon, 22 Dec 2014 13:54:21 +0100 Subject: Locale dependency of JMH, div by 0 in ListStatistics Message-ID: Hi, I couldn't find a way to report bugs in JIRA (I find navigating the openJDK web pages horrible, too), so I'm posting to the list. 1. Running JMH in a different locale prevents it from reporting its version number. I'm using the maven version (1.4) which apparently was built with C locale. But SimpleDateFormat defaults to using the system locale, and thus fails to parse the locale-depedant timestamp format that was used. IMHO the cleanest way is to use a non-locale timestamp, such as RFC 3339. 2. org.openjdk.jmh.util.ListStatistics#getVariance This will yield a division by 0 if getN() == 1. In order to compute variance, you need n > 1. The code is also inconsistent in the sense that it uses getN() in one place, and count in the other. Regards, Erich From aleksey.shipilev at oracle.com Mon Dec 22 14:12:42 2014 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Mon, 22 Dec 2014 14:12:42 +0000 Subject: hg: code-tools/jmh: 7901225: JMH converts the result file name to lower-case Message-ID: <201412221412.sBMECgBU020028@aojmv0008> Changeset: 340244351917 Author: shade Date: 2014-12-22 17:12 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/340244351917 7901225: JMH converts the result file name to lower-case Summary: Fix the operation order, only lower-case the result format suffix. + jmh-core-it/src/test/java/org/openjdk/jmh/it/result/ResultFileNameTest.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java From aleksey.shipilev at oracle.com Mon Dec 22 14:13:26 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Mon, 22 Dec 2014 17:13:26 +0300 Subject: Lowercase result file name In-Reply-To: <54981624.4080306@gmail.com> References: <54981624.4080306@gmail.com> Message-ID: <54982706.2040902@oracle.com> Hi Jozef, On 12/22/2014 04:01 PM, Jozef Hartinger wrote: > I noticed that JMH converts the result file name to lowercase before > writing the result. This clashes with our versioning strategy as we get > e.g: > > 3.0.0.alpha3.csv instead of 3.0.0.Alpha3 or > 3.0.0-snapshot instead of 3.0.0-SNAPSHOT > > I'm wondering what is the reason for doing this. It would make our life > (and scripts) a bit easier if JMH could preserve the original file name. It was supposed to lowercase the format type suffix. But I have fat fingers and can't get the precedence right: https://bugs.openjdk.java.net/browse/CODETOOLS-7901225 Fixed: http://hg.openjdk.java.net/code-tools/jmh/rev/340244351917 Should be the part of the next patch release (this week, if time allows), you can use self-built 1.5-SNAPSHOT meanwhile. Thanks, -Aleksey. From jozefhartinger at gmail.com Mon Dec 22 14:15:25 2014 From: jozefhartinger at gmail.com (Jozef Hartinger) Date: Mon, 22 Dec 2014 15:15:25 +0100 Subject: Lowercase result file name In-Reply-To: <54982706.2040902@oracle.com> References: <54981624.4080306@gmail.com> <54982706.2040902@oracle.com> Message-ID: <5498277D.7000806@gmail.com> Thanks a lot! On 12/22/2014 03:13 PM, Aleksey Shipilev wrote: > Hi Jozef, > > On 12/22/2014 04:01 PM, Jozef Hartinger wrote: >> I noticed that JMH converts the result file name to lowercase before >> writing the result. This clashes with our versioning strategy as we get >> e.g: >> >> 3.0.0.alpha3.csv instead of 3.0.0.Alpha3 or >> 3.0.0-snapshot instead of 3.0.0-SNAPSHOT >> >> I'm wondering what is the reason for doing this. It would make our life >> (and scripts) a bit easier if JMH could preserve the original file name. > It was supposed to lowercase the format type suffix. But I have fat > fingers and can't get the precedence right: > https://bugs.openjdk.java.net/browse/CODETOOLS-7901225 > > Fixed: > http://hg.openjdk.java.net/code-tools/jmh/rev/340244351917 > > Should be the part of the next patch release (this week, if time > allows), you can use self-built 1.5-SNAPSHOT meanwhile. > > Thanks, > -Aleksey. > > From aleksey.shipilev at oracle.com Mon Dec 22 14:17:54 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Mon, 22 Dec 2014 17:17:54 +0300 Subject: Locale dependency of JMH, div by 0 in ListStatistics In-Reply-To: References: Message-ID: <54982812.7060900@oracle.com> Hi Erich, On 12/22/2014 03:54 PM, Erich Schubert wrote: > I couldn't find a way to report bugs in JIRA (I find navigating the > openJDK web pages horrible, too), so I'm posting to the list. Yes, thanks. JIRA is open for submissions for OpenJDK authors only, sorry about that. I submitted the bug reports on your behalf. I agree both are bugs. Let me attend to them. > 1. Running JMH in a different locale prevents it from reporting its > version number. > I'm using the maven version (1.4) which apparently was built with C > locale. But SimpleDateFormat defaults to using the system locale, and > thus fails to parse the locale-depedant timestamp format that was > used. > > IMHO the cleanest way is to use a non-locale timestamp, such as RFC 3339. https://bugs.openjdk.java.net/browse/CODETOOLS-7901226 > 2. org.openjdk.jmh.util.ListStatistics#getVariance > > This will yield a division by 0 if getN() == 1. In order to compute > variance, you need n > 1. > The code is also inconsistent in the sense that it uses getN() in one > place, and count in the other. https://bugs.openjdk.java.net/browse/CODETOOLS-7901227 -Aleksey. From aleksey.shipilev at oracle.com Mon Dec 22 14:20:13 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Mon, 22 Dec 2014 17:20:13 +0300 Subject: Locale dependency of JMH, div by 0 in ListStatistics In-Reply-To: <54982812.7060900@oracle.com> References: <54982812.7060900@oracle.com> Message-ID: <5498289D.9070906@oracle.com> (resending, since Erich was accidentally dropped from To: line; one of these days I have to figure out how to keep people in loop without subscribing them...) -Aleksey. On 12/22/2014 05:17 PM, Aleksey Shipilev wrote: > Hi Erich, > > On 12/22/2014 03:54 PM, Erich Schubert wrote: >> I couldn't find a way to report bugs in JIRA (I find navigating the >> openJDK web pages horrible, too), so I'm posting to the list. > > Yes, thanks. JIRA is open for submissions for OpenJDK authors only, > sorry about that. I submitted the bug reports on your behalf. I agree > both are bugs. Let me attend to them. > >> 1. Running JMH in a different locale prevents it from reporting its >> version number. >> I'm using the maven version (1.4) which apparently was built with C >> locale. But SimpleDateFormat defaults to using the system locale, and >> thus fails to parse the locale-depedant timestamp format that was >> used. >> >> IMHO the cleanest way is to use a non-locale timestamp, such as RFC 3339. > > https://bugs.openjdk.java.net/browse/CODETOOLS-7901226 > >> 2. org.openjdk.jmh.util.ListStatistics#getVariance >> >> This will yield a division by 0 if getN() == 1. In order to compute >> variance, you need n > 1. >> The code is also inconsistent in the sense that it uses getN() in one >> place, and count in the other. > > https://bugs.openjdk.java.net/browse/CODETOOLS-7901227 > > -Aleksey. > > From aleksey.shipilev at oracle.com Mon Dec 22 14:47:55 2014 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Mon, 22 Dec 2014 14:47:55 +0000 Subject: hg: code-tools/jmh: 7901227: Division by zero in ListStatistics.getVariance() with N=1 Message-ID: <201412221447.sBMEluRh026235@aojmv0008> Changeset: df151f9c0071 Author: shade Date: 2014-12-22 17:47 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/df151f9c0071 7901227: Division by zero in ListStatistics.getVariance() with N=1 Summary: Fix the calculation and do more tests. ! jmh-core/src/main/java/org/openjdk/jmh/util/ListStatistics.java ! jmh-core/src/test/java/org/openjdk/jmh/util/TestListStatistics.java ! jmh-core/src/test/java/org/openjdk/jmh/util/TestMultisetStatistics.java From aleksey.shipilev at oracle.com Mon Dec 22 14:48:32 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Mon, 22 Dec 2014 17:48:32 +0300 Subject: Locale dependency of JMH, div by 0 in ListStatistics In-Reply-To: <5498289D.9070906@oracle.com> References: <54982812.7060900@oracle.com> <5498289D.9070906@oracle.com> Message-ID: <54982F40.90002@oracle.com> On 12/22/2014 05:20 PM, Aleksey Shipilev wrote: >>> 2. org.openjdk.jmh.util.ListStatistics#getVariance >>> >>> This will yield a division by 0 if getN() == 1. In order to compute >>> variance, you need n > 1. >>> The code is also inconsistent in the sense that it uses getN() in one >>> place, and count in the other. >> >> https://bugs.openjdk.java.net/browse/CODETOOLS-7901227 Ah, the hilarity of this "bug" is that current code does the floating-point division of 0.0/0.0 when N=1, which yields NaN, as expected. Fixed this anyway: http://hg.openjdk.java.net/code-tools/jmh/rev/df151f9c0071 Thanks, -Aleksey. From aleksey.shipilev at oracle.com Mon Dec 22 15:09:22 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Mon, 22 Dec 2014 18:09:22 +0300 Subject: Locale dependency of JMH, div by 0 in ListStatistics In-Reply-To: <5498289D.9070906@oracle.com> References: <54982812.7060900@oracle.com> <5498289D.9070906@oracle.com> Message-ID: <54983422.8050102@oracle.com> On 12/22/2014 05:20 PM, Aleksey Shipilev wrote: >>> 1. Running JMH in a different locale prevents it from reporting its >>> version number. >>> I'm using the maven version (1.4) which apparently was built with C >>> locale. But SimpleDateFormat defaults to using the system locale, and >>> thus fails to parse the locale-depedant timestamp format that was >>> used. >>> >>> IMHO the cleanest way is to use a non-locale timestamp, such as RFC 3339. >> >> https://bugs.openjdk.java.net/browse/CODETOOLS-7901226 Um. I am trying to understand this bug. Our build emits the timestamp in a preset format: ${maven.build.timestamp} EEE MMM dd HH:mm:ss zzz yyyy ...and Java code picks up the timestamp in the same format: Date parse = new SimpleDateFormat("EEE MMM dd HH:mm:ss zzz yyyy").parse(time); Does this mean Erich's Maven ignores maven.build.timestamp.format? Erich, can you show the run log? Thanks, -Aleksey. From aleksey.shipilev at oracle.com Mon Dec 22 15:59:18 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Mon, 22 Dec 2014 18:59:18 +0300 Subject: Understanding the code-generation strategy In-Reply-To: References: Message-ID: <54983FD6.4080800@oracle.com> On 12/22/2014 02:21 PM, Matt Warren wrote: > Is it purely related to the overhead of call a method via a reflection > call? I wrote a simple benchmark[2] (using JMH of course!) and assuming I > didn't mess it up, got the following numbers[3]: > > # Run complete. Total time: 00:06:47 > > Benchmark Mode Samples Score > Error Units > o.s.MyBenchmark.baseline avgt 50 0.283 > ? 0.004 ns/op > o.s.MyBenchmark.reflectionMethodCall avgt 50 80.220 > ? 1.082 ns/op > o.s.MyBenchmark.reflectionMethodCallNoCasting avgt 50 79.908 > ? 1.266 ns/op > o.s.MyBenchmark.regularMethodCall avgt 50 77.749 > ? 1.342 ns/op > > > Is it this difference alone that makes the Reflection call per invocation > not possible? For Java/JMH, there are three considerations: a) The cost of the reflective call itself. This is why people would try to "optimize" benchmark payload calls by introducing the repetition count, which breaks horribly in most of the cases. b) The boxing rules. Calling method that returns "int" would autobox the value into Integer, much to surprise the benchmark users. c) The compiler interaction. Hot reflective calls are optimized, and the performance characteristics of optimized reflective call drastically differs from that of the non-optimized call. All these considerations introduce the unwanted effects into the benchmarks. > I'm wondering if I'd need to use this technique in a .NET benchmark harness > as .NET gives you the ability to compile a method call (obtained via > reflection) into a regular delegate, so you don't pay the same penalty when > calling it, see [4] and [5]. Well, and JVM also routinely inflates the hot reflective calls into bytecode, and then optimizes them as the regular methods. But there is an inherent problem with this argument, which I know under the name of "magic compiler card". > My question is, what are the *technical* reasons for doing code generation > this way? I understand the nice benefits you get (can inspect the generated > code), but I see those as side-benefits. You can only play the magic compiler card sporadically. The best place to invoke it is when you argue against micro-optimizing all the code in the project. But when you are dealing with a very sensitive performance code, you have to assume the compiler is dumb. This puts you into the role of the compiler yourself. When you are building the benchmarking harness that, among other things, is here to quantify the compiler effects, you have to minimize the effects of the changing "compiler environment" on your code. This leads to actually generating the low-level code that has a better chance of being translated 1:1 to machine code by different compilers running under different conditions. Generating the synthetic code in JMH allows us to pre-optimize the code, e.g.: * omit the @Setup/@TearDown calls where they are not needed (under assumption the compiler is dumb, and can't inline empty methods or eliminate empty loops) * put the compiler hints over the methods we know are crucial for a sane environment and/or *generate* the parts of the infrastructure code right in the hot methods (under the assumption the compiler is dumb, and can't reliably figure out what to inline for performance) * violate best OOP practices and access the important fields directly (under the assumption the compiler is dumb, and can't devirtualize and/or inline the getters) * cache every important piece of data in locals (under the assumption the compiler is dumb, and can't figure out this by itself -- especially when the memory model issues are involved) * do primitive specializations (under the assumption the compiler is dumb, and can not optimize it reliably) * generate padding for our internal structures to avoid false sharing, (under the assumption the runtime is dumb, and can't avoid it itself) Some of those optimizations are downright non-conservative, and they only work in our isolated use cases. In other words, something that compiler would almost never optimize for; or would try to do so speculatively, which brings the "speculation failed"-checks and such. ------------------------------------------------------------------------ Bottom-line: if you want to have the best (= fast and predictable) performance in most cases, you have to write the specialized code. When you have to write lots of repetitive specialized code, it makes sense to use code generators. That's what JMH does for its benchmark stubs. Thanks, -Aleksey. From erich at debian.org Mon Dec 22 16:03:50 2014 From: erich at debian.org (Erich Schubert) Date: Mon, 22 Dec 2014 17:03:50 +0100 Subject: Locale dependency of JMH, div by 0 in ListStatistics In-Reply-To: <54983422.8050102@oracle.com> References: <54982812.7060900@oracle.com> <5498289D.9070906@oracle.com> <54983422.8050102@oracle.com> Message-ID: Hello Aleksey, I get the error message # Can not figure out JMH version, unable to parse the build time: Fri Dec 19 08:54:04 UTC 2014 And I guess it expects this to be "Fre Dez ..." because I'm in German locale. If I use this pattern to format, I get Mo Dez 22 16:57:48 MEZ 2014 It should be enough to use this: new SimpleDateFormat("EEE MMM dd HH:mm:ss zzz yyyy", Locale.ROOT); (Note the added Locale.ROOT!) As long as the maven build process also ensures root locale - e.g. by setting timestamp.locale. But IMHO the cleaner approach is to avoid any locale dependant field such as EEE MMM and zzz (!) in the timestamp, if you plan on parsing it later again. If parsing fails, it should still report the version number, even when the age is not available. Regards, Erich On Mon, Dec 22, 2014 at 4:09 PM, Aleksey Shipilev wrote: > On 12/22/2014 05:20 PM, Aleksey Shipilev wrote: >>>> 1. Running JMH in a different locale prevents it from reporting its >>>> version number. >>>> I'm using the maven version (1.4) which apparently was built with C >>>> locale. But SimpleDateFormat defaults to using the system locale, and >>>> thus fails to parse the locale-depedant timestamp format that was >>>> used. >>>> >>>> IMHO the cleanest way is to use a non-locale timestamp, such as RFC 3339. >>> >>> https://bugs.openjdk.java.net/browse/CODETOOLS-7901226 > > Um. I am trying to understand this bug. Our build emits the timestamp in > a preset format: > > ${maven.build.timestamp} > EEE MMM dd HH:mm:ss zzz > yyyy > > ...and Java code picks up the timestamp in the same format: > > Date parse = new SimpleDateFormat("EEE MMM dd HH:mm:ss zzz > yyyy").parse(time); > > Does this mean Erich's Maven ignores maven.build.timestamp.format? > Erich, can you show the run log? > > Thanks, > -Aleksey. > From aleksey.shipilev at oracle.com Mon Dec 22 16:27:58 2014 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Mon, 22 Dec 2014 16:27:58 +0000 Subject: hg: code-tools/jmh: 7901226: JMH version number reporting depends on the Locale Message-ID: <201412221627.sBMGRwlh017009@aojmv0008> Changeset: 21d5793de4c0 Author: shade Date: 2014-12-22 19:27 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/21d5793de4c0 7901226: JMH version number reporting depends on the Locale Summary: Remove the Locale-specific bits, store only neutral date, always print the version. ! jmh-core/pom.xml ! jmh-core/src/main/java/org/openjdk/jmh/util/Version.java ! jmh-core/src/main/resources/jmh.properties From aleksey.shipilev at oracle.com Mon Dec 22 16:29:16 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Mon, 22 Dec 2014 19:29:16 +0300 Subject: Locale dependency of JMH, div by 0 in ListStatistics In-Reply-To: References: <54982812.7060900@oracle.com> <5498289D.9070906@oracle.com> <54983422.8050102@oracle.com> Message-ID: <549846DC.90203@oracle.com> Ah, I can see now, thanks! Fixed: http://hg.openjdk.java.net/code-tools/jmh/rev/21d5793de4c0 Good call on always printing the version, fixed as well. -Aleksey. On 12/22/2014 07:03 PM, Erich Schubert wrote: > Hello Aleksey, > I get the error message > # Can not figure out JMH version, unable to parse the build time: Fri > Dec 19 08:54:04 UTC 2014 > > And I guess it expects this to be "Fre Dez ..." because I'm in German locale. > > If I use this pattern to format, I get > Mo Dez 22 16:57:48 MEZ 2014 > > It should be enough to use this: > new SimpleDateFormat("EEE MMM dd HH:mm:ss zzz yyyy", Locale.ROOT); > > (Note the added Locale.ROOT!) > > As long as the maven build process also ensures root locale - e.g. by > setting timestamp.locale. > > But IMHO the cleaner approach is to avoid any locale dependant field > such as EEE MMM and zzz (!) in the timestamp, if you plan on parsing > it later again. > > If parsing fails, it should still report the version number, even when > the age is not available. > > Regards, > Erich > > On Mon, Dec 22, 2014 at 4:09 PM, Aleksey Shipilev > wrote: >> On 12/22/2014 05:20 PM, Aleksey Shipilev wrote: >>>>> 1. Running JMH in a different locale prevents it from reporting its >>>>> version number. >>>>> I'm using the maven version (1.4) which apparently was built with C >>>>> locale. But SimpleDateFormat defaults to using the system locale, and >>>>> thus fails to parse the locale-depedant timestamp format that was >>>>> used. >>>>> >>>>> IMHO the cleanest way is to use a non-locale timestamp, such as RFC 3339. >>>> >>>> https://bugs.openjdk.java.net/browse/CODETOOLS-7901226 >> >> Um. I am trying to understand this bug. Our build emits the timestamp in >> a preset format: >> >> ${maven.build.timestamp} >> EEE MMM dd HH:mm:ss zzz >> yyyy >> >> ...and Java code picks up the timestamp in the same format: >> >> Date parse = new SimpleDateFormat("EEE MMM dd HH:mm:ss zzz >> yyyy").parse(time); >> >> Does this mean Erich's Maven ignores maven.build.timestamp.format? >> Erich, can you show the run log? >> >> Thanks, >> -Aleksey. >> From tom.deneau at amd.com Mon Dec 22 18:42:25 2014 From: tom.deneau at amd.com (Deneau, Tom) Date: Mon, 22 Dec 2014 18:42:25 +0000 Subject: contributed JMH suites Message-ID: Are there any central locations where people contribute JMH suites of micro benchmarks? -- Tom Deneau From staffan.friberg at oracle.com Mon Dec 22 19:03:55 2014 From: staffan.friberg at oracle.com (Staffan Friberg) Date: Mon, 22 Dec 2014 11:03:55 -0800 Subject: contributed JMH suites In-Reply-To: References: Message-ID: <54986B1B.9080607@oracle.com> Hi Tom, We are working on that as part of JEP-230, http://openjdk.java.net/jeps/230 The hope is to get it targeted for JDK 9 shortly and start reviews of builds scripts, structure and add an initial set of benchmarks. Regards, Staffan On 12/22/2014 10:42 AM, Deneau, Tom wrote: > Are there any central locations where people contribute JMH suites of micro benchmarks? > > -- Tom Deneau From matt.warren at live.co.uk Mon Dec 22 22:40:34 2014 From: matt.warren at live.co.uk (Matt Warren) Date: Mon, 22 Dec 2014 22:40:34 +0000 Subject: Understanding the code-generation strategy Message-ID: Hi Aleksey That makes more sense now, thanks for the extra info. I appreciate you taking the time to reply, Cheers Matt On 22 December 2014 at 11:21, Matt Warren wrote: > I'm thinking about writing a .NET version of JMH or at the very least > using several of the (applicable) ideas in JMH to write a .NET benchmarking > framework. > Fortunately (or unfortunately) depending on how you look at it, in .NET > things are a bit simpler, i.e. one-time JITting, no HotSpot, etc > > One thing that I've not been able to understand is the technical reasoning > behind the code-generation strategy used in JMH. I.e. emitting java code > that is then compiled into a single .jar, versus calling the compiled > benchmark code directly via reflection? > > I've read the canonical "JMH vs Caliper: reference thread"[1] and > in-particular the section: > > ------------------------------------------------------------------------- > A. Dynamic selection of benchmarks. > > Since you don't know at "harness" compile time what benchmarks it would > run, the obvious choice would be calling the benchmark methods via > Reflection. Back in the days, this pushed us to accept the same > "repetition" counter in the method to amortize the reflective costs. This > already introduces the major pitfall about looping, see below. > > But infrastructure-wise, harness then should intelligently choose the > repetition count. This almost always leads to calibrating mechanics, which > is almost always broken when loop optimizations are in effect. If one > benchmark is "slower" and autobalances with lower reps count, and another > benchmark is "faster" and autobalances with higher reps count, then > optimizer have more opportunity to optimize "faster" benchmark even > further. Which departs us from seeing how exactly the benchmark performs > and introduces another (hidden! and uncontrollable!) degree of freedom. > > In retrospect, the early days decision of JMH to generate synthetic > benchmark code around the method, which contains the loop (carefully chosen > by us to avoid the optimizations in current VMs -- separation on concerns, > basically), is paying off *very* nicely. We can then call that synthetic > stub via Reflection without even bothering about the costs. > > ...That is not to mention users can actually review the generated > benchmark code looking for the explanations for the weird effects. We do > that frequently as the additional control. > ------------------------------------------------------------------------- > > My question is, what are the *technical* reasons for doing code generation > this way? I understand the nice benefits you get (can inspect the generated > code), but I see those as side-benefits. > > Is it purely related to the overhead of call a method via a reflection > call? I wrote a simple benchmark[2] (using JMH of course!) and assuming I > didn't mess it up, got the following numbers[3]: > > # Run complete. Total time: 00:06:47 > > Benchmark Mode Samples Score Error Units > o.s.MyBenchmark.baseline avgt 50 0.283 ? 0.004 ns/op > o.s.MyBenchmark.reflectionMethodCall avgt 50 80.220 ? 1.082 ns/op > o.s.MyBenchmark.reflectionMethodCallNoCasting avgt 50 79.908 ? 1.266 ns/op > o.s.MyBenchmark.regularMethodCall avgt 50 77.749 ? 1.342 ns/op > > > Is it this difference alone that makes the Reflection call per invocation > not possible? > > I'm wondering if I'd need to use this technique in a .NET benchmark > harness as .NET gives you the ability to compile a method call (obtained > via reflection) into a regular delegate, so you don't pay the same penalty > when calling it, see [4] and [5]. > > Or have I missed the point entirely? > > Cheers > > Matt > > [1] > https://groups.google.com/d/msg/mechanical-sympathy/m4opvy4xq3U/7lY8x8SvHgwJ > [2] https://gist.github.com/mattwarren/75c969ce58e72883a9a0 (Benchmark) > [3] https://gist.github.com/mattwarren/15d88da37c57a222dfd7 (Full Results) > [4] http://msdn.microsoft.com/en-us/library/ms228976(v=vs.110).aspx > [5] > http://msdn.microsoft.com/en-us/library/system.delegate.createdelegate(v=vs.110).aspx > From arteamon at gmail.com Wed Dec 24 12:58:33 2014 From: arteamon at gmail.com (Artem Prigoda) Date: Wed, 24 Dec 2014 16:58:33 +0400 Subject: Possible leak in org.openjdk.jmh.util.Version Message-ID: Hi, Aleksey I don't know if this a convenient question in this mailing list, but during reading the JMH source code my eye caught one little problem. In the class Version http://hg.openjdk.java.net/code-tools/jmh/file/21d5793de4c0/jmh-core/src/main/java/org/openjdk/jmh/util/Version.java an input stream is being opened for reading the file jmh.properties. This stream is then passed to method Properties#load. But this method doesn't close the stream, so the stream is remained opened. Shouldn't it be closed manually or do I miss something? Artem From aleksey.shipilev at oracle.com Thu Dec 25 14:16:56 2014 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 25 Dec 2014 14:16:56 +0000 Subject: hg: code-tools/jmh: 7901230: org.openjdk.jmh.util.Version leaks the InputStream Message-ID: <201412251416.sBPEGusD022676@aojmv0008> Changeset: fd7425c74b22 Author: shade Date: 2014-12-25 17:16 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/fd7425c74b22 7901230: org.openjdk.jmh.util.Version leaks the InputStream ! jmh-core/src/main/java/org/openjdk/jmh/util/Version.java From aleksey.shipilev at oracle.com Thu Dec 25 14:18:27 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 25 Dec 2014 17:18:27 +0300 Subject: Possible leak in org.openjdk.jmh.util.Version In-Reply-To: References: Message-ID: <549C1CB3.4040701@oracle.com> Hi, On 24.12.2014 15:58, Artem Prigoda wrote: > I don't know if this a convenient question in this mailing list, but during > reading the JMH source code my eye caught one little problem. > > In the class Version > http://hg.openjdk.java.net/code-tools/jmh/file/21d5793de4c0/jmh-core/src/main/java/org/openjdk/jmh/util/Version.java > an input stream is being opened for reading the file jmh.properties. This > stream is then passed to method Properties#load. But this method doesn't > close the stream, so the stream is remained opened. Shouldn't it be closed > manually or do I miss something? That's a bug, fixed: https://bugs.openjdk.java.net/browse/CODETOOLS-7901230 Thanks, -Aleksey. From aleksey.shipilev at oracle.com Thu Dec 25 16:15:18 2014 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 25 Dec 2014 16:15:18 +0000 Subject: hg: code-tools/jmh: 7901231: Fix FindBugs warnings in JMH Message-ID: <201412251615.sBPGFIEF013744@aojmv0008> Changeset: c486d1241cc1 Author: shade Date: 2014-12-25 19:15 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/c486d1241cc1 7901231: Fix FindBugs warnings in JMH ! jmh-core/src/main/java/org/openjdk/jmh/generators/core/StateObjectHandler.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/HotspotCompilationProfiler.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/HotspotRuntimeProfiler.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/LinuxPerfAsmProfiler.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/LinuxPerfProfiler.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/AbstractResourceReader.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/BaseBenchmarkHandler.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/ForkedMain.java ! jmh-core/src/main/java/org/openjdk/jmh/util/FileUtils.java ! jmh-core/src/main/java/org/openjdk/jmh/util/Utils.java From aleksey.shipilev at oracle.com Fri Dec 26 11:12:18 2014 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Fri, 26 Dec 2014 14:12:18 +0300 Subject: JMH 1.4.1 Message-ID: <549D4292.1060401@oracle.com> Hi, JMH 1.4.1 is released and available at Maven Central (thanks to Evgeny, as usual). This is a minor patch release, and it fixes a few issues either introduced in 1.4, or lingering from the past versions. Summary of changes: * The result file name was erroneously lower-cased, even if requested by the user in the specific case. This is a lingering issue dating back to at least 1.3. Reported by Jozef Hartinger. Fixed: https://bugs.openjdk.java.net/browse/CODETOOLS-7901225 * Minor issues in statistics calculation are fixed. The results of any previous runs are not affected, since the external visible behavior is the same. Reported by Erich Schubert. Fixed: https://bugs.openjdk.java.net/browse/CODETOOLS-7901227 * JMH version printout introduced in 1.4 was broken when different locales are used for promoted release, and the system where the final JAR is used. Reported by Erich Schubert. Fixed by letting JMH be smarter about this: https://bugs.openjdk.java.net/browse/CODETOOLS-7901226 * JMH version printout also leaked the InputStream. Reported by Artem Prigoda. fixed: https://bugs.openjdk.java.net/browse/CODETOOLS-7901230 * Because of the latter bug report, we have looked into the FindBugs issues accumulated since 1.0 release. Fixed all the important ones: https://bugs.openjdk.java.net/browse/CODETOOLS-7901231 Enjoy! Thanks, -Aleksey From aleksey.shipilev at oracle.com Fri Dec 26 11:19:56 2014 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Fri, 26 Dec 2014 11:19:56 +0000 Subject: hg: code-tools/jmh: 3 new changesets Message-ID: <201412261119.sBQBJvYa024460@aojmv0008> Changeset: 553ca5963167 Author: shade Date: 2014-12-25 19:42 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/553ca5963167 JMH v1.4.1. ! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml ! jmh-archetypes/pom.xml ! jmh-core-benchmarks/pom.xml ! jmh-core-ct/pom.xml ! jmh-core-it/pom.xml ! jmh-core/pom.xml ! jmh-generator-annprocess/pom.xml ! jmh-generator-asm/pom.xml ! jmh-generator-bytecode/pom.xml ! jmh-generator-reflection/pom.xml ! jmh-samples/pom.xml ! pom.xml Changeset: 3f4ff39193a2 Author: shade Date: 2014-12-25 19:42 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/3f4ff39193a2 Added tag 1.4.1 for changeset 553ca5963167 ! .hgtags Changeset: 1ba1bb91b83c Author: shade Date: 2014-12-25 19:42 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/1ba1bb91b83c Continue in 1.5-SNAPSHOT. ! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml ! jmh-archetypes/pom.xml ! jmh-core-benchmarks/pom.xml ! jmh-core-ct/pom.xml ! jmh-core-it/pom.xml ! jmh-core/pom.xml ! jmh-generator-annprocess/pom.xml ! jmh-generator-asm/pom.xml ! jmh-generator-bytecode/pom.xml ! jmh-generator-reflection/pom.xml ! jmh-samples/pom.xml ! pom.xml From mandrikov at gmail.com Tue Dec 30 14:35:13 2014 From: mandrikov at gmail.com (Evgeny Mandrikov) Date: Tue, 30 Dec 2014 15:35:13 +0100 Subject: Unterminated StackProfiler thread in case of failure of benchmark Message-ID: Hi Aleksey, Seems that there is no notification for profilers in case of failure of benchmark ( LoopBenchmarkHandler can exit with BenchmarkException before reaching stopProfilers ) , so that forks won't be terminated if profiler spawns thread. This is the case for StackProfiler and can be reproduced by addition of ".addProfiler(StackProfiler.class)" into FailingForkedBenchTest. -- Best regards, Evgeny Mandrikov aka Godin http://twitter.com/_godin_