From valentin.male.kovalenko at gmail.com Tue Feb 5 12:20:07 2019 From: valentin.male.kovalenko at gmail.com (Valentin Kovalenko) Date: Tue, 5 Feb 2019 05:20:07 -0700 Subject: a situation when JHM 1.21 measurements are broken Message-ID: I encountered a simple situation where JMH 1.21 produces obviously incorrect results. The test (is also temporarily available at https://github.com/stIncMale/sandbox/blob/master/benchmarks/src/test/java/stincmale/sandbox/benchmarks/TmpTest.java , you can run it with `mvn clean test -f benchmarks/pom.xml -Dtest=TmpTest`): public class TmpTest { public TmpTest() { } public final void runThroughputBenchmarks(final int numberOfThreads) throws RunnerException { final OptionsBuilder opts = new OptionsBuilder(); opts.jvmArgs("-Xfuture", "--illegal-access=deny", "-Xshare:off", "-Xms1024m", "-Xmx1024m", "-server", "-disableassertions") .shouldDoGC(false) .syncIterations(true) .shouldFailOnError(true) .threads(1) .timeout(milliseconds(1000_000)) .forks(2) .warmupTime(milliseconds(1000)) .warmupIterations(6) .measurementTime(milliseconds(1000)) .measurementIterations(2) .include(includeBenchmarks(TmpTest.class)) .shouldDoGC(false) .mode(Mode.Throughput) .timeUnit(TimeUnit.MILLISECONDS) .threads(numberOfThreads); new Runner(opts.build()).run(); } @Test public final void throughputThreads4() throws RunnerException { runThroughputBenchmarks(4); } @Test public final void throughputThreads32() throws RunnerException { runThroughputBenchmarks(32); } @Benchmark public final long reentrantRwLock(final BenchmarkState s) throws InterruptedException { s.rwLock.readLock().lock(); try { if (s.counter.getAndIncrement() < 1000) { return s.counter.get(); } } finally { s.rwLock.readLock().unlock(); } s.rwLock.writeLock().lock(); try { Thread.sleep(10); s.counter.set(0); } finally { s.rwLock.writeLock().unlock(); } return s.counter.get(); } @Benchmark public final long stampedLock(final BenchmarkState state) throws InterruptedException { long stamp = state.lock.readLock(); try { if (state.counter.getAndIncrement() < 1000) { return state.counter.get(); } } finally { state.lock.unlockRead(stamp); } stamp = state.lock.writeLock(); try { Thread.sleep(10); state.counter.set(0); } finally { state.lock.unlockWrite(stamp); } return state.counter.get(); } @State(Scope.Benchmark) public static class BenchmarkState { private StampedLock lock; private ReentrantReadWriteLock rwLock; private AtomicLong counter; public BenchmarkState() { } @Setup(Level.Trial) public final void setup() { lock = new StampedLock(); rwLock = new ReentrantReadWriteLock(); counter = new AtomicLong(); } } } Obviously, there is no way the throughput for this benchmarks is greater than 1000 ops / 10ms = 100 ops/ms. And JMH produces stable results lesser than 50 ops/ms for 4 threads for both reentrantRwLock and stampedLock. However, for 32 threads JMH produces sane results only for reentrantRwLock, but for stampedLock the run looks like this: # Fork: 1 of 2 # Warmup Iteration 1: 665.471 ops/ms # Warmup Iteration 2: 782.521 ops/ms # Warmup Iteration 3: 32.830 ops/ms # Warmup Iteration 4: 43.356 ops/ms # Warmup Iteration 5: 2573.165 ops/ms # Warmup Iteration 6: 408.397 ops/ms Iteration 1: 9072.582 ops/ms Iteration 2: 2492.579 ops/ms # Fork: 2 of 2 # Warmup Iteration 1: 10.445 ops/ms # Warmup Iteration 2: 27.372 ops/ms # Warmup Iteration 3: 2632.625 ops/ms # Warmup Iteration 4: 13.445 ops/ms # Warmup Iteration 5: 16.657 ops/ms # Warmup Iteration 6: 28.452 ops/ms Iteration 1: 30.703 ops/ms Iteration 2: 27.807 ops/ms These numbers make no sense because as I mentioned, in these benchmarks the throughput can never be higher than 100 ops/ms. Is there a valid explanation for this JMH behaviour, or is this actually a bug in JMH? Regards, Valentin [image: LinkedIn] [image: GitHub] [image: YouTube] From shade at redhat.com Tue Feb 5 14:37:36 2019 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 5 Feb 2019 15:37:36 +0100 Subject: a situation when JHM 1.21 measurements are broken In-Reply-To: References: Message-ID: <9191cb86-7eb1-94b0-fe1c-69c466360a3d@redhat.com> On 2/5/19 1:20 PM, Valentin Kovalenko wrote: > I encountered a simple situation where JMH 1.21 produces obviously > incorrect results. The test (is also temporarily available at > https://github.com/stIncMale/sandbox/blob/master/benchmarks/src/test/java/stincmale/sandbox/benchmarks/TmpTest.java > , > you can run it with `mvn clean test -f benchmarks/pom.xml -Dtest=TmpTest`): It is customary to do MCVE that does not have any dependencies except JMH itself, like this: import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicLong; import java.util.concurrent.locks.StampedLock; import org.openjdk.jmh.annotations.*; @Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @Measurement(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @Fork(1) @BenchmarkMode(Mode.Throughput) @OutputTimeUnit(TimeUnit.SECONDS) @State(Scope.Benchmark) public class TmpTest { private StampedLock lock; private AtomicLong counter; @Setup(Level.Trial) public final void setup() { lock = new StampedLock(); counter = new AtomicLong(); } @Benchmark public final long stampedLock() throws InterruptedException { long stamp = lock.readLock(); try { if (counter.getAndIncrement() < 1000) { return counter.get(); } } finally { lock.unlockRead(stamp); } stamp = lock.writeLock(); try { Thread.sleep(10); counter.set(0); } finally { lock.unlockWrite(stamp); } return counter.get(); } } > Obviously, there is no way the throughput for this benchmarks is greater > than 1000 ops / 10ms = 100 ops/ms. And JMH produces stable results lesser > than 50 ops/ms for 4 threads for both reentrantRwLock and stampedLock. > However, for 32 threads JMH produces sane results only for reentrantRwLock, > but for stampedLock the run looks like this: Every time "obviously" is spelled somewhere, we need to challenge that "obviously" first. I can see how you can prove that _a single thread_ cannot do more than 100 ops/ms: indeed, it would have to sleep 10 ms each 1000 ops. But this does not extend to multiple threads, because other threads can "reset" the counter at unfortunate times. Consider N threads, all N threads have failed the increment, all of them queue up for writeLock acquisition. One of those threads succeeds, enters, resets the counter, and proceeds to do increments under readLock. But, other N-1 threads are entering and resetting the counter when first thread is doing its own increments. So, in this failure scenario, the worst case is incrementer thread doing 999 increments, and all N-1 threads reset the counter to zero "just then". Each of those threads would deliver no more than (N-1) resets each 10 ms. Therefore, the incrementer thread can perceive the throughput of 999 * (N-1) / 10 ops/ms. Or, it your example of 32 threads, up to around 3100 ops/ms. This would be exacerbated by more threads actually stalling on writeLock acquisition, so more threads do more outliers. Pretty sure it would be less pronunciated if pending writeLock acquisition starve the readLock acquisitions, which might explain why ReentrantRWLock "appears" unaffected. Pretty sure there are other failure modes in this benchmark, but this should get you the whiff of how non-obvious this benchmark really is. The outliers do not point to JMH bug, IMO. -Aleksey From sitnikov.vladimir at gmail.com Tue Feb 5 15:46:32 2019 From: sitnikov.vladimir at gmail.com (Vladimir Sitnikov) Date: Tue, 5 Feb 2019 18:46:32 +0300 Subject: a situation when JHM 1.21 measurements are broken In-Reply-To: <9191cb86-7eb1-94b0-fe1c-69c466360a3d@redhat.com> References: <9191cb86-7eb1-94b0-fe1c-69c466360a3d@redhat.com> Message-ID: Valentin, have you tried replacing counter.set(0); with counter.getAndAdd(-1000)? Vladimir From valentin.male.kovalenko at gmail.com Wed Feb 6 09:23:57 2019 From: valentin.male.kovalenko at gmail.com (Valentin Kovalenko) Date: Wed, 6 Feb 2019 02:23:57 -0700 Subject: a situation when JHM 1.21 measurements are broken In-Reply-To: <9191cb86-7eb1-94b0-fe1c-69c466360a3d@redhat.com> References: <9191cb86-7eb1-94b0-fe1c-69c466360a3d@redhat.com> Message-ID: Thanks for the answers! Sorry for the delayed reply, life strikes back. Aleksey, you are right, my interpretation of the benchmarks was completely broken. I will think of that more before continuing with this thread, but will most likely return :) Vladimir, state.counter.getAndAdd(-1000) instead of state.counter.set(0) does not make a difference: # Fork: 1 of 2 # Warmup Iteration 1: 74.487 ops/ms # Warmup Iteration 2: 56.800 ops/ms # Warmup Iteration 3: 56.952 ops/ms # Warmup Iteration 4: 69.635 ops/ms # Warmup Iteration 5: 8345.231 ops/ms # Warmup Iteration 6: 22419.179 ops/ms Iteration 1: 11798.305 ops/ms Iteration 2: 9974.925 ops/ms From grv87 at yandex.ru Sat Feb 9 03:46:52 2019 From: grv87 at yandex.ru (Basil Peace) Date: Sat, 09 Feb 2019 06:46:52 +0300 Subject: [PATCH]: Still a problem with long classpath on Windows In-Reply-To: References: <55598841546996798@myt6-fe24916a5562.qloud-c.yandex.net> Message-ID: <54948071549684012@sas2-2074c606c35d.qloud-c.yandex.net> > ?a) Sign the OCA, as per http://openjdk.java.net/contribute/ Done. > ?b) Put the patch somewhere on OpenJDK infra. Posting the simple patch in-line in this message would > be good enough. Improved patch, using URI for percent encoding: # HG changeset patch # User Basil Peace # Date 1549196743 -10800 # Sun Feb 03 15:25:43 2019 +0300 # Branch hotfix/long-classpath-absolute.v2 # Node ID fba2e675a44b2b8aaee3fd63aefe926e17fac945 # Parent 5984e353dca775da0e2208ddaed8427cd9a43acd fix: fix long classpath issue on Windows with different drives diff -r 5984e353dca7 -r fba2e675a44b jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java --- a/jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java Tue Jan 22 16:22:44 2019 +0100 +++ b/jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java Sun Feb 03 15:25:43 2019 +0300 @@ -42,10 +42,13 @@ import java.io.*; import java.lang.management.ManagementFactory; +import java.net.URI; +import java.net.URISyntaxException; import java.nio.channels.FileChannel; import java.nio.channels.FileLock; import java.nio.channels.OverlappingFileLockException; import java.nio.file.Path; +import java.nio.file.Paths; import java.util.*; import java.util.concurrent.TimeUnit; import java.util.jar.*; @@ -873,14 +876,25 @@ try { tmpFile = FileUtils.tempFile("classpath.jar"); Path tmpFileDir = tmpFile.toPath().getParent(); + String tmpFileRoot = tmpFileDir.toAbsolutePath().getRoot().toString(); StringBuilder sb = new StringBuilder(); for (String cp : cpProp.split(File.pathSeparator)) { - String rel = tmpFileDir.relativize(new File(cp).getAbsoluteFile().toPath()).toString(); - sb.append(rel.replace('\\', '/').replace(" ", "%20")); - if (!cp.endsWith(".jar")) { - sb.append('/'); + Path cpPath = Paths.get(cp).toAbsolutePath(); + URI cpUri; + if (cpPath.getRoot().toString().equalsIgnoreCase(tmpFileRoot)) { + String rel = tmpFileDir.relativize(cpPath).toString(); + if (File.separatorChar != '/') { + rel = rel.replace(File.separatorChar, '/'); + } + if (!rel.endsWith(".jar")) { + rel = rel + '/'; + } + cpUri = new URI(null, rel, null); + } else { + cpUri = cpPath.toUri(); } + sb.append(cpUri.toString()); sb.append(" "); } String classPath = sb.toString().trim(); @@ -893,8 +907,10 @@ try (JarOutputStream jos = new JarOutputStream(new FileOutputStream(tmpFile), manifest)) { jos.putNextEntry(new ZipEntry("META-INF/")); } - } catch (IOException ex) { - // Something is wrong in file generation, give up and fall-through to usual thing + } catch (IOException | URISyntaxException ex) { + out.println(""); + out.println(""); + out.println(""); tmpFile = null; } } -- Best regards, Basil Peace From grv87 at yandex.ru Thu Feb 28 06:19:13 2019 From: grv87 at yandex.ru (Basil Peace) Date: Thu, 28 Feb 2019 09:19:13 +0300 Subject: [PATCH]: Still a problem with long classpath on Windows In-Reply-To: <54948071549684012@sas2-2074c606c35d.qloud-c.yandex.net> References: <55598841546996798@myt6-fe24916a5562.qloud-c.yandex.net> <54948071549684012@sas2-2074c606c35d.qloud-c.yandex.net> Message-ID: <785341551334753@sas2-a271c7163765.qloud-c.yandex.net> Hi, Aleksey! Could you take a look at the proposed patch? -- Best regards, Basil Peace 09.02.2019, 06:46, "Basil Peace" : >> ??a) Sign the OCA, as per http://openjdk.java.net/contribute/ > > Done. > >> ??b) Put the patch somewhere on OpenJDK infra. Posting the simple patch in-line in this message would >> ?be good enough. > > Improved patch, using URI for percent encoding: > > # HG changeset patch > # User Basil Peace > # Date 1549196743 -10800 > # Sun Feb 03 15:25:43 2019 +0300 > # Branch hotfix/long-classpath-absolute.v2 > # Node ID fba2e675a44b2b8aaee3fd63aefe926e17fac945 > # Parent 5984e353dca775da0e2208ddaed8427cd9a43acd > fix: fix long classpath issue on Windows with different drives > > diff -r 5984e353dca7 -r fba2e675a44b jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java > --- a/jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java Tue Jan 22 16:22:44 2019 +0100 > +++ b/jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java Sun Feb 03 15:25:43 2019 +0300 > @@ -42,10 +42,13 @@ > > ?import java.io.*; > ?import java.lang.management.ManagementFactory; > +import java.net.URI; > +import java.net.URISyntaxException; > ?import java.nio.channels.FileChannel; > ?import java.nio.channels.FileLock; > ?import java.nio.channels.OverlappingFileLockException; > ?import java.nio.file.Path; > +import java.nio.file.Paths; > ?import java.util.*; > ?import java.util.concurrent.TimeUnit; > ?import java.util.jar.*; > @@ -873,14 +876,25 @@ > ?????????????try { > ?????????????????tmpFile = FileUtils.tempFile("classpath.jar"); > ?????????????????Path tmpFileDir = tmpFile.toPath().getParent(); > + String tmpFileRoot = tmpFileDir.toAbsolutePath().getRoot().toString(); > > ?????????????????StringBuilder sb = new StringBuilder(); > ?????????????????for (String cp : cpProp.split(File.pathSeparator)) { > - String rel = tmpFileDir.relativize(new File(cp).getAbsoluteFile().toPath()).toString(); > - sb.append(rel.replace('\\', '/').replace(" ", "%20")); > - if (!cp.endsWith(".jar")) { > - sb.append('/'); > + Path cpPath = Paths.get(cp).toAbsolutePath(); > + URI cpUri; > + if (cpPath.getRoot().toString().equalsIgnoreCase(tmpFileRoot)) { > + String rel = tmpFileDir.relativize(cpPath).toString(); > + if (File.separatorChar != '/') { > + rel = rel.replace(File.separatorChar, '/'); > + } > + if (!rel.endsWith(".jar")) { > + rel = rel + '/'; > + } > + cpUri = new URI(null, rel, null); > + } else { > + cpUri = cpPath.toUri(); > ?????????????????????} > + sb.append(cpUri.toString()); > ?????????????????????sb.append(" "); > ?????????????????} > ?????????????????String classPath = sb.toString().trim(); > @@ -893,8 +907,10 @@ > ?????????????????try (JarOutputStream jos = new JarOutputStream(new FileOutputStream(tmpFile), manifest)) { > ?????????????????????jos.putNextEntry(new ZipEntry("META-INF/")); > ?????????????????} > - } catch (IOException ex) { > - // Something is wrong in file generation, give up and fall-through to usual thing > + } catch (IOException | URISyntaxException ex) { > + out.println(""); > + out.println(""); > + out.println(""); > ?????????????????tmpFile = null; > ?????????????} > ?????????} > > -- > Best regards, > Basil Peace