From alex.averbuch at neotechnology.com Thu Jul 7 14:04:09 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Thu, 7 Jul 2016 17:04:09 +0300 Subject: Infra Params Message-ID: In the JMH samples there is an example of how to access BenchmarkParams during benchmark execution, e.g.: @Setup public void setup( BenchmarkParams params ) The sample I'm referring to is: http://hg.openjdk.java.net/code-tools/jmh/file/13ae22114e22/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_31_InfraParams.java I'd like to do this, e.g., for retrieving/then storing per-benchmark configuration data to the local file system, for later archiving. But, when I try in my own project I get the following error: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project jmh-benchmarks: Compilation failure: Compilation failure: [ERROR] /Users/me/IdeaProjects/jmh-benchmarks/target/generated-sources/annotations/org/db/bench/core/generated/FindNode_countNodesWithLabel_jmhTest.java:[417,16] method setUp in class org.db.bench.core.AbstractCoreBenchmark cannot be applied to given types; [ERROR] required: org.openjdk.jmh.infra.BenchmarkParams [ERROR] found: org.openjdk.jmh.infra.generated.BenchmarkParams_jmhType,org.openjdk.jmh.infra.generated.BenchmarkParams_jmhType [ERROR] reason: actual and formal argument lists differ in length [ERROR] /Users/me/IdeaProjects/jmh-benchmarks/target/generated-sources/annotations/org/db/bench/core/generated/FindNode_countNodesWithLabelAndKeyAndValue_jmhTest.java:[426,16] method setUp in class org.db.bench.core.AbstractCoreBenchmark cannot be applied to given types; AbstractCoreBenchmark is an abstract class with @State( Scope.Benchmark ) I've tried passing the BenchmarkParams instance into non-abstract children of AbstractCoreBenchmark but get the same error Thoughts? Thanks in advance! Alex From alex.averbuch at neotechnology.com Thu Jul 7 14:04:09 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Thu, 7 Jul 2016 17:04:09 +0300 Subject: Infra Params Message-ID: In the JMH samples there is an example of how to access BenchmarkParams during benchmark execution, e.g.: @Setup public void setup( BenchmarkParams params ) The sample I'm referring to is: http://hg.openjdk.java.net/code-tools/jmh/file/13ae22114e22/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_31_InfraParams.java I'd like to do this, e.g., for retrieving/then storing per-benchmark configuration data to the local file system, for later archiving. But, when I try in my own project I get the following error: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project jmh-benchmarks: Compilation failure: Compilation failure: [ERROR] /Users/me/IdeaProjects/jmh-benchmarks/target/generated-sources/annotations/org/db/bench/core/generated/FindNode_countNodesWithLabel_jmhTest.java:[417,16] method setUp in class org.db.bench.core.AbstractCoreBenchmark cannot be applied to given types; [ERROR] required: org.openjdk.jmh.infra.BenchmarkParams [ERROR] found: org.openjdk.jmh.infra.generated.BenchmarkParams_jmhType,org.openjdk.jmh.infra.generated.BenchmarkParams_jmhType [ERROR] reason: actual and formal argument lists differ in length [ERROR] /Users/me/IdeaProjects/jmh-benchmarks/target/generated-sources/annotations/org/db/bench/core/generated/FindNode_countNodesWithLabelAndKeyAndValue_jmhTest.java:[426,16] method setUp in class org.db.bench.core.AbstractCoreBenchmark cannot be applied to given types; AbstractCoreBenchmark is an abstract class with @State( Scope.Benchmark ) I've tried passing the BenchmarkParams instance into non-abstract children of AbstractCoreBenchmark but get the same error Thoughts? Thanks in advance! Alex From alex.averbuch at neotechnology.com Fri Jul 8 07:31:37 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Fri, 8 Jul 2016 10:31:37 +0300 Subject: Infra Params In-Reply-To: References: Message-ID: Just realized I didn't share the signatures for the related methods. They (methods from FindNode) are as follows: @Benchmark @BenchmarkMode( Mode.SampleTime ) public long countNodesWithLabel( TxState txState ) @Benchmark @BenchmarkMode( Mode.SampleTime ) public long countNodesWithLabelAndKeyAndValue( TxState txState, RNGState rngState ) and FindNode extends AbstractCoreBenchmark On Thu, Jul 7, 2016 at 5:04 PM, Alex Averbuch < alex.averbuch at neotechnology.com> wrote: > In the JMH samples there is an example of how to access BenchmarkParams > during benchmark execution, e.g.: > > @Setup > public void setup( BenchmarkParams params ) > > > The sample I'm referring to is: > > http://hg.openjdk.java.net/code-tools/jmh/file/13ae22114e22/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_31_InfraParams.java > > I'd like to do this, e.g., for retrieving/then storing per-benchmark > configuration data to the local file system, for later archiving. > > But, when I try in my own project I get the following error: > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:3.1:compile > (default-compile) on project jmh-benchmarks: Compilation failure: > Compilation failure: > [ERROR] > /Users/me/IdeaProjects/jmh-benchmarks/target/generated-sources/annotations/org/db/bench/core/generated/FindNode_countNodesWithLabel_jmhTest.java:[417,16] > method setUp in class org.db.bench.core.AbstractCoreBenchmark cannot be > applied to given types; > [ERROR] required: org.openjdk.jmh.infra.BenchmarkParams > [ERROR] found: > org.openjdk.jmh.infra.generated.BenchmarkParams_jmhType,org.openjdk.jmh.infra.generated.BenchmarkParams_jmhType > [ERROR] reason: actual and formal argument lists differ in length > [ERROR] > /Users/me/IdeaProjects/jmh-benchmarks/target/generated-sources/annotations/org/db/bench/core/generated/FindNode_countNodesWithLabelAndKeyAndValue_jmhTest.java:[426,16] > method setUp in class org.db.bench.core.AbstractCoreBenchmark cannot be > applied to given types; > > > AbstractCoreBenchmark is an abstract class with @State( Scope.Benchmark ) > > I've tried passing the BenchmarkParams instance into non-abstract children > of AbstractCoreBenchmark but get the same error > > Thoughts? > > Thanks in advance! > Alex > From alex.averbuch at neotechnology.com Fri Jul 8 09:01:53 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Fri, 8 Jul 2016 12:01:53 +0300 Subject: Infra Params In-Reply-To: References: Message-ID: both of the following variants result in similar errors --------------------------------------------------------------------------------- --- VARIANT 1 : BenchmarkParams in abstract base class --- --------------------------------------------------------------------------------- @State( Scope.Benchmark ) public abstract class AbstractCoreBenchmark { @Setup public void setUp( BenchmarkParams params ) { // setup } @TearDown public void tearDown() { // tear down } @State( Scope.Thread ) public static class RNGState { public final ThreadLocalRandom rng = ThreadLocalRandom.current(); } } public class RelationshipRead extends AbstractCoreBenchmark { @Setup public void setup( ) { // setup } @State( Scope.Thread ) public static class TxState { // open/close transactions } @Benchmark @BenchmarkMode( Mode.SampleTime ) public int countRelationships( TxState txState, RNGState rngState ) { // count relationships } } ERROR] /Users/me/IdeaProjects/jmh-benchmarks/target/generated-sources/annotations/org/db/bench/core/generated/RelationshipRead_countRelationships_jmhTest.java:[426,16] method setUp in class org.db.bench.core.AbstractCoreBenchmark cannot be applied to given types; required: org.openjdk.jmh.infra.BenchmarkParams found: org.openjdk.jmh.infra.generated.BenchmarkParams_jmhType,org.openjdk.jmh.infra.generated.BenchmarkParams_jmhType reason: actual and formal argument lists differ in length --------------------------------------------------------------------- --- VARIANT 2 : BenchmarkParams in child class --- --------------------------------------------------------------------- @State( Scope.Benchmark ) public abstract class AbstractCoreBenchmark { @Setup public void setUp( ) { // setup } @TearDown public void tearDown() { // tear down } @State( Scope.Thread ) public static class RNGState { public final ThreadLocalRandom rng = ThreadLocalRandom.current(); } } public class RelationshipRead extends AbstractCoreBenchmark { @Setup public void setup( BenchmarkParams params ) { // setup } @State( Scope.Thread ) public static class TxState { // open/close transactions } @Benchmark @BenchmarkMode( Mode.SampleTime ) public int countRelationships( TxState txState, RNGState rngState ) { // count relationships } } [ERROR] /Users/me/IdeaProjects/jmh-benchmarks/target/generated-sources/annotations/org/db/bench/core/generated/RelationshipRead_countRelationships_jmhTest.java:[427,16] method setup in class org.db.bench.core.RelationshipRead cannot be applied to given types; required: org.openjdk.jmh.infra.BenchmarkParams found: org.openjdk.jmh.infra.generated.BenchmarkParams_jmhType,org.openjdk.jmh.infra.generated.BenchmarkParams_jmhType reason: actual and formal argument lists differ in length From alex.averbuch at neotechnology.com Fri Jul 8 10:42:06 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Fri, 8 Jul 2016 13:42:06 +0300 Subject: Infra Params In-Reply-To: References: Message-ID: I have a minimal example for reproducing this error, it goes like this: (1) Define @State(Scope.Benchmark) class named Test <--- OK (2) Define @Benchmark method named nothing() <--- OK (3) Pass @State instance into @Benchmark method, e.g. public void nothing( Test test ) <--- ERROR! The error: [ERROR] /Users/me/IdeaProjects/jmh-benchmarks/target/generated-sources/annotations/org/db/bench/core/generated/Test_nothing_jmhTest.java:[390,16] method setUp in class org.db.bench.core.Test cannot be applied to given types; required: org.openjdk.jmh.infra.BenchmarkParams found: org.openjdk.jmh.infra.generated.BenchmarkParams_jmhType,org.openjdk.jmh.infra.generated.BenchmarkParams_jmhType reason: actual and formal argument lists differ in length The code: @State( Scope.Benchmark ) public class Test { @Setup public void setUp( BenchmarkParams params ){} @Benchmark public void nothing( Test test ){} } From alex.averbuch at neotechnology.com Fri Jul 8 14:42:21 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Fri, 8 Jul 2016 17:42:21 +0300 Subject: Infra Params In-Reply-To: References: Message-ID: http://stackoverflow.com/questions/38268510/jmh-accessing-benchmarkparams-from-setup-method-of-state-class From alex.averbuch at neotechnology.com Fri Jul 8 14:42:21 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Fri, 8 Jul 2016 17:42:21 +0300 Subject: Infra Params In-Reply-To: References: Message-ID: http://stackoverflow.com/questions/38268510/jmh-accessing-benchmarkparams-from-setup-method-of-state-class From alex.averbuch at neotechnology.com Fri Jul 8 10:42:06 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Fri, 8 Jul 2016 13:42:06 +0300 Subject: Infra Params In-Reply-To: References: Message-ID: I have a minimal example for reproducing this error, it goes like this: (1) Define @State(Scope.Benchmark) class named Test <--- OK (2) Define @Benchmark method named nothing() <--- OK (3) Pass @State instance into @Benchmark method, e.g. public void nothing( Test test ) <--- ERROR! The error: [ERROR] /Users/me/IdeaProjects/jmh-benchmarks/target/generated-sources/annotations/org/db/bench/core/generated/Test_nothing_jmhTest.java:[390,16] method setUp in class org.db.bench.core.Test cannot be applied to given types; required: org.openjdk.jmh.infra.BenchmarkParams found: org.openjdk.jmh.infra.generated.BenchmarkParams_jmhType,org.openjdk.jmh.infra.generated.BenchmarkParams_jmhType reason: actual and formal argument lists differ in length The code: @State( Scope.Benchmark ) public class Test { @Setup public void setUp( BenchmarkParams params ){} @Benchmark public void nothing( Test test ){} } From alex.averbuch at neotechnology.com Fri Jul 8 09:01:53 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Fri, 8 Jul 2016 12:01:53 +0300 Subject: Infra Params In-Reply-To: References: Message-ID: both of the following variants result in similar errors --------------------------------------------------------------------------------- --- VARIANT 1 : BenchmarkParams in abstract base class --- --------------------------------------------------------------------------------- @State( Scope.Benchmark ) public abstract class AbstractCoreBenchmark { @Setup public void setUp( BenchmarkParams params ) { // setup } @TearDown public void tearDown() { // tear down } @State( Scope.Thread ) public static class RNGState { public final ThreadLocalRandom rng = ThreadLocalRandom.current(); } } public class RelationshipRead extends AbstractCoreBenchmark { @Setup public void setup( ) { // setup } @State( Scope.Thread ) public static class TxState { // open/close transactions } @Benchmark @BenchmarkMode( Mode.SampleTime ) public int countRelationships( TxState txState, RNGState rngState ) { // count relationships } } ERROR] /Users/me/IdeaProjects/jmh-benchmarks/target/generated-sources/annotations/org/db/bench/core/generated/RelationshipRead_countRelationships_jmhTest.java:[426,16] method setUp in class org.db.bench.core.AbstractCoreBenchmark cannot be applied to given types; required: org.openjdk.jmh.infra.BenchmarkParams found: org.openjdk.jmh.infra.generated.BenchmarkParams_jmhType,org.openjdk.jmh.infra.generated.BenchmarkParams_jmhType reason: actual and formal argument lists differ in length --------------------------------------------------------------------- --- VARIANT 2 : BenchmarkParams in child class --- --------------------------------------------------------------------- @State( Scope.Benchmark ) public abstract class AbstractCoreBenchmark { @Setup public void setUp( ) { // setup } @TearDown public void tearDown() { // tear down } @State( Scope.Thread ) public static class RNGState { public final ThreadLocalRandom rng = ThreadLocalRandom.current(); } } public class RelationshipRead extends AbstractCoreBenchmark { @Setup public void setup( BenchmarkParams params ) { // setup } @State( Scope.Thread ) public static class TxState { // open/close transactions } @Benchmark @BenchmarkMode( Mode.SampleTime ) public int countRelationships( TxState txState, RNGState rngState ) { // count relationships } } [ERROR] /Users/me/IdeaProjects/jmh-benchmarks/target/generated-sources/annotations/org/db/bench/core/generated/RelationshipRead_countRelationships_jmhTest.java:[427,16] method setup in class org.db.bench.core.RelationshipRead cannot be applied to given types; required: org.openjdk.jmh.infra.BenchmarkParams found: org.openjdk.jmh.infra.generated.BenchmarkParams_jmhType,org.openjdk.jmh.infra.generated.BenchmarkParams_jmhType reason: actual and formal argument lists differ in length From alex.averbuch at neotechnology.com Fri Jul 8 07:31:37 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Fri, 8 Jul 2016 10:31:37 +0300 Subject: Infra Params In-Reply-To: References: Message-ID: Just realized I didn't share the signatures for the related methods. They (methods from FindNode) are as follows: @Benchmark @BenchmarkMode( Mode.SampleTime ) public long countNodesWithLabel( TxState txState ) @Benchmark @BenchmarkMode( Mode.SampleTime ) public long countNodesWithLabelAndKeyAndValue( TxState txState, RNGState rngState ) and FindNode extends AbstractCoreBenchmark On Thu, Jul 7, 2016 at 5:04 PM, Alex Averbuch < alex.averbuch at neotechnology.com> wrote: > In the JMH samples there is an example of how to access BenchmarkParams > during benchmark execution, e.g.: > > @Setup > public void setup( BenchmarkParams params ) > > > The sample I'm referring to is: > > http://hg.openjdk.java.net/code-tools/jmh/file/13ae22114e22/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_31_InfraParams.java > > I'd like to do this, e.g., for retrieving/then storing per-benchmark > configuration data to the local file system, for later archiving. > > But, when I try in my own project I get the following error: > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:3.1:compile > (default-compile) on project jmh-benchmarks: Compilation failure: > Compilation failure: > [ERROR] > /Users/me/IdeaProjects/jmh-benchmarks/target/generated-sources/annotations/org/db/bench/core/generated/FindNode_countNodesWithLabel_jmhTest.java:[417,16] > method setUp in class org.db.bench.core.AbstractCoreBenchmark cannot be > applied to given types; > [ERROR] required: org.openjdk.jmh.infra.BenchmarkParams > [ERROR] found: > org.openjdk.jmh.infra.generated.BenchmarkParams_jmhType,org.openjdk.jmh.infra.generated.BenchmarkParams_jmhType > [ERROR] reason: actual and formal argument lists differ in length > [ERROR] > /Users/me/IdeaProjects/jmh-benchmarks/target/generated-sources/annotations/org/db/bench/core/generated/FindNode_countNodesWithLabelAndKeyAndValue_jmhTest.java:[426,16] > method setUp in class org.db.bench.core.AbstractCoreBenchmark cannot be > applied to given types; > > > AbstractCoreBenchmark is an abstract class with @State( Scope.Benchmark ) > > I've tried passing the BenchmarkParams instance into non-abstract children > of AbstractCoreBenchmark but get the same error > > Thoughts? > > Thanks in advance! > Alex > From alex.averbuch at neotechnology.com Mon Jul 11 20:54:58 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Mon, 11 Jul 2016 23:54:58 +0300 Subject: Infra Params In-Reply-To: References: Message-ID: SOLVED. See: http://stackoverflow.com/questions/38268510/jmh-accessing-benchmarkparams-from-setup-method-of-state-class On Fri, Jul 8, 2016 at 5:42 PM, Alex Averbuch < alex.averbuch at neotechnology.com> wrote: > > http://stackoverflow.com/questions/38268510/jmh-accessing-benchmarkparams-from-setup-method-of-state-class > From aleksey.shipilev at oracle.com Tue Jul 12 19:09:29 2016 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Tue, 12 Jul 2016 19:09:29 +0000 Subject: hg: code-tools/jmh: 7901729: Usability: Print thread distribution for asymmetric benchmarks Message-ID: <201607121909.u6CJ9TmG020211@aojmv0008.oracle.com> Changeset: 086e57a61659 Author: shade Date: 2016-07-12 22:06 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/086e57a61659 7901729: Usability: Print thread distribution for asymmetric benchmarks + jmh-core-it/src/test/java/org/openjdk/jmh/it/threads/GroupThreadGroupOrderTest.java ! jmh-core/src/main/java/org/openjdk/jmh/generators/core/BenchmarkGenerator.java ! jmh-core/src/main/java/org/openjdk/jmh/generators/core/MethodGroup.java ! jmh-core/src/main/java/org/openjdk/jmh/infra/BenchmarkParams.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/BenchmarkListEntry.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/format/TextReportFormat.java ! jmh-core/src/test/java/org/openjdk/jmh/results/TestAggregateResult.java ! jmh-core/src/test/java/org/openjdk/jmh/results/format/ResultFormatTest.java ! jmh-core/src/test/java/org/openjdk/jmh/runner/RunnerTest.java ! jmh-core/src/test/resources/org/openjdk/jmh/runner/MicroBenchmarks From aleksey.shipilev at oracle.com Tue Jul 12 19:09:56 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Tue, 12 Jul 2016 22:09:56 +0300 Subject: GroupThreads all remainig In-Reply-To: References: <434440a933c8147f22f8351a2216e5b9@nomagicsoftware.com> <873060542.935159.1466797710639.JavaMail.yahoo@mail.yahoo.com> Message-ID: <57854084.3000508@oracle.com> Hi, Sorry for late reply, un-burying mail from a long vacation. On 06/24/2016 11:35 PM, thurston at nomagicsoftware.com wrote: > How are the comma-separated values assigned? > By line#? Or lexically, based on the benchmark methods names? Yes, should be lexicographical. > The info printed doesn't indicate the thread assignment. Yes, that's a usability bug, should be fixed with: https://bugs.openjdk.java.net/browse/CODETOOLS-7901729 It will now print, e.g.: # Threads: 4 threads (1 group; 1x "get", 3x "inc" in each group) Thanks, -Aleksey From aleksey.shipilev at oracle.com Tue Jul 12 19:34:12 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Tue, 12 Jul 2016 22:34:12 +0300 Subject: Blackhole user instantiation In-Reply-To: <5773834F.5030507@redhat.com> References: <5773834F.5030507@redhat.com> Message-ID: <57854634.6010806@oracle.com> On 06/29/2016 11:14 AM, Radim Vansa wrote: > I'd like to reuse Blackhole in project aimed at benchmarking distributed > systems (RadarGun [1]), but rather than just being tricky bastard [2] > I'd like to ask if there's any info what black magic does JMH use to > instantiate Blackhole properly and where be the dragons. First, you have to understand that Blackhole evolves along with JVM and our understanding of JVMs. Blindly copying Blackhole code to a separate project detaches the implementation from updates. Current Blackhole needs to avoid DCE-ing itself. That is, this runs into danger of not working: @Benchmark public void test() { new Blackhole().consume(someOp()); } ...because some Blackhole implementations may choose to store the consumed argument in the field. Inlining the BH instance runs at the risk of DCEing that part of BH mechanics, which breaks the contract. This is one realistic failure scenario, but there might be other bad ones. For example, at some point we would have to split Blackhole implementations on per-JVM/compiler version (e.g. Graal), trying to accommodate different compiler details. Injecting Blackhole to JMH users provide a simple and performant way to substitute BH implementations without forcing users to recompile. Thanks, -Aleksey From aleksey.shipilev at oracle.com Tue Jul 12 19:58:29 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Tue, 12 Jul 2016 22:58:29 +0300 Subject: Raw data output In-Reply-To: References: Message-ID: <57854BE5.407@oracle.com> Hi Petr, Sorry for late reply. On 06/21/2016 06:02 PM, Petr Stefan wrote: > But now, JMH doesn't support collecting very raw data (rawData list > from JSON output holds just mean from each iteration - except > SingleShot benchmarking mode where the mean is the actual value). But > we found out, that Statistics class (all it's implementations) has > the raw data inside, so we propose a patch, which adds a getter for > this data. Yes, I understand the problem. Current raw_data is not usable for most scenarios, so I don't mind changing it. > Please look at proposed changes and give me a feedback. We're open to > changes and improvements. Comments: *) Sign the OCA (http://openjdk.java.net/contribute/) as soon as possible -- processing takes time. *) No need for separate patches. The change would have go in as a single changeset anyway. *) No need for a separate JSON formatter. Keeping both non-raw and raw versions in sync would be a hassle, so let's just fix the current JSON formatter. *) We would need to be able to trim/disable raw_data printing in JSON output. The common way in JMH is to use system properties. Something like boolean "jmh.json.rawData" would be fine. *) It is better to return Iterator from Statistics.getRawData(). I understand you want Iterable for for-each loops. You can create an utility method in Utils that adapts Iterator to Iterable for this. *) Iterator.remove() should throw UnsupportedOperationException. *) RawJSONResultFormat.getConcatenation is replaceable with Utils.join. Thanks, -Aleksey From ilya.gorbunov at jetbrains.com Fri Jul 1 13:37:52 2016 From: ilya.gorbunov at jetbrains.com (Ilya Gorbunov) Date: Fri, 1 Jul 2016 16:37:52 +0300 Subject: Proposed patch: update kotlin jmh benchmarks maven archetype Message-ID: <850742d9-7df7-d030-9857-438128eb58d7@jetbrains.com> -- Ilya Gorbunov, Kotlin frameworks team From aleksey.shipilev at oracle.com Tue Jul 12 20:34:17 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Tue, 12 Jul 2016 23:34:17 +0300 Subject: Proposed patch: update kotlin jmh benchmarks maven archetype In-Reply-To: <850742d9-7df7-d030-9857-438128eb58d7@jetbrains.com> References: <850742d9-7df7-d030-9857-438128eb58d7@jetbrains.com> Message-ID: <57855449.7020906@oracle.com> On 07/01/2016 04:37 PM, Ilya Gorbunov wrote: > Terribly sorry, but the patch was apparently scrubbed. Can you send the patch inline and/or with .patch/.txt extension and/or appropriate MIME type? Thanks, -Aleksey From aleksey.shipilev at oracle.com Tue Jul 12 20:47:10 2016 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Tue, 12 Jul 2016 20:47:10 +0000 Subject: hg: code-tools/jmh: 7901708: Inconsistency in labelling and usage for dynamically discovered profilers Message-ID: <201607122047.u6CKlAE2013018@aojmv0008.oracle.com> Changeset: 70dfb4a93cf5 Author: shade Date: 2016-07-12 23:46 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/70dfb4a93cf5 7901708: Inconsistency in labelling and usage for dynamically discovered profilers Summary: print FQN for dynamically discovered profilers in profiler list. ! jmh-core/src/main/java/org/openjdk/jmh/profile/ProfilerFactory.java From aleksey.shipilev at oracle.com Tue Jul 12 22:00:59 2016 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Tue, 12 Jul 2016 22:00:59 +0000 Subject: hg: code-tools/jmh: 7901707: Ability to use different JVM options for different forks Message-ID: <201607122200.u6CM0xKI028740@aojmv0008.oracle.com> Changeset: 49e2fb0c9f3b Author: shade Date: 2016-07-13 01:00 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/49e2fb0c9f3b 7901707: Ability to use different JVM options for different forks + jmh-core-it/src/test/java/org/openjdk/jmh/it/profilers/ChangeJVMOptsExternalProfiler.java + jmh-core-it/src/test/java/org/openjdk/jmh/it/profilers/ChangeJVMOptsTest.java ! jmh-core-it/src/test/resources/META-INF/services/org.openjdk.jmh.profile.Profiler ! jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java ! jmh-core/src/main/java/org/openjdk/jmh/util/FileUtils.java ! jmh-core/src/test/java/org/openjdk/jmh/runner/RunnerTest.java From alex.averbuch at neotechnology.com Mon Jul 11 20:54:58 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Mon, 11 Jul 2016 23:54:58 +0300 Subject: Infra Params In-Reply-To: References: Message-ID: SOLVED. See: http://stackoverflow.com/questions/38268510/jmh-accessing-benchmarkparams-from-setup-method-of-state-class On Fri, Jul 8, 2016 at 5:42 PM, Alex Averbuch < alex.averbuch at neotechnology.com> wrote: > > http://stackoverflow.com/questions/38268510/jmh-accessing-benchmarkparams-from-setup-method-of-state-class > From rvansa at redhat.com Wed Jul 13 08:05:26 2016 From: rvansa at redhat.com (Radim Vansa) Date: Wed, 13 Jul 2016 10:05:26 +0200 Subject: Blackhole user instantiation In-Reply-To: <57854634.6010806@oracle.com> References: <5773834F.5030507@redhat.com> <57854634.6010806@oracle.com> Message-ID: <5785F646.2050006@redhat.com> On 07/12/2016 09:34 PM, Aleksey Shipilev wrote: > On 06/29/2016 11:14 AM, Radim Vansa wrote: >> I'd like to reuse Blackhole in project aimed at benchmarking distributed >> systems (RadarGun [1]), but rather than just being tricky bastard [2] >> I'd like to ask if there's any info what black magic does JMH use to >> instantiate Blackhole properly and where be the dragons. > First, you have to understand that Blackhole evolves along with JVM and > our understanding of JVMs. Blindly copying Blackhole code to a separate > project detaches the implementation from updates. I'd be ok with adding a maven dependency, though there's the code-generation step (fine, I could wire something like that into build as well). But mostly I am asking what trickery is employed there. I understand BlackholeL1+3 as padding, I could imagine that L0+4 use the markers to assert the offsets (haven't peeked where exactly), but I have no idea what are the Blackhole_jmh_B2 mutexes for (and why are these padded from the L2 fields though not separated from each other - that would probably be clearer once I know what are these used for). > > Current Blackhole needs to avoid DCE-ing itself. That is, this runs into > danger of not working: > > @Benchmark > public void test() { > new Blackhole().consume(someOp()); > } > > ...because some Blackhole implementations may choose to store the > consumed argument in the field. Inlining the BH instance runs at the > risk of DCEing that part of BH mechanics, which breaks the contract. So you prevent inlining by storing the BH in a field of the test-class - great, that's one thing that needs to be taken care of. Others? > > This is one realistic failure scenario, but there might be other bad > ones. For example, at some point we would have to split Blackhole > implementations on per-JVM/compiler version (e.g. Graal), trying to > accommodate different compiler details. Injecting Blackhole to JMH users > provide a simple and performant way to substitute BH implementations > without forcing users to recompile. I agree, that's very convenient for JMH users. But when reusing JMH code, there could be a factory method that would provide the correct implementation for given VM. I am not asking to add it to the API ATM - when the need comes, I am sure something like that will appear in codebase. Thanks for your insights Radim > > Thanks, > -Aleksey > > > > -- Radim Vansa JBoss Performance Team From ptr.stef at gmail.com Wed Jul 13 14:13:52 2016 From: ptr.stef at gmail.com (Petr Stefan) Date: Wed, 13 Jul 2016 16:13:52 +0200 Subject: Raw data output In-Reply-To: <57854BE5.407@oracle.com> References: <57854BE5.407@oracle.com> Message-ID: <34edc582-cbd7-1dc0-01b6-74e99425b571@gmail.com> Hi Aleksey, thanks for your feedback. Dne 12.7.2016 v 21:58 Aleksey Shipilev napsal(a): > *) Sign the OCA (http://openjdk.java.net/contribute/) as soon as > possible -- processing takes time. I did it today morning. > *) No need for a separate JSON formatter. Keeping both non-raw and raw > versions in sync would be a hassle, so let's just fix the current JSON > formatter. Sure, but I'd like to discuss the format. There are some options: - Print the raw data the same way as it's now. For most of the benchmarking modes the output remains the same, but for sample time mode it has (big) disadvantage in output file size because of huge amount of values (quite often repeated). - Print the raw data as a pair of number and count. In JSON it should look like "[ 4296.0, 129 ] ", eq. 129 times value of 4296.0. It's little more space consuming for other modes, but for sample time it saves a lot. - Print sample time mode differently than other modes. It's a little inconsistent, but probably not too much. Sample time mode will be printed in compact way, others will be printed with current format. For this, we'd need to split "rawData" key into two new keys like "rawDataSimple" and "rawDataCombined" and say that actual data are in one of this key, but not both (in which one depends on benchmarking mode). These are basic options I've thought of. I'd probably pick second one. Maybe there is even better solution? Another question - is it useful to have rawData values divided into multiple BenchmarkResult section? I'm talking about "rawData" : [ [ 942.0, 384.0 ], [ 351.0, 781.0 ], What if all the values were just in single JSON array as in "rawData" : [ 942.0, 384.0, 351.0, 781.0, It would be more compact and disk space saving. Maybe it is useful for something, but I don't see it now. However, preserve this formatting is quite easy. > *) We would need to be able to trim/disable raw_data printing in JSON > output. The common way in JMH is to use system properties. Something > like boolean "jmh.json.rawData" would be fine. Default will be off? Have a nice time. Cheers, Petr From aleksey.shipilev at oracle.com Wed Jul 13 16:35:16 2016 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Wed, 13 Jul 2016 16:35:16 +0000 Subject: hg: code-tools/jmh: 2 new changesets Message-ID: <201607131635.u6DGZGTP011407@aojmv0008.oracle.com> Changeset: cffb77974716 Author: shade Date: 2016-07-13 19:31 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/cffb77974716 7901730: jmh-core-ct should cleanup CLASS_OUTPUT before using a new compile session ! jmh-core-ct/src/test/java/org/openjdk/jmh/ct/CompileTest.java Changeset: c050a47b2b37 Author: shade Date: 2016-07-13 19:35 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/c050a47b2b37 7901421: Reconsider the marshalling scheme for BenchmarkListEntry ! jmh-core/src/main/java/org/openjdk/jmh/runner/AbstractResourceReader.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/BenchmarkList.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/BenchmarkListEntry.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/CompilerHints.java ! jmh-core/src/main/java/org/openjdk/jmh/util/Optional.java + jmh-core/src/main/java/org/openjdk/jmh/util/lines/Constants.java + jmh-core/src/main/java/org/openjdk/jmh/util/lines/TestLineReader.java + jmh-core/src/main/java/org/openjdk/jmh/util/lines/TestLineWriter.java ! jmh-core/src/test/java/org/openjdk/jmh/runner/TestBenchmarkList.java ! jmh-core/src/test/java/org/openjdk/jmh/util/TestFileUtils.java + jmh-core/src/test/java/org/openjdk/jmh/util/TestLineTest.java - jmh-core/src/test/resources/org/openjdk/jmh/runner/MicroBenchmarks From aleksey.shipilev at oracle.com Thu Jul 14 10:13:28 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 14 Jul 2016 13:13:28 +0300 Subject: Raw data output In-Reply-To: <34edc582-cbd7-1dc0-01b6-74e99425b571@gmail.com> References: <57854BE5.407@oracle.com> <34edc582-cbd7-1dc0-01b6-74e99425b571@gmail.com> Message-ID: <578765C8.3050205@oracle.com> Hi Petr, On 07/13/2016 05:13 PM, Petr Stefan wrote: > - Print sample time mode differently than other modes. It's a little > inconsistent, but probably not too much. Sample time mode will be > printed in compact way, others will be printed with current format. > For this, we'd need to split "rawData" key into two new keys like > "rawDataSimple" and "rawDataCombined" and say that actual data are in > one of this key, but not both (in which one depends on benchmarking > mode). I like this one. "rawData" and "rawDataHistogram" would seem to explain the intent better. Users would definitely need to have more steps for processing raw data, but raw data is not supposed to be convenient, it is supposed to be complete. > Another question - is it useful to have rawData values divided into > multiple BenchmarkResult section? I'm talking about > > "rawData" : [ > [ > 942.0, > 384.0 > ], > [ > 351.0, > 781.0 > ], > ... > It would be more compact and disk space saving. Maybe it is useful for > something, but I don't see it now. However, preserve this formatting is > quite easy. I think we added this to add finer granularity to raw data: [ [ run1-iteration1-data, run1-iteration2-data ], [ run2-iteration1-data, run2-iteration2-data ] ] This might help to analyze data progression from iteration to iteration and from run to run. We should keep this intact. >> *) We would need to be able to trim/disable raw_data printing in JSON >> output. The common way in JMH is to use system properties. Something >> like boolean "jmh.json.rawData" would be fine. > Default will be off? Default will be on. This is a safeguard switch in case JMH is accidentally generating gigabytes of data, chocking the output. Thanks, -Aleksey From aleksey.shipilev at oracle.com Thu Jul 14 10:59:21 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 14 Jul 2016 13:59:21 +0300 Subject: Blackhole user instantiation In-Reply-To: <5785F646.2050006@redhat.com> References: <5773834F.5030507@redhat.com> <57854634.6010806@oracle.com> <5785F646.2050006@redhat.com> Message-ID: <57877089.10109@oracle.com> On 07/13/2016 11:05 AM, Radim Vansa wrote: > I'd be ok with adding a maven dependency, though there's the > code-generation step (fine, I could wire something like that into build > as well). But mostly I am asking what trickery is employed there. I > understand BlackholeL1+3 as padding, I could imagine that L0+4 use the > markers to assert the offsets (haven't peeked where exactly), but I have > no idea what are the Blackhole_jmh_B2 mutexes for (and why are these > padded from the L2 fields though not separated from each other - that > would probably be clearer once I know what are these used for). _jmh_B* classes are the autogenerated for each @State class and keep the scaffolding needed for supporting @State functionality. Those mutexes guard @Setup/@TearDown methods. >> Current Blackhole needs to avoid DCE-ing itself. That is, this runs into >> danger of not working: >> >> @Benchmark >> public void test() { >> new Blackhole().consume(someOp()); >> } >> >> ...because some Blackhole implementations may choose to store the >> consumed argument in the field. Inlining the BH instance runs at the >> risk of DCEing that part of BH mechanics, which breaks the contract. > > So you prevent inlining by storing the BH in a field of the test-class - > great, that's one thing that needs to be taken care of. Others? Yes, we understand it works _now_. We cannot guarantee it will work forever. -Aleksey From ilya.gorbunov at jetbrains.com Thu Jul 14 19:04:27 2016 From: ilya.gorbunov at jetbrains.com (Ilya Gorbunov) Date: Thu, 14 Jul 2016 22:04:27 +0300 Subject: Proposed patch: update kotlin jmh benchmarks maven archetype In-Reply-To: <57855449.7020906@oracle.com> References: <850742d9-7df7-d030-9857-438128eb58d7@jetbrains.com> <57855449.7020906@oracle.com> Message-ID: 12.07.2016 23:34, Aleksey Shipilev ?????: > Terribly sorry, but the patch was apparently scrubbed. Can you send the > patch inline and/or with .patch/.txt extension and/or appropriate MIME type? > > Thanks, > -Aleksey > Ok, resending the patch inline: # HG changeset patch # User Ilya Gorbunov # Date 1463627690 -10800 # Node ID b98f6ce505b3763317a246e4276825c6a3ad1706 # Parent ccd7a1d6f9d041346618b2d75b441433002fea54 jmh-kotlin-benchmark-archetype: rename MyBenchmark.ktm file to .kt, update dependency on Kotlin compiler and stdlib. diff --git a/jmh-archetypes/jmh-kotlin-benchmark-archetype/src/main/resources/META-INF/maven/archetype-metadata.xml b/jmh-archetypes/jmh-kotlin-benchmark-archetype/src/main/resources/META-INF/maven/archetype-metadata.xml --- a/jmh-archetypes/jmh-kotlin-benchmark-archetype/src/main/resources/META-INF/maven/archetype-metadata.xml +++ b/jmh-archetypes/jmh-kotlin-benchmark-archetype/src/main/resources/META-INF/maven/archetype-metadata.xml @@ -30,7 +30,7 @@ src/main/kotlin - **/*.ktm + **/*.kt diff --git a/jmh-archetypes/jmh-kotlin-benchmark-archetype/src/main/resources/archetype-resources/pom.xml b/jmh-archetypes/jmh-kotlin-benchmark-archetype/src/main/resources/archetype-resources/pom.xml --- a/jmh-archetypes/jmh-kotlin-benchmark-archetype/src/main/resources/archetype-resources/pom.xml +++ b/jmh-archetypes/jmh-kotlin-benchmark-archetype/src/main/resources/archetype-resources/pom.xml @@ -50,10 +50,11 @@ jmh-core \${jmh.version} - + + org.jetbrains.kotlin + kotlin-stdlib + \${kotlin.version} + @@ -65,6 +66,12 @@ ${project.version} default 1.6 + + 1.0.2 benchmarks @@ -77,10 +84,7 @@ kotlin-maven-plugin org.jetbrains.kotlin - - PUT_THE_VERSION_HERE + \${kotlin.version} diff --git a/jmh-archetypes/jmh-kotlin-benchmark-archetype/src/main/resources/archetype-resources/src/main/kotlin/MyBenchmark.ktm b/jmh-archetypes/jmh-kotlin-benchmark-archetype/src/main/resources/archetype-resources/src/main/kotlin/MyBenchmark.kt rename from jmh-archetypes/jmh-kotlin-benchmark-archetype/src/main/resources/archetype-resources/src/main/kotlin/MyBenchmark.ktm rename to jmh-archetypes/jmh-kotlin-benchmark-archetype/src/main/resources/archetype-resources/src/main/kotlin/MyBenchmark.kt --- a/jmh-archetypes/jmh-kotlin-benchmark-archetype/src/main/resources/archetype-resources/src/main/kotlin/MyBenchmark.ktm +++ b/jmh-archetypes/jmh-kotlin-benchmark-archetype/src/main/resources/archetype-resources/src/main/kotlin/MyBenchmark.kt @@ -35,7 +35,7 @@ open class MyBenchmark { - [Benchmark] + @Benchmark fun testMethod() { // This is a demo/sample template for building your JMH benchmarks. Edit as needed. // Put your benchmark code here. From ptr.stef at gmail.com Fri Jul 15 12:35:41 2016 From: ptr.stef at gmail.com (Petr Stefan) Date: Fri, 15 Jul 2016 14:35:41 +0200 Subject: Raw data output In-Reply-To: <578765C8.3050205@oracle.com> References: <57854BE5.407@oracle.com> <34edc582-cbd7-1dc0-01b6-74e99425b571@gmail.com> <578765C8.3050205@oracle.com> Message-ID: Hi Aleksey, I've updated my patch according to your comments. Thanks. Cheers, Petr Dne 14.7.2016 v 12:13 Aleksey Shipilev napsal(a): > Hi Petr, > > On 07/13/2016 05:13 PM, Petr Stefan wrote: >> - Print sample time mode differently than other modes. It's a little >> inconsistent, but probably not too much. Sample time mode will be >> printed in compact way, others will be printed with current format. >> For this, we'd need to split "rawData" key into two new keys like >> "rawDataSimple" and "rawDataCombined" and say that actual data are in >> one of this key, but not both (in which one depends on benchmarking >> mode). > I like this one. "rawData" and "rawDataHistogram" would seem to explain > the intent better. Users would definitely need to have more steps for > processing raw data, but raw data is not supposed to be convenient, it > is supposed to be complete. > >> Another question - is it useful to have rawData values divided into >> multiple BenchmarkResult section? I'm talking about >> >> "rawData" : [ >> [ >> 942.0, >> 384.0 >> ], >> [ >> 351.0, >> 781.0 >> ], >> ... >> It would be more compact and disk space saving. Maybe it is useful for >> something, but I don't see it now. However, preserve this formatting is >> quite easy. > I think we added this to add finer granularity to raw data: > > [ > [ > run1-iteration1-data, > run1-iteration2-data > ], > [ > run2-iteration1-data, > run2-iteration2-data > ] > ] > > This might help to analyze data progression from iteration to iteration > and from run to run. We should keep this intact. > > >>> *) We would need to be able to trim/disable raw_data printing in JSON >>> output. The common way in JMH is to use system properties. Something >>> like boolean "jmh.json.rawData" would be fine. >> Default will be off? > Default will be on. This is a safeguard switch in case JMH is > accidentally generating gigabytes of data, chocking the output. > > Thanks, > -Aleksey > -------------- next part -------------- A non-text attachment was scrubbed... Name: Better_raw_data_in_JSON_output.patch Type: text/x-patch Size: 21753 bytes Desc: not available URL: From nobeh5 at gmail.com Fri Jul 15 16:24:04 2016 From: nobeh5 at gmail.com (Behrooz Nobakht) Date: Fri, 15 Jul 2016 18:24:04 +0200 Subject: JMH for Java IO benchmarking? Message-ID: Hi, I am looking for tests/libraries/frameworks to measure IO performance (essentially read/write) using different storage; e.g. local disk, NFS and etc. Although feels not like it, but can I use JMH for such purpose? If so, any examples? And, if not, do you happen to know such a tool/library/tool? Thanks, Behrooz ? From henri.tremblay at gmail.com Fri Jul 15 18:03:33 2016 From: henri.tremblay at gmail.com (Henri Tremblay) Date: Fri, 15 Jul 2016 14:03:33 -0400 Subject: JMH for Java IO benchmarking? In-Reply-To: References: Message-ID: If you want to test raw disk speed (not a Java implementation) I would use operating system tools. On linux: - hdparm - dd - bonnie++ On 15 July 2016 at 12:24, Behrooz Nobakht wrote: > Hi, > > I am looking for tests/libraries/frameworks to measure IO performance > (essentially read/write) > using different storage; e.g. local disk, NFS and etc. > > Although feels not like it, but can I use JMH for such purpose? If so, any > examples? > And, if not, do you happen to know such a tool/library/tool? > > Thanks, > Behrooz > ? > From nobeh5 at gmail.com Fri Jul 15 18:25:16 2016 From: nobeh5 at gmail.com (Behrooz Nobakht) Date: Fri, 15 Jul 2016 20:25:16 +0200 Subject: JMH for Java IO benchmarking? In-Reply-To: References: Message-ID: You're right. Maybe I should have been more clear. I really want to measure/benchmark JVM's performance when doing IO operations read/write. On Fri, Jul 15, 2016 at 8:03 PM, Henri Tremblay wrote: > If you want to test raw disk speed (not a Java implementation) I would use > operating system tools. > > On linux: > - hdparm > - dd > - bonnie++ > > > On 15 July 2016 at 12:24, Behrooz Nobakht wrote: > >> Hi, >> >> I am looking for tests/libraries/frameworks to measure IO performance >> (essentially read/write) >> using different storage; e.g. local disk, NFS and etc. >> >> Although feels not like it, but can I use JMH for such purpose? If so, any >> examples? >> And, if not, do you happen to know such a tool/library/tool? >> >> Thanks, >> Behrooz >> ? >> > > -- -- Behrooz Nobakht From ecki at zusammenkunft.net Fri Jul 15 19:14:49 2016 From: ecki at zusammenkunft.net (ecki at zusammenkunft.net) Date: Fri, 15 Jul 2016 21:14:49 +0200 Subject: JMH for Java IO benchmarking? In-Reply-To: References: Message-ID: <49526376-81d1-4dc6-b77c-f04baf553f23.maildroid@localhost> You can use JMH to execute repitetive operations with some control over parallelity. I did that myself. However if you want to do stuff with pending operations, mixed workloads, reactive tests JMH might not be the best option. But it really depends on the complexity of the tests. It does give you quite a lot of infrastructure to start with. Bernd -- http://bernd.eckenfels.net -----Original Message----- From: Behrooz Nobakht To: Henri Tremblay Cc: "jmh-dev at openjdk.java.net" Sent: Fr., 15 Juli 2016 20:55 Subject: Re: JMH for Java IO benchmarking? You're right. Maybe I should have been more clear. I really want to measure/benchmark JVM's performance when doing IO operations read/write. On Fri, Jul 15, 2016 at 8:03 PM, Henri Tremblay wrote: > If you want to test raw disk speed (not a Java implementation) I would use > operating system tools. > > On linux: > - hdparm > - dd > - bonnie++ > > > On 15 July 2016 at 12:24, Behrooz Nobakht wrote: > >> Hi, >> >> I am looking for tests/libraries/frameworks to measure IO performance >> (essentially read/write) >> using different storage; e.g. local disk, NFS and etc. >> >> Although feels not like it, but can I use JMH for such purpose? If so, any >> examples? >> And, if not, do you happen to know such a tool/library/tool? >> >> Thanks, >> Behrooz >> ? >> > > -- -- Behrooz Nobakht From aleksey.shipilev at oracle.com Tue Jul 19 21:56:08 2016 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Tue, 19 Jul 2016 21:56:08 +0000 Subject: hg: code-tools/jmh: 2 new changesets Message-ID: <201607192156.u6JLu8Bq021352@aojmv0008.oracle.com> Changeset: 1345ad5a0155 Author: shade Date: 2016-07-19 21:57 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/1345ad5a0155 7901735: JMH infrastructure classes (Blackhole, *Params, Control) should be treated specially + jmh-core-it/src/test/java/org/openjdk/jmh/it/infraparams/BenchmarkParamsTest.java + jmh-core-it/src/test/java/org/openjdk/jmh/it/infraparams/IterationParamsTest.java + jmh-core-it/src/test/java/org/openjdk/jmh/it/infraparams/ThreadParamsTest.java ! jmh-core/src/main/java/org/openjdk/jmh/generators/core/BenchmarkGenerator.java ! jmh-core/src/main/java/org/openjdk/jmh/generators/core/StateObjectHandler.java ! jmh-core/src/main/java/org/openjdk/jmh/infra/BenchmarkParams.java ! jmh-core/src/main/java/org/openjdk/jmh/infra/Blackhole.java ! jmh-core/src/main/java/org/openjdk/jmh/infra/Control.java ! jmh-core/src/main/java/org/openjdk/jmh/infra/IterationParams.java ! jmh-core/src/main/java/org/openjdk/jmh/infra/ThreadParams.java ! jmh-core/src/test/java/org/openjdk/jmh/BlackholeTest.java Changeset: 2d0b2fd926fb Author: shade Date: 2016-07-19 21:58 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/2d0b2fd926fb 7901736: JMH update interval should be greater ! jmh-core/src/main/java/org/openjdk/jmh/util/Version.java From aleksey.shipilev at oracle.com Wed Jul 20 20:30:11 2016 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Wed, 20 Jul 2016 20:30:11 +0000 Subject: hg: code-tools/jmh: 7901737: perfasm should print the compiler, compilation level, method version Message-ID: <201607202030.u6KKUBTv018708@aojmv0008.oracle.com> Changeset: e96cad1fc480 Author: shade Date: 2016-07-20 23:30 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/e96cad1fc480 7901737: perfasm should print the compiler, compilation level, method version ! jmh-core/src/main/java/org/openjdk/jmh/profile/AbstractPerfAsmProfiler.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/LinuxPerfAsmProfiler.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/WinPerfAsmProfiler.java + jmh-core/src/main/java/org/openjdk/jmh/util/Interval.java + jmh-core/src/main/java/org/openjdk/jmh/util/IntervalMap.java ! jmh-core/src/main/java/org/openjdk/jmh/util/Utils.java From aleksey.shipilev at oracle.com Thu Jul 21 14:28:33 2016 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 21 Jul 2016 14:28:33 +0000 Subject: hg: code-tools/jmh: 7901738: Update Kotlin benchmark archetype Message-ID: <201607211428.u6LESXnK006964@aojmv0008.oracle.com> Changeset: eead74fabfb4 Author: shade Date: 2016-07-21 17:02 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/eead74fabfb4 7901738: Update Kotlin benchmark archetype Contributed-by: Ilya Gorbunov ! jmh-archetypes/jmh-kotlin-benchmark-archetype/src/main/resources/META-INF/maven/archetype-metadata.xml ! jmh-archetypes/jmh-kotlin-benchmark-archetype/src/main/resources/archetype-resources/pom.xml + jmh-archetypes/jmh-kotlin-benchmark-archetype/src/main/resources/archetype-resources/src/main/kotlin/MyBenchmark.kt - jmh-archetypes/jmh-kotlin-benchmark-archetype/src/main/resources/archetype-resources/src/main/kotlin/MyBenchmark.ktm + jmh-archetypes/jmh-kotlin-benchmark-archetype/src/test/resources/projects/test/archetype.properties + jmh-archetypes/jmh-kotlin-benchmark-archetype/src/test/resources/projects/test/goal.txt From aleksey.shipilev at oracle.com Thu Jul 21 14:30:01 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 21 Jul 2016 17:30:01 +0300 Subject: Proposed patch: update kotlin jmh benchmarks maven archetype In-Reply-To: References: <850742d9-7df7-d030-9857-438128eb58d7@jetbrains.com> <57855449.7020906@oracle.com> Message-ID: <5790DC69.5050102@oracle.com> On 07/14/2016 10:04 PM, Ilya Gorbunov wrote: > 12.07.2016 23:34, Aleksey Shipilev ?????: >> Terribly sorry, but the patch was apparently scrubbed. Can you send the >> patch inline and/or with .patch/.txt extension and/or appropriate MIME >> type? >> >> Thanks, >> -Aleksey >> > > Ok, resending the patch inline: Thank you, pushed under: https://bugs.openjdk.java.net/browse/CODETOOLS-7901738 I also added an integration test for archetype. (Will do the same for other archetypes soon). Cheers, -Aleksey From aleksey.shipilev at oracle.com Thu Jul 21 14:42:37 2016 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 21 Jul 2016 14:42:37 +0000 Subject: hg: code-tools/jmh: 7901739: Update benchmark archetypes: Java version, suggested plugin versions, etc Message-ID: <201607211442.u6LEgcFT013432@aojmv0008.oracle.com> Changeset: fa510264b3f6 Author: shade Date: 2016-07-21 17:42 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/fa510264b3f6 7901739: Update benchmark archetypes: Java version, suggested plugin versions, etc ! jmh-archetypes/jmh-groovy-benchmark-archetype/src/main/resources/archetype-resources/pom.xml = jmh-archetypes/jmh-groovy-benchmark-archetype/src/test/resources/projects/test/archetype.properties < jmh-archetypes/jmh-kotlin-benchmark-archetype/src/test/resources/projects/test/archetype.properties = jmh-archetypes/jmh-groovy-benchmark-archetype/src/test/resources/projects/test/goal.txt < jmh-archetypes/jmh-kotlin-benchmark-archetype/src/test/resources/projects/test/goal.txt ! jmh-archetypes/jmh-java-benchmark-archetype/src/main/resources/archetype-resources/pom.xml ! jmh-archetypes/jmh-kotlin-benchmark-archetype/src/main/resources/archetype-resources/pom.xml ! jmh-archetypes/jmh-scala-benchmark-archetype/src/main/resources/archetype-resources/pom.xml ! jmh-archetypes/jmh-scala-benchmark-archetype/src/test/resources/projects/test/archetype.properties < jmh-archetypes/jmh-kotlin-benchmark-archetype/src/test/resources/projects/test/archetype.properties = jmh-archetypes/jmh-scala-benchmark-archetype/src/test/resources/projects/test/goal.txt < jmh-archetypes/jmh-kotlin-benchmark-archetype/src/test/resources/projects/test/goal.txt From aleksey.shipilev at oracle.com Thu Jul 21 15:49:51 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 21 Jul 2016 18:49:51 +0300 Subject: Raw data output In-Reply-To: References: <57854BE5.407@oracle.com> <34edc582-cbd7-1dc0-01b6-74e99425b571@gmail.com> <578765C8.3050205@oracle.com> Message-ID: <5790EF1F.6080502@oracle.com> Hi Petr, On 07/15/2016 03:35 PM, Petr Stefan wrote: > I've updated my patch according to your comments. Thanks. Any news about OCA? This is the thing that blocks us from going forward with this patch :) The patch looks OK. I have a few comments though: a) Utils.loopOver can be simplified; b) There is no need to emit empty "rawDataHistogram". If some data is not available, it's best to avoid printing out special values, like empty array. c) The grouping is incorrect: it adds up histograms from different iterations. This happens because the code reuses the old loop that prints out *scalar* per each iteration. But in histogram case, you have to wrap it up one more time. Hopefully fixed all three here: http://cr.openjdk.java.net/~shade/jmh/rawjson-1.patch Can you look into it, and verify it still works as intended? Thanks, -Aleksey From ptr.stef at gmail.com Thu Jul 21 16:35:07 2016 From: ptr.stef at gmail.com (Petr Stefan) Date: Thu, 21 Jul 2016 18:35:07 +0200 Subject: Raw data output In-Reply-To: <5790EF1F.6080502@oracle.com> References: <57854BE5.407@oracle.com> <34edc582-cbd7-1dc0-01b6-74e99425b571@gmail.com> <578765C8.3050205@oracle.com> <5790EF1F.6080502@oracle.com> Message-ID: Hi Aleksey, Dne 21.7.2016 v 17:49 Aleksey Shipilev napsal(a): > Any news about OCA? This is the thing that blocks us from going forward > with this patch :) No, I haven't heard from them yet. I can send you scan of the signed document, but I think before they approve it's not valid. > b) There is no need to emit empty "rawDataHistogram". If some data is > not available, it's best to avoid printing out special values, like > empty array. OK, I don't need it. My idea was to keep symetry, like empty secondaryMetrics when you don't use any profiler. But you have a point, better keep it simple :). > c) The grouping is incorrect: it adds up histograms from different > iterations. This happens because the code reuses the old loop that > prints out *scalar* per each iteration. But in histogram case, you have > to wrap it up one more time. Now I got it, you're right. My mistake, sorry. > Hopefully fixed all three here: > http://cr.openjdk.java.net/~shade/jmh/rawjson-1.patch > > Can you look into it, and verify it still works as intended? Yes, from my point it's perfect. Thank you very much. Cheers, Petr From aleksey.shipilev at oracle.com Thu Jul 21 19:01:54 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 21 Jul 2016 22:01:54 +0300 Subject: Raw data output In-Reply-To: References: <57854BE5.407@oracle.com> <34edc582-cbd7-1dc0-01b6-74e99425b571@gmail.com> <578765C8.3050205@oracle.com> <5790EF1F.6080502@oracle.com> Message-ID: <6213f93d-8832-bac8-19b8-e39f6c505745@oracle.com> On 07/21/2016 07:35 PM, Petr Stefan wrote: > Dne 21.7.2016 v 17:49 Aleksey Shipilev napsal(a): >> Any news about OCA? This is the thing that blocks us from going forward >> with this patch :) > > No, I haven't heard from them yet. I can send you scan of the signed > document, but I think before they approve it's not valid. Yes, that would not help us. But OCA folks usually OKay you a few days before you appear on "official" signatories list. >> Hopefully fixed all three here: >> http://cr.openjdk.java.net/~shade/jmh/rawjson-1.patch >> >> Can you look into it, and verify it still works as intended? > > Yes, from my point it's perfect. Thank you very much. Okay, good. Waiting for OCA now... Thanks, -Aleksey From aleksey.shipilev at oracle.com Fri Jul 22 12:21:11 2016 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Fri, 22 Jul 2016 12:21:11 +0000 Subject: hg: code-tools/jmh: 7901743: @State DAG machinery miscompiles/rejects valid DAGs Message-ID: <201607221221.u6MCLB2Z028675@aojmv0008.oracle.com> Changeset: c698f6c0e0fc Author: shade Date: 2016-07-22 15:20 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/c698f6c0e0fc 7901743: @State DAG machinery miscompiles/rejects valid DAGs - jmh-core-ct/src/test/java/org/openjdk/jmh/ct/states/dag/CycleBenchmarkTest.java - jmh-core-ct/src/test/java/org/openjdk/jmh/ct/states/dag/CycleThreadTest.java + jmh-core-ct/src/test/java/org/openjdk/jmh/ct/states/dag/cycles/ExplicitCycleBenchmarkTest.java + jmh-core-ct/src/test/java/org/openjdk/jmh/ct/states/dag/cycles/ExplicitCycleThreadTest.java + jmh-core-ct/src/test/java/org/openjdk/jmh/ct/states/dag/cycles/ImplicitCycleBenchmarkTest.java + jmh-core-ct/src/test/java/org/openjdk/jmh/ct/states/dag/cycles/ImplicitCycleThreadTest.java + jmh-core-ct/src/test/java/org/openjdk/jmh/ct/states/dag/doublets/BenchmarkBenchmarkTest.java + jmh-core-ct/src/test/java/org/openjdk/jmh/ct/states/dag/doublets/BenchmarkThreadTest.java + jmh-core-ct/src/test/java/org/openjdk/jmh/ct/states/dag/doublets/ThreadBenchmarkTest.java + jmh-core-ct/src/test/java/org/openjdk/jmh/ct/states/dag/doublets/ThreadThreadTest.java + jmh-core-ct/src/test/java/org/openjdk/jmh/ct/states/dag/implicit/BenchmarkBenchmarkTest.java + jmh-core-ct/src/test/java/org/openjdk/jmh/ct/states/dag/implicit/BenchmarkThreadTest.java + jmh-core-ct/src/test/java/org/openjdk/jmh/ct/states/dag/implicit/ThreadBenchmarkTest.java + jmh-core-ct/src/test/java/org/openjdk/jmh/ct/states/dag/implicit/ThreadThreadTest.java ! jmh-core/src/main/java/org/openjdk/jmh/generators/core/BenchmarkGenerator.java ! jmh-core/src/main/java/org/openjdk/jmh/generators/core/StateObject.java ! jmh-core/src/main/java/org/openjdk/jmh/generators/core/StateObjectHandler.java From aleksey.shipilev at oracle.com Fri Jul 22 12:25:03 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Fri, 22 Jul 2016 15:25:03 +0300 Subject: Infra Params In-Reply-To: References: Message-ID: <8131a1f8-bf25-5fbf-70ab-0d6523a3846b@oracle.com> Hi Alex, Thanks for the test case! On 07/08/2016 01:42 PM, Alex Averbuch wrote: > I have a minimal example for reproducing this error, it goes like this: > > (1) Define @State(Scope.Benchmark) class named Test <--- OK > (2) Define @Benchmark method named nothing() <--- OK > (3) Pass @State instance into @Benchmark method, e.g. public void nothing( > Test test ) <--- ERROR! Yes, that's a bug in @State DAG machinery, hopefully fixed with: https://bugs.openjdk.java.net/browse/CODETOOLS-7901743 @State(Scope.Benchmark) public class Test { @Setup public void setUp(BenchmarkParams params) {} @Benchmark public void nothing(Test test) {} } The compounding factor is that you reference a single @State class both implicitly (by the virtue of @Benchmark method being the member of @State class), and explicitly (by injecting via the argument). This made DAG machinery to believe the Test.setup() method should be fed with *two* instances of BenchmarkParams. The same thing may happen with two explicit copies of @State arguments, and it is fixed too. Also, BenchmarkParams and other infrastructure classes are not implicit @State-s anymore, which frees that particular example from failing even without fixing the DAG machinery: https://bugs.openjdk.java.net/browse/CODETOOLS-7901735 Thanks, -Aleksey From aleksey.shipilev at oracle.com Fri Jul 22 15:06:28 2016 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Fri, 22 Jul 2016 15:06:28 +0000 Subject: hg: code-tools/jmh: 7901744: Archetype verification should be enabled only on Java 8 Message-ID: <201607221506.u6MF6SWE010754@aojmv0008.oracle.com> Changeset: f3769cbfafee Author: shade Date: 2016-07-22 18:06 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/f3769cbfafee 7901744: Archetype verification should be enabled only on Java 8 ! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml ! jmh-archetypes/pom.xml From aleksey.shipilev at oracle.com Fri Jul 22 18:18:26 2016 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Fri, 22 Jul 2016 18:18:26 +0000 Subject: hg: code-tools/jmh: 7901745: perfasm treats address lines in the region print margin as intrajump Message-ID: <201607221818.u6MIIQCN001754@aojmv0008.oracle.com> Changeset: 31fa2612f8af Author: shade Date: 2016-07-22 21:18 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/31fa2612f8af 7901745: perfasm treats address lines in the region print margin as intrajump ! jmh-core/src/main/java/org/openjdk/jmh/profile/AbstractPerfAsmProfiler.java From aleksey.shipilev at oracle.com Sat Jul 23 11:47:49 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Sat, 23 Jul 2016 14:47:49 +0300 Subject: JMH 1.13 Message-ID: <2d5bd4d4-ab6c-11e0-b38b-a65232df6654@oracle.com> Hi, JMH 1.13 release is available at Maven Central (props to Evgeny Mandrikov, as usual). It includes several intrusive internal changes that bring interesting features and/or fix non-trivial bugs, but can also bring potential regressions. Please give them a ride! Summary of changes: *) -prof perfasm improvements: it now prints the compiler used to compile the method, compilation level, method version: https://bugs.openjdk.java.net/browse/CODETOOLS-7901737 *) Output improvements. Inconsistency in labeling dynamically discovered profilers was fixed -- prints class FQN name now, instead of "" label. Thread distribution for asymmetric benchmarks is now printed out. JMH update interval is increased. https://bugs.openjdk.java.net/browse/CODETOOLS-7901708 https://bugs.openjdk.java.net/browse/CODETOOLS-7901729 https://bugs.openjdk.java.net/browse/CODETOOLS-7901736 *) Maven Archetype improvements: benchmark projects generated from archetypes now default to source/target 1.8, have updated suggestions on language-specific versions, have more docs about the properties, etc: https://bugs.openjdk.java.net/browse/CODETOOLS-7901738 https://bugs.openjdk.java.net/browse/CODETOOLS-7901739 *) Profiler infrastructure improvements: JMH now polls external profilers for JVM options before each run, which enables profilers to have e.g. separate output files for each fork. https://bugs.openjdk.java.net/browse/CODETOOLS-7901707 *) Benchmark generator improvements. JMH used to reject some valid @State graphs. Also, frequently-used JMH infra classes (Blackhole, *Params, Control) are not implicitly @State-s anymore, and treated specially -- this frees JMH generators from generating unnecessary cruft (e.g. subclasses) around them. https://bugs.openjdk.java.net/browse/CODETOOLS-7901706 https://bugs.openjdk.java.net/browse/CODETOOLS-7901735 https://bugs.openjdk.java.net/browse/CODETOOLS-7901743 *) Other minor improvements. Some internal data marshaling improvements, fixing the leaks, etc. https://bugs.openjdk.java.net/browse/CODETOOLS-7901421 https://bugs.openjdk.java.net/browse/CODETOOLS-7901666 Enjoy! -Aleksey From aleksey.shipilev at oracle.com Sat Jul 23 11:48:51 2016 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Sat, 23 Jul 2016 11:48:51 +0000 Subject: hg: code-tools/jmh: 3 new changesets Message-ID: <201607231148.u6NBmpBg019684@aojmv0008.oracle.com> Changeset: 7ff584954008 Author: shade Date: 2016-07-22 21:31 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/7ff584954008 JMH v1.13. ! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml ! jmh-archetypes/pom.xml ! jmh-core-benchmarks/pom.xml ! jmh-core-ct/pom.xml ! jmh-core-it/pom.xml ! jmh-core/pom.xml ! jmh-generator-annprocess/pom.xml ! jmh-generator-asm/pom.xml ! jmh-generator-bytecode/pom.xml ! jmh-generator-reflection/pom.xml ! jmh-samples/pom.xml ! pom.xml Changeset: 1c69963cc116 Author: shade Date: 2016-07-22 21:31 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/1c69963cc116 Added tag 1.13 for changeset 7ff584954008 ! .hgtags Changeset: 8b80b24425a9 Author: shade Date: 2016-07-22 21:31 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/8b80b24425a9 Continue in 1.14-SNAPSHOT. ! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml ! jmh-archetypes/pom.xml ! jmh-core-benchmarks/pom.xml ! jmh-core-ct/pom.xml ! jmh-core-it/pom.xml ! jmh-core/pom.xml ! jmh-generator-annprocess/pom.xml ! jmh-generator-asm/pom.xml ! jmh-generator-bytecode/pom.xml ! jmh-generator-reflection/pom.xml ! jmh-samples/pom.xml ! pom.xml From alex.averbuch at neotechnology.com Mon Jul 25 13:27:13 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Mon, 25 Jul 2016 15:27:13 +0200 Subject: Leve.Iteration @TearDown method called one too many times Message-ID: Hi, I have a benchmark for measuring database index creation. Summary: * It is an expensive operation (5+ seconds per invocation), so Mode.SingleShotTime is used * The database runs 5 warmup iterations & 5 measurement iterations * Each iteration must create an index from scratch, so the index must be dropped/deleted between iterations * To delete the index after each iteration a Level.Iteration @TearDown method is used. Inside this method the index is dropped Problem: * It appears as though the Level.Iteration @TearDown method is being called 1 more time than the @Benchmark method * This causes the database (and benchmark) to crash, because the database tries to delete an index that does not exist When executing the benchmark code at the bottom of this email, the following is written to console: # Run progress: 0.00% complete, ETA 00:00:00 # Fork: 1 of 1 # Warmup Iteration 1: CREATE DROP # Warmup Iteration 2: CREATE DROP # Warmup Iteration 3: CREATE DROP # Warmup Iteration 4: CREATE DROP # Warmup Iteration 5: CREATE DROP Iteration 1: CREATE DROP Iteration 2: CREATE DROP Iteration 3: CREATE DROP Iteration 4: CREATE DROP Iteration 5: CREATE DROP DROP <--------- NOTICE THIS ADDITIONAL "DROP" Does this (extra DROP) make sense? Is there any way to prevent it from happening? Thanks in advance! @State( Scope.Benchmark ) public class CreateIndex { protected Database db; @Setup public void setUp() { db = new Database(); // populate db } @TearDown public void tearDown() { db.shutdown(); } @TearDown( Level.Iteration ) public void dropIndex() { System.out.println("DROP"); try ( Transaction tx = db.beginTx() ) { // drop index tx.success(); } } @Benchmark @BenchmarkMode( Mode.SingleShotTime ) public void createIndex() { System.out.println("CREATE"); try ( Transaction tx = db.beginTx() ) { // create index tx.success(); } } } From alex.averbuch at neotechnology.com Mon Jul 25 13:29:30 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Mon, 25 Jul 2016 15:29:30 +0200 Subject: Leve.Iteration @TearDown method called one too many times In-Reply-To: References: Message-ID: FYI, I'm using: org.openjdk.jmh jmh-core 1.1.3 On Mon, Jul 25, 2016 at 3:27 PM, Alex Averbuch < alex.averbuch at neotechnology.com> wrote: > Hi, > I have a benchmark for measuring database index creation. > > Summary: > * It is an expensive operation (5+ seconds per invocation), so > Mode.SingleShotTime is used > * The database runs 5 warmup iterations & 5 measurement iterations > * Each iteration must create an index from scratch, so the index must be > dropped/deleted between iterations > * To delete the index after each iteration a Level.Iteration @TearDown > method is used. Inside this method the index is dropped > > Problem: > * It appears as though the Level.Iteration @TearDown method is being > called 1 more time than the @Benchmark method > * This causes the database (and benchmark) to crash, because the > database tries to delete an index that does not exist > > When executing the benchmark code at the bottom of this email, the > following is written to console: > > # Run progress: 0.00% complete, ETA 00:00:00 > # Fork: 1 of 1 > # Warmup Iteration 1: > CREATE > DROP > # Warmup Iteration 2: > CREATE > DROP > # Warmup Iteration 3: > CREATE > DROP > # Warmup Iteration 4: > CREATE > DROP > # Warmup Iteration 5: > CREATE > DROP > Iteration 1: > CREATE > DROP > Iteration 2: > CREATE > DROP > Iteration 3: > CREATE > DROP > Iteration 4: > CREATE > DROP > Iteration 5: > CREATE > DROP > DROP <--------- NOTICE THIS ADDITIONAL > "DROP" > > > > Does this (extra DROP) make sense? > Is there any way to prevent it from happening? > Thanks in advance! > > > @State( Scope.Benchmark ) > public class CreateIndex > { > protected Database db; > > @Setup > public void setUp() > { > db = new Database(); > // populate db > } > > @TearDown > public void tearDown() > { > db.shutdown(); > } > > @TearDown( Level.Iteration ) > public void dropIndex() > { > System.out.println("DROP"); > try ( Transaction tx = db.beginTx() ) > { > // drop index > tx.success(); > } > } > > @Benchmark > @BenchmarkMode( Mode.SingleShotTime ) > public void createIndex() > { > System.out.println("CREATE"); > try ( Transaction tx = db.beginTx() ) > { > // create index > tx.success(); > } > } > } > From alex.averbuch at neotechnology.com Mon Jul 25 13:53:08 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Mon, 25 Jul 2016 15:53:08 +0200 Subject: Leve.Iteration @TearDown method called one too many times In-Reply-To: References: Message-ID: Hmmm... After trying other approaches and then returning to the one described in my original email everything seems to work fine. Unless the problem resurfaces please consider this as user error, I probably did something foolish. On Mon, Jul 25, 2016 at 3:29 PM, Alex Averbuch < alex.averbuch at neotechnology.com> wrote: > FYI, I'm using: > > > org.openjdk.jmh > jmh-core > 1.1.3 > > > > On Mon, Jul 25, 2016 at 3:27 PM, Alex Averbuch < > alex.averbuch at neotechnology.com> wrote: > >> Hi, >> I have a benchmark for measuring database index creation. >> >> Summary: >> * It is an expensive operation (5+ seconds per invocation), so >> Mode.SingleShotTime is used >> * The database runs 5 warmup iterations & 5 measurement iterations >> * Each iteration must create an index from scratch, so the index must >> be dropped/deleted between iterations >> * To delete the index after each iteration a Level.Iteration @TearDown >> method is used. Inside this method the index is dropped >> >> Problem: >> * It appears as though the Level.Iteration @TearDown method is being >> called 1 more time than the @Benchmark method >> * This causes the database (and benchmark) to crash, because the >> database tries to delete an index that does not exist >> >> When executing the benchmark code at the bottom of this email, the >> following is written to console: >> >> # Run progress: 0.00% complete, ETA 00:00:00 >> # Fork: 1 of 1 >> # Warmup Iteration 1: >> CREATE >> DROP >> # Warmup Iteration 2: >> CREATE >> DROP >> # Warmup Iteration 3: >> CREATE >> DROP >> # Warmup Iteration 4: >> CREATE >> DROP >> # Warmup Iteration 5: >> CREATE >> DROP >> Iteration 1: >> CREATE >> DROP >> Iteration 2: >> CREATE >> DROP >> Iteration 3: >> CREATE >> DROP >> Iteration 4: >> CREATE >> DROP >> Iteration 5: >> CREATE >> DROP >> DROP <--------- NOTICE THIS ADDITIONAL >> "DROP" >> >> >> >> Does this (extra DROP) make sense? >> Is there any way to prevent it from happening? >> Thanks in advance! >> >> >> @State( Scope.Benchmark ) >> public class CreateIndex >> { >> protected Database db; >> >> @Setup >> public void setUp() >> { >> db = new Database(); >> // populate db >> } >> >> @TearDown >> public void tearDown() >> { >> db.shutdown(); >> } >> >> @TearDown( Level.Iteration ) >> public void dropIndex() >> { >> System.out.println("DROP"); >> try ( Transaction tx = db.beginTx() ) >> { >> // drop index >> tx.success(); >> } >> } >> >> @Benchmark >> @BenchmarkMode( Mode.SingleShotTime ) >> public void createIndex() >> { >> System.out.println("CREATE"); >> try ( Transaction tx = db.beginTx() ) >> { >> // create index >> tx.success(); >> } >> } >> } >> > > From aleksey.shipilev at oracle.com Mon Jul 25 18:43:44 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Mon, 25 Jul 2016 21:43:44 +0300 Subject: Leve.Iteration @TearDown method called one too many times In-Reply-To: References: Message-ID: On 07/25/2016 04:29 PM, Alex Averbuch wrote: > FYI, I'm using: > > > org.openjdk.jmh > jmh-core > 1.1.3 The latest version is 1.13, not 1.1.3. Does the problem reproduce with 1.13? -Aleksey From travis.downs at gmail.com Sun Jul 24 23:54:50 2016 From: travis.downs at gmail.com (Travis Downs) Date: Sun, 24 Jul 2016 18:54:50 -0500 Subject: default values for options Message-ID: Running with the -h option gives a reasonable description of the various possible command line flags and options, but doesn't seem to mention which flag/option is the default if not explicit specified. Is this documented anywhere? From travis.downs at gmail.com Sun Jul 24 23:56:07 2016 From: travis.downs at gmail.com (Travis Downs) Date: Sun, 24 Jul 2016 18:56:07 -0500 Subject: JMH documentation Message-ID: Is there any JMH documentation beyond the intro at http://openjdk.java.net/projects/code-tools/jmh/, the information you get from -h and the samples? I'm not complaining - the samples are pretty good, and the options are reasonably well documented, but just checking if I'm missing a wiki or other resource that goes into more details on the various bits and pieces. From travis.downs at gmail.com Mon Jul 25 00:07:18 2016 From: travis.downs at gmail.com (Travis Downs) Date: Sun, 24 Jul 2016 19:07:18 -0500 Subject: Any option to use something other than "time" to measure benchmarks? Message-ID: Currently JMH uses "time" as the one part of the metric for all modes, as far as I can tell (e.g., ops/second, seconds/op, etc). This has the big downside of being wildly affected by all the fancy things that OSes and CPUs do these days. In particular, on the CPU end, you have: - downscaling of the CPU frequency due to thermal measurements, instruction use (e.g., AVX downscaling) - upscaling of the CPU if possible (aka "turbo") within thermal envelope - various scalings (up/down/whatever) of the CPU speed to maximize performance/watt (especially prevalent on mobile, but making inroads on bigger chips too) As a simple example, if I plug my laptop in, my benchmark scores more than double. If I change my power plan options, the effect increases or reduces depending on what I select. When I reduce most of the variability by running at "max frequency" and disabling most OS power management, I still see a turbo-boost related effect where the first iterations of a benchmark run at a high (turbo) speed that later ones (which throttle down) - which also depends on factors like the ambient air temperature, and the type of pants I am wearing (believe it or not - when I wear fleece pants, it causes higher temps and turbo throttles down much more quickly). There is a near-panacea for all this: have an option to use a measurement which directly measures CPU cycles, rather than time. For example, something like "unhalted cpu cycles" offered by the performance counters on x86 chips. This solves most of the problems above since this counter "ticks" at the same speed as the CPU. Some variability remains because some parts of the system (notably, the latency to RAM) may not scale in the same way, but in general it is much better. I don't know practical this is to do from JMH, so for now I'm just throwing the idea out there. Travis From aleksey.shipilev at oracle.com Mon Jul 25 19:51:46 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Mon, 25 Jul 2016 22:51:46 +0300 Subject: JMH documentation In-Reply-To: References: Message-ID: <920a88c4-2c12-da51-b15a-b15b6caa94ba@oracle.com> On 07/25/2016 02:56 AM, Travis Downs wrote: > Is there any JMH documentation beyond the intro at > http://openjdk.java.net/projects/code-tools/jmh/, the information you get > from -h and the samples? I'm not complaining - the samples are pretty good, > and the options are reasonably well documented, but just checking if I'm > missing a wiki or other resource that goes into more details on the various > bits and pieces. No, you are not missing anything. Our golden source of documentation is context help (-h and Javadoc) and runnable JMH Samples. Many other short introductions to JMH (mostly trimmed down Samples) are available on Internet, in different stages of being obsolete. Thanks, -Aleksey From ecki at zusammenkunft.net Mon Jul 25 20:35:02 2016 From: ecki at zusammenkunft.net (ecki at zusammenkunft.net) Date: Mon, 25 Jul 2016 22:35:02 +0200 Subject: AW: JMH documentation In-Reply-To: <920a88c4-2c12-da51-b15a-b15b6caa94ba@oracle.com> References: <920a88c4-2c12-da51-b15a-b15b6caa94ba@oracle.com> Message-ID: <579677f7.8aacc20a.ff124.8be6@mx.google.com> Aleksey is too humble to mention this, but some of his presentations and writing (on hotspot performance optimizations) also serve as a good example into benchmarking methodology and jmh application to those problems. In fact I think some writeup distilled from that would be a good introduction to jmh. https://shipilev.net/ It took me some time to understand the lifeczycle of benchmark and state objects and run iterations. Gruss Bernd -- http://bernd.eckenfels.net >From Win 10 Mobile Von: Aleksey Shipilev From aleksey.shipilev at oracle.com Mon Jul 25 21:02:46 2016 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Mon, 25 Jul 2016 21:02:46 +0000 Subject: hg: code-tools/jmh: 7901746: Command line options help should list default values Message-ID: <201607252102.u6PL2kt2010356@aojmv0008.oracle.com> Changeset: 397891feda1b Author: shade Date: 2016-07-26 00:02 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/397891feda1b 7901746: Command line options help should list default values ! jmh-core/src/main/java/org/openjdk/jmh/runner/BenchmarkList.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/Defaults.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/options/ChainedOptionsBuilder.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/options/CommandLineOptions.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/options/OptionFormatter.java From aleksey.shipilev at oracle.com Mon Jul 25 21:03:51 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Tue, 26 Jul 2016 00:03:51 +0300 Subject: default values for options In-Reply-To: References: Message-ID: On 07/25/2016 02:54 AM, Travis Downs wrote: > Running with the -h option gives a reasonable description of the various > possible command line flags and options, but doesn't seem to mention which > flag/option is the default if not explicit specified. Is this documented > anywhere? Ah, yes, this was not very easy to do, because "default" command line options are known by Runner itself, not by the command line option parsing code. This is because we don't want command line options defaults to override annotations. But, from UX standpoint, users don't care about that internal detail, and rightfully expect defaults to be spelled in -h. Fixed: https://bugs.openjdk.java.net/browse/CODETOOLS-7901746 Thanks, -Aleksey From aleksey.shipilev at oracle.com Mon Jul 25 21:40:19 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Tue, 26 Jul 2016 00:40:19 +0300 Subject: Any option to use something other than "time" to measure benchmarks? In-Reply-To: References: Message-ID: <4841ddf0-d0ee-5eb9-67e1-7a70b140ecca@oracle.com> On 07/25/2016 03:07 AM, Travis Downs wrote: > When I reduce most of the variability by running at "max frequency" and > disabling most OS power management, I still see a turbo-boost related > effect where the first iterations of a benchmark run at a high (turbo) > speed that later ones (which throttle down) - which also depends on factors > like the ambient air temperature, and the type of pants I am wearing > (believe it or not - when I wear fleece pants, it causes higher temps and > turbo throttles down much more quickly). To be blunt, it is a wishful thinking that you can throw the benchmark against the wall of non-prepared hardware and it will produce meaningful data. This is a problem with the benchmarking environment, not with the metric a harness uses. Software cannot fix a misbehaving hardware or non-cooperating software stack. It is a known thing to watch CPU frequency policy and thermals when doing benchmarking. JMH even has a "JMH Core Benchmarks" package that assesses the environment health, see e.g. runnable: http://central.maven.org/maven2/org/openjdk/jmh/jmh-core-benchmarks/1.13/jmh-core-benchmarks-1.13-full.jar > There is a near-panacea for all this: have an option to use a measurement > which directly measures CPU cycles, rather than time. For example, > something like "unhalted cpu cycles" offered by the performance counters on > x86 chips. This solves most of the problems above since this counter > "ticks" at the same speed as the CPU. Some variability remains because some > parts of the system (notably, the latency to RAM) may not scale in the same > way, but in general it is much better. > > I don't know practical this is to do from JMH, so for now I'm just throwing > the idea out there. There _is_ support for measuring cycles taken per benchmark op: perfnorm, which normalizes the perf counters per @Benchmark calls. This metric is not primary, but it is available. I don't believe measuring hardware cycles is panacea. First, performance counters support (esp. from userland) differs within hardware and OS variants. The known reliable thing is Linux/x86, with Linux/ARM and Linux/POWER closing up. Time measurement, on the other hand, is ubiquitously supported. It seems odd to require performance counters to be accessible on a platform to do performance measurements. Second, not-so-painfully-available PMU sampling is not as accurate as timing measurement over many samples. For example, simple HelloWorld test from Samples: $ java -jar jmh-samples/target/benchmarks.jar Hello -f 5 -wi 5 -i 5 -tu ns -bm avgt -prof perfnorm Benchmark Mode Cnt Score Error Units JMHSample_01_...wellHelloThere avgt 25 0.252 ? 0.002 ns/op JMHSample_01_...wellHelloThere:?cycles avgt 5 1.073 ? 0.043 #/op So in absolute values, timing measurement is 4x lower, and you would expect the error margin to be also 4x lower then, but instead it is 20x lower! This happens for several reasons: a) "cycles" perf is sampling, and so the absolute value is estimate; its accuracy is proportional to sampling frequency; overhead is proportional to that too! b) (harder) correlating external PMU counters that say "system did X cycles from time1 to time2" and internal @Benchmark invocation counters that say "we did Y ops from time3 to time4" (extend this to the number of running working threads!) is an art too; Synchronous per-thread timestamping works better. Granted, you may want something like RDPMC for reading the counters directly, and use that instead of timestamps. Assuming you somehow manhandled Java in doing this, now you have to solve the thread affinity problem, because: a) threads can migrate, making cycles difference bogus; b) several threads may have to be scheduled to the same core (e.g. when there are more threads than cores), at which point PMC difference has to be shared between all participating threads; etc, etc, etc. Thanks, -Aleksey From alex.averbuch at neotechnology.com Tue Jul 26 19:55:58 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Tue, 26 Jul 2016 21:55:58 +0200 Subject: Leve.Iteration @TearDown method called one too many times In-Reply-To: References: Message-ID: Sorry, that was a typo. The actual pom contents is: org.openjdk.jmh jmh-core ${jmh.version} Then while typing the actual version I added the extra "." -- for no good reason. On Mon, Jul 25, 2016 at 8:43 PM, Aleksey Shipilev < aleksey.shipilev at oracle.com> wrote: > On 07/25/2016 04:29 PM, Alex Averbuch wrote: > > FYI, I'm using: > > > > > > org.openjdk.jmh > > jmh-core > > 1.1.3 > > The latest version is 1.13, not 1.1.3. > > Does the problem reproduce with 1.13? > > -Aleksey > > > From travis.downs at gmail.com Thu Jul 28 00:32:31 2016 From: travis.downs at gmail.com (Travis Downs) Date: Wed, 27 Jul 2016 19:32:31 -0500 Subject: default values for options In-Reply-To: References: Message-ID: Awesome, thanks for the quick turnaround! On Mon, Jul 25, 2016 at 4:03 PM, Aleksey Shipilev < aleksey.shipilev at oracle.com> wrote: > On 07/25/2016 02:54 AM, Travis Downs wrote: > > Running with the -h option gives a reasonable description of the various > > possible command line flags and options, but doesn't seem to mention > which > > flag/option is the default if not explicit specified. Is this documented > > anywhere? > > Ah, yes, this was not very easy to do, because "default" command line > options are known by Runner itself, not by the command line option > parsing code. This is because we don't want command line options > defaults to override annotations. > > But, from UX standpoint, users don't care about that internal detail, > and rightfully expect defaults to be spelled in -h. > > Fixed: > https://bugs.openjdk.java.net/browse/CODETOOLS-7901746 > > Thanks, > -Aleksey > > From travis.downs at gmail.com Thu Jul 28 01:49:53 2016 From: travis.downs at gmail.com (Travis Downs) Date: Wed, 27 Jul 2016 20:49:53 -0500 Subject: AW: JMH documentation In-Reply-To: <579677f7.8aacc20a.ff124.8be6@mx.google.com> References: <920a88c4-2c12-da51-b15a-b15b6caa94ba@oracle.com> <579677f7.8aacc20a.ff124.8be6@mx.google.com> Message-ID: Thanks guys, I will dig in. I'm pretty up to date with benchmarking pitfalls (inclduing Java specific one) and have followed JMH for a while but am finally making the jump over from Caliper since it's pretty obvious at this point that JMH is the more active and feature-full framework. On Jul 25, 2016 3:35 PM, wrote: > Aleksey is too humble to mention this, but some of his presentations and > writing (on hotspot performance optimizations) also serve as a good example > into benchmarking methodology and jmh application to those problems. > > > > In fact I think some writeup distilled from that would be a good > introduction to jmh. > > > > https://shipilev.net/ > > > > It took me some time to understand the lifeczycle of benchmark and state > objects and run iterations. > > > Gruss > Bernd > -- > http://bernd.eckenfels.net > From Win 10 Mobile > > > > *Von: *Aleksey Shipilev > *Gesendet: *Montag, 25. Juli 2016 22:02 > *An: *Travis Downs ; jmh-dev at openjdk.java.net > *Betreff: *Re: JMH documentation > > > > On 07/25/2016 02:56 AM, Travis Downs wrote: > > > Is there any JMH documentation beyond the intro at > > > http://openjdk.java.net/projects/code-tools/jmh/, the information you > get > > > from -h and the samples? I'm not complaining - the samples are pretty > good, > > > and the options are reasonably well documented, but just checking if I'm > > > missing a wiki or other resource that goes into more details on the > various > > > bits and pieces. > > > > No, you are not missing anything. > > > > Our golden source of documentation is context help (-h and Javadoc) and > > runnable JMH Samples. Many other short introductions to JMH (mostly > > trimmed down Samples) are available on Internet, in different stages of > > being obsolete. > > > > Thanks, > > -Aleksey > > > > > > > From alex.averbuch at neotechnology.com Mon Jul 25 13:53:08 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Mon, 25 Jul 2016 15:53:08 +0200 Subject: Leve.Iteration @TearDown method called one too many times In-Reply-To: References: Message-ID: Hmmm... After trying other approaches and then returning to the one described in my original email everything seems to work fine. Unless the problem resurfaces please consider this as user error, I probably did something foolish. On Mon, Jul 25, 2016 at 3:29 PM, Alex Averbuch < alex.averbuch at neotechnology.com> wrote: > FYI, I'm using: > > > org.openjdk.jmh > jmh-core > 1.1.3 > > > > On Mon, Jul 25, 2016 at 3:27 PM, Alex Averbuch < > alex.averbuch at neotechnology.com> wrote: > >> Hi, >> I have a benchmark for measuring database index creation. >> >> Summary: >> * It is an expensive operation (5+ seconds per invocation), so >> Mode.SingleShotTime is used >> * The database runs 5 warmup iterations & 5 measurement iterations >> * Each iteration must create an index from scratch, so the index must >> be dropped/deleted between iterations >> * To delete the index after each iteration a Level.Iteration @TearDown >> method is used. Inside this method the index is dropped >> >> Problem: >> * It appears as though the Level.Iteration @TearDown method is being >> called 1 more time than the @Benchmark method >> * This causes the database (and benchmark) to crash, because the >> database tries to delete an index that does not exist >> >> When executing the benchmark code at the bottom of this email, the >> following is written to console: >> >> # Run progress: 0.00% complete, ETA 00:00:00 >> # Fork: 1 of 1 >> # Warmup Iteration 1: >> CREATE >> DROP >> # Warmup Iteration 2: >> CREATE >> DROP >> # Warmup Iteration 3: >> CREATE >> DROP >> # Warmup Iteration 4: >> CREATE >> DROP >> # Warmup Iteration 5: >> CREATE >> DROP >> Iteration 1: >> CREATE >> DROP >> Iteration 2: >> CREATE >> DROP >> Iteration 3: >> CREATE >> DROP >> Iteration 4: >> CREATE >> DROP >> Iteration 5: >> CREATE >> DROP >> DROP <--------- NOTICE THIS ADDITIONAL >> "DROP" >> >> >> >> Does this (extra DROP) make sense? >> Is there any way to prevent it from happening? >> Thanks in advance! >> >> >> @State( Scope.Benchmark ) >> public class CreateIndex >> { >> protected Database db; >> >> @Setup >> public void setUp() >> { >> db = new Database(); >> // populate db >> } >> >> @TearDown >> public void tearDown() >> { >> db.shutdown(); >> } >> >> @TearDown( Level.Iteration ) >> public void dropIndex() >> { >> System.out.println("DROP"); >> try ( Transaction tx = db.beginTx() ) >> { >> // drop index >> tx.success(); >> } >> } >> >> @Benchmark >> @BenchmarkMode( Mode.SingleShotTime ) >> public void createIndex() >> { >> System.out.println("CREATE"); >> try ( Transaction tx = db.beginTx() ) >> { >> // create index >> tx.success(); >> } >> } >> } >> > > From alex.averbuch at neotechnology.com Mon Jul 25 13:27:13 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Mon, 25 Jul 2016 15:27:13 +0200 Subject: Leve.Iteration @TearDown method called one too many times Message-ID: Hi, I have a benchmark for measuring database index creation. Summary: * It is an expensive operation (5+ seconds per invocation), so Mode.SingleShotTime is used * The database runs 5 warmup iterations & 5 measurement iterations * Each iteration must create an index from scratch, so the index must be dropped/deleted between iterations * To delete the index after each iteration a Level.Iteration @TearDown method is used. Inside this method the index is dropped Problem: * It appears as though the Level.Iteration @TearDown method is being called 1 more time than the @Benchmark method * This causes the database (and benchmark) to crash, because the database tries to delete an index that does not exist When executing the benchmark code at the bottom of this email, the following is written to console: # Run progress: 0.00% complete, ETA 00:00:00 # Fork: 1 of 1 # Warmup Iteration 1: CREATE DROP # Warmup Iteration 2: CREATE DROP # Warmup Iteration 3: CREATE DROP # Warmup Iteration 4: CREATE DROP # Warmup Iteration 5: CREATE DROP Iteration 1: CREATE DROP Iteration 2: CREATE DROP Iteration 3: CREATE DROP Iteration 4: CREATE DROP Iteration 5: CREATE DROP DROP <--------- NOTICE THIS ADDITIONAL "DROP" Does this (extra DROP) make sense? Is there any way to prevent it from happening? Thanks in advance! @State( Scope.Benchmark ) public class CreateIndex { protected Database db; @Setup public void setUp() { db = new Database(); // populate db } @TearDown public void tearDown() { db.shutdown(); } @TearDown( Level.Iteration ) public void dropIndex() { System.out.println("DROP"); try ( Transaction tx = db.beginTx() ) { // drop index tx.success(); } } @Benchmark @BenchmarkMode( Mode.SingleShotTime ) public void createIndex() { System.out.println("CREATE"); try ( Transaction tx = db.beginTx() ) { // create index tx.success(); } } } From alex.averbuch at neotechnology.com Tue Jul 26 19:55:58 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Tue, 26 Jul 2016 21:55:58 +0200 Subject: Leve.Iteration @TearDown method called one too many times In-Reply-To: References: Message-ID: Sorry, that was a typo. The actual pom contents is: org.openjdk.jmh jmh-core ${jmh.version} Then while typing the actual version I added the extra "." -- for no good reason. On Mon, Jul 25, 2016 at 8:43 PM, Aleksey Shipilev < aleksey.shipilev at oracle.com> wrote: > On 07/25/2016 04:29 PM, Alex Averbuch wrote: > > FYI, I'm using: > > > > > > org.openjdk.jmh > > jmh-core > > 1.1.3 > > The latest version is 1.13, not 1.1.3. > > Does the problem reproduce with 1.13? > > -Aleksey > > > From alex.averbuch at neotechnology.com Mon Jul 25 13:29:30 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Mon, 25 Jul 2016 15:29:30 +0200 Subject: Leve.Iteration @TearDown method called one too many times In-Reply-To: References: Message-ID: FYI, I'm using: org.openjdk.jmh jmh-core 1.1.3 On Mon, Jul 25, 2016 at 3:27 PM, Alex Averbuch < alex.averbuch at neotechnology.com> wrote: > Hi, > I have a benchmark for measuring database index creation. > > Summary: > * It is an expensive operation (5+ seconds per invocation), so > Mode.SingleShotTime is used > * The database runs 5 warmup iterations & 5 measurement iterations > * Each iteration must create an index from scratch, so the index must be > dropped/deleted between iterations > * To delete the index after each iteration a Level.Iteration @TearDown > method is used. Inside this method the index is dropped > > Problem: > * It appears as though the Level.Iteration @TearDown method is being > called 1 more time than the @Benchmark method > * This causes the database (and benchmark) to crash, because the > database tries to delete an index that does not exist > > When executing the benchmark code at the bottom of this email, the > following is written to console: > > # Run progress: 0.00% complete, ETA 00:00:00 > # Fork: 1 of 1 > # Warmup Iteration 1: > CREATE > DROP > # Warmup Iteration 2: > CREATE > DROP > # Warmup Iteration 3: > CREATE > DROP > # Warmup Iteration 4: > CREATE > DROP > # Warmup Iteration 5: > CREATE > DROP > Iteration 1: > CREATE > DROP > Iteration 2: > CREATE > DROP > Iteration 3: > CREATE > DROP > Iteration 4: > CREATE > DROP > Iteration 5: > CREATE > DROP > DROP <--------- NOTICE THIS ADDITIONAL > "DROP" > > > > Does this (extra DROP) make sense? > Is there any way to prevent it from happening? > Thanks in advance! > > > @State( Scope.Benchmark ) > public class CreateIndex > { > protected Database db; > > @Setup > public void setUp() > { > db = new Database(); > // populate db > } > > @TearDown > public void tearDown() > { > db.shutdown(); > } > > @TearDown( Level.Iteration ) > public void dropIndex() > { > System.out.println("DROP"); > try ( Transaction tx = db.beginTx() ) > { > // drop index > tx.success(); > } > } > > @Benchmark > @BenchmarkMode( Mode.SingleShotTime ) > public void createIndex() > { > System.out.println("CREATE"); > try ( Transaction tx = db.beginTx() ) > { > // create index > tx.success(); > } > } > } >