From alex.averbuch at neotechnology.com Tue Jun 7 15:24:09 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Tue, 7 Jun 2016 17:24:09 +0200 Subject: Concurrent database query benchmark Message-ID: Hi all, This is my first question to the list -- please excuse the idiocy. I am trying to write a benchmark that performs a very simple query against a database, concurrently from a fixed number of threads. The benchmark works fine with *@Threads( 1 )* but with *@Threads( 4 )* 3/4 threads find nothing in the database, as if it wasn't initialized or is a different database. When inspecting the database in *TxState.setUp()* I find that *System.identityHashCode(db)* is always the same, as is the path to the database, so I assume it is the same store -- a good thing. However, *db.getRecordById( id )* then fails due to empty database -- a bad thing. Can anyone see a reason why the different *randomRecordById()* benchmark method invocations would be receiving a different *db*? Also, how many other mistakes am I making with this benchmark? AFAIK, support for passing *ExampleDbBenchmark benchmarkState* into *TxState.setUp()* is experimental, could this be the cause of my troubles? Thanks in advance! Alex @State( Scope.Benchmark ) public class ExampleDbBenchmark { public static final int RECORD_COUNT = 1_000_000; File dbDir; Database db; @Setup public void setUp() throws IOException { dbDir = Files.createTempDirectory( "bench-db" ).toFile(); db = new Database( dbDir ); populateDb( db ); } void populateDb( Database db ) { try ( Transaction tx = db.beginTx() ) { for ( int i = 0; i < RECORD_COUNT; i++ ) { db.createRecord( i ); } tx.success(); } } @TearDown public void tearDown() throws IOException { db.shutdown(); FileUtils.deleteRecursively( dbDir ); } @State( Scope.Thread ) public static class TxState { ThreadLocalRandom rng; Transaction tx; Database db; @Setup public void setup( ExampleDbBenchmark benchmarkState ) throws InterruptedException { this.rng = ThreadLocalRandom.current(); this.db = benchmarkState.db; this.tx = db.beginTx(); } @TearDown public void tearDown() { tx.close(); } } @Benchmark @Threads( 4 ) @BenchmarkMode( Mode.SampleTime ) public Record randomRecordById( TxState txState ) { long id = txState.rng.nextLong( 1, RECORD_COUNT ); return txState.db.getRecordById( id ); } } From jw_list at headissue.com Tue Jun 7 16:18:24 2016 From: jw_list at headissue.com (Jens Wilke) Date: Tue, 07 Jun 2016 18:18:24 +0200 Subject: Concurrent database query benchmark In-Reply-To: References: Message-ID: <12219818.aWPgzAsaBG@tapsy> On Tuesday 07 June 2016 17:24:09 Alex Averbuch wrote: > When inspecting the database in *TxState.setUp()* I find that > *System.identityHashCode(db)* is always the same, as is the path to the > database, so I assume it is the same store -- a good thing. > However, *db.getRecordById( id )* then fails due to empty database -- a bad > thing. > I don't know what database you are using here. Is tx.success() really committing your initial population? Probably you see the content in one thread because the transaction joins your still open transaction for population, but the transactions in the other threads see nothing, of course. The general approach for the benchmark looks good! Cheers, Jens -- "Everything superfluous is wrong!" // Jens Wilke - headissue GmbH - Germany \// https://headissue.com From alex.averbuch at neotechnology.com Tue Jun 7 16:32:14 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Tue, 7 Jun 2016 18:32:14 +0200 Subject: Concurrent database query benchmark In-Reply-To: <5223497.pg2W80J2xQ@tapsy> References: <5223497.pg2W80J2xQ@tapsy> Message-ID: Thanks Jens, That was the clue I needed. The example I shared was a slightly simplified version of the actual benchmark, which populates the database as below. I was missing the *tx.close()* line. * void populateDb( Database db )* * {* * Transaction tx = db.beginTx();* * try* * {* * for ( int i = 0; i < RECORD_COUNT; i++ )* * {* * db.createRecord();* * if ( i % TX_SIZE == 0 )* * {* * tx.success();* * tx.close(); //<---- THIS WAS MISSING* * tx = db.beginTx();* * }* * }* * tx.success();* * }* * finally* * {* * tx.close();* * }* * }* On Tue, Jun 7, 2016 at 6:17 PM, Jens Wilke wrote: > On Tuesday 07 June 2016 17:24:09 Alex Averbuch wrote: > > When inspecting the database in *TxState.setUp()* I find that > > *System.identityHashCode(db)* is always the same, as is the path to the > > database, so I assume it is the same store -- a good thing. > > However, *db.getRecordById( id )* then fails due to empty database -- a > bad > > thing. > > > > I don't know what database you are using here. Is tx.success() really > committing your initial population? > > Probably you see the content in one thread because the transaction joins > your still open transaction for population, > but the transactions in the other threads see nothing, of course. > > The general approach for the benchmark looks good! > > Cheers, > > Jens > > -- > "Everything superfluous is wrong!" > > // Jens Wilke - headissue GmbH - Germany > \// https://headissue.com > From zolyfarkas at yahoo.com Thu Jun 16 16:24:28 2016 From: zolyfarkas at yahoo.com (Zoltan Farkas) Date: Thu, 16 Jun 2016 12:24:28 -0400 Subject: Question about iteration numbers Message-ID: I have a questions about iteration numbers: # Warmup Iteration 1: 62995.070 ops/s # Warmup Iteration 2: 70660.918 ops/s # Warmup Iteration 3: 73127.240 ops/s ... Iteration 1: 73277.416 ops/s Iteration 2: 72664.450 ops/s Iteration 3: 72385.558 ops/s Is there a way to obtain this number from void InternalProfiler.beforeIteration(BenchmarkParams benchmarkParams, IterationParams iterationParams); ? thanks ?Z From zolyfarkas at yahoo.com Sat Jun 18 12:48:56 2016 From: zolyfarkas at yahoo.com (Zoltan Farkas) Date: Sat, 18 Jun 2016 08:48:56 -0400 Subject: Improvement suggestion for Iteration printout. Message-ID: <813B9EC1-D5E3-4114-931B-223EAD721AD6@yahoo.com> Would it be possible to synchronize on the stdout when printing: Iteration 2: 72664.450 ops/s so that in case there is other output enabled, the line will not get interleaved like: Iteration 2: Bla bla bla 72664.450 ops/s ? let me know thank you ?Z From jw_list at headissue.com Sun Jun 19 11:31:18 2016 From: jw_list at headissue.com (Jens Wilke) Date: Sun, 19 Jun 2016 13:31:18 +0200 Subject: Improvement suggestion for Iteration printout. In-Reply-To: <813B9EC1-D5E3-4114-931B-223EAD721AD6@yahoo.com> References: <813B9EC1-D5E3-4114-931B-223EAD721AD6@yahoo.com> Message-ID: <2313419.0uT36DlEgR@tapsy> On Saturday 18 June 2016 08:48:56 Zoltan Farkas wrote: > Would it be possible to synchronize on the stdout when printing: > > Iteration 2: 72664.450 ops/s > > so that in case there is other output enabled, the line will not get interleaved like: > > Iteration 2: > > Bla bla bla > > 72664.450 ops/s "Iteration 2" gets printed before iteration starts and "ops/s" after the iteration is finished. Everything the iteration code prints, is printed in between. One possible solution would be to print two lines per iteration, like: Iteration 2 running ... Finished itaration 2: 72664.450 ops/s A variant could be to print the first line only with a CR, so when finished and no output happened in between the second line overwrites the first. Ultimately, I think the best way is to have a command line option for this. There are two different extremes here: - simple (real?) micro benchmarks => a compact one line for each iteration - verbose benchmark code => 2 lines with start and finish message for each iteration Cheers, Jens -- "Everything superfluous is wrong!" // Jens Wilke - headissue GmbH - Germany \// https://headissue.com From ptr.stef at gmail.com Tue Jun 21 15:02:04 2016 From: ptr.stef at gmail.com (Petr Stefan) Date: Tue, 21 Jun 2016 17:02:04 +0200 Subject: Raw data output Message-ID: Hi, I'm student of Mathematics and Physics Faculty, Charles University in Prague. I'm now working at Department of Distributed and Dependable Systems under the supervision of Petr Tuma, Lubomir Bulej and Vojtech Horky. We'd like to use JMH tool for collecting benchmarking data and use them for processing in our custom tools (like standalone part of SPL - http://d3s.mff.cuni.cz/research/development_awareness/). But now, JMH doesn't support collecting very raw data (rawData list from JSON output holds just mean from each iteration - except SingleShot benchmarking mode where the mean is the actual value). But we found out, that Statistics class (all it's implementations) has the raw data inside, so we propose a patch, which adds a getter for this data. The raw data aren't ordered by creation time, but that shouldn't be a problem for our purpose and fixing this will require bigger changes in JMH itself. On top of that, we propose second patch with a new file printer for collected raw values. It prints some common headers and then just the raw data with corresponding units. We chosen JSON format without pretty indentation, which is compromise between file size (we expect large amount of values) and human readability. The headers are at the same format as headers in current JSON printer. Please look at proposed changes and give me a feedback. We're open to changes and improvements. Cheers, Petr Stefan -------------- next part -------------- A non-text attachment was scrubbed... Name: New_formatter_for_raw_data_output_.patch Type: text/x-patch Size: 18611 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Support_for_getting_raw_data_out_of_Statistics_.patch Type: text/x-patch Size: 10709 bytes Desc: not available URL: From jw_list at headissue.com Wed Jun 22 12:25:42 2016 From: jw_list at headissue.com (Jens Wilke) Date: Wed, 22 Jun 2016 14:25:42 +0200 Subject: Raw data output In-Reply-To: References: Message-ID: <1680539.ZM4SAbd7Ld@tapsy> On Tuesday 21 June 2016 17:02:04 Petr Stefan wrote: > Hi, > > I'm student of Mathematics and Physics Faculty, Charles University in > Prague. I'm now working at Department of Distributed and Dependable > Systems under the supervision of Petr Tuma, Lubomir Bulej and Vojtech > Horky. We'd like to use JMH tool for collecting benchmarking data and > use them for processing in our custom tools (like standalone part of SPL > - http://d3s.mff.cuni.cz/research/development_awareness/). But now, JMH > doesn't support collecting very raw data (rawData list from JSON output > holds just mean from each iteration - except SingleShot benchmarking > mode where the mean is the actual value). Can you please double check. In my JSON results, raw data is included. Example: "primaryMetric" : { "score" : 2.6992631363634273E7, "scoreError" : 3362666.9122555815, "scoreConfidence" : [ 2.3629964451378692E7, 3.0355298275889855E7 ], "scorePercentiles" : { "0.0" : 2.549073134560909E7, "50.0" : 2.751281317032709E7, "90.0" : 2.8017559168625783E7, "95.0" : 2.8017559168625783E7, "99.0" : 2.8017559168625783E7, "99.9" : 2.8017559168625783E7, "99.99" : 2.8017559168625783E7, "99.999" : 2.8017559168625783E7, "99.9999" : 2.8017559168625783E7, "100.0" : 2.8017559168625783E7 }, "scoreUnit" : "ops/s", "rawData" : [ [ 2.712889602844093E7, 2.549073134560909E7, 2.7912823329979304E7 ], [ 2.789673031221325E7, 2.5509047996937286E7, 2.8017559168625783E7 ] ] }, Using JMH version 1.11.3. Cheers, Jens -- "Everything superfluous is wrong!" // Jens Wilke - headissue GmbH - Germany \// https://headissue.com From ptr.stef at gmail.com Wed Jun 22 13:39:13 2016 From: ptr.stef at gmail.com (Petr Stefan) Date: Wed, 22 Jun 2016 15:39:13 +0200 Subject: Raw data output In-Reply-To: <1680539.ZM4SAbd7Ld@tapsy> References: <1680539.ZM4SAbd7Ld@tapsy> Message-ID: <05418788-c0be-3ffa-4b2f-ddfd61f4df7f@gmail.com> Hi, you're right, but not for SampleTime mode of benchmarking. We could probably use SingleShot mode, but there are some issues with it. For example, SingleShot mode ignores "measurementTime" option, so you have to specify exact number of "measurementIterations" to get some data. The same holds for warmup phase, of course. So, in our opinion SampleTime mode is quite better. With proposed patch, it's easy to get that hidden data from Statistics (where they already are) without breaking any existing code. Then, we can extend the current JSON printer to serve these "real raw" data or add a new one as we propose in our second patch. We expect that you wouldn't like to change existing format, so we proposed a new one (but we provided 2 patches, to be flexible with both variants). Cheers, Petr Stefan P.S.: I just found misplaced comma in some cases of new raw JSON printer. I'm sorry, but fixed version is attached. The other patch is correct and you already have it, so I'm not resending it. Dne 22.6.2016 v 14:25 Jens Wilke napsal(a): > Can you please double check. In my JSON results, raw data is included. Example: > > "primaryMetric" : { > "score" : 2.6992631363634273E7, > "scoreError" : 3362666.9122555815, > "scoreConfidence" : [ > 2.3629964451378692E7, > 3.0355298275889855E7 > ], > "scorePercentiles" : { > "0.0" : 2.549073134560909E7, > "50.0" : 2.751281317032709E7, > "90.0" : 2.8017559168625783E7, > "95.0" : 2.8017559168625783E7, > "99.0" : 2.8017559168625783E7, > "99.9" : 2.8017559168625783E7, > "99.99" : 2.8017559168625783E7, > "99.999" : 2.8017559168625783E7, > "99.9999" : 2.8017559168625783E7, > "100.0" : 2.8017559168625783E7 > }, > "scoreUnit" : "ops/s", > "rawData" : [ > [ > 2.712889602844093E7, > 2.549073134560909E7, > 2.7912823329979304E7 > ], > [ > 2.789673031221325E7, > 2.5509047996937286E7, > 2.8017559168625783E7 > ] > ] > }, > > Using JMH version 1.11.3. > > Cheers, > > Jens > -------------- next part -------------- A non-text attachment was scrubbed... Name: New_formatter_for_raw_data_output_fix.patch Type: text/x-patch Size: 18634 bytes Desc: not available URL: From jw_list at headissue.com Wed Jun 22 14:33:49 2016 From: jw_list at headissue.com (Jens Wilke) Date: Wed, 22 Jun 2016 16:33:49 +0200 Subject: Raw data output In-Reply-To: <05418788-c0be-3ffa-4b2f-ddfd61f4df7f@gmail.com> References: <1680539.ZM4SAbd7Ld@tapsy> <05418788-c0be-3ffa-4b2f-ddfd61f4df7f@gmail.com> Message-ID: <1696085.eAq7gLzDLo@tapsy> Hi Petr, On Wednesday 22 June 2016 15:39:13 Petr Stefan wrote: > Hi, > > you're right, but not for SampleTime mode of benchmarking. We could > probably use SingleShot mode, but there are some issues with it. For > example, SingleShot mode ignores "measurementTime" option, so you have > to specify exact number of "measurementIterations" to get some data. > The same holds for warmup phase, of course. So, in our opinion SampleTime > mode is quite better. There are "issues" with _every_ benchmarking mode. That is why there are so many of them ;) Depending on the scenario I need to benchmark, I choose the benchmark mode. I would find it very limiting to be restricted to one, by some additional tooling. Maybe you want to share some benchmarks and discuss about that? > With proposed patch, it's easy to get that hidden > data from Statistics (where they already are) without breaking any > existing code. Then, we can extend the current JSON printer to serve > these "real raw" data or add a new one as we propose in our second > patch. We expect that you wouldn't like to change existing format, so we > proposed a new one (but we provided 2 patches, to be flexible with both > variants). Just for the record: I am not the author of JMH, that is Aleksey Shipilev. I am a humble user, just as you. In my opinion, the current JSON contains lots of different data, so I would be tempted to say it is actually meant to include "everything". Extending it in a compatible way should be possible. My personal preference would be to have an output which contains every peace of gathered information, so I can archive it and get back later and extract more things as I need it. Maybe you can show an example JSON output that you currently get from the existing output and make a proposal what additional values should be included? It is easier to discuss some data output or format with some examples. For Aleksey beeing able to accept your patches you need to sign the Oracle Contributor Agreement. Take a look at http://openjdk.java.net/contribute/ Cheers, Jens -- "Everything superfluous is wrong!" // Jens Wilke - headissue GmbH - Germany \// https://headissue.com From jw_list at headissue.com Wed Jun 22 14:41:56 2016 From: jw_list at headissue.com (Jens Wilke) Date: Wed, 22 Jun 2016 16:41:56 +0200 Subject: Raw data output In-Reply-To: <1696085.eAq7gLzDLo@tapsy> References: <05418788-c0be-3ffa-4b2f-ddfd61f4df7f@gmail.com> <1696085.eAq7gLzDLo@tapsy> Message-ID: <4365487.naB8KdU8ed@tapsy> On Wednesday 22 June 2016 16:33:49 Jens Wilke wrote: > My personal preference would be to have an output which contains every peace of gathered > information, so I can archive it and get back later and extract more things as I need it. Of course peace is always better then piece... :) -- "Everything superfluous is wrong!" // Jens Wilke - headissue GmbH - Germany \// https://headissue.com From alex.averbuch at neotechnology.com Wed Jun 22 21:51:01 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Wed, 22 Jun 2016 14:51:01 -0700 Subject: Benchmark Inheritance Message-ID: I would like for multiple "benchmark" classes to inherit from one base benchmark, where the base benchmark contains all @benchmark methods, and child classes specify the actual data to run against (it is a database benchmark). I've read through the JMH examples. From what I can tell the pattern I am using "should" work, but it does not. My code (three-class hierarchy) & the error are below. Any ideas? Thanks in advance! --- The Error --- [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project jmh-benchmarks: Compilation failure [ERROR] /Users/me/IdeaProjects/jmh-benchmarks/target/generated-sources/annotations/org/me/bench/generated/PropertyReadBenchmark_jmhType_B1.java:[3,8] org.me.bench.generated.PropertyReadBenchmark_jmhType_B1 is not abstract and does not override abstract method propertyDefinition() in org.me.bench.PropertyReadBenchmark --- The Code --- @State( Scope.Benchmark ) public abstract class BaseBenchmark { protected Database db; @Setup public void setUp() { db = new DataGenerator().generate(getConfig()); } protected DataGeneratorConfig getConfig() { return new DataGeneratorConfigBuilder().build(); } @TearDown public void tearDown() { db.shutdown(); } } public abstract class PropertyReadBenchmark extends BaseBenchmark { private final PropertyDefinition propertyDefinition; public PropertyReadBenchmark() { this.propertyDefinition = propertyDefinition(); } @Override protected DataGeneratorConfig getConfig() { return new DataGeneratorConfigBuilder() .withProperties( propertyDefinition ) .build(); } // NOTE: causes compilation failure! protected abstract PropertyDefinition propertyDefinition(); @State( Scope.Thread ) public static class TxState { Transaction tx; @Setup public void setUp( PropertyReadBenchmark benchmarkState ) { this.tx = benchmarkState.db.beginTx(); } @TearDown public void tearDown() { tx.close(); } } @Benchmark public Object getProperty( TxState txState ) { return db.getProperty( propertyDefinition.key() ); } } public class IntPropertyRead extends PropertyReadBenchmark { @Override protected PropertyDefinition propertyDefinition() { return new PropertyDefinition( "int", ValueGenerator.INTEGER ); } } From alex.averbuch at neotechnology.com Wed Jun 22 23:05:28 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Wed, 22 Jun 2016 16:05:28 -0700 Subject: Benchmark Inheritance In-Reply-To: References: Message-ID: FYI, replacing this: protected abstract PropertyDefinition propertyDefinition(); With this: protected PropertyDefinition propertyDefinition() { throw new RuntimeException( "Should never happen?" ); } Let's the project successfully compile but then fails at runtime because, it turns out, PropertyReadBenchmark.propertyDefinition() is called even though PropertyReadBenchmark is abstract. --- The Error --- java.lang.RuntimeException: Should never happen? at org.me.bench.PropertyReadBenchmark.propertyDefinition(PropertyReadBenchmark.java:57) at org.me.bench.PropertyReadBenchmark.(PropertyReadBenchmark.java:38) at org.me.bench.generated.PropertyReadBenchmark_jmhType_B1.(PropertyReadBenchmark_jmhType_B1.java:3) at org.me.bench.generated.PropertyReadBenchmark_jmhType_B2.(PropertyReadBenchmark_jmhType_B2.java:3) at org.me.bench.generated.PropertyReadBenchmark_jmhType_B3.(PropertyReadBenchmark_jmhType_B3.java:2) at org.me.bench.generated.PropertyReadBenchmark_jmhType.(PropertyReadBenchmark_jmhType.java:2) at org.me.bench.generated.IntPropertyRead_getProperty_jmhTest._jmh_tryInit_f_PropertyReadBenchmark3_G(IntPropertyRead_getProperty_jmhTest.java:482) at org.me.bench.generated.IntPropertyRead_getProperty_jmhTest.getProperty_SampleTime(IntPropertyRead_getProperty_jmhTest.java:266) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:430) at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:412) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Benchmark had encountered error, and fail on error was requested Exception in thread "main" java.lang.RuntimeException: Error running JMH benchmark at org.me.bench.client.RunCommand.run(RunCommand.java:156) at org.me.bench.client.Main.main(Main.java:17) Caused by: org.openjdk.jmh.runner.RunnerException: Benchmark caught the exception at org.openjdk.jmh.runner.Runner.runBenchmarks(Runner.java:558) at org.openjdk.jmh.runner.Runner.internalRun(Runner.java:313) at org.openjdk.jmh.runner.Runner.run(Runner.java:206) at org.me.bench.client.RunCommand.run(RunCommand.java:144) ... 1 more Caused by: java.lang.RuntimeException: Should never happen? at org.me.bench.PropertyReadBenchmark.propertyDefinition(PropertyReadBenchmark.java:57) at org.me.bench.PropertyReadBenchmark.(PropertyReadBenchmark.java:38) at org.me.bench.generated.PropertyReadBenchmark_jmhType_B1.(PropertyReadBenchmark_jmhType_B1.java:3) at org.me.bench.generated.PropertyReadBenchmark_jmhType_B2.(PropertyReadBenchmark_jmhType_B2.java:3) at org.me.bench.generated.PropertyReadBenchmark_jmhType_B3.(PropertyReadBenchmark_jmhType_B3.java:2) at org.me.bench.generated.PropertyReadBenchmark_jmhType.(PropertyReadBenchmark_jmhType.java:2) at org.me.bench.generated.IntPropertyRead_getProperty_jmhTest._jmh_tryInit_f_PropertyReadBenchmark3_G(IntPropertyRead_getProperty_jmhTest.java:482) at org.me.bench.generated.IntPropertyRead_getProperty_jmhTest.getProperty_SampleTime(IntPropertyRead_getProperty_jmhTest.java:266) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:430) at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:412) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) On Wed, Jun 22, 2016 at 2:51 PM, Alex Averbuch < alex.averbuch at neotechnology.com> wrote: > I would like for multiple "benchmark" classes to inherit from one base > benchmark, where the base benchmark contains all @benchmark methods, and > child classes specify the actual data to run against (it is a database > benchmark). > > I've read through the JMH examples. From what I can tell the pattern I am > using "should" work, but it does not. My code (three-class hierarchy) & the > error are below. > > Any ideas? > > Thanks in advance! > > --- The Error --- > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:3.1:compile > (default-compile) on project jmh-benchmarks: Compilation failure > [ERROR] > /Users/me/IdeaProjects/jmh-benchmarks/target/generated-sources/annotations/org/me/bench/generated/PropertyReadBenchmark_jmhType_B1.java:[3,8] > org.me.bench.generated.PropertyReadBenchmark_jmhType_B1 is not abstract and > does not override abstract method propertyDefinition() in > org.me.bench.PropertyReadBenchmark > > > --- The Code --- > > > @State( Scope.Benchmark ) > public abstract class BaseBenchmark > { > protected Database db; > > @Setup > public void setUp() > { > db = new DataGenerator().generate(getConfig()); > } > > protected DataGeneratorConfig getConfig() > { > return new DataGeneratorConfigBuilder().build(); > } > > @TearDown > public void tearDown() > { > db.shutdown(); > } > } > > public abstract class PropertyReadBenchmark extends BaseBenchmark > { > private final PropertyDefinition propertyDefinition; > > public PropertyReadBenchmark() > { > this.propertyDefinition = propertyDefinition(); > } > > @Override > protected DataGeneratorConfig getConfig() > { > return new DataGeneratorConfigBuilder() > .withProperties( propertyDefinition ) > .build(); > } > > // NOTE: causes compilation failure! > protected abstract PropertyDefinition propertyDefinition(); > > @State( Scope.Thread ) > public static class TxState > { > Transaction tx; > > @Setup > public void setUp( PropertyReadBenchmark benchmarkState ) > { > this.tx = benchmarkState.db.beginTx(); > } > > @TearDown > public void tearDown() > { > tx.close(); > } > > } > > @Benchmark > public Object getProperty( TxState txState ) > { > return db.getProperty( propertyDefinition.key() ); > } > } > > public class IntPropertyRead extends PropertyReadBenchmark > { > @Override > protected PropertyDefinition propertyDefinition() > { > return new PropertyDefinition( "int", ValueGenerator.INTEGER ); > } > } > > > > From alex.averbuch at neotechnology.com Thu Jun 23 00:32:30 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Wed, 22 Jun 2016 17:32:30 -0700 Subject: Benchmark Inheritance In-Reply-To: References: Message-ID: --- Update --- After replacing this: // PropertyReadBenchmark.propertyDefinition() protected PropertyDefinition propertyDefinition() { throw new RuntimeException( "Should never happen?" ); } With this: // PropertyReadBenchmark.propertyDefinition() protected PropertyDefinition propertyDefinition() { new PropertyDefinition( "int", ValueGenerator.INTEGER ); } The benchmarks get further, expectedly, but they still crash at the beginning of the second benchmark. Cause of error is that the database is trying to open a store that is already open (was not shutdown after the last time it was opened). Out of desperation I added printouts to get a clearer understanding of the lifecycle. This is what I see: <------ (1) why do these first series of calls occur, when all involved classes are abstract? BaseBenchmark() PropertyReadBenchmark() PropertyReadBenchmark.propertyDefinition() BaseBenchmark.setup() <------ (2) like (1), why is this even called on an abstract class (PropertyReadBenchmark)? PropertyReadBenchmark.getConfig() <------ (3) why was BaseBenchmark.tearDown() not called here? This is cause of error I am guessing. BaseBenchmark() PropertyReadBenchmark() IntPropertyRead.propertyDefinition() IntPropertyRead() BaseBenchmark.setup() PropertyReadBenchmark.getConfig() On Wed, Jun 22, 2016 at 4:05 PM, Alex Averbuch < alex.averbuch at neotechnology.com> wrote: > FYI, replacing this: > > protected abstract PropertyDefinition propertyDefinition(); > > With this: > > protected PropertyDefinition propertyDefinition() > { > throw new RuntimeException( "Should never happen?" ); > } > > Let's the project successfully compile but then fails at runtime because, > it turns out, PropertyReadBenchmark.propertyDefinition() is called even > though PropertyReadBenchmark is abstract. > > --- The Error --- > > java.lang.RuntimeException: Should never happen? > at > org.me.bench.PropertyReadBenchmark.propertyDefinition(PropertyReadBenchmark.java:57) > at > org.me.bench.PropertyReadBenchmark.(PropertyReadBenchmark.java:38) > at > org.me.bench.generated.PropertyReadBenchmark_jmhType_B1.(PropertyReadBenchmark_jmhType_B1.java:3) > at > org.me.bench.generated.PropertyReadBenchmark_jmhType_B2.(PropertyReadBenchmark_jmhType_B2.java:3) > at > org.me.bench.generated.PropertyReadBenchmark_jmhType_B3.(PropertyReadBenchmark_jmhType_B3.java:2) > at > org.me.bench.generated.PropertyReadBenchmark_jmhType.(PropertyReadBenchmark_jmhType.java:2) > at > org.me.bench.generated.IntPropertyRead_getProperty_jmhTest._jmh_tryInit_f_PropertyReadBenchmark3_G(IntPropertyRead_getProperty_jmhTest.java:482) > at > org.me.bench.generated.IntPropertyRead_getProperty_jmhTest.getProperty_SampleTime(IntPropertyRead_getProperty_jmhTest.java:266) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:430) > at > org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:412) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > > Benchmark had encountered error, and fail on error was requested > Exception in thread "main" java.lang.RuntimeException: Error running JMH > benchmark > at org.me.bench.client.RunCommand.run(RunCommand.java:156) > at org.me.bench.client.Main.main(Main.java:17) > Caused by: org.openjdk.jmh.runner.RunnerException: Benchmark caught the > exception > at org.openjdk.jmh.runner.Runner.runBenchmarks(Runner.java:558) > at org.openjdk.jmh.runner.Runner.internalRun(Runner.java:313) > at org.openjdk.jmh.runner.Runner.run(Runner.java:206) > at org.me.bench.client.RunCommand.run(RunCommand.java:144) > ... 1 more > Caused by: java.lang.RuntimeException: Should never happen? > at > org.me.bench.PropertyReadBenchmark.propertyDefinition(PropertyReadBenchmark.java:57) > at > org.me.bench.PropertyReadBenchmark.(PropertyReadBenchmark.java:38) > at > org.me.bench.generated.PropertyReadBenchmark_jmhType_B1.(PropertyReadBenchmark_jmhType_B1.java:3) > at > org.me.bench.generated.PropertyReadBenchmark_jmhType_B2.(PropertyReadBenchmark_jmhType_B2.java:3) > at > org.me.bench.generated.PropertyReadBenchmark_jmhType_B3.(PropertyReadBenchmark_jmhType_B3.java:2) > at > org.me.bench.generated.PropertyReadBenchmark_jmhType.(PropertyReadBenchmark_jmhType.java:2) > at > org.me.bench.generated.IntPropertyRead_getProperty_jmhTest._jmh_tryInit_f_PropertyReadBenchmark3_G(IntPropertyRead_getProperty_jmhTest.java:482) > at > org.me.bench.generated.IntPropertyRead_getProperty_jmhTest.getProperty_SampleTime(IntPropertyRead_getProperty_jmhTest.java:266) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:430) > at > org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:412) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > On Wed, Jun 22, 2016 at 2:51 PM, Alex Averbuch < > alex.averbuch at neotechnology.com> wrote: > >> I would like for multiple "benchmark" classes to inherit from one base >> benchmark, where the base benchmark contains all @benchmark methods, and >> child classes specify the actual data to run against (it is a database >> benchmark). >> >> I've read through the JMH examples. From what I can tell the pattern I am >> using "should" work, but it does not. My code (three-class hierarchy) & the >> error are below. >> >> Any ideas? >> >> Thanks in advance! >> >> --- The Error --- >> >> [ERROR] Failed to execute goal >> org.apache.maven.plugins:maven-compiler-plugin:3.1:compile >> (default-compile) on project jmh-benchmarks: Compilation failure >> [ERROR] >> /Users/me/IdeaProjects/jmh-benchmarks/target/generated-sources/annotations/org/me/bench/generated/PropertyReadBenchmark_jmhType_B1.java:[3,8] >> org.me.bench.generated.PropertyReadBenchmark_jmhType_B1 is not abstract and >> does not override abstract method propertyDefinition() in >> org.me.bench.PropertyReadBenchmark >> >> >> --- The Code --- >> >> >> @State( Scope.Benchmark ) >> public abstract class BaseBenchmark >> { >> protected Database db; >> >> @Setup >> public void setUp() >> { >> db = new DataGenerator().generate(getConfig()); >> } >> >> protected DataGeneratorConfig getConfig() >> { >> return new DataGeneratorConfigBuilder().build(); >> } >> >> @TearDown >> public void tearDown() >> { >> db.shutdown(); >> } >> } >> >> public abstract class PropertyReadBenchmark extends BaseBenchmark >> { >> private final PropertyDefinition propertyDefinition; >> >> public PropertyReadBenchmark() >> { >> this.propertyDefinition = propertyDefinition(); >> } >> >> @Override >> protected DataGeneratorConfig getConfig() >> { >> return new DataGeneratorConfigBuilder() >> .withProperties( propertyDefinition ) >> .build(); >> } >> >> // NOTE: causes compilation failure! >> protected abstract PropertyDefinition propertyDefinition(); >> >> @State( Scope.Thread ) >> public static class TxState >> { >> Transaction tx; >> >> @Setup >> public void setUp( PropertyReadBenchmark benchmarkState ) >> { >> this.tx = benchmarkState.db.beginTx(); >> } >> >> @TearDown >> public void tearDown() >> { >> tx.close(); >> } >> >> } >> >> @Benchmark >> public Object getProperty( TxState txState ) >> { >> return db.getProperty( propertyDefinition.key() ); >> } >> } >> >> public class IntPropertyRead extends PropertyReadBenchmark >> { >> @Override >> protected PropertyDefinition propertyDefinition() >> { >> return new PropertyDefinition( "int", ValueGenerator.INTEGER ); >> } >> } >> >> >> >> > From aleksey.shipilev at oracle.com Thu Jun 23 09:41:06 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 23 Jun 2016 12:41:06 +0300 Subject: Benchmark Inheritance In-Reply-To: References: Message-ID: <576BAEB2.6040706@oracle.com> On 06/23/2016 12:51 AM, Alex Averbuch wrote: > I would like for multiple "benchmark" classes to inherit from one base > benchmark, where the base benchmark contains all @benchmark methods, and > child classes specify the actual data to run against (it is a database > benchmark). > > I've read through the JMH examples. From what I can tell the pattern I am > using "should" work, but it does not. My code (three-class hierarchy) & the > error are below. > > Any ideas? You are caught in a very unlucky place with benchmark hierarchy, abstract classes, and @State DAGs. The problem goes like this: IntProperty.getProperty(TxState txState): implicit state: IntPropertyRead instance (this) states: TxState ...resolution continues: TxState.setUp: state: PropertyReadBenchmark The underlying reason is that @State dependency matching machinery is oblivious of class hierarchies: IntPropertyRead and PropertyReadBenchmark are *distinct* states for it. So it fails when PropertyReadBenchmark is abstract: it cannot be a @State on its own -- only its subclasses can be instantiated. The compiler would fail to produce an auxiliary JMH classes for PropertyReadBenchmark, without having all the abstract methods implemented. It fails if make PropertyReadBenchmark non-abstract: the compilation will proceed, and the dependency matching machinery will feed the *PropertyReadBenchmark* instance into TxState.setUp, not the *IntPropertyRead* one. Of course, it you put exception into PropertyReadBenchmark.propertyDefinition, it would throw. I am not sure it is easily fixable, so the immediate workaround it to avoid the complex hierarchy/dependency constructions. Thanks, -Aleksey From aleksey.shipilev at oracle.com Thu Jun 23 17:04:34 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 23 Jun 2016 20:04:34 +0300 Subject: Question about iteration numbers In-Reply-To: References: Message-ID: <576C16A2.2070008@oracle.com> On 16.06.2016 19:24, Zoltan Farkas wrote: > I have a questions about iteration numbers: > # Warmup Iteration 1: 62995.070 ops/s > # Warmup Iteration 2: 70660.918 ops/s > # Warmup Iteration 3: 73127.240 ops/s > ... > Iteration 1: 73277.416 ops/s > Iteration 2: 72664.450 ops/s > Iteration 3: 72385.558 ops/s > > Is there a way to obtain this number from void > InternalProfiler.beforeIteration(BenchmarkParams benchmarkParams, > IterationParams iterationParams)? No, there is no way to get this data from within the profiler. Both BenchmarkParams and IterationParams are the configuration parameters, not the on-going run state. Why would you need it? Thanks, -Aleksey From aleksey.shipilev at oracle.com Thu Jun 23 17:14:08 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 23 Jun 2016 20:14:08 +0300 Subject: Improvement suggestion for Iteration printout. In-Reply-To: <2313419.0uT36DlEgR@tapsy> References: <813B9EC1-D5E3-4114-931B-223EAD721AD6@yahoo.com> <2313419.0uT36DlEgR@tapsy> Message-ID: <576C18E0.2000408@oracle.com> On 19.06.2016 14:31, Jens Wilke wrote: > On Saturday 18 June 2016 08:48:56 Zoltan Farkas wrote: >> Would it be possible to synchronize on the stdout when printing: >> >> Iteration 2: 72664.450 ops/s >> >> so that in case there is other output enabled, the line will not get interleaved like: >> >> Iteration 2: >> >> Bla bla bla >> >> 72664.450 ops/s > > "Iteration 2" gets printed before iteration starts and "ops/s" after the iteration is finished. > Everything the iteration code prints, is printed in between. Exactly so. And the reason for that is simple: this provides enough indication that an iteration is in progress, plus it is invisible in the run logs after the iteration stops. I understand that workloads may want to print something out, but this is a corner case that we should not really handle with more garbage in the text output. The way out is to provide the facility to accumulate logs, e.g.: https://bugs.openjdk.java.net/browse/CODETOOLS-7901039 Thanks, -Aleksey From aleksey.shipilev at oracle.com Thu Jun 23 17:19:01 2016 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 23 Jun 2016 17:19:01 +0000 Subject: hg: code-tools/jmh: 7901706: @State classes are not validated when accessed as dependency Message-ID: <201606231719.u5NHJ18Z006992@aojmv0008.oracle.com> Changeset: 13ae22114e22 Author: shade Date: 2016-06-23 20:18 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/13ae22114e22 7901706: @State classes are not validated when accessed as dependency + jmh-core-ct/src/test/java/org/openjdk/jmh/ct/states/dag/AbstractDependencyTest.java ! jmh-core/src/main/java/org/openjdk/jmh/generators/core/BenchmarkGenerator.java ! jmh-core/src/main/java/org/openjdk/jmh/generators/core/StateObjectHandler.java ! jmh-generator-annprocess/src/main/java/org/openjdk/jmh/generators/annotations/APClassInfo.java From aleksey.shipilev at oracle.com Thu Jun 23 17:37:01 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 23 Jun 2016 20:37:01 +0300 Subject: Benchmark Inheritance In-Reply-To: <576BAEB2.6040706@oracle.com> References: <576BAEB2.6040706@oracle.com> Message-ID: <576C1E3D.8020401@oracle.com> On 23.06.2016 12:41, Aleksey Shipilev wrote: > You are caught in a very unlucky place with benchmark hierarchy, > abstract classes, and @State DAGs. The problem goes like this: > > IntProperty.getProperty(TxState txState): > implicit state: IntPropertyRead instance (this) > states: TxState > > ...resolution continues: > > TxState.setUp: > state: PropertyReadBenchmark > > The underlying reason is that @State dependency matching machinery is > oblivious of class hierarchies: IntPropertyRead and > PropertyReadBenchmark are *distinct* states for it. Thinking more about it, this is lesser of two evils. > So it fails when PropertyReadBenchmark is abstract: it cannot be a > @State on its own -- only its subclasses can be instantiated. The > compiler would fail to produce an auxiliary JMH classes for > PropertyReadBenchmark, without having all the abstract methods implemented. This case should have better diagnostics (fixed): https://bugs.openjdk.java.net/browse/CODETOOLS-7901706 > It fails if make PropertyReadBenchmark non-abstract: the compilation > will proceed, and the dependency matching machinery will feed the > *PropertyReadBenchmark* instance into TxState.setUp, not the > *IntPropertyRead* one. Of course, it you put exception into > PropertyReadBenchmark.propertyDefinition, it would throw. This is actually a correct behavior: we should keep matching by class name, regardless if there are classes that can be substituted for each other. This eliminates nasty ambiguities, e.g.: @State public class State1 {} @State public class State2 extends State1 {} @Benchmark public void test(State1 s1, State2 s2) { assert (s1 instanceof State1); // oops, can use State2 here? assert (s2 instanceof State2); } Thanks, -Aleksey From alex.averbuch at neotechnology.com Thu Jun 23 18:22:23 2016 From: alex.averbuch at neotechnology.com (Alex Averbuch) Date: Thu, 23 Jun 2016 11:22:23 -0700 Subject: Benchmark Inheritance In-Reply-To: <576C1E3D.8020401@oracle.com> References: <576BAEB2.6040706@oracle.com> <576C1E3D.8020401@oracle.com> Message-ID: Thanks Aleksey, Two questions.. --- 1 --- I'll see if I can restructure the project to get around these things. Given the following requirements, Are there any JMH patterns you could suggest? Simple database read benchmark * Simple schema: single-valued records, configurable value type (10-20 types) * 5-10 benchmark methods (per value type) * Benchmark methods must be run within long living transactions (to avoid overhead of transaction creation) --- 2 --- In one of my previous emails I wrote: "<------ (3) why was BaseBenchmark.tearDown() not called here? This is cause of error I am guessing." Do you know why this occurs? On Thu, Jun 23, 2016 at 10:37 AM, Aleksey Shipilev < aleksey.shipilev at oracle.com> wrote: > On 23.06.2016 12:41, Aleksey Shipilev wrote: > > You are caught in a very unlucky place with benchmark hierarchy, > > abstract classes, and @State DAGs. The problem goes like this: > > > > IntProperty.getProperty(TxState txState): > > implicit state: IntPropertyRead instance (this) > > states: TxState > > > > ...resolution continues: > > > > TxState.setUp: > > state: PropertyReadBenchmark > > > > The underlying reason is that @State dependency matching machinery is > > oblivious of class hierarchies: IntPropertyRead and > > PropertyReadBenchmark are *distinct* states for it. > > Thinking more about it, this is lesser of two evils. > > > > So it fails when PropertyReadBenchmark is abstract: it cannot be a > > @State on its own -- only its subclasses can be instantiated. The > > compiler would fail to produce an auxiliary JMH classes for > > PropertyReadBenchmark, without having all the abstract methods > implemented. > > This case should have better diagnostics (fixed): > https://bugs.openjdk.java.net/browse/CODETOOLS-7901706 > > > > It fails if make PropertyReadBenchmark non-abstract: the compilation > > will proceed, and the dependency matching machinery will feed the > > *PropertyReadBenchmark* instance into TxState.setUp, not the > > *IntPropertyRead* one. Of course, it you put exception into > > PropertyReadBenchmark.propertyDefinition, it would throw. > > This is actually a correct behavior: we should keep matching by class > name, regardless if there are classes that can be substituted for each > other. This eliminates nasty ambiguities, e.g.: > > @State > public class State1 {} > > @State > public class State2 extends State1 {} > > @Benchmark > public void test(State1 s1, State2 s2) { > assert (s1 instanceof State1); // oops, can use State2 here? > assert (s2 instanceof State2); > } > > Thanks, > -Aleksey > > > > > From zolyfarkas at yahoo.com Thu Jun 23 18:54:59 2016 From: zolyfarkas at yahoo.com (Zoltan Farkas) Date: Thu, 23 Jun 2016 14:54:59 -0400 Subject: Question about iteration numbers In-Reply-To: <576C16A2.2070008@oracle.com> References: <576C16A2.2070008@oracle.com> Message-ID: I was investigating some benchmark behavior where I was seeing: # Warmup Iteration 1: 70485.272 ops/s # Warmup Iteration 2: 77614.082 ops/s # Warmup Iteration 3: 4704.416 ops/s And I wanted to compare the benchmark profile between 2 iterations? I modified my profiler to save the iteration profile to disk in ?afterIteration? and I wanted to add the iteration number in the file name somewhere for correlation purposes? I ended up keeping a count myself :-), here is the actual implementation detail: https://github.com/zolyfarkas/spf4j/blob/master/spf4j-jmh/src/main/java/org/spf4j/stackmonitor/Spf4jJmhProfiler.java Ultimately this helped me narrow down the potential cause, if interested please see for more detail: http://zolyfarkas.github.io/spf4j/spf4j-benchmarks/CrazyJVM.pdf This is also related with my other question regarding the printing out of: ?# Warmup Iteration 1: 70485.272 ops/s? if it would be possible to print this out like: synchronized(System.out) { ... } this way it will(might) not be broken up by up by other output (like GC, JIT details?) assuming that the other output acquires the System.out monitor?(which might not be the case?) let me know if you have any questions. thank you ?Z > On Jun 23, 2016, at 1:04 PM, Aleksey Shipilev wrote: > > On 16.06.2016 19:24, Zoltan Farkas wrote: >> I have a questions about iteration numbers: >> # Warmup Iteration 1: 62995.070 ops/s >> # Warmup Iteration 2: 70660.918 ops/s >> # Warmup Iteration 3: 73127.240 ops/s >> ... >> Iteration 1: 73277.416 ops/s >> Iteration 2: 72664.450 ops/s >> Iteration 3: 72385.558 ops/s >> >> Is there a way to obtain this number from void >> InternalProfiler.beforeIteration(BenchmarkParams benchmarkParams, >> IterationParams iterationParams)? > > No, there is no way to get this data from within the profiler. Both > BenchmarkParams and IterationParams are the configuration parameters, > not the on-going run state. > > Why would you need it? > > Thanks, > -Aleksey > From thurston at nomagicsoftware.com Fri Jun 24 19:17:21 2016 From: thurston at nomagicsoftware.com (thurston at nomagicsoftware.com) Date: Fri, 24 Jun 2016 12:17:21 -0700 Subject: GroupThreads all remainig Message-ID: <434440a933c8147f22f8351a2216e5b9@nomagicsoftware.com> I was curious whether there is a way to specify like the following: @Group("abc") @GroupThreads(1) @Benchmark ...... @Group("abc") @GroupThreads(*) @Benchmark ..... Where the star would indicate "all of the rest of the threads" I realize the value parameter is an int, so something like -1 could serve as the indicator. I tend to set the benchmark threads from the command-line, and it would be a lot easier to just compile the benchmark once From nitsanw at yahoo.com Fri Jun 24 19:48:30 2016 From: nitsanw at yahoo.com (Nitsan Wakart) Date: Fri, 24 Jun 2016 19:48:30 +0000 (UTC) Subject: GroupThreads all remainig In-Reply-To: <434440a933c8147f22f8351a2216e5b9@nomagicsoftware.com> References: <434440a933c8147f22f8351a2216e5b9@nomagicsoftware.com> Message-ID: <873060542.935159.1466797710639.JavaMail.yahoo@mail.yahoo.com> You can use the '-tg' option which allows control on thread counts.For your benchmark you'd go from command line:java -jar benchy.jar -tg 1,127 FooBench On Friday, June 24, 2016 9:18 PM, "thurston at nomagicsoftware.com" wrote: I was curious whether there is a way to specify like the following: @Group("abc") @GroupThreads(1) @Benchmark ...... @Group("abc") @GroupThreads(*) @Benchmark ..... Where the star would indicate "all of the rest of the threads" I realize the value parameter is an int, so something like -1 could serve as the indicator. I tend to set the benchmark threads from the command-line, and it would be a lot easier to just compile the benchmark once From thurston at nomagicsoftware.com Fri Jun 24 20:35:29 2016 From: thurston at nomagicsoftware.com (thurston at nomagicsoftware.com) Date: Fri, 24 Jun 2016 13:35:29 -0700 Subject: GroupThreads all remainig In-Reply-To: <873060542.935159.1466797710639.JavaMail.yahoo@mail.yahoo.com> References: <434440a933c8147f22f8351a2216e5b9@nomagicsoftware.com> <873060542.935159.1466797710639.JavaMail.yahoo@mail.yahoo.com> Message-ID: How are the comma-separated values assigned? By line#? Or lexically, based on the benchmark methods names? The info printed doesn't indicate the thread assignment. On 2016-06-24 12:48, Nitsan Wakart wrote: > You can use the '-tg' option which allows control on thread counts. > For your benchmark you'd go from command line: > java -jar benchy.jar -tg 1,127 FooBench > > On Friday, June 24, 2016 9:18 PM, "thurston at nomagicsoftware.com" > wrote: > > I was curious whether there is a way to specify like the following: > > @Group("abc") > @GroupThreads(1) > @Benchmark > ...... > > @Group("abc") > @GroupThreads(*) > @Benchmark > ..... > > Where the star would indicate "all of the rest of the threads" > > I realize the value parameter is an int, so something like -1 could > serve as the indicator. > > I tend to set the benchmark threads from the command-line, and it > would > be a lot easier to just compile the benchmark once From ptr.stef at gmail.com Tue Jun 28 11:19:54 2016 From: ptr.stef at gmail.com (Petr Stefan) Date: Tue, 28 Jun 2016 13:19:54 +0200 Subject: Raw data output In-Reply-To: <1696085.eAq7gLzDLo@tapsy> References: <1680539.ZM4SAbd7Ld@tapsy> <05418788-c0be-3ffa-4b2f-ddfd61f4df7f@gmail.com> <1696085.eAq7gLzDLo@tapsy> Message-ID: <24bc7ab8-806d-28ab-0b39-c4abfb6bd025@gmail.com> Hi. > Depending on the scenario I need to benchmark, I choose the benchmark mode. > I would find it very limiting to be restricted to one, by some additional > tooling. > > Maybe you want to share some benchmarks and discuss about that? We are developing a tool, that collects benchmarking data, processes them and saves them depending on version (like git commit). Then you can specify some SPL formula (if you're interested, look at papers here http://d3s.mff.cuni.cz/research/development_awareness/#performance-testing) and evaluate it on top of the data. Typical use case is that you don't want your code to slow down, so your formula will reflect this requirement and our tool will find regressions in your code between commits. This is too big piece of code (and probably little out-of-topic) to nicely fit into JMH, so we want to get raw data out of JMH and process them ourselves. > Just for the record: I am not the author of JMH, that is Aleksey Shipilev. > I am a humble user, just as you. Sure, thanks for your opinion. We're looking forward to hear some comments from author, too. > In my opinion, the current JSON contains lots of different data, so I would be tempted > to say it is actually meant to include "everything". Extending it in a compatible > way should be possible. > > My personal preference would be to have an output which contains every peace of gathered > information, so I can archive it and get back later and extract more things as I need it. > > Maybe you can show an example JSON output that you currently get from the existing > output and make a proposal what additional values should be included? > It is easier to discuss some data output or format with some examples. It's a big question how to represent the raw data. If you print all the values one by one into current JSON file, you'll get huge amount of numbers per iteration (like tens of thousands). In our patch we're printing the values as pairs of value and quantity, which seems to be more space saving. Also, our JSON format has no preatty indentation, which adds a lot of spaces per line. However we don't want custom binary format, because it's hard to check it by hand. > For Aleksey beeing able to accept your patches you need to sign the Oracle Contributor > Agreement. Take a look at http://openjdk.java.net/contribute/ I think we could deal with the author when necessary. Thanks for mentioning. Cheers, Petr From jw_list at headissue.com Tue Jun 28 13:16:00 2016 From: jw_list at headissue.com (Jens Wilke) Date: Tue, 28 Jun 2016 15:16 +0200 Subject: Raw data output In-Reply-To: <24bc7ab8-806d-28ab-0b39-c4abfb6bd025@gmail.com> References: <1696085.eAq7gLzDLo@tapsy> <24bc7ab8-806d-28ab-0b39-c4abfb6bd025@gmail.com> Message-ID: <2523865.qqZhHd75pF@tapsy> On Tuesday 28 June 2016 13:19:54 Petr Stefan wrote: > Hi. > > > Depending on the scenario I need to benchmark, I choose the benchmark mode. > > I would find it very limiting to be restricted to one, by some additional > > tooling. > > > > Maybe you want to share some benchmarks and discuss about that? > > We are developing a tool, that collects benchmarking data, processes > them and saves them depending on version (like git commit). Then you can > specify some SPL formula (if you're interested, look at papers here > http://d3s.mff.cuni.cz/research/development_awareness/#performance-testing) > and evaluate it on top of the data. Typical use case is that you don't > want your code to slow down, so your formula will reflect this > requirement and our tool will find regressions in your code between > commits. This is too big piece of code (and probably little > out-of-topic) to nicely fit into JMH, so we want to get raw data out of > JMH and process them ourselves. Yes, I looked at your site. BTW: All the papers are hidden behind pay walls, so I cannot really asses the scientific approach in a glimpse. That wasn't the direction what I am aiming at. It's clear that this is out of the scope of JMH. Let's say I have two benchmarks: Throughput: https://github.com/cache2k/cache2k-benchmark/blob/master/jmh-suite/src/main/java/org/cache2k/benchmark/jmh/suite/noEviction/symmetrical/ReadOnlyBenchmark.java Single shot: https://github.com/cache2k/cache2k-benchmark/blob/master/jmh-suite/src/main/java/org/cache2k/benchmark/jmh/suite/noEviction/symmetrical/PopulateParallelOnceBenchmark.java The reason why I choose single shot or through put, is because this results in the most accurate result for the type of operation I want to measure. How can I use your tools to some comparison? If it is fixed to sample mode I can't. But why this limitation? > > Just for the record: I am not the author of JMH, that is Aleksey Shipilev. > > I am a humble user, just as you. > Sure, thanks for your opinion. We're looking forward to hear some > comments from author, too. > > > In my opinion, the current JSON contains lots of different data, so I would be tempted > > to say it is actually meant to include "everything". Extending it in a compatible > > way should be possible. > > > > My personal preference would be to have an output which contains every peace of gathered > > information, so I can archive it and get back later and extract more things as I need it. > > > > Maybe you can show an example JSON output that you currently get from the existing > > output and make a proposal what additional values should be included? > > It is easier to discuss some data output or format with some examples. > It's a big question how to represent the raw data. If you print all the > values one by one into current JSON file, you'll get huge amount of > numbers per iteration (like tens of thousands). In our patch we're > printing the values as pairs of value and quantity, which seems to be > more space saving. Also, our JSON format has no preatty indentation, > which adds a lot of spaces per line. However we don't want custom binary > format, because it's hard to check it by hand. My bad! Now I understand what you are trying to do. For example the ReadOnlyBenchmark yields 123378830 operations per seconds on latest hardware with four threads (with through put mode). For 30 seconds iterations time that would be 3701 million data points. It's a mental example, of course it couldn't be gathered with sample mode that way because it would heavily skew the result. But if it would work, why I should want to analyze the runtime of each operation? To be honest, the sample mode is the only mode I wonder how it can ever be of any use. That's why I am confused about the fact that you say it's the only useful one. Can you give an example? Cheers, Jens -- "Everything superfluous is wrong!" // Jens Wilke - headissue GmbH - Germany \// https://headissue.com From rvansa at redhat.com Wed Jun 29 08:14:07 2016 From: rvansa at redhat.com (Radim Vansa) Date: Wed, 29 Jun 2016 10:14:07 +0200 Subject: Blackhole user instantiation Message-ID: <5773834F.5030507@redhat.com> Hello, I'd like to reuse Blackhole in project aimed at benchmarking distributed systems (RadarGun [1]), but rather than just being tricky bastard [2] I'd like to ask if there's any info what black magic does JMH use to instantiate Blackhole properly and where be the dragons. Note that RadarGun is not meant to be as precise as JMH, most of the benchmarked calls involve RPCs and generally take hundreds of microseconds or more. I use JMH for single JVM benchmarks, RG is convenient to orchestrate many nodes. Thanks Radim [1] https://github.com/radargun/radargun/ [2] http://hg.openjdk.java.net/code-tools/jmh/file/13ae22114e22/jmh-core/src/main/java/org/openjdk/jmh/infra/Blackhole.java#l290 -- Radim Vansa JBoss Performance Team From ptr.stef at gmail.com Wed Jun 29 13:43:38 2016 From: ptr.stef at gmail.com (Petr Stefan) Date: Wed, 29 Jun 2016 15:43:38 +0200 Subject: Raw data output In-Reply-To: <2523865.qqZhHd75pF@tapsy> References: <1696085.eAq7gLzDLo@tapsy> <24bc7ab8-806d-28ab-0b39-c4abfb6bd025@gmail.com> <2523865.qqZhHd75pF@tapsy> Message-ID: <22212d72-003b-4756-b543-483382b7d397@gmail.com> Hi, Dne 28.6.2016 v 15:16 Jens Wilke napsal(a): > The reason why I choose single shot or through put, is because this results in the most accurate result > for the type of operation I want to measure. > > How can I use your tools to some comparison? If it is fixed to sample mode I can't. But why this limitation? > > For example the ReadOnlyBenchmark yields 123378830 operations per seconds on latest hardware with four threads > (with through put mode). For 30 seconds iterations time that would be 3701 million data points. It's a mental example, of course it couldn't be gathered with sample mode that way because it would heavily skew the result. But if it would work, why I should want to analyze the runtime of each operation? > > To be honest, the sample mode is the only mode I wonder how it can ever be of any use. That's > why I am confused about the fact that you say it's the only useful one. > no, it's not JUST the sample time mode, it's all INCLUDING sample time mode. This mode doesn't give us actual raw data while others modes do, so we're only extending the output. Main goal is that for custom processing we want to have raw data, not already preprocessed ones (as sample time mode sort of behaves now). Our tool is not ready yet for you, I'm currently developing it (which includes also extending JMH). Cheers, P.S.