From nitsanw at yahoo.com Sat Oct 1 12:38:45 2016 From: nitsanw at yahoo.com (Nitsan Wakart) Date: Sat, 1 Oct 2016 12:38:45 +0000 (UTC) Subject: volatile boolean isDone in InfraControl In-Reply-To: <57EEE28A.20208@oracle.com> References: <57EEE28A.20208@oracle.com> Message-ID: <1540909195.3714676.1475325525023@mail.yahoo.com> Hi James,Firstly, the isDone/volatileSpoiler effect is entirely intentional.It causes 2 things as you point out:1. Reload all fields in called benchmark method(which is force inlined)2. Prevent loop unrollingThis prevents the compiler from optimizing the method under measurement in an uncontrolled fashion (loop unroll count may vary, a non-volatile read will allow loads to be hoisted etc). The effects of allowing these optimizations to take place is confusing, misleading and fragile. JMH does the *right* thing for the general case by preventing this.This may not be exactly what you need for every case, but this is still the right choice for JMH(IMO). For more information on these decisions look for Shipilev talk on JMH from JVMLS and read through the unrolled look sample.Secondly, you ask how you may measure the cost of a register to register add. The answer IMO is to look it up in Agner Fog instruction cost tables, where you'll get a good approximation for each processor which added indication of reciprocal throughput (The average number of core clock cycles per instruction for a series of independent instructions of the same kind in the same thread). Measuring the cost of single instructions is not very meaningful IMO without the mix of instructions they get executed with on modern processors which can do crafty things like reordering of instructions, micro-op fusion and parallel execution of instructions.I'm not discouraging you from experimentation! you can have some very educational fun measuring stuff and refining your understanding of what the measurements mean, but measuring costs of the sub 10ns scale is tricky at best (in your context: ADD on a modern x86 is 1 cycle -> 0.25 to 0.5ns and the CPU can execute 4 or 5 in parallel).Good Luck,Nitsan From ecki at zusammenkunft.net Sat Oct 1 16:12:59 2016 From: ecki at zusammenkunft.net (ecki at zusammenkunft.net) Date: Sat, 1 Oct 2016 18:12:59 +0200 Subject: AW: volatile boolean isDone in InfraControl In-Reply-To: <57EEE28A.20208@oracle.com> References: <57EEE28A.20208@oracle.com> Message-ID: <57efe08e.a3adc20a.a0ebe.9759@mx.google.com> Hello James, I would think if you want to test unrolling and inlining effects inside loops it is best to have a loop inside your method under test: You dont have the volatile access then inside the hot code and can better emulate the structure of the code. (dont forget to actually return/blackhole a result from your loop) Blackholes have similiar effects: http://psy-lob-saw.blogspot.de/2014/08/the-volatile-read-suprise.html I think Aleksey wrote something about isDone effects on iterated code, but I cant find it right now. Gruss Bernd -- http://bernd.eckenfels.net >From Win 10 Mobile Von: James Cheng Gesendet: Samstag, 1. Oktober 2016 00:39 An: jmh-dev at openjdk.java.net Betreff: volatile boolean isDone in InfraControl Hi, For a benchmark as follows, public int i1 = 1, i2 = -2, i3 = 3, i4 = -4; @Benchmark public void testArithAddInt() { i1 += i4; i2 += i1; i3 += i2; i4 += i3; } the generated testArithAddInt_thrpt_jmhStub() looks like this: result.startTime = System.nanoTime(); do { l_jgarith0_0.testArithAddInt(); operations++; } while(!control.isDone); result.stopTime = System.nanoTime(); In jmh/runner/InfraControl.java, there are: public volatile boolean isDone; public volatile boolean volatileSpoiler; The problem we have is that the volatile boolean isDone appeared to cause the JIT compiler not to unroll the loop and to spill/refill those variables in every iteration. So this benchmark appeared to be dominated by memory ops rather than add ops. Other modes like "avgt" and "sample" also use InfraControl.isDone. The "ss" mode doesn't use InfraControl.isDone, but it uses InfraControl.volatileSpoiler in the loop. Is InfraControl.isDone used here for currency control, avoiding some compiler optimizations on the measurement loop, or both? Is there a different mode/way for benchmarking something like the register-to-register add operation? Thanks, -James From shade at redhat.com Mon Oct 3 09:42:00 2016 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 3 Oct 2016 11:42:00 +0200 Subject: volatile boolean isDone in InfraControl In-Reply-To: <57EEE28A.20208@oracle.com> References: <57EEE28A.20208@oracle.com> Message-ID: <325a0a61-da26-3e4b-0baa-682ad918af68@redhat.com> Hi James, On 10/01/2016 12:09 AM, James Cheng wrote: > For a benchmark as follows, > > public int i1 = 1, i2 = -2, i3 = 3, i4 = -4; > > @Benchmark > public void testArithAddInt() { > i1 += i4; > i2 += i1; > i3 += i2; > i4 += i3; > } ... > The problem we have is that the volatile boolean isDone appeared to > cause the JIT compiler not to unroll the loop and to spill/refill > those variables in every iteration. That's completely intentional. The first part of the story is constant fold elimination, which is prevented in Hotspot by reading the volatile field: http://hg.openjdk.java.net/code-tools/jmh/file/34407adf3d94/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_10_ConstantFold.java The second part of the story is unrollable loops, which breaks benchmarking: http://hg.openjdk.java.net/code-tools/jmh/file/34407adf3d94/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_11_Loops.java > So this benchmark appeared to be dominated by memory ops > rather than add ops. Yes, measuring nanosecond-scale events requires creativity. But I think you measure those for comparison for something else, not for the "absolute performance" alone. While the base overhead is comparable or larger than the operation itself, the difference against something else should be clearly measurable. Thanks, -Aleksey From james.cheng at oracle.com Mon Oct 3 16:45:02 2016 From: james.cheng at oracle.com (James Cheng) Date: Mon, 3 Oct 2016 09:45:02 -0700 Subject: volatile boolean isDone in InfraControl In-Reply-To: <325a0a61-da26-3e4b-0baa-682ad918af68@redhat.com> References: <57EEE28A.20208@oracle.com> <325a0a61-da26-3e4b-0baa-682ad918af68@redhat.com> Message-ID: <57F28B0E.2090509@oracle.com> Hi Aleksey, Many thanks for confirming my speculations and your insights on measuring nanosecond-scale events. And thanks for the pointers to the well-designed JMH samples which I should have read and played with more thoroughly. BTW, congratulations on the 1.15 release! Thanks, -James From james.cheng at oracle.com Mon Oct 3 16:50:01 2016 From: james.cheng at oracle.com (James Cheng) Date: Mon, 3 Oct 2016 09:50:01 -0700 Subject: volatile boolean isDone in InfraControl In-Reply-To: <1540909195.3714676.1475325525023@mail.yahoo.com> References: <57EEE28A.20208@oracle.com> <1540909195.3714676.1475325525023@mail.yahoo.com> Message-ID: <57F28C39.7080209@oracle.com> Hi Nitsan, Many thanks for your quick and insightful answers and suggestions. Have a great day! Thanks, -James From lev at serebryakov.spb.ru Mon Oct 3 17:12:34 2016 From: lev at serebryakov.spb.ru (Lev Serebryakov) Date: Mon, 3 Oct 2016 20:12:34 +0300 Subject: Custom and not-constant "Operations per invocation" (Re: JMH 1.15) In-Reply-To: <14cd3d5e-9683-8c6b-4691-91dadfde2a86@redhat.com> References: <14cd3d5e-9683-8c6b-4691-91dadfde2a86@redhat.com> Message-ID: On 30.09.2016 22:06, Aleksey Shipilev wrote: > *) @AuxCounters improvements. Note this is still a very much > experimental API. That said, it makes sense to improve it a little more: > now it handles more field/method types, has new counting mode "events" > that does not normalize to time, and does not require reset between > iterations. This API change is backwards-compatible. See the RFE: > https://bugs.openjdk.java.net/browse/CODETOOLS-7901810 > ...and new Javadoc: Looks like, it is way to implement custom "operations per invocation" metric, am I right? Problem I have at hands looks like this: I benchmark implementation, which could do batching of low-level operations. To make benchmark more "realistic" I choose size of batch on each invocation at random (really, from pre-filled big array of random numbers, as random generators, even bad ones, are slow). So, each benchmark invocation could be counted as different number of operations, from 1 to, say, 16. and default JMH's "average time per invocation" is not very meaningful in this situation, as I need to know time per basic operation. Looks like "@AuxCounters" is solution, am I right? I only need to count real number of basic operations, and I'm done! -- // Black Lion AKA Lev Serebryakov From james.cheng at oracle.com Mon Oct 3 17:18:39 2016 From: james.cheng at oracle.com (James Cheng) Date: Mon, 3 Oct 2016 10:18:39 -0700 Subject: AW: volatile boolean isDone in InfraControl In-Reply-To: <57efe08e.a3adc20a.a0ebe.9759@mx.google.com> References: <57EEE28A.20208@oracle.com> <57efe08e.a3adc20a.a0ebe.9759@mx.google.com> Message-ID: <57F292EF.90302@oracle.com> Hi Bernd, Many thanks for your quick suggestion and pointers. On 10/1/2016 9:12 AM, ecki at zusammenkunft.net wrote: > Hello James, > > I would think if you want to test unrolling and inlining effects inside loops it is best to have a loop inside your method under test: You dont have the volatile access > then inside the hot code and can better emulate the structure of the code. (dont forget to actually return/blackhole a result from your loop) I will try that and experiment with different loop counts as mentioned in JMHSample_11_Loops.java which Aleksey just pointed to me. Thanks, -James From lev at serebryakov.spb.ru Mon Oct 3 17:51:24 2016 From: lev at serebryakov.spb.ru (Lev Serebryakov) Date: Mon, 3 Oct 2016 20:51:24 +0300 Subject: Custom and not-constant "Operations per invocation" (Re: JMH 1.15) In-Reply-To: References: <14cd3d5e-9683-8c6b-4691-91dadfde2a86@redhat.com> Message-ID: <795456ff-1396-776d-78ab-9773273a114e@serebryakov.spb.ru> On 03.10.2016 20:12, Lev Serebryakov wrote: >> *) @AuxCounters improvements. Note this is still a very much >> experimental API. That said, it makes sense to improve it a little more: >> now it handles more field/method types, has new counting mode "events" >> that does not normalize to time, and does not require reset between >> iterations. This API change is backwards-compatible. See the RFE: >> https://bugs.openjdk.java.net/browse/CODETOOLS-7901810 >> ...and new Javadoc: > Looks like, it is way to implement custom "operations per invocation" > metric, am I right? > > Problem I have at hands looks like this: I benchmark implementation, > which could do batching of low-level operations. To make benchmark more > "realistic" I choose size of batch on each invocation at random (really, > from pre-filled big array of random numbers, as random generators, even > bad ones, are slow). So, each benchmark invocation could be counted as > different number of operations, from 1 to, say, 16. and default JMH's > "average time per invocation" is not very meaningful in this situation, > as I need to know time per basic operation. > > Looks like "@AuxCounters" is solution, am I right? I only need to count > real number of basic operations, and I'm done! It seems to work, but I don't understand how result is combined from different threads. Again, I have benchmark which contains of two threads in one group, doing different work. If I have same "counter" state class (Scope.Thread, of course) I get 19ns/count ("op"). If I have two different classes, I get 16ns/count at each thread. It looks strange and I could not do mental model what happens in case when each thread increments logically-the-same counter (in thread-local state). If all instances of this counter are summed up, it should be ~2 times faster than splitted counters (because combined counter is sum of splitted ones, and splitted counters are roughly the same). But it is slightly slower! -- // Black Lion AKA Lev Serebryakov From lev at serebryakov.spb.ru Mon Oct 3 18:07:33 2016 From: lev at serebryakov.spb.ru (Lev Serebryakov) Date: Mon, 3 Oct 2016 21:07:33 +0300 Subject: Is it possible to filter parameters space? Message-ID: <100d865e-ef0d-890e-6170-a65b241bf7d0@serebryakov.spb.ru> JMH have very handy @Param annotation, which allows to easily arrange multiple benchmarks across multi-dimension parameters space. But sometimes SOME combination of parameters doesn't have meaning for benchmark, or simply excessive. For example, if here is two numeric parameters, it could be the case, when they are interchangeable, and it is possible to get full picture by exploring only half of square (over or under main diagonal). Is here any way to register "parameter space predicate", which allows to skip some parameters combinations? IMHO, it could be useful. -- // Black Lion AKA Lev Serebryakov From lev at serebryakov.spb.ru Mon Oct 3 22:00:18 2016 From: lev at serebryakov.spb.ru (Lev Serebryakov) Date: Tue, 4 Oct 2016 01:00:18 +0300 Subject: Custom and not-constant "Operations per invocation" (Re: JMH 1.15) In-Reply-To: References: <14cd3d5e-9683-8c6b-4691-91dadfde2a86@redhat.com> Message-ID: <106391707.20161004005959@serebryakov.spb.ru> Hello Lev, Monday, October 3, 2016, 8:12:34 PM, you wrote: > Looks like "@AuxCounters" is solution, am I right? I only need to count > real number of basic operations, and I'm done! Yep, it works! Only one thing makes me sad panda: all these counters show up in CSV file as separate benchmarks, not as additional data (columns) to their respective parent benchmarks :( -- Best regards, Lev mailto:lev at serebryakov.spb.ru From shade at redhat.com Tue Oct 4 10:44:38 2016 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 4 Oct 2016 12:44:38 +0200 Subject: Is it possible to filter parameters space? In-Reply-To: <100d865e-ef0d-890e-6170-a65b241bf7d0@serebryakov.spb.ru> References: <100d865e-ef0d-890e-6170-a65b241bf7d0@serebryakov.spb.ru> Message-ID: <8e918f60-1056-1635-38ba-ddd7f01a97e2@redhat.com> On 10/03/2016 08:07 PM, Lev Serebryakov wrote: > JMH have very handy @Param annotation, which allows to easily arrange > multiple benchmarks across multi-dimension parameters space. But > sometimes SOME combination of parameters doesn't have meaning for > benchmark, or simply excessive. It is not possible now with annotations, but you can always make the Java API launcher and traverse the parameter space in any way you want. Extending JMH with more control creates another DSL over annotations, and it could be easier to set up an advanced experiment in plain Java to begin with. This is what Java API is for. See also: https://bugs.openjdk.java.net/browse/CODETOOLS-7901296 -Aleksey From shade at redhat.com Tue Oct 4 10:48:21 2016 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 4 Oct 2016 12:48:21 +0200 Subject: Custom and not-constant "Operations per invocation" (Re: JMH 1.15) In-Reply-To: References: <14cd3d5e-9683-8c6b-4691-91dadfde2a86@redhat.com> Message-ID: On 10/03/2016 07:12 PM, Lev Serebryakov wrote: > On 30.09.2016 22:06, Aleksey Shipilev wrote: > >> *) @AuxCounters improvements. Note this is still a very much >> experimental API. That said, it makes sense to improve it a little more: >> now it handles more field/method types, has new counting mode "events" >> that does not normalize to time, and does not require reset between >> iterations. This API change is backwards-compatible. See the RFE: >> https://bugs.openjdk.java.net/browse/CODETOOLS-7901810 >> ...and new Javadoc: > Looks like, it is way to implement custom "operations per invocation" > metric, am I right? No, it helps with auxiliary counters that need to count only the specific paths in the workload. These counters are not supposed to be as accurate as primary metric you get from the workload, and may only serve as the additional bits of data, together with the primary results. > Problem I have at hands looks like this: I benchmark implementation, > which could do batching of low-level operations. To make benchmark more > "realistic" I choose size of batch on each invocation at random (really, > from pre-filled big array of random numbers, as random generators, even > bad ones, are slow). So, each benchmark invocation could be counted as > different number of operations, from 1 to, say, 16. and default JMH's > "average time per invocation" is not very meaningful in this situation, > as I need to know time per basic operation. You have to use Java API for this, and override .operationsPerInvocation(...) to match your .params(...) if you want to normalize the operations like that. > Looks like "@AuxCounters" is solution, am I right? I only need to count > real number of basic operations, and I'm done! No, this is not a reliable solution. As said in both the changelog and the Javadoc, it is a very experimental API. The reliable solution measures the performance with primary metrics, and adjusts it accordingly. Thanks, -Aleksey From lev at serebryakov.spb.ru Tue Oct 4 11:24:52 2016 From: lev at serebryakov.spb.ru (Lev Serebryakov) Date: Tue, 4 Oct 2016 14:24:52 +0300 Subject: Is it possible to filter parameters space? In-Reply-To: <8e918f60-1056-1635-38ba-ddd7f01a97e2@redhat.com> References: <100d865e-ef0d-890e-6170-a65b241bf7d0@serebryakov.spb.ru> <8e918f60-1056-1635-38ba-ddd7f01a97e2@redhat.com> Message-ID: On 04.10.2016 13:44, Aleksey Shipilev wrote: > It is not possible now with annotations, but you can always make the > Java API launcher and traverse the parameter space in any way you want. > Extending JMH with more control creates another DSL over annotations, > and it could be easier to set up an advanced experiment in plain Java to > begin with. This is what Java API is for. I thought about something like @StateFilter boolean doWeNeedTestThisStaty() { ... } on state object. Not pure-annotation solution. > https://bugs.openjdk.java.net/browse/CODETOOLS-7901296 Yep, something like this! :) -- // Black Lion AKA Lev Serebryakov From lev at serebryakov.spb.ru Tue Oct 4 11:57:57 2016 From: lev at serebryakov.spb.ru (Lev Serebryakov) Date: Tue, 4 Oct 2016 14:57:57 +0300 Subject: Custom and not-constant "Operations per invocation" (Re: JMH 1.15) In-Reply-To: References: <14cd3d5e-9683-8c6b-4691-91dadfde2a86@redhat.com> Message-ID: <3d30c399-b4e6-ae54-d900-994768de53fa@serebryakov.spb.ru> On 04.10.2016 13:48, Aleksey Shipilev wrote: > You have to use Java API for this, and override > .operationsPerInvocation(...) to match your .params(...) if you want to > normalize the operations like that. Override on what class? I've read JHM sources, and looks like runner calls getOperationsPerInvocation() from options builder only once per benchmark (but I may be wrong, of course). And in my case, it is different on EACH call of @Bencmark'ed method. And only this method itself know how much work was done at this invocation. I don't see how building options by hands helps in such situation. And my situation is more complicated than that: it is non-symmetrical two-thread (consumer/producer) benchmark, where consumer and producer could use very different (random) values of OPI (and spent time, too!), and this difference could be 2 orders of magnutude. As result, "primary metrics" is completely meaningless now :( But, I should say, that @AuxCounters gives very stable "good-looking" results with very low stddev between runs. Maybe, they are not ns-precise (co, not suitable to estimating low-level characteristics like CPI and such), but looks suitable for comparing different implementations of same API. -- // Black Lion AKA Lev Serebryakov From ashipile at redhat.com Wed Oct 5 17:21:44 2016 From: ashipile at redhat.com (ashipile at redhat.com) Date: Wed, 05 Oct 2016 17:21:44 +0000 Subject: hg: code-tools/jmh: 7901813: @AuxCounters method constraints conflict with @State ones Message-ID: <201610051721.u95HLiCr029715@aojmv0008.oracle.com> Changeset: 7a25c71b43bf Author: shade Date: 2016-10-05 19:21 +0200 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/7a25c71b43bf 7901813: @AuxCounters method constraints conflict with @State ones Summary: Relax the requirement for "void" return type, implicitly allowing @Setup/@TearDown-s. + jmh-core-ct/src/test/java/org/openjdk/jmh/ct/other/auxcounters/HelperConflictTest.java ! jmh-core-ct/src/test/java/org/openjdk/jmh/ct/other/auxcounters/counterTypes/publicMethods/VoidTest.java ! jmh-core/src/main/java/org/openjdk/jmh/annotations/AuxCounters.java ! jmh-core/src/main/java/org/openjdk/jmh/generators/core/StateObjectHandler.java From joel.moberg at gmail.com Wed Oct 5 18:29:45 2016 From: joel.moberg at gmail.com (Joel Moberg) Date: Wed, 5 Oct 2016 20:29:45 +0200 Subject: jmh-dev Digest, Vol 42, Issue 2 In-Reply-To: References: Message-ID: help On Wed, Oct 5, 2016 at 7:21 PM, wrote: > Send jmh-dev mailing list submissions to > jmh-dev at openjdk.java.net > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.openjdk.java.net/mailman/listinfo/jmh-dev > or, via email, send a message with subject or body 'help' to > jmh-dev-request at openjdk.java.net > > You can reach the person managing the list at > jmh-dev-owner at openjdk.java.net > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of jmh-dev digest..." > > > Today's Topics: > > 1. Re: AW: volatile boolean isDone in InfraControl (James Cheng) > 2. Re: Custom and not-constant "Operations per invocation" (Re: > JMH 1.15) (Lev Serebryakov) > 3. Is it possible to filter parameters space? (Lev Serebryakov) > 4. Re: Custom and not-constant "Operations per invocation" (Re: > JMH 1.15) (Lev Serebryakov) > 5. Re: Is it possible to filter parameters space? (Aleksey Shipilev) > 6. Re: Custom and not-constant "Operations per invocation" (Re: > JMH 1.15) (Aleksey Shipilev) > 7. Re: Is it possible to filter parameters space? (Lev Serebryakov) > 8. Re: Custom and not-constant "Operations per invocation" (Re: > JMH 1.15) (Lev Serebryakov) > 9. hg: code-tools/jmh: 7901813: @AuxCounters method constraints > conflict with @State ones (ashipile at redhat.com) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 3 Oct 2016 10:18:39 -0700 > From: James Cheng > To: ecki at zusammenkunft.net, "jmh-dev at openjdk.java.net" > > Subject: Re: AW: volatile boolean isDone in InfraControl > Message-ID: <57F292EF.90302 at oracle.com> > Content-Type: text/plain; charset=utf-8; format=flowed > > Hi Bernd, > > Many thanks for your quick suggestion and pointers. > > On 10/1/2016 9:12 AM, ecki at zusammenkunft.net wrote: > > Hello James, > > > > I would think if you want to test unrolling and inlining effects inside > loops it is best to have a loop inside your method under test: You dont > have the volatile access > > then inside the hot code and can better emulate the structure of the > code. (dont forget to actually return/blackhole a result from your loop) > > I will try that and experiment with different loop counts as mentioned in > JMHSample_11_Loops.java which Aleksey just pointed to me. > > Thanks, > -James > > > ------------------------------ > > Message: 2 > Date: Mon, 3 Oct 2016 20:51:24 +0300 > From: Lev Serebryakov > To: Aleksey Shipilev , JMH Dev > > Subject: Re: Custom and not-constant "Operations per invocation" (Re: > JMH 1.15) > Message-ID: <795456ff-1396-776d-78ab-9773273a114e at serebryakov.spb.ru> > Content-Type: text/plain; charset=utf-8 > > On 03.10.2016 20:12, Lev Serebryakov wrote: > > >> *) @AuxCounters improvements. Note this is still a very much > >> experimental API. That said, it makes sense to improve it a little more: > >> now it handles more field/method types, has new counting mode "events" > >> that does not normalize to time, and does not require reset between > >> iterations. This API change is backwards-compatible. See the RFE: > >> https://bugs.openjdk.java.net/browse/CODETOOLS-7901810 > >> ...and new Javadoc: > > Looks like, it is way to implement custom "operations per invocation" > > metric, am I right? > > > > Problem I have at hands looks like this: I benchmark implementation, > > which could do batching of low-level operations. To make benchmark more > > "realistic" I choose size of batch on each invocation at random (really, > > from pre-filled big array of random numbers, as random generators, even > > bad ones, are slow). So, each benchmark invocation could be counted as > > different number of operations, from 1 to, say, 16. and default JMH's > > "average time per invocation" is not very meaningful in this situation, > > as I need to know time per basic operation. > > > > Looks like "@AuxCounters" is solution, am I right? I only need to count > > real number of basic operations, and I'm done! > It seems to work, but I don't understand how result is combined from > different threads. Again, I have benchmark which contains of two threads > in one group, doing different work. If I have same "counter" state class > (Scope.Thread, of course) I get 19ns/count ("op"). If I have two > different classes, I get 16ns/count at each thread. It looks strange and > I could not do mental model what happens in case when each thread > increments logically-the-same counter (in thread-local state). If all > instances of this counter are summed up, it should be ~2 times faster > than splitted counters (because combined counter is sum of splitted > ones, and splitted counters are roughly the same). But it is slightly > slower! > > -- > // Black Lion AKA Lev Serebryakov > > > > ------------------------------ > > Message: 3 > Date: Mon, 3 Oct 2016 21:07:33 +0300 > From: Lev Serebryakov > To: JMH Dev > Subject: Is it possible to filter parameters space? > Message-ID: <100d865e-ef0d-890e-6170-a65b241bf7d0 at serebryakov.spb.ru> > Content-Type: text/plain; charset=utf-8 > > > JMH have very handy @Param annotation, which allows to easily arrange > multiple benchmarks across multi-dimension parameters space. But > sometimes SOME combination of parameters doesn't have meaning for > benchmark, or simply excessive. > > For example, if here is two numeric parameters, it could be the case, > when they are interchangeable, and it is possible to get full picture by > exploring only half of square (over or under main diagonal). > > Is here any way to register "parameter space predicate", which allows > to skip some parameters combinations? > > IMHO, it could be useful. > > -- > // Black Lion AKA Lev Serebryakov > > > > ------------------------------ > > Message: 4 > Date: Tue, 4 Oct 2016 01:00:18 +0300 > From: Lev Serebryakov > To: Aleksey Shipilev , JMH Dev > > Subject: Re: Custom and not-constant "Operations per invocation" (Re: > JMH 1.15) > Message-ID: <106391707.20161004005959 at serebryakov.spb.ru> > Content-Type: text/plain; charset=us-ascii > > Hello Lev, > > Monday, October 3, 2016, 8:12:34 PM, you wrote: > > > Looks like "@AuxCounters" is solution, am I right? I only need to count > > real number of basic operations, and I'm done! > Yep, it works! Only one thing makes me sad panda: all these counters show > up in CSV file as separate benchmarks, not as additional data (columns) > to their > respective parent benchmarks :( > > > -- > Best regards, > Lev mailto:lev at serebryakov.spb.ru > > ------------------------------ > > Message: 5 > Date: Tue, 4 Oct 2016 12:44:38 +0200 > From: Aleksey Shipilev > To: Lev Serebryakov , JMH Dev > > Subject: Re: Is it possible to filter parameters space? > Message-ID: <8e918f60-1056-1635-38ba-ddd7f01a97e2 at redhat.com> > Content-Type: text/plain; charset=utf-8 > > On 10/03/2016 08:07 PM, Lev Serebryakov wrote: > > JMH have very handy @Param annotation, which allows to easily arrange > > multiple benchmarks across multi-dimension parameters space. But > > sometimes SOME combination of parameters doesn't have meaning for > > benchmark, or simply excessive. > > It is not possible now with annotations, but you can always make the > Java API launcher and traverse the parameter space in any way you want. > Extending JMH with more control creates another DSL over annotations, > and it could be easier to set up an advanced experiment in plain Java to > begin with. This is what Java API is for. > > See also: > https://bugs.openjdk.java.net/browse/CODETOOLS-7901296 > > -Aleksey > > > > ------------------------------ > > Message: 6 > Date: Tue, 4 Oct 2016 12:48:21 +0200 > From: Aleksey Shipilev > To: Lev Serebryakov , JMH Dev > > Subject: Re: Custom and not-constant "Operations per invocation" (Re: > JMH 1.15) > Message-ID: > Content-Type: text/plain; charset=utf-8 > > On 10/03/2016 07:12 PM, Lev Serebryakov wrote: > > On 30.09.2016 22:06, Aleksey Shipilev wrote: > > > >> *) @AuxCounters improvements. Note this is still a very much > >> experimental API. That said, it makes sense to improve it a little more: > >> now it handles more field/method types, has new counting mode "events" > >> that does not normalize to time, and does not require reset between > >> iterations. This API change is backwards-compatible. See the RFE: > >> https://bugs.openjdk.java.net/browse/CODETOOLS-7901810 > >> ...and new Javadoc: > > Looks like, it is way to implement custom "operations per invocation" > > metric, am I right? > > No, it helps with auxiliary counters that need to count only the > specific paths in the workload. These counters are not supposed to be as > accurate as primary metric you get from the workload, and may only serve > as the additional bits of data, together with the primary results. > > > Problem I have at hands looks like this: I benchmark implementation, > > which could do batching of low-level operations. To make benchmark more > > "realistic" I choose size of batch on each invocation at random (really, > > from pre-filled big array of random numbers, as random generators, even > > bad ones, are slow). So, each benchmark invocation could be counted as > > different number of operations, from 1 to, say, 16. and default JMH's > > "average time per invocation" is not very meaningful in this situation, > > as I need to know time per basic operation. > > You have to use Java API for this, and override > .operationsPerInvocation(...) to match your .params(...) if you want to > normalize the operations like that. > > > Looks like "@AuxCounters" is solution, am I right? I only need to count > > real number of basic operations, and I'm done! > > No, this is not a reliable solution. As said in both the changelog and > the Javadoc, it is a very experimental API. The reliable solution > measures the performance with primary metrics, and adjusts it accordingly. > > Thanks, > -Aleksey > > > > ------------------------------ > > Message: 7 > Date: Tue, 4 Oct 2016 14:24:52 +0300 > From: Lev Serebryakov > To: Aleksey Shipilev , JMH Dev > > Subject: Re: Is it possible to filter parameters space? > Message-ID: > Content-Type: text/plain; charset=utf-8 > > On 04.10.2016 13:44, Aleksey Shipilev wrote: > > > It is not possible now with annotations, but you can always make the > > Java API launcher and traverse the parameter space in any way you want. > > Extending JMH with more control creates another DSL over annotations, > > and it could be easier to set up an advanced experiment in plain Java to > > begin with. This is what Java API is for. > I thought about something like > > @StateFilter > boolean doWeNeedTestThisStaty() { > ... > } > > on state object. Not pure-annotation solution. > > > https://bugs.openjdk.java.net/browse/CODETOOLS-7901296 > Yep, something like this! :) > > -- > // Black Lion AKA Lev Serebryakov > > > > ------------------------------ > > Message: 8 > Date: Tue, 4 Oct 2016 14:57:57 +0300 > From: Lev Serebryakov > To: Aleksey Shipilev , JMH Dev > > Subject: Re: Custom and not-constant "Operations per invocation" (Re: > JMH 1.15) > Message-ID: <3d30c399-b4e6-ae54-d900-994768de53fa at serebryakov.spb.ru> > Content-Type: text/plain; charset=utf-8 > > On 04.10.2016 13:48, Aleksey Shipilev wrote: > > > You have to use Java API for this, and override > > .operationsPerInvocation(...) to match your .params(...) if you want to > > normalize the operations like that. > Override on what class? I've read JHM sources, and looks like runner > calls getOperationsPerInvocation() from options builder only once per > benchmark (but I may be wrong, of course). And in my case, it is > different on EACH call of @Bencmark'ed method. And only this method > itself know how much work was done at this invocation. I don't see how > building options by hands helps in such situation. > > And my situation is more complicated than that: it is non-symmetrical > two-thread (consumer/producer) benchmark, where consumer and producer > could use very different (random) values of OPI (and spent time, too!), > and this difference could be 2 orders of magnutude. As result, "primary > metrics" is completely meaningless now :( > > But, I should say, that @AuxCounters gives very stable "good-looking" > results with very low stddev between runs. Maybe, they are not > ns-precise (co, not suitable to estimating low-level characteristics > like CPI and such), but looks suitable for comparing different > implementations of same API. > > -- > // Black Lion AKA Lev Serebryakov > > > > ------------------------------ > > Message: 9 > Date: Wed, 05 Oct 2016 17:21:44 +0000 > From: ashipile at redhat.com > To: jmh-dev at openjdk.java.net > Subject: hg: code-tools/jmh: 7901813: @AuxCounters method constraints > conflict with @State ones > Message-ID: <201610051721.u95HLiCr029715 at aojmv0008.oracle.com> > > Changeset: 7a25c71b43bf > Author: shade > Date: 2016-10-05 19:21 +0200 > URL: http://hg.openjdk.java.net/code-tools/jmh/rev/7a25c71b43bf > > 7901813: @AuxCounters method constraints conflict with @State ones > Summary: Relax the requirement for "void" return type, implicitly allowing > @Setup/@TearDown-s. > > + jmh-core-ct/src/test/java/org/openjdk/jmh/ct/other/auxcounters/ > HelperConflictTest.java > ! jmh-core-ct/src/test/java/org/openjdk/jmh/ct/other/ > auxcounters/counterTypes/publicMethods/VoidTest.java > ! jmh-core/src/main/java/org/openjdk/jmh/annotations/AuxCounters.java > ! jmh-core/src/main/java/org/openjdk/jmh/generators/core/ > StateObjectHandler.java > > > > End of jmh-dev Digest, Vol 42, Issue 2 > ************************************** > From zolyfarkas at yahoo.com Fri Oct 7 01:04:05 2016 From: zolyfarkas at yahoo.com (Zoltan Farkas) Date: Thu, 6 Oct 2016 21:04:05 -0400 Subject: Question about exception I am seeing after upgrading to jmh 1.15 Message-ID: <9E05DB90-9AD8-4470-A214-1021FB17C810@yahoo.com> java.lang.Exception: java.lang.IllegalStateException: Cannot get another thread params at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:489) at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:448) at java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1424) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.concurrent.ForkJoinTask$AdaptedCallable.run(ForkJoinTask.java:1434) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) Caused by: java.lang.IllegalStateException: Cannot get another thread params at org.openjdk.jmh.runner.BenchmarkHandler$1.initialValue(BenchmarkHandler.java:85) at org.openjdk.jmh.runner.BenchmarkHandler$1.initialValue(BenchmarkHandler.java:78) at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:180) at java.lang.ThreadLocal.get(ThreadLocal.java:170) at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:463) ... 13 more I am using JDK 1.8_u102 + JMH 1.15?+ -Djmh.executor=FJP and pretty much get the exception for anything I am running. let me know ?Z From shade at redhat.com Fri Oct 7 10:24:42 2016 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 7 Oct 2016 12:24:42 +0200 Subject: Question about exception I am seeing after upgrading to jmh 1.15 In-Reply-To: <9E05DB90-9AD8-4470-A214-1021FB17C810@yahoo.com> References: <9E05DB90-9AD8-4470-A214-1021FB17C810@yahoo.com> Message-ID: <7db8f86c-ed71-d11b-66f0-cd1f6346ef86@redhat.com> Hi Zoltan, On 10/07/2016 03:04 AM, Zoltan Farkas wrote: > java.lang.Exception: java.lang.IllegalStateException: Cannot get another thread params > at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:489) > at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:448) > at java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1424) > at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) > at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) > at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) > at java.util.concurrent.ForkJoinTask$AdaptedCallable.run(ForkJoinTask.java:1434) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402) > at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) > at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) > at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) > at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) > Caused by: java.lang.IllegalStateException: Cannot get another thread params > at org.openjdk.jmh.runner.BenchmarkHandler$1.initialValue(BenchmarkHandler.java:85) > at org.openjdk.jmh.runner.BenchmarkHandler$1.initialValue(BenchmarkHandler.java:78) > at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:180) > at java.lang.ThreadLocal.get(ThreadLocal.java:170) > at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:463) > ... 13 more Mhm. That looks like there are more worker threads than harness has allocated the thread-local data for. I think FJP is a factor here: it probably instantiates another set of threads on the next iteration, and fails when those threads are trying to get new thread data. AFAIU, FJP would trim the workers (and then instantiate more) when this threshold is reached: /** * Initial timeout value (in milliseconds) for the thread * triggering quiescence to park waiting for new work. On timeout, * the thread will instead try to shrink the number of workers. * The value should be large enough to avoid overly aggressive * shrinkage during most transient stalls (long GCs etc). */ private static final long IDLE_TIMEOUT_MS = 2000L; // 2sec This, BTW, tells me that using non-standard executors really mess up the @State(Thread) invariants: you could execute the @Setup for those state objects several times, as your threads come and go. Which is what previous JMH version used to do. I think the long-term strategy for us is to consider "seamless iterations", where we don't release the worker thread between iterations, and thus do not set ourselves up for this failure. Short-term, you can keep using previous JMH version if you can accept @State(Thread) bug. Thanks, -Aleksey From zolyfarkas at yahoo.com Fri Oct 7 12:24:50 2016 From: zolyfarkas at yahoo.com (Zoltan Farkas) Date: Fri, 7 Oct 2016 08:24:50 -0400 Subject: Question about exception I am seeing after upgrading to jmh 1.15 In-Reply-To: <7db8f86c-ed71-d11b-66f0-cd1f6346ef86@redhat.com> References: <9E05DB90-9AD8-4470-A214-1021FB17C810@yahoo.com> <7db8f86c-ed71-d11b-66f0-cd1f6346ef86@redhat.com> Message-ID: <4AB93617-0E2C-42A7-B1BA-904FF86DF197@yahoo.com> Thank you, Once I disabled the FJP, everything started working again. Also FJP?s LIFO scheduling makes it more likely that a thread will be idle and will get retired? From what I understand, I believe JMH should probably take control of the thread lifecycle and not rely on executors. In any case I should have probably been using the JMH defaults? thank you for your help... ?Z > On Oct 7, 2016, at 6:24 AM, Aleksey Shipilev wrote: > > Hi Zoltan, > > On 10/07/2016 03:04 AM, Zoltan Farkas wrote: >> java.lang.Exception: java.lang.IllegalStateException: Cannot get another thread params >> at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:489) >> at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:448) >> at java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1424) >> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) >> at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) >> at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) >> at java.util.concurrent.ForkJoinTask$AdaptedCallable.run(ForkJoinTask.java:1434) >> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402) >> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) >> at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) >> at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) >> at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) >> Caused by: java.lang.IllegalStateException: Cannot get another thread params >> at org.openjdk.jmh.runner.BenchmarkHandler$1.initialValue(BenchmarkHandler.java:85) >> at org.openjdk.jmh.runner.BenchmarkHandler$1.initialValue(BenchmarkHandler.java:78) >> at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:180) >> at java.lang.ThreadLocal.get(ThreadLocal.java:170) >> at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:463) >> ... 13 more > > Mhm. > > That looks like there are more worker threads than harness has allocated > the thread-local data for. I think FJP is a factor here: it probably > instantiates another set of threads on the next iteration, and fails > when those threads are trying to get new thread data. > > AFAIU, FJP would trim the workers (and then instantiate more) when this > threshold is reached: > > /** > * Initial timeout value (in milliseconds) for the thread > * triggering quiescence to park waiting for new work. On timeout, > * the thread will instead try to shrink the number of workers. > * The value should be large enough to avoid overly aggressive > * shrinkage during most transient stalls (long GCs etc). > */ > private static final long IDLE_TIMEOUT_MS = 2000L; // 2sec > > This, BTW, tells me that using non-standard executors really mess up the > @State(Thread) invariants: you could execute the @Setup for those state > objects several times, as your threads come and go. Which is what > previous JMH version used to do. > > I think the long-term strategy for us is to consider "seamless > iterations", where we don't release the worker thread between > iterations, and thus do not set ourselves up for this failure. > Short-term, you can keep using previous JMH version if you can accept > @State(Thread) bug. > > Thanks, > -Aleksey From shade at redhat.com Fri Oct 7 12:58:20 2016 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 7 Oct 2016 14:58:20 +0200 Subject: Question about exception I am seeing after upgrading to jmh 1.15 In-Reply-To: <4AB93617-0E2C-42A7-B1BA-904FF86DF197@yahoo.com> References: <9E05DB90-9AD8-4470-A214-1021FB17C810@yahoo.com> <7db8f86c-ed71-d11b-66f0-cd1f6346ef86@redhat.com> <4AB93617-0E2C-42A7-B1BA-904FF86DF197@yahoo.com> Message-ID: On 10/07/2016 02:24 PM, Zoltan Farkas wrote: > From what I understand, I believe JMH should probably take control of > the thread lifecycle and not rely on executors. Yeah, well... there are cases where you *want* to make sure JMH threads are coming from a given executor. If your case is not one of them, then you will not suffer from JMH using whatever executor it thinks is okay. Thanks, -Aleksey From wizardedit at gentoo.org Sat Oct 15 03:29:28 2016 From: wizardedit at gentoo.org (Austin English) Date: Fri, 14 Oct 2016 22:29:28 -0500 Subject: jmh-core bug: fails to build with anything newer than jopt-4.6 Message-ID: <71b1f109-9e2f-376a-b50f-8a3b0ef9faac@gentoo.org> I'm attempting to package jmh for gentoo, but it's failing: [ERROR] /home/austin/src/jmh/jmh-core/src/main/java/org/openjdk/jmh/runner/options/IntegerValueConverter.java:[70,42] incompatible types: java.lang.Class cannot be converted to java.lang.Class I tracked this down to jopt. Gentoo has 4.8/4.9, forcing 4.6 worked. This behavior is broken in newer jopt, including the current 5.0.3. I tried to file a bug at https://bugs.openjdk.java.net/secure/Dashboard.jspa, but seems you must have contributed a patch to be able to file a bug, so, that's not available to me. -- -Austin GPG: 00B3 2957 B94B F3E1 From shade at redhat.com Mon Oct 17 11:00:12 2016 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 17 Oct 2016 13:00:12 +0200 Subject: jmh-core bug: fails to build with anything newer than jopt-4.6 In-Reply-To: <71b1f109-9e2f-376a-b50f-8a3b0ef9faac@gentoo.org> References: <71b1f109-9e2f-376a-b50f-8a3b0ef9faac@gentoo.org> Message-ID: <08050837-54b1-1b4a-4c9a-a952a2880c4b@redhat.com> Hi Austin, On 10/15/2016 05:29 AM, Austin English wrote: > I'm attempting to package jmh for gentoo, but it's failing: > > [ERROR] > /home/austin/src/jmh/jmh-core/src/main/java/org/openjdk/jmh/runner/options/IntegerValueConverter.java:[70,42] > incompatible types: java.lang.Class java.lang.Integer> cannot be converted to java.lang.Class > > I tracked this down to jopt. Gentoo has 4.8/4.9, forcing 4.6 worked. > This behavior is broken in newer jopt, including the current 5.0.3. Yes, JMH build is using 4.6. There is no guarantee it will work with the updated dependency, and I don't see the value in trying to do so. As much as I appreciate Gentoo way, trying to force the dependency version outside of what the actual project mentions in its build files is setting up Gentoo users for trouble. Thanks, -Aleksey From wizardedit at gentoo.org Mon Oct 17 16:43:56 2016 From: wizardedit at gentoo.org (Austin English) Date: Mon, 17 Oct 2016 11:43:56 -0500 Subject: jmh-core bug: fails to build with anything newer than jopt-4.6 In-Reply-To: <08050837-54b1-1b4a-4c9a-a952a2880c4b@redhat.com> References: <71b1f109-9e2f-376a-b50f-8a3b0ef9faac@gentoo.org> <08050837-54b1-1b4a-4c9a-a952a2880c4b@redhat.com> Message-ID: <3f3bbe2b-d257-f310-908a-e2ee40d185b3@gentoo.org> On 10/17/2016 06:00 AM, Aleksey Shipilev wrote: > Hi Austin, > > On 10/15/2016 05:29 AM, Austin English wrote: >> I'm attempting to package jmh for gentoo, but it's failing: >> >> [ERROR] >> /home/austin/src/jmh/jmh-core/src/main/java/org/openjdk/jmh/runner/options/IntegerValueConverter.java:[70,42] >> incompatible types: java.lang.Class> java.lang.Integer> cannot be converted to java.lang.Class >> >> I tracked this down to jopt. Gentoo has 4.8/4.9, forcing 4.6 worked. >> This behavior is broken in newer jopt, including the current 5.0.3. > > Yes, JMH build is using 4.6. There is no guarantee it will work with the > updated dependency, and I don't see the value in trying to do so. > > As much as I appreciate Gentoo way, trying to force the dependency > version outside of what the actual project mentions in its build files > is setting up Gentoo users for trouble. I'm not, as it won't build. Gentoo no longer has 4.6 in the tree, I would have to bring back an older version for this to work (and it sounds like that is what I'll have to do.. My goal was to file a bug to request that jmh support newer jopt, as depending on an older version of source code is generally considered a poor practice, but the OpenJDK tracker seems to only be open to OpenJDK developers. Is there some other place bugs can be filed? From jwhiting at redhat.com Fri Oct 21 10:01:31 2016 From: jwhiting at redhat.com (Jeremy Whiting) Date: Fri, 21 Oct 2016 11:01:31 +0100 Subject: BulkWarmup sample flag name. Message-ID: <2bba558e-d3a5-95db-a42c-967cad3d6c89@redhat.com> Hi, When running BulkWarmup sample this error is thrown using version 1.15 (and hg latest) $ java -jar target/benchmarks.jar JMHSample_32 -f 1 -wi 5 -i 5 -bm BULK Exception in thread "main" java.lang.IllegalStateException: Unable to parse benchmark mode: "BULK" Known values are [Throughput/thrpt, AverageTime/avgt, SampleTime/sample, SingleShotTime/ss, All/all] at org.openjdk.jmh.annotations.Mode.deepValueOf(Mode.java:125) at org.openjdk.jmh.runner.options.CommandLineOptions.(CommandLineOptions.java:338) at org.openjdk.jmh.Main.main(Main.java:41) Changing the parameter flag from -bm to -wm fixed it. $ hg diff diff -r 7a25c71b43bf jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_32_BulkWarmup.java --- a/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_32_BulkWarmup.java Wed Oct 05 19:21:18 2016 +0200 +++ b/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_32_BulkWarmup.java Fri Oct 21 09:53:06 2016 +0100 @@ -129,7 +129,7 @@ * * a) Via the command line: * $ mvn clean install - * $ java -jar target/benchmarks.jar JMHSample_32 -f 1 -wi 5 -i 5 -bm BULK + * $ java -jar target/benchmarks.jar JMHSample_32 -f 1 -wi 5 -i 5 -wm BULK * (we requested a single fork, 5 warmup/measurement iterations, and bulk warmup mode) * * b) Via the Java API: Regards, Jeremy -- Jeremy Whiting Senior Software Engineer, Middleware Performance Team JBoss, by Red Hat ------------------------------------------------------------ Registered Address: RED HAT UK LIMITED, Peninsular House, 30-36 Monument Street, 4th floor, London. EC3R 8NB United Kingdom Registered in UK and Wales under Company Registration No. 3798903 Directors: Michael Cunningham (US), Michael ("Mike") O'Neill (Ireland) and Eric Shander (US) From ashipile at redhat.com Fri Oct 21 18:29:01 2016 From: ashipile at redhat.com (ashipile at redhat.com) Date: Fri, 21 Oct 2016 18:29:01 +0000 Subject: hg: code-tools/jmh: Minor typo in JMHSample_32_BulkWarmup.java. Message-ID: <201610211829.u9LIT1KV000129@aojmv0008.oracle.com> Changeset: d0821c7627fb Author: shade Date: 2016-10-21 20:28 +0200 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/d0821c7627fb Minor typo in JMHSample_32_BulkWarmup.java. Contributed-by: Jeremy Whiting ! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_32_BulkWarmup.java From shade at redhat.com Fri Oct 21 18:29:21 2016 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 21 Oct 2016 20:29:21 +0200 Subject: BulkWarmup sample flag name. In-Reply-To: <2bba558e-d3a5-95db-a42c-967cad3d6c89@redhat.com> References: <2bba558e-d3a5-95db-a42c-967cad3d6c89@redhat.com> Message-ID: <546c5f54-875c-1ad4-2644-437faad52531@redhat.com> Hi Jeremy, On 10/21/2016 12:01 PM, Jeremy Whiting wrote: > $ java -jar target/benchmarks.jar JMHSample_32 -f 1 -wi 5 -i 5 -bm BULK > Exception in thread "main" java.lang.IllegalStateException: Unable to > parse benchmark mode: "BULK" > Known values are [Throughput/thrpt, AverageTime/avgt, SampleTime/sample, > SingleShotTime/ss, All/all] > at org.openjdk.jmh.annotations.Mode.deepValueOf(Mode.java:125) > at > org.openjdk.jmh.runner.options.CommandLineOptions.(CommandLineOptions.java:338) > > at org.openjdk.jmh.Main.main(Main.java:41) > > > Changing the parameter flag from -bm to -wm fixed it. Oops, thanks. Pushed: http://hg.openjdk.java.net/code-tools/jmh/rev/d0821c7627fb Thanks, -Aleksey From bruceteckel at gmail.com Fri Oct 21 21:16:42 2016 From: bruceteckel at gmail.com (Bruce Eckel) Date: Fri, 21 Oct 2016 14:16:42 -0700 Subject: Comparing performance of JDK List Implementations Message-ID: Attached is my current version which compares List implementations. I plan to do this for the other basic collection types (Set, Map, Queue, Dequeue), but would like some feedback on this one. Thanks. -- Bruce Eckel www.MindviewInc.com Blog: BruceEckel.github.io www.WinterTechForum.com www.AtomicScala.com www.Reinventing-Business.com http://www.TrustOrganizations.com From kirk.pepperdine at gmail.com Sat Oct 22 14:27:48 2016 From: kirk.pepperdine at gmail.com (kirk.pepperdine at gmail.com) Date: Sat, 22 Oct 2016 15:27:48 +0100 Subject: Comparing performance of JDK List Implementations In-Reply-To: References: Message-ID: <92A46810-CDC1-452D-8245-DBC6FD1D8242@gmail.com> Hi Bruce, Sorry, don?t see the attachment. Regards, Kirk > On Oct 21, 2016, at 10:16 PM, Bruce Eckel wrote: > > Attached is my current version which compares List implementations. I plan > to do this for the other basic collection types (Set, Map, Queue, Dequeue), > but would like some feedback on this one. Thanks. > > > > -- Bruce Eckel > www.MindviewInc.com > Blog: BruceEckel.github.io > www.WinterTechForum.com > www.AtomicScala.com > www.Reinventing-Business.com > http://www.TrustOrganizations.com From bruceteckel at gmail.com Sat Oct 22 15:30:00 2016 From: bruceteckel at gmail.com (Bruce Eckel) Date: Sat, 22 Oct 2016 08:30:00 -0700 Subject: Comparing performance of JDK List Implementations In-Reply-To: <92A46810-CDC1-452D-8245-DBC6FD1D8242@gmail.com> References: <92A46810-CDC1-452D-8245-DBC6FD1D8242@gmail.com> Message-ID: Trying again -- I see it as attached in the original message; does this list filter out attachments? -- Bruce Eckel www.MindviewInc.com Blog: BruceEckel.github.io www.WinterTechForum.com www.AtomicScala.com www.Reinventing-Business.com http://www.TrustOrganizations.com On Sat, Oct 22, 2016 at 7:27 AM, kirk.pepperdine at gmail.com < kirk.pepperdine at gmail.com> wrote: > Hi Bruce, > > Sorry, don?t see the attachment. > > Regards, > Kirk > > > On Oct 21, 2016, at 10:16 PM, Bruce Eckel wrote: > > > > Attached is my current version which compares List implementations. I > plan > > to do this for the other basic collection types (Set, Map, Queue, > Dequeue), > > but would like some feedback on this one. Thanks. > > > > > > > > -- Bruce Eckel > > www.MindviewInc.com > > Blog: BruceEckel.github.io > > www.WinterTechForum.com > > www.AtomicScala.com > > www.Reinventing-Business.com > > http://www.TrustOrganizations.com > > From henri.tremblay at gmail.com Sat Oct 22 15:35:47 2016 From: henri.tremblay at gmail.com (Henri Tremblay) Date: Sat, 22 Oct 2016 11:35:47 -0400 Subject: Comparing performance of JDK List Implementations In-Reply-To: References: <92A46810-CDC1-452D-8245-DBC6FD1D8242@gmail.com> Message-ID: No. It worked this time. Lists.java On 22 October 2016 at 11:30, Bruce Eckel wrote: > Trying again -- I see it as attached in the original message; does this > list filter out attachments? > > > > > -- Bruce Eckel > www.MindviewInc.com > Blog: BruceEckel.github.io > www.WinterTechForum.com > www.AtomicScala.com > www.Reinventing-Business.com > http://www.TrustOrganizations.com > > > On Sat, Oct 22, 2016 at 7:27 AM, kirk.pepperdine at gmail.com < > kirk.pepperdine at gmail.com> wrote: > > > Hi Bruce, > > > > Sorry, don?t see the attachment. > > > > Regards, > > Kirk > > > > > On Oct 21, 2016, at 10:16 PM, Bruce Eckel > wrote: > > > > > > Attached is my current version which compares List implementations. I > > plan > > > to do this for the other basic collection types (Set, Map, Queue, > > Dequeue), > > > but would like some feedback on this one. Thanks. > > > > > > > > > > > > -- Bruce Eckel > > > www.MindviewInc.com > > > Blog: BruceEckel.github.io > > > www.WinterTechForum.com > > > www.AtomicScala.com > > > www.Reinventing-Business.com > > > http://www.TrustOrganizations.com > > > > > From henri.tremblay at gmail.com Sat Oct 22 15:40:48 2016 From: henri.tremblay at gmail.com (Henri Tremblay) Date: Sat, 22 Oct 2016 11:40:48 -0400 Subject: Comparing performance of JDK List Implementations In-Reply-To: References: Message-ID: Hi, Quick question: Why are you using a BlackHole for get and returning a value for all the others? On 21 October 2016 at 17:16, Bruce Eckel wrote: > Attached is my current version which compares List implementations. I plan > to do this for the other basic collection types (Set, Map, Queue, Dequeue), > but would like some feedback on this one. Thanks. > > > > -- Bruce Eckel > www.MindviewInc.com > Blog: BruceEckel.github.io > www.WinterTechForum.com > www.AtomicScala.com > www.Reinventing-Business.com > http://www.TrustOrganizations.com > From bruceteckel at gmail.com Sat Oct 22 16:06:57 2016 From: bruceteckel at gmail.com (Bruce Eckel) Date: Sat, 22 Oct 2016 09:06:57 -0700 Subject: Comparing performance of JDK List Implementations In-Reply-To: References: Message-ID: I thought I was following the pattern I found on http://hg.openjdk.java.net/code-tools/jmh/file/d0821c7627fb/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_26_BatchSize.java with "measureRight()" -- Bruce Eckel www.MindviewInc.com Blog: BruceEckel.github.io www.WinterTechForum.com www.AtomicScala.com www.Reinventing-Business.com http://www.TrustOrganizations.com On Sat, Oct 22, 2016 at 8:40 AM, Henri Tremblay wrote: > Hi, > > Quick question: Why are you using a BlackHole for get and returning a > value for all the others? > > > On 21 October 2016 at 17:16, Bruce Eckel wrote: > >> Attached is my current version which compares List implementations. I plan >> to do this for the other basic collection types (Set, Map, Queue, >> Dequeue), >> but would like some feedback on this one. Thanks. >> >> >> >> -- Bruce Eckel >> www.MindviewInc.com >> Blog: BruceEckel.github.io >> www.WinterTechForum.com >> www.AtomicScala.com >> www.Reinventing-Business.com >> http://www.TrustOrganizations.com >> > > From bruceteckel at gmail.com Sat Oct 22 18:59:13 2016 From: bruceteckel at gmail.com (Bruce Eckel) Date: Sat, 22 Oct 2016 11:59:13 -0700 Subject: How do I shut everything down? Message-ID: If I call System.exit(), it prints a message about early JVM shutdown but then keeps going. Is there a call I can make to terminate everything? I'd like to be able to report a runtime error and then quit so it doesn't scroll off the screen. Thanks! -- Bruce Eckel www.MindviewInc.com Blog: BruceEckel.github.io www.WinterTechForum.com www.AtomicScala.com www.Reinventing-Business.com http://www.TrustOrganizations.com From bruceteckel at gmail.com Sat Oct 22 20:40:07 2016 From: bruceteckel at gmail.com (Bruce Eckel) Date: Sat, 22 Oct 2016 13:40:07 -0700 Subject: Review understandingcollections/jmh/ Message-ID: I've gotten through the first working draft of the JMH tests in the "Understanding Collections" chapter and would like feedback about how to improve it. To try it out, clone: https://github.com/BruceEckel/OnJava8-Examples.git Then run: gradlew understandingcollections:jmh The source code is in understandingcollections/jmh/ Everything seems to work; now I'm interested in tuning it appropriately (or fixing any misconceptions I might have). Thanks! -- Bruce Eckel www.MindviewInc.com Blog: BruceEckel.github.io www.WinterTechForum.com www.AtomicScala.com www.Reinventing-Business.com http://www.TrustOrganizations.com From henri.tremblay at gmail.com Mon Oct 24 02:18:59 2016 From: henri.tremblay at gmail.com (Henri Tremblay) Date: Sun, 23 Oct 2016 22:18:59 -0400 Subject: Comparing performance of JDK List Implementations In-Reply-To: References: Message-ID: Ok. What I meant by that is that returning a value from the benchmarked method does the same as putting it into a blackhole. So get could be public String get() { return list.get(list.size() / 2); } On 22 October 2016 at 12:06, Bruce Eckel wrote: > I thought I was following the pattern I found on > http://hg.openjdk.java.net/code-tools/jmh/file/d0821c7627fb/ > jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_ > 26_BatchSize.java > with "measureRight()" > > > -- Bruce Eckel > www.MindviewInc.com > Blog: BruceEckel.github.io > www.WinterTechForum.com > www.AtomicScala.com > www.Reinventing-Business.com > http://www.TrustOrganizations.com > > > On Sat, Oct 22, 2016 at 8:40 AM, Henri Tremblay > wrote: > >> Hi, >> >> Quick question: Why are you using a BlackHole for get and returning a >> value for all the others? >> >> >> On 21 October 2016 at 17:16, Bruce Eckel wrote: >> >>> Attached is my current version which compares List implementations. I >>> plan >>> to do this for the other basic collection types (Set, Map, Queue, >>> Dequeue), >>> but would like some feedback on this one. Thanks. >>> >>> >>> >>> -- Bruce Eckel >>> www.MindviewInc.com >>> Blog: BruceEckel.github.io >>> www.WinterTechForum.com >>> www.AtomicScala.com >>> www.Reinventing-Business.com >>> http://www.TrustOrganizations.com >>> >> >> > From bruceteckel at gmail.com Mon Oct 24 05:25:12 2016 From: bruceteckel at gmail.com (Bruce Eckel) Date: Sun, 23 Oct 2016 22:25:12 -0700 Subject: Comparing performance of JDK List Implementations In-Reply-To: References: Message-ID: Well that's certainly cleaner so I prefer it. Thanks. -- Bruce Eckel www.MindviewInc.com Blog: BruceEckel.github.io www.WinterTechForum.com www.AtomicScala.com www.Reinventing-Business.com http://www.TrustOrganizations.com On Sun, Oct 23, 2016 at 7:18 PM, Henri Tremblay wrote: > Ok. What I meant by that is that returning a value from the benchmarked > method does the same as putting it into a blackhole. So get could be > > public String get() { > return list.get(list.size() / 2); > } > > > On 22 October 2016 at 12:06, Bruce Eckel wrote: > >> I thought I was following the pattern I found on >> http://hg.openjdk.java.net/code-tools/jmh/file/d0821c7627fb/ >> jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_ >> 26_BatchSize.java >> with "measureRight()" >> >> >> -- Bruce Eckel >> www.MindviewInc.com >> Blog: BruceEckel.github.io >> www.WinterTechForum.com >> www.AtomicScala.com >> www.Reinventing-Business.com >> http://www.TrustOrganizations.com >> >> >> On Sat, Oct 22, 2016 at 8:40 AM, Henri Tremblay > > wrote: >> >>> Hi, >>> >>> Quick question: Why are you using a BlackHole for get and returning a >>> value for all the others? >>> >>> >>> On 21 October 2016 at 17:16, Bruce Eckel wrote: >>> >>>> Attached is my current version which compares List implementations. I >>>> plan >>>> to do this for the other basic collection types (Set, Map, Queue, >>>> Dequeue), >>>> but would like some feedback on this one. Thanks. >>>> >>>> >>>> >>>> -- Bruce Eckel >>>> www.MindviewInc.com >>>> Blog: BruceEckel.github.io >>>> www.WinterTechForum.com >>>> www.AtomicScala.com >>>> www.Reinventing-Business.com >>>> http://www.TrustOrganizations.com >>>> >>> >>> >> > From kirk.pepperdine at gmail.com Mon Oct 24 06:41:15 2016 From: kirk.pepperdine at gmail.com (kirk.pepperdine at gmail.com) Date: Mon, 24 Oct 2016 08:41:15 +0200 Subject: Comparing performance of JDK List Implementations In-Reply-To: References: <92A46810-CDC1-452D-8245-DBC6FD1D8242@gmail.com> Message-ID: Hi Bruce, It must as I still don?t see any attachments. Maybe you have the code in GitHub? ? Kirk > On Oct 22, 2016, at 4:30 PM, Bruce Eckel wrote: > > Trying again -- I see it as attached in the original message; does this > list filter out attachments? > > > > > -- Bruce Eckel > www.MindviewInc.com > Blog: BruceEckel.github.io > www.WinterTechForum.com > www.AtomicScala.com > www.Reinventing-Business.com > http://www.TrustOrganizations.com > > > On Sat, Oct 22, 2016 at 7:27 AM, kirk.pepperdine at gmail.com < > kirk.pepperdine at gmail.com> wrote: > >> Hi Bruce, >> >> Sorry, don?t see the attachment. >> >> Regards, >> Kirk >> >>> On Oct 21, 2016, at 10:16 PM, Bruce Eckel wrote: >>> >>> Attached is my current version which compares List implementations. I >> plan >>> to do this for the other basic collection types (Set, Map, Queue, >> Dequeue), >>> but would like some feedback on this one. Thanks. >>> >>> >>> >>> -- Bruce Eckel >>> www.MindviewInc.com >>> Blog: BruceEckel.github.io >>> www.WinterTechForum.com >>> www.AtomicScala.com >>> www.Reinventing-Business.com >>> http://www.TrustOrganizations.com >> >> From shade at redhat.com Mon Oct 24 10:23:21 2016 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 24 Oct 2016 12:23:21 +0200 Subject: How do I shut everything down? In-Reply-To: References: Message-ID: <5497f7aa-15b1-ddc6-5375-943fbc131f71@redhat.com> On 10/22/2016 08:59 PM, Bruce Eckel wrote: > If I call System.exit(), it prints a message about early JVM shutdown but > then keeps going. Is there a call I can make to terminate everything? I'd > like to be able to report a runtime error and then quit so it doesn't > scroll off the screen. Thanks! The default JMH policy is to handle the test failure gracefully, and move on: this helps a lot when running large suites of tests. This can be overriden, you can request harness to fail on benchmark error with -foe ("fail on error" cmdline switch) or the similar OptionsBuilder method. Thanks, -Aleksey P.S. Don't ever use System.exit() in the code driven by any framework. In JMH, you can throw the exception out of @Benchmark/@Setup/@TearDown to terminate the workload. From mr.chrisvest at gmail.com Mon Oct 24 10:25:33 2016 From: mr.chrisvest at gmail.com (Chris Vest) Date: Mon, 24 Oct 2016 12:25:33 +0200 Subject: Comparing performance of JDK List Implementations In-Reply-To: References: <92A46810-CDC1-452D-8245-DBC6FD1D8242@gmail.com> Message-ID: The mailing list is configured to scrub attachments. I suggest using a gist or pastebin instead. Cheers, Chris > On 24 Oct 2016, at 08.41, kirk.pepperdine at gmail.com wrote: > > Hi Bruce, > > It must as I still don?t see any attachments. Maybe you have the code in GitHub? > > ? Kirk > >> On Oct 22, 2016, at 4:30 PM, Bruce Eckel wrote: >> >> Trying again -- I see it as attached in the original message; does this >> list filter out attachments? >> >> >> >> >> -- Bruce Eckel >> www.MindviewInc.com >> Blog: BruceEckel.github.io >> www.WinterTechForum.com >> www.AtomicScala.com >> www.Reinventing-Business.com >> http://www.TrustOrganizations.com >> >> >> On Sat, Oct 22, 2016 at 7:27 AM, kirk.pepperdine at gmail.com < >> kirk.pepperdine at gmail.com> wrote: >> >>> Hi Bruce, >>> >>> Sorry, don?t see the attachment. >>> >>> Regards, >>> Kirk >>> >>>> On Oct 21, 2016, at 10:16 PM, Bruce Eckel wrote: >>>> >>>> Attached is my current version which compares List implementations. I >>> plan >>>> to do this for the other basic collection types (Set, Map, Queue, >>> Dequeue), >>>> but would like some feedback on this one. Thanks. >>>> >>>> >>>> >>>> -- Bruce Eckel >>>> www.MindviewInc.com >>>> Blog: BruceEckel.github.io >>>> www.WinterTechForum.com >>>> www.AtomicScala.com >>>> www.Reinventing-Business.com >>>> http://www.TrustOrganizations.com >>> >>> > From shade at redhat.com Mon Oct 24 10:30:50 2016 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 24 Oct 2016 12:30:50 +0200 Subject: Comparing performance of JDK List Implementations In-Reply-To: References: <92A46810-CDC1-452D-8245-DBC6FD1D8242@gmail.com> Message-ID: <0637b680-412c-235c-ff1e-b01a4db31820@redhat.com> On 10/24/2016 12:25 PM, Chris Vest wrote: > The mailing list is configured to scrub attachments. I suggest using > a gist or pastebin instead. Yes. mailman admins keep telling me that basic MIME types attachments are allowed here: multipart/mixed, multipart/alternative, text/plain, message/rfc822, multipart/signed, text/x-diff, text/x-patch. I guess Java MIME type (whatever that is), is not accepted. Thanks, -Aleksey From shade at redhat.com Mon Oct 24 10:46:50 2016 From: shade at redhat.com (Aleksey Shipilev) Date: Mon, 24 Oct 2016 12:46:50 +0200 Subject: Review understandingcollections/jmh/ In-Reply-To: References: Message-ID: <2f3d6933-df17-3882-c5ba-badf03b6ea93@redhat.com> Hi, On 10/22/2016 10:40 PM, Bruce Eckel wrote: > I've gotten through the first working draft of the JMH tests in the > "Understanding Collections" chapter and would like feedback about how to > improve it. > > To try it out, clone: > https://github.com/BruceEckel/OnJava8-Examples.git > > Then run: > gradlew understandingcollections:jmh > > The source code is in understandingcollections/jmh/ > > Everything seems to work; now I'm interested in tuning it appropriately (or > fixing any misconceptions I might have). I'm looking at tests at this path: https://github.com/BruceEckel/OnJava8-Examples/blob/master/understandingcollections/jmh/ Stylistic: *) Don't use System.exit, just throw the exception from the @Setup method. JMH will treat that as workload failure better than the abrupt VM termination. Ditto for println-ing in @Benchmark: just throw the exception out of @Benchmark. *) There is no particular reason to prefer explicit Blackhole rather than implicit return, if you have only a single result. E.g.: @Benchmark public void pollFirst(Blackhole bh) { bh.consume(deque.pollFirst()); } vs. @Benchmark public String pollFirst() { return deque.pollFirst(); } Okay, the problematic parts are: *) The benchmark like this is deceptive, because it will drain the deque on the first N invocations, and then you will measure the performance of polling from empty deque. This is probably not something you want. Deques, Lists, Queues, are affected. @Benchmark public String pollFirst() { return deque.pollFirst(); } *) Ditto for "add" tests: this workload would measure putting at the larger and larger queues, which in many cases would mean the performance would depend on iteration time... again, probably not something you want. Deques, Lists, Maps, Queues, are affected @Benchmark public Deque addLast() { deque.addLast("test"); return deque; } The benchmarks without steady state are difficult to do right. See e.g.: https://shipilev.net/blog/2014/nanotrusting-nanotime/#_steady_state_considerations. You might want to bite the bullet and repopulate the collection after you poll it dry + clear when it is full. Or, you might want to turn them into the steady state benchmarks by measuring add/poll pairs. *) For "size() / 2" index checks, are we assuming that the underlying implementations have simple size()? All four List implementations you have seem OKay, but this might not hold for any other List. *) It is better to precompute all non-measured parts in @Setup, to avoid this: @Benchmark public void Sets.contains(Blackhole bh) { String key = Integer.toString(size/2); // can do in @Setup bh.consume(set.contains(key)); } Thanks, -Aleksey From bruceteckel at gmail.com Mon Oct 24 15:36:25 2016 From: bruceteckel at gmail.com (Bruce Eckel) Date: Mon, 24 Oct 2016 08:36:25 -0700 Subject: Comparing performance of JDK List Implementations In-Reply-To: <0637b680-412c-235c-ff1e-b01a4db31820@redhat.com> References: <92A46810-CDC1-452D-8245-DBC6FD1D8242@gmail.com> <0637b680-412c-235c-ff1e-b01a4db31820@redhat.com> Message-ID: Kirk: clone: https://github.com/BruceEckel/OnJava8-Examples.git Then run: gradlew understandingcollections:jmh The source code is in understandingcollections/jmh/ -- Bruce Eckel www.MindviewInc.com Blog: BruceEckel.github.io www.WinterTechForum.com www.AtomicScala.com www.Reinventing-Business.com http://www.TrustOrganizations.com On Mon, Oct 24, 2016 at 3:30 AM, Aleksey Shipilev wrote: > On 10/24/2016 12:25 PM, Chris Vest wrote: > > The mailing list is configured to scrub attachments. I suggest using > > a gist or pastebin instead. > > Yes. mailman admins keep telling me that basic MIME types attachments > are allowed here: multipart/mixed, multipart/alternative, text/plain, > message/rfc822, multipart/signed, text/x-diff, text/x-patch. I guess > Java MIME type (whatever that is), is not accepted. > > Thanks, > -Aleksey > > From ashipile at redhat.com Tue Oct 25 15:55:36 2016 From: ashipile at redhat.com (ashipile at redhat.com) Date: Tue, 25 Oct 2016 15:55:36 +0000 Subject: hg: code-tools/jmh: Minor performance optimization in BenchmarkGenerator.buildAnnotatedSet(). Message-ID: <201610251555.u9PFta8L029915@aojmv0008.oracle.com> Changeset: cb9aa824b55a Author: shade Date: 2016-10-25 17:55 +0200 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/cb9aa824b55a Minor performance optimization in BenchmarkGenerator.buildAnnotatedSet(). ! jmh-core/src/main/java/org/openjdk/jmh/generators/core/BenchmarkGenerator.java From bruceteckel at gmail.com Tue Oct 25 16:02:07 2016 From: bruceteckel at gmail.com (Bruce Eckel) Date: Tue, 25 Oct 2016 09:02:07 -0700 Subject: How do I shut everything down? In-Reply-To: <5497f7aa-15b1-ddc6-5375-943fbc131f71@redhat.com> References: <5497f7aa-15b1-ddc6-5375-943fbc131f71@redhat.com> Message-ID: This is my gradle jmh section: jmh { jmhVersion = '1.15' duplicateClassesStrategy = 'warn' failOnError = true // See https://github.com/melix/jmh-gradle-plugin // for other options } However, that doesn't seem to have any effect: exceptions are reported as before but JMH just continues, as it did without the failOnError = true. -- Bruce Eckel www.MindviewInc.com Blog: BruceEckel.github.io www.WinterTechForum.com www.AtomicScala.com www.Reinventing-Business.com http://www.TrustOrganizations.com On Mon, Oct 24, 2016 at 3:23 AM, Aleksey Shipilev wrote: > On 10/22/2016 08:59 PM, Bruce Eckel wrote: > > If I call System.exit(), it prints a message about early JVM shutdown but > > then keeps going. Is there a call I can make to terminate everything? I'd > > like to be able to report a runtime error and then quit so it doesn't > > scroll off the screen. Thanks! > > The default JMH policy is to handle the test failure gracefully, and > move on: this helps a lot when running large suites of tests. This can > be overriden, you can request harness to fail on benchmark error with > -foe ("fail on error" cmdline switch) or the similar OptionsBuilder method. > > Thanks, > -Aleksey > > P.S. Don't ever use System.exit() in the code driven by any framework. > In JMH, you can throw the exception out of @Benchmark/@Setup/@TearDown > to terminate the workload. > > From shade at redhat.com Wed Oct 26 09:06:50 2016 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 26 Oct 2016 11:06:50 +0200 Subject: How do I shut everything down? In-Reply-To: References: <5497f7aa-15b1-ddc6-5375-943fbc131f71@redhat.com> Message-ID: Hi Bruce, So, our project page says this: "Make sure you tried these things before getting support: Archetypes provide the golden build configuration. Try to generate the clean JMH benchmark project and transplant the benchmark there. This is important to try when upgrading to the newer JMH versions, since the minute differences in the build configurations may attribute to the failures you are seeing." ...and you are using a 3rd party jmh-gradle-plugin. But if you copy e.g. Lists benchmark into the clean JMH archetype-bootstrapped project, then you will see how -foe works with either System.exit() or throwing the exceptions. Therefore, the fault is somewhere in jmh-gradle-plugin, you might want to ask plugin maintainers what's up. Thanks, -Aleksey On 10/25/2016 06:02 PM, Bruce Eckel wrote: > This is my gradle jmh section: > > jmh { > jmhVersion = '1.15' > duplicateClassesStrategy = 'warn' > failOnError = true > // See https://github.com/melix/jmh-gradle-plugin > // for other options > } > > > However, that doesn't seem to have any effect: exceptions are reported > as before but JMH just continues, as it did without the failOnError = true. > > > -- Bruce Eckel > www.MindviewInc.com > Blog: BruceEckel.github.io > www.WinterTechForum.com > www.AtomicScala.com > www.Reinventing-Business.com > http://www.TrustOrganizations.com > > > On Mon, Oct 24, 2016 at 3:23 AM, Aleksey Shipilev > wrote: > > On 10/22/2016 08:59 PM, Bruce Eckel wrote: > > If I call System.exit(), it prints a message about early JVM shutdown but > > then keeps going. Is there a call I can make to terminate everything? I'd > > like to be able to report a runtime error and then quit so it doesn't > > scroll off the screen. Thanks! > > The default JMH policy is to handle the test failure gracefully, and > move on: this helps a lot when running large suites of tests. This can > be overriden, you can request harness to fail on benchmark error with > -foe ("fail on error" cmdline switch) or the similar OptionsBuilder > method. > > Thanks, > -Aleksey > > P.S. Don't ever use System.exit() in the code driven by any framework. > In JMH, you can throw the exception out of @Benchmark/@Setup/@TearDown > to terminate the workload. > >