jmh-dev Digest, Vol 42, Issue 2
Joel Moberg
joel.moberg at gmail.com
Wed Oct 5 18:29:45 UTC 2016
help
On Wed, Oct 5, 2016 at 7:21 PM, <jmh-dev-request at openjdk.java.net> wrote:
> Send jmh-dev mailing list submissions to
> jmh-dev at openjdk.java.net
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://mail.openjdk.java.net/mailman/listinfo/jmh-dev
> or, via email, send a message with subject or body 'help' to
> jmh-dev-request at openjdk.java.net
>
> You can reach the person managing the list at
> jmh-dev-owner at openjdk.java.net
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of jmh-dev digest..."
>
>
> Today's Topics:
>
> 1. Re: AW: volatile boolean isDone in InfraControl (James Cheng)
> 2. Re: Custom and not-constant "Operations per invocation" (Re:
> JMH 1.15) (Lev Serebryakov)
> 3. Is it possible to filter parameters space? (Lev Serebryakov)
> 4. Re: Custom and not-constant "Operations per invocation" (Re:
> JMH 1.15) (Lev Serebryakov)
> 5. Re: Is it possible to filter parameters space? (Aleksey Shipilev)
> 6. Re: Custom and not-constant "Operations per invocation" (Re:
> JMH 1.15) (Aleksey Shipilev)
> 7. Re: Is it possible to filter parameters space? (Lev Serebryakov)
> 8. Re: Custom and not-constant "Operations per invocation" (Re:
> JMH 1.15) (Lev Serebryakov)
> 9. hg: code-tools/jmh: 7901813: @AuxCounters method constraints
> conflict with @State ones (ashipile at redhat.com)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 3 Oct 2016 10:18:39 -0700
> From: James Cheng <james.cheng at oracle.com>
> To: ecki at zusammenkunft.net, "jmh-dev at openjdk.java.net"
> <jmh-dev at openjdk.java.net>
> Subject: Re: AW: volatile boolean isDone in InfraControl
> Message-ID: <57F292EF.90302 at oracle.com>
> Content-Type: text/plain; charset=utf-8; format=flowed
>
> Hi Bernd,
>
> Many thanks for your quick suggestion and pointers.
>
> On 10/1/2016 9:12 AM, ecki at zusammenkunft.net wrote:
> > Hello James,
> >
> > I would think if you want to test unrolling and inlining effects inside
> loops it is best to have a loop inside your method under test: You dont
> have the volatile access
> > then inside the hot code and can better emulate the structure of the
> code. (dont forget to actually return/blackhole a result from your loop)
>
> I will try that and experiment with different loop counts as mentioned in
> JMHSample_11_Loops.java which Aleksey just pointed to me.
>
> Thanks,
> -James
>
>
> ------------------------------
>
> Message: 2
> Date: Mon, 3 Oct 2016 20:51:24 +0300
> From: Lev Serebryakov <lev at serebryakov.spb.ru>
> To: Aleksey Shipilev <shade at redhat.com>, JMH Dev
> <jmh-dev at openjdk.java.net>
> Subject: Re: Custom and not-constant "Operations per invocation" (Re:
> JMH 1.15)
> Message-ID: <795456ff-1396-776d-78ab-9773273a114e at serebryakov.spb.ru>
> Content-Type: text/plain; charset=utf-8
>
> On 03.10.2016 20:12, Lev Serebryakov wrote:
>
> >> *) @AuxCounters improvements. Note this is still a very much
> >> experimental API. That said, it makes sense to improve it a little more:
> >> now it handles more field/method types, has new counting mode "events"
> >> that does not normalize to time, and does not require reset between
> >> iterations. This API change is backwards-compatible. See the RFE:
> >> https://bugs.openjdk.java.net/browse/CODETOOLS-7901810
> >> ...and new Javadoc:
> > Looks like, it is way to implement custom "operations per invocation"
> > metric, am I right?
> >
> > Problem I have at hands looks like this: I benchmark implementation,
> > which could do batching of low-level operations. To make benchmark more
> > "realistic" I choose size of batch on each invocation at random (really,
> > from pre-filled big array of random numbers, as random generators, even
> > bad ones, are slow). So, each benchmark invocation could be counted as
> > different number of operations, from 1 to, say, 16. and default JMH's
> > "average time per invocation" is not very meaningful in this situation,
> > as I need to know time per basic operation.
> >
> > Looks like "@AuxCounters" is solution, am I right? I only need to count
> > real number of basic operations, and I'm done!
> It seems to work, but I don't understand how result is combined from
> different threads. Again, I have benchmark which contains of two threads
> in one group, doing different work. If I have same "counter" state class
> (Scope.Thread, of course) I get 19ns/count ("op"). If I have two
> different classes, I get 16ns/count at each thread. It looks strange and
> I could not do mental model what happens in case when each thread
> increments logically-the-same counter (in thread-local state). If all
> instances of this counter are summed up, it should be ~2 times faster
> than splitted counters (because combined counter is sum of splitted
> ones, and splitted counters are roughly the same). But it is slightly
> slower!
>
> --
> // Black Lion AKA Lev Serebryakov
>
>
>
> ------------------------------
>
> Message: 3
> Date: Mon, 3 Oct 2016 21:07:33 +0300
> From: Lev Serebryakov <lev at serebryakov.spb.ru>
> To: JMH Dev <jmh-dev at openjdk.java.net>
> Subject: Is it possible to filter parameters space?
> Message-ID: <100d865e-ef0d-890e-6170-a65b241bf7d0 at serebryakov.spb.ru>
> Content-Type: text/plain; charset=utf-8
>
>
> JMH have very handy @Param annotation, which allows to easily arrange
> multiple benchmarks across multi-dimension parameters space. But
> sometimes SOME combination of parameters doesn't have meaning for
> benchmark, or simply excessive.
>
> For example, if here is two numeric parameters, it could be the case,
> when they are interchangeable, and it is possible to get full picture by
> exploring only half of square (over or under main diagonal).
>
> Is here any way to register "parameter space predicate", which allows
> to skip some parameters combinations?
>
> IMHO, it could be useful.
>
> --
> // Black Lion AKA Lev Serebryakov
>
>
>
> ------------------------------
>
> Message: 4
> Date: Tue, 4 Oct 2016 01:00:18 +0300
> From: Lev Serebryakov <lev at serebryakov.spb.ru>
> To: Aleksey Shipilev <shade at redhat.com>, JMH Dev
> <jmh-dev at openjdk.java.net>
> Subject: Re: Custom and not-constant "Operations per invocation" (Re:
> JMH 1.15)
> Message-ID: <106391707.20161004005959 at serebryakov.spb.ru>
> Content-Type: text/plain; charset=us-ascii
>
> Hello Lev,
>
> Monday, October 3, 2016, 8:12:34 PM, you wrote:
>
> > Looks like "@AuxCounters" is solution, am I right? I only need to count
> > real number of basic operations, and I'm done!
> Yep, it works! Only one thing makes me sad panda: all these counters show
> up in CSV file as separate benchmarks, not as additional data (columns)
> to their
> respective parent benchmarks :(
>
>
> --
> Best regards,
> Lev mailto:lev at serebryakov.spb.ru
>
> ------------------------------
>
> Message: 5
> Date: Tue, 4 Oct 2016 12:44:38 +0200
> From: Aleksey Shipilev <shade at redhat.com>
> To: Lev Serebryakov <lev at serebryakov.spb.ru>, JMH Dev
> <jmh-dev at openjdk.java.net>
> Subject: Re: Is it possible to filter parameters space?
> Message-ID: <8e918f60-1056-1635-38ba-ddd7f01a97e2 at redhat.com>
> Content-Type: text/plain; charset=utf-8
>
> On 10/03/2016 08:07 PM, Lev Serebryakov wrote:
> > JMH have very handy @Param annotation, which allows to easily arrange
> > multiple benchmarks across multi-dimension parameters space. But
> > sometimes SOME combination of parameters doesn't have meaning for
> > benchmark, or simply excessive.
>
> It is not possible now with annotations, but you can always make the
> Java API launcher and traverse the parameter space in any way you want.
> Extending JMH with more control creates another DSL over annotations,
> and it could be easier to set up an advanced experiment in plain Java to
> begin with. This is what Java API is for.
>
> See also:
> https://bugs.openjdk.java.net/browse/CODETOOLS-7901296
>
> -Aleksey
>
>
>
> ------------------------------
>
> Message: 6
> Date: Tue, 4 Oct 2016 12:48:21 +0200
> From: Aleksey Shipilev <shade at redhat.com>
> To: Lev Serebryakov <lev at serebryakov.spb.ru>, JMH Dev
> <jmh-dev at openjdk.java.net>
> Subject: Re: Custom and not-constant "Operations per invocation" (Re:
> JMH 1.15)
> Message-ID: <e1552d04-e752-7b4f-4d63-62e4895d0dc8 at redhat.com>
> Content-Type: text/plain; charset=utf-8
>
> On 10/03/2016 07:12 PM, Lev Serebryakov wrote:
> > On 30.09.2016 22:06, Aleksey Shipilev wrote:
> >
> >> *) @AuxCounters improvements. Note this is still a very much
> >> experimental API. That said, it makes sense to improve it a little more:
> >> now it handles more field/method types, has new counting mode "events"
> >> that does not normalize to time, and does not require reset between
> >> iterations. This API change is backwards-compatible. See the RFE:
> >> https://bugs.openjdk.java.net/browse/CODETOOLS-7901810
> >> ...and new Javadoc:
> > Looks like, it is way to implement custom "operations per invocation"
> > metric, am I right?
>
> No, it helps with auxiliary counters that need to count only the
> specific paths in the workload. These counters are not supposed to be as
> accurate as primary metric you get from the workload, and may only serve
> as the additional bits of data, together with the primary results.
>
> > Problem I have at hands looks like this: I benchmark implementation,
> > which could do batching of low-level operations. To make benchmark more
> > "realistic" I choose size of batch on each invocation at random (really,
> > from pre-filled big array of random numbers, as random generators, even
> > bad ones, are slow). So, each benchmark invocation could be counted as
> > different number of operations, from 1 to, say, 16. and default JMH's
> > "average time per invocation" is not very meaningful in this situation,
> > as I need to know time per basic operation.
>
> You have to use Java API for this, and override
> .operationsPerInvocation(...) to match your .params(...) if you want to
> normalize the operations like that.
>
> > Looks like "@AuxCounters" is solution, am I right? I only need to count
> > real number of basic operations, and I'm done!
>
> No, this is not a reliable solution. As said in both the changelog and
> the Javadoc, it is a very experimental API. The reliable solution
> measures the performance with primary metrics, and adjusts it accordingly.
>
> Thanks,
> -Aleksey
>
>
>
> ------------------------------
>
> Message: 7
> Date: Tue, 4 Oct 2016 14:24:52 +0300
> From: Lev Serebryakov <lev at serebryakov.spb.ru>
> To: Aleksey Shipilev <shade at redhat.com>, JMH Dev
> <jmh-dev at openjdk.java.net>
> Subject: Re: Is it possible to filter parameters space?
> Message-ID: <a7d21b28-d50f-9985-f322-2c10ce193e44 at serebryakov.spb.ru>
> Content-Type: text/plain; charset=utf-8
>
> On 04.10.2016 13:44, Aleksey Shipilev wrote:
>
> > It is not possible now with annotations, but you can always make the
> > Java API launcher and traverse the parameter space in any way you want.
> > Extending JMH with more control creates another DSL over annotations,
> > and it could be easier to set up an advanced experiment in plain Java to
> > begin with. This is what Java API is for.
> I thought about something like
>
> @StateFilter
> boolean doWeNeedTestThisStaty() {
> ...
> }
>
> on state object. Not pure-annotation solution.
>
> > https://bugs.openjdk.java.net/browse/CODETOOLS-7901296
> Yep, something like this! :)
>
> --
> // Black Lion AKA Lev Serebryakov
>
>
>
> ------------------------------
>
> Message: 8
> Date: Tue, 4 Oct 2016 14:57:57 +0300
> From: Lev Serebryakov <lev at serebryakov.spb.ru>
> To: Aleksey Shipilev <shade at redhat.com>, JMH Dev
> <jmh-dev at openjdk.java.net>
> Subject: Re: Custom and not-constant "Operations per invocation" (Re:
> JMH 1.15)
> Message-ID: <3d30c399-b4e6-ae54-d900-994768de53fa at serebryakov.spb.ru>
> Content-Type: text/plain; charset=utf-8
>
> On 04.10.2016 13:48, Aleksey Shipilev wrote:
>
> > You have to use Java API for this, and override
> > .operationsPerInvocation(...) to match your .params(...) if you want to
> > normalize the operations like that.
> Override on what class? I've read JHM sources, and looks like runner
> calls getOperationsPerInvocation() from options builder only once per
> benchmark (but I may be wrong, of course). And in my case, it is
> different on EACH call of @Bencmark'ed method. And only this method
> itself know how much work was done at this invocation. I don't see how
> building options by hands helps in such situation.
>
> And my situation is more complicated than that: it is non-symmetrical
> two-thread (consumer/producer) benchmark, where consumer and producer
> could use very different (random) values of OPI (and spent time, too!),
> and this difference could be 2 orders of magnutude. As result, "primary
> metrics" is completely meaningless now :(
>
> But, I should say, that @AuxCounters gives very stable "good-looking"
> results with very low stddev between runs. Maybe, they are not
> ns-precise (co, not suitable to estimating low-level characteristics
> like CPI and such), but looks suitable for comparing different
> implementations of same API.
>
> --
> // Black Lion AKA Lev Serebryakov
>
>
>
> ------------------------------
>
> Message: 9
> Date: Wed, 05 Oct 2016 17:21:44 +0000
> From: ashipile at redhat.com
> To: jmh-dev at openjdk.java.net
> Subject: hg: code-tools/jmh: 7901813: @AuxCounters method constraints
> conflict with @State ones
> Message-ID: <201610051721.u95HLiCr029715 at aojmv0008.oracle.com>
>
> Changeset: 7a25c71b43bf
> Author: shade
> Date: 2016-10-05 19:21 +0200
> URL: http://hg.openjdk.java.net/code-tools/jmh/rev/7a25c71b43bf
>
> 7901813: @AuxCounters method constraints conflict with @State ones
> Summary: Relax the requirement for "void" return type, implicitly allowing
> @Setup/@TearDown-s.
>
> + jmh-core-ct/src/test/java/org/openjdk/jmh/ct/other/auxcounters/
> HelperConflictTest.java
> ! jmh-core-ct/src/test/java/org/openjdk/jmh/ct/other/
> auxcounters/counterTypes/publicMethods/VoidTest.java
> ! jmh-core/src/main/java/org/openjdk/jmh/annotations/AuxCounters.java
> ! jmh-core/src/main/java/org/openjdk/jmh/generators/core/
> StateObjectHandler.java
>
>
>
> End of jmh-dev Digest, Vol 42, Issue 2
> **************************************
>
More information about the jmh-dev
mailing list