From vyazelenko at yahoo.com  Mon Sep  1 14:48:35 2014
From: vyazelenko at yahoo.com (Dmitry Vyazelenko)
Date: Mon, 1 Sep 2014 07:48:35 -0700
Subject: Setup time is counted against iteration time in Mode.SingleShotTime
	(JMH-1.0)
Message-ID: <1409582915.16255.YahooMailNeo@web162205.mail.bf1.yahoo.com>

Hi,

I've discovered that when Mode.SingleShotTime is used the time in @Setup
is counted against iteration time. This results in benchmark being interrupted
if @Setup takes long time. 
What is also interesting is that iteration time for Mode.SingleShotTime is set 
to 10 minutes. I would expect that in this mode JMH will wait forever for iteration
to complete. Or at least such behavior should be documented somewhere. :)

Here is the benchmark to reproduce:
@Fork(1) 
@Warmup(iterations=10) 
@Measurement(iterations=10) 
@OutputTimeUnit(TimeUnit.NANOSECONDS) 
@State(Scope.Benchmark) 
@BenchmarkMode(Mode.SingleShotTime) 
public class SetupTimeCountedAgainstIterationTime_SingleShotTime { 
 @Setup(Level.Trial) 
    public void setUp() throws Exception { 
     System.out.println("Starting setup..."); 
        Thread.sleep(TimeUnit.MINUTES.toMillis(15)); 
     System.out.println("Setup finished!"); 
    } 
    @Benchmark 
    public void baseline() { 
    } 
}

When executed it fails with InterruptedException:
java -jar target\benchmarks.jar -foe true 
# VM invoker: C:\Program Files\Java\jre7\bin\java.exe 
# VM options: <none> 
# Warmup: 10 iterations, single-shot each 
# Measurement: 10 iterations, single-shot each 
# Threads: 1 thread 
# Benchmark mode: Single shot invocation time 
# Benchmark: org.sample.SetupTimeCountedAgainstIterationTime_SingleShotTime.baseline 
# Run progress: 0.00% complete, ETA 00:00:00 
# Fork: 1 of 1 
# Warmup Iteration   1: Starting setup... 
(*interrupt*) <failure> 
java.lang.InterruptedException: sleep interrupted 
        at java.lang.Thread.sleep(Native Method) 
        at org.sample.SetupTimeCountedAgainstIterationTime_SingleShotTime.setUp(SetupTimeCountedAgainstIterationTime_SingleShotTime.java:16) 
        at org.sample.generated.SetupTimeCountedAgainstIterationTime_SingleShotTime_baseline._jmh_tryInit_f_setuptimecountedagainstiterationtime_singleshottime0_G(SetupTimeCountedAgainstIterationTime_SingleShotTime_baseline.java:340) 
        at org.sample.generated.SetupTimeCountedAgainstIterationTime_SingleShotTime_baseline.baseline_SingleShotTime(SetupTimeCountedAgainstIterationTime_SingleShotTime_baseline.java:281) 
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) 
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) 
        at java.lang.reflect.Method.invoke(Unknown Source) 
        at org.openjdk.jmh.runner.LoopBenchmarkHandler$BenchmarkTask.call(LoopBenchmarkHandler.java:210) 
        at org.openjdk.jmh.runner.LoopBenchmarkHandler$BenchmarkTask.call(LoopBenchmarkHandler.java:192) 
        at java.util.concurrent.FutureTask.run(Unknown Source) 
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 
        at java.lang.Thread.run(Unknown Source) 

Expected: @Setup completes after 15 minutes and benchmark executes without exception.
 

Best regards,
Dmitry Vyazelenko

From aleksey.shipilev at oracle.com  Tue Sep  2 10:52:01 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Tue, 02 Sep 2014 14:52:01 +0400
Subject: Setup time is counted against iteration time in
	Mode.SingleShotTime (JMH-1.0)
In-Reply-To: <1409582915.16255.YahooMailNeo@web162205.mail.bf1.yahoo.com>
References: <1409582915.16255.YahooMailNeo@web162205.mail.bf1.yahoo.com>
Message-ID: <5405A151.6060609@oracle.com>

Hi,

On 09/01/2014 06:48 PM, Dmitry Vyazelenko wrote:
> I've discovered that when Mode.SingleShotTime is used the time in @Setup
> is counted against iteration time. This results in benchmark being interrupted
> if @Setup takes long time. 

Yes, there is the implementation reason for that: @Setup is executed by
the generated code, and so the harness control thread is oblivious of
the time spent in @Setup vs. @Benchmark.

Anyway, that is arguably not-the-issue, here is why. The timeout-ing is
done to recover from the misbehaved benchmarks, and the @Setup taking 10
minutes to run is arguably misbehaving. We can't afford waiting
indefinitely for broken benchmarks if we are running the large benchmark
suites. Now, when we are facing the misbehaving workload, it does not
seem all that necessary to accurately track the workload time?

> What is also interesting is that iteration time for Mode.SingleShotTime is set 
> to 10 minutes. I would expect that in this mode JMH will wait forever for iteration
> to complete. 

See above.

> Or at least such behavior should be documented somewhere. :)

And what would be the best place to document this? I am puzzled about
that, and since the issue is about the corner case, it does not seem to
belong in the generic documentation.

Thanks,
-Aleksey.



From aleksey.shipilev at oracle.com  Tue Sep  2 11:43:12 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Tue, 02 Sep 2014 11:43:12 +0000
Subject: hg: code-tools/jmh: 3 new changesets
Message-ID: <201409021143.s82BhDb9025817@aojmv0008>

Changeset: a85487efd84e
Author:    shade
Date:      2014-09-02 15:15 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/a85487efd84e

generators: generated code readability, omit the number suffix for some generated classes.

! jmh-core/src/main/java/org/openjdk/jmh/generators/core/Identifiers.java

Changeset: 239a7f170831
Author:    shade
Date:      2014-09-02 15:24 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/239a7f170831

generators: two minor indentation issues in generated code.

! jmh-core/src/main/java/org/openjdk/jmh/generators/core/BenchmarkGenerator.java

Changeset: 6a06014f7a38
Author:    shade
Date:      2014-09-02 15:39 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/6a06014f7a38

generators: save a few percents in generated code size by generating denser padding.

! jmh-core/src/main/java/org/openjdk/jmh/generators/core/BenchmarkGenerator.java
! jmh-core/src/main/java/org/openjdk/jmh/generators/core/StateObjectHandler.java


From aleksey.shipilev at oracle.com  Tue Sep  2 12:58:42 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Tue, 02 Sep 2014 12:58:42 +0000
Subject: hg: code-tools/jmh: generators: cache padding generators,
	saves a lot of benchmark generation time.
Message-ID: <201409021258.s82CwgOa007580@aojmv0008>

Changeset: e7f0bf479bcf
Author:    shade
Date:      2014-09-02 16:57 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/e7f0bf479bcf

generators: cache padding generators, saves a lot of benchmark generation time.

! jmh-core/src/main/java/org/openjdk/jmh/generators/core/BenchmarkGenerator.java
+ jmh-core/src/main/java/org/openjdk/jmh/generators/core/Paddings.java
! jmh-core/src/main/java/org/openjdk/jmh/generators/core/StateObjectHandler.java


From vyazelenko at yahoo.com  Tue Sep  2 13:18:12 2014
From: vyazelenko at yahoo.com (vyazelenko at yahoo.com)
Date: Tue, 2 Sep 2014 15:18:12 +0200
Subject: Setup time is counted against iteration time in
	Mode.SingleShotTime (JMH-1.0)
In-Reply-To: <5405A151.6060609@oracle.com>
References: <1409582915.16255.YahooMailNeo@web162205.mail.bf1.yahoo.com>
	<5405A151.6060609@oracle.com>
Message-ID: <3727E2AB-EAFC-4521-B10D-CC6280C64C86@yahoo.com>

Ok, that's interesting...

Sent from my iPhone

> On Sep 2, 2014, at 12:52, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:
> 
> Hi,
> 
>> On 09/01/2014 06:48 PM, Dmitry Vyazelenko wrote:
>> I've discovered that when Mode.SingleShotTime is used the time in @Setup
>> is counted against iteration time. This results in benchmark being interrupted
>> if @Setup takes long time.
> 
> Yes, there is the implementation reason for that: @Setup is executed by
> the generated code, and so the harness control thread is oblivious of
> the time spent in @Setup vs. @Benchmark.
I have a problem with that. It is neither intuitive nor expected that @Setup/@TearDown times are considered part of actual measurement. Especially if Level.Trial is used.

Consider the case of kilo-benchmark which inserts millions of entries into the database as part of @Setup(Level.Trial) and it takes 20 min to execute.
Now assume that benchmark methods are executing SELECTs against database (with different indices used). And they run 2-10 seconds.
With such setup and usage of SingleShotTime with 10 iterations reported execution times and reality will be quite different since very first iteration takes 1200 seconds longer than any other iteration.
> 
> Anyway, that is arguably not-the-issue, here is why. The timeout-ing is
> done to recover from the misbehaved benchmarks, and the @Setup taking 10
> minutes to run is arguably misbehaving. We can't afford waiting
> indefinitely for broken benchmarks if we are running the large benchmark
> suites. Now, when we are facing the misbehaving workload, it does not
> seem all that necessary to accurately track the workload time?
Well if Setup is not considered part of the workload then there would no problem to start with.

I think now with 1.0 released such change can be considered. Also waiting indefinitely for @Setup/@TearDown and maybe even benchmark proper should be considered.
> 
>> What is also interesting is that iteration time for Mode.SingleShotTime is set 
>> to 10 minutes. I would expect that in this mode JMH will wait forever for iteration
>> to complete.
> 
> See above.
> 
>> Or at least such behavior should be documented somewhere. :)
> 
> And what would be the best place to document this? I am puzzled about
> that, and since the issue is about the corner case, it does not seem to
> belong in the generic documentation.
Why not document it in "time" property of the @Warmup/@Measurement annotations? Saying something like:
"If time is undefined then depending on the BenchmarkMode JMH will:
  - wait at least 5 seconds...
  - wait for 10 minutes in case SingleShotTime...
To override defaults please put time value explicitly..."
> 
> Thanks,
> -Aleksey.
> 
> 
Best regards,
Dmitry Vyazelenko

From aleksey.shipilev at oracle.com  Tue Sep  2 13:32:51 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Tue, 02 Sep 2014 17:32:51 +0400
Subject: Setup time is counted against iteration time in
	Mode.SingleShotTime (JMH-1.0)
In-Reply-To: <3727E2AB-EAFC-4521-B10D-CC6280C64C86@yahoo.com>
References: <1409582915.16255.YahooMailNeo@web162205.mail.bf1.yahoo.com>
	<5405A151.6060609@oracle.com>
	<3727E2AB-EAFC-4521-B10D-CC6280C64C86@yahoo.com>
Message-ID: <5405C703.30706@oracle.com>

On 09/02/2014 05:18 PM, vyazelenko at yahoo.com wrote:
>> Yes, there is the implementation reason for that: @Setup is executed by
>> the generated code, and so the harness control thread is oblivious of
>> the time spent in @Setup vs. @Benchmark.
> I have a problem with that. It is neither intuitive nor expected
> that @Setup/@TearDown times are considered part of actual measurement
> Especially if Level.Trial is used.

@Setup/@TearDown-s are not considered the part of "measurement". @Setup
methods are not measured in any case, therefore your example about the
heavy-weight- at Setup kilo-benchmark is not applicable.

My point is about harness control thread, which guards the timeout, and
forcefully terminates the payload if it had timed out. I see no
particular reason for *control* thread to track the @Benchmark time in
isolation: once you about to time out with a large timeout, you already
have a problem :)

>> Anyway, that is arguably not-the-issue, here is why. The timeout-ing is
>> done to recover from the misbehaved benchmarks, and the @Setup taking 10
>> minutes to run is arguably misbehaving. We can't afford waiting
>> indefinitely for broken benchmarks if we are running the large benchmark
>> suites. Now, when we are facing the misbehaving workload, it does not
>> seem all that necessary to accurately track the workload time?
> Well if Setup is not considered part of the workload then there would no problem to start with.

See above, not an issue.

> I think now with 1.0 released such change can be considered. 
> Also waiting indefinitely for @Setup/@TearDown and maybe 
> even benchmark proper should be considered.

Maybe. But insofar, there is no strong evidence it is required. If you
have a long-running benchmark with large @Setup/@TearDown/@Benchmark,
you can set the larger "timeout" with -r (or it's equivalent in API and
annotations). Users will still have to do this step, and the property
"-Djmh.waitForever=true" is not really different from "-r 10d(ays)".


>> And what would be the best place to document this? I am puzzled about
>> that, and since the issue is about the corner case, it does not seem to
>> belong in the generic documentation.
> Why not document it in "time" property of the @Warmup/@Measurement annotations? Saying something like:
> "If time is undefined then depending on the BenchmarkMode JMH will:
>   - wait at least 5 seconds...
>   - wait for 10 minutes in case SingleShotTime...
> To override defaults please put time value explicitly..."

Alas, does not work that easily: there are also Java API-ish Options,
command line interface, etc.

Thanks,
-Aleksey.


From vyazelenko at yahoo.com  Tue Sep  2 14:16:20 2014
From: vyazelenko at yahoo.com (vyazelenko at yahoo.com)
Date: Tue, 2 Sep 2014 16:16:20 +0200
Subject: Setup time is counted against iteration time in
	Mode.SingleShotTime (JMH-1.0)
In-Reply-To: <5405C703.30706@oracle.com>
References: <1409582915.16255.YahooMailNeo@web162205.mail.bf1.yahoo.com>
	<5405A151.6060609@oracle.com>
	<3727E2AB-EAFC-4521-B10D-CC6280C64C86@yahoo.com>
	<5405C703.30706@oracle.com>
Message-ID: <39FF5F51-2F44-4C66-9914-4585398E0648@yahoo.com>

Ok, thanks for expanding on harness control thread and timeouts. I agree with all being said but still consider behavior of SingleShotTime magic.

Currently SingleShotTime is the only mode that seems to logically fit into description "timeout-agnostic". Up until now I thought that this mode should be used for kilo- and mega- benches but it turns out it's not the case (at least not in the default configuration). So it seems support for such benches is missing or should be made more explicit.

How about adding another command line switch -waitForever similar to -foe and -gc. At least it would allow altering behavior for all benches at once. ;)

Sent from my iPhone

> On Sep 2, 2014, at 15:32, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:
> 
> On 09/02/2014 05:18 PM, vyazelenko at yahoo.com wrote:
>>> Yes, there is the implementation reason for that: @Setup is executed by
>>> the generated code, and so the harness control thread is oblivious of
>>> the time spent in @Setup vs. @Benchmark.
>> I have a problem with that. It is neither intuitive nor expected
>> that @Setup/@TearDown times are considered part of actual measurement
>> Especially if Level.Trial is used.
> 
> @Setup/@TearDown-s are not considered the part of "measurement". @Setup
> methods are not measured in any case, therefore your example about the
> heavy-weight- at Setup kilo-benchmark is not applicable.
> 
> My point is about harness control thread, which guards the timeout, and
> forcefully terminates the payload if it had timed out. I see no
> particular reason for *control* thread to track the @Benchmark time in
> isolation: once you about to time out with a large timeout, you already
> have a problem :)
> 
>>> Anyway, that is arguably not-the-issue, here is why. The timeout-ing is
>>> done to recover from the misbehaved benchmarks, and the @Setup taking 10
>>> minutes to run is arguably misbehaving. We can't afford waiting
>>> indefinitely for broken benchmarks if we are running the large benchmark
>>> suites. Now, when we are facing the misbehaving workload, it does not
>>> seem all that necessary to accurately track the workload time?
>> Well if Setup is not considered part of the workload then there would no problem to start with.
> 
> See above, not an issue.
> 
>> I think now with 1.0 released such change can be considered. 
>> Also waiting indefinitely for @Setup/@TearDown and maybe 
>> even benchmark proper should be considered.
> 
> Maybe. But insofar, there is no strong evidence it is required. If you
> have a long-running benchmark with large @Setup/@TearDown/@Benchmark,
> you can set the larger "timeout" with -r (or it's equivalent in API and
> annotations). Users will still have to do this step, and the property
> "-Djmh.waitForever=true" is not really different from "-r 10d(ays)".
> 
> 
>>> And what would be the best place to document this? I am puzzled about
>>> that, and since the issue is about the corner case, it does not seem to
>>> belong in the generic documentation.
>> Why not document it in "time" property of the @Warmup/@Measurement annotations? Saying something like:
>> "If time is undefined then depending on the BenchmarkMode JMH will:
>>  - wait at least 5 seconds...
>>  - wait for 10 minutes in case SingleShotTime...
>> To override defaults please put time value explicitly..."
> 
> Alas, does not work that easily: there are also Java API-ish Options,
> command line interface, etc.
> 
> Thanks,
> -Aleksey.
> 

From tom.deneau at amd.com  Tue Sep  2 16:13:30 2014
From: tom.deneau at amd.com (Deneau, Tom)
Date: Tue, 2 Sep 2014 16:13:30 +0000
Subject: detecting jit compilation
Message-ID: <BC97738F8E7C8742BABED7F06FB9DF91554A0EA2@SATLEXDAG01.amd.com>

This may not be the right list for this question but...

Is there a way thru Management Beans that I can find out from the Java side whether a particular method has been JIT compiled?

-- Tom Deneau

-----Original Message-----
From: Deneau, Tom 
Sent: Friday, August 22, 2014 12:16 PM
To: jmh-dev at openjdk.java.net
Subject: specializing jvmArgs in a derived class

I have a base class with
    @Benchmark 
    public void myBenchmark()

and several derived classes that specialize that benchmark.
In one of the derived classes, I would like to specify that when that derived class forks, certain jvmArgs get added to the jvm command line.  Is there a way to do this?

When I tried this in the derived class

    @Override
    @Benchmark
    @Fork(jvmArgs="...") 
    public void myBenchmark()

I got this error which I don't really understand:
    Internal error: multiple methods per @Group, but not all methods have @Group
    
If I remove the @Benchmark annotation from the derived myBenchmark, that error goes away but the jvmArgs seems to also get ignored.

-- Tom

From kirk at kodewerk.com  Tue Sep  2 16:30:58 2014
From: kirk at kodewerk.com (Kirk Pepperdine)
Date: Tue, 2 Sep 2014 17:30:58 +0100
Subject: detecting jit compilation
In-Reply-To: <BC97738F8E7C8742BABED7F06FB9DF91554A0EA2@SATLEXDAG01.amd.com>
References: <BC97738F8E7C8742BABED7F06FB9DF91554A0EA2@SATLEXDAG01.amd.com>
Message-ID: <CC4070F7-9A4B-4D38-B21F-E47AFFF18DEB@kodewerk.com>

Unfortunately no MXBean for this. You?ll have to use -XX:+PrintCompilation

? Kirk

On Sep 2, 2014, at 5:13 PM, Deneau, Tom <tom.deneau at amd.com> wrote:

> This may not be the right list for this question but...
> 
> Is there a way thru Management Beans that I can find out from the Java side whether a particular method has been JIT compiled?
> 
> -- Tom Deneau
> 
> -----Original Message-----
> From: Deneau, Tom 
> Sent: Friday, August 22, 2014 12:16 PM
> To: jmh-dev at openjdk.java.net
> Subject: specializing jvmArgs in a derived class
> 
> I have a base class with
>    @Benchmark 
>    public void myBenchmark()
> 
> and several derived classes that specialize that benchmark.
> In one of the derived classes, I would like to specify that when that derived class forks, certain jvmArgs get added to the jvm command line.  Is there a way to do this?
> 
> When I tried this in the derived class
> 
>    @Override
>    @Benchmark
>    @Fork(jvmArgs="...") 
>    public void myBenchmark()
> 
> I got this error which I don't really understand:
>    Internal error: multiple methods per @Group, but not all methods have @Group
> 
> If I remove the @Benchmark annotation from the derived myBenchmark, that error goes away but the jvmArgs seems to also get ignored.
> 
> -- Tom


From aleksey.shipilev at oracle.com  Tue Sep  2 16:48:12 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Tue, 02 Sep 2014 20:48:12 +0400
Subject: detecting jit compilation
In-Reply-To: <BC97738F8E7C8742BABED7F06FB9DF91554A0EA2@SATLEXDAG01.amd.com>
References: <BC97738F8E7C8742BABED7F06FB9DF91554A0EA2@SATLEXDAG01.amd.com>
Message-ID: <5405F4CC.6080901@oracle.com>

On 09/02/2014 08:13 PM, Deneau, Tom wrote:
> This may not be the right list for this question but...

hotspot-compiler-dev@ is the best list to ask this.

> Is there a way thru Management Beans that I can find out from the
> Java side whether a particular method has been JIT compiled?

...and hotspot-compiler-dev@ would probably reply you are looking at
WhiteBox API, which will call into
http://hg.openjdk.java.net/jdk9/jdk9/hotspot/file/tip/src/share/vm/prims/whitebox.cpp#l358.

-Aleksey.


From aleksey.shipilev at oracle.com  Tue Sep  2 19:05:04 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Tue, 02 Sep 2014 19:05:04 +0000
Subject: hg: code-tools/jmh: generators: merge the @State override classes
	together,
	yields massive decrease in generated code size and compilation time.
Message-ID: <201409021905.s82J54XD008478@aojmv0008>

Changeset: a8f5b91bcefc
Author:    shade
Date:      2014-09-02 23:04 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/a8f5b91bcefc

generators: merge the @State override classes together, yields massive decrease in generated code size and compilation time.

! jmh-core/src/main/java/org/openjdk/jmh/generators/core/BenchmarkGenerator.java
+ jmh-core/src/main/java/org/openjdk/jmh/generators/core/BenchmarkGeneratorSession.java
! jmh-core/src/main/java/org/openjdk/jmh/generators/core/Identifiers.java
! jmh-core/src/main/java/org/openjdk/jmh/generators/core/StateObject.java
! jmh-core/src/main/java/org/openjdk/jmh/generators/core/StateObjectHandler.java


From aleksey.shipilev at oracle.com  Tue Sep  2 20:02:50 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Tue, 02 Sep 2014 20:02:50 +0000
Subject: hg: code-tools/jmh: generators: cache indent Strings,
	decreases the allocation pressure.
Message-ID: <201409022002.s82K2ooP017995@aojmv0008>

Changeset: 24f2032b147c
Author:    shade
Date:      2014-09-02 23:58 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/24f2032b147c

generators: cache indent Strings, decreases the allocation pressure.

! jmh-core/src/main/java/org/openjdk/jmh/generators/core/BenchmarkGenerator.java


From aleksey.shipilev at oracle.com  Tue Sep  2 20:36:15 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Tue, 02 Sep 2014 20:36:15 +0000
Subject: hg: code-tools/jmh: generators: more touchups in Paddings,
	denser padding again.
Message-ID: <201409022036.s82KaFjI023519@aojmv0008>

Changeset: 0a0ca0942f1d
Author:    shade
Date:      2014-09-03 00:33 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/0a0ca0942f1d

generators: more touchups in Paddings, denser padding again.

! jmh-core/src/main/java/org/openjdk/jmh/generators/core/BenchmarkGenerator.java
! jmh-core/src/main/java/org/openjdk/jmh/generators/core/Paddings.java
! jmh-core/src/main/java/org/openjdk/jmh/generators/core/StateObjectHandler.java


From aleksey.shipilev at oracle.com  Wed Sep  3 16:02:01 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Wed, 03 Sep 2014 16:02:01 +0000
Subject: hg: code-tools/jmh: generators: clean up imports in the generated
	code.
Message-ID: <201409031602.s83G21Te007772@aojmv0008>

Changeset: 77a66bcd6775
Author:    shade
Date:      2014-09-03 11:40 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/77a66bcd6775

generators: clean up imports in the generated code.

! jmh-core/src/main/java/org/openjdk/jmh/generators/core/BenchmarkGenerator.java


From aleksey.shipilev at oracle.com  Wed Sep  3 16:02:09 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Wed, 03 Sep 2014 16:02:09 +0000
Subject: hg: code-tools/jmh: generators: trim the number of compiler commands, 
	in attempt to dodge JDK-8057169.
Message-ID: <201409031602.s83G29JV007821@aojmv0008>

Changeset: da463d84dd41
Author:    shade
Date:      2014-09-03 20:01 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/da463d84dd41

generators: trim the number of compiler commands, in attempt to dodge JDK-8057169.

! jmh-core/src/main/java/org/openjdk/jmh/generators/core/BenchmarkGenerator.java


From aleksey.shipilev at oracle.com  Wed Sep  3 18:49:52 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Wed, 03 Sep 2014 18:49:52 +0000
Subject: hg: code-tools/jmh: jmh-core-it: fix two regressed tests after we
	changed the "jmhLoop" to "jmhStub" in generated method names.
Message-ID: <201409031849.s83Inqx1004578@aojmv0008>

Changeset: 53e0e7585434
Author:    shade
Date:      2014-09-03 22:49 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/53e0e7585434

jmh-core-it: fix two regressed tests after we changed the "jmhLoop" to "jmhStub" in generated method names.

! jmh-core-it/src/test/java/org/openjdk/jmh/it/synciter/SyncIterMeasurementOnlyTest.java
! jmh-core-it/src/test/java/org/openjdk/jmh/it/synciter/SyncIterNotOnlyMeasurementTest.java


From aleksey.shipilev at oracle.com  Thu Sep  4 06:16:17 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Thu, 04 Sep 2014 10:16:17 +0400
Subject: Setup time is counted against iteration time in
	Mode.SingleShotTime (JMH-1.0)
In-Reply-To: <39FF5F51-2F44-4C66-9914-4585398E0648@yahoo.com>
References: <1409582915.16255.YahooMailNeo@web162205.mail.bf1.yahoo.com>
	<5405A151.6060609@oracle.com>
	<3727E2AB-EAFC-4521-B10D-CC6280C64C86@yahoo.com>
	<5405C703.30706@oracle.com>
	<39FF5F51-2F44-4C66-9914-4585398E0648@yahoo.com>
Message-ID: <540803B1.4090809@oracle.com>

On 09/02/2014 06:16 PM, vyazelenko at yahoo.com wrote:
> Currently SingleShotTime is the only mode that seems to logically fit
> into description "timeout-agnostic". Up until now I thought that this
> mode should be used for kilo- and mega- benches but it turns out it's
> not the case (at least not in the default configuration). So it seems
> support for such benches is missing or should be made more explicit.

Well, any benchmark mode should be usable for kilo- and mega-
benchmarks. However, just like for time-based benchmarks, you have to
communicate how long you are expecting the benchmark to run.

> How about adding another command line switch -waitForever similar to
> -foe and -gc. At least it would allow altering behavior for all
> benches at once. ;)

-r $LARGE_VALUE also applies to all benchmarks at once.

Recorded:
 https://bugs.openjdk.java.net/browse/CODETOOLS-7901022

Thanks,
-Aleksey



From aleksey.shipilev at oracle.com  Thu Sep  4 12:45:26 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Thu, 04 Sep 2014 12:45:26 +0000
Subject: hg: code-tools/jmh: 7901013: JMH should safeguard against the
	concurrent execution with file lock
Message-ID: <201409041245.s84CjQvF001969@aojmv0008>

Changeset: 6a7ef879247a
Author:    shade
Date:      2014-09-04 16:45 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/6a7ef879247a

7901013: JMH should safeguard against the concurrent execution with file lock

! jmh-core-it/pom.xml
! jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java
! jmh-core/src/main/java/org/openjdk/jmh/util/FileUtils.java


From aleksey.shipilev at oracle.com  Thu Sep  4 13:02:33 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Thu, 04 Sep 2014 13:02:33 +0000
Subject: hg: code-tools/jmh: runners: provide user-settable property for the
	number of tailed lines to print on forked VM error.
Message-ID: <201409041302.s84D2Xwo004731@aojmv0008>

Changeset: bae8cd49a7c1
Author:    shade
Date:      2014-09-04 17:02 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/bae8cd49a7c1

runners: provide user-settable property for the number of tailed lines to print on forked VM error.

! jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java


From aleksey.shipilev at oracle.com  Thu Sep  4 20:05:59 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Fri, 05 Sep 2014 00:05:59 +0400
Subject: JMH 1.0.1
Message-ID: <5408C627.4080506@oracle.com>

Hi,

We have released the patch release, JMH 1.0.1. It is already available
in Maven Central, thanks to Evgeny as usual.

Fixes:

 * Massively improved compilation performance when dealing with large
benchmark corpus. On our internal corpus of 1200+ @Benchmarks, we have
3x improvement in compilation time:
  https://bugs.openjdk.java.net/browse/CODETOOLS-7901017

 * Another issue manifested in large corpus. CompilerOracle is not
scaling well with the number of compile commands, so we had to emit more
condensed patterns in our CompilerHints file.
  https://bugs.openjdk.java.net/browse/CODETOOLS-7901024

 * As suggested before on this mailing list, JMH now acquires the file
lock trying to forbid the concurrent runs of multiple JMH instances.
Users may opt to ignore that lock.
  https://bugs.openjdk.java.net/browse/CODETOOLS-7901013

 * User-settable property for the number of tailed lines to print on
forked VM error is added:
  http://hg.openjdk.java.net/code-tools/jmh/rev/bae8cd49a7c1

Enjoy this patch release!

Thanks,
-Aleksey.


From aleksey.shipilev at oracle.com  Thu Sep  4 20:06:56 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Thu, 04 Sep 2014 20:06:56 +0000
Subject: hg: code-tools/jmh: 3 new changesets
Message-ID: <201409042007.s84K6vWE017452@aojmv0008>

Changeset: 0d45525aa63e
Author:    shade
Date:      2014-09-04 17:15 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/0d45525aa63e

JMH v1.0.1.

! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml
! jmh-archetypes/pom.xml
! jmh-core-benchmarks/pom.xml
! jmh-core-ct/pom.xml
! jmh-core-it/pom.xml
! jmh-core/pom.xml
! jmh-generator-annprocess/pom.xml
! jmh-generator-asm/pom.xml
! jmh-generator-bytecode/pom.xml
! jmh-generator-reflection/pom.xml
! jmh-samples/pom.xml
! pom.xml

Changeset: 2ae1299af341
Author:    shade
Date:      2014-09-04 17:15 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/2ae1299af341

Added tag 1.0.1 for changeset 0d45525aa63e

! .hgtags

Changeset: df2964b5c7c7
Author:    shade
Date:      2014-09-04 17:16 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/df2964b5c7c7

Continue in 1.1-SNAPSHOT.

! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml
! jmh-archetypes/pom.xml
! jmh-core-benchmarks/pom.xml
! jmh-core-ct/pom.xml
! jmh-core-it/pom.xml
! jmh-core/pom.xml
! jmh-generator-annprocess/pom.xml
! jmh-generator-asm/pom.xml
! jmh-generator-bytecode/pom.xml
! jmh-generator-reflection/pom.xml
! jmh-samples/pom.xml
! pom.xml


From nitsanw at yahoo.com  Mon Sep  8 07:53:16 2014
From: nitsanw at yahoo.com (Nitsan Wakart)
Date: Mon, 8 Sep 2014 00:53:16 -0700
Subject: FR: summary stats to detect multi-modal behaviour of benchmarks
Message-ID: <1410162796.34666.YahooMailNeo@web120703.mail.ne1.yahoo.com>

Hi,
In my work I come across benchmarks that have 'modes' of performance. For instance a benchmark exercising code which suffers from false sharing usually has a probability of that false sharing manifesting. The run to run variance is very indicative of these sort of issues when they happen, but the summary of all the runs put together hides this behaviour from me. I've hit similar issues around unstable compilation results and timing sensitive benchmarks.
Currently I parse the output by hand/script to detect these anomalies and look out for any large error indicators in the summary, but I was wondering if we can make the summary statistics pluggable to better detect these behaviours, or if perhaps other people have better solutions to this problem.
Thanks,
Nitsan

From aleksey.shipilev at oracle.com  Mon Sep  8 08:37:49 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Mon, 08 Sep 2014 12:37:49 +0400
Subject: FR: summary stats to detect multi-modal behaviour of benchmarks
In-Reply-To: <1410162796.34666.YahooMailNeo@web120703.mail.ne1.yahoo.com>
References: <1410162796.34666.YahooMailNeo@web120703.mail.ne1.yahoo.com>
Message-ID: <540D6ADD.9080403@oracle.com>

On 09/08/2014 11:53 AM, Nitsan Wakart wrote:
> Hi, In my work I come across benchmarks that have 'modes' of
> performance. For instance a benchmark exercising code which suffers
> from false sharing usually has a probability of that false sharing
> manifesting. The run to run variance is very indicative of these sort
> of issues when they happen, but the summary of all the runs put
> together hides this behaviour from me. I've hit similar issues around
> unstable compilation results and timing sensitive benchmarks. 

So do we, but are there reliable statistical tests which can tell the
multi-modality, even if the baseline distribution is known? One can go
the other way around, and test for normality...

> Currently I parse the output by hand/script to detect these anomalies
> and look out for any large error indicators in the summary, but I was
> wondering if we can make the summary statistics pluggable to better
> detect these behaviours, or if perhaps other people have better
> solutions to this problem.

Well, pluggable how? Java API provides the access to raw statistical
data today. You can create your own runner to use the Java API and do
the results post-processing as you want.

I would like to see those statistic analysis approaches implemented
using Java API; then we can figure out if something is sensible to add
in the mainline JMH.

Thanks,
-Aleksey.



From aleksey.shipilev at oracle.com  Wed Sep 10 13:17:23 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Wed, 10 Sep 2014 13:17:23 +0000
Subject: hg: code-tools/jmh: 7901025: Prepend all internal settings with "jmh."
Message-ID: <201409101317.s8ADHO4P004865@aojmv0008>

Changeset: 9a66e03cd539
Author:    shade
Date:      2014-09-10 17:17 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/9a66e03cd539

7901025: Prepend all internal settings with "jmh."

! jmh-core-it/src/test/java/org/openjdk/jmh/it/Fixtures.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/BaseBenchmarkHandler.java


From aleksey.shipilev at oracle.com  Wed Sep 10 14:30:44 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Wed, 10 Sep 2014 18:30:44 +0400
Subject: RFR (S) 7901022: JMH runners should provide explicit timeout setting
Message-ID: <54106094.8060806@oracle.com>

Hi,

I need a second pair of eyes to look through the patch:
  http://cr.openjdk.java.net/~shade/7901022/webrev.01/

Please take a look. It removes the current "timeout" scheme that guesses
the timeout based on -r/measurementTime settings, and instead does the
explicit -to/timeout setting to control how much we wait. It also does
the waiting properly in multi-threaded workloads: do not wait for an
entire time, but only for the remaining part of it.

Context:
  https://bugs.openjdk.java.net/browse/CODETOOLS-7901022
  http://mail.openjdk.java.net/pipermail/jmh-dev/2014-September/001369.html

Thanks,
-Aleksey.


From sergey.kuksenko at oracle.com  Wed Sep 10 14:50:50 2014
From: sergey.kuksenko at oracle.com (Sergey Kuksenko)
Date: Wed, 10 Sep 2014 18:50:50 +0400
Subject: RFR (S) 7901022: JMH runners should provide explicit timeout
	setting
In-Reply-To: <54106094.8060806@oracle.com>
References: <54106094.8060806@oracle.com>
Message-ID: <5410654A.2060802@oracle.com>

Checked with one eye (you need yet another eye :))
Looks fine, except: What are you going to do if iteration time is larger 
than timeout?


On 09/10/2014 06:30 PM, Aleksey Shipilev wrote:
> Hi,
>
> I need a second pair of eyes to look through the patch:
>    http://cr.openjdk.java.net/~shade/7901022/webrev.01/
>
> Please take a look. It removes the current "timeout" scheme that guesses
> the timeout based on -r/measurementTime settings, and instead does the
> explicit -to/timeout setting to control how much we wait. It also does
> the waiting properly in multi-threaded workloads: do not wait for an
> entire time, but only for the remaining part of it.
>
> Context:
>    https://bugs.openjdk.java.net/browse/CODETOOLS-7901022
>    http://mail.openjdk.java.net/pipermail/jmh-dev/2014-September/001369.html
>
> Thanks,
> -Aleksey.
>

-- 
Best regards,
Sergey Kuksenko

From aleksey.shipilev at oracle.com  Wed Sep 10 14:59:40 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Wed, 10 Sep 2014 14:59:40 +0000
Subject: hg: code-tools/jmh: 7901030: -prof perf fails with
	NumberFormatException on some locales
Message-ID: <201409101459.s8AExeqO020484@aojmv0008>

Changeset: 76d7ea927d5d
Author:    shade
Date:      2014-09-10 18:59 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/76d7ea927d5d

7901030: -prof perf fails with NumberFormatException on some locales

! jmh-core/src/main/java/org/openjdk/jmh/profile/LinuxPerfProfiler.java


From aleksey.shipilev at oracle.com  Wed Sep 10 15:09:58 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Wed, 10 Sep 2014 19:09:58 +0400
Subject: RFR (S) 7901022: JMH runners should provide explicit timeout
	setting
In-Reply-To: <5410654A.2060802@oracle.com>
References: <54106094.8060806@oracle.com> <5410654A.2060802@oracle.com>
Message-ID: <541069C6.6050300@oracle.com>

Thanks!

Current code does not treat the case (runtime > timeout) in any specific
manner. You will get the interrupt right in the middle of the run. I can
think up the case when you actually want that to test the interrupt
mechanics, so current code seems to be what we want it to be. Thoughts?

-Aleksey.

On 09/10/2014 06:50 PM, Sergey Kuksenko wrote:
> Checked with one eye (you need yet another eye :))
> Looks fine, except: What are you going to do if iteration time is larger
> than timeout?
> 
> 
> On 09/10/2014 06:30 PM, Aleksey Shipilev wrote:
>> Hi,
>>
>> I need a second pair of eyes to look through the patch:
>>    http://cr.openjdk.java.net/~shade/7901022/webrev.01/
>>
>> Please take a look. It removes the current "timeout" scheme that guesses
>> the timeout based on -r/measurementTime settings, and instead does the
>> explicit -to/timeout setting to control how much we wait. It also does
>> the waiting properly in multi-threaded workloads: do not wait for an
>> entire time, but only for the remaining part of it.
>>
>> Context:
>>    https://bugs.openjdk.java.net/browse/CODETOOLS-7901022
>>   
>> http://mail.openjdk.java.net/pipermail/jmh-dev/2014-September/001369.html
>>
>> Thanks,
>> -Aleksey.
>>
> 



From sergey.kuksenko at oracle.com  Wed Sep 10 15:22:01 2014
From: sergey.kuksenko at oracle.com (Sergey Kuksenko)
Date: Wed, 10 Sep 2014 19:22:01 +0400
Subject: RFR (S) 7901022: JMH runners should provide explicit timeout
	setting
In-Reply-To: <541069C6.6050300@oracle.com>
References: <54106094.8060806@oracle.com> <5410654A.2060802@oracle.com>
	<541069C6.6050300@oracle.com>
Message-ID: <54106C99.7040002@oracle.com>

But  what to do when someone want to measure something for 12 minutes 
iteration and he/she didn't read demos/manuals correctly and/or forget 
about timeout?

On 09/10/2014 07:09 PM, Aleksey Shipilev wrote:
> Thanks!
>
> Current code does not treat the case (runtime > timeout) in any specific
> manner. You will get the interrupt right in the middle of the run. I can
> think up the case when you actually want that to test the interrupt
> mechanics, so current code seems to be what we want it to be. Thoughts?
>
> -Aleksey.
>
> On 09/10/2014 06:50 PM, Sergey Kuksenko wrote:
>> Checked with one eye (you need yet another eye :))
>> Looks fine, except: What are you going to do if iteration time is larger
>> than timeout?
>>
>>
>> On 09/10/2014 06:30 PM, Aleksey Shipilev wrote:
>>> Hi,
>>>
>>> I need a second pair of eyes to look through the patch:
>>>     http://cr.openjdk.java.net/~shade/7901022/webrev.01/
>>>
>>> Please take a look. It removes the current "timeout" scheme that guesses
>>> the timeout based on -r/measurementTime settings, and instead does the
>>> explicit -to/timeout setting to control how much we wait. It also does
>>> the waiting properly in multi-threaded workloads: do not wait for an
>>> entire time, but only for the remaining part of it.
>>>
>>> Context:
>>>     https://bugs.openjdk.java.net/browse/CODETOOLS-7901022
>>>
>>> http://mail.openjdk.java.net/pipermail/jmh-dev/2014-September/001369.html
>>>
>>> Thanks,
>>> -Aleksey.
>>>
>>
>
>

-- 
Best regards,
Sergey Kuksenko

From sergey.kuksenko at oracle.com  Wed Sep 10 15:25:32 2014
From: sergey.kuksenko at oracle.com (Sergey Kuksenko)
Date: Wed, 10 Sep 2014 19:25:32 +0400
Subject: RFR (S) 7901022: JMH runners should provide explicit timeout
	setting
In-Reply-To: <54106C99.7040002@oracle.com>
References: <54106094.8060806@oracle.com>
	<5410654A.2060802@oracle.com>	<541069C6.6050300@oracle.com>
	<54106C99.7040002@oracle.com>
Message-ID: <54106D6C.3050609@oracle.com>

JMH may print warning and continue executions as is.

On 09/10/2014 07:22 PM, Sergey Kuksenko wrote:
> But  what to do when someone want to measure something for 12 minutes
> iteration and he/she didn't read demos/manuals correctly and/or forget
> about timeout?
>
> On 09/10/2014 07:09 PM, Aleksey Shipilev wrote:
>> Thanks!
>>
>> Current code does not treat the case (runtime > timeout) in any specific
>> manner. You will get the interrupt right in the middle of the run. I can
>> think up the case when you actually want that to test the interrupt
>> mechanics, so current code seems to be what we want it to be. Thoughts?
>>
>> -Aleksey.
>>
>> On 09/10/2014 06:50 PM, Sergey Kuksenko wrote:
>>> Checked with one eye (you need yet another eye :))
>>> Looks fine, except: What are you going to do if iteration time is larger
>>> than timeout?
>>>
>>>
>>> On 09/10/2014 06:30 PM, Aleksey Shipilev wrote:
>>>> Hi,
>>>>
>>>> I need a second pair of eyes to look through the patch:
>>>>     http://cr.openjdk.java.net/~shade/7901022/webrev.01/
>>>>
>>>> Please take a look. It removes the current "timeout" scheme that
>>>> guesses
>>>> the timeout based on -r/measurementTime settings, and instead does the
>>>> explicit -to/timeout setting to control how much we wait. It also does
>>>> the waiting properly in multi-threaded workloads: do not wait for an
>>>> entire time, but only for the remaining part of it.
>>>>
>>>> Context:
>>>>     https://bugs.openjdk.java.net/browse/CODETOOLS-7901022
>>>>
>>>> http://mail.openjdk.java.net/pipermail/jmh-dev/2014-September/001369.html
>>>>
>>>>
>>>> Thanks,
>>>> -Aleksey.
>>>>
>>>
>>
>>
>

-- 
Best regards,
Sergey Kuksenko

From aleksey.shipilev at oracle.com  Wed Sep 10 19:43:05 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Wed, 10 Sep 2014 23:43:05 +0400
Subject: RFR (S) 7901022: JMH runners should provide explicit timeout
	setting
In-Reply-To: <54106D6C.3050609@oracle.com>
References: <54106094.8060806@oracle.com>
	<5410654A.2060802@oracle.com>	<541069C6.6050300@oracle.com>
	<54106C99.7040002@oracle.com> <54106D6C.3050609@oracle.com>
Message-ID: <5410A9C9.4090607@oracle.com>

Okay, printing the warning:
 http://cr.openjdk.java.net/~shade/7901022/webrev.02/

The printing itself here:

http://cr.openjdk.java.net/~shade/7901022/webrev.02/jmh-core/src/main/java/org/openjdk/jmh/runner/format/TextReportFormat.java.sdiff.html

-Aleksey.

On 09/10/2014 07:25 PM, Sergey Kuksenko wrote:
> JMH may print warning and continue executions as is.
> 
> On 09/10/2014 07:22 PM, Sergey Kuksenko wrote:
>> But  what to do when someone want to measure something for 12 minutes
>> iteration and he/she didn't read demos/manuals correctly and/or forget
>> about timeout?
>>
>> On 09/10/2014 07:09 PM, Aleksey Shipilev wrote:
>>> Thanks!
>>>
>>> Current code does not treat the case (runtime > timeout) in any specific
>>> manner. You will get the interrupt right in the middle of the run. I can
>>> think up the case when you actually want that to test the interrupt
>>> mechanics, so current code seems to be what we want it to be. Thoughts?
>>>
>>> -Aleksey.
>>>
>>> On 09/10/2014 06:50 PM, Sergey Kuksenko wrote:
>>>> Checked with one eye (you need yet another eye :))
>>>> Looks fine, except: What are you going to do if iteration time is
>>>> larger
>>>> than timeout?
>>>>
>>>>
>>>> On 09/10/2014 06:30 PM, Aleksey Shipilev wrote:
>>>>> Hi,
>>>>>
>>>>> I need a second pair of eyes to look through the patch:
>>>>>     http://cr.openjdk.java.net/~shade/7901022/webrev.01/
>>>>>
>>>>> Please take a look. It removes the current "timeout" scheme that
>>>>> guesses
>>>>> the timeout based on -r/measurementTime settings, and instead does the
>>>>> explicit -to/timeout setting to control how much we wait. It also does
>>>>> the waiting properly in multi-threaded workloads: do not wait for an
>>>>> entire time, but only for the remaining part of it.
>>>>>
>>>>> Context:
>>>>>     https://bugs.openjdk.java.net/browse/CODETOOLS-7901022
>>>>>
>>>>> http://mail.openjdk.java.net/pipermail/jmh-dev/2014-September/001369.html
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>> -Aleksey.
>>>>>
>>>>
>>>
>>>
>>
> 



From vyazelenko at yahoo.com  Wed Sep 10 22:31:52 2014
From: vyazelenko at yahoo.com (vyazelenko at yahoo.com)
Date: Thu, 11 Sep 2014 00:31:52 +0200
Subject: RFR (S) 7901022: JMH runners should provide explicit timeout
	setting
In-Reply-To: <5410A9C9.4090607@oracle.com>
References: <54106094.8060806@oracle.com> <5410654A.2060802@oracle.com>
	<541069C6.6050300@oracle.com> <54106C99.7040002@oracle.com>
	<54106D6C.3050609@oracle.com> <5410A9C9.4090607@oracle.com>
Message-ID: <B1EE559C-BCC3-43F3-8E78-9FD15E412123@yahoo.com>

Looks good! Great to see this issue being tackled!

When will it be available to play with? ;)

Best regards,
Dmitry

Sent from my iPhone

> On Sep 10, 2014, at 21:43, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:
> 
> Okay, printing the warning:
> http://cr.openjdk.java.net/~shade/7901022/webrev.02/
> 
> The printing itself here:
> 
> http://cr.openjdk.java.net/~shade/7901022/webrev.02/jmh-core/src/main/java/org/openjdk/jmh/runner/format/TextReportFormat.java.sdiff.html
> 
> -Aleksey.
> 
>> On 09/10/2014 07:25 PM, Sergey Kuksenko wrote:
>> JMH may print warning and continue executions as is.
>> 
>>> On 09/10/2014 07:22 PM, Sergey Kuksenko wrote:
>>> But  what to do when someone want to measure something for 12 minutes
>>> iteration and he/she didn't read demos/manuals correctly and/or forget
>>> about timeout?
>>> 
>>>> On 09/10/2014 07:09 PM, Aleksey Shipilev wrote:
>>>> Thanks!
>>>> 
>>>> Current code does not treat the case (runtime > timeout) in any specific
>>>> manner. You will get the interrupt right in the middle of the run. I can
>>>> think up the case when you actually want that to test the interrupt
>>>> mechanics, so current code seems to be what we want it to be. Thoughts?
>>>> 
>>>> -Aleksey.
>>>> 
>>>>> On 09/10/2014 06:50 PM, Sergey Kuksenko wrote:
>>>>> Checked with one eye (you need yet another eye :))
>>>>> Looks fine, except: What are you going to do if iteration time is
>>>>> larger
>>>>> than timeout?
>>>>> 
>>>>> 
>>>>>> On 09/10/2014 06:30 PM, Aleksey Shipilev wrote:
>>>>>> Hi,
>>>>>> 
>>>>>> I need a second pair of eyes to look through the patch:
>>>>>>    http://cr.openjdk.java.net/~shade/7901022/webrev.01/
>>>>>> 
>>>>>> Please take a look. It removes the current "timeout" scheme that
>>>>>> guesses
>>>>>> the timeout based on -r/measurementTime settings, and instead does the
>>>>>> explicit -to/timeout setting to control how much we wait. It also does
>>>>>> the waiting properly in multi-threaded workloads: do not wait for an
>>>>>> entire time, but only for the remaining part of it.
>>>>>> 
>>>>>> Context:
>>>>>>    https://bugs.openjdk.java.net/browse/CODETOOLS-7901022
>>>>>> 
>>>>>> http://mail.openjdk.java.net/pipermail/jmh-dev/2014-September/001369.html
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Thanks,
>>>>>> -Aleksey.
> 
> 

From aleksey.shipilev at oracle.com  Thu Sep 11 09:42:52 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Thu, 11 Sep 2014 09:42:52 +0000
Subject: hg: code-tools/jmh: 7901022: JMH runners should provide explicit
	timeout setting
Message-ID: <201409110942.s8B9gqGw021588@aojmv0008>

Changeset: 2e7682a5da28
Author:    shade
Date:      2014-09-11 13:40 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/2e7682a5da28

7901022: JMH runners should provide explicit timeout setting

! jmh-core/src/main/java/org/openjdk/jmh/infra/BenchmarkParams.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/Defaults.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/LoopBenchmarkHandler.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/format/TextReportFormat.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/options/ChainedOptionsBuilder.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/options/CommandLineOptions.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/options/Options.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/options/OptionsBuilder.java
! jmh-core/src/test/java/org/openjdk/jmh/results/TestAggregateResult.java
! jmh-core/src/test/java/org/openjdk/jmh/results/format/ResultFormatTest.java
! jmh-core/src/test/java/org/openjdk/jmh/runner/RunnerTest.java
! jmh-core/src/test/java/org/openjdk/jmh/runner/options/TestOptions.java
! jmh-core/src/test/java/org/openjdk/jmh/runner/options/TestParentOptions.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_30_Interrupts.java


From aleksey.shipilev at oracle.com  Thu Sep 11 09:44:16 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Thu, 11 Sep 2014 13:44:16 +0400
Subject: RFR (S) 7901022: JMH runners should provide explicit timeout
	setting
In-Reply-To: <B1EE559C-BCC3-43F3-8E78-9FD15E412123@yahoo.com>
References: <54106094.8060806@oracle.com> <5410654A.2060802@oracle.com>
	<541069C6.6050300@oracle.com> <54106C99.7040002@oracle.com>
	<54106D6C.3050609@oracle.com> <5410A9C9.4090607@oracle.com>
	<B1EE559C-BCC3-43F3-8E78-9FD15E412123@yahoo.com>
Message-ID: <54116EF0.5080908@oracle.com>

On 09/11/2014 02:31 AM, vyazelenko at yahoo.com wrote:
> Looks good! Great to see this issue being tackled!

Thanks for reviews!

> When will it be available to play with? ;)

It is available in master now:
  http://hg.openjdk.java.net/code-tools/jmh/rev/2e7682a5da28

Can you try this by Friday afternoon UTC? We will promote 1.1 this week
then.

-Aleksey.


From vyazelenko at yahoo.com  Thu Sep 11 10:51:59 2014
From: vyazelenko at yahoo.com (vyazelenko at yahoo.com)
Date: Thu, 11 Sep 2014 12:51:59 +0200
Subject: RFR (S) 7901022: JMH runners should provide explicit timeout
	setting
In-Reply-To: <54116EF0.5080908@oracle.com>
References: <54106094.8060806@oracle.com> <5410654A.2060802@oracle.com>
	<541069C6.6050300@oracle.com> <54106C99.7040002@oracle.com>
	<54106D6C.3050609@oracle.com> <5410A9C9.4090607@oracle.com>
	<B1EE559C-BCC3-43F3-8E78-9FD15E412123@yahoo.com>
	<54116EF0.5080908@oracle.com>
Message-ID: <00443A21-1E57-4CAB-94C5-A8747B1DF083@yahoo.com>

Sure! Will test later today.

Regards,
Dmitry

Sent from my iPhone

> On Sep 11, 2014, at 11:44, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:
> 
>> On 09/11/2014 02:31 AM, vyazelenko at yahoo.com wrote:
>> Looks good! Great to see this issue being tackled!
> 
> Thanks for reviews!
> 
>> When will it be available to play with? ;)
> 
> It is available in master now:
>  http://hg.openjdk.java.net/code-tools/jmh/rev/2e7682a5da28
> 
> Can you try this by Friday afternoon UTC? We will promote 1.1 this week
> then.
> 
> -Aleksey.
> 

From vyazelenko at yahoo.com  Thu Sep 11 21:28:51 2014
From: vyazelenko at yahoo.com (vyazelenko at yahoo.com)
Date: Thu, 11 Sep 2014 23:28:51 +0200
Subject: RFR (S) 7901022: JMH runners should provide explicit timeout
	setting
In-Reply-To: <00443A21-1E57-4CAB-94C5-A8747B1DF083@yahoo.com>
References: <54106094.8060806@oracle.com> <5410654A.2060802@oracle.com>
	<541069C6.6050300@oracle.com> <54106C99.7040002@oracle.com>
	<54106D6C.3050609@oracle.com> <5410A9C9.4090607@oracle.com>
	<B1EE559C-BCC3-43F3-8E78-9FD15E412123@yahoo.com>
	<54116EF0.5080908@oracle.com>
	<00443A21-1E57-4CAB-94C5-A8747B1DF083@yahoo.com>
Message-ID: <03BF9BD3-CF25-4BBF-818F-9A990AE876DB@yahoo.com>

I have tested new timeout feature. It works as expected. 

Looking forward to 1.1 release! 

Regards,
Dmitry

Sent from my iPhone

> On Sep 11, 2014, at 12:51, vyazelenko at yahoo.com wrote:
> 
> Sure! Will test later today.
> 
> Regards,
> Dmitry
> 
> Sent from my iPhone
> 
>>> On Sep 11, 2014, at 11:44, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:
>>> 
>>> On 09/11/2014 02:31 AM, vyazelenko at yahoo.com wrote:
>>> Looks good! Great to see this issue being tackled!
>> 
>> Thanks for reviews!
>> 
>>> When will it be available to play with? ;)
>> 
>> It is available in master now:
>> http://hg.openjdk.java.net/code-tools/jmh/rev/2e7682a5da28
>> 
>> Can you try this by Friday afternoon UTC? We will promote 1.1 this week
>> then.
>> 
>> -Aleksey.
>> 

From aleksey.shipilev at oracle.com  Fri Sep 12 08:38:57 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Fri, 12 Sep 2014 12:38:57 +0400
Subject: RFR (S) 7901022: JMH runners should provide explicit timeout
	setting
In-Reply-To: <03BF9BD3-CF25-4BBF-818F-9A990AE876DB@yahoo.com>
References: <54106094.8060806@oracle.com> <5410654A.2060802@oracle.com>
	<541069C6.6050300@oracle.com> <54106C99.7040002@oracle.com>
	<54106D6C.3050609@oracle.com> <5410A9C9.4090607@oracle.com>
	<B1EE559C-BCC3-43F3-8E78-9FD15E412123@yahoo.com>
	<54116EF0.5080908@oracle.com>
	<00443A21-1E57-4CAB-94C5-A8747B1DF083@yahoo.com>
	<03BF9BD3-CF25-4BBF-818F-9A990AE876DB@yahoo.com>
Message-ID: <5412B121.8090601@oracle.com>

Thanks for testing, Dmitry!

Cheers,
-Aleksey.

On 09/12/2014 01:28 AM, vyazelenko at yahoo.com wrote:
> I have tested new timeout feature. It works as expected. 
> 
> Looking forward to 1.1 release! 
> 
> Regards,
> Dmitry
> 
> Sent from my iPhone
> 
>> On Sep 11, 2014, at 12:51, vyazelenko at yahoo.com wrote:
>>
>> Sure! Will test later today.
>>
>> Regards,
>> Dmitry
>>
>> Sent from my iPhone
>>
>>>> On Sep 11, 2014, at 11:44, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:
>>>>
>>>> On 09/11/2014 02:31 AM, vyazelenko at yahoo.com wrote:
>>>> Looks good! Great to see this issue being tackled!
>>>
>>> Thanks for reviews!
>>>
>>>> When will it be available to play with? ;)
>>>
>>> It is available in master now:
>>> http://hg.openjdk.java.net/code-tools/jmh/rev/2e7682a5da28
>>>
>>> Can you try this by Friday afternoon UTC? We will promote 1.1 this week
>>> then.
>>>
>>> -Aleksey.
>>>



From aleksey.shipilev at oracle.com  Fri Sep 12 14:12:51 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Fri, 12 Sep 2014 18:12:51 +0400
Subject: JMH 1.1
Message-ID: <5412FF63.6080202@oracle.com>

Hi,

JMH 1.1 is released and available at Maven Central, props to Evgeny as
usual!

Highlights:

 * -to/timeout() options are now available. Before that, runner guessed
the time to wait for the workload from the expected execution time. This
was confusing at least for SingleShot benchmarks, and also some users
needed explicit timeouts.
   https://bugs.openjdk.java.net/browse/CODETOOLS-7901022

 * All internal JMH settings are prefixed with "jmh." now. If you have
used "harness.executor" and "harness.executor.class", please update the
options accordingly, see the change:
   http://hg.openjdk.java.net/code-tools/jmh/rev/9a66e03cd539
   https://bugs.openjdk.java.net/browse/CODETOOLS-7901025

 * "-prof perf" was failing on some locales, fixed:
   https://bugs.openjdk.java.net/browse/CODETOOLS-7901030

Since the first two changes affect the behavior of some current
benchmarks, we have bumped the minor version.

Enjoy!

-Aleksey.


From aleksey.shipilev at oracle.com  Fri Sep 12 14:14:45 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Fri, 12 Sep 2014 14:14:45 +0000
Subject: hg: code-tools/jmh: 3 new changesets
Message-ID: <201409121414.s8CEEjO3004016@aojmv0008>

Changeset: 9b4287ecb19c
Author:    shade
Date:      2014-09-12 12:39 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/9b4287ecb19c

JMH v1.1.

! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml
! jmh-archetypes/pom.xml
! jmh-core-benchmarks/pom.xml
! jmh-core-ct/pom.xml
! jmh-core-it/pom.xml
! jmh-core/pom.xml
! jmh-generator-annprocess/pom.xml
! jmh-generator-asm/pom.xml
! jmh-generator-bytecode/pom.xml
! jmh-generator-reflection/pom.xml
! jmh-samples/pom.xml
! pom.xml

Changeset: c0bca98b301f
Author:    shade
Date:      2014-09-12 12:39 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/c0bca98b301f

Added tag 1.1 for changeset 9b4287ecb19c

! .hgtags

Changeset: 712b5ce908ca
Author:    shade
Date:      2014-09-12 12:39 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/712b5ce908ca

Continue in 1.2-SNAPSHOT.

! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml
! jmh-archetypes/pom.xml
! jmh-core-benchmarks/pom.xml
! jmh-core-ct/pom.xml
! jmh-core-it/pom.xml
! jmh-core/pom.xml
! jmh-generator-annprocess/pom.xml
! jmh-generator-asm/pom.xml
! jmh-generator-bytecode/pom.xml
! jmh-generator-reflection/pom.xml
! jmh-samples/pom.xml
! pom.xml


From vyazelenko at yahoo.com  Sat Sep 13 18:28:26 2014
From: vyazelenko at yahoo.com (vyazelenko at yahoo.com)
Date: Sat, 13 Sep 2014 20:28:26 +0200
Subject: RFR (S) 7901022: JMH runners should provide explicit timeout
	setting
In-Reply-To: <5412B121.8090601@oracle.com>
References: <54106094.8060806@oracle.com> <5410654A.2060802@oracle.com>
	<541069C6.6050300@oracle.com> <54106C99.7040002@oracle.com>
	<54106D6C.3050609@oracle.com> <5410A9C9.4090607@oracle.com>
	<B1EE559C-BCC3-43F3-8E78-9FD15E412123@yahoo.com>
	<54116EF0.5080908@oracle.com>
	<00443A21-1E57-4CAB-94C5-A8747B1DF083@yahoo.com>
	<03BF9BD3-CF25-4BBF-818F-9A990AE876DB@yahoo.com>
	<5412B121.8090601@oracle.com>
Message-ID: <4D6D344E-8BC2-4AFD-9A63-F3A751770F5D@yahoo.com>

I'm glad to help!

Regards,
Dmitry

Sent from my iPhone

> On Sep 12, 2014, at 10:38, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:
> 
> Thanks for testing, Dmitry!
> 
> Cheers,
> -Aleksey.
> 
>> On 09/12/2014 01:28 AM, vyazelenko at yahoo.com wrote:
>> I have tested new timeout feature. It works as expected. 
>> 
>> Looking forward to 1.1 release! 
>> 
>> Regards,
>> Dmitry
>> 
>> Sent from my iPhone
>> 
>>> On Sep 11, 2014, at 12:51, vyazelenko at yahoo.com wrote:
>>> 
>>> Sure! Will test later today.
>>> 
>>> Regards,
>>> Dmitry
>>> 
>>> Sent from my iPhone
>>> 
>>>>> On Sep 11, 2014, at 11:44, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:
>>>>> 
>>>>> On 09/11/2014 02:31 AM, vyazelenko at yahoo.com wrote:
>>>>> Looks good! Great to see this issue being tackled!
>>>> 
>>>> Thanks for reviews!
>>>> 
>>>>> When will it be available to play with? ;)
>>>> 
>>>> It is available in master now:
>>>> http://hg.openjdk.java.net/code-tools/jmh/rev/2e7682a5da28
>>>> 
>>>> Can you try this by Friday afternoon UTC? We will promote 1.1 this week
>>>> then.
>>>> 
>>>> -Aleksey.
> 
> 

From mlists at juma.me.uk  Sun Sep 14 14:30:48 2014
From: mlists at juma.me.uk (Ismael Juma)
Date: Sun, 14 Sep 2014 15:30:48 +0100
Subject: Minor fix to JMHSample_24_Inheritance.java
Message-ID: <CAD5tkZaLav9dtSVWkzAe=7uWOC8-DyOg91+VknH+NdE++TurMw@mail.gmail.com>

Hi all,

I include a one-liner fix to what looks like a bug
in JMHSample_24_Inheritance.java at the end of this email. Hope it helps.

Best,
Ismael

diff -r 712b5ce908ca
jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_24_Inheritance.java
---
a/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_24_Inheritance.jav
Fri
Sep 12 12:39:52 2014 +0400
+++
b/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_24_Inheritance.jav
Sun
Sep 14 15:23:31 2014 +0100
@@ -95,7 +95,7 @@
     public static class BenchmarkCos extends AbstractBenchmark {
         @Override
         protected double doWork() {
-            return Math.sin(x);
+            return Math.cos(x);
         }
     }

From aleksey.shipilev at oracle.com  Mon Sep 15 09:41:26 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Mon, 15 Sep 2014 09:41:26 +0000
Subject: hg: code-tools/jmh: Rebuild JMHSample_24_Inheritance.
Message-ID: <201409150941.s8F9fQvU016747@aojmv0008>

Changeset: 2856725c3a81
Author:    shade
Date:      2014-09-15 13:41 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/2856725c3a81

Rebuild JMHSample_24_Inheritance.

! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_24_Inheritance.java


From aleksey.shipilev at oracle.com  Mon Sep 15 09:43:10 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Mon, 15 Sep 2014 13:43:10 +0400
Subject: Minor fix to JMHSample_24_Inheritance.java
In-Reply-To: <CAD5tkZaLav9dtSVWkzAe=7uWOC8-DyOg91+VknH+NdE++TurMw@mail.gmail.com>
References: <CAD5tkZaLav9dtSVWkzAe=7uWOC8-DyOg91+VknH+NdE++TurMw@mail.gmail.com>
Message-ID: <5416B4AE.2050008@oracle.com>

Hi Ismael,

On 09/14/2014 06:30 PM, Ismael Juma wrote:
> I include a one-liner fix to what looks like a bug
> in JMHSample_24_Inheritance.java at the end of this email. Hope it helps.

Thanks, that sample required a few other touchups. I did those here:
  http://hg.openjdk.java.net/code-tools/jmh/rev/2856725c3a81

-Aleksey.

P.S. If you are planning to contribute the code to OpenJDK and/or get
the credit for them, please consider signing the OCA, as per
http://openjdk.java.net/contribute.



From mlists at juma.me.uk  Mon Sep 15 09:53:56 2014
From: mlists at juma.me.uk (Ismael Juma)
Date: Mon, 15 Sep 2014 10:53:56 +0100
Subject: Minor fix to JMHSample_24_Inheritance.java
In-Reply-To: <5416B4AE.2050008@oracle.com>
References: <CAD5tkZaLav9dtSVWkzAe=7uWOC8-DyOg91+VknH+NdE++TurMw@mail.gmail.com>
	<5416B4AE.2050008@oracle.com>
Message-ID: <CAD5tkZaVOg_kotWt-Tg7cHAG+j=bfZprC1y22--hMSw2AM4dLg@mail.gmail.com>

Comments inline.

On Mon, Sep 15, 2014 at 10:43 AM, Aleksey Shipilev <
aleksey.shipilev at oracle.com> wrote:
>
> Thanks, that sample required a few other touchups. I did those here:
>   http://hg.openjdk.java.net/code-tools/jmh/rev/2856725c3a81


Excellent, thanks.

-Aleksey.
>
> P.S. If you are planning to contribute the code to OpenJDK and/or get
> the credit for them, please consider signing the OCA, as per
> http://openjdk.java.net/contribute.
>

Understood. Will do that for the next contribution (if and when it happens).

Best,
Ismael

From aleksey.shipilev at oracle.com  Sun Sep 21 06:02:32 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Sun, 21 Sep 2014 06:02:32 +0000
Subject: hg: code-tools/jmh: JMHSample_03_States: make the test more robust by
	contending over volatile fields.
Message-ID: <201409210602.s8L62WtY013773@aojmv0008>

Changeset: f481bc602a57
Author:    shade
Date:      2014-09-21 10:02 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/f481bc602a57

JMHSample_03_States: make the test more robust by contending over volatile fields.

! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_03_States.java


From ecki at zusammenkunft.net  Tue Sep 23 22:38:10 2014
From: ecki at zusammenkunft.net (Bernd Eckenfels)
Date: Wed, 24 Sep 2014 00:38:10 +0200
Subject: Maven old "Simple" Archetype still visible
Message-ID: <20140924003810.00004c2e.ecki@zusammenkunft.net>

Hello,

Eclipse shows by default only the latest version of each maven
archetype. However it also shows me version 0.4.2 of
jmh-simple-benchmark-archetype as contained in 
http://repo1.maven.org/maven2/archetype-catalog.xml:

<archetype><groupId>org.openjdk.jmh</groupId><artifactId>jmh-simple-benchmark-archetype</artifactId><version>0.4.2</version><description>Basic
archetype for simple JMH-driven benchmark.</description></archetype>

I think this is an old abandoned name (for jmh-java-benchmark). I
wonder if there is actually a way to deprecate or even remove the older
archetypes from this catalog. Or at least make Eclipse see that it is
an older verison of the new name? 

Gruss
Bernd

From ecki at zusammenkunft.net  Wed Sep 24 00:12:50 2014
From: ecki at zusammenkunft.net (Bernd Eckenfels)
Date: Wed, 24 Sep 2014 02:12:50 +0200
Subject: JMH branch testing
Message-ID: <20140924021250.0000509f.ecki@zusammenkunft.net>

Hello,

looking for some ideas. I want to benchmark different implementations
for a "remeber maximum value" statistics function (which is used multi
threaded). One behaviour is of course, that the performance might
differ for numbers which are and which are not higher than the current
maximum. And then also the impact of varrying branches.

What I currently have is not very nice, as the distribution of the
random numbers (and the creation of them) does not execise real world
cases. Before I spent some time having a better distribution I wonder
if using a deterministic "measure <currentmax and >currentmax cases"
solution would be better or worse?

https://gist.github.com/ecki/8350a276caa444a62dce

Gruss
Bernd

From aleksey.shipilev at oracle.com  Wed Sep 24 06:23:59 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Wed, 24 Sep 2014 10:23:59 +0400
Subject: Maven old "Simple" Archetype still visible
In-Reply-To: <20140924003810.00004c2e.ecki@zusammenkunft.net>
References: <20140924003810.00004c2e.ecki@zusammenkunft.net>
Message-ID: <5422637F.5050008@oracle.com>

Hi Bernd,

On 09/24/2014 02:38 AM, Bernd Eckenfels wrote:
> I think this is an old abandoned name (for jmh-java-benchmark). I 
> wonder if there is actually a way to deprecate or even remove the
> older archetypes from this catalog.

Alas, there seems to be no way. Maven Central is not forgetting anything.

> Or at least make Eclipse see that it is an older verison of the new
> name?

Well, the stale version should be enough of the deterrent for using the
archetype?

-Aleksey.


From thomas.wuerthinger at oracle.com  Wed Sep 24 17:15:37 2014
From: thomas.wuerthinger at oracle.com (Thomas Wuerthinger)
Date: Wed, 24 Sep 2014 19:15:37 +0200
Subject: Dynamic languages on the JVM / jmh
In-Reply-To: <17386119-4377-4CAC-97F0-0008BBAF33F7@oracle.com>
References: <1E892363-DD73-4B8D-AC8C-17758AD16200@oracle.com>
	<EDD61DDD-C5AC-44DB-8C78-1F276F26A561@oracle.com>
	<54209405.5070205@oracle.com>
	<8B2D2A79-5EF9-4BCE-95F3-0361BC9861AC@oracle.com>
	<54216B5E.80600@oracle.com>
	<06727EC4-9190-4AF7-AD7F-128C2B35449F@oracle.com>
	<54217A42.3090105@oracle.com>
	<D3427EBC-3B4F-4A98-A06F-F00887D467BC@oracle.com>
	<542191E8.6040909@oracle.com>
	<17386119-4377-4CAC-97F0-0008BBAF33F7@oracle.com>
Message-ID: <D2B2726E-E4C5-4B49-80BF-6806F515395D@oracle.com>

Moving this to jmh-dev then. The discussion topic is whether support for dynamic language on the JVM to include jmh APIs can be useful and how this could be added.

With JavaScript as the example: One could imagine providing a clojure with the benchmark code together with a JSON object with configuration data for executing a benchmark. Internally, the clojure could be represented via a method handle and the configuration data via a hash map. This would be the scenario (a) that you outline below. I believe that it would be easier to use and also less likely to measure additional overheads such as the javax.script implementation of the dynamic language. What I read from your message is that you feel that this is out of scope for JMH.

- thomas


On 23 Sep 2014, at 17:36, Marcus Lagergren <marcus.lagergren at oracle.com> wrote:

> Guys, guys! This is my review thread for JDK-8025435. Start another one for this, please?
> 
> /M
> 
> On 23 Sep 2014, at 17:29, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:
> 
>> On 09/23/2014 05:58 PM, Thomas Wuerthinger wrote:
>>> I understood your remark. I just believe that this restriction is not
>>> fundamental to JMH and there *could* be support for languages that do
>>> not statically compile to Java bytecodes as well.
>> 
>> I think this thread side-tracks.
>> 
>> Bottom-line: There are only two ways to hook up dynamically-compiled
>> language runtime into JMH (say, JS engine), depending who is in control:
>> 
>> a) Ask JS engine to start executing the JS benchmark, *and* call back
>> into JMH with an AST/bytecode/whatever in hands, asking to measure it.
>> We would need to generate the synthetic Java code, compile it, load it,
>> and only then run it, calling back to JS runtime for execution of our
>> payload -- all in flight, and requiring the tight cohesion.
>> 
>> b) Ask JMH to compile and execute a Java benchmark, and call into JS
>> engine with a JS script in hands, asking to compile and run it. There,
>> users statically compile the benchmark JAR, and it runs with minimal
>> dynamic dances, also going through the standard APIs (javax.script).
>> 
>> While there is nothing fundamental preventing us from exploring the
>> route (a), it is not as practical as going the well-established route
>> (b). In other words, "could be done" != "should be done".
>> 
>> -Aleksey.
>> 
> 


From aleksey.shipilev at oracle.com  Wed Sep 24 17:34:47 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Wed, 24 Sep 2014 21:34:47 +0400
Subject: Dynamic languages on the JVM / jmh
In-Reply-To: <D2B2726E-E4C5-4B49-80BF-6806F515395D@oracle.com>
References: <1E892363-DD73-4B8D-AC8C-17758AD16200@oracle.com>
	<EDD61DDD-C5AC-44DB-8C78-1F276F26A561@oracle.com>
	<54209405.5070205@oracle.com>
	<8B2D2A79-5EF9-4BCE-95F3-0361BC9861AC@oracle.com>
	<54216B5E.80600@oracle.com>
	<06727EC4-9190-4AF7-AD7F-128C2B35449F@oracle.com>
	<54217A42.3090105@oracle.com>
	<D3427EBC-3B4F-4A98-A06F-F00887D467BC@oracle.com>
	<542191E8.6040909@oracle.com>
	<17386119-4377-4CAC-97F0-0008BBAF33F7@oracle.com>
	<D2B2726E-E4C5-4B49-80BF-6806F515395D@oracle.com>
Message-ID: <542300B7.2090305@oracle.com>

On 09/24/2014 09:15 PM, Thomas Wuerthinger wrote:
> With JavaScript as the example: One could imagine providing a clojure
> with the benchmark code together with a JSON object with
> configuration data for executing a benchmark. Internally, the clojure
> could be represented via a method handle and the configuration data
> via a hash map. This would be the scenario (a) that you outline
> below. I believe that it would be easier to use and also less likely
> to measure additional overheads such as the javax.script
> implementation of the dynamic language. What I read from your message
> is that you feel that this is out of scope for JMH.

Once again, we already have a plausible solution in JMH. There is
nothing out-of-scope, but IMO we should not waste time building the
alternative solution, especially when all the mess is visible even
before laying the first line of code. Of course, I might be mistaken,
but is an open-source world, and people might certainly try to prove me
wrong. With code.

Major caveat is that the inverted control dance -- from language runtime
to JMH generators, then to javac, then to JMH runners, and ultimately
back to language runtime -- would not be cleaner than just calling the
language runtime from JMH payload.

Minor caveat is that naive closure with benchmark payload is not enough.
This approach is nothing new, and I have yet to see the implementation
that is not broken in subtle ways, e.g:
 http://www.reddit.com/r/java/comments/299vlz/nanobench_tiny_benchmarking_framework_for_java_8/cijhn6z

I'm still trying to wrap my head around your persistence on this option.
You mention the "additional overheads" of javax.script, may I ask what
are those overheads? In other words, have you got the benchmarks to back
up your claim? May we see them? Are they represent the real problem
benchmark users face?

I am asking because we do successfully use javax.script APIs for
measuring Nashorn/JS benchmarks without a trouble, and therefore it is
strange for me to discuss the solution without understanding what
problem it tries to solve.

Thanks,
-Aleksey.


From thomas.wuerthinger at oracle.com  Wed Sep 24 19:51:11 2014
From: thomas.wuerthinger at oracle.com (Thomas Wuerthinger)
Date: Wed, 24 Sep 2014 21:51:11 +0200
Subject: Dynamic languages on the JVM / jmh
In-Reply-To: <542300B7.2090305@oracle.com>
References: <1E892363-DD73-4B8D-AC8C-17758AD16200@oracle.com>
	<EDD61DDD-C5AC-44DB-8C78-1F276F26A561@oracle.com>
	<54209405.5070205@oracle.com>
	<8B2D2A79-5EF9-4BCE-95F3-0361BC9861AC@oracle.com>
	<54216B5E.80600@oracle.com>
	<06727EC4-9190-4AF7-AD7F-128C2B35449F@oracle.com>
	<54217A42.3090105@oracle.com>
	<D3427EBC-3B4F-4A98-A06F-F00887D467BC@oracle.com>
	<542191E8.6040909@oracle.com>
	<17386119-4377-4CAC-97F0-0008BBAF33F7@oracle.com>
	<D2B2726E-E4C5-4B49-80BF-6806F515395D@oracle.com>
	<542300B7.2090305@oracle.com>
Message-ID: <32144476-3839-45EB-9E3E-8AA696C042B6@oracle.com>

Just brainstorming here from the viewpoint of a JavaScript developer. Of course there are existing workarounds. The recent Nashorn pushback that is doing micro benchmarking without using jmh made me think whether there could be a better solution. Going through javax.script can have high overheads depending on the language runtime implementation. And I really do not believe that it is convenient to create a new Java class with a boiler plate call to javax.script for each JavaScript micro benchmark. I would be interested to see the solution of Marcus? code using jmh for a comparison. Or maybe jmh is just not suitable for his problem.

If you do not think that this is an issue to be worth thinking about, it is fine too :).

- thomas

On 24 Sep 2014, at 19:34, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:

> On 09/24/2014 09:15 PM, Thomas Wuerthinger wrote:
>> With JavaScript as the example: One could imagine providing a clojure
>> with the benchmark code together with a JSON object with
>> configuration data for executing a benchmark. Internally, the clojure
>> could be represented via a method handle and the configuration data
>> via a hash map. This would be the scenario (a) that you outline
>> below. I believe that it would be easier to use and also less likely
>> to measure additional overheads such as the javax.script
>> implementation of the dynamic language. What I read from your message
>> is that you feel that this is out of scope for JMH.
> 
> Once again, we already have a plausible solution in JMH. There is
> nothing out-of-scope, but IMO we should not waste time building the
> alternative solution, especially when all the mess is visible even
> before laying the first line of code. Of course, I might be mistaken,
> but is an open-source world, and people might certainly try to prove me
> wrong. With code.
> 
> Major caveat is that the inverted control dance -- from language runtime
> to JMH generators, then to javac, then to JMH runners, and ultimately
> back to language runtime -- would not be cleaner than just calling the
> language runtime from JMH payload.
> 
> Minor caveat is that naive closure with benchmark payload is not enough.
> This approach is nothing new, and I have yet to see the implementation
> that is not broken in subtle ways, e.g:
> http://www.reddit.com/r/java/comments/299vlz/nanobench_tiny_benchmarking_framework_for_java_8/cijhn6z
> 
> I'm still trying to wrap my head around your persistence on this option.
> You mention the "additional overheads" of javax.script, may I ask what
> are those overheads? In other words, have you got the benchmarks to back
> up your claim? May we see them? Are they represent the real problem
> benchmark users face?
> 
> I am asking because we do successfully use javax.script APIs for
> measuring Nashorn/JS benchmarks without a trouble, and therefore it is
> strange for me to discuss the solution without understanding what
> problem it tries to solve.
> 
> Thanks,
> -Aleksey.
> 


From aleksey.shipilev at oracle.com  Thu Sep 25 11:34:28 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Thu, 25 Sep 2014 11:34:28 +0000
Subject: hg: code-tools/jmh: 7901042: Separate "Score" and "Score Error" by
	some symbol
Message-ID: <201409251134.s8PBYS0H009228@aojmv0008>

Changeset: 5d05466092df
Author:    shade
Date:      2014-09-25 15:33 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/5d05466092df

7901042: Separate "Score" and "Score Error" by some symbol

! jmh-core/src/main/java/org/openjdk/jmh/results/format/TextResultFormat.java


From aleksey.shipilev at oracle.com  Thu Sep 25 14:21:41 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Thu, 25 Sep 2014 14:21:41 +0000
Subject: hg: code-tools/jmh: 7901043: perfasm parsing should be more robust in
	a face of multiple compiler writers
Message-ID: <201409251421.s8PELfnT009832@aojmv0008>

Changeset: 60ad9c971a64
Author:    shade
Date:      2014-09-25 18:21 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/60ad9c971a64

7901043: perfasm parsing should be more robust in a face of multiple compiler writers

! jmh-core/src/main/java/org/openjdk/jmh/profile/LinuxPerfAsmProfiler.java


From aleksey.shipilev at oracle.com  Thu Sep 25 20:24:30 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Thu, 25 Sep 2014 20:24:30 +0000
Subject: hg: code-tools/jmh: 3 new changesets
Message-ID: <201409252024.s8PKOUYn013546@aojmv0008>

Changeset: 42c0ba005980
Author:    shade
Date:      2014-09-25 23:37 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/42c0ba005980

JMH v1.1.1.

! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml
! jmh-archetypes/pom.xml
! jmh-core-benchmarks/pom.xml
! jmh-core-ct/pom.xml
! jmh-core-it/pom.xml
! jmh-core/pom.xml
! jmh-generator-annprocess/pom.xml
! jmh-generator-asm/pom.xml
! jmh-generator-bytecode/pom.xml
! jmh-generator-reflection/pom.xml
! jmh-samples/pom.xml
! pom.xml

Changeset: c33fe1b6bcc8
Author:    shade
Date:      2014-09-25 23:37 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/c33fe1b6bcc8

Added tag 1.1.1 for changeset 42c0ba005980

! .hgtags

Changeset: 7c70f079c95c
Author:    shade
Date:      2014-09-25 23:37 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/7c70f079c95c

Continue in 1.2-SNAPSHOT.

! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml
! jmh-archetypes/pom.xml
! jmh-core-benchmarks/pom.xml
! jmh-core-ct/pom.xml
! jmh-core-it/pom.xml
! jmh-core/pom.xml
! jmh-generator-annprocess/pom.xml
! jmh-generator-asm/pom.xml
! jmh-generator-bytecode/pom.xml
! jmh-generator-reflection/pom.xml
! jmh-samples/pom.xml
! pom.xml


From aleksey.shipilev at oracle.com  Thu Sep 25 20:27:52 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Fri, 26 Sep 2014 00:27:52 +0400
Subject: JMH 1.1.1
Message-ID: <54247AC8.1070403@oracle.com>

Hi,

JMH 1.1.1 is released and available at Maven Central (thanks to Evgeny).
This is a maintenance release, nothing very important in it, no major
bugfixes, just polishing:

 * https://bugs.openjdk.java.net/browse/CODETOOLS-7901042:
    We now separate score and score error with \plusminus sign. This
    plays well when you are copying the final lines without a header,
    or paste it into a monospace-hostile destination. Example:
      o.o.j.s.JMHSample_01_HW.wellHelloThere avgt 3 0.252 ? 0.023 ns/op

 * https://bugs.openjdk.java.net/browse/CODETOOLS-7901043:
    -prof perfasm should now be much more robust with figuring out the
    Java symbol names, and be more accurate with disambiguating the
    output from different compiler writer threads writing the log. If
    you ever experienced trouble with noise and/or outright lies in
    perfasm profiles (we did), update and try again.

 * Minor touchups in Samples: fixing an odd typo reported by Ismael
   Juma in JMHSample_24_Inheritance; and dodging weird performance
   issue with JMHSample_03_States reported on StackOverflow.

Enjoy!

Thanks,
-Aleksey.