From vyazelenko at yahoo.com  Fri Aug  1 10:29:58 2014
From: vyazelenko at yahoo.com (Dmitry Vyazelenko)
Date: Fri, 1 Aug 2014 12:29:58 +0200
Subject: Multiple JVMs, different numbers of threads and BenchmarkMode
Message-ID: <D32424E3-5F8C-42E4-9B13-36FFFC7E8C8C@yahoo.com>

Hi all,

I have several questions about JMH usage. In particular I?m interested in:
1) How do you handle need to run benchmarks on multiple JVMs
2) Using different  number of threads for the same benchmarks
3) Choosing BenchmarkMode/iteration config for dynamic workloads

Here what I?ve been doing:
1) To run on multiple JVMs I?m using -jvm parameter. Something like this:
  java -jar target/benchmarks.jar -jvm <path>/jdk1.7.0_65/bin/java -rf CSV -rff jdk1.7.0_65_results.csv
  java -jar target/benchmarks.jar -jvm <path>/jdk1.8.0_11/bin/java -rf CSV -rff jdk1.8.0_11_results.csv
I?ve been wondering is there any better way to do that? I mean in version 0.9.3 ability to specify JVM was added to the @Fork annotation. However it only allows single value to be specified (same as -jvm parameter). I was thinking maybe it should support multiple JVMs? This way one could start benchmark once and get results from all JVMs. Right now it requires writing bash scripts to run benchmarks on multiple JVMs.
I understand that this change would require changing output table to also include VM with which benchmarks were executed. Does anyone else think something like that would be a good idea?

2) Lately I found myself in a situation in which I want to execute the same benchmark with different number of threads. Up to this point I know of 3 different ways to do that:
    a) Invoke benchmark with -t parameter with given number of threads. This works fine for single benchmark class but does not work when I run benchmarks suite, because this parameter applies to all benchmarks.
    b) Using @Threads annotation on a class . The problem is that it only allows single value to be specified. Which means that either one has to change annotation and re-build benchmark. Or sub-class benchmark class and redefine @Threads annotation with different value.
    c) Using @Threads annotation on a method. This is probably the cleanest solution as I?m able to declare same method N times with different number of threads, The drawback is that one have to repeat benchmark method code or extract it into helper method and invoke from real benchmark methods annotated with @Benchmark.
I think it would be really great if JMH would allow specifying multiple values for the @Threads annotation. And will execute benchmarks with the number of threads defined there. Of course reporting of the results should then include number of threads that were used to produce result.

3) This last point is more of a question on which BenchmarkMode and/or iteration configuration should be used when work inside benchmark method is not constant but instead depends on the parameter for current run.
For example I have a benchmark that measures Map.put() performance for different number of "batch size?. Where batch size is a parameter with values 10, 100, 1000?10000000. The benchmark method is then looping over array of keys which are allocated based on the batch size and invoke Map.put(), e.g.:
    @Benchmark
    public void put(ThreadData data) {
        for (K key : data.keys) {
            map.put(key, data.value);
        }
    }
Now because of dynamic nature of the work done by a benchmark method what would be the best BenchmarkMode to use and how the best to configuration @Warmup/@Measurement iterations?
What I tried so far was to use AverageTime mode with default (i.e. time-based) configurations for warmup and measurement iterations. However I see the following issues: for small batches benchmark method is invoked lots of time but as the batch size increase it leads to maybe a single invocation. Also the target method (i.e. Map.put()) is invoked more time then expected, i.e. it should be called only ?batch size?-number of times but due to time-based nature of iterations it will be called much more often.
Therefore I?ve been looking for ways to mitigate these problems. To solve problem of ensuring fixed work per benchmark method I?m considering using either SingleShotTime mode or ?batchSize? feature to configure iterations (http://hg.openjdk.java.net/code-tools/jmh/file/6354acecccb7/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_26_BatchSize.java). 
 - While the former will allow keeping code as-is I?ll have to deal with the timers overhead on small batch sizes and also use bigger number of warmup and measurement iterations. Also SingleShotTime won?t aggregate any results for me which will complicate post-processing of the results. 
 - With the latter approach I?ll have to completely re-think my code, eliminate loops from measurement methods and rely on on fixed number of invocation defined by the ?batchSize? parameter (as shown in the http://hg.openjdk.java.net/code-tools/jmh/file/6354acecccb7/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_26_BatchSize.java example) of the @Warmaup/@Measurement annotation (or command flag). It would also mean that I can?t use my own ?batch size? parameter anymore. That would also mean that I won?t be able to run my benchmarks with set of values and instead will have to invoke them multiple times each time with different batchSize. And that would require writing script to drive benchmarks again.
Maybe it would be a good idea to allow specifying array of batchSize values via command line or annotations. At least this will eliminate scripting part but will require adding this information to the results table (e.g. batchSize 10 value1 batchSize 100 value2 etc.).

Thanks for the awesome tool and your help in advance!

Best regards,
Dmitry Vyazelenko

From Li.Guoyu at blackboard.com  Fri Aug  1 02:05:12 2014
From: Li.Guoyu at blackboard.com (Li Guoyu)
Date: Fri, 1 Aug 2014 02:05:12 +0000
Subject: Benchmark iteration time of SS mode when measurement iteration time
	is specifed
Message-ID: <D001171B.A5A5%li.guoyu@blackboard.com>

Hi Everyone,

The benchmark total time of the following dummy test doesn?t make sense to me, the benchmark duration is 21 seconds when the actual benchmark only take 10 seconds, it seems the benchmark duration depends on the measurement time annotation specified, is this expected?

BTW, the benchmark duration is 11 seconds when measurement time annotation is removed, it works pretty work.

I specified the measurement time for ss benchmark because one of my test took more than 10 minutes to complete, and it seems the max time is 10 minutes and otherwise it will be interrupted. Specifying the measurement time to 30 minutes resolved the interrupted exception, but every benchmark took 30 minutes to complete although the actual time needed is 10+ minutes.


@State( Scope.Benchmark )

@BenchmarkMode( Mode.SingleShotTime )

@Fork( value = 1 )

@OutputTimeUnit( TimeUnit.MILLISECONDS )

@Warmup( iterations = 0 )

@Measurement( iterations = 1, time = 20 )

public class JmhExample

{


  @Setup( Level.Trial )

  public void setup()

  {

    System.out.println( "+++++++Setup" );

  }


  @Benchmark

  public void test() throws Exception

  {

    System.out.println( "asdf" );

    Thread.sleep( 10 * 1000 );

  }


  public static void main( String[] args ) throws Exception

  {

    Options opt = new OptionsBuilder().include( ".*" + JmhExample.class.getSimpleName() + ".*" ).build();

    new Runner( opt ).run();

  }

}


# Run complete. Total time: 00:00:21


Benchmark             Mode   Samples        Score  Score error    Units

b.JmhExample.test       ss         1    10001.543          NaN       ms



This email and any attachments may contain confidential and proprietary information of Blackboard that is for the sole use of the intended recipient. If you are not the intended recipient, disclosure, copying, re-distribution or other use of any of this information is strictly prohibited. Please immediately notify the sender and delete this transmission if you received this email in error.

From vyazelenko at yahoo.com  Fri Aug  1 15:14:31 2014
From: vyazelenko at yahoo.com (Dmitry Vyazelenko)
Date: Fri, 1 Aug 2014 17:14:31 +0200
Subject: Benchmark iteration time of SS mode when measurement iteration
	time is specifed
In-Reply-To: <D001171B.A5A5%li.guoyu@blackboard.com>
References: <D001171B.A5A5%li.guoyu@blackboard.com>
Message-ID: <E10C1AA8-09CF-4BB1-B39B-A3CD88C736FA@yahoo.com>

Hi Li,

Actually your benchmark was executed in 10 seconds (i.e. Score reported as 10001.543 ms). When you use SingleShotTime it will ignore time specified and only do a single invocation of the benchmark method (it can actually do more invocation if batchSize is specified). 
What you see as 21 seconds total time is just a reporting bug. Apparently time element from annotation is used to print that?

Best regards,
Dmitry Vyazelenko

On Aug 1, 2014, at 04:05 , Li Guoyu <Li.Guoyu at blackboard.com> wrote:

> Hi Everyone,
> 
> The benchmark total time of the following dummy test doesn?t make sense to me, the benchmark duration is 21 seconds when the actual benchmark only take 10 seconds, it seems the benchmark duration depends on the measurement time annotation specified, is this expected?
> 
> BTW, the benchmark duration is 11 seconds when measurement time annotation is removed, it works pretty work.
> 
> I specified the measurement time for ss benchmark because one of my test took more than 10 minutes to complete, and it seems the max time is 10 minutes and otherwise it will be interrupted. Specifying the measurement time to 30 minutes resolved the interrupted exception, but every benchmark took 30 minutes to complete although the actual time needed is 10+ minutes.
> 
> 
> @State( Scope.Benchmark )
> 
> @BenchmarkMode( Mode.SingleShotTime )
> 
> @Fork( value = 1 )
> 
> @OutputTimeUnit( TimeUnit.MILLISECONDS )
> 
> @Warmup( iterations = 0 )
> 
> @Measurement( iterations = 1, time = 20 )
> 
> public class JmhExample
> 
> {
> 
> 
>  @Setup( Level.Trial )
> 
>  public void setup()
> 
>  {
> 
>    System.out.println( "+++++++Setup" );
> 
>  }
> 
> 
>  @Benchmark
> 
>  public void test() throws Exception
> 
>  {
> 
>    System.out.println( "asdf" );
> 
>    Thread.sleep( 10 * 1000 );
> 
>  }
> 
> 
>  public static void main( String[] args ) throws Exception
> 
>  {
> 
>    Options opt = new OptionsBuilder().include( ".*" + JmhExample.class.getSimpleName() + ".*" ).build();
> 
>    new Runner( opt ).run();
> 
>  }
> 
> }
> 
> 
> # Run complete. Total time: 00:00:21
> 
> 
> Benchmark             Mode   Samples        Score  Score error    Units
> 
> b.JmhExample.test       ss         1    10001.543          NaN       ms
> 
> 
> 
> This email and any attachments may contain confidential and proprietary information of Blackboard that is for the sole use of the intended recipient. If you are not the intended recipient, disclosure, copying, re-distribution or other use of any of this information is strictly prohibited. Please immediately notify the sender and delete this transmission if you received this email in error.


From vyazelenko at yahoo.com  Fri Aug  1 16:56:00 2014
From: vyazelenko at yahoo.com (Dmitry Vyazelenko)
Date: Fri, 1 Aug 2014 18:56:00 +0200
Subject: Benchmark iteration time of SS mode when measurement iteration
	time is specifed
In-Reply-To: <E10C1AA8-09CF-4BB1-B39B-A3CD88C736FA@yahoo.com>
References: <D001171B.A5A5%li.guoyu@blackboard.com>
	<E10C1AA8-09CF-4BB1-B39B-A3CD88C736FA@yahoo.com>
Message-ID: <04CC3449-3362-4F48-84A0-D43B18DACF73@yahoo.com>

Actually it seem that execution time is 20 seconds. So overall execution time seems to be based on the annotation setting even pin SingleShotTime case.

Running: time java -jar target/benchmarks.jar .*Jmh.*
Yields this:
# Run complete. Total time: 00:00:20

Benchmark              Mode  Samples      Score  Score error  Units
j.s.JmhExample.test      ss        1  10001.097          NaN     ms

real	0m20.466s
user	0m0.853s
sys	0m0.137s

Best regards,
Dmitry Vyazelenko

On Aug 1, 2014, at 17:14 , Dmitry Vyazelenko <vyazelenko at yahoo.com> wrote:

> Hi Li,
> 
> Actually your benchmark was executed in 10 seconds (i.e. Score reported as 10001.543 ms). When you use SingleShotTime it will ignore time specified and only do a single invocation of the benchmark method (it can actually do more invocation if batchSize is specified). 
> What you see as 21 seconds total time is just a reporting bug. Apparently time element from annotation is used to print that?
> 
> Best regards,
> Dmitry Vyazelenko
> 
> On Aug 1, 2014, at 04:05 , Li Guoyu <Li.Guoyu at blackboard.com> wrote:
> 
>> Hi Everyone,
>> 
>> The benchmark total time of the following dummy test doesn?t make sense to me, the benchmark duration is 21 seconds when the actual benchmark only take 10 seconds, it seems the benchmark duration depends on the measurement time annotation specified, is this expected?
>> 
>> BTW, the benchmark duration is 11 seconds when measurement time annotation is removed, it works pretty work.
>> 
>> I specified the measurement time for ss benchmark because one of my test took more than 10 minutes to complete, and it seems the max time is 10 minutes and otherwise it will be interrupted. Specifying the measurement time to 30 minutes resolved the interrupted exception, but every benchmark took 30 minutes to complete although the actual time needed is 10+ minutes.
>> 
>> 
>> @State( Scope.Benchmark )
>> 
>> @BenchmarkMode( Mode.SingleShotTime )
>> 
>> @Fork( value = 1 )
>> 
>> @OutputTimeUnit( TimeUnit.MILLISECONDS )
>> 
>> @Warmup( iterations = 0 )
>> 
>> @Measurement( iterations = 1, time = 20 )
>> 
>> public class JmhExample
>> 
>> {
>> 
>> 
>> @Setup( Level.Trial )
>> 
>> public void setup()
>> 
>> {
>> 
>>   System.out.println( "+++++++Setup" );
>> 
>> }
>> 
>> 
>> @Benchmark
>> 
>> public void test() throws Exception
>> 
>> {
>> 
>>   System.out.println( "asdf" );
>> 
>>   Thread.sleep( 10 * 1000 );
>> 
>> }
>> 
>> 
>> public static void main( String[] args ) throws Exception
>> 
>> {
>> 
>>   Options opt = new OptionsBuilder().include( ".*" + JmhExample.class.getSimpleName() + ".*" ).build();
>> 
>>   new Runner( opt ).run();
>> 
>> }
>> 
>> }
>> 
>> 
>> # Run complete. Total time: 00:00:21
>> 
>> 
>> Benchmark             Mode   Samples        Score  Score error    Units
>> 
>> b.JmhExample.test       ss         1    10001.543          NaN       ms
>> 
>> 
>> 
>> This email and any attachments may contain confidential and proprietary information of Blackboard that is for the sole use of the intended recipient. If you are not the intended recipient, disclosure, copying, re-distribution or other use of any of this information is strictly prohibited. Please immediately notify the sender and delete this transmission if you received this email in error.
> 


From aleksey.shipilev at oracle.com  Sun Aug  3 15:51:43 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Sun, 03 Aug 2014 15:51:43 +0000
Subject: hg: code-tools/jmh: runners: SingleShot mode should not sleep in
	control thread for run time.
Message-ID: <201408031551.s73FphTP014006@aojmv0008>

Changeset: 5d6eee917c11
Author:    shade
Date:      2014-08-03 19:48 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/5d6eee917c11

runners: SingleShot mode should not sleep in control thread for run time.

! jmh-core/src/main/java/org/openjdk/jmh/runner/LoopBenchmarkHandler.java


From aleksey.shipilev at oracle.com  Sun Aug  3 15:53:44 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Sun, 03 Aug 2014 19:53:44 +0400
Subject: Benchmark iteration time of SS mode when measurement iteration
	time is specifed
In-Reply-To: <D001171B.A5A5%li.guoyu@blackboard.com>
References: <D001171B.A5A5%li.guoyu@blackboard.com>
Message-ID: <53DE5B08.4020109@oracle.com>

Hi Li,

On 08/01/2014 06:05 AM, Li Guoyu wrote:
> I specified the measurement time for ss benchmark because one of my
> test took more than 10 minutes to complete, and it seems the max time
> is 10 minutes and otherwise it will be interrupted. Specifying the
> measurement time to 30 minutes resolved the interrupted exception,
> but every benchmark took 30 minutes to complete although the actual
> time needed is 10+ minutes.

Ah yes, thanks. In SS mode, specifying the run time is indeed lifts the
timeout. But, alas, the generic codepath that executes for all
benchmarks sleeps for the run time. This fits all the benchmark modes,
except from SS. Fixed:
 http://hg.openjdk.java.net/code-tools/jmh/rev/5d6eee917c11

Can you build 1.0-SNAPSHOT and test with it?

-Aleksey.

From aleksey.shipilev at oracle.com  Sun Aug  3 16:20:01 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Sun, 03 Aug 2014 20:20:01 +0400
Subject: Multiple JVMs, different numbers of threads and BenchmarkMode
In-Reply-To: <D32424E3-5F8C-42E4-9B13-36FFFC7E8C8C@yahoo.com>
References: <D32424E3-5F8C-42E4-9B13-36FFFC7E8C8C@yahoo.com>
Message-ID: <53DE6131.70208@oracle.com>

Hi Dmitry,

First, uncalled-for formatting advice: a) use text-wrapping tools in
your messenger, b) use an empty line to demarcate paragraphs. Otherwise,
your messages look like these wide walls of text in mailing clients:
 http://mail.openjdk.java.net/pipermail/jmh-dev/2014-August/001195.html

...with no hope for auto-reformat.


On 08/01/2014 02:29 PM, Dmitry Vyazelenko wrote:
> I have several questions about JMH usage. In particular I?m interested in:
> 1) How do you handle need to run benchmarks on multiple JVMs
> 2) Using different  number of threads for the same benchmarks
> 3) Choosing BenchmarkMode/iteration config for dynamic workloads

The rule of thumb is: annotations cover the generic cases; if current
annotations are too constraining, fall back to Java API. You should be
able to code non-trivial things there without much trouble. I tend to
think all three questions are answered by Java API.

A few suggestions otherwise:

> Here what I?ve been doing:
> 1) To run on multiple JVMs I?m using -jvm parameter. Something like this:
>   java -jar target/benchmarks.jar -jvm <path>/jdk1.7.0_65/bin/java -rf CSV -rff jdk1.7.0_65_results.csv
>   java -jar target/benchmarks.jar -jvm <path>/jdk1.8.0_11/bin/java -rf CSV -rff jdk1.8.0_11_results.csv

> I?ve been wondering is there any better way to do that? 

Most of us are invoking the benchmark.jar with the target VM to make
sure: a) we indeed running the requested VM, otherwise we need to
cross-check if JMH is actually running the given one; b) ensures
compatibility between hosted and forked VM. Granted, both are guaranteed
in sane environments, but this is defense-in-depth concern.

> I mean in version 0.9.3 ability to specify JVM was added to the @Fork
> annotation. However it only allows single value to be specified (same
> as -jvm parameter). I was thinking maybe it should support multiple
> JVMs?

...

> I understand that this change would require changing output table to
also include VM with which benchmarks were executed. Does anyone else
think something like that would be a good idea?

I think that we would need to rethink how benchmark parameters with
named annotations interact with @Param. It would be profitable to
adopt/convert named params to @Param-s, which will give us what you
suggest for free and in a clean manner.

> I think it would be really great if JMH would allow specifying 
> multiple values for the @Threads annotation. And will execute
> benchmarks with the numberof threads defined there. Of course
> reporting of the results should then include number of threads that
> were used to produce result.

Ditto, see above. Requires the connection with @Param-s.


> 3) This last point is more of a question on which BenchmarkMode
> and/or iteration configuration should be used when work inside
> benchmark method is not constant but instead depends on the parameter
> for current run.



> Now because of dynamic nature of the work done by a benchmark method
> what would be the best BenchmarkMode to use and how the best to
> configuration @Warmup/@Measurement iterations?

AverageTime or Throughput, obviously. I'm not sure how much of a problem
that is, actually. We should not care how many times @Benchmark was
called in throughput modes, we should only care it was called *enough*
times.

Sure, for large benchmarks warmup and measurement needs adjusting, but
that's in users' hands. We could theoretically go for something
SPECjvm2008 does, and enforce the minimum number of @Benchmark calls to
constitute the iteration. SPECjvm2008 experience tells us it can be
frustrating to users who need for predictable time, and those who have
large workloads, but don't know about that yet -- wasting hours waiting
for benchmark to complete.


> It would also mean that I can?t use my own ?batch size? parameter
> anymore. That would also mean that I won?t be able to run my
> benchmarks with set of values and instead will have to invoke them
> multiple times each time with different batchSize. And that would
> require writing script to drive benchmarks again.

See, you need advanced behavior with scripting. Java API provides you
with the opportunity to write "scripts" in Java. It should not be the
burden of JMH runner to encompass the user-defined benchmarking logic
for users. Instead, we enable users to tell JMH what to do in
coarse-grained (via annotations) or in fine-grained (via
API/command-line) fashion.

> Maybe it would be a good idea to allow specifying array of batchSize
> values via command line or annotations. At least this will eliminate
> scripting part but will require adding this information to the
> results table (e.g. batchSize 10 value1 batchSize 100 value2 etc.).

See above. Needs a proper connection with @Param-s.

Thanks,
-Aleksey.


From Li.Guoyu at blackboard.com  Mon Aug  4 06:30:29 2014
From: Li.Guoyu at blackboard.com (Li Guoyu)
Date: Mon, 4 Aug 2014 06:30:29 +0000
Subject: Benchmark iteration time of SS mode when measurement iteration
	time is specifed
In-Reply-To: <53DE5B08.4020109@oracle.com>
References: <D001171B.A5A5%li.guoyu@blackboard.com>
	<53DE5B08.4020109@oracle.com>
Message-ID: <D005464B.A6CC%li.guoyu@blackboard.com>

Hi Aleksey,

It works pretty good, thanks for the quick fix.

Thanks,
Li Guoyu


? 8/3/14, 11:53 PM? "Aleksey Shipilev" <aleksey.shipilev at oracle.com> ??:

>Hi Li,
>
>On 08/01/2014 06:05 AM, Li Guoyu wrote:
>> I specified the measurement time for ss benchmark because one of my
>> test took more than 10 minutes to complete, and it seems the max time
>> is 10 minutes and otherwise it will be interrupted. Specifying the
>> measurement time to 30 minutes resolved the interrupted exception,
>> but every benchmark took 30 minutes to complete although the actual
>> time needed is 10+ minutes.
>
>Ah yes, thanks. In SS mode, specifying the run time is indeed lifts the
>timeout. But, alas, the generic codepath that executes for all
>benchmarks sleeps for the run time. This fits all the benchmark modes,
>except from SS. Fixed:
> http://hg.openjdk.java.net/code-tools/jmh/rev/5d6eee917c11
>
>Can you build 1.0-SNAPSHOT and test with it?
>
>-Aleksey.


This email and any attachments may contain confidential and proprietary information of Blackboard that is for the sole use of the intended recipient. If you are not the intended recipient, disclosure, copying, re-distribution or other use of any of this information is strictly prohibited. Please immediately notify the sender and delete this transmission if you received this email in error.

From ecki at zusammenkunft.net  Mon Aug  4 07:22:30 2014
From: ecki at zusammenkunft.net (Bernd)
Date: Mon, 4 Aug 2014 09:22:30 +0200
Subject: Multiple JVMs, different numbers of threads and BenchmarkMode
In-Reply-To: <53DE6131.70208@oracle.com>
References: <D32424E3-5F8C-42E4-9B13-36FFFC7E8C8C@yahoo.com>
	<53DE6131.70208@oracle.com>
Message-ID: <CABOR3+yPo+p4F8sRsLVp_Lx4UnXUMa1GDBmrQkTpFJ=4Ujnu9g@mail.gmail.com>

Am 03.08.2014 18:30 schrieb "Aleksey Shipilev" <aleksey.shipilev at oracle.com
>:

> AverageTime or Throughput, obviously. I'm not sure how much of a problem
> that is, actually. We should not care how many times @Benchmark was
> called in throughput modes, we should only care it was called *enough*
> times.

Actually this is something, I was wondering about: when minimizing runtime
of warmup and benchmark I typically do some test runs to get a feeling for
the throughput and then calculate the number of repetitions for the chosen
time in head. I wonder why the number of executions for each iteration is
not printed.

Maybe if you dont want to clutter the output we can only print it if the
number is rather low?

Gruss
Bernd

From vyazelenko at yahoo.com  Mon Aug  4 20:48:09 2014
From: vyazelenko at yahoo.com (Dmitry Vyazelenko)
Date: Mon, 4 Aug 2014 22:48:09 +0200
Subject: Multiple JVMs, different numbers of threads and BenchmarkMode
In-Reply-To: <53DE6131.70208@oracle.com>
References: <D32424E3-5F8C-42E4-9B13-36FFFC7E8C8C@yahoo.com>
	<53DE6131.70208@oracle.com>
Message-ID: <3FADC9EA-BF57-4014-BF98-EC8B0752A0B7@yahoo.com>

Hi Aleksey,

Thanks for the reply. I?ll do my best formatting-wise in the future.
For other things, see my comment inline.


On Aug 3, 2014, at 18:20 , Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:

> Hi Dmitry,
> 
> First, uncalled-for formatting advice: a) use text-wrapping tools in
> your messenger, b) use an empty line to demarcate paragraphs. Otherwise,
> your messages look like these wide walls of text in mailing clients:
> http://mail.openjdk.java.net/pipermail/jmh-dev/2014-August/001195.html
> 
> ...with no hope for auto-reformat.
> 
> 
> On 08/01/2014 02:29 PM, Dmitry Vyazelenko wrote:
>> I have several questions about JMH usage. In particular I?m interested in:
>> 1) How do you handle need to run benchmarks on multiple JVMs
>> 2) Using different  number of threads for the same benchmarks
>> 3) Choosing BenchmarkMode/iteration config for dynamic workloads
> 
> The rule of thumb is: annotations cover the generic cases; if current
> annotations are too constraining, fall back to Java API. You should be
> able to code non-trivial things there without much trouble. I tend to
> think all three questions are answered by Java API.
> 
> A few suggestions otherwise:
> 
>> Here what I?ve been doing:
>> 1) To run on multiple JVMs I?m using -jvm parameter. Something like this:
>>  java -jar target/benchmarks.jar -jvm <path>/jdk1.7.0_65/bin/java -rf CSV -rff jdk1.7.0_65_results.csv
>>  java -jar target/benchmarks.jar -jvm <path>/jdk1.8.0_11/bin/java -rf CSV -rff jdk1.8.0_11_results.csv
> 
>> I?ve been wondering is there any better way to do that? 
> 
> Most of us are invoking the benchmark.jar with the target VM to make
> sure: a) we indeed running the requested VM, otherwise we need to
> cross-check if JMH is actually running the given one; b) ensures
> compatibility between hosted and forked VM. Granted, both are guaranteed
> in sane environments, but this is defense-in-depth concern.
I?ve been doing the same until recently when I discovered -jvm parameter. So it seems 
to be a better way anyway than specifying -jvm parameter. 
> 
>> I mean in version 0.9.3 ability to specify JVM was added to the @Fork
>> annotation. However it only allows single value to be specified (same
>> as -jvm parameter). I was thinking maybe it should support multiple
>> JVMs?
> 
> ...
> 
>> I understand that this change would require changing output table to
> also include VM with which benchmarks were executed. Does anyone else
> think something like that would be a good idea?
> 
> I think that we would need to rethink how benchmark parameters with
> named annotations interact with @Param. It would be profitable to
> adopt/convert named params to @Param-s, which will give us what you
> suggest for free and in a clean manner.
That would be great if named annotations and @Param can be unified. ;-)
> 
>> I think it would be really great if JMH would allow specifying 
>> multiple values for the @Threads annotation. And will execute
>> benchmarks with the numberof threads defined there. Of course
>> reporting of the results should then include number of threads that
>> were used to produce result.
> 
> Ditto, see above. Requires the connection with @Param-s.
> 
> 
>> 3) This last point is more of a question on which BenchmarkMode
>> and/or iteration configuration should be used when work inside
>> benchmark method is not constant but instead depends on the parameter
>> for current run.
> 
> 
> 
>> Now because of dynamic nature of the work done by a benchmark method
>> what would be the best BenchmarkMode to use and how the best to
>> configuration @Warmup/@Measurement iterations?
> 
> AverageTime or Throughput, obviously. I'm not sure how much of a problem
> that is, actually. We should not care how many times @Benchmark was
> called in throughput modes, we should only care it was called *enough*
> times.
Well, my use-case here is different one. I actually want to measure certain amount
of work/invocations. For example I want to benchmark how long it takes to add 1000
elements into my data structure. If I just rely on time-based modes like AverageTime
it means that my @Benchmark will be called unknown/unpredictable number of times.
If inside this method I put a loop which adds 1000 elements into collection then this
would not yield expected results (i.e. my collections will contain millions of entries and
not 1000).

Therefore I think that I should use @Warmup/@Measurment annotations with batchSize
defined and SingleShotTime mode, i.e. using similar technique shown in the 
JMHSample_26_BatchSize.java example. And of course I?ll have to use Java API to code
proper invocation of my benchmarks with batchSize set dynamically.
> 
> Sure, for large benchmarks warmup and measurement needs adjusting, but
> that's in users' hands. We could theoretically go for something
> SPECjvm2008 does, and enforce the minimum number of @Benchmark calls to
> constitute the iteration. SPECjvm2008 experience tells us it can be
> frustrating to users who need for predictable time, and those who have
> large workloads, but don't know about that yet -- wasting hours waiting
> for benchmark to complete.
> 
> 
>> It would also mean that I can?t use my own ?batch size? parameter
>> anymore. That would also mean that I won?t be able to run my
>> benchmarks with set of values and instead will have to invoke them
>> multiple times each time with different batchSize. And that would
>> require writing script to drive benchmarks again.
> 
> See, you need advanced behavior with scripting. Java API provides you
> with the opportunity to write "scripts" in Java. It should not be the
> burden of JMH runner to encompass the user-defined benchmarking logic
> for users. Instead, we enable users to tell JMH what to do in
> coarse-grained (via annotations) or in fine-grained (via
> API/command-line) fashion.
> 
>> Maybe it would be a good idea to allow specifying array of batchSize
>> values via command line or annotations. At least this will eliminate
>> scripting part but will require adding this information to the
>> results table (e.g. batchSize 10 value1 batchSize 100 value2 etc.).
> 
> See above. Needs a proper connection with @Param-s.
> 
> Thanks,
> -Aleksey.
> 


Regards,
Dmitry


From aleksey.shipilev at oracle.com  Tue Aug  5 09:48:43 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Tue, 05 Aug 2014 13:48:43 +0400
Subject: Multiple JVMs, different numbers of threads and BenchmarkMode
In-Reply-To: <CABOR3+yPo+p4F8sRsLVp_Lx4UnXUMa1GDBmrQkTpFJ=4Ujnu9g@mail.gmail.com>
References: <D32424E3-5F8C-42E4-9B13-36FFFC7E8C8C@yahoo.com>	<53DE6131.70208@oracle.com>
	<CABOR3+yPo+p4F8sRsLVp_Lx4UnXUMa1GDBmrQkTpFJ=4Ujnu9g@mail.gmail.com>
Message-ID: <53E0A87B.80108@oracle.com>

On 08/04/2014 11:22 AM, Bernd wrote:
> Am 03.08.2014 18:30 schrieb "Aleksey Shipilev" <aleksey.shipilev at oracle.com
>> :
> 
>> AverageTime or Throughput, obviously. I'm not sure how much of a problem
>> that is, actually. We should not care how many times @Benchmark was
>> called in throughput modes, we should only care it was called *enough*
>> times.
> 
> Actually this is something, I was wondering about: when minimizing runtime
> of warmup and benchmark I typically do some test runs to get a feeling for
> the throughput and then calculate the number of repetitions for the chosen
> time in head. I wonder why the number of executions for each iteration is
> not printed.

The issue is more complicated than that: one could be happy with the low
number of times @Benchmark is being called, because, e.g. it calls
thousands of leaf methods from there, and this brings the benchmark over
the warmup edge. The repetition count is meaningless here, and JMH has
no idea how to apply this generically.

This reiterates the lesson you need to do longer warmup for the unknown
workloads, and only then figure out how many warmup iterations are
enough to get to the steady state. Pretty much the thing you are doing
already.

> Maybe if you dont want to clutter the output we can only print it if the
> number is rather low?

That will clutter the output *and* confuse users. Just warmup longer
until you think you reached the steady state.

Thanks,
-Aleksey.


From aleksey.shipilev at oracle.com  Tue Aug  5 09:56:10 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Tue, 05 Aug 2014 13:56:10 +0400
Subject: Multiple JVMs, different numbers of threads and BenchmarkMode
In-Reply-To: <3FADC9EA-BF57-4014-BF98-EC8B0752A0B7@yahoo.com>
References: <D32424E3-5F8C-42E4-9B13-36FFFC7E8C8C@yahoo.com>
	<53DE6131.70208@oracle.com>
	<3FADC9EA-BF57-4014-BF98-EC8B0752A0B7@yahoo.com>
Message-ID: <53E0AA3A.7000903@oracle.com>

On 08/05/2014 12:48 AM, Dmitry Vyazelenko wrote:
>>> Now because of dynamic nature of the work done by a benchmark method
>>> what would be the best BenchmarkMode to use and how the best to
>>> configuration @Warmup/@Measurement iterations?
>>
>> AverageTime or Throughput, obviously. I'm not sure how much of a problem
>> that is, actually. We should not care how many times @Benchmark was
>> called in throughput modes, we should only care it was called *enough*
>> times.
> Well, my use-case here is different one. I actually want to measure certain amount
> of work/invocations. For example I want to benchmark how long it takes to add 1000
> elements into my data structure. If I just rely on time-based modes like AverageTime
> it means that my @Benchmark will be called unknown/unpredictable number of times.

If your workload is about doing 1000 operations over the map, then your
@Benchmark should do 1000 operations with a subsequent clear().

> If inside this method I put a loop which adds 1000 elements into collection then this
> would not yield expected results (i.e. my collections will contain millions of entries and
> not 1000).

The problem seems to be not with batching per se, but with the crud left
after a particular batch. SingleShot would not help you much as well,
since you would need @Setup(Iteration) with map.clear().

> Therefore I think that I should use @Warmup/@Measurment annotations with batchSize
> defined and SingleShotTime mode, i.e. using similar technique shown in the 
> JMHSample_26_BatchSize.java example. And of course I?ll have to use Java API to code
> proper invocation of my benchmarks with batchSize set dynamically.

That... overcomplicates the issue, doesn't it? Having @Param with
batchSize, @Benchmark method which does batchSize put/get-s, and the
following clear() seems to do the same as well.

Thanks,
-Aleksey.

From vyazelenko at yahoo.com  Tue Aug  5 11:23:51 2014
From: vyazelenko at yahoo.com (vyazelenko at yahoo.com)
Date: Tue, 5 Aug 2014 13:23:51 +0200
Subject: Multiple JVMs, different numbers of threads and BenchmarkMode
In-Reply-To: <53E0AA3A.7000903@oracle.com>
References: <D32424E3-5F8C-42E4-9B13-36FFFC7E8C8C@yahoo.com>
	<53DE6131.70208@oracle.com>
	<3FADC9EA-BF57-4014-BF98-EC8B0752A0B7@yahoo.com>
	<53E0AA3A.7000903@oracle.com>
Message-ID: <AFC93D4A-0991-4B55-876D-4E2D97A48BC6@yahoo.com>


> On Aug 5, 2014, at 11:56, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:
> 
> On 08/05/2014 12:48 AM, Dmitry Vyazelenko wrote:
>>>> Now because of dynamic nature of the work done by a benchmark method
>>>> what would be the best BenchmarkMode to use and how the best to
>>>> configuration @Warmup/@Measurement iterations?
>>> 
>>> AverageTime or Throughput, obviously. I'm not sure how much of a problem
>>> that is, actually. We should not care how many times @Benchmark was
>>> called in throughput modes, we should only care it was called *enough*
>>> times.
>> Well, my use-case here is different one. I actually want to measure certain amount
>> of work/invocations. For example I want to benchmark how long it takes to add 1000
>> elements into my data structure. If I just rely on time-based modes like AverageTime
>> it means that my @Benchmark will be called unknown/unpredictable number of times.
> 
> If your workload is about doing 1000 operations over the map, then your
> @Benchmark should do 1000 operations with a subsequent clear()
Actually the workload is 1K, 10K, 100K...100M. And yes the lower the value the more iterations I'll need. And naturally the map.clear() or creation of new map as part of @Setup(Iteration). ;)
> 
>> If inside this method I put a loop which adds 1000 elements into collection then this
>> would not yield expected results (i.e. my collections will contain millions of entries and
>> not 1000).
> 
> The problem seems to be not with batching per se, but with the crud left
> after a particular batch. SingleShot would not help you much as well,
> since you would need @Setup(Iteration) with map.clear().
But it helps ensuring exact number of invocations of the benchmark method .
> 
>> Therefore I think that I should use @Warmup/@Measurment annotations with batchSize
>> defined and SingleShotTime mode, i.e. using similar technique shown in the 
>> JMHSample_26_BatchSize.java example. And of course I?ll have to use Java API to code
>> proper invocation of my benchmarks with batchSize set dynamically.
> 
> That... overcomplicates the issue, doesn't it? Having @Param with
> batchSize, @Benchmark method which does batchSize put/get-s, and the
> following clear() seems to do the same as well.
Well, I thought that I'll remove my own @Param with batchSize and rely on @Benchmark behavior. That would also remove loops from the benchmark methods and the need to use Blackhole class (I'll just return value from the method).

On the other hand if I remove my @Param then in the result table I won't see batchSize anymore. However since I'll use Java API anyway that won't be an issue.
> 
> Thanks,
> -Aleksey.
Best regards,
Dmitry

From eugen.rabii at gmail.com  Tue Aug  5 19:08:52 2014
From: eugen.rabii at gmail.com (eugene rabii)
Date: Tue, 05 Aug 2014 22:08:52 +0300
Subject: Asymmetric confusion
Message-ID: <53E12BC4.5050500@gmail.com>

So I've looking at the JMH samples and pretty much all of them were 
somewhere clear to me until I reached asymmeteric and using groups.

I'm looking at the example here :

http://hg.openjdk.java.net/code-tools/jmh/file/5d6eee917c11/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_15_Asymmetric.java

And the first important comment starts with this:

"make two execution groups, each having 4 threads"

*two*? I can see just one that is called "g".
So each method that is annotated with @Group is an "execution group"? 
And by default it runs in 4 threads?

Sorry if this is a newbie question, but I simply don't get it.

Thx, Eugene.

From aleksey.shipilev at oracle.com  Wed Aug  6 08:19:39 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Wed, 06 Aug 2014 08:19:39 +0000
Subject: hg: code-tools/jmh: samples: improve JMHSample_15_Asymmetric
	description.
Message-ID: <201408060819.s768JdeH023047@aojmv0008>

Changeset: 1eff4c9d1ab0
Author:    shade
Date:      2014-08-06 12:19 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/1eff4c9d1ab0

samples: improve JMHSample_15_Asymmetric description.

! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_15_Asymmetric.java


From aleksey.shipilev at oracle.com  Wed Aug  6 08:20:19 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Wed, 06 Aug 2014 12:20:19 +0400
Subject: Asymmetric confusion
In-Reply-To: <53E12BC4.5050500@gmail.com>
References: <53E12BC4.5050500@gmail.com>
Message-ID: <53E1E543.3050902@oracle.com>

Hi,

On 08/05/2014 11:08 PM, eugene rabii wrote:
> And the first important comment starts with this:
> 
> "make two execution groups, each having 4 threads"
> 
> *two*? I can see just one that is called "g".

The key in that example is "Putting this all together, if we want to run
this with 8 threads, the example below means...". Note we talk about 8
total threads spread among 2 execution groups, having 4 threads each.

> So each method that is annotated with @Group is an "execution group"?

No, multiple @Benchmark methods are tied in the execution group, with
threads within the execution group distributed among these benchmarks.

> Sorry if this is a newbie question, but I simply don't get it.

That's OK. Tell me, does this sound more intuitive to you?
 http://hg.openjdk.java.net/code-tools/jmh/file/1eff4c9d1ab0/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_15_Asymmetric.java

-Aleksey.


From aleksey.shipilev at oracle.com  Wed Aug  6 13:00:58 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Wed, 06 Aug 2014 13:00:58 +0000
Subject: hg: code-tools/jmh: samples: add the caveat links for all samples
	running with Java API from the IDE.
Message-ID: <201408061300.s76D0wca010033@aojmv0008>

Changeset: 069fd4ea2c8e
Author:    shade
Date:      2014-08-06 17:00 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/069fd4ea2c8e

samples: add the caveat links for all samples running with Java API from the IDE.

! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_01_HelloWorld.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_02_BenchmarkModes.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_03_States.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_04_DefaultState.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_05_StateFixtures.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_06_FixtureLevel.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_07_FixtureLevelInvocation.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_08_DeadCode.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_09_Blackholes.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_10_ConstantFold.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_11_Loops.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_12_Forking.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_13_RunToRun.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_15_Asymmetric.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_16_CompilerControl.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_17_SyncIterations.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_18_Control.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_20_Annotations.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_21_ConsumeCPU.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_22_FalseSharing.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_23_AuxCounters.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_24_Inheritance.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_26_BatchSize.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_27_Params.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_28_BlackholeHelpers.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_29_StatesDAG.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_30_Interrupts.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_31_InfraParams.java


From vyazelenko at yahoo.com  Wed Aug  6 16:08:18 2014
From: vyazelenko at yahoo.com (Dmitry Vyazelenko)
Date: Wed, 6 Aug 2014 09:08:18 -0700
Subject: JMH does not handle signed JARs correctly
Message-ID: <1407341298.22312.YahooMailNeo@web162203.mail.bf1.yahoo.com>

Hi,

Attempt to execute benchmarks in a project that uses signed JARs fails with
"java.lang.SecurityException: Invalid signature file digest for Manifest main attributes".

This is a known problem with maven-shade-plugin (see http://zhentao-li.blogspot.ch/2012/06/maven-shade-plugin-invalid-signature.html and http://stackoverflow.com/questions/999489/invalid-signature-file-when-attempting-to-run-a-jar).
And the fix is to add the following to the plugin configuration:
? ? ? ?<configuration> 
? ? ? ? ? <filters> 
? ? ? ? ? ? <filter> 
? ? ? ? ? ? ? <artifact>*:*</artifact> 
? ? ? ? ? ? ? <excludes> 
? ? ? ? ? ? ? ? <exclude>META-INF/*.SF</exclude> 
? ? ? ? ? ? ? ? <exclude>META-INF/*.DSA</exclude> 
? ? ? ? ? ? ? ? <exclude>META-INF/*.RSA</exclude> 
? ? ? ? ? ? ? </excludes> 
? ? ? ? ? ? </filter> 
? ? ? ? ? </filters> 
? ? ? ? </configuration>

How to reproduce:
1) Create new JMH project
2) Add the following dependency:
? ? ? ? <dependency> 
? ? ? ? ? ? <groupId>org.bouncycastle</groupId> 
? ? ? ? ? ? <artifactId>bcprov-jdk16</artifactId> 
? ? ? ? ? ? <version>1.46</version> 
? ? ? ? </dependency> 
3) Execute mvn clean package
4) Try to run jar: java -jar target/benchmarks.jar => exception

Unrelated note, maven archetype for JMH uses old version 0.9.3 instead of current 0.9.5.

Best regards,
Dmitry Vyazelenko?

From aleksey.shipilev at oracle.com  Wed Aug  6 16:58:53 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Wed, 06 Aug 2014 20:58:53 +0400
Subject: JMH does not handle signed JARs correctly
In-Reply-To: <1407341298.22312.YahooMailNeo@web162203.mail.bf1.yahoo.com>
References: <1407341298.22312.YahooMailNeo@web162203.mail.bf1.yahoo.com>
Message-ID: <53E25ECD.7000209@oracle.com>

Hi,

On 08/06/2014 08:08 PM, Dmitry Vyazelenko wrote:
> This is a known problem with maven-shade-plugin (see http://zhentao-li.blogspot.ch/2012/06/maven-shade-plugin-invalid-signature.html and http://stackoverflow.com/questions/999489/invalid-signature-file-when-attempting-to-run-a-jar).
> And the fix is to add the following to the plugin configuration:
>        <configuration> 
>           <filters> 
>             <filter> 
>               <artifact>*:*</artifact> 
>               <excludes> 
>                 <exclude>META-INF/*.SF</exclude> 
>                 <exclude>META-INF/*.DSA</exclude> 
>                 <exclude>META-INF/*.RSA</exclude> 
>               </excludes> 
>             </filter> 
>           </filters> 
>         </configuration>

We use to have these filters in sample projects, but purged since it was
confusing people. Do we need to resurrect it? It seems to be only the
corner case of signed JARs (Bouncy Castle seems to be the only example I
ever saw experiencing this), and the solution is easily googlable.

> Unrelated note, maven archetype for JMH uses old version 0.9.3 instead of current 0.9.5.

Known issue:
  https://issues.sonatype.org/browse/MVNCENTRAL-482

-Aleksey.


From aleksey.shipilev at oracle.com  Wed Aug  6 20:02:42 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Wed, 06 Aug 2014 20:02:42 +0000
Subject: hg: code-tools/jmh: archetypes: filter signatures from final JAR
	(again).
Message-ID: <201408062002.s76K2gLP016058@aojmv0008>

Changeset: 9b2561a5e071
Author:    shade
Date:      2014-08-07 00:00 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/9b2561a5e071

archetypes: filter signatures from final JAR (again).

! jmh-archetypes/jmh-groovy-benchmark-archetype/src/main/resources/archetype-resources/pom.xml
! jmh-archetypes/jmh-java-benchmark-archetype/src/main/resources/archetype-resources/pom.xml
! jmh-archetypes/jmh-kotlin-benchmark-archetype/src/main/resources/archetype-resources/pom.xml
! jmh-archetypes/jmh-scala-benchmark-archetype/src/main/resources/archetype-resources/pom.xml


From aleksey.shipilev at oracle.com  Wed Aug  6 20:03:37 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Thu, 07 Aug 2014 00:03:37 +0400
Subject: JMH does not handle signed JARs correctly
In-Reply-To: <53E25ECD.7000209@oracle.com>
References: <1407341298.22312.YahooMailNeo@web162203.mail.bf1.yahoo.com>
	<53E25ECD.7000209@oracle.com>
Message-ID: <53E28A19.6050807@oracle.com>

On 08/06/2014 08:58 PM, Aleksey Shipilev wrote:
> Hi,
> 
> On 08/06/2014 08:08 PM, Dmitry Vyazelenko wrote:
>> This is a known problem with maven-shade-plugin (see http://zhentao-li.blogspot.ch/2012/06/maven-shade-plugin-invalid-signature.html and http://stackoverflow.com/questions/999489/invalid-signature-file-when-attempting-to-run-a-jar).
>> And the fix is to add the following to the plugin configuration:
>>        <configuration> 
>>           <filters> 
>>             <filter> 
>>               <artifact>*:*</artifact> 
>>               <excludes> 
>>                 <exclude>META-INF/*.SF</exclude> 
>>                 <exclude>META-INF/*.DSA</exclude> 
>>                 <exclude>META-INF/*.RSA</exclude> 
>>               </excludes> 
>>             </filter> 
>>           </filters> 
>>         </configuration>
> 
> We use to have these filters in sample projects, but purged since it was
> confusing people. Do we need to resurrect it? It seems to be only the
> corner case of signed JARs (Bouncy Castle seems to be the only example I
> ever saw experiencing this), and the solution is easily googlable.

Anyhow, pushing this to archetypes seems like a good idea:
 http://hg.openjdk.java.net/code-tools/jmh/rev/9b2561a5e071

Thanks,
-Aleksey.

From vyazelenko at yahoo.com  Wed Aug  6 21:59:50 2014
From: vyazelenko at yahoo.com (Dmitry Vyazelenko)
Date: Wed, 6 Aug 2014 23:59:50 +0200
Subject: JMH does not handle signed JARs correctly
In-Reply-To: <53E28A19.6050807@oracle.com>
References: <1407341298.22312.YahooMailNeo@web162203.mail.bf1.yahoo.com>
	<53E25ECD.7000209@oracle.com> <53E28A19.6050807@oracle.com>
Message-ID: <872E36FF-55CF-49E2-9C82-DD9C5C4C0D31@yahoo.com>

Hi Aleksey,

Thanks! You are fast! ;-)

P.S. I?ve had problems with Bounty Castle, Eclipse compiler and Janino compiler. They
all sign JARs. In that particular project 5 JARs from these libraries were causing problem.

Best regards,
Dmitry Vyazelenko
On Aug 6, 2014, at 22:03 , Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:

> On 08/06/2014 08:58 PM, Aleksey Shipilev wrote:
>> Hi,
>> 
>> On 08/06/2014 08:08 PM, Dmitry Vyazelenko wrote:
>>> This is a known problem with maven-shade-plugin (see http://zhentao-li.blogspot.ch/2012/06/maven-shade-plugin-invalid-signature.html and http://stackoverflow.com/questions/999489/invalid-signature-file-when-attempting-to-run-a-jar).
>>> And the fix is to add the following to the plugin configuration:
>>>       <configuration> 
>>>          <filters> 
>>>            <filter> 
>>>              <artifact>*:*</artifact> 
>>>              <excludes> 
>>>                <exclude>META-INF/*.SF</exclude> 
>>>                <exclude>META-INF/*.DSA</exclude> 
>>>                <exclude>META-INF/*.RSA</exclude> 
>>>              </excludes> 
>>>            </filter> 
>>>          </filters> 
>>>        </configuration>
>> 
>> We use to have these filters in sample projects, but purged since it was
>> confusing people. Do we need to resurrect it? It seems to be only the
>> corner case of signed JARs (Bouncy Castle seems to be the only example I
>> ever saw experiencing this), and the solution is easily googlable.
> 
> Anyhow, pushing this to archetypes seems like a good idea:
> http://hg.openjdk.java.net/code-tools/jmh/rev/9b2561a5e071
> 
> Thanks,
> -Aleksey.


From eugen.rabii at gmail.com  Fri Aug  8 08:43:39 2014
From: eugen.rabii at gmail.com (Eugen Rabii)
Date: Fri, 08 Aug 2014 11:43:39 +0300
Subject: Caliper CharMatcher Confusion
Message-ID: <53E48DBB.9030205@gmail.com>

So basically a weird piece of code in the famous guava library for 
CharMatcher.removeFrom
Utility method for removing a char from a String, sound not that 
complicated, until I looked at the sources:


   public String removeFrom(CharSequence sequence) {
     String string = sequence.toString();
     int pos = indexIn(string);
     if (pos == -1) {
       return string;
     }

     char[] chars = string.toCharArray();
     int spread = 1;

     // This unusual loop comes from extensive benchmarking
     OUT: while (true) {
       pos++;
       while (true) {
         if (pos == chars.length) {
           break OUT;
         }
         if (matches(chars[pos])) {
           break;
         }
         chars[pos - spread] = chars[pos];
         pos++;
       }
       spread++;
     }
     return new String(chars, 0, pos - spread);
   }

So that comment there : "This unusual loop comes from extensive 
benchmarking" got me thinking, I do not trust Caliper very much (JMH to 
blame), thus as a result I do not trust
their benchmarks too much, so I decided to code mine. In the end I 
reached almost the same logic that they did, almost. And decided to test 
it with JMH.


Can some of you professionals tell me if from a *JMH stand of point is 
this a correct testing approach*? I'm thinking to test at least cold 
start too.

package org.madmonky.guava;

import java.util.concurrent.TimeUnit;

import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Fork;
import org.openjdk.jmh.annotations.Measurement;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.annotations.Param;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.Setup;
import org.openjdk.jmh.annotations.State;
import org.openjdk.jmh.annotations.Warmup;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

import com.google.common.base.CharMatcher;

@Warmup(iterations=5, time=1, timeUnit=TimeUnit.SECONDS)
@BenchmarkMode(Mode.AverageTime)
@Measurement(iterations=3, time=1, timeUnit=TimeUnit.SECONDS)
@Fork(3)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
@State(Scope.Benchmark)
public class CharMatcherTest {
     public static void main(String[] args) throws RunnerException {
          Options opt = new OptionsBuilder()
                  .include(".*" + CharMatcherTest.class.getSimpleName() 
+ ".*")
                  .threads(4)
                  .build();

          new Runner(opt).run();
     }

     CharMatcher charMatcher;

     @Param({"e", "eeeee", "efefefefe", "eeeemeeeeseeeeer", "dfgrry", 
"wertyeoiuyeeeeeteee"})
     String input;

     char searched;

     @Setup
     public void prepare(){
         charMatcher = CharMatcher.is('e');
         searched = 'e';
     }


     @Benchmark
     public String mineFirstVersion(){
         char [] array = input.toCharArray();
         boolean reachedTheEnd = false;
         int totalCount = 0;
         for(int i=0;i<array.length;++i){
             int howManyInIteration = 0;
             while(array[i] == searched){
                 ++i;
                 ++howManyInIteration;
                 if(i == array.length) {
                     reachedTheEnd = true;
                     break;
                 }
             }
             totalCount += howManyInIteration;
             if(!reachedTheEnd) array[i-totalCount] = array[i];
         }
         return new String(array, 0, (array.length - totalCount));
     }


     @Benchmark
     public String guavaRemoveFrom(){
         return charMatcher.removeFrom(input);
     }
}


P.S. The results (if the approach is correct) show that their 
implementation is a bit slower.

Thank you very much for your time,
Eugene.





From ecki at zusammenkunft.net  Fri Aug  8 09:12:56 2014
From: ecki at zusammenkunft.net (Bernd Eckenfels)
Date: Fri, 8 Aug 2014 11:12:56 +0200
Subject: AW: Caliper CharMatcher Confusion
In-Reply-To: <53E48DBB.9030205@gmail.com>
References: <53E48DBB.9030205@gmail.com>
Message-ID: <53e4949e.147be00a.16fe.5c09@mx.google.com>

Hello,

Not a full analysis, but two comments:

first of all you should level the playing field by outlining your implementation into an external utilities class (like guava) which works in a CharSequence and secondly I wonder if benchmarking additionally with a much bigger haystack (longer search input by using a length and type parameter matrix (no match, all match, 10%match; len 10,100,200,500,2k) gives additional insight.

gruss
Bernd


-- 
http://bernd.eckenfels.net

----- Urspr?ngliche Nachricht -----
Von: "Eugen Rabii" <eugen.rabii at gmail.com>
Gesendet: ?08.?08.?2014 10:54
An: "jmh-dev at openjdk.java.net" <jmh-dev at openjdk.java.net>
Betreff: Caliper CharMatcher Confusion

So basically a weird piece of code in the famous guava library for 
CharMatcher.removeFrom
Utility method for removing a char from a String, sound not that 
complicated, until I looked at the sources:


   public String removeFrom(CharSequence sequence) {
     String string = sequence.toString();
     int pos = indexIn(string);
     if (pos == -1) {
       return string;
     }

     char[] chars = string.toCharArray();
     int spread = 1;

     // This unusual loop comes from extensive benchmarking
     OUT: while (true) {
       pos++;
       while (true) {
         if (pos == chars.length) {
           break OUT;
         }
         if (matches(chars[pos])) {
           break;
         }
         chars[pos - spread] = chars[pos];
         pos++;
       }
       spread++;
     }
     return new String(chars, 0, pos - spread);
   }

So that comment there : "This unusual loop comes from extensive 
benchmarking" got me thinking, I do not trust Caliper very much (JMH to 
blame), thus as a result I do not trust
their benchmarks too much, so I decided to code mine. In the end I 
reached almost the same logic that they did, almost. And decided to test 
it with JMH.


Can some of you professionals tell me if from a *JMH stand of point is 
this a correct testing approach*? I'm thinking to test at least cold 
start too.

package org.madmonky.guava;

import java.util.concurrent.TimeUnit;

import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Fork;
import org.openjdk.jmh.annotations.Measurement;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.annotations.Param;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.Setup;
import org.openjdk.jmh.annotations.State;
import org.openjdk.jmh.annotations.Warmup;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

import com.google.common.base.CharMatcher;

@Warmup(iterations=5, time=1, timeUnit=TimeUnit.SECONDS)
@BenchmarkMode(Mode.AverageTime)
@Measurement(iterations=3, time=1, timeUnit=TimeUnit.SECONDS)
@Fork(3)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
@State(Scope.Benchmark)
public class CharMatcherTest {
     public static void main(String[] args) throws RunnerException {
          Options opt = new OptionsBuilder()
                  .include(".*" + CharMatcherTest.class.getSimpleName() 
+ ".*")
                  .threads(4)
                  .build();

          new Runner(opt).run();
     }

     CharMatcher charMatcher;

     @Param({"e", "eeeee", "efefefefe", "eeeemeeeeseeeeer", "dfgrry", 
"wertyeoiuyeeeeeteee"})
     String input;

     char searched;

     @Setup
     public void prepare(){
         charMatcher = CharMatcher.is('e');
         searched = 'e';
     }


     @Benchmark
     public String mineFirstVersion(){
         char [] array = input.toCharArray();
         boolean reachedTheEnd = false;
         int totalCount = 0;
         for(int i=0;i<array.length;++i){
             int howManyInIteration = 0;
             while(array[i] == searched){
                 ++i;
                 ++howManyInIteration;
                 if(i == array.length) {
                     reachedTheEnd = true;
                     break;
                 }
             }
             totalCount += howManyInIteration;
             if(!reachedTheEnd) array[i-totalCount] = array[i];
         }
         return new String(array, 0, (array.length - totalCount));
     }


     @Benchmark
     public String guavaRemoveFrom(){
         return charMatcher.removeFrom(input);
     }
}


P.S. The results (if the approach is correct) show that their 
implementation is a bit slower.

Thank you very much for your time,
Eugene.





From aleksey.shipilev at oracle.com  Fri Aug  8 12:21:50 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Fri, 08 Aug 2014 16:21:50 +0400
Subject: Caliper CharMatcher Confusion
In-Reply-To: <53E48DBB.9030205@gmail.com>
References: <53E48DBB.9030205@gmail.com>
Message-ID: <53E4C0DE.50201@oracle.com>

This seems like the topic for StackOverflow/blog. jmh-dev is mostly
about developing JMH itself. We would need to set up the alias for
benchmark reviews...

On 08/08/2014 12:43 PM, Eugen Rabii wrote:
> Can some of you professionals tell me if from a *JMH stand of point is
> this a correct testing approach*? I'm thinking to test at least cold
> start too.

What Bernd already said, plus...

I think since you are dealing with nano/micro-benchmarks here, it needs
the dive into the generated code. The code generation quirks might
account for different results you are seeing.

In other words, have the explanation *why* the results are different,
not just the conclusion the results are different.

-Aleksey.



From tom.deneau at amd.com  Fri Aug  8 18:59:21 2014
From: tom.deneau at amd.com (Deneau, Tom)
Date: Fri, 8 Aug 2014 18:59:21 +0000
Subject: Forking and -server flag
Message-ID: <BC97738F8E7C8742BABED7F06FB9DF9155489D54@SATLEXDAG01.amd.com>

New to JMH...

My original command line specified -server, I am using forking and I noticed the forked command line does not contain -server, which on my configuration causes problems.  Is there a way to get the forked command line to contain -server?

-- Tom

From bernhard.urban at jku.at  Fri Aug  8 20:36:10 2014
From: bernhard.urban at jku.at (Bernhard Urban)
Date: Fri, 8 Aug 2014 22:36:10 +0200
Subject: Forking and -server flag
In-Reply-To: <BC97738F8E7C8742BABED7F06FB9DF9155489D54@SATLEXDAG01.amd.com>
References: <BC97738F8E7C8742BABED7F06FB9DF9155489D54@SATLEXDAG01.amd.com>
Message-ID: <CAAr_hs5ZhPYe9x+Qu94Jc38ozE7+SPdAY9SgwK29TwuuC7nh9A@mail.gmail.com>

take a look at --jvmArgs

-Bernhard
On Aug 8, 2014 8:59 PM, "Deneau, Tom" <tom.deneau at amd.com> wrote:

> New to JMH...
>
> My original command line specified -server, I am using forking and I
> noticed the forked command line does not contain -server, which on my
> configuration causes problems.  Is there a way to get the forked command
> line to contain -server?
>
> -- Tom
>

From aleksey.shipilev at oracle.com  Sat Aug  9 08:30:41 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Sat, 09 Aug 2014 12:30:41 +0400
Subject: Forking and -server flag
In-Reply-To: <BC97738F8E7C8742BABED7F06FB9DF9155489D54@SATLEXDAG01.amd.com>
References: <BC97738F8E7C8742BABED7F06FB9DF9155489D54@SATLEXDAG01.amd.com>
Message-ID: <53E5DC31.4020904@oracle.com>

Hi Tom,

On 08/08/2014 10:59 PM, Deneau, Tom wrote:
> My original command line specified -server, I am using forking and I 
> noticed the forked command line does not contain -server, which on
> my configuration causes problems.

Yes, known problem with Java launcher:
 https://bugs.openjdk.java.net/browse/JDK-8025700

> Is there a way to get the forked command line to contain -server?

--jvmArgs is your friend at the command line.
.jvmArgs(...) is your friend in the API.
@Fork(jvmArgs = ...) is your friend in annotations.

-Aleksey.


From aleksey.shipilev at oracle.com  Mon Aug 11 20:23:05 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Tue, 12 Aug 2014 00:23:05 +0400
Subject: JMH 0.9.6
Message-ID: <53E92629.8090701@oracle.com>

Hi,

JMH 0.9.6 is released and available in Maven Central (thanks again to
Evgeny, as usual). JMH 0.9.5 is our 2nd release candidate for 1.0,
please test! It turned out Maven Central had a weird bug which left
users with 0.9.3 only when bootstrapping from the archetype.

Highlights of JMH 0.9.6:

 * "-prof gc" overestimated the GC time 100x, fixed.

 * "-prof perfasm" now tries to recover when multiple compiler threads
are writing concurrently.


 * SingleShot mode does not sleep for the entire run time when -w/-r is
supplied now.

 * Benchmark archetypes are filtering the signatures from the uberjar,
allowing to merge signed JARs together.

Enjoy!

Thanks,
-Aleksey.


From clement at unportant.info  Mon Aug 11 21:36:35 2014
From: clement at unportant.info (=?ISO-8859-1?Q?Cl=E9ment?= MATHIEU)
Date: Mon, 11 Aug 2014 23:36:35 +0200
Subject: Perfasm fails on Fedora/OpenJDK
Message-ID: <1407792995.17345.22.camel@station>

Hi JMHers,

I played a bit with perfasm today and found an integration issue making
perfasm unusable on Fedora 20. I believe that my setup is not specific
and any Fedora user using the OpenJDK package will face the same issue.

Basically running perfasm always leads to this error message:

---
        WARNING: The perf event count is suspiciously low (1). The performance data might be
        inaccurate or misleading. Try to do the profiling again, or tune up the sampling frequency.
---

I did play a little bit with a debugger to troubleshoot the issue. The
outcome is that the name of the process returned by perf on Fedora is
java-abrt and not java. LinuxPerfAsmProfiler makes the assumption that
the name of the process is "java" and discard all the events. 

The following ugly patch against the main branch does solve the issue.
However I am not sure it will not cause harm on other platforms. 

diff -r 9b2561a5e071 jmh-core/src/main/java/org/openjdk/jmh/profile/LinuxPerfAsmProfiler.java
--- a/jmh-core/src/main/java/org/openjdk/jmh/profile/LinuxPerfAsmProfiler.java	Thu Aug 07 00:00:14 2014 +0400
+++ b/jmh-core/src/main/java/org/openjdk/jmh/profile/LinuxPerfAsmProfiler.java	Mon Aug 11 23:19:45 2014 +0200
@@ -724,7 +724,7 @@
                 String lib = elems[elems.length - 1];
                 lib = lib.substring(lib.lastIndexOf("/") + 1, lib.length()).replace("(", "").replace(")", "");
 
-                if (process.equalsIgnoreCase("java")) {
+                if (process.toLowerCase().startsWith("java")) {
                     try {
                         Double time = Double.valueOf(strTime);
                         if (startTime == null) {


Is this issue already known ? What would be the best way to solve it ?
Not sure if it should handled on the JMH side or if we should ask Fedora
to not change the name of the process. I can provide more information
about this java-abrt weirdness if needed.


Regards,

Cl?ment MATHIEU


From vyazelenko at yahoo.com  Mon Aug 11 21:48:25 2014
From: vyazelenko at yahoo.com (Dmitry Vyazelenko)
Date: Mon, 11 Aug 2014 23:48:25 +0200
Subject: Can't fork VM if option value contains space
Message-ID: <43C4DC37-9DAC-4341-917B-BB1E7D333827@yahoo.com>

Hi all,

Just got into some trouble passing JVM options to forked VMs when values have a space, i.e.
I want to pass the following option (via -jmvArgs) when executing benchmarks:
    -Dmy.option=?A B C?

What I get is the following error:
java -jar target/benchmarks.jar -jvmArgs -Dmy.property="A B C"
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/Home/jre/bin/java
# VM options: -Dmy.property=A B C
# Warmup: 20 iterations, 1 s each
# Measurement: 20 iterations, 1 s each
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: test.jmh.DummyBench.baseline

# Run progress: 0.00% complete, ETA 00:06:40
# Fork: 1 of 10
Error: Could not find or load main class B
<forked VM failed with exit code 1>
<stdout last='10 lines'>
</stdout>
<stderr last='10 lines'>
Error: Could not find or load main class B
</stderr>

OK, it seems that quotes were removed. Thus I try to force them which also fails:
java -jar target/benchmarks.jar -jvmArgs -Dmy.property="\"A B C\""
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/Home/jre/bin/java
# VM options: -Dmy.property="A B C"
# Warmup: 20 iterations, 1 s each
# Measurement: 20 iterations, 1 s each
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: test.jmh.DummyBench.baseline

# Run progress: 0.00% complete, ETA 00:06:40
# Fork: 1 of 10
Error: Could not find or load main class B
<forked VM failed with exit code 1>
<stdout last='10 lines'>
</stdout>
<stderr last='10 lines'>
Error: Could not find or load main class B
</stderr>

Note that this time ?VM options? shows command correctly but execution fails anyway.

The only workaround I?ve found is to use Java API:
public class DummyBench {
    @Benchmark
    public void baseline() {
    }

    public static void main(String[] args) throws Exception {
        Options options = new OptionsBuilder().jvmArgs("-Dmy.property=\"A B C\"").build();
        new Runner(options).run();
    }
}

When I run it via main method then it works:
java -cp target/benchmarks.jar test.jmh.DummyBench
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/Home/jre/bin/java
# VM options: -Dmy.property="A B C"
# Warmup: 20 iterations, 1 s each
# Measurement: 20 iterations, 1 s each
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: test.jmh.DummyBench.baseline

# Run progress: 0.00% complete, ETA 00:06:40
# Fork: 1 of 10
# Warmup Iteration   1: 3513969661.351 ops/s
?


Best regards,
Dmitry Vyazelenko

From vyazelenko at yahoo.com  Mon Aug 11 22:07:39 2014
From: vyazelenko at yahoo.com (Dmitry Vyazelenko)
Date: Tue, 12 Aug 2014 00:07:39 +0200
Subject: Multi-package benchmarks: package names 'merged' in the result table
Message-ID: <E478D272-A517-4F97-A26E-6118421517D3@yahoo.com>

Hi,

When using JMH with benchmarks located in multiple/different benchmarks
the results are incorrectly merged together.

Here are some benchmarks from multiple packages:

package com.some.even.longer.word;

import org.openjdk.jmh.annotations.Benchmark;

public class SomeEvenMoreWeirdBenchmark {
    @Benchmark
    public void baseline() {
    }
}


package my.sample.pkg;

import org.openjdk.jmh.annotations.Benchmark;

public class MySampleBenchmark {
    @Benchmark
    public void baseline() {
    }
}


package test.jmh;

import org.openjdk.jmh.annotations.Benchmark;

public class TestJmhBenchmark {
    @Benchmark
    public void baseline() {
    }
}


When I execute benchmarks I get the following results:
java -jar target/benchmarks.jar -f 1 -wi 0 -i 1
Benchmark                                                                  Mode  Samples           Score  Score error  Units
t.j.TestJmhBenchmark.baseline.word.SomeEvenMoreWeirdBenchmark.baseline    thrpt        1  3464644276.067          NaN  ops/s
t.j.TestJmhBenchmark.baseline.baseline                                    thrpt        1  3454747731.105          NaN  ops/s
t.j.TestJmhBenchmark.baseline                                             thrpt        1  3409380706.050          NaN  ops/s

So instead of using package name from every benchmark class it seems that
"test.jmh.TestJmhBenchmark" is used as prefix for all benchmarks which is wrong!


Best regards,
Dmitry Vyazelenko

From aleksey.shipilev at oracle.com  Mon Aug 11 22:23:58 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Mon, 11 Aug 2014 22:23:58 +0000
Subject: hg: code-tools/jmh: 3 new changesets
Message-ID: <201408112223.s7BMNwnm004526@aojmv0008>

Changeset: a275f55bcea0
Author:    shade
Date:      2014-08-11 12:29 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/a275f55bcea0

JMH v0.9.6.

! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml
! jmh-archetypes/pom.xml
! jmh-core-benchmarks/pom.xml
! jmh-core-ct/pom.xml
! jmh-core-it/pom.xml
! jmh-core/pom.xml
! jmh-generator-annprocess/pom.xml
! jmh-generator-asm/pom.xml
! jmh-generator-bytecode/pom.xml
! jmh-generator-reflection/pom.xml
! jmh-samples/pom.xml
! pom.xml

Changeset: 7252d7f84f5d
Author:    shade
Date:      2014-08-11 12:29 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/7252d7f84f5d

Added tag 0.9.6 for changeset a275f55bcea0

! .hgtags

Changeset: 5e330463c637
Author:    shade
Date:      2014-08-11 12:31 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/5e330463c637

Continue in 1.0-SNAPSHOT.

! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml
! jmh-archetypes/pom.xml
! jmh-core-benchmarks/pom.xml
! jmh-core-ct/pom.xml
! jmh-core-it/pom.xml
! jmh-core/pom.xml
! jmh-generator-annprocess/pom.xml
! jmh-generator-asm/pom.xml
! jmh-generator-bytecode/pom.xml
! jmh-generator-reflection/pom.xml
! jmh-samples/pom.xml
! pom.xml


From aleksey.shipilev at oracle.com  Mon Aug 11 22:25:06 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Mon, 11 Aug 2014 22:25:06 +0000
Subject: hg: code-tools/jmh: runners: ClassUtils.denseClassNames resets the
	prefix incorrectly.
Message-ID: <201408112225.s7BMP6kG004661@aojmv0008>

Changeset: 22d06efe91cb
Author:    shade
Date:      2014-08-12 02:24 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/22d06efe91cb

runners: ClassUtils.denseClassNames resets the prefix incorrectly.

! jmh-core/src/main/java/org/openjdk/jmh/util/ClassUtils.java
! jmh-core/src/test/java/org/openjdk/jmh/util/TestClassUtils.java


From aleksey.shipilev at oracle.com  Mon Aug 11 22:25:36 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Tue, 12 Aug 2014 02:25:36 +0400
Subject: Multi-package benchmarks: package names 'merged' in the result
	table
In-Reply-To: <E478D272-A517-4F97-A26E-6118421517D3@yahoo.com>
References: <E478D272-A517-4F97-A26E-6118421517D3@yahoo.com>
Message-ID: <53E942E0.1050800@oracle.com>

Hi Dmitry,

On 08/12/2014 02:07 AM, Dmitry Vyazelenko wrote:
> When using JMH with benchmarks located in multiple/different benchmarks
> the results are incorrectly merged together.
> 
> So instead of using package name from every benchmark class it seems that
> "test.jmh.TestJmhBenchmark" is used as prefix for all benchmarks which is wrong!

Thanks for the bug report! Indeed, this is a bug in class name
shortener. Should be working properly with:
 http://hg.openjdk.java.net/code-tools/jmh/rev/22d06efe91cb

-Aleksey.



From ecki at zusammenkunft.net  Mon Aug 11 22:45:45 2014
From: ecki at zusammenkunft.net (Bernd Eckenfels)
Date: Tue, 12 Aug 2014 00:45:45 +0200
Subject: Partial RE for Benchmark Tests
Message-ID: <20140812004545.00003937.ecki@zusammenkunft.net>

JMH uses a regular expression to select the Benchmarks. While doing so
(as I understand it) it reuqires a full match. This has the
"disadvantage" that you always need to specify ".*MyBenchmark.*". Not
everybody is used to this syntax and it is really seldomly needed.

How about changing this to a partial match with ^$ support: if you
specify an expression and it is found inside the benchmark name, the
benchmark will be selected. That way you can even specify Substrings
and be done 99% of the time: MyBenchmark. If you need to be more
speficic, you could: "^.*MyBenchmark\.[^.]+"

Is it worth coming up with a patch for it?

Bernd

From vyazelenko at yahoo.com  Mon Aug 11 22:51:34 2014
From: vyazelenko at yahoo.com (Dmitry Vyazelenko)
Date: Tue, 12 Aug 2014 00:51:34 +0200
Subject: Multi-package benchmarks: package names 'merged' in the result
	table
In-Reply-To: <53E94699.8010203@oracle.com>
References: <E478D272-A517-4F97-A26E-6118421517D3@yahoo.com>
	<53E942E0.1050800@oracle.com>
	<DABA3FCC-E9EB-49BA-B020-8180CEA46308@yahoo.com>
	<53E94699.8010203@oracle.com>
Message-ID: <E49339CB-4863-4D62-A045-34EB1A9C9BF0@yahoo.com>

Ok, good to know. ;-)

P.S. Replying to the list! :)

Best regards,
Dmitry
On Aug 12, 2014, at 00:41 , Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:

> Any reason you are not replying to the list? :)
> Yes, we shorten the common prefix, not exactly the package names.
> 
> -Aleksey.
> 
> On 08/12/2014 02:38 AM, Dmitry Vyazelenko wrote:
>> Hi Aleksey,
>> 
>> Thanks for the fix. 
>> 
>> I just tried it and got the following output:
>> Benchmark                                                         Mode  Samples           Score  Score error  Units
>> com.some.even.longer.word.SomeEvenMoreWeirdBenchmark.baseline    thrpt        1  3439169459.497          NaN  ops/s
>> my.sample.pkg.MySampleBenchmark.baseline                         thrpt        1  3426429841.206          NaN  ops/s
>> test.jmh.TestJmhBenchmark.baseline                               thrpt        1  3419601440.268          NaN  ops/s
>> 
>> So it seems that in case of multiple packages no shortening will be done, right? (that?s fine with me)
>> 
>> Best regards,
>> Dmitry
>> On Aug 12, 2014, at 00:25 , Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:
>> 
>>> Hi Dmitry,
>>> 
>>> On 08/12/2014 02:07 AM, Dmitry Vyazelenko wrote:
>>>> When using JMH with benchmarks located in multiple/different benchmarks
>>>> the results are incorrectly merged together.
>>>> 
>>>> So instead of using package name from every benchmark class it seems that
>>>> "test.jmh.TestJmhBenchmark" is used as prefix for all benchmarks which is wrong!
>>> 
>>> Thanks for the bug report! Indeed, this is a bug in class name
>>> shortener. Should be working properly with:
>>> http://hg.openjdk.java.net/code-tools/jmh/rev/22d06efe91cb
>>> 
>>> -Aleksey.
>>> 
>>> 
>> 
> 
> 


From aleksey.shipilev at oracle.com  Mon Aug 11 23:04:02 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Mon, 11 Aug 2014 23:04:02 +0000
Subject: hg: code-tools/jmh: profilers: perfasm,
	remove the redundant check for "java".
Message-ID: <201408112304.s7BN42l1010319@aojmv0008>

Changeset: e739c4bf5798
Author:    shade
Date:      2014-08-12 03:03 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/e739c4bf5798

profilers: perfasm, remove the redundant check for "java".
Additionally fixes the platforms where "java" gets symlinked in the weird ways.

! jmh-core/src/main/java/org/openjdk/jmh/profile/LinuxPerfAsmProfiler.java


From aleksey.shipilev at oracle.com  Mon Aug 11 23:04:41 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Tue, 12 Aug 2014 03:04:41 +0400
Subject: Perfasm fails on Fedora/OpenJDK
In-Reply-To: <1407792995.17345.22.camel@station>
References: <1407792995.17345.22.camel@station>
Message-ID: <53E94C09.3070907@oracle.com>

Hi Clement,

On 08/12/2014 01:36 AM, Cl?ment MATHIEU wrote:
> I did play a little bit with a debugger to troubleshoot the issue. The
> outcome is that the name of the process returned by perf on Fedora is
> java-abrt and not java. LinuxPerfAsmProfiler makes the assumption that
> the name of the process is "java" and discard all the events. 

Awwww.

> The following ugly patch against the main branch does solve the issue.
> However I am not sure it will not cause harm on other platforms. 
> 
> diff -r 9b2561a5e071 jmh-core/src/main/java/org/openjdk/jmh/profile/LinuxPerfAsmProfiler.java
> --- a/jmh-core/src/main/java/org/openjdk/jmh/profile/LinuxPerfAsmProfiler.java	Thu Aug 07 00:00:14 2014 +0400
> +++ b/jmh-core/src/main/java/org/openjdk/jmh/profile/LinuxPerfAsmProfiler.java	Mon Aug 11 23:19:45 2014 +0200
> @@ -724,7 +724,7 @@
>                  String lib = elems[elems.length - 1];
>                  lib = lib.substring(lib.lastIndexOf("/") + 1, lib.length()).replace("(", "").replace(")", "");
>  
> -                if (process.equalsIgnoreCase("java")) {
> +                if (process.toLowerCase().startsWith("java")) {
>                      try {
>                          Double time = Double.valueOf(strTime);
>                          if (startTime == null) {

This feels like an ugly fix, and it can capture more processes than
required...

But, the funny thing is, we don't need this check at all, since perf is
instrumenting the java process only. This check was used before to
separate the "data" lines from the other garbage, but now there is a
formatting check above. Therefore, we might as well remove it:
 http://hg.openjdk.java.net/code-tools/jmh/rev/e739c4bf5798

Please check it works on your Fedora installation.

Thanks,
-Aleksey.


From aleksey.shipilev at oracle.com  Mon Aug 11 23:17:08 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Tue, 12 Aug 2014 03:17:08 +0400
Subject: Partial RE for Benchmark Tests
In-Reply-To: <20140812004545.00003937.ecki@zusammenkunft.net>
References: <20140812004545.00003937.ecki@zusammenkunft.net>
Message-ID: <53E94EF4.7010801@oracle.com>

Hi Bernd,

On 08/12/2014 02:45 AM, Bernd Eckenfels wrote:
> JMH uses a regular expression to select the Benchmarks. While doing so
> (as I understand it) it reuqires a full match. This has the
> "disadvantage" that you always need to specify ".*MyBenchmark.*". Not
> everybody is used to this syntax and it is really seldomly needed.
> 
> How about changing this to a partial match with ^$ support: if you
> specify an expression and it is found inside the benchmark name, the
> benchmark will be selected. That way you can even specify Substrings
> and be done 99% of the time: MyBenchmark. If you need to be more
> speficic, you could: "^.*MyBenchmark\.[^.]+"
> 
> Is it worth coming up with a patch for it?

This is an interesting idea, but we need to think through what exactly
do we want: more use-cases, including the corner cases? Is there a
reference to what other regexp-accepting products do?

Your suggestion seems to amount to:
 1) Look for an explicit "^" at the beginning, add ".*" at the
beginning, otherwise.
 2) Look for an explicit "$" at the end, add ".*" at the end, otherwise.

Thanks,
-Aleksey.


From aleksey.shipilev at oracle.com  Mon Aug 11 23:22:16 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Tue, 12 Aug 2014 03:22:16 +0400
Subject: Can't fork VM if option value contains space
In-Reply-To: <43C4DC37-9DAC-4341-917B-BB1E7D333827@yahoo.com>
References: <43C4DC37-9DAC-4341-917B-BB1E7D333827@yahoo.com>
Message-ID: <53E95028.4060909@oracle.com>

Hi again Dmitry,

On 08/12/2014 01:48 AM, Dmitry Vyazelenko wrote:
> Just got into some trouble passing JVM options to forked VMs when values have a space, i.e.
> I want to pass the following option (via -jmvArgs) when executing benchmarks:
>     -Dmy.option=?A B C?
> 
> What I get is the following error:
> java -jar target/benchmarks.jar -jvmArgs -Dmy.property="A B C"
> # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/Home/jre/bin/java
> # VM options: -Dmy.property=A B C
> # Warmup: 20 iterations, 1 s each
> # Measurement: 20 iterations, 1 s each
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Throughput, ops/time
> # Benchmark: test.jmh.DummyBench.baseline
> 
> # Run progress: 0.00% complete, ETA 00:06:40
> # Fork: 1 of 10
> Error: Could not find or load main class B
> <forked VM failed with exit code 1>
> <stdout last='10 lines'>
> </stdout>
> <stderr last='10 lines'>
> Error: Could not find or load main class B
> </stderr>

This seems to be Mac OS specific, since on my Linux it works perfectly well:
 $ java -DmyOption="A B C" -jar jmh-samples/target/benchmarks.jar
".*01_HelloWorld.*"

> The only workaround I?ve found is to use Java API:
>   Options options = new OptionsBuilder().jvmArgs("-Dmy.property=\"A B C\"").build();

...this points to the shell escaping troubles in your shell, when you
pass the command line. Try to meddle with escaping both quotes and the
backslashes? E.g.:

 java -jar target/benchmarks.jar -jvmArgs -Dmy.property=\\\"A B C\\\"

Thanks,
-Aleksey.



From ecki at zusammenkunft.net  Mon Aug 11 23:40:30 2014
From: ecki at zusammenkunft.net (Bernd Eckenfels)
Date: Tue, 12 Aug 2014 01:40:30 +0200
Subject: Partial RE for Benchmark Tests
In-Reply-To: <53E94EF4.7010801@oracle.com>
References: <20140812004545.00003937.ecki@zusammenkunft.net>
	<53E94EF4.7010801@oracle.com>
Message-ID: <20140812014030.00007ecd.ecki@zusammenkunft.net>

Hello Aleksey,

I think from the point of implementing, it is enough to use
Matcher.find() instead of matches().

It will find substrings/patterns unless you anchor them forcefully with
"^" or "$". This is typical behaviour for grep:

bash-3.1$ echo "abcd" | grep "^.*b.*$"
abcd
bash-3.1$ echo "abcd" | grep "^.*b$"
bash-3.1$ echo "abcd" | grep "^.*d$"
abcd
bash-3.1$ echo "abcd" | grep "c"
abcd

Gruss
Bernd


 Am Tue, 12 Aug 2014 03:17:08 +0400
schrieb Aleksey Shipilev <aleksey.shipilev at oracle.com>:

> Hi Bernd,
> 
> On 08/12/2014 02:45 AM, Bernd Eckenfels wrote:
> > JMH uses a regular expression to select the Benchmarks. While doing
> > so (as I understand it) it reuqires a full match. This has the
> > "disadvantage" that you always need to specify ".*MyBenchmark.*".
> > Not everybody is used to this syntax and it is really seldomly
> > needed.
> > 
> > How about changing this to a partial match with ^$ support: if you
> > specify an expression and it is found inside the benchmark name, the
> > benchmark will be selected. That way you can even specify Substrings
> > and be done 99% of the time: MyBenchmark. If you need to be more
> > speficic, you could: "^.*MyBenchmark\.[^.]+"
> > 
> > Is it worth coming up with a patch for it?
> 
> This is an interesting idea, but we need to think through what exactly
> do we want: more use-cases, including the corner cases? Is there a
> reference to what other regexp-accepting products do?
> 
> Your suggestion seems to amount to:
>  1) Look for an explicit "^" at the beginning, add ".*" at the
> beginning, otherwise.
>  2) Look for an explicit "$" at the end, add ".*" at the end,
> otherwise.
> 
> Thanks,
> -Aleksey.
> 
> 


From ecki at zusammenkunft.net  Mon Aug 11 23:45:54 2014
From: ecki at zusammenkunft.net (Bernd Eckenfels)
Date: Tue, 12 Aug 2014 01:45:54 +0200
Subject: Can't fork VM if option value contains space
In-Reply-To: <53E95028.4060909@oracle.com>
References: <43C4DC37-9DAC-4341-917B-BB1E7D333827@yahoo.com>
	<53E95028.4060909@oracle.com>
Message-ID: <20140812014554.0000441a.ecki@zusammenkunft.net>

Am Tue, 12 Aug 2014 03:22:16 +0400
schrieb Aleksey Shipilev <aleksey.shipilev at oracle.com>:
> ...this points to the shell escaping troubles in your shell, when you
> pass the command line. Try to meddle with escaping both quotes and the
> backslashes? E.g.:
> 
>  java -jar target/benchmarks.jar -jvmArgs -Dmy.property=\\\"A B C\\\"

Or try this: java -jar target/benchmarks.jar -jvmArgs '-Dmy.property="A
B C"'

Gruss
Bernd

From clement at unportant.info  Tue Aug 12 06:36:27 2014
From: clement at unportant.info (=?ISO-8859-1?Q?Cl=E9ment?= MATHIEU)
Date: Tue, 12 Aug 2014 08:36:27 +0200
Subject: Perfasm fails on Fedora/OpenJDK
In-Reply-To: <53E94C09.3070907@oracle.com>
References: <1407792995.17345.22.camel@station> <53E94C09.3070907@oracle.com>
Message-ID: <1407825387.17345.27.camel@station>

On Tue, 2014-08-12 at 03:04 +0400, Aleksey Shipilev wrote:
> Hi Clement,

> But, the funny thing is, we don't need this check at all, since perf is
> instrumenting the java process only. This check was used before to
> separate the "data" lines from the other garbage, but now there is a
> formatting check above. Therefore, we might as well remove it:
>  http://hg.openjdk.java.net/code-tools/jmh/rev/e739c4bf5798

Yup make sense. I was unsure why we were filtering the lines.

> Please check it works on your Fedora installation.

Works flawlessly.

BTW: The 'process' variable line 719 is no longer used and could be
removed.

Thanks,
Cl?ment MATHIEU



From aleksey.shipilev at oracle.com  Tue Aug 12 07:53:04 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Tue, 12 Aug 2014 11:53:04 +0400
Subject: Partial RE for Benchmark Tests
In-Reply-To: <20140812014030.00007ecd.ecki@zusammenkunft.net>
References: <20140812004545.00003937.ecki@zusammenkunft.net>	<53E94EF4.7010801@oracle.com>
	<20140812014030.00007ecd.ecki@zusammenkunft.net>
Message-ID: <53E9C7E0.1010406@oracle.com>

Okay, let's see how that turns out.

I think this is your entry point:

http://hg.openjdk.java.net/code-tools/jmh/file/e739c4bf5798/jmh-core/src/main/java/org/openjdk/jmh/runner/BenchmarkList.java#l95

We would also need a few new unit tests against BenchmarkList here:
 http://hg.openjdk.java.net/code-tools/jmh/file/e739c4bf5798/jmh-core/src/test/java/org/openjdk/jmh/runner/TestBenchmarkList.java

Thanks,
-Aleksey.

P.S. I see your OCA from Seeburger (b_eckenfels), would you use that? We
would need the mail with the patch to originate from the address
supplied in your OCA.

On 08/12/2014 03:40 AM, Bernd Eckenfels wrote:
> Hello Aleksey,
> 
> I think from the point of implementing, it is enough to use
> Matcher.find() instead of matches().
> 
> It will find substrings/patterns unless you anchor them forcefully with
> "^" or "$". This is typical behaviour for grep:
> 
> bash-3.1$ echo "abcd" | grep "^.*b.*$"
> abcd
> bash-3.1$ echo "abcd" | grep "^.*b$"
> bash-3.1$ echo "abcd" | grep "^.*d$"
> abcd
> bash-3.1$ echo "abcd" | grep "c"
> abcd
> 
> Gruss
> Bernd
> 
> 
>  Am Tue, 12 Aug 2014 03:17:08 +0400
> schrieb Aleksey Shipilev <aleksey.shipilev at oracle.com>:
> 
>> Hi Bernd,
>>
>> On 08/12/2014 02:45 AM, Bernd Eckenfels wrote:
>>> JMH uses a regular expression to select the Benchmarks. While doing
>>> so (as I understand it) it reuqires a full match. This has the
>>> "disadvantage" that you always need to specify ".*MyBenchmark.*".
>>> Not everybody is used to this syntax and it is really seldomly
>>> needed.
>>>
>>> How about changing this to a partial match with ^$ support: if you
>>> specify an expression and it is found inside the benchmark name, the
>>> benchmark will be selected. That way you can even specify Substrings
>>> and be done 99% of the time: MyBenchmark. If you need to be more
>>> speficic, you could: "^.*MyBenchmark\.[^.]+"
>>>
>>> Is it worth coming up with a patch for it?
>>
>> This is an interesting idea, but we need to think through what exactly
>> do we want: more use-cases, including the corner cases? Is there a
>> reference to what other regexp-accepting products do?
>>
>> Your suggestion seems to amount to:
>>  1) Look for an explicit "^" at the beginning, add ".*" at the
>> beginning, otherwise.
>>  2) Look for an explicit "$" at the end, add ".*" at the end,
>> otherwise.
>>
>> Thanks,
>> -Aleksey.
>>
>>
> 



From aleksey.shipilev at oracle.com  Tue Aug 12 07:54:51 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Tue, 12 Aug 2014 11:54:51 +0400
Subject: Perfasm fails on Fedora/OpenJDK
In-Reply-To: <1407825387.17345.27.camel@station>
References: <1407792995.17345.22.camel@station>	 <53E94C09.3070907@oracle.com>
	<1407825387.17345.27.camel@station>
Message-ID: <53E9C84B.4050702@oracle.com>

On 08/12/2014 10:36 AM, Cl?ment MATHIEU wrote:
> On Tue, 2014-08-12 at 03:04 +0400, Aleksey Shipilev wrote:
>> Please check it works on your Fedora installation.
> 
> Works flawlessly.

Excellent. Would be the part of next patch release.

> BTW: The 'process' variable line 719 is no longer used and could be
> removed.

I would rather keep it to assign meaning to 0-th element of the parsed
array.

-Aleksey.



From vyazelenko at yahoo.com  Tue Aug 12 10:24:29 2014
From: vyazelenko at yahoo.com (vyazelenko at yahoo.com)
Date: Tue, 12 Aug 2014 12:24:29 +0200
Subject: Can't fork VM if option value contains space
In-Reply-To: <20140812014554.0000441a.ecki@zusammenkunft.net>
References: <43C4DC37-9DAC-4341-917B-BB1E7D333827@yahoo.com>
	<53E95028.4060909@oracle.com>
	<20140812014554.0000441a.ecki@zusammenkunft.net>
Message-ID: <9840718B-DAA4-4561-A5D7-F916E593EF0D@yahoo.com>

The solution from Bernd with single quotes around entire property does not work (at least on Windows), because entire thing is passed as String:
    VM options: '-Dmy.prop=A B C'
Note also that double quotes are gone.

@Aleksey:
- Original issue described is reproducible on Windows and Mac OS X. Try to pass property via -jvmArgs instead of passing it directly to java launcher.
- Quoting the quotes, i.e.:
    -jvmArgs -Dmy.prop="\"A B C\""
Works on Windows but fails on Mac OS X.
- Using runner API just works. :)

Best regards,
Dmitry

Sent from my iPhone

> On Aug 12, 2014, at 1:45, Bernd Eckenfels <ecki at zusammenkunft.net> wrote:
> 
> Am Tue, 12 Aug 2014 03:22:16 +0400
> schrieb Aleksey Shipilev <aleksey.shipilev at oracle.com>:
>> ...this points to the shell escaping troubles in your shell, when you
>> pass the command line. Try to meddle with escaping both quotes and the
>> backslashes? E.g.:
>> 
>> java -jar target/benchmarks.jar -jvmArgs -Dmy.property=\\\"A B C\\\"
> 
> Or try this: java -jar target/benchmarks.jar -jvmArgs '-Dmy.property="A
> B C"'
> 
> Gruss
> Bernd

From nikita.artyushov at gmail.com  Tue Aug 12 10:25:31 2014
From: nikita.artyushov at gmail.com (Nikita Artyushov)
Date: Tue, 12 Aug 2014 14:25:31 +0400
Subject: JMH plugin for IDEA
Message-ID: <CAO6=6RUuVE-8quqdcbL6+8RQO=QXAFf3yBPSkuWMGFkRyYct+A@mail.gmail.com>

Hi all,

I have built a prototype for an IntelliJ IDEA plugin for JMH. It
currently supports the following features:

* Creating new benchmarks via a shortcut
* Running existing benchmarks straight from your IDE

The plugin is available from the official JetBrains plugin repository.
Search for JMH and it will be there. You can also check the sources
out: https://github.com/artyushov/idea-jmh-plugin. The documentation
is there as well.

Everyone is very welcome to try it. All the feedback and feature
requests will be much appreciated. Feel free to submit an issue or a
pull request on GitHub or contact me directly.

Nikita

From ecki at zusammenkunft.net  Tue Aug 12 10:51:39 2014
From: ecki at zusammenkunft.net (Bernd Eckenfels)
Date: Tue, 12 Aug 2014 12:51:39 +0200
Subject: AW: Can't fork VM if option value contains space
In-Reply-To: <9840718B-DAA4-4561-A5D7-F916E593EF0D@yahoo.com>
References: <43C4DC37-9DAC-4341-917B-BB1E7D333827@yahoo.com>
	<53E95028.4060909@oracle.com>
	<20140812014554.0000441a.ecki@zusammenkunft.net>
	<9840718B-DAA4-4561-A5D7-F916E593EF0D@yahoo.com>
Message-ID: <53e9f1c4.e4578c0a.42fc.ffffeba6@mx.google.com>

It us not meant as a solution for Windows. There is no useful semantics in the command parser of Windows, just dont use it for stuff like that. It should work on most shells (mac,linux)

having said that i somewhat wonder why the original syntax did not work on OSX, it should preserv blanks when passed on with the right process API and no "/bin/sh -c".

gruss
Bernd

-- 
http://bernd.eckenfels.net

----- Urspr?ngliche Nachricht -----
Von: "vyazelenko at yahoo.com" <vyazelenko at yahoo.com>
Gesendet: ?12.?08.?2014 12:24
An: "Bernd Eckenfels" <ecki at zusammenkunft.net>
Cc: "Aleksey Shipilev" <aleksey.shipilev at oracle.com>; "jmh-dev at openjdk.java.net" <jmh-dev at openjdk.java.net>
Betreff: Re: Can't fork VM if option value contains space

The solution from Bernd with single quotes around entire property does not work (at least on Windows), because entire thing is passed as String:
    VM options: '-Dmy.prop=A B C'
Note also that double quotes are gone.

@Aleksey:
- Original issue described is reproducible on Windows and Mac OS X. Try to pass property via -jvmArgs instead of passing it directly to java launcher.
- Quoting the quotes, i.e.:
    -jvmArgs -Dmy.prop="\"A B C\""
Works on Windows but fails on Mac OS X.
- Using runner API just works. :)

Best regards,
Dmitry

Sent from my iPhone

> On Aug 12, 2014, at 1:45, Bernd Eckenfels <ecki at zusammenkunft.net> wrote:
> 
> Am Tue, 12 Aug 2014 03:22:16 +0400
> schrieb Aleksey Shipilev <aleksey.shipilev at oracle.com>:
>> ...this points to the shell escaping troubles in your shell, when you
>> pass the command line. Try to meddle with escaping both quotes and the
>> backslashes? E.g.:
>> 
>> java -jar target/benchmarks.jar -jvmArgs -Dmy.property=\\\"A B C\\\"
> 
> Or try this: java -jar target/benchmarks.jar -jvmArgs '-Dmy.property="A
> B C"'
> 
> Gruss
> Bernd

From vyazelenko at yahoo.com  Tue Aug 12 11:33:44 2014
From: vyazelenko at yahoo.com (Dmitry Vyazelenko)
Date: Tue, 12 Aug 2014 04:33:44 -0700
Subject: AW: Can't fork VM if option value contains space
In-Reply-To: <53e9f1c4.e4578c0a.42fc.ffffeba6@mx.google.com>
References: <43C4DC37-9DAC-4341-917B-BB1E7D333827@yahoo.com>	<53E95028.4060909@oracle.com>	<20140812014554.0000441a.ecki@zusammenkunft.net>	<9840718B-DAA4-4561-A5D7-F916E593EF0D@yahoo.com>
	<53e9f1c4.e4578c0a.42fc.ffffeba6@mx.google.com>
Message-ID: <1407843224.38893.YahooMailNeo@web162205.mail.bf1.yahoo.com>

Well it is even more strange on Windows:
- On command line:
? FAILS: java -jar target\benchmarks.jar -jvmArgs -Dmy.prop="A B C"
? WORKS: java -jar target\benchmarks.jar -jvmArgs -Dmy.prop="\"A B C\""

- Using Java API:
? FAILS: new OptionsBuilder() 
? ? ? ? ? ? ? ? .jvmArgs("-Dmy.prop=\"A B C\"").build();
??
? WORKS: new OptionsBuilder() 
? ? ? ? ? ? ? ? .jvmArgs("-Dmy.prop=\"\"A B C\"\"").build();

? FAILS: new OptionsBuilder() 
? ? ? ? ? ? ? ? .jvmArgs("-Dmy.prop=\"\\\"A B C\\\"\"").build();
?
Best regards,
Dmitry



On Tuesday, August 12, 2014 12:52 PM, Bernd Eckenfels <ecki at zusammenkunft.net> wrote:
It us not meant as a solution for Windows. There is no useful semantics in the command parser of Windows, just dont use it for stuff like that. It should work on most shells (mac,linux)

having said that i somewhat wonder why the original syntax did not work on OSX, it should preserv blanks when passed on with the right process API and no "/bin/sh -c".

gruss
Bernd

-- 
http://bernd.eckenfels.net




----- Urspr?ngliche Nachricht -----
Von: "vyazelenko at yahoo.com" <vyazelenko at yahoo.com>
Gesendet: ?12.?08.?2014 12:24
An: "Bernd Eckenfels" <ecki at zusammenkunft.net>
Cc: "Aleksey Shipilev" <aleksey.shipilev at oracle.com>; "jmh-dev at openjdk.java.net" <jmh-dev at openjdk.java.net>
Betreff: Re: Can't fork VM if option value contains space

The solution from Bernd with single quotes around entire property does not work (at least on Windows), because entire thing is passed as String:
? ? VM options: '-Dmy.prop=A B C'
Note also that double quotes are gone.

@Aleksey:
- Original issue described is reproducible on Windows and Mac OS X. Try to pass property via -jvmArgs instead of passing it directly to java launcher.
- Quoting the quotes, i.e.:
? ? -jvmArgs -Dmy.prop="\"A B C\""
Works on Windows but fails on Mac OS X.
- Using runner API just works. :)

Best regards,
Dmitry

Sent from my iPhone

> On Aug 12, 2014, at 1:45, Bernd Eckenfels <ecki at zusammenkunft.net> wrote:
> 
> Am Tue, 12 Aug 2014 03:22:16 +0400
> schrieb Aleksey Shipilev <aleksey.shipilev at oracle.com>:
>> ...this points to the shell escaping troubles in your shell, when you
>> pass the command line. Try to meddle with escaping both quotes and the
>> backslashes? E.g.:
>> 
>> java -jar target/benchmarks.jar -jvmArgs -Dmy.property=\\\"A B C\\\"
> 
> Or try this: java -jar target/benchmarks.jar -jvmArgs '-Dmy.property="A
> B C"'
> 
> Gruss
> Bernd

From vyazelenko at yahoo.com  Tue Aug 12 12:09:18 2014
From: vyazelenko at yahoo.com (Dmitry Vyazelenko)
Date: Tue, 12 Aug 2014 05:09:18 -0700
Subject: AW: Can't fork VM if option value contains space
In-Reply-To: <1407843224.38893.YahooMailNeo@web162205.mail.bf1.yahoo.com>
References: <43C4DC37-9DAC-4341-917B-BB1E7D333827@yahoo.com>	<53E95028.4060909@oracle.com>	<20140812014554.0000441a.ecki@zusammenkunft.net>	<9840718B-DAA4-4561-A5D7-F916E593EF0D@yahoo.com>	<53e9f1c4.e4578c0a.42fc.ffffeba6@mx.google.com>
	<1407843224.38893.YahooMailNeo@web162205.mail.bf1.yahoo.com>
Message-ID: <1407845358.64545.YahooMailNeo@web162203.mail.bf1.yahoo.com>

Update (see below).


On Tuesday, August 12, 2014 1:34 PM, Dmitry Vyazelenko <vyazelenko at yahoo.com> wrote:
Well it is even more strange on Windows:
- On command line:
? FAILS: java -jar target\benchmarks.jar -jvmArgs -Dmy.prop="A B C"
? Does not start benchmark execution.

? WORKS: java -jar target\benchmarks.jar -jvmArgs -Dmy.prop="\"A B C\""
? Runs and produces this result:
? ? ? ? ?Iteration   1: my.prop ==> A B C

- Using Java API:
? FAILS: new OptionsBuilder() 
? ? ? ? ? ? ? ? .jvmArgs("-Dmy.prop=\"A B C\"").build();
? Does not fork VM.
??
? WORKS: new OptionsBuilder() 
? ? ? ? ? ? ? ? .jvmArgs("-Dmy.prop=\"\"A B C\"\"").build();
? Runs but produces wrong result:
? ? ? ? ?Iteration   1: my.prop ==> "A B C"

? FAILS: new OptionsBuilder() 
? ? ? ? ? ? ? ? .jvmArgs("-Dmy.prop=\"\\\"A B C\\\"\"").build();
? Does not fork VM. 

? RUNS: new OptionsBuilder() 
? ? ? ? ? ? ? ? .jvmArgs("-Dmy.prop=\\\"A B C\\\"").build(); ?
? Runs but produces wrong result: 
? ? ? ? ?Iteration 1: my.prop ==> "A B C" 

? WORKS (no quoting): new OptionsBuilder() 
? ? ? ? ? ? ? ? .jvmArgs("-Dmy.prop=A B C").build();
? Runs and produces correct result: 
? ? ? ? ?Iteration 1: my.prop ==> A B C"
?

Best regards,
Dmitry






On Tuesday, August 12, 2014 12:52 PM, Bernd Eckenfels <ecki at zusammenkunft.net> wrote:
It us not meant as a solution for Windows. There is no useful semantics in the command parser of Windows, just dont use it for stuff like that. It should work on most shells (mac,linux)

having said that i somewhat wonder why the original syntax did not work on OSX, it should preserv blanks when passed on with the right process API and no "/bin/sh -c".

gruss
Bernd

-- 
http://bernd.eckenfels.net




----- Urspr?ngliche Nachricht -----
Von: "vyazelenko at yahoo.com" <vyazelenko at yahoo.com>
Gesendet: ?12.?08.?2014 12:24
An: "Bernd Eckenfels" <ecki at zusammenkunft.net>
Cc: "Aleksey Shipilev" <aleksey.shipilev at oracle.com>; "jmh-dev at openjdk.java.net" <jmh-dev at openjdk.java.net>
Betreff: Re: Can't fork VM if option value contains space

The solution from Bernd with single quotes around entire property does not work (at least on Windows), because entire thing is passed as String:
? ? VM options: '-Dmy.prop=A B C'
Note also that double quotes are gone.

@Aleksey:
- Original issue described is reproducible on Windows and Mac OS X. Try to pass property via -jvmArgs instead of passing it directly to java launcher.
- Quoting the quotes, i.e.:
? ? -jvmArgs -Dmy.prop="\"A B C\""
Works on Windows but fails on Mac OS X.
- Using runner API just works. :)

Best regards,
Dmitry

Sent from my iPhone

> On Aug 12, 2014, at 1:45, Bernd Eckenfels <ecki at zusammenkunft.net> wrote:
> 
> Am Tue, 12 Aug 2014 03:22:16 +0400
> schrieb Aleksey Shipilev <aleksey.shipilev at oracle.com>:
>> ...this points to the shell escaping troubles in your shell, when you
>> pass the command line. Try to meddle with escaping both quotes and the
>> backslashes? E.g.:
>> 
>> java -jar target/benchmarks.jar -jvmArgs -Dmy.property=\\\"A B C\\\"
> 
> Or try this: java -jar target/benchmarks.jar -jvmArgs '-Dmy.property="A
> B C"'
> 
> Gruss
> Bernd

From ecki at zusammenkunft.net  Wed Aug 13 00:21:48 2014
From: ecki at zusammenkunft.net (Bernd Eckenfels)
Date: Wed, 13 Aug 2014 02:21:48 +0200
Subject: AW: Can't fork VM if option value contains space
In-Reply-To: <1407843224.38893.YahooMailNeo@web162205.mail.bf1.yahoo.com>
References: <43C4DC37-9DAC-4341-917B-BB1E7D333827@yahoo.com>
	<53E95028.4060909@oracle.com>
	<20140812014554.0000441a.ecki@zusammenkunft.net>
	<9840718B-DAA4-4561-A5D7-F916E593EF0D@yahoo.com>
	<53e9f1c4.e4578c0a.42fc.ffffeba6@mx.google.com>
	<1407843224.38893.YahooMailNeo@web162205.mail.bf1.yahoo.com>
Message-ID: <20140813022148.00004deb.ecki@zusammenkunft.net>

Am Tue, 12 Aug 2014 04:33:44 -0700
schrieb Dmitry Vyazelenko <vyazelenko at yahoo.com>:

I would use (untested)

 ? ? ? ? ? ? ? ? .jvmArgs("-Dmy.prop=A B C").build();

Because the Process.execute(String[]) API works with blanks in
arguments.

Gruss
Bernd

From vyazelenko at yahoo.com  Wed Aug 13 06:02:49 2014
From: vyazelenko at yahoo.com (vyazelenko at yahoo.com)
Date: Wed, 13 Aug 2014 08:02:49 +0200
Subject: AW: Can't fork VM if option value contains space
In-Reply-To: <20140813022148.00004deb.ecki@zusammenkunft.net>
References: <43C4DC37-9DAC-4341-917B-BB1E7D333827@yahoo.com>
	<53E95028.4060909@oracle.com>
	<20140812014554.0000441a.ecki@zusammenkunft.net>
	<9840718B-DAA4-4561-A5D7-F916E593EF0D@yahoo.com>
	<53e9f1c4.e4578c0a.42fc.ffffeba6@mx.google.com>
	<1407843224.38893.YahooMailNeo@web162205.mail.bf1.yahoo.com>
	<20140813022148.00004deb.ecki@zusammenkunft.net>
Message-ID: <214FF88A-DE7B-4EA4-B7AE-580DE9983E4B@yahoo.com>

Yeah. That's exactly what I use.

Regards,
Dmitry

Sent from my iPhone

> On Aug 13, 2014, at 2:21, Bernd Eckenfels <ecki at zusammenkunft.net> wrote:
> 
> Am Tue, 12 Aug 2014 04:33:44 -0700
> schrieb Dmitry Vyazelenko <vyazelenko at yahoo.com>:
> 
> I would use (untested)
> 
>                 .jvmArgs("-Dmy.prop=A B C").build();
> 
> Because the Process.execute(String[]) API works with blanks in
> arguments.
> 
> Gruss
> Bernd

From aleksey.shipilev at oracle.com  Wed Aug 13 07:57:57 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Wed, 13 Aug 2014 07:57:57 +0000
Subject: hg: code-tools/jmh: Refresh BenchmarkMode meanings in
	JMHSample_02_BenchmarkModes.
Message-ID: <201408130757.s7D7vvCY017476@aojmv0008>

Changeset: 60e8aae7ac7d
Author:    shade
Date:      2014-08-13 11:57 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/60e8aae7ac7d

Refresh BenchmarkMode meanings in JMHSample_02_BenchmarkModes.

! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_02_BenchmarkModes.java


From aleksey.shipilev at oracle.com  Wed Aug 13 10:04:43 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Wed, 13 Aug 2014 14:04:43 +0400
Subject: JMH plugin for IDEA
In-Reply-To: <CAO6=6RUuVE-8quqdcbL6+8RQO=QXAFf3yBPSkuWMGFkRyYct+A@mail.gmail.com>
References: <CAO6=6RUuVE-8quqdcbL6+8RQO=QXAFf3yBPSkuWMGFkRyYct+A@mail.gmail.com>
Message-ID: <53EB383B.6040905@oracle.com>

Awesome!

I started the "Community-supported projects" section on JMH home page,
and linked the project there:
 http://openjdk.java.net/projects/code-tools/jmh/

-Aleksey.

On 08/12/2014 02:25 PM, Nikita Artyushov wrote:
> Hi all,
> 
> I have built a prototype for an IntelliJ IDEA plugin for JMH. It
> currently supports the following features:
> 
> * Creating new benchmarks via a shortcut
> * Running existing benchmarks straight from your IDE
> 
> The plugin is available from the official JetBrains plugin repository.
> Search for JMH and it will be there. You can also check the sources
> out: https://github.com/artyushov/idea-jmh-plugin. The documentation
> is there as well.
> 
> Everyone is very welcome to try it. All the feedback and feature
> requests will be much appreciated. Feel free to submit an issue or a
> pull request on GitHub or contact me directly.
> 
> Nikita
> 



From aleksey.shipilev at oracle.com  Wed Aug 13 10:45:59 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Wed, 13 Aug 2014 14:45:59 +0400
Subject: AW: Can't fork VM if option value contains space
In-Reply-To: <1407845358.64545.YahooMailNeo@web162203.mail.bf1.yahoo.com>
References: <43C4DC37-9DAC-4341-917B-BB1E7D333827@yahoo.com>	<53E95028.4060909@oracle.com>	<20140812014554.0000441a.ecki@zusammenkunft.net>	<9840718B-DAA4-4561-A5D7-F916E593EF0D@yahoo.com>	<53e9f1c4.e4578c0a.42fc.ffffeba6@mx.google.com>	<1407843224.38893.YahooMailNeo@web162205.mail.bf1.yahoo.com>
	<1407845358.64545.YahooMailNeo@web162203.mail.bf1.yahoo.com>
Message-ID: <53EB41E7.2040201@oracle.com>

Ah, got it. This is the impedance in -jvmArgs handling.

In all other places, including the Java API and annotations, we pass
String[], where each individual String is the argument (may be with
spaces). This fits nicely into Runtime.exec(String cmd, String[] args)
API, and it will handle the spaces correctly.

However, for -jvmArgs, we only get String, and then we blindly split it
by space to produce String[]:
 http://hg.openjdk.java.net/code-tools/jmh/file/60e8aae7ac7d/jmh-core/src/main/java/org/openjdk/jmh/runner/options/CommandLineOptions.java#l405

Therefore:

On 08/12/2014 04:09 PM, Dmitry Vyazelenko wrote:
> On Tuesday, August 12, 2014 1:34 PM, Dmitry Vyazelenko <vyazelenko at yahoo.com> wrote:
> Well it is even more strange on Windows:
> - On command line:
>   FAILS: java -jar target\benchmarks.jar -jvmArgs -Dmy.prop="A B C"
>   Does not start benchmark execution.

...forked command line has individual argument "B", which fails, because
VM thinks "B" is the Java class name.

>   WORKS: java -jar target\benchmarks.jar -jvmArgs -Dmy.prop="\"A B C\""
>   Runs and produces this result:
>          Iteration   1: my.prop ==> A B C

...forked command line also has "B" as the individual argument, but VM
launcher seem to reconstruct the "-DmyProp=\"A", "B", "C\"" into
"-DmyProp=A B C".

> - Using Java API:
>   FAILS: new OptionsBuilder() 
>                 .jvmArgs("-Dmy.prop=\"A B C\"").build();
>   Does not fork VM.

...one level of quotes seem to be digested by Runtime.exec.

>   WORKS: new OptionsBuilder() 
>                 .jvmArgs("-Dmy.prop=\"\"A B C\"\"").build();
>   Runs but produces wrong result:
>          Iteration   1: my.prop ==> "A B C"

...putting another level of quotes helps.

>   RUNS: new OptionsBuilder() 
>                 .jvmArgs("-Dmy.prop=\\\"A B C\\\"").build();  
>   Runs but produces wrong result: 
>          Iteration 1: my.prop ==> "A B C" 

...the quotes are properly escaped from Runtime.exec.

>   WORKS (no quoting): new OptionsBuilder() 
>                 .jvmArgs("-Dmy.prop=A B C").build();
>   Runs and produces correct result: 
>          Iteration 1: my.prop ==> A B C"

...and this is a correct and most straight-forward scenario.

All this feels rather messy cross-cutting issue across multiple shells,
VMs, etc. At this point, I am a bit puzzled if we can do anything about it.

Thanks,
-Aleksey.



From vyazelenko at yahoo.com  Wed Aug 13 10:48:59 2014
From: vyazelenko at yahoo.com (Dmitry Vyazelenko)
Date: Wed, 13 Aug 2014 03:48:59 -0700
Subject: AW: Can't fork VM if option value contains space
In-Reply-To: <53EB41E7.2040201@oracle.com>
References: <43C4DC37-9DAC-4341-917B-BB1E7D333827@yahoo.com>	<53E95028.4060909@oracle.com>	<20140812014554.0000441a.ecki@zusammenkunft.net>	<9840718B-DAA4-4561-A5D7-F916E593EF0D@yahoo.com>	<53e9f1c4.e4578c0a.42fc.ffffeba6@mx.google.com>	<1407843224.38893.YahooMailNeo@web162205.mail.bf1.yahoo.com>
	<1407845358.64545.YahooMailNeo@web162203.mail.bf1.yahoo.com>
	<53EB41E7.2040201@oracle.com>
Message-ID: <1407926939.15879.YahooMailNeo@web162205.mail.bf1.yahoo.com>

Hi Aleksey,

Using Java API is OK for me.
?
Best regards,
Dmitry



On Wednesday, August 13, 2014 12:45 PM, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:
Ah, got it. This is the impedance in -jvmArgs handling.

In all other places, including the Java API and annotations, we pass
String[], where each individual String is the argument (may be with
spaces). This fits nicely into Runtime.exec(String cmd, String[] args)
API, and it will handle the spaces correctly.

However, for -jvmArgs, we only get String, and then we blindly split it
by space to produce String[]:
http://hg.openjdk.java.net/code-tools/jmh/file/60e8aae7ac7d/jmh-core/src/main/java/org/openjdk/jmh/runner/options/CommandLineOptions.java#l405

Therefore:

On 08/12/2014 04:09 PM, Dmitry Vyazelenko wrote:
> On Tuesday, August 12, 2014 1:34 PM, Dmitry Vyazelenko <vyazelenko at yahoo.com> wrote:
> Well it is even more strange on Windows:
> - On command line:
>?  FAILS: java -jar target\benchmarks.jar -jvmArgs -Dmy.prop="A B C"
>?  Does not start benchmark execution.

...forked command line has individual argument "B", which fails, because
VM thinks "B" is the Java class name.

>?  WORKS: java -jar target\benchmarks.jar -jvmArgs -Dmy.prop="\"A B C\""
>?  Runs and produces this result:
>? ? ? ? ? Iteration?  1: my.prop ==> A B C

...forked command line also has "B" as the individual argument, but VM
launcher seem to reconstruct the "-DmyProp=\"A", "B", "C\"" into
"-DmyProp=A B C".

> - Using Java API:
>?  FAILS: new OptionsBuilder() 
>? ? ? ? ? ? ? ?  .jvmArgs("-Dmy.prop=\"A B C\"").build();
>?  Does not fork VM.

...one level of quotes seem to be digested by Runtime.exec.

>?  WORKS: new OptionsBuilder() 
>? ? ? ? ? ? ? ?  .jvmArgs("-Dmy.prop=\"\"A B C\"\"").build();
>?  Runs but produces wrong result:
>? ? ? ? ? Iteration?  1: my.prop ==> "A B C"

...putting another level of quotes helps.

>?  RUNS: new OptionsBuilder() 
>? ? ? ? ? ? ? ?  .jvmArgs("-Dmy.prop=\\\"A B C\\\"").build();? 
>?  Runs but produces wrong result: 
>? ? ? ? ? Iteration 1: my.prop ==> "A B C" 

...the quotes are properly escaped from Runtime.exec.

>?  WORKS (no quoting): new OptionsBuilder() 
>? ? ? ? ? ? ? ?  .jvmArgs("-Dmy.prop=A B C").build();
>?  Runs and produces correct result: 
>? ? ? ? ? Iteration 1: my.prop ==> A B C"

...and this is a correct and most straight-forward scenario.

All this feels rather messy cross-cutting issue across multiple shells,
VMs, etc. At this point, I am a bit puzzled if we can do anything about it.

Thanks,



-Aleksey.

From vyazelenko at yahoo.com  Wed Aug 13 11:06:00 2014
From: vyazelenko at yahoo.com (Dmitry Vyazelenko)
Date: Wed, 13 Aug 2014 04:06:00 -0700
Subject: Setup and TearDown executed by different threads
Message-ID: <1407927960.94775.YahooMailNeo@web162204.mail.bf1.yahoo.com>

Hi,

I've found a problem in multi-threaded benchmarks that @Setup and
@TearDown are executed by different threads. This breaks the teardown
code, because it assumes that same thread that executed @Setup will?
execute @TearDown method.

Here is benchmark that illustrates the issue:
@Fork(1) 
@Warmup(iterations = 0) 
@Measurement(iterations = 1) 
@State(Scope.Benchmark) 
@BenchmarkMode(Mode.SingleShotTime) 
public class SetupAndTearDownBenchmarkScope { 
? ? private String trialThreadName; 
? ? private String iterationsThreadName; 
? ? private String invocationThreadName; 
? ? @Benchmark 
? ? public void baseline() { 
? ? } 
? ? private void printThread(String msg) { 
? ? ? ? System.out.println("\r\n[" + Thread.currentThread().getName() + "] => " + msg); 
? ? } 
? ? @Setup(Level.Trial) 
? ? public void setUpTrial() { 
? ? ? ? printThread("setUpTrial()"); 
? ? } 
? ? @TearDown(Level.Trial) 
? ? public void tearDownTrial() { 
? ? ? ? printThread("tearDownTrial()"); 
? ? } 
? ? @Setup(Level.Iteration) 
? ? public void setUpIteration() { 
? ? ? ? printThread("setUpIteration()"); 
? ? } 
? ? @TearDown(Level.Iteration) 
? ? public void tearDownIteration() { 
? ? ? ? printThread("tearDownIteration()"); 
? ? } 
? ? @Setup(Level.Invocation) 
? ? public void setUpInvocation() { 
? ? ? ? printThread("setUpInvocation()"); 
? ? } 
? ? @TearDown(Level.Invocation) 
? ? public void tearDownInvocation() { 
? ? ? ? printThread("tearDownInvocation()"); 
? ? } 
}

When I execute it using single thread (i.e. java -jar target\benchmarks.jar -t 1) I get:
? [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-1] => setUpTrial() 
? [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-1] => setUpIteration() 
? [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-1] => setUpInvocation() 
? [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-1] => tearDownInvocation() 
? [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-1] => tearDownIteration() 
? [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-1] => tearDownTrial()
Which basically confirms that same thread used for everything.

However with multiple threads (i.e. java -jar target\benchmarks.jar -t 4):
? [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-3] => setUpTrial() 
? [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-1] => setUpIteration() 
? [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-1] => setUpInvocation() 
? [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-1] => tearDownInvocation() 
? [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-3] => setUpInvocation() 
? [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-3] => tearDownInvocation() 
? [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-2] => setUpInvocation() 
? [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-2] => tearDownInvocation() 
? [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-2] => tearDownIteration() 
? [org.sample.SetupAndTearDownBenchmarkScope.baseline-jmh-worker-2] => tearDownTrial()

This clearly shows that setup/tearDown pairs are executed by different threads!

What I expect:
- For Scope.Benchmark single thread will execute all @Setup/@TearDown methods
(at least for Level.Trial and Level.Iteration, not sure about Level.Invocation though)
- For Scope.Thread each thread will execute it's own @Setup/@TearDown methods
- For Scope.Group probably use one of threads from the group? Or dedicated @Setup/@TrearDown
thread created in addition.

?
Best regards,
Dmitry

From aleksey.shipilev at oracle.com  Wed Aug 13 11:25:25 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Wed, 13 Aug 2014 15:25:25 +0400
Subject: Setup and TearDown executed by different threads
In-Reply-To: <1407927960.94775.YahooMailNeo@web162204.mail.bf1.yahoo.com>
References: <1407927960.94775.YahooMailNeo@web162204.mail.bf1.yahoo.com>
Message-ID: <53EB4B25.6090708@oracle.com>

On 08/13/2014 03:06 PM, Dmitry Vyazelenko wrote:
> I've found a problem in multi-threaded benchmarks that @Setup and
> @TearDown are executed by different threads. This breaks the teardown
> code, because it assumes that same thread that executed @Setup will 
> execute @TearDown method.

If the @State is shared between the threads, you should not make
assumptions about who actually accessed the @State object and its
fixture methods.

> What I expect:
> - For Scope.Benchmark single thread will execute all @Setup/@TearDown methods
> (at least for Level.Trial and Level.Iteration, not sure about Level.Invocation though)

Ah, but that expectation is incorrect.

See
http://hg.openjdk.java.net/code-tools/jmh/file/60e8aae7ac7d/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_05_StateFixtures.java#l50.

"As with the State, fixture methods are only called by those benchmark
threads which are using the state." <-- That is, any thread which has
the access to @State can call the fixture method. This is analogous to
what you suggest from Scope.Group:

> - For Scope.Group probably use one of threads from the group? Or dedicated @Setup/@TrearDown
> thread created in addition.

Is there a convincing use case that shows we should track
@Setup/@TearDown callers for Scope.Benchmark?

Thanks,
-Aleksey


From vyazelenko at yahoo.com  Wed Aug 13 11:41:38 2014
From: vyazelenko at yahoo.com (vyazelenko at yahoo.com)
Date: Wed, 13 Aug 2014 13:41:38 +0200
Subject: Setup and TearDown executed by different threads
In-Reply-To: <53EB4B25.6090708@oracle.com>
References: <1407927960.94775.YahooMailNeo@web162204.mail.bf1.yahoo.com>
	<53EB4B25.6090708@oracle.com>
Message-ID: <DF545D37-5FD8-4492-85AB-2E5A3DAEBC42@yahoo.com>

My use-case: start server in @Setup(Level.Trial) and shutdown server in @TearDown(Level.Trial).
And of course I want to setup some data for entire benchmark and remove it at the end.
In other words I'm looking for a way to perform per-Benchmark work irrespective of @State configuration.
And this work should be done by the same thread. In my case there is a session attached via ThreadLocal in @Setup.

I hope this clarifies the intent.

Best regards,
Dmitry

Sent from my iPhone

> On Aug 13, 2014, at 13:25, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:
> 
>> On 08/13/2014 03:06 PM, Dmitry Vyazelenko wrote:
>> I've found a problem in multi-threaded benchmarks that @Setup and
>> @TearDown are executed by different threads. This breaks the teardown
>> code, because it assumes that same thread that executed @Setup will 
>> execute @TearDown method.
> 
> If the @State is shared between the threads, you should not make
> assumptions about who actually accessed the @State object and its
> fixture methods.
> 
>> What I expect:
>> - For Scope.Benchmark single thread will execute all @Setup/@TearDown methods
>> (at least for Level.Trial and Level.Iteration, not sure about Level.Invocation though)
> 
> Ah, but that expectation is incorrect.
> 
> See
> http://hg.openjdk.java.net/code-tools/jmh/file/60e8aae7ac7d/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_05_StateFixtures.java#l50.
> 
> "As with the State, fixture methods are only called by those benchmark
> threads which are using the state." <-- That is, any thread which has
> the access to @State can call the fixture method. This is analogous to
> what you suggest from Scope.Group:
> 
>> - For Scope.Group probably use one of threads from the group? Or dedicated @Setup/@TrearDown
>> thread created in addition.
> 
> Is there a convincing use case that shows we should track
> @Setup/@TearDown callers for Scope.Benchmark?
> 
> Thanks,
> -Aleksey
> 

From aleksey.shipilev at oracle.com  Wed Aug 13 11:48:50 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Wed, 13 Aug 2014 15:48:50 +0400
Subject: Setup and TearDown executed by different threads
In-Reply-To: <DF545D37-5FD8-4492-85AB-2E5A3DAEBC42@yahoo.com>
References: <1407927960.94775.YahooMailNeo@web162204.mail.bf1.yahoo.com>
	<53EB4B25.6090708@oracle.com>
	<DF545D37-5FD8-4492-85AB-2E5A3DAEBC42@yahoo.com>
Message-ID: <53EB50A2.8020401@oracle.com>

On 08/13/2014 03:41 PM, vyazelenko at yahoo.com wrote:
> My use-case: start server in @Setup(Level.Trial) and shutdown server
> in @TearDown(Level.Trial). And of course I want to setup some data
> for entire benchmark and remove it at the end. In other words I'm
> looking for a way to perform per-Benchmark work irrespective of
> @State configuration. 

Aha. @State(Scope.Benchmark) it is.

> And this work should be done by the same
> thread. In my case there is a session attached via ThreadLocal in
> @Setup.

Thinking out loud now.

Scope.Benchmark assumes the state is shared between the threads, which
forces users to do additional housekeeping when accessed from multiple
threads. My concern is that tracking the @Setup/@TearDown callers would
blur this for users: instead of blowing up early, users will experience
weird visibility problems caused by the lack of appropriate
synchronization.

I think you can do something like that with:
http://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_29_StatesDAG.java

-Aleksey.



From vyazelenko at yahoo.com  Wed Aug 13 12:40:20 2014
From: vyazelenko at yahoo.com (Dmitry Vyazelenko)
Date: Wed, 13 Aug 2014 05:40:20 -0700
Subject: Setup and TearDown executed by different threads
In-Reply-To: <53EB50A2.8020401@oracle.com>
References: <1407927960.94775.YahooMailNeo@web162204.mail.bf1.yahoo.com>
	<53EB4B25.6090708@oracle.com>
	<DF545D37-5FD8-4492-85AB-2E5A3DAEBC42@yahoo.com>
	<53EB50A2.8020401@oracle.com>
Message-ID: <1407933620.83089.YahooMailNeo@web162201.mail.bf1.yahoo.com>



On Wednesday, August 13, 2014 1:49 PM, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:
On 08/13/2014 03:41 PM, vyazelenko at yahoo.com wrote:
> > My use-case: start server in @Setup(Level.Trial) and shutdown server
> > in @TearDown(Level.Trial). And of course I want to setup some data
> > for entire benchmark and remove it at the end. In other words I'm
> > looking for a way to perform per-Benchmark work irrespective of
> > @State configuration. 

> Aha. @State(Scope.Benchmark) it is.

So using @State(Scope.Benchmark) is the correct way to setup/tearDown
for entire benchmark? And what about "executed by the same thread" part?
I mean I'm thinking in terms of @BeforeClass/@AfterClass from JUnit here.
Which allow me to do setup/tearDown for all tests inside my class. Maybe
something like that should be added to JMH. So that it is possible to use
it when no @State is defined at all. Also JUnit calls @BeforeClass/@AfterClass
from the same/main thread always.

> > And this work should be done by the same
> > thread. In my case there is a session attached via ThreadLocal in
> > @Setup.

> Thinking out loud now.

> Scope.Benchmark assumes the state is shared between the threads, which
> forces users to do additional housekeeping when accessed from multiple
> threads. My concern is that tracking the @Setup/@TearDown callers would
> blur this for users: instead of blowing up early, users will experience
> weird visibility problems caused by the lack of appropriate
> synchronization.

> I think you can do something like that with:
http://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_29_StatesDAG.java

Ca you be a bit more specific about JMHSample_29_StatesDAG example? How exactly
it should help me achieve that @Setup/@TearDown are executed by the same thread?
Or do you mean I should encapsulate @State(Scope.Benchmark)?

P.S. I made a test using JMHSample_29_StatesDAG approach:
@Fork(1) 
@Warmup(iterations = 0) 
@Measurement(iterations = 1) 
@State(Scope.Thread) 
@BenchmarkMode(Mode.SingleShotTime) 
public class SetupAndTearDownThreadBenchmarkScope { 
? ? @State(Scope.Benchmark) 
? ? public static class BenchmarkStateHolder { 
? ? ? ? private void printThread(String msg) { 
? ? ? ? ? ? System.out.println("\r\n[" + Thread.currentThread().getName() + "] => " + msg); 
? ? ? ? } 
? ? ? ? @Setup(Level.Trial) 
? ? ? ? public void setUpTrial() { 
? ? ? ? ? ? printThread("setUpTrial()"); 
? ? ? ? } 
? ? ? ? @TearDown(Level.Trial) 
? ? ? ? public void tearDownTrial() { 
? ? ? ? ? ? printThread("tearDownTrial()"); 
? ? ? ? } 
? ? ? ? @Setup(Level.Iteration) 
? ? ? ? public void setUpIteration() { 
? ? ? ? ? ? printThread("setUpIteration()"); 
? ? ? ? } 
? ? ? ? @TearDown(Level.Iteration) 
? ? ? ? public void tearDownIteration() { 
? ? ? ? ? ? printThread("tearDownIteration()"); 
? ? ? ? } 
? ? ? ? @Setup(Level.Invocation) 
? ? ? ? public void setUpInvocation() { 
? ? ? ? ? ? printThread("setUpInvocation()"); 
? ? ? ? } 
? ? ? ? @TearDown(Level.Invocation) 
? ? ? ? public void tearDownInvocation() { 
? ? ? ? ? ? printThread("tearDownInvocation()"); 
? ? ? ? } 
? ? } 
? ? @Benchmark 
? ? public void myBench(BenchmarkStateHolder holder) { 
? ? ? ? System.out.println(holder); 
? ? } 
} 

Here is the output:
[org.sample.SetupAndTearDownThreadBenchmarkScope.myBench-jmh-worker-3] => setUpTrial() 
[org.sample.SetupAndTearDownThreadBenchmarkScope.myBench-jmh-worker-1] => setUpIteration() 
[org.sample.SetupAndTearDownThreadBenchmarkScope.myBench-jmh-worker-1] => setUpInvocation() 
org.sample.generated.SetupAndTearDownThreadBenchmarkScope_myBench$BenchmarkStateHolder_1_jmh at 38cc4364 
org.sample.generated.SetupAndTearDownThreadBenchmarkScope_myBench$BenchmarkStateHolder_1_jmh at 38cc4364 
org.sample.generated.SetupAndTearDownThreadBenchmarkScope_myBench$BenchmarkStateHolder_1_jmh at 38cc4364 
org.sample.generated.SetupAndTearDownThreadBenchmarkScope_myBench$BenchmarkStateHolder_1_jmh at 38cc4364 
[org.sample.SetupAndTearDownThreadBenchmarkScope.myBench-jmh-worker-4] => tearDownInvocation() 
[org.sample.SetupAndTearDownThreadBenchmarkScope.myBench-jmh-worker-3] => tearDownIteration() 
[org.sample.SetupAndTearDownThreadBenchmarkScope.myBench-jmh-worker-4] => tearDownTrial()





> -Aleksey.


Best regards,
Dmitry

From biboudis at gmail.com  Wed Aug 13 12:53:37 2014
From: biboudis at gmail.com (Aggelos Biboudis)
Date: Wed, 13 Aug 2014 15:53:37 +0300
Subject: Running Java Flight Recorder with a profiler (-prof jfr)
Message-ID: <CAHLSyCn4Z3cTO5v+eP79F1gQ_xGuSGeb5q5OFF-_2WSxZheUmQ@mail.gmail.com>

Hi all!

Recently, I used jfr to profile microbenchmarks. My intentions were two: 1)
to profile right away from the beginning and 2) per each microbenchmark.

What we could do at the moment afaik is to pass the arguments manually (to
get a single jfr file) or programmatically as arguments of the @Fork
annotation to get them separately, e.g.,:

@Fork(jvmArgsAppend =
> { "-XX:+UnlockCommercialFeatures",
>   "-XX:+FlightRecorder",
>
> "-XX:StartFlightRecording=duration=60s,filename=./profiling-data.jfr,name=FULL,settings=profile",
>
> "-XX:FlightRecorderOptions=settings=./openjdk/jdk1.8.0/jre/lib/jfr/FULL.jfc,samplethreads=true"
> })


As a proof of concept, I created a profiler (jfr) that does this
automatically for all benchmarks. Currently, many settings are fixed and
jfr's output isn't processed but it demonstrates this idea. Finally, the
profiler cannot be executed unless the initial jvm command includes
explicitly the flag "-XX:+UnlockCommercialFeatures".

When jmh terminates you can then open the generated files with Java Mission
Control.

WDYT? Would you find something like that useful?

http://www.di.uoa.gr/~biboudis/cr/jmh/Adding_support_for_a_java_flight_recorder_(-prof_jfr)_profiler_.patch

Cheers,
Aggelos

From aleksey.shipilev at oracle.com  Wed Aug 13 14:54:24 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Wed, 13 Aug 2014 18:54:24 +0400
Subject: Setup and TearDown executed by different threads
In-Reply-To: <1407933620.83089.YahooMailNeo@web162201.mail.bf1.yahoo.com>
References: <1407927960.94775.YahooMailNeo@web162204.mail.bf1.yahoo.com>
	<53EB4B25.6090708@oracle.com>
	<DF545D37-5FD8-4492-85AB-2E5A3DAEBC42@yahoo.com>
	<53EB50A2.8020401@oracle.com>
	<1407933620.83089.YahooMailNeo@web162201.mail.bf1.yahoo.com>
Message-ID: <53EB7C20.70807@oracle.com>

On 08/13/2014 04:40 PM, Dmitry Vyazelenko wrote:
> On Wednesday, August 13, 2014 1:49 PM, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:
> On 08/13/2014 03:41 PM, vyazelenko at yahoo.com wrote:
>>> My use-case: start server in @Setup(Level.Trial) and shutdown server
>>> in @TearDown(Level.Trial). And of course I want to setup some data
>>> for entire benchmark and remove it at the end. In other words I'm
>>> looking for a way to perform per-Benchmark work irrespective of
>>> @State configuration. 
> 
>> Aha. @State(Scope.Benchmark) it is.
> 
> So using @State(Scope.Benchmark) is the correct way to setup/tearDown
> for entire benchmark?

Yes.

> I mean I'm thinking in terms of @BeforeClass/@AfterClass from JUnit here.
> Which allow me to do setup/tearDown for all tests inside my class. Maybe
> something like that should be added to JMH. So that it is possible to use
> it when no @State is defined at all. Also JUnit calls @BeforeClass/@AfterClass
> from the same/main thread always.

JUnit has a simplistic view of test state to allow this. In JMH, you
have given up the control over the test state to the harness, and
therefore harness asks you about the state lifecycle with the help of
fixture methods. Everything else is presumed to be inaccessible and
dead. (Up to the point we would like to completely isolate accesses for
everything beyond explicit @State-s).

Suppose we have @BeforeClass. How would you initialize the
@State(Benchmark)? How would @BeforeClass have the access to any state
in the run? Push the reference to @State to @BeforeClass method like DAG
sample does? If so, how's that different from having the auxiliary
@State with fixture methods which reference your @State(Benchmark)? Or,
in simpler way, how's that different from defining the fixture methods
directly in @State(Benchmark)?

>> Thinking out loud now.
> 
> Ca you be a bit more specific about JMHSample_29_StatesDAG example? How exactly
> it should help me achieve that @Setup/@TearDown are executed by the same thread?
> Or do you mean I should encapsulate @State(Scope.Benchmark)?

My thought was a straight-forward ripoff of JMHSample_29_StatesDAG
example. But that does not really work since every thread will fire the
@State(Thread) fixture method...

Let me see how messy it would be to track the @Setup/@TearDown callers.

Thanks,
-Aleksey.


From tom.deneau at amd.com  Wed Aug 13 15:09:18 2014
From: tom.deneau at amd.com (Deneau, Tom)
Date: Wed, 13 Aug 2014 15:09:18 +0000
Subject: Forking and -server flag
In-Reply-To: <53E5DC31.4020904@oracle.com>
References: <BC97738F8E7C8742BABED7F06FB9DF9155489D54@SATLEXDAG01.amd.com>
	<53E5DC31.4020904@oracle.com>
Message-ID: <BC97738F8E7C8742BABED7F06FB9DF9155499163@SATLEXDAG01.amd.com>

Thanks.

On a related topic, is there any support for controlling environment variables (used by native libraries) in the forked process?

-- Tom

-----Original Message-----
From: Aleksey Shipilev [mailto:aleksey.shipilev at oracle.com] 
Sent: Saturday, August 09, 2014 3:31 AM
To: Deneau, Tom; jmh-dev at openjdk.java.net
Subject: Re: Forking and -server flag

Hi Tom,

On 08/08/2014 10:59 PM, Deneau, Tom wrote:
> My original command line specified -server, I am using forking and I 
> noticed the forked command line does not contain -server, which on my 
> configuration causes problems.

Yes, known problem with Java launcher:
 https://bugs.openjdk.java.net/browse/JDK-8025700

> Is there a way to get the forked command line to contain -server?

--jvmArgs is your friend at the command line.
.jvmArgs(...) is your friend in the API.
@Fork(jvmArgs = ...) is your friend in annotations.

-Aleksey.


From vyazelenko at yahoo.com  Wed Aug 13 15:43:01 2014
From: vyazelenko at yahoo.com (vyazelenko at yahoo.com)
Date: Wed, 13 Aug 2014 17:43:01 +0200
Subject: Setup and TearDown executed by different threads
In-Reply-To: <53EB7C20.70807@oracle.com>
References: <1407927960.94775.YahooMailNeo@web162204.mail.bf1.yahoo.com>
	<53EB4B25.6090708@oracle.com>
	<DF545D37-5FD8-4492-85AB-2E5A3DAEBC42@yahoo.com>
	<53EB50A2.8020401@oracle.com>
	<1407933620.83089.YahooMailNeo@web162201.mail.bf1.yahoo.com>
	<53EB7C20.70807@oracle.com>
Message-ID: <5A6CB52E-1F47-42A3-A120-6F0B460098C6@yahoo.com>


> On Aug 13, 2014, at 16:54, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:
> 
>> On 08/13/2014 04:40 PM, Dmitry Vyazelenko wrote:
>> On Wednesday, August 13, 2014 1:49 PM, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:
>> On 08/13/2014 03:41 PM, vyazelenko at yahoo.com wrote:
>>>> My use-case: start server in @Setup(Level.Trial) and shutdown server
>>>> in @TearDown(Level.Trial). And of course I want to setup some data
>>>> for entire benchmark and remove it at the end. In other words I'm
>>>> looking for a way to perform per-Benchmark work irrespective of
>>>> @State configuration.
>> 
>>> Aha. @State(Scope.Benchmark) it is.
>> 
>> So using @State(Scope.Benchmark) is the correct way to setup/tearDown
>> for entire benchmark?
> 
> Yes.
> 
>> I mean I'm thinking in terms of @BeforeClass/@AfterClass from JUnit here.
>> Which allow me to do setup/tearDown for all tests inside my class. Maybe
>> something like that should be added to JMH. So that it is possible to use
>> it when no @State is defined at all. Also JUnit calls @BeforeClass/@AfterClass
>> from the same/main thread always.
> 
> JUnit has a simplistic view of test state to allow this. In JMH, you
> have given up the control over the test state to the harness, and
> therefore harness asks you about the state lifecycle with the help of
> fixture methods. Everything else is presumed to be inaccessible and
> dead. (Up to the point we would like to completely isolate accesses for
> everything beyond explicit @State-s).
> 
> Suppose we have @BeforeClass. How would you initialize the
> @State(Benchmark)? How would @BeforeClass have the access to any state
> in the run? Push the reference to @State to @BeforeClass method like DAG
> sample does? If so, how's that different from having the auxiliary
> @State with fixture methods which reference your @State(Benchmark)? Or,
> in simpler way, how's that different from defining the fixture methods
> directly in @State(Benchmark)?
> 
>>> Thinking out loud now.
>> 
>> Ca you be a bit more specific about JMHSample_29_StatesDAG example? How exactly
>> it should help me achieve that @Setup/@TearDown are executed by the same thread?
>> Or do you mean I should encapsulate @State(Scope.Benchmark)?
> 
> My thought was a straight-forward ripoff of JMHSample_29_StatesDAG
> example. But that does not really work since every thread will fire the
> @State(Thread) fixture method...
> 
> Let me see how messy it would be to track the @Setup/@TearDown callers.
Ok, my comparison of JUnit annotation and JMH was simplistic to say the least. But one thing out if JUnit seems to be valuable - notion of the main thread.

What if JMH would simply have a main thread that would be used to execute @Setup/@TearDown calls on @State(Scope.Benchmark)?
Then you won't need to track callers, because there will be only one caller.
Having such thread will basically make @Setup(Level.Iteration) like @BeforeClass in Junit and @Setup(Level.Iteration) like @Before.

One thing I'm not sure about is @Setup(Level.Invocation). Logically for Scope.Benchmark this method should also be called by main thread. But that would mean that threads would ping pong, i.e. setup in one thread and immediate execution in another.

This is my view on the issue. But I'm definitely missing all the JMH-internal context to know better.
> 
> Thanks,
> -Aleksey.
> 
Aleksey, thanks for looking into it. :)

Regards,
Dmitry

From aleksey.shipilev at oracle.com  Wed Aug 13 19:00:02 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Wed, 13 Aug 2014 19:00:02 +0000
Subject: hg: code-tools/jmh: Mention the JVM executable for jvm(), -jvm.
Message-ID: <201408131900.s7DJ028p008458@aojmv0008>

Changeset: bdfb6ec5251c
Author:    shade
Date:      2014-08-13 22:59 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/bdfb6ec5251c

Mention the JVM executable for jvm(), -jvm.

! jmh-core/src/main/java/org/openjdk/jmh/runner/options/CommandLineOptions.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/options/Options.java


From aleksey.shipilev at oracle.com  Wed Aug 13 19:10:25 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Wed, 13 Aug 2014 23:10:25 +0400
Subject: Forking and -server flag
In-Reply-To: <BC97738F8E7C8742BABED7F06FB9DF9155499163@SATLEXDAG01.amd.com>
References: <BC97738F8E7C8742BABED7F06FB9DF9155489D54@SATLEXDAG01.amd.com>
	<53E5DC31.4020904@oracle.com>
	<BC97738F8E7C8742BABED7F06FB9DF9155499163@SATLEXDAG01.amd.com>
Message-ID: <53EBB821.9020204@oracle.com>

On 08/13/2014 07:09 PM, Deneau, Tom wrote:
> On a related topic, is there any support for controlling environment
> variables (used by native libraries) in the forked process?

Nope, sorry. Export your environment variables, and then invoke JMH.
That would require shell scripting to sequence the envvars.

-Aleksey.


From aleksey.shipilev at oracle.com  Wed Aug 13 19:20:37 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Wed, 13 Aug 2014 19:20:37 +0000
Subject: hg: code-tools/jmh: runners: produce the result file if either
	-rf/result() or -rrf/resultFormat() is supplied.
Message-ID: <201408131920.s7DJKbTc011885@aojmv0008>

Changeset: 0234208765da
Author:    shade
Date:      2014-08-13 23:20 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/0234208765da

runners: produce the result file if either -rf/result() or -rrf/resultFormat() is supplied.

- jmh-core/src/main/java/org/openjdk/jmh/results/format/NoneResultFormat.java
! jmh-core/src/main/java/org/openjdk/jmh/results/format/ResultFormatFactory.java
! jmh-core/src/main/java/org/openjdk/jmh/results/format/ResultFormatType.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/Defaults.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java


From aleksey.shipilev at oracle.com  Wed Aug 13 19:37:56 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Wed, 13 Aug 2014 19:37:56 +0000
Subject: hg: code-tools/jmh: runners: print user-friendly error messages when
	forked VM is not executable or something else goes wrong.
Message-ID: <201408131937.s7DJbuPl014951@aojmv0008>

Changeset: 32d412b1693a
Author:    shade
Date:      2014-08-13 23:37 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/32d412b1693a

runners: print user-friendly error messages when forked VM is not executable or something else goes wrong.

! jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java


From ecki at zusammenkunft.net  Thu Aug 14 00:19:06 2014
From: ecki at zusammenkunft.net (Bernd Eckenfels)
Date: Thu, 14 Aug 2014 02:19:06 +0200
Subject: AW: AW: Can't fork VM if option value contains space
In-Reply-To: <53EB41E7.2040201@oracle.com>
References: <43C4DC37-9DAC-4341-917B-BB1E7D333827@yahoo.com>
	<53E95028.4060909@oracle.com>
	<20140812014554.0000441a.ecki@zusammenkunft.net>
	<9840718B-DAA4-4561-A5D7-F916E593EF0D@yahoo.com>
	<53e9f1c4.e4578c0a.42fc.ffffeba6@mx.google.com>
	<1407843224.38893.YahooMailNeo@web162205.mail.bf1.yahoo.com>
	<1407845358.64545.YahooMailNeo@web162203.mail.bf1.yahoo.com>
	<53EB41E7.2040201@oracle.com>
Message-ID: <53ec0082.6c2a8c0a.58bb.ffff90b2@mx.google.com>

Hello,

I think it is a bit hard to solve because the Java Launcher had to do its own parsing on Windows but not Unix (and it seems to do more than needed on OSX?)

(Don't even look at the apache commons exec, it does use /bin/sh -c and breaks even the simple Unix case*)

Maybe the requirement for passing hard to parse system properties can be reduced (to passing only VM and system lib controlling switches but no dynamic benchmark parameters) by offering a better interface to environment variables? having said that, what is the OP's use case?

Gruss
bernd

* https://issues.apache.org/jira/browse/EXEC-82?jql=project%20%3D%20%22EXEC%22%20AND%20resolution%20%3D%20Unresolved%20AND%20status%20%3D%20Open%20AND%20text%20~%20%22quote%22%20ORDER%20BY%20priority%20DESC%2C%20key%20DESC
-- 
http://bernd.eckenfels.net

----- Urspr?ngliche Nachricht -----
Von: "Aleksey Shipilev" <aleksey.shipilev at oracle.com>
Gesendet: ?13.?08.?2014 12:46
An: "Dmitry Vyazelenko" <vyazelenko at yahoo.com>; "Bernd Eckenfels" <ecki at zusammenkunft.net>; "jmh-dev at openjdk.java.net" <jmh-dev at openjdk.java.net>
Betreff: Re: AW: Can't fork VM if option value contains space

Ah, got it. This is the impedance in -jvmArgs handling.

In all other places, including the Java API and annotations, we pass
String[], where each individual String is the argument (may be with
spaces). This fits nicely into Runtime.exec(String cmd, String[] args)
API, and it will handle the spaces correctly.

However, for -jvmArgs, we only get String, and then we blindly split it
by space to produce String[]:
 http://hg.openjdk.java.net/code-tools/jmh/file/60e8aae7ac7d/jmh-core/src/main/java/org/openjdk/jmh/runner/options/CommandLineOptions.java#l405

Therefore:

On 08/12/2014 04:09 PM, Dmitry Vyazelenko wrote:
> On Tuesday, August 12, 2014 1:34 PM, Dmitry Vyazelenko <vyazelenko at yahoo.com> wrote:
> Well it is even more strange on Windows:
> - On command line:
>   FAILS: java -jar target\benchmarks.jar -jvmArgs -Dmy.prop="A B C"
>   Does not start benchmark execution.

...forked command line has individual argument "B", which fails, because
VM thinks "B" is the Java class name.

>   WORKS: java -jar target\benchmarks.jar -jvmArgs -Dmy.prop="\"A B C\""
>   Runs and produces this result:
>          Iteration   1: my.prop ==> A B C

...forked command line also has "B" as the individual argument, but VM
launcher seem to reconstruct the "-DmyProp=\"A", "B", "C\"" into
"-DmyProp=A B C".

> - Using Java API:
>   FAILS: new OptionsBuilder() 
>                 .jvmArgs("-Dmy.prop=\"A B C\"").build();
>   Does not fork VM.

...one level of quotes seem to be digested by Runtime.exec.

>   WORKS: new OptionsBuilder() 
>                 .jvmArgs("-Dmy.prop=\"\"A B C\"\"").build();
>   Runs but produces wrong result:
>          Iteration   1: my.prop ==> "A B C"

...putting another level of quotes helps.

>   RUNS: new OptionsBuilder() 
>                 .jvmArgs("-Dmy.prop=\\\"A B C\\\"").build();  
>   Runs but produces wrong result: 
>          Iteration 1: my.prop ==> "A B C" 

...the quotes are properly escaped from Runtime.exec.

>   WORKS (no quoting): new OptionsBuilder() 
>                 .jvmArgs("-Dmy.prop=A B C").build();
>   Runs and produces correct result: 
>          Iteration 1: my.prop ==> A B C"

...and this is a correct and most straight-forward scenario.

All this feels rather messy cross-cutting issue across multiple shells,
VMs, etc. At this point, I am a bit puzzled if we can do anything about it.

Thanks,
-Aleksey.



From vyazelenko at yahoo.com  Thu Aug 14 07:29:22 2014
From: vyazelenko at yahoo.com (Dmitry Vyazelenko)
Date: Thu, 14 Aug 2014 00:29:22 -0700
Subject: Deadlock on attempt to shutdown forked JVM/worker threads
Message-ID: <1408001362.12517.YahooMailNeo@web162202.mail.bf1.yahoo.com>

Hi,

I've got a deadlock from a benchmark that uses Spring framework (i.e. it creates Spring ApplicationContext
in @Setup(Level.Trial)/Scope.Benchmark). The benchmark itself is started with 4 threads. And finally
the benchmark method was failing with exception (internal issue) and "-foe true" was specified.

At the point when exception is thrown there is attempt to shutdown worker threads and also Spring
tries to shutdown itself. This results in the deadlock:
Found one Java-level deadlock:
=============================
"Thread-0":
? waiting to lock monitor 0x000000001131f088 (object 0x00000007059573f0, a org.openjdk.jmh.runner.link.BinaryLinkClient$ForwardingPrintStream),
? which is held by "Thread-5"
"Thread-5":
? waiting to lock monitor 0x0000000011321398 (object 0x00000007058d89a0, a java.lang.Object),
? which is held by "Thread-0"

Java stack information for the threads listed above:
===================================================
"Thread-0":
?at java.io.PrintStream.flush(PrintStream.java:337)
?- waiting to lock <0x00000007059573f0> (a org.openjdk.jmh.runner.link.BinaryLinkClient$ForwardingPrintStream)
?at org.openjdk.jmh.util.FileUtils.safelyClose(FileUtils.java:202)
?at org.openjdk.jmh.runner.link.BinaryLinkClient.close(BinaryLinkClient.java:77)
?- locked <0x00000007058d89a0> (a java.lang.Object)
?at org.openjdk.jmh.runner.ForkedMain$1.run(ForkedMain.java:103)
"Thread-5":
?at org.openjdk.jmh.runner.link.BinaryLinkClient.pushFrame(BinaryLinkClient.java:69)
?- waiting to lock <0x00000007058d89a0> (a java.lang.Object)
?at org.openjdk.jmh.runner.link.BinaryLinkClient.access$000(BinaryLinkClient.java:48)
?at org.openjdk.jmh.runner.link.BinaryLinkClient$ForwardingPrintStream$1.write(BinaryLinkClient.java:156)
?at java.io.PrintStream.write(PrintStream.java:480)
?- locked <0x00000007059573f0> (a org.openjdk.jmh.runner.link.BinaryLinkClient$ForwardingPrintStream)
?at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
?at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
?at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
?at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
?- locked <0x0000000705a5bc08> (a java.io.OutputStreamWriter)
?at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
?at org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:59)
?at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:324)
?at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
?at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
?- locked <0x0000000705a5b3f0> (a org.apache.log4j.ConsoleAppender)
?at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
?at org.apache.log4j.Category.callAppenders(Category.java:206)
?- locked <0x0000000705a4c760> (a org.apache.log4j.spi.RootLogger)
?at com.xxx.xxx.logging.DispatcherLogHandler.append(DispatcherLogHandler.java:81)
?at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
?- locked <0x0000000705a5bd80> (a com.xxx.xxx.logging.DispatcherLogHandler)
?at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
?at org.apache.log4j.Category.callAppenders(Category.java:206)
?- locked <0x0000000705a4c6b0> (a org.apache.log4j.Logger)
?at org.apache.log4j.Category.forcedLog(Category.java:391)
?at org.apache.log4j.Category.log(Category.java:856)
?at org.slf4j.impl.Log4jLoggerAdapter.log(Log4jLoggerAdapter.java:601)
?at org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:159)
?at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:873)
?at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:809)

I'm also attaching full stacktrace (the internal package names were changed to protect the innocent).

Unfortunately I don't have a standalone reproducible benchmark that I can share.

Best regards,
Dmitry
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: ForkedMain-full-stack-trace.txt
URL: <http://mail.openjdk.java.net/pipermail/jmh-dev/attachments/20140814/53d69fda/ForkedMain-full-stack-trace-0001.txt>

From aleksey.shipilev at oracle.com  Thu Aug 14 08:29:28 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Thu, 14 Aug 2014 08:29:28 +0000
Subject: hg: code-tools/jmh: runners: fix the deadlock in BinaryLinkClient.
Message-ID: <201408140829.s7E8TSUc015636@aojmv0008>

Changeset: f90aef7f1d2c
Author:    shade
Date:      2014-08-14 12:28 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/f90aef7f1d2c

runners: fix the deadlock in BinaryLinkClient.

! jmh-core/src/main/java/org/openjdk/jmh/runner/link/BinaryLinkClient.java


From aleksey.shipilev at oracle.com  Thu Aug 14 08:30:00 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Thu, 14 Aug 2014 12:30:00 +0400
Subject: Deadlock on attempt to shutdown forked JVM/worker threads
In-Reply-To: <1408001362.12517.YahooMailNeo@web162202.mail.bf1.yahoo.com>
References: <1408001362.12517.YahooMailNeo@web162202.mail.bf1.yahoo.com>
Message-ID: <53EC7388.9000703@oracle.com>

On 08/14/2014 11:29 AM, Dmitry Vyazelenko wrote:
> Found one Java-level deadlock:
> =============================
> "Thread-0":
>   waiting to lock monitor 0x000000001131f088 (object 0x00000007059573f0, a org.openjdk.jmh.runner.link.BinaryLinkClient$ForwardingPrintStream),
>   which is held by "Thread-5"
> "Thread-5":
>   waiting to lock monitor 0x0000000011321398 (object 0x00000007058d89a0, a java.lang.Object),
>   which is held by "Thread-0"
> 
> Java stack information for the threads listed above:
> ===================================================
> "Thread-0":
>  at java.io.PrintStream.flush(PrintStream.java:337)
>  - waiting to lock <0x00000007059573f0> (a org.openjdk.jmh.runner.link.BinaryLinkClient$ForwardingPrintStream)
>  at org.openjdk.jmh.util.FileUtils.safelyClose(FileUtils.java:202)
>  at org.openjdk.jmh.runner.link.BinaryLinkClient.close(BinaryLinkClient.java:77)
>  - locked <0x00000007058d89a0> (a java.lang.Object)
>  at org.openjdk.jmh.runner.ForkedMain$1.run(ForkedMain.java:103)
> "Thread-5":
>  at org.openjdk.jmh.runner.link.BinaryLinkClient.pushFrame(BinaryLinkClient.java:69)
>  - waiting to lock <0x00000007058d89a0> (a java.lang.Object)
>  at org.openjdk.jmh.runner.link.BinaryLinkClient.access$000(BinaryLinkClient.java:48)
>  at org.openjdk.jmh.runner.link.BinaryLinkClient$ForwardingPrintStream$1.write(BinaryLinkClient.java:156)
>  at java.io.PrintStream.write(PrintStream.java:480)
>  - locked <0x00000007059573f0> (a org.openjdk.jmh.runner.link.BinaryLinkClient$ForwardingPrintStream)
>  at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
>  at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)

Thanks, this is enough to dissect the issue. Fixed:
  http://hg.openjdk.java.net/code-tools/jmh/rev/f90aef7f1d2c

Can you run the patched version on your workload again?

-Aleksey.


From dalibor.topic at oracle.com  Thu Aug 14 09:36:10 2014
From: dalibor.topic at oracle.com (dalibor topic)
Date: Thu, 14 Aug 2014 11:36:10 +0200
Subject: Running Java Flight Recorder with a profiler (-prof jfr)
In-Reply-To: <CAHLSyCn4Z3cTO5v+eP79F1gQ_xGuSGeb5q5OFF-_2WSxZheUmQ@mail.gmail.com>
References: <CAHLSyCn4Z3cTO5v+eP79F1gQ_xGuSGeb5q5OFF-_2WSxZheUmQ@mail.gmail.com>
Message-ID: <53EC830A.7020100@oracle.com>

Hi Aggelos,

I don't think that it makes a lot of sense for jmh in OpenJDK to accept 
patches that integrate it with commercial features of the Oracle JDK.

cheers,
dalibor topic

-- 
<http://www.oracle.com> Dalibor Topic | Principal Product Manager
Phone: +494089091214 <tel:+494089091214> | Mobile: +491737185961
<tel:+491737185961>

ORACLE Deutschland B.V. & Co. KG | K?hneh?fe 5 | 22761 Hamburg

ORACLE Deutschland B.V. & Co. KG
Hauptverwaltung: Riesstr. 25, D-80992 M?nchen
Registergericht: Amtsgericht M?nchen, HRA 95603
Gesch?ftsf?hrer: J?rgen Kunz

Komplement?rin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Gesch?ftsf?hrer: Alexander van der Ven, Astrid Kepper, Val Maher

<http://www.oracle.com/commitment> Oracle is committed to developing
practices and products that help protect the environment

From vyazelenko at yahoo.com  Thu Aug 14 12:14:41 2014
From: vyazelenko at yahoo.com (Dmitry Vyazelenko)
Date: Thu, 14 Aug 2014 05:14:41 -0700
Subject: Deadlock on attempt to shutdown forked JVM/worker threads
In-Reply-To: <53EC7388.9000703@oracle.com>
References: <1408001362.12517.YahooMailNeo@web162202.mail.bf1.yahoo.com>
	<53EC7388.9000703@oracle.com>
Message-ID: <1408018481.3581.YahooMailNeo@web162201.mail.bf1.yahoo.com>

This is awesome! I came from the meeting and the problem is fixed!

Thanks, Aleksey!
?
Best regards,
Dmitry



On Thursday, August 14, 2014 10:30 AM, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:
On 08/14/2014 11:29 AM, Dmitry Vyazelenko wrote:



> Found one Java-level deadlock:
> =============================
> "Thread-0":
>?  waiting to lock monitor 0x000000001131f088 (object 0x00000007059573f0, a org.openjdk.jmh.runner.link.BinaryLinkClient$ForwardingPrintStream),
>?  which is held by "Thread-5"
> "Thread-5":
>?  waiting to lock monitor 0x0000000011321398 (object 0x00000007058d89a0, a java.lang.Object),
>?  which is held by "Thread-0"
> 
> Java stack information for the threads listed above:
> ===================================================
> "Thread-0":
>? at java.io.PrintStream.flush(PrintStream.java:337)
>? - waiting to lock <0x00000007059573f0> (a org.openjdk.jmh.runner.link.BinaryLinkClient$ForwardingPrintStream)
>? at org.openjdk.jmh.util.FileUtils.safelyClose(FileUtils.java:202)
>? at org.openjdk.jmh.runner.link.BinaryLinkClient.close(BinaryLinkClient.java:77)
>? - locked <0x00000007058d89a0> (a java.lang.Object)
>? at org.openjdk.jmh.runner.ForkedMain$1.run(ForkedMain.java:103)
> "Thread-5":
>? at org.openjdk.jmh.runner.link.BinaryLinkClient.pushFrame(BinaryLinkClient.java:69)
>? - waiting to lock <0x00000007058d89a0> (a java.lang.Object)
>? at org.openjdk.jmh.runner.link.BinaryLinkClient.access$000(BinaryLinkClient.java:48)
>? at org.openjdk.jmh.runner.link.BinaryLinkClient$ForwardingPrintStream$1.write(BinaryLinkClient.java:156)
>? at java.io.PrintStream.write(PrintStream.java:480)
>? - locked <0x00000007059573f0> (a org.openjdk.jmh.runner.link.BinaryLinkClient$ForwardingPrintStream)
>? at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
>? at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)

Thanks, this is enough to dissect the issue. Fixed:
? http://hg.openjdk.java.net/code-tools/jmh/rev/f90aef7f1d2c

Can you run the patched version on your workload again?

-Aleksey.


From biboudis at gmail.com  Thu Aug 14 15:02:31 2014
From: biboudis at gmail.com (Aggelos Biboudis)
Date: Thu, 14 Aug 2014 18:02:31 +0300
Subject: Running Java Flight Recorder with a profiler (-prof jfr)
In-Reply-To: <53EC830A.7020100@oracle.com>
References: <CAHLSyCn4Z3cTO5v+eP79F1gQ_xGuSGeb5q5OFF-_2WSxZheUmQ@mail.gmail.com>
	<53EC830A.7020100@oracle.com>
Message-ID: <CAHLSyCk90_nWjfBpnsBiKowQ_t23dCR9wbTps+X1AANqbgx6YA@mail.gmail.com>

Good point! Thank you for your answer.

External profilers' discovery would be a better thing to support, I
suppose. I may work on it a little and propose an alternative.

Cheers!
Aggelos Biboudis.



On Thu, Aug 14, 2014 at 12:36 PM, dalibor topic <dalibor.topic at oracle.com>
wrote:

> Hi Aggelos,
>
> I don't think that it makes a lot of sense for jmh in OpenJDK to accept
> patches that integrate it with commercial features of the Oracle JDK.
>
> cheers,
> dalibor topic
>
> --
> <http://www.oracle.com> Dalibor Topic | Principal Product Manager
> Phone: +494089091214 <tel:+494089091214> | Mobile: +491737185961
> <tel:+491737185961>
>
> ORACLE Deutschland B.V. & Co. KG | K?hneh?fe 5 | 22761 Hamburg
>
> ORACLE Deutschland B.V. & Co. KG
> Hauptverwaltung: Riesstr. 25, D-80992 M?nchen
> Registergericht: Amtsgericht M?nchen, HRA 95603
> Gesch?ftsf?hrer: J?rgen Kunz
>
> Komplement?rin: ORACLE Deutschland Verwaltung B.V.
> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
> Gesch?ftsf?hrer: Alexander van der Ven, Astrid Kepper, Val Maher
>
> <http://www.oracle.com/commitment> Oracle is committed to developing
> practices and products that help protect the environment
>

From biboudis at gmail.com  Fri Aug 15 11:23:18 2014
From: biboudis at gmail.com (Aggelos Biboudis)
Date: Fri, 15 Aug 2014 14:23:18 +0300
Subject: Running Java Flight Recorder with a profiler (-prof jfr)
In-Reply-To: <CAHLSyCk90_nWjfBpnsBiKowQ_t23dCR9wbTps+X1AANqbgx6YA@mail.gmail.com>
References: <CAHLSyCn4Z3cTO5v+eP79F1gQ_xGuSGeb5q5OFF-_2WSxZheUmQ@mail.gmail.com>
	<53EC830A.7020100@oracle.com>
	<CAHLSyCk90_nWjfBpnsBiKowQ_t23dCR9wbTps+X1AANqbgx6YA@mail.gmail.com>
Message-ID: <CAHLSyC=A9UMw5va5rP_hdm2Thv0Xvv3mLLsvK4kH69qt3qobQw@mail.gmail.com>

Hi again!

This contains my alternative proposal:
https://github.com/biboudis/jmh-profilers

It includes the patch that enables SPI
<https://github.com/biboudis/jmh-profilers/blob/master/cr/Lightweight_external_profiler_discovery_through_SPI_.patch>
and
the profiler that was migrated there. Maybe this would open JMH up to
community-maintained *external* profilers for Linux/Win/Mac.

WDYT?

Cheers!
Aggelos.



On Thu, Aug 14, 2014 at 6:02 PM, Aggelos Biboudis <biboudis at gmail.com>
wrote:

> Good point! Thank you for your answer.
>
> External profilers' discovery would be a better thing to support, I
> suppose. I may work on it a little and propose an alternative.
>
> Cheers!
> Aggelos Biboudis.
>
>
>
> On Thu, Aug 14, 2014 at 12:36 PM, dalibor topic <dalibor.topic at oracle.com>
> wrote:
>
>> Hi Aggelos,
>>
>> I don't think that it makes a lot of sense for jmh in OpenJDK to accept
>> patches that integrate it with commercial features of the Oracle JDK.
>>
>> cheers,
>> dalibor topic
>>
>> --
>> <http://www.oracle.com> Dalibor Topic | Principal Product Manager
>> Phone: +494089091214 <tel:+494089091214> | Mobile: +491737185961
>> <tel:+491737185961>
>>
>> ORACLE Deutschland B.V. & Co. KG | K?hneh?fe 5 | 22761 Hamburg
>>
>> ORACLE Deutschland B.V. & Co. KG
>> Hauptverwaltung: Riesstr. 25, D-80992 M?nchen
>> Registergericht: Amtsgericht M?nchen, HRA 95603
>> Gesch?ftsf?hrer: J?rgen Kunz
>>
>> Komplement?rin: ORACLE Deutschland Verwaltung B.V.
>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
>> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
>> Gesch?ftsf?hrer: Alexander van der Ven, Astrid Kepper, Val Maher
>>
>> <http://www.oracle.com/commitment> Oracle is committed to developing
>> practices and products that help protect the environment
>>
>
>

From aleksey.shipilev at oracle.com  Fri Aug 15 11:58:12 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Fri, 15 Aug 2014 15:58:12 +0400
Subject: Running Java Flight Recorder with a profiler (-prof jfr)
In-Reply-To: <CAHLSyC=A9UMw5va5rP_hdm2Thv0Xvv3mLLsvK4kH69qt3qobQw@mail.gmail.com>
References: <CAHLSyCn4Z3cTO5v+eP79F1gQ_xGuSGeb5q5OFF-_2WSxZheUmQ@mail.gmail.com>	<53EC830A.7020100@oracle.com>	<CAHLSyCk90_nWjfBpnsBiKowQ_t23dCR9wbTps+X1AANqbgx6YA@mail.gmail.com>
	<CAHLSyC=A9UMw5va5rP_hdm2Thv0Xvv3mLLsvK4kH69qt3qobQw@mail.gmail.com>
Message-ID: <53EDF5D4.7000004@oracle.com>

Hi Aggelos,

On 08/15/2014 03:23 PM, Aggelos Biboudis wrote:
> It includes the patch that enables SPI
> <https://github.com/biboudis/jmh-profilers/blob/master/cr/Lightweight_external_profiler_discovery_through_SPI_.patch>
> and
> the profiler that was migrated there. Maybe this would open JMH up to
> community-maintained *external* profilers for Linux/Win/Mac.

I think the external API for profilers would be awesome to have. SPI
seems to be the good option to do this. Please extend your patch to
InternalProfiler.class as well, or just Profiler.class.

Also, as per OpenJDK TOU, we accept the patches as the attachments to
this mailing list.

Thanks,
-Aleksey.



From biboudis at gmail.com  Fri Aug 15 12:22:49 2014
From: biboudis at gmail.com (Aggelos Biboudis)
Date: Fri, 15 Aug 2014 15:22:49 +0300
Subject: Running Java Flight Recorder with a profiler (-prof jfr)
In-Reply-To: <53EDF5D4.7000004@oracle.com>
References: <CAHLSyCn4Z3cTO5v+eP79F1gQ_xGuSGeb5q5OFF-_2WSxZheUmQ@mail.gmail.com>
	<53EC830A.7020100@oracle.com>
	<CAHLSyCk90_nWjfBpnsBiKowQ_t23dCR9wbTps+X1AANqbgx6YA@mail.gmail.com>
	<CAHLSyC=A9UMw5va5rP_hdm2Thv0Xvv3mLLsvK4kH69qt3qobQw@mail.gmail.com>
	<53EDF5D4.7000004@oracle.com>
Message-ID: <CAHLSyCmF8GVS1PG7_+NTz8qbu+c6R4baaRUZFB49pGuT0uQtfQ@mail.gmail.com>

Here it is.

Thanks!

Aggelos.


On Fri, Aug 15, 2014 at 2:58 PM, Aleksey Shipilev <
aleksey.shipilev at oracle.com> wrote:

> Hi Aggelos,
>
> On 08/15/2014 03:23 PM, Aggelos Biboudis wrote:
> > It includes the patch that enables SPI
> > <
> https://github.com/biboudis/jmh-profilers/blob/master/cr/Lightweight_external_profiler_discovery_through_SPI_.patch
> >
> > and
> > the profiler that was migrated there. Maybe this would open JMH up to
> > community-maintained *external* profilers for Linux/Win/Mac.
>
> I think the external API for profilers would be awesome to have. SPI
> seems to be the good option to do this. Please extend your patch to
> InternalProfiler.class as well, or just Profiler.class.
>
> Also, as per OpenJDK TOU, we accept the patches as the attachments to
> this mailing list.
>
> Thanks,
> -Aleksey.
>
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Lightweight_profiler_discovery_through_SPI.patch
Type: text/x-patch
Size: 3388 bytes
Desc: not available
URL: <http://mail.openjdk.java.net/pipermail/jmh-dev/attachments/20140815/e70fdb66/Lightweight_profiler_discovery_through_SPI-0001.patch>

From aleksey.shipilev at oracle.com  Fri Aug 15 13:19:43 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Fri, 15 Aug 2014 13:19:43 +0000
Subject: hg: code-tools/jmh: samples: rephrase two starting samples.
Message-ID: <201408151319.s7FDJh6d001314@aojmv0008>

Changeset: 606df305ada6
Author:    shade
Date:      2014-08-15 13:58 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/606df305ada6

samples: rephrase two starting samples.

! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_01_HelloWorld.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_02_BenchmarkModes.java


From aleksey.shipilev at oracle.com  Fri Aug 15 13:21:05 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Fri, 15 Aug 2014 13:21:05 +0000
Subject: hg: code-tools/jmh: Re-touch JMH Javadocs.
Message-ID: <201408151321.s7FDL5BM001536@aojmv0008>

Changeset: 858587557d96
Author:    shade
Date:      2014-08-15 17:19 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/858587557d96

Re-touch JMH Javadocs.

! jmh-core/src/main/java/org/openjdk/jmh/annotations/AuxCounters.java
! jmh-core/src/main/java/org/openjdk/jmh/annotations/Benchmark.java
! jmh-core/src/main/java/org/openjdk/jmh/annotations/CompilerControl.java
! jmh-core/src/main/java/org/openjdk/jmh/annotations/Fork.java
! jmh-core/src/main/java/org/openjdk/jmh/annotations/Group.java
! jmh-core/src/main/java/org/openjdk/jmh/annotations/GroupThreads.java
! jmh-core/src/main/java/org/openjdk/jmh/annotations/Param.java
! jmh-core/src/main/java/org/openjdk/jmh/annotations/Setup.java
! jmh-core/src/main/java/org/openjdk/jmh/annotations/State.java
! jmh-core/src/main/java/org/openjdk/jmh/annotations/TearDown.java


From aleksey.shipilev at oracle.com  Fri Aug 15 13:37:17 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Fri, 15 Aug 2014 13:37:17 +0000
Subject: hg: code-tools/jmh: runners: profilers are pluggable through SPI.
Message-ID: <201408151337.s7FDbHeG005772@aojmv0008>

Changeset: a0b4a12cdc55
Author:    shade
Date:      2014-08-15 17:36 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/a0b4a12cdc55

runners: profilers are pluggable through SPI.
Contributed-by: Aggelos Biboudis <biboudis at gmail.com>

! jmh-core/src/main/java/org/openjdk/jmh/profile/ProfilerFactory.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/options/CommandLineOptions.java


From aleksey.shipilev at oracle.com  Fri Aug 15 13:38:25 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Fri, 15 Aug 2014 17:38:25 +0400
Subject: Running Java Flight Recorder with a profiler (-prof jfr)
In-Reply-To: <CAHLSyCmF8GVS1PG7_+NTz8qbu+c6R4baaRUZFB49pGuT0uQtfQ@mail.gmail.com>
References: <CAHLSyCn4Z3cTO5v+eP79F1gQ_xGuSGeb5q5OFF-_2WSxZheUmQ@mail.gmail.com>	<53EC830A.7020100@oracle.com>	<CAHLSyCk90_nWjfBpnsBiKowQ_t23dCR9wbTps+X1AANqbgx6YA@mail.gmail.com>	<CAHLSyC=A9UMw5va5rP_hdm2Thv0Xvv3mLLsvK4kH69qt3qobQw@mail.gmail.com>	<53EDF5D4.7000004@oracle.com>
	<CAHLSyCmF8GVS1PG7_+NTz8qbu+c6R4baaRUZFB49pGuT0uQtfQ@mail.gmail.com>
Message-ID: <53EE0D51.7080105@oracle.com>

On 08/15/2014 04:22 PM, Aggelos Biboudis wrote:
> Here it is.

Thank you, SPI support is committed:
 http://hg.openjdk.java.net/code-tools/jmh/rev/a0b4a12cdc55

-Aleksey.


From aleksey.shipilev at oracle.com  Fri Aug 15 13:59:00 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Fri, 15 Aug 2014 17:59:00 +0400
Subject: Setup and TearDown executed by different threads
In-Reply-To: <53EB7C20.70807@oracle.com>
References: <1407927960.94775.YahooMailNeo@web162204.mail.bf1.yahoo.com>	<53EB4B25.6090708@oracle.com>	<DF545D37-5FD8-4492-85AB-2E5A3DAEBC42@yahoo.com>	<53EB50A2.8020401@oracle.com>	<1407933620.83089.YahooMailNeo@web162201.mail.bf1.yahoo.com>
	<53EB7C20.70807@oracle.com>
Message-ID: <53EE1224.10507@oracle.com>

On 08/13/2014 06:54 PM, Aleksey Shipilev wrote:
> Let me see how messy it would be to track the @Setup/@TearDown callers.

Okay, it seems messier than I expected at the first glance. There are
multiple things to consider when tracking the @State owner:
 - do we share the owner across multiple Levels?
 - what do we do if the owner exits abruptly?
 - how do we handle the absence of paired @Setup/@TearDowns?
 - ...?

Not that isn't not doable, but the change seems too intrusive to do in a
Release Candidate. It also intersects with Chris Vest's example of
benchmark being stuck if one of the threads in asymmetric benchmarks
throws the exception. Let us consider this for post 1.0 revisions.

I had updated the @Setup/@TearDown Javadocs to (un)specify which thread
is going to call the fixture methods though.

Thanks,
-Aleksey.


From aleksey.shipilev at oracle.com  Fri Aug 15 15:43:33 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Fri, 15 Aug 2014 19:43:33 +0400
Subject: JMH 0.9.7
Message-ID: <53EE2AA5.7000201@oracle.com>

Hi,

JMH 0.9.7 is released and available in Maven Central (thanks to
Evgeny, as usual). We have been having a very active discussion and many
sensible small tweaks for 0.9.6, and therefore JMH 0.9.7 is our 3nd
release candidate for 1.0, please test!

Highlights of JMH 0.9.7:

 * JMH may have experienced the deadlock on forked VM shutdown when
there were extensive users of stdout/stderr in the forked VM.

 * "-prof perfasm" does not look for "java" in the perf datafile now,
which enables it to run on the systems where the final executable was
renamed from "java" to something else.

 * JMH will now print the user-friendly error messages when forked VM is
unable to start.

 * The benchmark name prefix shortener in the final result table was
having a bug could generate wrong prefixes, fixed now.

 * Supplying either -rf/resultFormat() or -rff/result() will dump the
machine-readable result into the file. Before, users were required to
supply -rf in any case

 * Profilers are now pluggable through SPI (thanks Aggelos!)

 * Documentation improvements in Javadocs and Samples.

Enjoy!

Thanks,


From aleksey.shipilev at oracle.com  Fri Aug 15 16:04:26 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Fri, 15 Aug 2014 16:04:26 +0000
Subject: hg: code-tools/jmh: 3 new changesets
Message-ID: <201408151604.s7FG4Q1x000120@aojmv0008>

Changeset: 6e702a334f2e
Author:    shade
Date:      2014-08-15 18:38 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/6e702a334f2e

JMH v0.9.7.

! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml
! jmh-archetypes/pom.xml
! jmh-core-benchmarks/pom.xml
! jmh-core-ct/pom.xml
! jmh-core-it/pom.xml
! jmh-core/pom.xml
! jmh-generator-annprocess/pom.xml
! jmh-generator-asm/pom.xml
! jmh-generator-bytecode/pom.xml
! jmh-generator-reflection/pom.xml
! jmh-samples/pom.xml
! pom.xml

Changeset: a25bd72b92f5
Author:    shade
Date:      2014-08-15 18:38 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/a25bd72b92f5

Added tag 0.9.7 for changeset 6e702a334f2e

! .hgtags

Changeset: d5526434fd4a
Author:    shade
Date:      2014-08-15 18:38 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/d5526434fd4a

Continue in 1.0-SNAPSHOT.

! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml
! jmh-archetypes/pom.xml
! jmh-core-benchmarks/pom.xml
! jmh-core-ct/pom.xml
! jmh-core-it/pom.xml
! jmh-core/pom.xml
! jmh-generator-annprocess/pom.xml
! jmh-generator-asm/pom.xml
! jmh-generator-bytecode/pom.xml
! jmh-generator-reflection/pom.xml
! jmh-samples/pom.xml
! pom.xml


From vyazelenko at yahoo.com  Fri Aug 15 17:22:44 2014
From: vyazelenko at yahoo.com (vyazelenko at yahoo.com)
Date: Fri, 15 Aug 2014 19:22:44 +0200
Subject: Setup and TearDown executed by different threads
In-Reply-To: <53EE1224.10507@oracle.com>
References: <1407927960.94775.YahooMailNeo@web162204.mail.bf1.yahoo.com>
	<53EB4B25.6090708@oracle.com>
	<DF545D37-5FD8-4492-85AB-2E5A3DAEBC42@yahoo.com>
	<53EB50A2.8020401@oracle.com>
	<1407933620.83089.YahooMailNeo@web162201.mail.bf1.yahoo.com>
	<53EB7C20.70807@oracle.com> <53EE1224.10507@oracle.com>
Message-ID: <7D8ACA9E-C296-446A-89CC-BDD76531E0E3@yahoo.com>

Ok, I'm said panda now. ;( I'll have to workaround the issue on my side. Most likely by dropping @TearDowns and moving all logic into @Setups...

P.S. Thanks for JavaDoc update and 0.9.7 release! :)

Best regards,
Dmitry

Sent from my iPhone

> On Aug 15, 2014, at 15:59, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:
> 
>> On 08/13/2014 06:54 PM, Aleksey Shipilev wrote:
>> Let me see how messy it would be to track the @Setup/@TearDown callers.
> 
> Okay, it seems messier than I expected at the first glance. There are
> multiple things to consider when tracking the @State owner:
> - do we share the owner across multiple Levels?
> - what do we do if the owner exits abruptly?
> - how do we handle the absence of paired @Setup/@TearDowns?
> - ...?
> 
> Not that isn't not doable, but the change seems too intrusive to do in a
> Release Candidate. It also intersects with Chris Vest's example of
> benchmark being stuck if one of the threads in asymmetric benchmarks
> throws the exception. Let us consider this for post 1.0 revisions.
> 
> I had updated the @Setup/@TearDown Javadocs to (un)specify which thread
> is going to call the fixture methods though.
> 
> Thanks,
> -Aleksey.
> 

From vyazelenko at yahoo.com  Fri Aug 15 19:12:56 2014
From: vyazelenko at yahoo.com (Dmitry Vyazelenko)
Date: Fri, 15 Aug 2014 21:12:56 +0200
Subject: Runner closes System.out stream
Message-ID: <1A3E8500-267B-4130-B8B1-010502EDB4D6@yahoo.com>

Hi,

I?ve got the problem with JMH 0.9.7 when using Runner API. It appears that System.out stream is being closed once Runner#run() method is executed.

Here is simple benchmark to reproduce:
public class OutStreamClosedBenchmark {
    @Benchmark
    public void dummy() {
    }

    public static void main(String[] args) throws Exception {
        System.out.println("[OUT] BEFORE");
        System.err.println("[ERR] BEFORE.");
        Runner r = new Runner(new OptionsBuilder()
                .forks(1).warmupIterations(0).measurementIterations(1)
                .include(".*OutStreamClosedBenchmark.*")
                .build());
        r.run();
        System.out.println("[OUT] AFTER");
        System.err.println("[ERR] AFTER");
    }
}

And here is the output when executing java -cp target/benchmarks.jar jmh.test.OutStreamClosedBenchmark:
[OUT] BEFORE
[ERR] BEFORE.
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/Home/jre/bin/java
# VM options: <none>
# Warmup: <none>
# Measurement: 1 iterations, 1 s each
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: jmh.test.OutStreamClosedBenchmark.dummy
...
Benchmark                              Mode  Samples           Score  Score error  Units
j.t.OutStreamClosedBenchmark.dummy    thrpt        1  3573392858.280          NaN  ops/s
[ERR] AFTER

As can be seen the "[OUT] AFTER? is missing in the output.

Best regards,
Dmitry

From ecki at zusammenkunft.net  Fri Aug 15 19:13:53 2014
From: ecki at zusammenkunft.net (Bernd Eckenfels)
Date: Fri, 15 Aug 2014 21:13:53 +0200
Subject: JMH 1.0 SNAPSHOT Failed ResultFormatTest
Message-ID: <20140815211353.00002790.ecki@zusammenkunft.net>

Hello,

in preparation for the patch I want to submit to JMH for enabling
substring RE matching I tried to compile JMH. I am not very familiar
with Hg, but I am pretty sure the clone was not messed up. 

parent: 955:d5526434fd4a tip

Anyway, when I run "mvn clean install" on Win7 with Mvn3.1.2 or 3.0.5
and Oracle JDK 1.8.0_11 or 1.7.0_65 (x64) I get:

Failed tests:
  ResultFormatTest.jsonTest_Stream:137->compare:109 Mismatch
expected:<...         "rawData" :[ []> but was:<...
"rawData" :[]> ResultFormatTest.jsonTest:123->compare:109 Mismatch
expected:<...         "rawData" :[ []> but was:<...
"rawData" :[]>

Tests run: 250, Failures: 2, Errors: 0, Skipped: 1

43551 [INFO] Java Microbenchmark Harness Parent ................ SUCCESS [  5.458 s]
43553 [INFO] JMH Core .......................................... FAILURE [ 30.620 s]

It does compile with skipTests. The result does work to benchmark my
sample projects, so maybe its only a test problem.

BTW: the jmh-core-its do crash the JVM (OutOfMemory, native out of virtual, etc) Java, but I
need to check if this is a platform problem (memory related?). Will followup on that.

Gruss
Bernd



From aleksey.shipilev at oracle.com  Fri Aug 15 19:32:16 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Fri, 15 Aug 2014 23:32:16 +0400
Subject: JMH 1.0 SNAPSHOT Failed ResultFormatTest
In-Reply-To: <20140815211353.00002790.ecki@zusammenkunft.net>
References: <20140815211353.00002790.ecki@zusammenkunft.net>
Message-ID: <53EE6040.9050303@oracle.com>

Hi,

On 08/15/2014 11:13 PM, Bernd Eckenfels wrote:
> Anyway, when I run "mvn clean install" on Win7 with Mvn3.1.2 or 3.0.5
> and Oracle JDK 1.8.0_11 or 1.7.0_65 (x64) I get:

This must be specific to Windows, since CI builds on Linux are fine.

> Failed tests:
>   ResultFormatTest.jsonTest_Stream:137->compare:109 Mismatch
> expected:<...         "rawData" :[ []> but was:<...
> "rawData" :[]> ResultFormatTest.jsonTest:123->compare:109 Mismatch
> expected:<...         "rawData" :[ []> but was:<...
> "rawData" :[]>

Looks like a trailing whitespace to me... CRLF? Will check on some
Windows machines around.

Thanks,
-Aleksey.


From aleksey.shipilev at oracle.com  Fri Aug 15 20:26:03 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Fri, 15 Aug 2014 20:26:03 +0000
Subject: hg: code-tools/jmh: runners: TextReportFormat accidentally closes the
	underlying stream; this may close System.out.
Message-ID: <201408152026.s7FKQ33h013758@aojmv0008>

Changeset: 8e332f3bc3bf
Author:    shade
Date:      2014-08-16 00:13 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/8e332f3bc3bf

runners: TextReportFormat accidentally closes the underlying stream; this may close System.out.

! jmh-core/src/main/java/org/openjdk/jmh/runner/format/TextReportFormat.java


From aleksey.shipilev at oracle.com  Fri Aug 15 20:26:10 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Fri, 15 Aug 2014 20:26:10 +0000
Subject: hg: code-tools/jmh: runners: clean up System.out intercept logic a
	bit, future work.
Message-ID: <201408152026.s7FKQAac013814@aojmv0008>

Changeset: 130caa12e774
Author:    shade
Date:      2014-08-16 00:22 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/130caa12e774

runners: clean up System.out intercept logic a bit, future work.

! jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java


From aleksey.shipilev at oracle.com  Fri Aug 15 20:26:36 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Sat, 16 Aug 2014 00:26:36 +0400
Subject: Runner closes System.out stream
In-Reply-To: <1A3E8500-267B-4130-B8B1-010502EDB4D6@yahoo.com>
References: <1A3E8500-267B-4130-B8B1-010502EDB4D6@yahoo.com>
Message-ID: <53EE6CFC.2080401@oracle.com>

On 08/15/2014 11:12 PM, Dmitry Vyazelenko wrote:
> I?ve got the problem with JMH 0.9.7 when using Runner API. It appears
> that System.out stream is being closed once Runner#run() method is
> executed.

Found the bastard that does it:
 http://hg.openjdk.java.net/code-tools/jmh/rev/8e332f3bc3bf

This seems to pass your test. But all in all, we should be more careful,
and set a few trip-wires around System.out. That's for next week:
 http://hg.openjdk.java.net/code-tools/jmh/rev/130caa12e774

Thanks,
-Aleksey.


From ecki at zusammenkunft.net  Fri Aug 15 20:27:59 2014
From: ecki at zusammenkunft.net (Bernd Eckenfels)
Date: Fri, 15 Aug 2014 22:27:59 +0200
Subject: JMH 1.0 SNAPSHOT Failed ResultFormatTest
In-Reply-To: <20140815211353.00002790.ecki@zusammenkunft.net>
References: <20140815211353.00002790.ecki@zusammenkunft.net>
Message-ID: <20140815222759.00003322.ecki@zusammenkunft.net>

Am Fri, 15 Aug 2014 21:13:53 +0200
schrieb Bernd Eckenfels <ecki at zusammenkunft.net>:
> BTW: the jmh-core-its do crash the JVM (OutOfMemory, native out of
> virtual, etc) Java, but I need to check if this is a platform problem
> (memory related?). Will followup on that.

I have 8GB Ram and only 10GB virtual on this Win7, so it looks like the
<forkCount>2C was too much for that poor Notebook. It caused malloc
failures (in the C1 and C2 compiler threads of the JVM).

This machine uses HT and therefore Java sees 8 cores, so 16 Forks is rather heavy. 
Using forkCount=8 finishes all ITs with no failures or crashes,

      Tests run: 219, Failures: 0, Errors: 0, Skipped: 1
INFO] Total time: 9:14.908s
[INFO] Finished at: Fri Aug 15 22:25:42 CEST 2014
[INFO] Final Memory: 16M/441M
[
(but the unit test failure still happen)

Gruss
Bernd

From vyazelenko at yahoo.com  Fri Aug 15 20:30:43 2014
From: vyazelenko at yahoo.com (vyazelenko at yahoo.com)
Date: Fri, 15 Aug 2014 22:30:43 +0200
Subject: Runner closes System.out stream
In-Reply-To: <53EE6CFC.2080401@oracle.com>
References: <1A3E8500-267B-4130-B8B1-010502EDB4D6@yahoo.com>
	<53EE6CFC.2080401@oracle.com>
Message-ID: <5A20B9F4-AECB-4931-8B4D-67A957D46587@yahoo.com>

Thanks for quick fix!

Best regards,
Dmitry

Sent from my iPhone

> On Aug 15, 2014, at 22:26, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:
> 
>> On 08/15/2014 11:12 PM, Dmitry Vyazelenko wrote:
>> I?ve got the problem with JMH 0.9.7 when using Runner API. It appears
>> that System.out stream is being closed once Runner#run() method is
>> executed.
> 
> Found the bastard that does it:
> http://hg.openjdk.java.net/code-tools/jmh/rev/8e332f3bc3bf
> 
> This seems to pass your test. But all in all, we should be more careful,
> and set a few trip-wires around System.out. That's for next week:
> http://hg.openjdk.java.net/code-tools/jmh/rev/130caa12e774
> 
> Thanks,
> -Aleksey.
> 

From aleksey.shipilev at oracle.com  Fri Aug 15 20:31:26 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Fri, 15 Aug 2014 20:31:26 +0000
Subject: hg: code-tools/jmh: jmh-core-it: lower the number of parallel forks
	to 1 fork per available core.
Message-ID: <201408152031.s7FKVQgf014448@aojmv0008>

Changeset: e777272ea3e2
Author:    shade
Date:      2014-08-16 00:31 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/e777272ea3e2

jmh-core-it: lower the number of parallel forks to 1 fork per available core.

! jmh-core-it/pom.xml


From aleksey.shipilev at oracle.com  Fri Aug 15 20:32:27 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Sat, 16 Aug 2014 00:32:27 +0400
Subject: JMH 1.0 SNAPSHOT Failed ResultFormatTest
In-Reply-To: <20140815222759.00003322.ecki@zusammenkunft.net>
References: <20140815211353.00002790.ecki@zusammenkunft.net>
	<20140815222759.00003322.ecki@zusammenkunft.net>
Message-ID: <53EE6E5B.9020903@oracle.com>

On 08/16/2014 12:27 AM, Bernd Eckenfels wrote:
> Am Fri, 15 Aug 2014 21:13:53 +0200
> schrieb Bernd Eckenfels <ecki at zusammenkunft.net>:
>> BTW: the jmh-core-its do crash the JVM (OutOfMemory, native out of
>> virtual, etc) Java, but I need to check if this is a platform problem
>> (memory related?). Will followup on that.
> 
> I have 8GB Ram and only 10GB virtual on this Win7, so it looks like the
> <forkCount>2C was too much for that poor Notebook. It caused malloc
> failures (in the C1 and C2 compiler threads of the JVM).
> 
> This machine uses HT and therefore Java sees 8 cores, so 16 Forks is rather heavy. 
> Using forkCount=8 finishes all ITs with no failures or crashes,

Got it, we were using 2C to run something in the "shadow" of sleepy
tests. But the tests should be really fast now, therefore, we can just
go with 1C:
  http://hg.openjdk.java.net/code-tools/jmh/rev/e777272ea3e2

Thanks,
-Aleksey.



From aleksey.shipilev at oracle.com  Fri Aug 15 20:55:15 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Fri, 15 Aug 2014 20:55:15 +0000
Subject: hg: code-tools/jmh: results: JSON result tidyer should take care of
	CR symbol on Windows as well.
Message-ID: <201408152055.s7FKtFAC017977@aojmv0008>

Changeset: 892e9d690259
Author:    shade
Date:      2014-08-16 00:54 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/892e9d690259

results: JSON result tidyer should take care of CR symbol on Windows as well.

! jmh-core/src/main/java/org/openjdk/jmh/results/format/JSONResultFormat.java


From aleksey.shipilev at oracle.com  Fri Aug 15 20:55:44 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Sat, 16 Aug 2014 00:55:44 +0400
Subject: JMH 1.0 SNAPSHOT Failed ResultFormatTest
In-Reply-To: <20140815211353.00002790.ecki@zusammenkunft.net>
References: <20140815211353.00002790.ecki@zusammenkunft.net>
Message-ID: <53EE73D0.3070906@oracle.com>

On 08/15/2014 11:13 PM, Bernd Eckenfels wrote:
> Anyway, when I run "mvn clean install" on Win7 with Mvn3.1.2 or 3.0.5
> and Oracle JDK 1.8.0_11 or 1.7.0_65 (x64) I get:
> 
> Failed tests:
>   ResultFormatTest.jsonTest_Stream:137->compare:109 Mismatch
> expected:<...         "rawData" :[ []> but was:<...
> "rawData" :[]> ResultFormatTest.jsonTest:123->compare:109 Mismatch
> expected:<...         "rawData" :[ []> but was:<...
> "rawData" :[]>

Yeah, that was our JSON tidy-er mistreating the CR symbol, which only
occurs on Windows. Fixed:
  http://hg.openjdk.java.net/code-tools/jmh/rev/892e9d690259

Tested it works on Windows 7. Confirm?

Thanks,
-Aleksey.


From ecki at zusammenkunft.net  Fri Aug 15 21:07:58 2014
From: ecki at zusammenkunft.net (Bernd Eckenfels)
Date: Fri, 15 Aug 2014 23:07:58 +0200
Subject: JMH 1.0 SNAPSHOT Failed ResultFormatTest
In-Reply-To: <53EE6040.9050303@oracle.com>
References: <20140815211353.00002790.ecki@zusammenkunft.net>
	<53EE6040.9050303@oracle.com>
Message-ID: <20140815230758.00002cb3.ecki@zusammenkunft.net>

Am Fri, 15 Aug 2014 23:32:16 +0400
schrieb Aleksey Shipilev <aleksey.shipilev at oracle.com>:

> This must be specific to Windows, since CI builds on Linux are fine.
> 
> > Failed tests:
> >   ResultFormatTest.jsonTest_Stream:137->compare:109 Mismatch
> > expected:<...         "rawData" :[ []> but was:<...
> > "rawData" :[]> ResultFormatTest.jsonTest:123->compare:109 Mismatch
> > expected:<...         "rawData" :[ []> but was:<...
> > "rawData" :[]>
> 
> Looks like a trailing whitespace to me... CRLF? Will check on some
> Windows machines around.

Hm no, does not look line line endings, you use lines with the
readLine. When I change the golden file to contain CRLF instead of LF
(which it gets from mercurial) then the test fails as well.

What I noticed is, the golden file contains different spacing for
rawData array:


L37         "scoreUnit" : "ops/ms",
            "rawData" : [
                [
                    942.0,
                    384.0
                ],

...
L89             "scoreUnit" : "ops/ms",
                "rawData" :[
                    [
                        390.0,
                        787.0
                    ],

And when I add code to print out the data you can see the file starts
to differ at line 38 with different formatting (. means actual line):

 0037             "scoreUnit" : "ops/ms",
.0037             "scoreUnit" : "ops/ms",
 0038             "rawData" :[
.0038             "rawData" :
 0039                 [
.0039  [
 0040                     942.0,
.0040                 [
 0041                     384.0
.0041                     942.0,
 0042                 ],
.0042                     384.0

Maybe different JSON toolkit versions?

Gruss
Bernd

diff -r d5526434fd4a
jmh-core/src/test/java/org/openjdk/jmh/results/format/ResultFormatTest.java
---
a/jmh-core/src/test/java/org/openjdk/jmh/results/format/ResultFormatTest.java
Fri Aug 15 18:38:54 2014 +0400 +++
b/jmh-core/src/test/java/org/openjdk/jmh/results/format/ResultFormatTest.java
Fri Aug 15 23:07:24 2014 +0200 @@ -101,12 +101,17 @@ }

     private void compare(String actualFile, String goldenFile) throws
     IOException {
+        System.out.printf("%s vs. %s%n", goldenFile, actualFile);
         BufferedReader actualReader = new BufferedReader(new
     FileReader(actualFile)); BufferedReader goldenReader = new
     BufferedReader(new
     InputStreamReader(ResultFormatTest.class.getResourceAsStream("/org/openjdk/jmh/results/format/"
     + goldenFile)));
+        int line=0;
         while (true) {
             String goldenLine = goldenReader.readLine();
             String actualLine = actualReader.readLine();
-            Assert.assertEquals("Mismatch", goldenLine, actualLine);
+            line++;
+            System.out.printf(" %04d %s%n", line, goldenLine);
+            System.out.printf(".%04d %s%n", line, actualLine);
+            //Assert.assertEquals("Mismatch L" + line, goldenLine,
     actualLine); if (goldenLine == null && actualLine == null) break;
         }
     }

C

From ecki at zusammenkunft.net  Fri Aug 15 21:14:19 2014
From: ecki at zusammenkunft.net (Bernd Eckenfels)
Date: Fri, 15 Aug 2014 23:14:19 +0200
Subject: JMH 1.0 SNAPSHOT Failed ResultFormatTest
In-Reply-To: <53EE73D0.3070906@oracle.com>
References: <20140815211353.00002790.ecki@zusammenkunft.net>
	<53EE73D0.3070906@oracle.com>
Message-ID: <20140815231419.00007aff.ecki@zusammenkunft.net>

Am Sat, 16 Aug 2014 00:55:44 +0400
schrieb Aleksey Shipilev <aleksey.shipilev at oracle.com>:

> Yeah, that was our JSON tidy-er mistreating the CR symbol, which only
> occurs on Windows. Fixed:
>   http://hg.openjdk.java.net/code-tools/jmh/rev/892e9d690259
> 
> Tested it works on Windows 7. Confirm?

Yes, confirmed. Thanks! Passes with 959:892e9d690259. Strange enough
the different spacing in rawdata in the golden file is still present?

Gruss
Bernd

From aleksey.shipilev at oracle.com  Sat Aug 16 11:00:32 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Sat, 16 Aug 2014 11:00:32 +0000
Subject: hg: code-tools/jmh: Apply SUIDs for all Serializable classes: this
	protects from the benign instrumentation.
Message-ID: <201408161100.s7GB0Wit021523@aojmv0008>

Changeset: 3d44d68e45be
Author:    shade
Date:      2014-08-16 15:00 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/3d44d68e45be

Apply SUIDs for all Serializable classes: this protects from the benign instrumentation.

! jmh-core/src/main/java/org/openjdk/jmh/generators/core/GenerationException.java
! jmh-core/src/main/java/org/openjdk/jmh/infra/BenchmarkParams.java
! jmh-core/src/main/java/org/openjdk/jmh/infra/IterationParams.java
! jmh-core/src/main/java/org/openjdk/jmh/profile/LinuxPerfAsmProfiler.java
! jmh-core/src/main/java/org/openjdk/jmh/profile/LinuxPerfProfiler.java
! jmh-core/src/main/java/org/openjdk/jmh/profile/ProfilerResult.java
! jmh-core/src/main/java/org/openjdk/jmh/profile/StackProfiler.java
! jmh-core/src/main/java/org/openjdk/jmh/results/AverageTimeResult.java
! jmh-core/src/main/java/org/openjdk/jmh/results/IterationResult.java
! jmh-core/src/main/java/org/openjdk/jmh/results/Result.java
! jmh-core/src/main/java/org/openjdk/jmh/results/SampleTimeResult.java
! jmh-core/src/main/java/org/openjdk/jmh/results/SingleShotResult.java
! jmh-core/src/main/java/org/openjdk/jmh/results/ThroughputResult.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/Action.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/ActionPlan.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/BenchmarkException.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/NoBenchmarksException.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/RunnerException.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/WorkloadParams.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/link/ActionPlanFrame.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/link/ExceptionFrame.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/link/FinishingFrame.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/link/InfraFrame.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/link/OptionsFrame.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/link/OutputFrame.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/link/ResultsFrame.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/options/CommandLineOptionException.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/options/CommandLineOptions.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/options/OptionsBuilder.java
! jmh-core/src/main/java/org/openjdk/jmh/util/AbstractStatistics.java
! jmh-core/src/main/java/org/openjdk/jmh/util/HashMultimap.java
! jmh-core/src/main/java/org/openjdk/jmh/util/HashMultiset.java
! jmh-core/src/main/java/org/openjdk/jmh/util/HashsetMultimap.java
! jmh-core/src/main/java/org/openjdk/jmh/util/MultisetStatistics.java
! jmh-core/src/main/java/org/openjdk/jmh/util/Optional.java
! jmh-core/src/main/java/org/openjdk/jmh/util/SampleBuffer.java
! jmh-core/src/main/java/org/openjdk/jmh/util/TreeMultimap.java
! jmh-core/src/main/java/org/openjdk/jmh/util/TreeMultiset.java


From aleksey.shipilev at oracle.com  Mon Aug 18 11:13:25 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Mon, 18 Aug 2014 11:13:25 +0000
Subject: hg: code-tools/jmh: runners: touch the result file early.
Message-ID: <201408181113.s7IBDPAW016616@aojmv0008>

Changeset: 0e898fece0ec
Author:    shade
Date:      2014-08-18 15:13 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/0e898fece0ec

runners: touch the result file early.

! jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java
! jmh-core/src/main/java/org/openjdk/jmh/util/FileUtils.java


From aleksey.shipilev at oracle.com  Mon Aug 18 11:41:29 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Mon, 18 Aug 2014 11:41:29 +0000
Subject: hg: code-tools/jmh: output: use "---" instead of "N/A" all over the
	place.
Message-ID: <201408181141.s7IBfUh1020852@aojmv0008>

Changeset: b6b8161b42a2
Author:    shade
Date:      2014-08-18 15:30 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/b6b8161b42a2

output: use "---" instead of "N/A" all over the place.

! jmh-core/src/main/java/org/openjdk/jmh/profile/LinuxPerfAsmProfiler.java
! jmh-core/src/main/java/org/openjdk/jmh/results/format/TextResultFormat.java


From aleksey.shipilev at oracle.com  Mon Aug 18 11:42:07 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Mon, 18 Aug 2014 11:42:07 +0000
Subject: hg: code-tools/jmh: output: shun the results without any values (e.g.
	perfasm @asm, and stack @stack).
Message-ID: <201408181142.s7IBg7WO021045@aojmv0008>

Changeset: 83849e54f773
Author:    shade
Date:      2014-08-18 15:41 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/83849e54f773

output: shun the results without any values (e.g. perfasm @asm, and stack @stack).

! jmh-core/src/main/java/org/openjdk/jmh/profile/LinuxPerfAsmProfiler.java
! jmh-core/src/main/java/org/openjdk/jmh/profile/StackProfiler.java
! jmh-core/src/main/java/org/openjdk/jmh/results/ResultRole.java


From aleksey.shipilev at oracle.com  Mon Aug 18 11:54:19 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Mon, 18 Aug 2014 11:54:19 +0000
Subject: hg: code-tools/jmh: Revert accidental "N/A" -> "---" commit.
Message-ID: <201408181154.s7IBsJZU022536@aojmv0008>

Changeset: da149d95b9cc
Author:    shade
Date:      2014-08-18 15:45 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/da149d95b9cc

Revert accidental "N/A" -> "---" commit.

! jmh-core/src/main/java/org/openjdk/jmh/results/format/TextResultFormat.java


From aleksey.shipilev at oracle.com  Mon Aug 18 12:22:26 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Mon, 18 Aug 2014 12:22:26 +0000
Subject: hg: code-tools/jmh: profilers: print perfasm/stack profilers again.
Message-ID: <201408181222.s7ICMQti028093@aojmv0008>

Changeset: e32fd30f13db
Author:    shade
Date:      2014-08-18 16:18 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/e32fd30f13db

profilers: print perfasm/stack profilers again.

! jmh-core/src/main/java/org/openjdk/jmh/profile/LinuxPerfAsmProfiler.java
! jmh-core/src/main/java/org/openjdk/jmh/profile/StackProfiler.java


From aleksey.shipilev at oracle.com  Mon Aug 18 12:35:20 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Mon, 18 Aug 2014 12:35:20 +0000
Subject: hg: code-tools/jmh: profilers: perfasm,
	reliably detect the absence of assembly output.
Message-ID: <201408181235.s7ICZKWU000292@aojmv0008>

Changeset: c941eda8b87f
Author:    shade
Date:      2014-08-18 16:35 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/c941eda8b87f

profilers: perfasm, reliably detect the absence of assembly output.

! jmh-core/src/main/java/org/openjdk/jmh/profile/LinuxPerfAsmProfiler.java


From aleksey.shipilev at oracle.com  Mon Aug 18 12:50:02 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Mon, 18 Aug 2014 12:50:02 +0000
Subject: hg: code-tools/jmh: runners: print shorter message about profilers
	consuming stdout/stderr.
Message-ID: <201408181250.s7ICo2wT002840@aojmv0008>

Changeset: 3c1528368e9f
Author:    shade
Date:      2014-08-18 16:49 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/3c1528368e9f

runners: print shorter message about profilers consuming stdout/stderr.

! jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java


From brianfromoregon at gmail.com  Mon Aug 18 14:31:32 2014
From: brianfromoregon at gmail.com (Brian Harris)
Date: Mon, 18 Aug 2014 07:31:32 -0700
Subject: hg: code-tools/jmh: runners: print shorter message about
	profilers consuming stdout/stderr.
In-Reply-To: <201408181250.s7ICo2wT002840@aojmv0008>
References: <201408181250.s7ICo2wT002840@aojmv0008>
Message-ID: <CAFtUM9Zn6Hk5Yi5XFV7Kpa7ub1FJ6abF+dK+NOftq876hm8T2w@mail.gmail.com>

In case it hasn't been requested before, consider sending commit msgs to a
different group than the discussion forum. Thanks!


On Mon, Aug 18, 2014 at 5:50 AM, <aleksey.shipilev at oracle.com> wrote:

> Changeset: 3c1528368e9f
> Author:    shade
> Date:      2014-08-18 16:49 +0400
> URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/3c1528368e9f
>
> runners: print shorter message about profilers consuming stdout/stderr.
>
> ! jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java
>
>

From aleksey.shipilev at oracle.com  Wed Aug 20 08:59:07 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Wed, 20 Aug 2014 08:59:07 +0000
Subject: hg: code-tools/jmh: runners: protect from accidentally closing the
	System.out.
Message-ID: <201408200859.s7K8x81m011544@aojmv0008>

Changeset: ed4b2a0461b2
Author:    shade
Date:      2014-08-20 12:25 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/ed4b2a0461b2

runners: protect from accidentally closing the System.out.

! jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java
+ jmh-core/src/main/java/org/openjdk/jmh/util/UnCloseablePrintStream.java


From aleksey.shipilev at oracle.com  Wed Aug 20 08:59:16 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Wed, 20 Aug 2014 08:59:16 +0000
Subject: hg: code-tools/jmh: runners: do not intercept System.out in a host VM
	at all.
Message-ID: <201408200859.s7K8xG70011598@aojmv0008>

Changeset: b9151815e306
Author:    shade
Date:      2014-08-20 12:47 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/b9151815e306

runners: do not intercept System.out in a host VM at all.
Do that only in forked VMs where we are the exclusive users.

! jmh-core/src/main/java/org/openjdk/jmh/runner/ForkedMain.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/ForkedRunner.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java
! jmh-core/src/main/java/org/openjdk/jmh/runner/link/BinaryLinkClient.java


From aleksey.shipilev at oracle.com  Wed Aug 20 09:01:08 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Wed, 20 Aug 2014 09:01:08 +0000
Subject: hg: code-tools/jmh: runners: declare Runner to be usable only once.
Message-ID: <201408200901.s7K918vK011947@aojmv0008>

Changeset: a2f6cce2c08a
Author:    shade
Date:      2014-08-20 12:58 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/a2f6cce2c08a

runners: declare Runner to be usable only once.

! jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java


From aleksey.shipilev at oracle.com  Wed Aug 20 12:08:06 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Wed, 20 Aug 2014 12:08:06 +0000
Subject: hg: code-tools/jmh: results: fix annoying formatting issue in JSON
	output.
Message-ID: <201408201208.s7KC86hZ011164@aojmv0008>

Changeset: 4a845244cdfc
Author:    shade
Date:      2014-08-20 16:07 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/4a845244cdfc

results: fix annoying formatting issue in JSON output.

! jmh-core/src/main/java/org/openjdk/jmh/results/format/JSONResultFormat.java
! jmh-core/src/test/resources/org/openjdk/jmh/results/format/output-golden.json


From aleksey.shipilev at oracle.com  Wed Aug 20 12:08:59 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Wed, 20 Aug 2014 16:08:59 +0400
Subject: JMH 1.0 SNAPSHOT Failed ResultFormatTest
In-Reply-To: <20140815231419.00007aff.ecki@zusammenkunft.net>
References: <20140815211353.00002790.ecki@zusammenkunft.net>	<53EE73D0.3070906@oracle.com>
	<20140815231419.00007aff.ecki@zusammenkunft.net>
Message-ID: <53F48FDB.8050708@oracle.com>

On 08/16/2014 01:14 AM, Bernd Eckenfels wrote:
> Strange enough the different spacing in rawdata in the golden file is
> still present?

Yes, the perils of constructing JSON by hand. Fixed:
  http://hg.openjdk.java.net/code-tools/jmh/rev/4a845244cdfc

-Aleksey.


From aleksey.shipilev at oracle.com  Wed Aug 20 12:23:23 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Wed, 20 Aug 2014 12:23:23 +0000
Subject: hg: code-tools/jmh: Remove obsolete scratch/ files.
Message-ID: <201408201223.s7KCNOFo013352@aojmv0008>

Changeset: 0d74d2f85299
Author:    shade
Date:      2014-08-20 16:23 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/0d74d2f85299

Remove obsolete scratch/ files.

- jmh-core/src/scratch/JFRProfiler.java


From brianfromoregon at gmail.com  Thu Aug 21 17:06:45 2014
From: brianfromoregon at gmail.com (Brian Harris)
Date: Thu, 21 Aug 2014 10:06:45 -0700
Subject: Temp files
Message-ID: <CAFtUM9ZVELWu5eJd9zym1xtFH5ZQ3kSvULQXT_f1xV4DhaH4=g@mail.gmail.com>

Hi, wanted to report the inability to run jmh when user doesnt have write
access to tmp dir. Must you create a temp file on the real fs? Maybe there
could be a switch to change the dir used.

java.lang.IllegalStateException: Error creating compiler hints file
	at org.openjdk.jmh.runner.CompilerHints.hintsFile(CompilerHints.java:74)
	at org.openjdk.jmh.runner.CompilerHints.addCompilerHints(CompilerHints.java:199)
	at org.openjdk.jmh.runner.Runner.getSeparateExecutionCommand(Runner.java:698)
	at org.openjdk.jmh.runner.Runner.runSeparate(Runner.java:513)
	at org.openjdk.jmh.runner.Runner.runBenchmarks(Runner.java:449)
	at org.openjdk.jmh.runner.Runner.run(Runner.java:232)
	at HelloJmhTest.run(HelloJmhTest.java:26)
Caused by: java.io.IOException: Permission denied
	at java.io.UnixFileSystem.createFileExclusively(Native Method)
	at java.io.File.createTempFile(File.java:1879)
	at java.io.File.createTempFile(File.java:1923)
	at org.openjdk.jmh.util.FileUtils.tempFile(FileUtils.java:53)
	at org.openjdk.jmh.util.FileUtils.createTempFileWithLines(FileUtils.java:111)
	at org.openjdk.jmh.runner.CompilerHints.hintsFile(CompilerHints.java:72)

From staffan.friberg at oracle.com  Thu Aug 21 17:27:19 2014
From: staffan.friberg at oracle.com (Staffan Friberg)
Date: Thu, 21 Aug 2014 10:27:19 -0700
Subject: Temp files
In-Reply-To: <CAFtUM9ZVELWu5eJd9zym1xtFH5ZQ3kSvULQXT_f1xV4DhaH4=g@mail.gmail.com>
References: <CAFtUM9ZVELWu5eJd9zym1xtFH5ZQ3kSvULQXT_f1xV4DhaH4=g@mail.gmail.com>
Message-ID: <53F62BF7.6050703@oracle.com>

You should be able to control the temp directory the JDK uses with 
-Djava.io.tmpdir.

Cheers,
Staffan

On 08/21/2014 10:06 AM, Brian Harris wrote:
> Hi, wanted to report the inability to run jmh when user doesnt have write
> access to tmp dir. Must you create a temp file on the real fs? Maybe there
> could be a switch to change the dir used.
>
> java.lang.IllegalStateException: Error creating compiler hints file
> 	at org.openjdk.jmh.runner.CompilerHints.hintsFile(CompilerHints.java:74)
> 	at org.openjdk.jmh.runner.CompilerHints.addCompilerHints(CompilerHints.java:199)
> 	at org.openjdk.jmh.runner.Runner.getSeparateExecutionCommand(Runner.java:698)
> 	at org.openjdk.jmh.runner.Runner.runSeparate(Runner.java:513)
> 	at org.openjdk.jmh.runner.Runner.runBenchmarks(Runner.java:449)
> 	at org.openjdk.jmh.runner.Runner.run(Runner.java:232)
> 	at HelloJmhTest.run(HelloJmhTest.java:26)
> Caused by: java.io.IOException: Permission denied
> 	at java.io.UnixFileSystem.createFileExclusively(Native Method)
> 	at java.io.File.createTempFile(File.java:1879)
> 	at java.io.File.createTempFile(File.java:1923)
> 	at org.openjdk.jmh.util.FileUtils.tempFile(FileUtils.java:53)
> 	at org.openjdk.jmh.util.FileUtils.createTempFileWithLines(FileUtils.java:111)
> 	at org.openjdk.jmh.runner.CompilerHints.hintsFile(CompilerHints.java:72)


From B.Eckenfels at seeburger.de  Fri Aug 22 00:40:11 2014
From: B.Eckenfels at seeburger.de (Eckenfels. Bernd)
Date: Fri, 22 Aug 2014 00:40:11 +0000
Subject: Partial RE for Benchmark Tests
Message-ID: <ED12D06EC205E3458FC3E23CD84D4ED50103A90E14@dedcexch02.seeburger.de>

Hello,

as discussed before I propose to change the benchmark name filter from a all-match to a regular expression pattern find. This is a very small code change. The TestBenchmarkList is currently @Ignored because (I think) the test input file is in a old/wrong format.

I changed the BenchmarkList file by adding some (random) benchmark parameters to make it comply the line format. Currently the tests all work on the user-visible benchmark name of these lines, so it doesn't really matter what's in the remaining 19 args. But keep that in mind if testing for more features will be added.

In addition to actually enabling the tests again I also added some test cases around ^$ as well as lists, the empty list case and the "typical" '.' abuse case.

All the tests (including IT) pass with my modification.

Here is a sample usage, which shows you don't need .* for substrings (but you can still use it)

jmh\jmh-core-benchmarks\target>java -jar benchmarks.jar -l Int

Benchmarks:
org.openjdk.jmh.benchmarks.BlackholeBench.explicit_testInt
org.openjdk.jmh.benchmarks.BlackholeBench.implicit_testInt
org.openjdk.jmh.benchmarks.BlackholePipelineBench.test_Int_1
org.openjdk.jmh.benchmarks.BlackholePipelineBench.test_Int_10
org.openjdk.jmh.benchmarks.BlackholePipelineBench.test_Int_100
org.openjdk.jmh.benchmarks.BlackholePipelineBench.test_Int_1000


jmh\jmh-core-benchmarks\target>java -jar benchmarks.jar -l Int -e BlackholeBench

Benchmarks:
org.openjdk.jmh.benchmarks.BlackholePipelineBench.test_Int_1
org.openjdk.jmh.benchmarks.BlackholePipelineBench.test_Int_10
org.openjdk.jmh.benchmarks.BlackholePipelineBench.test_Int_100
org.openjdk.jmh.benchmarks.BlackholePipelineBench.test_Int_1000


jmh\jmh-core-benchmarks\target>java -jar benchmarks.jar -l Int$

Benchmarks:
org.openjdk.jmh.benchmarks.BlackholeBench.explicit_testInt
org.openjdk.jmh.benchmarks.BlackholeBench.implicit_testInt


jmh\jmh-core-benchmarks\target>java -jar benchmarks.jar -l .*Object.*

Benchmarks:
org.openjdk.jmh.benchmarks.BlackholeBench.explicit_testObject
org.openjdk.jmh.benchmarks.BlackholeBench.implicit_testObject

There is one thing unclear, the exclude pattern is matched against the while line, not only the user visible benchmark name, is this intentional - the verbose statement contradicts this somewhat.

This is my account where we have submitted a contributors agreement, feel free to use the patch or let me know what should be improved.

Greetings
Bernd
--
www.seeburger.com












SEEBURGER AG            Vorstand/Seeburger Executive Board:
Sitz der Gesellschaft/Registered Office:                Bernd Seeburger, Axel Haas, Michael Kleeberg
Edisonstr. 1
D-75015 Bretten         Vorsitzender des Aufsichtsrats/Chairperson of the Seeburger Supervisory Board:
Tel.: 07252 / 96 - 0            Dr. Franz Scherer
Fax: 07252 / 96 - 2222
Internet: http://www.seeburger.de               Registergericht/Commercial Register:
e-mail: info at seeburger.de               HRB 240708 Mannheim


Dieses E-Mail ist nur f?r den Empf?nger bestimmt, an den es gerichtet ist und kann vertrauliches bzw. unter das Berufsgeheimnis fallendes Material enthalten. Jegliche darin enthaltene Ansicht oder Meinungs?u?erung ist die des Autors und stellt nicht notwendigerweise die Ansicht oder Meinung der SEEBURGER AG dar. Sind Sie nicht der Empf?nger, so haben Sie diese E-Mail irrt?mlich erhalten und jegliche Verwendung, Ver?ffentlichung, Weiterleitung, Abschrift oder jeglicher Druck dieser E-Mail ist strengstens untersagt. Weder die SEEBURGER AG noch der Absender (Eckenfels. Bernd) ?bernehmen die Haftung f?r Viren; es obliegt Ihrer Verantwortung, die E-Mail und deren Anh?nge auf Viren zu pr?fen.


This email is intended only for the recipient(s) to whom it is addressed. This email may contain confidential material that may be protected by professional secrecy. Any fact or opinion contained, or expression of the material herein, does not necessarily reflect that of SEEBURGER AG. If you are not the addressee or if you have received this email in error, any use, publication or distribution including forwarding, copying or printing is strictly prohibited. Neither SEEBURGER AG, nor the sender (Eckenfels. Bernd) accept liability for viruses; it is your responsibility to check this email and its attachments for viruses.


From B.Eckenfels at seeburger.de  Fri Aug 22 01:00:08 2014
From: B.Eckenfels at seeburger.de (Eckenfels. Bernd)
Date: Fri, 22 Aug 2014 01:00:08 +0000
Subject: Partial RE for Benchmark Tests
Message-ID: <ED12D06EC205E3458FC3E23CD84D4ED50103A90E48@dedcexch02.seeburger.de>

Hm, I thought it was required to submit the patch via the list - looks like the attachment was stripped. Here it is as an download:

https://gist.github.com/ecki/1e3ea203dba902407843

Sorry I don?t have a full working Hg stack here, so I hope this is enough.

Gruss
Bernd
--
www.seeburger.com











SEEBURGER AG            Vorstand/Seeburger Executive Board:
Sitz der Gesellschaft/Registered Office:                Bernd Seeburger, Axel Haas, Michael Kleeberg
Edisonstr. 1
D-75015 Bretten         Vorsitzender des Aufsichtsrats/Chairperson of the Seeburger Supervisory Board:
Tel.: 07252 / 96 - 0            Dr. Franz Scherer
Fax: 07252 / 96 - 2222
Internet: http://www.seeburger.de               Registergericht/Commercial Register:
e-mail: info at seeburger.de               HRB 240708 Mannheim


Dieses E-Mail ist nur f?r den Empf?nger bestimmt, an den es gerichtet ist und kann vertrauliches bzw. unter das Berufsgeheimnis fallendes Material enthalten. Jegliche darin enthaltene Ansicht oder Meinungs?u?erung ist die des Autors und stellt nicht notwendigerweise die Ansicht oder Meinung der SEEBURGER AG dar. Sind Sie nicht der Empf?nger, so haben Sie diese E-Mail irrt?mlich erhalten und jegliche Verwendung, Ver?ffentlichung, Weiterleitung, Abschrift oder jeglicher Druck dieser E-Mail ist strengstens untersagt. Weder die SEEBURGER AG noch der Absender (Eckenfels. Bernd) ?bernehmen die Haftung f?r Viren; es obliegt Ihrer Verantwortung, die E-Mail und deren Anh?nge auf Viren zu pr?fen.


This email is intended only for the recipient(s) to whom it is addressed. This email may contain confidential material that may be protected by professional secrecy. Any fact or opinion contained, or expression of the material herein, does not necessarily reflect that of SEEBURGER AG. If you are not the addressee or if you have received this email in error, any use, publication or distribution including forwarding, copying or printing is strictly prohibited. Neither SEEBURGER AG, nor the sender (Eckenfels. Bernd) accept liability for viruses; it is your responsibility to check this email and its attachments for viruses.


From ecki at zusammenkunft.net  Fri Aug 22 01:31:13 2014
From: ecki at zusammenkunft.net (Bernd Eckenfels)
Date: Fri, 22 Aug 2014 03:31:13 +0200
Subject: Maven Archetype License and Versioning
Message-ID: <20140822033113.00005a11.ecki@zusammenkunft.net>

Hello,

there is one thing which bothers me a bit about the created
test-harness by the maven archetype. The generated MyBenchmark class
and the POM file has an Oracle copyright header.

JMH is supposed to be GPL with classpath exception, so strictly
speaking the created benchmarks do not have to be GPL. However it is
unclear how this relates to the MyBenchmark template and the POM.

Would it be possible to remove the license headers of those files or
adjust them to make them modifyable?


While speaking of changes to the archetype, I was thinking it would be
a good idea to actually add a property like
"<jmh.templateversion>1.0</>" which is then added to the Manifest of
the generated benchmark. This way when someone updates the JMH version,
JMH itself can warn if a too old maven archetype template was abused.

Gruss
Bernd

From ssb687 at gmail.com  Thu Aug 21 17:52:44 2014
From: ssb687 at gmail.com (Samuel Bishop)
Date: Thu, 21 Aug 2014 13:52:44 -0400
Subject: Exception in Benchmark isn't marked as failure in CSV file.
Message-ID: <CADOP4RbWiwfMJAUWi8+UoM793AOOxRwu7WFzqha+aSukWfa2tA@mail.gmail.com>

Hello JMH-mailing list,

I've been reading the provided JMH samples, and I see that in sample 2,
that JMH will stop measuring if an exception is thrown.  To test this I
wrote a small benchmark to intentionally throw an exception when given a
particular parameter. I was surprised to see that the CSV file didn't mark
that anything had gone wrong when given my "bad" parameter.

For convenience, I will post my benchmark and CSV output below:

@State(Scope.Thread)
 public class BenchmarkException {

    @Param({"0","1","2"})
    int l;

    private BogusItem i;

    @Setup
    public void setUp()
    {
        i = new BogusItem();
    }

    @Benchmark
    @BenchmarkMode(Mode.Throughput)
    @Warmup(iterations = 2)
    @Measurement(iterations = 2)
    public void testExcept() throws BogusException {
        i.expensiveCalculation(l);
    }


    private class BogusException extends Throwable {

    }

    private class BogusItem {

        private BogusItem(){

        }

        private int expensiveCalculation(int l) throws BogusException
        {
            int answer = 0;
            switch (l)
            {
                case ( 0 ):
                    answer = answer + l + 10;
                    break;
                case ( 1 ):
                    answer = answer + l;
                    break;
                case ( 2 ):
                    throw new BogusException();
                default:
                    answer = 0;
                    break;
            }
            return answer;
        }
    }


}




With the output CSV file of :

"Benchmark","Mode","Threads","Samples","Score","Score Error
(99.9%)","Unit","Param: l"
"benchmark.BenchmarkException.testExcept","thrpt",1,20,8.824403593474023E8,3449877.7951812423,"ops/s","0"
"benchmark.BenchmarkException.testExcept","thrpt",1,20,1.1786938536323738E9,2625966.0416936204,"ops/s","1"

Notice the complete absence of an error for my parameter = 2. Is this a
bug, or intended?
I was expecting something to be there in the file, even if just a string of
"benchmark failed for parameter == 2".

From aleksey.shipilev at oracle.com  Fri Aug 22 07:21:44 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Fri, 22 Aug 2014 11:21:44 +0400
Subject: Temp files
In-Reply-To: <CAFtUM9ZVELWu5eJd9zym1xtFH5ZQ3kSvULQXT_f1xV4DhaH4=g@mail.gmail.com>
References: <CAFtUM9ZVELWu5eJd9zym1xtFH5ZQ3kSvULQXT_f1xV4DhaH4=g@mail.gmail.com>
Message-ID: <53F6EF88.90601@oracle.com>

Hi Brian,

On 08/21/2014 09:06 PM, Brian Harris wrote:
> Hi, wanted to report the inability to run jmh when user doesnt have write
> access to tmp dir. Must you create a temp file on the real fs? Maybe there
> could be a switch to change the dir used.

Yes, JMH expects a writable temp dir, since we are communicating many
things to/from the forked VM via the files. Moving out to completely
read-only execution will introduce quite a mess in the harness
infrastructure. The use case of non-writable tmp dir is minuscule in
proportion, as far as I can see. Therefore, we wouldn't reconsider in
nearest future.

As Staffan said, -Djava.io.tmpdir is your friend here :)

-Aleksey.


From aleksey.shipilev at oracle.com  Fri Aug 22 07:44:21 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Fri, 22 Aug 2014 07:44:21 +0000
Subject: hg: code-tools/jmh: runners: partially match the benchmarks with
	non-anchored regular expressions.
Message-ID: <201408220744.s7M7iLad004718@aojmv0008>

Changeset: 8e48b96c25e2
Author:    shade
Date:      2014-08-22 11:44 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/8e48b96c25e2

runners: partially match the benchmarks with non-anchored regular expressions.
Contributed-by: Bernd Eckenfels <b.eckenfels at seeburger.de>

! jmh-core/src/main/java/org/openjdk/jmh/runner/BenchmarkList.java
! jmh-core/src/test/java/org/openjdk/jmh/runner/TestBenchmarkList.java
! jmh-core/src/test/resources/org/openjdk/jmh/runner/MicroBenchmarks


From aleksey.shipilev at oracle.com  Fri Aug 22 07:44:54 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Fri, 22 Aug 2014 07:44:54 +0000
Subject: hg: code-tools/jmh: runners: exclude pattern should match the
	benchmark name only.
Message-ID: <201408220744.s7M7isRk004833@aojmv0008>

Changeset: 1fe1a9936fa8
Author:    shade
Date:      2014-08-22 11:44 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/1fe1a9936fa8

runners: exclude pattern should match the benchmark name only.

! jmh-core/src/main/java/org/openjdk/jmh/runner/BenchmarkList.java


From aleksey.shipilev at oracle.com  Fri Aug 22 07:45:19 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Fri, 22 Aug 2014 11:45:19 +0400
Subject: Partial RE for Benchmark Tests
In-Reply-To: <ED12D06EC205E3458FC3E23CD84D4ED50103A90E14@dedcexch02.seeburger.de>
References: <ED12D06EC205E3458FC3E23CD84D4ED50103A90E14@dedcexch02.seeburger.de>
Message-ID: <53F6F50F.3010308@oracle.com>

On 08/22/2014 04:40 AM, Eckenfels. Bernd wrote:
> as discussed before I propose to change the benchmark name filter
> from a all-match to a regular expression pattern find. This is a very
> small code change. The TestBenchmarkList is currently @Ignored
> because (I think) the test input file is in a old/wrong format.

THANK YOU, that was an embarrassing test omission :) Re-enabling that
test warrants the entire change to go in release candidate.

Committed your patch verbatim:
  http://hg.openjdk.java.net/code-tools/jmh/rev/8e48b96c25e2

-Aleksey.


From aleksey.shipilev at oracle.com  Fri Aug 22 07:45:34 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Fri, 22 Aug 2014 11:45:34 +0400
Subject: Partial RE for Benchmark Tests
In-Reply-To: <ED12D06EC205E3458FC3E23CD84D4ED50103A90E14@dedcexch02.seeburger.de>
References: <ED12D06EC205E3458FC3E23CD84D4ED50103A90E14@dedcexch02.seeburger.de>
Message-ID: <53F6F51E.1050308@oracle.com>

On 08/22/2014 04:40 AM, Eckenfels. Bernd wrote:
> There is one thing unclear, the exclude pattern is matched against
> the while line, not only the user visible benchmark name, is this
> intentional - the verbose statement contradicts this somewhat.

That's a bug, fixed:
  http://hg.openjdk.java.net/code-tools/jmh/rev/1fe1a9936fa8

-Aleksey.


From aleksey.shipilev at oracle.com  Fri Aug 22 08:16:33 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Fri, 22 Aug 2014 08:16:33 +0000
Subject: hg: code-tools/jmh: runners: clean up BenchmarkList and its tests.
Message-ID: <201408220816.s7M8GXMY009843@aojmv0008>

Changeset: 928634dbc9ec
Author:    shade
Date:      2014-08-22 12:08 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/928634dbc9ec

runners: clean up BenchmarkList and its tests.

! jmh-core/src/main/java/org/openjdk/jmh/runner/BenchmarkList.java
! jmh-core/src/test/java/org/openjdk/jmh/runner/TestBenchmarkList.java


From aleksey.shipilev at oracle.com  Fri Aug 22 09:25:35 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Fri, 22 Aug 2014 09:25:35 +0000
Subject: hg: code-tools/jmh: jmh-core-it: fix build failure in Parameters.
Message-ID: <201408220925.s7M9PZAP020883@aojmv0008>

Changeset: a128b58999b1
Author:    shade
Date:      2014-08-22 13:25 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/a128b58999b1

jmh-core-it: fix build failure in Parameters.

! jmh-core-it/src/test/java/org/openjdk/jmh/it/parameters/Parameters.java


From aleksey.shipilev at oracle.com  Fri Aug 22 09:48:39 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Fri, 22 Aug 2014 09:48:39 +0000
Subject: hg: code-tools/jmh: samples: use the shorter matches "A" instead of
	".*A.*".
Message-ID: <201408220948.s7M9mdbk024587@aojmv0008>

Changeset: 962a1d288d71
Author:    shade
Date:      2014-08-22 13:47 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/962a1d288d71

samples: use the shorter matches "A" instead of ".*A.*".

! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_01_HelloWorld.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_02_BenchmarkModes.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_03_States.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_04_DefaultState.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_05_StateFixtures.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_06_FixtureLevel.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_07_FixtureLevelInvocation.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_08_DeadCode.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_09_Blackholes.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_10_ConstantFold.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_11_Loops.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_12_Forking.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_13_RunToRun.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_15_Asymmetric.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_16_CompilerControl.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_17_SyncIterations.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_18_Control.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_20_Annotations.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_21_ConsumeCPU.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_22_FalseSharing.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_23_AuxCounters.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_24_Inheritance.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_25_API_GA.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_26_BatchSize.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_27_Params.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_28_BlackholeHelpers.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_29_StatesDAG.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_30_Interrupts.java
! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_31_InfraParams.java


From aleksey.shipilev at oracle.com  Fri Aug 22 10:03:56 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Fri, 22 Aug 2014 14:03:56 +0400
Subject: Exception in Benchmark isn't marked as failure in CSV file.
In-Reply-To: <CADOP4RbWiwfMJAUWi8+UoM793AOOxRwu7WFzqha+aSukWfa2tA@mail.gmail.com>
References: <CADOP4RbWiwfMJAUWi8+UoM793AOOxRwu7WFzqha+aSukWfa2tA@mail.gmail.com>
Message-ID: <53F7158C.1080808@oracle.com>

Hi Samuel,

Thanks for the bug report!

On 08/21/2014 09:52 PM, Samuel Bishop wrote:
> I've been reading the provided JMH samples, and I see that in sample 2,
> that JMH will stop measuring if an exception is thrown.  To test this I
> wrote a small benchmark to intentionally throw an exception when given a
> particular parameter. I was surprised to see that the CSV file didn't mark
> that anything had gone wrong when given my "bad" parameter.

This affects not only the CSV file, but also any other facility that
feeds on JMH results, including the final table in the human-readable
output.

Notice the result is not produced for a faulty benchmark at all, which
is the easiest of the sane things for harness to do. Too bad users have
to refer to the benchmark log to figure out what went wrong, and why the
result is missing.

I have looked into printing some human-readable message in the cases
like this, and concluded the change is too disruptive for current
release candidate. It also adds to the bucket of fixes we have to have
for treating the failing benchmarks well. Since failing benchmarks are
not the things we want to measure anyway, we will revisit this behavior
after 1.0 ships.

Thanks,
-Aleksey.


From brianfromoregon at gmail.com  Fri Aug 22 16:24:58 2014
From: brianfromoregon at gmail.com (Brian Harris)
Date: Fri, 22 Aug 2014 09:24:58 -0700
Subject: Temp files
In-Reply-To: <53F6EF88.90601@oracle.com>
References: <CAFtUM9ZVELWu5eJd9zym1xtFH5ZQ3kSvULQXT_f1xV4DhaH4=g@mail.gmail.com>
	<53F6EF88.90601@oracle.com>
Message-ID: <CAFtUM9Y3NoEHVj-pNcAiDRmYLVoWVmnK-Zn0T4Cr_ha-Xtfadg@mail.gmail.com>

Yep, that works for me. Random thought: I guess this would be one thing on
the heap of issues with running jmh on android.



On Fri, Aug 22, 2014 at 12:21 AM, Aleksey Shipilev <
aleksey.shipilev at oracle.com> wrote:

> Hi Brian,
>
> On 08/21/2014 09:06 PM, Brian Harris wrote:
> > Hi, wanted to report the inability to run jmh when user doesnt have write
> > access to tmp dir. Must you create a temp file on the real fs? Maybe
> there
> > could be a switch to change the dir used.
>
> Yes, JMH expects a writable temp dir, since we are communicating many
> things to/from the forked VM via the files. Moving out to completely
> read-only execution will introduce quite a mess in the harness
> infrastructure. The use case of non-writable tmp dir is minuscule in
> proportion, as far as I can see. Therefore, we wouldn't reconsider in
> nearest future.
>
> As Staffan said, -Djava.io.tmpdir is your friend here :)
>
> -Aleksey.
>
>

From brianfromoregon at gmail.com  Fri Aug 22 16:36:12 2014
From: brianfromoregon at gmail.com (Brian Harris)
Date: Fri, 22 Aug 2014 09:36:12 -0700
Subject: Gotchas with avoiding benchmarks.jar
Message-ID: <CAFtUM9ZBEiY6MNeHhrJh0QFsaX6pK+cMTTTXx3jqZBaDrm+obQ@mail.gmail.com>

I have jmh integrated with our build environment at work now. The template
benchmark is this

public class HelloJmhTest {
  @Benchmark public void wellHelloThere() {}

  @Test public void run() throws RunnerException {
    Options opt = new ...
    new Runner(opt).run();
  }
}

That is, same as your samples except replacing psvm with junit @Test. Rest
assured, these jmh "tests" are segregated from the real tests and they are
run on a dedicated Jenkins slave which is otherwise completely quiet. I
just use junit as the runner hook because it's so damn convenient.

I'm wondering what are the gotchas with not using the benchmarks.jar setup
that you recommend. The classpath will be huge, will that be a problem?

Thanks,
Brian

From tom.deneau at amd.com  Fri Aug 22 17:16:26 2014
From: tom.deneau at amd.com (Deneau, Tom)
Date: Fri, 22 Aug 2014 17:16:26 +0000
Subject: specializing jvmArgs in a derived class
Message-ID: <BC97738F8E7C8742BABED7F06FB9DF915549CC6B@SATLEXDAG01.amd.com>

I have a base class with
    @Benchmark 
    public void myBenchmark()

and several derived classes that specialize that benchmark.
In one of the derived classes, I would like to specify that when that derived class forks, certain jvmArgs get added to the jvm command line.  Is there a way to do this?

When I tried this in the derived class

    @Override
    @Benchmark
    @Fork(jvmArgs="...") 
    public void myBenchmark()

I got this error which I don't really understand:
    Internal error: multiple methods per @Group, but not all methods have @Group
    
If I remove the @Benchmark annotation from the derived myBenchmark, that error goes away but the jvmArgs seems to also get ignored.

-- Tom

From aleksey.shipilev at oracle.com  Fri Aug 22 19:53:10 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Fri, 22 Aug 2014 23:53:10 +0400
Subject: specializing jvmArgs in a derived class
In-Reply-To: <BC97738F8E7C8742BABED7F06FB9DF915549CC6B@SATLEXDAG01.amd.com>
References: <BC97738F8E7C8742BABED7F06FB9DF915549CC6B@SATLEXDAG01.amd.com>
Message-ID: <53F79FA6.5050208@oracle.com>

Hi Tom,

On 08/22/2014 09:16 PM, Deneau, Tom wrote:
> I have a base class with
>     @Benchmark 
>     public void myBenchmark()
> 
> and several derived classes that specialize that benchmark.
> In one of the derived classes, I would like to specify that when that derived class forks, certain jvmArgs get added to the jvm command line.  Is there a way to do this?
> 
> When I tried this in the derived class
> 
>     @Override
>     @Benchmark
>     @Fork(jvmArgs="...") 
>     public void myBenchmark()
> 
> I got this error which I don't really understand:
>     Internal error: multiple methods per @Group, but not all methods have @Group

This is because JMH treats super-class @Benchmark as the benchmark
sibling on par with the sub-class @Benchmark, ignoring the fact the
super-class is overridden by Java rules. This should be fixed to at
least print a sane error message.

In your case, I would rather make the non- at Benchmark method in the
super-class, and override it with @Benchmark in each subclass. Do you
really need the @Benchmark on the super-class method in your scenario?

-Aleksey.



From tom.deneau at amd.com  Fri Aug 22 20:05:46 2014
From: tom.deneau at amd.com (Deneau, Tom)
Date: Fri, 22 Aug 2014 20:05:46 +0000
Subject: specializing jvmArgs in a derived class
In-Reply-To: <53F79FA6.5050208@oracle.com>
References: <BC97738F8E7C8742BABED7F06FB9DF915549CC6B@SATLEXDAG01.amd.com>
	<53F79FA6.5050208@oracle.com>
Message-ID: <BC97738F8E7C8742BABED7F06FB9DF915549CDD6@SATLEXDAG01.amd.com>

Aleksey --

I see your point about making @Benchmark be in the derived classes.
I only did it the other way because 90% of the time, it doesn't need to be overridden with anything.
But perhaps some restructuring would be better...

-- Tom

-----Original Message-----
From: Aleksey Shipilev [mailto:aleksey.shipilev at oracle.com] 
Sent: Friday, August 22, 2014 2:53 PM
To: Deneau, Tom; jmh-dev at openjdk.java.net
Subject: Re: specializing jvmArgs in a derived class

Hi Tom,

On 08/22/2014 09:16 PM, Deneau, Tom wrote:
> I have a base class with
>     @Benchmark 
>     public void myBenchmark()
> 
> and several derived classes that specialize that benchmark.
> In one of the derived classes, I would like to specify that when that derived class forks, certain jvmArgs get added to the jvm command line.  Is there a way to do this?
> 
> When I tried this in the derived class
> 
>     @Override
>     @Benchmark
>     @Fork(jvmArgs="...") 
>     public void myBenchmark()
> 
> I got this error which I don't really understand:
>     Internal error: multiple methods per @Group, but not all methods 
> have @Group

This is because JMH treats super-class @Benchmark as the benchmark sibling on par with the sub-class @Benchmark, ignoring the fact the super-class is overridden by Java rules. This should be fixed to at least print a sane error message.

In your case, I would rather make the non- at Benchmark method in the super-class, and override it with @Benchmark in each subclass. Do you really need the @Benchmark on the super-class method in your scenario?

-Aleksey.



From aleksey.shipilev at oracle.com  Fri Aug 22 20:12:09 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Sat, 23 Aug 2014 00:12:09 +0400
Subject: specializing jvmArgs in a derived class
In-Reply-To: <BC97738F8E7C8742BABED7F06FB9DF915549CDD6@SATLEXDAG01.amd.com>
References: <BC97738F8E7C8742BABED7F06FB9DF915549CC6B@SATLEXDAG01.amd.com>
	<53F79FA6.5050208@oracle.com>
	<BC97738F8E7C8742BABED7F06FB9DF915549CDD6@SATLEXDAG01.amd.com>
Message-ID: <53F7A419.7000502@oracle.com>

On 08/23/2014 12:05 AM, Deneau, Tom wrote:
> I see your point about making @Benchmark be in the derived classes.
> I only did it the other way because 90% of the time, it doesn't need to be overridden with anything.
> But perhaps some restructuring would be better...

Understood. Well, since the JMH parameters are resolved through the
inheritance hierarchy [1], it might be fruitful to just introduce the
intermediary class, a la:

 @Fork(jvmArgs = ...)
 class MyCustomSharedSettings {
 }

 class OverridenSettingsBenchmark extends MyCustomSharedSettings {
    @Benchmark
    public void bench() {
    }
 }

 class JustABenchmark {
    @Benchmark
    public void bench() {
    }
 }

Thanks,
-Aleksey.

[1]
http://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_24_Inheritance.java


From aleksey.shipilev at oracle.com  Fri Aug 22 21:42:29 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Sat, 23 Aug 2014 01:42:29 +0400
Subject: JMH 0.9.8
Message-ID: <53F7B945.6090702@oracle.com>

Hi,

JMH 0.9.8 is released and available in Maven Central (thanks to
Evgeny, as usual). JMH 0.9.9 is our 4th release candidate for 1.0, and
we are on the straight line to 1.0 release, barring critical issues.
Please test!

Highlights of JMH 0.9.8:

 * Re-enabled and re-tested pattern matching in runners. Runners now
also support non-anchored regular expressions, e.g. you can call
JMHSample_01 instead of ".*JMHSample_01.*". Providing the explicit
anchors ^ and $ will get you the exact match, if you want. (Thanks Bernd
Eckenfels for the patch).

 * System.out is not closed after Runner.run() now.

 * Assorted readability and formatting fixes.

Enjoy!

Thanks,
-Aleksey


From aleksey.shipilev at oracle.com  Fri Aug 22 21:46:06 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Sat, 23 Aug 2014 01:46:06 +0400
Subject: JMH 0.9.8
In-Reply-To: <53F7B945.6090702@oracle.com>
References: <53F7B945.6090702@oracle.com>
Message-ID: <53F7BA1E.7060304@oracle.com>

On 08/23/2014 01:42 AM, Aleksey Shipilev wrote:
> Highlights of JMH 0.9.8:

...and one more thing:

 * All Serializable classes are now aided with SUIDs, which lets forked
VM to withstand instrumentation via javaagents and profilers.

-Aleksey.



From aleksey.shipilev at oracle.com  Fri Aug 22 21:54:13 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Fri, 22 Aug 2014 21:54:13 +0000
Subject: hg: code-tools/jmh: 3 new changesets
Message-ID: <201408222154.s7MLsDPh024783@aojmv0008>

Changeset: 9a19d4380a95
Author:    shade
Date:      2014-08-22 17:31 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/9a19d4380a95

JMH v0.9.8.

! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml
! jmh-archetypes/pom.xml
! jmh-core-benchmarks/pom.xml
! jmh-core-ct/pom.xml
! jmh-core-it/pom.xml
! jmh-core/pom.xml
! jmh-generator-annprocess/pom.xml
! jmh-generator-asm/pom.xml
! jmh-generator-bytecode/pom.xml
! jmh-generator-reflection/pom.xml
! jmh-samples/pom.xml
! pom.xml

Changeset: 82464ba31f1d
Author:    shade
Date:      2014-08-22 17:31 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/82464ba31f1d

Added tag 0.9.8 for changeset 9a19d4380a95

! .hgtags

Changeset: fea2e0888ac5
Author:    shade
Date:      2014-08-22 17:32 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/fea2e0888ac5

Continue 1.0-SNAPSHOT.

! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml
! jmh-archetypes/pom.xml
! jmh-core-benchmarks/pom.xml
! jmh-core-ct/pom.xml
! jmh-core-it/pom.xml
! jmh-core/pom.xml
! jmh-generator-annprocess/pom.xml
! jmh-generator-asm/pom.xml
! jmh-generator-bytecode/pom.xml
! jmh-generator-reflection/pom.xml
! jmh-samples/pom.xml
! pom.xml


From volkan.yazici at gmail.com  Sat Aug 23 12:05:14 2014
From: volkan.yazici at gmail.com (Volkan YAZICI)
Date: Sat, 23 Aug 2014 15:05:14 +0300
Subject: Reporting Internal Functions
Message-ID: <CAP7pH7s7YW=+fiGb=ZqDeBDRE+K2OH0hSgfNdXhxUv8AJUFmOA@mail.gmail.com>

Hi,

I have a function as follows.

    @Benchmark
    void f() {
        g(h());
    }

Put another way, f() is an entry point for g() and h(). Is it possible to
make JMH to measure the performance of g() and h() individually?

Best.

From ecki at zusammenkunft.net  Sat Aug 23 14:21:22 2014
From: ecki at zusammenkunft.net (Bernd)
Date: Sat, 23 Aug 2014 16:21:22 +0200
Subject: Reporting Internal Functions
In-Reply-To: <CAP7pH7s7YW=+fiGb=ZqDeBDRE+K2OH0hSgfNdXhxUv8AJUFmOA@mail.gmail.com>
References: <CAP7pH7s7YW=+fiGb=ZqDeBDRE+K2OH0hSgfNdXhxUv8AJUFmOA@mail.gmail.com>
Message-ID: <CABOR3+yNGQAj_Q4D9TFE3hjPgzRayx6sCXFMRMfsYDihjzVeLQ@mail.gmail.com>

Hello,

I guess the obvious thing is to put a @Benchmark on g() and h(). ;)

But I think what you are asking for is one of the profilers. I suspect they
will either be inaccurate or affect your result, so it is better to try to
measure the code segments you care about seperate (but this has its own
issues as they might get differently optimized and inlined).

Gruss
Bernd
Am 23.08.2014 14:15 schrieb "Volkan YAZICI" <volkan.yazici at gmail.com>:

> Hi,
>
> I have a function as follows.
>
>     @Benchmark
>     void f() {
>         g(h());
>     }
>
> Put another way, f() is an entry point for g() and h(). Is it possible to
> make JMH to measure the performance of g() and h() individually?
>
> Best.
>

From brianfromoregon at gmail.com  Mon Aug 25 06:25:02 2014
From: brianfromoregon at gmail.com (Brian Harris)
Date: Sun, 24 Aug 2014 23:25:02 -0700
Subject: Jenkins plugin
Message-ID: <CAFtUM9ZuF-Yy0iaVDCN4kOLi2TgS0MORdf4yC14Tif6an1O2+g@mail.gmail.com>

Hi,

Just sharing what I'm up to. A simple plugin to collect jmh results in the
workspace and publish an index of them across builds. I plan to run jmh in
Jenkins on a dedicated slave, then have d3.js consume the index file
published by the plugin and give some basic trending charts.

https://github.com/brianfromoregon/jmh-plugin

Didn't see a logo, so used Aleksey's mug!

Brian

From aleksey.shipilev at oracle.com  Mon Aug 25 08:14:15 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Mon, 25 Aug 2014 12:14:15 +0400
Subject: Gotchas with avoiding benchmarks.jar
In-Reply-To: <CAFtUM9ZBEiY6MNeHhrJh0QFsaX6pK+cMTTTXx3jqZBaDrm+obQ@mail.gmail.com>
References: <CAFtUM9ZBEiY6MNeHhrJh0QFsaX6pK+cMTTTXx3jqZBaDrm+obQ@mail.gmail.com>
Message-ID: <53FAF057.60705@oracle.com>

Hi Brian,

On 08/22/2014 08:36 PM, Brian Harris wrote:
> I have jmh integrated with our build environment at work now. The template
> benchmark is this
> 
> public class HelloJmhTest {
>   @Benchmark public void wellHelloThere() {}
> 
>   @Test public void run() throws RunnerException {
>     Options opt = new ...
>     new Runner(opt).run();
>   }
> }
> 
> That is, same as your samples except replacing psvm with junit @Test. Rest
> assured, these jmh "tests" are segregated from the real tests and they are
> run on a dedicated Jenkins slave which is otherwise completely quiet. I
> just use junit as the runner hook because it's so damn convenient.

Yes, our jmh-core-it (core integration tests) are built in the same
manner -- but we care about functionality there, not performance.

> I'm wondering what are the gotchas with not using the benchmarks.jar setup
> that you recommend. The classpath will be huge, will that be a problem?

There are multiple things one should consider while breaking from the
sweet cradle of self-contained prebuilt JARs:

 a) Isolation. When you have an uberjar, you are most probably running
it from the separate JVM, which you explicitly control w.r.t. what
executable you are using, what command line options you are passing, and
basically where and when the benchmarks are run. In the "exploded"
configuration you will have to accurately replicate all the pieces that
are needed to run the benchmark, possibly adding more than actually
required. In other words, if you are calling Runner from @Test, then
Runner and forked VM will inherit all the JVM options that were
(accidentally) supplied to @Test, possibly screwing your results.

 b) Exclusivity. It is very convenient for, say, JUnit @Tests to run
concurrently, because most well-behaved functional tests tolerate the
external concurrency. Benchmarks do not tolerate external contenders. It
is arguably easy to accidentally run multiple @Tests that will
concurrently call into multiple JMH runners, and they will interfere
with each others' scores. I know e.g. JMH IDEA plugin stomps hard on
running multiple JMH sessions at once, can you guarantee the same with
@Test-s?

 c) Consistency. Uberjar also packs the essential metadata within it.
First, the benchmark list that enumerates all the benchmarks in you are
able to run along with their default options. Failure to locate the
benchmark list is easily detectable (even though people tend to get
completely catatonic when they face "No benchmarks to run" when their
build configuration misbehaves). The detectability is *not* the case for
compiler hints, which you may accidentally omit from the resource
location, and then things break silently on performance front (e.g.
@CompilerControl annotations suddenly stop working, the forced inlining
of JMH infrastructure methods is not working... silently, etc.)

TL;DR: Running without uberjar is probably OK, but truly paranoid people
should prefer the uberjar to avoid the avoidable accidents.

Thanks,
-Aleksey.


From aleksey.shipilev at oracle.com  Mon Aug 25 08:16:32 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Mon, 25 Aug 2014 12:16:32 +0400
Subject: Reporting Internal Functions
In-Reply-To: <CABOR3+yNGQAj_Q4D9TFE3hjPgzRayx6sCXFMRMfsYDihjzVeLQ@mail.gmail.com>
References: <CAP7pH7s7YW=+fiGb=ZqDeBDRE+K2OH0hSgfNdXhxUv8AJUFmOA@mail.gmail.com>
	<CABOR3+yNGQAj_Q4D9TFE3hjPgzRayx6sCXFMRMfsYDihjzVeLQ@mail.gmail.com>
Message-ID: <53FAF0E0.9040402@oracle.com>

Hi Volkan,

What Bernd said.

JMH is not a magic wand which can introspect your code, you have to
explicitly mark what you want to measure with @Benchmark. If you are
looking for a way to see where the time is spent within the @Benchmark
call, use either bundled JMH profilers (-prof stack, -prof perfasm), or
external profilers (VisualVM, JMC, etc.).

Thanks,
-Aleksey.

On 08/23/2014 06:21 PM, Bernd wrote:
> Hello,
> 
> I guess the obvious thing is to put a @Benchmark on g() and h(). ;)
> 
> But I think what you are asking for is one of the profilers. I suspect they
> will either be inaccurate or affect your result, so it is better to try to
> measure the code segments you care about seperate (but this has its own
> issues as they might get differently optimized and inlined).
> 
> Gruss
> Bernd
> Am 23.08.2014 14:15 schrieb "Volkan YAZICI" <volkan.yazici at gmail.com>:
> 
>> Hi,
>>
>> I have a function as follows.
>>
>>     @Benchmark
>>     void f() {
>>         g(h());
>>     }
>>
>> Put another way, f() is an entry point for g() and h(). Is it possible to
>> make JMH to measure the performance of g() and h() individually?
>>
>> Best.
>>



From aleksey.shipilev at oracle.com  Mon Aug 25 08:36:58 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Mon, 25 Aug 2014 12:36:58 +0400
Subject: Jenkins plugin
In-Reply-To: <CAFtUM9ZuF-Yy0iaVDCN4kOLi2TgS0MORdf4yC14Tif6an1O2+g@mail.gmail.com>
References: <CAFtUM9ZuF-Yy0iaVDCN4kOLi2TgS0MORdf4yC14Tif6an1O2+g@mail.gmail.com>
Message-ID: <53FAF5AA.9090601@oracle.com>

Hi again,

On 08/25/2014 10:25 AM, Brian Harris wrote:
> Just sharing what I'm up to. A simple plugin to collect jmh results in the
> workspace and publish an index of them across builds. I plan to run jmh in
> Jenkins on a dedicated slave, then have d3.js consume the index file
> published by the plugin and give some basic trending charts.

Good, I see you are capturing JSON output, which is the intended use
case for JSON dumpers. I think the project lacks the essential info how
to produce the JSON output to be captured by this plugin?

Recorded a link to this project on JMH homepage:
  http://openjdk.java.net/projects/code-tools/jmh/

-Aleksey.



From aleksey.shipilev at oracle.com  Mon Aug 25 11:19:34 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Mon, 25 Aug 2014 11:19:34 +0000
Subject: hg: code-tools/jmh: runners: omit error from simple result toString, 
	if the error is undefined.
Message-ID: <201408251119.s7PBJZn1017097@aojmv0008>

Changeset: 2756664a5c82
Author:    shade
Date:      2014-08-25 15:19 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/2756664a5c82

runners: omit error from simple result toString, if the error is undefined.

! jmh-core/src/main/java/org/openjdk/jmh/results/Result.java


From aleksey.shipilev at oracle.com  Mon Aug 25 11:59:46 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Mon, 25 Aug 2014 11:59:46 +0000
Subject: hg: code-tools/jmh: profilers: perfasm,
	able to switch the classload trace off, if that becomes a bottleneck.
Message-ID: <201408251159.s7PBxl9Z023493@aojmv0008>

Changeset: f3a68bbd0f15
Author:    shade
Date:      2014-08-25 15:59 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/f3a68bbd0f15

profilers: perfasm, able to switch the classload trace off, if that becomes a bottleneck.

! jmh-core/src/main/java/org/openjdk/jmh/profile/LinuxPerfAsmProfiler.java


From brianfromoregon at gmail.com  Mon Aug 25 18:09:10 2014
From: brianfromoregon at gmail.com (Brian Harris)
Date: Mon, 25 Aug 2014 11:09:10 -0700
Subject: Jenkins plugin
In-Reply-To: <53FAF5AA.9090601@oracle.com>
References: <CAFtUM9ZuF-Yy0iaVDCN4kOLi2TgS0MORdf4yC14Tif6an1O2+g@mail.gmail.com>
	<53FAF5AA.9090601@oracle.com>
Message-ID: <CAFtUM9b1TMX2mYoussXGf7Oz1Jhu0+nm-jzPuOhrJfUJqV1cMA@mail.gmail.com>

Sure, added an example in readme.


On Mon, Aug 25, 2014 at 1:36 AM, Aleksey Shipilev <
aleksey.shipilev at oracle.com> wrote:

> Hi again,
>
> On 08/25/2014 10:25 AM, Brian Harris wrote:
> > Just sharing what I'm up to. A simple plugin to collect jmh results in
> the
> > workspace and publish an index of them across builds. I plan to run jmh
> in
> > Jenkins on a dedicated slave, then have d3.js consume the index file
> > published by the plugin and give some basic trending charts.
>
> Good, I see you are capturing JSON output, which is the intended use
> case for JSON dumpers. I think the project lacks the essential info how
> to produce the JSON output to be captured by this plugin?
>
> Recorded a link to this project on JMH homepage:
>   http://openjdk.java.net/projects/code-tools/jmh/
>
> -Aleksey.
>
>
>

From brianfromoregon at gmail.com  Mon Aug 25 18:08:51 2014
From: brianfromoregon at gmail.com (Brian Harris)
Date: Mon, 25 Aug 2014 11:08:51 -0700
Subject: Gotchas with avoiding benchmarks.jar
In-Reply-To: <53FAF057.60705@oracle.com>
References: <CAFtUM9ZBEiY6MNeHhrJh0QFsaX6pK+cMTTTXx3jqZBaDrm+obQ@mail.gmail.com>
	<53FAF057.60705@oracle.com>
Message-ID: <CAFtUM9aKNAU6nmg1pDpE-gfgyh9xh4cY+PEWtSYuj4j3V5VMuw@mail.gmail.com>

Thanks for the response!

a) looking at handling of jvm args in Runner the safe thing for me is to
always explicitly set OptionsBuilder#jvmArgs (perhaps to an empty set,
even) so that the host JVM args aren't automatically used
.
b) junit allows concurrent execution with custom @RunWith but by default
it's always single threaded, so good point tho out of the box behavior is
safe. to be extra sure, Runner (or our wrapper) could acquire a static lock

c) if jmh always generates these two files, Runner could assert they are
present to prevent this case.


On Mon, Aug 25, 2014 at 1:14 AM, Aleksey Shipilev <
aleksey.shipilev at oracle.com> wrote:

> Hi Brian,
>
> On 08/22/2014 08:36 PM, Brian Harris wrote:
> > I have jmh integrated with our build environment at work now. The
> template
> > benchmark is this
> >
> > public class HelloJmhTest {
> >   @Benchmark public void wellHelloThere() {}
> >
> >   @Test public void run() throws RunnerException {
> >     Options opt = new ...
> >     new Runner(opt).run();
> >   }
> > }
> >
> > That is, same as your samples except replacing psvm with junit @Test.
> Rest
> > assured, these jmh "tests" are segregated from the real tests and they
> are
> > run on a dedicated Jenkins slave which is otherwise completely quiet. I
> > just use junit as the runner hook because it's so damn convenient.
>
> Yes, our jmh-core-it (core integration tests) are built in the same
> manner -- but we care about functionality there, not performance.
>
> > I'm wondering what are the gotchas with not using the benchmarks.jar
> setup
> > that you recommend. The classpath will be huge, will that be a problem?
>
> There are multiple things one should consider while breaking from the
> sweet cradle of self-contained prebuilt JARs:
>
>  a) Isolation. When you have an uberjar, you are most probably running
> it from the separate JVM, which you explicitly control w.r.t. what
> executable you are using, what command line options you are passing, and
> basically where and when the benchmarks are run. In the "exploded"
> configuration you will have to accurately replicate all the pieces that
> are needed to run the benchmark, possibly adding more than actually
> required. In other words, if you are calling Runner from @Test, then
> Runner and forked VM will inherit all the JVM options that were
> (accidentally) supplied to @Test, possibly screwing your results.
>
>  b) Exclusivity. It is very convenient for, say, JUnit @Tests to run
> concurrently, because most well-behaved functional tests tolerate the
> external concurrency. Benchmarks do not tolerate external contenders. It
> is arguably easy to accidentally run multiple @Tests that will
> concurrently call into multiple JMH runners, and they will interfere
> with each others' scores. I know e.g. JMH IDEA plugin stomps hard on
> running multiple JMH sessions at once, can you guarantee the same with
> @Test-s?
>
>  c) Consistency. Uberjar also packs the essential metadata within it.
> First, the benchmark list that enumerates all the benchmarks in you are
> able to run along with their default options. Failure to locate the
> benchmark list is easily detectable (even though people tend to get
> completely catatonic when they face "No benchmarks to run" when their
> build configuration misbehaves). The detectability is *not* the case for
> compiler hints, which you may accidentally omit from the resource
> location, and then things break silently on performance front (e.g.
> @CompilerControl annotations suddenly stop working, the forced inlining
> of JMH infrastructure methods is not working... silently, etc.)
>
> TL;DR: Running without uberjar is probably OK, but truly paranoid people
> should prefer the uberjar to avoid the avoidable accidents.
>
> Thanks,
> -Aleksey.
>
>

From aleksey.shipilev at oracle.com  Wed Aug 27 10:43:23 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Wed, 27 Aug 2014 14:43:23 +0400
Subject: When an exception kills a thread in a Group
In-Reply-To: <53AAC9AE.5030507@oracle.com>
References: <F42FA2C0-5391-407A-B7CD-E28DC95E3008@gmail.com>
	<53AA8846.70704@oracle.com>
	<E223C774-B136-4F80-A12F-33505421544B@gmail.com>
	<53AAC9AE.5030507@oracle.com>
Message-ID: <53FDB64B.4040909@oracle.com>

On 06/25/2014 05:07 PM, Aleksey Shipilev wrote:
> Thanks, my original hunch was almost correct. The exceptional thread
> indeed exits, and the second thread is stuck at StampedLock acquire. The
> *control* thread which reads the results from the benchmark threads
> (through the futures), waits for synchronize iteration epilog to finish,
> which will never finish since second thread is stuck.

This is recorded as
 https://bugs.openjdk.java.net/browse/CODETOOLS-7901008

We will try to take care of it after 1.0.

Thanks,
-Aleksey.


From aleksey.shipilev at oracle.com  Wed Aug 27 11:13:16 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Wed, 27 Aug 2014 15:13:16 +0400
Subject: specializing jvmArgs in a derived class
In-Reply-To: <53F7A419.7000502@oracle.com>
References: <BC97738F8E7C8742BABED7F06FB9DF915549CC6B@SATLEXDAG01.amd.com>	<53F79FA6.5050208@oracle.com>	<BC97738F8E7C8742BABED7F06FB9DF915549CDD6@SATLEXDAG01.amd.com>
	<53F7A419.7000502@oracle.com>
Message-ID: <53FDBD4C.3060700@oracle.com>

On 08/23/2014 12:12 AM, Aleksey Shipilev wrote:
> On 08/23/2014 12:05 AM, Deneau, Tom wrote:
>> I see your point about making @Benchmark be in the derived classes.
>> I only did it the other way because 90% of the time, it doesn't need to be overridden with anything.
>> But perhaps some restructuring would be better...
> 
> Understood. Well, since the JMH parameters are resolved through the
> inheritance hierarchy [1], it might be fruitful to just introduce the
> intermediary class, a la:

I haven't seen the reply if that method works.

Anyhow, I recorded the original issue in:
 https://bugs.openjdk.java.net/browse/CODETOOLS-7901009

Thanks,
-Aleksey.



From aleksey.shipilev at oracle.com  Wed Aug 27 12:13:11 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Wed, 27 Aug 2014 16:13:11 +0400
Subject: Exception in Benchmark isn't marked as failure in CSV file.
In-Reply-To: <53F7158C.1080808@oracle.com>
References: <CADOP4RbWiwfMJAUWi8+UoM793AOOxRwu7WFzqha+aSukWfa2tA@mail.gmail.com>
	<53F7158C.1080808@oracle.com>
Message-ID: <53FDCB57.3070009@oracle.com>

On 08/22/2014 02:03 PM, Aleksey Shipilev wrote:
> I have looked into printing some human-readable message in the cases
> like this, and concluded the change is too disruptive for current
> release candidate. It also adds to the bucket of fixes we have to have
> for treating the failing benchmarks well. Since failing benchmarks are
> not the things we want to measure anyway, we will revisit this behavior
> after 1.0 ships.

Recorded as:
 https://bugs.openjdk.java.net/browse/CODETOOLS-7901010

Thanks,
-Aleksey.


From aleksey.shipilev at oracle.com  Wed Aug 27 12:24:39 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Wed, 27 Aug 2014 16:24:39 +0400
Subject: Setup and TearDown executed by different threads
In-Reply-To: <53EE1224.10507@oracle.com>
References: <1407927960.94775.YahooMailNeo@web162204.mail.bf1.yahoo.com>	<53EB4B25.6090708@oracle.com>	<DF545D37-5FD8-4492-85AB-2E5A3DAEBC42@yahoo.com>	<53EB50A2.8020401@oracle.com>	<1407933620.83089.YahooMailNeo@web162201.mail.bf1.yahoo.com>
	<53EB7C20.70807@oracle.com> <53EE1224.10507@oracle.com>
Message-ID: <53FDCE07.6010800@oracle.com>

On 08/15/2014 05:59 PM, Aleksey Shipilev wrote:
> Not that isn't not doable, but the change seems too intrusive to do in a
> Release Candidate. It also intersects with Chris Vest's example of
> benchmark being stuck if one of the threads in asymmetric benchmarks
> throws the exception. Let us consider this for post 1.0 revisions.

So recorded:
  https://bugs.openjdk.java.net/browse/CODETOOLS-7901011

-Aleksey.



From aleksey.shipilev at oracle.com  Wed Aug 27 12:30:55 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Wed, 27 Aug 2014 16:30:55 +0400
Subject: Multiple JVMs, different numbers of threads and BenchmarkMode
In-Reply-To: <53DE6131.70208@oracle.com>
References: <D32424E3-5F8C-42E4-9B13-36FFFC7E8C8C@yahoo.com>
	<53DE6131.70208@oracle.com>
Message-ID: <53FDCF7F.3040203@oracle.com>

On 08/03/2014 08:20 PM, Aleksey Shipilev wrote:
> On 08/01/2014 02:29 PM, Dmitry Vyazelenko wrote:
>> Maybe it would be a good idea to allow specifying array of batchSize
>> values via command line or annotations. At least this will eliminate
>> scripting part but will require adding this information to the
>> results table (e.g. batchSize 10 value1 batchSize 100 value2 etc.).
> 
> See above. Needs a proper connection with @Param-s.

So recorded:
  https://bugs.openjdk.java.net/browse/CODETOOLS-7901012

-Aleksey.


From aleksey.shipilev at oracle.com  Wed Aug 27 12:49:10 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Wed, 27 Aug 2014 16:49:10 +0400
Subject: Gotchas with avoiding benchmarks.jar
In-Reply-To: <CAFtUM9aKNAU6nmg1pDpE-gfgyh9xh4cY+PEWtSYuj4j3V5VMuw@mail.gmail.com>
References: <CAFtUM9ZBEiY6MNeHhrJh0QFsaX6pK+cMTTTXx3jqZBaDrm+obQ@mail.gmail.com>
	<53FAF057.60705@oracle.com>
	<CAFtUM9aKNAU6nmg1pDpE-gfgyh9xh4cY+PEWtSYuj4j3V5VMuw@mail.gmail.com>
Message-ID: <53FDD3C6.2020200@oracle.com>

On 08/25/2014 10:08 PM, Brian Harris wrote:
> a) looking at handling of jvm args in Runner the safe thing for me is to
> always explicitly set OptionsBuilder#jvmArgs (perhaps to an empty set,
> even) so that the host JVM args aren't automatically used

Yes, that works.

> b) junit allows concurrent execution with custom @RunWith but by default
> it's always single threaded, so good point tho out of the box behavior
> is safe. to be extra sure, Runner (or our wrapper) could acquire a
> static lock

Static lock does not guard from multiple concurrent JVMs, and many
@JUnit test runners I know of are "isolating" the concurrently running
tests by forking the VMs.

> c) if jmh always generates these two files, Runner could assert they are
> present to prevent this case.

Oh yes, we need to assert these things.

-Aleksey.


From aleksey.shipilev at oracle.com  Wed Aug 27 14:36:58 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Wed, 27 Aug 2014 14:36:58 +0000
Subject: hg: code-tools/jmh: runners: assert the existence of critical
	resources.
Message-ID: <201408271436.s7REawfa012120@aojmv0008>

Changeset: 7dd59a0f6d71
Author:    shade
Date:      2014-08-27 18:36 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/7dd59a0f6d71

runners: assert the existence of critical resources.

! jmh-core/src/main/java/org/openjdk/jmh/runner/AbstractResourceReader.java
+ jmh-core/src/test/resources/META-INF/CompilerHints


From aleksey.shipilev at oracle.com  Wed Aug 27 14:37:25 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Wed, 27 Aug 2014 18:37:25 +0400
Subject: Gotchas with avoiding benchmarks.jar
In-Reply-To: <53FDD3C6.2020200@oracle.com>
References: <CAFtUM9ZBEiY6MNeHhrJh0QFsaX6pK+cMTTTXx3jqZBaDrm+obQ@mail.gmail.com>	<53FAF057.60705@oracle.com>	<CAFtUM9aKNAU6nmg1pDpE-gfgyh9xh4cY+PEWtSYuj4j3V5VMuw@mail.gmail.com>
	<53FDD3C6.2020200@oracle.com>
Message-ID: <53FDED25.9000409@oracle.com>

On 08/27/2014 04:49 PM, Aleksey Shipilev wrote:
> On 08/25/2014 10:08 PM, Brian Harris wrote:
>> c) if jmh always generates these two files, Runner could assert they are
>> present to prevent this case.
> 
> Oh yes, we need to assert these things.

Surprisingly easy to fix:
  http://hg.openjdk.java.net/code-tools/jmh/rev/7dd59a0f6d71

-Aleksey.



From brianfromoregon at gmail.com  Wed Aug 27 16:16:32 2014
From: brianfromoregon at gmail.com (Brian Harris)
Date: Wed, 27 Aug 2014 09:16:32 -0700
Subject: Gotchas with avoiding benchmarks.jar
In-Reply-To: <53FDD3C6.2020200@oracle.com>
References: <CAFtUM9ZBEiY6MNeHhrJh0QFsaX6pK+cMTTTXx3jqZBaDrm+obQ@mail.gmail.com>
	<53FAF057.60705@oracle.com>
	<CAFtUM9aKNAU6nmg1pDpE-gfgyh9xh4cY+PEWtSYuj4j3V5VMuw@mail.gmail.com>
	<53FDD3C6.2020200@oracle.com>
Message-ID: <CAFtUM9ZMbmAxK-bH80tvev5c4U=TjUmNV_vjcFykA1S_KRHsog@mail.gmail.com>

If you like the idea of jmh detecting and aborting concurrent execution,
consider introducing a file system lock such as this
https://gist.github.com/brianfromoregon/707ce3a94e45c933f126

On Wed, Aug 27, 2014 at 5:49 AM, Aleksey Shipilev <
aleksey.shipilev at oracle.com> wrote:

> On 08/25/2014 10:08 PM, Brian Harris wrote:
> > a) looking at handling of jvm args in Runner the safe thing for me is to
> > always explicitly set OptionsBuilder#jvmArgs (perhaps to an empty set,
> > even) so that the host JVM args aren't automatically used
>
> Yes, that works.
>
> > b) junit allows concurrent execution with custom @RunWith but by default
> > it's always single threaded, so good point tho out of the box behavior
> > is safe. to be extra sure, Runner (or our wrapper) could acquire a
> > static lock
>
> Static lock does not guard from multiple concurrent JVMs, and many
> @JUnit test runners I know of are "isolating" the concurrently running
> tests by forking the VMs.
>
> > c) if jmh always generates these two files, Runner could assert they are
> > present to prevent this case.
>
> Oh yes, we need to assert these things.
>
> -Aleksey.
>
>

From aleksey.shipilev at oracle.com  Wed Aug 27 17:40:01 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Wed, 27 Aug 2014 21:40:01 +0400
Subject: Gotchas with avoiding benchmarks.jar
In-Reply-To: <CAFtUM9ZMbmAxK-bH80tvev5c4U=TjUmNV_vjcFykA1S_KRHsog@mail.gmail.com>
References: <CAFtUM9ZBEiY6MNeHhrJh0QFsaX6pK+cMTTTXx3jqZBaDrm+obQ@mail.gmail.com>
	<53FAF057.60705@oracle.com>
	<CAFtUM9aKNAU6nmg1pDpE-gfgyh9xh4cY+PEWtSYuj4j3V5VMuw@mail.gmail.com>
	<53FDD3C6.2020200@oracle.com>
	<CAFtUM9ZMbmAxK-bH80tvev5c4U=TjUmNV_vjcFykA1S_KRHsog@mail.gmail.com>
Message-ID: <53FE17F1.4080208@oracle.com>

So recorded:
 https://bugs.openjdk.java.net/browse/CODETOOLS-7901013

-Aleksey.

On 08/27/2014 08:16 PM, Brian Harris wrote:
> If you like the idea of jmh detecting and aborting concurrent execution,
> consider introducing a file system lock such as
> this https://gist.github.com/brianfromoregon/707ce3a94e45c933f126
> 
> On Wed, Aug 27, 2014 at 5:49 AM, Aleksey Shipilev
> <aleksey.shipilev at oracle.com <mailto:aleksey.shipilev at oracle.com>> wrote:
> 
>     On 08/25/2014 10:08 PM, Brian Harris wrote:
>     > a) looking at handling of jvm args in Runner the safe thing for me
>     is to
>     > always explicitly set OptionsBuilder#jvmArgs (perhaps to an empty set,
>     > even) so that the host JVM args aren't automatically used
> 
>     Yes, that works.
> 
>     > b) junit allows concurrent execution with custom @RunWith but by
>     default
>     > it's always single threaded, so good point tho out of the box behavior
>     > is safe. to be extra sure, Runner (or our wrapper) could acquire a
>     > static lock
> 
>     Static lock does not guard from multiple concurrent JVMs, and many
>     @JUnit test runners I know of are "isolating" the concurrently running
>     tests by forking the VMs.
> 
>     > c) if jmh always generates these two files, Runner could assert
>     they are
>     > present to prevent this case.
> 
>     Oh yes, we need to assert these things.
> 
>     -Aleksey.
> 
> 



From aleksey.shipilev at oracle.com  Thu Aug 28 12:57:32 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Thu, 28 Aug 2014 12:57:32 +0000
Subject: hg: code-tools/jmh: runners: minor formatting quirk,
	"10days" should be "10 days".
Message-ID: <201408281257.s7SCvWcY009374@aojmv0008>

Changeset: c97ad3d4b507
Author:    shade
Date:      2014-08-28 16:50 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/c97ad3d4b507

runners: minor formatting quirk, "10days" should be "10 days".

! jmh-core/src/main/java/org/openjdk/jmh/runner/BaseRunner.java


From aleksey.shipilev at oracle.com  Thu Aug 28 13:42:41 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Thu, 28 Aug 2014 13:42:41 +0000
Subject: hg: code-tools/jmh: runners: print more tailed lines when forked VM
	fails.
Message-ID: <201408281342.s7SDgfH9020134@aojmv0008>

Changeset: 31c75cfd7b06
Author:    shade
Date:      2014-08-28 17:42 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/31c75cfd7b06

runners: print more tailed lines when forked VM fails.

! jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java


From aleksey.shipilev at oracle.com  Fri Aug 29 18:12:33 2014
From: aleksey.shipilev at oracle.com (Aleksey Shipilev)
Date: Fri, 29 Aug 2014 22:12:33 +0400
Subject: JMH 1.0
Message-ID: <5400C291.6010901@oracle.com>

Hi,

On March 28, 2013, we had released the very first public version of JMH.
Today, seventeen months and almost a thousand commits later, we hit the
major milestone in our project: the release of JMH 1.0. This marks our
belief the API and the harness itself is stable, reliable, and
dependable. The artefacts for JMH 1.0 are already available at Maven
Central [1].

We will continue working towards improving the harness and/or APIs, and
we already have quite a few of non-critical issues [2] found by
community to fix and polish. Please continue using JMH and contribute
the issues, fixes, and general suggestions back! You, the community, are
essential to keep the project alive.

We would like to take the rest of this note to credit those involved in
the effort, in any capacity:

 * Anders Astrand, Staffan Friberg, and Henrik Loef, who have developed
the initial version of the harness for internal use back in 2011.

 * Donald Smith, Dalibor Topic, and Cecilia Borg, who have shepherded
the initial contribution of JMH under Codetools umbrella in OpenJDK, and
who are still providing the project management guidance for it.

 * Sergey Kuksenko, who contributed immensely in refactoring the JMH
internals before it became public, as well as providing the internal
reviews and feedback for the JMH features and improvements

 * Evgeny Mandrikov, who consistently helps us with the releases to
Maven Central.

 * Nitsan Wakart, Daniel Mitterdorfer, Julien Ponge, Richard Warburton,
and other authors who provided the solid stories and tutorials on using
JMH, as well as providing the invaluable feedback on user experience.

 * Bernd Eckenfels, Joe Kearney, Aggelos Biboudis, Chris Vest, Dmitry
Vyazelenko, Claes Redestad, Dmitry Chuyko, Julien Nicolaund, Tom Deneau,
Roman Leventov, Henri Tremblay, Ivan Gerasimov, Gilles Duboscq, Clement
Mathieu and many others participating in mailing list discussions, often
culminating in code contributions.

 * Konrad Malawski, Cedric Champeau, Gleb Smirnov, Nikita Artyushov,
Brian Harris and many others contributing to various integrations to
SBT, Gradle, Jenkins, IntelliJ IDEA, etc.

Cheers, and Thank You!

Enjoy the JMH 1.0 :)

-Aleksey.

[1] http://central.maven.org/maven2/org/openjdk/jmh/jmh-core/
[2]
https://bugs.openjdk.java.net/issues/?jql=project%20in%20(CODETOOLS)%20AND%20component%20in%20(tools)%20AND%20Subcomponent%20in%20(jmh)


From aleksey.shipilev at oracle.com  Fri Aug 29 18:23:09 2014
From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com)
Date: Fri, 29 Aug 2014 18:23:09 +0000
Subject: hg: code-tools/jmh: 3 new changesets
Message-ID: <201408291823.s7TIN937023454@aojmv0008>

Changeset: 7257b0afa007
Author:    shade
Date:      2014-08-29 14:50 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/7257b0afa007

JMH v1.0.

! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml
! jmh-archetypes/pom.xml
! jmh-core-benchmarks/pom.xml
! jmh-core-ct/pom.xml
! jmh-core-it/pom.xml
! jmh-core/pom.xml
! jmh-generator-annprocess/pom.xml
! jmh-generator-asm/pom.xml
! jmh-generator-bytecode/pom.xml
! jmh-generator-reflection/pom.xml
! jmh-samples/pom.xml
! pom.xml

Changeset: 0a71e94b355c
Author:    shade
Date:      2014-08-29 14:50 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/0a71e94b355c

Added tag 1.0 for changeset 7257b0afa007

! .hgtags

Changeset: 0c828f0e6001
Author:    shade
Date:      2014-08-29 14:55 +0400
URL:       http://hg.openjdk.java.net/code-tools/jmh/rev/0c828f0e6001

Continue in 2.0-SNAPSHOT.

! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml
! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml
! jmh-archetypes/pom.xml
! jmh-core-benchmarks/pom.xml
! jmh-core-ct/pom.xml
! jmh-core-it/pom.xml
! jmh-core/pom.xml
! jmh-generator-annprocess/pom.xml
! jmh-generator-asm/pom.xml
! jmh-generator-bytecode/pom.xml
! jmh-generator-reflection/pom.xml
! jmh-samples/pom.xml
! pom.xml


From georges.gomes at gmail.com  Fri Aug 29 18:45:00 2014
From: georges.gomes at gmail.com (Georges Gomes)
Date: Fri, 29 Aug 2014 14:45:00 -0400
Subject: JMH 1.0
In-Reply-To: <5400C291.6010901@oracle.com>
References: <5400C291.6010901@oracle.com>
Message-ID: <CAKOqNJ-dff8tHVEdEv0dbxD5HEsMOb0O1et1jCOJiCZ4cg5rZg@mail.gmail.com>

Hi Aleksey
I think the entire community thank you. Your reactivity has made this
project extremely alive. The result is fenomenal IMHO. It everyday helps
our firm in delivering better software and also learning a lot with the
underlying [sympathic] mechanics with the magical perfasm.
Thank you
Georges



On Fri, Aug 29, 2014 at 2:12 PM, Aleksey Shipilev <
aleksey.shipilev at oracle.com> wrote:

> Hi,
>
> On March 28, 2013, we had released the very first public version of JMH.
> Today, seventeen months and almost a thousand commits later, we hit the
> major milestone in our project: the release of JMH 1.0. This marks our
> belief the API and the harness itself is stable, reliable, and
> dependable. The artefacts for JMH 1.0 are already available at Maven
> Central [1].
>
> We will continue working towards improving the harness and/or APIs, and
> we already have quite a few of non-critical issues [2] found by
> community to fix and polish. Please continue using JMH and contribute
> the issues, fixes, and general suggestions back! You, the community, are
> essential to keep the project alive.
>
> We would like to take the rest of this note to credit those involved in
> the effort, in any capacity:
>
>  * Anders Astrand, Staffan Friberg, and Henrik Loef, who have developed
> the initial version of the harness for internal use back in 2011.
>
>  * Donald Smith, Dalibor Topic, and Cecilia Borg, who have shepherded
> the initial contribution of JMH under Codetools umbrella in OpenJDK, and
> who are still providing the project management guidance for it.
>
>  * Sergey Kuksenko, who contributed immensely in refactoring the JMH
> internals before it became public, as well as providing the internal
> reviews and feedback for the JMH features and improvements
>
>  * Evgeny Mandrikov, who consistently helps us with the releases to
> Maven Central.
>
>  * Nitsan Wakart, Daniel Mitterdorfer, Julien Ponge, Richard Warburton,
> and other authors who provided the solid stories and tutorials on using
> JMH, as well as providing the invaluable feedback on user experience.
>
>  * Bernd Eckenfels, Joe Kearney, Aggelos Biboudis, Chris Vest, Dmitry
> Vyazelenko, Claes Redestad, Dmitry Chuyko, Julien Nicolaund, Tom Deneau,
> Roman Leventov, Henri Tremblay, Ivan Gerasimov, Gilles Duboscq, Clement
> Mathieu and many others participating in mailing list discussions, often
> culminating in code contributions.
>
>  * Konrad Malawski, Cedric Champeau, Gleb Smirnov, Nikita Artyushov,
> Brian Harris and many others contributing to various integrations to
> SBT, Gradle, Jenkins, IntelliJ IDEA, etc.
>
> Cheers, and Thank You!
>
> Enjoy the JMH 1.0 :)
>
> -Aleksey.
>
> [1] http://central.maven.org/maven2/org/openjdk/jmh/jmh-core/
> [2]
>
> https://bugs.openjdk.java.net/issues/?jql=project%20in%20(CODETOOLS)%20AND%20component%20in%20(tools)%20AND%20Subcomponent%20in%20(jmh)
>
>

From vyazelenko at yahoo.com  Fri Aug 29 20:04:32 2014
From: vyazelenko at yahoo.com (vyazelenko at yahoo.com)
Date: Fri, 29 Aug 2014 23:04:32 +0300
Subject: JMH 1.0
In-Reply-To: <5400C291.6010901@oracle.com>
References: <5400C291.6010901@oracle.com>
Message-ID: <96692581-7ADB-4EB1-90CB-D8A461ADBF75@yahoo.com>

Awesome news! Thanks Aleksey and the team for great harness!

Best regards,
Dmitry Vyazelenko

Sent from my iPhone

> On Aug 29, 2014, at 21:12, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:
> 
> Hi,
> 
> On March 28, 2013, we had released the very first public version of JMH.
> Today, seventeen months and almost a thousand commits later, we hit the
> major milestone in our project: the release of JMH 1.0. This marks our
> belief the API and the harness itself is stable, reliable, and
> dependable. The artefacts for JMH 1.0 are already available at Maven
> Central [1].
> 
> We will continue working towards improving the harness and/or APIs, and
> we already have quite a few of non-critical issues [2] found by
> community to fix and polish. Please continue using JMH and contribute
> the issues, fixes, and general suggestions back! You, the community, are
> essential to keep the project alive.
> 
> We would like to take the rest of this note to credit those involved in
> the effort, in any capacity:
> 
> * Anders Astrand, Staffan Friberg, and Henrik Loef, who have developed
> the initial version of the harness for internal use back in 2011.
> 
> * Donald Smith, Dalibor Topic, and Cecilia Borg, who have shepherded
> the initial contribution of JMH under Codetools umbrella in OpenJDK, and
> who are still providing the project management guidance for it.
> 
> * Sergey Kuksenko, who contributed immensely in refactoring the JMH
> internals before it became public, as well as providing the internal
> reviews and feedback for the JMH features and improvements
> 
> * Evgeny Mandrikov, who consistently helps us with the releases to
> Maven Central.
> 
> * Nitsan Wakart, Daniel Mitterdorfer, Julien Ponge, Richard Warburton,
> and other authors who provided the solid stories and tutorials on using
> JMH, as well as providing the invaluable feedback on user experience.
> 
> * Bernd Eckenfels, Joe Kearney, Aggelos Biboudis, Chris Vest, Dmitry
> Vyazelenko, Claes Redestad, Dmitry Chuyko, Julien Nicolaund, Tom Deneau,
> Roman Leventov, Henri Tremblay, Ivan Gerasimov, Gilles Duboscq, Clement
> Mathieu and many others participating in mailing list discussions, often
> culminating in code contributions.
> 
> * Konrad Malawski, Cedric Champeau, Gleb Smirnov, Nikita Artyushov,
> Brian Harris and many others contributing to various integrations to
> SBT, Gradle, Jenkins, IntelliJ IDEA, etc.
> 
> Cheers, and Thank You!
> 
> Enjoy the JMH 1.0 :)
> 
> -Aleksey.
> 
> [1] http://central.maven.org/maven2/org/openjdk/jmh/jmh-core/
> [2]
> https://bugs.openjdk.java.net/issues/?jql=project%20in%20(CODETOOLS)%20AND%20component%20in%20(tools)%20AND%20Subcomponent%20in%20(jmh)
> 

From daniel.mitterdorfer at gmail.com  Sat Aug 30 13:15:30 2014
From: daniel.mitterdorfer at gmail.com (Daniel Mitterdorfer)
Date: Sat, 30 Aug 2014 15:15:30 +0200
Subject: JMH 1.0
In-Reply-To: <5400C291.6010901@oracle.com>
References: <5400C291.6010901@oracle.com>
Message-ID: <CAJmnTFjhdW9GcgiAxw=gr9H6=rU-PNb5jf4=3DoCmNRLpqzm7g@mail.gmail.com>

Hi Aleksey,

JMH is 1.0, awesome! Kudos to you and all contributors to JMH for all the
hard work you've put into building such a great framework.

Bye

Daniel



2014-08-29 20:12 GMT+02:00 Aleksey Shipilev <aleksey.shipilev at oracle.com>:

> Hi,
>
> On March 28, 2013, we had released the very first public version of JMH.
> Today, seventeen months and almost a thousand commits later, we hit the
> major milestone in our project: the release of JMH 1.0. This marks our
> belief the API and the harness itself is stable, reliable, and
> dependable. The artefacts for JMH 1.0 are already available at Maven
> Central [1].
>
> We will continue working towards improving the harness and/or APIs, and
> we already have quite a few of non-critical issues [2] found by
> community to fix and polish. Please continue using JMH and contribute
> the issues, fixes, and general suggestions back! You, the community, are
> essential to keep the project alive.
>
> We would like to take the rest of this note to credit those involved in
> the effort, in any capacity:
>
>  * Anders Astrand, Staffan Friberg, and Henrik Loef, who have developed
> the initial version of the harness for internal use back in 2011.
>
>  * Donald Smith, Dalibor Topic, and Cecilia Borg, who have shepherded
> the initial contribution of JMH under Codetools umbrella in OpenJDK, and
> who are still providing the project management guidance for it.
>
>  * Sergey Kuksenko, who contributed immensely in refactoring the JMH
> internals before it became public, as well as providing the internal
> reviews and feedback for the JMH features and improvements
>
>  * Evgeny Mandrikov, who consistently helps us with the releases to
> Maven Central.
>
>  * Nitsan Wakart, Daniel Mitterdorfer, Julien Ponge, Richard Warburton,
> and other authors who provided the solid stories and tutorials on using
> JMH, as well as providing the invaluable feedback on user experience.
>
>  * Bernd Eckenfels, Joe Kearney, Aggelos Biboudis, Chris Vest, Dmitry
> Vyazelenko, Claes Redestad, Dmitry Chuyko, Julien Nicolaund, Tom Deneau,
> Roman Leventov, Henri Tremblay, Ivan Gerasimov, Gilles Duboscq, Clement
> Mathieu and many others participating in mailing list discussions, often
> culminating in code contributions.
>
>  * Konrad Malawski, Cedric Champeau, Gleb Smirnov, Nikita Artyushov,
> Brian Harris and many others contributing to various integrations to
> SBT, Gradle, Jenkins, IntelliJ IDEA, etc.
>
> Cheers, and Thank You!
>
> Enjoy the JMH 1.0 :)
>
> -Aleksey.
>
> [1] http://central.maven.org/maven2/org/openjdk/jmh/jmh-core/
> [2]
>
> https://bugs.openjdk.java.net/issues/?jql=project%20in%20(CODETOOLS)%20AND%20component%20in%20(tools)%20AND%20Subcomponent%20in%20(jmh)
>
>