From sitnikov.vladimir at gmail.com Mon Jun 1 14:02:34 2015 From: sitnikov.vladimir at gmail.com (Vladimir Sitnikov) Date: Mon, 1 Jun 2015 17:02:34 +0300 Subject: Ability to add "jvm arguments" based on "@Param" In-Reply-To: References: <5566D512.2050108@oracle.com> Message-ID: I see two main use cases: 1) Generate unique set of logs per iteration. For instance: 1,1) being able to use unique names for -Xloggc, -XX:+FlightRecorder options 1.2) use different jar files in the classpath (in case of non-shaded dependency) 1.3) being able to calculate OperationsPerInvocation. 2) Explore search space. For instance, if parameters are dependent, it could make sense to investigate different combinations. In case #1, additional configuration of OptionsBuilder is performed _after_ jmh picks the values for all the parameters. We can have even some kind of predefined "post-processors": "save gc logs to gc_param1value1_param2value2.log like files". In case #2 some kind of Supplier is required. It looks case #1 happens more often, and it is somewhat different from #2, so they can be treated differently. Any thoughts on that? Vladimir From sitnikov.vladimir at gmail.com Tue Jun 2 18:49:04 2015 From: sitnikov.vladimir at gmail.com (Vladimir Sitnikov) Date: Tue, 2 Jun 2015 21:49:04 +0300 Subject: "Nice" report for failed forks/benchmarks Message-ID: Hi, I happen to test some finalizer performance, and it looks like jmh's report is not very nice there. Does anybody have idea how that should be depicted? I guess it might need some un-biasing in case just a couple of runs fails. For instance, if 3 of 5 forks survive we would have to average apples with exceptions somehow. # Warmup Iteration 4: 202029,343 ns/op # Warmup Iteration 5: 846066,057 ns/op # Warmup Iteration 6: java.lang.OutOfMemoryError: GC overhead limit exceeded ... # Run complete. Total time: 00:00:18 Benchmark Mode Cnt Score Error Units -- Regards, Vladimir Sitnikov From aleksey.shipilev at oracle.com Wed Jun 3 11:02:45 2015 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Wed, 03 Jun 2015 14:02:45 +0300 Subject: "Nice" report for failed forks/benchmarks In-Reply-To: References: Message-ID: <556EDED5.1080000@oracle.com> On 06/02/2015 09:49 PM, Vladimir Sitnikov wrote: > I happen to test some finalizer performance, and it looks like jmh's > report is not very nice there. Yes, it is tracked by: https://bugs.openjdk.java.net/browse/CODETOOLS-7901010 > Does anybody have idea how that should be depicted? > I guess it might need some un-biasing in case just a couple of runs > fails. For instance, if 3 of 5 forks survive we would have to average > apples with exceptions somehow. I'd think if benchmark fails, then the result should be marked as failure as well. No recovery is possible. > # Warmup Iteration 4: 202029,343 ns/op > # Warmup Iteration 5: 846066,057 ns/op > # Warmup Iteration 6: > java.lang.OutOfMemoryError: GC overhead limit exceeded > > ... > > # Run complete. Total time: 00:00:18 > > Benchmark Mode Cnt Score Error Units Should probably be: Benchmark Mode Cnt Score Error Units MyBenchmark.test avgt Thanks, -Aleksey From sitnikov.vladimir at gmail.com Wed Jun 3 11:28:42 2015 From: sitnikov.vladimir at gmail.com (Vladimir Sitnikov) Date: Wed, 3 Jun 2015 14:28:42 +0300 Subject: "Nice" report for failed forks/benchmarks In-Reply-To: <556EDED5.1080000@oracle.com> References: <556EDED5.1080000@oracle.com> Message-ID: > I'd think if benchmark fails, then the result should be marked as failure as well. No recovery is possible. If warmup fails, then no luck. If at least single measurement iteration succeeds, then we might be interested in some of the profiling results. Overall benchmark might fail due to "OutOfMemory", however "?gc.alloc.rate.norm" is still relevant. What if we add secondary sample that tells user that particular benchmark failed at iteration X? For instance: Benchmark (autoCloseStatements) (leakPct) Mode Cnt Score Error FinalizeStatement.createAndLeak true 0 avgt 2 OutOfMemoryError at iteration 3 FinalizeStatement.createAndLeak:?gc.alloc.rate.norm true 0 avgt 2 192,026 Vladimir From sitnikov.vladimir at gmail.com Wed Jun 3 11:30:47 2015 From: sitnikov.vladimir at gmail.com (Vladimir Sitnikov) Date: Wed, 3 Jun 2015 14:30:47 +0300 Subject: "Nice" report for failed forks/benchmarks In-Reply-To: References: <556EDED5.1080000@oracle.com> Message-ID: Here's git for the suggestion: https://gist.github.com/vlsi/d46a6c8e5ba2abc5e83a Vladimir From aleksey.shipilev at oracle.com Thu Jun 4 12:32:35 2015 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 04 Jun 2015 15:32:35 +0300 Subject: "Nice" report for failed forks/benchmarks In-Reply-To: References: <556EDED5.1080000@oracle.com> Message-ID: <55704563.3000206@oracle.com> On 06/03/2015 02:28 PM, Vladimir Sitnikov wrote: > If at least single measurement iteration succeeds, then we might be > interested in some of the profiling results. > Overall benchmark might fail due to "OutOfMemory", however > "?gc.alloc.rate.norm" is still relevant. I think it does not worth an effort, trying to breadcrumb the results of unstable/failing benchmark. There are things that just do not worth the development and support time. > What if we add secondary sample that tells user that particular > benchmark failed at iteration X? IMO, failing benchmark should be examined by looking at the execution logs. There, users will see where it failed, and what exception had occurred. Let's not overcomplicate things. Also, the scores are implicitly numerical scalars (we are talking performance metrics here), so there is no clean way to encode an arbitrary string there. We can still print "" recognizing the shape of failing result. Thanks, -Aleksey. From sitnikov.vladimir at gmail.com Thu Jun 4 15:17:39 2015 From: sitnikov.vladimir at gmail.com (Vladimir Sitnikov) Date: Thu, 4 Jun 2015 18:17:39 +0300 Subject: "Nice" report for failed forks/benchmarks In-Reply-To: <55704563.3000206@oracle.com> References: <556EDED5.1080000@oracle.com> <55704563.3000206@oracle.com> Message-ID: > failing benchmark should be examined by looking at the execution logs I do not follow you. Do you suggest to kill all the results for a particular benchmark even if there is just a single failure? Well, that's debatable, however I see no need in digging deeper. Killing all the results for a failed benchmark is OK for me provided there is a line in report that tells us that the benchmark did fail. Vladimir From aleksey.shipilev at oracle.com Fri Jun 5 07:23:23 2015 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Fri, 05 Jun 2015 10:23:23 +0300 Subject: "Nice" report for failed forks/benchmarks In-Reply-To: References: <556EDED5.1080000@oracle.com> <55704563.3000206@oracle.com> Message-ID: <55714E6B.8080307@oracle.com> On 06/04/2015 06:17 PM, Vladimir Sitnikov wrote: > Do you suggest to kill all the results for a particular benchmark even > if there is just a single failure? Yes. The performance of a failing benchmark is not something you should be caring about. -Aleksey. From aleksey.shipilev at oracle.com Fri Jun 5 07:29:20 2015 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Fri, 05 Jun 2015 10:29:20 +0300 Subject: Ability to add "jvm arguments" based on "@Param" In-Reply-To: References: <5566D512.2050108@oracle.com> Message-ID: <55714FD0.8060103@oracle.com> On 06/01/2015 05:02 PM, Vladimir Sitnikov wrote: > I see two main use cases: > 1) Generate unique set of logs per iteration. > For instance: > 1,1) being able to use unique names for -Xloggc, -XX:+FlightRecorder options > 1.2) use different jar files in the classpath (in case of non-shaded dependency) > 1.3) being able to calculate OperationsPerInvocation. > > 2) Explore search space. For instance, if parameters are dependent, it > could make sense to investigate different combinations. > > In case #1, additional configuration of OptionsBuilder is performed > _after_ jmh picks the values for all the parameters. > We can have even some kind of predefined "post-processors": "save gc > logs to gc_param1value1_param2value2.log like files". > > In case #2 some kind of Supplier is required. > > > It looks case #1 happens more often, and it is somewhat different from > #2, so they can be treated differently. > > Any thoughts on that? I think all these use cases are handled by API with a minimal coding. So, whatever you come up with can be prototyped as the wrapper around the JMH API. Then we can talk about what parts of it should be migrated under the JMH API. Thanks, -Aleksey From sitnikov.vladimir at gmail.com Fri Jun 5 07:54:03 2015 From: sitnikov.vladimir at gmail.com (Vladimir Sitnikov) Date: Fri, 5 Jun 2015 10:54:03 +0300 Subject: Ability to add "jvm arguments" based on "@Param" In-Reply-To: <55714FD0.8060103@oracle.com> References: <5566D512.2050108@oracle.com> <55714FD0.8060103@oracle.com> Message-ID: Can you please clarify how can I achieve "operationsPerInvocation==@Param" without resorting to my own @Param space walker? Vladimir From aleksey.shipilev at oracle.com Fri Jun 5 07:56:44 2015 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Fri, 05 Jun 2015 10:56:44 +0300 Subject: Ability to add "jvm arguments" based on "@Param" In-Reply-To: References: <5566D512.2050108@oracle.com> <55714FD0.8060103@oracle.com> Message-ID: <5571563C.5010107@oracle.com> On 06/05/2015 10:54 AM, Vladimir Sitnikov wrote: > Can you please clarify how can I achieve > "operationsPerInvocation==@Param" without resorting to my own @Param > space walker? By using .operationsPerInvocation() API? If you wonder how would you make operationsPerInvocation to match the value of a particular @Param, then your wrapper should set *both* .operationsPerInvocation() and the corresponding .param(). -Aleksey. From sitnikov.vladimir at gmail.com Fri Jun 5 08:08:05 2015 From: sitnikov.vladimir at gmail.com (Vladimir Sitnikov) Date: Fri, 5 Jun 2015 11:08:05 +0300 Subject: Ability to add "jvm arguments" based on "@Param" In-Reply-To: <5571563C.5010107@oracle.com> References: <5566D512.2050108@oracle.com> <55714FD0.8060103@oracle.com> <5571563C.5010107@oracle.com> Message-ID: rrr. I want the following: java -jar benchmarks.jar. It runs several combinations of the parameters, and for each combination it sets relevant operationsPerInvocation. I do like how set of parameters to test can be specified at the command line. I do not want reimplementing my own org.openjdk.jmh.runner.Runner#explodeAllParams I do want to get all the results in the same report table. On top of that, it would be great if jmh could use certain parameter as a "treatment" boolean, so it pivots "before"/"after" measurements to "before"/"after" columns. That's a bit different story, but I think it is rather simple/common use case. Vladimir From aleksey.shipilev at oracle.com Fri Jun 5 08:48:30 2015 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Fri, 05 Jun 2015 11:48:30 +0300 Subject: Ability to add "jvm arguments" based on "@Param" In-Reply-To: References: <5566D512.2050108@oracle.com> <55714FD0.8060103@oracle.com> <5571563C.5010107@oracle.com> Message-ID: <5571625E.7050105@oracle.com> On 06/05/2015 11:08 AM, Vladimir Sitnikov wrote: > I want the following: java -jar benchmarks.jar. > It runs several combinations of the parameters, and for each > combination it sets relevant operationsPerInvocation. > I do like how set of parameters to test can be specified at the command line. > I do not want reimplementing my own > org.openjdk.jmh.runner.Runner#explodeAllParams > I do want to get all the results in the same report table. Is it really *that* hard? @Param("1") private int p1; @Param("1") private int p2; @Param("1") private int p3; @Benchmark public void test() throws InterruptedException { TimeUnit.MILLISECONDS.sleep(1); } public static void main(String... args) throws Exception { CommandLineOptions opts = new CommandLineOptions(args); Collection rrs = new ArrayList(); for (String p1 : opts.getParameter("p1").get()) { for (String p2 : opts.getParameter("p2").get()) { for (String p3 : opts.getParameter("p3").get()) { int opi = Integer.valueOf(p1) + Integer.valueOf(p2) + Integer.valueOf(p3); Options o = new OptionsBuilder() .parent(opts) .param("p1", p1) .param("p2", p2) .param("p3", p3) .operationsPerInvocation(opi) .build(); rrs.add(new Runner(o).runSingle()); } } } ResultFormatFactory.getInstance( ResultFormatType.TEXT, System.out).writeOut(rrs); } > On top of that, it would be great if jmh could use certain parameter > as a "treatment" boolean, so it pivots "before"/"after" measurements > to "before"/"after" columns. That's a bit different story, but I think > it is rather simple/common use case. This is the way it was envisioned long ago: 1. Annotations are limiting for you? Use API. 2a. Have an API problem, and you can work better if API provides something more? Suggest to extend the API to expose more knobs. (More likely to fly) 2b. Have an API use case with the code that looks generic enough? Suggest to include it into JMH core. (Less likely to fly) The changes in JMH core are painful and time-consuming, they require working out many mundane details, close the possibilities for misuse and abuse, etc. This is why we keep JMH core arguably nice and tidy, *and* provide APIs for the extension. What you want is to introduce the intrusive change within the JMH core without a compelling reason (yet). This is how it should be done: you need a coherent argument stating "this is how it can be done now", "this is what is minimally desireable", "this is what we would like to see at best". All with code samples please. Handwaving has a drawback that most kinks are not visible until you try to put it into the code... Thanks, -Aleksey From sitnikov.vladimir at gmail.com Fri Jun 5 10:31:28 2015 From: sitnikov.vladimir at gmail.com (Vladimir Sitnikov) Date: Fri, 5 Jun 2015 13:31:28 +0300 Subject: Ability to add "jvm arguments" based on "@Param" In-Reply-To: <5571625E.7050105@oracle.com> References: <5566D512.2050108@oracle.com> <55714FD0.8060103@oracle.com> <5571563C.5010107@oracle.com> <5571625E.7050105@oracle.com> Message-ID: >Is it really *that* hard? a) This raises a bar significantly. I can code that, no problem. The problem is if you "just want bind operationsPerInvocation to a single parameter", then you have to enumerate all the parameters. If you just want having different flight recorder files (or garbage collector log files), you have to write your own profiler. Well, recording jfrs via ExternalProfiler is not that bad. However, running tests with different concurrency levels is not "just adding an option". b) "progress bar" feature is broken when you run in separate Runners, isn't it? >Suggest to extend the API to expose more knobs. (More likely to fly) What if ChainedOptionsBuilder#postConfigurator is added to post-process the options? It is just a proof of concept (no documentation, very little testing, naming issues not solved, etc etc). As far as I understand, even command line can support this API extension via JSR 223 scripting API :) .postConfigurator(new OptionsPostConfigurator() { @Override public void configure(PerBenchmarkOptionsBuilder builder) { builder.operationsPerInvocation(Integer.valueOf(builder.getParam("x"))); // builder.threads(Integer.valueOf(builder.getParam("x"))), etc, etc } }) The idea is to allow customization options that are part of BenchmarkListEntry (excluding WorkloadParams, and, probably, some others) Do you feel it is an "intrusive change"? As for me it looks like a straightforward change (modulo naming, documentation and testing). Vladimir -------------- next part -------------- diff -r 13ffd1b8c9a1 jmh-core-it/src/test/java/org/openjdk/jmh/it/postprocessor/VariableOperationsPerInvocation.java --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/jmh-core-it/src/test/java/org/openjdk/jmh/it/postprocessor/VariableOperationsPerInvocation.java Fri Jun 05 12:20:30 2015 +0300 @@ -0,0 +1,86 @@ +/* + * Copyright (c) 2005, 2014, Oracle and/or its affiliates. All rights reserved. + * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. + * + * This code is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 only, as + * published by the Free Software Foundation. Oracle designates this + * particular file as subject to the "Classpath" exception as provided + * by Oracle in the LICENSE file that accompanied this code. + * + * This code is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License + * version 2 for more details (a copy is included in the LICENSE file that + * accompanied this code). + * + * You should have received a copy of the GNU General Public License version + * 2 along with this work; if not, write to the Free Software Foundation, + * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + * + * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA + * or visit www.oracle.com if you need additional information or have any + * questions. + */ +package org.openjdk.jmh.it.postprocessor; + +import junit.framework.Assert; +import org.junit.Test; +import org.openjdk.jmh.annotations.*; +import org.openjdk.jmh.it.Fixtures; +import org.openjdk.jmh.results.RunResult; +import org.openjdk.jmh.runner.Runner; +import org.openjdk.jmh.runner.RunnerException; +import org.openjdk.jmh.runner.options.*; + +import java.util.Collection; +import java.util.concurrent.TimeUnit; + + at Measurement(iterations = 1, time = 100, timeUnit = TimeUnit.MILLISECONDS) + at Warmup(iterations = 1, time = 100, timeUnit = TimeUnit.MILLISECONDS) + at Fork(1) + at State(Scope.Thread) +public class VariableOperationsPerInvocation { + @Param({"1", "2", "3"}) + public static int x; + + @Benchmark + public void test() { + Fixtures.work(); + } + + @Test + public void full() throws RunnerException { + Options opts = new OptionsBuilder() + .include(Fixtures.getTestMask(this.getClass())) + .shouldFailOnError(true) + .postConfigurator(new OptionsPostConfigurator() { + @Override + public void configure(PerBenchmarkOptionsBuilder builder) { + builder.operationsPerInvocation(Integer.valueOf(builder.getParam("x"))); + } + }) + .build(); + + assertOperationsPerInvocation(new Runner(opts).run()); + } + + @Test + public void constrainedX() throws RunnerException { + Options opts = new OptionsBuilder() + .include(Fixtures.getTestMask(this.getClass())) + .shouldFailOnError(true) + .param("x", "2", "3") + .build(); + + assertOperationsPerInvocation(new Runner(opts).run()); + } + + private void assertOperationsPerInvocation(Collection run) { + for (RunResult result : run) { + int opsPerInvocation = result.getParams().getOpsPerInvocation(); + String x = result.getParams().getParam("x"); + Assert.assertEquals(x, Integer.toString(opsPerInvocation)); + } + } +} diff -r 13ffd1b8c9a1 jmh-core/src/main/java/org/openjdk/jmh/runner/BenchmarkListEntry.java --- a/jmh-core/src/main/java/org/openjdk/jmh/runner/BenchmarkListEntry.java Thu May 21 18:45:31 2015 +0300 +++ b/jmh-core/src/main/java/org/openjdk/jmh/runner/BenchmarkListEntry.java Fri Jun 05 12:20:30 2015 +0300 @@ -25,6 +25,7 @@ package org.openjdk.jmh.runner; import org.openjdk.jmh.annotations.Mode; +import org.openjdk.jmh.runner.options.Options; import org.openjdk.jmh.runner.options.TimeValue; import org.openjdk.jmh.util.Optional; import org.openjdk.jmh.util.Utils; @@ -170,6 +171,19 @@ return br; } + public BenchmarkListEntry cloneWith(Options options) { + BenchmarkListEntry br = new BenchmarkListEntry(userName, generatedName, options.getBenchModes().iterator().next(), + options.getThreadGroups().get(), options.getThreads(), + options.getWarmupIterations(), options.getWarmupTime(), options.getWarmupBatchSize(), + options.getMeasurementIterations(), options.getMeasurementTime(), options.getMeasurementBatchSize(), + options.getForkCount(), options.getWarmupForkCount(), + options.getJvm(), options.getJvmArgs(), options.getJvmArgsPrepend(), options.getJvmArgsAppend(), + params, options.getTimeUnit(), options.getOperationsPerInvocation(), + options.getTimeout()); + br.workloadParams = workloadParams; + return br; + } + public WorkloadParams getWorkloadParams() { return workloadParams; } diff -r 13ffd1b8c9a1 jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java --- a/jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java Thu May 21 18:45:31 2015 +0300 +++ b/jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java Fri Jun 05 12:20:30 2015 +0300 @@ -40,17 +40,8 @@ import org.openjdk.jmh.runner.format.OutputFormat; import org.openjdk.jmh.runner.format.OutputFormatFactory; import org.openjdk.jmh.runner.link.BinaryLinkServer; -import org.openjdk.jmh.runner.options.Options; -import org.openjdk.jmh.runner.options.TimeValue; -import org.openjdk.jmh.runner.options.VerboseMode; -import org.openjdk.jmh.util.FileUtils; -import org.openjdk.jmh.util.HashMultimap; -import org.openjdk.jmh.util.InputStreamDrainer; -import org.openjdk.jmh.util.Multimap; -import org.openjdk.jmh.util.TreeMultimap; -import org.openjdk.jmh.util.UnCloseablePrintStream; -import org.openjdk.jmh.util.Utils; -import org.openjdk.jmh.util.Version; +import org.openjdk.jmh.runner.options.*; +import org.openjdk.jmh.util.*; import java.io.File; import java.io.FileNotFoundException; @@ -305,6 +296,19 @@ benchmarks.addAll(newBenchmarks); } + Optional postConfigurator = options.getPostConfigurator(); + if (postConfigurator.hasValue()) { + OptionsPostConfigurator configurator = postConfigurator.get(); + List newBenchmarks = new ArrayList(benchmarks.size()); + for (BenchmarkListEntry br : benchmarks) { + PerBenchmarkOptionsBuilderImpl opts = PerBenchmarkOptionsBuilderImpl.of(br, options); + configurator.configure(opts); + newBenchmarks.add(br.cloneWith(opts)); + } + benchmarks.clear(); + benchmarks.addAll(newBenchmarks); + } + Collection results = runBenchmarks(benchmarks); // If user requested the result file, write it out. diff -r 13ffd1b8c9a1 jmh-core/src/main/java/org/openjdk/jmh/runner/options/ChainedOptionsBuilder.java --- a/jmh-core/src/main/java/org/openjdk/jmh/runner/options/ChainedOptionsBuilder.java Thu May 21 18:45:31 2015 +0300 +++ b/jmh-core/src/main/java/org/openjdk/jmh/runner/options/ChainedOptionsBuilder.java Fri Jun 05 12:20:30 2015 +0300 @@ -305,4 +305,5 @@ */ ChainedOptionsBuilder timeout(TimeValue value); + ChainedOptionsBuilder postConfigurator(OptionsPostConfigurator value); } diff -r 13ffd1b8c9a1 jmh-core/src/main/java/org/openjdk/jmh/runner/options/CommandLineOptions.java --- a/jmh-core/src/main/java/org/openjdk/jmh/runner/options/CommandLineOptions.java Thu May 21 18:45:31 2015 +0300 +++ b/jmh-core/src/main/java/org/openjdk/jmh/runner/options/CommandLineOptions.java Fri Jun 05 12:20:30 2015 +0300 @@ -632,4 +632,9 @@ public Optional getTimeout() { return timeout; } + + @Override + public Optional getPostConfigurator() { + return Optional.none(); // TODO: Use JSR-223 to instantiate OptionsPostConfigurator from the command line? + } } diff -r 13ffd1b8c9a1 jmh-core/src/main/java/org/openjdk/jmh/runner/options/Options.java --- a/jmh-core/src/main/java/org/openjdk/jmh/runner/options/Options.java Thu May 21 18:45:31 2015 +0300 +++ b/jmh-core/src/main/java/org/openjdk/jmh/runner/options/Options.java Fri Jun 05 12:20:30 2015 +0300 @@ -245,4 +245,10 @@ */ Optional getTimeout(); + /** + * Configurator of additional options. + * @return configurator + */ + Optional getPostConfigurator(); + } diff -r 13ffd1b8c9a1 jmh-core/src/main/java/org/openjdk/jmh/runner/options/OptionsBuilder.java --- a/jmh-core/src/main/java/org/openjdk/jmh/runner/options/OptionsBuilder.java Thu May 21 18:45:31 2015 +0300 +++ b/jmh-core/src/main/java/org/openjdk/jmh/runner/options/OptionsBuilder.java Fri Jun 05 12:20:30 2015 +0300 @@ -699,4 +699,23 @@ // --------------------------------------------------------------------------- + private Optional postConfigurator = Optional.none(); + + @Override + public ChainedOptionsBuilder postConfigurator(OptionsPostConfigurator value) { + this.postConfigurator = Optional.of(value); + return this; + } + + @Override + public Optional getPostConfigurator() { + if (otherOptions != null) { + return postConfigurator.orAnother(otherOptions.getPostConfigurator()); + } else { + return postConfigurator; + } + } + + // --------------------------------------------------------------------------- + } diff -r 13ffd1b8c9a1 jmh-core/src/main/java/org/openjdk/jmh/runner/options/OptionsPostConfigurator.java --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/jmh-core/src/main/java/org/openjdk/jmh/runner/options/OptionsPostConfigurator.java Fri Jun 05 12:20:30 2015 +0300 @@ -0,0 +1,36 @@ +/* + * Copyright (c) 2014, 2015, Oracle and/or its affiliates. All rights reserved. + * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. + * + * This code is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 only, as + * published by the Free Software Foundation. Oracle designates this + * particular file as subject to the "Classpath" exception as provided + * by Oracle in the LICENSE file that accompanied this code. + * + * This code is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License + * version 2 for more details (a copy is included in the LICENSE file that + * accompanied this code). + * + * You should have received a copy of the GNU General Public License version + * 2 along with this work; if not, write to the Free Software Foundation, + * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + * + * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA + * or visit www.oracle.com if you need additional information or have any + * questions. + */ +package org.openjdk.jmh.runner.options; + +public interface OptionsPostConfigurator { + /** + * Post configure given set of options for a particular benchmark. + * For instance, you can use argument that is customized based on the test parameters. + * + *

Note: not all the options can be overridden + * @param builder + */ + void configure(PerBenchmarkOptionsBuilder builder); +} diff -r 13ffd1b8c9a1 jmh-core/src/main/java/org/openjdk/jmh/runner/options/PerBenchmarkOptionsBuilder.java --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/jmh-core/src/main/java/org/openjdk/jmh/runner/options/PerBenchmarkOptionsBuilder.java Fri Jun 05 12:20:30 2015 +0300 @@ -0,0 +1,31 @@ +/* + * Copyright (c) 2014, 2015, Oracle and/or its affiliates. All rights reserved. + * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. + * + * This code is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 only, as + * published by the Free Software Foundation. Oracle designates this + * particular file as subject to the "Classpath" exception as provided + * by Oracle in the LICENSE file that accompanied this code. + * + * This code is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License + * version 2 for more details (a copy is included in the LICENSE file that + * accompanied this code). + * + * You should have received a copy of the GNU General Public License version + * 2 along with this work; if not, write to the Free Software Foundation, + * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + * + * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA + * or visit www.oracle.com if you need additional information or have any + * questions. + */ +package org.openjdk.jmh.runner.options; + +public interface PerBenchmarkOptionsBuilder extends ChainedOptionsBuilder, Options { + String getUsername(); + String getGeneratedTarget(); + String getParam(String param); +} diff -r 13ffd1b8c9a1 jmh-core/src/main/java/org/openjdk/jmh/runner/options/PerBenchmarkOptionsBuilderImpl.java --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/jmh-core/src/main/java/org/openjdk/jmh/runner/options/PerBenchmarkOptionsBuilderImpl.java Fri Jun 05 12:20:30 2015 +0300 @@ -0,0 +1,77 @@ +/* + * Copyright (c) 2014, 2015, Oracle and/or its affiliates. All rights reserved. + * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. + * + * This code is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 only, as + * published by the Free Software Foundation. Oracle designates this + * particular file as subject to the "Classpath" exception as provided + * by Oracle in the LICENSE file that accompanied this code. + * + * This code is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License + * version 2 for more details (a copy is included in the LICENSE file that + * accompanied this code). + * + * You should have received a copy of the GNU General Public License version + * 2 along with this work; if not, write to the Free Software Foundation, + * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + * + * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA + * or visit www.oracle.com if you need additional information or have any + * questions. + */ +package org.openjdk.jmh.runner.options; + +import org.openjdk.jmh.runner.BenchmarkListEntry; +import org.openjdk.jmh.runner.WorkloadParams; + +public class PerBenchmarkOptionsBuilderImpl extends OptionsBuilder implements PerBenchmarkOptionsBuilder { + private final BenchmarkListEntry br; + + public PerBenchmarkOptionsBuilderImpl(BenchmarkListEntry br) { + this.br = br; + } + + public static PerBenchmarkOptionsBuilderImpl of(BenchmarkListEntry br, Options parent) { + PerBenchmarkOptionsBuilderImpl out = new PerBenchmarkOptionsBuilderImpl(br); + out.parent(parent); + out.mode(br.getMode()); + if (br.getThreads().hasValue()) out.threads(br.getThreads().get()); + if (br.getJvmArgsAppend().hasValue()) out.jvmArgsAppend(br.getJvmArgsAppend().get().toArray(new String[0])); + out.threadGroups(br.getThreadGroups()); + if (br.getWarmupIterations().hasValue()) out.warmupIterations(br.getWarmupIterations().get()); + if (br.getWarmupBatchSize().hasValue()) out.warmupBatchSize(br.getWarmupBatchSize().get()); + if (br.getWarmupTime().hasValue()) out.warmupTime(br.getWarmupTime().get()); + if (br.getMeasurementIterations().hasValue()) out.measurementIterations(br.getMeasurementIterations().get()); + if (br.getMeasurementBatchSize().hasValue()) out.measurementBatchSize(br.getMeasurementBatchSize().get()); + if (br.getMeasurementTime().hasValue()) out.measurementTime(br.getMeasurementTime().get()); + if (br.getForks().hasValue()) out.forks(br.getForks().get()); + if (br.getWarmupForks().hasValue()) out.warmupForks(br.getWarmupForks().get()); + if (br.getJvm().hasValue()) out.jvm(br.getJvm().get()); + if (br.getJvmArgs().hasValue()) out.jvmArgs(br.getJvmArgs().get().toArray(new String[0])); + if (br.getJvmArgsAppend().hasValue()) out.jvmArgsAppend(br.getJvmArgsAppend().get().toArray(new String[0])); + if (br.getJvmArgsPrepend().hasValue()) out.jvmArgsPrepend(br.getJvmArgsPrepend().get().toArray(new String[0])); + if (br.getTimeUnit().hasValue()) out.timeUnit(br.getTimeUnit().get()); + if (br.getOperationsPerInvocation().hasValue()) out.operationsPerInvocation(br.getOperationsPerInvocation().get()); + WorkloadParams params = br.getWorkloadParams(); + // Just in case, pretend we are running the exact param combination + for (String param : params.keys()) { + out.param(param, params.get(param)); + } + return out; + } + + public String getUsername() { + return br.getUsername(); + } + + public String getGeneratedTarget() { + return br.generatedTarget(); + } + + public String getParam(String param) { + return br.getWorkloadParams().get(param); + } +} From sebastian.ramacher at iaik.tugraz.at Fri Jun 5 11:06:01 2015 From: sebastian.ramacher at iaik.tugraz.at (Sebastian Ramacher) Date: Fri, 05 Jun 2015 13:06:01 +0200 Subject: CSV result writer saves samples as floating point number Message-ID: <55718299.7040400@iaik.tugraz.at> [Please CC me on replies, I'm not subscribed.] Hi Result's getSampleCount returns a long, but if the result is exported as CSV the sample count is written as floating point number. I suppose this is due to org.openjdk.jmh.results.format.XSVResultFormat's emit methods which is available for ints and doubles. So for the sample count emit(double) is called instead of the integer variant. Could emit(int) please be changed to emit(long) so that the result is exported as integer? Cheers From aleksey.shipilev at oracle.com Mon Jun 8 07:47:34 2015 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Mon, 08 Jun 2015 07:47:34 +0000 Subject: hg: code-tools/jmh: 7901443: CSV result writer saves samples as floating point number Message-ID: <201506080747.t587lYbd014629@aojmv0008.oracle.com> Changeset: 6609eea91d87 Author: shade Date: 2015-06-08 10:47 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/6609eea91d87 7901443: CSV result writer saves samples as floating point number ! jmh-core/src/main/java/org/openjdk/jmh/results/format/XSVResultFormat.java ! jmh-core/src/test/resources/org/openjdk/jmh/results/format/output-golden.csv.root ! jmh-core/src/test/resources/org/openjdk/jmh/results/format/output-golden.csv.ru ! jmh-core/src/test/resources/org/openjdk/jmh/results/format/output-golden.csv.us ! jmh-core/src/test/resources/org/openjdk/jmh/results/format/output-golden.scsv.root ! jmh-core/src/test/resources/org/openjdk/jmh/results/format/output-golden.scsv.ru ! jmh-core/src/test/resources/org/openjdk/jmh/results/format/output-golden.scsv.us From aleksey.shipilev at oracle.com Mon Jun 8 07:48:14 2015 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Mon, 08 Jun 2015 10:48:14 +0300 Subject: CSV result writer saves samples as floating point number In-Reply-To: <55718299.7040400@iaik.tugraz.at> References: <55718299.7040400@iaik.tugraz.at> Message-ID: <557548BE.2090108@oracle.com> On 06/05/2015 02:06 PM, Sebastian Ramacher wrote: > Result's getSampleCount returns a long, but if the result is exported as > CSV the sample count is written as floating point number. I suppose this > is due to org.openjdk.jmh.results.format.XSVResultFormat's emit methods > which is available for ints and doubles. So for the sample count > emit(double) is called instead of the integer variant. Could emit(int) > please be changed to emit(long) so that the result is exported as integer? Thanks, fixed: https://bugs.openjdk.java.net/browse/CODETOOLS-7901443 http://hg.openjdk.java.net/code-tools/jmh/rev/6609eea91d87 -Aleksey From aleksey.shipilev at oracle.com Mon Jun 8 08:55:00 2015 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Mon, 08 Jun 2015 11:55:00 +0300 Subject: Ability to add "jvm arguments" based on "@Param" In-Reply-To: References: <5566D512.2050108@oracle.com> <55714FD0.8060103@oracle.com> <5571563C.5010107@oracle.com> <5571625E.7050105@oracle.com> Message-ID: <55755864.4040802@oracle.com> On 06/05/2015 01:31 PM, Vladimir Sitnikov wrote: >> Is it really *that* hard? > > a) This raises a bar significantly. I can code that, no problem. The > problem is if you "just want bind operationsPerInvocation to a single > parameter", then you have to enumerate all the parameters. Yes. Using the API would *always* be harder than using any implicit thing the framework has. The trick is to recognize whether it is hard enough to warrant complicating the framework implementation. There is always a consideration where to draw a line in the framework interface. > If you just want having different flight recorder files (or garbage > collector log files), you have to write your own profiler. > Well, recording jfrs via ExternalProfiler is not that bad. However, > running tests with different concurrency levels is not "just adding an > option". > > b) "progress bar" feature is broken when you run in separate Runners, isn't it? Yes, but only if you presume the progress should be tracked across Runner calls. >> Suggest to extend the API to expose more knobs. (More > likely to fly) > > What if ChainedOptionsBuilder#postConfigurator is added to > post-process the options? > It is just a proof of concept (no documentation, very little testing, > naming issues not solved, etc etc). Oh, good one. I wonder if you can simplify it by recognizing you want to tap into (Options,BenchmarkListEntry) -> BenchmarkParams conversion, like this: http://cr.openjdk.java.net/~shade/jmh/postconfigurator-1.patch The upside for this approach is that it does not introduce more concepts than required, and uses the public BenchmarkParams [1] to communicate the actual parameters. Thanks, -Aleksey [1] http://hg.openjdk.java.net/code-tools/jmh/file/6609eea91d87/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_31_InfraParams.java From sitnikov.vladimir at gmail.com Mon Jun 8 12:35:27 2015 From: sitnikov.vladimir at gmail.com (Vladimir Sitnikov) Date: Mon, 8 Jun 2015 15:35:27 +0300 Subject: Ability to add "jvm arguments" based on "@Param" In-Reply-To: <55755864.4040802@oracle.com> References: <5566D512.2050108@oracle.com> <55714FD0.8060103@oracle.com> <5571563C.5010107@oracle.com> <5571625E.7050105@oracle.com> <55755864.4040802@oracle.com> Message-ID: >I wonder if you can simplify it by recognizing you want to tap into (Options,BenchmarkListEntry) -> BenchmarkParams conversion, This seems to be a better approach. Frankly speaking, I was very confused by newBenchmarkParams vs cloneWith. Currently it makes more sense for me, so I'm just confused (without "very") :) 1) It would be good to be able to use multiple post processors. For instance, we could add OOB processor that "binds @Threads to specific param value". In other words: "-t threadCount" would use the value of "threadCount" parameter for "number of threads". Should it be postConfigurator (just set postconfigurator and ignore others) and postConfiguratorAdd (add configurator to the list) methods? 2) getActionPlans uses BenchmarkParams params = newBenchmarkParams(br, ActionMode.UNDEF); So PostProcessor sees invalid values for BenchmarkParams#getMeasurement. This looks like a slight glitch as it exposes "strange values for getMeasurement and getWarmup)" to the post processor. Vladimir From aleksey.shipilev at oracle.com Tue Jun 9 07:27:37 2015 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Tue, 09 Jun 2015 07:27:37 +0000 Subject: hg: code-tools/jmh: 7901444: Diagnostic message for duplicate @Benchmark method should point to exact duplicate Message-ID: <201506090727.t597RbwU000590@aojmv0008.oracle.com> Changeset: 0dcde834de07 Author: shade Date: 2015-06-09 10:27 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/0dcde834de07 7901444: Diagnostic message for duplicate @Benchmark method should point to exact duplicate ! jmh-core/src/main/java/org/openjdk/jmh/generators/core/MethodGroup.java From aleksey.shipilev at oracle.com Wed Jun 10 19:28:51 2015 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Wed, 10 Jun 2015 19:28:51 +0000 Subject: hg: code-tools/jmh: 7901387: UX: profiler options should be specifiable on command line Message-ID: <201506101928.t5AJSpRJ004595@aojmv0008.oracle.com> Changeset: 15cd0bde0cee Author: shade Date: 2015-06-10 22:28 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/15cd0bde0cee 7901387: UX: profiler options should be specifiable on command line ! jmh-core-it/src/test/java/org/openjdk/jmh/it/ccontrol/LogConsumeProfiler.java ! jmh-core-it/src/test/java/org/openjdk/jmh/it/profilers/GCProfilerTest.java ! jmh-core-it/src/test/java/org/openjdk/jmh/it/profilers/ItExternalProfiler.java ! jmh-core-it/src/test/java/org/openjdk/jmh/it/profilers/ItInternalProfiler.java ! jmh-core-it/src/test/java/org/openjdk/jmh/it/profilers/ProfilerTest.java ! jmh-core-it/src/test/java/org/openjdk/jmh/it/profilers/order/AbstractExternalProfiler.java ! jmh-core-it/src/test/java/org/openjdk/jmh/it/profilers/order/AbstractInternalProfiler.java ! jmh-core-it/src/test/java/org/openjdk/jmh/it/profilers/order/ExternalProfiler1.java ! jmh-core-it/src/test/java/org/openjdk/jmh/it/profilers/order/ExternalProfiler2.java ! jmh-core-it/src/test/java/org/openjdk/jmh/it/profilers/order/ExternalProfiler3.java ! jmh-core-it/src/test/java/org/openjdk/jmh/it/profilers/order/InternalProfiler1.java ! jmh-core-it/src/test/java/org/openjdk/jmh/it/profilers/order/InternalProfiler2.java ! jmh-core-it/src/test/java/org/openjdk/jmh/it/profilers/order/InternalProfiler3.java ! jmh-core-it/src/test/java/org/openjdk/jmh/it/profilers/order/ProfilerOrderTest.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/AbstractHotspotProfiler.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/AbstractPerfAsmProfiler.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/ClassloaderProfiler.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/CompilerProfiler.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/GCProfiler.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/HotspotClassloadingProfiler.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/HotspotCompilationProfiler.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/HotspotMemoryProfiler.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/HotspotRuntimeProfiler.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/HotspotThreadProfiler.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/LinuxPerfAsmProfiler.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/LinuxPerfNormProfiler.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/LinuxPerfProfiler.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/Profiler.java + jmh-core/src/main/java/org/openjdk/jmh/profile/ProfilerException.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/ProfilerFactory.java + jmh-core/src/main/java/org/openjdk/jmh/profile/ProfilerOptionFormatter.java + jmh-core/src/main/java/org/openjdk/jmh/profile/ProfilerUtils.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/StackProfiler.java ! jmh-core/src/main/java/org/openjdk/jmh/profile/WinPerfAsmProfiler.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/BenchmarkHandler.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/options/ChainedOptionsBuilder.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/options/CommandLineOptions.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/options/Options.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/options/OptionsBuilder.java + jmh-core/src/main/java/org/openjdk/jmh/runner/options/ProfilerConfig.java ! jmh-core/src/test/java/org/openjdk/jmh/runner/options/TestOptions.java ! jmh-core/src/test/java/org/openjdk/jmh/runner/options/TestParentOptions.java From aleksey.shipilev at oracle.com Wed Jun 10 19:31:09 2015 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Wed, 10 Jun 2015 22:31:09 +0300 Subject: Heads-up: profiler UX changes Message-ID: <5578907D.5000805@oracle.com> Hi, We have pushed the fix for: https://bugs.openjdk.java.net/browse/CODETOOLS-7901387 ...which now enables profilers to use profiler options from the command line and/or API. There are a few things left to do to smooth the edges, but I would appreciate the testing this week (next 36 hrs would be best!). Try, for example, -prof stack:help or -prof perfasm:help. Thanks, -Aleksey. From sitnikov.vladimir at gmail.com Wed Jun 10 20:00:51 2015 From: sitnikov.vladimir at gmail.com (Vladimir Sitnikov) Date: Wed, 10 Jun 2015 23:00:51 +0300 Subject: Heads-up: profiler UX changes In-Reply-To: <5578907D.5000805@oracle.com> References: <5578907D.5000805@oracle.com> Message-ID: Hi, java -jar benchmarks.jar -prof perfasm:help and -prof stack:help from jmh-samples/target resulted in ERROR: org.openjdk.jmh.runner.RunnerException: Some profilers have failed to initialize at org.openjdk.jmh.runner.Runner.internalRun(Runner.java:209) at org.openjdk.jmh.runner.Runner.run(Runner.java:178) at org.openjdk.jmh.Main.main(Main.java:69) However the help itself looks wonderful. PS. osx, java 1.8 Vladimir From aleksey.shipilev at oracle.com Wed Jun 10 20:07:36 2015 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Wed, 10 Jun 2015 23:07:36 +0300 Subject: Heads-up: profiler UX changes In-Reply-To: References: <5578907D.5000805@oracle.com> Message-ID: <55789908.8090900@oracle.com> On 06/10/2015 11:00 PM, Vladimir Sitnikov wrote: > java -jar benchmarks.jar -prof perfasm:help and -prof stack:help from > jmh-samples/target resulted in > > ERROR: org.openjdk.jmh.runner.RunnerException: Some profilers have > failed to initialize > at org.openjdk.jmh.runner.Runner.internalRun(Runner.java:209) > at org.openjdk.jmh.runner.Runner.run(Runner.java:178) > at org.openjdk.jmh.Main.main(Main.java:69) > > However the help itself looks wonderful. But it printed the help before failing with "ERROR", right? We should shun the error message in this case. Thanks, -Aleksey. From sitnikov.vladimir at gmail.com Wed Jun 10 20:08:45 2015 From: sitnikov.vladimir at gmail.com (Vladimir Sitnikov) Date: Wed, 10 Jun 2015 23:08:45 +0300 Subject: Heads-up: profiler UX changes In-Reply-To: <55789908.8090900@oracle.com> References: <5578907D.5000805@oracle.com> <55789908.8090900@oracle.com> Message-ID: Sure it did. Vladimir From aleksey.shipilev at oracle.com Thu Jun 11 14:57:05 2015 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 11 Jun 2015 14:57:05 +0000 Subject: hg: code-tools/jmh: 7901448: Profiler help should not yield an ERROR message Message-ID: <201506111457.t5BEv5pl019538@aojmv0008.oracle.com> Changeset: 3378c82927e0 Author: shade Date: 2015-06-11 17:56 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/3378c82927e0 7901448: Profiler help should not yield an ERROR message ! jmh-core/src/main/java/org/openjdk/jmh/Main.java + jmh-core/src/main/java/org/openjdk/jmh/runner/ProfilersFailedException.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java From aleksey.shipilev at oracle.com Thu Jun 11 14:58:32 2015 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 11 Jun 2015 14:58:32 +0000 Subject: hg: code-tools/jmh: Amend 7901448 with better formatting. Message-ID: <201506111458.t5BEwWb6019735@aojmv0008.oracle.com> Changeset: ec073422d692 Author: shade Date: 2015-06-11 17:57 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/ec073422d692 Amend 7901448 with better formatting. ! jmh-core/src/main/java/org/openjdk/jmh/Main.java ! jmh-core/src/main/java/org/openjdk/jmh/runner/ProfilersFailedException.java From aleksey.shipilev at oracle.com Thu Jun 11 15:07:32 2015 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 11 Jun 2015 15:07:32 +0000 Subject: hg: code-tools/jmh: Profiler javadoc to explain the new interface. Message-ID: <201506111507.t5BF7W6b022185@aojmv0008.oracle.com> Changeset: c247d464d17f Author: shade Date: 2015-06-11 18:07 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/c247d464d17f Profiler javadoc to explain the new interface. ! jmh-core/src/main/java/org/openjdk/jmh/profile/Profiler.java ! jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_35_Profilers.java From aleksey.shipilev at oracle.com Fri Jun 12 23:37:16 2015 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Sat, 13 Jun 2015 02:37:16 +0300 Subject: JMH 1.10 Message-ID: <557B6D2C.4070803@oracle.com> Hi, It has been a while since the last release. Rejoice! Double-digit release is coming. JMH 1.10 is released and available at Maven Central (props to Evgeny Mandrikov, as usual). This is a feature release, and it contains the following changes (in order of importance/visibility): * Profilers are now accepting the options. Try, for example, -prof stack:help. The treatment of the option line is profiler-specific, but bundled JMH profilers accept "-prof :key1=val1,val2;key2=val3". So far only stack, (x)perfasm and perfnorm are accepting the options, but we fully expect more profilers to follow suit. There are also the relevant changes in the API, to accept the option line. Because of the new option line, the profiler interface had changed (again), so pluggable profilers would need adjustment, see Profiler javadoc. Tracking: https://bugs.openjdk.java.net/browse/CODETOOLS-7901387 https://bugs.openjdk.java.net/browse/CODETOOLS-7901448 * Command line options checking was overhauled to range-check the options. It saves you from weird things like "-5 warmup iterations", that wrecks the internals. Contributed by Vladimir Sitnikov, thanks! Tracking: https://bugs.openjdk.java.net/browse/CODETOOLS-7901431 * CSV writer was saving "samples" as floating-point number, while the value is always integral: https://bugs.openjdk.java.net/browse/CODETOOLS-7901443 * Diagnostic message for duplicate @Benchmark was misleading. We now print the exact duplicate we are colliding with: https://bugs.openjdk.java.net/browse/CODETOOLS-7901444 * (x)perfasm profilers were not treating negative error codes as errors, fixed: https://bugs.openjdk.java.net/browse/CODETOOLS-7901434 * ThreadFactory that was naming the threads did instantiate the defaultThreadFactory every time (duh), messing up thread names: https://bugs.openjdk.java.net/browse/CODETOOLS-7901430 * We are now using ProcessBuilders instead of raw Runtime.exec, which hopefully improves reliability, and escapes corner cases with command line and argument handling: https://bugs.openjdk.java.net/browse/CODETOOLS-7901435 Enjoy! Thanks, -Aleksey From aleksey.shipilev at oracle.com Fri Jun 12 23:37:43 2015 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Fri, 12 Jun 2015 23:37:43 +0000 Subject: hg: code-tools/jmh: 3 new changesets Message-ID: <201506122337.t5CNbhat015434@aojmv0008.oracle.com> Changeset: 665a832d5061 Author: shade Date: 2015-06-12 13:28 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/665a832d5061 JMH v1.10. ! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml ! jmh-archetypes/pom.xml ! jmh-core-benchmarks/pom.xml ! jmh-core-ct/pom.xml ! jmh-core-it/pom.xml ! jmh-core/pom.xml ! jmh-generator-annprocess/pom.xml ! jmh-generator-asm/pom.xml ! jmh-generator-bytecode/pom.xml ! jmh-generator-reflection/pom.xml ! jmh-samples/pom.xml ! pom.xml Changeset: 07bbe543b2e5 Author: shade Date: 2015-06-12 13:29 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/07bbe543b2e5 Added tag 1.10 for changeset 665a832d5061 ! .hgtags Changeset: 1db18e73d21c Author: shade Date: 2015-06-12 13:29 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/1db18e73d21c Continue in 1.11-SNAPSHOT. ! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml ! jmh-archetypes/pom.xml ! jmh-core-benchmarks/pom.xml ! jmh-core-ct/pom.xml ! jmh-core-it/pom.xml ! jmh-core/pom.xml ! jmh-generator-annprocess/pom.xml ! jmh-generator-asm/pom.xml ! jmh-generator-bytecode/pom.xml ! jmh-generator-reflection/pom.xml ! jmh-samples/pom.xml ! pom.xml From me at bramp.net Thu Jun 18 16:17:42 2015 From: me at bramp.net (Andrew Brampton) Date: Thu, 18 Jun 2015 09:17:42 -0700 Subject: Creating a Blackhole Message-ID: I have a complex benchmark, that takes >1 hour to run. I kept hitting an issue where the benchmark would crash during one of the setup/teardown phase (due to mistakes in my code). So I figured why don't I unit test my benchmark, to ensure the assumptions I make hold true. During my unit tests I wanted to create a Blackhole to pass to one of the benchmark methods. However, I found out that I'm not allowed to create a Blackhole: IllegalStateException("Blackholes should not be instantiated directly."). I can either do a nasty hack to create the Blackhole, or I'm requesting you add a new static method, Blackhole.newBlackholeYesIdoKnowWhatImDoing(), or similar. Since in my case, I don't care about the benchmarking, and I'm calling this from a single junit thread, and just testing the correctness of my benchmark code. Alternatively if you create a Blackhole interface, I could easily create mock blackhole, for my testing. Suggestions? thanks Andrew P.S For the curious, my code lives here https://github.com/bramp/unsafe under unsafe-benchmark. From dawid.weiss at gmail.com Thu Jun 18 16:25:24 2015 From: dawid.weiss at gmail.com (Dawid Weiss) Date: Thu, 18 Jun 2015 18:25:24 +0200 Subject: Creating a Blackhole In-Reply-To: References: Message-ID: This may not be the quickest solution... but you can postprocess your bytecode after compilation (with asmlib, aspectj or whatever) and remove blackhole calls. With AspectJ it's a relatively easy call pointcut and an "around" advice that doesn't propagate the call. You could replace the default compiler with ajc and apply test-specific aspects during compilation. Dawid On Thu, Jun 18, 2015 at 6:17 PM, Andrew Brampton wrote: > I have a complex benchmark, that takes >1 hour to run. I kept hitting an > issue where the benchmark would crash during one of the setup/teardown > phase (due to mistakes in my code). So I figured why don't I unit test my > benchmark, to ensure the assumptions I make hold true. > > During my unit tests I wanted to create a Blackhole to pass to one of the > benchmark methods. However, I found out that I'm not allowed to create a > Blackhole: IllegalStateException("Blackholes should not be instantiated > directly."). > > I can either do a nasty hack to create the Blackhole, or I'm requesting you > add a new static method, Blackhole.newBlackholeYesIdoKnowWhatImDoing(), or > similar. Since in my case, I don't care about the benchmarking, and I'm > calling this from a single junit thread, and just testing the correctness > of my benchmark code. Alternatively if you create a Blackhole interface, I > could easily create mock blackhole, for my testing. > > Suggestions? > thanks > Andrew > > P.S For the curious, my code lives here https://github.com/bramp/unsafe > under unsafe-benchmark. From me at bramp.net Fri Jun 19 20:21:41 2015 From: me at bramp.net (Andrew Brampton) Date: Fri, 19 Jun 2015 13:21:41 -0700 Subject: Creating a Blackhole In-Reply-To: References: Message-ID: Thanks Dawid, what I ended up doing was: https://github.com/bramp/unsafe/blob/master/unsafe-benchmark/src/test/java/net/bramp/unsafe/JMHHelper.java I would still like an official way to create a Blackhole, or be able to mock a Blackhole, without hacks. I didn't find a bug tracker where I could make this request. thanks Andrew On 18 June 2015 at 09:25, Dawid Weiss wrote: > This may not be the quickest solution... but you can postprocess your > bytecode after compilation (with asmlib, aspectj or whatever) and > remove blackhole calls. > > With AspectJ it's a relatively easy call pointcut and an "around" > advice that doesn't > propagate the call. You could replace the default compiler with ajc > and apply test-specific aspects during compilation. > > Dawid > > On Thu, Jun 18, 2015 at 6:17 PM, Andrew Brampton wrote: > > I have a complex benchmark, that takes >1 hour to run. I kept hitting an > > issue where the benchmark would crash during one of the setup/teardown > > phase (due to mistakes in my code). So I figured why don't I unit test my > > benchmark, to ensure the assumptions I make hold true. > > > > During my unit tests I wanted to create a Blackhole to pass to one of the > > benchmark methods. However, I found out that I'm not allowed to create a > > Blackhole: IllegalStateException("Blackholes should not be instantiated > > directly."). > > > > I can either do a nasty hack to create the Blackhole, or I'm requesting > you > > add a new static method, Blackhole.newBlackholeYesIdoKnowWhatImDoing(), > or > > similar. Since in my case, I don't care about the benchmarking, and I'm > > calling this from a single junit thread, and just testing the correctness > > of my benchmark code. Alternatively if you create a Blackhole interface, > I > > could easily create mock blackhole, for my testing. > > > > Suggestions? > > thanks > > Andrew > > > > P.S For the curious, my code lives here https://github.com/bramp/unsafe > > under unsafe-benchmark. > From aleksey.shipilev at oracle.com Mon Jun 22 09:27:43 2015 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Mon, 22 Jun 2015 12:27:43 +0300 Subject: Creating a Blackhole In-Reply-To: References: Message-ID: <5587D50F.10902@oracle.com> Hi Andrew, On 06/18/2015 07:17 PM, Andrew Brampton wrote: > I have a complex benchmark, that takes >1 hour to run. I kept hitting an > issue where the benchmark would crash during one of the setup/teardown > phase (due to mistakes in my code). So I figured why don't I unit test my > benchmark, to ensure the assumptions I make hold true. I understand the use case. However, mocking Blackhole seems not to be enough to test the generic JMH-driven benchmark? E.g. using other @State-s would require mocking them as well, including special {Benchmark,Iteration,Thread}Params, making sure all threads get their relevant instance of state object, @Setup/@TearDown run in the same order JMH does it, etc. Given that, would it be easier to put the Asserts straight into @Setup/@TearDown code, and run the benchmark with shorter data set? Our entire integration testing suite is driven like that: http://hg.openjdk.java.net/code-tools/jmh/file/1db18e73d21c/jmh-core-it/src/test/java/org/openjdk/jmh/it > I can either do a nasty hack to create the Blackhole, or I'm requesting you > add a new static method, Blackhole.newBlackholeYesIdoKnowWhatImDoing(), or > similar. Since in my case, I don't care about the benchmarking, and I'm > calling this from a single junit thread, and just testing the correctness > of my benchmark code. Alternatively if you create a Blackhole interface, I > could easily create mock blackhole, for my testing. Factory method lends itself for abuse: no matter how you call the method, people would eventually call it in some weird scenario, without reading the documentation. Interface is out of the question: Blackhole.consume invocation performance is very important. > Suggestions? a) (Boot-)classpath your own version of Blackhole class with proper public constructor, or do some other kind of bytecode magic. b) Ask us to provide a system property that disables the check in Blackhole constructor. Can your testing environment set the Java options? c) Ask us to ditch the exception from Blackhole constructor, and make the constructor private/protected instead. Then you can circumvent the protection offered for common use case by using Reflection with setAccessible(true)? > P.S For the curious, my code lives here https://github.com/bramp/unsafe > under unsafe-benchmark. 404. -Aleksey From aleksey.shipilev at oracle.com Wed Jun 24 09:24:29 2015 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Wed, 24 Jun 2015 09:24:29 +0000 Subject: hg: code-tools/jmh: 7901454: -prof perfasm should render the jumps Message-ID: <201506240924.t5O9OTbN027402@aojmv0008.oracle.com> Changeset: 7f20879715eb Author: shade Date: 2015-06-24 12:24 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/7f20879715eb 7901454: -prof perfasm should render the jumps ! jmh-core/src/main/java/org/openjdk/jmh/profile/AbstractPerfAsmProfiler.java From aleksey.shipilev at oracle.com Wed Jun 24 10:55:21 2015 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Wed, 24 Jun 2015 10:55:21 +0000 Subject: hg: code-tools/jmh: Make JMH compilable with JDK 6 again. Message-ID: <201506241055.t5OAtLAe015660@aojmv0008.oracle.com> Changeset: 4863dfc3e61e Author: shade Date: 2015-06-24 13:55 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/4863dfc3e61e Make JMH compilable with JDK 6 again. ! jmh-core/src/main/java/org/openjdk/jmh/profile/AbstractPerfAsmProfiler.java From skoehler at demandware.com Wed Jun 24 11:29:14 2015 From: skoehler at demandware.com (Steffen Koehler) Date: Wed, 24 Jun 2015 11:29:14 +0000 Subject: Use my own ResultFormat Message-ID: <6acd8fe36890474587cf51f9cb6f7aa5@DWU16EXCH2.int.demandware.com> Hi, it's simple to create an own ResultFormat class by implementing ResultFormat. But how can I use this class? The OptionBuilder offers only the possibility to add an ResultFormatType. Thanks Steffen This e-mail message and all attachments transmitted with it may contain privileged and/or confidential information intended solely for the use of the addressee(s). If the reader of this message is not the intended recipient, you are hereby notified that any reading, dissemination, distribution, copying, forwarding or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately and delete this message, all attachments and all copies and backups thereof. From aleksey.shipilev at oracle.com Wed Jun 24 14:14:59 2015 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Wed, 24 Jun 2015 17:14:59 +0300 Subject: Use my own ResultFormat In-Reply-To: <6acd8fe36890474587cf51f9cb6f7aa5@DWU16EXCH2.int.demandware.com> References: <6acd8fe36890474587cf51f9cb6f7aa5@DWU16EXCH2.int.demandware.com> Message-ID: <558ABB63.2020802@oracle.com> Hi Steffen, On 06/24/2015 02:29 PM, Steffen Koehler wrote: > it's simple to create an own ResultFormat class by implementing > ResultFormat. But how can I use this class? The OptionBuilder offers > only the possibility to add an ResultFormatType. I am trying to understand the use case for having a custom ResultFormat. Can you describe what you are trying to accomplish? If you are using API, then the benchmark results are already available from Runner.run call, you can feed them into your own formatter. If you are using CLI, then you would need some way to make JMH recognize the ResultFormat class, instantiate it, and feed the data there. Our pluggable profilers are like that, we can make ResultFormat that way as well. Thanks, -Aleksey From skoehler at demandware.com Wed Jun 24 14:28:37 2015 From: skoehler at demandware.com (Steffen Koehler) Date: Wed, 24 Jun 2015 14:28:37 +0000 Subject: Use my own ResultFormat In-Reply-To: <558ABB63.2020802@oracle.com> References: <6acd8fe36890474587cf51f9cb6f7aa5@DWU16EXCH2.int.demandware.com> <558ABB63.2020802@oracle.com> Message-ID: <01288a722c8446b5b16972bd585649f8@DWU16EXCH2.int.demandware.com> Hi Aleksey, thanks for the quick response. You're right, I have miss that the Runner.run returns the result collection, that will solve my use case, which is to add the result to a database. I was focused at the ResultFormat :-) Thanks Steffen -----Original Message----- From: Aleksey Shipilev [mailto:aleksey.shipilev at oracle.com] Sent: Mittwoch, 24. Juni 2015 16:15 To: Steffen Koehler; jmh-dev at openjdk.java.net Subject: Re: Use my own ResultFormat Hi Steffen, On 06/24/2015 02:29 PM, Steffen Koehler wrote: > it's simple to create an own ResultFormat class by implementing > ResultFormat. But how can I use this class? The OptionBuilder offers > only the possibility to add an ResultFormatType. I am trying to understand the use case for having a custom ResultFormat. Can you describe what you are trying to accomplish? If you are using API, then the benchmark results are already available from Runner.run call, you can feed them into your own formatter. If you are using CLI, then you would need some way to make JMH recognize the ResultFormat class, instantiate it, and feed the data there. Our pluggable profilers are like that, we can make ResultFormat that way as well. Thanks, -Aleksey This e-mail message and all attachments transmitted with it may contain privileged and/or confidential information intended solely for the use of the addressee(s). If the reader of this message is not the intended recipient, you are hereby notified that any reading, dissemination, distribution, copying, forwarding or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately and delete this message, all attachments and all copies and backups thereof. From zolyfarkas at yahoo.com Thu Jun 25 12:29:07 2015 From: zolyfarkas at yahoo.com (Zoltan Farkas) Date: Thu, 25 Jun 2015 08:29:07 -0400 Subject: Question about benchmark finish and non daemon threads Message-ID: <0F4EA1B4-F445-4F9A-BB0E-7B3601037BC1@yahoo.com> I see at the end of the benchmark: Would it be possible before to add some code to list the non daemon threads still running? would help with troubleshooting? let me know thank you ?Z From aleksey.shipilev at oracle.com Thu Jun 25 13:18:32 2015 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 25 Jun 2015 16:18:32 +0300 Subject: Question about benchmark finish and non daemon threads In-Reply-To: <0F4EA1B4-F445-4F9A-BB0E-7B3601037BC1@yahoo.com> References: <0F4EA1B4-F445-4F9A-BB0E-7B3601037BC1@yahoo.com> Message-ID: <558BFFA8.40000@oracle.com> Hi Zoltan, On 06/25/2015 03:29 PM, Zoltan Farkas wrote: > I see at the end of the benchmark: > > running threads? Waiting 14 seconds more?> > > Would it be possible before to add some code to list the non daemon > threads still running? would help with troubleshooting? Is there a reason you can't use "jstack" against the forked VM to figure this out? -Aleksey. From zolyfarkas at yahoo.com Thu Jun 25 15:05:06 2015 From: zolyfarkas at yahoo.com (Zoltan Farkas) Date: Thu, 25 Jun 2015 11:05:06 -0400 Subject: Question about benchmark finish and non daemon threads In-Reply-To: <558BFFA8.40000@oracle.com> References: <0F4EA1B4-F445-4F9A-BB0E-7B3601037BC1@yahoo.com> <558BFFA8.40000@oracle.com> Message-ID: <8E36946A-CD6C-429F-9C9A-C6F8F7DE4A10@yahoo.com> I run benchmarks continuously, by the time I look at the results the process is long gone. When this happens Java flight recorder data is not persisted, which is inconvenient. To help with this, I am printing out all non daemon threads on benchmark teardown... Thought it would be useful of jmh would provide this detail in case the benchmark is not finishing properly... --z > On Jun 25, 2015, at 9:18 AM, Aleksey Shipilev wrote: > > Hi Zoltan, > >> On 06/25/2015 03:29 PM, Zoltan Farkas wrote: >> I see at the end of the benchmark: >> >> > running threads? Waiting 14 seconds more?> >> >> Would it be possible before to add some code to list the non daemon >> threads still running? would help with troubleshooting? > > Is there a reason you can't use "jstack" against the forked VM to figure > this out? > > -Aleksey. > From aleksey.shipilev at oracle.com Thu Jun 25 15:15:22 2015 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 25 Jun 2015 18:15:22 +0300 Subject: Question about benchmark finish and non daemon threads In-Reply-To: <8E36946A-CD6C-429F-9C9A-C6F8F7DE4A10@yahoo.com> References: <0F4EA1B4-F445-4F9A-BB0E-7B3601037BC1@yahoo.com> <558BFFA8.40000@oracle.com> <8E36946A-CD6C-429F-9C9A-C6F8F7DE4A10@yahoo.com> Message-ID: <558C1B0A.10700@oracle.com> Got it, thanks. I assumed JMH process is stuck waiting for your intervention anyway. But now I remember we kill off the JMH process once the timeout is reached. Submitted: https://bugs.openjdk.java.net/browse/CODETOOLS-7901456 -Aleksey On 06/25/2015 06:05 PM, Zoltan Farkas wrote: > I run benchmarks continuously, by the time I look at the results the > process is long gone. > > When this happens Java flight recorder data is not persisted, which > is inconvenient. > > To help with this, I am printing out all non daemon threads on > benchmark teardown... > > Thought it would be useful of jmh would provide this detail in case > the benchmark is not finishing properly... > > --z > > >> On Jun 25, 2015, at 9:18 AM, Aleksey Shipilev >> wrote: >> >> Hi Zoltan, >> >>> On 06/25/2015 03:29 PM, Zoltan Farkas wrote: I see at the end of >>> the benchmark: >>> >>> >> running threads? Waiting 14 seconds more?> >>> >>> Would it be possible before to add some code to list the non >>> daemon threads still running? would help with troubleshooting? >> >> Is there a reason you can't use "jstack" against the forked VM to >> figure this out? >> >> -Aleksey. >> From aleksey.shipilev at oracle.com Thu Jun 25 18:00:34 2015 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 25 Jun 2015 18:00:34 +0000 Subject: hg: code-tools/jmh: 7901456: JMH should print stuck threads before forcefully exiting Message-ID: <201506251800.t5PI0YX0011708@aojmv0008.oracle.com> Changeset: 9440a466ee8c Author: shade Date: 2015-06-25 20:58 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/9440a466ee8c 7901456: JMH should print stuck threads before forcefully exiting ! jmh-core/src/main/java/org/openjdk/jmh/runner/ForkedMain.java From aleksey.shipilev at oracle.com Thu Jun 25 18:03:24 2015 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 25 Jun 2015 21:03:24 +0300 Subject: Question about benchmark finish and non daemon threads In-Reply-To: <558C1B0A.10700@oracle.com> References: <0F4EA1B4-F445-4F9A-BB0E-7B3601037BC1@yahoo.com> <558BFFA8.40000@oracle.com> <8E36946A-CD6C-429F-9C9A-C6F8F7DE4A10@yahoo.com> <558C1B0A.10700@oracle.com> Message-ID: <558C426C.8040604@oracle.com> On 25.06.2015 18:15, Aleksey Shipilev wrote: > Got it, thanks. I assumed JMH process is stuck waiting for your > intervention anyway. But now I remember we kill off the JMH process once > the timeout is reached. > > Submitted: > https://bugs.openjdk.java.net/browse/CODETOOLS-7901456 Okay, done: http://hg.openjdk.java.net/code-tools/jmh/rev/9440a466ee8c -Aleksey. From aleksey.shipilev at oracle.com Thu Jun 25 18:21:22 2015 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 25 Jun 2015 18:21:22 +0000 Subject: hg: code-tools/jmh: Add two integration tests to verify forceful shutdown works. Message-ID: <201506251821.t5PILMAB018904@aojmv0008.oracle.com> Changeset: bc6354c5d6ce Author: shade Date: 2015-06-25 21:20 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/bc6354c5d6ce Add two integration tests to verify forceful shutdown works. + jmh-core-it/src/test/java/org/openjdk/jmh/it/errors/ForkedStuckShutdownHookTest.java + jmh-core-it/src/test/java/org/openjdk/jmh/it/errors/ForkedStuckThreadTest.java From zolyfarkas at yahoo.com Thu Jun 25 20:03:51 2015 From: zolyfarkas at yahoo.com (Zoltan Farkas) Date: Thu, 25 Jun 2015 16:03:51 -0400 Subject: Question about benchmark finish and non daemon threads In-Reply-To: <558C426C.8040604@oracle.com> References: <0F4EA1B4-F445-4F9A-BB0E-7B3601037BC1@yahoo.com> <558BFFA8.40000@oracle.com> <8E36946A-CD6C-429F-9C9A-C6F8F7DE4A10@yahoo.com> <558C1B0A.10700@oracle.com> <558C426C.8040604@oracle.com> Message-ID: <42A9ED81-3BB3-4C54-B7E4-869EF1510FDA@yahoo.com> Great! Thank you! ?Z > On Jun 25, 2015, at 2:03 PM, Aleksey Shipilev wrote: > > On 25.06.2015 18:15, Aleksey Shipilev wrote: >> Got it, thanks. I assumed JMH process is stuck waiting for your >> intervention anyway. But now I remember we kill off the JMH process once >> the timeout is reached. >> >> Submitted: >> https://bugs.openjdk.java.net/browse/CODETOOLS-7901456 > > Okay, done: > http://hg.openjdk.java.net/code-tools/jmh/rev/9440a466ee8c > > -Aleksey. > From aleksey.shipilev at oracle.com Thu Jun 25 21:13:27 2015 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Fri, 26 Jun 2015 00:13:27 +0300 Subject: JMH 1.10.1 Message-ID: <558C6EF7.2090709@oracle.com> Hi, JMH 1.10.1 patch update is released and available at Maven Central (props to Evgeny Mandrikov, as usual). It includes two changes: * "perfasm" now draws jumps in its output, like this: http://cr.openjdk.java.net/~shade/jmh/perfasm-branch-jumps.png This greatly aids reading the generated code, especially in the presence of large loops. Two new perfasm options control what jumps are drawn: "drawIntraJumps" and "drawInterJumps". By default, we draw intra-region jumps only, since the hot jumps are what we usually interested in. Inter-region jumps are abundant, and may clutter the output; enable them as you see fit. Tracking: https://bugs.openjdk.java.net/browse/CODETOOLS-7901454 * JMH now prints out stuck non-daemon threads while counting down the time to forcefully exit the VM. This should help to diagnose the runaway threads when running the benchmarks remotely, or in the course of continuous testing. Tracking: https://bugs.openjdk.java.net/browse/CODETOOLS-7901456 Enjoy! Thanks, -Aleksey From aleksey.shipilev at oracle.com Thu Jun 25 21:15:57 2015 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 25 Jun 2015 21:15:57 +0000 Subject: hg: code-tools/jmh: 3 new changesets Message-ID: <201506252115.t5PLFvTc015329@aojmv0008.oracle.com> Changeset: d79c787a80c0 Author: shade Date: 2015-06-25 22:31 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/d79c787a80c0 JMH v1.10.1. ! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml ! jmh-archetypes/pom.xml ! jmh-core-benchmarks/pom.xml ! jmh-core-ct/pom.xml ! jmh-core-it/pom.xml ! jmh-core/pom.xml ! jmh-generator-annprocess/pom.xml ! jmh-generator-asm/pom.xml ! jmh-generator-bytecode/pom.xml ! jmh-generator-reflection/pom.xml ! jmh-samples/pom.xml ! pom.xml Changeset: fa12dc9b7c81 Author: shade Date: 2015-06-25 22:31 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/fa12dc9b7c81 Added tag 1.10.1 for changeset d79c787a80c0 ! .hgtags Changeset: 29564342b418 Author: shade Date: 2015-06-25 22:31 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/29564342b418 Continue in 1.11-SNAPSHOT. ! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml ! jmh-archetypes/pom.xml ! jmh-core-benchmarks/pom.xml ! jmh-core-ct/pom.xml ! jmh-core-it/pom.xml ! jmh-core/pom.xml ! jmh-generator-annprocess/pom.xml ! jmh-generator-asm/pom.xml ! jmh-generator-bytecode/pom.xml ! jmh-generator-reflection/pom.xml ! jmh-samples/pom.xml ! pom.xml