From Matthias.Huber at novatec-gmbh.de Thu Jan 7 13:18:28 2016 From: Matthias.Huber at novatec-gmbh.de (Huber Matthias) Date: Thu, 7 Jan 2016 13:18:28 +0000 Subject: Printing of batchsize + single execution time in Results for Batching Message-ID: <1452172709500.67586@novatec-gmbh.de> Hello, I do some benchmarking of non-steady-state benchmarks. Therefore, I use the batching mechanism as described in the JMH Samples. The benchmarks will be run with different batch sizes (controlled with -wbs/bs). Afterwards I want to process the results (csv) most likely with R. Right now the CSV output (or outputs in general) do not printout the batchsize nor a time for a single execution (= score(error) / batchsize). I want to have this figures for chart generation in R. I could include the batchsize into the CSV File name and calculate the single execution time in R. However, I don't like the dependency to the filename. I want to see within the results which batchsize has been used. As I haven't found an option to enable this I have extended the 'XSVResultFormat' to printout these figures in case the Mode SingleShot time with a batchsize > 1 is used. Is there any other way to easily accomplish the same without extending JMH (preferred way) like using the JMH Java API? If not, I could provide an extension if this is useful in general. Mit freundlichen Gr??en / Kind Regards Matthias Huber Senior Consultant CA Application Performance Management NOVATEC // Consulting GmbH Dieselstra?e 18/1 // D-70771 Leinfelden-Echterdingen phone: +49 711 22040-700 fax: +49 711 22040-899 mobile: +49 170 5682609 e-mail: matthias.huber at novatec-gmbh.de web: www.novatec-gmbh.de Sitz der Gesellschaft: Waldenbuch // Handelsregister: Stuttgart - HRB 739078 Gesch?ftsf?hrer: Stefan Bleicher // Hans-Dieter Brenner // Konrad Pfeilsticker // Michael Schuchart inspectIT // freies Java Performance und Monitoringwerkzeug (www.inspectit.rocks) From aleksey.shipilev at oracle.com Mon Jan 11 12:47:30 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Mon, 11 Jan 2016 15:47:30 +0300 Subject: Printing of batchsize + single execution time in Results for Batching In-Reply-To: <1452172709500.67586@novatec-gmbh.de> References: <1452172709500.67586@novatec-gmbh.de> Message-ID: <5693A462.2090306@oracle.com> Hi, On 01/07/2016 04:18 PM, Huber Matthias wrote: > Is there any other way to easily accomplish the same without > extending JMH (preferred way) like using the JMH Java API? If not, I > could provide an extension if this is useful in general. No, there is no other way than using the Java API. It would always be the case that "out of the box" scenarios are not covering all the possible corner cases like yours. Extending XSV outputs to include more benchmark metadata is troublesome, since it breaks the format (and therefore, the tools that depend on it). But, I think we can extend JSON output to include more benchmark metadata, including batch sizes. R should be able to pull in JSON dataset? Contributions welcome ;) Thanks, -Aleksey From aleksey.shipilev at oracle.com Mon Jan 11 12:54:23 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Mon, 11 Jan 2016 15:54:23 +0300 Subject: Sample time with "inverted" histogram? In-Reply-To: <87D45DCE-26CA-4151-BF97-E32AF1B769DC@gmail.com> References: <87D45DCE-26CA-4151-BF97-E32AF1B769DC@gmail.com> Message-ID: <5693A5FF.4040209@oracle.com> Hi Chris, On 11/27/2015 07:08 PM, Chris Vest wrote: > I sometimes find myself wishing the sample time benchmark mode had an ?inverse? histogram, because I find that this makes it easier to spot when response times are modal. > > To demonstrate what I mean, the current histogram output looks like this: (numbers are fictive) > Samples, N = 3184 > mean = 2.531 ?(99.9%) 0.014 us/op > min = 0.119 us/op > p( 0.0000) = 0.119 us/op > p(50.0000) = 2.420 us/op > p(90.0000) = 2.644 us/op > p(95.0000) = 2.816 us/op > p(99.0000) = 4.408 us/op > p(99.9000) = 13.152 us/op > p(99.9900) = 36.901 us/op > p(99.9990) = 254.240 us/op > p(99.9999) = 260.864 us/op > max = 260.864 us/op > > And I would like a configuration option that would make it look more like this: (numbers even more fictive) > Samples, N = 3184 > mean = 2.531 ?(99.9%) 0.014 us/op > min = 0.119 us/op > us/op = samples added in increment > 1 = 1081 > 2 = 194 > 4 = 959 > 8 = 357 > 16 = 3 > 32 = 3 > 64 = 118 > 128 = 410 > 256 = 58 > 512 = 1 > This way, I can more easily see that the numbers in my made-up > example fall into 2 or 3 groups. When I have needed something like > this, I have thus far reached for HdrHistogram and its > LogarithmicIterator, and the > HistogramIterationValue.getCountAddedInThisIterationStep method. It might be worthwhile to explore. JMH's SampleBuffer is recording the HDR data, so those histogram values are available. Therefore it is a question of presentation: maybe print the histogram alongside with quantiles? Thanks, -Aleksey From aleksey.shipilev at oracle.com Mon Jan 11 13:17:32 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Mon, 11 Jan 2016 16:17:32 +0300 Subject: Benchmarking with heavily-assymetric threads? (full message) In-Reply-To: <564DC6B7.6000806@serebryakov.spb.ru> References: <1279128937.20151114195239@serebryakov.spb.ru> <1056521024.20151114195956@serebryakov.spb.ru> <5647881B.2050108@serebryakov.spb.ru> <564DA132.6070803@azulsystems.com> <564DC6B7.6000806@serebryakov.spb.ru> Message-ID: <5693AB6C.40906@oracle.com> Hi Lev, Sorry for a very late reply! On 11/19/2015 03:55 PM, Lev Serebryakov wrote: > I'm using @Group("name") feature of JMH to start different threads in > one benchmark, I don't start threads by myself. But I'm blocking > benchmarked methods, as I try to benchmark different implementations > and strategies for blocking! > > Benchmark is rather simple (see at the end of message complete > example, you could not run it as you don't have my buffer > implementation, but it is straightforward *blocking* ring buffer for > single producer and single consumer). > > Now I'm understanding, that there is one major problem: if one of > group threads stops slightly before other one (perfectly possible), > other benchmark method could block forever (till JMH timeout), because > it will not have data to "consume" (or space in buffer to "produce"). Unfortunately, there is no way to properly solve this. JMH does not "equalize" the number of benchmark method calls per subtests in most measurement modes. Therefore, if you need the rendezvous tests that should be perfectly matched, you are in the bind. When faced with the problems like these, it is very instructive to ask yourself: how would you do this without JMH? In other words, how exactly would you want to measure things? There are few approaches: a) Count reads and writes in both threads, and run until both counts are matched. If readers and writers are symmetrical (heavily unlikely), then you just run until time expires, and pray all pairs rendezvous perfectly well. If they are not, then a hard cap on reader/writer counts should be imposed, otherwise it would never converge. This can be simulated with SingleShot mode with large batch sizes [1]. b) Run without counting, and shutdown the parties in orderly fashion. E.g. shutdown readers first, then shutdown writers -- that will avoid the trouble you are describing. But, this trick cannot be applied consistently. I.e. if the queue/ring buffer storage is bounded and writers can block, you have to shutdown *writers* first. Oops. c) Run without counting, but make readers/writers to respond to interrupts, and break away gracefully once interrupt is caught. This is what JMHSample_30_Interrupts [2] sample tries to show. Those "interrupted" messages is JMH trying to communicate to workload that time is up, and workload should exit @Benchmark right now. (Technical detail: interrupts are usually received in ramp-down catchup iterations, and should have no effect on measurement). Thanks, -Aleksey [1] http://hg.openjdk.java.net/code-tools/jmh/file/bcec9a03787f/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_26_BatchSize.java [2] http://hg.openjdk.java.net/code-tools/jmh/file/bcec9a03787f/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_30_Interrupts.java From mr.chrisvest at gmail.com Mon Jan 11 15:05:01 2016 From: mr.chrisvest at gmail.com (Chris Vest) Date: Mon, 11 Jan 2016 16:05:01 +0100 Subject: Sample time with "inverted" histogram? In-Reply-To: <5693A5FF.4040209@oracle.com> References: <87D45DCE-26CA-4151-BF97-E32AF1B769DC@gmail.com> <5693A5FF.4040209@oracle.com> Message-ID: <2A6F4F06-D033-4DF8-AE12-2BFE411FA37D@gmail.com> > On 11 Jan 2016, at 13:54, Aleksey Shipilev wrote: > > It might be worthwhile to explore. JMH's SampleBuffer is recording the > HDR data, so those histogram values are available. Therefore it is a > question of presentation: maybe print the histogram alongside with > quantiles? You meaning putting both presentations side by side in the print out? That would work for me. Cheers, Chris From Matthias.Huber at novatec-gmbh.de Mon Jan 11 16:06:22 2016 From: Matthias.Huber at novatec-gmbh.de (Huber Matthias) Date: Mon, 11 Jan 2016 16:06:22 +0000 Subject: Printing of batchsize + single execution time in Results for Batching In-Reply-To: <5693A462.2090306@oracle.com> References: <1452172709500.67586@novatec-gmbh.de>, <5693A462.2090306@oracle.com> Message-ID: <1452528385901.4269@novatec-gmbh.de> Hello Aleksey, I see your point with breaking the XSV format and its impact. I will have a look at the Java API to deal with this corner case. Right now I think that only the batch size is interesting for me to distinguish the results based on the used batch size from each other. Using the calculated time for one execution (total avg time per batch / batch size) could be dangerous for comparison. Comparing two values with each other that are calculated by dividing an average time by different batch sizes feels incorrect to me. On 01/11/2016 01:47 PM, Shipilev Aleksey wrote: < But, I think we can extend JSON output to include more benchmark < metadata, including batch sizes. R should be able to pull in JSON dataset? I haven't used the JSON DataSet so far. It's worth to have a look at it and its interoperability with R. I might come back with a contribution. Thanks, Matthias From alun at badgerous.net Tue Jan 12 17:55:52 2016 From: alun at badgerous.net (Alun Evans) Date: Tue, 12 Jan 2016 09:55:52 -0800 Subject: Question on jmh.lock Message-ID: <39A95014-90EE-4274-B1C1-0842A4305E4D@badgerous.net> Hello, Firstly, thanks for all the great work on JMH, it?s really useful for me. I?ve got a question about this fix: https://bugs.openjdk.java.net/browse/CODETOOLS-7901013 http://hg.openjdk.java.net/code-tools/jmh/file/bcec9a03787f/jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java#l63 This java.io.tmp + jmh.lock doesn?t seemed to be deleted at the end of a test run, meaning that when a different user comes to run the test, the file already exists. I was debating setting `java.io.tmp` to something else, but I think that the goal is that only 1 user per machine should really be running a JMH test at the same time, so it?s probably just better to cleanup the lock file ? thanks, A. -- Alun Evans From aleksey.shipilev at oracle.com Tue Jan 12 18:46:53 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Tue, 12 Jan 2016 21:46:53 +0300 Subject: Question on jmh.lock In-Reply-To: <39A95014-90EE-4274-B1C1-0842A4305E4D@badgerous.net> References: <39A95014-90EE-4274-B1C1-0842A4305E4D@badgerous.net> Message-ID: <56954A1D.5020406@oracle.com> Hi, It's unclear if you actually run into the problem with this mechanism? On 01/12/2016 08:55 PM, Alun Evans wrote: > This java.io.tmp + jmh.lock doesn?t seemed to be deleted at the end > of a test run, meaning that when a different user comes to run the > test, the file already exists. It's irrelevant that a file exists, what matters is that we are java.nio.channels.FileLock-ing it. Note that this is different from "file locking" that shell scripts do: write out the tombstone file, then remove it at the end; plus always check for the existence of the tombstone file on entry. Therefore, it is irrelevant that jmh.lock file is kept after JMH exits. If that was a problem, then consecutive JMH invocation by the same user would not also work. > I was debating setting `java.io.tmp` to something else, but I think > that the goal is that only 1 user per machine should really be > running a JMH test at the same time, so it?s probably just better to > cleanup the lock file ? The intent was to disallow accidental JMH session when other session is already running (think running JMH from the IDE). The side effect of disallowing concurrent JMH runs on the entire machine is a nice special case, and I think we should continue doing that. Thanks, -Aleksey From alun at badgerous.net Tue Jan 12 21:19:37 2016 From: alun at badgerous.net (Alun Evans) Date: Tue, 12 Jan 2016 13:19:37 -0800 Subject: Question on jmh.lock In-Reply-To: <56954A1D.5020406@oracle.com> References: <39A95014-90EE-4274-B1C1-0842A4305E4D@badgerous.net> <56954A1D.5020406@oracle.com> Message-ID: <80742529-CA11-4213-9798-F98D437EA709@badgerous.net> Hi Aleksey, > On 12 Jan 2016, at 10:46, Aleksey Shipilev wrote: > > Hi, > > It's unclear if you actually run into the problem with this mechanism? Sorry, yes I have, I?m hitting a permissions problem: [alun at poweredge ~]$ ll /tmp/jmh.lock -rw-rw-r--. 1 misha misha 0 Jan 11 20:26 /tmp/jmh.lock [alun at poweredge ~]$ rm /tmp/jmh.lock rm: remove write-protected regular empty file `/tmp/jmh.lock'? y rm: cannot remove `/tmp/jmh.lock': Operation not permitted [alun at poweredge ~]$ [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running com.xockets.InterfaceBenchmark [INFO] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.081 sec <<< FAILURE! - in com.xockets.InterfaceBenchmark [INFO] run(com.xockets.InterfaceBenchmark) Time elapsed: 0.079 sec <<< ERROR! [INFO] org.openjdk.jmh.runner.RunnerException: ERROR: Exception while trying to acquire the JMH lock (/tmp/jmh.lock): /tmp/jmh.lock (Permission denied), exiting. Use -Djmh.ignoreLock=true to forcefully continue. [INFO] at org.openjdk.jmh.runner.Runner.run(Runner.java:209) [INFO] at com.xockets.AbstractBenchmark.run(AbstractBenchmark.java:92) [INFO] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [INFO] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > On 01/12/2016 08:55 PM, Alun Evans wrote: >> This java.io.tmp + jmh.lock doesn?t seemed to be deleted at the end >> of a test run, meaning that when a different user comes to run the >> test, the file already exists. > > It's irrelevant that a file exists, what matters is that we are > java.nio.channels.FileLock-ing it. Note that this is different from > "file locking" that shell scripts do: write out the tombstone file, then > remove it at the end; plus always check for the existence of the > tombstone file on entry. > > Therefore, it is irrelevant that jmh.lock file is kept after JMH exits. > If that was a problem, then consecutive JMH invocation by the same user > would not also work. I think the permissions on the file aren?t right for a subsequent different user to be able to FileLock it? >> I was debating setting `java.io.tmp` to something else, but I think >> that the goal is that only 1 user per machine should really be >> running a JMH test at the same time, so it?s probably just better to >> cleanup the lock file ? > > The intent was to disallow accidental JMH session when other session is > already running (think running JMH from the IDE). The side effect of > disallowing concurrent JMH runs on the entire machine is a nice special > case, and I think we should continue doing that. Yes, it does make sense to me. I?ve been developing inside a VM, but then when we want *actual* numbers, we need to run it on a bare metal machine. thanks, A. -- Alun Evans From aleksey.shipilev at oracle.com Tue Jan 12 22:33:28 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Wed, 13 Jan 2016 01:33:28 +0300 Subject: Question on jmh.lock In-Reply-To: <80742529-CA11-4213-9798-F98D437EA709@badgerous.net> References: <39A95014-90EE-4274-B1C1-0842A4305E4D@badgerous.net> <56954A1D.5020406@oracle.com> <80742529-CA11-4213-9798-F98D437EA709@badgerous.net> Message-ID: <56957F38.40600@oracle.com> On 01/13/2016 12:19 AM, Alun Evans wrote: >> On 12 Jan 2016, at 10:46, Aleksey Shipilev wrote: >> It's unclear if you actually run into the problem with this mechanism? > > Sorry, yes I have, I?m hitting a permissions problem: > > [alun at poweredge ~]$ ll /tmp/jmh.lock > -rw-rw-r--. 1 misha misha 0 Jan 11 20:26 /tmp/jmh.lock > [alun at poweredge ~]$ rm /tmp/jmh.lock > rm: remove write-protected regular empty file `/tmp/jmh.lock'? y > rm: cannot remove `/tmp/jmh.lock': Operation not permitted > [alun at poweredge ~]$ > > [INFO] ------------------------------------------------------- > [INFO] T E S T S > [INFO] ------------------------------------------------------- > [INFO] Running com.xockets.InterfaceBenchmark > [INFO] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.081 sec <<< FAILURE! - in com.xockets.InterfaceBenchmark > [INFO] run(com.xockets.InterfaceBenchmark) Time elapsed: 0.079 sec <<< ERROR! > [INFO] org.openjdk.jmh.runner.RunnerException: ERROR: Exception while trying to acquire the JMH lock (/tmp/jmh.lock): /tmp/jmh.lock (Permission denied), exiting. Use -Djmh.ignoreLock=true to forcefully continue. > [INFO] at org.openjdk.jmh.runner.Runner.run(Runner.java:209) > [INFO] at com.xockets.AbstractBenchmark.run(AbstractBenchmark.java:92) > [INFO] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > [INFO] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) I see. That's indeed problematic! The file should indeed be world-writeable for this to work properly. I'll take a look how to fix/workaround this. Thanks, -Aleksey From aleksey.shipilev at oracle.com Tue Jan 12 22:49:54 2016 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Tue, 12 Jan 2016 22:49:54 +0000 Subject: hg: code-tools/jmh: 7901577: JMH file lock works only for the current user, prevents any future runs by other users Message-ID: <201601122249.u0CMntO6015965@aojmv0008.oracle.com> Changeset: 4efff1f3f56f Author: shade Date: 2016-01-13 01:49 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/4efff1f3f56f 7901577: JMH file lock works only for the current user, prevents any future runs by other users ! jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java From aleksey.shipilev at oracle.com Tue Jan 12 22:51:20 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Wed, 13 Jan 2016 01:51:20 +0300 Subject: Question on jmh.lock In-Reply-To: <56957F38.40600@oracle.com> References: <39A95014-90EE-4274-B1C1-0842A4305E4D@badgerous.net> <56954A1D.5020406@oracle.com> <80742529-CA11-4213-9798-F98D437EA709@badgerous.net> <56957F38.40600@oracle.com> Message-ID: <56958368.4060709@oracle.com> On 01/13/2016 01:33 AM, Aleksey Shipilev wrote: > On 01/13/2016 12:19 AM, Alun Evans wrote: >>> On 12 Jan 2016, at 10:46, Aleksey Shipilev wrote: >>> It's unclear if you actually run into the problem with this mechanism? >> >> Sorry, yes I have, I?m hitting a permissions problem: >> >> [alun at poweredge ~]$ ll /tmp/jmh.lock >> -rw-rw-r--. 1 misha misha 0 Jan 11 20:26 /tmp/jmh.lock >> [alun at poweredge ~]$ rm /tmp/jmh.lock >> rm: remove write-protected regular empty file `/tmp/jmh.lock'? y >> rm: cannot remove `/tmp/jmh.lock': Operation not permitted >> [alun at poweredge ~]$ >> >> [INFO] ------------------------------------------------------- >> [INFO] T E S T S >> [INFO] ------------------------------------------------------- >> [INFO] Running com.xockets.InterfaceBenchmark >> [INFO] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.081 sec <<< FAILURE! - in com.xockets.InterfaceBenchmark >> [INFO] run(com.xockets.InterfaceBenchmark) Time elapsed: 0.079 sec <<< ERROR! >> [INFO] org.openjdk.jmh.runner.RunnerException: ERROR: Exception while trying to acquire the JMH lock (/tmp/jmh.lock): /tmp/jmh.lock (Permission denied), exiting. Use -Djmh.ignoreLock=true to forcefully continue. >> [INFO] at org.openjdk.jmh.runner.Runner.run(Runner.java:209) >> [INFO] at com.xockets.AbstractBenchmark.run(AbstractBenchmark.java:92) >> [INFO] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> [INFO] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > I see. That's indeed problematic! The file should indeed be > world-writeable for this to work properly. I'll take a look how to > fix/workaround this. That seems simple: https://bugs.openjdk.java.net/browse/CODETOOLS-7901577 Can you test? The owner would need to run JMH at least once to set the permissions right, or remove the lock file completely. -Aleksey From ecki at zusammenkunft.net Tue Jan 12 23:02:32 2016 From: ecki at zusammenkunft.net (ecki at zusammenkunft.net) Date: Wed, 13 Jan 2016 00:02:32 +0100 Subject: Question on jmh.lock In-Reply-To: <56957F38.40600@oracle.com> References: <39A95014-90EE-4274-B1C1-0842A4305E4D@badgerous.net> <56954A1D.5020406@oracle.com> <80742529-CA11-4213-9798-F98D437EA709@badgerous.net> <56957F38.40600@oracle.com> Message-ID: Hello, World-writeable files are a problem on multi user systems (for quota reasons). At least I would not chmod the file by default. Maybe printing some instructions on how to chmod or delete the file? (And an option to use a user-specific lock?) Bernd -- http://bernd.eckenfels.net -----Original Message----- From: Aleksey Shipilev To: Alun Evans Cc: jmh-dev at openjdk.java.net Sent: Di., 12 Jan. 2016 23:44 Subject: Re: Question on jmh.lock On 01/13/2016 12:19 AM, Alun Evans wrote: >> On 12 Jan 2016, at 10:46, Aleksey Shipilev wrote: >> It's unclear if you actually run into the problem with this mechanism? > > Sorry, yes I have, I?m hitting a permissions problem: > > [alun at poweredge ~]$ ll /tmp/jmh.lock > -rw-rw-r--. 1 misha misha 0 Jan 11 20:26 /tmp/jmh.lock > [alun at poweredge ~]$ rm /tmp/jmh.lock > rm: remove write-protected regular empty file `/tmp/jmh.lock'? y > rm: cannot remove `/tmp/jmh.lock': Operation not permitted > [alun at poweredge ~]$ > > [INFO] ------------------------------------------------------- > [INFO] T E S T S > [INFO] ------------------------------------------------------- > [INFO] Running com.xockets.InterfaceBenchmark > [INFO] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.081 sec <<< FAILURE! - in com.xockets.InterfaceBenchmark > [INFO] run(com.xockets.InterfaceBenchmark) Time elapsed: 0.079 sec <<< ERROR! > [INFO] org.openjdk.jmh.runner.RunnerException: ERROR: Exception while trying to acquire the JMH lock (/tmp/jmh.lock): /tmp/jmh.lock (Permission denied), exiting. Use -Djmh.ignoreLock=true to forcefully continue. > [INFO] at org.openjdk.jmh.runner.Runner.run(Runner.java:209) > [INFO] at com.xockets.AbstractBenchmark.run(AbstractBenchmark.java:92) > [INFO] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > [INFO] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) I see. That's indeed problematic! The file should indeed be world-writeable for this to work properly. I'll take a look how to fix/workaround this. Thanks, -Aleksey From alun at badgerous.net Tue Jan 12 23:06:28 2016 From: alun at badgerous.net (Alun Evans) Date: Tue, 12 Jan 2016 15:06:28 -0800 Subject: Question on jmh.lock In-Reply-To: <56958368.4060709@oracle.com> References: <39A95014-90EE-4274-B1C1-0842A4305E4D@badgerous.net> <56954A1D.5020406@oracle.com> <80742529-CA11-4213-9798-F98D437EA709@badgerous.net> <56957F38.40600@oracle.com> <56958368.4060709@oracle.com> Message-ID: <6330ED57-AFA7-4C86-BFC9-5EBD2C078F0F@badgerous.net> > On 12 Jan 2016, at 14:51, Aleksey Shipilev wrote: > > On 01/13/2016 01:33 AM, Aleksey Shipilev wrote: >> On 01/13/2016 12:19 AM, Alun Evans wrote: >>>> On 12 Jan 2016, at 10:46, Aleksey Shipilev wrote: >>>> It's unclear if you actually run into the problem with this mechanism? >>> >>> Sorry, yes I have, I?m hitting a permissions problem: >>> >>> [alun at poweredge ~]$ ll /tmp/jmh.lock >>> -rw-rw-r--. 1 misha misha 0 Jan 11 20:26 /tmp/jmh.lock >>> [alun at poweredge ~]$ rm /tmp/jmh.lock >>> rm: remove write-protected regular empty file `/tmp/jmh.lock'? y >>> rm: cannot remove `/tmp/jmh.lock': Operation not permitted >>> [alun at poweredge ~]$ >>> >>> [INFO] ------------------------------------------------------- >>> [INFO] T E S T S >>> [INFO] ------------------------------------------------------- >>> [INFO] Running com.xockets.InterfaceBenchmark >>> [INFO] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.081 sec <<< FAILURE! - in com.xockets.InterfaceBenchmark >>> [INFO] run(com.xockets.InterfaceBenchmark) Time elapsed: 0.079 sec <<< ERROR! >>> [INFO] org.openjdk.jmh.runner.RunnerException: ERROR: Exception while trying to acquire the JMH lock (/tmp/jmh.lock): /tmp/jmh.lock (Permission denied), exiting. Use -Djmh.ignoreLock=true to forcefully continue. >>> [INFO] at org.openjdk.jmh.runner.Runner.run(Runner.java:209) >>> [INFO] at com.xockets.AbstractBenchmark.run(AbstractBenchmark.java:92) >>> [INFO] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> [INFO] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> >> I see. That's indeed problematic! The file should indeed be >> world-writeable for this to work properly. I'll take a look how to >> fix/workaround this. > > That seems simple: > https://bugs.openjdk.java.net/browse/CODETOOLS-7901577 > > Can you test? The owner would need to run JMH at least once to set the > permissions right, or remove the lock file completely. Sure, just following "Building the "Bleeding Edge? JMH? http://openjdk.java.net/projects/code-tools/jmh/ A. -- Alun Evans From alun at badgerous.net Tue Jan 12 23:20:28 2016 From: alun at badgerous.net (Alun Evans) Date: Tue, 12 Jan 2016 15:20:28 -0800 Subject: Question on jmh.lock In-Reply-To: <6330ED57-AFA7-4C86-BFC9-5EBD2C078F0F@badgerous.net> References: <39A95014-90EE-4274-B1C1-0842A4305E4D@badgerous.net> <56954A1D.5020406@oracle.com> <80742529-CA11-4213-9798-F98D437EA709@badgerous.net> <56957F38.40600@oracle.com> <56958368.4060709@oracle.com> <6330ED57-AFA7-4C86-BFC9-5EBD2C078F0F@badgerous.net> Message-ID: > On 12 Jan 2016, at 15:06, Alun Evans wrote: > >> >> On 12 Jan 2016, at 14:51, Aleksey Shipilev wrote: >> >> On 01/13/2016 01:33 AM, Aleksey Shipilev wrote: >>> On 01/13/2016 12:19 AM, Alun Evans wrote: >>>>> On 12 Jan 2016, at 10:46, Aleksey Shipilev wrote: >>>>> It's unclear if you actually run into the problem with this mechanism? >>>> >>>> Sorry, yes I have, I?m hitting a permissions problem: >>>> >>>> [alun at poweredge ~]$ ll /tmp/jmh.lock >>>> -rw-rw-r--. 1 misha misha 0 Jan 11 20:26 /tmp/jmh.lock >>>> [alun at poweredge ~]$ rm /tmp/jmh.lock >>>> rm: remove write-protected regular empty file `/tmp/jmh.lock'? y >>>> rm: cannot remove `/tmp/jmh.lock': Operation not permitted >>>> [alun at poweredge ~]$ >>>> >>>> [INFO] ------------------------------------------------------- >>>> [INFO] T E S T S >>>> [INFO] ------------------------------------------------------- >>>> [INFO] Running com.xockets.InterfaceBenchmark >>>> [INFO] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.081 sec <<< FAILURE! - in com.xockets.InterfaceBenchmark >>>> [INFO] run(com.xockets.InterfaceBenchmark) Time elapsed: 0.079 sec <<< ERROR! >>>> [INFO] org.openjdk.jmh.runner.RunnerException: ERROR: Exception while trying to acquire the JMH lock (/tmp/jmh.lock): /tmp/jmh.lock (Permission denied), exiting. Use -Djmh.ignoreLock=true to forcefully continue. >>>> [INFO] at org.openjdk.jmh.runner.Runner.run(Runner.java:209) >>>> [INFO] at com.xockets.AbstractBenchmark.run(AbstractBenchmark.java:92) >>>> [INFO] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> [INFO] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> >>> I see. That's indeed problematic! The file should indeed be >>> world-writeable for this to work properly. I'll take a look how to >>> fix/workaround this. >> >> That seems simple: >> https://bugs.openjdk.java.net/browse/CODETOOLS-7901577 >> >> Can you test? The owner would need to run JMH at least once to set the >> permissions right, or remove the lock file completely. > > Sure, just following "Building the "Bleeding Edge? JMH? > > http://openjdk.java.net/projects/code-tools/jmh/ > Looks good so far, just getting a colleague to run ?on-top? of my file: [alun at poweredge ximm]$ git diff src/it/benchmark/pom.xml diff --git a/ximm/src/it/benchmark/pom.xml b/ximm/src/it/benchmark/pom.xml index 6ff0daa..74ba530 100644 --- a/ximm/src/it/benchmark/pom.xml +++ b/ximm/src/it/benchmark/pom.xml @@ -37,7 +37,7 @@ UTF-8 1.8 - 1.6.3 + 1.12-SNAPSHOT [alun at poweredge ximm]$ mvn clean install [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:58 min [INFO] Finished at: 2016-01-12T15:15:09-08:00 [INFO] Final Memory: 36M/401M [INFO] ------------------------------------------------------------------------ [alun at poweredge ximm]$ ll /tmp/jmh.lock -rw-rw-rw-. 1 alun alun 0 Jan 12 15:12 /tmp/jmh.lock [alun at poweredge ximm]$ A. -- Alun Evans From alun at badgerous.net Wed Jan 13 00:25:32 2016 From: alun at badgerous.net (Alun Evans) Date: Tue, 12 Jan 2016 16:25:32 -0800 Subject: Question on jmh.lock In-Reply-To: References: <39A95014-90EE-4274-B1C1-0842A4305E4D@badgerous.net> <56954A1D.5020406@oracle.com> <80742529-CA11-4213-9798-F98D437EA709@badgerous.net> <56957F38.40600@oracle.com> <56958368.4060709@oracle.com> <6330ED57-AFA7-4C86-BFC9-5EBD2C078F0F@badgerous.net> Message-ID: > On 12 Jan 2016, at 15:20, Alun Evans wrote: > > Looks good so far, just getting a colleague to run ?on-top? of my file: It worked for him, so it looks good to me. thanks! A. -- Alun Evans From aleksey.shipilev at oracle.com Thu Jan 14 14:07:28 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 14 Jan 2016 17:07:28 +0300 Subject: Sample time with "inverted" histogram? In-Reply-To: <2A6F4F06-D033-4DF8-AE12-2BFE411FA37D@gmail.com> References: <87D45DCE-26CA-4151-BF97-E32AF1B769DC@gmail.com> <5693A5FF.4040209@oracle.com> <2A6F4F06-D033-4DF8-AE12-2BFE411FA37D@gmail.com> Message-ID: <5697ABA0.3030706@oracle.com> On 01/11/2016 06:05 PM, Chris Vest wrote: >> On 11 Jan 2016, at 13:54, Aleksey Shipilev >> > wrote: >> >> It might be worthwhile to explore. JMH's SampleBuffer is recording the >> HDR data, so those histogram values are available. Therefore it is a >> question of presentation: maybe print the histogram alongside with >> quantiles? > > You meaning putting both presentations side by side in the print out? > That would work for me. So I have a patch that produces this. Is that what you are looking for? ----------------------------------------------------------- Result "test": N = 67526 mean = 16.117 ?(99.9%) 0.039 ns/op Percentiles: p(0.0000) = 15.000 ns/op p(50.0000) = 16.000 ns/op p(90.0000) = 16.000 ns/op p(95.0000) = 16.000 ns/op p(99.0000) = 17.000 ns/op p(99.9000) = 67.000 ns/op p(99.9900) = 98.247 ns/op p(99.9990) = 201.000 ns/op p(99.9999) = 201.000 ns/op p(100.0000) = 201.000 ns/op Histogram, ns/op: [ 0.000, 25.000) = 66908 [ 25.000, 50.000) = 462 [ 50.000, 75.000) = 140 [ 75.000, 100.000) = 11 [100.000, 125.000) = 3 [125.000, 150.000) = 1 [150.000, 175.000) = 0 [175.000, 200.000) = 0 [200.000, 225.000) = 1 [225.000, 250.000) = 0 [250.000, 275.000) = 0 ----------------------------------------------------------- Result "test": N = 94842 mean = 52662.517 ?(99.9%) 19.244 ns/op Percentiles: p(0.0000) = 2216.000 ns/op p(50.0000) = 52608.000 ns/op p(90.0000) = 52672.000 ns/op p(95.0000) = 52672.000 ns/op p(99.0000) = 53312.000 ns/op p(99.9000) = 66304.000 ns/op p(99.9900) = 174724.019 ns/op p(99.9990) = 211456.000 ns/op p(99.9999) = 211456.000 ns/op p(100.0000) = 211456.000 ns/op Histogram, ns/op: [ 0.000, 25000.000) = 12 [ 25000.000, 50000.000) = 22 [ 50000.000, 75000.000) = 94774 [ 75000.000, 100000.000) = 13 [100000.000, 125000.000) = 7 [125000.000, 150000.000) = 2 [150000.000, 175000.000) = 4 [175000.000, 200000.000) = 7 [200000.000, 225000.000) = 1 [225000.000, 250000.000) = 0 [250000.000, 275000.000) = 0 ----------------------------------------------------------- Thanks, -Aleksey From mr.chrisvest at gmail.com Thu Jan 14 14:37:01 2016 From: mr.chrisvest at gmail.com (Chris Vest) Date: Thu, 14 Jan 2016 15:37:01 +0100 Subject: Sample time with "inverted" histogram? In-Reply-To: <5697ABA0.3030706@oracle.com> References: <87D45DCE-26CA-4151-BF97-E32AF1B769DC@gmail.com> <5693A5FF.4040209@oracle.com> <2A6F4F06-D033-4DF8-AE12-2BFE411FA37D@gmail.com> <5697ABA0.3030706@oracle.com> Message-ID: <54F66791-23CD-464B-B0AF-E013381C476E@gmail.com> Yes, that?s perfect. Cheers, Chris > On 14 Jan 2016, at 15:07, Aleksey Shipilev wrote: > > On 01/11/2016 06:05 PM, Chris Vest wrote: >>> On 11 Jan 2016, at 13:54, Aleksey Shipilev >>> > wrote: >>> >>> It might be worthwhile to explore. JMH's SampleBuffer is recording the >>> HDR data, so those histogram values are available. Therefore it is a >>> question of presentation: maybe print the histogram alongside with >>> quantiles? >> >> You meaning putting both presentations side by side in the print out? >> That would work for me. > > So I have a patch that produces this. Is that what you are looking for? > > ----------------------------------------------------------- > > Result "test": > N = 67526 > mean = 16.117 ?(99.9%) 0.039 ns/op > > Percentiles: > p(0.0000) = 15.000 ns/op > p(50.0000) = 16.000 ns/op > p(90.0000) = 16.000 ns/op > p(95.0000) = 16.000 ns/op > p(99.0000) = 17.000 ns/op > p(99.9000) = 67.000 ns/op > p(99.9900) = 98.247 ns/op > p(99.9990) = 201.000 ns/op > p(99.9999) = 201.000 ns/op > p(100.0000) = 201.000 ns/op > > Histogram, ns/op: > [ 0.000, 25.000) = 66908 > [ 25.000, 50.000) = 462 > [ 50.000, 75.000) = 140 > [ 75.000, 100.000) = 11 > [100.000, 125.000) = 3 > [125.000, 150.000) = 1 > [150.000, 175.000) = 0 > [175.000, 200.000) = 0 > [200.000, 225.000) = 1 > [225.000, 250.000) = 0 > [250.000, 275.000) = 0 > > ----------------------------------------------------------- > > Result "test": > N = 94842 > mean = 52662.517 ?(99.9%) 19.244 ns/op > > Percentiles: > p(0.0000) = 2216.000 ns/op > p(50.0000) = 52608.000 ns/op > p(90.0000) = 52672.000 ns/op > p(95.0000) = 52672.000 ns/op > p(99.0000) = 53312.000 ns/op > p(99.9000) = 66304.000 ns/op > p(99.9900) = 174724.019 ns/op > p(99.9990) = 211456.000 ns/op > p(99.9999) = 211456.000 ns/op > p(100.0000) = 211456.000 ns/op > > Histogram, ns/op: > [ 0.000, 25000.000) = 12 > [ 25000.000, 50000.000) = 22 > [ 50000.000, 75000.000) = 94774 > [ 75000.000, 100000.000) = 13 > [100000.000, 125000.000) = 7 > [125000.000, 150000.000) = 2 > [150000.000, 175000.000) = 4 > [175000.000, 200000.000) = 7 > [200000.000, 225000.000) = 1 > [225000.000, 250000.000) = 0 > [250000.000, 275000.000) = 0 > > ----------------------------------------------------------- > > Thanks, > -Aleksey > From nitsanw at yahoo.com Thu Jan 14 14:59:04 2016 From: nitsanw at yahoo.com (Nitsan Wakart) Date: Thu, 14 Jan 2016 14:59:04 +0000 (UTC) Subject: Enhancement: Report number of threads in summary report References: <2070781663.4048985.1452783544786.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <2070781663.4048985.1452783544786.JavaMail.yahoo@mail.yahoo.com> JMH allows us to set the number of benchmark threads, and to set up thread groups. The number of threads has a potentially large impact on the results but is not reported in the summary. I would like to have the number of threads reported e.g(ignore the values): Benchmark Ts Mode Cnt Score Error Units Contentention.getAndAdd 1 avgt 5 27.300 ? 0.406 ns/op Contentention.getAndAdd 2 avgt 5 270.300 ? 0.406 ns/op Contentention.getAndAdd 20 avgt 5 2700.300 ? 0.406 ns/op The report will be look like this for groups: Benchmark Ts Mode Cnt Score Error Units Contentention.g 22 avgt 5 2270.300 ? 0.406 ns/op Contentention.g:get 2 avgt 5 270.300 ? 0.406 ns/op Contentention.g:getAndAdd 20 avgt 5 2700.300 ? 0.406 ns/op From aleksey.shipilev at oracle.com Thu Jan 14 15:09:22 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 14 Jan 2016 18:09:22 +0300 Subject: Enhancement: Report number of threads in summary report In-Reply-To: <2070781663.4048985.1452783544786.JavaMail.yahoo@mail.yahoo.com> References: <2070781663.4048985.1452783544786.JavaMail.yahoo.ref@mail.yahoo.com> <2070781663.4048985.1452783544786.JavaMail.yahoo@mail.yahoo.com> Message-ID: <5697BA22.8020300@oracle.com> On 01/14/2016 05:59 PM, Nitsan Wakart wrote: > JMH allows us to set the number of benchmark threads, and to set up > thread groups. The number of threads has a potentially large impact > on the results but is not reported in the summary. I would like to > have the number of threads reported e.g(ignore the values): We actually had that long before, but removed it. While it is true that a number of threads indeed affects the result, many other run options do that too (e.g. OpsPerInvocation, batch sizes, etc) -- where do you stop? Last time we decided that only the performance data itself would be printed out in the report. > Benchmark Ts Mode Cnt Score Error Units > Contentention.getAndAdd 1 avgt 5 27.300 ? 0.406 ns/op > Contentention.getAndAdd 2 avgt 5 270.300 ? 0.406 ns/op > Contentention.getAndAdd 20 avgt 5 2700.300 ? 0.406 ns/op This suggests that a number of threads is changing within a single trial, how exactly that is accomplished? Before, we had the sequencing -t, where you could list the number of threads, but we don't anymore. > The report will be look like this for groups: > Benchmark Ts Mode Cnt Score Error Units > Contentention.g 22 avgt 5 2270.300 ? 0.406 ns/op > Contentention.g:get 2 avgt 5 270.300 ? 0.406 ns/op > Contentention.g:getAndAdd 20 avgt 5 2700.300 ? 0.406 ns/op This report is subtly misleading. Is that (2x get, 20x getAndAdd), or 2x(1x get, 10x getAndAdd)? This confused users before, and was the trigger for removing threads column as well. Thanks, -Aleksey From aleksey.shipilev at oracle.com Thu Jan 14 15:23:40 2016 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 14 Jan 2016 15:23:40 +0000 Subject: hg: code-tools/jmh: 7901580: Print samples histogram alongside with percentiles Message-ID: <201601141523.u0EFNeZh024214@aojmv0008.oracle.com> Changeset: 2290f828ce5c Author: shade Date: 2016-01-14 18:23 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/2290f828ce5c 7901580: Print samples histogram alongside with percentiles ! jmh-core/src/main/java/org/openjdk/jmh/results/Result.java ! jmh-core/src/main/java/org/openjdk/jmh/results/SampleTimeResult.java ! jmh-core/src/main/java/org/openjdk/jmh/results/SingleShotResult.java ! jmh-core/src/main/java/org/openjdk/jmh/util/ListStatistics.java ! jmh-core/src/main/java/org/openjdk/jmh/util/MultisetStatistics.java ! jmh-core/src/main/java/org/openjdk/jmh/util/ScoreFormatter.java ! jmh-core/src/main/java/org/openjdk/jmh/util/SingletonStatistics.java ! jmh-core/src/main/java/org/openjdk/jmh/util/Statistics.java ! jmh-core/src/test/java/org/openjdk/jmh/util/TestListStatistics.java ! jmh-core/src/test/java/org/openjdk/jmh/util/TestMultisetStatistics.java ! jmh-core/src/test/java/org/openjdk/jmh/util/TestSingletonStatistics.java + jmh-core/src/test/java/org/openjdk/jmh/util/Util.java From aleksey.shipilev at oracle.com Thu Jan 14 15:24:45 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Thu, 14 Jan 2016 18:24:45 +0300 Subject: Sample time with "inverted" histogram? In-Reply-To: <54F66791-23CD-464B-B0AF-E013381C476E@gmail.com> References: <87D45DCE-26CA-4151-BF97-E32AF1B769DC@gmail.com> <5693A5FF.4040209@oracle.com> <2A6F4F06-D033-4DF8-AE12-2BFE411FA37D@gmail.com> <5697ABA0.3030706@oracle.com> <54F66791-23CD-464B-B0AF-E013381C476E@gmail.com> Message-ID: <5697BDBD.7040108@oracle.com> On 01/14/2016 05:37 PM, Chris Vest wrote: > Yes, that?s perfect. Good, pushed. -Aleksey From aleksey.shipilev at oracle.com Thu Jan 14 18:47:35 2016 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 14 Jan 2016 18:47:35 +0000 Subject: hg: code-tools/jmh: 7901581: @CompilerControl(DONT_INLINE) on a class does not work for @Benchmark method Message-ID: <201601141847.u0EIlZtD020452@aojmv0008.oracle.com> Changeset: 80b666fe8548 Author: shade Date: 2016-01-14 21:47 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/80b666fe8548 7901581: @CompilerControl(DONT_INLINE) on a class does not work for @Benchmark method + jmh-core-it/src/test/java/org/openjdk/jmh/it/ccontrol/CompilerControlDontInlineClassTest.java ! jmh-core-it/src/test/java/org/openjdk/jmh/it/ccontrol/CompilerControlDontInlineTest.java + jmh-core-it/src/test/java/org/openjdk/jmh/it/ccontrol/CompilerControlExcludeClassTest.java ! jmh-core-it/src/test/java/org/openjdk/jmh/it/ccontrol/CompilerControlExcludeTest.java + jmh-core-it/src/test/java/org/openjdk/jmh/it/ccontrol/CompilerControlInlineClassTest.java ! jmh-core-it/src/test/java/org/openjdk/jmh/it/ccontrol/CompilerControlInlineTest.java ! jmh-core-it/src/test/java/org/openjdk/jmh/it/ccontrol/CompilerControlNestedTest.java ! jmh-core-it/src/test/java/org/openjdk/jmh/it/ccontrol/CompilerControlUtils.java ! jmh-core/src/main/java/org/openjdk/jmh/generators/core/CompilerControlPlugin.java From aleksey.shipilev at oracle.com Thu Jan 14 20:26:17 2016 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 14 Jan 2016 20:26:17 +0000 Subject: hg: code-tools/jmh: Complete Javadoc in Statistics interface. Message-ID: <201601142026.u0EKQHLg024415@aojmv0008.oracle.com> Changeset: ce0559bf7731 Author: shade Date: 2016-01-14 23:23 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/ce0559bf7731 Complete Javadoc in Statistics interface. ! jmh-core/src/main/java/org/openjdk/jmh/util/Statistics.java From aleksey.shipilev at oracle.com Thu Jan 14 22:09:47 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Fri, 15 Jan 2016 01:09:47 +0300 Subject: JMH 1.11.3 Message-ID: <56981CAB.7040008@oracle.com> Hi, JMH 1.11.3 patch release is available at Maven Central (props to Evgeny Mandrikov, as usual). It includes a few minor improvements: * The locking facility that prevents multiple concurrent JMH runs was not working properly with multiple users on the machine (lock file was created with first user's permissions). Fixed: https://bugs.openjdk.java.net/browse/CODETOOLS-7901577 * SampleTime and SingleShot modes are now printing the samples histogram along with percentile data, like this: http://mail.openjdk.java.net/pipermail/jmh-dev/2016-January/002130.html https://bugs.openjdk.java.net/browse/CODETOOLS-7901580 * @CompilerControl(DONT_INLINE) did not work on as a class-level annotation, because it was overridden by the implicit "inline" hint we put for all @Benchmarks. Fixed: https://bugs.openjdk.java.net/browse/CODETOOLS-7901581 * A few touchups in tests, javadocs, etc Enjoy! -Aleksey From aleksey.shipilev at oracle.com Thu Jan 14 22:10:06 2016 From: aleksey.shipilev at oracle.com (aleksey.shipilev at oracle.com) Date: Thu, 14 Jan 2016 22:10:06 +0000 Subject: hg: code-tools/jmh: 3 new changesets Message-ID: <201601142210.u0EMA6YT001101@aojmv0008.oracle.com> Changeset: 534d83d9137f Author: shade Date: 2016-01-14 23:28 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/534d83d9137f JMH v1.11.3. ! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml ! jmh-archetypes/pom.xml ! jmh-core-benchmarks/pom.xml ! jmh-core-ct/pom.xml ! jmh-core-it/pom.xml ! jmh-core/pom.xml ! jmh-generator-annprocess/pom.xml ! jmh-generator-asm/pom.xml ! jmh-generator-bytecode/pom.xml ! jmh-generator-reflection/pom.xml ! jmh-samples/pom.xml ! pom.xml Changeset: 05f36d230b84 Author: shade Date: 2016-01-14 23:28 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/05f36d230b84 Added tag 1.11.3 for changeset 534d83d9137f ! .hgtags Changeset: f4e8d0d61f1f Author: shade Date: 2016-01-14 23:28 +0300 URL: http://hg.openjdk.java.net/code-tools/jmh/rev/f4e8d0d61f1f Continue in 1.12-SNAPSHOT. ! jmh-archetypes/jmh-groovy-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-java-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-kotlin-benchmark-archetype/pom.xml ! jmh-archetypes/jmh-scala-benchmark-archetype/pom.xml ! jmh-archetypes/pom.xml ! jmh-core-benchmarks/pom.xml ! jmh-core-ct/pom.xml ! jmh-core-it/pom.xml ! jmh-core/pom.xml ! jmh-generator-annprocess/pom.xml ! jmh-generator-asm/pom.xml ! jmh-generator-bytecode/pom.xml ! jmh-generator-reflection/pom.xml ! jmh-samples/pom.xml ! pom.xml From Matthias.Huber at novatec-gmbh.de Fri Jan 29 15:22:23 2016 From: Matthias.Huber at novatec-gmbh.de (Huber Matthias) Date: Fri, 29 Jan 2016 15:22:23 +0000 Subject: Printing of batchsize + single execution time in Results for Batching In-Reply-To: <1452528385901.4269@novatec-gmbh.de> References: <1452172709500.67586@novatec-gmbh.de>, <5693A462.2090306@oracle.com>,<1452528385901.4269@novatec-gmbh.de> Message-ID: <1454080944968.35904@novatec-gmbh.de> Hello Aleksey, I'm short before submitting the OCA. However, I'm not sure about the correct project name. Is it "OpenJDK", "JMH" or something else. Kind Regards Matthias Huber ________________________________________ From: jmh-dev on behalf of Huber Matthias Sent: 11 January 2016 17:06 To: Aleksey Shipilev; jmh-dev at openjdk.java.net Subject: Re: Printing of batchsize + single execution time in Results for Batching Hello Aleksey, I see your point with breaking the XSV format and its impact. I will have a look at the Java API to deal with this corner case. Right now I think that only the batch size is interesting for me to distinguish the results based on the used batch size from each other. Using the calculated time for one execution (total avg time per batch / batch size) could be dangerous for comparison. Comparing two values with each other that are calculated by dividing an average time by different batch sizes feels incorrect to me. On 01/11/2016 01:47 PM, Shipilev Aleksey wrote: < But, I think we can extend JSON output to include more benchmark < metadata, including batch sizes. R should be able to pull in JSON dataset? I haven't used the JSON DataSet so far. It's worth to have a look at it and its interoperability with R. I might come back with a contribution. Thanks, Matthias From aleksey.shipilev at oracle.com Fri Jan 29 15:23:41 2016 From: aleksey.shipilev at oracle.com (Aleksey Shipilev) Date: Fri, 29 Jan 2016 18:23:41 +0300 Subject: Printing of batchsize + single execution time in Results for Batching In-Reply-To: <1454080944968.35904@novatec-gmbh.de> References: <1452172709500.67586@novatec-gmbh.de> <5693A462.2090306@oracle.com> <1452528385901.4269@novatec-gmbh.de> <1454080944968.35904@novatec-gmbh.de> Message-ID: <56AB83FD.9050302@oracle.com> OpenJDK. -Aleksey On 01/29/2016 06:22 PM, Huber Matthias wrote: > Hello Aleksey, > > I'm short before submitting the OCA. However, I'm not sure about the correct project name. Is it "OpenJDK", "JMH" or something else. > > > Kind Regards > > Matthias Huber > > ________________________________________ > From: jmh-dev on behalf of Huber Matthias > Sent: 11 January 2016 17:06 > To: Aleksey Shipilev; jmh-dev at openjdk.java.net > Subject: Re: Printing of batchsize + single execution time in Results for Batching > > Hello Aleksey, > > I see your point with breaking the XSV format and its impact. I will have a look at the Java API to deal with this corner case. > > Right now I think that only the batch size is interesting for me to distinguish the results based on the used batch size from each other. Using the calculated time for one execution (total avg time per batch / batch size) could be dangerous for comparison. Comparing two values with each other that are calculated by dividing an average time by different batch sizes feels incorrect to me. > > On 01/11/2016 01:47 PM, Shipilev Aleksey wrote: > < But, I think we can extend JSON output to include more benchmark > < metadata, including batch sizes. R should be able to pull in JSON dataset? > > I haven't used the JSON DataSet so far. It's worth to have a look at it and its interoperability with R. I might come back with a contribution. > > Thanks, > Matthias >