Nashorn Octane JMH code publicly available?

[Chris Newland]

Hi Volker,

I thought I'd posted my findings back here but turns out I'd tweeted them:

https://twitter.com/chriswhocodes/status/596723775779012608

Here is my JMH Octane wrapper:
https://gist.github.com/chriswhocodes/c5bf3632fb7272e994e9

The gist doesn't incorporate incorporate Aleksey's @Setup suggestion.

Given the time taken to reach 00s of runs I think it might be better to
benchmark each Octane test in isolation rather than wait for the whole
suite to stabilise?

-Chris

On Thu, June 25, 2015 09:04, Marcus Lagergren wrote:
> No results are very unstable - using JMH will bring significant stability
> to them, but you have to realize that in some cases HotSpot needs to do
> hundreds of runs before plateauing out because of LambdaForms etc.
> Aleksey knows more.
>
>
> /M
>
>
>> On 24 Jun 2015, at 17:50, Volker Simonis <volker.simonis at gmail.com>
>> wrote:
>>
>>
>> Hi Chris, Marcus,
>>
>>
>> thanks for your responses.
>>
>> Some colleagues of mine who set up Octane in our environment
>> complained about the test results being extremely unstable. And even more
>> so if comparing two different JDK versions (like our internal build with
>> an "official" Oracle-JDK build" which should "theoretically" behave the
>> same).
>>
>> So I wanted to look at the differences and what I found was the quote
>> from the Nashorn Wiki which says: "the octane base.js harness .. has a
>> very primitive and deviation prone way of measuring benchmark time."
>> That's why I thought to better use JMH harness as described on that
>> page before starting to look at "real" problems.
>>
>>
>> @Marcus: do you really see stable benchmark results when using your
>> internal JMH Octane harness or is this a pointless undertaking anywhere?
>>
>> @Chris: if it isn't too much of an effort for you, I'd appreciate to
>> have a look at your harness.
>>
>> Regards,
>> Volker
>>
>>
>> On Wed, Jun 24, 2015 at 5:38 PM, Chris Newland
>> <cnewland at chrisnewland.com> wrote:
>>
>>> Hi Volker,
>>>
>>>
>>> In the end I wrote my own JMH harness around octane but I can't get
>>> the results stable even with 3 day runs so my JMH code could be buggy.
>>>
>>>
>>> Will share later off list if you're interested.
>>>
>>>
>>> Regards,
>>>
>>>
>>> Chris
>>> @chriswhocodes
>>>
>>>
>>> On Wed, June 24, 2015 16:00, Marcus Lagergren wrote:
>>>
>>>> Not sure if it’s available publically as part of JMH, but you can
>>>> get the octane benchmark code itself from
>>>>
>>>> svn co http://octane-benchmark.googlecode.com/svn/trunk
>>>> <http://octane-benchmark.googlecode.com/svn/trunk>
>>>>
>>>>
>>>>
>>>> /M
>>>>
>>>>
>>>>
>>>>> On 24 Jun 2015, at 16:41, Volker Simonis
>>>>> <volker.simonis at gmail.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>> I just had the same question and couldn't find any answer with
>>>>> Google
>>>>> except this mail thread.
>>>>>
>>>>> So I'll just as again: is the Octane benchmark as part of JMH as
>>>>> described in the Nashorn Wiki [1] publicly available somewhere?
>>>>>
>>>>> Thanks,
>>>>> Volker
>>>>>
>>>>>
>>>>>
>>>>> [1]
>>>>> https://wiki.openjdk.java.net/display/Nashorn/Monitoring+Nashorn+P
>>>>> erfor mance
>>>>>
>>>>> On Thu, Apr 30, 2015 at 9:18 AM, Marcus Lagergren
>>>>> <marcus.lagergren at oracle.com> wrote:
>>>>>
>>>>>
>>>>>>
>>>>>>> On 29 Apr 2015, at 09:08, Marcus Lagergren
>>>>>>> <marcus.lagergren at oracle.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> The benchmark code itself is public, but I don’t know if
>>>>>>> the benchmarks as part of JMH are. CC Aleksey /M
>>>>>>>
>>>>>>>
>>>>>>>> On 28 Apr 2015, at 19:24, Chris Newland
>>>>>>>> <cnewland at chrisnewland.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> https://wiki.openjdk.java.net/display/Nashorn/Monitoring+Na
>>>>>>>> shorn+ Performance
>>>>>>>> indicates the JMH microbenchmarks for octane are internal to
>>>>>>>>  Oracle (VPN
>>>>>>>> needed).
>>>>>>>>
>>>>>>>> I can't see them elsewhere in the OpenJDK repos, are they
>>>>>>>> public?
>>>>>>>>
>>>>>>>>
>>>>>>>> Asking because the results I'm getting from
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> nashorn/test/script/external/octane$ jjs run.js
>>>>>>>>
>>>>>>>> are wildly variable (as per above wiki page).
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Chris
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>>
>>>
>>>
>
>