Nashorn performance regression from JDK8u5 to JDK8u25?

Wed Nov 26 16:43:00 UTC 2014

Thanks Bernard.

As to your question about differences between u5, u25 and u40, there are 
quite a few, and it's hard to guess what caused a regression without 
looking at the code running. We mostly rely on Octane for performance 
benchmarking, it's quite possible that this is a bit too narrow and 
leaves a few blind spots.

I have used Dust before and I think I can get it running. I'll let you 
know when I find out more.

Regards,
Hannes

Am 2014-11-25 um 20:28 schrieb Bernard Liang:
> Hannes,
>
> Sure, I¹ve reproduced the data you requested below, as they are small
> enough to fit comfortably in a standard message. However, in addition I
> would still like to reiterate the questions from my previous reply,
> regarding across-build Nashorn-related changes and performance benchmarks.
>
> Your reply also suggests that you have some familiarity with Dust, but if
> that is not the case, I can certainly provide some more details.
>
> Regards,
> Bernard Liang
>
> ===== begin basic.tl =====
> <html>{~n}
> <head>{~n}
>    <title>{page_title}</title>{~n}
> </head>{~n}
> <body>{~n}
>    <ul>{~n}
>    {#names}
>      <li>{title} {name}</li>{~n}
>    {/names}
>    </ul>{~n}
> </body>{~n}
> </html>
>
> ===== end basic.tl =====
> ===== begin basic.json =====
> {
>    "page_title": "Benchmark",
>    "title": "Sir",
>    "names": [{
>      "name": "Moe"
>    },
>    {
>      "name": "Larry"
>    },
>    {
>      "name": "Curly"
>    }]
> }
> ===== end basic.json =====
>
>
> On 11/25/14, 2:30 AM, "Hannes Wallnoefer" <hannes.wallnoefer at oracle.com>
> wrote:
>
>> Hi Bernard,
>>
>> I'm trying to reproduce your problems. In your first mail to the list
>> you wrote you attached a Dust template and JSON data. I think you either
>> forgot to attach it, or it was stripped by the list software. Can you
>> please try to send it again, or put it somewhere we can download it?
>>
>> Thanks,
>> Hannes
>>
>> Am 2014-11-18 um 03:21 schrieb Bernard Liang:
>>> Michel et al,
>>>
>>> I¹ve run our local test battery using the link you provided, and while
>>> in some cases there is improvement, overall the performance still seems
>>> to be closer to u25 levels than u5 levels. For what it¹s worth, I did
>>> notice that the performance improvements from u25 to u40 were generally
>>> better in pooled environments than ones where a single instance of the
>>> execution environment was running per thread. This leads itself to a few
>>> questions, some of which are reiterated from the original inquiry:
>>>
>>>     *   Is anyone familiar with (significant) specific changes in the
>>> Nashorn libraries from u5 => u25 => u40 that might be related to this
>>> regression and could explain the u25 and/or u40 changes in more detail
>>> (that might have led to the recommendation to use u40)?
>>>     *   Do you have any performance suites (internal or other) that test
>>> various Nashorn benchmarks across different releases (of JDK8, for
>>> instance)? Do the results of those correlate with our findings?
>>>
>>> Regards,
>>> Bernard Liang
>>>
>>> PS. The output of `java -version` most recently tested was as follows:
>>>
>>> java version "1.8.0_40-ea"
>>>
>>> Java(TM) SE Runtime Environment (build 1.8.0_40-ea-b12)
>>>
>>> Java HotSpot(TM) 64-Bit Server VM (build 25.40-b16, mixed mode)
>>>
>>> Previous versions tested:
>>>
>>> java version "1.8.0_25"
>>>
>>> Java(TM) SE Runtime Environment (build 1.8.0_25-b17)
>>>
>>> Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)
>>>
>>> java version "1.8.0_05"
>>>
>>> Java(TM) SE Runtime Environment (build 1.8.0_05-b13)
>>>
>>> Java HotSpot(TM) 64-Bit Server VM (build 25.5-b02, mixed mode)
>>>
>>> From: Michel Trudeau
>>> <michel.trudeau at oracle.com<mailto:michel.trudeau at oracle.com>>
>>> Date: Monday, November 17, 2014 at 1:12 PM
>>> To: Bernard Liang <bliang at linkedin.com<mailto:bliang at linkedin.com>>
>>> Cc: "nashorn-dev at openjdk.java.net<mailto:nashorn-dev at openjdk.java.net>"
>>> <nashorn-dev at openjdk.java.net<mailto:nashorn-dev at openjdk.java.net>>
>>> Subject: Re: Nashorn performance regression from JDK8u5 to JDK8u25?
>>>
>>> Bernard,
>>>
>>> It'd be great if you could try the latest 8u40 stable build.   We are
>>> planning to release 8u40 early in the new year.
>>>
>>> https://jdk8.java.net/download.html
>>>
>>> We also have an optional optimizer in 8u40, enable it with the command
>>> line argument '-ot'.
>>>
>>> Thanks,
>>> Michel
>>>
>>>
>>>
>>>
>>>
>>> Bernard Liang wrote:
>>>
>>> Hello,
>>>
>>> After running some performance tests on the Cartesian product of
>>> ([JDK8u5, JDK8u25] x [simple template, complex templates] x
>>> [all-or-nothing, streaming chunks] x [single dust instance per thread,
>>> pooled dust instances] x [blank Dust instances, Dust instances with
>>> templates preloaded]), we find that JDK8u25 performance is very
>>> consistently considerably worse than JDK8u5 (by roughly 10-100%, with
>>> the average falling somewhere between there). The relevant code has been
>>> executed enough times (on the order of 10,000 times) to reach reasonably
>>> warmed-up states. If some of the items on the axes of the Cartesian
>>> product don¹t make much sense, you can ignore the fuzzy parts of the
>>> detailed breakdown for now, with the general understanding that various
>>> different environments have been tested and shown to yield the same
>>> results.
>>>
>>> Some additional high-level context:
>>>
>>> Dust is basically a templating language used to render JSON data into
>>> HTML with compilable ³templates": https://github.com/linkedin/dustjs (we
>>> are at the v2.4.2 tag)
>>>
>>> ³simple template² = ~150 bytes each of one (precompiled) template +
>>> context JSON (attached)
>>> ³complex templates² = ~350 compiled templates spanning ~245KB in
>>> compiled JS + ~75KB of JSON context (proprietary data)
>>>
>>> >From
>>> https://blogs.oracle.com/nashorn/entry/latest_news_from_the_nashorn, it
>>> sounded like there were some recent updates made to Nashorn performance
>>> around the u20 mark, but that seems to have caused a regression rather
>>> than an improvement. Is this something that nashorn-dev is aware of? Is
>>> there any way we can help diagnose the issue further using publicly safe
>>> data? (If you¹re looking for a way to reproduce this, the attached basic
>>> Dust template + JSON context should be adequate under almost any
>>> environment.)
>>>
>>> Regards,
>>> Bernard Liang
>>>