Nashorn performance regression from JDK8u5 to JDK8u25?

Tue Nov 25 19:28:35 UTC 2014

Hannes,

Sure, I¹ve reproduced the data you requested below, as they are small
enough to fit comfortably in a standard message. However, in addition I
would still like to reiterate the questions from my previous reply,
regarding across-build Nashorn-related changes and performance benchmarks.

Your reply also suggests that you have some familiarity with Dust, but if
that is not the case, I can certainly provide some more details.

Regards,
Bernard Liang

===== begin basic.tl =====
<html>{~n}
<head>{~n}
  <title>{page_title}</title>{~n}
</head>{~n}
<body>{~n}
  <ul>{~n}
  {#names}
    <li>{title} {name}</li>{~n}
  {/names}
  </ul>{~n}
</body>{~n}
</html>

===== end basic.tl =====
===== begin basic.json =====
{
  "page_title": "Benchmark",
  "title": "Sir",
  "names": [{
    "name": "Moe"
  },
  {
    "name": "Larry"
  },
  {
    "name": "Curly"
  }]
}
===== end basic.json =====

On 11/25/14, 2:30 AM, "Hannes Wallnoefer" <hannes.wallnoefer at oracle.com>
wrote:

>Hi Bernard,
>
>I'm trying to reproduce your problems. In your first mail to the list
>you wrote you attached a Dust template and JSON data. I think you either
>forgot to attach it, or it was stripped by the list software. Can you
>please try to send it again, or put it somewhere we can download it?
>
>Thanks,
>Hannes
>
>Am 2014-11-18 um 03:21 schrieb Bernard Liang:
>> Michel et al,
>>
>> I¹ve run our local test battery using the link you provided, and while
>>in some cases there is improvement, overall the performance still seems
>>to be closer to u25 levels than u5 levels. For what it¹s worth, I did
>>notice that the performance improvements from u25 to u40 were generally
>>better in pooled environments than ones where a single instance of the
>>execution environment was running per thread. This leads itself to a few
>>questions, some of which are reiterated from the original inquiry:
>>
>>    *   Is anyone familiar with (significant) specific changes in the
>>Nashorn libraries from u5 => u25 => u40 that might be related to this
>>regression and could explain the u25 and/or u40 changes in more detail
>>(that might have led to the recommendation to use u40)?
>>    *   Do you have any performance suites (internal or other) that test
>>various Nashorn benchmarks across different releases (of JDK8, for
>>instance)? Do the results of those correlate with our findings?
>>
>> Regards,
>> Bernard Liang
>>
>> PS. The output of `java -version` most recently tested was as follows:
>>
>> java version "1.8.0_40-ea"
>>
>> Java(TM) SE Runtime Environment (build 1.8.0_40-ea-b12)
>>
>> Java HotSpot(TM) 64-Bit Server VM (build 25.40-b16, mixed mode)
>>
>> Previous versions tested:
>>
>> java version "1.8.0_25"
>>
>> Java(TM) SE Runtime Environment (build 1.8.0_25-b17)
>>
>> Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)
>>
>> java version "1.8.0_05"
>>
>> Java(TM) SE Runtime Environment (build 1.8.0_05-b13)
>>
>> Java HotSpot(TM) 64-Bit Server VM (build 25.5-b02, mixed mode)
>>
>> From: Michel Trudeau
>><michel.trudeau at oracle.com<mailto:michel.trudeau at oracle.com>>
>> Date: Monday, November 17, 2014 at 1:12 PM
>> To: Bernard Liang <bliang at linkedin.com<mailto:bliang at linkedin.com>>
>> Cc: "nashorn-dev at openjdk.java.net<mailto:nashorn-dev at openjdk.java.net>"
>><nashorn-dev at openjdk.java.net<mailto:nashorn-dev at openjdk.java.net>>
>> Subject: Re: Nashorn performance regression from JDK8u5 to JDK8u25?
>>
>> Bernard,
>>
>> It'd be great if you could try the latest 8u40 stable build.   We are
>>planning to release 8u40 early in the new year.
>>
>> https://jdk8.java.net/download.html
>>
>> We also have an optional optimizer in 8u40, enable it with the command
>>line argument '-ot'.
>>
>> Thanks,
>> Michel
>>
>>
>>
>>
>>
>> Bernard Liang wrote:
>>
>> Hello,
>>
>> After running some performance tests on the Cartesian product of
>>([JDK8u5, JDK8u25] x [simple template, complex templates] x
>>[all-or-nothing, streaming chunks] x [single dust instance per thread,
>>pooled dust instances] x [blank Dust instances, Dust instances with
>>templates preloaded]), we find that JDK8u25 performance is very
>>consistently considerably worse than JDK8u5 (by roughly 10-100%, with
>>the average falling somewhere between there). The relevant code has been
>>executed enough times (on the order of 10,000 times) to reach reasonably
>>warmed-up states. If some of the items on the axes of the Cartesian
>>product don¹t make much sense, you can ignore the fuzzy parts of the
>>detailed breakdown for now, with the general understanding that various
>>different environments have been tested and shown to yield the same
>>results.
>>
>> Some additional high-level context:
>>
>> Dust is basically a templating language used to render JSON data into
>>HTML with compilable ³templates": https://github.com/linkedin/dustjs (we
>>are at the v2.4.2 tag)
>>
>> ³simple template² = ~150 bytes each of one (precompiled) template +
>>context JSON (attached)
>> ³complex templates² = ~350 compiled templates spanning ~245KB in
>>compiled JS + ~75KB of JSON context (proprietary data)
>>
>> >From 
>>https://blogs.oracle.com/nashorn/entry/latest_news_from_the_nashorn, it
>>sounded like there were some recent updates made to Nashorn performance
>>around the u20 mark, but that seems to have caused a regression rather
>>than an improvement. Is this something that nashorn-dev is aware of? Is
>>there any way we can help diagnose the issue further using publicly safe
>>data? (If you¹re looking for a way to reproduce this, the attached basic
>>Dust template + JSON context should be adequate under almost any
>>environment.)
>>
>> Regards,
>> Bernard Liang
>>
>