performance of less compilation

[Michel Trudeau]

Hannes,

Thank you for working on this and figuring this one out.   It'd be nice if we'd have a JVM mode that would force full opt at a 4-5 iterations instead of 15.

Michel

On Jul 18, 2013, at 6:43 AM, Hannes Wallnoefer <hannes.wallnoefer at oracle.com> wrote:

Hi Ivo,

I've looked at the less.js performance issues. What I could reproduce is that Nashorn is initially slower than Rhino. This is due to the heavy use of MethodHandles which in current JDK8 generate a lot of intermediate code that needs to be JIT-compiled. In fact, of the first run of less.js more than 90% of time is spent compiling bytecode. This is something we're aware of and we're trying to improve.

However, I do see Nashorn speeding up and passing by Rhino after a few runs. Roughly speaking I see Nashorn 3 times slower than Rhino on the first run, but 3 times faster once the code is fully compiled and optimized. I've pasted the benchmark results below.

You'll notice that this is not from your benchmarking code. For some reason I couldn't get your code to locate the script files, probably due to some setup problem or my inexperience with gradle, so I decided to start with the original less-rhino-1.3.3.js from the main less.js repository, adapting it to Nashorn and adding a loop around the main code. I ran this with the benchmark/benchmark.less file from the main less.js repository. I've uploaded my benchmark scripts to gist.github.com:

https://gist.github.com/hns/6029223

I'm not exactly sure why you don't see Nashorn catching up and getting faster to Rhino. It may be something that has improved in Nashorn or the JDK since then. At least I haven't found anything suspicious in your source code. Maybe you could rerun your tests with a current JDK8 from https://jdk8.java.net/, or help me get your test running?

cheers,
Hannes

Below are my results with current Nashorn tip and Rhino master and JDK8-b98 (this is with default Rhino optlevel but results for optlevel 9 is basically the same):

$ /home/hannes/local/jdk1.8.0/bin/java -jar ../nashorn/dist/nashorn.jar -scripting less-nashorn-1.3.3.js -- benchmark.less
benchmarking nashorn...
run 1:    7973 millis
run 2:    3971 millis
run 3:    1514 millis
run 4:    1533 millis
run 5:    782 millis
run 6:    651 millis
run 7:    434 millis
run 8:    437 millis
run 9:    378 millis
run 10:    309 millis
run 11:    338 millis
run 12:    294 millis
run 13:    305 millis
run 14:    313 millis
run 15:    419 millis
run 16:    261 millis
run 17:    247 millis
run 18:    251 millis
run 19:    234 millis
run 20:    235 millis
run 21:    250 millis
run 22:    258 millis
run 23:    236 millis
run 24:    232 millis
run 25:    227 millis
run 26:    237 millis
run 27:    238 millis
run 28:    229 millis
run 29:    225 millis
run 30:    276 millis
done

$ /home/hannes/local/jdk1.8.0/bin/java -jar ../rhino/build/rhino1_7R5pre/js.jar less-rhino-1.3.3.js benchmark.less
benchmarking rhino...
run 1:    2585 millis
run 2:    1437 millis
run 3:    1173 millis
run 4:    1058 millis
run 5:    856 millis
run 6:    824 millis
run 7:    872 millis
run 8:    779 millis
run 9:    808 millis
run 10:    756 millis
run 11:    748 millis
run 12:    751 millis
run 13:    785 millis
run 14:    773 millis
run 15:    758 millis
run 16:    739 millis
run 17:    750 millis
run 18:    736 millis
run 19:    743 millis
run 20:    739 millis
run 21:    739 millis
run 22:    742 millis
run 23:    742 millis
run 24:    751 millis
run 25:    737 millis
run 26:    748 millis
run 27:    743 millis
run 28:    746 millis
run 29:    739 millis
run 30:    739 millis
done


Am 2013-06-25 22:11, schrieb Ivo Houbrechts:
>> Thanks for the heads up.  JDK8 is only API frozen, so we have some time to address performance issues as they come up.  There are some fixes in the pipe that will address some of the things I see here, but we should examine in more detail.  Therefore, I have a few questions and comments.
>> 
>> How did you run less-1.3.3.min.js independently of DOM/CSS? Running straight up I get several reference errors.  Would you post the exact code you used to produce these results?
> The source code is on github:https://github.com/houbie/lesscss/tree/nashorn, in the nashorn branch
> The main class is LessCompiler.java, which can compile an input fie to an output file.
> The project is build with gradle 1.6, but since it has only one test dependency, it can be compiled/executed straight from an IDE.
> The src/main/resources/js directory contains the 3 javascript files that get merged and compiled: environment.js (minimal stubs for browser/dom), less-1.3.3(.min).js and compile.js (java callback functions)
> 
> PerformanceComparison.groovy in src/test/groovy was used as a basis for the test runs to compile the bundled bootstrap less
> 
> 
>> The chart you provide doesn't  show Nashorn JDK7 completing.  How did it fail?  It is possible that Nashorn might converge differently than Rhino (focus on server side.)  I'll see if I can get an in house version of JDK7  to run (we don't use the github back port - it has issues.)
> It failed with a javascript error (don't remember the exact error) during the 3th run within the same context, but I wouldn't bother too much because the back port is based on an older Nashorn version (although I was intrigued by the fact that it was faster then jdk8 for the first 2 runs)
> 
>> You should also note that there is a bug in JDK7  javax.script that causes javascript to be chosen randomly (rhino/nashorn.)  This has been addressed in later releases (fix is in the pipe.)
> I explicitly load the engine by name and I don't think I ran into this.
> 
>> Comparing JDK 7 and JDK 8 is apples and oranges at this point.  JDK8 suffers a high start up cost of JSR-292 Lambda Forms.  There is also a fix in the pipe for this.
> As you can see in PerformanceComparison.groovy, I started to measure after creating the compiler object, so jdk startup should not matter. Furthermore, when I run the code from the master branch, that uses Rhino 1.7R4, I get the same results in both jdk7 and jdk8
> 
> Grtz,
> Ivo
>