Dynamic languages on the JVM / jmh

[Thomas Wuerthinger]

Just brainstorming here from the viewpoint of a JavaScript developer. Of course there are existing workarounds. The recent Nashorn pushback that is doing micro benchmarking without using jmh made me think whether there could be a better solution. Going through javax.script can have high overheads depending on the language runtime implementation. And I really do not believe that it is convenient to create a new Java class with a boiler plate call to javax.script for each JavaScript micro benchmark. I would be interested to see the solution of Marcus’ code using jmh for a comparison. Or maybe jmh is just not suitable for his problem.

If you do not think that this is an issue to be worth thinking about, it is fine too :).

- thomas

On 24 Sep 2014, at 19:34, Aleksey Shipilev <aleksey.shipilev at oracle.com> wrote:

> On 09/24/2014 09:15 PM, Thomas Wuerthinger wrote:
>> With JavaScript as the example: One could imagine providing a clojure
>> with the benchmark code together with a JSON object with
>> configuration data for executing a benchmark. Internally, the clojure
>> could be represented via a method handle and the configuration data
>> via a hash map. This would be the scenario (a) that you outline
>> below. I believe that it would be easier to use and also less likely
>> to measure additional overheads such as the javax.script
>> implementation of the dynamic language. What I read from your message
>> is that you feel that this is out of scope for JMH.
> 
> Once again, we already have a plausible solution in JMH. There is
> nothing out-of-scope, but IMO we should not waste time building the
> alternative solution, especially when all the mess is visible even
> before laying the first line of code. Of course, I might be mistaken,
> but is an open-source world, and people might certainly try to prove me
> wrong. With code.
> 
> Major caveat is that the inverted control dance -- from language runtime
> to JMH generators, then to javac, then to JMH runners, and ultimately
> back to language runtime -- would not be cleaner than just calling the
> language runtime from JMH payload.
> 
> Minor caveat is that naive closure with benchmark payload is not enough.
> This approach is nothing new, and I have yet to see the implementation
> that is not broken in subtle ways, e.g:
> http://www.reddit.com/r/java/comments/299vlz/nanobench_tiny_benchmarking_framework_for_java_8/cijhn6z
> 
> I'm still trying to wrap my head around your persistence on this option.
> You mention the "additional overheads" of javax.script, may I ask what
> are those overheads? In other words, have you got the benchmarks to back
> up your claim? May we see them? Are they represent the real problem
> benchmark users face?
> 
> I am asking because we do successfully use javax.script APIs for
> measuring Nashorn/JS benchmarks without a trouble, and therefore it is
> strange for me to discuss the solution without understanding what
> problem it tries to solve.
> 
> Thanks,
> -Aleksey.
>