Dynamic languages on the JVM / jmh

[Aleksey Shipilev]

On 09/24/2014 09:15 PM, Thomas Wuerthinger wrote:
> With JavaScript as the example: One could imagine providing a clojure
> with the benchmark code together with a JSON object with
> configuration data for executing a benchmark. Internally, the clojure
> could be represented via a method handle and the configuration data
> via a hash map. This would be the scenario (a) that you outline
> below. I believe that it would be easier to use and also less likely
> to measure additional overheads such as the javax.script
> implementation of the dynamic language. What I read from your message
> is that you feel that this is out of scope for JMH.

Once again, we already have a plausible solution in JMH. There is
nothing out-of-scope, but IMO we should not waste time building the
alternative solution, especially when all the mess is visible even
before laying the first line of code. Of course, I might be mistaken,
but is an open-source world, and people might certainly try to prove me
wrong. With code.

Major caveat is that the inverted control dance -- from language runtime
to JMH generators, then to javac, then to JMH runners, and ultimately
back to language runtime -- would not be cleaner than just calling the
language runtime from JMH payload.

Minor caveat is that naive closure with benchmark payload is not enough.
This approach is nothing new, and I have yet to see the implementation
that is not broken in subtle ways, e.g:
 http://www.reddit.com/r/java/comments/299vlz/nanobench_tiny_benchmarking_framework_for_java_8/cijhn6z

I'm still trying to wrap my head around your persistence on this option.
You mention the "additional overheads" of javax.script, may I ask what
are those overheads? In other words, have you got the benchmarks to back
up your claim? May we see them? Are they represent the real problem
benchmark users face?

I am asking because we do successfully use javax.script APIs for
measuring Nashorn/JS benchmarks without a trouble, and therefore it is
strange for me to discuss the solution without understanding what
problem it tries to solve.

Thanks,
-Aleksey.