MethodHandle lookup&invocation performance

[Hiroshi Nakamura]

Hello,

I heard that jsr292 makes dynamic method lookup&invocation faster than
reflection so I did some performance comparison against plain
reflection. I'm sending this mail since the result looks strange to
me.

Code is here:
https://raw.github.com/nahi/jsr292-sandbox/master/src/jp/gr/java_conf/jruby/MethodHandleTest.java

And here's the result of above code on my linux 64bit box.

% uname -a
Linux ubuntu 2.6.39-0-generic #5~20110427-Ubuntu SMP Wed Apr 27
15:27:41 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
% /home/nahi/java/jdk1.7.0b147/bin/java -version
java version "1.7.0"
Java(TM) SE Runtime Environment (build 1.7.0-b147)
Java HotSpot(TM) 64-Bit Server VM (build 21.0-b17, mixed mode)
% /home/nahi/java/jdk1.7.0b147/bin/java -cp
out/production/jsr292-sandbox/ jp.gr.java_conf.jruby.MethodHandleTest
-- 0
methodhandle lookup        * 100000: 584.48 [msec], average:   5.84 [nsec]
methodhandle lookup+invoke * 100000: 420.05 [msec], average:   4.20 [nsec]
reflection   lookup        * 100000:  96.72 [msec], average:   0.97 [nsec]
reflection   lookup+invoke * 100000: 161.51 [msec], average:   1.62 [nsec]
-- 1
methodhandle lookup        * 100000: 184.66 [msec], average:   1.85 [nsec]
methodhandle lookup+invoke * 100000: 290.20 [msec], average:   2.90 [nsec]
reflection   lookup        * 100000:  27.19 [msec], average:   0.27 [nsec]
reflection   lookup+invoke * 100000:  62.92 [msec], average:   0.63 [nsec]
-- 2
methodhandle lookup        * 100000: 182.42 [msec], average:   1.82 [nsec]
methodhandle lookup+invoke * 100000: 293.39 [msec], average:   2.93 [nsec]
reflection   lookup        * 100000:  30.18 [msec], average:   0.30 [nsec]
reflection   lookup+invoke * 100000:  62.91 [msec], average:   0.63 [nsec]
-- 3
methodhandle lookup        * 100000: 182.37 [msec], average:   1.82 [nsec]
methodhandle lookup+invoke * 100000: 245.75 [msec], average:   2.46 [nsec]
reflection   lookup        * 100000:  27.13 [msec], average:   0.27 [nsec]
reflection   lookup+invoke * 100000:  58.63 [msec], average:   0.59 [nsec]
-- 4
methodhandle lookup        * 100000: 173.80 [msec], average:   1.74 [nsec]
methodhandle lookup+invoke * 100000: 239.77 [msec], average:   2.40 [nsec]
reflection   lookup        * 100000:  29.21 [msec], average:   0.29 [nsec]
reflection   lookup+invoke * 100000:  63.65 [msec], average:   0.64 [nsec]
-- 5
methodhandle lookup        * 100000: 188.05 [msec], average:   1.88 [nsec]
methodhandle lookup+invoke * 100000: 247.05 [msec], average:   2.47 [nsec]
reflection   lookup        * 100000:  27.80 [msec], average:   0.28 [nsec]
reflection   lookup+invoke * 100000:  61.39 [msec], average:   0.61 [nsec]
-- 6
methodhandle lookup        * 100000: 181.78 [msec], average:   1.82 [nsec]
methodhandle lookup+invoke * 100000: 248.66 [msec], average:   2.49 [nsec]
reflection   lookup        * 100000:  29.58 [msec], average:   0.30 [nsec]
reflection   lookup+invoke * 100000:  64.29 [msec], average:   0.64 [nsec]
-- 7
methodhandle lookup        * 100000: 178.12 [msec], average:   1.78 [nsec]
methodhandle lookup+invoke * 100000: 255.79 [msec], average:   2.56 [nsec]
reflection   lookup        * 100000:  28.21 [msec], average:   0.28 [nsec]
reflection   lookup+invoke * 100000:  60.76 [msec], average:   0.61 [nsec]
-- 8
methodhandle lookup        * 100000: 172.35 [msec], average:   1.72 [nsec]
methodhandle lookup+invoke * 100000: 252.54 [msec], average:   2.53 [nsec]
reflection   lookup        * 100000:  28.52 [msec], average:   0.29 [nsec]
reflection   lookup+invoke * 100000:  67.25 [msec], average:   0.67 [nsec]
-- 9
methodhandle lookup        * 100000: 181.18 [msec], average:   1.81 [nsec]
methodhandle lookup+invoke * 100000: 250.97 [msec], average:   2.51 [nsec]
reflection   lookup        * 100000:  27.14 [msec], average:   0.27 [nsec]
reflection   lookup+invoke * 100000:  62.53 [msec], average:   0.63 [nsec]

Hotspot benchmarking is hard and I would be missing something. Am I
doing it wrong? Can I expect jsr292 invocation to be faster than
reflection?

Regards,
// NaHi