LLVM vs stock GCC

[Henri Gomez]

I rebuilded latest OpenJDK 7 from bsd-port with gcc/g++ 4.2 (4.2.1)

I redo the dacapobench, increasing some benchs looping, now :

avrora : 10 pass
eclipse : 5 pass
fop : 10 pass
h2 : 5 pass
jython : 5 pass
luindex : 10 pass
lusearch : 10 pass
pmd : 10 pass
sunflow : 10 pass
tomcat : 5 pass
tradebeans : 5 pass
tradesoap : 5 pass
xalan : 10 pass

Here are the results (on i7 2.8Ghz), a serie of 3 consecutives benchs
for bsd and then osx OpenJDK.

OpenJDK 7 (bsd) - gcc 4.2

===== DaCapo 9.12 avrora PASSED in 3430 msec =====
===== DaCapo 9.12 eclipse PASSED in 20227 msec =====
===== DaCapo 9.12 fop PASSED in 300 msec =====
===== DaCapo 9.12 h2 PASSED in 4618 msec =====
===== DaCapo 9.12 luindex PASSED in 626 msec =====
===== DaCapo 9.12 lusearch PASSED in 1214 msec =====
===== DaCapo 9.12 pmd PASSED in 1579 msec =====
===== DaCapo 9.12 sunflow PASSED in 2261 msec =====
===== DaCapo 9.12 tomcat PASSED in 1661 msec =====
===== DaCapo 9.12 tradebeans PASSED in 4933 msec =====
===== DaCapo 9.12 tradesoap PASSED in 21193 msec =====
===== DaCapo 9.12 xalan PASSED in 1842 msec =====

imac-hgomez-exo:Documents henri$ ./dacapobench.sh
===== DaCapo 9.12 avrora PASSED in 3276 msec =====
===== DaCapo 9.12 eclipse PASSED in 17462 msec =====
===== DaCapo 9.12 fop PASSED in 314 msec =====
===== DaCapo 9.12 h2 PASSED in 4658 msec =====
===== DaCapo 9.12 luindex PASSED in 773 msec =====
===== DaCapo 9.12 lusearch PASSED in 1305 msec =====
===== DaCapo 9.12 pmd PASSED in 1566 msec =====
===== DaCapo 9.12 sunflow PASSED in 2354 msec =====
===== DaCapo 9.12 tomcat PASSED in 1678 msec =====
===== DaCapo 9.12 tradebeans PASSED in 4875 msec =====
===== DaCapo 9.12 tradesoap PASSED in 24030 msec =====
===== DaCapo 9.12 xalan PASSED in 727 msec =====

===== DaCapo 9.12 avrora PASSED in 3249 msec =====
===== DaCapo 9.12 eclipse PASSED in 17728 msec =====
===== DaCapo 9.12 fop PASSED in 299 msec =====
===== DaCapo 9.12 h2 PASSED in 4399 msec =====
===== DaCapo 9.12 luindex PASSED in 649 msec =====
===== DaCapo 9.12 lusearch PASSED in 4128 msec =====
===== DaCapo 9.12 pmd PASSED in 1572 msec =====
===== DaCapo 9.12 sunflow PASSED in 2269 msec =====
===== DaCapo 9.12 tomcat PASSED in 1640 msec =====
===== DaCapo 9.12 tradebeans PASSED in 4952 msec =====
===== DaCapo 9.12 tradesoap PASSED in 19845 msec =====
===== DaCapo 9.12 xalan PASSED in 782 msec =====

OpenJDK 7 osx

===== DaCapo 9.12 avrora PASSED in 3431 msec =====
===== DaCapo 9.12 eclipse PASSED in 20261 msec =====
===== DaCapo 9.12 fop PASSED in 302 msec =====
===== DaCapo 9.12 h2 PASSED in 4684 msec =====
===== DaCapo 9.12 jython PASSED in 2227 msec =====
===== DaCapo 9.12 luindex PASSED in 826 msec =====
===== DaCapo 9.12 lusearch PASSED in 3974 msec =====
===== DaCapo 9.12 pmd PASSED in 1891 msec =====
===== DaCapo 9.12 sunflow PASSED in 2224 msec =====
===== DaCapo 9.12 tomcat PASSED in 1740 msec =====
===== DaCapo 9.12 tradebeans PASSED in 5159 msec =====
===== DaCapo 9.12 tradesoap PASSED in 19105 msec =====
===== DaCapo 9.12 xalan PASSED in 798 msec =====

===== DaCapo 9.12 avrora PASSED in 3407 msec =====
===== DaCapo 9.12 eclipse PASSED in 20655 msec =====
===== DaCapo 9.12 fop PASSED in 301 msec =====
===== DaCapo 9.12 h2 PASSED in 4652 msec =====
===== DaCapo 9.12 jython PASSED in 2347 msec =====
===== DaCapo 9.12 luindex PASSED in 2291 msec =====
===== DaCapo 9.12 lusearch PASSED in 3629 msec =====
===== DaCapo 9.12 pmd PASSED in 1888 msec =====
===== DaCapo 9.12 sunflow PASSED in 2362 msec =====
===== DaCapo 9.12 tomcat PASSED in 1755 msec =====
===== DaCapo 9.12 tradebeans PASSED in 5184 msec =====
===== DaCapo 9.12 tradesoap PASSED in 19272 msec =====
===== DaCapo 9.12 xalan PASSED in 1948 msec =====

===== DaCapo 9.12 avrora PASSED in 3436 msec =====
===== DaCapo 9.12 eclipse PASSED in 19694 msec =====
===== DaCapo 9.12 fop PASSED in 318 msec =====
===== DaCapo 9.12 h2 PASSED in 4724 msec =====
===== DaCapo 9.12 jython PASSED in 2320 msec =====
===== DaCapo 9.12 luindex PASSED in 2214 msec =====
===== DaCapo 9.12 lusearch PASSED in 1193 msec =====
===== DaCapo 9.12 pmd PASSED in 1888 msec =====
===== DaCapo 9.12 sunflow PASSED in 2271 msec =====
===== DaCapo 9.12 tomcat PASSED in 1763 msec =====
===== DaCapo 9.12 tradebeans PASSED in 5120 msec =====
===== DaCapo 9.12 tradesoap PASSED in 18341 msec =====
===== DaCapo 9.12 xalan PASSED in 789 msec =====

If we take a CPU intensive bench, like avovra, where variation between
tests is low, less than 1%, we could see that bsd-port is about 4%
faster.
My concern is less about LLVM vs GCC but about possible LLVM
optimization flags, and how to set it globally at make time.

Regards

2011/4/19 Henri Gomez <henri.gomez at gmail.com>:
>>> The core functionality that Henri's test are evaluating are effectively complete. I would speculate that the minor drop in performance is likely the combination of forcing stack pointer alignment to 16-byte boundaries, and using -Os instead of -O3, as opposed to any systemic issue with LLVM.
>>>
>>> It would be interesting to find the combination of options that cause xalan, lusearch, and eclipse to fall behind the other OpenJDK variants. If there is an issue with using the LLVM backend, a bug should be filed against it ASAP (of course comparing -Os to -Os or -O3 to -O3, and ensuring all other factors are equal).
>>
>> I could redo the test of course. Did there is a way to change globally
>> LLVM CC+C++ flags ?
>
> I was thinking about CFLAGS and CXXFLAGS, if they are provided to make
> command line.
>
> BTW, I checked my open-bsd port, and I'm using gcc-4.0 (and not
> gcc-4.2). I'll redo a new build to check if the perf diff couldn't
> came from here.
>