[loc-en-dev] jdk test logs with the locale changes

[Yoshito Umaoka]

First of all, I'm not expecting such tests represent actual performance 
changes very well. If we really want to measure performance changes, we 
need to write dedicated test cases. But, when something really bad for 
performance is introduced, I think we could see the sign of performance 
degrade even from such test.

When I ran the test on my system, I did not run other apps at the same 
time. But it's not easy to eliminate other factors. I noticed something 
like below (both negative/positive changes). The number is the average 
time of 5 attempts. As you can see, it took 0.1050 sec for some reasons 
once, which impacts the average number. However, the minimum time is 
unchanged (0.007 sec), that indicates the number increase was caused by 
other factors. Also, this test case does nothing with Locale obviously.

So, if an average number is changed a lot, I check min/max. If the delta 
between min and max is big, I assume it was caused by other factors 
unless min times are not changed.

-Yoshito

Andy Staudacher wrote:
> On Wed, Sep 1, 2010 at 12:47 AM, Masayoshi Okutsu 
> <masayoshi.okutsu at oracle.com <mailto:masayoshi.okutsu at oracle.com>> wrote:
>
>      I noticed there are some big changes where the Locale changes
>     shouldn't affect, like this one.
>
>     Test  : java/util/AbstractMap/ToString.java
>     Old   : 0.0084 (Min=0.0070, Max=0.0110)
>     New   : 0.0270 (Min=0.0070, Max=0.1050)
>     Change: +221.43%
>
>     Is the performance measurement reliable?
>
>
> Good question. 
> Related: http://code.google.com/p/caliper/wiki/JavaMicrobenchmarks. 
>
> Looking at this example's source code 
> <http://www.google.com/codesearch/p?hl=en#SWrQ0LlxhHw/test/java/util/AbstractMap/ToString.java&q=java/util/AbstractMap/ToString.java&sa=N&cd=1&ct=rc>, 
> it's clear that it's a test and not a micro-benchmark. Huge variations 
> are to be expected. These performance numbers cannot be meaningful. 
> The measured time is probably dominated by initializing and warming up 
> the JVM.
>
> Are there actual micro benchmarks, or are the tests the best framework 
> / tool we have to measure performance regressions?
>
> Thanks,
>  - Andy
>
>
>     Thanks,
>     Masayoshi
>
>
>     On 9/1/2010 7:55 AM, Yoshito Umaoka wrote:
>
>         I wanted to make sure if we do not introduce any performance
>         regression. So I ran the standard test target jdk_lang,
>         jdk_text and jdk_util on our latest code and its base build
>         (b106). I ran the test 5 times for each and posted the raw log
>         files below -
>
>         b106 -> http://icu-project.org/~yoshito/LocaleTestLogs/b106/
>         <http://icu-project.org/%7Eyoshito/LocaleTestLogs/b106/>
>         b106 + locale changes ->
>         http://icu-project.org/~yoshito/LocaleTestLogs/le/
>         <http://icu-project.org/%7Eyoshito/LocaleTestLogs/le/>
>
>         I wrote a small code to collect execution time of each text
>         case (average/min/max). You can see the changes below -
>
>         jdk_lang ->
>         http://icu-project.org/~yoshito/LocaleTestLogs/lang.txt
>         <http://icu-project.org/%7Eyoshito/LocaleTestLogs/lang.txt>
>         jdk_text ->
>         http://icu-project.org/~yoshito/LocaleTestLogs/text.txt
>         <http://icu-project.org/%7Eyoshito/LocaleTestLogs/text.txt>
>         jdk_util ->
>         http://icu-project.org/~yoshito/LocaleTestLogs/util.txt
>         <http://icu-project.org/%7Eyoshito/LocaleTestLogs/util.txt>
>
>         I reviewed the output briefly. So far, I did not see any
>         obvious performance regression problems.
>
>         -Yoshito
>
>