[loc-en-dev] jdk test logs with the locale changes

[Andy Staudacher]

On Wed, Sep 1, 2010 at 12:47 AM, Masayoshi Okutsu <
masayoshi.okutsu at oracle.com> wrote:

>  I noticed there are some big changes where the Locale changes shouldn't
> affect, like this one.
>
> Test  : java/util/AbstractMap/ToString.java
> Old   : 0.0084 (Min=0.0070, Max=0.0110)
> New   : 0.0270 (Min=0.0070, Max=0.1050)
> Change: +221.43%
>
> Is the performance measurement reliable?
>

Good question. Related:
http://code.google.com/p/caliper/wiki/JavaMicrobenchmarks.

Looking at this example's source
code<http://www.google.com/codesearch/p?hl=en#SWrQ0LlxhHw/test/java/util/AbstractMap/ToString.java&q=java/util/AbstractMap/ToString.java&sa=N&cd=1&ct=rc>,
it's clear that it's a test and not a micro-benchmark. Huge variations are
to be expected. These performance numbers cannot be meaningful. The measured
time is probably dominated by initializing and warming up the JVM.

Are there actual micro benchmarks, or are the tests the best framework /
tool we have to measure performance regressions?

Thanks,
 - Andy

>
> Thanks,
> Masayoshi
>
>
> On 9/1/2010 7:55 AM, Yoshito Umaoka wrote:
>
>> I wanted to make sure if we do not introduce any performance regression.
>> So I ran the standard test target jdk_lang, jdk_text and jdk_util on our
>> latest code and its base build (b106). I ran the test 5 times for each and
>> posted the raw log files below -
>>
>> b106 -> http://icu-project.org/~yoshito/LocaleTestLogs/b106/
>> b106 + locale changes ->
>> http://icu-project.org/~yoshito/LocaleTestLogs/le/
>>
>> I wrote a small code to collect execution time of each text case
>> (average/min/max). You can see the changes below -
>>
>> jdk_lang -> http://icu-project.org/~yoshito/LocaleTestLogs/lang.txt
>> jdk_text -> http://icu-project.org/~yoshito/LocaleTestLogs/text.txt
>> jdk_util -> http://icu-project.org/~yoshito/LocaleTestLogs/util.txt
>>
>> I reviewed the output briefly. So far, I did not see any obvious
>> performance regression problems.
>>
>> -Yoshito
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/locale-enhancement-dev/attachments/20100901/b2480def/attachment-0001.html