[9] RFR(S): 8050407: Add jtreg compiler tests to Hotspot JPRT jobs

Fri Sep 5 18:34:54 UTC 2014

Zoltan,

Christian Tornqvist gave a great suggestion in chat room. We can split 
tests in several sets each taking no more than 10 min. Runtime has 
hotspot_runtime and hotspot_runtime_closed. So we can have several 
hotspot_compiler_tests1, etc too.

Could you, please, investigate this?

Thanks,
Vladimir

On 9/5/14 7:17 AM, Zoltán Majó wrote:
> Hi,
>
>
> Thank you for the feedback.
>
> On 09/04/2014 09:33 PM, Vladimir Kozlov wrote:
>> http://cr.openjdk.java.net/~zmajo/8050407/webrev.01/
>>
>> This looks better.
>>
>> On 9/4/14 1:42 AM, Albert wrote:
>>> Hi,
>>>
>>> doesn't it make more sense to select tests based on their 'usefulness'?
>>
>> I agree. But we need to take into account its running time too.
>> I assume Zoltan did a quick review of tests on usefulness.
>
> I went through the list of tests once more. I found 15 tests not that
> useful (at least in my view), so I replaced them with 14 tests that I
> found more useful.
>
> We have now 83 tests in total, here is the new webrev:
>
> http://cr.openjdk.java.net/~zmajo/8050407/webrev.02/
>
> I ran the tests three times on both Stockholm and West. Here are the
> timings for the runs:
>
> WEST:
>
> solaris_sparcv9-fastdebug-c2-hotspot_compiler     success(09m 49s)
>        USED:     hostname=sca00bvw platform=solaris_sparc_5.11
> osname=solaris osarch=sparc cpus=6 parallelcount=6 ram=31744MB
> cpufreqmhz=2848 instance=P1
>
> solaris_sparcv9-fastdebug-c2-hotspot_compiler     success(06m 36s)
>        USED:     hostname=sca00bnp platform=solaris_sparc_5.11
> osname=solaris osarch=sparc cpus=16 parallelcount=16 ram=32768MB
> cpufreqmhz=3600 instance=P1
>
> solaris_sparcv9-fastdebug-c2-hotspot_compiler     success(09m 32s)
>        USED:     hostname=sc11152346 platform=solaris_sparc_5.11
> osname=solaris osarch=sparc cpus=2 parallelcount=2 ram=7900MB
> cpufreqmhz=2848 instance=T1
>
>
> STOCKHOLM:
>
> solaris_sparcv9-fastdebug-c2-hotspot_compiler     success(08m 05s)
>        USED:     hostname=sthjprtsol40 platform=solaris_sparc_5.11
> osname=solaris osarch=sparc cpus=12 parallelcount=12 ram=24576MB
> cpufreqmhz=2848 instance=P1
>
> solaris_sparcv9-fastdebug-c2-hotspot_compiler     success(08m 27s)
>        USED:     hostname=sthjprtsol32 platform=solaris_sparc_5.11
> osname=solaris osarch=sparc cpus=12 parallelcount=12 ram=24576MB
> cpufreqmhz=2848 instance=P1
>
> solaris_sparcv9-fastdebug-c2-hotspot_compiler     success(08m 14s)
>        USED:     hostname=sthjprtsol38 platform=solaris_sparc_5.11
> osname=solaris osarch=sparc cpus=12 parallelcount=12 ram=24576MB
> cpufreqmhz=2848 instance=P1
>
>
> We get close to the 10-minute limit on both queues (9:49 on West and
> 8:27 on Stockholm).
>
> Thank you and best regards,
>
>
> Zoltan
>
>>
>>>
>>> For example, it seems to me that adding a regression test that was
>>> introduced
>>> based on a fix of a segmentation fault in compiled code makes more sense
>>> than adding a test that checks the consistency of rarely used command
>>> line
>>> arguments.
>>
>> Agree. Which tests in the list do flags testing?
>>
>> We should also exclude unstable tests (have to look on history of the
>> test).
>>
>>>
>>> How about adding tests based on their priorities? Maybe it makes more
>>> sense
>>> to add a test of a corresponding P1 bug than adding a test of a P5 bug?
>>
>> We can't add P1 test if it takes more then 1 min to run. I would
>> prefer to run 5 x 10 sec P3 tests instead. We would test more code
>> this way.
>> Also Bug's priority was not accurate before as it is now.
>>
>> Thanks,
>> Vladimir
>>
>>>
>>> Thanks,
>>> Albert
>>>
>>> On 08/28/2014 08:02 PM, Zoltán Majó wrote:
>>>> Hi,
>>>>
>>>>
>>>> please review the following patch.
>>>>
>>>>
>>>> Bug: Add jtreg compiler tests to Hotspot JPRT jobs
>>>>
>>>>
>>>> Problem: The test/TEST.groups file lists JTREG compiler tests executed
>>>> in JPRT (target name: 'hotspot_compiler'). We need to determine the
>>>> list of tests from test/compiler that should be executed in JPRT. The
>>>> total time of execution should be less than 10 minutes on slowest
>>>> platform.
>>>>
>>>>
>>>> Solution: The slowest platform in JPRT is currently solaris_sparcv9. I
>>>> executed all open JTREG tests from test/compiler on solaris_sparcv9
>>>> and measured the "work time" of each test. Then, tests were sorted in
>>>> ascending order of their work time. To construct the subset, I first
>>>> added the test with the lowest work time to the subset and then
>>>> continued adding tests until a time limit L is reached.
>>>>
>>>> Limit L is set to 80% of the original time budget (10 minutes). L is
>>>> set conservatively to account also for JPRT "cleanup", "init", and
>>>> "finishing" time, as well as for the variation of tests' "work time".
>>>> (The profiling measurements contain only "work time".)
>>>>
>>>> The subset contains 77 tests. The longest executing test in the subset
>>>> takes 6.9 seconds.
>>>>
>>>>
>>>> Webrev: http://cr.openjdk.java.net/~zmajo/8050407/webrev.00/
>>>>
>>>>
>>>> Testing: Ran subset on all platforms in both west and stockholm JPRT.
>>>> Execution times of tests are as follows:
>>>>
>>>> West:
>>>>   linux_i586-fastdebug-c1-hotspot_compiler         success(03m 01s)
>>>>   linux_i586-fastdebug-c2-hotspot_compiler         success(03m 39s)
>>>>   linux_x64-fastdebug-c2-hotspot_compiler          success(03m 32s)
>>>>   macosx_x64-fastdebug-c2-hotspot_compiler         success(03m 48s)
>>>>   solaris_sparcv9-fastdebug-c2-hotspot_compiler success(08m 41s)
>>>>   solaris_x64-fastdebug-c2-hotspot_compiler        success(03m 00s)
>>>>   windows_i586-fastdebug-c1-hotspot_compiler       success(03m 39s)
>>>>   windows_i586-fastdebug-c2-hotspot_compiler       success(03m 54s)
>>>>   windows_x64-fastdebug-c2-hotspot_compiler        success(04m 51s)
>>>>
>>>> Stockholm:
>>>>   linux_i586-fastdebug-c1-hotspot_compiler         success(02m 31s)
>>>>   linux_i586-fastdebug-c2-hotspot_compiler         success(02m 39s)
>>>>   linux_x64-fastdebug-c2-hotspot_compiler          success(02m 53s)
>>>>   macosx_x64-fastdebug-c2-hotspot_compiler         success(04m 10s)
>>>>   solaris_sparcv9-fastdebug-c2-hotspot_compiler success(05m 55s)
>>>>   solaris_x64-fastdebug-c2-hotspot_compiler        success(03m 06s)
>>>>   windows_i586-fastdebug-c1-hotspot_compiler       success(02m 34s)
>>>>   windows_i586-fastdebug-c2-hotspot_compiler       success(03m 07s)
>>>>   windows_x64-fastdebug-c2-hotspot_compiler        success(04m 33s)
>>>>
>>>> We get close to the 10 minute budget (87% usage on West for
>>>> solaris_sparcv9).
>>>>
>>>>
>>>> Thank you and best regards,
>>>>
>>>>
>>>> Zoltan
>>>>
>>>>
>>>>
>>>
>