[9] RFR(S): 8050407: Add jtreg compiler tests to Hotspot JPRT jobs

Fri Sep 5 18:50:21 UTC 2014

Note, we still need to include only useful quick-running tests. And I 
think we should have no more then 4 sets. If you can squeeze into 3 sets 
it will be great.

Thanks,
Vladimir

On 9/5/14 11:34 AM, Vladimir Kozlov wrote:
> Zoltan,
>
> Christian Tornqvist gave a great suggestion in chat room. We can split
> tests in several sets each taking no more than 10 min. Runtime has
> hotspot_runtime and hotspot_runtime_closed. So we can have several
> hotspot_compiler_tests1, etc too.
>
> Could you, please, investigate this?
>
> Thanks,
> Vladimir
>
> On 9/5/14 7:17 AM, Zoltán Majó wrote:
>> Hi,
>>
>>
>> Thank you for the feedback.
>>
>> On 09/04/2014 09:33 PM, Vladimir Kozlov wrote:
>>> http://cr.openjdk.java.net/~zmajo/8050407/webrev.01/
>>>
>>> This looks better.
>>>
>>> On 9/4/14 1:42 AM, Albert wrote:
>>>> Hi,
>>>>
>>>> doesn't it make more sense to select tests based on their 'usefulness'?
>>>
>>> I agree. But we need to take into account its running time too.
>>> I assume Zoltan did a quick review of tests on usefulness.
>>
>> I went through the list of tests once more. I found 15 tests not that
>> useful (at least in my view), so I replaced them with 14 tests that I
>> found more useful.
>>
>> We have now 83 tests in total, here is the new webrev:
>>
>> http://cr.openjdk.java.net/~zmajo/8050407/webrev.02/
>>
>> I ran the tests three times on both Stockholm and West. Here are the
>> timings for the runs:
>>
>> WEST:
>>
>> solaris_sparcv9-fastdebug-c2-hotspot_compiler     success(09m 49s)
>>        USED:     hostname=sca00bvw platform=solaris_sparc_5.11
>> osname=solaris osarch=sparc cpus=6 parallelcount=6 ram=31744MB
>> cpufreqmhz=2848 instance=P1
>>
>> solaris_sparcv9-fastdebug-c2-hotspot_compiler     success(06m 36s)
>>        USED:     hostname=sca00bnp platform=solaris_sparc_5.11
>> osname=solaris osarch=sparc cpus=16 parallelcount=16 ram=32768MB
>> cpufreqmhz=3600 instance=P1
>>
>> solaris_sparcv9-fastdebug-c2-hotspot_compiler     success(09m 32s)
>>        USED:     hostname=sc11152346 platform=solaris_sparc_5.11
>> osname=solaris osarch=sparc cpus=2 parallelcount=2 ram=7900MB
>> cpufreqmhz=2848 instance=T1
>>
>>
>> STOCKHOLM:
>>
>> solaris_sparcv9-fastdebug-c2-hotspot_compiler     success(08m 05s)
>>        USED:     hostname=sthjprtsol40 platform=solaris_sparc_5.11
>> osname=solaris osarch=sparc cpus=12 parallelcount=12 ram=24576MB
>> cpufreqmhz=2848 instance=P1
>>
>> solaris_sparcv9-fastdebug-c2-hotspot_compiler     success(08m 27s)
>>        USED:     hostname=sthjprtsol32 platform=solaris_sparc_5.11
>> osname=solaris osarch=sparc cpus=12 parallelcount=12 ram=24576MB
>> cpufreqmhz=2848 instance=P1
>>
>> solaris_sparcv9-fastdebug-c2-hotspot_compiler     success(08m 14s)
>>        USED:     hostname=sthjprtsol38 platform=solaris_sparc_5.11
>> osname=solaris osarch=sparc cpus=12 parallelcount=12 ram=24576MB
>> cpufreqmhz=2848 instance=P1
>>
>>
>> We get close to the 10-minute limit on both queues (9:49 on West and
>> 8:27 on Stockholm).
>>
>> Thank you and best regards,
>>
>>
>> Zoltan
>>
>>>
>>>>
>>>> For example, it seems to me that adding a regression test that was
>>>> introduced
>>>> based on a fix of a segmentation fault in compiled code makes more
>>>> sense
>>>> than adding a test that checks the consistency of rarely used command
>>>> line
>>>> arguments.
>>>
>>> Agree. Which tests in the list do flags testing?
>>>
>>> We should also exclude unstable tests (have to look on history of the
>>> test).
>>>
>>>>
>>>> How about adding tests based on their priorities? Maybe it makes more
>>>> sense
>>>> to add a test of a corresponding P1 bug than adding a test of a P5 bug?
>>>
>>> We can't add P1 test if it takes more then 1 min to run. I would
>>> prefer to run 5 x 10 sec P3 tests instead. We would test more code
>>> this way.
>>> Also Bug's priority was not accurate before as it is now.
>>>
>>> Thanks,
>>> Vladimir
>>>
>>>>
>>>> Thanks,
>>>> Albert
>>>>
>>>> On 08/28/2014 08:02 PM, Zoltán Majó wrote:
>>>>> Hi,
>>>>>
>>>>>
>>>>> please review the following patch.
>>>>>
>>>>>
>>>>> Bug: Add jtreg compiler tests to Hotspot JPRT jobs
>>>>>
>>>>>
>>>>> Problem: The test/TEST.groups file lists JTREG compiler tests executed
>>>>> in JPRT (target name: 'hotspot_compiler'). We need to determine the
>>>>> list of tests from test/compiler that should be executed in JPRT. The
>>>>> total time of execution should be less than 10 minutes on slowest
>>>>> platform.
>>>>>
>>>>>
>>>>> Solution: The slowest platform in JPRT is currently solaris_sparcv9. I
>>>>> executed all open JTREG tests from test/compiler on solaris_sparcv9
>>>>> and measured the "work time" of each test. Then, tests were sorted in
>>>>> ascending order of their work time. To construct the subset, I first
>>>>> added the test with the lowest work time to the subset and then
>>>>> continued adding tests until a time limit L is reached.
>>>>>
>>>>> Limit L is set to 80% of the original time budget (10 minutes). L is
>>>>> set conservatively to account also for JPRT "cleanup", "init", and
>>>>> "finishing" time, as well as for the variation of tests' "work time".
>>>>> (The profiling measurements contain only "work time".)
>>>>>
>>>>> The subset contains 77 tests. The longest executing test in the subset
>>>>> takes 6.9 seconds.
>>>>>
>>>>>
>>>>> Webrev: http://cr.openjdk.java.net/~zmajo/8050407/webrev.00/
>>>>>
>>>>>
>>>>> Testing: Ran subset on all platforms in both west and stockholm JPRT.
>>>>> Execution times of tests are as follows:
>>>>>
>>>>> West:
>>>>>   linux_i586-fastdebug-c1-hotspot_compiler         success(03m 01s)
>>>>>   linux_i586-fastdebug-c2-hotspot_compiler         success(03m 39s)
>>>>>   linux_x64-fastdebug-c2-hotspot_compiler          success(03m 32s)
>>>>>   macosx_x64-fastdebug-c2-hotspot_compiler         success(03m 48s)
>>>>>   solaris_sparcv9-fastdebug-c2-hotspot_compiler success(08m 41s)
>>>>>   solaris_x64-fastdebug-c2-hotspot_compiler        success(03m 00s)
>>>>>   windows_i586-fastdebug-c1-hotspot_compiler       success(03m 39s)
>>>>>   windows_i586-fastdebug-c2-hotspot_compiler       success(03m 54s)
>>>>>   windows_x64-fastdebug-c2-hotspot_compiler        success(04m 51s)
>>>>>
>>>>> Stockholm:
>>>>>   linux_i586-fastdebug-c1-hotspot_compiler         success(02m 31s)
>>>>>   linux_i586-fastdebug-c2-hotspot_compiler         success(02m 39s)
>>>>>   linux_x64-fastdebug-c2-hotspot_compiler          success(02m 53s)
>>>>>   macosx_x64-fastdebug-c2-hotspot_compiler         success(04m 10s)
>>>>>   solaris_sparcv9-fastdebug-c2-hotspot_compiler success(05m 55s)
>>>>>   solaris_x64-fastdebug-c2-hotspot_compiler        success(03m 06s)
>>>>>   windows_i586-fastdebug-c1-hotspot_compiler       success(02m 34s)
>>>>>   windows_i586-fastdebug-c2-hotspot_compiler       success(03m 07s)
>>>>>   windows_x64-fastdebug-c2-hotspot_compiler        success(04m 33s)
>>>>>
>>>>> We get close to the 10 minute budget (87% usage on West for
>>>>> solaris_sparcv9).
>>>>>
>>>>>
>>>>> Thank you and best regards,
>>>>>
>>>>>
>>>>> Zoltan
>>>>>
>>>>>
>>>>>
>>>>
>>