[9] RFR(S): 8050407: Add jtreg compiler tests to Hotspot JPRT jobs

[Zoltán Majó]

Hi Nils,


thank your for looking into this.

On 09/05/2014 04:12 PM, Nils Eliasson wrote:
> Have you considered excluding the expensive tests instead of including 
> the cheap?

Yes, I did. But I think results would not get better (we would not get a 
larger number of tests and we would most likely not have better tests). 
The reasons are as follows.

I considered 408 tests in total. If you consider a test with an 
execution time < 10 second as cheap, then 253 tests are cheap. So by 
eliminating expensive tests we have to create a subset from these.

I selected 83 tests with a total execution time of around 10 minutes. 
The execution time of 94% of the selected tests is between 6 and 7 seconds.

But there are 170 tests remaining that I could have selected.

The total execution time of the remaining tests is 22 minutes, each test 
takes  between 7 and 10 seconds to execute. Taking any of the remaining 
tests (with an execution time between 7 and 10 seconds) would most 
likely "kick out" more than one test from the subset (at least in the 
average case). That would most likely reduce test coverage.

Maybe there exists an important test for which that is worth doing, but 
I don't know which tests that would be. If you know any such test, I 
would be glad to update the subset.

> Then we could avoid mistakes when people write new jtreg tests, run 
> them through JPRT and forgetting the tests will be disabled by default.

If people want to run new JTREG tests through JPRT they have to update 
the JPRT configuration files. So I think new tests won't be forgotten. 
But please correct me if I'm wrong.

Thank you!

Best regards,


Zoltan

>
> //Nils
>
> On 2014-08-28 20:02, Zoltán Majó wrote:
>> Hi,
>>
>>
>> please review the following patch.
>>
>>
>> Bug: Add jtreg compiler tests to Hotspot JPRT jobs
>>
>>
>> Problem: The test/TEST.groups file lists JTREG compiler tests 
>> executed in JPRT (target name: 'hotspot_compiler'). We need to 
>> determine the list of tests from test/compiler that should be 
>> executed in JPRT. The total time of execution should be less than 10 
>> minutes on slowest platform.
>>
>>
>> Solution: The slowest platform in JPRT is currently solaris_sparcv9. 
>> I executed all open JTREG tests from test/compiler on solaris_sparcv9 
>> and measured the "work time" of each test. Then, tests were sorted in 
>> ascending order of their work time. To construct the subset, I first 
>> added the test with the lowest work time to the subset and then 
>> continued adding tests until a time limit L is reached.
>>
>> Limit L is set to 80% of the original time budget (10 minutes). L is 
>> set conservatively to account also for JPRT "cleanup", "init", and 
>> "finishing" time, as well as for the variation of tests' "work time". 
>> (The profiling measurements contain only "work time".)
>>
>> The subset contains 77 tests. The longest executing test in the 
>> subset takes 6.9 seconds.
>>
>>
>> Webrev: http://cr.openjdk.java.net/~zmajo/8050407/webrev.00/
>>
>>
>> Testing: Ran subset on all platforms in both west and stockholm JPRT. 
>> Execution times of tests are as follows:
>>
>> West:
>>   linux_i586-fastdebug-c1-hotspot_compiler         success(03m 01s)
>>   linux_i586-fastdebug-c2-hotspot_compiler         success(03m 39s)
>>   linux_x64-fastdebug-c2-hotspot_compiler          success(03m 32s)
>>   macosx_x64-fastdebug-c2-hotspot_compiler         success(03m 48s)
>>   solaris_sparcv9-fastdebug-c2-hotspot_compiler     success(08m 41s)
>>   solaris_x64-fastdebug-c2-hotspot_compiler        success(03m 00s)
>>   windows_i586-fastdebug-c1-hotspot_compiler       success(03m 39s)
>>   windows_i586-fastdebug-c2-hotspot_compiler       success(03m 54s)
>>   windows_x64-fastdebug-c2-hotspot_compiler        success(04m 51s)
>>
>> Stockholm:
>>   linux_i586-fastdebug-c1-hotspot_compiler         success(02m 31s)
>>   linux_i586-fastdebug-c2-hotspot_compiler         success(02m 39s)
>>   linux_x64-fastdebug-c2-hotspot_compiler          success(02m 53s)
>>   macosx_x64-fastdebug-c2-hotspot_compiler         success(04m 10s)
>>   solaris_sparcv9-fastdebug-c2-hotspot_compiler     success(05m 55s)
>>   solaris_x64-fastdebug-c2-hotspot_compiler        success(03m 06s)
>>   windows_i586-fastdebug-c1-hotspot_compiler       success(02m 34s)
>>   windows_i586-fastdebug-c2-hotspot_compiler       success(03m 07s)
>>   windows_x64-fastdebug-c2-hotspot_compiler        success(04m 33s)
>>
>> We get close to the 10 minute budget (87% usage on West for 
>> solaris_sparcv9).
>>
>>
>> Thank you and best regards,
>>
>>
>> Zoltan
>>
>>
>>
>