RFR: JDK-8212028: Use run-test makefile framework for testing in Oracle's Mach5

[Jonathan Gibbons]

On 10/11/18 3:40 PM, David Holmes wrote:
> Hi Erik,
>
> On 12/10/2018 8:29 AM, Erik Joelsson wrote:
>> Hello,
>>
>> (adding serviceability-dev and hotspot-dev for test changes)
>>
>> Bug: https://bugs.openjdk.java.net/browse/JDK-8212028
>>
>> Webrev: 
>> http://cr.openjdk.java.net/~erikj/8212028/webrev.01/index.html (From 
>> ihse-runtestprebuilt-branch in jdk-sandbox)
>>
>> In order to fully adopt the new run-test framework, we need to switch 
>> over the automated and distributed testing system at Oracle to the 
>> new framework. To get this to work, there are number of issues that 
>> needed to be fixed. Here follows a brief explanation, see bug for 
>> more details.
>>
>> For RunTest.gmk and related makefiles there are a number of minor 
>> tweaks to support all the necessary control variables that are 
>> currently used for the old test makefiles, as well as correcting some 
>> test setup settings.
>>
>> In addition to that, some tests also needed to be modified:
>>
>> Timeouts
>> The current default timeoutFactor in the makefiles is 4. However, the 
>> old Mach5 executor overrides that to 10. I don't think it should 
>> dabble with such things and leave it to the makefiles, the user, or a 
>> specific job definition, so with the new run-test executor, it no 
>> longer does. This means many tests now have a much shorter effective 
>> timeout. Because of this, we need to increase the timeout on some 
>> that are now prone to timing out. I have run tier1-5 a few times to 
>> try and find these and added /timeout=300 (which will result in the 
>> same effective timeout as before) when specific tests seemed 
>> problematic.
>
> This should be fixed in the tier job definitions not the individual 
> tests. We have moved away from putting explicit timeouts on individual 
> tests and instead rely on the framework timeout being set appropriately.
>
> David
> -----

David,

That's a suboptimal policy. because it means you're relying on the 
framework handling the worst case test.

As far as jtreg goes, the default timeout for each step is 2 mins, which 
is intended to be enough for the test to reliably run within that time 
on a reasonably modern developer-class machine.  A test which always 
times out on a good machine should use a test-specific increased timeout.

Where the framework can help is, if tests are being run on an old or 
slow machine, or if test run args are provided that will cause the test 
to run significantly slower than usual, then the framework can/should 
start scaling up the timeout factor.

-- Jon
>
>> test/hotspot/jtreg/runtime/appcds/jvmti/InstrumentationTest.java
>> This test spawns a child process and tries to locate it using the 
>> attach api, by looking for a unique token in the command line string 
>> of the spawned JVM. The problem is that the command line string it 
>> gets from the attach api is truncated and the token is last on the 
>> command line. This normally works well, but the arguments before it 
>> are 3 files, with full absolute paths inside the jtreg work 
>> directory. With Mach5 we have pretty deep work directories, and with 
>> run-test, we make them even deeper. This unfortunately trips the 
>> limit and the test fails. I have fixed this by reordering the 
>> arguments to the child process.
>>
>> /Erik
>>