Passing time factor to tests run under jtreg

[Gary Adams]

On 11/15/11 8:33 PM, David Holmes wrote:
> Hi Gary,
>
> On 16/11/2011 6:14 AM, Gary Adams wrote:
>> I've been scanning a number of the slow machine test
>> bugs that are reported and wanted to check to see if
>> anyone has looked into time dependencies in the regression
>> tests previously. From what I've been able to learn so far
>> individual bugs can use the "timeout" parameter to indicate to
>> the test harness an expected time to run.
>>
>> The test harness has command line arguments where it can
>> filter out tests that take too long (timelimit) or can apply a
>> multiplier to
>> to the timeout when conditions are known to slow down the process
>> (timeoutFactor). e.g. 8X for a slow machine or running with -Xcomp
>>
>> I see that there are some wrappers that can be applied around running
>> a particular test to allow processing before main(). Could this 
>> mechanism
>> be exploited so the harness command line options could be made known
>> to the time dependent tests as command line arguments or as system
>> properties?
>>
>> My thought is the current timeout granularity is too large and only 
>> applies
>> to the full test execution. If a test knew that a timeoutFactor was to
>> be applied,
>> it could internally adjust the time dependent delays appropriately. e.g.
>> not every sleep(), await(), join() with timeouts would need the
>> timeoutFactor applied.
>
> I don't quite get what you mean about the timeouts applied to sleeps, 
> awaits etc. Depending on the test some of these are delays (eg sleep 
> is usually used this way) because it may not be feasible (or even 
> possible) to coordinate the threads directly; while others (await, 
> wait etc) are actual timeouts - if they expire it is an error because 
> something appears to have gone wrong somewhere (of course this can be 
> wrong because the timeout was too short for a given situation).
The timeout being too short for a slow machine is the specific
condition I'd like to address.
>
> As many of these tests have evolved along with the testing 
> infrastructure it isn't always very clear who has responsibility for 
> programming defensive timeouts. And many tests are designed to be run 
> stand-alone or under a test harness, where exceptions due to timeouts 
> are preferable to hangs.
Some of the bug reports state an intermittent failure.
Could be failing due to busy machine, slow machine or
aggressive command line options.

If a test has an advertized timeout, then it needs to be run stand-alone.

If aggressive command line options are used the timefactor multiplier
can compensate for the whole test timeout.

I believe we can identify slow machines and apply the same timefactor
multiplier.

>
> Further, while we can add a scale factor for known retarding factors - 
> like Xcomp - there's no general way to assess the target machine 
> capability (# cores, speed) and load, as it may impact a given test. 
> And historically there has been a lack of resources to investigate and 
> solve these issues.
I'm not sure why test machine characteristics are not known at test time.
Obviously some agent setup has been done to identify chip
architecture and operating system. The cpu speed is just another
attribute of the test machine.

I understand the lack of resources to address this area in the past,
but resources are being applied to ensure jdk8 is appropriate for
embedded deployments. This seems like another low hanging
fruit. e.g. not technically difficult, not particularly controversial
>
> Cheers,
> David
>
>> Before any test could be updated the information would need to be 
>> available
>> from the test context.
>>
>> Any feedback/pointers appreciated!
>>
>>
>> See
>> timeoutFactorArg
>> jtreg/src/share/classes/com/sun/javatest/regtest/Main.java
>> runOtherJVM()
>> jtreg/src/share/classes/com/sun/javatest/regtest/MainAction.java
>> maxTimeoutValue
>> jtreg/src/share/classes/com/sun/javatest/regtest/RegressionParameters.java 
>>
>>
>>