Defaults for @ignored

[Jonathan Gibbons]

On 08/04/2014 01:22 PM, roger riggs wrote:
> Hi,
>
> The default status for jtreg tests that are marked @ignore is to list 
> them as errors
> (instead of quietly ignoring them).
>
> There is a use for putting stress tests into the test directory 
> hierarchy but being able to
> mark them not to be run by default.  A combination of putting them in 
> a group and marking them
> with @ignore would be useful but with the current defaults behavior it 
> reports them as errors.
> It would inconvenient if every developer needed to say -ignore:quiet 
> to run all the other tests.
>
> Suggestions?
>
> Thanks, Roger

Well, I'm not sure the "inconvenient" argument has any significant legs, 
since that is the way it has been since forever. Originally, you had to 
say -k:!ignore (select those tests which don't have the ignore keyword) 
and now you can say -ignore:quiet.  We've always had @ignore-d test and 
we've always had to run the un-ignore'd tests with a command line option.

Personally, I think that putting @ignore on valid stress tests is a poor 
way to go.   I would put a different keyword on the test and then use 
keyword selection to select the tests you want to run. But at some 
point, the fundamental "problem" is that you're putting different types 
of tests into the test suite, and different communities will have 
different expectations of what should be the "default" set of tests.

An alternate direction is for the stress tests to run by default in some 
minimal mode, so that by default, they run but without causing undue 
stress on the system, but if they see an enabling property like 
-Dstress.mode=high then they can kick into overdrive and seriously test 
the system.

You might also want to ask, do you want them to run as stress tests 
under jtreg?   In the langtools/test suite, we have some tests which are 
small and simple when run under jtreg, but which can be run manually to 
behave in a different manner.  Checking for a jtreg property like 
"test.src" is a good way to check if a test is (probably) running under 
jtreg. Or, you can use the args passed to main() to tell whether to do 
default activity or heavyweight activity.

-- Jon