<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
On 11/12/2013 23:10, Jonathan Gibbons wrote:
<blockquote cite="mid:52A8F0EA.2050008@oracle.com" type="cite">
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
:<br>
<br>
Yes, the high order point I was trying to make is that something
is wrong if you need to specify a long list of tests to run.
While we all may take whatever short cuts we choose to get our day
to day work done, there should be a standard set of tests[1] that
we agree should be run, and which can be run with reasonably
concise command line args.<br>
<br>
-- Jon<br>
<br>
Ideally, "all" but maybe we're not there yet.<br>
<br>
</blockquote>
Suppose we create a jdk_stable (or other name) group in TEST.groups
for what "we" consider are the stable tests. Once it is defined in
the groups file then it means it can be used by anyone that runs
jtreg directly or anyone that uses the make file to run tests ("make
test TEST=jdk_stable" for example). Whether it deserves its own make
file is another question.<br>
<br>
So suppose we create such a group then what would be the criteria to
be in that group? Clearly the test should be stable in the sense
that it should pass when we don't have a bug. It should also clean
up after itself. Things that come to mind are:<br>
<br>
- should be usable with -agentvm? We have /othervm option for @run
and we also have othervm.dirs, the main point is that they can be
run with either in othervm or agentvm modes and they should just
work. I have deliberately not mentioned -samevm here as it's not
suitable for the jdk tests.<br>
<br>
- needs to work with -concurrency, assuming sufficient resources. Is
that reasonable to require? We have /exclusive and
exclusiveAccess.dirs available for areas that have issues.<br>
<br>
- needs to run in a reasonable time. I don't think we have
guidelines for what is reasonable in the jdk tests but clearly a
test that runs for more than a few minutes needs to be looked at.<br>
<br>
- needs to run headless? Maybe this is controversial but it is
somewhat appealing to skip Xvfb or other setup. Also being selfish,
I'd like to run tests in a terminal window and not have windows
dancing on my desktop.<br>
<br>
- should not require special configuration? Maybe this is
controversial too but there are tests javax/print that fail when
there isn't a printer configured.<br>
<br>
Anything else? If we do something like this then such a group would
need to be maintained, at least for period until all tests are
stable and fast.<br>
<br>
-Alan.<br>
<br>
</body>
</html>