JDK 8 b118 ea test results are now available

[Alan Bateman]

On 11/12/2013 23:10, Jonathan Gibbons wrote:
> :
>
> Yes, the high order point I was trying to make is that something is 
> wrong if you need to specify a long list of tests to run.   While we 
> all may take whatever short cuts we choose to get our day to day work 
> done, there should be a standard set of tests[1] that we agree should 
> be run, and which can be run with reasonably concise command line args.
>
> -- Jon
>
> Ideally, "all" but maybe we're not there yet.
>
Suppose we create a jdk_stable (or other name) group in TEST.groups for 
what "we" consider are the stable tests. Once it is defined in the 
groups file then it means it can be used by anyone that runs jtreg 
directly or anyone that uses the make file to run tests ("make test 
TEST=jdk_stable" for example). Whether it deserves its own make file is 
another question.

So suppose we create such a group then what would be the criteria to be 
in that group? Clearly the test should be stable in the sense that it 
should pass when we don't have a bug. It should also clean up after 
itself. Things that come to mind are:

- should be usable with -agentvm? We have /othervm option for @run and 
we also have othervm.dirs, the main point is that they can be run with 
either in othervm or agentvm modes and they should just work. I have 
deliberately not mentioned -samevm here as it's not suitable for the jdk 
tests.

- needs to work with -concurrency, assuming sufficient resources. Is 
that reasonable to require? We have /exclusive and exclusiveAccess.dirs 
available for areas that have issues.

- needs to run in a reasonable time. I don't think we have guidelines 
for what is reasonable in the jdk tests but clearly a test that runs for 
more than a few minutes needs to be looked at.

- needs to run headless? Maybe this is controversial but it is somewhat 
appealing to skip Xvfb or other setup. Also being selfish, I'd like to 
run tests in a terminal window and not have windows dancing on my desktop.

- should not require special configuration? Maybe this is controversial 
too but there are tests javax/print that fail when there isn't a printer 
configured.

Anything else? If we do something like this then such a group would need 
to be maintained, at least for period until all tests are stable and fast.

-Alan.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/quality-discuss/attachments/20131212/62dccb94/attachment.html