jtreg testing integrated

Jonathan Gibbons Jonathan.Gibbons at Sun.COM
Thu May 22 10:00:25 PDT 2008


Mark,

The suggestion regarding -ignore:try is interesting, but I'd have to  
think about
how it could be done. The underlying JT Harness does not support such a
concept, and I doubt that it would be easy to add it. We'd have to  
create a
side table in jtreg of @ignored tests that were executed and which  
passed.
I'll have to go talk to the JT Harness folk to see if I could add that  
info into
a report.

One idea that has been on the table for a while is the idea of a  
"known failure
list".  You do a test run and then compare results against a list  
containing
tests which are regrettably known to fail.  Seems to me we could use
-ignore:run for jtreg, then invoke jtdiff against the KFL to get a  
report of
which tests did not behave as expected, including tests which now pass  
which
previously did not.

-- Jon

On May 22, 2008, at 7:16 AM, Mark Wielaard wrote:

> Hi Jonathan,
>
> On Mon, 2008-05-19 at 16:07 -0700, Jonathan Gibbons wrote:
>> Separately, check out the options for handling @ignore tests. Even on
>> older versions
>> of jtreg you can use "-k:!ignore" to ignore @ignore tests.  (This  
>> works
>> because @ignore
>> tests are given an implicit "ignore" keyword.)   With later  
>> versions of
>> jtreg, you can use
>> -Ignore:{quiet,error,run} to control how @ignore tests should be
>> handled. Using this
>> option, you should be to get closer to the goal of "all tests should
>> pass", meaning
>> that there are less failures and so less need to compare the output
>> results with jtdiff.
>
> This is a really a great feature! For icedtea we now use "-v1 -a
> -ignore:quiet", that give output and results that should be pretty
> familiar to people. And this is the set that I hope we can get to be  
> all
> PASS in the default case.
>
> One extension might be to have a "-ignore:try" that does try to run  
> it,
> that doesn't report it failure, but that does flag it as unexpected
> XPASS to alert people to bugs that are (accidentally) fixed but where
> the testcase was not yet enabled.
>
> Cheers,
>
> Mark
>




More information about the distro-pkg-dev mailing list