[icedtea-web] RFC: disable tests that are known to fail

[Jiri Vanek]

On 03/21/2012 03:57 PM, Omair Majid wrote:
> Hi,
>
> IcedTea-Web contains a number of unit tests. Their usefulness is
> lessened because some of the tests always fail and so many (most?)
> developers will not actually run them. Even if developers do run the
> tests, they may miss a regression or ignore it ("X other tests fail,
> what's the big deal with one more failure"). I would like to have all
> the unit tests pass. One way is to fix the underlying problem, of
> course. The other is to disable the test temporarily.
>
> The attached patch disables a few parser tests that we know are broken
> and we wont be fixing soon. The disabled tests all check if the parser
> can handle mixed content in xml files (the embedded nanoxml cant) and if
> the parser can handle malformed xml files (pretty much all xml parsers
> wont).
>
> I would like to keep these tests around (because they do represent
> actual bugs) but disable them to give developers greater incentive to
> run all tests and always check that they pass.
>

Personally I'm against the disabling. My opinion is that once thy will be marked as @Ignored, they 
will be forgotten. Falling test is still representing the issue, what I consider as correct. In this 
matter I'm really missing  TestNg grouping, and put them to some group "should fail" with 
description "until fix xyz is done". But I'm willing to add such annotation rather then disabling them.

> Any thoughts or comments?
>
> Thanks,
> Omair

J.