[icedtea-web] RFC: disable tests that are known to fail

[Omair Majid]

Hi Jiri,

Thanks for your sharing your thoughts.

On 03/21/2012 11:12 AM, Jiri Vanek wrote:
> Personally I'm against the disabling. My opinion is that once thy will
> be marked as @Ignored, they will be forgotten. 

Our command line output, as well as most IDEs, should show the number of
tests that have been ignored.

> In this matter I'm
> really missing  TestNg grouping, and put them to some group "should
> fail" with description "until fix xyz is done". But I'm willing to add
> such annotation rather then disabling them.
> 

JUnit does support grouping, but the interface to it (if you are not
using maven) is rather crappy.

> Falling test is still
> representing the issue, what I consider as correct. 

I agree. But if these tests always fail, then developers (I am guilty of
this too) will often ignore running the tests or not pay too much
attention to its output. For a developer, the impact of going from 0
failures to 1 failure is significant, but the impact of going from 20
failures to 21 is not so much. And as a counter-argument, these tests
have been failing for a while now and they haven't been fixed. What's
the value of these tests?

To be honest, I don't think there is a "right" answer here, but I am
trying to figure out what's the lesser evil and go with that.

Thanks,
Omair