[icedtea-web] RFC: disable tests that are known to fail

[Deepak Bhole]

* Jiri Vanek <jvanek at redhat.com> [2012-03-21 14:27]:
> On 03/21/2012 04:54 PM, Omair Majid wrote:
> >Hi Jiri,
> >
> >Thanks for your sharing your thoughts.
> >
> >On 03/21/2012 11:12 AM, Jiri Vanek wrote:
> >>Personally I'm against the disabling. My opinion is that once thy will
> >>be marked as @Ignored, they will be forgotten.
> >
> >Our command line output, as well as most IDEs, should show the number of
> >tests that have been ignored.
> 
> Yap, but noone will ever care any more why those tests are ignored.
> >
> >>In this matter I'm
> >>really missing  TestNg grouping, and put them to some group "should
> >>fail" with description "until fix xyz is done". But I'm willing to add
> >>such annotation rather then disabling them.
> >>
> >
> >JUnit does support grouping, but the interface to it (if you are not
> >using maven) is rather crappy.
> >
> >>Falling test is still
> >>representing the issue, what I consider as correct.
> >
> >I agree. But if these tests always fail, then developers (I am guilty of
> >this too) will often ignore running the tests or not pay too much
> >attention to its output. For a developer, the impact of going from 0
> >failures to 1 failure is significant, but the impact of going from 20
> >failures to 21 is not so much. And as a counter-argument, these tests
> 
> To much true!
> 
> >have been failing for a while now and they haven't been fixed. What's
> >the value of these tests?
> Thy are still representing the issue.
> 

I think that we should then open a bug for them and disable the tests
then.

> >
> >To be honest, I don't think there is a "right" answer here, but I am
> >trying to figure out what's the lesser evil and go with that.
> 
> As an intersection looks to annotate them @KnownToBeFailing, and
> allow to run make tests with configured something like
> skipKnownToBeFailing, which will skip all KnownToBeFailing
> testmethods.
> 
> Spam & Advertisement:
> This have close connection to @Bug annotation I have posted yesterday O:)
 
I don't think there should ever be a "known to fail" scenario because as
Omair said, it will lead to developers being more lax and ignoring the
output. 

Cheers,
Deepak