[icedtea-web] RFC: disable tests that are known to fail

[Jiri Vanek]

On 03/21/2012 04:54 PM, Omair Majid wrote:
> Hi Jiri,
>
> Thanks for your sharing your thoughts.
>
> On 03/21/2012 11:12 AM, Jiri Vanek wrote:
>> Personally I'm against the disabling. My opinion is that once thy will
>> be marked as @Ignored, they will be forgotten.
>
> Our command line output, as well as most IDEs, should show the number of
> tests that have been ignored.

Yap, but noone will ever care any more why those tests are ignored.
>
>> In this matter I'm
>> really missing  TestNg grouping, and put them to some group "should
>> fail" with description "until fix xyz is done". But I'm willing to add
>> such annotation rather then disabling them.
>>
>
> JUnit does support grouping, but the interface to it (if you are not
> using maven) is rather crappy.
>
>> Falling test is still
>> representing the issue, what I consider as correct.
>
> I agree. But if these tests always fail, then developers (I am guilty of
> this too) will often ignore running the tests or not pay too much
> attention to its output. For a developer, the impact of going from 0
> failures to 1 failure is significant, but the impact of going from 20
> failures to 21 is not so much. And as a counter-argument, these tests

To much true!

> have been failing for a while now and they haven't been fixed. What's
> the value of these tests?
Thy are still representing the issue.

>
> To be honest, I don't think there is a "right" answer here, but I am
> trying to figure out what's the lesser evil and go with that.

As an intersection looks to annotate them @KnownToBeFailing, and allow to run make tests with 
configured something like skipKnownToBeFailing, which will skip all KnownToBeFailing testmethods.

Spam & Advertisement:
This have close connection to @Bug annotation I have posted yesterday O:)



>
> Thanks,
> Omair

J.