[icedtea-web] RFC: disable tests that are known to fail

[Jiri Vanek]

On 03/21/2012 10:06 PM, Deepak Bhole wrote:
> * Jiri Vanek<jvanek at redhat.com>  [2012-03-21 14:27]:
>> On 03/21/2012 04:54 PM, Omair Majid wrote:
>>> Hi Jiri,
>>>
>>> Thanks for your sharing your thoughts.
>>>
>>> On 03/21/2012 11:12 AM, Jiri Vanek wrote:
>>>> Personally I'm against the disabling. My opinion is that once thy will
>>>> be marked as @Ignored, they will be forgotten.
>>>
>>> Our command line output, as well as most IDEs, should show the number of
>>> tests that have been ignored.
>>
>> Yap, but noone will ever care any more why those tests are ignored.
>>>
>>>> In this matter I'm
>>>> really missing  TestNg grouping, and put them to some group "should
>>>> fail" with description "until fix xyz is done". But I'm willing to add
>>>> such annotation rather then disabling them.
>>>>
>>>
>>> JUnit does support grouping, but the interface to it (if you are not
>>> using maven) is rather crappy.
>>>
>>>> Falling test is still
>>>> representing the issue, what I consider as correct.
>>>
>>> I agree. But if these tests always fail, then developers (I am guilty of
>>> this too) will often ignore running the tests or not pay too much
>>> attention to its output. For a developer, the impact of going from 0
>>> failures to 1 failure is significant, but the impact of going from 20
>>> failures to 21 is not so much. And as a counter-argument, these tests
>>
>> To much true!
>>
>>> have been failing for a while now and they haven't been fixed. What's
>>> the value of these tests?
>> Thy are still representing the issue.
>>
>
> I think that we should then open a bug for them and disable the tests
> then.

Sure! And for most of them is already opened. The Bug annotation should help to ensure bidirectional 
tracking.

>
>>>
>>> To be honest, I don't think there is a "right" answer here, but I am
>>> trying to figure out what's the lesser evil and go with that.
>>
>> As an intersection looks to annotate them @KnownToBeFailing, and
>> allow to run make tests with configured something like
>> skipKnownToBeFailing, which will skip all KnownToBeFailing
>> testmethods.
>>
>> Spam&  Advertisement:
>> This have close connection to @Bug annotation I have posted yesterday O:)
>
> I don't think there should ever be a "known to fail" scenario because as
> Omair said, it will lead to developers being more lax and ignoring the
> output.

Yes. That's why I'm suggesting to make them  marked, and allow them to to be ignored. But not to 
ignore them so forcibly and so  "without-return".

To be clear - I'm for some kind of skipping, but do not like the idea of used @ignored annotation.

J.