[rfc][icedtea-web] known to fail annotation

[Jiri Vanek]

On 05/15/2012 04:16 PM, Omair Majid wrote:
> Hi Jiri,

Hi! Answered as expected ;)
>
> On 05/14/2012 10:39 AM, Jiri Vanek wrote:
>> Hi! This annotation should serve for marking tests which are
>> representing the issue and so are constantly failing.
>
> Sorry, but I don't follow. If tests are failing, surely we can see that
> in the test reports (make check and friends), right? What advantage does
> this annotation give us over that?

Well. The reason why it should be included was recently discussed. A lot of tests are failing, 
because they are reproducing issue. Also I have plenty of new reproducers which should be definitely 
  pushed. And if there will be to much failing reproducers then no one will see some issue.

Now when somebody will break something by his changeset, and eg new failure will arise, it is easy 
to overlook new red line between reproducers. (both unit and "my")

By on sentence - the reason is to separate regressions from reproducers.

>
>> I have dared also to extend it a bit for marking of tests which are not
>> failing always, but are failing often by some abstract percentage of how
>> often they are failing.
>
> I disagree with the concept of percentage. I don't see the point of
> tests that (occasionally) fail. If tests fails all the time, that's
> great. If tests don't fail all the time, the test is broken. I don't see
> how the percentage helps in either case.

Answer expected:) And I agree it it very discutable. The issue here is that I'm not able to 
reproduce the failure. And in case of cache reproducers (which are 90% of randomly failing 
reproducers) are actually representing the issue. Because cache is not protected from simultaneous 
read/write from multiple javaws. It is just luck when they sometimes pass (because of some delayed 
r/w on system)

>
>> I believe that this annotation will be very useful, as Reproducers
>> representing the issue are strong leverage (for me;) to fix the
>> underlying issue.
>
> Okay, but aren't the reproducers failing enough reason (or leverage, if
> you will) to fix it?

yes. they are. But can  hide regression.

hmhm.. Ready for looong discussion :)

J.
>
> Cheers,
> Omair