[rfc][icedtea-web] known to fail annotation

[Omair Majid]

Hi Jiri,

On 05/15/2012 10:47 AM, Jiri Vanek wrote:
> On 05/15/2012 04:16 PM, Omair Majid wrote:
> Hi! Answered as expected ;)

:)

> Well. The reason why it should be included was recently discussed. A lot
> of tests are failing, because they are reproducing issue. Also I have
> plenty of new reproducers which should be definitely  pushed. And if
> there will be to much failing reproducers then no one will see some issue.
> 
> Now when somebody will break something by his changeset, and eg new
> failure will arise, it is easy to overlook new red line between
> reproducers. (both unit and "my")

Ah. I see. So will things marked with this annotation not be run by
default? This sounds like a good idea to me.

>>> I have dared also to extend it a bit for marking of tests which are not
>>> failing always, but are failing often by some abstract percentage of how
>>> often they are failing.
>>
>> I disagree with the concept of percentage. I don't see the point of
>> tests that (occasionally) fail. If tests fails all the time, that's
>> great. If tests don't fail all the time, the test is broken. I don't see
>> how the percentage helps in either case.
> 
> Answer expected:) And I agree it it very discutable. The issue here is
> that I'm not able to reproduce the failure. And in case of cache
> reproducers (which are 90% of randomly failing reproducers) are actually
> representing the issue. Because cache is not protected from simultaneous
> read/write from multiple javaws. It is just luck when they sometimes
> pass (because of some delayed r/w on system)
> 

I am trying to figure out how the percentage helps. How do you plan to
use it?

I still think the test is broken in this case. Perhaps we should fix the
test? Or consider rewriting/refactoring some code to make it more easily
testable? Or to put it another way, wouldn't effort spent adding this be
better spent fixing the code/tests?

Cheers,
Omair