[icedtea-web] RFC: disable tests that are known to fail

[Jiri Vanek]

On 03/22/2012 04:14 PM, Omair Majid wrote:
> On 03/22/2012 04:13 AM, Jiri Vanek wrote:
>> To be clear - I'm for some kind of skipping, but do not like the idea of
>> used @ignored annotation.
>
> Assume that most developers are not running the junit tests through the
> command line but using their ide's junit integration. Any ideas on how
> skipping can be accomplished? (Without configuring their ide). We want a
> default 'make check' (and whatever method an ide might use to run the
> junit tests) to show that all tests have passed.

Eg I'm not doing so. I'm running in ide just suite or two which I expect to be affected by my work, and then run make all-possible-test and check reports/coverage.
But yes, from this point of view own annotation is not an solution.

>
> To look at it from the other point of view, why dont we have a flag/env
> var that says ignore the @Ignore annotation during make check (or
> something like that)?

ha! Nice reversed idea :) In this case My opinion is to  have configure-able variable to ignore @ignore ( :D ) during make.
(As I'm thinking about implementation of ignore @Ignore ... it can be tricky!)

But when this reversed idea is connected with the "preferring ide" mentioned above,  hmhm my-mind-is-deadlocked.


I'm still more preferring own annotation:
  - in ide it will fail, developer will check, why the hell, will see all the reasons (eg that it is representing an issue) and ignoring through rest of his patching.
  - After his patch is done, he will return to the discovered known-to-be-failing test and will make new patch :)
  - will run make check/distcheck both with enabled/disabled "ignore KnownToBeFailing" and so will know all the impacts


(Ignore) @ignore
  - in ide will run testsuite, will see that nothing failed, something was ignored, will never check what and will not care why...
  - will see both in ide and make all-tests that everything is passing, something is ignored (but who cares!)
  - will never be "forced" to discover amazing --ignooreIgnore configure option :)


I'm definietly for marking known-too-be-failing tests, but whether own annotation or Ignore (with ignore @ignore) I'm 50/50 now .


J.

>
> Thanks,
> Omair