[rfc][icedtea-web] KnownToFailInBrowsers annotation patch

[Jana Fabrikova]

Hi Jiri and Adam,

as Jiri suggested I have modified the annotation (this time no new 
annotation KnownToFailInBrowsers, only a new optional parameter 
Browsers[] failsIn() added to the old KnownToFail annotation).

I have modified the JunitLikeXmlOutputListener and 
LessVerboseTextListener also, they identify the combinations of 
testcase+browser that are declared to fail and write 
known-to-fail="true" into the xml file, which is then processed by the 
same xslt as usual.

I have run one reproducer (modification of JToJSEval) in opera, midori 
and epiphany with a combination of failing/passing testcases.

The results in terminal were (screenshot1):

-For the testcase that really failed in midori and epiphany but was 
annotated with @KnownToFail(failsIn={}) :

Passed: JToJSEvalTest.AppletJToJSEval_AnnotatedAsFailsInAll_1plus1_Test 
- opera
  - WARNING This test is known to fail, but have passed!
FAILED: AppletJToJSEval_AnnotatedAsFailsInAll_1plus1_Test - 
midori(JToJSEvalTest) JToJSEval: the J and JS outputs are not equal!
  - This test is known to fail
FAILED: AppletJToJSEval_AnnotatedAsFailsInAll_1plus1_Test - 
epiphany(JToJSEvalTest) JToJSEval: the J and JS outputs are not equal!
  - This test is known to fail

-For the testcase that really failed in midori and epiphany but was 
annotated with @KnownToFail :

Passed: 
JToJSEvalTest.AppletJToJSEval_AnnotatedAsDefaultFailsInAll_1plus1_Test - 
opera
  - WARNING This test is known to fail, but have passed!
FAILED: AppletJToJSEval_AnnotatedAsDefaultFailsInAll_1plus1_Test - 
midori(JToJSEvalTest) JToJSEval: the J and JS outputs are not equal!
  - This test is known to fail
FAILED: AppletJToJSEval_AnnotatedAsDefaultFailsInAll_1plus1_Test - 
epiphany(JToJSEvalTest) JToJSEval: the J and JS outputs are not equal!
  - This test is known to fail

-For the testcase that really failed in midori and epiphany and was 
annotated with @KnownToFail(failsIn={Browsers.midori, Browsers.epiphany})

Passed: JToJSEvalTest.AppletJToJSEval_FailsInSomeBrowsers_1plus1_Test - 
opera
FAILED: AppletJToJSEval_FailsInSomeBrowsers_1plus1_Test - 
midori(JToJSEvalTest) JToJSEval: the J and JS outputs are not equal!
  - This test is known to fail
FAILED: AppletJToJSEval_FailsInSomeBrowsers_1plus1_Test - 
epiphany(JToJSEvalTest) JToJSEval: the J and JS outputs are not equal!
  - This test is known to fail

-For the testcase that really failed in opera (only because opera 
crashed, it does not normally fail) and was not annotated:

FAILED: AppletJToJSEval_typeof_Test - opera(JToJSEvalTest) JToJSEvalTest 
stdout should contain JToJSEval applet initialized. but it didnt.
Passed: JToJSEvalTest.AppletJToJSEval_typeof_Test - midori
Passed: JToJSEvalTest.AppletJToJSEval_typeof_Test - epiphany

-For the testcase that passes in all browsers and was not annotated:

Passed: JToJSEvalTest.AppletJToJSEval_location_Test - opera
Passed: JToJSEvalTest.AppletJToJSEval_location_Test - midori
Passed: JToJSEvalTest.AppletJToJSEval_location_Test - epiphany


-Total summary:

Total tests run: 15; From  those : 8 known to fail
Test known to fail: passed: 2; failed: 6; ignored: 0
Test results: passed: 8; failed: 7; ignored: 0


The results in html are attached as index_reproducers.html and screenshot2).


Please see the attached patch,
cheers,
Jana


On 03/13/2013 05:45 PM, Jiri Vanek wrote:
> On 03/13/2013 05:13 PM, Jana Fabrikova wrote:
>> Hello,
>>
>> I am sending for your consideration a patch that introduces the
>> @KnownToFailInBrowsers annotation, which may be useful in some of the
>> reproducers (namely passing arrays from JS->J does work in opera and
>> firefox and does not work in google-chrome, chromium, midori and
>> epiphany, and i think it is more problem of the browsers than of
>> icedtea-web plugin).
>>
>> The annotation is detected in
>> junit-runner/JunitLikeXmlOutputListener.java and
>> junit-runner/LessVerboseTextListener.java and reflected in the console
>> and xml output of run-netx-dist-tests.
>>
>> Cheers,
>> Jana
>>
>> 2013-03-13 Jana Fabrikova <jfabriko at redhat.com>
>>
>> *
>> /tests/test-extensions/net/sourceforge/jnlp/annotations/KnownToFailInBrowsers.java
>>
>> the implementation of new annotation, which has an array of
>> browsers of type Browsers[] named failsIn
>>
>> * /tests/junit-runner/JunitLikeXmlOutputListener.java
>> in method (testDone) the testcases that are known to fail in
>> current browser are detected in addition to the tests that are
>> k2f in all browsers
>>
>> * /tests/junit-runner/LessVerboseTextListener.java
>> added method (getK2FinB) reading the annotion,
>> in method (printK2F) the testcases that are known to fail in
>> current browser are detected in addition to the tests that are
>> k2f in all browsers
>
> hmhm... I'm not sure If new type of annotation is needed. I would prefer
> extension of KnownTofail as it is.
> Now it have no parameter. In your new implementation it will have
> optional  Browsers[] failsIn(). If it will be  null/empty, then it will
> behave as we are  used now. Otherwise it will be more sophisticated
> What do you think?
>
> For the implementation itself - uless I missed something, I'm terribly
> missing the browser recognition  handling in xml/plaintext generation
> and in xslt sheet.
> Also some example usage of annotaion would be nice and as a top candy
> the screenshots of outputs can be attached.
>
>
> J.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: annotk2f-version2.patch
Type: text/x-patch
Size: 7039 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/distro-pkg-dev/attachments/20130318/05c7e3b1/annotk2f-version2.patch 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/distro-pkg-dev/attachments/20130318/05c7e3b1/index_reproducers.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: screenshot1.png
Type: image/png
Size: 183774 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/distro-pkg-dev/attachments/20130318/05c7e3b1/screenshot1.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: screenshot2.png
Type: image/png
Size: 147489 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/distro-pkg-dev/attachments/20130318/05c7e3b1/screenshot2.png