[rfc] [icedtea-web] finally and proudly... TestInBrowsers annotation

[Jiri Vanek]

As introduced in  "Extending jnlp reproducers for html+applet reproducers" from 20.3 2012, after 
make links, configurable browsers, whitelist, multiple signatures, custom makefiles, gtk3 (midori 
and epiphany), knowntofail,  logging and many other improvements here is final version 
@TestInBrowser annotation on "future" head. This patch need to have applied  "[rfc][icedtea-web] 
finish to get rid of std out/err in testsuites" patch (as connection between log-id and browser-id 
is very close)

Other logic remains same (just adapted original text):

At first all available (supported and available) browsers are discovered by configure or set by 
user. Make links can ten ensure that all used browsers are configured properly

So just before running reproducers those links created (bad luck that for opera this linking need 
roots privileges :( ) are ensured and used browsers are passed  into testing framework by 
-Dused.browsers=$(FIREFOX):$(CHROMIUM):$(CHROME):$(OPERA):$(MIDORI):$(EPIPHANY)

This variable is inside tranlsated by BrowserFactory to proxies representing individual browser 
(providing theirs binary, theirs plugin dirs, theirs individual settings.

Not-likely in original one, one global switch is available for cofigure - --with-browser-tests -
yes - default behaviour, all browser tests behave as designed
no - ignore all browser tests
all - is handling all @TestInBrowsers as Browsers.all (good for daily report)
one  - is handling all @TestInBrowsers as Browsers.one  (good for quick tests report), with 
exception  when some set of concrete browsers is set, then this "one" is selected from them

Tests are invoked in browser by lunching already introduced server.executeBrowser(resource).
But there is little bit of magic  around this procedure - annotation @TestInBrowsers(testIn={...}) 
Where values are singletons pointing to individual concrete proxies. or special case ONE, which 
represents any one of configured browsers - so method will be invoked just once times with this 
browser injected, NONE - which is mostly just for testing purposes and will inject non existing 
browser. Or ALL which will inject all configured browsers.
In case that ALL returns zero of browsers, or individual requested proxies are not configured, or 
when requested is one ONE  and no Browser is returned, then NONE browser is injected. (causing 
none-browser result)

This injection is very real, and if Eg FIREFOX,CHROMIUM and OPERA are requested by annotation, then 
the testing method is really launched three times - once for each browser. if ALL is requested by 
annotation, then method is really iterated for each configured browser.
This injection is also propagated into reporting, where is displayed method - browser. When no 
browser is configured, then all "browser" tests will fail as for "unset_browser" (which is 
representation of mentioned NONE)
This injection and iterating over methods is done inside custom junit runner BrowserTestRunner, 
which is invoking the tested method several times, and is always injecting desired browser into its 
  ServerAccess (included by inheriting of BrowserTest). So the only thing which final reproducer 
writer is to do is:
  - prepare reproducer as is common now
  - make his testcase to extends BrowserTest
  - annotate methods which have something to do with browser which browsers they should be tested 
against
      - there is still possibility to work without this annotation and get browsers directly from 
factory or to inject custom browsers, or to run whatever he wants :)
-the port handling and other deployment stuff have been used form current jnlp reproducers



browserAnnotation-impl

     introduced possibility to run comfortably applets+html reproducers
     * Makefile.am: used BROWSER_TESTS_MODIFICATION variable to pass global switch from configure
     * acinclude.m4: (IT_SET_GLOBAL_BROWSERTESTS_BHAVIOUR) new method handling --with-browser-tests
     * configure.ac: used IT_SET_GLOBAL_BROWSERTESTS_BHAVIOUR
     switch and passing  BROWSER_TESTS_MODIFICATION variable to Makefile.
     * tests/netx/jnlp_testsengine/net/sourceforge/jnlp/ServerAccess.java:
     (currentBrowser) variable holding injected browser for ServerAccess instance
     (loggedBrowser) static variable keeping id of (currentBrowser) for static logging purposes
     (modifyMethodWithForBrowser) new function changing the name of method to "method - browser" for 
logging purposes
     (getBrowserLocation) - returning path to process to be launched when browser requested
     (getBrowserParams) - gathering set default's browser settings
     (set/getCurrentBrowsers) - set browser by id/return id of set browser
     (set/getCurrentBrowser) - set browser instance /returns instance of current browser
     (executeBrowser) family of methods now cooperate with above methods for default set browser
     (executeBrowser(Browser) family to work with implicit browser
     * tests/netx/jnlp_testsengine/net/sourceforge/jnlp/annotations/TestInBrowsers.java: annotation 
for determining which browser(s) to use with annotated method
     * tests/netx/jnlp_testsengine/net/sourceforge/jnlp/browsertesting/Browser.java: interface for 
dealing with various browsers
     * tests/netx/jnlp_testsengine/net/sourceforge/jnlp/browsertesting/BrowserFactory.java:
     singleton for mapping configured browsers x requested browser x browsers proxies
     * tests/netx/jnlp_testsengine/net/sourceforge/jnlp/browsertesting/BrowserTest.java:
     Forefather of all testcases which have methods to do tests iniside browser. Is allowing correct 
annotation -> proxy trasnver to VirtualServer for selected method and is requesting custom runner 
from junit framework
     * tests/netx/jnlp_testsengine/net/sourceforge/jnlp/browsertesting/BrowserTestRunner.java:
     custom test Ruuner which is responsible for translating annotation and run the method 
mutlipletimes for each requested browser and to name it properly
     * tests/netx/jnlp_testsengine/net/sourceforge/jnlp/browsertesting/Browsers.java: enumeration of 
abstract browsers and theirs sets or subsets.
     * tests/netx/jnlp_testsengine/net/sourceforge/jnlp/browsertesting/browsers/*: individual 
browsers proxies and theirs abstractions,namely:
     * tests/netx/jnlp_testsengine/net/sourceforge/jnlp/browsertesting/browsers/Opera.java:
     * tests/netx/jnlp_testsengine/net/sourceforge/jnlp/browsertesting/browsers/Firefox.java:
     * tests/netx/jnlp_testsengine/net/sourceforge/jnlp/browsertesting/browsers/Chrome.java:
     * tests/netx/jnlp_testsengine/net/sourceforge/jnlp/browsertesting/browsers/Chromium.java:
     * tests/netx/jnlp_testsengine/net/sourceforge/jnlp/browsertesting/browsers/Midori.java:
     * tests/netx/jnlp_testsengine/net/sourceforge/jnlp/browsertesting/browsers/Epiphany.java: 
proxies for  browsers as name suggests
     * tests/netx/jnlp_testsengine/net/sourceforge/jnlp/browsertesting/browsers/LinuxBrowser.java:
     abstract forefather for all browsers implementing Browser interface. Is setting /usr/bin as bin 
path, libjavaplugin.so as default plugin library name, intorducing stubs for methods (eg 32/64 bit libs)
     * 
tests/netx/jnlp_testsengine/net/sourceforge/jnlp/browsertesting/browsers/MozillaFamilyLinuxBrowser: 
forefather for all browsers except Opera. Is unifying .mozilla/plugins directories


browserAnnotation-tests:

     various test for browser engine
     * tests/jnlp_tests/simple/AppletTest/resources/appletAutoTests.html: fixed missing parenthesis
     * tests/jnlp_tests/simple/AppletTest/resources/appletAutoTests2.html: new test excluding XslowX 
for applets
     * tests/jnlp_tests/simple/AppletTest/testcases/AppletTestTests:
     (doubleChrome) test for ensuring that two chrome browsers launched behind themselves will not 
cause errors as they vere withhout criticalFixes patch
      (AppletInBrowserTest) and (AppletInBrowserTestXslowX) testing methods for all browser
     * tests/jnlp_tests/simple/deadlocktest/srcs/DeadlockTest.java: improved to print sometimes 
something out
     * tests/jnlp_tests/simple/deadlocktest/testcases/DeadLockTestTest.java: 
(testDeadLockTestTerminatedBody) enchanced to ensure that not so much is lost when process is 
terminated, but showing that something can be lost (which is correct)
     * tests/netx/jnlp_testsengine/net/sourceforge/jnlp/ResourcesTest.java:
     (testNonExisitngBrowserWillNotDeadlock) testing taht when no browser is set, then no deadlock 
happens as happen before criticalFixes's deadlyException
     (testUnexistingProcessWillFailRecognizedly)  is actually testing deadlyException
     (testNonExisitngBrowserWillNotCauseMess) some but with annotation which was also harming output 
of tool little bit without TestInBrowsersAnnotation fixed.
     (testBrowsers2) is testing all browsers configuration without annotation  and (testBrowser3) do 
the same configuration tests annotation driven
     (testBrowser) body of above two methods. Is testing whether used browsers are correctly linked 
with latest build
     (testBrowsers1) is testing parsing of -D variable
     (testListeners) annotated that needs display
     * tests/netx/unit/net/sourceforge/jnlp/runtime/CodeBaseClassLoaderTest.java: annotated with Bug 
annotation


I hope that it will fulfil our needs as much as possible
   J.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: browserAnnotation-impl.diff
Type: text/x-patch
Size: 35876 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/distro-pkg-dev/attachments/20120611/b240ba8a/browserAnnotation-impl.diff 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: browserAnnotation-tests.diff
Type: text/x-patch
Size: 28077 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/distro-pkg-dev/attachments/20120611/b240ba8a/browserAnnotation-tests.diff