Extending jnlp reproducers for html+applet reproducers
Pavel Tisnovsky
ptisnovs at redhat.com
Tue Mar 20 10:19:47 PDT 2012
Hi Jiri,
I'm going to review all your changes when I'll have some free time :-) As Omair said
we are likely pushing JUnit to its limit and it's probably that we will need another
solution (still JUnit is usable for unit tests, not integration or smoke tests).
Pavel
PS: I need to buy +6 hr/day but can't find it on Amazon!
Jiri Vanek wrote:
> Hi All!
>
> As promised some time ago, and "proof of concepted" week ago, I have
> written small extension to reproducers framework, which allows to launch
> applet reproducers inside browsers.
>
> The hg diff in attachment is not considered for review (however any
> brave soul *can do this!*), but is added for overview information. I
> will post individual parts of this one by one for next months.
> Also individual parts into to which I have split browserTests.diff have
> small overleaps.
>
> _concept_
> At first all available (supported and avaiable) browsers are discovered
> by configure: eg AC_CHECK_PROGS([OPERA],[opera],[], []) .... for opera,
> chrome, chromium and firefox.
>
> *New make targets* for creating *symlinks* to --prefixed location were
> added (stamps/user-links.stamp and stamps/global-links.stamp). Those
> depends on make install and stamps/run-netx-dist-tests.stamp depends on
> global-links. Those targets are mentioned only for testing purposes.
> Those ensure that tested browsers will use just built plugin. There is
> test for this correctness inside reproducers (testBrowser family of
> tests inside jnlp_testsengine/net/sourceforge/jnlp/ResourcesTest.java)
>
> So just before running reproducers the links are created (bad luck that
> for opera this linking need roots privileges :( )and used browsers are
> passed into testing framework by
> -Dused.browsers=$(FIREFOX):$(CHROMIUM):$(CHROME):$(OPERA)
>
> This variable is inside tranlsated by BrowserFactory to proxies
> representing individual browser (providing theirs binary, theirs plugin
> dirs, theirs individual settings.
>
> Tests are invoked in browser by lunching already introduced
> server.executeBrowser(resource).
> But there is little bit of magic around this procedure - annotation
> @TestInBrowsers(testIn={...}) Where values are singletons pointing to
> individual concrete proxies. or special case ONE, which represents any
> one of configured browsers - so method will be invoked just once times
> with this browser injected, NONE - which is mostly just for testing
> purposes and will inject non existing browser. Or ALL which will inject
> all configured browsers.
> In case that ALL returns zero of browsers, or individual requested
> proxies are not configured, or when requested is one ONE and no Browser
> is returned, then NONE browser is injected. (causing none-browser result)
>
> This injection is very real, and if Eg FIREFOX,CHROMIUM and OPERA are
> requested by annotation, then the testing method is really launched
> three times - once for each browser. if ALL is requested by annotation,
> then method is really iterated for each configured browser.
> This injection is also propagated into reporting, where is displayed
> method - browser. When no browser is configured, then all "browser"
> tests will fail as for "unset_browser" (which is representation of
> mentioned NONE)
> This injection and iterating over methods is done inside custom junit
> runner BrowserTestRunner, which is invoking the tested method several
> times, and is always injecting desired browser into its ServerAccess
> (included by inheriting of BrowserTest). So the only thing which final
> reproducer writer is to do is:
> - prepare reproducer as is common now
> - instead of jnlp file prepare html file which will launch appelt
> - make his testcase to extends BrowserTest
> - annotate methods which have something to do with browser which
> browsers they should be tested against
> - there is still possibility to work without this annotation and
> get browsers directly from factory or to inject custom browsers, or tu
> run whatever he wants :)
> -the port handling and other deployment stuff have been used form
> current jnlp reproducers
>
> _fixes_
> There was bug in ProcessAssasin which leaked killed process. It acctual
> did not kill it, just left it be and return control to main theread :-/
> It is fixed by correctly added (!to proper place!) destroy() method of
> process. This leak have come to light when I started to launch browsers.
>
> _other minor and useful stuff_
> *@Bug* annotation - can handle IDs of bugs or urls to web which
> contains more info about "why this reproducer or unit test was added",
> are *nicely* propagated also to html report (see attached report.zip).
> NAme convention is as we are used to from NEWS files.
> *@NeedsDispaly* - annotation which tell that this method can not be run
> if no display is specified.
> *UnitTests* also have benefits of Bug annotation and of *ServerAccess*.
> - added classpath dependence
> *Blacklist* to exclude some reproducers is introduced
> (REPRODUCERS_CLASS_BLACKLIST)
> *enabled decision weather to kill the process or just to skip him
> *embeded virtual server now returns correct mimetypes (this was really
> mayor issue just wrongly ignored by firefox)
> *fied *leaking* proceses (missing process.destroy()
>
> Changelogs: in order of expected order of to be applied
> browserTests-criticalFixes:
>
> fixing issue when process was not launched at all and when was
> killed but left behind hanging
> *tests/netx/jnlp_testsengine/net/sourceforge/jnlp/ServerAccess.java:
> (getContentOfStream) this method overlaoded with possibility to
> specify encoding (I needed to set aASCI in one test)
> (deadlyException) field intoruced in ThreadedProcess to record
> exception caused by impassibility of launching the process. And so
> process have been null without any sign why.
> (TinyHtmlServer) now correctly returns known mime types
> (ProcessAssasin) can now skip or smoothly (and finally correctly)
> destroy its process, and all his logging messages were done null-proof
> (as deadlyException now allows)
> Asynchronous (ContentReader) have been silenced when complaining
> about closed streams by Assassin.
>
>
> browserTests-MakeLinks:
>
> Added detection of installed browsers and added targets to create
> symbolic links from install dir to browsers' plugin directories.
> Primarily for testing purposes
> *Makefile.am: (clean-local) optionally delets links' stamps
> (stamps/user-links.stamp) with alias (links) - new target for
> creating symlinks for all users. One must be root to execute this target.
> (stamps/global-links.stamp) with alias (user-links) - new target for
> creating symlinks for logged user only. Because opera is missing this
> feature, quite useless.
> *configure.ac: added basic check whether and which browsers are
> installed
>
>
> browserTests-blacklist:
>
> Added possibility to exclude some classes) from being processed by
> reproducers engine. Usefull when testing just one testcase or when
> interfaces or abstract classes are included (throwing initialization
> exception)
> *Makefile.am: (REPRODUCERS_CLASS_BLACKLIST) new variable holding the
> file with list
> (REPRODUCERS_CLASS_NAMES) added loop through
> REPRODUCERS_CLASS_BLACKLIST file which contains regular expressions
> which, when matched, causing file not to be included in list.
> *netx-dist-tests-blacklist: new file containing expressions.
>
>
> browserTests-BugAnnotation:
>
> Intorduced new annotations Bug (to connect test/reproducer with
> documentation) and NeedsDisplay which tells the launching engine thet
> this particualr test needs Display. BAsed on Pavel's ideas and jtreg
> experiences
> *Makefile.am: (JUNIT_RUNNER_JAR), (stamps/run-netx-unit-tests.stamp)
> and (stamps/run-unit-test-code-coverage.stamp) are now dependent on
> (stamps/netx-dist-tests-compile) and classpaths inside them have been
> enriched for JNLP_TESTS_ENGINE_DIRwhich contains definitions of those
> annotations
> *tests/jnlp_tests/simple/CheckServices/testcases/CheckServicesTests.java:
> and
> *tests/jnlp_tests/simple/ManifestedJar1/testcases/ManifestedJar1Test.java:
> and
> *tests/jnlp_tests/simple/Spaces can be
> everywhere/testcases/SpacesCanBeEverywhereTests.java: filled Bug
> annotations
> *tests/junit-runner/JunitLikeXmlOutputListener.java: made to
> understan Bug annotation
> *tests/netx/jnlp_testsengine/net/sourceforge/jnlp/annotations/NeedsDisplay.java:
> and
> *tests/netx/jnlp_testsengine/net/sourceforge/jnlp/annotations/Bug.java:
> annotations definitions
> *tests/report-styles/jreport.xsl" made nice links from bug
> annotation prepared by JunitLikeXmlOutputListener
>
>
> browserTests-TestInBrowsersAnnotation
>
> introduced possibility to run comfortably applets+html reproducers
> *tests/netx/jnlp_testsengine/net/sourceforge/jnlp/ServerAccess.java:
> (currentBrowser) variable holding injected browser
> (getBrowserLocation) - returning path to process to be launched when
> browser requested
> (getBrowserParams) - gathering set default's browser settings
> (set/getCurrentBrowsers) - set browser by id/return id of set browser
> (set/getCurrentBrowser) - set browser instance /returns instance of
> current browser
> (executeBrowser) family of methods now cooperate with above methods
> for default set browser
> (executeBrowser(Browser) family to work with implicit browser
> *tests/netx/jnlp_testsengine/net/sourceforge/jnlp/annotations/TestInBrowsers.java:
> annotation for determining which browser(s) to use with annotated method
> *tests/netx/jnlp_testsengine/net/sourceforge/jnlp/browsertesting/Browser.java:
> interface for dealing with various browsers
> *tests/netx/jnlp_testsengine/net/sourceforge/jnlp/browsertesting/BrowserFactory.java:
>
> singleton for mapping configured browsers x requested browser x
> browsers proxies
> *tests/netx/jnlp_testsengine/net/sourceforge/jnlp/browsertesting/BrowserTest.java:
>
> Forefather of all testcases which have methods to do tests iniside
> browser. Is allowing correct annotation -> proxy trasnver to
> VirtualServer for selected method and is requesting custom runner from
> junit framework
> *tests/netx/jnlp_testsengine/net/sourceforge/jnlp/browsertesting/BrowserTestRunner.java:
>
> custom test Ruuner which is responsible for translating annotation
> and run the method mutlipletimes for each requested browser and to name
> it properly
> *tests/netx/jnlp_testsengine/net/sourceforge/jnlp/browsertesting/Browsers.java:
> enumeration of abstract browsers and theirs sets or subsets.
> tests/netx/jnlp_testsengine/net/sourceforge/jnlp/browsertesting/browsers/*:
> individual browsers proxies
>
>
> browserTests-tests:
>
> various test for browser engine
> *tests/jnlp_tests/simple/AppletTest/resources/appletAutoTests.html:
> fixed missing parenthesis
> *tests/jnlp_tests/simple/AppletTest/resources/appletAutoTests2.html:
> new test excluding XslowX for applets
> *tests/jnlp_tests/simple/AppletTest/testcases/AppletTestTests:
> (doubleChrome) test for ensuring that two chrome browsers launched
> behind themselves will not cause errors as they vere withhout
> criticalFixes patch
> (AppletInBrowserTest) and (AppletInBrowserTestXslowX) testing
> methods for all browser
> */tests/jnlp_tests/simple/deadlocktest/srcs/DeadlockTest.java:
> improved to print sometimes something out
> *tests/jnlp_tests/simple/deadlocktest/testcases/DeadLockTestTest.java:
> (testDeadLockTestTerminatedBody) enchanced to ensure that not so much is
> lost when process is terminated, but showing that something can be lost
> (which is correct)
> *tests/netx/jnlp_testsengine/net/sourceforge/jnlp/ResourcesTest.java:
> (testNonExisitngBrowserWillNotDeadlock) testing taht when no browser
> is set, then no deadlock happens as happend before criticalFixes's
> deadlyException
> (testUnexistingProcessWillFailRecognizedly) is actually testing
> deadlyException
> (testNonExisitngBrowserWillNotCauseMess) some but with annotation
> which was also harmiing output of tool little bit without
> TestInBrowsersAnnotation fixed.
> (testBrowsers2) is testing all browsers configuration without
> annotation and (testBrowser3) do the same configuration tests
> annotation driven
> (testBrowser) body of above two methods. Is testing whether used
> browsers are correctly linked with latest build
> (testBrowsers1) is testing parsing of -D variable
> (testListeners) annotated that needs display
> *tests/netx/unit/net/sourceforge/jnlp/runtime/CodeBaseClassLoaderTest.java:
> annotated with Bug annotation
>
>
>
> browserTests is gathering all previous patches and can be applied and
> tested
> See samples in atached archive!-)
>
>
>
> However.... Omair have raised an opinion against this approach (damn..
> after it was written) -
> http://mail.openjdk.java.net/pipermail/distro-pkg-dev/2012-March/017768.html
>
>
> Personally I disagree with searching for new engine and I think it is
> worthy to keep maintain our framework (as I have made quite progress on
> it) and is designed exactly for our needs. If it will be extended for
> awt robot (soon) and for remote tests (faaar future) it will be quite
> really nice!-)
> And maintaining it is quite fun O:)
>
> I'm also not aware of any framework matching our needs at least from 50%
> and bending something so much ...I don't like this idea.
>
> Looking forward to opinions!
>
>
> Best regards and good luck
> J.
>
>
>
>
> Ps: mjw is cced just for case it need approve to distro-pkg O:)
>
>
>
>
More information about the distro-pkg-dev
mailing list