[RFC][icedtea-web] emma and testcoveragefor testsuites

[Jiri Vanek]

On 11/14/2011 01:35 PM, Jiri Vanek wrote:
> On 11/10/2011 05:04 PM, Jiri Vanek wrote:
>> This patch add code-coverage of our testsuits functionality into makefile. It is adding optional dependence for emma.jar, which is providing whole functionality of arround investigating coverage and generate reports.
>> It add three new top level targets into makefile.am It run-test-code-coverage, run-reproducers-test-code-coverage and run-unit-test-code-coverage. Except this, it is modifying touch $@ of several more targets. The reason is as follows.
>> *run-reproducers-test-code-coverage and run-unit-test-code-coverage:
>> * each must run all tests again, with (by emma) marrked code (so the results can not be as correct as we can wish)
>> * it depends on all preparations which proceed normal testing
>> * when they are run after this tests, they do not invoke them again (just run the marrked version)
>> * when they are run without previoous make check or make run-netx-dist-tests they invoke them
>> * all four make check , make run-netx-dist-tests run-reproducers-test-code-coverage and run-unit-test-code-coverage are considered as top level targets
>> *run-test-code-coverage, on the other side just merges results generated by run-reproducers-test-code-coverage and run-unit-test-code-coverage. When it is run after this two targets, then he just reuse the results. If he ie run without, then he invoke them. It ia lso to be considered top level target.
>> *Each code-coverege target creates html report.
>>
>> Reproducer tests are not covered 100% they will need more hacking, which I will post as new patch. (in some longer time, it will not be easy)
>>
>> Best regards
>> J.
>>
>>
>> changelog:
>>
>> 2011-11-10 Jiri Vanek <jvanek at redhat.com>
>> Added code-coverage generation targets
>> * configure.ac: added search for optional emma.jar
>> * makefile.am: added UNIT_CLASS_NAMES and REPRODUCERS_CLASS_NAMES variables
>> to store tests clases for reuse in emmarun
>> (run-netx-dist-tests): made dependent on reused stamped version
>> (stamps/run-netx-dist-tests): stamped rusable version of run-netx-dist-tests
>> ($(NETX_UNIT_TEST_DIR)/coverage.es) and (run-unit-test-code-coverage) targets
>> to generate report from unit-tests. Both result binary file and html report in
>> tests.build/netx/unit
>> ($(JNLP_TESTS_ENGINE_DIR)/coverage.es) and (run-reproducers-test-code-coverage)
>> targets to generate report from reproducers-test. Both result binary file
>> and html report in tests.build/netx/jnlp_testsengine
>> (run-test-code-coverage): merges binary results from unit and reproducers
>> code-coverage runs and save html report in abs_top_builddir
>> (clean-netx-dist-tests) removes REPRODUCERS_CLASS_NAMES file, and
>> run-reproducers-test-code-coverage and stamps/run-netx-dist-tests stamps
>> (clean-netx-unit-tests) remove UNIT_CLASS_NAMES fiel and
>> run-unit-test-code-coverage and run-netx-unit-tests stamps
>>
>
> Updated for head, added patching of javaws and netx as mentioned above
>
> 2011-11-14 Jiri Vanek <jvanek at redhat.com>
> Added code-coverage generation targets
> * configure.ac: added search for optional emma.jar
> * makefile.am: added UNIT_CLASS_NAMES and REPRODUCERS_CLASS_NAMES variables
> to store tests clases for reuse in emmarun
> (run-netx-dist-tests): made dependent on reused stamped version
> (stamps/run-netx-dist-tests): stamped rusable version of run-netx-dist-tests
> ($(NETX_UNIT_TEST_DIR)/coverage.es) and (run-unit-test-code-coverage) targets
> to generate report from unit-tests. Both result binary file, xml and html report in
> tests.build/netx/unit
> ($(JNLP_TESTS_ENGINE_DIR)/coverage.es) and (run-reproducers-test-code-coverage)
> targets to generate report from reproducers-test. They also backup,
> change javaws and netx.jar, run tests and restore them. Both result binary file, xml
> and html report in tests.build/netx/jnlp_testsengine
> (run-test-code-coverage): merges binary results from unit and reproducers
> code-coverage runs and save xml and html report in abs_top_builddir
> (clean-netx-dist-tests) removes REPRODUCERS_CLASS_NAMES file, and
> run-reproducers-test-code-coverage and stamps/run-netx-dist-tests stamps
> (clean-netx-unit-tests) remove UNIT_CLASS_NAMES file and
> run-unit-test-code-coverage and run-netx-unit-tests stamps
> * tests/netx/jnlp_testsengine/net/sourceforge/jnlp/ServerAccess.java
> (ProcessResult) added filtering on stdout to remove EMMA messages
> (testsProcessResultFiltering) test for this behaviour
>
>
> There are several things, I would like to pick up:
> touch @$ (madness?) rmains
> EMMA is very badly configurable in verbosity upon instructed classes and output file names.In fact - in its source codes there are todo marks:)
> -verbosityl.level property is not working at all, so I'm filtering EMMA messages for reproducers - if not done so, then most of reproducers fail, and coverage is then not-so-according. Unfiltered output remains inside ProcessResult also. I have added small test for this behaviour.
> -output fielnames, are working, but do not follow merging correctly. So instead of setting output filenames I'm mv-ing files after each run and using this moved versions (because -in is working correctly, and outputs will not override inputs (or better - will not fail)
>
> Sample output can be found at http://10.34.2.200/icedtea-web/
> -There is iso-88..1. encoding. In this patch I have already used utf-8, but still it was little bit pain to force Emma to encode properly :-/
>
> Best regards
> J.
>

I made an typo on line 155 of the patch. There was missing -D in property definition.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: testCoverageUponHead2.diff
Type: text/x-patch
Size: 15217 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/distro-pkg-dev/attachments/20111119/42e98fec/testCoverageUponHead2.diff