[RFC][icedtea-web] emma and testcoveragefor testsuites

Mon Nov 28 10:25:52 PST 2011

On 11/28/2011 11:44 AM, Jiri Vanek wrote:
> TYVM for review!
>
> I have fixed most of the issues you suggested.
> The one we talked about on IRC  - touching files in prefix/bin . I still
> think its the best aproach. Test coverage should be run in "test run"
> prefix, and with  configuration nearest to final one. I have found no
> better solution (copying prefix...) then this one to fulfil it more.
>
> Except fixes you have suggested, there  are (inspired by your review)
> two mayor changes - final result is now in test.built instead of
> abs_top_build_dir and coverage targets have their own clean, instead of
> chaotic '-rm '  as it was before.
>

>>> CLASSPATH=$(NETX_DIR)/lib/classes.jar:$(JUNIT_JAR):$(JUNIT_RUNNER_JAR):.
>>> \
>>> $(BOOT_DIR)/bin/java -Xbootclasspath:$(RUNTIME) CommandLine
>>> $$class_names \
>>> > stdout.log 2> stderr.log ; \
>>> cat stdout.log ; \
>>> - cat stderr.log>&2
>>> + cat stderr.log>&2 ;
>>> -xsltproc $(TESTS_SRCDIR)/$(REPORT_STYLES_DIRNAME)/jreport.xsl
>>> $(NETX_UNIT_TEST_DIR)/tests-output.xml> $(TESTS_DIR)/index_unit.html
>>
>> Hm... perhaps we should add a configure check for xsltproc?
> I will do it later in separate patch, ok?

Sure.

>>
>>> + mv $(NETX_UNIT_TEST_DIR)/tests-output.xml
>>> $(NETX_UNIT_TEST_DIR)/tests-output.xml_noEmma ; \
>>> + cd $(NETX_UNIT_TEST_DIR) ; \
>>> + class_names=`cat $(UNIT_CLASS_NAMES)` ; \
>>> + $(BOOT_DIR)/bin/java -Xbootclasspath:$(RUNTIME) -cp $(EMMA_JAR)
>>> -Demma.report.html.out.encoding=UTF-8 emmarun \
>>> + -Dreport.html.out.encoding=UTF-8 \
>>
>> Are the properties duplicated on purpose?
> Yes. By some bug in emma, the first is setting encoding of output
> stream, and second is setting value of html tag encoding:-/

Wow, emma's documentation 
(http://emma.sourceforge.net/reference/ch03.html) is _very_ misleading then.

>>
>>> + mv $(NETX_UNIT_TEST_DIR)/tests-output.xml
>>> $(NETX_UNIT_TEST_DIR)/tests-output_withEmma.xml ; \
>>> + mv $(NETX_UNIT_TEST_DIR)/tests-output.xml_noEmma
>>> $(NETX_UNIT_TEST_DIR)/tests-output.xml ; \
>>> + else \
>>> + echo "Sorry, coverage report cant be run without emma installed.
>>> Try install emma or specify with-emma value" ; \
>>
>> Perhaps you might want to invoke false here so "make
>> run-unit-test-code-coverage" fails?
> I have added exit -5. Enough?

Exit codes are normally in the range [0,255].

>>
>>> + fi
>>> + touch $@
>>> +
>>> +$(JNLP_TESTS_ENGINE_DIR)/coverage.es:
>>> run-reproducers-test-code-coverage
>>> +
>>> +run-reproducers-test-code-coverage: stamps/run-netx-dist-tests
>>> + if test "$(EMMA_AVAILABLE)" = "true"; then \
>>> + echo "warning, this can rewrite
>>> tests.build/netx/jnlp_testsengine/tests-output.xml, but not
>>> coresponding html file" ; \
>>> + echo "xml results run from emma sandbox, however, can be wrong" ; \
>>> + mv $(JNLP_TESTS_ENGINE_DIR)/tests-output.xml
>>> $(JNLP_TESTS_ENGINE_DIR)/tests-output.xml_noEmma ; \
>>> + echo "backuping javaws and netx.jar in $(DESTDIR)" ; \
>>> + netx_backup=$(DESTDIR)$(datadir)/$(PACKAGE_NAME)/netx_backup.jar ; \
>>> + javaws_backup=$(DESTDIR)$(bindir)/javaws_backup ; \
>>> + mv $(DESTDIR)$(bindir)/javaws $$javaws_backup ; \
>>> + mv $(DESTDIR)$(datadir)/$(PACKAGE_NAME)/netx.jar $$netx_backup ; \
>>
>> I dont think playing with files inside $(bindir) and/or $(datadir) is
>> a good idea. The files may be read-only at this point if the user does
>> something like this:
>
> According our discussion on IRC, I still think this is best idea.
> Any more ideas more then welcomed.

Okay, I guess we can do this for now.

>> $ make
>> # make install
>> $ make run-reproducers-test-code-coverage
>>
>>> @@ -611,6 +772,9 @@
>>> clean-netx-unit-tests: clean_tests_reports
>>> rm -f netx-unit-tests-source-files.txt
>>> rm -rf $(NETX_UNIT_TEST_DIR)
>>> + rm -f $(UNIT_CLASS_NAMES)
>>> + -rm -f run-unit-test-code-coverage
>>> + -rm -f run-netx-unit-tests
>>
>> Is the "-" in "-rm" required? -f means force so rm should not fail if
>> the file is missing.
> Yes, it was  required

I am quite surprised. Do you have a way to reproduce this?

>>
>>> rm -f stamps/netx-unit-tests-compile.stamp
>>>
>>> clean_tests_reports:
>>> @@ -629,7 +793,10 @@
>>> rm -f stamps/netx-dist-tests-sign-some-reproducers.stamp
>>> rm -f junit-jnlp-dist-simple.txt
>>> rm -f junit-jnlp-dist-signed.txt
>>> + rm -f $(REPRODUCERS_CLASS_NAMES)
>>> rm -f $(abs_top_builddir)/$(KEYSTORE_NAME)
>>> + -rm -f run-reproducers-test-code-coverage
>>> + rm -f stamps/run-netx-dist-tests
>>>
>>> # plugin tests
>>>
>>
>> Cheers,
>> Omair
>
> Best regards, and TYVM for review , J
>
>
> testCoverageUponHead3.diff
>
>
> diff -r 22db4e09bbc7 Makefile.am
> --- a/Makefile.am	Fri Nov 11 14:40:26 2011 +0100
> +++ b/Makefile.am	Mon Nov 28 17:26:35 2011 +0100
> @@ -25,6 +25,8 @@
>   KEYSTORE_NAME=teststore.ks
>
>   JUNIT_RUNNER_JAR=$(abs_top_builddir)/junit-runner.jar
> +UNIT_CLASS_NAMES = $(abs_top_builddir)/unit_class_names
> +REPRODUCERS_CLASS_NAMES = $(abs_top_builddir)/reproducers_class_names
>
>   # Build directories
>
> @@ -106,7 +108,7 @@
>   endif
>
>   if WITH_JUNIT
> -  JUNIT_TESTS=run-netx-unit-tests
> +  JUNIT_TESTS=stamps/run-netx-unit-tests.stamp
>   else
>     JUNIT_TESTS=
>   endif
> @@ -146,7 +148,7 @@
>
>   .PHONY: clean-IcedTeaPlugin clean-add-netx clean-add-netx-debug clean-add-plugin clean-add-plugin-debug \
>    clean-bootstrap-directory clean-native-ecj clean-desktop-files clean-netx-docs clean-docs clean-plugin-docs \
> - clean-tests check-local clean-launchers check-pac-functions run-netx-unit-tests clean-netx-tests \
> + clean-tests check-local clean-launchers check-pac-functions stamps/run-netx-unit-tests.stamp clean-netx-tests \
>    clean-junit-runner clean-netx-unit-tests
>
>   install-exec-local:
> @@ -539,7 +541,10 @@
>   	mkdir -p stamps&&  \
>   	touch $@
>
> -run-netx-dist-tests: all-local stamps/netx.stamp stamps/junit-jnlp-dist-dirs stamps/netx-dist-tests-sign-some-reproducers.stamp \
> +
> +stamps/run-netx-dist-tests.stamp: stamps/netx-dist.stamp extra-lib/about.jar stamps/plugin.stamp launcher.build/$(javaws) \
> + javaws.desktop stamps/docs.stamp launcher.build/$(itweb_settings) itweb-settings.desktop \
> + stamps/netx.stamp stamps/junit-jnlp-dist-dirs stamps/netx-dist-tests-sign-some-reproducers.stamp \
>    stamps/netx-dist-tests-compile.stamp stamps/netx-dist-tests-compile-testcases.stamp $(JUNIT_RUNNER_JAR) $(TESTS_DIR)/$(REPORT_STYLES_DIRNAME)
>   	cd $(JNLP_TESTS_ENGINE_DIR) ; \
>   	class_names= ; \
> @@ -548,7 +553,7 @@
>   	  class_name=`echo $$class_name | sed -e 's|/|.|g' ` ; \
>   	  class_names="$$class_names $$class_name" ; \
>   	done ; \
> -	echo $$class_names ; \
> +	echo $$class_names>  $(REPRODUCERS_CLASS_NAMES)	; \

Can we also make this a separate target?

Cheers,
Omair