[rfc][icedtea-web] @Remote and automated manual testcases
Adam Domurad
adomurad at redhat.com
Mon Dec 17 11:07:47 PST 2012
On 08/20/2012 01:25 PM, Jiri Vanek wrote:
> hi!
>
> I would like to introduce Remote annotation for launching remote jnlp
> files (and appelts). Especially for automation of
> http://icedtea.classpath.org/wiki/IcedTea-Web-Tests .
>
> For jnlp files I would like to continue in way as suggested in patch
> (as most of them for now do not have any output :-/ [but I know that a
> lot of *have* output to check]).
>
> For applets there will be challenge to deal with various certficates,
> but it will be possible by new robot-like engien we are trying now in
> brno.
>
> Currently the remote tests are run in default and no sign of @remote
> is in html report. But both will be fixed in case that this annotation
> and automatism will be found worthy.
> Also some changes in wiki I would like to introduce after this
> automation is done. eg for now it will be like:
>
> ...
> IcedTea-Web webstart test applications
>
> Note: A lot of these are not Free Software.
> http://www.cs.brown.edu/exploratories/freeSoftware/repository/edu/brown/cs/exploratories/applets/fft1DApp/1d_fast_fourier_transform_java_jnlp.jnlp
> (automated)
> http://www.arbores.ca/AnnuityCalc.jnlp (automated)
> http://www.ucware.com/apev/jaws/apev.jnlp (not working, one needs
> paied account)
> http://phetsims.colorado.edu/sims/circuit-construction-kit/circuit-construction-kit-dc_en.jnlp
> (sometimes do not start, always starts with -Xtrustall, automated)
> http://www.topcoder.com/contest/arena/ContestAppletProd.jnlp
> (automatised, with -Xnofork, otherwise do not close correctly)
> http://java.sun.com/docs/books/tutorialJWS/uiswing/events/ex6/ComponentEventDemo.jnlp
> (automated)
> http://www.crossftp.com/crossftp.jnlp (trial, can't be automated)
> http://www.arbores.ca/Deposit.jnlp
> http://java.sun.com/javase/technologies/desktop/javawebstart/apps/draw.jnlp
>
> ...
>
> Where automatised can be links to test files.
>
> Hope that this is worthy.
>
> There is close connection with [rfc][icedtea-web] do exec java
> instead of plain java in javaws. If it will not able to go in, then I
> will need to add some killing mechanism (eg as the one in DeadlockTest
> - Xfork tests)
Thanks for the ping! This would be nice to have. My current solution for
remote tests is a little bash script :-)
I assume everything needed is in now, yes ?
>
>
> Regards,
> J.
> diff -r 6e84854aecba
> tests/test-extensions/net/sourceforge/jnlp/annotations/Remote.java
> --- /dev/null Thu Jan 01 00:00:00 1970 +0000
> +++
> b/tests/test-extensions/net/sourceforge/jnlp/annotations/Remote.java
> Mon Aug 20 18:42:19 2012 +0200
> @@ -0,0 +1,52 @@
> +/* Bug.java
> +Copyright (C) 2012 Red Hat, Inc.
> +
> +This file is part of IcedTea.
> +
> +IcedTea is free software; you can redistribute it and/or
> +modify it under the terms of the GNU General Public License as
> published by
> +the Free Software Foundation, version 2.
> +
> +IcedTea is distributed in the hope that it will be useful,
> +but WITHOUT ANY WARRANTY; without even the implied warranty of
> +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> +General Public License for more details.
> +
> +You should have received a copy of the GNU General Public License
> +along with IcedTea; see the file COPYING. If not, write to
> +the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
> Boston, MA
> +02110-1301 USA.
> +
> +Linking this library statically or dynamically with other modules is
> +making a combined work based on this library. Thus, the terms and
> +conditions of the GNU General Public License cover the whole
> +combination.
> +
> +As a special exception, the copyright holders of this library give you
> +permission to link this library with independent modules to produce an
> +executable, regardless of the license terms of these independent
> +modules, and to copy and distribute the resulting executable under
> +terms of your choice, provided that you also meet, for each linked
> +independent module, the terms and conditions of the license of that
> +module. An independent module is a module which is not derived from
> +or based on this library. If you modify this library, you may extend
> +this exception to your version of the library, but you are not
> +obligated to do so. If you do not wish to do so, delete this
> +exception statement from your version.
> + */
> +
> +package net.sourceforge.jnlp.annotations;
> +
> +import java.lang.annotation.ElementType;
> +import java.lang.annotation.Retention;
> +import java.lang.annotation.RetentionPolicy;
> +import java.lang.annotation.Target;
> +
> +/**
> + * Mark for tests running content on remote servers
> + */
> + at Target({ElementType.TYPE})
> + at Retention(RetentionPolicy.RUNTIME)
> +public @interface Remote {
> +
> +}
> diff -r 6e84854aecba tests/junit-runner/JunitLikeXmlOutputListener.java
> --- a/tests/junit-runner/JunitLikeXmlOutputListener.java Sun Aug 19
> 16:50:30 2012 +0200
> +++ b/tests/junit-runner/JunitLikeXmlOutputListener.java Mon Aug 20
> 18:24:47 2012 +0200
> @@ -23,6 +23,7 @@
> import java.util.concurrent.TimeUnit;
> import net.sourceforge.jnlp.annotations.Bug;
> import net.sourceforge.jnlp.annotations.KnownToFail;
> +import net.sourceforge.jnlp.annotations.Remote;
>
>
> import org.junit.internal.JUnitSystem;
> @@ -49,6 +50,7 @@
> private static final String BUGS = "bugs";
> private static final String BUG = "bug";
> private static final String K2F = "known-to-fail";
> + private static final String REMOTE = "remote";
> private static final String TEST_NAME_ATTRIBUTE = "name";
> private static final String TEST_TIME_ATTRIBUTE = "time";
> private static final String TEST_IGNORED_ATTRIBUTE = "ignored";
> @@ -198,6 +200,7 @@
> testcaseAtts.put(TEST_IGNORED_ATTRIBUTE,
> Boolean.TRUE.toString());
> }
> KnownToFail k2f=null;
> + Remote remote=null;
> try {
> if (testClass != null && testMethod != null) {
> k2f = testMethod.getAnnotation(KnownToFail.class);
> @@ -208,6 +211,16 @@
> } catch (Exception ex) {
> ex.printStackTrace();
> }
> + try {
> + if (testClass != null && testMethod != null) {
> + remote =
> testMethod.getClass().getAnnotation(Remote.class);
> + if (k2f != null) {
> + testcaseAtts.put(REMOTE, Boolean.TRUE.toString());
> + }
> + }
> + } catch (Exception ex) {
> + ex.printStackTrace();
> + }
> openElement(TEST_ELEMENT, testcaseAtts);
> if (testFailed != null) {
> if (k2f != null) {
> diff -r 6e84854aecba tests/junit-runner/LessVerboseTextListener.java
> --- a/tests/junit-runner/LessVerboseTextListener.java Sun Aug 19
> 16:50:30 2012 +0200
> +++ b/tests/junit-runner/LessVerboseTextListener.java Mon Aug 20
> 18:24:47 2012 +0200
> @@ -8,6 +8,7 @@
> import java.io.PrintStream;
> import java.lang.reflect.Method;
> import net.sourceforge.jnlp.annotations.KnownToFail;
> +import net.sourceforge.jnlp.annotations.Remote;
>
> import org.junit.internal.JUnitSystem;
> import org.junit.runner.Description;
> @@ -37,6 +38,7 @@
> public void testIgnored(Description description) throws Exception {
> writer.println("Ignored: " + description.getClassName() + "."
> + description.getMethodName());
> printK2F(writer, null, description);
> + printRemote(writer, description);
> }
>
>
> @@ -45,6 +47,7 @@
> testFailed = true;
> writer.println("FAILED: " + failure.getTestHeader() + " " +
> failure.getMessage());
> printK2F(writer,true,failure.getDescription());
> + printRemote(writer, failure.getDescription());
> }
>
> @Override
> @@ -52,6 +55,7 @@
> if (!testFailed) {
> writer.println("Passed: " + description.getClassName() +
> "." + description.getMethodName());
> printK2F(writer,false,description);
> + printRemote(writer, description);
> }
> }
>
> @@ -114,4 +118,30 @@
> return null;
> }
>
> +
> +
> + public static Remote getRemote(Description description) {
> + try {
> + Class q = description.getTestClass();
> + if (q != null) {
> + Remote rem = (Remote) q.getAnnotation(Remote.class);
> + return rem;
> + }
> + } catch (Exception ex) {
> + ex.printStackTrace();
> + }
> + return null;
> + }
> +
> +
> + private void printRemote(PrintStream writer, Description
> description) {
> + try {
> + Remote rem = getRemote(description);
> + if (rem!=null) {
> + writer.println(" - This test is running remote
> content, can fail without any internal reason");
Maybe clearer is 'This test is running remote content, note that
failures may be caused by broken applications"
> + }
> + } catch (Exception ex) {
> + ex.printStackTrace();
> + }
> + }
> }
This part looks OK otherwise.
> diff -r 6e84854aecba tests/reproducers/custom/remote/srcs/Makefile
Good idea treating this as custom-reproducer
> --- /dev/null Thu Jan 01 00:00:00 1970 +0000
> +++ b/tests/reproducers/custom/remote/srcs/Makefile Mon Aug 20
> 18:42:19 2012 +0200
> @@ -0,0 +1,5 @@
> +prepare-reproducer:
> + echo "Nothing to do to prepare remote reproducers now"
> +
> +clean-reproducer:
> + echo "Nothing to do to clean remote reproducers now"
> diff -r 6e84854aecba
> tests/reproducers/custom/remote/testcases/RemoteApplicationSettings.java
> --- /dev/null Thu Jan 01 00:00:00 1970 +0000
> +++
> b/tests/reproducers/custom/remote/testcases/RemoteApplicationSettings.java
> Mon Aug 20 18:42:19 2012 +0200
> @@ -0,0 +1,126 @@
> +/* RemoteApplicationTests.java
> +Copyright (C) 2011 Red Hat, Inc.
> +
> +This file is part of IcedTea.
> +
> +IcedTea is free software; you can redistribute it and/or
> +modify it under the terms of the GNU General Public License as
> published by
> +the Free Software Foundation, version 2.
> +
> +IcedTea is distributed in the hope that it will be useful,
> +but WITHOUT ANY WARRANTY; without even the implied warranty of
> +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> +General Public License for more details.
> +
> +You should have received a copy of the GNU General Public License
> +along with IcedTea; see the file COPYING. If not, write to
> +the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
> Boston, MA
> +02110-1301 USA.
> +
> +Linking this library statically or dynamically with other modules is
> +making a combined work based on this library. Thus, the terms and
> +conditions of the GNU General Public License cover the whole
> +combination.
> +
> +As a special exception, the copyright holders of this library give you
> +permission to link this library with independent modules to produce an
> +executable, regardless of the license terms of these independent
> +modules, and to copy and distribute the resulting executable under
> +terms of your choice, provided that you also meet, for each linked
> +independent module, the terms and conditions of the license of that
> +module. An independent module is a module which is not derived from
> +or based on this library. If you modify this library, you may extend
> +this exception to your version of the library, but you are not
> +obligated to do so. If you do not wish to do so, delete this
> +exception statement from your version.
> + */
> +
> +import java.net.MalformedURLException;
> +import java.net.URL;
> +import net.sourceforge.jnlp.ProcessResult;
> +import org.junit.Assert;
> +import org.junit.Test;
> +
> +;
> +
> +public class RemoteApplicationSettings {
> +
> + public static interface RemoteApplicationTestcaseSettings {
> +
> + public URL getUrl() throws MalformedURLException;
Checked exceptions suck ... no point here, just ranting :-D
> +
> + public void evaluate(ProcessResult pr);
> + }
> +
> + public static class FourierTransform implements
> RemoteApplicationTestcaseSettings {
> +
> + @Override
> + public URL getUrl() throws MalformedURLException {
> + return new
> URL("http://www.cs.brown.edu/exploratories/freeSoftware/repository/edu/brown/cs/exploratories/applets/fft1DApp/1d_fast_fourier_transform_java_jnlp.jnlp");
> + }
> +
> + @Override
> + public void evaluate(ProcessResult pr) {
> + Assert.assertTrue(pr.stdout.length() == 0);
> + Assert.assertTrue(pr.stderr.length() == 0 ||
> pr.stderr.contains(IllegalStateException.class.getName()));
Why is this illegal state exception valid ?
> +
> + }
> + }
> +
> + public abstract static class NoOutputs implements
> RemoteApplicationTestcaseSettings {
> +
> + @Override
> + public void evaluate(ProcessResult pr) {
> + Assert.assertTrue(pr.stdout.length() == 0);
> + Assert.assertTrue(pr.stderr.length() == 0);
> +
> + }
> + }
Although I'd make this a concrete class that accepts the URL string, I
guess tons of classes is fine if you like that sort of thing :-)
> +
> + public static class Arbores extends NoOutputs {
> +
> + @Override
> + public URL getUrl() throws MalformedURLException {
> + return new URL("http://www.arbores.ca/AnnuityCalc.jnlp");
> + }
> + }
> +
> + public static class PhetSims extends NoOutputs {
> +
> + @Override
> + public URL getUrl() throws MalformedURLException {
> + return new
> URL("http://phetsims.colorado.edu/sims/circuit-construction-kit/circuit-construction-kit-dc_en.jnlp");
> + }
> + }
> +
> + public static class TopCoder extends NoOutputs {
> +
> + @Override
> + public URL getUrl() throws MalformedURLException {
> + return new
> URL("http://www.topcoder.com/contest/arena/ContestAppletProd.jnlp");
> + }
> + }
> +
> + public static class SunSwingDemo extends NoOutputs {
> +
> + @Override
> + public URL getUrl() throws MalformedURLException {
> + return new
> URL("http://java.sun.com/docs/books/tutorialJWS/uiswing/events/ex6/ComponentEventDemo.jnlp");
> + }
> + }
> +
> + @Test
> + public void remoteApplicationSettingsAreWorking() throws Exception {
> + RemoteApplicationTestcaseSettings s5 = new FourierTransform();
> + Assert.assertNotNull(s5.getUrl());
> + RemoteApplicationTestcaseSettings s4 = new Arbores();
> + Assert.assertNotNull(s4.getUrl());
> + RemoteApplicationTestcaseSettings s3 = new PhetSims();
> + Assert.assertNotNull(s3.getUrl());
> + RemoteApplicationTestcaseSettings s2 = new TopCoder();
> + Assert.assertNotNull(s2.getUrl());
> + RemoteApplicationTestcaseSettings s1 = new SunSwingDemo();
> + Assert.assertNotNull(s1.getUrl());
> +
> + }
> +}
> diff -r 6e84854aecba
> tests/reproducers/custom/remote/testcases/RemoteApplicationTests.java
> --- /dev/null Thu Jan 01 00:00:00 1970 +0000
> +++
> b/tests/reproducers/custom/remote/testcases/RemoteApplicationTests.java Mon
> Aug 20 18:42:19 2012 +0200
> @@ -0,0 +1,89 @@
> +/* RemoteApplicationTests.java
> +Copyright (C) 2011 Red Hat, Inc.
> +
> +This file is part of IcedTea.
> +
> +IcedTea is free software; you can redistribute it and/or
> +modify it under the terms of the GNU General Public License as
> published by
> +the Free Software Foundation, version 2.
> +
> +IcedTea is distributed in the hope that it will be useful,
> +but WITHOUT ANY WARRANTY; without even the implied warranty of
> +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> +General Public License for more details.
> +
> +You should have received a copy of the GNU General Public License
> +along with IcedTea; see the file COPYING. If not, write to
> +the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
> Boston, MA
> +02110-1301 USA.
> +
> +Linking this library statically or dynamically with other modules is
> +making a combined work based on this library. Thus, the terms and
> +conditions of the GNU General Public License cover the whole
> +combination.
> +
> +As a special exception, the copyright holders of this library give you
> +permission to link this library with independent modules to produce an
> +executable, regardless of the license terms of these independent
> +modules, and to copy and distribute the resulting executable under
> +terms of your choice, provided that you also meet, for each linked
> +independent module, the terms and conditions of the license of that
> +module. An independent module is a module which is not derived from
> +or based on this library. If you modify this library, you may extend
> +this exception to your version of the library, but you are not
> +obligated to do so. If you do not wish to do so, delete this
> +exception statement from your version.
> + */
> +
> +import java.util.Arrays;
> +import java.util.Collections;
> +import java.util.List;
> +import net.sourceforge.jnlp.ProcessResult;
> +import net.sourceforge.jnlp.ServerAccess;
> +import net.sourceforge.jnlp.annotations.Remote;
> +import org.junit.Test;
> +
> + at Remote
> +public class RemoteApplicationTests {
> +
> + private static ServerAccess server = new ServerAccess();
> + private final List<String> l =
> Collections.unmodifiableList(Arrays.asList(new String[]{"-Xtrustall"}));
> + private final List<String> ll =
> Collections.unmodifiableList(Arrays.asList(new String[]{"-Xtrustall",
> "-Xnofork"}));
> +
> +
> + @Test
> + public void topCoderRemoteTest() throws Exception {
> + RemoteApplicationSettings.RemoteApplicationTestcaseSettings settings
> = new RemoteApplicationSettings.TopCoder();
> + ProcessResult pr = server.executeJavawsUponUrl(ll,
> settings.getUrl());
> + settings.evaluate(pr);
> + }
> +
> + @Test
> + public void sunSwingRemoteTest() throws Exception {
> + RemoteApplicationSettings.RemoteApplicationTestcaseSettings settings
> = new RemoteApplicationSettings.SunSwingDemo();
> + ProcessResult pr = server.executeJavawsUponUrl(l,
> settings.getUrl());
> + settings.evaluate(pr);
> + }
> +
> +
> + @Test
> + public void fourierTransformRemoteTest() throws Exception {
> + RemoteApplicationSettings.RemoteApplicationTestcaseSettings settings
> = new RemoteApplicationSettings.FourierTransform();
> + ProcessResult pr = server.executeJavawsUponUrl(null,
> settings.getUrl());
> + settings.evaluate(pr);
> + }
> +
> + @Test
> + public void arboresRemoteTest() throws Exception {
> + RemoteApplicationSettings.RemoteApplicationTestcaseSettings settings
> = new RemoteApplicationSettings.Arbores();
> + ProcessResult pr = server.executeJavawsUponUrl(l,
> settings.getUrl());
> + settings.evaluate(pr);
> + }
> +
> + @Test
> + public void phetsimsRemoteTest() throws Exception {
> + RemoteApplicationSettings.RemoteApplicationTestcaseSettings settings
> = new RemoteApplicationSettings.PhetSims();
> + ProcessResult pr = server.executeJavawsUponUrl(l,
> settings.getUrl());
> + settings.evaluate(pr);
> + }
> +}
Looks good, OK for HEAD
-Adam
More information about the distro-pkg-dev
mailing list