From mark at klomp.org  Sun Nov 23 09:17:54 2008
From: mark at klomp.org (Mark Wielaard)
Date: Sun, 23 Nov 2008 18:17:54 +0100
Subject: Allowing partial jtreg samejvm safe runs
Message-ID: <1227460674.29485.26.camel@hermans.wildebeest.org>

Hi,

Jonathan Gibbons has been extending jtreg so that tests can be run in
the same vm. He also made it so that most langtools tests are run that
way. That brings an enormous speed up. Since running all jdk tests takes
ages it would be great if we could use this same speedup there.

Problem is that -samevm only works if all tests have been adapted to
behave well (or marked as needing /othervm). So my idea was to add a
"whitelist mechanism" so you can indicate which (sub)dirs are ready to
be run with samevm.

Luckily jtreg has mostly been setup for this already. All Actions ask
their Script whether or not they should be running in an othervm. So we
hook into that to check the scripts location against a whitelist.

I choose the TEST.ROOT file as the location of the whitelist. Since that
is also the root of the test suite. I made the code so that the
whitelist is only used when running the full test suite from the root.
That way you can still experiment with -samevm or -othervm when running
subtrees of tests.

Also CCed jdk-regtest at sun.com since that address is mentioned in the
TEST.ROOT file with an request to contact them if you change anything in
that file.

Below is the implementation as I checked into icedtea6 (patch also
attached).

2008-11-23  Mark Wielaard  <mark at klomp.org>

    * test/jtreg/com/sun/javatest/regtest/Main.java
    (createParameters): Set same jvm safe dirs when non-null.
    (getSameJVMSafeDirs): New method that reads samejvmsafe property
    from TEST.ROOT.
    (sameJVMSafeDirs): New private field.
    * test/jtreg/com/sun/javatest/regtest/RegressionParameters.java
    (SAME_JVM_SAFE_DIRS): New static final field.
    (load): Read same jvm safe dirs.
    (save0): Write same jvm safe dirs.
    (getSameJVMSafeDirs): New method.
    (setSameJVMSafeDirs): New method.
    * test/jtreg/com/sun/javatest/regtest/RegressionScript.java
    (run): Set testDirPath.
    (isOtherJVM): Take same jvm safe into account.
    (isSameJVMSafe): New method.
    (testDirPath): New private field.

For icedtea6 I did a quick scan of the openjdk/jdk/test subdirs and
listed an initial set that work with -samejvm and added a patch to
augment the TEST.ROOT file (attached).

2008-11-23  Mark Wielaard  <mark at klomp.org>

    * patches/icedtea-samejvm-safe.patch: New patch.
    * Makefile.am (ICEDTEA_PATCHES): Add new patch.
    (check-jdk): Run jtreg with -samevm.
    * HACKING: Document new patch.

Comments on the idea and the implementation very appreciated.

With this 'make check-jdk' takes a little bit less than 3 hours on my
machine now. It was almost 3.5 hours, so it already saves a lot of time.
But we need to make much more tests work with samevm to bring the total
test running time down to something an ordinary developer would be
comfortable with running every day.

We have to see whether or not that is really possible. It seems jdi, rmi
and awt take up a lot of the time needed to run all tests, and I am not
sure those tests can be speed up much.

Cheers,

Mark
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jtreg-samejvm-safe.patch
Type: text/x-patch
Size: 6370 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20081123/eecf8590/attachment.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: icedtea-samejvm-safe.patch
Type: text/x-patch
Size: 1489 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20081123/eecf8590/attachment-0001.bin 

From mark at klomp.org  Sun Nov 23 09:37:19 2008
From: mark at klomp.org (Mark Wielaard)
Date: Sun, 23 Nov 2008 18:37:19 +0100
Subject: Allowing partial jtreg samejvm safe runs
In-Reply-To: <1227460674.29485.26.camel@hermans.wildebeest.org>
References: <1227460674.29485.26.camel@hermans.wildebeest.org>
Message-ID: <1227461839.29485.29.camel@hermans.wildebeest.org>

On Sun, 2008-11-23 at 18:17 +0100, Mark Wielaard wrote:
> Also CCed jdk-regtest at sun.com since that address is mentioned in the
> TEST.ROOT file with an request to contact them if you change anything in
> that file.

This address bounces. Should we change it to discuss at openjdk.java.net?

Cheers,

Mark



From Jonathan.Gibbons at Sun.COM  Sun Nov 23 09:57:13 2008
From: Jonathan.Gibbons at Sun.COM (Jonathan Gibbons)
Date: Sun, 23 Nov 2008 09:57:13 -0800
Subject: Allowing partial jtreg samejvm safe runs
In-Reply-To: <1227460674.29485.26.camel@hermans.wildebeest.org>
References: <1227460674.29485.26.camel@hermans.wildebeest.org>
Message-ID: <EB6434B6-6268-4806-97DC-0DC094BCF487@sun.com>

Mark,

I like this approach in that we don't mess up the tests with /othervm  
keywords
unnecessarily.  I think it is worth distinguishing tests which are  
inherently
"othervm" from those which are "currently othervm but potentially  
samevm with
a bit of effort".

-- Jon

On Nov 23, 2008, at 9:17 AM, Mark Wielaard wrote:

> Hi,
>
> Jonathan Gibbons has been extending jtreg so that tests can be run in
> the same vm. He also made it so that most langtools tests are run that
> way. That brings an enormous speed up. Since running all jdk tests  
> takes
> ages it would be great if we could use this same speedup there.
>
> Problem is that -samevm only works if all tests have been adapted to
> behave well (or marked as needing /othervm). So my idea was to add a
> "whitelist mechanism" so you can indicate which (sub)dirs are ready to
> be run with samevm.
>
> Luckily jtreg has mostly been setup for this already. All Actions ask
> their Script whether or not they should be running in an othervm. So  
> we
> hook into that to check the scripts location against a whitelist.
>
> I choose the TEST.ROOT file as the location of the whitelist. Since  
> that
> is also the root of the test suite. I made the code so that the
> whitelist is only used when running the full test suite from the root.
> That way you can still experiment with -samevm or -othervm when  
> running
> subtrees of tests.
>
> Also CCed jdk-regtest at sun.com since that address is mentioned in the
> TEST.ROOT file with an request to contact them if you change  
> anything in
> that file.
>
> Below is the implementation as I checked into icedtea6 (patch also
> attached).
>
> 2008-11-23  Mark Wielaard  <mark at klomp.org>
>
>    * test/jtreg/com/sun/javatest/regtest/Main.java
>    (createParameters): Set same jvm safe dirs when non-null.
>    (getSameJVMSafeDirs): New method that reads samejvmsafe property
>    from TEST.ROOT.
>    (sameJVMSafeDirs): New private field.
>    * test/jtreg/com/sun/javatest/regtest/RegressionParameters.java
>    (SAME_JVM_SAFE_DIRS): New static final field.
>    (load): Read same jvm safe dirs.
>    (save0): Write same jvm safe dirs.
>    (getSameJVMSafeDirs): New method.
>    (setSameJVMSafeDirs): New method.
>    * test/jtreg/com/sun/javatest/regtest/RegressionScript.java
>    (run): Set testDirPath.
>    (isOtherJVM): Take same jvm safe into account.
>    (isSameJVMSafe): New method.
>    (testDirPath): New private field.
>
> For icedtea6 I did a quick scan of the openjdk/jdk/test subdirs and
> listed an initial set that work with -samejvm and added a patch to
> augment the TEST.ROOT file (attached).
>
> 2008-11-23  Mark Wielaard  <mark at klomp.org>
>
>    * patches/icedtea-samejvm-safe.patch: New patch.
>    * Makefile.am (ICEDTEA_PATCHES): Add new patch.
>    (check-jdk): Run jtreg with -samevm.
>    * HACKING: Document new patch.
>
> Comments on the idea and the implementation very appreciated.
>
> With this 'make check-jdk' takes a little bit less than 3 hours on my
> machine now. It was almost 3.5 hours, so it already saves a lot of  
> time.
> But we need to make much more tests work with samevm to bring the  
> total
> test running time down to something an ordinary developer would be
> comfortable with running every day.
>
> We have to see whether or not that is really possible. It seems jdi,  
> rmi
> and awt take up a lot of the time needed to run all tests, and I am  
> not
> sure those tests can be speed up much.
>
> Cheers,
>
> Mark
> diff -r 9aae858397f9 test/jtreg/com/sun/javatest/regtest/Main.java
> --- a/test/jtreg/com/sun/javatest/regtest/Main.java	Fri Nov 21  
> 18:35:27 2008 -0500
> +++ b/test/jtreg/com/sun/javatest/regtest/Main.java	Sun Nov 23  
> 16:56:34 2008 +0100
> @@ -29,6 +29,7 @@
> import java.io.BufferedReader;
> import java.io.BufferedWriter;
> import java.io.File;
> +import java.io.FileInputStream;
> import java.io.FileNotFoundException;
> import java.io.FileReader;
> import java.io.FileWriter;
> @@ -52,6 +53,7 @@
> import java.util.Iterator;
> import java.util.List;
> import java.util.Map;
> +import java.util.Properties;
> import java.util.TreeMap;
>
> import com.sun.javatest.CompositeFilter;
> @@ -1354,6 +1356,10 @@
>             if (ignoreKind != null)
>                 rp.setIgnoreKind(ignoreKind);
>
> +            sameJVMSafeDirs = getSameJVMSafeDirs(ts);
> +            if (sameJVMSafeDirs != null)
> +                rp.setSameJVMSafeDirs(sameJVMSafeDirs);
> +
>             return rp;
>         } catch (TestSuite.Fault f) {
>             f.printStackTrace();
> @@ -1372,6 +1378,35 @@
>         } catch (IOException e) {
>             return file.getAbsoluteFile();
>         }
> +    }
> +
> +    // Returns directory (prefix) for tests that are same jvm safe
> +    // read from amejvmsafe property in TEST.ROOT file. Returning  
> null
> +    // means all tests are considered same jvm safe. null is returned
> +    // when there is no samejvmsafe property, or there was a problem
> +    // reading it, and when anything else than the test root was  
> given
> +    // as test file argument. Meaning that this only returns  
> something
> +    // non-null if anything was actually specified as same jvm safe  
> and
> +    // the whole test suite is being tested.
> +    private List<String> getSameJVMSafeDirs(File testRoot) {
> +	// Only use the same jvm safe dirs when running from the root.
> +	if (testFileArgs.size() != 1
> +	    || ! 
> canon(testFileArgs.iterator().next()).equals(canon(testRoot)))
> +	    return null;
> +
> +	try {
> +	    File file = new File(testRoot, "TEST.ROOT");
> +            if (file.exists()) {
> +		Properties testRootProps = new Properties();
> +		testRootProps.load(new FileInputStream(file));
> +		String safedirs = testRootProps.getProperty("samejvmsafe");
> +		if ((safedirs != null) && (safedirs.trim().length() > 0))
> +		    return Arrays.asList(StringArray.splitWS(safedirs));
> +	    }
> +	} catch (IOException ioe) {
> +	    // Bah, then just assume everything is safe.
> +	}
> +	return null;
>     }
>
>     private String getRelativePath(File base, File f) {
> @@ -1762,6 +1797,7 @@
>     // these args are jtreg extras
>     private File baseDirArg;
>     private boolean sameJVMFlag;
> +    private List<String> sameJVMSafeDirs;
>     private JDK jdk;
>     private boolean guiFlag;
>     private boolean reportOnlyFlag;
> diff -r 9aae858397f9 test/jtreg/com/sun/javatest/regtest/ 
> RegressionParameters.java
> --- a/test/jtreg/com/sun/javatest/regtest/RegressionParameters.java	 
> Fri Nov 21 18:35:27 2008 -0500
> +++ b/test/jtreg/com/sun/javatest/regtest/RegressionParameters.java	 
> Sun Nov 23 16:56:34 2008 +0100
> @@ -139,6 +139,7 @@
>     private static final String TEST_JAVA_OPTIONS = ".testJavaOpts";
>     private static final String IGNORE = ".ignore";
>     private static final String RETAIN_ARGS = ".retain";
> +    private static final String SAME_JVM_SAFE_DIRS =  
> ".samejvmsafedirs";
>
>     @Override
>     public void load(Map data, boolean checkChecksum) throws  
> Interview.Fault {
> @@ -182,6 +183,10 @@
>         v = (String) data.get(prefix + RETAIN_ARGS);
>         if (v != null)
>              
> setRetainArgs(Arrays.asList(StringArray.splitSeparator("\n", v)));
> +
> +        v = (String) data.get(prefix + SAME_JVM_SAFE_DIRS);
> +        if (v != null)
> +             
> setSameJVMSafeDirs(Arrays.asList(StringArray.splitSeparator("\n",  
> v)));
>     }
>
>     @SuppressWarnings("unchecked")
> @@ -203,6 +208,9 @@
>
>         if (jdk != null)
>             data.put(prefix + JDK, jdk.getPath());
> +
> +        if (sameJVMSafeDirs != null)
> +            data.put(prefix + SAME_JVM_SAFE_DIRS,  
> StringUtils.join(sameJVMSafeDirs, "\n"));
>
>         if (retainArgs != null)
>             data.put(prefix + RETAIN_ARGS,  
> StringUtils.join(retainArgs, "\n"));
> @@ -592,6 +600,18 @@
>     private Pattern retainFilesPattern;
>
>     //---------------------------------------------------------------------
> +
> +    List<String> getSameJVMSafeDirs() {
> +        return sameJVMSafeDirs;
> +    }
> +
> +    void setSameJVMSafeDirs(List<String> sameJVMSafeDirs) {
> +        this.sameJVMSafeDirs= sameJVMSafeDirs;
> +    }
> +
> +    private List<String> sameJVMSafeDirs;
> +
> + 
>     //---------------------------------------------------------------------
>
>     private static final String PATHSEP  =  
> System.getProperty("path.separator");
>     private static final String LINESEP  =  
> System.getProperty("line.separator");
> diff -r 9aae858397f9 test/jtreg/com/sun/javatest/regtest/ 
> RegressionScript.java
> --- a/test/jtreg/com/sun/javatest/regtest/RegressionScript.java	Fri  
> Nov 21 18:35:27 2008 -0500
> +++ b/test/jtreg/com/sun/javatest/regtest/RegressionScript.java	Sun  
> Nov 23 16:56:34 2008 +0100
> @@ -69,6 +69,10 @@
>     public Status run(String[] argv, TestDescription td,  
> TestEnvironment env) {
>         if (!(env instanceof RegressionEnvironment))
>             throw new AssertionError();
> +
> +	String testFilePath = td.getRootRelativePath();
> +	int lastSlash = testFilePath.lastIndexOf('/');
> +	testDirPath = testFilePath.substring(0, lastSlash);
>
>         regEnv = (RegressionEnvironment) env;
>         params = regEnv.params;
> @@ -833,7 +837,26 @@
>     }
>
>     boolean isOtherJVM() {
> -        return params.isOtherJVM();
> +	boolean samevm = !params.isOtherJVM();
> +	if (samevm)
> +	    return !isSameJVMSafe();
> +	else
> +	    return true;
> +    }
> +
> +    // Whether the actions of this script can safely run in the  
> same jvm.
> +    // No same jvm safe dirs given means they are all assumed safe.
> +    // If our actions come from a file in a subdir of a safe dir  
> that is ok.
> +    boolean isSameJVMSafe() {
> +        List<String> dirs = params.getSameJVMSafeDirs();
> +	if (dirs == null)
> +	    return true;
> +
> +	for (String dir : dirs)
> +	    if (testDirPath.startsWith(dir))
> +		return true;
> +
> +	return false;
>     }
>
>     String getJavaProg() {
> @@ -921,5 +944,6 @@
>
>     private RegressionEnvironment regEnv;
>     private RegressionParameters params;
> +    private String testDirPath;
> }
>
> <icedtea-samejvm-safe.patch>