From jonathan.gibbons at oracle.com  Fri Mar  2 19:33:07 2018
From: jonathan.gibbons at oracle.com (jonathan.gibbons at oracle.com)
Date: Fri, 02 Mar 2018 19:33:07 +0000
Subject: hg: code-tools/jtreg: 7902127: IllegalStateException in
 getReportDirectory when -noreport is used
Message-ID: <201803021933.w22JX7kv026533@aojmv0008.oracle.com>

Changeset: 4e8a76b6763c
Author:    jjg
Date:      2018-03-02 11:27 -0800
URL:       http://hg.openjdk.java.net/code-tools/jtreg/rev/4e8a76b6763c

7902127: IllegalStateException in getReportDirectory when -noreport is used

! src/share/classes/com/sun/javatest/regtest/tool/Tool.java
! test/multirun/MultiRunTest.gmk


From jonathan.gibbons at oracle.com  Fri Mar  2 19:44:35 2018
From: jonathan.gibbons at oracle.com (jonathan.gibbons at oracle.com)
Date: Fri, 02 Mar 2018 19:44:35 +0000
Subject: hg: code-tools/jtreg: Fix HTML for <tr> in report file
Message-ID: <201803021944.w22JiZ05002067@aojmv0008.oracle.com>

Changeset: f6757d07705e
Author:    jjg
Date:      2018-03-02 11:39 -0800
URL:       http://hg.openjdk.java.net/code-tools/jtreg/rev/f6757d07705e

Fix HTML for <tr> in report file

! src/share/classes/com/sun/javatest/regtest/report/RegressionReporter.java


From jonathan.gibbons at oracle.com  Fri Mar  2 20:15:28 2018
From: jonathan.gibbons at oracle.com (jonathan.gibbons at oracle.com)
Date: Fri, 02 Mar 2018 20:15:28 +0000
Subject: hg: code-tools/jtreg: 2 new changesets
Message-ID: <201803022015.w22KFSna015861@aojmv0008.oracle.com>

Changeset: 49082d1ff391
Author:    afedorch
Date:      2018-03-01 18:28 -0800
URL:       http://hg.openjdk.java.net/code-tools/jtreg/rev/49082d1ff391

update HTML output to use HTML5

! src/share/classes/com/sun/javatest/diff/HTMLReporter.java
! src/share/classes/com/sun/javatest/diff/SuperDiff.java
! src/share/classes/com/sun/javatest/regtest/report/RegressionReporter.java

Changeset: 5f71ac9f999f
Author:    jjg
Date:      2018-03-02 12:10 -0800
URL:       http://hg.openjdk.java.net/code-tools/jtreg/rev/5f71ac9f999f

Merge



From jonathan.gibbons at oracle.com  Sat Mar  3 00:10:50 2018
From: jonathan.gibbons at oracle.com (jonathan.gibbons at oracle.com)
Date: Sat, 03 Mar 2018 00:10:50 +0000
Subject: hg: code-tools/jtreg: 2 new changesets
Message-ID: <201803030010.w230Aojb006931@aojmv0008.oracle.com>

Changeset: 4e56945f1ff2
Author:    jjg
Date:      2018-03-02 16:02 -0800
URL:       http://hg.openjdk.java.net/code-tools/jtreg/rev/4e56945f1ff2

Update README; now same as content on website

! README

Changeset: 51bd85270bd9
Author:    jjg
Date:      2018-03-02 16:04 -0800
URL:       http://hg.openjdk.java.net/code-tools/jtreg/rev/51bd85270bd9

Minor updates to build-all.sh for configurability

! make/build-all.sh


From jonathan.gibbons at oracle.com  Sat Mar  3 00:44:50 2018
From: jonathan.gibbons at oracle.com (jonathan.gibbons at oracle.com)
Date: Sat, 03 Mar 2018 00:44:50 +0000
Subject: hg: code-tools/jtreg: Change default Ant to 1.9.4
Message-ID: <201803030044.w230ipj3022556@aojmv0008.oracle.com>

Changeset: 7f6a84fad097
Author:    jjg
Date:      2018-03-02 16:40 -0800
URL:       http://hg.openjdk.java.net/code-tools/jtreg/rev/7f6a84fad097

Change default Ant to 1.9.4

! make/Defs.gmk
! make/build-all.sh


From jonathan.gibbons at oracle.com  Sat Mar  3 02:04:09 2018
From: jonathan.gibbons at oracle.com (jonathan.gibbons at oracle.com)
Date: Sat, 03 Mar 2018 02:04:09 +0000
Subject: hg: code-tools/jtreg: build-all.sh: use downloaded Ant to compile as
 to compile against
Message-ID: <201803030204.w2324Are027833@aojmv0008.oracle.com>

Changeset: b394d526666c
Author:    jjg
Date:      2018-03-02 17:59 -0800
URL:       http://hg.openjdk.java.net/code-tools/jtreg/rev/b394d526666c

build-all.sh: use downloaded Ant to compile as to compile against

! make/build-all.sh


From jonathan.gibbons at oracle.com  Thu Mar  8 01:13:10 2018
From: jonathan.gibbons at oracle.com (jonathan.gibbons at oracle.com)
Date: Thu, 08 Mar 2018 01:13:10 +0000
Subject: hg: code-tools/jtreg: 7902131: Generation of stack traces when "Error
 while cleaning up threads after test" occurred
Message-ID: <201803080113.w281DBe0028322@aojmv0008.oracle.com>

Changeset: 6e7a7ae9d3f2
Author:    jjg
Date:      2018-03-07 17:08 -0800
URL:       http://hg.openjdk.java.net/code-tools/jtreg/rev/6e7a7ae9d3f2

7902131: Generation of stack traces when "Error while cleaning up threads after test" occurred

! src/share/classes/com/sun/javatest/regtest/agent/ActionHelper.java
! src/share/classes/com/sun/javatest/regtest/agent/MainActionHelper.java
! test/6517728/T6517728.gmk
! test/6517728/T6517728B.java
+ test/threadCleanup/TEST.ROOT
+ test/threadCleanup/Test.java
+ test/threadCleanup/ThreadCleanupTest.gmk


From jonathan.gibbons at oracle.com  Tue Mar 13 18:01:42 2018
From: jonathan.gibbons at oracle.com (jonathan.gibbons at oracle.com)
Date: Tue, 13 Mar 2018 18:01:42 +0000
Subject: hg: code-tools/jtreg: 7902093: IntelliJ plugin does not build against
 latest SDK
Message-ID: <201803131801.w2DI1gYs020230@aojmv0008.oracle.com>

Changeset: b5f8dbaf9dcd
Author:    mcimadamore
Date:      2018-01-09 18:05 +0000
URL:       http://hg.openjdk.java.net/code-tools/jtreg/rev/b5f8dbaf9dcd

7902093: IntelliJ plugin does not build against latest SDK
* Add missing overrides
* Fix unchecked warnings
* Fix deprecation warnings

! plugins/idea/.idea/misc.xml
! plugins/idea/resources/META-INF/plugin.xml
! plugins/idea/src/com/oracle/plugin/jtreg/components/JTRegFileManagerListener.java
! plugins/idea/src/com/oracle/plugin/jtreg/configuration/JTRegConfiguration.java
! plugins/idea/src/com/oracle/plugin/jtreg/service/ui/JTRegServiceConfigurable.java


From jonathan.gibbons at oracle.com  Wed Mar 14 20:32:52 2018
From: jonathan.gibbons at oracle.com (Jonathan Gibbons)
Date: Wed, 14 Mar 2018 13:32:52 -0700
Subject: jtreg JDK7 support
In-Reply-To: <55892bf5-0747-3bf9-411c-d5ab7bf30fdc@redhat.com>
References: <55892bf5-0747-3bf9-411c-d5ab7bf30fdc@redhat.com>
Message-ID: <3a0ac226-e527-490b-d6b6-027fc65c0ccb@oracle.com>

The intent is that the OpenJDK tests for OpenJDK 6 and 7 should be 
runnable with jtreg using JDK 7.

Since IIRC there are no OpenJDK tests using asmtools and TestNG in 
OpenJDK 6 and 7, the issue of asmtools and TestNG using a newer 
classfile version has not arisen before.

If you want to run some new/additional TestNG tests on JDK 7, you'll 
have to build jtreg with an older, compatible version of TestNG. If you 
want to run some new/additional tests using asmtools files, you'll have 
to work with the AsmTools folk (asmtools-dev at openjdk.java.net) to ensure 
the code is compatible with JDK 7.

-- Jon


On 2/26/18 8:38 AM, Zden?k ?ambersk? wrote:
> Hi,
>
> I have noticed that latest build of jtreg (tip) on adoptopenjdk is not 
> compatible with jdk7.
> Jtreg.jar itself has class versions compatible with JDK 7, but some 
> dependencies
> require JDK >=8 (namely testng.jar and asmtools.jar).
>
>
> this caused some tests fail (on JDK 7) with e.g.
>
> Failed. Execution failed: `main' threw exception: 
> java.lang.UnsupportedClassVersionError: org/testng/IReporter : 
> Unsupported major.minor version 52.0
>
>
> I tried to build jtreg myself using make/build-all.sh with JDK 7, but 
> script complained
> it requires jdk8. I tried to remove this check :) , but it failed when 
> building
> asmtools-7.0-b02. However page about building jtreg still mentions 
> both JDK 7 or JDK 8 [1].
>
> Could you please clarify what is the current status of jtreg, when it 
> comes to JDK 7 support ?
>
>
> [1] http://openjdk.java.net/jtreg/build.html
>
>
> thank you
>
> best regards,
>


From alexandre.iline at oracle.com  Sat Mar 24 00:54:54 2018
From: alexandre.iline at oracle.com (Alexandre (Shura) Iline)
Date: Fri, 23 Mar 2018 17:54:54 -0700
Subject: Tool suggestion: JTReg test module dependencies verifier. 
Message-ID: <1BE70BC9-B15C-498E-B3C2-7E01C36AFC65@oracle.com>

Hi.

This is an invitation to discuss requirements for a ?JTReg test module dependencies verifier? (for not having a better name).

Background: test module dependencies are declared with @modules JTReg tag and modules entry in test suite configuration files. Please consult JTReg documentation for more information.

Problem statement: At the current state there is no effective way for a test developer to verify correctness of module declarations for a test or a group of tests. To verify that information, a test developer needs to manually limit module availability to those declared as module dependencies. This could be achieved either through VM options or by building a custom JDK image. That operation has to be repeated for every subset of tests having unique set of module dependencies. This is a tedious, error prone manual process.

Intent of the tool is to assess correctness of declared module dependencies of JTReg tests.

I. Functional requirements
1. The tool must be able to detect tests which, for execution, depend on presence of more modules than declared.
2. There must be a way to specify a subset of tests for analysis in a way that is compatible with JTReg.
3. Results of the module dependencies assessment should be stored in a machine parse-able format.
3. Results of the module dependencies assessment should be provided in a human readable format.
4. The tool must allow to specify additional test execution parameters which influence test behavior.
5. The tool may provide some analysis of test artifacts in order to identify missing module dependencies.
6. The tool may provide some analysis of test artifacts in order to identify extra, not needed module dependencies.
7. The tool must provide a command-line interface.

II. Non-functional requirements
1. The tool must use JTReg tool or JTReg API to perform test selection and test execution.

III. Non-requirement
1. The tool does not have to be a part of JTReg source repository.
2. The tool does not have to be a part of binary JTReg distribution.
3. It is not a requirement for the tool to verify other JTReg test tags or any other aspects of test suite consistency.


Shura

From alexey.fedorchenko at oracle.com  Mon Mar 26 18:37:43 2018
From: alexey.fedorchenko at oracle.com (alexey.fedorchenko at oracle.com)
Date: Mon, 26 Mar 2018 18:37:43 +0000
Subject: hg: code-tools/jtreg: 2 new changesets
Message-ID: <201803261837.w2QIbhos018562@aojmv0008.oracle.com>

Changeset: 98954d23271b
Author:    afedorch
Date:      2018-03-26 11:11 -0700
URL:       http://hg.openjdk.java.net/code-tools/jtreg/rev/98954d23271b

build-all.sh: update the latest JCov version
Reviewed-by: jjg

! make/build-all.sh

Changeset: d8152011e008
Author:    afedorch
Date:      2018-03-26 11:16 -0700
URL:       http://hg.openjdk.java.net/code-tools/jtreg/rev/d8152011e008

build-all.sh: minor changes to make the script less environment dependent
Reviewed-by: jjg

! make/build-all.sh


From jonathan.gibbons at oracle.com  Mon Mar 26 20:55:19 2018
From: jonathan.gibbons at oracle.com (jonathan.gibbons at oracle.com)
Date: Mon, 26 Mar 2018 20:55:19 +0000
Subject: hg: code-tools/jtreg: update to asmtools 7.0 b04
Message-ID: <201803262055.w2QKtJqh029025@aojmv0008.oracle.com>

Changeset: fc971d6132c3
Author:    jjg
Date:      2018-03-26 13:54 -0700
URL:       http://hg.openjdk.java.net/code-tools/jtreg/rev/fc971d6132c3

update to asmtools 7.0 b04

! make/Defs.gmk
! make/build-all.sh


From jonathan.gibbons at oracle.com  Tue Mar 27 22:39:58 2018
From: jonathan.gibbons at oracle.com (Jonathan Gibbons)
Date: Tue, 27 Mar 2018 15:39:58 -0700
Subject: Tool suggestion: JTReg test module dependencies verifier.
In-Reply-To: <1BE70BC9-B15C-498E-B3C2-7E01C36AFC65@oracle.com>
References: <1BE70BC9-B15C-498E-B3C2-7E01C36AFC65@oracle.com>
Message-ID: <531159e8-ffb5-283b-d167-b766cc79211a@oracle.com>

On 3/23/18 5:54 PM, Alexandre (Shura) Iline wrote:

> Hi.
>
> This is an invitation to discuss requirements for a ?JTReg test module dependencies verifier? (for not having a better name).
>
> Background: test module dependencies are declared with @modules JTReg tag and modules entry in test suite configuration files. Please consult JTReg documentation for more information.
>
> Problem statement: At the current state there is no effective way for a test developer to verify correctness of module declarations for a test or a group of tests. To verify that information, a test developer needs to manually limit module availability to those declared as module dependencies. This could be achieved either through VM options or by building a custom JDK image. That operation has to be repeated for every subset of tests having unique set of module dependencies. This is a tedious, error prone manual process.
>
> Intent of the tool is to assess correctness of declared module dependencies of JTReg tests.
>
> I. Functional requirements
> 1. The tool must be able to detect tests which, for execution, depend on presence of more modules than declared.
> 2. There must be a way to specify a subset of tests for analysis in a way that is compatible with JTReg.
> 3. Results of the module dependencies assessment should be stored in a machine parse-able format.
> 3. Results of the module dependencies assessment should be provided in a human readable format.
> 4. The tool must allow to specify additional test execution parameters which influence test behavior.
> 5. The tool may provide some analysis of test artifacts in order to identify missing module dependencies.
> 6. The tool may provide some analysis of test artifacts in order to identify extra, not needed module dependencies.
> 7. The tool must provide a command-line interface.
>
> II. Non-functional requirements
> 1. The tool must use JTReg tool or JTReg API to perform test selection and test execution.
>
> III. Non-requirement
> 1. The tool does not have to be a part of JTReg source repository.
> 2. The tool does not have to be a part of binary JTReg distribution.
> 3. It is not a requirement for the tool to verify other JTReg test tags or any other aspects of test suite consistency.
>
>
> Shura

Shura,

Good start. Are there ever any issues with platform dependencies? Or is 
the answer inherently "no" because we only support Java SE and JDK 
modules in the @modules line.

-- Jon

From weijun.wang at oracle.com  Wed Mar 28 00:21:36 2018
From: weijun.wang at oracle.com (Weijun Wang)
Date: Wed, 28 Mar 2018 08:21:36 +0800
Subject: Tool suggestion: JTReg test module dependencies verifier.
In-Reply-To: <1BE70BC9-B15C-498E-B3C2-7E01C36AFC65@oracle.com>
References: <1BE70BC9-B15C-498E-B3C2-7E01C36AFC65@oracle.com>
Message-ID: <2F893DB7-07C0-451B-A247-600B6F11FD17@oracle.com>

Two questions:

1. The modules tag looks like "@module module.name/package:+open". Does this tool intend to check the necessity of the package name and +open option?

2. How does the tool deal with optional dependencies or maybe just useless dependencies? For example, both modules A and B provide the XYZ algorithm and the test wants to ensure the one in A is the preferred one when both modules are available. Will this tool complain B is useless? How can I tell the tool this is intended?

Thanks
Max

> On Mar 24, 2018, at 8:54 AM, Alexandre (Shura) Iline <alexandre.iline at oracle.com> wrote:
> 
> Hi.
> 
> This is an invitation to discuss requirements for a ?JTReg test module dependencies verifier? (for not having a better name).
> 
> Background: test module dependencies are declared with @modules JTReg tag and modules entry in test suite configuration files. Please consult JTReg documentation for more information.
> 
> Problem statement: At the current state there is no effective way for a test developer to verify correctness of module declarations for a test or a group of tests. To verify that information, a test developer needs to manually limit module availability to those declared as module dependencies. This could be achieved either through VM options or by building a custom JDK image. That operation has to be repeated for every subset of tests having unique set of module dependencies. This is a tedious, error prone manual process.
> 
> Intent of the tool is to assess correctness of declared module dependencies of JTReg tests.
> 
> I. Functional requirements
> 1. The tool must be able to detect tests which, for execution, depend on presence of more modules than declared.
> 2. There must be a way to specify a subset of tests for analysis in a way that is compatible with JTReg.
> 3. Results of the module dependencies assessment should be stored in a machine parse-able format.
> 3. Results of the module dependencies assessment should be provided in a human readable format.
> 4. The tool must allow to specify additional test execution parameters which influence test behavior.
> 5. The tool may provide some analysis of test artifacts in order to identify missing module dependencies.
> 6. The tool may provide some analysis of test artifacts in order to identify extra, not needed module dependencies.
> 7. The tool must provide a command-line interface.
> 
> II. Non-functional requirements
> 1. The tool must use JTReg tool or JTReg API to perform test selection and test execution.
> 
> III. Non-requirement
> 1. The tool does not have to be a part of JTReg source repository.
> 2. The tool does not have to be a part of binary JTReg distribution.
> 3. It is not a requirement for the tool to verify other JTReg test tags or any other aspects of test suite consistency.
> 
> 
> Shura


From jonathan.gibbons at oracle.com  Wed Mar 28 15:00:40 2018
From: jonathan.gibbons at oracle.com (Jonathan Gibbons)
Date: Wed, 28 Mar 2018 08:00:40 -0700
Subject: Tool suggestion: JTReg test module dependencies verifier.
In-Reply-To: <2F893DB7-07C0-451B-A247-600B6F11FD17@oracle.com>
References: <1BE70BC9-B15C-498E-B3C2-7E01C36AFC65@oracle.com>
 <2F893DB7-07C0-451B-A247-600B6F11FD17@oracle.com>
Message-ID: <8ff8ea7e-5a93-7d8d-b52a-43674fb9d08c@oracle.com>

Max,

As I understand it, this proposal about being able to run tests on 
custom systems with a subset of the full set of JDK modules. For 
example, we might want to start test a JRE (or smaller) image, instead 
of always and only testing the full JDK image. Thus, it is about the 
accuracy of the list of module names specified in the @modules tag.

* An unnecessary module name means that the test will not be run on 
systems without that module, even though it is not required. This is 
because jtreg will automatically not select the test for execution when 
the test platform does not include any of the specified modules.

* A missing module name means that the test will fail when run on a 
system without that module.

Generally speaking, as I understand it, the testing is about running 
each test individually in a system that has a reduced set of modules. 
This is an expensive operation, because while --limit-modules can be 
used for some tests, it is potentially necessary to run jlink to create 
custom images for tests when --limit-modules can be used.

As regards, the ability to specify packages to be exported or opened, 
missing entries will cause the test to fail, and should be caught by the 
test developer or whenever the test is run; excess entries are mostly 
harmless, and will not prevent the test from running, or from being 
selected to be run.

-- Jon


On 3/27/18 5:21 PM, Weijun Wang wrote:
> Two questions:
>
> 1. The modules tag looks like "@module module.name/package:+open". Does this tool intend to check the necessity of the package name and +open option?
>
> 2. How does the tool deal with optional dependencies or maybe just useless dependencies? For example, both modules A and B provide the XYZ algorithm and the test wants to ensure the one in A is the preferred one when both modules are available. Will this tool complain B is useless? How can I tell the tool this is intended?
>
> Thanks
> Max
>
>> On Mar 24, 2018, at 8:54 AM, Alexandre (Shura) Iline <alexandre.iline at oracle.com> wrote:
>>
>> Hi.
>>
>> This is an invitation to discuss requirements for a ?JTReg test module dependencies verifier? (for not having a better name).
>>
>> Background: test module dependencies are declared with @modules JTReg tag and modules entry in test suite configuration files. Please consult JTReg documentation for more information.
>>
>> Problem statement: At the current state there is no effective way for a test developer to verify correctness of module declarations for a test or a group of tests. To verify that information, a test developer needs to manually limit module availability to those declared as module dependencies. This could be achieved either through VM options or by building a custom JDK image. That operation has to be repeated for every subset of tests having unique set of module dependencies. This is a tedious, error prone manual process.
>>
>> Intent of the tool is to assess correctness of declared module dependencies of JTReg tests.
>>
>> I. Functional requirements
>> 1. The tool must be able to detect tests which, for execution, depend on presence of more modules than declared.
>> 2. There must be a way to specify a subset of tests for analysis in a way that is compatible with JTReg.
>> 3. Results of the module dependencies assessment should be stored in a machine parse-able format.
>> 3. Results of the module dependencies assessment should be provided in a human readable format.
>> 4. The tool must allow to specify additional test execution parameters which influence test behavior.
>> 5. The tool may provide some analysis of test artifacts in order to identify missing module dependencies.
>> 6. The tool may provide some analysis of test artifacts in order to identify extra, not needed module dependencies.
>> 7. The tool must provide a command-line interface.
>>
>> II. Non-functional requirements
>> 1. The tool must use JTReg tool or JTReg API to perform test selection and test execution.
>>
>> III. Non-requirement
>> 1. The tool does not have to be a part of JTReg source repository.
>> 2. The tool does not have to be a part of binary JTReg distribution.
>> 3. It is not a requirement for the tool to verify other JTReg test tags or any other aspects of test suite consistency.
>>
>>
>> Shura


From alexandre.iline at oracle.com  Wed Mar 28 17:34:27 2018
From: alexandre.iline at oracle.com (Alexandre (Shura) Iline)
Date: Wed, 28 Mar 2018 10:34:27 -0700
Subject: Tool suggestion: JTReg test module dependencies verifier.
In-Reply-To: <2F893DB7-07C0-451B-A247-600B6F11FD17@oracle.com>
References: <1BE70BC9-B15C-498E-B3C2-7E01C36AFC65@oracle.com>
 <2F893DB7-07C0-451B-A247-600B6F11FD17@oracle.com>
Message-ID: <BA03BF64-B8E4-452B-8C50-E23131F029E1@oracle.com>



> On Mar 27, 2018, at 5:21 PM, Weijun Wang <weijun.wang at oracle.com> wrote:
> 
> Two questions:
> 
> 1. The modules tag looks like "@module module.name/package:+open". Does this tool intend to check the necessity of the package name and +open option?

You are correct. I will add this (or similar) to the requirements: "The tool must be capable of detecting unneeded internal package dependencies."

As far as [+]?open goes, however, I see no reasonable way to detect that, other than some deep instrumentation of JDK code.

> 
> 2. How does the tool deal with optional dependencies or maybe just useless dependencies? For example, both modules A and B provide the XYZ algorithm and the test wants to ensure the one in A is the preferred one when both modules are available. Will this tool complain B is useless? How can I tell the tool this is intended?

I think we can only go part of the way, which is why I have put this item under ?may?: "The tool may provide some analysis of test artifacts in order to identify extra, not needed module dependencies."

Indeed, any test using API from a module which use a service may, in reality, have a runtime dependency on any number of modules which provide that service. That is a very common scenario, indeed. Almost every module using java.compiler is actually requiring jdk.compiler, to give one example.

So, in my current implementation I do this:
For every non-API dependency (that is, for every module which is not detected by jdeps), I check if any of the of the API dependency modules declares a service which is provided by the module in question. If none, I report the dependency as not used. Let me use an example to explain. Assume a test has @modules jdk.compiler. I see that jdk.compiler provides java.util.spi.ToolProvider, com.sun.tools.javac.platform.PlatformProvider, javax.tools.JavaCompiler, com.sun.tools.javac.api.JavacTool. If among the API dependencies there is java.compiler (most often through java.tools.JavaCompiler), than the dependency on jdk.compiler is justified. If none of the services provided by jdk.compiler is  used by any of the API dependencies, than the module jdk.compiler is reported as not used.

Needless to say, this is imperfect, as
1. there could be cases when no service implementation provided by the module is actually used. Such as, for example, no module which provides any service from java.base will ever be reported as not used.
2. some modules reported as not used while they actually are used - I have a handful of cases like that in jdk tier1 tests.

I know not if a better way to detect this, other than deep instrumentation of JDK code.

I was saving this topic for design discussion. We seem to be having it already, which is perfectly normal.

Thanks.

Shura 


> 
> Thanks
> Max
> 
>> On Mar 24, 2018, at 8:54 AM, Alexandre (Shura) Iline <alexandre.iline at oracle.com> wrote:
>> 
>> Hi.
>> 
>> This is an invitation to discuss requirements for a ?JTReg test module dependencies verifier? (for not having a better name).
>> 
>> Background: test module dependencies are declared with @modules JTReg tag and modules entry in test suite configuration files. Please consult JTReg documentation for more information.
>> 
>> Problem statement: At the current state there is no effective way for a test developer to verify correctness of module declarations for a test or a group of tests. To verify that information, a test developer needs to manually limit module availability to those declared as module dependencies. This could be achieved either through VM options or by building a custom JDK image. That operation has to be repeated for every subset of tests having unique set of module dependencies. This is a tedious, error prone manual process.
>> 
>> Intent of the tool is to assess correctness of declared module dependencies of JTReg tests.
>> 
>> I. Functional requirements
>> 1. The tool must be able to detect tests which, for execution, depend on presence of more modules than declared.
>> 2. There must be a way to specify a subset of tests for analysis in a way that is compatible with JTReg.
>> 3. Results of the module dependencies assessment should be stored in a machine parse-able format.
>> 3. Results of the module dependencies assessment should be provided in a human readable format.
>> 4. The tool must allow to specify additional test execution parameters which influence test behavior.
>> 5. The tool may provide some analysis of test artifacts in order to identify missing module dependencies.
>> 6. The tool may provide some analysis of test artifacts in order to identify extra, not needed module dependencies.
>> 7. The tool must provide a command-line interface.
>> 
>> II. Non-functional requirements
>> 1. The tool must use JTReg tool or JTReg API to perform test selection and test execution.
>> 
>> III. Non-requirement
>> 1. The tool does not have to be a part of JTReg source repository.
>> 2. The tool does not have to be a part of binary JTReg distribution.
>> 3. It is not a requirement for the tool to verify other JTReg test tags or any other aspects of test suite consistency.
>> 
>> 
>> Shura
> 


From alexandre.iline at oracle.com  Wed Mar 28 17:39:18 2018
From: alexandre.iline at oracle.com (Alexandre (Shura) Iline)
Date: Wed, 28 Mar 2018 10:39:18 -0700
Subject: Tool suggestion: JTReg test module dependencies verifier.
In-Reply-To: <8ff8ea7e-5a93-7d8d-b52a-43674fb9d08c@oracle.com>
References: <1BE70BC9-B15C-498E-B3C2-7E01C36AFC65@oracle.com>
 <2F893DB7-07C0-451B-A247-600B6F11FD17@oracle.com>
 <8ff8ea7e-5a93-7d8d-b52a-43674fb9d08c@oracle.com>
Message-ID: <39F10A82-FE81-4C9F-8F8D-27B3BE8171D5@oracle.com>



> On Mar 28, 2018, at 8:00 AM, Jonathan Gibbons <jonathan.gibbons at oracle.com> wrote:
> 
> Max,
> 
> As I understand it, this proposal about being able to run tests on custom systems with a subset of the full set of JDK modules. For example, we might want to start test a JRE (or smaller) image, instead of always and only testing the full JDK image. Thus, it is about the accuracy of the list of module names specified in the @modules tag.
> 
> * An unnecessary module name means that the test will not be run on systems without that module, even though it is not required. This is because jtreg will automatically not select the test for execution when the test platform does not include any of the specified modules.
> 
> * A missing module name means that the test will fail when run on a system without that module.
> 
> Generally speaking, as I understand it, the testing is about running each test individually in a system that has a reduced set of modules. This is an expensive operation, because while --limit-modules can be used for some tests, it is potentially necessary to run jlink to create custom images for tests when --limit-modules can be used.
> 
> As regards, the ability to specify packages to be exported or opened, missing entries will cause the test to fail, and should be caught by the test developer or whenever the test is run; excess entries are mostly harmless, and will not prevent the test from running, or from being selected to be run.

All correct, Jon.

I, however, think that detecting extra internal package decencies is also useful. Since it is easy to implement, we can have it. After all, " the tool is to assess correctness of declared module dependencies? and that is a part of the ?correctness?.

Shura

> 
> -- Jon
> 
> 
> On 3/27/18 5:21 PM, Weijun Wang wrote:
>> Two questions:
>> 
>> 1. The modules tag looks like "@module module.name/package:+open". Does this tool intend to check the necessity of the package name and +open option?
>> 
>> 2. How does the tool deal with optional dependencies or maybe just useless dependencies? For example, both modules A and B provide the XYZ algorithm and the test wants to ensure the one in A is the preferred one when both modules are available. Will this tool complain B is useless? How can I tell the tool this is intended?
>> 
>> Thanks
>> Max
>> 
>>> On Mar 24, 2018, at 8:54 AM, Alexandre (Shura) Iline <alexandre.iline at oracle.com> wrote:
>>> 
>>> Hi.
>>> 
>>> This is an invitation to discuss requirements for a ?JTReg test module dependencies verifier? (for not having a better name).
>>> 
>>> Background: test module dependencies are declared with @modules JTReg tag and modules entry in test suite configuration files. Please consult JTReg documentation for more information.
>>> 
>>> Problem statement: At the current state there is no effective way for a test developer to verify correctness of module declarations for a test or a group of tests. To verify that information, a test developer needs to manually limit module availability to those declared as module dependencies. This could be achieved either through VM options or by building a custom JDK image. That operation has to be repeated for every subset of tests having unique set of module dependencies. This is a tedious, error prone manual process.
>>> 
>>> Intent of the tool is to assess correctness of declared module dependencies of JTReg tests.
>>> 
>>> I. Functional requirements
>>> 1. The tool must be able to detect tests which, for execution, depend on presence of more modules than declared.
>>> 2. There must be a way to specify a subset of tests for analysis in a way that is compatible with JTReg.
>>> 3. Results of the module dependencies assessment should be stored in a machine parse-able format.
>>> 3. Results of the module dependencies assessment should be provided in a human readable format.
>>> 4. The tool must allow to specify additional test execution parameters which influence test behavior.
>>> 5. The tool may provide some analysis of test artifacts in order to identify missing module dependencies.
>>> 6. The tool may provide some analysis of test artifacts in order to identify extra, not needed module dependencies.
>>> 7. The tool must provide a command-line interface.
>>> 
>>> II. Non-functional requirements
>>> 1. The tool must use JTReg tool or JTReg API to perform test selection and test execution.
>>> 
>>> III. Non-requirement
>>> 1. The tool does not have to be a part of JTReg source repository.
>>> 2. The tool does not have to be a part of binary JTReg distribution.
>>> 3. It is not a requirement for the tool to verify other JTReg test tags or any other aspects of test suite consistency.
>>> 
>>> 
>>> Shura
> 


From jonathan.gibbons at oracle.com  Wed Mar 28 17:45:02 2018
From: jonathan.gibbons at oracle.com (Jonathan Gibbons)
Date: Wed, 28 Mar 2018 10:45:02 -0700
Subject: Tool suggestion: JTReg test module dependencies verifier.
In-Reply-To: <BA03BF64-B8E4-452B-8C50-E23131F029E1@oracle.com>
References: <1BE70BC9-B15C-498E-B3C2-7E01C36AFC65@oracle.com>
 <2F893DB7-07C0-451B-A247-600B6F11FD17@oracle.com>
 <BA03BF64-B8E4-452B-8C50-E23131F029E1@oracle.com>
Message-ID: <5fbf458b-8ef0-f0fe-b71f-d038e313d5a2@oracle.com>

Shura,

I think you should start ranking your requirements, in terms of "must 
have", "nice to have" etc, or any similar scale.

I think that "must have" should be to detect conditions which may cause 
the test to fail on a restricted system. Think of them as the 
hard-to-detect errors that might upset a test run on a custom image.

I think that "nice to have" should be lint-like checks for conditions 
which will not cause the test to fail, but which are not necessary for 
the test to succeed.

There's a 3rd (weaker?) category of "implementation desires", such as 
whether or not or how much this is integrated into jtreg.

--

There is potentially a difference between "package not found" and 
"export|open not required" in a `@module some.module/some.package`

-- Jon


On 3/28/18 10:34 AM, Alexandre (Shura) Iline wrote:
>
>> On Mar 27, 2018, at 5:21 PM, Weijun Wang <weijun.wang at oracle.com> wrote:
>>
>> Two questions:
>>
>> 1. The modules tag looks like "@module module.name/package:+open". Does this tool intend to check the necessity of the package name and +open option?
> You are correct. I will add this (or similar) to the requirements: "The tool must be capable of detecting unneeded internal package dependencies."
>
> As far as [+]?open goes, however, I see no reasonable way to detect that, other than some deep instrumentation of JDK code.
>
>> 2. How does the tool deal with optional dependencies or maybe just useless dependencies? For example, both modules A and B provide the XYZ algorithm and the test wants to ensure the one in A is the preferred one when both modules are available. Will this tool complain B is useless? How can I tell the tool this is intended?
> I think we can only go part of the way, which is why I have put this item under ?may?: "The tool may provide some analysis of test artifacts in order to identify extra, not needed module dependencies."
>
> Indeed, any test using API from a module which use a service may, in reality, have a runtime dependency on any number of modules which provide that service. That is a very common scenario, indeed. Almost every module using java.compiler is actually requiring jdk.compiler, to give one example.
>
> So, in my current implementation I do this:
> For every non-API dependency (that is, for every module which is not detected by jdeps), I check if any of the of the API dependency modules declares a service which is provided by the module in question. If none, I report the dependency as not used. Let me use an example to explain. Assume a test has @modules jdk.compiler. I see that jdk.compiler provides java.util.spi.ToolProvider, com.sun.tools.javac.platform.PlatformProvider, javax.tools.JavaCompiler, com.sun.tools.javac.api.JavacTool. If among the API dependencies there is java.compiler (most often through java.tools.JavaCompiler), than the dependency on jdk.compiler is justified. If none of the services provided by jdk.compiler is  used by any of the API dependencies, than the module jdk.compiler is reported as not used.
>
> Needless to say, this is imperfect, as
> 1. there could be cases when no service implementation provided by the module is actually used. Such as, for example, no module which provides any service from java.base will ever be reported as not used.
> 2. some modules reported as not used while they actually are used - I have a handful of cases like that in jdk tier1 tests.
>
> I know not if a better way to detect this, other than deep instrumentation of JDK code.
>
> I was saving this topic for design discussion. We seem to be having it already, which is perfectly normal.
>
> Thanks.
>
> Shura
>
>
>> Thanks
>> Max
>>
>>> On Mar 24, 2018, at 8:54 AM, Alexandre (Shura) Iline <alexandre.iline at oracle.com> wrote:
>>>
>>> Hi.
>>>
>>> This is an invitation to discuss requirements for a ?JTReg test module dependencies verifier? (for not having a better name).
>>>
>>> Background: test module dependencies are declared with @modules JTReg tag and modules entry in test suite configuration files. Please consult JTReg documentation for more information.
>>>
>>> Problem statement: At the current state there is no effective way for a test developer to verify correctness of module declarations for a test or a group of tests. To verify that information, a test developer needs to manually limit module availability to those declared as module dependencies. This could be achieved either through VM options or by building a custom JDK image. That operation has to be repeated for every subset of tests having unique set of module dependencies. This is a tedious, error prone manual process.
>>>
>>> Intent of the tool is to assess correctness of declared module dependencies of JTReg tests.
>>>
>>> I. Functional requirements
>>> 1. The tool must be able to detect tests which, for execution, depend on presence of more modules than declared.
>>> 2. There must be a way to specify a subset of tests for analysis in a way that is compatible with JTReg.
>>> 3. Results of the module dependencies assessment should be stored in a machine parse-able format.
>>> 3. Results of the module dependencies assessment should be provided in a human readable format.
>>> 4. The tool must allow to specify additional test execution parameters which influence test behavior.
>>> 5. The tool may provide some analysis of test artifacts in order to identify missing module dependencies.
>>> 6. The tool may provide some analysis of test artifacts in order to identify extra, not needed module dependencies.
>>> 7. The tool must provide a command-line interface.
>>>
>>> II. Non-functional requirements
>>> 1. The tool must use JTReg tool or JTReg API to perform test selection and test execution.
>>>
>>> III. Non-requirement
>>> 1. The tool does not have to be a part of JTReg source repository.
>>> 2. The tool does not have to be a part of binary JTReg distribution.
>>> 3. It is not a requirement for the tool to verify other JTReg test tags or any other aspects of test suite consistency.
>>>
>>>
>>> Shura


From alexandre.iline at oracle.com  Wed Mar 28 17:57:52 2018
From: alexandre.iline at oracle.com (Alexandre (Shura) Iline)
Date: Wed, 28 Mar 2018 10:57:52 -0700
Subject: Tool suggestion: JTReg test module dependencies verifier.
In-Reply-To: <5fbf458b-8ef0-f0fe-b71f-d038e313d5a2@oracle.com>
References: <1BE70BC9-B15C-498E-B3C2-7E01C36AFC65@oracle.com>
 <2F893DB7-07C0-451B-A247-600B6F11FD17@oracle.com>
 <BA03BF64-B8E4-452B-8C50-E23131F029E1@oracle.com>
 <5fbf458b-8ef0-f0fe-b71f-d038e313d5a2@oracle.com>
Message-ID: <5B213A07-BF68-4CE8-A386-6C02BCF71427@oracle.com>



> On Mar 28, 2018, at 10:45 AM, Jonathan Gibbons <jonathan.gibbons at oracle.com> wrote:
> 
> Shura,
> 
> I think you should start ranking your requirements, in terms of "must have", "nice to have" etc, or any similar scale.
> 
> I think that "must have" should be to detect conditions which may cause the test to fail on a restricted system. Think of them as the hard-to-detect errors that might upset a test run on a custom image.
> 
> I think that "nice to have" should be lint-like checks for conditions which will not cause the test to fail, but which are not necessary for the test to succeed.

Yes, point taken. 

When I look on the list, then, those functional requirements which are ?must? belong to the ?must have? group. 

Having (human|machine)-readable output is also "mush have" - perhaps this should also be saying ?must?. 

And the internal package dependency detection requirement (which I think have triggered your response), should not really be ?must? or belong to the ?must have" group.

> 
> There's a 3rd (weaker?) category of "implementation desires", such as whether or not or how much this is integrated into jtreg.

Correct, these are the non-functional requirements, of which at least one is missing right now:
2. The tool must use jdeps tool or jdeps API to perform test selection and test execution.


I will rearrange and resend.

Shura

> 
> --
> 
> There is potentially a difference between "package not found" and "export|open not required" in a `@module some.module/some.package`
> 
> -- Jon
> 
> 
> On 3/28/18 10:34 AM, Alexandre (Shura) Iline wrote:
>> 
>>> On Mar 27, 2018, at 5:21 PM, Weijun Wang <weijun.wang at oracle.com> wrote:
>>> 
>>> Two questions:
>>> 
>>> 1. The modules tag looks like "@module module.name/package:+open". Does this tool intend to check the necessity of the package name and +open option?
>> You are correct. I will add this (or similar) to the requirements: "The tool must be capable of detecting unneeded internal package dependencies."
>> 
>> As far as [+]?open goes, however, I see no reasonable way to detect that, other than some deep instrumentation of JDK code.
>> 
>>> 2. How does the tool deal with optional dependencies or maybe just useless dependencies? For example, both modules A and B provide the XYZ algorithm and the test wants to ensure the one in A is the preferred one when both modules are available. Will this tool complain B is useless? How can I tell the tool this is intended?
>> I think we can only go part of the way, which is why I have put this item under ?may?: "The tool may provide some analysis of test artifacts in order to identify extra, not needed module dependencies."
>> 
>> Indeed, any test using API from a module which use a service may, in reality, have a runtime dependency on any number of modules which provide that service. That is a very common scenario, indeed. Almost every module using java.compiler is actually requiring jdk.compiler, to give one example.
>> 
>> So, in my current implementation I do this:
>> For every non-API dependency (that is, for every module which is not detected by jdeps), I check if any of the of the API dependency modules declares a service which is provided by the module in question. If none, I report the dependency as not used. Let me use an example to explain. Assume a test has @modules jdk.compiler. I see that jdk.compiler provides java.util.spi.ToolProvider, com.sun.tools.javac.platform.PlatformProvider, javax.tools.JavaCompiler, com.sun.tools.javac.api.JavacTool. If among the API dependencies there is java.compiler (most often through java.tools.JavaCompiler), than the dependency on jdk.compiler is justified. If none of the services provided by jdk.compiler is  used by any of the API dependencies, than the module jdk.compiler is reported as not used.
>> 
>> Needless to say, this is imperfect, as
>> 1. there could be cases when no service implementation provided by the module is actually used. Such as, for example, no module which provides any service from java.base will ever be reported as not used.
>> 2. some modules reported as not used while they actually are used - I have a handful of cases like that in jdk tier1 tests.
>> 
>> I know not if a better way to detect this, other than deep instrumentation of JDK code.
>> 
>> I was saving this topic for design discussion. We seem to be having it already, which is perfectly normal.
>> 
>> Thanks.
>> 
>> Shura
>> 
>> 
>>> Thanks
>>> Max
>>> 
>>>> On Mar 24, 2018, at 8:54 AM, Alexandre (Shura) Iline <alexandre.iline at oracle.com> wrote:
>>>> 
>>>> Hi.
>>>> 
>>>> This is an invitation to discuss requirements for a ?JTReg test module dependencies verifier? (for not having a better name).
>>>> 
>>>> Background: test module dependencies are declared with @modules JTReg tag and modules entry in test suite configuration files. Please consult JTReg documentation for more information.
>>>> 
>>>> Problem statement: At the current state there is no effective way for a test developer to verify correctness of module declarations for a test or a group of tests. To verify that information, a test developer needs to manually limit module availability to those declared as module dependencies. This could be achieved either through VM options or by building a custom JDK image. That operation has to be repeated for every subset of tests having unique set of module dependencies. This is a tedious, error prone manual process.
>>>> 
>>>> Intent of the tool is to assess correctness of declared module dependencies of JTReg tests.
>>>> 
>>>> I. Functional requirements
>>>> 1. The tool must be able to detect tests which, for execution, depend on presence of more modules than declared.
>>>> 2. There must be a way to specify a subset of tests for analysis in a way that is compatible with JTReg.
>>>> 3. Results of the module dependencies assessment should be stored in a machine parse-able format.
>>>> 3. Results of the module dependencies assessment should be provided in a human readable format.
>>>> 4. The tool must allow to specify additional test execution parameters which influence test behavior.
>>>> 5. The tool may provide some analysis of test artifacts in order to identify missing module dependencies.
>>>> 6. The tool may provide some analysis of test artifacts in order to identify extra, not needed module dependencies.
>>>> 7. The tool must provide a command-line interface.
>>>> 
>>>> II. Non-functional requirements
>>>> 1. The tool must use JTReg tool or JTReg API to perform test selection and test execution.
>>>> 
>>>> III. Non-requirement
>>>> 1. The tool does not have to be a part of JTReg source repository.
>>>> 2. The tool does not have to be a part of binary JTReg distribution.
>>>> 3. It is not a requirement for the tool to verify other JTReg test tags or any other aspects of test suite consistency.
>>>> 
>>>> 
>>>> Shura
> 


From jonathan.gibbons at oracle.com  Wed Mar 28 18:22:25 2018
From: jonathan.gibbons at oracle.com (Jonathan Gibbons)
Date: Wed, 28 Mar 2018 11:22:25 -0700
Subject: Tool suggestion: JTReg test module dependencies verifier.
In-Reply-To: <5B213A07-BF68-4CE8-A386-6C02BCF71427@oracle.com>
References: <1BE70BC9-B15C-498E-B3C2-7E01C36AFC65@oracle.com>
 <2F893DB7-07C0-451B-A247-600B6F11FD17@oracle.com>
 <BA03BF64-B8E4-452B-8C50-E23131F029E1@oracle.com>
 <5fbf458b-8ef0-f0fe-b71f-d038e313d5a2@oracle.com>
 <5B213A07-BF68-4CE8-A386-6C02BCF71427@oracle.com>
Message-ID: <667b3445-2b55-60ac-e919-e27976f42109@oracle.com>



On 3/28/18 10:57 AM, Alexandre (Shura) Iline wrote:
>> There's a 3rd (weaker?) category of "implementation desires", such as whether or not or how much this is integrated into jtreg.
> Correct, these are the non-functional requirements, of which at least one is missing right now:
> 2. The tool must use jdeps tool or jdeps API to perform test selection and test execution.
It's quibbling, but I don't think these are requirements. If jdeps or 
the jdeps API did not exist, you could still implement what you are 
proposing.? jdeps and the jdeps API are just technologies which you may 
elect to use to make your task simpler.

-- Jon

From alexandre.iline at oracle.com  Wed Mar 28 18:34:35 2018
From: alexandre.iline at oracle.com (Alexandre (Shura) Iline)
Date: Wed, 28 Mar 2018 11:34:35 -0700
Subject: Tool suggestion: JTReg test module dependencies verifier.
In-Reply-To: <667b3445-2b55-60ac-e919-e27976f42109@oracle.com>
References: <1BE70BC9-B15C-498E-B3C2-7E01C36AFC65@oracle.com>
 <2F893DB7-07C0-451B-A247-600B6F11FD17@oracle.com>
 <BA03BF64-B8E4-452B-8C50-E23131F029E1@oracle.com>
 <5fbf458b-8ef0-f0fe-b71f-d038e313d5a2@oracle.com>
 <5B213A07-BF68-4CE8-A386-6C02BCF71427@oracle.com>
 <667b3445-2b55-60ac-e919-e27976f42109@oracle.com>
Message-ID: <3FC0524C-00F2-4261-900B-D05AED689324@oracle.com>



> On Mar 28, 2018, at 11:22 AM, Jonathan Gibbons <jonathan.gibbons at oracle.com> wrote:
> 
> 
> 
> On 3/28/18 10:57 AM, Alexandre (Shura) Iline wrote:
>>> There's a 3rd (weaker?) category of "implementation desires", such as whether or not or how much this is integrated into jtreg.
>> Correct, these are the non-functional requirements, of which at least one is missing right now:
>> 2. The tool must use jdeps tool or jdeps API to perform test selection and test execution.
> It's quibbling, but I don't think these are requirements. If jdeps or the jdeps API did not exist, you could still implement what you are proposing.  jdeps and the jdeps API are just technologies which you may elect to use to make your task simpler.

Well, it must be a requirement to treat modules in a way compatible with JDK. By requiring what I am requiring, I am guarantying that and also, in addition, since I am strongly opposed to re-implement what is already implemented in jdeps, this is my proposal for the tool: it must use jdeps.

Same, word for word, goes for JTReg, FWIW.

Shura
> 
> -- Jon


From jonathan.gibbons at oracle.com  Wed Mar 28 19:05:15 2018
From: jonathan.gibbons at oracle.com (Jonathan Gibbons)
Date: Wed, 28 Mar 2018 12:05:15 -0700
Subject: Tool suggestion: JTReg test module dependencies verifier.
In-Reply-To: <3FC0524C-00F2-4261-900B-D05AED689324@oracle.com>
References: <1BE70BC9-B15C-498E-B3C2-7E01C36AFC65@oracle.com>
 <2F893DB7-07C0-451B-A247-600B6F11FD17@oracle.com>
 <BA03BF64-B8E4-452B-8C50-E23131F029E1@oracle.com>
 <5fbf458b-8ef0-f0fe-b71f-d038e313d5a2@oracle.com>
 <5B213A07-BF68-4CE8-A386-6C02BCF71427@oracle.com>
 <667b3445-2b55-60ac-e919-e27976f42109@oracle.com>
 <3FC0524C-00F2-4261-900B-D05AED689324@oracle.com>
Message-ID: <5ABBE76B.8080702@oracle.com>



On 03/28/2018 11:34 AM, Alexandre (Shura) Iline wrote:
>
>
>> On Mar 28, 2018, at 11:22 AM, Jonathan Gibbons 
>> <jonathan.gibbons at oracle.com <mailto:jonathan.gibbons at oracle.com>> wrote:
>>
>>
>>
>> On 3/28/18 10:57 AM, Alexandre (Shura) Iline wrote:
>>>> There's a 3rd (weaker?) category of "implementation desires", such as whether or not or how much this is integrated into jtreg.
>>> Correct, these are the non-functional requirements, of which at least one is missing right now:
>>> 2. The tool must use jdeps tool or jdeps API to perform test selection and test execution.
>> It's quibbling, but I don't think these are requirements. If jdeps or 
>> the jdeps API did not exist, you could still implement what you are 
>> proposing.  jdeps and the jdeps API are just technologies which you 
>> may elect to use to make your task simpler.
>
> Well, it must be a requirement to treat modules in a way compatible 
> with JDK. By requiring what I am requiring, I am guarantying that and 
> also, in addition, since I am strongly opposed to re-implement what is 
> already implemented in jdeps, this is my proposal for the tool: it 
> must use jdeps.
>
> Same, word for word, goes for JTReg, FWIW.
>
> Shura
>>
>> -- Jon
>

I'm not against using jdeps, or jtreg. But there is a difference between 
a Requirement and an implementation choice. But this is just an email 
thread, and not any more formal document such as a CSR or JEP, so I'll 
not push on this any more.

-- Jon

From alexandre.iline at oracle.com  Thu Mar 29 18:05:53 2018
From: alexandre.iline at oracle.com (Alexandre (Shura) Iline)
Date: Thu, 29 Mar 2018 11:05:53 -0700
Subject: Tool suggestion: JTReg test module dependencies verifier.
In-Reply-To: <5ABBE76B.8080702@oracle.com>
References: <1BE70BC9-B15C-498E-B3C2-7E01C36AFC65@oracle.com>
 <2F893DB7-07C0-451B-A247-600B6F11FD17@oracle.com>
 <BA03BF64-B8E4-452B-8C50-E23131F029E1@oracle.com>
 <5fbf458b-8ef0-f0fe-b71f-d038e313d5a2@oracle.com>
 <5B213A07-BF68-4CE8-A386-6C02BCF71427@oracle.com>
 <667b3445-2b55-60ac-e919-e27976f42109@oracle.com>
 <3FC0524C-00F2-4261-900B-D05AED689324@oracle.com>
 <5ABBE76B.8080702@oracle.com>
Message-ID: <39DAB207-36D8-405B-A334-0F075D465355@oracle.com>

Jon, do you think I should start JEP for this tool?

I think now is the right time.

Shura

> On Mar 28, 2018, at 12:05 PM, Jonathan Gibbons <jonathan.gibbons at oracle.com> wrote:
> 
> 
> 
> On 03/28/2018 11:34 AM, Alexandre (Shura) Iline wrote:
>> 
>> 
>>> On Mar 28, 2018, at 11:22 AM, Jonathan Gibbons <jonathan.gibbons at oracle.com <mailto:jonathan.gibbons at oracle.com>> wrote:
>>> 
>>> 
>>> 
>>> On 3/28/18 10:57 AM, Alexandre (Shura) Iline wrote:
>>>>> There's a 3rd (weaker?) category of "implementation desires", such as whether or not or how much this is integrated into jtreg.
>>>> Correct, these are the non-functional requirements, of which at least one is missing right now:
>>>> 2. The tool must use jdeps tool or jdeps API to perform test selection and test execution.
>>> It's quibbling, but I don't think these are requirements. If jdeps or the jdeps API did not exist, you could still implement what you are proposing.  jdeps and the jdeps API are just technologies which you may elect to use to make your task simpler.
>> 
>> Well, it must be a requirement to treat modules in a way compatible with JDK. By requiring what I am requiring, I am guarantying that and also, in addition, since I am strongly opposed to re-implement what is already implemented in jdeps, this is my proposal for the tool: it must use jdeps.
>> 
>> Same, word for word, goes for JTReg, FWIW.
>> 
>> Shura
>>> 
>>> -- Jon
>> 
> 
> I'm not against using jdeps, or jtreg. But there is a difference between a Requirement and an implementation choice. But this is just an email thread, and not any more formal document such as a CSR or JEP, so I'll not push on this any more.
> 
> -- Jon


From jonathan.gibbons at oracle.com  Thu Mar 29 18:42:34 2018
From: jonathan.gibbons at oracle.com (Jonathan Gibbons)
Date: Thu, 29 Mar 2018 11:42:34 -0700
Subject: Tool suggestion: JTReg test module dependencies verifier.
In-Reply-To: <39DAB207-36D8-405B-A334-0F075D465355@oracle.com>
References: <1BE70BC9-B15C-498E-B3C2-7E01C36AFC65@oracle.com>
 <2F893DB7-07C0-451B-A247-600B6F11FD17@oracle.com>
 <BA03BF64-B8E4-452B-8C50-E23131F029E1@oracle.com>
 <5fbf458b-8ef0-f0fe-b71f-d038e313d5a2@oracle.com>
 <5B213A07-BF68-4CE8-A386-6C02BCF71427@oracle.com>
 <667b3445-2b55-60ac-e919-e27976f42109@oracle.com>
 <3FC0524C-00F2-4261-900B-D05AED689324@oracle.com>
 <5ABBE76B.8080702@oracle.com>
 <39DAB207-36D8-405B-A334-0F075D465355@oracle.com>
Message-ID: <5ABD339A.70002@oracle.com>

No. This should not be a JEP.

JEPs are for enhancements to the Java platform.

-- Jon

On 03/29/2018 11:05 AM, Alexandre (Shura) Iline wrote:
> Jon, do you think I should start JEP for this tool?
>
> I think now is the right time.
>
> Shura
>
>> On Mar 28, 2018, at 12:05 PM, Jonathan Gibbons 
>> <jonathan.gibbons at oracle.com <mailto:jonathan.gibbons at oracle.com>> wrote:
>>
>>
>>
>> On 03/28/2018 11:34 AM, Alexandre (Shura) Iline wrote:
>>>
>>>
>>>> On Mar 28, 2018, at 11:22 AM, Jonathan Gibbons 
>>>> <jonathan.gibbons at oracle.com <mailto:jonathan.gibbons at oracle.com>> 
>>>> wrote:
>>>>
>>>>
>>>>
>>>> On 3/28/18 10:57 AM, Alexandre (Shura) Iline wrote:
>>>>>> There's a 3rd (weaker?) category of "implementation desires", such as whether or not or how much this is integrated into jtreg.
>>>>> Correct, these are the non-functional requirements, of which at least one is missing right now:
>>>>> 2. The tool must use jdeps tool or jdeps API to perform test selection and test execution.
>>>> It's quibbling, but I don't think these are requirements. If jdeps 
>>>> or the jdeps API did not exist, you could still implement what you 
>>>> are proposing.  jdeps and the jdeps API are just technologies which 
>>>> you may elect to use to make your task simpler.
>>>
>>> Well, it must be a requirement to treat modules in a way compatible 
>>> with JDK. By requiring what I am requiring, I am guarantying that 
>>> and also, in addition, since I am strongly opposed to re-implement 
>>> what is already implemented in jdeps, this is my proposal for the 
>>> tool: it must use jdeps.
>>>
>>> Same, word for word, goes for JTReg, FWIW.
>>>
>>> Shura
>>>>
>>>> -- Jon
>>>
>>
>> I'm not against using jdeps, or jtreg. But there is a difference 
>> between a Requirement and an implementation choice. But this is just 
>> an email thread, and not any more formal document such as a CSR or 
>> JEP, so I'll not push on this any more.
>>
>> -- Jon
>


From jonathan.gibbons at oracle.com  Thu Mar 29 22:51:52 2018
From: jonathan.gibbons at oracle.com (Jonathan Gibbons)
Date: Thu, 29 Mar 2018 15:51:52 -0700
Subject: Tool suggestion: JTReg test module dependencies verifier.
In-Reply-To: <1BE70BC9-B15C-498E-B3C2-7E01C36AFC65@oracle.com>
References: <1BE70BC9-B15C-498E-B3C2-7E01C36AFC65@oracle.com>
Message-ID: <de446dea-9819-a50d-3180-ed3b3e2a9a2c@oracle.com>

Shura,

I think the tool you are proposing could be the first in a series to 
help check for issues in jtreg test descriptions that for whatever 
reason (typically performance) we do not want to check every time we use 
jtreg to run tests.

Other (future) checks could be, using @compile where @build would be 
more appropriate, checking the set of @build directives, and so on.

-- Jon

On 3/23/18 5:54 PM, Alexandre (Shura) Iline wrote:
> Hi.
>
> This is an invitation to discuss requirements for a ?JTReg test module dependencies verifier? (for not having a better name).
>
> Background: test module dependencies are declared with @modules JTReg tag and modules entry in test suite configuration files. Please consult JTReg documentation for more information.
>
> Problem statement: At the current state there is no effective way for a test developer to verify correctness of module declarations for a test or a group of tests. To verify that information, a test developer needs to manually limit module availability to those declared as module dependencies. This could be achieved either through VM options or by building a custom JDK image. That operation has to be repeated for every subset of tests having unique set of module dependencies. This is a tedious, error prone manual process.
>
> Intent of the tool is to assess correctness of declared module dependencies of JTReg tests.
>
> I. Functional requirements
> 1. The tool must be able to detect tests which, for execution, depend on presence of more modules than declared.
> 2. There must be a way to specify a subset of tests for analysis in a way that is compatible with JTReg.
> 3. Results of the module dependencies assessment should be stored in a machine parse-able format.
> 3. Results of the module dependencies assessment should be provided in a human readable format.
> 4. The tool must allow to specify additional test execution parameters which influence test behavior.
> 5. The tool may provide some analysis of test artifacts in order to identify missing module dependencies.
> 6. The tool may provide some analysis of test artifacts in order to identify extra, not needed module dependencies.
> 7. The tool must provide a command-line interface.
>
> II. Non-functional requirements
> 1. The tool must use JTReg tool or JTReg API to perform test selection and test execution.
>
> III. Non-requirement
> 1. The tool does not have to be a part of JTReg source repository.
> 2. The tool does not have to be a part of binary JTReg distribution.
> 3. It is not a requirement for the tool to verify other JTReg test tags or any other aspects of test suite consistency.
>
>
> Shura


From jonathan.gibbons at oracle.com  Fri Mar 30 22:09:29 2018
From: jonathan.gibbons at oracle.com (jonathan.gibbons at oracle.com)
Date: Fri, 30 Mar 2018 22:09:29 +0000
Subject: hg: code-tools/jtreg: Provide ability to set the maxOutputSize for a
 test
Message-ID: <201803302209.w2UM9TEF026473@aojmv0008.oracle.com>

Changeset: a74a10626041
Author:    jjg
Date:      2018-03-30 15:08 -0700
URL:       http://hg.openjdk.java.net/code-tools/jtreg/rev/a74a10626041

Provide ability to set the maxOutputSize for a test

! src/share/classes/com/sun/javatest/regtest/config/RegressionTestSuite.java
! src/share/classes/com/sun/javatest/regtest/config/TestProperties.java
! src/share/classes/com/sun/javatest/regtest/config/i18n.properties
! src/share/classes/com/sun/javatest/regtest/exec/RegressionScript.java
! src/share/doc/javatest/regtest/tag-spec.html
+ test/maxOutputSize/MaxOutputSize.gmk
+ test/maxOutputSize/TEST.ROOT
+ test/maxOutputSize/defaultMax/Test.java
+ test/maxOutputSize/overrideMax/TEST.properties
+ test/maxOutputSize/overrideMax/Test.java