Tool suggestion: JTReg test module dependencies verifier.

[Jonathan Gibbons]

Max,

As I understand it, this proposal about being able to run tests on 
custom systems with a subset of the full set of JDK modules. For 
example, we might want to start test a JRE (or smaller) image, instead 
of always and only testing the full JDK image. Thus, it is about the 
accuracy of the list of module names specified in the @modules tag.

* An unnecessary module name means that the test will not be run on 
systems without that module, even though it is not required. This is 
because jtreg will automatically not select the test for execution when 
the test platform does not include any of the specified modules.

* A missing module name means that the test will fail when run on a 
system without that module.

Generally speaking, as I understand it, the testing is about running 
each test individually in a system that has a reduced set of modules. 
This is an expensive operation, because while --limit-modules can be 
used for some tests, it is potentially necessary to run jlink to create 
custom images for tests when --limit-modules can be used.

As regards, the ability to specify packages to be exported or opened, 
missing entries will cause the test to fail, and should be caught by the 
test developer or whenever the test is run; excess entries are mostly 
harmless, and will not prevent the test from running, or from being 
selected to be run.

-- Jon


On 3/27/18 5:21 PM, Weijun Wang wrote:
> Two questions:
>
> 1. The modules tag looks like "@module module.name/package:+open". Does this tool intend to check the necessity of the package name and +open option?
>
> 2. How does the tool deal with optional dependencies or maybe just useless dependencies? For example, both modules A and B provide the XYZ algorithm and the test wants to ensure the one in A is the preferred one when both modules are available. Will this tool complain B is useless? How can I tell the tool this is intended?
>
> Thanks
> Max
>
>> On Mar 24, 2018, at 8:54 AM, Alexandre (Shura) Iline <alexandre.iline at oracle.com> wrote:
>>
>> Hi.
>>
>> This is an invitation to discuss requirements for a “JTReg test module dependencies verifier” (for not having a better name).
>>
>> Background: test module dependencies are declared with @modules JTReg tag and modules entry in test suite configuration files. Please consult JTReg documentation for more information.
>>
>> Problem statement: At the current state there is no effective way for a test developer to verify correctness of module declarations for a test or a group of tests. To verify that information, a test developer needs to manually limit module availability to those declared as module dependencies. This could be achieved either through VM options or by building a custom JDK image. That operation has to be repeated for every subset of tests having unique set of module dependencies. This is a tedious, error prone manual process.
>>
>> Intent of the tool is to assess correctness of declared module dependencies of JTReg tests.
>>
>> I. Functional requirements
>> 1. The tool must be able to detect tests which, for execution, depend on presence of more modules than declared.
>> 2. There must be a way to specify a subset of tests for analysis in a way that is compatible with JTReg.
>> 3. Results of the module dependencies assessment should be stored in a machine parse-able format.
>> 3. Results of the module dependencies assessment should be provided in a human readable format.
>> 4. The tool must allow to specify additional test execution parameters which influence test behavior.
>> 5. The tool may provide some analysis of test artifacts in order to identify missing module dependencies.
>> 6. The tool may provide some analysis of test artifacts in order to identify extra, not needed module dependencies.
>> 7. The tool must provide a command-line interface.
>>
>> II. Non-functional requirements
>> 1. The tool must use JTReg tool or JTReg API to perform test selection and test execution.
>>
>> III. Non-requirement
>> 1. The tool does not have to be a part of JTReg source repository.
>> 2. The tool does not have to be a part of binary JTReg distribution.
>> 3. It is not a requirement for the tool to verify other JTReg test tags or any other aspects of test suite consistency.
>>
>>
>> Shura