Tool suggestion: JTReg test module dependencies verifier.

[Alexandre (Shura) Iline]

> On Mar 28, 2018, at 10:45 AM, Jonathan Gibbons <jonathan.gibbons at oracle.com> wrote:
> 
> Shura,
> 
> I think you should start ranking your requirements, in terms of "must have", "nice to have" etc, or any similar scale.
> 
> I think that "must have" should be to detect conditions which may cause the test to fail on a restricted system. Think of them as the hard-to-detect errors that might upset a test run on a custom image.
> 
> I think that "nice to have" should be lint-like checks for conditions which will not cause the test to fail, but which are not necessary for the test to succeed.

Yes, point taken. 

When I look on the list, then, those functional requirements which are “must” belong to the “must have” group. 

Having (human|machine)-readable output is also "mush have" - perhaps this should also be saying “must”. 

And the internal package dependency detection requirement (which I think have triggered your response), should not really be “must” or belong to the “must have" group.

> 
> There's a 3rd (weaker?) category of "implementation desires", such as whether or not or how much this is integrated into jtreg.

Correct, these are the non-functional requirements, of which at least one is missing right now:
2. The tool must use jdeps tool or jdeps API to perform test selection and test execution.


I will rearrange and resend.

Shura

> 
> --
> 
> There is potentially a difference between "package not found" and "export|open not required" in a `@module some.module/some.package`
> 
> -- Jon
> 
> 
> On 3/28/18 10:34 AM, Alexandre (Shura) Iline wrote:
>> 
>>> On Mar 27, 2018, at 5:21 PM, Weijun Wang <weijun.wang at oracle.com> wrote:
>>> 
>>> Two questions:
>>> 
>>> 1. The modules tag looks like "@module module.name/package:+open". Does this tool intend to check the necessity of the package name and +open option?
>> You are correct. I will add this (or similar) to the requirements: "The tool must be capable of detecting unneeded internal package dependencies."
>> 
>> As far as [+]?open goes, however, I see no reasonable way to detect that, other than some deep instrumentation of JDK code.
>> 
>>> 2. How does the tool deal with optional dependencies or maybe just useless dependencies? For example, both modules A and B provide the XYZ algorithm and the test wants to ensure the one in A is the preferred one when both modules are available. Will this tool complain B is useless? How can I tell the tool this is intended?
>> I think we can only go part of the way, which is why I have put this item under “may”: "The tool may provide some analysis of test artifacts in order to identify extra, not needed module dependencies."
>> 
>> Indeed, any test using API from a module which use a service may, in reality, have a runtime dependency on any number of modules which provide that service. That is a very common scenario, indeed. Almost every module using java.compiler is actually requiring jdk.compiler, to give one example.
>> 
>> So, in my current implementation I do this:
>> For every non-API dependency (that is, for every module which is not detected by jdeps), I check if any of the of the API dependency modules declares a service which is provided by the module in question. If none, I report the dependency as not used. Let me use an example to explain. Assume a test has @modules jdk.compiler. I see that jdk.compiler provides java.util.spi.ToolProvider, com.sun.tools.javac.platform.PlatformProvider, javax.tools.JavaCompiler, com.sun.tools.javac.api.JavacTool. If among the API dependencies there is java.compiler (most often through java.tools.JavaCompiler), than the dependency on jdk.compiler is justified. If none of the services provided by jdk.compiler is  used by any of the API dependencies, than the module jdk.compiler is reported as not used.
>> 
>> Needless to say, this is imperfect, as
>> 1. there could be cases when no service implementation provided by the module is actually used. Such as, for example, no module which provides any service from java.base will ever be reported as not used.
>> 2. some modules reported as not used while they actually are used - I have a handful of cases like that in jdk tier1 tests.
>> 
>> I know not if a better way to detect this, other than deep instrumentation of JDK code.
>> 
>> I was saving this topic for design discussion. We seem to be having it already, which is perfectly normal.
>> 
>> Thanks.
>> 
>> Shura
>> 
>> 
>>> Thanks
>>> Max
>>> 
>>>> On Mar 24, 2018, at 8:54 AM, Alexandre (Shura) Iline <alexandre.iline at oracle.com> wrote:
>>>> 
>>>> Hi.
>>>> 
>>>> This is an invitation to discuss requirements for a “JTReg test module dependencies verifier” (for not having a better name).
>>>> 
>>>> Background: test module dependencies are declared with @modules JTReg tag and modules entry in test suite configuration files. Please consult JTReg documentation for more information.
>>>> 
>>>> Problem statement: At the current state there is no effective way for a test developer to verify correctness of module declarations for a test or a group of tests. To verify that information, a test developer needs to manually limit module availability to those declared as module dependencies. This could be achieved either through VM options or by building a custom JDK image. That operation has to be repeated for every subset of tests having unique set of module dependencies. This is a tedious, error prone manual process.
>>>> 
>>>> Intent of the tool is to assess correctness of declared module dependencies of JTReg tests.
>>>> 
>>>> I. Functional requirements
>>>> 1. The tool must be able to detect tests which, for execution, depend on presence of more modules than declared.
>>>> 2. There must be a way to specify a subset of tests for analysis in a way that is compatible with JTReg.
>>>> 3. Results of the module dependencies assessment should be stored in a machine parse-able format.
>>>> 3. Results of the module dependencies assessment should be provided in a human readable format.
>>>> 4. The tool must allow to specify additional test execution parameters which influence test behavior.
>>>> 5. The tool may provide some analysis of test artifacts in order to identify missing module dependencies.
>>>> 6. The tool may provide some analysis of test artifacts in order to identify extra, not needed module dependencies.
>>>> 7. The tool must provide a command-line interface.
>>>> 
>>>> II. Non-functional requirements
>>>> 1. The tool must use JTReg tool or JTReg API to perform test selection and test execution.
>>>> 
>>>> III. Non-requirement
>>>> 1. The tool does not have to be a part of JTReg source repository.
>>>> 2. The tool does not have to be a part of binary JTReg distribution.
>>>> 3. It is not a requirement for the tool to verify other JTReg test tags or any other aspects of test suite consistency.
>>>> 
>>>> 
>>>> Shura
>