Tool suggestion: JTReg test module dependencies verifier.

[Jonathan Gibbons]

Shura,

I think you should start ranking your requirements, in terms of "must 
have", "nice to have" etc, or any similar scale.

I think that "must have" should be to detect conditions which may cause 
the test to fail on a restricted system. Think of them as the 
hard-to-detect errors that might upset a test run on a custom image.

I think that "nice to have" should be lint-like checks for conditions 
which will not cause the test to fail, but which are not necessary for 
the test to succeed.

There's a 3rd (weaker?) category of "implementation desires", such as 
whether or not or how much this is integrated into jtreg.

--

There is potentially a difference between "package not found" and 
"export|open not required" in a `@module some.module/some.package`

-- Jon


On 3/28/18 10:34 AM, Alexandre (Shura) Iline wrote:
>
>> On Mar 27, 2018, at 5:21 PM, Weijun Wang <weijun.wang at oracle.com> wrote:
>>
>> Two questions:
>>
>> 1. The modules tag looks like "@module module.name/package:+open". Does this tool intend to check the necessity of the package name and +open option?
> You are correct. I will add this (or similar) to the requirements: "The tool must be capable of detecting unneeded internal package dependencies."
>
> As far as [+]?open goes, however, I see no reasonable way to detect that, other than some deep instrumentation of JDK code.
>
>> 2. How does the tool deal with optional dependencies or maybe just useless dependencies? For example, both modules A and B provide the XYZ algorithm and the test wants to ensure the one in A is the preferred one when both modules are available. Will this tool complain B is useless? How can I tell the tool this is intended?
> I think we can only go part of the way, which is why I have put this item under “may”: "The tool may provide some analysis of test artifacts in order to identify extra, not needed module dependencies."
>
> Indeed, any test using API from a module which use a service may, in reality, have a runtime dependency on any number of modules which provide that service. That is a very common scenario, indeed. Almost every module using java.compiler is actually requiring jdk.compiler, to give one example.
>
> So, in my current implementation I do this:
> For every non-API dependency (that is, for every module which is not detected by jdeps), I check if any of the of the API dependency modules declares a service which is provided by the module in question. If none, I report the dependency as not used. Let me use an example to explain. Assume a test has @modules jdk.compiler. I see that jdk.compiler provides java.util.spi.ToolProvider, com.sun.tools.javac.platform.PlatformProvider, javax.tools.JavaCompiler, com.sun.tools.javac.api.JavacTool. If among the API dependencies there is java.compiler (most often through java.tools.JavaCompiler), than the dependency on jdk.compiler is justified. If none of the services provided by jdk.compiler is  used by any of the API dependencies, than the module jdk.compiler is reported as not used.
>
> Needless to say, this is imperfect, as
> 1. there could be cases when no service implementation provided by the module is actually used. Such as, for example, no module which provides any service from java.base will ever be reported as not used.
> 2. some modules reported as not used while they actually are used - I have a handful of cases like that in jdk tier1 tests.
>
> I know not if a better way to detect this, other than deep instrumentation of JDK code.
>
> I was saving this topic for design discussion. We seem to be having it already, which is perfectly normal.
>
> Thanks.
>
> Shura
>
>
>> Thanks
>> Max
>>
>>> On Mar 24, 2018, at 8:54 AM, Alexandre (Shura) Iline <alexandre.iline at oracle.com> wrote:
>>>
>>> Hi.
>>>
>>> This is an invitation to discuss requirements for a “JTReg test module dependencies verifier” (for not having a better name).
>>>
>>> Background: test module dependencies are declared with @modules JTReg tag and modules entry in test suite configuration files. Please consult JTReg documentation for more information.
>>>
>>> Problem statement: At the current state there is no effective way for a test developer to verify correctness of module declarations for a test or a group of tests. To verify that information, a test developer needs to manually limit module availability to those declared as module dependencies. This could be achieved either through VM options or by building a custom JDK image. That operation has to be repeated for every subset of tests having unique set of module dependencies. This is a tedious, error prone manual process.
>>>
>>> Intent of the tool is to assess correctness of declared module dependencies of JTReg tests.
>>>
>>> I. Functional requirements
>>> 1. The tool must be able to detect tests which, for execution, depend on presence of more modules than declared.
>>> 2. There must be a way to specify a subset of tests for analysis in a way that is compatible with JTReg.
>>> 3. Results of the module dependencies assessment should be stored in a machine parse-able format.
>>> 3. Results of the module dependencies assessment should be provided in a human readable format.
>>> 4. The tool must allow to specify additional test execution parameters which influence test behavior.
>>> 5. The tool may provide some analysis of test artifacts in order to identify missing module dependencies.
>>> 6. The tool may provide some analysis of test artifacts in order to identify extra, not needed module dependencies.
>>> 7. The tool must provide a command-line interface.
>>>
>>> II. Non-functional requirements
>>> 1. The tool must use JTReg tool or JTReg API to perform test selection and test execution.
>>>
>>> III. Non-requirement
>>> 1. The tool does not have to be a part of JTReg source repository.
>>> 2. The tool does not have to be a part of binary JTReg distribution.
>>> 3. It is not a requirement for the tool to verify other JTReg test tags or any other aspects of test suite consistency.
>>>
>>>
>>> Shura