RFC: add starred module(s)

[Jonathan Gibbons]

Jiri,

I have read all your comments. I don't think it helps to respond on a 
line by line basis, and so instead, I will say this ...

I am sympathetic to the problem of the complexity of configuring 
individual tests, and it will come as no surprise to anyone that we have 
encountered some of those same problems as well, writing tests for JDK.  
But, I want to separate the issue of determining the appropriate content 
for @modules from the syntactic presentation of that content.

In the discussions we've had regarding the addition of @modules for all 
the existing JDK tests, we have focused more on providing and using 
tools to analyse a test and suggest possible entries for @modules. This 
analysis is more than can reasonably be done on the fly by jtreg, 
involving tools like jdeprscan, and running tests in minimal JVMs with 
just the listed modules.   For example, as well as verifying that we 
have enough entries in @modules, we also want to make sure we don't have 
any unnecessary entries, which might preclude running the test on a 
smaller JVM (because of an advertised dependence on module that is not 
actually required for the test to execute successfully.)

In addition, we also want to write negative tests, such as being able to 
test the behavior of the system when modules A, B and C are present, but 
not modules D, E, and F ... even if there are references to those 
modules in the code.

The bottom line is that I do not think it is appropriate to much too 
much smarts into jtreg to infer more than it is already doing. It's 
already close to being too complicated, which is why we added the 
"Configuration" section in the output (.jtr) file, to help users verify 
the setup being provided by jtreg.  But maybe we can address the general 
need by focussing on providing better lint-like tools for validating the 
module setup for a test.

And, if we have tools like that, then the presentation issues, such as 
wildcard entries in test descriptions, becomes less important.

-- Jon

On 12/14/2017 07:56 AM, Jiri Vanek wrote:
> Hi Jonathan!
>
> Thank you very much for great answer. Few remarks inline.
>
>
> On 12/09/2017 02:51 AM, Jonathan Gibbons wrote:
>> Jiri, Mila,
>>
>> There are three significant questions here.
>>
>> 1. @modules all-needed
>>
>> This is a non-starter. @modules participates in a relatively 
>> light-weight test selection mechanism 
>
> Sure. It is last step, if it will be ever started.
>
>> ... i.e. tests are not selected if the specified modules are not 
>> available in the system image. Having jtreg analyse the source code 
>> to determine the set of necessary modules is just not practical.
>>
>> 2. Starred modules, as in @modules 
>> java.security.jgss/sun.security.krb5.internal.*
>>
>> Setting aside whether this is desirable or not, this is not 
>> necessarily an easy thing to do. It requires that we can list the 
>> (internal) packages of a module. Yes, there may be a attribute that 
>> provides the info, but the attribute is not guaranteed to be 
>> available, and if it's not, you're 
> terrible truth. see at bottom.
>
>> reduced to scanning the module contents. On the desirability point of 
>> view, the use of this sort of syntax was discussed, and rejected, for 
>> module declarations, and while that is separate from what is 
>
> I was reading most of this discussion, and I can agree with it for 
> final deployment, but not for testing.
>> proposed here, it does lend guidance. Neither is the '*' syntax 
>> supported by the options for the Java launcher or javac, and this 
>> construct is intended to be a thinly-veiled wrapper for those options.
>
> This is probably the only sentence I moreover disagree with you. Jtreg 
> is far away fro being tinny wrapper, and it always  did huge help for 
> tests authors.  So helping testers with setting up module path a bit, 
> may be good step.
>>
>> One minor comment ... you can reduce the "wordiness" of your examples 
>> by almost 50% by just having a single @modules directive with many 
>> arg values, possibly spanning many lines, as in ...
>>
>>    * ....
>>    * @modules java.security.jgss/sun.security.krb5.internal
>>    * java.security.jgss/sun.security.krb5.internal.ccache
>>    * java.security.jgss/sun.security.krb5.internal.crypto
>>    * java.security.jgss/sun.security.krb5.internal.ktab
>>    * java.base/sun.security.util
>>    * java.security.jgss/sun.security.jgss
>
> Still you need to enumerate them all. That mean to run javac and jmod 
> in loops  one by one, untill you get collect them.
>>
>> 3. Shell scripts.
>>
>> Using shell scripts for jtreg tests is generally discouraged, these 
>> days, to the point that we have made efforts to convert many/most JDK 
>> shell tests to Java code. The general experience was that such tests 
>> were very fragile, and were often wrong/broken. They typically 
>> execute slower than equivalent Java code, especially when using 
>> libraries providing commonly-used or domain-specific functionality. 
>
> Well, shell lunchers have theirs dark sides, thats right. But are 
> awesome feature without replacement.
>> And as you note below, processing variables like TESTMODULES is 
>> problematic. (It is a bug if present for JDK 8 tests).
>
> Its a bug :)
>>
>> When using Java code instead of shell scripts, you can look to 
>> libraries in the various test suites for suggestions on useful 
>> methods to provide, such as invoking generating and compiling source 
>> code, providing simple equivalents for regex search (simple grep), 
>> comparison (simple diff), etc, as well as support for running 
>> subprocesses when necessary. One common pattern is to use Java code 
>> to validate whether it is appropriate to run a test in the current 
>> configuration and to then launch a JVM with a selected combination of 
>> options. This sort of mode has explicit support in jtreg:  "@run 
>> driver" is similar to "@run main" for running a class, but it is not 
>> run with any JVM options intended for test JVMs. This makes it 
>> suitable for running "test driver" classes which spawn JVMs to run 
>> the "real" test code.  And if you're just writing API tests, then I 
>> recommend using plain old Java code and "agentvm" mode to have the 
>> tests as fast as possible.
>
>
> You are mostly right. But consider this: The locating of modules is 
> painful when you are dealing with private apis. And that is what you 
> do in jtregs for half of the time...
> So naturally, the help would be easy.
> On contrary, the help do not come from JDK itself in any "supported" 
> way (or am I missing something?)
> Still the javac can do so. So it would be natural to enahnce javac to 
> help with this a bit more via some more solid api. Maybe you know 
> something here?
>
> One would say, that writing private api jtreg is one or two modules 
> usage only. Unluckily, oposite is true, and Mila's * idea is another 
> proof of it. And If I look to past to my recent work, I was facing the 
> same. Just got angry later.
>
>
> So generally - some help should be provided in JDK to deal with it. 
> And as JDK looks to have its reasons to not to do so, Jtregs should.
>
>
> Thanx for again for your reply!
>   J.
>
>
>
>
>
>
>