jtreg shell tests

Sat Feb 2 01:16:59 UTC 2019

Sergey's suggestion to force the elimination of shell tests may not be 
far off the mark. :-)

It is getting insanely complicated to support maintain these shell 
tests.  Not impossible, but very, very fragile.

That being said, I have a small demo test suite that mimics the options 
1-4 that I enumerated before, and I have an updated version of jtreg to 
run these tests.

This version of jtreg supports the following:

  * bin/jtreg has been updated to work correctly on WSL

  * new option -cygwin, as well as recent new -wsl option, to force the
    use of Cygwin, but neither option is necessary in normal use: jtreg
    will autodetect whether to use Cygwin or WSL (or MKS, untested)

  * The following combinations are supported on Windows:

      * Cygwin, jtreg running on a Windows JDK, test JDK is a Windows JDK.
      * WSL, jtreg running on a Windows JDK, test JDK is a Windows JDK.
      * WSL, jtreg running on a Linux JDK, test JDK is a Linux JDK.

  * jtreg will /not/ support mixed JDKs ... running on one kind of JDK,
    and the test JDK is the other kind.

Part of the reason this is absurdly fragile is because of the need to 
use "wslpath" to convert command line args, which (I'm presuming) we 
don't want to put in our shell scripts.  WSLEnv only does some of the 
work; it only converts environment variables; it does nothing to help 
fix up command lines. Therefore the only way to get shell scripts to 
work is for jtreg to "guess" whether an environment variable is going to 
be used by WSL (e.g. a path to invoke the test JDK) or whether the 
environment variable is going to be used by a Windows binary invoked 
from WSL (e.g. a value for a classpath option.)

It is worth noting that we don't have this problem on Cygwin, because 
Cygwin will tolerate the use of Windows-style paths, and so 
consistent-looking environment variables can be used within the script 
and passed to programs.  That's not true in the WSL world at this point.

I need to do more testing on the new version of jtreg; I'll push changes 
next week.

-- Jon

On 01/26/2019 08:40 AM, Jonathan Gibbons wrote:
> I don't think we should force the elimination of shell tests, but we 
> should definitely encourage it. (i.e. Option 1.)
>
> The underlying theme in options 1-4 is to minimize the effort required 
> to accommodate the changes needed to support the use of WSL to run 
> tests; not to impose effort. Carrots, not sticks.
>
> Of course, Sergey, if you'd like to volunteer to convert all the 
> client tests, that would be your prerogative. Just yesterday, I had a 
> chat with Phil, giving anecdotes about how we converted almost all of 
> the langtools shell tests, by writing a library that provided methods 
> based on the shell commands that we saw in our shell scripts: cp, mv, 
> rm, diff, grep, etc. Others have done the same for some of the 
> core-libs tests. We can start a separate thread if you'd like to 
> discuss such techniques.
>
> -- Jon
>
>
> On 1/25/19 5:16 PM, Sergey Bylokhov wrote:
>> No my point was radical drop of all such tests.
>>
>> On 25/01/2019 17:05, Andrew Luo wrote:
>>> Isn't that option 1?
>>>
>>> Thanks,
>>>
>>> -Andrew
>>>
>>> -----Original Message-----
>>> From: quality-discuss <quality-discuss-bounces at openjdk.java.net> On 
>>> Behalf Of Sergey Bylokhov
>>> Sent: Friday, January 25, 2019 5:00 PM
>>> To: Jonathan Gibbons <jonathan.gibbons at oracle.com>; 
>>> quality-discuss at openjdk.java.net
>>> Subject: Re: jtreg shell tests
>>>
>>> There is one more Option 5.
>>>
>>> Drop shell tests from the workspace and provide some examples on how 
>>> to write such logic using java api.
>>>
>>> On 25/01/2019 16:43, Jonathan Gibbons wrote:
>>>> With all the recent discussion regarding how to support the use of
>>>> Windows Subsystem for Linux (WSL) as an alternate to Cygwin, it 
>>>> seems worth writing up some recommendations on writing jtreg shell 
>>>> tests.
>>>> The intent of these notes is that they will evolve into a page in 
>>>> the jtreg section on the OpenJDK website.
>>>>
>>>> The focus is specifically about different approaches to providing the
>>>> ability to run a shell test on all supported platforms, by means of
>>>> abstracting the significant differences into a series of 
>>>> environment variables that are set according to the environment in 
>>>> which the test is running.
>>>>
>>>> Option 1.
>>>>
>>>> Convert the test to Java. In general, this continues to be the 
>>>> recommended alternative.
>>>>
>>>>
>>>> Option 2.
>>>>
>>>> Use a shell `case` statement, like the following, or a variant 
>>>> thereof:
>>>>
>>>>      OS=`uname -s`;
>>>>      case "$OS" in
>>>>           Windows* | CYGWIN* )
>>>>               FS="\\"
>>>>               PS=";"
>>>>               NULL=NUL
>>>>               ;;
>>>>
>>>>           Linux )
>>>>               if [ -r $TESTJAVA/bin/java.exe ]; then
>>>>                   FS="\\"
>>>>                   PS=";"
>>>>                   EXE_SUFFIX=".exe"
>>>>               else
>>>>                   FS="/"
>>>>                   PS=":"
>>>>               fi
>>>>               NULL=/dev/null
>>>>               ;;
>>>>
>>>>           * )
>>>>               FS="/"
>>>>               PS=":"
>>>>               NULL=/dev/null
>>>>      esac
>>>>
>>>> Option 3.
>>>>
>>>> Use a shared library script to embody the behavior in the previous 
>>>> example.  jtreg now provides a new `TESTROOT` environment variable, 
>>>> which makes it easy to reference a shared script in a constant 
>>>> manner from any shell test, wherever the test is within the test 
>>>> suite. Since the library script is used to set environment 
>>>> variables like `FS`, `PS`, and `NULL`, it should be executed with 
>>>> `source` and not `bash` or `sh`.
>>>>
>>>>
>>>> Option 4.
>>>>
>>>> jtreg now sets the following environment variables when running a 
>>>> shell script: `FS`, `PS`, `NULL` and `EXE_SUFFIX`. This may be 
>>>> enough to completely avoid the need for a `case` statement in each 
>>>> shell script or the use of a shared library script to set these 
>>>> variables.
>>>>
>>>>
>>>> Running scripts standalone.
>>>>
>>>> One concern when working with shell tests has been the ability to 
>>>> run the test "stand-alone", without the use of jtreg. In the past, 
>>>> this was seen as justification for the explicit use of the `case` 
>>>> statement in each shell test. However, the need to run shell tests 
>>>> standalone no longer seems to be a significant concern. For those 
>>>> that do want to run shell tests by themselves, it is worth noting 
>>>> that once a test has been run by jtreg, the ".jtr" file contains 
>>>> "rerun" sections with details on how to run each action of the 
>>>> test. You can either copy/paste/edit from the ".jtr" file directly, 
>>>> or use the jtreg `-show:rerun` option to output the information to 
>>>> the standard output stream.
>>>>
>>>>
>>>>
>>>
>>>
>>> -- 
>>> Best regards, Sergey.
>>>
>>
>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.java.net/pipermail/quality-discuss/attachments/20190201/6fc5c0bc/attachment.html>