From andrewluotechnologies at outlook.com Sun Jan 6 19:29:44 2019 From: andrewluotechnologies at outlook.com (Andrew Luo) Date: Sun, 6 Jan 2019 19:29:44 +0000 Subject: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) Message-ID: We recently added support for building OpenJDK on WSL (targeting Windows; targeting Linux always worked). However, running tests on WSL did not work, and we determined that some changes would be needed to jtreg to support this use case. In particular, jtreg needs to know if it is running on WSL targeting Windows or WSL targeting Linux (when targeting Linux, no changes are needed as it is the same as any other Linux system, but when targeting Windows jtreg needs to run shell scripts in a special manner - in particular, calling "wsl.exe sh" instead of just "sh" since it's a Windows boot JDK, as well as translating environment variables using WSLENV...) I've attached my patch with my proposed changes. I welcome any thoughts/feedback. (See https://bugs.openjdk.java.net/browse/JDK-8215445 for more details about supporting WSL for building OpenJDK) Thanks, -Andrew -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: diff.txt URL: From andrewluotechnologies at outlook.com Mon Jan 7 05:02:48 2019 From: andrewluotechnologies at outlook.com (Andrew Luo) Date: Mon, 7 Jan 2019 05:02:48 +0000 Subject: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) In-Reply-To: References: Message-ID: Adding jtreg-dev. Didn't realize that there was such a mailing list until later (the jtreg code lives under code-tools repository) Thanks, -Andrew -----Original Message----- From: code-tools-dev On Behalf Of Andrew Luo Sent: Sunday, January 6, 2019 11:30 AM To: code-tools-dev at openjdk.java.net Subject: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) We recently added support for building OpenJDK on WSL (targeting Windows; targeting Linux always worked). However, running tests on WSL did not work, and we determined that some changes would be needed to jtreg to support this use case. In particular, jtreg needs to know if it is running on WSL targeting Windows or WSL targeting Linux (when targeting Linux, no changes are needed as it is the same as any other Linux system, but when targeting Windows jtreg needs to run shell scripts in a special manner - in particular, calling "wsl.exe sh" instead of just "sh" since it's a Windows boot JDK, as well as translating environment variables using WSLENV...) I've attached my patch with my proposed changes. I welcome any thoughts/feedback. (See https://bugs.openjdk.java.net/browse/JDK-8215445 for more details about supporting WSL for building OpenJDK) Thanks, -Andrew -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: diff.txt URL: From jonathan.gibbons at oracle.com Mon Jan 7 15:27:26 2019 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Mon, 7 Jan 2019 07:27:26 -0800 Subject: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) In-Reply-To: References: Message-ID: Andrew, I'll take a look at this in detail later today. -- Jon On 1/6/19 9:02 PM, Andrew Luo wrote: > Adding jtreg-dev. Didn't realize that there was such a mailing list until later (the jtreg code lives under code-tools repository) > > Thanks, > > -Andrew > > -----Original Message----- > From: code-tools-dev On Behalf Of Andrew Luo > Sent: Sunday, January 6, 2019 11:30 AM > To: code-tools-dev at openjdk.java.net > Subject: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) > > We recently added support for building OpenJDK on WSL (targeting Windows; targeting Linux always worked). However, running tests on WSL did not work, and we determined that some changes would be needed to jtreg to support this use case. In particular, jtreg needs to know if it is running on WSL targeting Windows or WSL targeting Linux (when targeting Linux, no changes are needed as it is the same as any other Linux system, but when targeting Windows jtreg needs to run shell scripts in a special manner - in particular, calling "wsl.exe sh" instead of just "sh" since it's a Windows boot JDK, as well as translating environment variables using WSLENV...) > > I've attached my patch with my proposed changes. I welcome any thoughts/feedback. > > (See https://bugs.openjdk.java.net/browse/JDK-8215445 for more details about supporting WSL for building OpenJDK) > > Thanks, > > -Andrew From jonathan.gibbons at oracle.com Mon Jan 7 15:53:47 2019 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Mon, 7 Jan 2019 07:53:47 -0800 Subject: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) In-Reply-To: References: Message-ID: <658a4caf-0400-e31b-d3b1-79e7f00a2c06@oracle.com> Andrew, I've been generally following the progress for supporting WSL, but now that I need to get more involved, (i.e. reviewing jtreg updates) can you point me at any reasonable general introductory material for WSL? -- Jon On 1/6/19 11:29 AM, Andrew Luo wrote: > We recently added support for building OpenJDK on WSL (targeting Windows; targeting Linux always worked). However, running tests on WSL did not work, and we determined that some changes would be needed to jtreg to support this use case. In particular, jtreg needs to know if it is running on WSL targeting Windows or WSL targeting Linux (when targeting Linux, no changes are needed as it is the same as any other Linux system, but when targeting Windows jtreg needs to run shell scripts in a special manner - in particular, calling "wsl.exe sh" instead of just "sh" since it's a Windows boot JDK, as well as translating environment variables using WSLENV...) > > I've attached my patch with my proposed changes. I welcome any thoughts/feedback. > > (See https://bugs.openjdk.java.net/browse/JDK-8215445 for more details about supporting WSL for building OpenJDK) > > Thanks, > > -Andrew From andrewluotechnologies at outlook.com Mon Jan 7 20:36:37 2019 From: andrewluotechnologies at outlook.com (Andrew Luo) Date: Mon, 7 Jan 2019 20:36:37 +0000 Subject: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) In-Reply-To: <658a4caf-0400-e31b-d3b1-79e7f00a2c06@oracle.com> References: <658a4caf-0400-e31b-d3b1-79e7f00a2c06@oracle.com> Message-ID: Hi Jon, For a general intro to WSL, see here: https://blogs.msdn.microsoft.com/wsl/2016/04/22/windows-subsystem-for-linux-overview/ https://docs.microsoft.com/en-us/windows/wsl/faq These links might also be useful for reviewing these changes specifically: https://docs.microsoft.com/en-us/windows/wsl/interop https://blogs.msdn.microsoft.com/commandline/2017/12/22/share-environment-vars-between-wsl-and-windows/ Thanks, -Andrew -----Original Message----- From: Jonathan Gibbons Sent: Monday, January 7, 2019 7:54 AM To: Andrew Luo ; code-tools-dev at openjdk.java.net Subject: Re: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) Andrew, I've been generally following the progress for supporting WSL, but now that I need to get more involved, (i.e. reviewing jtreg updates) can you point me at any reasonable general introductory material for WSL? -- Jon On 1/6/19 11:29 AM, Andrew Luo wrote: > We recently added support for building OpenJDK on WSL (targeting > Windows; targeting Linux always worked). However, running tests on > WSL did not work, and we determined that some changes would be needed > to jtreg to support this use case. In particular, jtreg needs to know > if it is running on WSL targeting Windows or WSL targeting Linux (when > targeting Linux, no changes are needed as it is the same as any other > Linux system, but when targeting Windows jtreg needs to run shell > scripts in a special manner - in particular, calling "wsl.exe sh" > instead of just "sh" since it's a Windows boot JDK, as well as > translating environment variables using WSLENV...) > > I've attached my patch with my proposed changes. I welcome any thoughts/feedback. > > (See https://bugs.openjdk.java.net/browse/JDK-8215445 for more details > about supporting WSL for building OpenJDK) > > Thanks, > > -Andrew From jonathan.gibbons at oracle.com Mon Jan 7 20:41:14 2019 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Mon, 7 Jan 2019 12:41:14 -0800 Subject: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) In-Reply-To: References: <658a4caf-0400-e31b-d3b1-79e7f00a2c06@oracle.com> Message-ID: Andrew, Thank you for the references. -- Jon On 01/07/2019 12:36 PM, Andrew Luo wrote: > Hi Jon, > > For a general intro to WSL, see here: > > https://blogs.msdn.microsoft.com/wsl/2016/04/22/windows-subsystem-for-linux-overview/ > https://docs.microsoft.com/en-us/windows/wsl/faq > > These links might also be useful for reviewing these changes specifically: > > https://docs.microsoft.com/en-us/windows/wsl/interop > https://blogs.msdn.microsoft.com/commandline/2017/12/22/share-environment-vars-between-wsl-and-windows/ > > Thanks, > > -Andrew > > -----Original Message----- > From: Jonathan Gibbons > Sent: Monday, January 7, 2019 7:54 AM > To: Andrew Luo ; code-tools-dev at openjdk.java.net > Subject: Re: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) > > Andrew, > > I've been generally following the progress for supporting WSL, but now that I need to get more involved, (i.e. reviewing jtreg updates) can you point me at any reasonable general introductory material for WSL? > > -- Jon > > On 1/6/19 11:29 AM, Andrew Luo wrote: >> We recently added support for building OpenJDK on WSL (targeting >> Windows; targeting Linux always worked). However, running tests on >> WSL did not work, and we determined that some changes would be needed >> to jtreg to support this use case. In particular, jtreg needs to know >> if it is running on WSL targeting Windows or WSL targeting Linux (when >> targeting Linux, no changes are needed as it is the same as any other >> Linux system, but when targeting Windows jtreg needs to run shell >> scripts in a special manner - in particular, calling "wsl.exe sh" >> instead of just "sh" since it's a Windows boot JDK, as well as >> translating environment variables using WSLENV...) >> >> I've attached my patch with my proposed changes. I welcome any thoughts/feedback. >> >> (See https://bugs.openjdk.java.net/browse/JDK-8215445 for more details >> about supporting WSL for building OpenJDK) >> >> Thanks, >> >> -Andrew From jonathan.gibbons at oracle.com Mon Jan 7 22:20:27 2019 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Mon, 7 Jan 2019 14:20:27 -0800 Subject: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) In-Reply-To: References: <658a4caf-0400-e31b-d3b1-79e7f00a2c06@oracle.com> Message-ID: <71ed2368-0051-8243-ced9-27598d6c7368@oracle.com> Andrew, There are some initial problems with your patch. 1.? In ShellAction.joinWSLEnv ... it is still a self-imposed requirement that we can build jtreg with JDK 7 (to help support testing older releases of JDK.) In joinWSLEnv, you use String.join, which is @since 8, and so not available on 7.?? For now, I've written a replacement, but in doing do. I wonder if there is a bug in your original.? In particular, at the end of the method, why do you add `suffix` if `result.isEmpty())`.? I would have expected this to be a test if the result was _not_ empty. Anyway, here is my proposed replacement for the method: private String joinWSLENV(List list, String suffix) { StringBuilder sb =new StringBuilder(); String sep =""; for (String item : list) { if (item.trim().isEmpty() || item.toLowerCase().equals("path")) { continue; } sb.append(sep).append(item).append(suffix); sep =":"; } return sb.toString(); } The code appends the ":" separator after the first item, and always appends the suffix.?? Is that what you believe it should do? 2. You added a new option, but did not add an entry into the resource file. This causes the jtreg build to fail one of its self-tests.? I've created a placeholder entry, but it is at the semantic level I hate: "getFo(): gets a foo".? If you can provide a more informative string, that would be better. $ hg diff src/share/classes/com/sun/javatest/regtest/tool/i18n.properties diff -r b6fabf6bf942 src/share/classes/com/sun/javatest/regtest/tool/i18n.properties --- a/src/share/classes/com/sun/javatest/regtest/tool/i18n.properties Thu Dec 20 14:43:41 2018 -0800 +++ b/src/share/classes/com/sun/javatest/regtest/tool/i18n.properties Mon Jan 07 13:38:40 2019 -0800 @@ -257,6 +257,7 @@ ???? The name may contain '*' to match any sequence of characters. \ ???? For example, result.* or *.err. ?help.main.retain.arg=,... +help.main.wslWindowsTarget.desc=(Windows only) Run with WSL targeting Windows ?help.main.dir.desc=Specify a base directory for test files and directories ?help.main.dir.arg= ?help.main.allowSetSecurityManager.desc=Allow agentVM tests to set a security manager. \ 3. README. It's not OK to change the baseline JDK used to build jtreg.? If there is a WSL-reason why JDK 8 is required to run jtreg in a WSL environment, that should be done as a runtime check. Beyond that, the changes to ShellAction feel a bit clunky, changing the non-WSL flow of the code more than might be thought necessary. After all, there's really only two changes here: inserting wsl.exe at the begtinning of the command, and computing/inserting WSLENV into the environment. Given that WSLENV only contains the names of the env variables, with suffixes as needed, I would have thought that a simpler impl would be to scan the keys in env to build WSLENV, using strings-in-switch to identify the suffixes, and do that all in a relatively simple helper method, without having to affect the mainline code to put values into the env. I've created CODETOOLS-7902357 to track this work. https://bugs.openjdk.java.net/browse/CODETOOLS-7902357 I've put a webrev containing your patch at http://cr.openjdk.java.net/~jjg/7902357/webrev.00/ I've put a followup containing your patch + fixes for 1-2-3 above at http://cr.openjdk.java.net/~jjg/7902357/webrev.01/ -- Jon On 01/07/2019 12:36 PM, Andrew Luo wrote: > Hi Jon, > > For a general intro to WSL, see here: > > https://blogs.msdn.microsoft.com/wsl/2016/04/22/windows-subsystem-for-linux-overview/ > https://docs.microsoft.com/en-us/windows/wsl/faq > > These links might also be useful for reviewing these changes specifically: > > https://docs.microsoft.com/en-us/windows/wsl/interop > https://blogs.msdn.microsoft.com/commandline/2017/12/22/share-environment-vars-between-wsl-and-windows/ > > Thanks, > > -Andrew > > -----Original Message----- > From: Jonathan Gibbons > Sent: Monday, January 7, 2019 7:54 AM > To: Andrew Luo ; code-tools-dev at openjdk.java.net > Subject: Re: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) > > Andrew, > > I've been generally following the progress for supporting WSL, but now that I need to get more involved, (i.e. reviewing jtreg updates) can you point me at any reasonable general introductory material for WSL? > > -- Jon > > On 1/6/19 11:29 AM, Andrew Luo wrote: >> We recently added support for building OpenJDK on WSL (targeting >> Windows; targeting Linux always worked). However, running tests on >> WSL did not work, and we determined that some changes would be needed >> to jtreg to support this use case. In particular, jtreg needs to know >> if it is running on WSL targeting Windows or WSL targeting Linux (when >> targeting Linux, no changes are needed as it is the same as any other >> Linux system, but when targeting Windows jtreg needs to run shell >> scripts in a special manner - in particular, calling "wsl.exe sh" >> instead of just "sh" since it's a Windows boot JDK, as well as >> translating environment variables using WSLENV...) >> >> I've attached my patch with my proposed changes. I welcome any thoughts/feedback. >> >> (See https://bugs.openjdk.java.net/browse/JDK-8215445 for more details >> about supporting WSL for building OpenJDK) >> >> Thanks, >> >> -Andrew From andrewluotechnologies at outlook.com Mon Jan 7 23:23:40 2019 From: andrewluotechnologies at outlook.com (Andrew Luo) Date: Mon, 7 Jan 2019 23:23:40 +0000 Subject: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) In-Reply-To: <71ed2368-0051-8243-ced9-27598d6c7368@oracle.com> References: <658a4caf-0400-e31b-d3b1-79e7f00a2c06@oracle.com> <71ed2368-0051-8243-ced9-27598d6c7368@oracle.com> Message-ID: Hi Jon, Thanks for reviewing this. Yes, there is a bug there, it should be !result.isEmpty() at the end of joinWSLENV. Is it still the case that we support JDK 7? I got the impression from build-all.sh:48 that JDK 7 is no longer supported: if case ${JAVA_VERSION} in 1.8*) false ;; *) true; esac; then echo "Error: Expected a path to JDK with version 1.8, got version ${JAVA_VERSION}" >&2 exit 1 fi Or perhaps is this only the build-all.sh script that doesn?t support JDK 7, build jtreg itself does? As for the changes in ShellAction, the reason I did it that way instead of scanning through env, is because some environment variables have to be translated raw, others are paths, and others are path lists (multiple paths separated by ; in Windows). These all have to be added into WSLENV differently (in particular with the /p or /l suffix). Scanning through would be a bit error prone, as you?d have to decide if the variable is a raw string, path, or path list just using the value. Although it?s possible, it seems a bit error prone/clunky. If you have some preferred solution to this though, I?d be happy to make those changes. Thanks, -Andrew From: Jonathan Gibbons Sent: Monday, January 7, 2019 2:20 PM To: Andrew Luo ; code-tools-dev at openjdk.java.net Subject: Re: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) Andrew, There are some initial problems with your patch. 1. In ShellAction.joinWSLEnv ... it is still a self-imposed requirement that we can build jtreg with JDK 7 (to help support testing older releases of JDK.) In joinWSLEnv, you use String.join, which is @since 8, and so not available on 7. For now, I've written a replacement, but in doing do. I wonder if there is a bug in your original. In particular, at the end of the method, why do you add `suffix` if `result.isEmpty())`. I would have expected this to be a test if the result was _not_ empty. Anyway, here is my proposed replacement for the method: private String joinWSLENV(List list, String suffix) { StringBuilder sb = new StringBuilder(); String sep = ""; for (String item : list) { if (item.trim().isEmpty() || item.toLowerCase().equals("path")) { continue; } sb.append(sep).append(item).append(suffix); sep = ":"; } return sb.toString(); } The code appends the ":" separator after the first item, and always appends the suffix. Is that what you believe it should do? 2. You added a new option, but did not add an entry into the resource file. This causes the jtreg build to fail one of its self-tests. I've created a placeholder entry, but it is at the semantic level I hate: "getFo(): gets a foo". If you can provide a more informative string, that would be better. $ hg diff src/share/classes/com/sun/javatest/regtest/tool/i18n.properties diff -r b6fabf6bf942 src/share/classes/com/sun/javatest/regtest/tool/i18n.properties --- a/src/share/classes/com/sun/javatest/regtest/tool/i18n.properties Thu Dec 20 14:43:41 2018 -0800 +++ b/src/share/classes/com/sun/javatest/regtest/tool/i18n.properties Mon Jan 07 13:38:40 2019 -0800 @@ -257,6 +257,7 @@ The name may contain '*' to match any sequence of characters. \ For example, result.* or *.err. help.main.retain.arg=,... +help.main.wslWindowsTarget.desc=(Windows only) Run with WSL targeting Windows help.main.dir.desc=Specify a base directory for test files and directories help.main.dir.arg= help.main.allowSetSecurityManager.desc=Allow agentVM tests to set a security manager. \ 3. README. It's not OK to change the baseline JDK used to build jtreg. If there is a WSL-reason why JDK 8 is required to run jtreg in a WSL environment, that should be done as a runtime check. Beyond that, the changes to ShellAction feel a bit clunky, changing the non-WSL flow of the code more than might be thought necessary. After all, there's really only two changes here: inserting wsl.exe at the begtinning of the command, and computing/inserting WSLENV into the environment. Given that WSLENV only contains the names of the env variables, with suffixes as needed, I would have thought that a simpler impl would be to scan the keys in env to build WSLENV, using strings-in-switch to identify the suffixes, and do that all in a relatively simple helper method, without having to affect the mainline code to put values into the env. I've created CODETOOLS-7902357 to track this work. https://bugs.openjdk.java.net/browse/CODETOOLS-7902357 I've put a webrev containing your patch at http://cr.openjdk.java.net/~jjg/7902357/webrev.00/ I've put a followup containing your patch + fixes for 1-2-3 above at http://cr.openjdk.java.net/~jjg/7902357/webrev.01/ -- Jon On 01/07/2019 12:36 PM, Andrew Luo wrote: Hi Jon, For a general intro to WSL, see here: https://blogs.msdn.microsoft.com/wsl/2016/04/22/windows-subsystem-for-linux-overview/ https://docs.microsoft.com/en-us/windows/wsl/faq These links might also be useful for reviewing these changes specifically: https://docs.microsoft.com/en-us/windows/wsl/interop https://blogs.msdn.microsoft.com/commandline/2017/12/22/share-environment-vars-between-wsl-and-windows/ Thanks, -Andrew -----Original Message----- From: Jonathan Gibbons Sent: Monday, January 7, 2019 7:54 AM To: Andrew Luo ; code-tools-dev at openjdk.java.net Subject: Re: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) Andrew, I've been generally following the progress for supporting WSL, but now that I need to get more involved, (i.e. reviewing jtreg updates) can you point me at any reasonable general introductory material for WSL? -- Jon On 1/6/19 11:29 AM, Andrew Luo wrote: We recently added support for building OpenJDK on WSL (targeting Windows; targeting Linux always worked). However, running tests on WSL did not work, and we determined that some changes would be needed to jtreg to support this use case. In particular, jtreg needs to know if it is running on WSL targeting Windows or WSL targeting Linux (when targeting Linux, no changes are needed as it is the same as any other Linux system, but when targeting Windows jtreg needs to run shell scripts in a special manner - in particular, calling "wsl.exe sh" instead of just "sh" since it's a Windows boot JDK, as well as translating environment variables using WSLENV...) I've attached my patch with my proposed changes. I welcome any thoughts/feedback. (See https://bugs.openjdk.java.net/browse/JDK-8215445 for more details about supporting WSL for building OpenJDK) Thanks, -Andrew From jonathan.gibbons at oracle.com Mon Jan 7 23:32:43 2019 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Mon, 7 Jan 2019 15:32:43 -0800 Subject: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) In-Reply-To: References: <658a4caf-0400-e31b-d3b1-79e7f00a2c06@oracle.com> <71ed2368-0051-8243-ced9-27598d6c7368@oracle.com> Message-ID: <634cce18-fad7-64bf-cfb9-208c2b330e07@oracle.com> The core way to build jtreg is to run "make" directly. build-all.sh is primarily intended as a convenient wrapper for those that want to use it. Oracle still supports older releases of JDK for paying customers, which is why we want to retain the option to build jtreg with JDK 7, as well as with JDK 8. I'll play with the ShellAction code to see if I can come up with anything better. -- Jon On 01/07/2019 03:23 PM, Andrew Luo wrote: > > Hi Jon, > > Thanks for reviewing this.? Yes, there is a bug there, it should be > !result.isEmpty() at the end of joinWSLENV.? Is it still the case that > we support JDK 7?? I got the impression from build-all.sh:48 that JDK > 7 is no longer supported: > > ifcase${JAVA_VERSION} in 1.8*) false;; *) true; esac; then > > echo"Error: Expected a path to JDK with version 1.8, got version > ${JAVA_VERSION}">&2 > > exit1 > > fi > > Or perhaps is this only the build-all.sh script that doesn?t support > JDK 7, build jtreg itself does? > > As for the changes in ShellAction, the reason I did it that way > instead of scanning through env, is because some environment variables > have to be translated raw, others are paths, and others are path lists > (multiple paths separated by ; in Windows). ?These all have to be > added into WSLENV differently (in particular with the /p or /l > suffix). Scanning through would be a bit error prone, as you?d have to > decide if the variable is a raw string, path, or path list just using > the value.? Although it?s possible, it seems a bit error > prone/clunky.? If you have some preferred solution to this though, I?d > be happy to make those changes. > > Thanks, > > -Andrew > > *From:*Jonathan Gibbons > *Sent:* Monday, January 7, 2019 2:20 PM > *To:* Andrew Luo ; > code-tools-dev at openjdk.java.net > *Subject:* Re: [PATCH] Enable jtreg tests to run on WSL (Windows > Subsystem for Linux) > > Andrew, > > There are some initial problems with your patch. > > 1.? In ShellAction.joinWSLEnv ... it is still a self-imposed > requirement that we can build jtreg with JDK 7 (to help support > testing older releases of JDK.) In joinWSLEnv, you use String.join, > which is @since 8, and so not available on 7. For now, I've written a > replacement, but in doing do. I wonder if there is a bug in your > original.? In particular, at the end of the method, why do you add > `suffix` if `result.isEmpty())`.? I would have expected this to be a > test if the result was _not_ empty. > > Anyway, here is my proposed replacement for the method: > > *private *String joinWSLENV(List list, String suffix) { > > ?? ?StringBuilder sb = *new *StringBuilder(); > > ?? ?String sep = *""*; > > *for *(String item : list) { > > *if *(item.trim().isEmpty() || item.toLowerCase().equals(*"path"*)) { > > *continue*; > > ?????? ?} > > ?????? ?sb.append(sep).append(item).append(suffix); > > ?????? ?sep = *":"*; > > ?? ?} > > *return *sb.toString(); > > } > > The code appends the ":" separator after the first item, and always > appends the suffix.?? Is that what you believe it should do? > > 2. You added a new option, but did not add an entry into the resource > file. This causes the jtreg build to fail one of its self-tests.? I've > created a placeholder entry, but it is at the semantic level I hate: > "getFo(): gets a foo".? If you can provide a more informative string, > that would be better. > > $ hg diff src/share/classes/com/sun/javatest/regtest/tool/i18n.properties > diff -r b6fabf6bf942 > src/share/classes/com/sun/javatest/regtest/tool/i18n.properties > --- a/src/share/classes/com/sun/javatest/regtest/tool/i18n.properties > Thu Dec 20 14:43:41 2018 -0800 > +++ b/src/share/classes/com/sun/javatest/regtest/tool/i18n.properties > Mon Jan 07 13:38:40 2019 -0800 > @@ -257,6 +257,7 @@ > ???? The name may contain '*' to match any sequence of characters. \ > ???? For example, result.* or *.err. > ?help.main.retain.arg=,... > +help.main.wslWindowsTarget.desc=(Windows only) Run with WSL targeting > Windows > ?help.main.dir.desc=Specify a base directory for test files and > directories > ?help.main.dir.arg= > ?help.main.allowSetSecurityManager.desc=Allow agentVM tests to set a > security manager. \ > > 3. README. It's not OK to change the baseline JDK used to build jtreg. > If there is a WSL-reason why JDK 8 is required to run jtreg in a WSL > environment, that should be done as a runtime check. > > Beyond that, the changes to ShellAction feel a bit clunky, changing > the non-WSL flow of the code more than might be thought necessary.? > After all, there's really only two changes here: inserting wsl.exe at > the begtinning of the command, and computing/inserting WSLENV into the > environment. Given that WSLENV only contains the names of the env > variables, with suffixes as needed, I would have thought that a > simpler impl would be to scan the keys in env to build WSLENV, using > strings-in-switch to identify the suffixes, and do that all in a > relatively simple helper method, without having to affect the mainline > code to put values into the env. > > I've created CODETOOLS-7902357 to track this work. > https://bugs.openjdk.java.net/browse/CODETOOLS-7902357 > > I've put a webrev containing your patch at > http://cr.openjdk.java.net/~jjg/7902357/webrev.00/ > > I've put a followup containing your patch + fixes for 1-2-3 above at > http://cr.openjdk.java.net/~jjg/7902357/webrev.01/ > > > -- Jon > > On 01/07/2019 12:36 PM, Andrew Luo wrote: > > Hi Jon, > > For a general intro to WSL, see here: > > https://blogs.msdn.microsoft.com/wsl/2016/04/22/windows-subsystem-for-linux-overview/ > > https://docs.microsoft.com/en-us/windows/wsl/faq > > These links might also be useful for reviewing these changes specifically: > > https://docs.microsoft.com/en-us/windows/wsl/interop > > https://blogs.msdn.microsoft.com/commandline/2017/12/22/share-environment-vars-between-wsl-and-windows/ > > Thanks, > > -Andrew > > -----Original Message----- > > From: Jonathan Gibbons > > Sent: Monday, January 7, 2019 7:54 AM > > To: Andrew Luo > ;code-tools-dev at openjdk.java.net > > > Subject: Re: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) > > Andrew, > > I've been generally following the progress for supporting WSL, but now that I need to get more involved, (i.e. reviewing jtreg updates) can you point me at any reasonable general introductory material for WSL? > > -- Jon > > On 1/6/19 11:29 AM, Andrew Luo wrote: > > We recently added support for building OpenJDK on WSL (targeting > > Windows; targeting Linux always worked).? However, running tests on > > WSL did not work, and we determined that some changes would be needed > > to jtreg to support this use case.? In particular, jtreg needs to know > > if it is running on WSL targeting Windows or WSL targeting Linux (when > > targeting Linux, no changes are needed as it is the same as any other > > Linux system, but when targeting Windows jtreg needs to run shell > > scripts in a special manner - in particular, calling "wsl.exe sh" > > instead of just "sh" since it's a Windows boot JDK, as well as > > translating environment variables using WSLENV...) > > I've attached my patch with my proposed changes.? I welcome any thoughts/feedback. > > (Seehttps://bugs.openjdk.java.net/browse/JDK-8215445 for more details > > about supporting WSL for building OpenJDK) > > Thanks, > > -Andrew > From jonathan.gibbons at oracle.com Tue Jan 8 00:25:33 2019 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Mon, 7 Jan 2019 16:25:33 -0800 Subject: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) In-Reply-To: References: <658a4caf-0400-e31b-d3b1-79e7f00a2c06@oracle.com> <71ed2368-0051-8243-ced9-27598d6c7368@oracle.com> Message-ID: Andrew, I've put out an updated webrev, that is somewhat simpler than your version, although it should be functionally equivalent. In particular, it has a much smaller impact on the non-WSL code. Some questions: * Why is it important to distinguish "path" and "path list"? Isn't a "path" just a "path list" with a single entry? In other words, can we always use suffix `/l` for a list of one-or-more paths? * Is it important to ignore case when checking for "PATH"? Would the Windows environment ever contain an env variable "path"? * The env should never have entries in it with "blank" names; was it just defensive programming on your part to check for blank names and ignore them, or did you see them in practice? * I'm not set up here right now to be able to test with WSL; can you try building and testing this version of jtreg? -- Jon Webrev: http://cr.openjdk.java.net/~jjg/7902357/webrev.02/ On 01/07/2019 03:23 PM, Andrew Luo wrote: > > Hi Jon, > > Thanks for reviewing this.? Yes, there is a bug there, it should be > !result.isEmpty() at the end of joinWSLENV.? Is it still the case that > we support JDK 7?? I got the impression from build-all.sh:48 that JDK > 7 is no longer supported: > > ifcase${JAVA_VERSION} in 1.8*) false;; *) true; esac; then > > echo"Error: Expected a path to JDK with version 1.8, got version > ${JAVA_VERSION}">&2 > > exit1 > > fi > > Or perhaps is this only the build-all.sh script that doesn?t support > JDK 7, build jtreg itself does? > > As for the changes in ShellAction, the reason I did it that way > instead of scanning through env, is because some environment variables > have to be translated raw, others are paths, and others are path lists > (multiple paths separated by ; in Windows). ?These all have to be > added into WSLENV differently (in particular with the /p or /l > suffix). Scanning through would be a bit error prone, as you?d have to > decide if the variable is a raw string, path, or path list just using > the value.? Although it?s possible, it seems a bit error > prone/clunky.? If you have some preferred solution to this though, I?d > be happy to make those changes. > > Thanks, > > -Andrew > > *From:*Jonathan Gibbons > *Sent:* Monday, January 7, 2019 2:20 PM > *To:* Andrew Luo ; > code-tools-dev at openjdk.java.net > *Subject:* Re: [PATCH] Enable jtreg tests to run on WSL (Windows > Subsystem for Linux) > > Andrew, > > There are some initial problems with your patch. > > 1.? In ShellAction.joinWSLEnv ... it is still a self-imposed > requirement that we can build jtreg with JDK 7 (to help support > testing older releases of JDK.) In joinWSLEnv, you use String.join, > which is @since 8, and so not available on 7. For now, I've written a > replacement, but in doing do. I wonder if there is a bug in your > original.? In particular, at the end of the method, why do you add > `suffix` if `result.isEmpty())`.? I would have expected this to be a > test if the result was _not_ empty. > > Anyway, here is my proposed replacement for the method: > > *private *String joinWSLENV(List list, String suffix) { > > ?? ?StringBuilder sb = *new *StringBuilder(); > > ?? ?String sep = *""*; > > *for *(String item : list) { > > *if *(item.trim().isEmpty() || item.toLowerCase().equals(*"path"*)) { > > *continue*; > > ?????? ?} > > ?????? ?sb.append(sep).append(item).append(suffix); > > ?????? ?sep = *":"*; > > ?? ?} > > *return *sb.toString(); > > } > > The code appends the ":" separator after the first item, and always > appends the suffix.?? Is that what you believe it should do? > > 2. You added a new option, but did not add an entry into the resource > file. This causes the jtreg build to fail one of its self-tests.? I've > created a placeholder entry, but it is at the semantic level I hate: > "getFo(): gets a foo".? If you can provide a more informative string, > that would be better. > > $ hg diff src/share/classes/com/sun/javatest/regtest/tool/i18n.properties > diff -r b6fabf6bf942 > src/share/classes/com/sun/javatest/regtest/tool/i18n.properties > --- a/src/share/classes/com/sun/javatest/regtest/tool/i18n.properties > Thu Dec 20 14:43:41 2018 -0800 > +++ b/src/share/classes/com/sun/javatest/regtest/tool/i18n.properties > Mon Jan 07 13:38:40 2019 -0800 > @@ -257,6 +257,7 @@ > ???? The name may contain '*' to match any sequence of characters. \ > ???? For example, result.* or *.err. > ?help.main.retain.arg=,... > +help.main.wslWindowsTarget.desc=(Windows only) Run with WSL targeting > Windows > ?help.main.dir.desc=Specify a base directory for test files and > directories > ?help.main.dir.arg= > ?help.main.allowSetSecurityManager.desc=Allow agentVM tests to set a > security manager. \ > > 3. README. It's not OK to change the baseline JDK used to build jtreg. > If there is a WSL-reason why JDK 8 is required to run jtreg in a WSL > environment, that should be done as a runtime check. > > Beyond that, the changes to ShellAction feel a bit clunky, changing > the non-WSL flow of the code more than might be thought necessary.? > After all, there's really only two changes here: inserting wsl.exe at > the begtinning of the command, and computing/inserting WSLENV into the > environment. Given that WSLENV only contains the names of the env > variables, with suffixes as needed, I would have thought that a > simpler impl would be to scan the keys in env to build WSLENV, using > strings-in-switch to identify the suffixes, and do that all in a > relatively simple helper method, without having to affect the mainline > code to put values into the env. > > I've created CODETOOLS-7902357 to track this work. > https://bugs.openjdk.java.net/browse/CODETOOLS-7902357 > > I've put a webrev containing your patch at > http://cr.openjdk.java.net/~jjg/7902357/webrev.00/ > > I've put a followup containing your patch + fixes for 1-2-3 above at > http://cr.openjdk.java.net/~jjg/7902357/webrev.01/ > > > -- Jon > > On 01/07/2019 12:36 PM, Andrew Luo wrote: > > Hi Jon, > > For a general intro to WSL, see here: > > https://blogs.msdn.microsoft.com/wsl/2016/04/22/windows-subsystem-for-linux-overview/ > > https://docs.microsoft.com/en-us/windows/wsl/faq > > These links might also be useful for reviewing these changes specifically: > > https://docs.microsoft.com/en-us/windows/wsl/interop > > https://blogs.msdn.microsoft.com/commandline/2017/12/22/share-environment-vars-between-wsl-and-windows/ > > Thanks, > > -Andrew > > -----Original Message----- > > From: Jonathan Gibbons > > Sent: Monday, January 7, 2019 7:54 AM > > To: Andrew Luo > ;code-tools-dev at openjdk.java.net > > > Subject: Re: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) > > Andrew, > > I've been generally following the progress for supporting WSL, but now that I need to get more involved, (i.e. reviewing jtreg updates) can you point me at any reasonable general introductory material for WSL? > > -- Jon > > On 1/6/19 11:29 AM, Andrew Luo wrote: > > We recently added support for building OpenJDK on WSL (targeting > > Windows; targeting Linux always worked).? However, running tests on > > WSL did not work, and we determined that some changes would be needed > > to jtreg to support this use case.? In particular, jtreg needs to know > > if it is running on WSL targeting Windows or WSL targeting Linux (when > > targeting Linux, no changes are needed as it is the same as any other > > Linux system, but when targeting Windows jtreg needs to run shell > > scripts in a special manner - in particular, calling "wsl.exe sh" > > instead of just "sh" since it's a Windows boot JDK, as well as > > translating environment variables using WSLENV...) > > I've attached my patch with my proposed changes.? I welcome any thoughts/feedback. > > (Seehttps://bugs.openjdk.java.net/browse/JDK-8215445 for more details > > about supporting WSL for building OpenJDK) > > Thanks, > > -Andrew > From andrewluotechnologies at outlook.com Tue Jan 8 18:30:59 2019 From: andrewluotechnologies at outlook.com (Andrew Luo) Date: Tue, 8 Jan 2019 18:30:59 +0000 Subject: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) In-Reply-To: References: <658a4caf-0400-e31b-d3b1-79e7f00a2c06@oracle.com> <71ed2368-0051-8243-ced9-27598d6c7368@oracle.com> Message-ID: Hi Jon, Thanks, looks good to me. I guess that a path is a subset of path list, although I?m not sure why Microsoft gives them separate suffixes? In Windows, technically the correct case is ?Path?. However often times people use PATH (or path), and Linux environment variables are case sensitive, while Windows environment variables are not, so if someone set Linux PATH, path, and Path, we don?t want any of them to be translated to Windows, as I believe any of them would override the Windows path. I didn?t see any blank env elements, the .isEmpty() check must have been by habit. I did test your changes on my WSL environment and everything compiles/works. Thanks, -Andrew From: Jonathan Gibbons Sent: Monday, January 7, 2019 4:26 PM To: Andrew Luo ; code-tools-dev at openjdk.java.net Subject: Re: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) Andrew, I've put out an updated webrev, that is somewhat simpler than your version, although it should be functionally equivalent. In particular, it has a much smaller impact on the non-WSL code. Some questions: * Why is it important to distinguish "path" and "path list"? Isn't a "path" just a "path list" with a single entry? In other words, can we always use suffix `/l` for a list of one-or-more paths? * Is it important to ignore case when checking for "PATH"? Would the Windows environment ever contain an env variable "path"? * The env should never have entries in it with "blank" names; was it just defensive programming on your part to check for blank names and ignore them, or did you see them in practice? * I'm not set up here right now to be able to test with WSL; can you try building and testing this version of jtreg? -- Jon Webrev: http://cr.openjdk.java.net/~jjg/7902357/webrev.02/ From jonathan.gibbons at oracle.com Tue Jan 8 19:21:04 2019 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Tue, 8 Jan 2019 11:21:04 -0800 Subject: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) In-Reply-To: References: <658a4caf-0400-e31b-d3b1-79e7f00a2c06@oracle.com> <71ed2368-0051-8243-ced9-27598d6c7368@oracle.com> Message-ID: <63a6ee11-4a98-7caa-013c-cd486e7c04a6@oracle.com> On 1/8/19 10:30 AM, Andrew Luo wrote: > In Windows, technically the correct case is ?Path?.? However often > times people use PATH (or path), and Linux environment variables are > case sensitive, while Windows environment variables are not, so if > someone set Linux PATH, path, and Path, we don?t want any of them to > be translated to Windows, as I believe any of them would override the > Windows path. OK, thanks for the explanation.?? Here, we're going from Windows binary (the JDK running jtreg) to WSL, and while jtreg is typically invoked by script (implying WSL or Cygwin) it is not required to be so, so I guess we could theoretically see any of the variants of "path", "Path", "PATH" etc, Is that right? -- Jon From jonathan.gibbons at oracle.com Tue Jan 8 21:21:26 2019 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Tue, 8 Jan 2019 13:21:26 -0800 Subject: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) In-Reply-To: References: <658a4caf-0400-e31b-d3b1-79e7f00a2c06@oracle.com> <71ed2368-0051-8243-ced9-27598d6c7368@oracle.com> Message-ID: <4a0e038a-249a-b542-1731-93bca8c80555@oracle.com> Andrew, One more enhancement, in webrev.03. The new command line option is rejected if jtreg is not running on Windows. -- Jon On 01/08/2019 10:30 AM, Andrew Luo wrote: > > Hi Jon, > > Thanks, looks good to me. > > I guess that a path is a subset of path list, although I?m not sure > why Microsoft gives them separate suffixes? > > In Windows, technically the correct case is ?Path?.? However often > times people use PATH (or path), and Linux environment variables are > case sensitive, while Windows environment variables are not, so if > someone set Linux PATH, path, and Path, we don?t want any of them to > be translated to Windows, as I believe any of them would override the > Windows path. > > I didn?t see any blank env elements, the .isEmpty() check must have > been by habit. > > I did test your changes on my WSL environment and everything > compiles/works. > > Thanks, > > -Andrew > > *From:*Jonathan Gibbons > *Sent:* Monday, January 7, 2019 4:26 PM > *To:* Andrew Luo ; > code-tools-dev at openjdk.java.net > *Subject:* Re: [PATCH] Enable jtreg tests to run on WSL (Windows > Subsystem for Linux) > > Andrew, > > I've put out an updated webrev, that is somewhat simpler than your > version, although it should be functionally equivalent. In particular, > it has a much smaller impact on the non-WSL code. > > Some questions: > > * Why is it important to distinguish "path" and "path list"? Isn't a > "path" just a "path list" with a single entry? In other words, can > we always use suffix `/l` for a list of one-or-more paths? > * Is it important to ignore case when checking for "PATH"? Would the > Windows environment ever contain an env variable "path"? > * The env should never have entries in it with "blank" names; was it > just defensive programming on your part to check for blank names > and ignore them, or did you see them in practice? > * I'm not set up here right now to be able to test with WSL; can you > try building and testing this version of jtreg? > > -- Jon > > Webrev: http://cr.openjdk.java.net/~jjg/7902357/webrev.02/ > > From andrewluotechnologies at outlook.com Tue Jan 8 22:17:39 2019 From: andrewluotechnologies at outlook.com (Andrew Luo) Date: Tue, 8 Jan 2019 22:17:39 +0000 Subject: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) In-Reply-To: <63a6ee11-4a98-7caa-013c-cd486e7c04a6@oracle.com> References: <658a4caf-0400-e31b-d3b1-79e7f00a2c06@oracle.com> <71ed2368-0051-8243-ced9-27598d6c7368@oracle.com> <63a6ee11-4a98-7caa-013c-cd486e7c04a6@oracle.com> Message-ID: When running on WSL targeting Windows, the boot JDK is a Windows JDK, so there is only one Path. However, the env map could have PATH, Path, path, etc. from the environment of the specific test case or from the native library path, etc (I believe, although I?m not sure ? I?m not that familiar with jtreg but I assumed it would be possible). We don?t need to add any form of Path to WSLENV; it is auto-translated from Windows -> WSL by appending. However, if you add Path to WSLENV, then it will be translated both ways (Windows -> WSL and WSL -> Windows). Translating Path from WSL -> Windows is almost never what you want to do, since it would overwrite (rather than append to) the Windows Path (which includes things you need such as System32, etc.) Thanks, -Andrew From: Jonathan Gibbons Sent: Tuesday, January 8, 2019 11:21 AM To: Andrew Luo ; code-tools-dev at openjdk.java.net Subject: Re: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) On 1/8/19 10:30 AM, Andrew Luo wrote: In Windows, technically the correct case is ?Path?. However often times people use PATH (or path), and Linux environment variables are case sensitive, while Windows environment variables are not, so if someone set Linux PATH, path, and Path, we don?t want any of them to be translated to Windows, as I believe any of them would override the Windows path. OK, thanks for the explanation. Here, we're going from Windows binary (the JDK running jtreg) to WSL, and while jtreg is typically invoked by script (implying WSL or Cygwin) it is not required to be so, so I guess we could theoretically see any of the variants of "path", "Path", "PATH" etc, Is that right? -- Jon From andrewluotechnologies at outlook.com Tue Jan 8 22:22:38 2019 From: andrewluotechnologies at outlook.com (Andrew Luo) Date: Tue, 8 Jan 2019 22:22:38 +0000 Subject: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) In-Reply-To: <36154e0c-d466-d516-0856-c23b7e4cfef9@oracle.com> References: <658a4caf-0400-e31b-d3b1-79e7f00a2c06@oracle.com> <71ed2368-0051-8243-ced9-27598d6c7368@oracle.com> <36154e0c-d466-d516-0856-c23b7e4cfef9@oracle.com> Message-ID: Jon, I?m not sure what exactly you mean that the ?test effectively require either Cygwin or WSL?? I believe it would be possible (ideal actually) to write the tests such that they can run on all platforms (Linux, Cygwin, WSL Windows target) ? most tests today run on multiple platforms already, right? The option needs to be provided only when you use WSL to target Windows. If you use WSL to target Linux then the option should not be set; the behavior will be the same as Linux. I don?t think it?s possible for jtreg or the test itself to infer this information, because it depends on whether you called ./configure in OpenJDK to build a Linux JDK on WSL or a Windows JDK on WSL. Perhaps I?m misunderstanding what you?re saying though? Thanks, -Andrew From: Jonathan Gibbons Sent: Tuesday, January 8, 2019 11:25 AM To: Andrew Luo ; code-tools-dev at openjdk.java.net Subject: Re: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) Andrew, One final question ... You introduce a new jtreg option to sepcify the use of WSL. Given that the test effectively require either Cygwin or WSL, would it be reasonable to try and infer which subsystem is required, perhaps leaving the new option to be used only in the case of uncertainty/ambiguity? Not everyone runs jtreg tests through the Makefiles, and for those that don't, requiring the new option will mean that they need to specify the new option to use WSL. It would be nice if they didn't have to. -- Jon From andrewluotechnologies at outlook.com Tue Jan 8 22:26:00 2019 From: andrewluotechnologies at outlook.com (Andrew Luo) Date: Tue, 8 Jan 2019 22:26:00 +0000 Subject: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) In-Reply-To: References: <658a4caf-0400-e31b-d3b1-79e7f00a2c06@oracle.com> <71ed2368-0051-8243-ced9-27598d6c7368@oracle.com> <36154e0c-d466-d516-0856-c23b7e4cfef9@oracle.com> Message-ID: Oh, now that I look at your 3rd revision, I think I understand what you meant. jtreg can detect whether or not it is targeting Windows or Linux based on the current OS of the JRE being used to run jtreg - is that what you mean? If so, that might work, but on the build-dev discussion while adding WSL we did think about it being possible (later on; see https://mail.openjdk.java.net/pipermail/build-dev/2018-December/024542.html) to use either a Windows or Linux boot JDK to build on WSL. Thanks, -Andrew -----Original Message----- From: code-tools-dev On Behalf Of Andrew Luo Sent: Tuesday, January 8, 2019 2:23 PM To: Jonathan Gibbons ; code-tools-dev at openjdk.java.net Subject: RE: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) Jon, I?m not sure what exactly you mean that the ?test effectively require either Cygwin or WSL?? I believe it would be possible (ideal actually) to write the tests such that they can run on all platforms (Linux, Cygwin, WSL Windows target) ? most tests today run on multiple platforms already, right? The option needs to be provided only when you use WSL to target Windows. If you use WSL to target Linux then the option should not be set; the behavior will be the same as Linux. I don?t think it?s possible for jtreg or the test itself to infer this information, because it depends on whether you called ./configure in OpenJDK to build a Linux JDK on WSL or a Windows JDK on WSL. Perhaps I?m misunderstanding what you?re saying though? Thanks, -Andrew From: Jonathan Gibbons Sent: Tuesday, January 8, 2019 11:25 AM To: Andrew Luo ; code-tools-dev at openjdk.java.net Subject: Re: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) Andrew, One final question ... You introduce a new jtreg option to sepcify the use of WSL. Given that the test effectively require either Cygwin or WSL, would it be reasonable to try and infer which subsystem is required, perhaps leaving the new option to be used only in the case of uncertainty/ambiguity? Not everyone runs jtreg tests through the Makefiles, and for those that don't, requiring the new option will mean that they need to specify the new option to use WSL. It would be nice if they didn't have to. -- Jon From jonathan.gibbons at oracle.com Tue Jan 8 23:15:37 2019 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Tue, 8 Jan 2019 15:15:37 -0800 Subject: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) In-Reply-To: References: <658a4caf-0400-e31b-d3b1-79e7f00a2c06@oracle.com> <71ed2368-0051-8243-ced9-27598d6c7368@oracle.com> <36154e0c-d466-d516-0856-c23b7e4cfef9@oracle.com> Message-ID: <5ef11986-a5bd-8c00-a471-ecd81f468946@oracle.com> Well, the issue is about how to run shell tests when running on Windows, because the shell tests are written to be run with bash, which means that on Windows you need either WSL or Cygwin, right? But, you write:? "it depends on whether you called ./configure in OpenJDK to build a Linux JDK on WSL or a Windows JDK on WSL."? Would it not be reasonable to expect that you can query the JVM to know if it is a "Linux JDK on WSL" or a "Windows JDK on WSL"?? For example, is there any information in the output of `java -version` or `java -fullversion` or a system property that identifies which of these variants the JDK is? -- Jon On 01/08/2019 02:22 PM, Andrew Luo wrote: > > Jon, > > I?m not sure what exactly you mean that the ?test effectively require > either Cygwin or WSL??? I believe it would be possible (ideal > actually) to write the tests such that they can run on all platforms > (Linux, Cygwin, WSL Windows target) ? most tests today run on multiple > platforms already, right? > > The option needs to be provided only when you use WSL to target > Windows.? If you use WSL to target Linux then the option should not be > set; the behavior will be the same as Linux. I don?t think it?s > possible for jtreg or the test itself to infer this information, > because it depends on whether you called ./configure in OpenJDK to > build a Linux JDK on WSL or a Windows JDK on WSL. > > Perhaps I?m misunderstanding what you?re saying though? > > Thanks, > > -Andrew > > *From:*Jonathan Gibbons > *Sent:* Tuesday, January 8, 2019 11:25 AM > *To:* Andrew Luo ; > code-tools-dev at openjdk.java.net > *Subject:* Re: [PATCH] Enable jtreg tests to run on WSL (Windows > Subsystem for Linux) > > Andrew, > > One final question ... > > You introduce a new jtreg option to sepcify the use of WSL. Given that > the test effectively require either Cygwin or WSL, would it be > reasonable to try and infer which subsystem is required, perhaps > leaving the new option to be used only in the case of > uncertainty/ambiguity? > > Not everyone runs jtreg tests through the Makefiles, and for those > that don't, requiring the new option will mean that they need to > specify the new option to use WSL. It would be nice if they didn't > have to. > > -- Jon > From jonathan.gibbons at oracle.com Tue Jan 8 23:27:46 2019 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Tue, 8 Jan 2019 15:27:46 -0800 Subject: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) In-Reply-To: References: <658a4caf-0400-e31b-d3b1-79e7f00a2c06@oracle.com> <71ed2368-0051-8243-ced9-27598d6c7368@oracle.com> <36154e0c-d466-d516-0856-c23b7e4cfef9@oracle.com> Message-ID: `targetting` is a strange word to use in this context, since (to me) `targetting` implies `action at a distance`. Any Java program can query its environment to determine the OS on which it it running. There are various system properties, all beginning "os.":? os.name, os.arch and os.version. Listed here: https://docs.oracle.com/javase/9/docs/api/java/lang/System.html#getProperties-- So part of the question is, what are the values of these properties when using WSL, or (more accurately) when running a JRE that was built by using "/./configure in OpenJDK to build a Linux JDK on WSL or a Windows JDK on WSL./" Then, the question becomes, can jtreg detect if a compatibility layer is required, and if so can it infer whether to use WSL or Cygwin??? I'm not sure if this question makes sense, but maybe jtreg could infer which to use by seeing whether wsl.exe or sh.exe is on $PATH. And, maybe that's "too magic" ... I'm just asking. -- Jon On 01/08/2019 02:26 PM, Andrew Luo wrote: > Oh, now that I look at your 3rd revision, I think I understand what you meant. jtreg can detect whether or not it is targeting Windows or Linux based on the current OS of the JRE being used to run jtreg - is that what you mean? > > If so, that might work, but on the build-dev discussion while adding WSL we did think about it being possible (later on; see https://mail.openjdk.java.net/pipermail/build-dev/2018-December/024542.html) to use either a Windows or Linux boot JDK to build on WSL. > > Thanks, > > -Andrew > > -----Original Message----- > From: code-tools-dev On Behalf Of Andrew Luo > Sent: Tuesday, January 8, 2019 2:23 PM > To: Jonathan Gibbons ; code-tools-dev at openjdk.java.net > Subject: RE: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) > > Jon, > > I?m not sure what exactly you mean that the ?test effectively require either Cygwin or WSL?? I believe it would be possible (ideal actually) to write the tests such that they can run on all platforms (Linux, Cygwin, WSL Windows target) ? most tests today run on multiple platforms already, right? > > The option needs to be provided only when you use WSL to target Windows. If you use WSL to target Linux then the option should not be set; the behavior will be the same as Linux. I don?t think it?s possible for jtreg or the test itself to infer this information, because it depends on whether you called ./configure in OpenJDK to build a Linux JDK on WSL or a Windows JDK on WSL. > > Perhaps I?m misunderstanding what you?re saying though? > > Thanks, > > -Andrew > > From: Jonathan Gibbons > Sent: Tuesday, January 8, 2019 11:25 AM > To: Andrew Luo ; code-tools-dev at openjdk.java.net > Subject: Re: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) > > > Andrew, > > One final question ... > > You introduce a new jtreg option to sepcify the use of WSL. Given that the test effectively require either Cygwin or WSL, would it be reasonable to try and infer which subsystem is required, perhaps leaving the new option to be used only in the case of uncertainty/ambiguity? > > Not everyone runs jtreg tests through the Makefiles, and for those that don't, requiring the new option will mean that they need to specify the new option to use WSL. It would be nice if they didn't have to. > > -- Jon > From jonathan.gibbons at oracle.com Wed Jan 9 21:34:38 2019 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Wed, 9 Jan 2019 13:34:38 -0800 Subject: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) In-Reply-To: <4a0e038a-249a-b542-1731-93bca8c80555@oracle.com> References: <658a4caf-0400-e31b-d3b1-79e7f00a2c06@oracle.com> <71ed2368-0051-8243-ced9-27598d6c7368@oracle.com> <4a0e038a-249a-b542-1731-93bca8c80555@oracle.com> Message-ID: <237acedd-1eda-5c36-9c36-31d1f8aa8bc0@oracle.com> I have pushed a version of this patch to the jtreg repo. I did some renaming: I've simplified the command-line option to just `-wsl`, improved the command line help, and internally, used more verbose names like "useWindowsSubsystemForLinux" instead of the more cryptic "wslWindowsTarget". It may still be possible to refine the use of the option, so that it may not be necessary to specify the option if jtreg can determine whether to use Cygwin or WSL. But this can be done as a followup changeset. We'll do some additional testing of this feature before promoting a new version of jtreg for more widespread use. -- Jon On 01/08/2019 01:21 PM, Jonathan Gibbons wrote: > Andrew, > > One more enhancement, in webrev.03. The new command line option is > rejected if jtreg is not running on Windows. > > -- Jon > > > On 01/08/2019 10:30 AM, Andrew Luo wrote: >> >> Hi Jon, >> >> Thanks, looks good to me. >> >> I guess that a path is a subset of path list, although I?m not sure >> why Microsoft gives them separate suffixes? >> >> In Windows, technically the correct case is ?Path?.? However often >> times people use PATH (or path), and Linux environment variables are >> case sensitive, while Windows environment variables are not, so if >> someone set Linux PATH, path, and Path, we don?t want any of them to >> be translated to Windows, as I believe any of them would override the >> Windows path. >> >> I didn?t see any blank env elements, the .isEmpty() check must have >> been by habit. >> >> I did test your changes on my WSL environment and everything >> compiles/works. >> >> Thanks, >> >> -Andrew >> >> *From:*Jonathan Gibbons >> *Sent:* Monday, January 7, 2019 4:26 PM >> *To:* Andrew Luo ; >> code-tools-dev at openjdk.java.net >> *Subject:* Re: [PATCH] Enable jtreg tests to run on WSL (Windows >> Subsystem for Linux) >> >> Andrew, >> >> I've put out an updated webrev, that is somewhat simpler than your >> version, although it should be functionally equivalent. In >> particular, it has a much smaller impact on the non-WSL code. >> >> Some questions: >> >> ? * Why is it important to distinguish "path" and "path list"? Isn't a >> ??? "path" just a "path list" with a single entry? In other words, can >> ??? we always use suffix `/l` for a list of one-or-more paths? >> ? * Is it important to ignore case when checking for "PATH"? Would the >> ??? Windows environment ever contain an env variable "path"? >> ? * The env should never have entries in it with "blank" names; was it >> ??? just defensive programming on your part to check for blank names >> ??? and ignore them, or did you see them in practice? >> ? * I'm not set up here right now to be able to test with WSL; can you >> ??? try building and testing this version of jtreg? >> >> -- Jon >> >> Webrev: http://cr.openjdk.java.net/~jjg/7902357/webrev.02/ >> >> > From andrewluotechnologies at outlook.com Thu Jan 10 00:42:18 2019 From: andrewluotechnologies at outlook.com (Andrew Luo) Date: Thu, 10 Jan 2019 00:42:18 +0000 Subject: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) In-Reply-To: References: <658a4caf-0400-e31b-d3b1-79e7f00a2c06@oracle.com> <71ed2368-0051-8243-ced9-27598d6c7368@oracle.com> <36154e0c-d466-d516-0856-c23b7e4cfef9@oracle.com> Message-ID: I see what you mean. From my understanding, there are 2 JDKs here: 1. Boot JDK ? which is used to build the new OpenJDK, and run jtreg (is that right, or does the JDK under test itself run jtreg?) 2. JDK under test ? the JDK that was just built from the OpenJDK source tree, what jtreg is actually testing Currently the boot JDK can only be Windows when running a WSL Windows build. But I think this could change in the future. So if we do potentially allow either a Windows or Linux boot JDK, we wouldn?t be able to test in-process by just checking os.name/os.version properties. However, we could execute the JDK under test and check os.name/os.version out-of-process. If os.name is Windows, then we need either Cygwin, or WSL (or some other compatibility layer ? MSYS or MKS, but my understanding is these are no longer supported). Or potentially, we could just check to see if the java executable is named exactly ?java? or ?java.exe?. Once we determine our JDK under test is a Windows JDK, we have to figure out whether we?re in Cygwin or WSL. I haven?t used Cygwin in a long time, but if I remember right, Cygwin uses the same Path environment variables as Windows (Cygwin bash is a Win32 process). So when invoking the JDK, all the Cygwin executables will live on the Path, so one can just call uname to check if one is in Cygwin. In WSL, this call to uname will either fail (Windows Boot JDK) or return Linux (Linux Boot JDK), but assuming we support no other compatibility layers, if we?re not in Cygwin, we can assume we?re in WSL. Checking for wsl.exe being on the Path, in my opinion, would not be ideal, since if both Cygwin and WSL are installed on a system, wsl.exe would be on the Path, even if you are in Cygwin (or some other compatibility layer, assuming it shares the Windows Path). Thanks, -Andrew From: Jonathan Gibbons Sent: Tuesday, January 8, 2019 3:28 PM To: Andrew Luo ; code-tools-dev at openjdk.java.net Subject: Re: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) `targetting` is a strange word to use in this context, since (to me) `targetting` implies `action at a distance`. Any Java program can query its environment to determine the OS on which it it running. There are various system properties, all beginning "os.": os.name, os.arch and os.version. Listed here: https://docs.oracle.com/javase/9/docs/api/java/lang/System.html#getProperties-- So part of the question is, what are the values of these properties when using WSL, or (more accurately) when running a JRE that was built by using "./configure in OpenJDK to build a Linux JDK on WSL or a Windows JDK on WSL." Then, the question becomes, can jtreg detect if a compatibility layer is required, and if so can it infer whether to use WSL or Cygwin? I'm not sure if this question makes sense, but maybe jtreg could infer which to use by seeing whether wsl.exe or sh.exe is on $PATH. And, maybe that's "too magic" ... I'm just asking. -- Jon On 01/08/2019 02:26 PM, Andrew Luo wrote: Oh, now that I look at your 3rd revision, I think I understand what you meant. jtreg can detect whether or not it is targeting Windows or Linux based on the current OS of the JRE being used to run jtreg - is that what you mean? If so, that might work, but on the build-dev discussion while adding WSL we did think about it being possible (later on; see https://mail.openjdk.java.net/pipermail/build-dev/2018-December/024542.html) to use either a Windows or Linux boot JDK to build on WSL. Thanks, -Andrew -----Original Message----- From: code-tools-dev On Behalf Of Andrew Luo Sent: Tuesday, January 8, 2019 2:23 PM To: Jonathan Gibbons ; code-tools-dev at openjdk.java.net Subject: RE: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) Jon, I?m not sure what exactly you mean that the ?test effectively require either Cygwin or WSL?? I believe it would be possible (ideal actually) to write the tests such that they can run on all platforms (Linux, Cygwin, WSL Windows target) ? most tests today run on multiple platforms already, right? The option needs to be provided only when you use WSL to target Windows. If you use WSL to target Linux then the option should not be set; the behavior will be the same as Linux. I don?t think it?s possible for jtreg or the test itself to infer this information, because it depends on whether you called ./configure in OpenJDK to build a Linux JDK on WSL or a Windows JDK on WSL. Perhaps I?m misunderstanding what you?re saying though? Thanks, -Andrew From: Jonathan Gibbons Sent: Tuesday, January 8, 2019 11:25 AM To: Andrew Luo ; code-tools-dev at openjdk.java.net Subject: Re: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) Andrew, One final question ... You introduce a new jtreg option to sepcify the use of WSL. Given that the test effectively require either Cygwin or WSL, would it be reasonable to try and infer which subsystem is required, perhaps leaving the new option to be used only in the case of uncertainty/ambiguity? Not everyone runs jtreg tests through the Makefiles, and for those that don't, requiring the new option will mean that they need to specify the new option to use WSL. It would be nice if they didn't have to. -- Jon From jonathan.gibbons at oracle.com Thu Jan 10 00:58:47 2019 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Wed, 9 Jan 2019 16:58:47 -0800 Subject: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) In-Reply-To: References: <658a4caf-0400-e31b-d3b1-79e7f00a2c06@oracle.com> <71ed2368-0051-8243-ced9-27598d6c7368@oracle.com> <36154e0c-d466-d516-0856-c23b7e4cfef9@oracle.com> Message-ID: Andrew, You are right that there are typically 2 JDKs involved, but jtreg is a standalone app, and does not need to be run from the Makefiles, and so is not tied to the Boot JDK. But it is true to say that there is the JDK used to run jtreg, and the JDK under test. Until your recent work with WSL, Cygwin was the only supported environment for running shell tests. (MKS was the official solution for Sun, back before JDK was open source; MSYS never quite took hold.) If we get to the point of widely adopting/supporting WSL, then we could have jtreg default to using WSL if WSL is found to be installed and on the path, or we could figure a way to determine that Cygwin is available and should be used. Either way, I would retain the ability to specify which to use with an option, in case people want to override the default choice made by jtreg. And, Erik suggested that when running jtreg from the Makefiles, it would make sense to always specify which compatibility layer to use. But, for now, we can defer these design decisions, and rely on requiring the option to be specified. -- Jon On 01/09/2019 04:42 PM, Andrew Luo wrote: > > I see what you mean. > > From my understanding, there are 2 JDKs here: > > 1.Boot JDK ? which is used to build the new OpenJDK, and run jtreg (is > that right, or does the JDK under test itself run jtreg?) > > 2.JDK under test ? the JDK that was just built from the OpenJDK source > tree, what jtreg is actually testing > > Currently the boot JDK can only be Windows when running a WSL Windows > build.? But I think this could change in the future.? So if we do > potentially allow either a Windows or Linux boot JDK, we wouldn?t be > able to test in-process by just checking os.name/os.version > properties.? However, we could execute the JDK under test and check > os.name/os.version out-of-process.? If os.name is Windows, then we > need either Cygwin, or WSL (or some other compatibility layer ? MSYS > or MKS, but my understanding is these are no longer supported).? Or > potentially, we could just check to see if the java executable is > named exactly ?java? or ?java.exe?. > > Once we determine our JDK under test is a Windows JDK, we have to > figure out whether we?re in Cygwin or WSL.? I haven?t used Cygwin in a > long time, but if I remember right, Cygwin uses the same Path > environment variables as Windows (Cygwin bash is a Win32 process).? So > when invoking the JDK, all the Cygwin executables will live on the > Path, so one can just call uname to check if one is in Cygwin.? In > WSL, this call to uname will either fail (Windows Boot JDK) or return > Linux (Linux Boot JDK), but assuming we support no other compatibility > layers, if we?re not in Cygwin, we can assume we?re in WSL. > > Checking for wsl.exe being on the Path, in my opinion, would not be > ideal, since if both Cygwin and WSL are installed on a system, wsl.exe > would be on the Path, even if you are in Cygwin (or some other > compatibility layer, assuming it shares the Windows Path). > > Thanks, > > -Andrew > > *From:*Jonathan Gibbons > *Sent:* Tuesday, January 8, 2019 3:28 PM > *To:* Andrew Luo ; > code-tools-dev at openjdk.java.net > *Subject:* Re: [PATCH] Enable jtreg tests to run on WSL (Windows > Subsystem for Linux) > > `targetting` is a strange word to use in this context, since (to me) > `targetting` implies `action at a distance`. > > Any Java program can query its environment to determine the OS on > which it it running. There are various system properties, all > beginning "os.":? os.name, os.arch and os.version. Listed here: > https://docs.oracle.com/javase/9/docs/api/java/lang/System.html#getProperties-- > > So part of the question is, what are the values of these properties > when using WSL, or (more accurately) when running a JRE that was built > by using "/./configure in OpenJDK to build a Linux JDK on WSL or a > Windows JDK on WSL./" > > Then, the question becomes, can jtreg detect if a compatibility layer > is required, and if so can it infer whether to use WSL or Cygwin??? > I'm not sure if this question makes sense, but maybe jtreg could infer > which to use by seeing whether wsl.exe or sh.exe is on $PATH. And, > maybe that's "too magic" ... I'm just asking. > > -- Jon > > On 01/08/2019 02:26 PM, Andrew Luo wrote: > > Oh, now that I look at your 3rd revision, I think I understand what you meant.? jtreg can detect whether or not it is targeting Windows or Linux based on the current OS of the JRE being used to run jtreg - is that what you mean? > > If so, that might work, but on the build-dev discussion while adding WSL we did think about it being possible (later on; seehttps://mail.openjdk.java.net/pipermail/build-dev/2018-December/024542.html) to use either a Windows or Linux boot JDK to build on WSL. > > Thanks, > > -Andrew > > -----Original Message----- > > From: code-tools-dev > On Behalf Of Andrew Luo > > Sent: Tuesday, January 8, 2019 2:23 PM > > To: Jonathan Gibbons ;code-tools-dev at openjdk.java.net > > > Subject: RE: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) > > Jon, > > I?m not sure what exactly you mean that the ?test effectively require either Cygwin or WSL??? I believe it would be possible (ideal actually) to write the tests such that they can run on all platforms (Linux, Cygwin, WSL Windows target) ? most tests today run on multiple platforms already, right? > > The option needs to be provided only when you use WSL to target Windows.? If you use WSL to target Linux then the option should not be set; the behavior will be the same as Linux.? I don?t think it?s possible for jtreg or the test itself to infer this information, because it depends on whether you called ./configure in OpenJDK to build a Linux JDK on WSL or a Windows JDK on WSL. > > Perhaps I?m misunderstanding what you?re saying though? > > Thanks, > > -Andrew > > From: Jonathan Gibbons > > Sent: Tuesday, January 8, 2019 11:25 AM > > To: Andrew Luo > ;code-tools-dev at openjdk.java.net > > > Subject: Re: [PATCH] Enable jtreg tests to run on WSL (Windows Subsystem for Linux) > > Andrew, > > One final question ... > > You introduce a new jtreg option to sepcify the use of WSL. Given that the test effectively require either Cygwin or WSL, would it be reasonable to try and infer which subsystem is required, perhaps leaving the new option to be used only in the case of uncertainty/ambiguity? > > Not everyone runs jtreg tests through the Makefiles, and for those that don't, requiring the new option will mean that they need to specify the new option to use WSL. It would be nice if they didn't have to. > > -- Jon > From adam.farley at uk.ibm.com Fri Jan 11 10:06:53 2019 From: adam.farley at uk.ibm.com (Adam Farley8) Date: Fri, 11 Jan 2019 10:06:53 +0000 Subject: RFR: JDK-8197541: Close as Unable to Reproduce Message-ID: Hi All, Can't reproduce this bug locally after many tries, and after contacting the originator, he agrees that the issue hasn't been seen since April 2018. I recommend we resolve this as "Unable to Reproduce". All in favor? Bug: https://bugs.openjdk.java.net/browse/JDK-8197541 Best Regards Adam Farley IBM Runtimes Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU From jonathan.gibbons at oracle.com Fri Jan 11 17:00:48 2019 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Fri, 11 Jan 2019 09:00:48 -0800 Subject: RFR: JDK-8197541: Close as Unable to Reproduce In-Reply-To: References: Message-ID: Kulla folk, This looks like your test, and so is your issue to close. I know you have been fixing issues like this, so it may not be surprising that this issue has not been seen recently. -- Jon On 1/11/19 2:06 AM, Adam Farley8 wrote: > Hi All, > > Can't reproduce this bug locally after many tries, and after contacting > the originator, he agrees that the issue hasn't been seen since April > 2018. > > I recommend we resolve this as "Unable to Reproduce". > > All in favor? > > Bug: https://bugs.openjdk.java.net/browse/JDK-8197541 > > Best Regards > > Adam Farley > IBM Runtimes > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU From jan.lahoda at oracle.com Fri Jan 11 17:19:45 2019 From: jan.lahoda at oracle.com (Jan Lahoda) Date: Fri, 11 Jan 2019 18:19:45 +0100 Subject: RFR: JDK-8197541: Close as Unable to Reproduce In-Reply-To: References: Message-ID: <5C38D031.8040200@oracle.com> Hi, Closed as Cannot Reproduce. Please feel free to reopen if it happens again or if you disagree. Jan On 11.1.2019 18:00, Jonathan Gibbons wrote: > Kulla folk, > > This looks like your test, and so is your issue to close. I know you > have been fixing issues like this, so it may not be surprising that this > issue has not been seen recently. > > -- Jon > > On 1/11/19 2:06 AM, Adam Farley8 wrote: >> Hi All, >> >> Can't reproduce this bug locally after many tries, and after contacting >> the originator, he agrees that the issue hasn't been seen since April >> 2018. >> >> I recommend we resolve this as "Unable to Reproduce". >> >> All in favor? >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8197541 >> >> Best Regards >> >> Adam Farley >> IBM Runtimes >> >> Unless stated otherwise above: >> IBM United Kingdom Limited - Registered in England and Wales with number >> 741598. >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 >> 3AU From Leonid.Kuskov at Oracle.com Tue Jan 15 22:55:04 2019 From: Leonid.Kuskov at Oracle.com (Leonid Kuskov) Date: Tue, 15 Jan 2019 14:55:04 -0800 Subject: RFR: CODETOOLS-7902350: JCov 3.0,method mode: The instrumented OpenJDK 12 fails on starting with StackOverflowError Message-ID: <206ec33d-da54-b757-13b8-a97caa25de86@Oracle.com> Hi, Starting from the version 12 instrumented JDK fails with the StackOverflow: Internal Error (/scratch/mesos/slaves/2dd962d0-8988-479b-a804-57ab764ada59-S1642/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/a33a9ec7-fbdb-4eb2-ac32-2e8e766f18b0/runs/0466c2ad-f030-4120-995b-bad4b331f824/workspace/open/src/hotspot/share/utilities/exceptions.cpp:231), pid=72260, tid=8195 #? assert(k->is_initialized()) failed: need to increase java_thread_min_stack_allowed calculation ... Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) j? com.sun.tdk.jcov.runtime.CollectDetect.hit(III)V+3 java.base j? java.lang.NullPointerException.()V+6 java.base v? ~StubRoutines::call_stub j? com.sun.tdk.jcov.runtime.CollectDetect.hit(III)V+3 java.base j? java.lang.NullPointerException.()V+6 java.base v? ~StubRoutines::call_stub ... com.sun.tdk.jcov.runtime.CollectDetect.hit(III)V+3 java.base j? java.lang.Throwable.()V+9 java.base v? ~StubRoutines::call_stub j? com.sun.tdk.jcov.runtime.CollectDetect.hit(III)V+3 java.base j? java.lang.System.allowSecurityManager()Z+9 java.base j java.lang.System.getSecurityManager()Ljava/lang/SecurityManager;+6 java.base j? java.lang.ThreadGroup.checkAccess()V+6 java.base j java.lang.ThreadGroup.checkParentAccess(Ljava/lang/ThreadGroup;)Ljava/lang/Void;+7 java.base j java.lang.ThreadGroup.(Ljava/lang/ThreadGroup;Ljava/lang/String;)V+8 java.base v? ~StubRoutines::call_stub ... There is a trivial fix for this: Bug: https://bugs.openjdk.java.net/browse/CODETOOLS-7902350 Webrev: http://cr.openjdk.java.net/~lkuskov/7902350/webrev.00/ Please review it. Thanks, Leonid From alexandre.iline at oracle.com Tue Jan 15 23:58:21 2019 From: alexandre.iline at oracle.com (Alexandre (Shura) Iline) Date: Tue, 15 Jan 2019 15:58:21 -0800 Subject: RFR: CODETOOLS-7902350: JCov 3.0,method mode: The instrumented OpenJDK 12 fails on starting with StackOverflowError In-Reply-To: <206ec33d-da54-b757-13b8-a97caa25de86@Oracle.com> References: <206ec33d-da54-b757-13b8-a97caa25de86@Oracle.com> Message-ID: The changes look good, Leonid. Perhaps later, as a separate enhancement, it would make sense to completely rewrite com.sun.tdk.jcov.util.Utils.isAdvanceStaticInstrAllowed(String, String) method as it is written poorly. I am guessing it started when there was only one class to exclude, and then it was OK, but now it does not look good. Shura > On Jan 15, 2019, at 2:55 PM, Leonid Kuskov wrote: > > Hi, > > Starting from the version 12 instrumented JDK fails with the StackOverflow: > > Internal Error (/scratch/mesos/slaves/2dd962d0-8988-479b-a804-57ab764ada59-S1642/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/a33a9ec7-fbdb-4eb2-ac32-2e8e766f18b0/runs/0466c2ad-f030-4120-995b-bad4b331f824/workspace/open/src/hotspot/share/utilities/exceptions.cpp:231), pid=72260, tid=8195 > # assert(k->is_initialized()) failed: need to increase java_thread_min_stack_allowed calculation > ... > > Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) > j com.sun.tdk.jcov.runtime.CollectDetect.hit(III)V+3 java.base > j java.lang.NullPointerException.()V+6 java.base > v ~StubRoutines::call_stub > j com.sun.tdk.jcov.runtime.CollectDetect.hit(III)V+3 java.base > j java.lang.NullPointerException.()V+6 java.base > v ~StubRoutines::call_stub > > ... > > com.sun.tdk.jcov.runtime.CollectDetect.hit(III)V+3 java.base > j java.lang.Throwable.()V+9 java.base > v ~StubRoutines::call_stub > j com.sun.tdk.jcov.runtime.CollectDetect.hit(III)V+3 java.base > j java.lang.System.allowSecurityManager()Z+9 java.base > j java.lang.System.getSecurityManager()Ljava/lang/SecurityManager;+6 java.base > j java.lang.ThreadGroup.checkAccess()V+6 java.base > j java.lang.ThreadGroup.checkParentAccess(Ljava/lang/ThreadGroup;)Ljava/lang/Void;+7 java.base > j java.lang.ThreadGroup.(Ljava/lang/ThreadGroup;Ljava/lang/String;)V+8 java.base > v ~StubRoutines::call_stub > ... > > There is a trivial fix for this: > > Bug: https://bugs.openjdk.java.net/browse/CODETOOLS-7902350 > Webrev: http://cr.openjdk.java.net/~lkuskov/7902350/webrev.00/ > > Please review it. > > Thanks, > Leonid > > > From andrewluotechnologies at outlook.com Thu Jan 17 23:28:58 2019 From: andrewluotechnologies at outlook.com (Andrew Luo) Date: Thu, 17 Jan 2019 23:28:58 +0000 Subject: [PATCH] Fix for EXE_SUFFIX being set for WSL having no effect Message-ID: While fixing shell tests, I noticed that EXE_SUFFIX was having no effect... That's because we're appending to env after setting WSLENV... Simple patch attached. Thanks, -Andrew -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: diff.txt URL: From jonathan.gibbons at oracle.com Fri Jan 18 01:40:16 2019 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Thu, 17 Jan 2019 17:40:16 -0800 Subject: [PATCH] Fix for EXE_SUFFIX being set for WSL having no effect In-Reply-To: References: Message-ID: <7f886009-4d4a-e429-ce2e-a19dab055ce3@oracle.com> On 01/17/2019 03:28 PM, Andrew Luo wrote: > While fixing shell tests, I noticed that EXE_SUFFIX was having no effect... That's because we're appending to env after setting WSLENV... Simple patch attached. > > Thanks, > > -Andrew Thanks for testing and reporting. Please don't update test/langtools; I'll be doing those. I recommend getting consensus from other teams before we go ahead with this solution and do a mass update of tests. -- Jon From jonathan.gibbons at oracle.com Fri Jan 18 18:12:48 2019 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Fri, 18 Jan 2019 10:12:48 -0800 Subject: [PATCH] Fix for EXE_SUFFIX being set for WSL having no effect In-Reply-To: <7f886009-4d4a-e429-ce2e-a19dab055ce3@oracle.com> References: <7f886009-4d4a-e429-ce2e-a19dab055ce3@oracle.com> Message-ID: On 01/17/2019 05:40 PM, Jonathan Gibbons wrote: > > > On 01/17/2019 03:28 PM, Andrew Luo wrote: >> While fixing shell tests, I noticed that EXE_SUFFIX was having no >> effect... That's because we're appending to env after setting >> WSLENV... Simple patch attached. >> >> Thanks, >> >> -Andrew > > Thanks for testing and reporting. > > Please don't update test/langtools; I'll be doing those. > > I recommend getting consensus from other teams before we go ahead with > this solution and do a mass update of tests. > > -- Jon Just for the record, here are some stats about the number of shell files in the test/jdk directories 59: test/jdk/sun/security 47: test/jdk/java/lang 30: test/jdk/sun/tools 12: test/jdk/java/awt 10: test/jdk/tools/jjs 8: test/jdk/sun/rmi 8: test/jdk/java/security 6: test/jdk/jdk/jfr 6: test/jdk/javax/imageio 5: test/jdk/sun/management 5: test/jdk/java/util 4: test/jdk/javax/script 3: test/jdk/tools/launcher 3: test/jdk/java/nio 3: test/jdk/com/sun 2: test/jdk/javax/swing 2: test/jdk/javax/crypto 1: test/jdk/sun/net 1: test/jdk/sun/jvmstat 1: test/jdk/sun/java2d 1: test/jdk/jdk/security 1: test/jdk/javax/xml 1: test/jdk/javax/security 1: test/jdk/javax/rmi 1: test/jdk/javax/print 1: test/jdk/javax/accessibility 1: test/jdk/java/rmi 1: test/jdk/java/io 1: test/jdk/com/oracle and hotspot ... 8: test/hotspot/jtreg/compiler/aot 3: test/hotspot/jtreg/vmTestbase/vm 2: test/hotspot/jtreg/vmTestbase/jit 1: test/hotspot/jtreg/vmTestbase/nsk 1: test/hotspot/jtreg/vmTestbase/metaspace 1: test/hotspot/jtreg/vmTestbase/gc 1: test/hotspot/jtreg/serviceability/7170638 1: test/hotspot/jtreg/runtime/StackGuardPages 1: test/hotspot/jtreg/runtime/StackGap 1: test/hotspot/jtreg/runtime/CommandLine 1: test/hotspot/jtreg/runtime/7162488 1: test/hotspot/jtreg/compiler/unsafe 1: test/hotspot/jtreg/compiler/graalunit and langtools... 7: test/langtools/tools/javac 1: test/langtools/tools/doclint From andrewluotechnologies at outlook.com Fri Jan 18 21:34:13 2019 From: andrewluotechnologies at outlook.com (Andrew Luo) Date: Fri, 18 Jan 2019 21:34:13 +0000 Subject: [PATCH] Fix for EXE_SUFFIX being set for WSL having no effect In-Reply-To: References: <7f886009-4d4a-e429-ce2e-a19dab055ce3@oracle.com> Message-ID: Sounds good. My plan is to first try to address most of the failures in tier 1 rather than doing all tests at once. There aren't that many shell tests in tier 1 so it won't be a lot of work wasted if other teams prefer a different approach. Anyways, I want to have something that I know works before I propose it... Regarding the previous topic of whether or not we need the -wsl flag - I noticed in some places we are actually checking the type of JVM, in particular in test/hotspot/jtreg/test_env.sh: ${TESTJAVA}${FS}bin${FS}java${EXE_SUFFIX} ${TESTOPTS} -Xinternalversion | sed -e 's/[(][^)]*[)]//g' -e 's/ by "[^"]*"//g' > vm_version.out 2>&1 echo "INT_VERSION=`cat vm_version.out 2>&1`" ... VM_OS="unknown" grep "aix" vm_version.out > ${NULL} if [ $? = 0 ] then VM_OS="aix" fi grep "bsd" vm_version.out > ${NULL} if [ $? = 0 ] then VM_OS="bsd" fi grep "linux" vm_version.out > ${NULL} if [ $? = 0 ] then VM_OS="linux" fi grep "solaris" vm_version.out > ${NULL} if [ $? = 0 ] then VM_OS="solaris" fi grep "windows" vm_version.out > ${NULL} if [ $? = 0 ] then VM_OS="windows" fi This is potentially a way shell scripts can determine themselves whether or not they are in WSL, or perhaps if we want to remove the need for the -wsl flag to jtreg, we could implement some similar logic. Just a thought though. (I do also wonder why someone implemented this - for cross-compilation? As far as I'm aware, the OS running == JDK target OS in all cases except WSL) On a (somewhat related note), another thing I noticed is that some scripts have logic such as: OS=`uname -s`; # Set classpath separator case "$OS" in Windows* | CYGWIN* ) SEP=";" ;; * ) SEP=":" esac However, the classpath separator would depend on the platform of the JVM under test (; for a Windows JVM under test, : for a Linux JVM under test). I could copy the above logic to detect the JVM under test in the script, but this seems like overkill just to check whether your target JVM is Windows or Linux. So perhaps we could add another environment variable, say WSL_TARGET, which would either be set to "windows" for a Windows WSL target, or empty for Linux WSL target/non-WSL. That way the above logic could be rewritten: OS=`uname -s`; # Set classpath separator case "$OS" in Windows* | CYGWIN* ) SEP=";" ;; * ) if [ "x${WSL_TARGET}" = "xwindows" ]; then SEP=";" else SEP=":" fi esac Of course, this could be implemented using VM_OS using the code above, but it seems to me to be pretty clumsy to have to invoke the JVM in every test just to determine whether you're target JVM is Windows or Linux, given that jtreg knows this information anyways and could just tell the scripts, via an environment variable. Let me know your thoughts/what you prefer. Thanks, -Andrew From jonathan.gibbons at oracle.com Fri Jan 18 23:15:07 2019 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Fri, 18 Jan 2019 15:15:07 -0800 Subject: [PATCH] Fix for EXE_SUFFIX being set for WSL having no effect In-Reply-To: References: <7f886009-4d4a-e429-ce2e-a19dab055ce3@oracle.com> Message-ID: <1ea47c49-83ca-b455-2d55-53d7bb447890@oracle.com> Andrew, This is the place to discuss what we might or might not need in jtreg, but this is not really the place to discuss style issues within the content of tests. You are touching on a number of issues here. 1. With respect to the provision and use of a `-wsl` option, the suggestion to do without it, if possible, was intended for the convenience of the person running jtreg.? If you're running jtreg from the makefiles, it's not a big deal for the makefile to use the option, but not everyone runs tests that way; many folk have private scripts to run jtreg, which can be especially convenient when developing and debugging new tests.?? It is reasonable for jtreg to provide the option, but it would be a bonus if jtreg was clever enough to determine when it was not necessary (e.g. running on Windows, with WSL installed, and with no Cygwin installed.) 2. With respect to the content of tests, for the purposes of this exercise of getting tests to run with WSL, we should be doing the minimum necessary to get the tests to function correctly when run in a WSL world. Based on earlier emails in this thread, it seemed like it was going to be enough to introduce and use `${EXE_SUFFIX}` as needed, which is why it was worth adding that into jtreg. 3. IMO, doing any non-trivial refactoring or rewriting of shell tests is out of scope for this work, and out of scope for this list. For historical reasons different teams have developed different styles for their tests, and anything beyond minimal fixup (like using ${EXE_SUFFIX}) needs to be discussed with the appropriate team. If you need help identifying teams and corresponding email lists, just ask. And, even for minimal fixup, individual teams will want to be involved in the review process for any changes to their tests. 4. An equally if not more important reason not to do work beyond the minimum for shell tests is that, in general, there is an ongoing trend away from using shell tests at all, converting any such tests into Java code. The best way to make a shell test better is to convert it to Java!? See https://openjdk.java.net/jtreg/faq.html#can-i-and-should-i-write-shell-tests But again, such conversion work should be decoupled from WSL work and needs to be coordinated with the relevant component team. -- Jon On 01/18/2019 01:34 PM, Andrew Luo wrote: > Sounds good. My plan is to first try to address most of the failures in tier 1 rather than doing all tests at once. There aren't that many shell tests in tier 1 so it won't be a lot of work wasted if other teams prefer a different approach. Anyways, I want to have something that I know works before I propose it... > > Regarding the previous topic of whether or not we need the -wsl flag - I noticed in some places we are actually checking the type of JVM, in particular in test/hotspot/jtreg/test_env.sh: > > ${TESTJAVA}${FS}bin${FS}java${EXE_SUFFIX} ${TESTOPTS} -Xinternalversion | sed -e 's/[(][^)]*[)]//g' -e 's/ by "[^"]*"//g' > vm_version.out 2>&1 > echo "INT_VERSION=`cat vm_version.out 2>&1`" > > ... > > VM_OS="unknown" > grep "aix" vm_version.out > ${NULL} > if [ $? = 0 ] > then > VM_OS="aix" > fi > grep "bsd" vm_version.out > ${NULL} > if [ $? = 0 ] > then > VM_OS="bsd" > fi > grep "linux" vm_version.out > ${NULL} > if [ $? = 0 ] > then > VM_OS="linux" > fi > grep "solaris" vm_version.out > ${NULL} > if [ $? = 0 ] > then > VM_OS="solaris" > fi > grep "windows" vm_version.out > ${NULL} > if [ $? = 0 ] > then > VM_OS="windows" > fi > > This is potentially a way shell scripts can determine themselves whether or not they are in WSL, or perhaps if we want to remove the need for the -wsl flag to jtreg, we could implement some similar logic. Just a thought though. (I do also wonder why someone implemented this - for cross-compilation? As far as I'm aware, the OS running == JDK target OS in all cases except WSL) > > On a (somewhat related note), another thing I noticed is that some scripts have logic such as: > > OS=`uname -s`; > # Set classpath separator > case "$OS" in > Windows* | CYGWIN* ) > SEP=";" > ;; > > * ) > SEP=":" > esac > > However, the classpath separator would depend on the platform of the JVM under test (; for a Windows JVM under test, : for a Linux JVM under test). I could copy the above logic to detect the JVM under test in the script, but this seems like overkill just to check whether your target JVM is Windows or Linux. So perhaps we could add another environment variable, say WSL_TARGET, which would either be set to "windows" for a Windows WSL target, or empty for Linux WSL target/non-WSL. That way the above logic could be rewritten: > > OS=`uname -s`; > # Set classpath separator > case "$OS" in > Windows* | CYGWIN* ) > SEP=";" > ;; > > * ) > if [ "x${WSL_TARGET}" = "xwindows" ]; then > SEP=";" > else > SEP=":" > fi > esac > > Of course, this could be implemented using VM_OS using the code above, but it seems to me to be pretty clumsy to have to invoke the JVM in every test just to determine whether you're target JVM is Windows or Linux, given that jtreg knows this information anyways and could just tell the scripts, via an environment variable. Let me know your thoughts/what you prefer. > > Thanks, > > -Andrew > From andrewluotechnologies at outlook.com Fri Jan 18 23:33:10 2019 From: andrewluotechnologies at outlook.com (Andrew Luo) Date: Fri, 18 Jan 2019 23:33:10 +0000 Subject: [PATCH] Fix for EXE_SUFFIX being set for WSL having no effect In-Reply-To: <1ea47c49-83ca-b455-2d55-53d7bb447890@oracle.com> References: <7f886009-4d4a-e429-ce2e-a19dab055ce3@oracle.com> <1ea47c49-83ca-b455-2d55-53d7bb447890@oracle.com> Message-ID: I generally agree with the points you made. I wasn't meaning to discuss the test here and propose specific changes to them (that will of course need the relevant team to review), just was giving a few examples of places we could use WSL_TARGET. With regards to converting shell tests to Java code, I do realize that shell tests are, generally speaking, deprecated, however, I don't know how long it will take to fully remove them (in my experience at various companies, we've had "deprecated" features/code retained for sometimes as long as 5-10 years). But if it's not happening anytime soon, I think it's important to have tests work on WSL if we want WSL to be a fully-supported platform for building OpenJDK. Anyways, I can make the jtreg WSL_TARGET changes as well as the shell scripts changes locally, and then seek review from the involved teams, but I wanted to make sure that it's okay with you guys to add WSL_TARGET to jtreg in the first place, if that indeed does solve our issues. Thanks, -Andrew -----Original Message----- From: Jonathan Gibbons Sent: Friday, January 18, 2019 3:15 PM To: Andrew Luo ; code-tools-dev at openjdk.java.net Subject: Re: [PATCH] Fix for EXE_SUFFIX being set for WSL having no effect Andrew, This is the place to discuss what we might or might not need in jtreg, but this is not really the place to discuss style issues within the content of tests. You are touching on a number of issues here. 1. With respect to the provision and use of a `-wsl` option, the suggestion to do without it, if possible, was intended for the convenience of the person running jtreg.? If you're running jtreg from the makefiles, it's not a big deal for the makefile to use the option, but not everyone runs tests that way; many folk have private scripts to run jtreg, which can be especially convenient when developing and debugging new tests.?? It is reasonable for jtreg to provide the option, but it would be a bonus if jtreg was clever enough to determine when it was not necessary (e.g. running on Windows, with WSL installed, and with no Cygwin installed.) 2. With respect to the content of tests, for the purposes of this exercise of getting tests to run with WSL, we should be doing the minimum necessary to get the tests to function correctly when run in a WSL world. Based on earlier emails in this thread, it seemed like it was going to be enough to introduce and use `${EXE_SUFFIX}` as needed, which is why it was worth adding that into jtreg. 3. IMO, doing any non-trivial refactoring or rewriting of shell tests is out of scope for this work, and out of scope for this list. For historical reasons different teams have developed different styles for their tests, and anything beyond minimal fixup (like using ${EXE_SUFFIX}) needs to be discussed with the appropriate team. If you need help identifying teams and corresponding email lists, just ask. And, even for minimal fixup, individual teams will want to be involved in the review process for any changes to their tests. 4. An equally if not more important reason not to do work beyond the minimum for shell tests is that, in general, there is an ongoing trend away from using shell tests at all, converting any such tests into Java code. The best way to make a shell test better is to convert it to Java!? See https://openjdk.java.net/jtreg/faq.html#can-i-and-should-i-write-shell-tests But again, such conversion work should be decoupled from WSL work and needs to be coordinated with the relevant component team. -- Jon On 01/18/2019 01:34 PM, Andrew Luo wrote: > Sounds good. My plan is to first try to address most of the failures in tier 1 rather than doing all tests at once. There aren't that many shell tests in tier 1 so it won't be a lot of work wasted if other teams prefer a different approach. Anyways, I want to have something that I know works before I propose it... > > Regarding the previous topic of whether or not we need the -wsl flag - I noticed in some places we are actually checking the type of JVM, in particular in test/hotspot/jtreg/test_env.sh: > > ${TESTJAVA}${FS}bin${FS}java${EXE_SUFFIX} ${TESTOPTS} -Xinternalversion | sed -e 's/[(][^)]*[)]//g' -e 's/ by "[^"]*"//g' > vm_version.out 2>&1 > echo "INT_VERSION=`cat vm_version.out 2>&1`" > > ... > > VM_OS="unknown" > grep "aix" vm_version.out > ${NULL} > if [ $? = 0 ] > then > VM_OS="aix" > fi > grep "bsd" vm_version.out > ${NULL} > if [ $? = 0 ] > then > VM_OS="bsd" > fi > grep "linux" vm_version.out > ${NULL} > if [ $? = 0 ] > then > VM_OS="linux" > fi > grep "solaris" vm_version.out > ${NULL} > if [ $? = 0 ] > then > VM_OS="solaris" > fi > grep "windows" vm_version.out > ${NULL} > if [ $? = 0 ] > then > VM_OS="windows" > fi > > This is potentially a way shell scripts can determine themselves whether or not they are in WSL, or perhaps if we want to remove the need for the -wsl flag to jtreg, we could implement some similar logic. Just a thought though. (I do also wonder why someone implemented this - for cross-compilation? As far as I'm aware, the OS running == JDK target OS in all cases except WSL) > > On a (somewhat related note), another thing I noticed is that some scripts have logic such as: > > OS=`uname -s`; > # Set classpath separator > case "$OS" in > Windows* | CYGWIN* ) > SEP=";" > ;; > > * ) > SEP=":" > esac > > However, the classpath separator would depend on the platform of the JVM under test (; for a Windows JVM under test, : for a Linux JVM under test). I could copy the above logic to detect the JVM under test in the script, but this seems like overkill just to check whether your target JVM is Windows or Linux. So perhaps we could add another environment variable, say WSL_TARGET, which would either be set to "windows" for a Windows WSL target, or empty for Linux WSL target/non-WSL. That way the above logic could be rewritten: > > OS=`uname -s`; > # Set classpath separator > case "$OS" in > Windows* | CYGWIN* ) > SEP=";" > ;; > > * ) > if [ "x${WSL_TARGET}" = "xwindows" ]; then > SEP=";" > else > SEP=":" > fi > esac > > Of course, this could be implemented using VM_OS using the code above, but it seems to me to be pretty clumsy to have to invoke the JVM in every test just to determine whether you're target JVM is Windows or Linux, given that jtreg knows this information anyways and could just tell the scripts, via an environment variable. Let me know your thoughts/what you prefer. > > Thanks, > > -Andrew > From jonathan.gibbons at oracle.com Fri Jan 18 23:56:11 2019 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Fri, 18 Jan 2019 15:56:11 -0800 Subject: [PATCH] Fix for EXE_SUFFIX being set for WSL having no effect In-Reply-To: References: <7f886009-4d4a-e429-ce2e-a19dab055ce3@oracle.com> <1ea47c49-83ca-b455-2d55-53d7bb447890@oracle.com> Message-ID: Andrew, I guess I don't yet understand why we would need WSL_TARGET. If a shell script has been started by jtreg, can you provide a table describing the observable operational differences (e.g. `os.*` system properties, `uname` output etc) in the following environments: * Real Linux * Windows with Cygwin * Windows with WSL In addition, assuming the existence of EXE_SUFFIX, can you explain why some tests may still not work?? I think you were trying to explain that earlier, but it seemed to be mixed up with a discussion of the jtreg -wsl option, which ought to be a separable issue.?? For tests that use the "case ${OS}" convention, what is the simplest way to modify the code pattern to cope with WSL? Likewise, setting efficiency aside, what is the simplest way to extend the code pattern in test/hotspot/jtreg/test_env.sh for WSL? I guess I'm looking for indications that WSL either behaves the same as an existing case (e.g. it's like Linux) or it can be handled as a new case in its own right, (e.g. like Cygwin). -- Jon On 01/18/2019 03:33 PM, Andrew Luo wrote: > I generally agree with the points you made. I wasn't meaning to discuss the test here and propose specific changes to them (that will of course need the relevant team to review), just was giving a few examples of places we could use WSL_TARGET. > > With regards to converting shell tests to Java code, I do realize that shell tests are, generally speaking, deprecated, however, I don't know how long it will take to fully remove them (in my experience at various companies, we've had "deprecated" features/code retained for sometimes as long as 5-10 years). But if it's not happening anytime soon, I think it's important to have tests work on WSL if we want WSL to be a fully-supported platform for building OpenJDK. > > Anyways, I can make the jtreg WSL_TARGET changes as well as the shell scripts changes locally, and then seek review from the involved teams, but I wanted to make sure that it's okay with you guys to add WSL_TARGET to jtreg in the first place, if that indeed does solve our issues. > > Thanks, > > -Andrew > > -----Original Message----- > From: Jonathan Gibbons > Sent: Friday, January 18, 2019 3:15 PM > To: Andrew Luo ; code-tools-dev at openjdk.java.net > Subject: Re: [PATCH] Fix for EXE_SUFFIX being set for WSL having no effect > > Andrew, > > This is the place to discuss what we might or might not need in jtreg, but this is not really the place to discuss style issues within the content of tests. > > You are touching on a number of issues here. > > 1. With respect to the provision and use of a `-wsl` option, the suggestion to do without it, if possible, was intended for the convenience of the person running jtreg.? If you're running jtreg from the makefiles, it's not a big deal for the makefile to use the option, but not everyone runs tests that way; many folk have private scripts to run jtreg, which can be especially convenient when developing and debugging new tests.?? It is reasonable for jtreg to provide the option, but it would be a bonus if jtreg was clever enough to determine when it was not necessary (e.g. running on Windows, with WSL installed, and with no Cygwin installed.) > > 2. With respect to the content of tests, for the purposes of this exercise of getting tests to run with WSL, we should be doing the minimum necessary to get the tests to function correctly when run in a WSL world. Based on earlier emails in this thread, it seemed like it was going to be enough to introduce and use `${EXE_SUFFIX}` as needed, which is why it was worth adding that into jtreg. > > 3. IMO, doing any non-trivial refactoring or rewriting of shell tests is out of scope for this work, and out of scope for this list. For historical reasons different teams have developed different styles for their tests, and anything beyond minimal fixup (like using > ${EXE_SUFFIX}) needs to be discussed with the appropriate team. If you need help identifying teams and corresponding email lists, just ask. > And, even for minimal fixup, individual teams will want to be involved in the review process for any changes to their tests. > > 4. An equally if not more important reason not to do work beyond the minimum for shell tests is that, in general, there is an ongoing trend away from using shell tests at all, converting any such tests into Java code. The best way to make a shell test better is to convert it to Java!? See https://openjdk.java.net/jtreg/faq.html#can-i-and-should-i-write-shell-tests > But again, such conversion work should be decoupled from WSL work and needs to be coordinated with the relevant component team. > > -- Jon > > > On 01/18/2019 01:34 PM, Andrew Luo wrote: >> Sounds good. My plan is to first try to address most of the failures in tier 1 rather than doing all tests at once. There aren't that many shell tests in tier 1 so it won't be a lot of work wasted if other teams prefer a different approach. Anyways, I want to have something that I know works before I propose it... >> >> Regarding the previous topic of whether or not we need the -wsl flag - I noticed in some places we are actually checking the type of JVM, in particular in test/hotspot/jtreg/test_env.sh: >> >> ${TESTJAVA}${FS}bin${FS}java${EXE_SUFFIX} ${TESTOPTS} -Xinternalversion | sed -e 's/[(][^)]*[)]//g' -e 's/ by "[^"]*"//g' > vm_version.out 2>&1 >> echo "INT_VERSION=`cat vm_version.out 2>&1`" >> >> ... >> >> VM_OS="unknown" >> grep "aix" vm_version.out > ${NULL} >> if [ $? = 0 ] >> then >> VM_OS="aix" >> fi >> grep "bsd" vm_version.out > ${NULL} >> if [ $? = 0 ] >> then >> VM_OS="bsd" >> fi >> grep "linux" vm_version.out > ${NULL} >> if [ $? = 0 ] >> then >> VM_OS="linux" >> fi >> grep "solaris" vm_version.out > ${NULL} >> if [ $? = 0 ] >> then >> VM_OS="solaris" >> fi >> grep "windows" vm_version.out > ${NULL} >> if [ $? = 0 ] >> then >> VM_OS="windows" >> fi >> >> This is potentially a way shell scripts can determine themselves whether or not they are in WSL, or perhaps if we want to remove the need for the -wsl flag to jtreg, we could implement some similar logic. Just a thought though. (I do also wonder why someone implemented this - for cross-compilation? As far as I'm aware, the OS running == JDK target OS in all cases except WSL) >> >> On a (somewhat related note), another thing I noticed is that some scripts have logic such as: >> >> OS=`uname -s`; >> # Set classpath separator >> case "$OS" in >> Windows* | CYGWIN* ) >> SEP=";" >> ;; >> >> * ) >> SEP=":" >> esac >> >> However, the classpath separator would depend on the platform of the JVM under test (; for a Windows JVM under test, : for a Linux JVM under test). I could copy the above logic to detect the JVM under test in the script, but this seems like overkill just to check whether your target JVM is Windows or Linux. So perhaps we could add another environment variable, say WSL_TARGET, which would either be set to "windows" for a Windows WSL target, or empty for Linux WSL target/non-WSL. That way the above logic could be rewritten: >> >> OS=`uname -s`; >> # Set classpath separator >> case "$OS" in >> Windows* | CYGWIN* ) >> SEP=";" >> ;; >> >> * ) >> if [ "x${WSL_TARGET}" = "xwindows" ]; then >> SEP=";" >> else >> SEP=":" >> fi >> esac >> >> Of course, this could be implemented using VM_OS using the code above, but it seems to me to be pretty clumsy to have to invoke the JVM in every test just to determine whether you're target JVM is Windows or Linux, given that jtreg knows this information anyways and could just tell the scripts, via an environment variable. Let me know your thoughts/what you prefer. >> >> Thanks, >> >> -Andrew >> From jonathan.gibbons at oracle.com Sat Jan 19 00:36:16 2019 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Fri, 18 Jan 2019 16:36:16 -0800 Subject: [PATCH] Fix for EXE_SUFFIX being set for WSL having no effect In-Reply-To: References: <7f886009-4d4a-e429-ce2e-a19dab055ce3@oracle.com> <1ea47c49-83ca-b455-2d55-53d7bb447890@oracle.com> Message-ID: <1a9e3468-e6c3-9944-bc18-94c8106fb1a2@oracle.com> On 01/18/2019 03:56 PM, Jonathan Gibbons wrote: > > On a (somewhat related note), another thing I noticed is that some > scripts have logic such as: > > OS=`uname -s`; > # Set classpath separator > case "$OS" in > ????????? Windows* | CYGWIN* ) > ????SEP=";" > ????????? ;; > > ????* ) > ????SEP=":" > esac > > However, the classpath separator would depend on the platform of the > JVM under test (; for a Windows JVM under test, : for a Linux JVM > under test).? I could copy the above logic to detect the JVM under > test in the script, but this seems like overkill just to check whether > your target JVM is Windows or Linux.? So perhaps we could add another > environment variable, say WSL_TARGET, which would either be set to > "windows" for a Windows WSL target, or empty for Linux WSL > target/non-WSL.? That way the above logic could be rewritten: > > OS=`uname -s`; > # Set classpath separator > case "$OS" in > ????????? Windows* | CYGWIN* ) > ????SEP=";" > ????????? ;; > > ????* ) > ????if [ "x${WSL_TARGET}" = "xwindows" ]; then > ??????? SEP=";" > ????else > ??????? SEP=":" > ????fi > esac Just reading this fragment of your message, I think I'm beginning to see what you may be trying to do.? But WSL_TARGET seems like the wrong way to go.? If you need to use WSL_TARGET to fix up the "*" case, that means you executed "case" on the wrong value in the first place. How about the following direction: OS=`if [ "x${EXE_SUFFIX}" = "x.exe" ]; then echo "Windows_WSL" ; else uname -s; fi`; # Set classpath separator case "$OS" in ??? Windows* | CYGWIN* ) ????????? SEP=";" ????????? ;; ????* ) ????SEP=":" esac or finesse it a bit: SYSTEM=`if [ "x${EXE_SUFFIX}" = "x.exe" ]; then echo "WSL" ; else uname -s; fi`; # Set classpath separator case "$SYSTEM" in ??? Windows* | CYGWIN* | WSL ) ????????? SEP=";" ????????? ;; ????* ) ????SEP=":" esac In other words, isn't EXE_SUFFIX already indirectly telling you that its a Windows JVM under test? -- Jon From andrewluotechnologies at outlook.com Sat Jan 19 03:59:59 2019 From: andrewluotechnologies at outlook.com (Andrew Luo) Date: Sat, 19 Jan 2019 03:59:59 +0000 Subject: [PATCH] Fix for EXE_SUFFIX being set for WSL having no effect In-Reply-To: <1a9e3468-e6c3-9944-bc18-94c8106fb1a2@oracle.com> References: <7f886009-4d4a-e429-ce2e-a19dab055ce3@oracle.com> <1ea47c49-83ca-b455-2d55-53d7bb447890@oracle.com> <1a9e3468-e6c3-9944-bc18-94c8106fb1a2@oracle.com> Message-ID: EXE_SUFFIX being .exe guarantees you that it is a Windows JVM under test (WSL currently), but I think EXE_SUFFIX could also be used for Cygwin as well (even though it is not used currently). Or perhaps another emulation layer, if we add support for yet another one in the future. But aside from that, I think from the perspective of clarity, it's not obviously (to someone that has no context) that EXE_SUFFIX == .exe => WSL and not Cygwin, because EXE_SUFFIX could be .exe on all Windows platforms. But if we set WSL_TARGET == windows, it's pretty obvious that that environment variable means WSL. Of course, the possibility of supporting another emulation layer is purely theoretical at this point, so if you have a strong preference, it doesn't matter that much to me. But if I had a choice between the above (EXE_SUFFIX == .exe => WSL_TARGET = windows) and WSL_TARGET == windows => EXE_SUFFIX = .exe, I would prefer the latter, because EXE_SUFFIX doesn't explicitly guarantee you are on WSL. Anyways, I want to hear any other thoughts on this... Thanks, -Andrew -----Original Message----- From: Jonathan Gibbons Sent: Friday, January 18, 2019 4:36 PM To: Andrew Luo ; code-tools-dev at openjdk.java.net Subject: Re: [PATCH] Fix for EXE_SUFFIX being set for WSL having no effect On 01/18/2019 03:56 PM, Jonathan Gibbons wrote: > > On a (somewhat related note), another thing I noticed is that some > scripts have logic such as: > > OS=`uname -s`; > # Set classpath separator > case "$OS" in > ????????? Windows* | CYGWIN* ) > ????SEP=";" > ????????? ;; > > ????* ) > ????SEP=":" > esac > > However, the classpath separator would depend on the platform of the > JVM under test (; for a Windows JVM under test, : for a Linux JVM > under test).? I could copy the above logic to detect the JVM under > test in the script, but this seems like overkill just to check whether > your target JVM is Windows or Linux.? So perhaps we could add another > environment variable, say WSL_TARGET, which would either be set to > "windows" for a Windows WSL target, or empty for Linux WSL > target/non-WSL.? That way the above logic could be rewritten: > > OS=`uname -s`; > # Set classpath separator > case "$OS" in > ????????? Windows* | CYGWIN* ) > ????SEP=";" > ????????? ;; > > ????* ) > ????if [ "x${WSL_TARGET}" = "xwindows" ]; then > ??????? SEP=";" > ????else > ??????? SEP=":" > ????fi > esac Just reading this fragment of your message, I think I'm beginning to see what you may be trying to do.? But WSL_TARGET seems like the wrong way to go.? If you need to use WSL_TARGET to fix up the "*" case, that means you executed "case" on the wrong value in the first place. How about the following direction: OS=`if [ "x${EXE_SUFFIX}" = "x.exe" ]; then echo "Windows_WSL" ; else uname -s; fi`; # Set classpath separator case "$OS" in ??? Windows* | CYGWIN* ) ????????? SEP=";" ????????? ;; ????* ) ????SEP=":" esac or finesse it a bit: SYSTEM=`if [ "x${EXE_SUFFIX}" = "x.exe" ]; then echo "WSL" ; else uname -s; fi`; # Set classpath separator case "$SYSTEM" in ??? Windows* | CYGWIN* | WSL ) ????????? SEP=";" ????????? ;; ????* ) ????SEP=":" esac In other words, isn't EXE_SUFFIX already indirectly telling you that its a Windows JVM under test? -- Jon From jonathan.gibbons at oracle.com Wed Jan 23 00:16:42 2019 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Tue, 22 Jan 2019 16:16:42 -0800 Subject: Supporting WSL in shell tests; was Re: [PATCH] Fix for EXE_SUFFIX being set for WSL having no effect In-Reply-To: References: <7f886009-4d4a-e429-ce2e-a19dab055ce3@oracle.com> <1ea47c49-83ca-b455-2d55-53d7bb447890@oracle.com> <1a9e3468-e6c3-9944-bc18-94c8106fb1a2@oracle.com> Message-ID: cc: quality-discuss bcc: code-tools-dev This is now turning into a discussion on the way we want to modify tests to accommodate WSL, and so I think we should carry on this discussion on the quality-discuss at openjdk.java.net alias. I want to make sure that we get input from the folk that write, debug and maintain these tests. Once we decide on how we would like to modify the coding patter in tests, then we can determine how best to support that from jtreg. Of particular interest, it would help to know how these tests are used these days. It used to be the case that it was important for these shell tests to be able to run standalone, without the use of jtreg at all. If that is still the case, we cannot overly rely on jtreg to set lots of variables. If we only ever run shell tests from jtreg these days, they we could remove a bunch of boilerplate and have jtreg set more info in env variables. The reality is probably somewhere in between, which is why we need test authors to chime in here. -- Jon On 01/18/2019 07:59 PM, Andrew Luo wrote: > EXE_SUFFIX being .exe guarantees you that it is a Windows JVM under test (WSL currently), but I think EXE_SUFFIX could also be used for Cygwin as well (even though it is not used currently). Or perhaps another emulation layer, if we add support for yet another one in the future. But aside from that, I think from the perspective of clarity, it's not obviously (to someone that has no context) that EXE_SUFFIX == .exe => WSL and not Cygwin, because EXE_SUFFIX could be .exe on all Windows platforms. But if we set WSL_TARGET == windows, it's pretty obvious that that environment variable means WSL. Of course, the possibility of supporting another emulation layer is purely theoretical at this point, so if you have a strong preference, it doesn't matter that much to me. > > But if I had a choice between the above (EXE_SUFFIX == .exe => WSL_TARGET = windows) and WSL_TARGET == windows => EXE_SUFFIX = .exe, I would prefer the latter, because EXE_SUFFIX doesn't explicitly guarantee you are on WSL. > > Anyways, I want to hear any other thoughts on this... > > Thanks, > > -Andrew > > -----Original Message----- > From: Jonathan Gibbons > Sent: Friday, January 18, 2019 4:36 PM > To: Andrew Luo ; code-tools-dev at openjdk.java.net > Subject: Re: [PATCH] Fix for EXE_SUFFIX being set for WSL having no effect > > > > On 01/18/2019 03:56 PM, Jonathan Gibbons wrote: >> On a (somewhat related note), another thing I noticed is that some >> scripts have logic such as: >> >> OS=`uname -s`; >> # Set classpath separator >> case "$OS" in >> ????????? Windows* | CYGWIN* ) >> ????SEP=";" >> ????????? ;; >> >> ????* ) >> ????SEP=":" >> esac >> >> However, the classpath separator would depend on the platform of the >> JVM under test (; for a Windows JVM under test, : for a Linux JVM >> under test).? I could copy the above logic to detect the JVM under >> test in the script, but this seems like overkill just to check whether >> your target JVM is Windows or Linux.? So perhaps we could add another >> environment variable, say WSL_TARGET, which would either be set to >> "windows" for a Windows WSL target, or empty for Linux WSL >> target/non-WSL.? That way the above logic could be rewritten: >> >> OS=`uname -s`; >> # Set classpath separator >> case "$OS" in >> ????????? Windows* | CYGWIN* ) >> ????SEP=";" >> ????????? ;; >> >> ????* ) >> ????if [ "x${WSL_TARGET}" = "xwindows" ]; then >> ??????? SEP=";" >> ????else >> ??????? SEP=":" >> ????fi >> esac > Just reading this fragment of your message, I think I'm beginning to see what you may be trying to do.? But WSL_TARGET seems like the wrong way to go.? If you need to use WSL_TARGET to fix up the "*" case, that means you executed "case" on the wrong value in the first place. > > How about the following direction: > > OS=`if [ "x${EXE_SUFFIX}" = "x.exe" ]; then echo "Windows_WSL" ; else uname -s; fi`; # Set classpath separator case "$OS" in > ??? Windows* | CYGWIN* ) > ????????? SEP=";" > ????????? ;; > > ????* ) > ????SEP=":" > esac > > or finesse it a bit: > > SYSTEM=`if [ "x${EXE_SUFFIX}" = "x.exe" ]; then echo "WSL" ; else uname -s; fi`; # Set classpath separator case "$SYSTEM" in > ??? Windows* | CYGWIN* | WSL ) > ????????? SEP=";" > ????????? ;; > > ????* ) > ????SEP=":" > esac > > In other words, isn't EXE_SUFFIX already indirectly telling you that its a Windows JVM under test? > > -- Jon From eric.caspole at oracle.com Wed Jan 23 20:23:40 2019 From: eric.caspole at oracle.com (eric.caspole at oracle.com) Date: Wed, 23 Jan 2019 15:23:40 -0500 Subject: RFR: New JMH for class redefintion Message-ID: Hi everybody, Could I have any reviews or comments on this new JMH to be able to test the work for class redefinition in JDK-8139551. thanks, Eric Related JBS: https://bugs.openjdk.java.net/browse/JDK-8139551 Webrev: http://cr.openjdk.java.net/~ecaspole/redefine_jmh/webrev/