From: Martin Buchholz [mailto:martinrb@google.com] "default unicode charset" is a bizarre concept to me.  Do windows users really have a "regular" charset and a "unicode" charset?  If they do, we probably want to keep that concept out of the public non-Windows API somehow.  Also, the "W" API is implicitly UTF-16, not any other flavor of Unicode.  You probably want to capitalize "unicode".

The defaultUnicodeCharset is not a windows concept. This is the way we chose to ensure that the previous charset selection in the openJDK was unchanged unless windows.UnicodeConsole == true and that all the places that needed to override the charset on windows were covered by one new parameter. We could change the implementation such that the existing sun.jnu.encoding and file.encoding properties were used instead of adding the new file.encoding.unicode property. This might require that users set both on the commandline instead of just one. Other ideas for how to safely modify the charset selection when windows.UnicodeConsole == true are welcome. Thanks, Kirk Developer Microsoft Open Technologies, Inc.

-----Original Message----- From: Anthony Vanelverdinghe [mailto:anthony.vanelverdinghe@gmail.com] Sent: Tuesday, June 23, 2015 12:12 PM

I would like to be able to associate file types with a Java program [1]. Currently, the Java program doesn't receive the argument (i.e. the path to the file that was double-clicked) correctly if it contains Unicode characters. I assume this would be possible by specifying - Dwindows.UnicodeConsole=true in the ftype command string, right?

I have not tested setting a file association. This change is intended to allow the argument to be parsed correctly. There may still be other bugs related to file operations on Unicode paths.

I would also like for System.out and System.err to use the specified Unicode charset. Currently, System.out and System.err use "Windows-1252" (on my system), so they can't readily be used for Unicode output. A quick look at the webrev suggests that this hasn't been addressed yet.

Java.exe should not change the windows codepage. To get this scenario to work, make the association change the codepage and then start java. 'chcp 65001 && java . . .' With this change, if the codepage is correct, then System.out and System.err should support Unicode. Kirk Developer Microsoft Open Technologies, Inc.

-----Original Message----- From: Kumar Srinivasan [mailto:kumar.x.srinivasan@oracle.com]

Hi Kirk,

Thanks for proposing this change.

If you notice all the posix calls are wrapped in JLI_* this gives us the ability to use "W" functions. I almost got it done, several years ago, but we upgraded to VS2010 and my work based on VS2003 keeled over, meanwhile my focus was "shifted" to something else.

main.c: is really envisioned to be a stub compiled by the tool launchers, like java, javac, javah, jar etc. I prefer to see all the heavy logic in this file moved to the platform specific file windows/java_md.*

Heavy logic is moved out of main.c file.

For the reason specified above we need to move fprintf or any naked posix calls to JLI_* indirections.

Posix calls are wrapped in JLI_* functions.

I don't see any tests ? The tests must be written in java and placed in jdk/test/tools/launcher, there is a helper framework TestHelper.java.

A new test for our changes is added.

There are other changes in nio, charsets etc, this will be reviewed by my colleague specializing in that area (Sherman) cc'ed.

Thanks

Kumar

Thank you for the feedback! Valery prepared a new webrev which I placed here: http://cr.openjdk.java.net/~kshoop/8124977/webrev.01 Thanks, Kirk

Hi Kirk, Two questions (1) Why do you need to change the "encoding" used by the java.io.Console class. My understanding is that the console encoding is specifically used to "talk" to the underlying terminal, it should just be the one used by the underlying terminal/console. I don't think the proposed change updates the underlying console encoding (something like chcp) when -Dwindows.UnicodeConsole=true specified, if I read the webrev correctly. Instead, the Java_java_io_Console_encoding() probably need to be updated to return utf8 if the cp == 65001, so if the underlying terminal/console is using cp65001, the java.io.Console should encode/decode in utf8. I would assume the encoding of java.io.Condole should have nothing to do with using GetCommandLIneW() to parse the arguments in unicode in launcher? (2) Why do you need a defaultUnicodeCharset() in Charset class? Seems to me the scope should/could be limited inside LauncherHelper.java? Thanks, -Sherman On 7/8/15 9:50 AM, Kirk Shoop wrote:

...

-----Original Message----- From: Kumar Srinivasan [mailto:kumar.x.srinivasan@oracle.com]

Hi Kirk,

Thanks for proposing this change.

If you notice all the posix calls are wrapped in JLI_* this gives us the ability to use "W" functions. I almost got it done, several years ago, but we upgraded to VS2010 and my work based on VS2003 keeled over, meanwhile my focus was "shifted" to something else.

main.c: is really envisioned to be a stub compiled by the tool launchers, like java, javac, javah, jar etc. I prefer to see all the heavy logic in this file moved to the platform specific file windows/java_md.*

Heavy logic is moved out of main.c file.

...

For the reason specified above we need to move fprintf or any naked posix calls to JLI_* indirections.

Posix calls are wrapped in JLI_* functions.

...

I don't see any tests ? The tests must be written in java and placed in jdk/test/tools/launcher, there is a helper framework TestHelper.java.

A new test for our changes is added.

...

There are other changes in nio, charsets etc, this will be reviewed by my colleague specializing in that area (Sherman) cc'ed.

Thanks

Kumar Thank you for the feedback!

Valery prepared a new webrev which I placed here: http://cr.openjdk.java.net/~kshoop/8124977/webrev.01

Thanks, Kirk

On 07/20/2015 10:22 AM, Kirk Shoop wrote:

So when default system locale differs from the active one, we have different behavior on Linux and Windows. The new options allow a windows user to select the same behavior that one would expect on unix. The switches can certainly be removed, if the compatibility impact is acceptable.

Kirk, on Windows file.encoding is from the user locale and the sun.jnu.encoding is from the system locale setting. sun.jnu.encoding is purely for those text encoding sensitive jnu functiond to communicate with the underlying windows system api, when the system locale and the user locale are set to different value. On unix/linux/osx, these two are always set to the same value. Yes, they might be input/output issue if the encoding used by the console (oem codepage) is not compatible with the encoding used by the "user locale" and you are trying to use System.in/out/err for the input/output to the console. Here is the original CCC request regarding the sun.jnu.encoding, which might provide some background info. http://cr.openjdk.java.net/~sherman/4958170.html If you/we are NOT going to change the encoding used by the underlying console, I don't think we need/should change the "encoding" used by the java.io.Console. As I suggested in my previously email, the Java_java_io_Console_encoding() implementation probably need to update to return utf8 if the cp == 65001 (that was 10 years ago, I'm not sure if the 65001 was really used back then when we wrote this code). My understanding of the issue here is that if you continue to use the "A" version of the API to parse/get the arguments, and try to solve the possible issue triggered by the "incompatibility" of the oem encoding used by the console and the user locale encoding used by the System.in/ out/err, it's fine to define a new system property to specify a preferred encoding for the launcher to use, but this "preferred" encoding should not be used by java.io.Console. But isn't it more reasonable to simply always use the "W" version for this purpose in launcher? -Sherman

(2) The defaultUnicodeCharset() method because it is called from java.io.Console() constructor as well as from LauncherHelper. So its scope should be wider than just LauncherHelper.java.

Kirk and Valery

...

-----Original Message----- From: Xueming Shen [mailto:xueming.shen@oracle.com] Sent: Saturday, July 11, 2015 11:51 AM

Hi Kirk,

Two questions

(1) Why do you need to change the "encoding" used by the java.io.Console class. My understanding is that the console encoding is specifically used to "talk" to the underlying terminal, it should just be the one used by the underlying terminal/console. I don't think the proposed change updates the underlying console encoding (something like chcp) when -Dwindows.UnicodeConsole=true specified, if I read the webrev correctly. Instead, the Java_java_io_Console_encoding() probably need to be updated to return utf8 if the cp == 65001, so if the underlying terminal/console is using cp65001, the java.io.Console should encode/decode in utf8. I would assume the encoding of java.io.Condole should have nothing to do with using GetCommandLIneW() to parse the arguments in unicode in launcher?

(2) Why do you need a defaultUnicodeCharset() in Charset class? Seems to me the scope should/could be limited inside LauncherHelper.java?

Thanks, -Sherman

Hi Kirk, We don't allow any shell tests. There are existing shell based tests, I have not had a chance to port all of them. Your test must be rewritten to use the TestHelper framework, please see the other tests in jdk/test/tools/launcher for patterns. Kumar

...

-----Original Message----- From: Xueming Shen [mailto:xueming.shen@oracle.com] Sent: Monday, July 20, 2015 11:50 AM

On 07/20/2015 10:22 AM, Kirk Shoop wrote:

...

So when default system locale differs from the active one, we have different behavior on Linux and Windows. The new options allow a windows user to select the same behavior that one would expect on unix. The switches can certainly be removed, if the compatibility impact is acceptable.

Kirk, on Windows file.encoding is from the user locale and the sun.jnu.encoding is from the system locale setting. sun.jnu.encoding is purely for those text encoding sensitive jnu functiond to communicate with the underlying windows system api, when the system locale and the user locale are set to different value. On unix/linux/osx, these two are always set to the same value. Yes, they might be input/output issue if the encoding used by the console (oem codepage) is not compatible with the encoding used by the "user locale" and you are trying to use System.in/out/err for the input/output to the console.

Here is the original CCC request regarding the sun.jnu.encoding, which might provide some background info.

http://cr.openjdk.java.net/~sherman/4958170.html

If you/we are NOT going to change the encoding used by the underlying console, I don't think we need/should change the "encoding" used by the java.io.Console. As I suggested in my previously email, the Java_java_io_Console_encoding() implementation probably need to update to return utf8 if the cp == 65001 (that was 10 years ago, I'm not sure if the 65001 was really used back then when we wrote this code). My understanding of the issue here is that if you continue to use the "A" version of the API to parse/get the arguments, and try to solve the possible issue triggered by the "incompatibility" of the oem encoding used by the console and the user locale encoding used by the System.in/ out/err, it's fine to define a new system property to specify a preferred encoding for the launcher to use, but this "preferred" encoding should not be used by java.io.Console. But isn't it more reasonable to simply always use the "W" version for this purpose in launcher?

-Sherman

Thank you for the valuable feedback. We have vastly simplified the original patch. The new webrev is here: http://cr.openjdk.java.net/~kshoop/8124977/webrev.02/

This webrev uses GetCommandLineW on windows to retrieve the UCS16 commandline and also supports the 65001(UTF-8) codepage (set by chcp 65001) so that when -Dsun.jnu.encoding="UTF-8" is supplied the console output (stdout & stderr) will be in UTF8.

There are no new commandline switches.

Please let us know if there is anything else that needs improvement.

Thanks! Kirk and Valery

Hi Kirk, I took another pass over the changes. IMHO, we need to make it easier for people to use this feature, and eliminate these two, atleast externally. "-Dwindows.UnicodeConsole=true", "-Dfile.encoding.unicode=\"UTF-8\"", Can we invent a new command line arg ? we can make it an -X flag, by inspecting the args, we can set the two properties, preferably one ? and also make the decision to use Wide character translations etc. 1. java_md.c a.> + ConvertWinArgsToCommonFormat(cmdLineUtf8, pargc, pargv); + JLI_MemFree(cmdLineUtf8); The free may not happen under error conditions, though we return immediately memory checking tools are likely to complain about memory leak. b.> + if (stricmp(arg, "-Dwindows.UnicodeConsole=true") == 0) { For consistency sake there there is a wrapper for this. 69 #define JLI_StrCaseCmp(p1, p2) stricmp((p1), (p2)) As for the test you can delete the shell test and all you need is the following, I have not tested it, but it should work. ##################### UnicodeCmdVerifier.java ##################### <Copyright deleted for clarity> /* * @test * @bug 8124977 * @summary ensures that the command line argument, passed as Unicode, * matches the desired value. * @compile -XDignore.symbol.file UnicodeCmdVerifier.java * @run main UnicodeCmdVerifier */ public class UnicodeCmdVerifier extends TestHelper { public static void main(String[] args) { if (!isWindows) { // test is applicable only on Windows return; } if (args != null && args.length > 0) { String desired = new String("\u042e\u043d\u0438\u043a\u043e\u0434"); if (args[1].equals(desired)) { System.out.println("The argument matches the desired text: " + args[0]); return; } else { throw new Error("The argument does not match the desired text: " + args[0] + " != " + desired); } return; } TestResult tr = doExec(javaCmd, "-cp", TEST_CLASSES_DIR.getAbsolutePath(), "-Dwindows.UnicodeConsole=true", "-Dfile.encoding.unicode=\"UTF-8\"", "UnicodeCmdVerifier", "testme", "Юникод"); if (!tr.isOK()) { throw new Error("test failed with a non-zero exit"); } } } Thanks Kumar On 7/8/2015 9:50 AM, Kirk Shoop wrote:

...

-----Original Message----- From: Kumar Srinivasan [mailto:kumar.x.srinivasan@oracle.com]

Hi Kirk,

Thanks for proposing this change.

If you notice all the posix calls are wrapped in JLI_* this gives us the ability to use "W" functions. I almost got it done, several years ago, but we upgraded to VS2010 and my work based on VS2003 keeled over, meanwhile my focus was "shifted" to something else.

main.c: is really envisioned to be a stub compiled by the tool launchers, like java, javac, javah, jar etc. I prefer to see all the heavy logic in this file moved to the platform specific file windows/java_md.*

Heavy logic is moved out of main.c file.

...

For the reason specified above we need to move fprintf or any naked posix calls to JLI_* indirections.

Posix calls are wrapped in JLI_* functions.

...

I don't see any tests ? The tests must be written in java and placed in jdk/test/tools/launcher, there is a helper framework TestHelper.java.

A new test for our changes is added.

...

There are other changes in nio, charsets etc, this will be reviewed by my colleague specializing in that area (Sherman) cc'ed.

Thanks

Kumar Thank you for the feedback!

Valery prepared a new webrev which I placed here: http://cr.openjdk.java.net/~kshoop/8124977/webrev.01

Thanks, Kirk

Hi all Here's an updated webrev attempting to take into account the various pieces of feedback we have received: Issue: https://bugs.openjdk.java.net/browse/JDK-8124977 Webrev: http://cr.openjdk.java.net/~kshoop/8124977/webrev.03/ (Vladimir Shcherbakov is now working on this from our side) Looking forward to any other feedback. Thanks -----Original Message----- From: core-libs-dev [mailto:core-libs-dev-bounces@openjdk.java.net] On Behalf Of Kumar Srinivasan Sent: Thursday, June 25, 2015 6:26 AM To: Kirk Shoop (MS OPEN TECH) <Kirk.Shoop@microsoft.com> Cc: Valery Kopylov (Akvelon) <v-valkop@microsoft.com>; core-libs-dev Libs <core-libs-dev@openjdk.java.net> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows Hi Kirk, Thanks for proposing this change. If you notice all the posix calls are wrapped in JLI_* this gives us the ability to use "W" functions. I almost got it done, several years ago, but we upgraded to VS2010 and my work based on VS2003 keeled over, meanwhile my focus was "shifted" to something else. main.c: is really envisioned to be a stub compiled by the tool launchers, like java, javac, javah, jar etc. I prefer to see all the heavy logic in this file moved to the platform specific file windows/java_md.* For the reason specified above we need to move fprintf or any naked posix calls to JLI_* indirections. I don't see any tests ? The tests must be written in java and placed in jdk/test/tools/launcher, there is a helper framework TestHelper.java. There are other changes in nio, charsets etc, this will be reviewed by my colleague specializing in that area (Sherman) cc'ed. Thanks Kumar On 6/22/2015 2:01 PM, Kirk Shoop (MS OPEN TECH) wrote:

Hi,

Issue: https://bugs.openjdk.java.net/browse/JDK-8124977

Webrev: http://cr.openjdk.java.net/~kshoop/8124977/

This webrev intends to address interaction between Windows console and java apps.

Two switches were added that change the behavior of the launcher. The defaults do not change the launcher behavior.

-Dwindows.UnicodeConsole=true - switches on Unicode support in the Windows console. This optional switch causes the launcher to call GetCommandLineW() and parse the arguments in unicode. It also modifies how the codepage for console output is selected.

-Dfile.encoding.unicode="UTF-8" - identifies Unicode charset to use; If not specified, UTF-8 is used by default. Ignored when windows.UnicodeConsole is not set to true. When the first switch is used, this optional switch allows the codepage for console output to be controlled.

I would like to get feedback on the approach here and any additional work that is required solve these particular Unicode issues on Windows.

Kirk

Hi Kumar, Please find updated webreview here: http://cr.openjdk.java.net/~kshoop/8124977/webrev.04/ Thanks, Vladimir. -----Original Message----- From: Kumar Srinivasan [mailto:kumar.x.srinivasan@oracle.com] Sent: Sunday, November 22, 2015 8:14 AM To: Martin Sawicki <marcins@microsoft.com> Cc: Kirk Shoop <Kirk.Shoop@microsoft.com>; Vladimir Shcherbakov <vlashch@microsoft.com>; core-libs-dev Libs <core-libs-dev@openjdk.java.net> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows Hi Martin, et. al., Sorry for not getting back earlier, I am very busy right now with my other large commitments for JDK9. I will sponsor this "enhancement/bug fix" sometime in the new year, meanwhile, there is the changeset [1] which is likely to cause merge conflicts, and perhaps logic issues. Thanks Kumar [1] https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fhg.openjdk.j...

Hi all Here's an updated webrev attempting to take into account the various pieces of feedback we have received:

Issue:

https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fbugs. openjdk.java.net%2fbrowse%2fJDK-8124977&data=01%7c01%7cvlashch%40micro soft.com%7c4d49ae546dba4d29b7be08d2f3589ee1%7c72f988bf86f141af91ab2d7c d011db47%7c1&sdata=FjmfM%2fnPbWB%2fMsUU8uDzAUo3aPu3zOELVsJO%2fsUIq9E%3 d Webrev:

https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr.openj dk.java.net%2f~kshoop%2f8124977%2fwebrev.03%2f&data=01%7C01%7Cvlashch% 40microsoft.com%7C4d49ae546dba4d29b7be08d2f3589ee1%7C72f988bf86f141af9 1ab2d7cd011db47%7C1&sdata=101HBPar2AZ63GJWyubWH0DiKmNI%2bOxknN667BJnWY 0%3d

(Vladimir Shcherbakov is now working on this from our side)

Looking forward to any other feedback. Thanks

-----Original Message----- From: core-libs-dev [mailto:core-libs-dev-bounces@openjdk.java.net] On Behalf Of Kumar Srinivasan Sent: Thursday, June 25, 2015 6:26 AM To: Kirk Shoop (MS OPEN TECH) <Kirk.Shoop@microsoft.com> Cc: Valery Kopylov (Akvelon) <v-valkop@microsoft.com>; core-libs-dev Libs <core-libs-dev@openjdk.java.net> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows

Hi Kirk,

Thanks for proposing this change.

If you notice all the posix calls are wrapped in JLI_* this gives us the ability to use "W" functions. I almost got it done, several years ago, but we upgraded to VS2010 and my work based on VS2003 keeled over, meanwhile my focus was "shifted" to something else.

main.c: is really envisioned to be a stub compiled by the tool launchers, like java, javac, javah, jar etc. I prefer to see all the heavy logic in this file moved to the platform specific file windows/java_md.*

For the reason specified above we need to move fprintf or any naked posix calls to JLI_* indirections.

I don't see any tests ? The tests must be written in java and placed in jdk/test/tools/launcher, there is a helper framework TestHelper.java.

There are other changes in nio, charsets etc, this will be reviewed by my colleague specializing in that area (Sherman) cc'ed.

Thanks

Kumar

On 6/22/2015 2:01 PM, Kirk Shoop (MS OPEN TECH) wrote:

...

Hi,

Issue:

https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fbugs .openjdk.java.net%2fbrowse%2fJDK-8124977&data=01%7c01%7cvlashch%40mic rosoft.com%7c4d49ae546dba4d29b7be08d2f3589ee1%7c72f988bf86f141af91ab2 d7cd011db47%7c1&sdata=FjmfM%2fnPbWB%2fMsUU8uDzAUo3aPu3zOELVsJO%2fsUIq 9E%3d

Webrev:

https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr.open jdk.java.net%2f~kshoop%2f8124977%2f&data=01%7C01%7Cvlashch%40microsof t.com%7C4d49ae546dba4d29b7be08d2f3589ee1%7C72f988bf86f141af91ab2d7cd0 11db47%7C1&sdata=RAA%2b5aIzKtrk5X85oLXKlPzbpSk%2bgJZRI%2b0QSI11B0M%3d

This webrev intends to address interaction between Windows console and java apps.

Two switches were added that change the behavior of the launcher. The defaults do not change the launcher behavior.

-Dwindows.UnicodeConsole=true - switches on Unicode support in the Windows console. This optional switch causes the launcher to call GetCommandLineW() and parse the arguments in unicode. It also modifies how the codepage for console output is selected.

-Dfile.encoding.unicode="UTF-8" - identifies Unicode charset to use; If not specified, UTF-8 is used by default. Ignored when windows.UnicodeConsole is not set to true. When the first switch is used, this optional switch allows the codepage for console output to be controlled.

I would like to get feedback on the approach here and any additional work that is required solve these particular Unicode issues on Windows.

Kirk

Hi, Was on vacation, I started to prepare the patch from webrev.04 for integration. Please note: made some adjustments to your patch to pass jcheck, ie. usage of tabs and space at line endings, and modifications to Copyright dates. Also fixed a minor bug on unix replaced JLI_TRUE with JNI_TRUE. I have attached a patch to for your reference. However, there is a regression test failure on Windows, jdk/test/tools/launcher/I18NTest.java ---Test info---- Executed command: C:\mmm\jdk\bin\javac.exe i18nH▒lloWorld.java ++++Test Output++++ javac: file not found: i18nHélloWorld.java ----End test info----- Have you run all the launcher regression tests with this changeset ? Thanks Kumar

Hi Kumar, just wondering if there are any updates on processing this submission. Thanks!

-----Original Message----- From: Vladimir Shcherbakov Sent: Wednesday, November 25, 2015 2:38 PM To: Kumar Srinivasan <kumar.x.srinivasan@oracle.com>; Martin Sawicki <marcins@microsoft.com> Cc: Kirk Shoop <Kirk.Shoop@microsoft.com>; core-libs-dev Libs <core-libs-dev@openjdk.java.net> Subject: RE: RFR 8124977 cmdline encoding challenges on Windows

Hi Kumar,

Please find updated webreview here: http://cr.openjdk.java.net/~kshoop/8124977/webrev.04/

Thanks, Vladimir.

-----Original Message----- From: Kumar Srinivasan [mailto:kumar.x.srinivasan@oracle.com] Sent: Sunday, November 22, 2015 8:14 AM To: Martin Sawicki <marcins@microsoft.com> Cc: Kirk Shoop <Kirk.Shoop@microsoft.com>; Vladimir Shcherbakov <vlashch@microsoft.com>; core-libs-dev Libs <core-libs-dev@openjdk.java.net> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows

Hi Martin, et. al.,

Sorry for not getting back earlier, I am very busy right now with my other large commitments for JDK9.

I will sponsor this "enhancement/bug fix" sometime in the new year, meanwhile, there is the changeset [1] which is likely to cause merge conflicts, and perhaps logic issues.

Thanks Kumar

[1] https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fhg.openjdk.j...

...

Hi all Here's an updated webrev attempting to take into account the various pieces of feedback we have received:

Issue:

https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fbugs. openjdk.java.net%2fbrowse%2fJDK-8124977&data=01%7c01%7cvlashch%40micro soft.com%7c4d49ae546dba4d29b7be08d2f3589ee1%7c72f988bf86f141af91ab2d7c d011db47%7c1&sdata=FjmfM%2fnPbWB%2fMsUU8uDzAUo3aPu3zOELVsJO%2fsUIq9E%3 d Webrev:

https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr.openj dk.java.net%2f~kshoop%2f8124977%2fwebrev.03%2f&data=01%7C01%7Cvlashch% 40microsoft.com%7C4d49ae546dba4d29b7be08d2f3589ee1%7C72f988bf86f141af9 1ab2d7cd011db47%7C1&sdata=101HBPar2AZ63GJWyubWH0DiKmNI%2bOxknN667BJnWY 0%3d

(Vladimir Shcherbakov is now working on this from our side)

Looking forward to any other feedback. Thanks

-----Original Message----- From: core-libs-dev [mailto:core-libs-dev-bounces@openjdk.java.net] On Behalf Of Kumar Srinivasan Sent: Thursday, June 25, 2015 6:26 AM To: Kirk Shoop (MS OPEN TECH) <Kirk.Shoop@microsoft.com> Cc: Valery Kopylov (Akvelon) <v-valkop@microsoft.com>; core-libs-dev Libs <core-libs-dev@openjdk.java.net> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows

Hi Kirk,

Thanks for proposing this change.

If you notice all the posix calls are wrapped in JLI_* this gives us the ability to use "W" functions. I almost got it done, several years ago, but we upgraded to VS2010 and my work based on VS2003 keeled over, meanwhile my focus was "shifted" to something else.

main.c: is really envisioned to be a stub compiled by the tool launchers, like java, javac, javah, jar etc. I prefer to see all the heavy logic in this file moved to the platform specific file windows/java_md.*

For the reason specified above we need to move fprintf or any naked posix calls to JLI_* indirections.

I don't see any tests ? The tests must be written in java and placed in jdk/test/tools/launcher, there is a helper framework TestHelper.java.

There are other changes in nio, charsets etc, this will be reviewed by my colleague specializing in that area (Sherman) cc'ed.

Thanks

Kumar

On 6/22/2015 2:01 PM, Kirk Shoop (MS OPEN TECH) wrote:

...

Hi,

Issue:

https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fbugs .openjdk.java.net%2fbrowse%2fJDK-8124977&data=01%7c01%7cvlashch%40mic rosoft.com%7c4d49ae546dba4d29b7be08d2f3589ee1%7c72f988bf86f141af91ab2 d7cd011db47%7c1&sdata=FjmfM%2fnPbWB%2fMsUU8uDzAUo3aPu3zOELVsJO%2fsUIq 9E%3d

Webrev:

https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr.open jdk.java.net%2f~kshoop%2f8124977%2f&data=01%7C01%7Cvlashch%40microsof t.com%7C4d49ae546dba4d29b7be08d2f3589ee1%7C72f988bf86f141af91ab2d7cd0 11db47%7C1&sdata=RAA%2b5aIzKtrk5X85oLXKlPzbpSk%2bgJZRI%2b0QSI11B0M%3d

This webrev intends to address interaction between Windows console and java apps.

Two switches were added that change the behavior of the launcher. The defaults do not change the launcher behavior.

-Dwindows.UnicodeConsole=true - switches on Unicode support in the Windows console. This optional switch causes the launcher to call GetCommandLineW() and parse the arguments in unicode. It also modifies how the codepage for console output is selected.

-Dfile.encoding.unicode="UTF-8" - identifies Unicode charset to use; If not specified, UTF-8 is used by default. Ignored when windows.UnicodeConsole is not set to true. When the first switch is used, this optional switch allows the codepage for console output to be controlled.

I would like to get feedback on the approach here and any additional work that is required solve these particular Unicode issues on Windows.

Kirk

Thanks for the feedback. Investigating the regression failure. We'll get back as soon as we figure this out. (and yes, we'll run this through some localized Windows VMs) Cheers -----Original Message----- From: Kumar Srinivasan [mailto:kumar.x.srinivasan@oracle.com] Sent: Tuesday, January 12, 2016 2:35 PM To: Martin Sawicki <marcins@microsoft.com>; Vladimir Shcherbakov <vlashch@microsoft.com> Cc: core-libs-dev Libs <core-libs-dev@openjdk.java.net>; Naoto Sato <naoto.sato@oracle.com> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows Hi Martin, Vladimir, It was suggested that this patch be tested on localized Windows machines and/or trying with the various Windows native encodings, appreciate if you can verify this as well. Thanks Kumar On 1/11/2016 1:10 PM, Kumar Srinivasan wrote:

Hi,

Was on vacation, I started to prepare the patch from webrev.04 for integration. Please note: made some adjustments to your patch to pass jcheck, ie. usage of tabs and space at line endings, and modifications to Copyright dates.

Also fixed a minor bug on unix replaced JLI_TRUE with JNI_TRUE. I have attached a patch to for your reference.

However, there is a regression test failure on Windows, jdk/test/tools/launcher/I18NTest.java

---Test info---- Executed command: C:\mmm\jdk\bin\javac.exe i18nH▒lloWorld.java

++++Test Output++++ javac: file not found: i18nHélloWorld.java ----End test info-----

Have you run all the launcher regression tests with this changeset ?

Thanks Kumar

...

Hi Kumar, just wondering if there are any updates on processing this submission. Thanks!

-----Original Message----- From: Vladimir Shcherbakov Sent: Wednesday, November 25, 2015 2:38 PM To: Kumar Srinivasan <kumar.x.srinivasan@oracle.com>; Martin Sawicki <marcins@microsoft.com> Cc: Kirk Shoop <Kirk.Shoop@microsoft.com>; core-libs-dev Libs <core-libs-dev@openjdk.java.net> Subject: RE: RFR 8124977 cmdline encoding challenges on Windows

Hi Kumar,

Please find updated webreview here: https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr.openjdk.jav...

Thanks, Vladimir.

-----Original Message----- From: Kumar Srinivasan [mailto:kumar.x.srinivasan@oracle.com] Sent: Sunday, November 22, 2015 8:14 AM To: Martin Sawicki <marcins@microsoft.com> Cc: Kirk Shoop <Kirk.Shoop@microsoft.com>; Vladimir Shcherbakov <vlashch@microsoft.com>; core-libs-dev Libs <core-libs-dev@openjdk.java.net> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows

Hi Martin, et. al.,

Sorry for not getting back earlier, I am very busy right now with my other large commitments for JDK9.

I will sponsor this "enhancement/bug fix" sometime in the new year, meanwhile, there is the changeset [1] which is likely to cause merge conflicts, and perhaps logic issues.

Thanks Kumar

[1] https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fhg.openjdk.j...

...

Hi all Here's an updated webrev attempting to take into account the various pieces of feedback we have received:

Issue: https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fbugs. openjdk.java.net%2fbrowse%2fJDK-8124977&data=01%7c01%7cvlashch%40micro soft.com%7c4d49ae546dba4d29b7be08d2f3589ee1%7c72f988bf86f141af91ab2d7c d011db47%7c1&sdata=FjmfM%2fnPbWB%2fMsUU8uDzAUo3aPu3zOELVsJO%2fsUIq9E%3 d Webrev: https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr.openj dk.java.net%2f~kshoop%2f8124977%2fwebrev.03%2f&data=01%7C01%7Cvlashch% 40microsoft.com%7C4d49ae546dba4d29b7be08d2f3589ee1%7C72f988bf86f141af9 1ab2d7cd011db47%7C1&sdata=101HBPar2AZ63GJWyubWH0DiKmNI%2bOxknN667BJnWY 0%3d

(Vladimir Shcherbakov is now working on this from our side)

Looking forward to any other feedback. Thanks

-----Original Message----- From: core-libs-dev [mailto:core-libs-dev-bounces@openjdk.java.net] On Behalf Of Kumar Srinivasan Sent: Thursday, June 25, 2015 6:26 AM To: Kirk Shoop (MS OPEN TECH) <Kirk.Shoop@microsoft.com> Cc: Valery Kopylov (Akvelon) <v-valkop@microsoft.com>; core-libs-dev Libs <core-libs-dev@openjdk.java.net> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows

Hi Kirk,

Thanks for proposing this change.

If you notice all the posix calls are wrapped in JLI_* this gives us the ability to use "W" functions. I almost got it done, several years ago, but we upgraded to VS2010 and my work based on VS2003 keeled over, meanwhile my focus was "shifted" to something else.

main.c: is really envisioned to be a stub compiled by the tool launchers, like java, javac, javah, jar etc. I prefer to see all the heavy logic in this file moved to the platform specific file windows/java_md.*

For the reason specified above we need to move fprintf or any naked posix calls to JLI_* indirections.

I don't see any tests ? The tests must be written in java and placed in jdk/test/tools/launcher, there is a helper framework TestHelper.java.

There are other changes in nio, charsets etc, this will be reviewed by my colleague specializing in that area (Sherman) cc'ed.

Thanks

Kumar

On 6/22/2015 2:01 PM, Kirk Shoop (MS OPEN TECH) wrote:

...

Hi,

Issue: https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fbugs .openjdk.java.net%2fbrowse%2fJDK-8124977&data=01%7c01%7cvlashch%40mic rosoft.com%7c4d49ae546dba4d29b7be08d2f3589ee1%7c72f988bf86f141af91ab2 d7cd011db47%7c1&sdata=FjmfM%2fnPbWB%2fMsUU8uDzAUo3aPu3zOELVsJO%2fsUIq 9E%3d

Webrev: https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr.open jdk.java.net%2f~kshoop%2f8124977%2f&data=01%7C01%7Cvlashch%40microsof t.com%7C4d49ae546dba4d29b7be08d2f3589ee1%7C72f988bf86f141af91ab2d7cd0 11db47%7C1&sdata=RAA%2b5aIzKtrk5X85oLXKlPzbpSk%2bgJZRI%2b0QSI11B0M%3d

This webrev intends to address interaction between Windows console and java apps.

Two switches were added that change the behavior of the launcher. The defaults do not change the launcher behavior.

-Dwindows.UnicodeConsole=true - switches on Unicode support in the Windows console. This optional switch causes the launcher to call GetCommandLineW() and parse the arguments in unicode. It also modifies how the codepage for console output is selected.

-Dfile.encoding.unicode="UTF-8" - identifies Unicode charset to use; If not specified, UTF-8 is used by default. Ignored when windows.UnicodeConsole is not set to true. When the first switch is used, this optional switch allows the codepage for console output to be controlled.

I would like to get feedback on the approach here and any additional work that is required solve these particular Unicode issues on Windows.

Kirk

Hi Naoto, Sherman, can you please take a look. I tested with the jprt build and test all tests pass. Hi Vladimir, et. al., It appears that there has been more simplifications from the previous webrev.04. :-) It would've helped if you highlight the changes you have made from the previous revision, unfortunately this is one of the deficiencies of webrev. There are some inconsistencies in the coding conventions: parse_manifest.c + if (q == 0) return -1; we expect the return to be on the next line. similarly main.c if (0 == q) { I can fix those up. If I were to push this change, who should I attribute the changes to ? ie. in the Contributed-by: line of the commit info ? Please note these have to be email addresses of the contributors. Thanks Kumar

Hi Kumar,

We posted another web review here: http://cr.openjdk.java.net/~kshoop/8124977/webrev.05/

The patch was successfully tested.

Test details: * Regression tests folder: jdk/test/tools/launcher/ * Builds were used: windows-x86_64-normal-server-fastdebug, windows-x86_64-normal-server-release, windows-x86-normal-server-release; * Platforms were used: Windows 7(64 bit), Windows 8.1, Windows Server 2012 R2 DC, Windows 10 ; * System locales were used: English (United States), Persian, Japanese (Japan), Chinese (Traditional, Taiwan), Russian (Russia);

Thanks, Vladimir.

-----Original Message----- From: Martin Sawicki Sent: Thursday, January 14, 2016 11:34 AM To: Kumar Srinivasan <kumar.x.srinivasan@oracle.com>; Vladimir Shcherbakov <vlashch@microsoft.com> Cc: core-libs-dev Libs <core-libs-dev@openjdk.java.net>; Naoto Sato <naoto.sato@oracle.com> Subject: RE: RFR 8124977 cmdline encoding challenges on Windows

Thanks for the feedback. Investigating the regression failure. We'll get back as soon as we figure this out. (and yes, we'll run this through some localized Windows VMs)

Cheers

-----Original Message----- From: Kumar Srinivasan [mailto:kumar.x.srinivasan@oracle.com] Sent: Tuesday, January 12, 2016 2:35 PM To: Martin Sawicki <marcins@microsoft.com>; Vladimir Shcherbakov <vlashch@microsoft.com> Cc: core-libs-dev Libs <core-libs-dev@openjdk.java.net>; Naoto Sato <naoto.sato@oracle.com> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows

Hi Martin, Vladimir,

It was suggested that this patch be tested on localized Windows machines and/or trying with the various Windows native encodings, appreciate if you can verify this as well.

Thanks Kumar

On 1/11/2016 1:10 PM, Kumar Srinivasan wrote:

...

Hi,

Was on vacation, I started to prepare the patch from webrev.04 for integration. Please note: made some adjustments to your patch to pass jcheck, ie. usage of tabs and space at line endings, and modifications to Copyright dates.

Also fixed a minor bug on unix replaced JLI_TRUE with JNI_TRUE. I have attached a patch to for your reference.

However, there is a regression test failure on Windows, jdk/test/tools/launcher/I18NTest.java

---Test info---- Executed command: C:\mmm\jdk\bin\javac.exe i18nH▒lloWorld.java

++++Test Output++++ javac: file not found: i18nHélloWorld.java ----End test info-----

Have you run all the launcher regression tests with this changeset ?

Thanks Kumar

...

Hi Kumar, just wondering if there are any updates on processing this submission. Thanks!

-----Original Message----- From: Vladimir Shcherbakov Sent: Wednesday, November 25, 2015 2:38 PM To: Kumar Srinivasan <kumar.x.srinivasan@oracle.com>; Martin Sawicki <marcins@microsoft.com> Cc: Kirk Shoop <Kirk.Shoop@microsoft.com>; core-libs-dev Libs <core-libs-dev@openjdk.java.net> Subject: RE: RFR 8124977 cmdline encoding challenges on Windows

Hi Kumar,

Please find updated webreview here: https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr.open jdk.java.net%2f~kshoop%2f8124977%2fwebrev.04%2f&data=01%7C01%7Cmarcin s%40microsoft.com%7C13ff309b775c4c019fc308d31ba0c43c%7C72f988bf86f141 af91ab2d7cd011db47%7C1&sdata=3hhbO5mNPyTvtrTb4kCR42zsWGPGzDhqnmjpNfwn bIw%3d

Thanks, Vladimir.

-----Original Message----- From: Kumar Srinivasan [mailto:kumar.x.srinivasan@oracle.com] Sent: Sunday, November 22, 2015 8:14 AM To: Martin Sawicki <marcins@microsoft.com> Cc: Kirk Shoop <Kirk.Shoop@microsoft.com>; Vladimir Shcherbakov <vlashch@microsoft.com>; core-libs-dev Libs <core-libs-dev@openjdk.java.net> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows

Hi Martin, et. al.,

Sorry for not getting back earlier, I am very busy right now with my other large commitments for JDK9.

I will sponsor this "enhancement/bug fix" sometime in the new year, meanwhile, there is the changeset [1] which is likely to cause merge conflicts, and perhaps logic issues.

Thanks Kumar

[1] https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fhg.op enjdk.java.net%2fjdk9%2fdev%2fjdk%2frev%2f3b201a9ef918&data=01%7c01%7 cvlashch%40microsoft.com%7c4d49ae546dba4d29b7be08d2f3589ee1%7c72f988b f86f141af91ab2d7cd011db47%7c1&sdata=I2FKvBn82%2fxhW3D%2fi%2bRWaNOJk7M g4lt2P0sdzLS%2fT9Q%3d

...

Hi all Here's an updated webrev attempting to take into account the various pieces of feedback we have received:

Issue: https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fbugs. openjdk.java.net%2fbrowse%2fJDK-8124977&data=01%7c01%7cvlashch%40mic ro soft.com%7c4d49ae546dba4d29b7be08d2f3589ee1%7c72f988bf86f141af91ab2d 7c d011db47%7c1&sdata=FjmfM%2fnPbWB%2fMsUU8uDzAUo3aPu3zOELVsJO%2fsUIq9E %3 d Webrev: https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr.ope nj dk.java.net%2f~kshoop%2f8124977%2fwebrev.03%2f&data=01%7C01%7Cvlashc h% 40microsoft.com%7C4d49ae546dba4d29b7be08d2f3589ee1%7C72f988bf86f141a f9 1ab2d7cd011db47%7C1&sdata=101HBPar2AZ63GJWyubWH0DiKmNI%2bOxknN667BJn WY 0%3d

(Vladimir Shcherbakov is now working on this from our side)

Looking forward to any other feedback. Thanks

-----Original Message----- From: core-libs-dev [mailto:core-libs-dev-bounces@openjdk.java.net] On Behalf Of Kumar Srinivasan Sent: Thursday, June 25, 2015 6:26 AM To: Kirk Shoop (MS OPEN TECH) <Kirk.Shoop@microsoft.com> Cc: Valery Kopylov (Akvelon) <v-valkop@microsoft.com>; core-libs-dev Libs <core-libs-dev@openjdk.java.net> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows

Hi Kirk,

Thanks for proposing this change.

If you notice all the posix calls are wrapped in JLI_* this gives us the ability to use "W" functions. I almost got it done, several years ago, but we upgraded to VS2010 and my work based on VS2003 keeled over, meanwhile my focus was "shifted" to something else.

main.c: is really envisioned to be a stub compiled by the tool launchers, like java, javac, javah, jar etc. I prefer to see all the heavy logic in this file moved to the platform specific file windows/java_md.*

For the reason specified above we need to move fprintf or any naked posix calls to JLI_* indirections.

I don't see any tests ? The tests must be written in java and placed in jdk/test/tools/launcher, there is a helper framework TestHelper.java.

There are other changes in nio, charsets etc, this will be reviewed by my colleague specializing in that area (Sherman) cc'ed.

Thanks

Kumar

On 6/22/2015 2:01 PM, Kirk Shoop (MS OPEN TECH) wrote:

...

Hi,

Issue: https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fbu gs .openjdk.java.net%2fbrowse%2fJDK-8124977&data=01%7c01%7cvlashch%40m ic rosoft.com%7c4d49ae546dba4d29b7be08d2f3589ee1%7c72f988bf86f141af91a b2 d7cd011db47%7c1&sdata=FjmfM%2fnPbWB%2fMsUU8uDzAUo3aPu3zOELVsJO%2fsU Iq 9E%3d

Webrev: https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr.op en jdk.java.net%2f~kshoop%2f8124977%2f&data=01%7C01%7Cvlashch%40micros of t.com%7C4d49ae546dba4d29b7be08d2f3589ee1%7C72f988bf86f141af91ab2d7c d0 11db47%7C1&sdata=RAA%2b5aIzKtrk5X85oLXKlPzbpSk%2bgJZRI%2b0QSI11B0M% 3d

This webrev intends to address interaction between Windows console and java apps.

Two switches were added that change the behavior of the launcher. The defaults do not change the launcher behavior.

-Dwindows.UnicodeConsole=true - switches on Unicode support in the Windows console. This optional switch causes the launcher to call GetCommandLineW() and parse the arguments in unicode. It also modifies how the codepage for console output is selected.

-Dfile.encoding.unicode="UTF-8" - identifies Unicode charset to use; If not specified, UTF-8 is used by default. Ignored when windows.UnicodeConsole is not set to true. When the first switch is used, this optional switch allows the codepage for console output to be controlled.

I would like to get feedback on the approach here and any additional work that is required solve these particular Unicode issues on Windows.

Kirk

Hi Kumar, (*) Highlights of the changes from the previous revision: 1) Conversion from UTF-16 to UTF-8. a) in webrev 4 was implemented in a standalone function JLI_DecodeArgs - the declaration and definition and usage affected 5 files: src/java.base/share/native/libjli/jli_util.h make/lib/CoreLibraries.gmk src/java.base/unix/native/libjli/java_md_common.c src/java.base/windows/native/libjli/java_md.c src/java.base/share/native/launcher/main.c b) webrew 5: was implemented directly in the main method and affected only 1 file: src/java.base/share/native/launcher/main.c 2) sprops.sun_stdout_encoding property setting this property turned out to be unnecessary to address the problem and all related changed were abolished - this affected 2 files: src/java.base/windows/native/libjava/java_props_md.c src/java.base/windows/native/libjava/Console_md.c 3) Conversion from UTF-8 to UTF-16. to interact from Java launcher with Windows file system - changes to the file were added in webrew 5: src/java.base/share/native/libjli/parse_manifest.c (*) All your comments have been fixed. Our only representative who has the right to post web reviews (Kirk Shoop) is on vacation now and will return on Monday 02/29/2016. We'll post the updated web review (6) as soon as he is back. (*) We should give credit to: - Vladimir Shcherbakov <vlashch@microsoft.com> - Valeriy Kopylov <valery.kopylov@akvelon.com> - Kirk Shoop <Kirk.Shoop@microsoft.com> Thanks, Vladimir. -----Original Message----- From: Kumar Srinivasan [mailto:kumar.x.srinivasan@oracle.com] Sent: Monday, February 22, 2016 8:18 AM To: Vladimir Shcherbakov <vlashch@microsoft.com> Cc: Naoto Sato <naoto.sato@oracle.com>; SHEN,XUEMING <xueming.shen@oracle.com>; core-libs-dev Libs <core-libs-dev@openjdk.java.net> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows Few more comments on the tests: UnicodeCmdTestRun.java: The summary is fine, but it would be good to add a comment explaining what this test really does, random folks look at the tests when a test failure occurs, such a comment should help, similar to what you have in https://na01.safelinks.protection.outlook.com/?url=UnicodeCmdTest.java&data=... UnicodeCmdTest.java: it would be good add an error at line 33, + System.out.println("Error: 0 length argument"); System.exit(1); Thanks Kumar

Hi Naoto, Sherman, can you please take a look. I tested with the jprt build and test all tests pass.

Hi Vladimir, et. al.,

It appears that there has been more simplifications from the previous webrev.04. :-)

It would've helped if you highlight the changes you have made from the previous revision, unfortunately this is one of the deficiencies of webrev.

There are some inconsistencies in the coding conventions:

parse_manifest.c + if (q == 0) return -1;

we expect the return to be on the next line.

similarly main.c

if (0 == q) {

I can fix those up. If I were to push this change, who should I attribute the changes to ? ie. in the Contributed-by: line of the commit info ? Please note these have to be email addresses of the contributors.

Thanks Kumar

...

Hi Kumar,

We posted another web review here: https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr.open jdk.java.net%2f~kshoop%2f8124977%2fwebrev.05%2f&data=01%7C01%7Cvlashc h%40microsoft.com%7C3ec7d35c50cc44070a4108d33ba3d6ea%7C72f988bf86f141 af91ab2d7cd011db47%7C1&sdata=UWa60mcs9nyfqxB1CpJA%2b6h%2f6fPks0aDofCv k9ojssU%3d

The patch was successfully tested.

Test details: * Regression tests folder: jdk/test/tools/launcher/ * Builds were used: windows-x86_64-normal-server-fastdebug, windows-x86_64-normal-server-release, windows-x86-normal-server-release; * Platforms were used: Windows 7(64 bit), Windows 8.1, Windows Server 2012 R2 DC, Windows 10 ; * System locales were used: English (United States), Persian, Japanese (Japan), Chinese (Traditional, Taiwan), Russian (Russia);

Thanks, Vladimir.

-----Original Message----- From: Martin Sawicki Sent: Thursday, January 14, 2016 11:34 AM To: Kumar Srinivasan <kumar.x.srinivasan@oracle.com>; Vladimir Shcherbakov <vlashch@microsoft.com> Cc: core-libs-dev Libs <core-libs-dev@openjdk.java.net>; Naoto Sato <naoto.sato@oracle.com> Subject: RE: RFR 8124977 cmdline encoding challenges on Windows

Thanks for the feedback. Investigating the regression failure. We'll get back as soon as we figure this out. (and yes, we'll run this through some localized Windows VMs)

Cheers

-----Original Message----- From: Kumar Srinivasan [mailto:kumar.x.srinivasan@oracle.com] Sent: Tuesday, January 12, 2016 2:35 PM To: Martin Sawicki <marcins@microsoft.com>; Vladimir Shcherbakov <vlashch@microsoft.com> Cc: core-libs-dev Libs <core-libs-dev@openjdk.java.net>; Naoto Sato <naoto.sato@oracle.com> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows

Hi Martin, Vladimir,

It was suggested that this patch be tested on localized Windows machines and/or trying with the various Windows native encodings, appreciate if you can verify this as well.

Thanks Kumar

On 1/11/2016 1:10 PM, Kumar Srinivasan wrote:

...

Hi,

Was on vacation, I started to prepare the patch from webrev.04 for integration. Please note: made some adjustments to your patch to pass jcheck, ie. usage of tabs and space at line endings, and modifications to Copyright dates.

Also fixed a minor bug on unix replaced JLI_TRUE with JNI_TRUE. I have attached a patch to for your reference.

However, there is a regression test failure on Windows, jdk/test/tools/launcher/I18NTest.java

---Test info---- Executed command: C:\mmm\jdk\bin\javac.exe i18nH▒lloWorld.java

++++Test Output++++ javac: file not found: i18nHélloWorld.java ----End test info-----

Have you run all the launcher regression tests with this changeset ?

Thanks Kumar

...

Hi Kumar, just wondering if there are any updates on processing this submission. Thanks!

-----Original Message----- From: Vladimir Shcherbakov Sent: Wednesday, November 25, 2015 2:38 PM To: Kumar Srinivasan <kumar.x.srinivasan@oracle.com>; Martin Sawicki <marcins@microsoft.com> Cc: Kirk Shoop <Kirk.Shoop@microsoft.com>; core-libs-dev Libs <core-libs-dev@openjdk.java.net> Subject: RE: RFR 8124977 cmdline encoding challenges on Windows

Hi Kumar,

Please find updated webreview here: https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr.op en jdk.java.net%2f~kshoop%2f8124977%2fwebrev.04%2f&data=01%7C01%7Cmarc in s%40microsoft.com%7C13ff309b775c4c019fc308d31ba0c43c%7C72f988bf86f1 41 af91ab2d7cd011db47%7C1&sdata=3hhbO5mNPyTvtrTb4kCR42zsWGPGzDhqnmjpNf wn bIw%3d

Thanks, Vladimir.

-----Original Message----- From: Kumar Srinivasan [mailto:kumar.x.srinivasan@oracle.com] Sent: Sunday, November 22, 2015 8:14 AM To: Martin Sawicki <marcins@microsoft.com> Cc: Kirk Shoop <Kirk.Shoop@microsoft.com>; Vladimir Shcherbakov <vlashch@microsoft.com>; core-libs-dev Libs <core-libs-dev@openjdk.java.net> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows

Hi Martin, et. al.,

Sorry for not getting back earlier, I am very busy right now with my other large commitments for JDK9.

I will sponsor this "enhancement/bug fix" sometime in the new year, meanwhile, there is the changeset [1] which is likely to cause merge conflicts, and perhaps logic issues.

Thanks Kumar

[1] https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fhg. op enjdk.java.net%2fjdk9%2fdev%2fjdk%2frev%2f3b201a9ef918&data=01%7c01 %7 cvlashch%40microsoft.com%7c4d49ae546dba4d29b7be08d2f3589ee1%7c72f98 8b f86f141af91ab2d7cd011db47%7c1&sdata=I2FKvBn82%2fxhW3D%2fi%2bRWaNOJk 7M g4lt2P0sdzLS%2fT9Q%3d

...

Hi all Here's an updated webrev attempting to take into account the various pieces of feedback we have received:

Issue: https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fbugs.

openjdk.java.net%2fbrowse%2fJDK-8124977&data=01%7c01%7cvlashch%40m ic ro soft.com%7c4d49ae546dba4d29b7be08d2f3589ee1%7c72f988bf86f141af91ab 2d 7c d011db47%7c1&sdata=FjmfM%2fnPbWB%2fMsUU8uDzAUo3aPu3zOELVsJO%2fsUIq 9E %3 d Webrev: https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr.o pe nj dk.java.net%2f~kshoop%2f8124977%2fwebrev.03%2f&data=01%7C01%7Cvlas hc h% 40microsoft.com%7C4d49ae546dba4d29b7be08d2f3589ee1%7C72f988bf86f14 1a f9 1ab2d7cd011db47%7C1&sdata=101HBPar2AZ63GJWyubWH0DiKmNI%2bOxknN667B Jn WY 0%3d

(Vladimir Shcherbakov is now working on this from our side)

Looking forward to any other feedback. Thanks

-----Original Message----- From: core-libs-dev [mailto:core-libs-dev-bounces@openjdk.java.net] On Behalf Of Kumar Srinivasan Sent: Thursday, June 25, 2015 6:26 AM To: Kirk Shoop (MS OPEN TECH) <Kirk.Shoop@microsoft.com> Cc: Valery Kopylov (Akvelon) <v-valkop@microsoft.com>; core-libs-dev Libs <core-libs-dev@openjdk.java.net> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows

Hi Kirk,

Thanks for proposing this change.

If you notice all the posix calls are wrapped in JLI_* this gives us the ability to use "W" functions. I almost got it done, several years ago, but we upgraded to VS2010 and my work based on VS2003 keeled over, meanwhile my focus was "shifted" to something else.

main.c: is really envisioned to be a stub compiled by the tool launchers, like java, javac, javah, jar etc. I prefer to see all the heavy logic in this file moved to the platform specific file windows/java_md.*

For the reason specified above we need to move fprintf or any naked posix calls to JLI_* indirections.

I don't see any tests ? The tests must be written in java and placed in jdk/test/tools/launcher, there is a helper framework TestHelper.java.

There are other changes in nio, charsets etc, this will be reviewed by my colleague specializing in that area (Sherman) cc'ed.

Thanks

Kumar

On 6/22/2015 2:01 PM, Kirk Shoop (MS OPEN TECH) wrote:

...

Hi,

Issue: https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2f bu gs .openjdk.java.net%2fbrowse%2fJDK-8124977&data=01%7c01%7cvlashch%4 0m ic rosoft.com%7c4d49ae546dba4d29b7be08d2f3589ee1%7c72f988bf86f141af9 1a b2 d7cd011db47%7c1&sdata=FjmfM%2fnPbWB%2fMsUU8uDzAUo3aPu3zOELVsJO%2f sU Iq 9E%3d

Webrev: https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr. op en jdk.java.net%2f~kshoop%2f8124977%2f&data=01%7C01%7Cvlashch%40micr os of t.com%7C4d49ae546dba4d29b7be08d2f3589ee1%7C72f988bf86f141af91ab2d 7c d0 11db47%7C1&sdata=RAA%2b5aIzKtrk5X85oLXKlPzbpSk%2bgJZRI%2b0QSI11B0 M% 3d

This webrev intends to address interaction between Windows console and java apps.

Two switches were added that change the behavior of the launcher. The defaults do not change the launcher behavior.

-Dwindows.UnicodeConsole=true - switches on Unicode support in the Windows console. This optional switch causes the launcher to call GetCommandLineW() and parse the arguments in unicode. It also modifies how the codepage for console output is selected.

-Dfile.encoding.unicode="UTF-8" - identifies Unicode charset to use; If not specified, UTF-8 is used by default. Ignored when windows.UnicodeConsole is not set to true. When the first switch is used, this optional switch allows the codepage for console output to be controlled.

I would like to get feedback on the approach here and any additional work that is required solve these particular Unicode issues on Windows.

Kirk

Hi Naoto, 1) The system locale determines which code page is used on the system by default on operating systems that use Unicode as their native encoding (all OSes from Windows 2000 to Windows 10) to convert text data from Unicode to code page whenever dealing with legacy non-Unicode applications. Only applications that do not use Unicode as their default character-encoding mechanism are affected by this setting; therefore, applications that are already Unicode-encoded can safely ignore the value and functionality of this setting. 2) The fundamental representation of text in Windows NT-based operating systems is UTF-16, and the WCHAR data type is a UTF-16 code unit. Java launcher, from the other side, uses CHAR as a code unit - so to use UNICODE charset with Java launcher we had to encode entire command line with UTF-8 (convert from UTF-16 to UTF-8). After that step we can state that Java launcher is Unicode-encoded and can safely ignore the value and functionality of the system locale. To let JVM know that we use UTF-8 as a default UNICODE encoding for platform string - we assign the value to sprops.sun_jnu_encoding property (mac osx does the same) instead of reading system locale code page. The main idea of the fix was to change the way of how java and javac works with so called platform string on Windows. Before the fix the platform string was read as ANSI encoded - that's why the system locale code page was very important. The sun.jnu.encoding property is responsible for storing the platform string encoding. On Windows the property could be set with the system locale but the system locale doesn't support (by design) UTF-8 or with -Dsun.jnu.encoding switch, but the switch only works with java not with javac, and the switch was useless for ANSI encoded platform string. Thanks, Vladimir. -----Original Message----- From: Naoto Sato [mailto:naoto.sato@oracle.com] Sent: Tuesday, February 23, 2016 10:47 AM To: Kumar Srinivasan <kumar.x.srinivasan@oracle.com>; Vladimir Shcherbakov <vlashch@microsoft.com>; SHEN,XUEMING <xueming.shen@oracle.com> Cc: Martin Sawicki <marcins@microsoft.com>; core-libs-dev Libs <core-libs-dev@openjdk.java.net> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows Hello, Sorry if this has already been discussed, but this is my first time looking at the fix. In java_props_md.c, sprops.sun_jnu_encoding is now always "UTF-8". Is it always the case? What if the system admin switches the locale for "non-Unicode" applications in the Windows control panel? Naoto On 2/22/16 8:00 AM, Kumar Srinivasan wrote:

Hi Naoto, Sherman, can you please take a look. I tested with the jprt build and test all tests pass.

Hi Vladimir, et. al.,

It appears that there has been more simplifications from the previous webrev.04. :-)

It would've helped if you highlight the changes you have made from the previous revision, unfortunately this is one of the deficiencies of webrev.

There are some inconsistencies in the coding conventions:

parse_manifest.c + if (q == 0) return -1;

we expect the return to be on the next line.

similarly main.c

if (0 == q) {

I can fix those up. If I were to push this change, who should I attribute the changes to ? ie. in the Contributed-by: line of the commit info ? Please note these have to be email addresses of the contributors.

Thanks Kumar

...

Hi Kumar,

We posted another web review here: https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr.open jdk.java.net%2f~kshoop%2f8124977%2fwebrev.05%2f&data=01%7C01%7Cvlashc h%40microsoft.com%7Cf33316507f214e013a4008d33c81c785%7C72f988bf86f141 af91ab2d7cd011db47%7C1&sdata=%2fTQaWH0KGurgvZcdCQRZHSyaftjlMsW5FVc%2f 14Wc5fA%3d

The patch was successfully tested.

Test details: * Regression tests folder: jdk/test/tools/launcher/ * Builds were used: windows-x86_64-normal-server-fastdebug, windows-x86_64-normal-server-release, windows-x86-normal-server-release; * Platforms were used: Windows 7(64 bit), Windows 8.1, Windows Server 2012 R2 DC, Windows 10 ; * System locales were used: English (United States), Persian, Japanese (Japan), Chinese (Traditional, Taiwan), Russian (Russia);

Thanks, Vladimir.

-----Original Message----- From: Martin Sawicki Sent: Thursday, January 14, 2016 11:34 AM To: Kumar Srinivasan <kumar.x.srinivasan@oracle.com>; Vladimir Shcherbakov <vlashch@microsoft.com> Cc: core-libs-dev Libs <core-libs-dev@openjdk.java.net>; Naoto Sato <naoto.sato@oracle.com> Subject: RE: RFR 8124977 cmdline encoding challenges on Windows

Thanks for the feedback. Investigating the regression failure. We'll get back as soon as we figure this out. (and yes, we'll run this through some localized Windows VMs)

Cheers

-----Original Message----- From: Kumar Srinivasan [mailto:kumar.x.srinivasan@oracle.com] Sent: Tuesday, January 12, 2016 2:35 PM To: Martin Sawicki <marcins@microsoft.com>; Vladimir Shcherbakov <vlashch@microsoft.com> Cc: core-libs-dev Libs <core-libs-dev@openjdk.java.net>; Naoto Sato <naoto.sato@oracle.com> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows

Hi Martin, Vladimir,

It was suggested that this patch be tested on localized Windows machines and/or trying with the various Windows native encodings, appreciate if you can verify this as well.

Thanks Kumar

On 1/11/2016 1:10 PM, Kumar Srinivasan wrote:

...

Hi,

Was on vacation, I started to prepare the patch from webrev.04 for integration. Please note: made some adjustments to your patch to pass jcheck, ie. usage of tabs and space at line endings, and modifications to Copyright dates.

Also fixed a minor bug on unix replaced JLI_TRUE with JNI_TRUE. I have attached a patch to for your reference.

However, there is a regression test failure on Windows, jdk/test/tools/launcher/I18NTest.java

---Test info---- Executed command: C:\mmm\jdk\bin\javac.exe i18nH▒lloWorld.java

++++Test Output++++ javac: file not found: i18nHélloWorld.java ----End test info-----

Have you run all the launcher regression tests with this changeset ?

Thanks Kumar

...

Hi Kumar, just wondering if there are any updates on processing this submission. Thanks!

-----Original Message----- From: Vladimir Shcherbakov Sent: Wednesday, November 25, 2015 2:38 PM To: Kumar Srinivasan <kumar.x.srinivasan@oracle.com>; Martin Sawicki <marcins@microsoft.com> Cc: Kirk Shoop <Kirk.Shoop@microsoft.com>; core-libs-dev Libs <core-libs-dev@openjdk.java.net> Subject: RE: RFR 8124977 cmdline encoding challenges on Windows

Hi Kumar,

Please find updated webreview here: https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr.op en jdk.java.net%2f~kshoop%2f8124977%2fwebrev.04%2f&data=01%7C01%7Cmarc in s%40microsoft.com%7C13ff309b775c4c019fc308d31ba0c43c%7C72f988bf86f1 41 af91ab2d7cd011db47%7C1&sdata=3hhbO5mNPyTvtrTb4kCR42zsWGPGzDhqnmjpNf wn bIw%3d

Thanks, Vladimir.

-----Original Message----- From: Kumar Srinivasan [mailto:kumar.x.srinivasan@oracle.com] Sent: Sunday, November 22, 2015 8:14 AM To: Martin Sawicki <marcins@microsoft.com> Cc: Kirk Shoop <Kirk.Shoop@microsoft.com>; Vladimir Shcherbakov <vlashch@microsoft.com>; core-libs-dev Libs <core-libs-dev@openjdk.java.net> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows

Hi Martin, et. al.,

Sorry for not getting back earlier, I am very busy right now with my other large commitments for JDK9.

I will sponsor this "enhancement/bug fix" sometime in the new year, meanwhile, there is the changeset [1] which is likely to cause merge conflicts, and perhaps logic issues.

Thanks Kumar

[1] https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fhg. op enjdk.java.net%2fjdk9%2fdev%2fjdk%2frev%2f3b201a9ef918&data=01%7c01 %7 cvlashch%40microsoft.com%7c4d49ae546dba4d29b7be08d2f3589ee1%7c72f98 8b f86f141af91ab2d7cd011db47%7c1&sdata=I2FKvBn82%2fxhW3D%2fi%2bRWaNOJk 7M g4lt2P0sdzLS%2fT9Q%3d

...

Hi all Here's an updated webrev attempting to take into account the various pieces of feedback we have received:

Issue: https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fbugs. openjdk.java.net%2fbrowse%2fJDK-8124977&data=01%7c01%7cvlashch%40m ic ro soft.com%7c4d49ae546dba4d29b7be08d2f3589ee1%7c72f988bf86f141af91ab 2d 7c d011db47%7c1&sdata=FjmfM%2fnPbWB%2fMsUU8uDzAUo3aPu3zOELVsJO%2fsUIq 9E %3 d Webrev: https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr.o pe nj dk.java.net%2f~kshoop%2f8124977%2fwebrev.03%2f&data=01%7C01%7Cvlas hc h% 40microsoft.com%7C4d49ae546dba4d29b7be08d2f3589ee1%7C72f988bf86f14 1a f9 1ab2d7cd011db47%7C1&sdata=101HBPar2AZ63GJWyubWH0DiKmNI%2bOxknN667B Jn WY 0%3d

(Vladimir Shcherbakov is now working on this from our side)

Looking forward to any other feedback. Thanks

-----Original Message----- From: core-libs-dev [mailto:core-libs-dev-bounces@openjdk.java.net] On Behalf Of Kumar Srinivasan Sent: Thursday, June 25, 2015 6:26 AM To: Kirk Shoop (MS OPEN TECH) <Kirk.Shoop@microsoft.com> Cc: Valery Kopylov (Akvelon) <v-valkop@microsoft.com>; core-libs-dev Libs <core-libs-dev@openjdk.java.net> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows

Hi Kirk,

Thanks for proposing this change.

If you notice all the posix calls are wrapped in JLI_* this gives us the ability to use "W" functions. I almost got it done, several years ago, but we upgraded to VS2010 and my work based on VS2003 keeled over, meanwhile my focus was "shifted" to something else.

main.c: is really envisioned to be a stub compiled by the tool launchers, like java, javac, javah, jar etc. I prefer to see all the heavy logic in this file moved to the platform specific file windows/java_md.*

For the reason specified above we need to move fprintf or any naked posix calls to JLI_* indirections.

I don't see any tests ? The tests must be written in java and placed in jdk/test/tools/launcher, there is a helper framework TestHelper.java.

There are other changes in nio, charsets etc, this will be reviewed by my colleague specializing in that area (Sherman) cc'ed.

Thanks

Kumar

On 6/22/2015 2:01 PM, Kirk Shoop (MS OPEN TECH) wrote:

...

Hi,

Issue: https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2f bu gs .openjdk.java.net%2fbrowse%2fJDK-8124977&data=01%7c01%7cvlashch%4 0m ic rosoft.com%7c4d49ae546dba4d29b7be08d2f3589ee1%7c72f988bf86f141af9 1a b2 d7cd011db47%7c1&sdata=FjmfM%2fnPbWB%2fMsUU8uDzAUo3aPu3zOELVsJO%2f sU Iq 9E%3d

Webrev: https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr. op en jdk.java.net%2f~kshoop%2f8124977%2f&data=01%7C01%7Cvlashch%40micr os of t.com%7C4d49ae546dba4d29b7be08d2f3589ee1%7C72f988bf86f141af91ab2d 7c d0 11db47%7C1&sdata=RAA%2b5aIzKtrk5X85oLXKlPzbpSk%2bgJZRI%2b0QSI11B0 M% 3d

This webrev intends to address interaction between Windows console and java apps.

Two switches were added that change the behavior of the launcher. The defaults do not change the launcher behavior.

-Dwindows.UnicodeConsole=true - switches on Unicode support in the Windows console. This optional switch causes the launcher to call GetCommandLineW() and parse the arguments in unicode. It also modifies how the codepage for console output is selected.

-Dfile.encoding.unicode="UTF-8" - identifies Unicode charset to use; If not specified, UTF-8 is used by default. Ignored when windows.UnicodeConsole is not set to true. When the first switch is used, this optional switch allows the codepage for console output to be controlled.

I would like to get feedback on the approach here and any additional work that is required solve these particular Unicode issues on Windows.

Kirk

Vladimir, sun.jnu.encoding is used by JNU_NewStringPlatform/JNU_GetStringPlatformChars. The JNU_ pair is "widely" used by the various native library code to convert between the jstring and native char*, with the assumption that the "platform encoding" for the native char* is the "default" encoding used by the underlying platform/os APIs that takes char* parameters or return char* values, in case of Windows, it's the code page decided by the system locale. We have migrated certain areas completely to use the "W" version/WChar APIs, such as the java.io, the system properties initialization, but I'm think lots of areas still work on the "A" apis, especially I think the "char*" interface between the jvm and the libraries is still the the "ansi" codepage, not the utf8. Those work on utf8 have their names explicitly named as "xyzUTF" or similar. For example, the "java_home_dir" path used in libjava/TimeZone.c/getSystemTimeZoneID/ TimeZone_md.c/findJavaTZ_md is encoded frm jstring java_home to char* via JNU_GetSTringPlatformChars. Simply change/hardcode the jnu_sun.encoding to utf8 probably will cause the timezone code stop to work if the java_home_dir path has some non-ascii characters in it (the jdk/jre is installed in a Japanese/Chinese directory, for example). A quick "grep" indicates java.desktop/windows/native/libawt/windows package has a heavily usage of the JNU_ pair as well. I'm not sure if this awt implementation is still being used though :-) Before we clear all these internal "StringPlatform" use cases (I'm not sure if they are also used by external), I don't think we can simply set the sun.jnu.encoding to utf8, though it's very attractive. Thanks, -Sherman On 2/23/16 4:34 PM, Naoto Sato wrote:

Hi Vladimir,

I think it would work fine with the Java launcher, but what about other areas, which may rely on the native encodings? Java runtime is in itself a "non-Unicode" application, so still there may be the area affected by hardcoding "UTF-8" as the native encoding. Have you checked in such cases? Sherman, will you comment on this too?

Naoto

On 2/23/16 2:12 PM, Vladimir Shcherbakov wrote:

...

Hi Naoto,

1) The system locale determines which code page is used on the system by default on operating systems that use Unicode as their native encoding (all OSes from Windows 2000 to Windows 10) to convert text data from Unicode to code page whenever dealing with legacy non-Unicode applications. Only applications that do not use Unicode as their default character-encoding mechanism are affected by this setting; therefore, applications that are already Unicode-encoded can safely ignore the value and functionality of this setting.

2) The fundamental representation of text in Windows NT-based operating systems is UTF-16, and the WCHAR data type is a UTF-16 code unit. Java launcher, from the other side, uses CHAR as a code unit - so to use UNICODE charset with Java launcher we had to encode entire command line with UTF-8 (convert from UTF-16 to UTF-8). After that step we can state that Java launcher is Unicode-encoded and can safely ignore the value and functionality of the system locale. To let JVM know that we use UTF-8 as a default UNICODE encoding for platform string - we assign the value to sprops.sun_jnu_encoding property (mac osx does the same) instead of reading system locale code page.

The main idea of the fix was to change the way of how java and javac works with so called platform string on Windows. Before the fix the platform string was read as ANSI encoded - that's why the system locale code page was very important. The sun.jnu.encoding property is responsible for storing the platform string encoding. On Windows the property could be set with the system locale but the system locale doesn't support (by design) UTF-8 or with -Dsun.jnu.encoding switch, but the switch only works with java not with javac, and the switch was useless for ANSI encoded platform string.

Thanks, Vladimir.

-----Original Message----- From: Naoto Sato [mailto:naoto.sato@oracle.com] Sent: Tuesday, February 23, 2016 10:47 AM To: Kumar Srinivasan <kumar.x.srinivasan@oracle.com>; Vladimir Shcherbakov <vlashch@microsoft.com>; SHEN,XUEMING <xueming.shen@oracle.com> Cc: Martin Sawicki <marcins@microsoft.com>; core-libs-dev Libs <core-libs-dev@openjdk.java.net> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows

Hello,

Sorry if this has already been discussed, but this is my first time looking at the fix. In java_props_md.c, sprops.sun_jnu_encoding is now always "UTF-8". Is it always the case? What if the system admin switches the locale for "non-Unicode" applications in the Windows control panel?

Naoto

On 2/22/16 8:00 AM, Kumar Srinivasan wrote:

...

Hi Naoto, Sherman, can you please take a look. I tested with the jprt build and test all tests pass.

Hi Vladimir, et. al.,

It appears that there has been more simplifications from the previous webrev.04. :-)

It would've helped if you highlight the changes you have made from the previous revision, unfortunately this is one of the deficiencies of webrev.

There are some inconsistencies in the coding conventions:

parse_manifest.c + if (q == 0) return -1;

we expect the return to be on the next line.

similarly main.c

if (0 == q) {

I can fix those up. If I were to push this change, who should I attribute the changes to ? ie. in the Contributed-by: line of the commit info ? Please note these have to be email addresses of the contributors.

Thanks Kumar

...

Hi Kumar,

We posted another web review here: https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr.open jdk.java.net%2f~kshoop%2f8124977%2fwebrev.05%2f&data=01%7C01%7Cvlashc h%40microsoft.com%7Cf33316507f214e013a4008d33c81c785%7C72f988bf86f141 af91ab2d7cd011db47%7C1&sdata=%2fTQaWH0KGurgvZcdCQRZHSyaftjlMsW5FVc%2f 14Wc5fA%3d

The patch was successfully tested.

Test details: * Regression tests folder: jdk/test/tools/launcher/ * Builds were used: windows-x86_64-normal-server-fastdebug, windows-x86_64-normal-server-release, windows-x86-normal-server-release; * Platforms were used: Windows 7(64 bit), Windows 8.1, Windows Server 2012 R2 DC, Windows 10 ; * System locales were used: English (United States), Persian, Japanese (Japan), Chinese (Traditional, Taiwan), Russian (Russia);

Thanks, Vladimir.

-----Original Message----- From: Martin Sawicki Sent: Thursday, January 14, 2016 11:34 AM To: Kumar Srinivasan <kumar.x.srinivasan@oracle.com>; Vladimir Shcherbakov <vlashch@microsoft.com> Cc: core-libs-dev Libs <core-libs-dev@openjdk.java.net>; Naoto Sato <naoto.sato@oracle.com> Subject: RE: RFR 8124977 cmdline encoding challenges on Windows

Thanks for the feedback. Investigating the regression failure. We'll get back as soon as we figure this out. (and yes, we'll run this through some localized Windows VMs)

Cheers

-----Original Message----- From: Kumar Srinivasan [mailto:kumar.x.srinivasan@oracle.com] Sent: Tuesday, January 12, 2016 2:35 PM To: Martin Sawicki <marcins@microsoft.com>; Vladimir Shcherbakov <vlashch@microsoft.com> Cc: core-libs-dev Libs <core-libs-dev@openjdk.java.net>; Naoto Sato <naoto.sato@oracle.com> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows

Hi Martin, Vladimir,

It was suggested that this patch be tested on localized Windows machines and/or trying with the various Windows native encodings, appreciate if you can verify this as well.

Thanks Kumar

On 1/11/2016 1:10 PM, Kumar Srinivasan wrote:

...

Hi,

Was on vacation, I started to prepare the patch from webrev.04 for integration. Please note: made some adjustments to your patch to pass jcheck, ie. usage of tabs and space at line endings, and modifications to Copyright dates.

Also fixed a minor bug on unix replaced JLI_TRUE with JNI_TRUE. I have attached a patch to for your reference.

However, there is a regression test failure on Windows, jdk/test/tools/launcher/I18NTest.java

---Test info---- Executed command: C:\mmm\jdk\bin\javac.exe i18nH▒lloWorld.java

++++Test Output++++ javac: file not found: i18nHélloWorld.java ----End test info-----

Have you run all the launcher regression tests with this changeset ?

Thanks Kumar

...

Hi Kumar, just wondering if there are any updates on processing this submission. Thanks!

-----Original Message----- From: Vladimir Shcherbakov Sent: Wednesday, November 25, 2015 2:38 PM To: Kumar Srinivasan <kumar.x.srinivasan@oracle.com>; Martin Sawicki <marcins@microsoft.com> Cc: Kirk Shoop <Kirk.Shoop@microsoft.com>; core-libs-dev Libs <core-libs-dev@openjdk.java.net> Subject: RE: RFR 8124977 cmdline encoding challenges on Windows

Hi Kumar,

Please find updated webreview here: https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr.op en jdk.java.net%2f~kshoop%2f8124977%2fwebrev.04%2f&data=01%7C01%7Cmarc in s%40microsoft.com%7C13ff309b775c4c019fc308d31ba0c43c%7C72f988bf86f1 41 af91ab2d7cd011db47%7C1&sdata=3hhbO5mNPyTvtrTb4kCR42zsWGPGzDhqnmjpNf wn bIw%3d

Thanks, Vladimir.

-----Original Message----- From: Kumar Srinivasan [mailto:kumar.x.srinivasan@oracle.com] Sent: Sunday, November 22, 2015 8:14 AM To: Martin Sawicki <marcins@microsoft.com> Cc: Kirk Shoop <Kirk.Shoop@microsoft.com>; Vladimir Shcherbakov <vlashch@microsoft.com>; core-libs-dev Libs <core-libs-dev@openjdk.java.net> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows

Hi Martin, et. al.,

Sorry for not getting back earlier, I am very busy right now with my other large commitments for JDK9.

I will sponsor this "enhancement/bug fix" sometime in the new year, meanwhile, there is the changeset [1] which is likely to cause merge conflicts, and perhaps logic issues.

Thanks Kumar

[1] https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fhg. op enjdk.java.net%2fjdk9%2fdev%2fjdk%2frev%2f3b201a9ef918&data=01%7c01 %7 cvlashch%40microsoft.com%7c4d49ae546dba4d29b7be08d2f3589ee1%7c72f98 8b f86f141af91ab2d7cd011db47%7c1&sdata=I2FKvBn82%2fxhW3D%2fi%2bRWaNOJk 7M g4lt2P0sdzLS%2fT9Q%3d > Hi all > Here's an updated webrev attempting to take into account the > various pieces of feedback we have received: > > Issue: > https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fbugs. > > openjdk.java.net%2fbrowse%2fJDK-8124977&data=01%7c01%7cvlashch%40m > ic > ro > soft.com%7c4d49ae546dba4d29b7be08d2f3589ee1%7c72f988bf86f141af91ab > 2d > 7c > d011db47%7c1&sdata=FjmfM%2fnPbWB%2fMsUU8uDzAUo3aPu3zOELVsJO%2fsUIq > 9E > %3 > d > Webrev: > https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr.o > pe > nj > dk.java.net%2f~kshoop%2f8124977%2fwebrev.03%2f&data=01%7C01%7Cvlas > hc > h% > 40microsoft.com%7C4d49ae546dba4d29b7be08d2f3589ee1%7C72f988bf86f14 > 1a > f9 > 1ab2d7cd011db47%7C1&sdata=101HBPar2AZ63GJWyubWH0DiKmNI%2bOxknN667B > Jn > WY > 0%3d > > (Vladimir Shcherbakov is now working on this from our side) > > Looking forward to any other feedback. > Thanks > > -----Original Message----- > From: core-libs-dev > [mailto:core-libs-dev-bounces@openjdk.java.net] > On Behalf Of Kumar Srinivasan > Sent: Thursday, June 25, 2015 6:26 AM > To: Kirk Shoop (MS OPEN TECH) <Kirk.Shoop@microsoft.com> > Cc: Valery Kopylov (Akvelon) <v-valkop@microsoft.com>; > core-libs-dev Libs <core-libs-dev@openjdk.java.net> > Subject: Re: RFR 8124977 cmdline encoding challenges on Windows > > Hi Kirk, > > Thanks for proposing this change. > > If you notice all the posix calls are wrapped in JLI_* this gives > us the ability to use "W" functions. I almost got it done, > several years ago, but we upgraded to VS2010 and my work based on > VS2003 keeled over, meanwhile my focus was "shifted" to > something else. > > main.c: is really envisioned to be a stub compiled by the tool > launchers, like java, javac, javah, jar etc. I prefer to see all > the heavy logic in this file moved to the platform specific file > windows/java_md.* > > For the reason specified above we need to move fprintf or any > naked posix calls to JLI_* indirections. > > I don't see any tests ? The tests must be written in java and > placed in jdk/test/tools/launcher, there is a helper framework > TestHelper.java. > > There are other changes in nio, charsets etc, this will be > reviewed by my colleague specializing in that area (Sherman) cc'ed. > > > Thanks > > Kumar > > > > > > > On 6/22/2015 2:01 PM, Kirk Shoop (MS OPEN TECH) wrote: >> Hi, >> >> Issue: >> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2f >> bu >> gs >> .openjdk.java.net%2fbrowse%2fJDK-8124977&data=01%7c01%7cvlashch%4 >> 0m >> ic >> rosoft.com%7c4d49ae546dba4d29b7be08d2f3589ee1%7c72f988bf86f141af9 >> 1a >> b2 >> d7cd011db47%7c1&sdata=FjmfM%2fnPbWB%2fMsUU8uDzAUo3aPu3zOELVsJO%2f >> sU >> Iq >> 9E%3d >> >> Webrev: >> https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr. >> op >> en >> jdk.java.net%2f~kshoop%2f8124977%2f&data=01%7C01%7Cvlashch%40micr >> os >> of >> t.com%7C4d49ae546dba4d29b7be08d2f3589ee1%7C72f988bf86f141af91ab2d >> 7c >> d0 >> 11db47%7C1&sdata=RAA%2b5aIzKtrk5X85oLXKlPzbpSk%2bgJZRI%2b0QSI11B0 >> M% >> 3d >> >> This webrev intends to address interaction between Windows >> console and java apps. >> >> Two switches were added that change the behavior of the launcher. >> The defaults do not change the launcher behavior. >> >> -Dwindows.UnicodeConsole=true - switches on Unicode support >> in the Windows console. This optional switch causes the launcher >> to call GetCommandLineW() and parse the arguments in unicode. It >> also modifies how the codepage for console output is selected. >> >> -Dfile.encoding.unicode="UTF-8" - identifies Unicode >> charset to use; If not specified, UTF-8 is used by default. >> Ignored when windows.UnicodeConsole is not set to true. When the >> first switch is used, this optional switch allows the codepage >> for console output to be controlled. >> >> I would like to get feedback on the approach here and any >> additional work that is required solve these particular Unicode >> issues on Windows. >> >> Kirk

On 2/23/16 9:52 PM, Vladimir Shcherbakov wrote:

Hi Sherman,

1) If you can point out the regression test cases that are compromised by the fix - it would be very helpful;

I don't have a specific regression test for now. Guess running a "old" awt app with Japanese/Chinese characters as menu item might help show the issue? if I read those awt lines correctly.

2) From my understanding you can change default encoding by starting java with -Dsun.jnu.encoding=UTF-8 - this is well known feature that never caused problems (javac doesn't have such a switch );

I have been telling people for decade -Dfile.encoding is not a "supported" usage/feature :-) as it does cause "inconsistent" behavior on different platforms with different use scenario. And then the sun.jnu.encoding, definitely with no intention to be specified via -D. That's a pure contract between the java runtime and the underlying os on how the string/text should be encoded when using those platform APIs. I think I had forwarded the internal CCC doc for sun.jnu.encoding a while back, no -Dsun.jnu.encoding=XYZ please :-)

3) If you state that java is non-Unicode on Windows by nature - the issue JDK-8124977 is a feature not a bug :)

Ideally we should run the java runtime as a unicode app. launcher is not a big issue. The concern is the interface with jvm for those "char*". Sherman

Thanks, Vladimir.

-----Original Message----- From: Xueming Shen [mailto:xueming.shen@oracle.com] Sent: Tuesday, February 23, 2016 8:54 PM To: Vladimir Shcherbakov <vlashch@microsoft.com> Cc: Naoto Sato <naoto.sato@oracle.com>; Kumar Srinivasan <kumar.x.srinivasan@oracle.com>; Martin Sawicki <marcins@microsoft.com>; core-libs-dev Libs <core-libs-dev@openjdk.java.net> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows

Vladimir,

sun.jnu.encoding is used by JNU_NewStringPlatform/JNU_GetStringPlatformChars. The JNU_ pair is "widely" used by the various native library code to convert between the jstring and native char*, with the assumption that the "platform encoding" for the native char* is the "default" encoding used by the underlying platform/os APIs that takes char* parameters or return char* values, in case of Windows, it's the code page decided by the system locale. We have migrated certain areas completely to use the "W" version/WChar APIs, such as the https://na01.safelinks.protection.outlook.com/?url=java.io&data=01%7c01%7cvl..., the system properties initialization, but I'm think lots of areas still work on the "A" apis, especially I think the "char*" interface between the jvm and the libraries is still the the "ansi" codepage, not the utf8. Those work on utf8 have their names explicitly named as "xyzUTF" or similar.

For example, the "java_home_dir" path used in libjava/TimeZone.c/getSystemTimeZoneID/ TimeZone_md.c/findJavaTZ_md is encoded frm jstring java_home to char* via JNU_GetSTringPlatformChars. Simply change/hardcode the jnu_sun.encoding to utf8 probably will cause the timezone code stop to work if the java_home_dir path has some non-ascii characters in it (the jdk/jre is installed in a Japanese/Chinese directory, for example).

A quick "grep" indicates java.desktop/windows/native/libawt/windows package has a heavily usage of the JNU_ pair as well. I'm not sure if this awt implementation is still being used though :-)

Before we clear all these internal "StringPlatform" use cases (I'm not sure if they are also used by external), I don't think we can simply set the sun.jnu.encoding to utf8, though it's very attractive.

Thanks, -Sherman

On 2/23/16 4:34 PM, Naoto Sato wrote:

...

Hi Vladimir,

I think it would work fine with the Java launcher, but what about other areas, which may rely on the native encodings? Java runtime is in itself a "non-Unicode" application, so still there may be the area affected by hardcoding "UTF-8" as the native encoding. Have you checked in such cases? Sherman, will you comment on this too?

Naoto

On 2/23/16 2:12 PM, Vladimir Shcherbakov wrote:

...

Hi Naoto,

1) The system locale determines which code page is used on the system by default on operating systems that use Unicode as their native encoding (all OSes from Windows 2000 to Windows 10) to convert text data from Unicode to code page whenever dealing with legacy non-Unicode applications. Only applications that do not use Unicode as their default character-encoding mechanism are affected by this setting; therefore, applications that are already Unicode-encoded can safely ignore the value and functionality of this setting.

2) The fundamental representation of text in Windows NT-based operating systems is UTF-16, and the WCHAR data type is a UTF-16 code unit. Java launcher, from the other side, uses CHAR as a code unit - so to use UNICODE charset with Java launcher we had to encode entire command line with UTF-8 (convert from UTF-16 to UTF-8). After that step we can state that Java launcher is Unicode-encoded and can safely ignore the value and functionality of the system locale. To let JVM know that we use UTF-8 as a default UNICODE encoding for platform string - we assign the value to sprops.sun_jnu_encoding property (mac osx does the same) instead of reading system locale code page.

The main idea of the fix was to change the way of how java and javac works with so called platform string on Windows. Before the fix the platform string was read as ANSI encoded - that's why the system locale code page was very important. The sun.jnu.encoding property is responsible for storing the platform string encoding. On Windows the property could be set with the system locale but the system locale doesn't support (by design) UTF-8 or with -Dsun.jnu.encoding switch, but the switch only works with java not with javac, and the switch was useless for ANSI encoded platform string.

Thanks, Vladimir.

-----Original Message----- From: Naoto Sato [mailto:naoto.sato@oracle.com] Sent: Tuesday, February 23, 2016 10:47 AM To: Kumar Srinivasan <kumar.x.srinivasan@oracle.com>; Vladimir Shcherbakov <vlashch@microsoft.com>; SHEN,XUEMING <xueming.shen@oracle.com> Cc: Martin Sawicki <marcins@microsoft.com>; core-libs-dev Libs <core-libs-dev@openjdk.java.net> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows

Hello,

Sorry if this has already been discussed, but this is my first time looking at the fix. In java_props_md.c, sprops.sun_jnu_encoding is now always "UTF-8". Is it always the case? What if the system admin switches the locale for "non-Unicode" applications in the Windows control panel?

Naoto

On 2/22/16 8:00 AM, Kumar Srinivasan wrote:

...

Hi Naoto, Sherman, can you please take a look. I tested with the jprt build and test all tests pass.

Hi Vladimir, et. al.,

It appears that there has been more simplifications from the previous webrev.04. :-)

It would've helped if you highlight the changes you have made from the previous revision, unfortunately this is one of the deficiencies of webrev.

There are some inconsistencies in the coding conventions:

parse_manifest.c + if (q == 0) return -1;

we expect the return to be on the next line.

similarly main.c

if (0 == q) {

I can fix those up. If I were to push this change, who should I attribute the changes to ? ie. in the Contributed-by: line of the commit info ? Please note these have to be email addresses of the contributors.

Thanks Kumar

...

Hi Kumar,

We posted another web review here: https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr.op en jdk.java.net%2f~kshoop%2f8124977%2fwebrev.05%2f&data=01%7C01%7Cvlas hc h%40microsoft.com%7Cf33316507f214e013a4008d33c81c785%7C72f988bf86f1 41 af91ab2d7cd011db47%7C1&sdata=%2fTQaWH0KGurgvZcdCQRZHSyaftjlMsW5FVc% 2f 14Wc5fA%3d

The patch was successfully tested.

Test details: * Regression tests folder: jdk/test/tools/launcher/ * Builds were used: windows-x86_64-normal-server-fastdebug, windows-x86_64-normal-server-release, windows-x86-normal-server-release; * Platforms were used: Windows 7(64 bit), Windows 8.1, Windows Server 2012 R2 DC, Windows 10 ; * System locales were used: English (United States), Persian, Japanese (Japan), Chinese (Traditional, Taiwan), Russian (Russia);

Thanks, Vladimir.

-----Original Message----- From: Martin Sawicki Sent: Thursday, January 14, 2016 11:34 AM To: Kumar Srinivasan <kumar.x.srinivasan@oracle.com>; Vladimir Shcherbakov <vlashch@microsoft.com> Cc: core-libs-dev Libs <core-libs-dev@openjdk.java.net>; Naoto Sato <naoto.sato@oracle.com> Subject: RE: RFR 8124977 cmdline encoding challenges on Windows

Thanks for the feedback. Investigating the regression failure. We'll get back as soon as we figure this out. (and yes, we'll run this through some localized Windows VMs)

Cheers

-----Original Message----- From: Kumar Srinivasan [mailto:kumar.x.srinivasan@oracle.com] Sent: Tuesday, January 12, 2016 2:35 PM To: Martin Sawicki <marcins@microsoft.com>; Vladimir Shcherbakov <vlashch@microsoft.com> Cc: core-libs-dev Libs <core-libs-dev@openjdk.java.net>; Naoto Sato <naoto.sato@oracle.com> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows

Hi Martin, Vladimir,

It was suggested that this patch be tested on localized Windows machines and/or trying with the various Windows native encodings, appreciate if you can verify this as well.

Thanks Kumar

On 1/11/2016 1:10 PM, Kumar Srinivasan wrote:

...

Hi,

Was on vacation, I started to prepare the patch from webrev.04 for integration. Please note: made some adjustments to your patch to pass jcheck, ie. usage of tabs and space at line endings, and modifications to Copyright dates.

Also fixed a minor bug on unix replaced JLI_TRUE with JNI_TRUE. I have attached a patch to for your reference.

However, there is a regression test failure on Windows, jdk/test/tools/launcher/I18NTest.java

---Test info---- Executed command: C:\mmm\jdk\bin\javac.exe i18nH▒lloWorld.java

++++Test Output++++ javac: file not found: i18nHélloWorld.java ----End test info-----

Have you run all the launcher regression tests with this changeset ?

Thanks Kumar

> Hi Kumar, just wondering if there are any updates on processing > this submission. > Thanks! > > -----Original Message----- > From: Vladimir Shcherbakov > Sent: Wednesday, November 25, 2015 2:38 PM > To: Kumar Srinivasan <kumar.x.srinivasan@oracle.com>; Martin > Sawicki <marcins@microsoft.com> > Cc: Kirk Shoop <Kirk.Shoop@microsoft.com>; core-libs-dev Libs > <core-libs-dev@openjdk.java.net> > Subject: RE: RFR 8124977 cmdline encoding challenges on Windows > > Hi Kumar, > > Please find updated webreview here: > https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr. > op > en > jdk.java.net%2f~kshoop%2f8124977%2fwebrev.04%2f&data=01%7C01%7Cma > rc > in > s%40microsoft.com%7C13ff309b775c4c019fc308d31ba0c43c%7C72f988bf86 > f1 > 41 > af91ab2d7cd011db47%7C1&sdata=3hhbO5mNPyTvtrTb4kCR42zsWGPGzDhqnmjp > Nf > wn > bIw%3d > > Thanks, > Vladimir. > > -----Original Message----- > From: Kumar Srinivasan [mailto:kumar.x.srinivasan@oracle.com] > Sent: Sunday, November 22, 2015 8:14 AM > To: Martin Sawicki <marcins@microsoft.com> > Cc: Kirk Shoop <Kirk.Shoop@microsoft.com>; Vladimir Shcherbakov > <vlashch@microsoft.com>; core-libs-dev Libs > <core-libs-dev@openjdk.java.net> > Subject: Re: RFR 8124977 cmdline encoding challenges on Windows > > > Hi Martin, et. al., > > Sorry for not getting back earlier, I am very busy right now with > my other large commitments for JDK9. > > I will sponsor this "enhancement/bug fix" sometime in the new > year, meanwhile, there is the changeset [1] which is likely to > cause merge conflicts, and perhaps logic issues. > > Thanks > Kumar > > [1] > https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fhg. > op > enjdk.java.net%2fjdk9%2fdev%2fjdk%2frev%2f3b201a9ef918&data=01%7c > 01 > %7 > cvlashch%40microsoft.com%7c4d49ae546dba4d29b7be08d2f3589ee1%7c72f > 98 > 8b > f86f141af91ab2d7cd011db47%7c1&sdata=I2FKvBn82%2fxhW3D%2fi%2bRWaNO > Jk > 7M > g4lt2P0sdzLS%2fT9Q%3d >> Hi all >> Here's an updated webrev attempting to take into account the >> various pieces of feedback we have received: >> >> Issue: >> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fbugs. >> >> openjdk.java.net%2fbrowse%2fJDK-8124977&data=01%7c01%7cvlashch%4 >> 0m >> ic >> ro >> soft.com%7c4d49ae546dba4d29b7be08d2f3589ee1%7c72f988bf86f141af91 >> ab >> 2d >> 7c >> d011db47%7c1&sdata=FjmfM%2fnPbWB%2fMsUU8uDzAUo3aPu3zOELVsJO%2fsU >> Iq >> 9E >> %3 >> d >> Webrev: >> https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr >> .o >> pe >> nj >> dk.java.net%2f~kshoop%2f8124977%2fwebrev.03%2f&data=01%7C01%7Cvl >> as >> hc >> h% >> 40microsoft.com%7C4d49ae546dba4d29b7be08d2f3589ee1%7C72f988bf86f >> 14 >> 1a >> f9 >> 1ab2d7cd011db47%7C1&sdata=101HBPar2AZ63GJWyubWH0DiKmNI%2bOxknN66 >> 7B >> Jn >> WY >> 0%3d >> >> (Vladimir Shcherbakov is now working on this from our side) >> >> Looking forward to any other feedback. >> Thanks >> >> -----Original Message----- >> From: core-libs-dev >> [mailto:core-libs-dev-bounces@openjdk.java.net] >> On Behalf Of Kumar Srinivasan >> Sent: Thursday, June 25, 2015 6:26 AM >> To: Kirk Shoop (MS OPEN TECH) <Kirk.Shoop@microsoft.com> >> Cc: Valery Kopylov (Akvelon) <v-valkop@microsoft.com>; >> core-libs-dev Libs <core-libs-dev@openjdk.java.net> >> Subject: Re: RFR 8124977 cmdline encoding challenges on Windows >> >> Hi Kirk, >> >> Thanks for proposing this change. >> >> If you notice all the posix calls are wrapped in JLI_* this >> gives us the ability to use "W" functions. I almost got it >> done, several years ago, but we upgraded to VS2010 and my work >> based on >> VS2003 keeled over, meanwhile my focus was "shifted" to >> something else. >> >> main.c: is really envisioned to be a stub compiled by the tool >> launchers, like java, javac, javah, jar etc. I prefer to see all >> the heavy logic in this file moved to the platform specific file >> windows/java_md.* >> >> For the reason specified above we need to move fprintf or any >> naked posix calls to JLI_* indirections. >> >> I don't see any tests ? The tests must be written in java and >> placed in jdk/test/tools/launcher, there is a helper framework >> TestHelper.java. >> >> There are other changes in nio, charsets etc, this will be >> reviewed by my colleague specializing in that area (Sherman) cc'ed. >> >> >> Thanks >> >> Kumar >> >> >> >> >> >> >> On 6/22/2015 2:01 PM, Kirk Shoop (MS OPEN TECH) wrote: >>> Hi, >>> >>> Issue: >>> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f% >>> 2f >>> bu >>> gs >>> .openjdk.java.net%2fbrowse%2fJDK-8124977&data=01%7c01%7cvlashch >>> %4 >>> 0m >>> ic >>> rosoft.com%7c4d49ae546dba4d29b7be08d2f3589ee1%7c72f988bf86f141a >>> f9 >>> 1a >>> b2 >>> d7cd011db47%7c1&sdata=FjmfM%2fnPbWB%2fMsUU8uDzAUo3aPu3zOELVsJO% >>> 2f >>> sU >>> Iq >>> 9E%3d >>> >>> Webrev: >>> https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fcr. >>> op >>> en >>> jdk.java.net%2f~kshoop%2f8124977%2f&data=01%7C01%7Cvlashch%40mi >>> cr >>> os >>> of >>> t.com%7C4d49ae546dba4d29b7be08d2f3589ee1%7C72f988bf86f141af91ab >>> 2d >>> 7c >>> d0 >>> 11db47%7C1&sdata=RAA%2b5aIzKtrk5X85oLXKlPzbpSk%2bgJZRI%2b0QSI11 >>> B0 >>> M% >>> 3d >>> >>> This webrev intends to address interaction between Windows >>> console and java apps. >>> >>> Two switches were added that change the behavior of the launcher. >>> The defaults do not change the launcher behavior. >>> >>> -Dwindows.UnicodeConsole=true - switches on Unicode >>> support in the Windows console. This optional switch causes the >>> launcher to call GetCommandLineW() and parse the arguments in >>> unicode. It also modifies how the codepage for console output is selected. >>> >>> -Dfile.encoding.unicode="UTF-8" - identifies Unicode >>> charset to use; If not specified, UTF-8 is used by default. >>> Ignored when windows.UnicodeConsole is not set to true. When >>> the first switch is used, this optional switch allows the >>> codepage for console output to be controlled. >>> >>> I would like to get feedback on the approach here and any >>> additional work that is required solve these particular Unicode >>> issues on Windows. >>> >>> Kirk