<html><body><div dir="ltr">PREAMBLE: Due to not being sure where to post it, this was posted to amber-dev before. I have updated it to take into account Ron Pressler’s notes on System.console, and Brian Goetz’s notes on steering clear of shoving deadlines in debate posts like this one.</div><div dir="ltr">—</div><div dir="ltr"><br></div><div dir="ltr"><br></div>JDK18 brought JEP400 which changes the default charset encoding to UTF-8. This, probably out of necessity, goes quite far, in that <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">Charset.defaultCharset()</code> is now more or less a constant - always returns UTF_8. It’s now quite difficult to retrieve the OS-configured encoding (the ’native’ encoding).<div><br></div><div dir="ltr">However, that does mean one of the most common lines in all of java’s history, is now necessarily buggy: <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">new Scanner(System.in)</code> is now broken. Always, unless your docs specifically state that you must feed the app UTF_8 data. Linting tools ought to flag it down as incorrect. It’s incorrect In a nasty way too: Initially it seems to work fine, but if you’re on an OS whose native encoding isn’t UTF-8, this is subtly broken; enter non-ASCII characters on the command line and the app doesn’t handle them appropriately. A bug that is literally utterly undiscoverable on macs and most linux computers, even. How can you figure out your code is broken if all the machines you test it on use UTF-8 as an OS default?</div><div dir="ltr"><br></div><div dir="ltr">This affects beginning java programmers particularly (who tend to be writing some command line-interactive apps at first). In light of Brian Goetz’s post “Paving the Onramp” (<a href="https://openjdk.org/projects/amber/design-notes/on-ramp">https://openjdk.org/projects/amber/design-notes/on-ramp</a>) - the experience for new users is evidently of some importance to the OpenJDK team. In light of that, the current state of writing command line interactive java apps is inconsistent with that goal.</div><div dir="ltr"><br></div><div dir="ltr">The right way to read system input in a way that works in both pre- and post-JEP400 JVM editions appears to be, as far as I can tell:<br></div><div dir="ltr"><br></div><div dir="ltr"><pre dir="ltr" style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:10px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px;margin-top:0px;margin-bottom:0px">Charset nativeCharset = Charset.forName(System.getProperty("native.encoding", Charset.defaultEncoding().name());
Scanner sc = new Scanner(<a href="https://system.in/" class="linkified" target="_blank">System.in</a>, nativeCharset);</pre></div><div dir="ltr"><div><br></div><div dir="ltr">I’ll risk the hyperbole: That’s.. atrocious. Hopefully I’m missing something!</div><div dir="ltr"><br></div><div dir="ltr">Breaking _thousands_ of blogs, tutorials, stack overflow answers, and books in the process, everything that contains <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">new Scanner(System.in)</code>. Even sysin interaction that doesn’t use scanner is likely broken; the general strategy then becomes:</div><div dir="ltr"><br></div><div dir="ltr"><pre dir="ltr" style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:10px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px;margin-top:0px;margin-bottom:0px">new InputStreamReader(<a href="https://system.in/" class="linkified" target="_blank">System.in</a>);</pre></div><div dir="ltr"><br></div><div dir="ltr">which suffers from the same problem.</div><div dir="ltr"><br></div><div dir="ltr">I see a few directions for trying to address this; I’m not quite sure which way would be most appropriate:</div><div dir="ltr"><br></div><div dir="ltr"><ul dir="ltr" style="margin:0px"><li>Completely re-work keyboard input, in light of <i>Paving the on-ramp</i>. Scanner has always been a problematic API if used for keyboard input, in that the default delimiter isn’t convenient. I think the single most common beginner java stackoverflow question is the bizarre interaction between scanner’s <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">nextLine()</code> and scanner’s <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">next()</code>, and to make matters considerably worse, the proper fix (which is to call <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">.useDelimiter(“\\R”)</code> on the scanner first) is said in less than 1% of answers; the vast majority of tutorials and answers tell you to call <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">.nextLine()</code> after every <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">.nextX()</code> call. A suboptimal suggestion (it now means using space to delimit your input is broken). Scanner is now also quite inconsistent: The constructor goes for ‘internet standard’, using UTF-8 as a default even if the OS does not, but the locale <i>does</i> go by platform default, which affects double parsing amongst other things: <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">scanner.nextDouble()</code> will require you to use commas as fractions separator if your OS is configured to use the Dutch locale, for example. It’s weird that scanner neither fully follows common platform-independent expectations (english locale, UTF-8), nor local-platform expectation (OS-configured locale and OS-configured charset). One way out is to make a new API for ‘command line apps’ and take into account Paving the on-ramp’s plans when designing it.</li><li>Rewrite specifically the <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">new Scanner(InputStream)</code> constructor as defaulting to native encoding even when everything else in java defaults to UTF-8 now, because that constructor is 99% used for <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">System.in</code>. Scanner has its own File-based constructor, so <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">new Scanner(Files.newInputStream(..))</code> is quite rare.</li><li>Define that constructor to act as follows: the charset used is the platform default (i.e., from JDK18 and up, UTF-8), <i>unless</i> <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">arg == System.in</code> is <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">true</code>, in which case the scanner uses native encoding. This is a bit bizarre to write in the spec but does the right thing in the most circumstances and unbreaks thousands of tutorials, blogs, and answer sites, and is most convenient to code against. That’s usually the case with voodoo magic (because this surely risks being ’too magical’): It’s convenient and does the right thing almost always, at the risk of being hard to fathom and producing convoluted spec documentation.</li><li>Attach the problem that what’s really broken isn’t so much scanner, it’s <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">System.in</code> itself: byte based, of course, but now that all java methods default to UTF-8, almost all interactions with it (given that most <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">System.in</code> interaction is char-based, not byte-based) are now also broken. Create a second field or method in <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">System</code> that gives you a <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">Reader</code> instead of an <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">InputStream</code>, with the OS-native encoding applied to make it. This still leaves those thousands of tutorials broken, but at least the proper code is now simply <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">new Scanner(System.charIn())</code> or whatnot, instead of the atrocious snippet above.</li><li>Even less impactful, make a new method in <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">Charset</code> to get the native encoding without having to delve into <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">System.getProperty()</code>. <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">Charset.nativeEncoding()</code> seems like a method that should exist. Unfortunately this would be of no help to create code that works pre- and post-JEP400, but in time, having code that only works post-JEP400 is fine, I assume.</li><li>Create a new concept ‘represents a stream that would use platform native encoding if characters are read/written to it’, have <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">System.in</code> return <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">true</code> for this, and have filterstreams like <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">BufferedInputStream</code> just pass the call through, then redefine relevant APIs such as <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">Scanner</code> and <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">PrintStream</code> (e.g. anything that internalises conversion from bytes to characters) to pick charset encoding (native vs UTF8) based on that property. This is a more robust take on ‘<code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">new Scanner(System.in)</code> should do the right thing'. Possibly the in/out/err streams that <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">Process</code> gives you should also have this flag set.</li><li>(based on feedback from Ron Pressler in amber-dev) Try to move the community away from treating <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">System.in</code> and <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">System.out</code> as the streams to be used for ‘command line apps’, and towards using <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">System.console()</code> instead, which is already char based, and is better positioned to take care of picking the right charset for you. However, this is quite a big job, given that virtually all tutorials, books, q&a sites like Stack Overflow talk about <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">System.in/out</code> and not about <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">Console</code>. Even if somehow the message gets out and these start using <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">Console</code> instead, the experience for java developers would be deplorable, given that <b>no IDE supports Console!</b> - possibly because it is maybe difficult for them to set it up properly? At any rate, just like the JDBC group works together with DB vendors to ensure JDBC actually is fit for purpose, there would have to be something set up to ensure tool developers like the eclipse team or IntelliJ update their templates and support Console for their run/debug-inside-IDE features. An open question then comes up: How does the OpenJDK team move the community in the direction that the OpenJDK wants them to move? “Build it and they will come”? I highly doubt that would work here; <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">System.in</code> works well enough for the base case at first glance. At the very least a statement by the OpenJDK that <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">new Scanner(System.in)</code> is a bad idea would help to start the decades-long work of trying to break down established Stack Overflow answers, mark tutorials as obsolete, etc. I have no idea if the OpenJDK even wants to meddle with community interaction like this, but if it does not, then “it’s fine, <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">Console</code> exists, it’s not our problem the community doesn’t use it” seems a bit hollow.</li></ul><div><br></div><div><br></div><div dir="ltr">If it was up to me, I think a multitude of steps are warranted, each relatively simple.</div><div dir="ltr"><br></div><div dir="ltr"><ul dir="ltr" style="margin:0px"><li>Create <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">Charset.nativeEncoding()</code>. Which simply returns <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">Charset.forName(System.getProperty(“native.encoding”)</code>. But with the advantage that its shorter, doesn’t require knowing a magic string, and will fail at compile time if compiled against versions that predate the existence of the native.encoding property, instead of NPEs at runtime.</li><li>Create <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">System.charIn()</code>. Which just returns an InputStreamReader wrapped around <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">System.in</code>, but with native encoding applied.</li><li>Put the job of how java apps do basic command line stuff on the agenda as a thing that should probably be addressed in the next 5 years or so, maybe after the steps laid out in Paving the on-ramp are more fleshed out.</li><li>In order to avoid problems, re-spec <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">new Scanner(System.in)</code> to default to native encoding, specifically when the passed inputstream is identical to <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">System.in</code>. Don’t bother with trying to introduce an abstracted ‘prefers native encoding’ flag system.</li><li>Contact IntelliJ, eclipse, and possibly maven/gradle (insofar that Console doesn’t work when using <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">mvn run</code> and the like) and ask them what they need to add console support, keeping in mind encoding is important, and possibly, to rewire their <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">syso</code> (eclipse) and <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">sysout</code> (intellij) template shortcuts away from <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">System.out.println</code> and towards <code style="border:1px solid rgb(206,206,206);background-color:rgb(244,244,244);padding:0px 2px;border-top-left-radius:2px;border-top-right-radius:2px;border-bottom-right-radius:2px;border-bottom-left-radius:2px">System.console().printf</code> instead.</li></ul></div><div><br></div></div><div dir="ltr"> --Reinier Zwitserloot</div></div><div><div class="gmail_signature" data-smartmail="gmail_signature"><br></div></div></body></html>