java.io.File#toPath() on Windows doesn't understand "NUL:" (null device)?
Please consider this trivial code: import java.io.*; import java.nio.file.*; public class FileTest { public static void main(final String[] args) throws Exception { System.getProperties().list(System.out); final File f = new File("NUL:"); try (final InputStream fis = Files.newInputStream(f.toPath())) { int numBytes = 0; while (fis.read() != -1) { System.out.println("Files.newInputStream - Read a byte from " + f); numBytes++; } System.out.println("Files.newInputStream - Done reading " + numBytes + " bytes from " + f); } } } The code tries to read from NUL: on a Windows setup. This code runs into the following exception on Windows when the java.io.File#toPath() gets invoked: Exception in thread "main" java.nio.file.InvalidPathException: Illegal char <:> at index 3: NUL: at java.base/sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182) at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153) at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77) at java.base/sun.nio.fs.WindowsPath.parse(WindowsPath.java:92) at java.base/sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:230) at java.base/java.io.File.toPath(File.java:2316) at FileTest.main(FileTest.java:18) So it looks like java.io.File.toPath() on Windows isn't able to recognize the null device construct? Just to make sure this issue resides only this specific API implementation, I switched the above code to use new FileInputStream(f) as follows and that works as expected - reads 0 bytes and completes successfully. So the NUL: construct is indeed understood by the other APIs. public class FileTest { public static void main(final String[] args) throws Exception { System.getProperties().list(System.out); final File f = new File("NUL:"); try (final FileInputStream fis = new FileInputStream(f)) { int numBytes = 0; while (fis.read() != -1) { System.out.println("FileInputStream() - Read a byte from " + f); numBytes++; } System.out.println("FileInputStream() - Done reading " + numBytes + " bytes from " + f); } } } Output: FileInputStream() - Done reading 0 bytes from NUL: Test results are from latest Java 16 release on a Windows setup. -Jaikiran
On 17/03/21 8:51 am, Jaikiran Pai wrote:
Test results are from latest Java 16 release on a Windows setup.
Just gave a quick try against Java 8 (java.version=1.8.0_252) and it fails on that version too with the same error: Exception in thread "main" java.nio.file.InvalidPathException: Illegal char <:> at index 3: NUL: at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182) at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153) at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77) at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94) at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255) at java.io.File.toPath(File.java:2234) at FileTest.main(FileTest.java:18) -Jaikiran
https://bugs.openjdk.java.net/browse/JDK-8022671 Cheers, David On 17/03/2021 1:21 pm, Jaikiran Pai wrote:
Please consider this trivial code:
import java.io.*; import java.nio.file.*;
public class FileTest { public static void main(final String[] args) throws Exception { System.getProperties().list(System.out); final File f = new File("NUL:"); try (final InputStream fis = Files.newInputStream(f.toPath())) { int numBytes = 0; while (fis.read() != -1) { System.out.println("Files.newInputStream - Read a byte from " + f); numBytes++; } System.out.println("Files.newInputStream - Done reading " + numBytes + " bytes from " + f); } } }
The code tries to read from NUL: on a Windows setup. This code runs into the following exception on Windows when the java.io.File#toPath() gets invoked:
Exception in thread "main" java.nio.file.InvalidPathException: Illegal char <:> at index 3: NUL: at java.base/sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153) at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77) at java.base/sun.nio.fs.WindowsPath.parse(WindowsPath.java:92) at java.base/sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:230) at java.base/java.io.File.toPath(File.java:2316) at FileTest.main(FileTest.java:18)
So it looks like java.io.File.toPath() on Windows isn't able to recognize the null device construct?
Just to make sure this issue resides only this specific API implementation, I switched the above code to use new FileInputStream(f) as follows and that works as expected - reads 0 bytes and completes successfully. So the NUL: construct is indeed understood by the other APIs.
public class FileTest { public static void main(final String[] args) throws Exception { System.getProperties().list(System.out); final File f = new File("NUL:"); try (final FileInputStream fis = new FileInputStream(f)) { int numBytes = 0; while (fis.read() != -1) { System.out.println("FileInputStream() - Read a byte from " + f); numBytes++; } System.out.println("FileInputStream() - Done reading " + numBytes + " bytes from " + f); } } }
Output:
FileInputStream() - Done reading 0 bytes from NUL:
Test results are from latest Java 16 release on a Windows setup.
-Jaikiran
On 17/03/2021 03:21, Jaikiran Pai wrote:
:
The code tries to read from NUL: on a Windows setup. This code runs into the following exception on Windows when the java.io.File#toPath() gets invoked:
Exception in thread "main" java.nio.file.InvalidPathException: Illegal char <:> at index 3: NUL: at java.base/sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182) at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153) at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77) at java.base/sun.nio.fs.WindowsPath.parse(WindowsPath.java:92) at java.base/sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:230) at java.base/java.io.File.toPath(File.java:2316) at FileTest.main(FileTest.java:18)
So it looks like java.io.File.toPath() on Windows isn't able to recognize the null device construct?
Special devices, esp. those historical devices from the DOS era, are very problematic. NUL is somewhat benign compared to the other and you use "NUL" (not "NUL:") then should work as you expect. Changing the path parser to allow ":" in places other than after drive letters is a slippery slope as it brings all a lot of the issues that plagued the older code. -Alan
On 17/03/21 1:26 pm, Alan Bateman wrote:
On 17/03/2021 03:21, Jaikiran Pai wrote:
:
The code tries to read from NUL: on a Windows setup. This code runs into the following exception on Windows when the java.io.File#toPath() gets invoked:
Exception in thread "main" java.nio.file.InvalidPathException: Illegal char <:> at index 3: NUL: at java.base/sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182) at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153) at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77) at java.base/sun.nio.fs.WindowsPath.parse(WindowsPath.java:92) at java.base/sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:230) at java.base/java.io.File.toPath(File.java:2316) at FileTest.main(FileTest.java:18)
So it looks like java.io.File.toPath() on Windows isn't able to recognize the null device construct?
Special devices, esp. those historical devices from the DOS era, are very problematic.
NUL is somewhat benign compared to the other and you use "NUL" (not "NUL:") then should work as you expect.
Thank you David and Alan. I can confirm that using "NUL" or "nul" work fine in the above code, with the FileInputStream/FileOutputStream constructors as well as Files.newInputStream(f.toPath()) and Files.newOutputStream(f.toPath()).
Changing the path parser to allow ":" in places other than after drive letters is a slippery slope as it brings all a lot of the issues that plagued the older code.
Understood. -Jaikiran
On 17/03/2021 08:21, Jaikiran Pai wrote:
:
I can confirm that using "NUL" or "nul" work fine in the above code,
I don't know the context for your question but just to mention InputStream.nullInputStream() or Reader.nullReader() may be useful if you have something that wants to read from a null stream. -Alan.
Hello Alan, On 17/03/21 2:45 pm, Alan Bateman wrote:
On 17/03/2021 08:21, Jaikiran Pai wrote:
:
I can confirm that using "NUL" or "nul" work fine in the above code,
I don't know the context for your question
A while back Apache Ant switched to using the Files.newInputStream/Files.newOutputStream construct as a replacement for the FileInputStream and FileOutputStream constructors[1]. That commit seems to have introduced an regression in Ant as noted here[2]. It appears that users of Ant were using "nul" (and even "nul:") as destination file to have Ant write the data to that destination. Internally Ant constructs an instance of java.io.File from the user provided path string ("nul" or "nul:" in this case) and constructs a OutputStream out of it. Previously (before we made that commit in Ant), it would be using the FileOutputStream constructor (and would succeed) and now it uses the Files.newOuputStream(...) which expects a Path instance and this where our usage of java.io.File.toPath() ran into the issue I note in this mail, when I started investigating this. I didn't mention this context in my mail because the error noted in those user reports is surprisingly a bit different than what I have seen in my experiments on Windows and I don't think I have so far narrowed it down to a JDK issue (if any). In their case, it appears the call went past the issue we are discussing this mail and instead failed later with something like: java.nio.file.FileSystemException: E:\nul: Incorrect function. at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) at sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:230) at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434) at java.nio.file.Files.newOutputStream(Files.java:216) I'm not yet sure how they managed to get to that stage and am waiting to see if they can provide us with a reproducible build file to reproduce this.
but just to mention InputStream.nullInputStream() or Reader.nullReader() may be useful if you have something that wants to read from a null stream.
We have had a recent discussion in Ant dev mailing list[3] to introduce such a construct in some of our Ant tasks where users can tell us that they want the output/error output discarded (that's what they are using /dev/null and "nul:" for). That will prevent all these platform specific usages of null device representations. Internally, in the implementation, we will use the APIs like the one you note and avoid having to deal with null devices. [1] https://github.com/apache/ant/commit/af74d1f6b882cef5f4167d972638ad886d12d58... [2] https://bz.apache.org/bugzilla/show_bug.cgi?id=62330 [3] https://www.mail-archive.com/dev@ant.apache.org/msg48730.html -Jaikiran
On 17/03/21 3:10 pm, Jaikiran Pai wrote:
Hello Alan,
On 17/03/21 2:45 pm, Alan Bateman wrote:
On 17/03/2021 08:21, Jaikiran Pai wrote:
:
I can confirm that using "NUL" or "nul" work fine in the above code,
I don't know the context for your question
A while back Apache Ant switched to using the Files.newInputStream/Files.newOutputStream construct as a replacement for the FileInputStream and FileOutputStream constructors[1]. That commit seems to have introduced an regression in Ant as noted here[2]. It appears that users of Ant were using "nul" (and even "nul:") as destination file to have Ant write the data to that destination. Internally Ant constructs an instance of java.io.File from the user provided path string ("nul" or "nul:" in this case) and constructs a OutputStream out of it. Previously (before we made that commit in Ant), it would be using the FileOutputStream constructor (and would succeed) and now it uses the Files.newOuputStream(...) which expects a Path instance and this where our usage of java.io.File.toPath() ran into the issue I note in this mail, when I started investigating this.
I didn't mention this context in my mail because the error noted in those user reports is surprisingly a bit different than what I have seen in my experiments on Windows and I don't think I have so far narrowed it down to a JDK issue (if any). In their case, it appears the call went past the issue we are discussing this mail and instead failed later with something like:
java.nio.file.FileSystemException: E:\nul: Incorrect function. at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) at sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:230) at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434) at java.nio.file.Files.newOutputStream(Files.java:216)
I'm not yet sure how they managed to get to that stage and am waiting to see if they can provide us with a reproducible build file to reproduce this.
FWIW - that bug report states that they ran into this even when using "nul" and not just "nul:". So there might be something more going on here and am just waiting to see if they can provide us a build file to reproduce this issue. -Jaikiran
On 17/03/21 3:16 pm, Jaikiran Pai wrote:
On 17/03/21 3:10 pm, Jaikiran Pai wrote:
Hello Alan,
On 17/03/21 2:45 pm, Alan Bateman wrote:
On 17/03/2021 08:21, Jaikiran Pai wrote:
:
I can confirm that using "NUL" or "nul" work fine in the above code,
I don't know the context for your question
...
FWIW - that bug report states that they ran into this even when using "nul" and not just "nul:". So there might be something more going on here and am just waiting to see if they can provide us a build file to reproduce this issue.
I received some inputs on that Ant bugzilla issue. Based on that, I was able to reproduce the exception and IMO it's a bug in Files.newOutputStream() API. I have opened https://bugs.openjdk.java.net/browse/JDK-8263898 with the relevant details. I considered this a bug and took the liberty of opening that JBS issue because as I note in that issue, this only happens specifically when TRUNCATE_EXISTING (default) option gets used against "nul" on Windows. -Jaikiran
On 20/03/2021 07:16, Jaikiran Pai wrote:
:
I received some inputs on that Ant bugzilla issue. Based on that, I was able to reproduce the exception and IMO it's a bug in Files.newOutputStream() API. I have opened https://bugs.openjdk.java.net/browse/JDK-8263898 with the relevant details. I considered this a bug and took the liberty of opening that JBS issue because as I note in that issue, this only happens specifically when TRUNCATE_EXISTING (default) option gets used against "nul" on Windows.
OutputStream.nullOutputStream() may be a better and more portable alternative. In general, the DOS era reserved names (including NUL) are very under specified and many file operations lead to surprising behavior (this isn't solely a JDK issue, I think other runtimes and languages also get tripped up). In this case, the attempt to truncate the file to zero length is failing. Sure, it can be be worked around but workarounds like this tend to cause issues in other unusual cases so care is required. -Alan.
On 22/03/21 2:36 pm, Alan Bateman wrote:
On 20/03/2021 07:16, Jaikiran Pai wrote:
:
I received some inputs on that Ant bugzilla issue. Based on that, I was able to reproduce the exception and IMO it's a bug in Files.newOutputStream() API. I have opened https://bugs.openjdk.java.net/browse/JDK-8263898 with the relevant details. I considered this a bug and took the liberty of opening that JBS issue because as I note in that issue, this only happens specifically when TRUNCATE_EXISTING (default) option gets used against "nul" on Windows.
OutputStream.nullOutputStream() may be a better and more portable alternative. In general, the DOS era reserved names (including NUL) are very under specified and many file operations lead to surprising behavior (this isn't solely a JDK issue, I think other runtimes and languages also get tripped up). In this case, the attempt to truncate the file to zero length is failing. Sure, it can be be worked around but workarounds like this tend to cause issues in other unusual cases so care is required.
Understood. Thank you Alan for those inputs. Just yesterday, Stefan (one of the Ant developers) has implemented a way where we allow users to discard output without relying on null devices. It uses the approach that you note (although we use an internal NullOutputStream, since we want Ant to be usable in Java 8 where OutputStream.nullOutputStream() isn't present). We will be asking our users to move to this new way/attribute instead of relying on null devices. -Jaikiran
participants (3)
-
Alan Bateman
-
David Holmes
-
Jaikiran Pai