From Alan.Bateman at Sun.COM Tue Sep 1 01:28:08 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Tue, 01 Sep 2009 09:28:08 +0100 Subject: Wrong exception thrown? In-Reply-To: References: Message-ID: <4A9CDB18.6020201@sun.com> John Hendrikx wrote: > Exception in thread "Thread-8": java.nio.file.AccessDeniedException: > c:\Copy\Macros2 -> c:\Copy2\Macros2 > at > sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83) > at > sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) > at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301) > at sun.nio.fs.WindowsPath.implMoveTo(WindowsPath.java:1004) > at sun.nio.fs.AbstractPath.moveTo(AbstractPath.java:201) > at > hs.filemanager.transfer.RenameFileVisitor.rename(RenameFileVisitor.java:67) > > This occurs when moveTo() is called with the arguments indicated + > ATOMIC_MOVE. > > I would expect the FileAlreadyExistsException to be thrown. It's not a > permission issue (and the files are not 'read only'). > > When the ATOMIC_MOVE option is specified then it is allowed to replace the target file if it exists. In other words, it is not required to fail if the target exists. An implementation can't first check if the file exists because the overall operation must be atomic. One annoyance, as Andrew suspected, is that Windows returns its equivalent of "access denied" for both the permission denied case and the case that the target is a directory. Is C:\Copy2\Macros2 is a directory? -Alan. From hjohn at xs4all.nl Tue Sep 1 04:43:05 2009 From: hjohn at xs4all.nl (John Hendrikx) Date: Tue, 1 Sep 2009 13:43:05 +0200 (CEST) Subject: Wrong exception thrown? In-Reply-To: <4A9CDB18.6020201@sun.com> References: <4A9CDB18.6020201@sun.com> Message-ID: > John Hendrikx wrote: >> Exception in thread "Thread-8": java.nio.file.AccessDeniedException: >> c:\Copy\Macros2 -> c:\Copy2\Macros2 >> at >> sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83) >> at >> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) >> at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301) >> at sun.nio.fs.WindowsPath.implMoveTo(WindowsPath.java:1004) >> at sun.nio.fs.AbstractPath.moveTo(AbstractPath.java:201) >> at >> hs.filemanager.transfer.RenameFileVisitor.rename(RenameFileVisitor.java:67) >> >> This occurs when moveTo() is called with the arguments indicated + >> ATOMIC_MOVE. >> >> I would expect the FileAlreadyExistsException to be thrown. It's not a >> permission issue (and the files are not 'read only'). >> >> > When the ATOMIC_MOVE option is specified then it is allowed to replace > the target file if it exists. In other words, it is not required to fail > if the target exists. An implementation can't first check if the file > exists because the overall operation must be atomic. One annoyance, as > Andrew suspected, is that Windows returns its equivalent of "access > denied" for both the permission denied case and the case that the target > is a directory. Is C:\Copy2\Macros2 is a directory? Yes, correct it was a directory. I didn't however specify REPLACE_EXISTING, only ATOMIC_MOVE (and the latter is documented to "ignore all other options"). I'm actually only trying to accomplish a simple rename of the directory -- but I couldn't find any other function that accomplishes this in nio2, so I suspected that srcPath.moveTo(dstPath, ATOMIC_MOVE) is the closest thing to it. I must specify ATOMIC_MOVE to prevent it actually moving any data, only a rename is desired. Would it be better to instead check if srcPath and dstPath are on the same FileStore and only then call srcPath.moveTo(dstPath) without any further arguments? Best regards, John Hendrikx From Alan.Bateman at Sun.COM Tue Sep 1 06:30:40 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Tue, 01 Sep 2009 14:30:40 +0100 Subject: Wrong exception thrown? In-Reply-To: References: <4A9CDB18.6020201@sun.com> Message-ID: <4A9D2200.1010508@sun.com> John Hendrikx wrote: > : > Yes, correct it was a directory. > > I didn't however specify REPLACE_EXISTING, only ATOMIC_MOVE (and the > latter is documented to "ignore all other options"). > > I'm actually only trying to accomplish a simple rename of the directory -- > but I couldn't find any other function that accomplishes this in nio2, so > I suspected that srcPath.moveTo(dstPath, ATOMIC_MOVE) is the closest thing > to it. I must specify ATOMIC_MOVE to prevent it actually moving any data, > only a rename is desired. > > Would it be better to instead check if srcPath and dstPath are on the same > FileStore and only then call srcPath.moveTo(dstPath) without any further > arguments? > It's good to bring this up as the assumption has been most applications will use source.moveTo(target) or source.moveTo(target, REPLACE_EXISTING) and not care if the target is on a different volume/file-system. The right thing is probably to introduce something like a NOCOPY_ALLOWED or some such option. For you, you are correct, that you can check if the source and target FileStore are equal. -Alan. From mthornton at optrak.co.uk Tue Sep 1 06:42:06 2009 From: mthornton at optrak.co.uk (Mark Thornton) Date: Tue, 01 Sep 2009 14:42:06 +0100 Subject: Wrong exception thrown? In-Reply-To: <4A9D2200.1010508@sun.com> References: <4A9CDB18.6020201@sun.com> <4A9D2200.1010508@sun.com> Message-ID: <4A9D24AE.8040009@optrak.co.uk> Alan Bateman wrote: > John Hendrikx wrote: >> : >> Yes, correct it was a directory. >> >> I didn't however specify REPLACE_EXISTING, only ATOMIC_MOVE (and the >> latter is documented to "ignore all other options"). >> >> I'm actually only trying to accomplish a simple rename of the >> directory -- >> but I couldn't find any other function that accomplishes this in >> nio2, so >> I suspected that srcPath.moveTo(dstPath, ATOMIC_MOVE) is the closest >> thing >> to it. I must specify ATOMIC_MOVE to prevent it actually moving any >> data, >> only a rename is desired. >> >> Would it be better to instead check if srcPath and dstPath are on the >> same >> FileStore and only then call srcPath.moveTo(dstPath) without any further >> arguments? >> > It's good to bring this up as the assumption has been most > applications will use source.moveTo(target) or source.moveTo(target, > REPLACE_EXISTING) and not care if the target is on a different > volume/file-system. The right thing is probably to introduce something > like a NOCOPY_ALLOWED or some such option. For you, you are correct, > that you can check if the source and target FileStore are equal. Surely that is also subject to potential races --- another process could change the FileStore associated with a path. Mark Thornton From Alan.Bateman at Sun.COM Tue Sep 1 07:00:45 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Tue, 01 Sep 2009 15:00:45 +0100 Subject: Wrong exception thrown? In-Reply-To: <4A9D24AE.8040009@optrak.co.uk> References: <4A9CDB18.6020201@sun.com> <4A9D2200.1010508@sun.com> <4A9D24AE.8040009@optrak.co.uk> Message-ID: <4A9D290D.8080807@sun.com> Mark Thornton wrote: > Surely that is also subject to potential races --- another process > could change the FileStore associated with a path. Yes, this can happen. Same for this: if (target.notExist()) source.moveTo(target, ATOMIC_MOVE); It would be desirable if ATOMIC_MOVE could guarantee not to replace an existing file but that may not be feasible everywhere (esp. implementations with rename). -Alan. From Alan.Bateman at Sun.COM Fri Sep 4 06:26:52 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Fri, 04 Sep 2009 14:26:52 +0100 Subject: Need review for 6873621: (file) FileStore.supportsFileAttributeView(Class type) returns wrong result Message-ID: <4AA1159C.5040902@sun.com> I need a reviewer for this one - it's very simple. FileStore.supportsFileAttributeView(Class) should return true if the file store supports the given file attribute view. The method needs to be overridden in the Solaris and Linux implementations so that it returns the right answer for the more specialized views. Also, the Windows implementation is missing one case. The webrev is here: http://cr.openjdk.java.net/~alanb/6873621/webrev.00/ Thanks, Alan. From gnu_andrew at member.fsf.org Fri Sep 4 06:42:57 2009 From: gnu_andrew at member.fsf.org (Andrew John Hughes) Date: Fri, 4 Sep 2009 14:42:57 +0100 Subject: Need review for 6873621: (file) FileStore.supportsFileAttributeView(Class type) returns wrong result In-Reply-To: <4AA1159C.5040902@sun.com> References: <4AA1159C.5040902@sun.com> Message-ID: <17c6771e0909040642p1d29a20dw5dc767fe3ebf9ff7@mail.gmail.com> 2009/9/4 Alan Bateman : > I need a reviewer for this one - it's very simple. > FileStore.supportsFileAttributeView(Class) > should return true if the file store supports the given file attribute view. > The method needs to be overridden in the Solaris and Linux implementations > so that it returns the right answer for the more specialized views. Also, > the Windows implementation is missing one case. The webrev is here: > ? http://cr.openjdk.java.net/~alanb/6873621/webrev.00/ > > Thanks, > Alan. > Looks good to me. I like the idea of making this safer by checking against a type rather than an arbitrary String. This also makes these classes match UnixFileStore which already does this for the PosixFileAttributeView. As an aside, I noticed that Solaris is the only implementation to use AclFileAttributeView. Don't both Linux and Windows have some form of ACL support? Something to fix long term I guess. -- Andrew :-) Free Java Software Engineer Red Hat, Inc. (http://www.redhat.com) Support Free Java! Contribute to GNU Classpath and the OpenJDK http://www.gnu.org/software/classpath http://openjdk.java.net PGP Key: 94EFD9D8 (http://subkeys.pgp.net) Fingerprint: F8EF F1EA 401E 2E60 15FA 7927 142C 2591 94EF D9D8 From gnu_andrew at member.fsf.org Fri Sep 4 06:45:47 2009 From: gnu_andrew at member.fsf.org (Andrew John Hughes) Date: Fri, 4 Sep 2009 14:45:47 +0100 Subject: Need review for 6873621: (file) FileStore.supportsFileAttributeView(Class type) returns wrong result In-Reply-To: <17c6771e0909040642p1d29a20dw5dc767fe3ebf9ff7@mail.gmail.com> References: <4AA1159C.5040902@sun.com> <17c6771e0909040642p1d29a20dw5dc767fe3ebf9ff7@mail.gmail.com> Message-ID: <17c6771e0909040645j48bfe704u73f84e681fbc0ca5@mail.gmail.com> 2009/9/4 Andrew John Hughes : > 2009/9/4 Alan Bateman : >> I need a reviewer for this one - it's very simple. >> FileStore.supportsFileAttributeView(Class) >> should return true if the file store supports the given file attribute view. >> The method needs to be overridden in the Solaris and Linux implementations >> so that it returns the right answer for the more specialized views. Also, >> the Windows implementation is missing one case. The webrev is here: >> ? http://cr.openjdk.java.net/~alanb/6873621/webrev.00/ >> >> Thanks, >> Alan. >> > > Looks good to me. ?I like the idea of making this safer by checking > against a type rather than an arbitrary String. ?This also makes these > classes match UnixFileStore which already does this for the > PosixFileAttributeView. > > As an aside, I noticed that Solaris is the only implementation to use > AclFileAttributeView. ?Don't both Linux and Windows have some form of > ACL support? ?Something to fix long term I guess. > -- > Andrew :-) > > Free Java Software Engineer > Red Hat, Inc. (http://www.redhat.com) > > Support Free Java! > Contribute to GNU Classpath and the OpenJDK > http://www.gnu.org/software/classpath > http://openjdk.java.net > > PGP Key: 94EFD9D8 (http://subkeys.pgp.net) > Fingerprint: F8EF F1EA 401E 2E60 15FA ?7927 142C 2591 94EF D9D8 > In supportsFileAttributeView(Class type) returns wrong result In-Reply-To: <17c6771e0909040642p1d29a20dw5dc767fe3ebf9ff7@mail.gmail.com> References: <4AA1159C.5040902@sun.com> <17c6771e0909040642p1d29a20dw5dc767fe3ebf9ff7@mail.gmail.com> Message-ID: <4AA11AE3.9060007@optrak.co.uk> Andrew John Hughes wrote: > 2009/9/4 Alan Bateman : > >> I need a reviewer for this one - it's very simple. >> FileStore.supportsFileAttributeView(Class) >> should return true if the file store supports the given file attribute view. >> The method needs to be overridden in the Solaris and Linux implementations >> so that it returns the right answer for the more specialized views. Also, >> the Windows implementation is missing one case. The webrev is here: >> http://cr.openjdk.java.net/~alanb/6873621/webrev.00/ >> >> Thanks, >> Alan. >> >> > > Looks good to me. I like the idea of making this safer by checking > against a type rather than an arbitrary String. This also makes these > classes match UnixFileStore which already does this for the > PosixFileAttributeView. > > As an aside, I noticed that Solaris is the only implementation to use > AclFileAttributeView. Don't both Linux and Windows have some form of > ACL support? Something to fix long term I guess. > AclFileAttributeView *is* in the Windows implementation: 158 if (type == AclFileAttributeView.class || type == FileOwnerAttributeView.class) 159 return ((volInfo.flags() & FILE_PERSISTENT_ACLS) != 0); Looks good to me too, although I can only really comment on the Windows case. Regards, Mark Thornton -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20090904/ab581c61/attachment.html From Alan.Bateman at Sun.COM Fri Sep 4 07:03:00 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Fri, 04 Sep 2009 15:03:00 +0100 Subject: Need review for 6873621: (file) FileStore.supportsFileAttributeView(Class type) returns wrong result In-Reply-To: <17c6771e0909040642p1d29a20dw5dc767fe3ebf9ff7@mail.gmail.com> References: <4AA1159C.5040902@sun.com> <17c6771e0909040642p1d29a20dw5dc767fe3ebf9ff7@mail.gmail.com> Message-ID: <4AA11E14.9050109@sun.com> Andrew John Hughes wrote: > : > Looks good to me. I like the idea of making this safer by checking > against a type rather than an arbitrary String. This also makes these > classes match UnixFileStore which already does this for the > PosixFileAttributeView. > Thanks. > As an aside, I noticed that Solaris is the only implementation to use > AclFileAttributeView. Don't both Linux and Windows have some form of > ACL support? Something to fix long term I guess. > It's supported on Windows too - the API is based on the NFSv4 ACL model (RFC 3530) which was based on the Windows NT ACL model (a few differences but well understood). We don't have any support for ACLs on Linux, not yet anyway. It's a while since I looked into this but there was a patch available to support them on ext3 but not NFS. Also, at the time, I didn't find any distributions that shipped with the support. My understanding might be stale, but definitely something to add in the future. -Alan. From Alan.Bateman at Sun.COM Fri Sep 4 07:03:36 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Fri, 04 Sep 2009 15:03:36 +0100 Subject: Need review for 6873621: (file) FileStore.supportsFileAttributeView(Class type) returns wrong result In-Reply-To: <17c6771e0909040645j48bfe704u73f84e681fbc0ca5@mail.gmail.com> References: <4AA1159C.5040902@sun.com> <17c6771e0909040642p1d29a20dw5dc767fe3ebf9ff7@mail.gmail.com> <17c6771e0909040645j48bfe704u73f84e681fbc0ca5@mail.gmail.com> Message-ID: <4AA11E38.9020501@sun.com> Andrew John Hughes wrote: > : > In supportsFileAttributeView(Class UnixFileStore, the following: > > FeatureStatus status = checkIfFeaturePresent("posix"); > if (status == FeatureStatus.NOT_PRESENT) > return false; > return true; > > could be changed to simply: > > FeatureStatus status = checkIfFeaturePresent("posix"); > return (status != FeatureStatus.NOT_PRESENT) > > or even: > > return checkIfFeaturePresent("posix") != FeatureStatus.NOT_PRESENT; > > as part of your patch. > Makes sense - I'll do that - thanks! -Alan. From Alan.Bateman at Sun.COM Fri Sep 4 07:23:26 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Fri, 04 Sep 2009 15:23:26 +0100 Subject: Need review for 6873621: (file) FileStore.supportsFileAttributeView(Class type) returns wrong result In-Reply-To: <4AA11AE3.9060007@optrak.co.uk> References: <4AA1159C.5040902@sun.com> <17c6771e0909040642p1d29a20dw5dc767fe3ebf9ff7@mail.gmail.com> <4AA11AE3.9060007@optrak.co.uk> Message-ID: <4AA122DE.90904@sun.com> Mark Thornton wrote: > AclFileAttributeView *is* in the Windows implementation: > > 158 if (type == AclFileAttributeView.class || type == FileOwnerAttributeView.class) > 159 return ((volInfo.flags() & FILE_PERSISTENT_ACLS) != 0); > > Looks good to me too, although I can only really comment on the Windows case. > Yes, that's right - thanks! -Alan. From hjohn at xs4all.nl Fri Sep 4 17:11:46 2009 From: hjohn at xs4all.nl (John Hendrikx) Date: Sat, 05 Sep 2009 02:11:46 +0200 Subject: NIO performance Message-ID: <4AA1ACC2.5020801@xs4all.nl> I noticed that using a FileVisitor to iterate over a single directory is far faster than doing a DirectoryStream + readBasicFileAttributes combination. Results for 9000 file directory: Simple DirectoryStream iteration = 300 ms DirectoryStream iteration + readBasicFileAttributes for each entry = 9000 ms FileVisitor which skips all subdirectories (but does return BasicFileAttributes for each entry) = 480 ms Using a FileVisitor to iterate over a single directory seems somewhat clumsy so I looked at the implementation to see if there was a better way, but I found that it is basically cheating (Path seems to be an instance of BasicFileAttributesHolder which obviously is a lot faster than doing your own Attributes.readBasicFileAttributes(path) call). I guess what I'm saying is that I didn't really expect that -- I would have expected that for reading a single directory (+ attributes) there would be a simple way to do it like DirectoryStream currently provides. Currently, I think that many would fall for the trap of iterating over a DirectoryStream and calling readBasicFileAttributes on each entry which is very slow. Of course now that I figured this out it is no real problem to just wrap a FileVisitor in my own class to read a single directory. I hope this feedback is useful. From Alan.Bateman at Sun.COM Sat Sep 5 07:14:35 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Sat, 05 Sep 2009 15:14:35 +0100 Subject: NIO performance In-Reply-To: <4AA1ACC2.5020801@xs4all.nl> References: <4AA1ACC2.5020801@xs4all.nl> Message-ID: <4AA2724B.3040600@sun.com> John Hendrikx wrote: > I noticed that using a FileVisitor to iterate over a single directory > is far faster than doing a DirectoryStream + readBasicFileAttributes > combination. > > Results for 9000 file directory: > > Simple DirectoryStream iteration = 300 ms > DirectoryStream iteration + readBasicFileAttributes for each entry = > 9000 ms > FileVisitor which skips all subdirectories (but does return > BasicFileAttributes for each entry) = 480 ms > > Using a FileVisitor to iterate over a single directory seems somewhat > clumsy so I looked at the implementation to see if there was a better > way, but I found that it is basically cheating (Path seems to be an > instance of BasicFileAttributesHolder which obviously is a lot faster > than doing your own Attributes.readBasicFileAttributes(path) call). > > I guess what I'm saying is that I didn't really expect that -- I would > have expected that for reading a single directory (+ attributes) there > would be a simple way to do it like DirectoryStream currently > provides. Currently, I think that many would fall for the trap of > iterating over a DirectoryStream and calling readBasicFileAttributes > on each entry which is very slow. Of course now that I figured this > out it is no real problem to just wrap a FileVisitor in my own class > to read a single directory. > > I hope this feedback is useful. Files.walkFileTree is essentially an internal iterator built on an external iterator (DirectoryStream). So for the maxDepth == 1 case then it is reasonable to expect the performance to be the same as using DirectoryStream to iterate over all entries in the directory, calling Attributes.readBasicFileAttributes to read the attributes of each file. The anomaly you are seeing is a Windows only anomaly. Elsewhere (on Solaris and Linux at least) the performance will be as you would expect. For example, I did a quick test on Solaris with a directory of 9000 files and the simple iteration took 22ms, the iteration + reading the attributes took 88ms, and walkFileTree with maxDepth==1 took 83ms. On Windows, the anomaly (or why is Files.walkFileTree so much faster) is because the attributes are obtained during the directory traversal so the implementation can avoid re-reading them - if it re-read the attributes for each file then it would take about the same time as calling Attributes.readBasicFileAttributes for each file in the directory, an operation that is expensive on Windows. One thing to say is that difference isn't as obvious with NTFS - for example, I repeated your test with a directory of 9000 files and the simple iteration took 21ms, the iteration + reading the attributes took 237ms, and Files.walkFileTree took 20ms. With FAT32 or when the volume is remote then the difference is very obvious - I'll guess this is what you are testing on. This issue does bring up the question as to if we need a method that returns a DirectoryStream where the elements are a pair consisting of the entry and its attributes. It's come up once or twice. From a performance point of view it helps Windows, and maybe some custom file systems. The other potential justification is convenience in that the basic attributes will often be required when iterating over a directory. It's worth looking at. -Alan. From hjohn at xs4all.nl Sat Sep 5 07:40:17 2009 From: hjohn at xs4all.nl (John Hendrikx) Date: Sat, 05 Sep 2009 16:40:17 +0200 Subject: NIO performance In-Reply-To: <4AA2724B.3040600@sun.com> References: <4AA1ACC2.5020801@xs4all.nl> <4AA2724B.3040600@sun.com> Message-ID: <4AA27851.3040106@xs4all.nl> Alan Bateman wrote: > Files.walkFileTree is essentially an internal iterator built on an > external iterator (DirectoryStream). So for the maxDepth == 1 case > then it is reasonable to expect the performance to be the same as > using DirectoryStream to iterate over all entries in the directory, > calling Attributes.readBasicFileAttributes to read the attributes of > each file. The anomaly you are seeing is a Windows only anomaly. > Elsewhere (on Solaris and Linux at least) the performance will be as > you would expect. For example, I did a quick test on Solaris with a > directory of 9000 files and the simple iteration took 22ms, the > iteration + reading the attributes took 88ms, and walkFileTree with > maxDepth==1 took 83ms. On Windows, the anomaly (or why is > Files.walkFileTree so much faster) is because the attributes are > obtained during the directory traversal so the implementation can > avoid re-reading them - if it re-read the attributes for each file > then it would take about the same time as calling > Attributes.readBasicFileAttributes for each file in the directory, an > operation that is expensive on Windows. One thing to say is that > difference isn't as obvious with NTFS - for example, I repeated your > test with a directory of 9000 files and the simple iteration took > 21ms, the iteration + reading the attributes took 237ms, and > Files.walkFileTree took 20ms. With FAT32 or when the volume is remote > then the difference is very obvious - I'll guess this is what you are > testing on. > > This issue does bring up the question as to if we need a method that > returns a DirectoryStream where the elements are a pair consisting of > the entry and its attributes. It's come up once or twice. From a > performance point of view it helps Windows, and maybe some custom file > systems. The other potential justification is convenience in that the > basic attributes will often be required when iterating over a > directory. It's worth looking at. > > -Alan. > You are right, I hadn't realized this was important, but the files were on a Samba share (Java running on Windows, scanning a network share on Linux using a path like: //Server/BigDir). So, the volume was remote as you suspected. --John From hjohn at xs4all.nl Sat Sep 5 16:55:22 2009 From: hjohn at xs4all.nl (John Hendrikx) Date: Sun, 06 Sep 2009 01:55:22 +0200 Subject: NIO performance In-Reply-To: <4AA2724B.3040600@sun.com> References: <4AA1ACC2.5020801@xs4all.nl> <4AA2724B.3040600@sun.com> Message-ID: <4AA2FA6A.5050509@xs4all.nl> Alan Bateman wrote: > Files.walkFileTree is essentially an internal iterator built on an > external iterator (DirectoryStream). So for the maxDepth == 1 case > then it is reasonable to expect the performance to be the same as > using DirectoryStream to iterate over all entries in the directory, > calling Attributes.readBasicFileAttributes to read the attributes of > each file. The anomaly you are seeing is a Windows only anomaly. > Elsewhere (on Solaris and Linux at least) the performance will be as > you would expect. For example, I did a quick test on Solaris with a > directory of 9000 files and the simple iteration took 22ms, the > iteration + reading the attributes took 88ms, and walkFileTree with > maxDepth==1 took 83ms. On Windows, the anomaly (or why is > Files.walkFileTree so much faster) is because the attributes are > obtained during the directory traversal so the implementation can > avoid re-reading them - if it re-read the attributes for each file > then it would take about the same time as calling > Attributes.readBasicFileAttributes for each file in the directory, an > operation that is expensive on Windows. One thing to say is that > difference isn't as obvious with NTFS - for example, I repeated your > test with a directory of 9000 files and the simple iteration took > 21ms, the iteration + reading the attributes took 237ms, and > Files.walkFileTree took 20ms. With FAT32 or when the volume is remote > then the difference is very obvious - I'll guess this is what you are > testing on. > > This issue does bring up the question as to if we need a method that > returns a DirectoryStream where the elements are a pair consisting of > the entry and its attributes. It's come up once or twice. From a > performance point of view it helps Windows, and maybe some custom file > systems. The other potential justification is convenience in that the > basic attributes will often be required when iterating over a > directory. It's worth looking at. > > -Alan. > I forgot one thing I thought of yesterday. Files.walkFileTree does not provide BasicFileAttributes for directories (preVisitDirectory does not supply them). This would mean the performance problems would return if the directory being scanned contains a lot of subdirectories, as in that case even with the FileVisitor I would be forced to call readBasicFileAttributes. It might be worth consideration to supply preVisitDirectory with BasicFileAttributes as well. --John From Alan.Bateman at Sun.COM Sun Sep 6 13:41:07 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Sun, 06 Sep 2009 21:41:07 +0100 Subject: NIO performance In-Reply-To: <4AA2FA6A.5050509@xs4all.nl> References: <4AA1ACC2.5020801@xs4all.nl> <4AA2724B.3040600@sun.com> <4AA2FA6A.5050509@xs4all.nl> Message-ID: <4AA41E63.9000509@sun.com> John Hendrikx wrote: > I forgot one thing I thought of yesterday. Files.walkFileTree does > not provide BasicFileAttributes for directories (preVisitDirectory > does not supply them). This would mean the performance problems would > return if the directory being scanned contains a lot of > subdirectories, as in that case even with the FileVisitor I would be > forced to call readBasicFileAttributes. > > It might be worth consideration to supply preVisitDirectory with > BasicFileAttributes as well. Right, it could help for the case where the directory attributes are needed, reading them is slow (such as using SMB), and where the ratio of directories to other file types is high. Even if the ratio isn't that high, there there may be some small benefit. So yes, worth looking at. Also postVisitDirectory (such as the "cp -r" case) but that isn't as clear cut as it would only be beneficial if the attributes from before the directory is visited are useful. -Alan. From Alan.Bateman at Sun.COM Wed Sep 9 05:22:20 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Wed, 09 Sep 2009 13:22:20 +0100 Subject: Need reviewer for 6876541: (file) Files.walkFileTree(...): no SecurityException if read access to the starting file is denied Message-ID: <4AA79DFC.8090006@sun.com> I need a reviewer for this faux pas in Files.walkFileTree. SecurityException should be thrown if access to the starting file is denied by the security manager. Instead, the exception is silently consumed and the starting file ignored (ie: walkFileTree does nothing). A secondary issue (Windows only) arises where a security manager allows access to a directory but not to the all entries in that directory. Such entries should be skipped but non-directory entries are still notified to the FileVisitor. The webrev is here: http://cr.openjdk.java.net/~alanb/6876541/webrev.00/ Thanks, Alan. From Christopher.Hegarty at Sun.COM Mon Sep 14 01:53:30 2009 From: Christopher.Hegarty at Sun.COM (Christopher Hegarty -Sun Microsystems Ireland) Date: Mon, 14 Sep 2009 09:53:30 +0100 Subject: Need reviewer for 6876541: (file) Files.walkFileTree(...): no SecurityException if read access to the starting file is denied In-Reply-To: <4AA79DFC.8090006@sun.com> References: <4AA79DFC.8090006@sun.com> Message-ID: <4AAE048A.7060308@sun.com> These changes look good. I have just one minor question about the testcase. Why does it disallow user.dir == test.src. Will this mean that the directory has implicit permission or is it just catching the case where the user tries to run the test outside of jtreg? The reason I ask is that in the case of failure it is sometimes useful to be able to run the test standalone. -Chris. Alan Bateman wrote: > > I need a reviewer for this faux pas in Files.walkFileTree. > SecurityException should be thrown if access to the starting file is > denied by the security manager. Instead, the exception is silently > consumed and the starting file ignored (ie: walkFileTree does nothing). > A secondary issue (Windows only) arises where a security manager allows > access to a directory but not to the all entries in that directory. Such > entries should be skipped but non-directory entries are still notified > to the FileVisitor. The webrev is here: > http://cr.openjdk.java.net/~alanb/6876541/webrev.00/ > > Thanks, > Alan. From Alan.Bateman at Sun.COM Mon Sep 14 06:52:45 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Mon, 14 Sep 2009 14:52:45 +0100 Subject: Need reviewer for 6876541: (file) Files.walkFileTree(...): no SecurityException if read access to the starting file is denied In-Reply-To: <4AAE048A.7060308@sun.com> References: <4AA79DFC.8090006@sun.com> <4AAE048A.7060308@sun.com> Message-ID: <4AAE4AAD.7020000@sun.com> Christopher Hegarty -Sun Microsystems Ireland wrote: > These changes look good. > > I have just one minor question about the testcase. Why does it > disallow user.dir == test.src. Will this mean that the directory has > implicit permission or is it just catching the case where the user > tries to run the test outside of jtreg? > > The reason I ask is that in the case of failure it is sometimes useful > to be able to run the test standalone. Thanks for reviewing. We could split this test into 3 if you think its worth it. For now, it needs to be run by jtreg and will be run 3 times, each with a different policy. To keep it simple it attempts to walk test.src (an alternative would have been to create a file tree, perhaps in the tmp dir and corresponding policy files). As the default policy grants reads access to the current directory then it would otherwise cause the "denyAll" and "grantTopOnly" tests to fail. -Alan. From Christopher.Hegarty at Sun.COM Mon Sep 14 07:14:22 2009 From: Christopher.Hegarty at Sun.COM (Christopher Hegarty -Sun Microsystems Ireland) Date: Mon, 14 Sep 2009 15:14:22 +0100 Subject: Need reviewer for 6876541: (file) Files.walkFileTree(...): no SecurityException if read access to the starting file is denied In-Reply-To: <4AAE4AAD.7020000@sun.com> References: <4AA79DFC.8090006@sun.com> <4AAE048A.7060308@sun.com> <4AAE4AAD.7020000@sun.com> Message-ID: <4AAE4FBE.9050706@sun.com> Alan Bateman wrote: > Christopher Hegarty -Sun Microsystems Ireland wrote: >> These changes look good. >> >> I have just one minor question about the testcase. Why does it >> disallow user.dir == test.src. Will this mean that the directory has >> implicit permission or is it just catching the case where the user >> tries to run the test outside of jtreg? >> >> The reason I ask is that in the case of failure it is sometimes useful >> to be able to run the test standalone. > Thanks for reviewing. > > We could split this test into 3 if you think its worth it. For now, it > needs to be run by jtreg and will be run 3 times, each with a different > policy. To keep it simple it attempts to walk test.src (an alternative > would have been to create a file tree, perhaps in the tmp dir and > corresponding policy files). As the default policy grants reads access > to the current directory then it would otherwise cause the "denyAll" and > "grantTopOnly" tests to fail. That answers my question. I just missed this the fact that the current dir has read access by default. -Chris. > > -Alan. From Alan.Bateman at Sun.COM Mon Sep 28 10:55:27 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Mon, 28 Sep 2009 18:55:27 +0100 Subject: 6884800: Path.newInputStream does not usefully implement available() Message-ID: <4AC0F88F.2050401@sun.com> I need a reviewer for this small change. A long standing issue (4648049) is that the InputStream returned by Channels.newInputStream(ReadableByteChannel) doesn't usefully implement the available method. 6884800 is the case where the channel is to the file, a case where available can be easily implemented. The webrev is here: http://cr.openjdk.java.net/~alanb/6884800/webrev.00/ Thanks, Alan. From esmond.pitt at bigpond.com Mon Sep 28 16:29:38 2009 From: esmond.pitt at bigpond.com (Esmond Pitt FACS) Date: Tue, 29 Sep 2009 09:29:38 +1000 Subject: Path.newInputStream does not usefully implement In-Reply-To: Message-ID: <20090928232538.HREI28976.nschwotgx01p.mx.bigpond.com@toshibakx1wgjb> > I need a reviewer for this small change. I don't know whether I qualify but it looks good to me. EJP From martinrb at google.com Mon Sep 28 20:22:21 2009 From: martinrb at google.com (Martin Buchholz) Date: Mon, 28 Sep 2009 20:22:21 -0700 Subject: 6884800: Path.newInputStream does not usefully implement available() In-Reply-To: <4AC0F88F.2050401@sun.com> References: <4AC0F88F.2050401@sun.com> Message-ID: <1ccfd1c10909282022p2537ae7bv8ea837194e6508f1@mail.gmail.com> You could check that available always reports all the remaining bytes e.g. after 239 totalRead += bytesRead; available() should equal fc.size() - totalRead for maximal usefulness. Martin On Mon, Sep 28, 2009 at 10:55, Alan Bateman wrote: > > I need a reviewer for this small change. ?A long standing issue (4648049) is > that the InputStream returned by > Channels.newInputStream(ReadableByteChannel) doesn't usefully implement the > available method. 6884800 is the case where the channel is to the file, a > case where available can be easily implemented. The webrev is here: > ?http://cr.openjdk.java.net/~alanb/6884800/webrev.00/ > > Thanks, > Alan. > > > From Christopher.Hegarty at Sun.COM Tue Sep 29 01:40:33 2009 From: Christopher.Hegarty at Sun.COM (Christopher Hegarty -Sun Microsystems Ireland) Date: Tue, 29 Sep 2009 09:40:33 +0100 Subject: 6884800: Path.newInputStream does not usefully implement available() In-Reply-To: <4AC0F88F.2050401@sun.com> References: <4AC0F88F.2050401@sun.com> Message-ID: <4AC1C801.4000305@sun.com> Looks fine. There is of course a minor change in behavior in that these InputStreams will now throw ChannelClosedException instead of returning 0, when the channel is closed. But I don't expect this will surprise anyone. -Chris. Alan Bateman wrote: > > I need a reviewer for this small change. A long standing issue > (4648049) is that the InputStream returned by > Channels.newInputStream(ReadableByteChannel) doesn't usefully implement > the available method. 6884800 is the case where the channel is to the > file, a case where available can be easily implemented. The webrev is here: > http://cr.openjdk.java.net/~alanb/6884800/webrev.00/ > > Thanks, > Alan. > >