NIO performance

[John Hendrikx]

Alan Bateman wrote:
> Files.walkFileTree is essentially an internal iterator built on an 
> external iterator (DirectoryStream). So for the maxDepth == 1 case 
> then it is reasonable to expect the performance to be  the same as 
> using DirectoryStream to iterate over all entries in the directory, 
> calling Attributes.readBasicFileAttributes to read the attributes of 
> each file. The anomaly you are seeing is a Windows only anomaly. 
> Elsewhere (on Solaris and Linux at least) the performance will be as 
> you would expect. For example, I did a quick test on Solaris with a 
> directory of 9000 files and the simple iteration took 22ms, the 
> iteration + reading the attributes took 88ms, and walkFileTree with 
> maxDepth==1 took 83ms. On Windows, the anomaly (or why is 
> Files.walkFileTree so much faster) is because the attributes are 
> obtained during the directory traversal so the implementation can 
> avoid re-reading them - if it re-read the attributes for each file 
> then it would take about the same time as calling 
> Attributes.readBasicFileAttributes for each file in the directory, an 
> operation that is expensive on Windows. One thing to say is that 
> difference isn't as obvious with NTFS - for example, I repeated your 
> test with a directory of 9000 files and the simple iteration took 
> 21ms, the iteration + reading the attributes took 237ms, and 
> Files.walkFileTree took 20ms. With FAT32 or when the volume is remote 
> then the difference is very obvious - I'll guess this is what you are 
> testing on.
>
> This issue does bring up the question as to if we need a method that 
> returns a DirectoryStream where the elements are a pair consisting of 
> the entry and its attributes. It's come up once or twice. From a 
> performance point of view it helps Windows, and maybe some custom file 
> systems. The other potential justification is convenience in that the 
> basic attributes will often be required when iterating over a 
> directory. It's worth looking at.
>
> -Alan.
>
I forgot one thing I thought of yesterday.  Files.walkFileTree does not 
provide BasicFileAttributes for directories (preVisitDirectory does not 
supply them).  This would mean the performance problems would return if 
the directory being scanned contains a lot of subdirectories, as in that 
case even with the FileVisitor I would be forced to call 
readBasicFileAttributes.

It might be worth consideration to supply preVisitDirectory with 
BasicFileAttributes as well.

--John