Batch operations

[Alan Bateman]

On 17/05/2015 15:00, Christian Beikov wrote:
> Hello!
>
> I am new on this list and I am not sure if this is the right place, 
> but I have a question regarding "batch" like operations.
>
> The file system implementation I use is for AWS S3 and it has to do a 
> network roundtrip for every operation.
> I recently have encountered the need to delete multiple files at once 
> and obviously, the overhead for the roundtrips is unnecessary since 
> the S3 SDK supports a batch delete operation. I then implemented a 
> utility method that used the S3 SDK to delete a list of paths when the 
> first path is from a S3 file system. This works for me, but I was 
> curious if there is a need for such batch operations in the 
> FileSystemProvider class.
>
> I think it is necessary to add some more higher level methods like the 
> batch operations to the FileSystemProvider and let the 
> java.nio.files.Files delegate to these instead of implementing it in a 
> utility class. By doing so, file system providers can do a lot of 
> operations much more efficiently if they like to override it.
>
> What do you think about that? I would very much like to help improve 
> the API and see more file system provider implementations with good 
> performance!

Out of curiosity, is this a FUSE-based file system or a 
FileSystemProvider that uses the Amazon REST API?

I'm also curious if you've attempted use a thread pool so that you do 
the file operations asynchronously. That should help with the latency 
when doing multiple deletes, assuming they are independent.

In general then the file system API was originally intended to provide 
primitives that map what most native file systems support. Once you 
extend it to do multiple operations at the same time then it requires a 
slightly different API, particularly when you get into dealing with 
errors/exceptions. Cloud is clearly important but I think it would be 
good to see how far you can get, and see how common a need this is, 
before looking at extending the API.

-Alan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20150518/e0e3a436/attachment.html>