AIO and some advice needed.

[Alan Bateman]

Avinash Lakshman wrote:
> Thanks Alan. Do you have any performance benchmark numbers that you 
> can share comparing the performance of synchronous and asynchronous 
> approaches on Linux?
I don't have numbers to share but one thing to say is that kernel AIO on 
Linux isn't supported (to my knowledge) for files that aren't opened for 
direct I/O. Direct I/O is very much for databases and a few cases that 
do their own caching. This means that when you are using an asynchronous 
file channel it is actually doing synchronous I/O with a thread pool. 
This works surprisingly well but you need to play around with sizing of 
the thread pool to get the right concurrency. In the future we can make 
use of the kernel support if it changes. I don't think I understand your 
environment sufficiently to make any reasonable recommendations. I 
understand there is a big 45GB file but it's not clear to me how it is 
used. If indeed you can read and process areas of the file in parallel 
then try AsynchronousFileChannel and send feedback. FileChannel also 
defines a positional read method and so can support concurrent access 
without concern for the global file position.

-Alan.