AIO and some advice needed.

[David M. Lloyd]

On 11/28/2008 08:25 PM, Avinash Lakshman wrote:
> Looks like Linux has AIO support only if the file has been open with 
> O_DIRECT flag. I guess AIO is moot at this point. However what does it 
> take for us to request O_DIRECT support in this NIO release? We are 
> building a large scale storage system called Cassandra, here at 
> Facebook, details of which can be found here 

Here's some comments by Linus about O_DIRECT as it pertains to regular files:

"There really is no valid reason for EVER using O_DIRECT. You need a buffer 
whatever IO you do, and it might as well be the page cache. There are 
better ways to control the page cache than play games and think that a page 
cache isn't necessary."

http://lkml.org/lkml/2007/1/10/233

Read the whole thread - the following comments, and the questions that lead 
up to it - it's full of interesting tidbits, like this one about the 
realities of what it means to bypass OS caching: 
http://lkml.org/lkml/2007/1/11/129

O_DIRECT only makes sense on raw devices: http://lkml.org/lkml/2007/1/11/157

The main problem with O_DIRECT is that userspace pages cannot be "stolen" 
to act as buffer pages for the filesystem write, thus the write() has to 
block until the data can be written out all the way: 
http://lkml.org/lkml/2007/1/11/157

That said, later on in the thread there are some good counterpoints as well.

One thing to consider is instead using FileChannel.map() and 
FileChannel.force() or MappedByteBuffer.force() instead, possibly in 
combination with a thread pool... this should have similar characteristics 
to O_DIRECT and has the futher advantage of being available since 1.4 as well.

- DML