Proposal for adding O_DIRECT support into JDK 9

[David Holmes]

Hi Paul,

On 5/11/2016 2:58 AM, Paul Sandoz wrote:
> Hi Lucy,
>
> I am struggling to find an entirely satisfactory location to place a method such as getPageSize, but java.lang.Runtime seems less worse than other places :-)

This does not seem an appropriate method to expose in this form at the 
java.lang.Runtime level. If we want a public, platform agnostic API for 
exposing virtual memory page sizes (plural) then we should design such 
an API IMHO.

David

> Paul.
>
>> On 3 Nov 2016, at 12:49, Lu, Yingqi <yingqi.lu at intel.com> wrote:
>>
>> Hi Paul and Peter,
>>
>> Thank you very much for pointing this out. We did not know about it J
>>
>> We will modify the patch accordingly and will send it here for review.
>>
>> Thanks,
>> Lucy
>>
>> From: Paul Sandoz [mailto:paul.sandoz at oracle.com]
>> Sent: Thursday, November 03, 2016 12:46 PM
>> To: Peter Levart <peter.levart at gmail.com>
>> Cc: Lu, Yingqi <yingqi.lu at intel.com>; Alan Bateman <Alan.Bateman at oracle.com>; nio-dev at openjdk.java.net; Kaczmarek, Eric <eric.kaczmarek at intel.com>; core-libs-dev at openjdk.java.net; Kharbas, Kishor <kishor.kharbas at intel.com>
>> Subject: Re: Proposal for adding O_DIRECT support into JDK 9
>>
>> Hi Peter,
>>
>> Thanks for pointing this out, you beat me to it :-)
>>
>> Note that the alignment for a unit size corresponding to page size will only be stable for direct byte buffers, since for heap byte buffers the GC can move the underlying byte array and the alignment properties may change.
>>
>> Paul.
>>
>> On 3 Nov 2016, at 12:38, Peter Levart <peter.levart at gmail.com> wrote:
>>
>> Hi Lucy,
>>
>> Did you know that in JDK 9 the following two methods have been added to java.nio.ByteBuffer:
>>
>> /** @return  The indexed byte's memory address modulus the unit size
>>   */
>> public final int alignmentOffset(int index, int unitSize);
>>
>> /** @return new byte buffer whose content is a shared and aligned subsequence of this buffer's content
>>   */
>> public final ByteBuffer alignedSlice(int unitSize);
>>
>>
>> So you could express your proposed methods ByteBuffer.allocateDirectAligned() and ByteBuffer.isAligned() with the above two methods in the following way:
>>
>> instead of:
>>
>>     ByteBuffer bb = ByteBuffer.allocateDirectAligned(capacity)
>>
>> you could do:
>>
>>     int pageSize = ...
>>     ByteBuffer bb = ByteBuffer.allocateDirect(capacity + pageSize - 1).alignedSlice(pageSize);
>>
>> And instead of:
>>
>>     bb.isAligned()
>>
>> you could do:
>>
>>     bb.alignmentOffset(pageSize) == 0
>>
>>
>>
>> The only thing that needs to be added is a public API to return the pageSize (i.e. Bits.pageSize()).
>>
>>
>> What do you think?
>>
>> Regards, Peter
>>
>> On 11/03/2016 12:34 AM, Lu, Yingqi wrote:
>> Hi All,
>>
>> Our most recent DirectIO patch is available at http://cr.openjdk.java.net/~igraves/8164900-3/
>>
>> In this version, we made following changes:
>>
>> 1. Removed the flag "direct" from FileDescriptor class. Instead, moved it to the FileChannelImpl class.
>>
>> 2. Provided a way for user to allocate a page aligned direct ByteBuffer.
>>     1) Added a constructor DirectByteBuffer(int cap, boolean direct) to allocate a direct ByteBuffer that is aligned to the page size.
>>     2) Added Util.getTemporaryAlignedDirectBuffer(int size)
>>     3) Added DirectByteBuffer.isAligned(int pos) to check if the buffer is aligned before doing native IO with it.
>>
>> 3. Moved all the alignment check from C code to Java code (mainly FileChannelImpl and IOUtil.java).
>>
>> 4. Made the DirectIO functionality consistent between read and write operations. With current version of the patch, user would be responsible for the alignment with file offset and IO size.
>>
>> 5. Made the API for DirectIO more extensible to all the supporting platforms.
>>     1) Unix OS specific code are done through UnixConstants.java.template and FileChannelImpl.c.
>>     2) Coded and tested for Linux and OS X (OS X testing is done through VirtualBox with OS X VM on top of Linux OS).
>>     3) Coded for Solaris. We do not have environment to test it so that we commented the changes out.
>>
>> 6. We added 4 test cases following the existing non-direct IO examples.
>>
>> 7. We did jtreg test for the entire nio package and no errors were found due to our changes.
>>
>> Please let us know your feedback and comment. Thank you very much for your time and consideration!
>>
>> Thanks,
>> Lucy
>>
>> -----Original Message-----
>> From: Alan Bateman [mailto:Alan.Bateman at oracle.com]
>> Sent: Monday, October 17, 2016 7:59 AM
>> To: Lu, Yingqi <yingqi.lu at intel.com>
>> Cc: nio-dev at openjdk.java.net; Kaczmarek, Eric <eric.kaczmarek at intel.com>;
>> Kharbas, Kishor <kishor.kharbas at intel.com>
>> Subject: Re: Proposal for adding O_DIRECT support into JDK 9
>>
>> On 12/10/2016 17:41, Lu, Yingqi wrote:
>>
>> :
>>
>> You are correct about the "extra copy" with DirectIO. Will it be acceptable if we
>> add a function "Util.getAlignedTemporaryDirectBuffer" and use that for the
>> DirectIO operation? In this case, I think we should be able to avoid the additional
>> copy?
>> Yes, that should work but it still lacks a way for the user to get an aligned buffer
>> and so you will always be copying in and out of an aligned buffer. The other thing
>> is the sizing of the I/O operation where I think you will also need a way to expose
>> the multiple (or block size) to the user.
>>
>> -Alan.
>