Proposal for adding O_DIRECT support into JDK 9

[Lu, Yingqi]

Dear All,

This is a proposal for adding O_DIRECT support into JDK 9 for reading/writing from/to a file on Linux platform. O_DIRECT is a file-open flag to pass to OPEN (2). It tries to minimize cache effects of the I/O to and from this file. File I/O is done directly to/from user-space buffers. Please refer to http://man7.org/linux/man-pages/man2/open.2.html for more details.

The advantage of this feature is to provide consistent and predictable IO throughput, bypassing the influence of Linux filesystem cache. It is useful for applications which already have their own caching layers, such as databases Apache HBase and Apache Cassandra. In addition, for applications like Apache Spark shuffle phase that have random file accesses and no access locality (read the file only once from a random location), this feature also helps performance by minimizing trashing the filesystem cache.

The proposal adds 4 API methods to java/io/FileInputStream.java, java/io/OutputStream.java to enable the feature. In addition, it add O_DIRECT with two more modes "ro" (read-only and direct) and "rwo" (read-write and direct) to java/io/RandomAccessFile.java.

public FileInputStream(String name, boolean direct) throws FileNotFoundException

public FileInputStream(File file, boolean direct) throws FileNotFoundException

public FileOutputStream(String name, boolean append, boolean direct) throws FileNotFoundException

public FileOutputStream(File file, boolean append, boolean direct) throws FileNotFoundException

Many database applications written in C/C++ such as MySQL and MongoDB have already used O_DIRECT for certain DB operations. Many Java based BigData applications such as Cassandra are already using it from third party library named JNA. (https://github.com/java-native-access/jna) Our internal version of Hadoop Distributed File System (HDFS) that has O_DIRECT enabled via JNI functions also show significant throughput improvements.

We opened a JBS ticket at https://bugs.openjdk.java.net/browse/JDK-8164900 and the initial version of the JDK9 patch is available at http://cr.openjdk.java.net/~vdeshpande/8164900/webrev.00/. By using this patch, we saw good results up to 1.8x improvement on random read throughput on a micro workload named IOMeter with both SATA SSDs and Intel PCIeSSDs. The workload is available at https://github.com/persado/iometer/blob/master/src/main/java/com/persado/oss/iometer/IOMeter.java

Please review the patch and let us know your feedback.

Thanks,
Yingqi Lu (Lucy)