Proposal for adding O_DIRECT support into JDK 9

[Lu, Yingqi]

Hi Volker,

Thank you very much for testing it on AIX and patching the current version for AIX. This is really helpful!

Regarding to setting O_DIRECT to a file descriptor, I remember Alan had some concerns before. I will let either Alan or Brian to double confirm. Alan/Brian, please also provide some guidance on how should we work this feature with AIX.

I also found that Windows flag setting FILE_FLAG_NO_BUFFERING and FILE_FLAG_WRITE_THROUGH are missing in the most recent version of the patch. I will add it back in the next version together with the new test cases and hopefully changes for AIX.

Thanks, 
Lucy

>-----Original Message-----
>From: Volker Simonis [mailto:volker.simonis at gmail.com]
>Sent: Wednesday, August 30, 2017 10:22 AM
>To: Lu, Yingqi <yingqi.lu at intel.com>
>Cc: Alan Bateman <Alan.Bateman at oracle.com>; Brian Burkhalter
><brian.burkhalter at oracle.com>; nio-dev at openjdk.java.net; Markle, Seth
><smarkle at amazon.com>
>Subject: Re: Proposal for adding O_DIRECT support into JDK 9
>
>Hi Lucy,
>
>I've tried your changes on AIX. The good news is that it compiles fine. The bad
>news is that the current version doesn't really work on AIX. That's because AIX
>only allows to set O_DIRECT when calling
>open() but not later on, on a file descriptor with fcntl() (see [1]).
>I could fix at least the current regression tests by setting the O_DIRECT flag
>right in UnixChannelFactory.open() as demonstrated in the following addon to
>your latest change:
>
>http://cr.openjdk.java.net/~simonis/webrevs/2017/8164900_AIX_addon/
>
>Unfortunately I'm not that deep into this topic to foresee if my fix catches
>really all paths or if it is really necessary to set O_DIRECT on a plain file
>descriptor which we're not opening ourselves. In the latter case we will either
>have to switch O_DIRECT support on AIX completely off or support it only in
>the case where we open the file ourselves and disallow it for plain file
>descriptors.
>
>Besides that, I've also updated the tests to support AIX. The changes in the
>Makefile are just for reference because its contents will have to be moved to
>jdk/make/test/JtregNative.gmk anyway.
>
>Also please use jdk.test.lib.Platform from the testlibrary for platform
>detection instead of parsing the properties yourself. You can look for example
>at test/java/lang/System/OsVersionTest.java to see how it can be used.
>
>Regards,
>Volker
>
>[1]
>https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.a
>ix.files/fcntl.h.htm
>
>On Wed, Aug 30, 2017 at 6:16 PM, Lu, Yingqi <yingqi.lu at intel.com> wrote:
>> Hi Volker,
>>
>> Thank you very much for the detailed instructions.
>>
>> Last night, I tried to do "make all" (normally I only do "make jdk") and found
>the native libraries were built in the build/linux-x86_64-normal-server-
>release/support/test/jdk/jtreg/native/lib. When using "-
>nativepath:PATH_TO_LIB" with jtreg, the test passed. I will try "make test-
>image-jdk-jtreg-native" as well to make sure it works too.
>>
>> Thank you!
>> Lucy
>>
>>>-----Original Message-----
>>>From: Volker Simonis [mailto:volker.simonis at gmail.com]
>>>Sent: Wednesday, August 30, 2017 12:48 AM
>>>To: Lu, Yingqi <yingqi.lu at intel.com>
>>>Cc: Alan Bateman <Alan.Bateman at oracle.com>; Brian Burkhalter
>>><brian.burkhalter at oracle.com>; nio-dev at openjdk.java.net; Markle, Seth
>>><smarkle at amazon.com>
>>>Subject: Re: Proposal for adding O_DIRECT support into JDK 9
>>>
>>>Hi Lucy,
>>>
>>>running the tests with native libs is a little tricky and not very
>>>well documented but if you know how to do it it is quite simple :)
>>>
>>>- after doing a build (i.e. make images) you can build the native
>>>parts which are required for running the corresponding tests by making
>>>the following
>>>targets:
>>>  make test-image-jdk-jtreg-native (for the jdk/test native parts)
>>>  make test-image-hotspot-jtreg-native (for the hotspot/test native
>>>parts)
>>>- this will create the following subdirectories in your build
>>>directory which can be directly used as arguments to -nativepath when
>>>running the corresponding JTreg tests:
>>>  ./support/test/jdk/jtreg/native/lib (for the jdk/test native parts)
>>>  ./support/test/hotspot/jtreg/native/lib (for the hotspot/test native
>>>parts)
>>>
>>>What Alan meant was that he wanted you to build the native part of
>>>your tests with the make targets mentioned above (i.e.
>>>test-image-jdk-jtreg-native) by putting the build instructions into
>>>jdk/make/test/JtregNative.gmk. StringPlatformChars.java was meant as
>>>an example of a test which needs a native library which is built in
>>>the described way. I hearby second Alan's suggestion :)
>>>
>>>Regards,
>>>Volker
>>>
>>>On Tue, Aug 29, 2017 at 7:19 PM, Lu, Yingqi <yingqi.lu at intel.com> wrote:
>>>> Hi Alan,
>>>>
>>>> Thank you for your suggestion. I first tried to run
>>>> StringPlatformChars.java
>>>with jtreg and wanted to understand how it works. However the test
>>>failed with error message " test result: Error. Use -nativepath to
>>>specify the location of native code"
>>>>
>>>> The command I use to run is:
>>>> jtreg
>>>> jdk/test/java/lang/String/nativeEncoding/StringPlatformChars.java
>>>>
>>>> Please let me know if I missed anything.
>>>>
>>>> Thanks,
>>>> Lucy
>>>>
>>>>>-----Original Message-----
>>>>>From: Alan Bateman [mailto:Alan.Bateman at oracle.com]
>>>>>Sent: Tuesday, August 29, 2017 3:24 AM
>>>>>To: Brian Burkhalter <brian.burkhalter at oracle.com>; Lu, Yingqi
>>>>><yingqi.lu at intel.com>
>>>>>Cc: Markle, Seth <smarkle at amazon.com>; nio-dev at openjdk.java.net
>>>>>Subject: Re: Proposal for adding O_DIRECT support into JDK 9
>>>>>
>>>>>On 28/08/2017 21:28, Brian Burkhalter wrote:
>>>>>> Hi Lucy,
>>>>>>
>>>>>> I am unsure of the best way to get the native code to build as
>>>>>> part of the test build. One example however of where this is done
>>>>>> is in <JDK_REPO>/jdk/test/java/awt/JAWT but the test is a shell
>>>>>> test and we are trying to move away from those. I do not know
>>>>>> exactly what Alan intends by his comment. There are other tests
>>>>>> which have pre-compiled shared libraries checked into the SCM. If
>>>>>> there is a problem with the native library itself or in locating
>>>>>> it, then I would think the test should fail and we would be
>>>>>> alerted that there is a
>>>problem.
>>>>>An example to study is the test in
>>>>>jdk/test/java/lang/String/nativeEncoding. The make file to build it
>>>>>is jdk/make/test/JtregNative.gmk. No issue with splitting this to a
>>>>>separate issue if you decide to include it.
>>>>>
>>>>>-Alan
>>>>