From Alan.Bateman at oracle.com Fri Sep 1 14:36:44 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 1 Sep 2017 15:36:44 +0100 Subject: RFR JDK-8186751: Add ISO-8859-16 Charset support In-Reply-To: <59A7711F.9060801@oracle.com> References: <59A7711F.9060801@oracle.com> Message-ID: <13fc862e-60d8-d6e7-df48-28f62f2e451e@oracle.com> On 31/08/2017 03:14, Xueming Shen wrote: > Hi, > > Please help review the changeset to add the 8859_1 charset support. > > issue: https://bugs.openjdk.java.net/browse/JDK-8186751 > webrev: http://cr.openjdk.java.net/~sherman/8186751/webrev Looks good. -Alan From brian.burkhalter at oracle.com Fri Sep 1 14:46:35 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 1 Sep 2017 07:46:35 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> References: <0EE00E4F-EB7F-4C25-8955-E06AA4659BA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EAA14@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EF8EE@ORSMSX113.amr.corp.intel.com> <9F95A48E-EF98-4EB0-93F4-39CC24A02867@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EFC81@ORSMSX113.amr.corp.intel.com> <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> Message-ID: <2F421C67-CA19-45C4-A6C7-FC9749B6D2CD@oracle.com> Hi Lucy, On Aug 31, 2017, at 3:49 PM, Lu, Yingqi wrote: > Here is the webrev.07 version of the patch:http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/ > > In this version, I have addressed the following issues: > 2. Added the filesystem type check on Solaris for DirectIOTest. > This check should be in all the tests as they all fail on Solaris. Perhaps the code could be put in a static method on DirectIOTest which could be called by all five tests? > 3. Checked if there is an equivalent system call like mincore in Windows to enable DirectIOTest for Windows. I could not find anything unfortunately. > OK I do not think that is a big deal as this is not the target area anyway. > Issues that are still open: > > 1. How should we address the issue of AIX OS does not support setting O_DIRECT after the open() > It would be good to hear others comments as well on this. > Please take a look at this version and let us know your feedback. I would also like to learn what will be a proper approach to solve the AIX issue and what kind of details we need to provide for the exception messages. I will try to incorporate all of those changes into the next version. > I?ll try to get back to you on these points today. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Fri Sep 1 14:56:43 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 1 Sep 2017 07:56:43 -0700 Subject: JDK 10 RFR of 8147615: (fc) FileChannelImpl has no finalizer Message-ID: Please review at your convenience: https://bugs.openjdk.java.net/browse/JDK-8147615 http://cr.openjdk.java.net/~bpb/8147615/webrev.00/ There is a question at line 106 of FileChannelImpl.java which needs to be resolved. Another question is whether a synchronized block should be used in or around invalidateAndClose() in addition to relying on the pseudo-synchronization of checking the FileDescriptor validity. It is necessary to invalidate the FileDescriptor in any case as otherwise ?Bad file descriptor? exceptions occur. This is likely due to the cleaning action attempting to close a native file descriptor which was already closed when the channel was closed. That this actually works was verified by instrumenting FileChannelImpl$Closer#run() with a print statement which was since removed. The issue has however been labelled ?noreg-hard? as there appears to be no way to verify the fix without such instrumentation or via existing public APIs. Thanks, Brian From yingqi.lu at intel.com Fri Sep 1 16:13:30 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Fri, 1 Sep 2017 16:13:30 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <2F421C67-CA19-45C4-A6C7-FC9749B6D2CD@oracle.com> References: <0EE00E4F-EB7F-4C25-8955-E06AA4659BA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EAA14@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EF8EE@ORSMSX113.amr.corp.intel.com> <9F95A48E-EF98-4EB0-93F4-39CC24A02867@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EFC81@ORSMSX113.amr.corp.intel.com> <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <2F421C67-CA19-45C4-A6C7-FC9749B6D2CD@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD11F5BC3@ORSMSX113.amr.corp.intel.com> Hi Brian, I will add the Solaris check for all the tests now and wait for your feedback on the additional tests. All these changes will go into webrev.08. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Friday, September 01, 2017 7:47 AM To: Lu, Yingqi Cc: Alan Bateman ; nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, On Aug 31, 2017, at 3:49 PM, Lu, Yingqi > wrote: Here is the webrev.07 version of the patch:http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/ In this version, I have addressed the following issues: 2. Added the filesystem type check on Solaris for DirectIOTest. This check should be in all the tests as they all fail on Solaris. Perhaps the code could be put in a static method on DirectIOTest which could be called by all five tests? 3. Checked if there is an equivalent system call like mincore in Windows to enable DirectIOTest for Windows. I could not find anything unfortunately. OK I do not think that is a big deal as this is not the target area anyway. Issues that are still open: 1. How should we address the issue of AIX OS does not support setting O_DIRECT after the open() It would be good to hear others comments as well on this. Please take a look at this version and let us know your feedback. I would also like to learn what will be a proper approach to solve the AIX issue and what kind of details we need to provide for the exception messages. I will try to incorporate all of those changes into the next version. I'll try to get back to you on these points today. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ulf.Zibis at CoSoCo.de Fri Sep 1 18:22:39 2017 From: Ulf.Zibis at CoSoCo.de (Ulf Zibis) Date: Fri, 1 Sep 2017 20:22:39 +0200 Subject: RFR JDK-8186751: Add ISO-8859-16 Charset support In-Reply-To: <59A87FC0.4090505@oracle.com> References: <59A7711F.9060801@oracle.com> <59A87FC0.4090505@oracle.com> Message-ID: Hi again, Am 31.08.2017 um 23:29 schrieb Xueming Shen: > The 8859-16 aliases are the copy/paste from > https://www.iana.org/assignments/character-sets/character-sets.xhtml > > From the same page we have the followings for 8859-15 > ISO_8859-15 > Latin-9 > csISO885915 > > It appears only the first one is listed in our alias list (explicitly > commented). > Don't have the memory for the history (8859-15 was added a long time ago), > not sure whether the other two were missed at the very beginning or were > added later in the iana. Since we are here, I added them in. Fine! > http://cr.openjdk.java.net/~sherman/8186751/webrev > (the old one is copied to webrev.00) - I think, the comment should now be in plural: # IANA alias*es *- Also I think, it would make sense to at least add "ISO8859-16" and "ISO8859_16" as # Other aliases - "ISO-8859-15" must not be listed as alias, as it already is the canonical name. - Not sure, where "LATIN0" and "csISOlatin0" come from. -Ulf -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Fri Sep 1 19:09:06 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 1 Sep 2017 12:09:06 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> References: <0EE00E4F-EB7F-4C25-8955-E06AA4659BA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EAA14@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EF8EE@ORSMSX113.amr.corp.intel.com> <9F95A48E-EF98-4EB0-93F4-39CC24A02867@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EFC81@ORSMSX113.amr.corp.intel.com> <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> Message-ID: <5B82CAC0-CBC5-4C48-8B36-83F8ECB19020@oracle.com> Hi Lucy, DirectIOTest fails for me on Linux, macOS, and Solaris with the message "java.lang.UnsatisfiedLinkError: no DirectIO in java.library.path.? There is a compilation error on Windows which is fixed by the patch included below. With this patch applied however the four read/write direct tests fail with this error: java.lang.IllegalArgumentException: Unit size not a power of two: -1 at java.base/java.nio.ByteBuffer.alignmentOffset(ByteBuffer.java:1659) at java.base/java.nio.ByteBuffer.alignedSlice(ByteBuffer.java:1729) at java.base/sun.nio.ch.Util.getTemporaryAlignedDirectBuffer(Util.java:263) at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:72) at java.base/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:277) Thanks, Brian On Aug 31, 2017, at 3:49 PM, Lu, Yingqi wrote: > 1. Following the example of stringPlatformChars, made the DirectIOTest native library being compiled automatically. Removed DirectIOTest.java, README file and Makefile. --- a/src/java.base/windows/classes/sun/nio/ch/FileDispatcherImpl.java +++ b/src/java.base/windows/classes/sun/nio/ch/FileDispatcherImpl.java @@ -132,7 +132,7 @@ CharBuffer buffer = CharBuffer.allocate(filePath.length()); buffer.put(filePath); try { - result = setDirect0(buffer); + result = setDirect0(fd, buffer); } catch (IOException e) { throw new UnsupportedOperationException ("Error setting up DirectIO", e); @@ -195,5 +195,6 @@ static native long duplicateHandle(long fd) throws IOException; - static native int setDirect0(CharBuffer buffer) throws IOException; + static native int setDirect0(FileDescriptor fd, CharBuffer buffer) + throws IOException; } --- a/src/java.base/windows/native/libnio/ch/FileDispatcherImpl.c +++ b/src/java.base/windows/native/libnio/ch/FileDispatcherImpl.c @@ -459,11 +459,11 @@ JNIEXPORT jint JNICALL Java_sun_nio_ch_FileDispatcherImpl_setDirect0(JNIEnv *env, jclass this, - jobject buffer) + jobject fdObj, jobject buffer) { jint result = -1; - HANDLE orig = (HANDLE)(handleval(env, fObj)); + HANDLE orig = (HANDLE)(handleval(env, fdObj)); HANDLE modify = ReOpenFile(orig, 0, 0, FILE_FLAG_NO_BUFFERING | FILE_FLAG_WRITE_THROUGH); -------------- next part -------------- An HTML attachment was scrubbed... URL: From fguillaume at nuxeo.com Fri Sep 1 19:16:13 2017 From: fguillaume at nuxeo.com (Florent Guillaume) Date: Fri, 1 Sep 2017 21:16:13 +0200 Subject: RFR JDK-8186751: Add ISO-8859-16 Charset support In-Reply-To: References: <59A7711F.9060801@oracle.com> <59A87FC0.4090505@oracle.com> Message-ID: Hi, >From https://www.cs.tut.fi/~jkorpela/latin9.html: > Originally (http://www.indigo.ie/egt/standards/iso8859/latin00.html) the project > which lead to the creation of ISO Latin 9 used the working name "Latin Alphabet > Number Zero" for it. Therefore it has often been referred to as "Latin 0". I don't think it should be included in the aliases. Florent On Fri, Sep 1, 2017 at 8:22 PM, Ulf Zibis wrote: > Hi again, > > Am 31.08.2017 um 23:29 schrieb Xueming Shen: > > The 8859-16 aliases are the copy/paste from > https://www.iana.org/assignments/character-sets/character-sets.xhtml > > From the same page we have the followings for 8859-15 > ISO_8859-15 > Latin-9 > csISO885915 > > It appears only the first one is listed in our alias list (explicitly > commented). > Don't have the memory for the history (8859-15 was added a long time ago), > not sure whether the other two were missed at the very beginning or were > added later in the iana. Since we are here, I added them in. > > > Fine! > > http://cr.openjdk.java.net/~sherman/8186751/webrev > (the old one is copied to webrev.00) > > > - I think, the comment should now be in plural: # IANA alias > *es *- Also I think, it would make sense to at least add "ISO8859-16" and > "ISO8859_16" as # Other aliases > - "ISO-8859-15" must not be listed as alias, as it already is the > canonical name. > - Not sure, where "LATIN0" and "csISOlatin0" come from. > > -Ulf > > -- [image: Nuxeo] Florent Guillaume Head of R&D Twitter: @efge -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Fri Sep 1 19:18:01 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Fri, 1 Sep 2017 19:18:01 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <5B82CAC0-CBC5-4C48-8B36-83F8ECB19020@oracle.com> References: <0EE00E4F-EB7F-4C25-8955-E06AA4659BA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EAA14@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EF8EE@ORSMSX113.amr.corp.intel.com> <9F95A48E-EF98-4EB0-93F4-39CC24A02867@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EFC81@ORSMSX113.amr.corp.intel.com> <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <5B82CAC0-CBC5-4C48-8B36-83F8ECB19020@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD11F5F96@ORSMSX113.amr.corp.intel.com> Hi Brain, I had that issue before. I solved it by provide the -nativepath: to jtreg (only test it on Linux). Are you already using the parameter? I will have to look into the Windows code. Will let you know soon. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Friday, September 01, 2017 12:09 PM To: Lu, Yingqi Cc: Alan Bateman ; nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, DirectIOTest fails for me on Linux, macOS, and Solaris with the message "java.lang.UnsatisfiedLinkError: no DirectIO in java.library.path." There is a compilation error on Windows which is fixed by the patch included below. With this patch applied however the four read/write direct tests fail with this error: java.lang.IllegalArgumentException: Unit size not a power of two: -1 at java.base/java.nio.ByteBuffer.alignmentOffset(ByteBuffer.java:1659) at java.base/java.nio.ByteBuffer.alignedSlice(ByteBuffer.java:1729) at java.base/sun.nio.ch.Util.getTemporaryAlignedDirectBuffer(Util.java:263) at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:72) at java.base/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:277) Thanks, Brian On Aug 31, 2017, at 3:49 PM, Lu, Yingqi > wrote: 1. Following the example of stringPlatformChars, made the DirectIOTest native library being compiled automatically. Removed DirectIOTest.java, README file and Makefile. --- a/src/java.base/windows/classes/sun/nio/ch/FileDispatcherImpl.java +++ b/src/java.base/windows/classes/sun/nio/ch/FileDispatcherImpl.java @@ -132,7 +132,7 @@ CharBuffer buffer = CharBuffer.allocate(filePath.length()); buffer.put(filePath); try { - result = setDirect0(buffer); + result = setDirect0(fd, buffer); } catch (IOException e) { throw new UnsupportedOperationException ("Error setting up DirectIO", e); @@ -195,5 +195,6 @@ static native long duplicateHandle(long fd) throws IOException; - static native int setDirect0(CharBuffer buffer) throws IOException; + static native int setDirect0(FileDescriptor fd, CharBuffer buffer) + throws IOException; } --- a/src/java.base/windows/native/libnio/ch/FileDispatcherImpl.c +++ b/src/java.base/windows/native/libnio/ch/FileDispatcherImpl.c @@ -459,11 +459,11 @@ JNIEXPORT jint JNICALL Java_sun_nio_ch_FileDispatcherImpl_setDirect0(JNIEnv *env, jclass this, - jobject buffer) + jobject fdObj, jobject buffer) { jint result = -1; - HANDLE orig = (HANDLE)(handleval(env, fObj)); + HANDLE orig = (HANDLE)(handleval(env, fdObj)); HANDLE modify = ReOpenFile(orig, 0, 0, FILE_FLAG_NO_BUFFERING | FILE_FLAG_WRITE_THROUGH); -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Fri Sep 1 20:14:13 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 1 Sep 2017 13:14:13 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD11F5F96@ORSMSX113.amr.corp.intel.com> References: <0EE00E4F-EB7F-4C25-8955-E06AA4659BA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EAA14@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EF8EE@ORSMSX113.amr.corp.intel.com> <9F95A48E-EF98-4EB0-93F4-39CC24A02867@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EFC81@ORSMSX113.amr.corp.intel.com> <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! ! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <5B82CAC0-CBC5-4C48-8B36-83F8ECB19020@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F5F96@ORSMSX113.amr.corp.intel.com> Message-ID: <88907196-8AC4-4111-BCFF-4ED037C66FEB@oracle.com> Hi Lucy, On Sep 1, 2017, at 12:18 PM, Lu, Yingqi wrote: > I had that issue before. I solved it by provide the -nativepath: to jtreg (only test it on Linux). Are you already using the parameter? It works on macOS when I run it manually with that parameter. I do not know however to get it to work inside our harness but that is not your problem. > I will have to look into the Windows code. Will let you know soon. OK. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Fri Sep 1 22:15:53 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Fri, 1 Sep 2017 22:15:53 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD11F5F96@ORSMSX113.amr.corp.intel.com> References: <0EE00E4F-EB7F-4C25-8955-E06AA4659BA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EAA14@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EF8EE@ORSMSX113.amr.corp.intel.com> <9F95A48E-EF98-4EB0-93F4-39CC24A02867@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EFC81@ORSMSX113.amr.corp.intel.com> <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <5B82CAC0-CBC5-4C48-8B36-83F8ECB19020@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F5F96@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD11F648A@ORSMSX113.amr.corp.intel.com> Hi Brian, When I add back the Windows flags, I forgot to check on the file descriptor. Your patch perfectly solved the issue. Thank you!! I also patched the jdk/make/test/JtregNative.gmk to make sure the native library is not going to be built on Windows. Interestingly, I tried all 4 tests on Windows and they all passed. $ ./jtreg/bin/jtreg jdk/test/java/nio/channels/FileChannel/directio Directory "JTwork" not found: creating Directory "JTreport" not found: creating Test results: passed: 4 Report written to C:\cygwin64\home\ylu41\jdk10-directio-0831\JTreport\html\report.html Results written to C:\cygwin64\home\ylu41\jdk10-directio-0831\JTwork The test environment is Windows 10 enterprise 64 bit. I did everything through Cygwin. Thanks, Lucy From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of Lu, Yingqi Sent: Friday, September 01, 2017 12:18 PM To: Brian Burkhalter Cc: nio-dev at openjdk.java.net Subject: RE: Proposal for adding O_DIRECT support into JDK 9 Hi Brain, I had that issue before. I solved it by provide the -nativepath: to jtreg (only test it on Linux). Are you already using the parameter? I will have to look into the Windows code. Will let you know soon. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Friday, September 01, 2017 12:09 PM To: Lu, Yingqi > Cc: Alan Bateman >; nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, DirectIOTest fails for me on Linux, macOS, and Solaris with the message "java.lang.UnsatisfiedLinkError: no DirectIO in java.library.path." There is a compilation error on Windows which is fixed by the patch included below. With this patch applied however the four read/write direct tests fail with this error: java.lang.IllegalArgumentException: Unit size not a power of two: -1 at java.base/java.nio.ByteBuffer.alignmentOffset(ByteBuffer.java:1659) at java.base/java.nio.ByteBuffer.alignedSlice(ByteBuffer.java:1729) at java.base/sun.nio.ch.Util.getTemporaryAlignedDirectBuffer(Util.java:263) at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:72) at java.base/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:277) Thanks, Brian On Aug 31, 2017, at 3:49 PM, Lu, Yingqi > wrote: 1. Following the example of stringPlatformChars, made the DirectIOTest native library being compiled automatically. Removed DirectIOTest.java, README file and Makefile. --- a/src/java.base/windows/classes/sun/nio/ch/FileDispatcherImpl.java +++ b/src/java.base/windows/classes/sun/nio/ch/FileDispatcherImpl.java @@ -132,7 +132,7 @@ CharBuffer buffer = CharBuffer.allocate(filePath.length()); buffer.put(filePath); try { - result = setDirect0(buffer); + result = setDirect0(fd, buffer); } catch (IOException e) { throw new UnsupportedOperationException ("Error setting up DirectIO", e); @@ -195,5 +195,6 @@ static native long duplicateHandle(long fd) throws IOException; - static native int setDirect0(CharBuffer buffer) throws IOException; + static native int setDirect0(FileDescriptor fd, CharBuffer buffer) + throws IOException; } --- a/src/java.base/windows/native/libnio/ch/FileDispatcherImpl.c +++ b/src/java.base/windows/native/libnio/ch/FileDispatcherImpl.c @@ -459,11 +459,11 @@ JNIEXPORT jint JNICALL Java_sun_nio_ch_FileDispatcherImpl_setDirect0(JNIEnv *env, jclass this, - jobject buffer) + jobject fdObj, jobject buffer) { jint result = -1; - HANDLE orig = (HANDLE)(handleval(env, fObj)); + HANDLE orig = (HANDLE)(handleval(env, fdObj)); HANDLE modify = ReOpenFile(orig, 0, 0, FILE_FLAG_NO_BUFFERING | FILE_FLAG_WRITE_THROUGH); -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Fri Sep 1 22:17:07 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 1 Sep 2017 15:17:07 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD11F648A@ORSMSX113.amr.corp.intel.com> References: <0EE00E4F-EB7F-4C25-8955-E06AA4659BA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EAA14@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EF8EE@ORSMSX113.amr.corp.intel.com> <9F95A48E-EF98-4EB0-93F4-39CC24A02867@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EFC81@ORSMSX113.amr.corp.intel.com> <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! ! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <5B82CAC0-CBC5-4C48-8B36-83F8ECB19020@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F5F96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F648A@ORSMSX113.amr.corp.intel.com> Message-ID: Hi Lucy, On Sep 1, 2017, at 3:15 PM, Lu, Yingqi wrote: > When I add back the Windows flags, I forgot to check on the file descriptor. Your patch perfectly solved the issue. Thank you!! You?re welcome! > I also patched the jdk/make/test/JtregNative.gmk to make sure the native library is not going to be built on Windows. > > Interestingly, I tried all 4 tests on Windows and they all passed. > $ ./jtreg/bin/jtreg jdk/test/java/nio/channels/FileChannel/directio > Directory "JTwork" not found: creating > Directory "JTreport" not found: creating > Test results: passed: 4 > Report written to C:\cygwin64\home\ylu41\jdk10-directio-0831\JTreport\html\report.html > Results written to C:\cygwin64\home\ylu41\jdk10-directio-0831\JTwork > > The test environment is Windows 10 enterprise 64 bit. I did everything through Cygwin. Weird. I?ll try it again. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Fri Sep 1 22:53:16 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 1 Sep 2017 15:53:16 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <0EE00E4F-EB7F-4C25-8955-E06AA4659BA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EAA14@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EF8EE@ORSMSX113.amr.corp.intel.com> <9F95A48E-EF98-4EB0-93F4-39CC24A02867@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EFC81@ORSMSX113.amr.corp.intel.com> <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! ! ! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <5B82CAC0-CBC5-4C48-8B36-83F8ECB19020@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F5F96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F648A@ORSMSX113.amr.corp.intel.com> Message-ID: <208D0D05-E0B1-430D-A3BC-3BD9C57FBA55@oracle.com> Hi Lucy, All four tests pass on Windows for me, too. I think I had the wrong build of nio.dll in place. FWIW it is Windows 7 Pro 64. Thanks, Brian On Sep 1, 2017, at 3:17 PM, Brian Burkhalter wrote: >> >> Interestingly, I tried all 4 tests on Windows and they all passed. >> $ ./jtreg/bin/jtreg jdk/test/java/nio/channels/FileChannel/directio >> Directory "JTwork" not found: creating >> Directory "JTreport" not found: creating >> Test results: passed: 4 >> Report written to C:\cygwin64\home\ylu41\jdk10-directio-0831\JTreport\html\report.html >> Results written to C:\cygwin64\home\ylu41\jdk10-directio-0831\JTwork >> >> The test environment is Windows 10 enterprise 64 bit. I did everything through Cygwin. > > Weird. I?ll try it again. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Fri Sep 1 22:55:01 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Fri, 1 Sep 2017 22:55:01 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <208D0D05-E0B1-430D-A3BC-3BD9C57FBA55@oracle.com> References: <0EE00E4F-EB7F-4C25-8955-E06AA4659BA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EAA14@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EF8EE@ORSMSX113.amr.corp.intel.com> <9F95A48E-EF98-4EB0-93F4-39CC24A02867@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EFC81@ORSMSX113.amr.corp.intel.com> <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! ! ! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <5B82CAC0-CBC5-4C48-8B36-83F8ECB19020@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F5F96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F648A@ORSMSX113.amr.corp.intel.com> <208D0D05-E0B1-430D-A3BC-3BD9C57FBA55@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD11F6606@ORSMSX113.amr.corp.intel.com> Hi Brian, Glad to hear it and thank you for confirming on this! Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Friday, September 01, 2017 3:53 PM To: Lu, Yingqi Cc: nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, All four tests pass on Windows for me, too. I think I had the wrong build of nio.dll in place. FWIW it is Windows 7 Pro 64. Thanks, Brian On Sep 1, 2017, at 3:17 PM, Brian Burkhalter > wrote: Interestingly, I tried all 4 tests on Windows and they all passed. $ ./jtreg/bin/jtreg jdk/test/java/nio/channels/FileChannel/directio Directory "JTwork" not found: creating Directory "JTreport" not found: creating Test results: passed: 4 Report written to C:\cygwin64\home\ylu41\jdk10-directio-0831\JTreport\html\report.html Results written to C:\cygwin64\home\ylu41\jdk10-directio-0831\JTwork The test environment is Windows 10 enterprise 64 bit. I did everything through Cygwin. Weird. I'll try it again. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Fri Sep 1 23:11:03 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 1 Sep 2017 16:11:03 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <2F421C67-CA19-45C4-A6C7-FC9749B6D2CD@oracle.com> References: <0EE00E4F-EB7F-4C25-8955-E06AA4659BA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EAA14@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EF8EE@ORSMSX113.amr.corp.intel.com> <9F95A48E-EF98-4EB0-93F4-39CC24A02867@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EFC81@ORSMSX113.amr.corp.intel.com> <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! ! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <2F421C67-CA19-45C4-A6C7-FC9749B6D2CD@oracle.com> Message-ID: <7697CAEB-0367-41C6-B668-A8D38039C3C8@oracle.com> Hi Lucy, On Sep 1, 2017, at 7:46 AM, Brian Burkhalter wrote: >> Please take a look at this version and let us know your feedback. I would also like to learn what will be a proper approach to solve the AIX issue and what kind of details we need to provide for the exception messages. I will try to incorporate all of those changes into the next version. >> > I?ll try to get back to you on these points today. Given that the only API-level call which can create a direct FileChannel requires the path and opens the channel then perhaps Volker?s patch is OK as-is. The use of fcntl() and ReOpenFile() were intended to provide a more flexible approach better to accommodate potential future changes. As for the exception messages, for example in the case of a number of bytes not equalling the block size, it might not be a bad idea to include the actual values of the number of bytes and the block size. On Sep 1, 2017, at 9:13 AM, Lu, Yingqi wrote: > I will add the Solaris check for all the tests now and wait for your feedback on the additional tests. All these changes will go into webrev.08. As of now the tests look to be sufficient but I should take another look next week after the holiday weekend. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Sat Sep 2 16:05:39 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Sat, 2 Sep 2017 16:05:39 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <7697CAEB-0367-41C6-B668-A8D38039C3C8@oracle.com> References: <0EE00E4F-EB7F-4C25-8955-E06AA4659BA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EAA14@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EF8EE@ORSMSX113.amr.corp.intel.com> <9F95A48E-EF98-4EB0-93F4-39CC24A02867@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EFC81@ORSMSX113.amr.corp.intel.com> <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! ! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <2F421C67-CA19-45C4-A6C7-FC9749B6D2CD@oracle.com> <7697CAEB-0367-41C6-B668-A8D38039C3C8@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD11F6EEA@ORSMSX113.amr.corp.intel.com> Hi Brian/Volker, I have already addressed all the existing issues into webrev.08. However, I found after applying Volker's patch, JDK does not compile on Linux. Error message is listed below: error: cannot find symbol oflags |= O_DIRECT; ^ symbol: variable O_DIRECT location: class UnixChannelFactory In addition, I have two more questions: 1. UnixChannelFactory.java is a common class for all the UNIX based OS, MacOS and Solaris do not have O_DIRECT flag defined. If we specifically use O_DIRECT, will it be an issue? 2. Does this approach has conflict with the current implementation with fcntl and ReOpenFile() or it is OK to use it on top of the current approach? Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Friday, September 01, 2017 4:11 PM To: Lu, Yingqi Cc: nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, On Sep 1, 2017, at 7:46 AM, Brian Burkhalter > wrote: Please take a look at this version and let us know your feedback. I would also like to learn what will be a proper approach to solve the AIX issue and what kind of details we need to provide for the exception messages. I will try to incorporate all of those changes into the next version. I'll try to get back to you on these points today. Given that the only API-level call which can create a direct FileChannel requires the path and opens the channel then perhaps Volker's patch is OK as-is. The use of fcntl() and ReOpenFile() were intended to provide a more flexible approach better to accommodate potential future changes. As for the exception messages, for example in the case of a number of bytes not equalling the block size, it might not be a bad idea to include the actual values of the number of bytes and the block size. On Sep 1, 2017, at 9:13 AM, Lu, Yingqi > wrote: I will add the Solaris check for all the tests now and wait for your feedback on the additional tests. All these changes will go into webrev.08. As of now the tests look to be sufficient but I should take another look next week after the holiday weekend. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From huaming.li at oracle.com Mon Sep 4 07:16:52 2017 From: huaming.li at oracle.com (huaming.li at oracle.com) Date: Mon, 4 Sep 2017 15:16:52 +0800 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD11EFC81@ORSMSX113.amr.corp.intel.com> <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> Message-ID: Hi Lucy, Thank you for the great work. I tried to review based on http://cr.openjdk.java.net/~kkharbas/8164900/webrev.08/, but can not view file change diffs there, so following comments are based on webrev.07, hope it's helpful. ====================== Comments about implementation ====================== http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/src/java.base/share/classes/sun/nio/ch/Util.java.frames.html In line 250, should it use a different cache than non-direct buffer? (Can different files use direct & non-direct io at the same time?) In line 254, what?s the possibility of to get here, if it?s very low, maybe 1) another cache is necessary? 2) or no cache at all 243 public static ByteBuffer getTemporaryAlignedDirectBuffer(int size, 244 int alignment) { 245 if (isBufferTooLarge(size)) { 246 return ByteBuffer.allocateDirect(size + alignment - 1) 247 .alignedSlice(alignment); 248 } 249 250 BufferCache cache = bufferCache.get(); 251 ByteBuffer buf = cache.get(size); 252 if (buf != null) { 253 if (buf.alignmentOffset(0, alignment) == 0) { 254 return buf; 255 } 256 } else { 257 if (!cache.isEmpty()) { 258 buf = cache.removeFirst(); 259 free(buf); 260 } 261 } http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/src/java.base/windows/classes/sun/nio/fs/WindowsFileStore.java.frames.html 137 private DiskFreeSpace readDiskFreeSpace() throws IOException { 138 try { 139 return GetDiskFreeSpace(root); // should this or below one be cached? 140 } catch (WindowsException x) { 141 x.rethrowAsIOException(root); 142 return null; 143 } 144 } 156 public long getBlockSize() throws IOException { 157 return readDiskFreeSpace().bytesPerSector(); // should this or above one be cached? 158 } http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java.frames.html Alignment is checked for both remained length and position in lots places (for example below 2 blocks of code), could they be extracted and made 2 functions calls? 179 if (dst.remaining() % alignment != 0) { 180 throw new IOException("Number of remaining bytes is not " 181 + "a multiple of the block size"); 182 } 183 if (position() % alignment != 0) { 184 throw new IOException("Channel position is not a multiple " 185 + "of the block size"); 186 } http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/src/java.base/share/classes/sun/nio/ch/IOUtil.java.frames.html Alignment is checked for both remained length and position in lots places, could they be extracted and made 2 functions calls? Are the checks in IOUtil.java and FileChannelImpl.java duplicated with each other, Should they be all moved and merged into IOUtil.java? In java.nio.ByteBuffer Should a new method be introduced in ByteBuffer to combine operations like ByteBuffer.allocateDirect(SIZE + alignment - 1).alignedSlice(alignment); Besides of above comments, I have some questions about this new direct io feature: What?s the expected FileChannel.read behavior when remaining bytes in file is less than block size? For example, read from a file which is only 10 bytes long through?ExtendedOpenOption.DIRECT. What?s the expected FileChannel.write behavior when remaining valid bytes in buffer is less than block size? For example, write just 10 bytes to a file through?ExtendedOpenOption.DIRECT. What?s the expected FileChannel.position behavior after FileChannel.read/write? What?s the expected FileChannel.read behavior when remaining bytes in file is more than block size? For example, read 4096 bytes from a file which is 8192 long, is it guaranteed to read out 4096 bytes at once? Similar question for write. Is it necessary to add test case for above scenarios? And, I think in some place, it should document the usage of ExtendedOpenOption.DIRECT and cautions for example constraints on file position, buffer position/remaning... ====================== Comments about tests ====================== http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/test/java/nio/channels/FileChannel/directio/DirectIOTest.java.html Should add ?return? after print out message? else it will not skip the test 97 if (!fsType.equals("nfs") && !fsType.equals("ufs")) { 98 // print a message and return without failing 99 System.out.format("Skipping test: file system type %s of " 100 + "FileStore of %s is neither nfs nor ufs.%n", fsType, p); 101 } http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/test/java/nio/channels/FileChannel/directio/libDirectIO.c.html Inhttp://man7.org/linux/man-pages/man2/mincore.2.html, it says: The/vec/ argument must point to an array containing at least /(length+PAGE_SIZE-1) / PAGE_SIZE/ bytes. On return, the least significant bit of each byte will be set if the corresponding page is currently resident in memory, and be clear otherwise. (The settings of the other bits in each byte are undefined; these bits are reserved for possible later use.) Of course the information returned in/vec/ is only a snapshot: pages that are not locked in memory can come and go at any moment, and the contents of/vec/ may already be stale by the time this call returns. So, is it possible that DirectIOTest.java may report false positive failure intermittently when Java_DirectIOTest_isFileInCache0 return true? http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/test/java/nio/channels/FileChannel/directio/WriteDirect.java.html Do you need to add negative test cases for?below exception in IOUtil.java? Similar suggestion for ReadDirect.java 105 if (bb.alignmentOffset(pos, alignment) != 0) { 106 throw new IOException("Current location of the bytebuffer " 107 + "is not a multiple of the block size"); 108 } Should .alignedSlice(alignment); be appended at line 93? 92 try { 93 ByteBuffer bb = ByteBuffer.allocateDirect(8192); 94 fc.read(bb); http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/test/java/nio/channels/FileChannel/directio/PreadDirect.java.html The direct read/write in PreadDirect/PwriteDirect.java are put in a while loop, is it necessary? If yes, then is it necessary for similar read/write in ReadDirect/WriteDirect.java be put in a while loop too? 130 while (totalRead < 4096) { 131 int read = c.read(block, offset); 132 if (read < 0) 133 throw new Exception("Read failed"); 134 totalRead += read; 135 } And I don?t quite understand the logic of above code. General comments for tests: 1. It would be better to replace magic number such as 4096 with meaningfully named variables 2. It would be better to rename test methods with self-explained names, for example, testNegativePosition is a positive case, genericTest2 is a negative case Thank you -Hamlin On 01/09/2017 6:49 AM, Lu, Yingqi wrote: > > Hi Brian, > > Here is the webrev.07 version of the patch: > http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/ > > > In this version, I have addressed the following issues: > > 1.Following the example of stringPlatformChars, made the DirectIOTest > native library being compiled automatically. Removed > DirectIOTest.java, README file and Makefile. > > 2.Added the filesystem type check on Solaris for DirectIOTest. > > 3.Checked if there is an equivalent system call like mincore in > Windows to enable DirectIOTest for Windows. I could not find anything > unfortunately. > > 4.Added the DirectIO flag FILE_FLAG_NO_BUFFERING and > FILE_FLAG_WRITE_THROUGH back to Windows version of setDirect0() function. > > 5.Addressed the comments from you on DirectIO.c and DirectIO.java (now > being renamed as DirectIOTest.java) > > 6.Added the tests for the read() and write() methods which take as > parameter an array of buffers ByteBuffer[]. > > 7.Changed the indentation for continuous lines. > > Issues that are still open: > > 1.How should we address the issue of AIX OS does not support setting > O_DIRECT after the open() > > 2.Provide more detailed messages for IOExceptions with DirectIO > > 3.Are there any more tests we need to add? > > Please take a look at this version and let us know your feedback. I > would also like to learn what will be a proper approach to solve the > AIX issue and what kind of details we need to provide for the > exception messages. I will try to incorporate all of those changes > into the next version. > > Thanks, > > Lucy > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Mon Sep 4 11:39:14 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 4 Sep 2017 12:39:14 +0100 Subject: JDK 10 RFR of 8147615: (fc) FileChannelImpl has no finalizer In-Reply-To: References: Message-ID: On 01/09/2017 15:56, Brian Burkhalter wrote: > Please review at your convenience: > > https://bugs.openjdk.java.net/browse/JDK-8147615 > http://cr.openjdk.java.net/~bpb/8147615/webrev.00/ > > There is a question at line 106 of FileChannelImpl.java which needs to be resolved. > > Another question is whether a synchronized block should be used in or around invalidateAndClose() in addition to relying on the pseudo-synchronization of checking the FileDescriptor validity. It is necessary to invalidate the FileDescriptor in any case as otherwise ?Bad file descriptor? exceptions occur. This is likely due to the cleaning action attempting to close a native file descriptor which was already closed when the channel was closed. > > That this actually works was verified by instrumenting FileChannelImpl$Closer#run() with a print statement which was since removed. The issue has however been labelled ?noreg-hard? as there appears to be no way to verify the fix without such instrumentation or via existing public APIs. > Yes, this is hard to test as you will likely run out of file descriptors before the cleaner runs. One thing you could do is use the management API and check if the OperatingSystemMXBean is a UnixOperatingSystemMXBean as that will give you an easy way to get the maximum and open file descriptors. As regards the patch then I assume you can skip creating the Cleaner when there is a FileInputStream or FileOutputStream parent (as they have finalizers to close the file, also a reference to the channel). If you do that then it reduces the scenarios to think about to just the FileChannels that are created directly and then unreferenced without calling close. As regards setting the fd to -1 then this is needed to avoid the cleaner closing a file descriptor that has been recycled to another file or socket or something else. You can synchronize that code. The "what is close fails" is difficult. In other areas then failure will terminate the VM but I don't think this make sense here close might fail with EIO. Minimally EINTR will need special handling as you won't want an exception or logging for that case. The simplest might be to just for ignore and pick up again if/when logging in core libraries is examined. -Alan From volker.simonis at gmail.com Mon Sep 4 17:42:03 2017 From: volker.simonis at gmail.com (Volker Simonis) Date: Mon, 4 Sep 2017 19:42:03 +0200 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD11F6EEA@ORSMSX113.amr.corp.intel.com> References: <0EE00E4F-EB7F-4C25-8955-E06AA4659BA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EAA14@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EF8EE@ORSMSX113.amr.corp.intel.com> <9F95A48E-EF98-4EB0-93F4-39CC24A02867@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EFC81@ORSMSX113.amr.corp.intel.com> <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <2F421C67-CA19-45C4-A6C7-FC9749B6D2CD@oracle.com> <7697CAEB-0367-41C6-B668-A8D38039C3C8@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F6EEA@ORSMSX113.amr.corp.intel.com> Message-ID: Hi Lucy, On Sat, Sep 2, 2017 at 6:05 PM, Lu, Yingqi wrote: > Hi Brian/Volker, > > > > I have already addressed all the existing issues into webrev.08. However, I > found after applying Volker?s patch, JDK does not compile on Linux. Error > message is listed below: > > > > error: cannot find symbol > > oflags |= O_DIRECT; > > ^ > > symbol: variable O_DIRECT > > location: class UnixChannelFactory > Sorry for the trouble :( I've just run into this myself now. I've initially tested my proposal on AIX only (because it was just a proposal). Now that it seems to get accepted, I'll have to look at it more generally :) The problem on Linux is caused by the fact that the build doesn't use "-D_GNU_SOURCE" when processing UnixConstants.java.template and therefore "O_DIRECT" isn't visible. In contrast, the compilation of "FileDispatcherImpl.c" always uses "-D_GNU_SOURCE", so you can freely access "O_DIRECT". A quick fix is the following: diff -r 0c49baac8805 make/gensrc/GensrcMisc.gmk --- a/make/gensrc/GensrcMisc.gmk Wed Aug 30 18:54:50 2017 +0200 +++ b/make/gensrc/GensrcMisc.gmk Mon Sep 04 19:37:35 2017 +0200 @@ -63,7 +63,7 @@ define generate-preproc-src $(call MakeDir, $(@D)) ( $(NAWK) '/@@END_COPYRIGHT@@/{exit}1' $< && \ - $(CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $< \ + $(CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $(CFLAGS_JDKLIB) $< \ 2> >($(GREP) -v '^$(&2) \ | $(NAWK) '/@@START_HERE@@/,0' \ | $(SED) -e 's/@@START_HERE@@/\/\/ AUTOMATICALLY GENERATED FILE - DO NOT EDIT/' \ which worked for me on Linux/Solaris/AIX (haven't tested on Mac). I think it was always a mistake that we haven't generated the UnixConstants with the same C-Flags that we've also used to compile the files which used them. It was probably just luck that we've not run into problems until now. I'll try your newest version tomorrow and also check the tests on AIX. > > In addition, I have two more questions: > > 1. UnixChannelFactory.java is a common class for all the UNIX based OS, > MacOS and Solaris do not have O_DIRECT flag defined. If we specifically use > O_DIRECT, will it be an issue? > I think it depends. If we only use it on top of the current fcntl() approach it will be probably fine (but we'll have to define O_DIRECT to '0' on platforms which don't know it in that case). Otherwise we'll have to use the corresponding platform flags in UnixChannelFactory.java > 2. Does this approach has conflict with the current implementation with > fcntl and ReOpenFile() or it is OK to use it on top of the current approach? > I'll leave this question up to Brian :) > > Thanks, > > Lucy > > > > From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] > Sent: Friday, September 01, 2017 4:11 PM > To: Lu, Yingqi > Cc: nio-dev at openjdk.java.net > Subject: Re: Proposal for adding O_DIRECT support into JDK 9 > > > > Hi Lucy, > > > > On Sep 1, 2017, at 7:46 AM, Brian Burkhalter > wrote: > > > > Please take a look at this version and let us know your feedback. I would > also like to learn what will be a proper approach to solve the AIX issue and > what kind of details we need to provide for the exception messages. I will > try to incorporate all of those changes into the next version. > > I?ll try to get back to you on these points today. > > > > Given that the only API-level call which can create a direct FileChannel > requires the path and opens the channel then perhaps Volker?s patch is OK > as-is. The use of fcntl() and ReOpenFile() were intended to provide a more > flexible approach better to accommodate potential future changes. > > > > As for the exception messages, for example in the case of a number of bytes > not equalling the block size, it might not be a bad idea to include the > actual values of the number of bytes and the block size. > > > > On Sep 1, 2017, at 9:13 AM, Lu, Yingqi wrote: > > > > I will add the Solaris check for all the tests now and wait for your > feedback on the additional tests. All these changes will go into webrev.08. > > > > As of now the tests look to be sufficient but I should take another look > next week after the holiday weekend. > > > > Thanks, > > > > Brian From yingqi.lu at intel.com Mon Sep 4 17:50:10 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Mon, 4 Sep 2017 17:50:10 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AD11EFC81@ORSMSX113.amr.corp.intel.com> <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> Hi Hamlin, Thank you very much for reviewing the patch. It is very helpful! webrev.08 has not been published yet as I am waiting on the feedback to resolve to the AIX issue. I sent a separate email couple days ago with the details. Please see my answers below: http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/src/java.base/share/classes/sun/nio/ch/Util.java.frames.html In line 250, should it use a different cache than non-direct buffer? (Can different files use direct & non-direct io at the same time?) In line 254, what's the possibility of to get here, if it's very low, maybe 1) another cache is necessary? 2) or no cache at all Yes, direct and non-direct IO can be used at the same time. I am not familiar on how often the code is accessed. I will let Alan/Brian and others comment on the proper approach http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java.frames.html Alignment is checked for both remained length and position in lots places (for example below 2 blocks of code), could they be extracted and made 2 functions calls? Good point! As you suggested, I will create 2 static functions inside IOUtil.java, so that both FileChannelImpl and IOUtil class can use them. In java.nio.ByteBuffer Should a new method be introduced in ByteBuffer to combine operations like ByteBuffer.allocateDirect(SIZE + alignment - 1).alignedSlice(alignment); I only see ByteBuffer.allocateDirect() being called twice. I am not totally convinced whether we need to introduce a new API. However, I am surely open to this suggestion if people here agree :) http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/test/java/nio/channels/FileChannel/directio/DirectIOTest.java.html Should add "return" after print out message? else it will not skip the test This is already solved in my local webrev.08. http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/test/java/nio/channels/FileChannel/directio/libDirectIO.c.html In http://man7.org/linux/man-pages/man2/mincore.2.html, it says: The vec argument must point to an array containing at least (length+PAGE_SIZE-1) / PAGE_SIZE bytes. On return, the least significant bit of each byte will be set if the corresponding page is currently resident in memory, and be clear otherwise. (The settings of the other bits in each byte are undefined; these bits are reserved for possible later use.) Of course the information returned in vec is only a snapshot: pages that are not locked in memory can come and go at any moment, and the contents of vec may already be stale by the time this call returns. So, is it possible that DirectIOTest.java may report false positive failure intermittently when Java_DirectIOTest_isFileInCache0 return true? Yes, pages are not locked in memory and they can move freely. However, I think this is the nature of the test. Given the temp file size is very small compare to the memory size and the page check happens right after the IO operation, I think the test should be good in most of the cases. http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/test/java/nio/channels/FileChannel/directio/WriteDirect.java.html Do you need to add negative test cases for below exception in IOUtil.java? Similar suggestion for ReadDirect.java 105 if (bb.alignmentOffset(pos, alignment) != 0) { 106 throw new IOException("Current location of the bytebuffer " 107 + "is not a multiple of the block size"); 108 } Should .alignedSlice(alignment); be appended at line 93? 92 try { 93 ByteBuffer bb = ByteBuffer.allocateDirect(8192); 94 fc.read(bb); The channel is opened as a non-Direct channel. It is just to read the content and confirm the write operation is performed correctly. I think there is no need to append .alignedSlice() I think the negative test for non-aligned position are added in PreadDirect.java and PwriteDirect.java. Please let me know if I missed anything though. http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/test/java/nio/channels/FileChannel/directio/PreadDirect.java.html The direct read/write in PreadDirect/PwriteDirect.java are put in a while loop, is it necessary? If yes, then is it necessary for similar read/write in ReadDirect/WriteDirect.java be put in a while loop too? 130 while (totalRead < 4096) { 131 int read = c.read(block, offset); 132 if (read < 0) 133 throw new Exception("Read failed"); 134 totalRead += read; 135 } And I don't quite understand the logic of above code. I followed the example of Read.java and Write.java on the while loop. I can modify it accordingly. What's the expected FileChannel.read behavior when remaining bytes in file is less than block size? For example, read from a file which is only 10 bytes long through ExtendedOpenOption.DIRECT. It will read 10 bytes and return. What's the expected FileChannel.write behavior when remaining valid bytes in buffer is less than block size? For example, write just 10 bytes to a file through ExtendedOpenOption.DIRECT. It will throw exception. The remaining buffer size needs to a multiple of the alignment value. What's the expected FileChannel.position behavior after FileChannel.read/write? Similarly to the non-DirectIO case, it will move to the current position. What's the expected FileChannel.read behavior when remaining bytes in file is more than block size? For example, read 4096 bytes from a file which is 8192 long, is it guaranteed to read out 4096 bytes at once? Similar question for write. It depends on the current position of the file. If the position is at the beginning of the file, yes, it will read 4096 bytes. If the position is > 0, but not aligned, then it will throw error. If the position is at 4096, it will read 4096 bytes. If the position is at the end of the file, it will read 0. Everything is similar to non-Direct channel, except the current position needs to be aligned. Is it necessary to add test case for above scenarios? I can add tests for case 1. I think other cases are covered with the existing tests. Please let me know if I missed anything though. Thanks, Lucy From: huaming.li at oracle.com [mailto:huaming.li at oracle.com] Sent: Monday, September 04, 2017 12:17 AM To: nio-dev at openjdk.java.net; Lu, Yingqi Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, Thank you for the great work. I tried to review based on http://cr.openjdk.java.net/~kkharbas/8164900/webrev.08/, but can not view file change diffs there, so following comments are based on webrev.07, hope it's helpful. ====================== Comments about implementation ====================== http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/src/java.base/share/classes/sun/nio/ch/Util.java.frames.html In line 250, should it use a different cache than non-direct buffer? (Can different files use direct & non-direct io at the same time?) In line 254, what's the possibility of to get here, if it's very low, maybe 1) another cache is necessary? 2) or no cache at all 243 public static ByteBuffer getTemporaryAlignedDirectBuffer(int size, 244 int alignment) { 245 if (isBufferTooLarge(size)) { 246 return ByteBuffer.allocateDirect(size + alignment - 1) 247 .alignedSlice(alignment); 248 } 249 250 BufferCache cache = bufferCache.get(); 251 ByteBuffer buf = cache.get(size); 252 if (buf != null) { 253 if (buf.alignmentOffset(0, alignment) == 0) { 254 return buf; 255 } 256 } else { 257 if (!cache.isEmpty()) { 258 buf = cache.removeFirst(); 259 free(buf); 260 } 261 } http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/src/java.base/windows/classes/sun/nio/fs/WindowsFileStore.java.frames.html 137 private DiskFreeSpace readDiskFreeSpace() throws IOException { 138 try { 139 return GetDiskFreeSpace(root); // should this or below one be cached? 140 } catch (WindowsException x) { 141 x.rethrowAsIOException(root); 142 return null; 143 } 144 } 156 public long getBlockSize() throws IOException { 157 return readDiskFreeSpace().bytesPerSector(); // should this or above one be cached? 158 } http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java.frames.html Alignment is checked for both remained length and position in lots places (for example below 2 blocks of code), could they be extracted and made 2 functions calls? 179 if (dst.remaining() % alignment != 0) { 180 throw new IOException("Number of remaining bytes is not " 181 + "a multiple of the block size"); 182 } 183 if (position() % alignment != 0) { 184 throw new IOException("Channel position is not a multiple " 185 + "of the block size"); 186 } http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/src/java.base/share/classes/sun/nio/ch/IOUtil.java.frames.html Alignment is checked for both remained length and position in lots places, could they be extracted and made 2 functions calls? Are the checks in IOUtil.java and FileChannelImpl.java duplicated with each other, Should they be all moved and merged into IOUtil.java? In java.nio.ByteBuffer Should a new method be introduced in ByteBuffer to combine operations like ByteBuffer.allocateDirect(SIZE + alignment - 1).alignedSlice(alignment); Besides of above comments, I have some questions about this new direct io feature: What's the expected FileChannel.read behavior when remaining bytes in file is less than block size? For example, read from a file which is only 10 bytes long through ExtendedOpenOption.DIRECT. What's the expected FileChannel.write behavior when remaining valid bytes in buffer is less than block size? For example, write just 10 bytes to a file through ExtendedOpenOption.DIRECT. What's the expected FileChannel.position behavior after FileChannel.read/write? What's the expected FileChannel.read behavior when remaining bytes in file is more than block size? For example, read 4096 bytes from a file which is 8192 long, is it guaranteed to read out 4096 bytes at once? Similar question for write. Is it necessary to add test case for above scenarios? And, I think in some place, it should document the usage of ExtendedOpenOption.DIRECT and cautions for example constraints on file position, buffer position/remaning... ====================== Comments about tests ====================== http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/test/java/nio/channels/FileChannel/directio/DirectIOTest.java.html Should add "return" after print out message? else it will not skip the test 97 if (!fsType.equals("nfs") && !fsType.equals("ufs")) { 98 // print a message and return without failing 99 System.out.format("Skipping test: file system type %s of " 100 + "FileStore of %s is neither nfs nor ufs.%n", fsType, p); 101 } http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/test/java/nio/channels/FileChannel/directio/libDirectIO.c.html In http://man7.org/linux/man-pages/man2/mincore.2.html, it says: The vec argument must point to an array containing at least (length+PAGE_SIZE-1) / PAGE_SIZE bytes. On return, the least significant bit of each byte will be set if the corresponding page is currently resident in memory, and be clear otherwise. (The settings of the other bits in each byte are undefined; these bits are reserved for possible later use.) Of course the information returned in vec is only a snapshot: pages that are not locked in memory can come and go at any moment, and the contents of vec may already be stale by the time this call returns. So, is it possible that DirectIOTest.java may report false positive failure intermittently when Java_DirectIOTest_isFileInCache0 return true? http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/test/java/nio/channels/FileChannel/directio/WriteDirect.java.html Do you need to add negative test cases for below exception in IOUtil.java? Similar suggestion for ReadDirect.java 105 if (bb.alignmentOffset(pos, alignment) != 0) { 106 throw new IOException("Current location of the bytebuffer " 107 + "is not a multiple of the block size"); 108 } Should .alignedSlice(alignment); be appended at line 93? 92 try { 93 ByteBuffer bb = ByteBuffer.allocateDirect(8192); 94 fc.read(bb); http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/test/java/nio/channels/FileChannel/directio/PreadDirect.java.html The direct read/write in PreadDirect/PwriteDirect.java are put in a while loop, is it necessary? If yes, then is it necessary for similar read/write in ReadDirect/WriteDirect.java be put in a while loop too? 130 while (totalRead < 4096) { 131 int read = c.read(block, offset); 132 if (read < 0) 133 throw new Exception("Read failed"); 134 totalRead += read; 135 } And I don't quite understand the logic of above code. General comments for tests: 1. It would be better to replace magic number such as 4096 with meaningfully named variables 2. It would be better to rename test methods with self-explained names, for example, testNegativePosition is a positive case, genericTest2 is a negative case Thank you -Hamlin On 01/09/2017 6:49 AM, Lu, Yingqi wrote: Hi Brian, Here is the webrev.07 version of the patch: http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/ In this version, I have addressed the following issues: 1. Following the example of stringPlatformChars, made the DirectIOTest native library being compiled automatically. Removed DirectIOTest.java, README file and Makefile. 2. Added the filesystem type check on Solaris for DirectIOTest. 3. Checked if there is an equivalent system call like mincore in Windows to enable DirectIOTest for Windows. I could not find anything unfortunately. 4. Added the DirectIO flag FILE_FLAG_NO_BUFFERING and FILE_FLAG_WRITE_THROUGH back to Windows version of setDirect0() function. 5. Addressed the comments from you on DirectIO.c and DirectIO.java (now being renamed as DirectIOTest.java) 6. Added the tests for the read() and write() methods which take as parameter an array of buffers ByteBuffer[]. 7. Changed the indentation for continuous lines. Issues that are still open: 1. How should we address the issue of AIX OS does not support setting O_DIRECT after the open() 2. Provide more detailed messages for IOExceptions with DirectIO 3. Are there any more tests we need to add? Please take a look at this version and let us know your feedback. I would also like to learn what will be a proper approach to solve the AIX issue and what kind of details we need to provide for the exception messages. I will try to incorporate all of those changes into the next version. Thanks, Lucy -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Mon Sep 4 17:55:10 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Mon, 4 Sep 2017 17:55:10 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <0EE00E4F-EB7F-4C25-8955-E06AA4659BA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EAA14@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EF8EE@ORSMSX113.amr.corp.intel.com> <9F95A48E-EF98-4EB0-93F4-39CC24A02867@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11EFC81@ORSMSX113.amr.corp.intel.com> <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <2F421C67-CA19-45C4-A6C7-FC9749B6D2CD@oracle.com> <7697CAEB-0367-41C6-B668-A8D38039C3C8@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F6EEA@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C56@ORSMSX113.amr.corp.intel.com> Hi Volker, Thank you for the quick reply. I will try your fix today and I am sure it will work! Please hold on to the testing since webrev.08 will be out hopefully tomorrow. In the new version, we fixed a handful issues Brain, you and Hamlin pointed out. You might want to directly test the coming new version :) Thanks to everyone for all your great feedback. Really appreciate!! Thanks! Lucy >-----Original Message----- >From: Volker Simonis [mailto:volker.simonis at gmail.com] >Sent: Monday, September 04, 2017 10:42 AM >To: Lu, Yingqi >Cc: Brian Burkhalter ; nio-dev at openjdk.java.net >Subject: Re: Proposal for adding O_DIRECT support into JDK 9 > >Hi Lucy, > >On Sat, Sep 2, 2017 at 6:05 PM, Lu, Yingqi wrote: >> Hi Brian/Volker, >> >> >> >> I have already addressed all the existing issues into webrev.08. >> However, I found after applying Volker?s patch, JDK does not compile >> on Linux. Error message is listed below: >> >> >> >> error: cannot find symbol >> >> oflags |= O_DIRECT; >> >> ^ >> >> symbol: variable O_DIRECT >> >> location: class UnixChannelFactory >> > >Sorry for the trouble :( >I've just run into this myself now. I've initially tested my proposal on AIX only >(because it was just a proposal). Now that it seems to get accepted, I'll have to >look at it more generally :) > >The problem on Linux is caused by the fact that the build doesn't use "- >D_GNU_SOURCE" when processing UnixConstants.java.template and therefore >"O_DIRECT" isn't visible. In contrast, the compilation of "FileDispatcherImpl.c" >always uses "-D_GNU_SOURCE", so you can freely access "O_DIRECT". > >A quick fix is the following: > >diff -r 0c49baac8805 make/gensrc/GensrcMisc.gmk >--- a/make/gensrc/GensrcMisc.gmk Wed Aug 30 18:54:50 2017 +0200 >+++ b/make/gensrc/GensrcMisc.gmk Mon Sep 04 19:37:35 2017 +0200 >@@ -63,7 +63,7 @@ > define generate-preproc-src > $(call MakeDir, $(@D)) > ( $(NAWK) '/@@END_COPYRIGHT@@/{exit}1' $< && \ >- $(CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $< \ >+ $(CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $(CFLAGS_JDKLIB) $< \ > 2> >($(GREP) -v '^$(&2) \ > | $(NAWK) '/@@START_HERE@@/,0' \ > | $(SED) -e 's/@@START_HERE@@/\/\/ AUTOMATICALLY GENERATED >FILE - DO NOT EDIT/' \ > >which worked for me on Linux/Solaris/AIX (haven't tested on Mac). > >I think it was always a mistake that we haven't generated the UnixConstants with >the same C-Flags that we've also used to compile the files which used them. It >was probably just luck that we've not run into problems until now. > >I'll try your newest version tomorrow and also check the tests on AIX. > >> >> In addition, I have two more questions: >> >> 1. UnixChannelFactory.java is a common class for all the UNIX based OS, >> MacOS and Solaris do not have O_DIRECT flag defined. If we >> specifically use O_DIRECT, will it be an issue? >> > >I think it depends. If we only use it on top of the current fcntl() approach it will be >probably fine (but we'll have to define O_DIRECT to '0' on platforms which don't >know it in that case). Otherwise we'll have to use the corresponding platform >flags in UnixChannelFactory.java > >> 2. Does this approach has conflict with the current implementation with >> fcntl and ReOpenFile() or it is OK to use it on top of the current approach? >> > >I'll leave this question up to Brian :) > >> >> Thanks, >> >> Lucy >> >> >> >> From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] >> Sent: Friday, September 01, 2017 4:11 PM >> To: Lu, Yingqi >> Cc: nio-dev at openjdk.java.net >> Subject: Re: Proposal for adding O_DIRECT support into JDK 9 >> >> >> >> Hi Lucy, >> >> >> >> On Sep 1, 2017, at 7:46 AM, Brian Burkhalter >> >> wrote: >> >> >> >> Please take a look at this version and let us know your feedback. I >> would also like to learn what will be a proper approach to solve the >> AIX issue and what kind of details we need to provide for the >> exception messages. I will try to incorporate all of those changes into the next >version. >> >> I?ll try to get back to you on these points today. >> >> >> >> Given that the only API-level call which can create a direct >> FileChannel requires the path and opens the channel then perhaps >> Volker?s patch is OK as-is. The use of fcntl() and ReOpenFile() were >> intended to provide a more flexible approach better to accommodate potential >future changes. >> >> >> >> As for the exception messages, for example in the case of a number of >> bytes not equalling the block size, it might not be a bad idea to >> include the actual values of the number of bytes and the block size. >> >> >> >> On Sep 1, 2017, at 9:13 AM, Lu, Yingqi wrote: >> >> >> >> I will add the Solaris check for all the tests now and wait for your >> feedback on the additional tests. All these changes will go into webrev.08. >> >> >> >> As of now the tests look to be sufficient but I should take another >> look next week after the holiday weekend. >> >> >> >> Thanks, >> >> >> >> Brian From huaming.li at oracle.com Tue Sep 5 03:44:20 2017 From: huaming.li at oracle.com (huaming.li at oracle.com) Date: Tue, 5 Sep 2017 11:44:20 +0800 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> References: <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> Message-ID: Hi Lucy, Thank you for response, please check my comments inline. On 05/09/2017 1:50 AM, Lu, Yingqi wrote: > > Hi Hamlin, > > Thank you very much for reviewing the patch. It is very helpful! > :-) > > webrev.08 has not been published yet as I am waiting on the feedback > to resolve to the AIX issue. I sent a separate email couple days ago > with the details. > > Please see my answers below: > > http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/src/java.base/share/classes/sun/nio/ch/Util.java.frames.html > > In line 250, should it use a different cache than non-direct buffer? (Can different files use direct & non-direct io at the same time?) > In line 254, what?s the possibility of to get here, if it?s very low, maybe 1) another cache is necessary? 2) or no cache at all > > Yes, direct and non-direct IO can be used at the same time. I am not > familiar on how often the code is accessed. I will let Alan/Brian and > others comment on the proper approach > OK. > http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java.frames.html > > Alignment is checked for both remained length and position in lots places (for example below 2 blocks of code), could they be extracted and made 2 functions calls? > > Good point! As you suggested, I will create 2 static functions inside > IOUtil.java, so that both FileChannelImpl and IOUtil class can use them. > OK. > In java.nio.ByteBuffer > Should a new method be introduced in ByteBuffer to combine operations like ByteBuffer.allocateDirect(SIZE + alignment - 1).alignedSlice(alignment); > > I only see ByteBuffer.allocateDirect() being called twice. I am not > totally convinced whether we need to introduce a new API. However, I > am surely open to this suggestion if people here agree J > Maybe there is misunderstanding here, I don't mean we should add such method in test, I means to add a new API in java.nio.ByteBuffer class, the reason is that "ByteBuffer.allocateDirect(SIZE + alignment - 1).alignedSlice(alignment);" is necessary for a direct read/write, a new API for example ByteBuffer.allocateAlignedDirect(size, alignment) or allocateAlignedDirect(size) might be helpful for direct io end user. > > http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/test/java/nio/channels/FileChannel/directio/DirectIOTest.java.html > > Should add ?return? after print out message? else it will not skip the test > > This is already solved in my local webrev.08. > OK. > > http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/test/java/nio/channels/FileChannel/directio/libDirectIO.c.html > > In http://man7.org/linux/man-pages/man2/mincore.2.html, it says: > ?????? The/vec/ argument must point to an array containing at least > /(length+PAGE_SIZE-1) / PAGE_SIZE/ bytes.? On return, the least > ?????? significant bit of each byte will be set if the corresponding page is > ?????? currently resident in memory, and be clear otherwise.? (The settings > ?????? of the other bits in each byte are undefined; these bits are reserved > ?????? for possible later use.)? Of course the information returned in/vec/ > ?????? is only a snapshot: pages that are not locked in memory can come and > ?????? go at any moment, and the contents of/vec/ may already be stale by the > ?????? time this call returns. > So, is it possible that DirectIOTest.java may report false positive > failure intermittently when Java_DirectIOTest_isFileInCache0 return true? > > Yes, pages are not locked in memory and they can move freely. However, > I think this is the nature of the test. Given the temp file size is > very small compare to the memory size and the page check happens right > after the IO operation, I think the test should be good in most of the > cases. > Maybe we need to add some comments in the test, I'm not sure if @intermittent is needed to be added into test. So when test fails in the future, it can be analyzed quickly. > > http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/test/java/nio/channels/FileChannel/directio/WriteDirect.java.html > > Do you need to add negative test cases for?below exception in > IOUtil.java? Similar suggestion for ReadDirect.java > *105???????????? if (bb.alignmentOffset(pos, alignment) != 0) {* > *106???????????????? throw new IOException("Current location of the > bytebuffer "* > *107???????????????????? + "is not a multiple of the block size");* > *108???????????? }* > Should .alignedSlice(alignment); be appended at line 93? > ? 92???????? try { > ? 93???????????? ByteBuffer bb = ByteBuffer.allocateDirect(8192); > ? 94???????????? fc.read(bb); > > The channel is opened as a non-Direct channel. It is just to read the > content and confirm the write operation is performed correctly. I > think there is no need to append .alignedSlice() > Thank you for clarifying, I miss the line 91 " fc = FileChannel.open(p);". > > I think the negative test for non-aligned position are added in > PreadDirect.java and PwriteDirect.java. Please let me know if I missed > anything though. > The negative tests in P[read|write]Direct.java are checking channel position, but in IOUtil.java line 106, it's throwing exception when checking buffer position. > > http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/test/java/nio/channels/FileChannel/directio/PreadDirect.java.html > > The direct read/write in PreadDirect/PwriteDirect.java are put in a > while loop, is it necessary? If yes, then is it necessary for similar > read/write in ReadDirect/WriteDirect.java be put in a while loop too? > 130???????????????? while (totalRead < 4096) { > 131???????????????????? int read = c.read(block, offset); > 132??????????????? ?????if (read < 0) > 133???????????????????????? throw new Exception("Read failed"); > 134???????????????????? totalRead += read; > 135???????????????? } > And I don?t quite understand the logic of above code. > > I followed the example of Read.java and Write.java on the while loop. > I can modify it accordingly. > OK. > > What?s the expected FileChannel.read behavior when remaining bytes in file is less than block size? For example, read from a file which is only 10 bytes long through?ExtendedOpenOption.DIRECT. > It will read 10 bytes and return. > What?s the expected FileChannel.write behavior when remaining valid bytes in buffer is less than block size? For example, write just 10 bytes to a file through?ExtendedOpenOption.DIRECT. > It will throw exception. The remaining buffer size needs to a multiple > of the alignment value. Does this mean that there is no way for direct io to write a file the length of which is not multiple of block size? > What?s the expected FileChannel.position behavior after FileChannel.read/write? > Similarly to the non-DirectIO case, it will move to the current position. Maybe I missed something. But in http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/test/java/nio/channels/FileChannel/directio/PreadDirect.java.html, there is following check, 150 // Ensure that file pointer position has not changed 151 if (originalPosition != newPosition) 152 throw new Exception("File position modified"); > What?s the expected FileChannel.read behavior when remaining bytes in file is more than block size? For example, read 4096 bytes from a file which is 8192 long, is it guaranteed to read out 4096 bytes at once? Similar question for write. > ?????????????? It depends on the current position of the file. If the > position is at the beginning of the file, yes, it will read 4096 > bytes. If the position is > 0, but not aligned, then it will throw > error. If the position is at 4096, it will read 4096 bytes. If the > position is at the end of the file, it will read 0. Everything is > similar to non-Direct channel, except the current position needs to be > aligned. > Is it necessary to add test case for above scenarios? > > ?? I can add tests for case 1. I think other cases are covered with > the existing tests. Please let me know if I missed anything though. > The major reason I asked above several questions is that direct io works a little different from non-direct io, but currently we don't have a java doc to state this behavior difference, I think it would be helpful for direct io end users if we do so, but I'm not sure where to add this java doc, and how detailed it should be. Thank you -Hamlin > > Thanks, > > Lucy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Tue Sep 5 06:01:34 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Tue, 5 Sep 2017 06:01:34 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD11F914D@ORSMSX113.amr.corp.intel.com> Hi Hamlin, Thank you again for the quick feedback. Regarding to add an API into ByteBuffer, I like your suggestion. It makes DirectIO coding much easier. If everyone agrees, I can add the API "ByteBuffer.allocateAlignedDirect(size, alignment)" into ByteBuffer. Please let me know if this is appropriate. Regarding to DirectIOTest with mincore, if the test runs on an idle system with a reasonable amount of memory, I do not think it will be a problem. However, I agree with you that it might be a good idea to add some comments. Is something like this sufficient? "This test is to check whether a file or part of the file exists in the kernel file system cache. As memory pages are not locked in the file system cache, result of the test might be stale before the function returns. The test should not be run concurrently with other programs that clear file system cache" I will add one more test for "not aligned buffer position" check. Thank you for pointing this out. Base on my understanding, we cannot write to a file the length of which is not aligned without padding. I think this the expected behavior of DirectIO. In Java applications, user need to work on their own padding. Thanks, Lucy From: huaming.li at oracle.com [mailto:huaming.li at oracle.com] Sent: Monday, September 04, 2017 8:44 PM To: Lu, Yingqi ; nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, Thank you for response, please check my comments inline. On 05/09/2017 1:50 AM, Lu, Yingqi wrote: Hi Hamlin, Thank you very much for reviewing the patch. It is very helpful! :-) webrev.08 has not been published yet as I am waiting on the feedback to resolve to the AIX issue. I sent a separate email couple days ago with the details. Please see my answers below: http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/src/java.base/share/classes/sun/nio/ch/Util.java.frames.html In line 250, should it use a different cache than non-direct buffer? (Can different files use direct & non-direct io at the same time?) In line 254, what's the possibility of to get here, if it's very low, maybe 1) another cache is necessary? 2) or no cache at all Yes, direct and non-direct IO can be used at the same time. I am not familiar on how often the code is accessed. I will let Alan/Brian and others comment on the proper approach OK. http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java.frames.html Alignment is checked for both remained length and position in lots places (for example below 2 blocks of code), could they be extracted and made 2 functions calls? Good point! As you suggested, I will create 2 static functions inside IOUtil.java, so that both FileChannelImpl and IOUtil class can use them. OK. In java.nio.ByteBuffer Should a new method be introduced in ByteBuffer to combine operations like ByteBuffer.allocateDirect(SIZE + alignment - 1).alignedSlice(alignment); I only see ByteBuffer.allocateDirect() being called twice. I am not totally convinced whether we need to introduce a new API. However, I am surely open to this suggestion if people here agree :) Maybe there is misunderstanding here, I don't mean we should add such method in test, I means to add a new API in java.nio.ByteBuffer class, the reason is that "ByteBuffer.allocateDirect(SIZE + alignment - 1).alignedSlice(alignment);" is necessary for a direct read/write, a new API for example ByteBuffer.allocateAlignedDirect(size, alignment) or allocateAlignedDirect(size) might be helpful for direct io end user. http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/test/java/nio/channels/FileChannel/directio/DirectIOTest.java.html Should add "return" after print out message? else it will not skip the test This is already solved in my local webrev.08. OK. http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/test/java/nio/channels/FileChannel/directio/libDirectIO.c.html In http://man7.org/linux/man-pages/man2/mincore.2.html, it says: The vec argument must point to an array containing at least (length+PAGE_SIZE-1) / PAGE_SIZE bytes. On return, the least significant bit of each byte will be set if the corresponding page is currently resident in memory, and be clear otherwise. (The settings of the other bits in each byte are undefined; these bits are reserved for possible later use.) Of course the information returned in vec is only a snapshot: pages that are not locked in memory can come and go at any moment, and the contents of vec may already be stale by the time this call returns. So, is it possible that DirectIOTest.java may report false positive failure intermittently when Java_DirectIOTest_isFileInCache0 return true? Yes, pages are not locked in memory and they can move freely. However, I think this is the nature of the test. Given the temp file size is very small compare to the memory size and the page check happens right after the IO operation, I think the test should be good in most of the cases. Maybe we need to add some comments in the test, I'm not sure if @intermittent is needed to be added into test. So when test fails in the future, it can be analyzed quickly. http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/test/java/nio/channels/FileChannel/directio/WriteDirect.java.html Do you need to add negative test cases for below exception in IOUtil.java? Similar suggestion for ReadDirect.java 105 if (bb.alignmentOffset(pos, alignment) != 0) { 106 throw new IOException("Current location of the bytebuffer " 107 + "is not a multiple of the block size"); 108 } Should .alignedSlice(alignment); be appended at line 93? 92 try { 93 ByteBuffer bb = ByteBuffer.allocateDirect(8192); 94 fc.read(bb); The channel is opened as a non-Direct channel. It is just to read the content and confirm the write operation is performed correctly. I think there is no need to append .alignedSlice() Thank you for clarifying, I miss the line 91 " fc = FileChannel.open(p);". I think the negative test for non-aligned position are added in PreadDirect.java and PwriteDirect.java. Please let me know if I missed anything though. The negative tests in P[read|write]Direct.java are checking channel position, but in IOUtil.java line 106, it's throwing exception when checking buffer position. http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/test/java/nio/channels/FileChannel/directio/PreadDirect.java.html The direct read/write in PreadDirect/PwriteDirect.java are put in a while loop, is it necessary? If yes, then is it necessary for similar read/write in ReadDirect/WriteDirect.java be put in a while loop too? 130 while (totalRead < 4096) { 131 int read = c.read(block, offset); 132 if (read < 0) 133 throw new Exception("Read failed"); 134 totalRead += read; 135 } And I don't quite understand the logic of above code. I followed the example of Read.java and Write.java on the while loop. I can modify it accordingly. OK. What's the expected FileChannel.read behavior when remaining bytes in file is less than block size? For example, read from a file which is only 10 bytes long through ExtendedOpenOption.DIRECT. It will read 10 bytes and return. What's the expected FileChannel.write behavior when remaining valid bytes in buffer is less than block size? For example, write just 10 bytes to a file through ExtendedOpenOption.DIRECT. It will throw exception. The remaining buffer size needs to a multiple of the alignment value. Does this mean that there is no way for direct io to write a file the length of which is not multiple of block size? What's the expected FileChannel.position behavior after FileChannel.read/write? Similarly to the non-DirectIO case, it will move to the current position. Maybe I missed something. But in http://cr.openjdk.java.net/~kkharbas/8164900/webrev.07/test/java/nio/channels/FileChannel/directio/PreadDirect.java.html, there is following check, 150 // Ensure that file pointer position has not changed 151 if (originalPosition != newPosition) 152 throw new Exception("File position modified"); What's the expected FileChannel.read behavior when remaining bytes in file is more than block size? For example, read 4096 bytes from a file which is 8192 long, is it guaranteed to read out 4096 bytes at once? Similar question for write. It depends on the current position of the file. If the position is at the beginning of the file, yes, it will read 4096 bytes. If the position is > 0, but not aligned, then it will throw error. If the position is at 4096, it will read 4096 bytes. If the position is at the end of the file, it will read 0. Everything is similar to non-Direct channel, except the current position needs to be aligned. Is it necessary to add test case for above scenarios? I can add tests for case 1. I think other cases are covered with the existing tests. Please let me know if I missed anything though. The major reason I asked above several questions is that direct io works a little different from non-direct io, but currently we don't have a java doc to state this behavior difference, I think it would be helpful for direct io end users if we do so, but I'm not sure where to add this java doc, and how detailed it should be. Thank you -Hamlin Thanks, Lucy -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Sep 5 06:38:09 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 5 Sep 2017 07:38:09 +0100 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> Message-ID: On 05/09/2017 04:44, huaming.li at oracle.com wrote: > Maybe there is misunderstanding here, I don't mean we should add such > method in test, I means to add a new API in java.nio.ByteBuffer class, > the reason is that "ByteBuffer.allocateDirect(SIZE + alignment - > 1).alignedSlice(alignment);" is necessary for a direct read/write, a > new API for example ByteBuffer.allocateAlignedDirect(size, alignment) > or allocateAlignedDirect(size) might be helpful for direct io end user. This topic was discussed previously and I think we decided not rev the ByteBuffer API in this iteration. One reason is that direct I/O is a super advanced topic and not something that regular developers will use. So I think the right thing is to move forward on the API as is and re-examine the ByteBuffer API at a future time in the event that advanced developers still need additions to help them use the API. -Alan From huaming.li at oracle.com Tue Sep 5 07:00:42 2017 From: huaming.li at oracle.com (huaming.li at oracle.com) Date: Tue, 5 Sep 2017 15:00:42 +0800 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> Message-ID: <5d3dee45-5a77-388e-7201-74c54c989ca2@oracle.com> On 05/09/2017 2:38 PM, Alan Bateman wrote: > > > On 05/09/2017 04:44, huaming.li at oracle.com wrote: >> Maybe there is misunderstanding here, I don't mean we should add such >> method in test, I means to add a new API in java.nio.ByteBuffer >> class, the reason is that "ByteBuffer.allocateDirect(SIZE + alignment >> - 1).alignedSlice(alignment);" is necessary for a direct read/write, >> a new API for example ByteBuffer.allocateAlignedDirect(size, >> alignment) or allocateAlignedDirect(size) might be helpful for direct >> io end user. > This topic was discussed previously and I think we decided not rev the > ByteBuffer API in this iteration. One reason is that direct I/O is a > super advanced topic and not something that regular developers will > use. So I think the right thing is to move forward on the API as is > and re-examine the ByteBuffer API at a future time in the event that > advanced developers still need additions to help them use the API. Got it, thank you for clarifying. -Hamlin > > -Alan From huaming.li at oracle.com Tue Sep 5 07:06:18 2017 From: huaming.li at oracle.com (huaming.li at oracle.com) Date: Tue, 5 Sep 2017 15:06:18 +0800 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD11F914D@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F914D@ORSMSX113.amr.corp.intel.com> Message-ID: Hi Lucy, On 05/09/2017 2:01 PM, Lu, Yingqi wrote: > > Hi Hamlin, > > Thank you again for the quick feedback. > > Regarding to add an API into ByteBuffer, I like your suggestion. It > makes DirectIO coding much easier. If everyone agrees, I can add the > API ?ByteBuffer.allocateAlignedDirect(size, alignment)? into > ByteBuffer. Please let me know if this is appropriate. > I think Alan just answered this question. > > Regarding to DirectIOTest with mincore, if the test runs on an idle > system with a reasonable amount of memory, I do not think it will be a > problem. However, I agree with you that it might be a good idea to add > some comments. Is something like this sufficient? ?This test is to > check whether a file or part of the file exists in the kernel file > system cache. As memory pages are not locked in the file system cache, > result of the test might be stale before the function returns. The > test should not be run concurrently with other programs that clear > file system cache? As I know, if the test is put as part of jdk regression test, we can not control the test environment where it's busy or idle. But the clarifying comment is better. Maybe we can address the issue later when we face the real failure in test environment. > > I will add one more test for ?not aligned buffer position? check. > Thank you for pointing this out. > OK. > > Base on my understanding, we cannot write to a file the length of > which is not aligned without padding. I think this the expected > behavior of DirectIO. In Java applications, user need to work on their > own padding. > Got it. Thank you -Hamlin > > Thanks, > > Lucy > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Sep 5 18:34:07 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 5 Sep 2017 11:34:07 -0700 Subject: JDK 10 RFR of 8147615: (fc) FileChannelImpl has no finalizer In-Reply-To: References: Message-ID: On Sep 4, 2017, at 4:39 AM, Alan Bateman wrote: > Yes, this is hard to test as you will likely run out of file descriptors before the cleaner runs. One thing you could do is use the management API and check if the OperatingSystemMXBean is a UnixOperatingSystemMXBean as that will give you an easy way to get the maximum and open file descriptors. But we can?t have java.base depending on jdk.management, can we? > As regards the patch then I assume you can skip creating the Cleaner when there is a FileInputStream or FileOutputStream parent (as they have finalizers to close the file, also a reference to the channel). If you do that then it reduces the scenarios to think about to just the FileChannels that are created directly and then unreferenced without calling close. Good point. I understand the close() interrelationship with the parent here but overlooked this. > As regards setting the fd to -1 then this is needed to avoid the cleaner closing a file descriptor that has been recycled to another file or socket or something else. Precisely. > You can synchronize that code. The "what is close fails" is difficult. In other areas then failure will terminate the VM but I don't think this make sense here close might fail with EIO. Minimally EINTR will need special handling as you won't want an exception or logging for that case. The simplest might be to just for ignore and pick up again if/when logging in core libraries is examined. This concerns only the re-thrown exception at FileChannelImpl line 106 or in both invocations of invalidateAndClose()? Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Sep 5 19:31:22 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 5 Sep 2017 12:31:22 -0700 Subject: JDK 10 RFR of 8147615: (fc) FileChannelImpl has no finalizer In-Reply-To: References: Message-ID: On Sep 5, 2017, at 11:34 AM, Brian Burkhalter wrote: > On Sep 4, 2017, at 4:39 AM, Alan Bateman wrote: > >> Yes, this is hard to test as you will likely run out of file descriptors before the cleaner runs. One thing you could do is use the management API and check if the OperatingSystemMXBean is a UnixOperatingSystemMXBean as that will give you an easy way to get the maximum and open file descriptors. > > But we can?t have java.base depending on jdk.management, can we? Never mind: I am sure you meant to use that in a test, not java.nio. Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Sep 5 21:06:21 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 5 Sep 2017 14:06:21 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> Message-ID: <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> On Sep 4, 2017, at 8:44 PM, huaming.li at oracle.com wrote: > Maybe we need to add some comments in the test, I'm not sure if @intermittent is needed to be added into test. So when test fails in the future, it can be analyzed quickly. I don?t know that it?s a good idea to add @intermittent until we know that it is so. >> I can add tests for case 1. I think other cases are covered with the existing tests. Please let me know if I missed anything though. > The major reason I asked above several questions is that direct io works a little different from non-direct io, but currently we don't have a java doc to state this behavior difference, I think it would be helpful for direct io end users if we do so, but I'm not sure where to add this java doc, and how detailed it should be. I think that the javadoc of ExtendedOpenOption#DIRECT is sufficient. We are assuming that users of this feature are advanced and know how direct I/O works and how to use it. I don?t know that we need to weigh down the documentation with too many details. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Roger.Riggs at Oracle.com Tue Sep 5 21:28:52 2017 From: Roger.Riggs at Oracle.com (Roger Riggs) Date: Tue, 5 Sep 2017 17:28:52 -0400 Subject: JDK 10 RFR of 8147615: (fc) FileChannelImpl has no finalizer In-Reply-To: References: Message-ID: <35dedfa6-8fa5-c50e-5933-73bec0ac7c73@Oracle.com> Hi Brian, Can the function to close and invalidate the FileDescriptor be encapsulated in FileDescriptor itself? ? It would be another method on JavaIOFileDescriptorAccess but would bring the handling of close together in one place. Currently, there are several places that replicate the code to do that and FileDescriptor already has Windows and Unix versions. It might also reasonably handle the case where the FileDescriptor has a parent and should or should not call the closeables. [fyi, in parallel, I'm working on changing FIS/FOS/RandomAccess file to use the Cleaner.] Thanks, Roger On 9/1/2017 10:56 AM, Brian Burkhalter wrote: > Please review at your convenience: > > https://bugs.openjdk.java.net/browse/JDK-8147615 > http://cr.openjdk.java.net/~bpb/8147615/webrev.00/ > > There is a question at line 106 of FileChannelImpl.java which needs to be resolved. > > Another question is whether a synchronized block should be used in or around invalidateAndClose() in addition to relying on the pseudo-synchronization of checking the FileDescriptor validity. It is necessary to invalidate the FileDescriptor in any case as otherwise ?Bad file descriptor? exceptions occur. This is likely due to the cleaning action attempting to close a native file descriptor which was already closed when the channel was closed. > > That this actually works was verified by instrumenting FileChannelImpl$Closer#run() with a print statement which was since removed. The issue has however been labelled ?noreg-hard? as there appears to be no way to verify the fix without such instrumentation or via existing public APIs. > > Thanks, > > Brian From brian.burkhalter at oracle.com Tue Sep 5 21:47:21 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 5 Sep 2017 14:47:21 -0700 Subject: JDK 10 RFR of 8147615: (fc) FileChannelImpl has no finalizer In-Reply-To: <35dedfa6-8fa5-c50e-5933-73bec0ac7c73@Oracle.com> References: <35dedfa6-8fa5-c50e-5933-73bec0ac7c73@Oracle.com> Message-ID: Hi Roger, On Sep 5, 2017, at 2:28 PM, Roger Riggs wrote: > Can the function to close and invalidate the FileDescriptor be encapsulated > in FileDescriptor itself? It would be another method on JavaIOFileDescriptorAccess > but would bring the handling of close together in one place. Yes, I have thought about that also. Currently JavaIOFileDescriptorAccess only contains accessor and mutator methods. I do not know whether this implies that it would be unsafe to add methods there which actually perform some function such as closing. > Currently, there are several places that replicate the code to do that and FileDescriptor > already has Windows and Unix versions. I have observed that as well. In particular I compared attach() and closeAll() and they are identical. > It might also reasonably handle the case where the FileDescriptor has a parent and should or should > not call the closeables. Agreed. > [fyi, in parallel, I'm working on changing FIS/FOS/RandomAccess file to use the Cleaner.] Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Roger.Riggs at Oracle.com Tue Sep 5 21:51:56 2017 From: Roger.Riggs at Oracle.com (Roger Riggs) Date: Tue, 5 Sep 2017 17:51:56 -0400 Subject: JDK 10 RFR of 8147615: (fc) FileChannelImpl has no finalizer In-Reply-To: References: <35dedfa6-8fa5-c50e-5933-73bec0ac7c73@Oracle.com> Message-ID: <216a9480-f6e1-e9e6-2ba4-fc1db60ddc9b@Oracle.com> Hi Brian, On 9/5/2017 5:47 PM, Brian Burkhalter wrote: > Hi Roger, > > On Sep 5, 2017, at 2:28 PM, Roger Riggs > wrote: > >> Can the function to close and invalidate the FileDescriptor be >> encapsulated >> in FileDescriptor itself? ? It would be another method on >> JavaIOFileDescriptorAccess >> but would bring the handling of close together in one place. > > Yes, I have thought about that also. Currently > JavaIOFileDescriptorAccess only contains accessor and mutator methods. > I do not know whether this implies that it would be unsafe to add > methods there which actually perform some function such as closing. Adding public methods would increase the chance of mis-use and potential incompatibilities so they can/should only be introduced as package private to be used in the java.io package or via the SharedSecrets interface/mechanism. Thanks, Roger -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Sep 5 22:02:34 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 5 Sep 2017 15:02:34 -0700 Subject: JDK 10 RFR of 8147615: (fc) FileChannelImpl has no finalizer In-Reply-To: <216a9480-f6e1-e9e6-2ba4-fc1db60ddc9b@Oracle.com> References: <35dedfa6-8fa5-c50e-5933-73bec0ac7c73@Oracle.com> <216a9480-f6e1-e9e6-2ba4-fc1db60ddc9b@Oracle.com> Message-ID: Hi Roger, On Sep 5, 2017, at 2:51 PM, Roger Riggs wrote: >> Yes, I have thought about that also. Currently JavaIOFileDescriptorAccess only contains accessor and mutator methods. I do not know whether this implies that it would be unsafe to add methods there which actually perform some function such as closing. > Adding public methods would increase the chance of mis-use and potential incompatibilities so > they can/should only be introduced as package private to be used in the java.io package > or via the SharedSecrets interface/mechanism. I understand. I have not looked through all the jdk.internal.misc.JavaAccess classes, but it seems like in general the methods defined allow only modification of the state of an instance of and not the state of an instance of some other class to which the instance of refers. Having some sort of close() on a FileDescriptor which could be invoked via JavaIOFileDescriptorAccess would allow any attached FIS/FOS/RAF to be closed as well and it was not clear to me whether this sort of behavior was intended. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Tue Sep 5 22:27:17 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Tue, 5 Sep 2017 22:27:17 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> References: <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.corp.intel.com> Hi Brain, Based on the recent feedback I receive, I am going to make following changes into the next version of the patch. Please let me know if there is anything missing or not correctly understood. 1. Apply Volker's updated patch for AIX DirectIO support. 2. Add 2 static functions inside IOUtil.java for position check and remaining buffer size check. Change IOUtil.java and FileChannelImpl.java accordingly. 3. Put Solaris Filesystem check into DirectIOTest.java and use it for all the existing tests. 4. Add 1 additional test for both read and write to test buffer location alignment. 5. Add 1 test case into ReadDirect test represent the case file size is smaller than block size. 6. Remove 4096 from all the tests and replace it with a meaningful variable. 7. Change the function name inside the tests to be more meaningful. Please let me know if there is anything missing. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Tuesday, September 05, 2017 2:06 PM To: huaming.li at oracle.com Cc: Lu, Yingqi ; nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 On Sep 4, 2017, at 8:44 PM, huaming.li at oracle.com wrote: Maybe we need to add some comments in the test, I'm not sure if @intermittent is needed to be added into test. So when test fails in the future, it can be analyzed quickly. I don't know that it's a good idea to add @intermittent until we know that it is so. I can add tests for case 1. I think other cases are covered with the existing tests. Please let me know if I missed anything though. The major reason I asked above several questions is that direct io works a little different from non-direct io, but currently we don't have a java doc to state this behavior difference, I think it would be helpful for direct io end users if we do so, but I'm not sure where to add this java doc, and how detailed it should be. I think that the javadoc of ExtendedOpenOption#DIRECT is sufficient. We are assuming that users of this feature are advanced and know how direct I/O works and how to use it. I don't know that we need to weigh down the documentation with too many details. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Sep 5 22:34:38 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 5 Sep 2017 15:34:38 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.corp.intel.com> References: <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.corp.intel.com> Message-ID: <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> Hi Lucy, Based on what I recall seeing in the e-mail thread since the previous version this sounds to be about right. Thanks, Brian On Sep 5, 2017, at 3:27 PM, Lu, Yingqi wrote: > Based on the recent feedback I receive, I am going to make following changes into the next version of the patch. Please let me know if there is anything missing or not correctly understood. > > 1. Apply Volker?s updated patch for AIX DirectIO support. > > 2. Add 2 static functions inside IOUtil.java for position check and remaining buffer size check. Change IOUtil.java and FileChannelImpl.java accordingly. > > 3. Put Solaris Filesystem check into DirectIOTest.java and use it for all the existing tests. > > 4. Add 1 additional test for both read and write to test buffer location alignment. > > 5. Add 1 test case into ReadDirect test represent the case file size is smaller than block size. > > 6. Remove 4096 from all the tests and replace it with a meaningful variable. > > 7. Change the function name inside the tests to be more meaningful. > > Please let me know if there is anything missing. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Wed Sep 6 00:21:23 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 5 Sep 2017 17:21:23 -0700 Subject: JDK 10 RFR of 8147615: (fc) FileChannelImpl has no finalizer In-Reply-To: References: Message-ID: <0C2FD652-FD96-48D0-B6C1-38C27620D402@oracle.com> On Sep 4, 2017, at 4:39 AM, Alan Bateman wrote: > Yes, this is hard to test as you will likely run out of file descriptors before the cleaner runs. One thing you could do is use the management API and check if the OperatingSystemMXBean is a UnixOperatingSystemMXBean as that will give you an easy way to get the maximum and open file descriptors. > > As regards the patch then I assume you can skip creating the Cleaner when there is a FileInputStream or FileOutputStream parent (as they have finalizers to close the file, also a reference to the channel). If you do that then it reduces the scenarios to think about to just the FileChannels that are created directly and then unreferenced without calling close. > > As regards setting the fd to -1 then this is needed to avoid the cleaner closing a file descriptor that has been recycled to another file or socket or something else. You can synchronize that code. The "what is close fails" is difficult. In other areas then failure will terminate the VM but I don't think this make sense here close might fail with EIO. Minimally EINTR will need special handling as you won't want an exception or logging for that case. The simplest might be to just for ignore and pick up again if/when logging in core libraries is examined. Here is a version updated per the foregoing comments: http://cr.openjdk.java.net/~bpb/8147615/webrev.01/ This continues the path taken by version .00 and does not attempt to incorporate any concept of close() on the FileDescriptor itself. In consideration of the latter it might be that the current patch represents only an interim solution. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From huaming.li at oracle.com Wed Sep 6 00:43:01 2017 From: huaming.li at oracle.com (huaming.li at oracle.com) Date: Wed, 6 Sep 2017 08:43:01 +0800 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> Message-ID: <08dbe7b4-fd52-8f92-cd3d-0093d362962a@oracle.com> Got it, thanks for clarifying. -Hamlin On 06/09/2017 5:06 AM, Brian Burkhalter wrote: > On Sep 4, 2017, at 8:44 PM, huaming.li at oracle.com > wrote: > >> Maybe we need to add some comments in the test, I'm not sure if >> @intermittent is needed to be added into test. So when test fails in >> the future, it can be analyzed quickly. > > I don?t know that it?s a good idea to add @intermittent until we know > that it is so. > >>> I can add tests for case 1. I think other cases are covered with the >>> existing tests. Please let me know if I missed anything though. >> The major reason I asked above several questions is that direct io >> works a little different from non-direct io, but currently we don't >> have a java doc to state this behavior difference, I think it would >> be helpful for direct io end users if we do so, but I'm not sure >> where to add this java doc, and how detailed it should be. > > I think that the javadoc of ExtendedOpenOption#DIRECT is sufficient. > We are assuming that users of this feature are advanced and know how > direct I/O works and how to use it. I don?t know that we need to weigh > down the documentation with too many details. > > Thanks, > > Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Wed Sep 6 18:38:23 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Wed, 6 Sep 2017 18:38:23 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> References: <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.corp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> Hi Brian, The webrev.08 is available at http://cr.openjdk.java.net/~kkharbas/8164900/webrev.08/ It supposes to address the following the issues. Please let me know if there is anything missing. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Tuesday, September 05, 2017 3:35 PM To: Lu, Yingqi Cc: huaming.li at oracle.com; nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, Based on what I recall seeing in the e-mail thread since the previous version this sounds to be about right. Thanks, Brian On Sep 5, 2017, at 3:27 PM, Lu, Yingqi > wrote: Based on the recent feedback I receive, I am going to make following changes into the next version of the patch. Please let me know if there is anything missing or not correctly understood. 1. Apply Volker's updated patch for AIX DirectIO support. 2. Add 2 static functions inside IOUtil.java for position check and remaining buffer size check. Change IOUtil.java and FileChannelImpl.java accordingly. 3. Put Solaris Filesystem check into DirectIOTest.java and use it for all the existing tests. 4. Add 1 additional test for both read and write to test buffer location alignment. 5. Add 1 test case into ReadDirect test represent the case file size is smaller than block size. 6. Remove 4096 from all the tests and replace it with a meaningful variable. 7. Change the function name inside the tests to be more meaningful. Please let me know if there is anything missing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From huaming.li at oracle.com Thu Sep 7 07:52:17 2017 From: huaming.li at oracle.com (huaming.li at oracle.com) Date: Thu, 7 Sep 2017 15:52:17 +0800 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.corp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> Message-ID: <64f93e2d-e1cc-c88a-9cec-271964ee3f65@oracle.com> Hi Lucy, Just for your information. I applied the patch, face following compilation error, I worked on Mac, Darwin Kernel Version 16.7.0. /workspace/jdk/jdk10/jdk/src/java.base/unix/classes/sun/nio/fs/UnixChannelFactory.java:247: error: cannot find symbol ??????????? oflags |= O_DIRECT; ????????????????????? ^ ? symbol:?? variable O_DIRECT ? location: class UnixChannelFactory Thank you -Hamlin On 07/09/2017 2:38 AM, Lu, Yingqi wrote: > > Hi Brian, > > The webrev.08 is available at > http://cr.openjdk.java.net/~kkharbas/8164900/webrev.08/ > > > It supposes to address the following the issues. Please let me know if > there is anything missing. > > Thanks, > > Lucy > > *From:*Brian Burkhalter [mailto:brian.burkhalter at oracle.com] > *Sent:* Tuesday, September 05, 2017 3:35 PM > *To:* Lu, Yingqi > *Cc:* huaming.li at oracle.com; nio-dev at openjdk.java.net > *Subject:* Re: Proposal for adding O_DIRECT support into JDK 9 > > Hi Lucy, > > Based on what I recall seeing in the e-mail thread since the previous > version this sounds to be about right. > > Thanks, > > Brian > > On Sep 5, 2017, at 3:27 PM, Lu, Yingqi > wrote: > > > > Based on the recent feedback I receive, I am going to make > following changes into the next version of the patch. Please let > me know if there is anything missing or not correctly understood. > > 1.Apply Volker?s updated patch for AIX DirectIO support. > > 2.Add 2 static functions inside IOUtil.java for position check and > remaining buffer size check. Change IOUtil.java and > FileChannelImpl.java accordingly. > > 3.Put Solaris Filesystem check into DirectIOTest.java and use it > for all the existing tests. > > 4.Add 1 additional test for both read and write to test buffer > location alignment. > > 5.Add 1 test case into ReadDirect test represent the case file > size is smaller than block size. > > 6.Remove 4096 from all the tests and replace it with a meaningful > variable. > > 7.Change the function name inside the tests to be more meaningful. > > Please let me know if there is anything missing. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From huaming.li at oracle.com Thu Sep 7 08:55:06 2017 From: huaming.li at oracle.com (huaming.li at oracle.com) Date: Thu, 7 Sep 2017 16:55:06 +0800 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <64f93e2d-e1cc-c88a-9cec-271964ee3f65@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.corp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <64f93e2d-e1cc-c88a-9cec-271964ee3f65@oracle.com> Message-ID: <857f5834-3e59-7887-3dc0-babd39d00299@oracle.com> Besides of functionality of this feature, I did some benchmark based on webrev.07, result is at the bottom. jmh benchmark code is at http://cr.openjdk.java.net/~mli/8164900/benchmark.test/ I'm running test on locally built jdk, which is $ hg incoming comparing with http://hg.openjdk.java.net/jdk10/jdk10 searching for changes changeset:?? 2796:13a1041e2950 tag:???????? tip date:??????? Wed Sep 06 16:05:49 2017 +0200 my mac is $ uname -a Darwin huamli-mac 16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64 x86_64 Seems it's not improving performance, maybe there is some degression. I'm not sure if I'm measuring the expected scenario or not, please let me know if I miss something, or misuse jmh or new DirectIO feature. Thank you -Hamlin ------------------------------------------------------------------------ =========================== direct IO =========================== Result "directio.DirectIOBlockSizeFileBenchmarkTest.measureRead": ? 767108.639 ops/s Result "directio.DirectIOBlockSizeFileBenchmarkTest.measureWrite": ? 815677.653 ops/s Result "directio.DirectIOPageSizeFileBenchmarkTest.measureRead": ? 851990.726 ops/s Result "directio.DirectIOPageSizeFileBenchmarkTest.measureWrite": ? 848017.535 ops/s Result "directio.DirectIOMedianFileBenchmarkTest.measureRead": ? 729527.785 ops/s Result "directio.DirectIOMedianFileBenchmarkTest.measureWrite": ? 812342.645 ops/s Result "directio.DirectIOHugeFileBenchmarkTest.measureRead": ? 1331178.163 ops/s Result "directio.DirectIOHugeFileBenchmarkTest.measureWrite": ? 1345364.562 ops/s =========================== normal IO =========================== Result "directio.DirectIOBlockSizeFileBenchmarkTest.measureRead": ? 1110321.245 ops/s Result "directio.DirectIOBlockSizeFileBenchmarkTest.measureWrite": ? 1038699.077 ops/s Result "directio.DirectIOPageSizeFileBenchmarkTest.measureRead": ? 1049606.473 ops/s Result "directio.DirectIOPageSizeFileBenchmarkTest.measureWrite": ? 1212248.812 ops/s Result "directio.DirectIOMedianFileBenchmarkTest.measureRead": ? 1263085.875 ops/s Result "directio.DirectIOMedianFileBenchmarkTest.measureWrite": ? 1225198.455 ops/s Result "directio.DirectIOHugeFileBenchmarkTest.measureRead": ? 2630492.405 ops/s Result "directio.DirectIOHugeFileBenchmarkTest.measureWrite": ? 2647528.429 ops/s On 07/09/2017 3:52 PM, huaming.li at oracle.com wrote: > > Hi Lucy, > > Just for your information. > > I applied the patch, face following compilation error, I worked on > Mac, Darwin Kernel Version 16.7.0. > > /workspace/jdk/jdk10/jdk/src/java.base/unix/classes/sun/nio/fs/UnixChannelFactory.java:247: > error: cannot find symbol > ??????????? oflags |= O_DIRECT; > ????????????????????? ^ > ? symbol:?? variable O_DIRECT > ? location: class UnixChannelFactory > > Thank you > > -Hamlin > > > On 07/09/2017 2:38 AM, Lu, Yingqi wrote: >> >> Hi Brian, >> >> The webrev.08 is available at >> http://cr.openjdk.java.net/~kkharbas/8164900/webrev.08/ >> >> >> It supposes to address the following the issues. Please let me know >> if there is anything missing. >> >> Thanks, >> >> Lucy >> >> *From:*Brian Burkhalter [mailto:brian.burkhalter at oracle.com] >> *Sent:* Tuesday, September 05, 2017 3:35 PM >> *To:* Lu, Yingqi >> *Cc:* huaming.li at oracle.com; nio-dev at openjdk.java.net >> *Subject:* Re: Proposal for adding O_DIRECT support into JDK 9 >> >> Hi Lucy, >> >> Based on what I recall seeing in the e-mail thread since the previous >> version this sounds to be about right. >> >> Thanks, >> >> Brian >> >> On Sep 5, 2017, at 3:27 PM, Lu, Yingqi > > wrote: >> >> >> >> Based on the recent feedback I receive, I am going to make >> following changes into the next version of the patch. Please let >> me know if there is anything missing or not correctly understood. >> >> 1.Apply Volker?s updated patch for AIX DirectIO support. >> >> 2.Add 2 static functions inside IOUtil.java for position check >> and remaining buffer size check. Change IOUtil.java and >> FileChannelImpl.java accordingly. >> >> 3.Put Solaris Filesystem check into DirectIOTest.java and use it >> for all the existing tests. >> >> 4.Add 1 additional test for both read and write to test buffer >> location alignment. >> >> 5.Add 1 test case into ReadDirect test represent the case file >> size is smaller than block size. >> >> 6.Remove 4096 from all the tests and replace it with a meaningful >> variable. >> >> 7.Change the function name inside the tests to be more meaningful. >> >> Please let me know if there is anything missing. >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Thu Sep 7 16:11:55 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Thu, 7 Sep 2017 16:11:55 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <64f93e2d-e1cc-c88a-9cec-271964ee3f65@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.corp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <64f93e2d-e1cc-c88a-9cec-271964ee3f65@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD11FC2C3@ORSMSX113.amr.corp.intel.com> Hi Hamlin, Thank you for the information. I will look into this issue. Volker, please take a look as well and let us know if you know a solution to it. Thanks, Lucy From: huaming.li at oracle.com [mailto:huaming.li at oracle.com] Sent: Thursday, September 07, 2017 12:52 AM To: Lu, Yingqi ; Brian Burkhalter Cc: nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, Just for your information. I applied the patch, face following compilation error, I worked on Mac, Darwin Kernel Version 16.7.0. /workspace/jdk/jdk10/jdk/src/java.base/unix/classes/sun/nio/fs/UnixChannelFactory.java:247: error: cannot find symbol oflags |= O_DIRECT; ^ symbol: variable O_DIRECT location: class UnixChannelFactory Thank you -Hamlin On 07/09/2017 2:38 AM, Lu, Yingqi wrote: Hi Brian, The webrev.08 is available at http://cr.openjdk.java.net/~kkharbas/8164900/webrev.08/ It supposes to address the following the issues. Please let me know if there is anything missing. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Tuesday, September 05, 2017 3:35 PM To: Lu, Yingqi Cc: huaming.li at oracle.com; nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, Based on what I recall seeing in the e-mail thread since the previous version this sounds to be about right. Thanks, Brian On Sep 5, 2017, at 3:27 PM, Lu, Yingqi > wrote: Based on the recent feedback I receive, I am going to make following changes into the next version of the patch. Please let me know if there is anything missing or not correctly understood. 1. Apply Volker's updated patch for AIX DirectIO support. 2. Add 2 static functions inside IOUtil.java for position check and remaining buffer size check. Change IOUtil.java and FileChannelImpl.java accordingly. 3. Put Solaris Filesystem check into DirectIOTest.java and use it for all the existing tests. 4. Add 1 additional test for both read and write to test buffer location alignment. 5. Add 1 test case into ReadDirect test represent the case file size is smaller than block size. 6. Remove 4096 from all the tests and replace it with a meaningful variable. 7. Change the function name inside the tests to be more meaningful. Please let me know if there is anything missing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Thu Sep 7 16:43:10 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Thu, 7 Sep 2017 16:43:10 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <857f5834-3e59-7887-3dc0-babd39d00299@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.corp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <64f93e2d-e1cc-c88a-9cec-271964ee3f65@oracle.com> <857f5834-3e59-7887-3dc0-babd39d00299@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD11FC349@ORSMSX113.amr.corp.intel.com> Hi Hamlin, Thank you for testing the feature. Here is my thinking on the results: 1. In general, DirectIO is an advanced performance feature that needs to be used with caution. It provides most benefits on modern fast storage, such as PCIeSSDs with random reads. In addition, performance benefits also vary with different IO Size and the ratio between dataset and memory capacity. 2. Besides the throughput increase and latency reduction DirectIO might provide, the feature also makes sure the IO performance is consistent and predictable. This is because with DirectIO, IO operations bypass kernel file system cache and read ahead buffer. The page cache thrashing is avoid. 3. In your test, what is the storage type you have on the system? HDD or faster SSDs? This might be the first and main reason that is why you do not see benefits with DirectIO. 4. In addition, I see the maximum dataset size is 2GB (it might contain a single file or a group of files). I am not sure how much memory you have on the system, but I guess a good chunk of the data will end up reside in the memory page cache very soon after the run starts. This might be another reason why you do not see benefits. Please let me know if you have any additional questions. Thanks, Lucy From: huaming.li at oracle.com [mailto:huaming.li at oracle.com] Sent: Thursday, September 07, 2017 1:55 AM To: Lu, Yingqi ; Brian Burkhalter Cc: nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Besides of functionality of this feature, I did some benchmark based on webrev.07, result is at the bottom. jmh benchmark code is at http://cr.openjdk.java.net/~mli/8164900/benchmark.test/ I'm running test on locally built jdk, which is $ hg incoming comparing with http://hg.openjdk.java.net/jdk10/jdk10 searching for changes changeset: 2796:13a1041e2950 tag: tip date: Wed Sep 06 16:05:49 2017 +0200 my mac is $ uname -a Darwin huamli-mac 16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64 x86_64 Seems it's not improving performance, maybe there is some degression. I'm not sure if I'm measuring the expected scenario or not, please let me know if I miss something, or misuse jmh or new DirectIO feature. Thank you -Hamlin ________________________________ =========================== direct IO =========================== Result "directio.DirectIOBlockSizeFileBenchmarkTest.measureRead": 767108.639 ops/s Result "directio.DirectIOBlockSizeFileBenchmarkTest.measureWrite": 815677.653 ops/s Result "directio.DirectIOPageSizeFileBenchmarkTest.measureRead": 851990.726 ops/s Result "directio.DirectIOPageSizeFileBenchmarkTest.measureWrite": 848017.535 ops/s Result "directio.DirectIOMedianFileBenchmarkTest.measureRead": 729527.785 ops/s Result "directio.DirectIOMedianFileBenchmarkTest.measureWrite": 812342.645 ops/s Result "directio.DirectIOHugeFileBenchmarkTest.measureRead": 1331178.163 ops/s Result "directio.DirectIOHugeFileBenchmarkTest.measureWrite": 1345364.562 ops/s =========================== normal IO =========================== Result "directio.DirectIOBlockSizeFileBenchmarkTest.measureRead": 1110321.245 ops/s Result "directio.DirectIOBlockSizeFileBenchmarkTest.measureWrite": 1038699.077 ops/s Result "directio.DirectIOPageSizeFileBenchmarkTest.measureRead": 1049606.473 ops/s Result "directio.DirectIOPageSizeFileBenchmarkTest.measureWrite": 1212248.812 ops/s Result "directio.DirectIOMedianFileBenchmarkTest.measureRead": 1263085.875 ops/s Result "directio.DirectIOMedianFileBenchmarkTest.measureWrite": 1225198.455 ops/s Result "directio.DirectIOHugeFileBenchmarkTest.measureRead": 2630492.405 ops/s Result "directio.DirectIOHugeFileBenchmarkTest.measureWrite": 2647528.429 ops/s On 07/09/2017 3:52 PM, huaming.li at oracle.com wrote: Hi Lucy, Just for your information. I applied the patch, face following compilation error, I worked on Mac, Darwin Kernel Version 16.7.0. /workspace/jdk/jdk10/jdk/src/java.base/unix/classes/sun/nio/fs/UnixChannelFactory.java:247: error: cannot find symbol oflags |= O_DIRECT; ^ symbol: variable O_DIRECT location: class UnixChannelFactory Thank you -Hamlin On 07/09/2017 2:38 AM, Lu, Yingqi wrote: Hi Brian, The webrev.08 is available at http://cr.openjdk.java.net/~kkharbas/8164900/webrev.08/ It supposes to address the following the issues. Please let me know if there is anything missing. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Tuesday, September 05, 2017 3:35 PM To: Lu, Yingqi Cc: huaming.li at oracle.com; nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, Based on what I recall seeing in the e-mail thread since the previous version this sounds to be about right. Thanks, Brian On Sep 5, 2017, at 3:27 PM, Lu, Yingqi > wrote: Based on the recent feedback I receive, I am going to make following changes into the next version of the patch. Please let me know if there is anything missing or not correctly understood. 1. Apply Volker's updated patch for AIX DirectIO support. 2. Add 2 static functions inside IOUtil.java for position check and remaining buffer size check. Change IOUtil.java and FileChannelImpl.java accordingly. 3. Put Solaris Filesystem check into DirectIOTest.java and use it for all the existing tests. 4. Add 1 additional test for both read and write to test buffer location alignment. 5. Add 1 test case into ReadDirect test represent the case file size is smaller than block size. 6. Remove 4096 from all the tests and replace it with a meaningful variable. 7. Change the function name inside the tests to be more meaningful. Please let me know if there is anything missing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Thu Sep 7 18:38:14 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 7 Sep 2017 11:38:14 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> References: <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.corp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> Message-ID: <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> Hi Lucy, On Sep 6, 2017, at 11:38 AM, Lu, Yingqi wrote: > The webrev.08 is available at http://cr.openjdk.java.net/~kkharbas/8164900/webrev.08/ > > It supposes to address the following the issues. Please let me know if there is anything missing. I was off yesterday so I need to catch up here. On Sep 7, 2017, at 12:52 AM, huaming.li at oracle.com wrote: > I applied the patch, face following compilation error, I worked on Mac, Darwin Kernel Version 16.7.0. > > /workspace/jdk/jdk10/jdk/src/java.base/unix/classes/sun/nio/fs/UnixChannelFactory.java:247: error: cannot find symbol > oflags |= O_DIRECT; > ^ > symbol: variable O_DIRECT > location: class UnixChannelFactory > The same failure occurs of course on Solaris. In the build phase it is resolved by --- a/src/java.base/unix/classes/sun/nio/fs/UnixConstants.java.template +++ b/src/java.base/unix/classes/sun/nio/fs/UnixConstants.java.template @@ -72,6 +72,9 @@ #ifdef O_DIRECT static final int PREFIX_O_DIRECT = O_DIRECT; +#else + // not supported (dummy values will not be used at runtime). + static final int PREFIX_O_DIRECT = 00; #endif static final int PREFIX_S_IAMB = I am currently running a build+regression job on Linux, macOS, Solaris, and Windows. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Thu Sep 7 19:41:59 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Thu, 7 Sep 2017 19:41:59 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> References: <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.corp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD11FC80A@ORSMSX113.amr.corp.intel.com> I tested your patch and it works on both MacOS and Linux both. I will include it in the next version of the patch. Please also let me know if you see anything missing from the build+regession test. Thank you very much for your help! Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Thursday, September 07, 2017 11:38 AM To: Lu, Yingqi Cc: huaming.li at oracle.com; nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, On Sep 6, 2017, at 11:38 AM, Lu, Yingqi > wrote: The webrev.08 is available at http://cr.openjdk.java.net/~kkharbas/8164900/webrev.08/ It supposes to address the following the issues. Please let me know if there is anything missing. I was off yesterday so I need to catch up here. On Sep 7, 2017, at 12:52 AM, huaming.li at oracle.com wrote: I applied the patch, face following compilation error, I worked on Mac, Darwin Kernel Version 16.7.0. /workspace/jdk/jdk10/jdk/src/java.base/unix/classes/sun/nio/fs/UnixChannelFactory.java:247: error: cannot find symbol oflags |= O_DIRECT; ^ symbol: variable O_DIRECT location: class UnixChannelFactory The same failure occurs of course on Solaris. In the build phase it is resolved by --- a/src/java.base/unix/classes/sun/nio/fs/UnixConstants.java.template +++ b/src/java.base/unix/classes/sun/nio/fs/UnixConstants.java.template @@ -72,6 +72,9 @@ #ifdef O_DIRECT static final int PREFIX_O_DIRECT = O_DIRECT; +#else + // not supported (dummy values will not be used at runtime). + static final int PREFIX_O_DIRECT = 00; #endif static final int PREFIX_S_IAMB = I am currently running a build+regression job on Linux, macOS, Solaris, and Windows. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Thu Sep 7 20:52:51 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 7 Sep 2017 13:52:51 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD11FC80A@ORSMSX113.amr.corp.intel.com> References: <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.corp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FC80A@ORSMSX113.amr.corp.intel.com> Message-ID: On Sep 7, 2017, at 12:41 PM, Lu, Yingqi wrote: > I tested your patch and it works on both MacOS and Linux both. I will include it in the next version of the patch. > > Please also let me know if you see anything missing from the build+regession test. This patch fixes the build problem on all Unix platforms, but when I run the Direct I/O tests locally on macOS I get this error java.lang.UnsatisfiedLinkError: Native Library /Users/bpb/Work/CoreLibs/jdk/jdk10/dev/build/macosx-x86_64-normal-server-fastdebug/support/test/jdk/jtreg/native/lib/libDirectIO.dylib already loaded in another classloader for the four read and writes tests. The DirectIOTests passes. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Thu Sep 7 21:36:26 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Thu, 7 Sep 2017 21:36:26 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.corp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FC80A@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD11FCA13@ORSMSX113.amr.corp.intel.com> Hi Brian, I might be missing something here, but I think the four read/write tests do not need the libDirectIO.dylib file? Only DirectIOTest.java needs the native library. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Thursday, September 07, 2017 1:53 PM To: Lu, Yingqi Cc: huaming.li at oracle.com; nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 On Sep 7, 2017, at 12:41 PM, Lu, Yingqi > wrote: I tested your patch and it works on both MacOS and Linux both. I will include it in the next version of the patch. Please also let me know if you see anything missing from the build+regession test. This patch fixes the build problem on all Unix platforms, but when I run the Direct I/O tests locally on macOS I get this error java.lang.UnsatisfiedLinkError: Native Library /Users/bpb/Work/CoreLibs/jdk/jdk10/dev/build/macosx-x86_64-normal-server-fastdebug/support/test/jdk/jtreg/native/lib/libDirectIO.dylib already loaded in another classloader for the four read and writes tests. The DirectIOTests passes. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Thu Sep 7 22:05:43 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 7 Sep 2017 15:05:43 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD11FCA13@ORSMSX113.amr.corp.intel.com> References: <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.corp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FC80A@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FCA13@ORSMSX113.amr.corp.intel.com> Message-ID: Hi Lucy, Correct but the other tests call DirectIOTest.isDirectIOSupportedByFS() which causes the static initializer of DirectIOTest to load the library. Thanks, Brian On Sep 7, 2017, at 2:36 PM, Lu, Yingqi wrote: > I might be missing something here, but I think the four read/write tests do not need the libDirectIO.dylib file? Only DirectIOTest.java needs the native library. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Thu Sep 7 22:17:50 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Thu, 7 Sep 2017 22:17:50 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.corp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FC80A@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FCA13@ORSMSX113.amr.corp.intel.com>, Message-ID: Hi Brian, That is very true! I forgot about the solaris check. My bad! I do not see the issue on my Linux box. I am not familiar with the jtreg framework. One easy way I can think of is to have the 4 read/write tests independent of DirectIOTest case by having their own checks. Will this work? Thanks, Lucy Sent from my iPhone On Sep 7, 2017, at 3:05 PM, Brian Burkhalter > wrote: Hi Lucy, Correct but the other tests call DirectIOTest.isDirectIOSupportedByFS() which causes the static initializer of DirectIOTest to load the library. Thanks, Brian On Sep 7, 2017, at 2:36 PM, Lu, Yingqi > wrote: I might be missing something here, but I think the four read/write tests do not need the libDirectIO.dylib file? Only DirectIOTest.java needs the native library. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Thu Sep 7 22:20:06 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 7 Sep 2017 15:20:06 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.corp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FC80A@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FCA13@ORSMSX113.amr.corp.intel.com>, Message-ID: <7061377A-5119-44A9-AE8C-3100FE88BFA5@oracle.com> Hi Lucy, On Sep 7, 2017, at 3:17 PM, Lu, Yingqi wrote: > I do not see the issue on my Linux box. I am not familiar with the jtreg framework. One easy way I can think of is to have the 4 read/write tests independent of DirectIOTest case by having their own checks. Will this work? That would work, or perhaps you could add it to just one of the read/write tests or add a separate Util class to contain the method. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Thu Sep 7 22:23:52 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 7 Sep 2017 15:23:52 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <7061377A-5119-44A9-AE8C-3100FE88BFA5@oracle.com> References: <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.corp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FC80A@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FCA13@ORSMSX113.amr.corp.intel.com>, <7061377A-5119-44A9-AE8C-3100FE88BFA5@oracle.com> Message-ID: Another option which appears to work is to simply move the loadLibrary() call to inside main(): --- a/test/java/nio/channels/FileChannel/directio/DirectIOTest.java +++ b/test/java/nio/channels/FileChannel/directio/DirectIOTest.java @@ -49,10 +49,6 @@ private static final int SIZE = 4096; - static { - System.loadLibrary("DirectIO"); - } - private static void testWrite(Path p) throws Exception { FileChannel fc = FileChannel.open(p, StandardOpenOption.WRITE, ExtendedOpenOption.DIRECT); @@ -113,6 +109,8 @@ return; } + System.loadLibrary("DirectIO"); + try { testWrite(p); if (isFileInCache(p)) { Thanks, Brian On Sep 7, 2017, at 3:20 PM, Brian Burkhalter wrote: > That would work, or perhaps you could add it to just one of the read/write tests or add a separate Util class to contain the method. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Thu Sep 7 22:26:07 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Thu, 7 Sep 2017 22:26:07 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.corp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FC80A@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FCA13@ORSMSX113.amr.corp.intel.com>, <7061377A-5119-44A9-AE8C-3100FE88BFA5@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD11FCBF2@ORSMSX113.amr.corp.intel.com> I like both of them, but second option seems to be simpler. I will use it :) Thank you for the help! Please let me know if you see other issues. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Thursday, September 07, 2017 3:24 PM To: Lu, Yingqi Cc: huaming.li at oracle.com; nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Another option which appears to work is to simply move the loadLibrary() call to inside main(): --- a/test/java/nio/channels/FileChannel/directio/DirectIOTest.java +++ b/test/java/nio/channels/FileChannel/directio/DirectIOTest.java @@ -49,10 +49,6 @@ private static final int SIZE = 4096; - static { - System.loadLibrary("DirectIO"); - } - private static void testWrite(Path p) throws Exception { FileChannel fc = FileChannel.open(p, StandardOpenOption.WRITE, ExtendedOpenOption.DIRECT); @@ -113,6 +109,8 @@ return; } + System.loadLibrary("DirectIO"); + try { testWrite(p); if (isFileInCache(p)) { Thanks, Brian On Sep 7, 2017, at 3:20 PM, Brian Burkhalter > wrote: That would work, or perhaps you could add it to just one of the read/write tests or add a separate Util class to contain the method. -------------- next part -------------- An HTML attachment was scrubbed... URL: From volker.simonis at gmail.com Fri Sep 8 09:57:53 2017 From: volker.simonis at gmail.com (Volker Simonis) Date: Fri, 8 Sep 2017 11:57:53 +0200 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> References: <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.corp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> Message-ID: On Thu, Sep 7, 2017 at 8:38 PM, Brian Burkhalter wrote: > Hi Lucy, > > On Sep 6, 2017, at 11:38 AM, Lu, Yingqi wrote: > > The webrev.08 is available at > http://cr.openjdk.java.net/~kkharbas/8164900/webrev.08/ > > It supposes to address the following the issues. Please let me know if there > is anything missing. > > > I was off yesterday so I need to catch up here. > > On Sep 7, 2017, at 12:52 AM, huaming.li at oracle.com wrote: > > I applied the patch, face following compilation error, I worked on Mac, > Darwin Kernel Version 16.7.0. > > /workspace/jdk/jdk10/jdk/src/java.base/unix/classes/sun/nio/fs/UnixChannelFactory.java:247: > error: cannot find symbol > oflags |= O_DIRECT; > ^ > symbol: variable O_DIRECT > location: class UnixChannelFactory > > The same failure occurs of course on Solaris. In the build phase it is > resolved by > > --- a/src/java.base/unix/classes/sun/nio/fs/UnixConstants.java.template > +++ b/src/java.base/unix/classes/sun/nio/fs/UnixConstants.java.template > @@ -72,6 +72,9 @@ > > > > #ifdef O_DIRECT > static final int PREFIX_O_DIRECT = O_DIRECT; > +#else > + // not supported (dummy values will not be used at runtime). > + static final int PREFIX_O_DIRECT = 00; > #endif > > > > static final int PREFIX_S_IAMB = > Sorry but this is now getting a little confusing as it seems we have two different mail threads for this topic. To resolve the build issue we not only need the patch proposed by Brian, but also the following build patch which I've already sent you four days before on the other mail thread [1]: The problem on Linux is caused by the fact that the build doesn't use "-D_GNU_SOURCE" when processing UnixConstants.java.template and therefore "O_DIRECT" isn't visible. In contrast, the compilation of "FileDispatcherImpl.c" always uses "-D_GNU_SOURCE", so you can freely access "O_DIRECT". A quick fix is the following: diff -r 0c49baac8805 make/gensrc/GensrcMisc.gmk --- a/make/gensrc/GensrcMisc.gmk Wed Aug 30 18:54:50 2017 +0200 +++ b/make/gensrc/GensrcMisc.gmk Mon Sep 04 19:37:35 2017 +0200 @@ -63,7 +63,7 @@ define generate-preproc-src $(call MakeDir, $(@D)) ( $(NAWK) '/@@END_COPYRIGHT@@/{exit}1' $< && \ - $(CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $< \ + $(CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $(CFLAGS_JDKLIB) $< \ 2> >($(GREP) -v '^$(&2) \ | $(NAWK) '/@@START_HERE@@/,0' \ | $(SED) -e 's/@@START_HERE@@/\/\/ AUTOMATICALLY GENERATED FILE - DO NOT EDIT/' \ which worked for me on Linux/Solaris/AIX (haven't tested on Mac). I think it was always a mistake that we haven't generated the UnixConstants with the same C-Flags that we've also used to compile the files which used them. It was probably just luck that we've not run into problems until now. [1] http://mail.openjdk.java.net/pipermail/nio-dev/2017-September/004478.html > I am currently running a build+regression job on Linux, macOS, Solaris, and > Windows. > > Thanks, > > Brian From huaming.li at oracle.com Fri Sep 8 10:22:16 2017 From: huaming.li at oracle.com (huaming.li at oracle.com) Date: Fri, 8 Sep 2017 18:22:16 +0800 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD11FC349@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD11F30C8@ORSMSX113.amr.c! ! orp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.corp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <64f93e2d-e1cc-c88a-9cec-271964ee3f65@oracle.com> <857f5834-3e59-7887-3dc0-babd39d00299@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FC349@ORSMSX113.amr.corp.intel.com> Message-ID: <361448f1-99e5-e4bd-13e8-b33278f4317e@oracle.com> Hi Lucy, Thank you for follow-up. I'm not familiar with direct io, it may report false positive issue, or testing unexpected scenario, so the test result is just for your information my local dist type: Device / Media Name:????? APPLE SSD SM0512G Thank you -Hamlin On 08/09/2017 12:43 AM, Lu, Yingqi wrote: > > Hi Hamlin, > > Thank you for testing the feature. Here is my thinking on the results: > > 1.In general, DirectIO is an advanced performance feature that needs > to be used with caution. It provides most benefits on modern fast > storage, such as PCIeSSDs with random reads. In addition, performance > benefits also vary with different IO Size and the ratio between > dataset and memory capacity. > > 2.Besides the throughput increase and latency reduction DirectIO might > provide, the feature also makes sure the IO performance is consistent > and predictable. This is because with DirectIO, IO operations bypass > kernel file system cache and read ahead buffer. The page cache > thrashing is avoid. > > 3.In your test, what is the storage type you have on the system? HDD > or faster SSDs? This might be the first and main reason that is why > you do not see benefits with DirectIO. > > 4.In addition, I see the maximum dataset size is 2GB (it might contain > a single file or a group of files). I am not sure how much memory you > have on the system, but I guess a good chunk of the data will end up > reside in the memory page cache very soon after the run starts. This > might be another reason why you do not see benefits. > > Please let me know if you have any additional questions. > > Thanks, > > Lucy > > *From:*huaming.li at oracle.com [mailto:huaming.li at oracle.com] > *Sent:* Thursday, September 07, 2017 1:55 AM > *To:* Lu, Yingqi ; Brian Burkhalter > > *Cc:* nio-dev at openjdk.java.net > *Subject:* Re: Proposal for adding O_DIRECT support into JDK 9 > > Besides of functionality of this feature, I did some benchmark based > on webrev.07, result is at the bottom. > > jmh benchmark code is at > http://cr.openjdk.java.net/~mli/8164900/benchmark.test/ > > > I'm running test on locally built jdk, which is > > $ hg incoming > comparing with http://hg.openjdk.java.net/jdk10/jdk10 > searching for changes > changeset:?? 2796:13a1041e2950 > tag:???????? tip > date:??????? Wed Sep 06 16:05:49 2017 +0200 > > my mac is > $ uname -a > Darwin huamli-mac 16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 > 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64 x86_64 > > Seems it's not improving performance, maybe there is some degression. > I'm not sure if I'm measuring the expected scenario or not, please let > me know if I miss something, or misuse jmh or new DirectIO feature. > > Thank you > -Hamlin > > ------------------------------------------------------------------------ > > > =========================== direct IO =========================== > > Result "directio.DirectIOBlockSizeFileBenchmarkTest.measureRead": > ? 767108.639 ops/s > Result "directio.DirectIOBlockSizeFileBenchmarkTest.measureWrite": > ? 815677.653 ops/s > > Result "directio.DirectIOPageSizeFileBenchmarkTest.measureRead": > ? 851990.726 ops/s > Result "directio.DirectIOPageSizeFileBenchmarkTest.measureWrite": > ? 848017.535 ops/s > > Result "directio.DirectIOMedianFileBenchmarkTest.measureRead": > ? 729527.785 ops/s > Result "directio.DirectIOMedianFileBenchmarkTest.measureWrite": > ? 812342.645 ops/s > > Result "directio.DirectIOHugeFileBenchmarkTest.measureRead": > ? 1331178.163 ops/s > Result "directio.DirectIOHugeFileBenchmarkTest.measureWrite": > ? 1345364.562 ops/s > > =========================== normal IO =========================== > Result "directio.DirectIOBlockSizeFileBenchmarkTest.measureRead": > ? 1110321.245 ops/s > Result "directio.DirectIOBlockSizeFileBenchmarkTest.measureWrite": > ? 1038699.077 ops/s > > Result "directio.DirectIOPageSizeFileBenchmarkTest.measureRead": > ? 1049606.473 ops/s > Result "directio.DirectIOPageSizeFileBenchmarkTest.measureWrite": > ? 1212248.812 ops/s > > Result "directio.DirectIOMedianFileBenchmarkTest.measureRead": > ? 1263085.875 ops/s > Result "directio.DirectIOMedianFileBenchmarkTest.measureWrite": > ? 1225198.455 ops/s > > Result "directio.DirectIOHugeFileBenchmarkTest.measureRead": > ? 2630492.405 ops/s > Result "directio.DirectIOHugeFileBenchmarkTest.measureWrite": > ? 2647528.429 ops/s > > On 07/09/2017 3:52 PM, huaming.li at oracle.com > wrote: > > Hi Lucy, > > Just for your information. > > I applied the patch, face following compilation error, I worked on > Mac, Darwin Kernel Version 16.7.0. > > /workspace/jdk/jdk10/jdk/src/java.base/unix/classes/sun/nio/fs/UnixChannelFactory.java:247: > error: cannot find symbol > ??????????? oflags |= O_DIRECT; > ????????????????????? ^ > ? symbol:?? variable O_DIRECT > ? location: class UnixChannelFactory > > Thank you > > -Hamlin > > On 07/09/2017 2:38 AM, Lu, Yingqi wrote: > > Hi Brian, > > The webrev.08 is available at > http://cr.openjdk.java.net/~kkharbas/8164900/webrev.08/ > > > It supposes to address the following the issues. Please let me > know if there is anything missing. > > Thanks, > > Lucy > > *From:*Brian Burkhalter [mailto:brian.burkhalter at oracle.com] > *Sent:* Tuesday, September 05, 2017 3:35 PM > *To:* Lu, Yingqi > > *Cc:* huaming.li at oracle.com ; > nio-dev at openjdk.java.net > *Subject:* Re: Proposal for adding O_DIRECT support into JDK 9 > > Hi Lucy, > > Based on what I recall seeing in the e-mail thread since the > previous version this sounds to be about right. > > Thanks, > > Brian > > On Sep 5, 2017, at 3:27 PM, Lu, Yingqi > wrote: > > > > > Based on the recent feedback I receive, I am going to make > following changes into the next version of the patch. > Please let me know if there is anything missing or not > correctly understood. > > 1.Apply Volker?s updated patch for AIX DirectIO support. > > 2.Add 2 static functions inside IOUtil.java for position > check and remaining buffer size check. Change IOUtil.java > and FileChannelImpl.java accordingly. > > 3.Put Solaris Filesystem check into DirectIOTest.java and > use it for all the existing tests. > > 4.Add 1 additional test for both read and write to test > buffer location alignment. > > 5.Add 1 test case into ReadDirect test represent the case > file size is smaller than block size. > > 6.Remove 4096 from all the tests and replace it with a > meaningful variable. > > 7.Change the function name inside the tests to be more > meaningful. > > Please let me know if there is anything missing. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Fri Sep 8 16:12:10 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Fri, 8 Sep 2017 16:12:10 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.corp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD11FD7DD@ORSMSX113.amr.corp.intel.com> Hi Volker, Sorry for the confusion. Brian's patch is patching on top of your patch. I already included your patch in webrev.08. However, it has issue building on MacOS. Brian's patch fixed it. I will incorporate Brian's patch into webrev.09 which will be online sometime today. Please let me know if you have any questions. Thanks, Lucy >-----Original Message----- >From: Volker Simonis [mailto:volker.simonis at gmail.com] >Sent: Friday, September 08, 2017 2:58 AM >To: Brian Burkhalter >Cc: Lu, Yingqi ; nio-dev at openjdk.java.net >Subject: Re: Proposal for adding O_DIRECT support into JDK 9 > >On Thu, Sep 7, 2017 at 8:38 PM, Brian Burkhalter >wrote: >> Hi Lucy, >> >> On Sep 6, 2017, at 11:38 AM, Lu, Yingqi wrote: >> >> The webrev.08 is available at >> http://cr.openjdk.java.net/~kkharbas/8164900/webrev.08/ >> >> It supposes to address the following the issues. Please let me know if >> there is anything missing. >> >> >> I was off yesterday so I need to catch up here. >> >> On Sep 7, 2017, at 12:52 AM, huaming.li at oracle.com wrote: >> >> I applied the patch, face following compilation error, I worked on >> Mac, Darwin Kernel Version 16.7.0. >> >> >/workspace/jdk/jdk10/jdk/src/java.base/unix/classes/sun/nio/fs/UnixChannelFac >tory.java:247: >> error: cannot find symbol >> oflags |= O_DIRECT; >> ^ >> symbol: variable O_DIRECT >> location: class UnixChannelFactory >> >> The same failure occurs of course on Solaris. In the build phase it is >> resolved by >> >> --- >> a/src/java.base/unix/classes/sun/nio/fs/UnixConstants.java.template >> +++ b/src/java.base/unix/classes/sun/nio/fs/UnixConstants.java.templat >> +++ e >> @@ -72,6 +72,9 @@ >> >> >> >> #ifdef O_DIRECT >> static final int PREFIX_O_DIRECT = O_DIRECT; >> +#else >> + // not supported (dummy values will not be used at runtime). >> + static final int PREFIX_O_DIRECT = 00; >> #endif >> >> >> >> static final int PREFIX_S_IAMB = >> > >Sorry but this is now getting a little confusing as it seems we have two different >mail threads for this topic. > >To resolve the build issue we not only need the patch proposed by Brian, but also >the following build patch which I've already sent you four days before on the >other mail thread [1]: > >The problem on Linux is caused by the fact that the build doesn't use "- >D_GNU_SOURCE" when processing UnixConstants.java.template and therefore >"O_DIRECT" isn't visible. In contrast, the compilation of "FileDispatcherImpl.c" >always uses "-D_GNU_SOURCE", so you can freely access "O_DIRECT". > >A quick fix is the following: > >diff -r 0c49baac8805 make/gensrc/GensrcMisc.gmk >--- a/make/gensrc/GensrcMisc.gmk Wed Aug 30 18:54:50 2017 +0200 >+++ b/make/gensrc/GensrcMisc.gmk Mon Sep 04 19:37:35 2017 +0200 >@@ -63,7 +63,7 @@ > define generate-preproc-src > $(call MakeDir, $(@D)) > ( $(NAWK) '/@@END_COPYRIGHT@@/{exit}1' $< && \ >- $(CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $< \ >+ $(CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $(CFLAGS_JDKLIB) $< \ > 2> >($(GREP) -v '^$(&2) \ > | $(NAWK) '/@@START_HERE@@/,0' \ > | $(SED) -e 's/@@START_HERE@@/\/\/ AUTOMATICALLY GENERATED >FILE - DO NOT EDIT/' \ > >which worked for me on Linux/Solaris/AIX (haven't tested on Mac). > >I think it was always a mistake that we haven't generated the UnixConstants with >the same C-Flags that we've also used to compile the files which used them. It >was probably just luck that we've not run into problems until now. > >[1] http://mail.openjdk.java.net/pipermail/nio-dev/2017-September/004478.html > >> I am currently running a build+regression job on Linux, macOS, >> Solaris, and Windows. >> >> Thanks, >> >> Brian From volker.simonis at gmail.com Fri Sep 8 16:36:39 2017 From: volker.simonis at gmail.com (Volker Simonis) Date: Fri, 8 Sep 2017 18:36:39 +0200 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD11FD7DD@ORSMSX113.amr.corp.intel.com> References: <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.corp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD7DD@ORSMSX113.amr.corp.intel.com> Message-ID: On Fri, Sep 8, 2017 at 6:12 PM, Lu, Yingqi wrote: > Hi Volker, > > Sorry for the confusion. Brian's patch is patching on top of your patch. I already included your patch in webrev.08. However, it has issue building on MacOS. Brian's patch fixed it. I will incorporate Brian's patch into webrev.09 which will be online sometime today. > Sorry, but I can't see my patch in webrev.08. My fix was in make/gensrc/GensrcMisc.gmk which isn't touched in http://cr.openjdk.java.net/~kkharbas/8164900/webrev.08/ Brian's fix is OK for platforms which don't define O_DIRECT at all. But for Linux it will simply define O_DIRECT to 00 which is wrong! We have to generate UnixConstants.java from UnixConstants.java.template with the same CPP defines we use for the compilation of the actual C-sources because O_DIRECT is only defined on Linux if we are using "-D_GNU_SOURCE". Regards, Volker > Please let me know if you have any questions. > > Thanks, > Lucy > >>-----Original Message----- >>From: Volker Simonis [mailto:volker.simonis at gmail.com] >>Sent: Friday, September 08, 2017 2:58 AM >>To: Brian Burkhalter >>Cc: Lu, Yingqi ; nio-dev at openjdk.java.net >>Subject: Re: Proposal for adding O_DIRECT support into JDK 9 >> >>On Thu, Sep 7, 2017 at 8:38 PM, Brian Burkhalter >>wrote: >>> Hi Lucy, >>> >>> On Sep 6, 2017, at 11:38 AM, Lu, Yingqi wrote: >>> >>> The webrev.08 is available at >>> http://cr.openjdk.java.net/~kkharbas/8164900/webrev.08/ >>> >>> It supposes to address the following the issues. Please let me know if >>> there is anything missing. >>> >>> >>> I was off yesterday so I need to catch up here. >>> >>> On Sep 7, 2017, at 12:52 AM, huaming.li at oracle.com wrote: >>> >>> I applied the patch, face following compilation error, I worked on >>> Mac, Darwin Kernel Version 16.7.0. >>> >>> >>/workspace/jdk/jdk10/jdk/src/java.base/unix/classes/sun/nio/fs/UnixChannelFac >>tory.java:247: >>> error: cannot find symbol >>> oflags |= O_DIRECT; >>> ^ >>> symbol: variable O_DIRECT >>> location: class UnixChannelFactory >>> >>> The same failure occurs of course on Solaris. In the build phase it is >>> resolved by >>> >>> --- >>> a/src/java.base/unix/classes/sun/nio/fs/UnixConstants.java.template >>> +++ b/src/java.base/unix/classes/sun/nio/fs/UnixConstants.java.templat >>> +++ e >>> @@ -72,6 +72,9 @@ >>> >>> >>> >>> #ifdef O_DIRECT >>> static final int PREFIX_O_DIRECT = O_DIRECT; >>> +#else >>> + // not supported (dummy values will not be used at runtime). >>> + static final int PREFIX_O_DIRECT = 00; >>> #endif >>> >>> >>> >>> static final int PREFIX_S_IAMB = >>> >> >>Sorry but this is now getting a little confusing as it seems we have two different >>mail threads for this topic. >> >>To resolve the build issue we not only need the patch proposed by Brian, but also >>the following build patch which I've already sent you four days before on the >>other mail thread [1]: >> >>The problem on Linux is caused by the fact that the build doesn't use "- >>D_GNU_SOURCE" when processing UnixConstants.java.template and therefore >>"O_DIRECT" isn't visible. In contrast, the compilation of "FileDispatcherImpl.c" >>always uses "-D_GNU_SOURCE", so you can freely access "O_DIRECT". >> >>A quick fix is the following: >> >>diff -r 0c49baac8805 make/gensrc/GensrcMisc.gmk >>--- a/make/gensrc/GensrcMisc.gmk Wed Aug 30 18:54:50 2017 +0200 >>+++ b/make/gensrc/GensrcMisc.gmk Mon Sep 04 19:37:35 2017 +0200 >>@@ -63,7 +63,7 @@ >> define generate-preproc-src >> $(call MakeDir, $(@D)) >> ( $(NAWK) '/@@END_COPYRIGHT@@/{exit}1' $< && \ >>- $(CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $< \ >>+ $(CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $(CFLAGS_JDKLIB) $< \ >> 2> >($(GREP) -v '^$(&2) \ >> | $(NAWK) '/@@START_HERE@@/,0' \ >> | $(SED) -e 's/@@START_HERE@@/\/\/ AUTOMATICALLY GENERATED >>FILE - DO NOT EDIT/' \ >> >>which worked for me on Linux/Solaris/AIX (haven't tested on Mac). >> >>I think it was always a mistake that we haven't generated the UnixConstants with >>the same C-Flags that we've also used to compile the files which used them. It >>was probably just luck that we've not run into problems until now. >> >>[1] http://mail.openjdk.java.net/pipermail/nio-dev/2017-September/004478.html >> >>> I am currently running a build+regression job on Linux, macOS, >>> Solaris, and Windows. >>> >>> Thanks, >>> >>> Brian From yingqi.lu at intel.com Fri Sep 8 16:45:35 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Fri, 8 Sep 2017 16:45:35 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.corp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD7DD@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD11FD8C3@ORSMSX113.amr.corp.intel.com> Hi Volker, Sorry for the confusion again. The change you refer to is the following one or something else? --- old/make/gensrc/GensrcMisc.gmk 2017-09-06 01:38:06.005049364 -0400 +++ new/make/gensrc/GensrcMisc.gmk 2017-09-06 01:38:05.885049363 -0400 @@ -63,7 +63,7 @@ define generate-preproc-src $(call MakeDir, $(@D)) ( $(NAWK) '/@@END_COPYRIGHT@@/{exit}1' $< && \ - $(CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $< \ + $(CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $(CFLAGS_JDKLIB) $< \ 2> >($(GREP) -v '^$(&2) \ | $(NAWK) '/@@START_HERE@@/,0' \ | $(SED) -e 's/@@START_HERE@@/\/\/ AUTOMATICALLY GENERATED FILE - DO NOT EDIT/' \ If it is above, it is already in the http://cr.openjdk.java.net/~kkharbas/8164900/webrev.08/make/gensrc/GensrcMisc.gmk.patch. If it is not, please let me know. I think Brian's patch still define O_DIRECT correctly on Linux since O_DIRECT is defined on Linux, but I might miss something. Please let me know if I misunderstand anything here. +#ifdef O_DIRECT + static final int PREFIX_O_DIRECT = O_DIRECT; +#else + // not supported (dummy values will not be used at runtime). + static final int PREFIX_O_DIRECT = 00; +#endif Thanks, Lucy >-----Original Message----- >From: Volker Simonis [mailto:volker.simonis at gmail.com] >Sent: Friday, September 08, 2017 9:37 AM >To: Lu, Yingqi >Cc: Brian Burkhalter ; nio-dev at openjdk.java.net >Subject: Re: Proposal for adding O_DIRECT support into JDK 9 > >On Fri, Sep 8, 2017 at 6:12 PM, Lu, Yingqi wrote: >> Hi Volker, >> >> Sorry for the confusion. Brian's patch is patching on top of your patch. I already >included your patch in webrev.08. However, it has issue building on MacOS. >Brian's patch fixed it. I will incorporate Brian's patch into webrev.09 which will be >online sometime today. >> > >Sorry, but I can't see my patch in webrev.08. My fix was in >make/gensrc/GensrcMisc.gmk which isn't touched in >http://cr.openjdk.java.net/~kkharbas/8164900/webrev.08/ > >Brian's fix is OK for platforms which don't define O_DIRECT at all. >But for Linux it will simply define O_DIRECT to 00 which is wrong! We have to >generate UnixConstants.java from UnixConstants.java.template with the same >CPP defines we use for the compilation of the actual C-sources because >O_DIRECT is only defined on Linux if we are using "-D_GNU_SOURCE". > >Regards, >Volker > >> Please let me know if you have any questions. >> >> Thanks, >> Lucy >> >>>-----Original Message----- >>>From: Volker Simonis [mailto:volker.simonis at gmail.com] >>>Sent: Friday, September 08, 2017 2:58 AM >>>To: Brian Burkhalter >>>Cc: Lu, Yingqi ; nio-dev at openjdk.java.net >>>Subject: Re: Proposal for adding O_DIRECT support into JDK 9 >>> >>>On Thu, Sep 7, 2017 at 8:38 PM, Brian Burkhalter >>> >>>wrote: >>>> Hi Lucy, >>>> >>>> On Sep 6, 2017, at 11:38 AM, Lu, Yingqi wrote: >>>> >>>> The webrev.08 is available at >>>> http://cr.openjdk.java.net/~kkharbas/8164900/webrev.08/ >>>> >>>> It supposes to address the following the issues. Please let me know >>>> if there is anything missing. >>>> >>>> >>>> I was off yesterday so I need to catch up here. >>>> >>>> On Sep 7, 2017, at 12:52 AM, huaming.li at oracle.com wrote: >>>> >>>> I applied the patch, face following compilation error, I worked on >>>> Mac, Darwin Kernel Version 16.7.0. >>>> >>>> >>>/workspace/jdk/jdk10/jdk/src/java.base/unix/classes/sun/nio/fs/UnixCha >>>nnelFac >>>tory.java:247: >>>> error: cannot find symbol >>>> oflags |= O_DIRECT; >>>> ^ >>>> symbol: variable O_DIRECT >>>> location: class UnixChannelFactory >>>> >>>> The same failure occurs of course on Solaris. In the build phase it >>>> is resolved by >>>> >>>> --- >>>> a/src/java.base/unix/classes/sun/nio/fs/UnixConstants.java.template >>>> +++ b/src/java.base/unix/classes/sun/nio/fs/UnixConstants.java.templ >>>> +++ at >>>> +++ e >>>> @@ -72,6 +72,9 @@ >>>> >>>> >>>> >>>> #ifdef O_DIRECT >>>> static final int PREFIX_O_DIRECT = O_DIRECT; >>>> +#else >>>> + // not supported (dummy values will not be used at runtime). >>>> + static final int PREFIX_O_DIRECT = 00; >>>> #endif >>>> >>>> >>>> >>>> static final int PREFIX_S_IAMB = >>>> >>> >>>Sorry but this is now getting a little confusing as it seems we have >>>two different mail threads for this topic. >>> >>>To resolve the build issue we not only need the patch proposed by >>>Brian, but also the following build patch which I've already sent you >>>four days before on the other mail thread [1]: >>> >>>The problem on Linux is caused by the fact that the build doesn't use >>>"- D_GNU_SOURCE" when processing UnixConstants.java.template and >>>therefore "O_DIRECT" isn't visible. In contrast, the compilation of >"FileDispatcherImpl.c" >>>always uses "-D_GNU_SOURCE", so you can freely access "O_DIRECT". >>> >>>A quick fix is the following: >>> >>>diff -r 0c49baac8805 make/gensrc/GensrcMisc.gmk >>>--- a/make/gensrc/GensrcMisc.gmk Wed Aug 30 18:54:50 2017 +0200 >>>+++ b/make/gensrc/GensrcMisc.gmk Mon Sep 04 19:37:35 2017 +0200 >>>@@ -63,7 +63,7 @@ >>> define generate-preproc-src >>> $(call MakeDir, $(@D)) >>> ( $(NAWK) '/@@END_COPYRIGHT@@/{exit}1' $< && \ >>>- $(CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $< \ >>>+ $(CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $(CFLAGS_JDKLIB) $< \ >>> 2> >($(GREP) -v '^$(&2) \ >>> | $(NAWK) '/@@START_HERE@@/,0' \ >>> | $(SED) -e 's/@@START_HERE@@/\/\/ AUTOMATICALLY >>>GENERATED FILE - DO NOT EDIT/' \ >>> >>>which worked for me on Linux/Solaris/AIX (haven't tested on Mac). >>> >>>I think it was always a mistake that we haven't generated the >>>UnixConstants with the same C-Flags that we've also used to compile >>>the files which used them. It was probably just luck that we've not run into >problems until now. >>> >>>[1] >>>http://mail.openjdk.java.net/pipermail/nio-dev/2017-September/004478.h >>>tml >>> >>>> I am currently running a build+regression job on Linux, macOS, >>>> Solaris, and Windows. >>>> >>>> Thanks, >>>> >>>> Brian From brian.burkhalter at oracle.com Fri Sep 8 18:53:38 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 8 Sep 2017 11:53:38 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD11FD8C3@ORSMSX113.amr.corp.intel.com> References: <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.c! orp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD7DD@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD8C3@ORSMSX113.amr.corp.intel.com> Message-ID: It?s in there. Thanks, Brian On Sep 8, 2017, at 9:45 AM, Lu, Yingqi wrote: > The change you refer to is the following one or something else? > > --- old/make/gensrc/GensrcMisc.gmk 2017-09-06 01:38:06.005049364 -0400 > +++ new/make/gensrc/GensrcMisc.gmk 2017-09-06 01:38:05.885049363 -0400 > @@ -63,7 +63,7 @@ > define generate-preproc-src > $(call MakeDir, $(@D)) > ( $(NAWK) '/@@END_COPYRIGHT@@/{exit}1' $< && \ > - $(CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $< \ > + $(CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $(CFLAGS_JDKLIB) $< \ > 2> >($(GREP) -v '^$(&2) \ > | $(NAWK) '/@@START_HERE@@/,0' \ > | $(SED) -e 's/@@START_HERE@@/\/\/ AUTOMATICALLY GENERATED FILE - DO NOT EDIT/' \ > > If it is above, it is already in the http://cr.openjdk.java.net/~kkharbas/8164900/webrev.08/make/gensrc/GensrcMisc.gmk.patch. If it is not, please let me know. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Fri Sep 8 20:41:28 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Fri, 8 Sep 2017 20:41:28 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.c! orp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD7DD@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD8C3@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD11FDC74@ORSMSX113.amr.corp.intel.com> Hi Brian, Please find the most recent version of the patch at http://cr.openjdk.java.net/~kkharbas/8164900/webrev.09/ In this version, following issues get addressed: 1. Fixed the issue with compilation on AIX 2. Fixed the issue on the tests with loading the native library Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Friday, September 08, 2017 11:54 AM To: Lu, Yingqi Cc: Volker Simonis ; nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 It's in there. Thanks, Brian On Sep 8, 2017, at 9:45 AM, Lu, Yingqi > wrote: The change you refer to is the following one or something else? --- old/make/gensrc/GensrcMisc.gmk 2017-09-06 01:38:06.005049364 -0400 +++ new/make/gensrc/GensrcMisc.gmk 2017-09-06 01:38:05.885049363 -0400 @@ -63,7 +63,7 @@ define generate-preproc-src $(call MakeDir, $(@D)) ( $(NAWK) '/@@END_COPYRIGHT@@/{exit}1' $< && \ - $(CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $< \ + $(CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $(CFLAGS_JDKLIB) $< \ 2> >($(GREP) -v '^$(&2) \ | $(NAWK) '/@@START_HERE@@/,0' \ | $(SED) -e 's/@@START_HERE@@/\/\/ AUTOMATICALLY GENERATED FILE - DO NOT EDIT/' \ If it is above, it is already in the http://cr.openjdk.java.net/~kkharbas/8164900/webrev.08/make/gensrc/GensrcMisc.gmk.patch. If it is not, please let me know. -------------- next part -------------- An HTML attachment was scrubbed... URL: From volker.simonis at gmail.com Sat Sep 9 10:30:40 2017 From: volker.simonis at gmail.com (Volker Simonis) Date: Sat, 09 Sep 2017 10:30:40 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <20dd0be6-d9f1-5e32-9333-2d936a123251@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F184D@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2C52@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F2F7F@ORSMSX113.amr.corp.intel.com> <3E075F2E-E0C9-4404-A4D2-1AFAAB79B692@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F328C@ORSMSX113.amr.corp.intel.com> <5BD9C1D1-F7EE-4402-BEAD-F6F48A09D1EE@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD7DD@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD8C3@ORSMSX113.amr.corp.intel.com> Message-ID: Brian Burkhalter schrieb am Fr. 8. Sep. 2017 um 20:53: > It?s in there. > Yes, it's in. Just verfied it from home. So this was a caching problem with my browser. I had that before, although I did several reloads. I therefore usually strongly discourage people from changing webrevs in-place on cr.openjdk.java.net. > > Thanks, > > Brian > > On Sep 8, 2017, at 9:45 AM, Lu, Yingqi wrote: > > The change you refer to is the following one or something else? > > --- old/make/gensrc/GensrcMisc.gmk 2017-09-06 01:38:06.005049364 -0400 > +++ new/make/gensrc/GensrcMisc.gmk 2017-09-06 01:38:05.885049363 -0400 > @@ -63,7 +63,7 @@ > define generate-preproc-src > $(call MakeDir, $(@D)) > ( $(NAWK) '/@@END_COPYRIGHT@@/{exit}1' $< && \ > - $(CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $< \ > + $(CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $(CFLAGS_JDKLIB) $< \ > 2> >($(GREP) -v '^$(&2) \ > | $(NAWK) '/@@START_HERE@@/,0' \ > | $(SED) -e 's/@@START_HERE@@/\/\/ AUTOMATICALLY GENERATED FILE - > DO NOT EDIT/' \ > > If it is above, it is already in the > http://cr.openjdk.java.net/~kkharbas/8164900/webrev.08/make/gensrc/GensrcMisc.gmk.patch. > If it is not, please let me know. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Mon Sep 11 11:21:12 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 11 Sep 2017 12:21:12 +0100 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD11FDC74@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.c! orp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD7DD@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD8C3@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FDC74@ORSMSX113.amr.corp.intel.com> Message-ID: On 08/09/2017 21:41, Lu, Yingqi wrote: > > Hi Brian, > > Please find the most recent version of the patch at > http://cr.openjdk.java.net/~kkharbas/8164900/webrev.09/ > > > I've looked through the latest version. One thing that has changed is that buffer utility methods have been added to IOUtil. I assume this is a mistake and you meant to add them to Util to go with the other buffer utility methods (IOUtil is for I/O methods of course). Related is that validation of the alignment is now done twice - once in the FileChannel read/write methods and again in IOUtil. As buffers are mutable then the alignment can only be done after taking a copy? of the position + limit. Good to see that the native test has been integrated into the build. -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Mon Sep 11 11:34:10 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 11 Sep 2017 12:34:10 +0100 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD11FDC74@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.c! orp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD7DD@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD8C3@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FDC74@ORSMSX113.amr.corp.intel.com> Message-ID: On 08/09/2017 21:41, Lu, Yingqi wrote: > > Hi Brian, > > Please find the most recent version of the patch at > http://cr.openjdk.java.net/~kkharbas/8164900/webrev.09/ > > One additional comment on FileStore::getBlockSize. We haven't defined the notion of "block" anywhere so I think we need to add some wording to this method to explain that FileStore will typically organized storage into blocks. One or two sentences should be enough. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From vyom.tewari at oracle.com Mon Sep 11 15:38:39 2017 From: vyom.tewari at oracle.com (vyom tewari) Date: Mon, 11 Sep 2017 21:08:39 +0530 Subject: RFR 8145635 : Add TCP_QUICKACK socket option In-Reply-To: <56CD7C0A.2080402@oracle.com> References: <56CD74E1.60309@oracle.com> <56CD7C0A.2080402@oracle.com> Message-ID: <0b0989b5-c791-a476-a82f-6c686b19c9d3@oracle.com> Hi All, As jdk.net API already moved out of java.base, Please review the below code change for jdk10. Bug: https://bugs.openjdk.java.net/browse/JDK-8145635 Webrev: http://cr.openjdk.java.net/~vtewari/8145635/webrev0.1/index.html Thanks, Vyom On Wednesday 24 February 2016 03:16 PM, Alan Bateman wrote: > > On 24/02/2016 09:16, vyom wrote: >> Hi All, >> >> Please review my code changes for the below issue. >> >> Bug: JDK-8145635 : Add TCP_QUICKACK socket option >> >> Webrev: http://cr.openjdk.java.net/~vtewari/8145635/webrev0.0/index.html >> >> Currently TCP_QUICKACK is only supported on Linux( since Linux 2.4.4) >> so i implemented is as "ExtendedSocketOption". > > Is there any urgency on this one? I'm just wondering if we should try > to the jdk.net API moved out of java.base before we add more socket > options. This is currently tracked as JDK-8044773 and important to get > into JDK 9. > > -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Mon Sep 11 16:10:27 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Mon, 11 Sep 2017 16:10:27 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.c! orp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD7DD@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD8C3@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FDC74@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD11FFCE1@ORSMSX113.amr.corp.intel.com> Hi Alan, I will move the alignment checks to Util class. One thing that has changed is that buffer utility methods have been added to IOUtil. I assume this is a mistake and you meant to add them to Util to go with the other buffer utility methods (IOUtil is for I/O methods of course). Do you mean remove the validation from FileChannel, only do it in IOUtil? Or, you have another suggested location? Related is that validation of the alignment is now done twice - once in the FileChannel read/write methods and again in IOUtil. As buffers are mutable then the alignment can only be done after taking a copy of the position + limit. Thanks, Lucy From: Alan Bateman [mailto:Alan.Bateman at oracle.com] Sent: Monday, September 11, 2017 4:21 AM To: Lu, Yingqi Cc: nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 On 08/09/2017 21:41, Lu, Yingqi wrote: Hi Brian, Please find the most recent version of the patch at http://cr.openjdk.java.net/~kkharbas/8164900/webrev.09/ I've looked through the latest version. One thing that has changed is that buffer utility methods have been added to IOUtil. I assume this is a mistake and you meant to add them to Util to go with the other buffer utility methods (IOUtil is for I/O methods of course). Related is that validation of the alignment is now done twice - once in the FileChannel read/write methods and again in IOUtil. As buffers are mutable then the alignment can only be done after taking a copy of the position + limit. Good to see that the native test has been integrated into the build. -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Mon Sep 11 18:16:34 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Mon, 11 Sep 2017 18:16:34 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.c! orp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD7DD@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD8C3@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FDC74@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> Hi Alan/Brian, Here is a draft of the description. Please let me know if this is sufficient. /** * FileStore typically organizes storage into number of blocks. A block is * the smallest storage unit of the file store. Every read and write * operations are done on a multiple of blocks. * * @implSpec The implementation in this class throws an * {@code UnsupportedOperationException}. * * @return the block size of this file store, in bytes * * @throws IOException * if an I/O error occurs * * @throws UnsupportedOperationException * if operation is not supported * * @since 10 */ Thanks, Lucy From: Alan Bateman [mailto:Alan.Bateman at oracle.com] Sent: Monday, September 11, 2017 4:34 AM To: Lu, Yingqi ; Brian Burkhalter Cc: nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 On 08/09/2017 21:41, Lu, Yingqi wrote: Hi Brian, Please find the most recent version of the patch at http://cr.openjdk.java.net/~kkharbas/8164900/webrev.09/ One additional comment on FileStore::getBlockSize. We haven't defined the notion of "block" anywhere so I think we need to add some wording to this method to explain that FileStore will typically organized storage into blocks. One or two sentences should be enough. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Mon Sep 11 19:25:39 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 11 Sep 2017 12:25:39 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.c! orp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD7DD@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD8C3@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FDC74@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> Message-ID: Hi Lucy, Here is a slightly reworded version of your draft. Thanks, Brian /** * File storage is typically organized into discrete sequences of bytes called * blocks. A block is the smallest storage unit of a file store. Each read * and write operation is performed on a multiple of blocks. * * @implSpec The implementation in this class throws an * {@code UnsupportedOperationException}. * * @return the block size of this file store, in bytes * * @throws IOException * if an I/O error occurs * * @throws UnsupportedOperationException * if the operation is not supported * * @since 10 */ On Sep 11, 2017, at 11:16 AM, Lu, Yingqi wrote: > Here is a draft of the description. Please let me know if this is sufficient. > > /** > * FileStore typically organizes storage into number of blocks. A block is > * the smallest storage unit of the file store. Every read and write > * operations are done on a multiple of blocks. > * > * @implSpec The implementation in this class throws an > * {@code UnsupportedOperationException}. > * > * @return the block size of this file store, in bytes > * > * @throws IOException > * if an I/O error occurs > * > * @throws UnsupportedOperationException > * if operation is not supported > * > * @since 10 > */ -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Mon Sep 11 20:13:33 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Mon, 11 Sep 2017 20:13:33 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.c! orp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD7DD@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD8C3@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FDC74@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> Hi Brian, Thank you for the feedback. I will incorporate it into the next version of the webrev. Right now, we still have one open issue which was from Alan's previous email regarding to the 2-times check on alignment. Once we resolve that, I will submit webrev version 10 for review. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Monday, September 11, 2017 12:26 PM To: Lu, Yingqi Cc: Alan Bateman ; nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, Here is a slightly reworded version of your draft. Thanks, Brian /** * File storage is typically organized into discrete sequences of bytes called * blocks. A block is the smallest storage unit of a file store. Each read * and write operation is performed on a multiple of blocks. * * @implSpec The implementation in this class throws an * {@code UnsupportedOperationException}. * * @return the block size of this file store, in bytes * * @throws IOException * if an I/O error occurs * * @throws UnsupportedOperationException * if the operation is not supported * * @since 10 */ On Sep 11, 2017, at 11:16 AM, Lu, Yingqi > wrote: Here is a draft of the description. Please let me know if this is sufficient. /** * FileStore typically organizes storage into number of blocks. A block is * the smallest storage unit of the file store. Every read and write * operations are done on a multiple of blocks. * * @implSpec The implementation in this class throws an * {@code UnsupportedOperationException}. * * @return the block size of this file store, in bytes * * @throws IOException * if an I/O error occurs * * @throws UnsupportedOperationException * if operation is not supported * * @since 10 */ -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Tue Sep 12 22:58:03 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Tue, 12 Sep 2017 22:58:03 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.c! orp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD7DD@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD8C3@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FDC74@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> Hi Brian/Alan, Here is the most recent version of the webrev: http://cr.openjdk.java.net/~kkharbas/8164900/webrev.10/ In this version, two issues have been addressed: 1. Add description on the notion "block" in FileStore:getBlockSize 2. Move alignment check from IOUtil to Util There is one remaining issue: we check alignment on buffer size and channel position inside both FileChannelImpl and IOUtil. I looked at the code and think we might be able to remove the checks inside readFromNativeBuffer and writeFromNativeBuffer. However, I want to double check with you on your feedback. Please let me know. Thanks, Lucy From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of Lu, Yingqi Sent: Monday, September 11, 2017 1:14 PM To: Brian Burkhalter Cc: nio-dev at openjdk.java.net Subject: RE: Proposal for adding O_DIRECT support into JDK 9 Hi Brian, Thank you for the feedback. I will incorporate it into the next version of the webrev. Right now, we still have one open issue which was from Alan's previous email regarding to the 2-times check on alignment. Once we resolve that, I will submit webrev version 10 for review. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Monday, September 11, 2017 12:26 PM To: Lu, Yingqi > Cc: Alan Bateman >; nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, Here is a slightly reworded version of your draft. Thanks, Brian /** * File storage is typically organized into discrete sequences of bytes called * blocks. A block is the smallest storage unit of a file store. Each read * and write operation is performed on a multiple of blocks. * * @implSpec The implementation in this class throws an * {@code UnsupportedOperationException}. * * @return the block size of this file store, in bytes * * @throws IOException * if an I/O error occurs * * @throws UnsupportedOperationException * if the operation is not supported * * @since 10 */ On Sep 11, 2017, at 11:16 AM, Lu, Yingqi > wrote: Here is a draft of the description. Please let me know if this is sufficient. /** * FileStore typically organizes storage into number of blocks. A block is * the smallest storage unit of the file store. Every read and write * operations are done on a multiple of blocks. * * @implSpec The implementation in this class throws an * {@code UnsupportedOperationException}. * * @return the block size of this file store, in bytes * * @throws IOException * if an I/O error occurs * * @throws UnsupportedOperationException * if operation is not supported * * @since 10 */ -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Wed Sep 13 01:03:40 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 12 Sep 2017 18:03:40 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.c! orp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD7DD@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD8C3@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FDC74@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> Message-ID: <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> Hi Lucy, On Sep 12, 2017, at 3:58 PM, Lu, Yingqi wrote: > http://cr.openjdk.java.net/~kkharbas/8164900/webrev.10/ > > In this version, two issues have been addressed: > > 1. Add description on the notion ?block? in FileStore:getBlockSize FileStore.java 116: ?isthe smallest stoage? -> ?is the smallest storage? 128: ?if operation? -> ?if the operation? > 2. Move alignment check from IOUtil to Util > > There is one remaining issue: we check alignment on buffer size and channel position inside both FileChannelImpl and IOUtil. I looked at the code and think we might be able to remove the checks inside readFromNativeBuffer readIntoNativeBuffer() > and writeFromNativeBuffer. It does look as if either the FileChannelImpl or the IOUtil checks could be removed. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Wed Sep 13 16:52:39 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Wed, 13 Sep 2017 16:52:39 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.c! orp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD7DD@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD8C3@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FDC74@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> Hi Brian, Thank you for your feedback! Please find the most recent version of the patch is available at http://cr.openjdk.java.net/~kkharbas/8164900/webrev.11/ In this version, I fixed the typos in the FileStore:getBlockSize. I also removed alignment checks with checkBufferPositionAligned and checkRemainingBufferSizeAligned from FileChannelImpl (only have them in IOUtil), but still keep checkChannelPositionAligned there. This way, I think the duplicated checks are removed. Please let me know if I miss anything. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Tuesday, September 12, 2017 6:04 PM To: Lu, Yingqi Cc: nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, On Sep 12, 2017, at 3:58 PM, Lu, Yingqi > wrote: http://cr.openjdk.java.net/~kkharbas/8164900/webrev.10/ In this version, two issues have been addressed: 1. Add description on the notion "block" in FileStore:getBlockSize FileStore.java 116: "isthe smallest stoage" -> "is the smallest storage" 128: "if operation" -> "if the operation" 2. Move alignment check from IOUtil to Util There is one remaining issue: we check alignment on buffer size and channel position inside both FileChannelImpl and IOUtil. I looked at the code and think we might be able to remove the checks inside readFromNativeBuffer readIntoNativeBuffer() and writeFromNativeBuffer. It does look as if either the FileChannelImpl or the IOUtil checks could be removed. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Wed Sep 13 23:20:57 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 13 Sep 2017 16:20:57 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.c! orp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD7DD@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD8C3@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FDC74@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> Message-ID: <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> Hi Lucy, On Sep 13, 2017, at 9:52 AM, Lu, Yingqi wrote: > Please find the most recent version of the patch is available athttp://cr.openjdk.java.net/~kkharbas/8164900/webrev.11/ GensrcMisc.gmk and UnixConstants.java.template 2: Copyright year 2016 -> 2017 jtregNative.gmk 66: Is the ?-lc? necessary? FileStore.java 116: ?stoage? -> ?storage? (missing an ?r?) > In this version, I fixed the typos in the FileStore:getBlockSize. I also removed alignment checks with checkBufferPositionAligned and checkRemainingBufferSizeAligned from FileChannelImpl (only have them in IOUtil), but still keep checkChannelPositionAligned there. This way, I think the duplicated checks are removed. Please let me know if I miss anything. I don?t see any obvious problems. I have not been able to test it today due to some problem with my connection; will try again tomorrow. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Wed Sep 13 23:50:01 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Wed, 13 Sep 2017 23:50:01 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD11F4F76@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11F8C11@ORSMSX113.amr.corp.intel.com> <5E0715EF-7354-4DAC-9C44-03A8EE55F79A@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FA3C5@ORSMSX113.amr.c! orp.intel.com> <400F298E-8C8C-4E92-8E0A-F79664B66151@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FB31A@ORSMSX113.amr.corp.intel.com> <89838B08-3C7D-46C6-9DB0-72B1D2BE7C88@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD7DD@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD8C3@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FDC74@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> Hi Brian, Thank you for the feedback. Here is a newer version of the patch http://cr.openjdk.java.net/~kkharbas/8164900/webrev.12/ Please let me know if you find anything unexpected from the testing. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Wednesday, September 13, 2017 4:21 PM To: Lu, Yingqi Cc: nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, On Sep 13, 2017, at 9:52 AM, Lu, Yingqi > wrote: Please find the most recent version of the patch is available athttp://cr.openjdk.java.net/~kkharbas/8164900/webrev.11/ GensrcMisc.gmk and UnixConstants.java.template 2: Copyright year 2016 -> 2017 jtregNative.gmk 66: Is the "-lc" necessary? FileStore.java 116: "stoage" -> "storage" (missing an "r") In this version, I fixed the typos in the FileStore:getBlockSize. I also removed alignment checks with checkBufferPositionAligned and checkRemainingBufferSizeAligned from FileChannelImpl (only have them in IOUtil), but still keep checkChannelPositionAligned there. This way, I think the duplicated checks are removed. Please let me know if I miss anything. I don't see any obvious problems. I have not been able to test it today due to some problem with my connection; will try again tomorrow. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Sep 14 12:25:06 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 14 Sep 2017 13:25:06 +0100 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD11FD7DD@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD8C3@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FDC74@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> Message-ID: <5b6d7a90-5564-d24a-aa67-9c7a47b7e577@oracle.com> On 14/09/2017 00:50, Lu, Yingqi wrote: > > Hi Brian, > > Thank you for the feedback. Here is a newer version of the patch > http://cr.openjdk.java.net/~kkharbas/8164900/webrev.12/ > > Just catching up on this again. The check for alignment of the position and size of the remaining in readInto/writeFrom{NativeBuffer} looks right now. There is still a duplicate check in FileChannelImpl.read/write that I assume can be removed (it is easily defeated). I assume the protected modifier can be dropped from the new check methods in Util. It looks like the first sentence in the FileStore::getBlockSize javadoc has been dropped in this version. A sentence to say that to returns the block size would be good to have as the first sentence. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Thu Sep 14 16:39:42 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Thu, 14 Sep 2017 16:39:42 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <5b6d7a90-5564-d24a-aa67-9c7a47b7e577@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD11FD7DD@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD8C3@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FDC74@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9c7a47b7e577@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> Hi Alan, The remaining check in FileChannelImpl is to check if channel/file position is aligned. I think we still need the check, but we can move it to IOUtil class. Please let me know if there is anything I misunderstand. I will submit version 13 today to incorporate your feedback. Thanks, Lucy From: Alan Bateman [mailto:Alan.Bateman at oracle.com] Sent: Thursday, September 14, 2017 5:25 AM To: Lu, Yingqi ; Brian Burkhalter Cc: nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 On 14/09/2017 00:50, Lu, Yingqi wrote: Hi Brian, Thank you for the feedback. Here is a newer version of the patch http://cr.openjdk.java.net/~kkharbas/8164900/webrev.12/ Just catching up on this again. The check for alignment of the position and size of the remaining in readInto/writeFrom{NativeBuffer} looks right now. There is still a duplicate check in FileChannelImpl.read/write that I assume can be removed (it is easily defeated). I assume the protected modifier can be dropped from the new check methods in Util. It looks like the first sentence in the FileStore::getBlockSize javadoc has been dropped in this version. A sentence to say that to returns the block size would be good to have as the first sentence. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Thu Sep 14 17:20:30 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Thu, 14 Sep 2017 17:20:30 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD11FD7DD@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD8C3@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FDC74@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9c7a47b7e577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> Hi Brian/Alan, After taking another look at the code, I could not find an easy way to check the channel/file position inside IOUtil class for non-positional read/write (position is not provided as a parameter to FileChannel.read/write). Do you think it is proper to just leave the channel position check inside FileChannelImpl like it is now? Thanks, Lucy From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of Lu, Yingqi Sent: Thursday, September 14, 2017 9:40 AM To: Alan Bateman ; Brian Burkhalter Cc: nio-dev at openjdk.java.net Subject: RE: Proposal for adding O_DIRECT support into JDK 9 Hi Alan, The remaining check in FileChannelImpl is to check if channel/file position is aligned. I think we still need the check, but we can move it to IOUtil class. Please let me know if there is anything I misunderstand. I will submit version 13 today to incorporate your feedback. Thanks, Lucy From: Alan Bateman [mailto:Alan.Bateman at oracle.com] Sent: Thursday, September 14, 2017 5:25 AM To: Lu, Yingqi >; Brian Burkhalter > Cc: nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 On 14/09/2017 00:50, Lu, Yingqi wrote: Hi Brian, Thank you for the feedback. Here is a newer version of the patch http://cr.openjdk.java.net/~kkharbas/8164900/webrev.12/ Just catching up on this again. The check for alignment of the position and size of the remaining in readInto/writeFrom{NativeBuffer} looks right now. There is still a duplicate check in FileChannelImpl.read/write that I assume can be removed (it is easily defeated). I assume the protected modifier can be dropped from the new check methods in Util. It looks like the first sentence in the FileStore::getBlockSize javadoc has been dropped in this version. A sentence to say that to returns the block size would be good to have as the first sentence. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Thu Sep 14 20:17:09 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 14 Sep 2017 13:17:09 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD11FD7DD@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD8C3@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FDC74@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9! c7a47b7e577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> Message-ID: Hi Lucy, It would be awkward to add another parameter to the IOUtil methods just to be able to move this check down a level. I do not see anything wrong with leaving it as is. Thanks, Brian On Sep 14, 2017, at 10:20 AM, Lu, Yingqi wrote: > After taking another look at the code, I could not find an easy way to check the channel/file position inside IOUtil class for non-positional read/write (position is not provided as a parameter to FileChannel.read/write). > > Do you think it is proper to just leave the channel position check inside FileChannelImpl like it is now? -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Thu Sep 14 22:28:42 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Thu, 14 Sep 2017 22:28:42 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AD11FD7DD@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD8C3@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FDC74@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9! c7a47b7e577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> Hi Brian and Alan, Please find the version 13 of the patch is available at http://cr.openjdk.java.net/~kkharbas/8164900/webrev.13/ In this version, I have addressed Alan's most recent feedback. Please let me know if you have any further feedback and comments. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Thursday, September 14, 2017 1:17 PM To: Lu, Yingqi Cc: Alan Bateman ; nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, It would be awkward to add another parameter to the IOUtil methods just to be able to move this check down a level. I do not see anything wrong with leaving it as is. Thanks, Brian On Sep 14, 2017, at 10:20 AM, Lu, Yingqi > wrote: After taking another look at the code, I could not find an easy way to check the channel/file position inside IOUtil class for non-positional read/write (position is not provided as a parameter to FileChannel.read/write). Do you think it is proper to just leave the channel position check inside FileChannelImpl like it is now? -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Fri Sep 15 11:53:06 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 15 Sep 2017 12:53:06 +0100 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9! c7a47b7e577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> Message-ID: <1422ea82-d6ae-a5e2-2d06-5bfb35eb1557@oracle.com> On 14/09/2017 23:28, Lu, Yingqi wrote: > > Hi Brian and Alan, > > Please find the version 13 of the patch is available at > http://cr.openjdk.java.net/~kkharbas/8164900/webrev.13/ > > > In this version, I have addressed Alan?s most recent feedback. > > I think we are close but I'm still concerned about the position check in FileChannel.read/write. This has to be done while holding positionLock, otherwise there is no guarantee that the position hasn't changed (I wasn't clear in my mail yesterday). One other issue is in IOUtil.read where the alignment check is done on dst.remaining(), it should be "rem". The other checks look correct. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From volker.simonis at gmail.com Fri Sep 15 12:29:19 2017 From: volker.simonis at gmail.com (Volker Simonis) Date: Fri, 15 Sep 2017 14:29:19 +0200 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD11FD7DD@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD8C3@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FDC74@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> Message-ID: Hi Lucy, I just wanted to report that your last version (i.e. webrev.13) compiles fine on AIX and passes all the new directio tests. Thanks, Volker On Fri, Sep 15, 2017 at 12:28 AM, Lu, Yingqi wrote: > Hi Brian and Alan, > > > > Please find the version 13 of the patch is available at > http://cr.openjdk.java.net/~kkharbas/8164900/webrev.13/ > > > > In this version, I have addressed Alan?s most recent feedback. > > > > Please let me know if you have any further feedback and comments. > > > > Thanks, > > Lucy > > > > From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] > Sent: Thursday, September 14, 2017 1:17 PM > To: Lu, Yingqi > Cc: Alan Bateman ; nio-dev at openjdk.java.net > Subject: Re: Proposal for adding O_DIRECT support into JDK 9 > > > > Hi Lucy, > > > > It would be awkward to add another parameter to the IOUtil methods just to > be able to move this check down a level. I do not see anything wrong with > leaving it as is. > > > > Thanks, > > > > Brian > > > > On Sep 14, 2017, at 10:20 AM, Lu, Yingqi wrote: > > > > After taking another look at the code, I could not find an easy way to check > the channel/file position inside IOUtil class for non-positional read/write > (position is not provided as a parameter to FileChannel.read/write). > > > > Do you think it is proper to just leave the channel position check inside > FileChannelImpl like it is now? > > From yingqi.lu at intel.com Fri Sep 15 15:50:23 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Fri, 15 Sep 2017 15:50:23 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AD11FD7DD@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FD8C3@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD11FDC74@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD1207708@ORSMSX113.amr.corp.intel.com> Hi Volker, Thank you very much for testing the patch. Really appreciate your help :) Thanks, Lucy >-----Original Message----- >From: Volker Simonis [mailto:volker.simonis at gmail.com] >Sent: Friday, September 15, 2017 5:29 AM >To: Lu, Yingqi >Cc: Brian Burkhalter ; nio-dev at openjdk.java.net >Subject: Re: Proposal for adding O_DIRECT support into JDK 9 > >Hi Lucy, > >I just wanted to report that your last version (i.e. webrev.13) compiles fine on AIX >and passes all the new directio tests. > >Thanks, >Volker > > >On Fri, Sep 15, 2017 at 12:28 AM, Lu, Yingqi wrote: >> Hi Brian and Alan, >> >> >> >> Please find the version 13 of the patch is available at >> http://cr.openjdk.java.net/~kkharbas/8164900/webrev.13/ >> >> >> >> In this version, I have addressed Alan?s most recent feedback. >> >> >> >> Please let me know if you have any further feedback and comments. >> >> >> >> Thanks, >> >> Lucy >> >> >> >> From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] >> Sent: Thursday, September 14, 2017 1:17 PM >> To: Lu, Yingqi >> Cc: Alan Bateman ; nio-dev at openjdk.java.net >> Subject: Re: Proposal for adding O_DIRECT support into JDK 9 >> >> >> >> Hi Lucy, >> >> >> >> It would be awkward to add another parameter to the IOUtil methods >> just to be able to move this check down a level. I do not see anything >> wrong with leaving it as is. >> >> >> >> Thanks, >> >> >> >> Brian >> >> >> >> On Sep 14, 2017, at 10:20 AM, Lu, Yingqi wrote: >> >> >> >> After taking another look at the code, I could not find an easy way to >> check the channel/file position inside IOUtil class for non-positional >> read/write (position is not provided as a parameter to FileChannel.read/write). >> >> >> >> Do you think it is proper to just leave the channel position check >> inside FileChannelImpl like it is now? >> >> From yingqi.lu at intel.com Fri Sep 15 18:00:51 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Fri, 15 Sep 2017 18:00:51 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <1422ea82-d6ae-a5e2-2d06-5bfb35eb1557@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9! c7a47b7e577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> <1422ea82-d6ae-a5e2-2d06-5bfb35eb1557@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD1207BDA@ORSMSX113.amr.corp.intel.com> Hi Alan and Brian, Please find version 14 of the patch is available at http://cr.openjdk.java.net/~kkharbas/8164900/webrev.14/ In this version, I did following two changes: 1. Move position alignment check for non-positional read/write inside synchronized (positionLock) block. For positional read/write (position is provided as a parameter from user), I keep the check outside the synchronized block. Since the position is an absolute number, I think it should be OK? 2. In IOUtil.read of version 13, I saw alignment check was done on "rem" where rem = dst.remaining(). However, getTemporaryAlignedDirectBuffer and getTemporaryDirectBuffer were using dst.remaining(). I am not sure if this is the place you refer to in your email. In this version, I make all three functions use rem. Please let me know if I misunderstand. Thanks, Lucy From: Alan Bateman [mailto:Alan.Bateman at oracle.com] Sent: Friday, September 15, 2017 4:53 AM To: Lu, Yingqi ; Brian Burkhalter Cc: nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 On 14/09/2017 23:28, Lu, Yingqi wrote: Hi Brian and Alan, Please find the version 13 of the patch is available at http://cr.openjdk.java.net/~kkharbas/8164900/webrev.13/ In this version, I have addressed Alan's most recent feedback. I think we are close but I'm still concerned about the position check in FileChannel.read/write. This has to be done while holding positionLock, otherwise there is no guarantee that the position hasn't changed (I wasn't clear in my mail yesterday). One other issue is in IOUtil.read where the alignment check is done on dst.remaining(), it should be "rem". The other checks look correct. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Fri Sep 15 22:18:56 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 15 Sep 2017 15:18:56 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD1207BDA@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9! c7a47b7e577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> <1422ea82-d6ae-a5e2-2d06-5bfb35eb1557@oracl! e.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1207BDA@ORSMSX113.amr.corp.intel.com> Message-ID: Hi Lucy, I?ve not re-examined the patch yet but I ran some tests. I observed some problems on my system. When I do 'make test-image-jdk-jtreg-native? in the repo root directory in my Ubuntu 16.04 64-bit VM, I get this error: In file included from /usr/include/fcntl.h:289:0, from /home/bpb/Work/jdk10-dev/jdk/test/java/nio/channels/FileChannel/directio/libDirectIO.c:28: In function ?open?, inlined from ?Java_DirectIOTest_isFileInCache0? at /home/bpb/Work/jdk10-dev/jdk/test/java/nio/channels/FileChannel/directio/libDirectIO.c:74:9: /usr/include/x86_64-linux-gnu/bits/fcntl2.h:50:4: error: call to ?__open_missing_mode? declared with attribute error: open with O_CREAT or O_TMPFILE in second argument needs 3 arguments __open_missing_mode (); On macOS native I get a failure for ReadDirect: java.lang.NumberFormatException: For input string: "aa" at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.base/java.lang.Integer.parseInt(Integer.java:652) at java.base/java.lang.Integer.parseInt(Integer.java:770) at ReadDirect.testWithArrayOfBuffer(ReadDirect.java:203) at ReadDirect.main(ReadDirect.java:262) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at com.sun.javatest.regtest.agent.MainActionHelper$SameVMRunnable.run(MainActionHelper.java:230) at java.base/java.lang.Thread.run(Thread.java:844) In my Windows 7 64-bit VM all four read/write direct I/O tests pass. The foregoing results were on my own system, as mentioned. In a job run through our regression test harness I see that the four read/write direct I/O tests are all failing on Solaris. The initTests() methods in these tests check on Solaris whether the file system type supports direct I/O and return but this verification is not propagated to the main() which continues. Probably you should instead call DirectIOTest.isDirectIOSupportedByFS() in each of these four tests and check the return value and if false the test should return as if it had succeeded. Also in the regression run it looks as if all four read/write tests passed on Windows but the harness flagged them as failed with the message ? failed to clean up files after test? but I think that this could be a problem in the harness as I do not see this in my local VM tests. In the regression run DirectIOTest fails on all the Unix platforms but this is because the nativepath is not set on jtreg. I do not as yet know how to make this work in the automatic test runs but that is not your problem. Thanks, Brian On Sep 15, 2017, at 11:00 AM, Lu, Yingqi wrote: > Please find version 14 of the patch is available athttp://cr.openjdk.java.net/~kkharbas/8164900/webrev.14/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Fri Sep 15 22:28:35 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Fri, 15 Sep 2017 22:28:35 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9! c7a47b7e577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> <1422ea82-d6ae-a5e2-2d06-5bfb35eb1557@oracl! e.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1207BDA@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD120837B@ORSMSX113.amr.corp.intel.com> Hi Brian, I will look into all the issues. Thank you very much running all these tests. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Friday, September 15, 2017 3:19 PM To: Lu, Yingqi Cc: Alan Bateman ; nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, I've not re-examined the patch yet but I ran some tests. I observed some problems on my system. When I do 'make test-image-jdk-jtreg-native' in the repo root directory in my Ubuntu 16.04 64-bit VM, I get this error: In file included from /usr/include/fcntl.h:289:0, from /home/bpb/Work/jdk10-dev/jdk/test/java/nio/channels/FileChannel/directio/libDirectIO.c:28: In function 'open', inlined from 'Java_DirectIOTest_isFileInCache0' at /home/bpb/Work/jdk10-dev/jdk/test/java/nio/channels/FileChannel/directio/libDirectIO.c:74:9: /usr/include/x86_64-linux-gnu/bits/fcntl2.h:50:4: error: call to '__open_missing_mode' declared with attribute error: open with O_CREAT or O_TMPFILE in second argument needs 3 arguments __open_missing_mode (); On macOS native I get a failure for ReadDirect: java.lang.NumberFormatException: For input string: "aa" at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.base/java.lang.Integer.parseInt(Integer.java:652) at java.base/java.lang.Integer.parseInt(Integer.java:770) at ReadDirect.testWithArrayOfBuffer(ReadDirect.java:203) at ReadDirect.main(ReadDirect.java:262) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at com.sun.javatest.regtest.agent.MainActionHelper$SameVMRunnable.run(MainActionHelper.java:230) at java.base/java.lang.Thread.run(Thread.java:844) In my Windows 7 64-bit VM all four read/write direct I/O tests pass. The foregoing results were on my own system, as mentioned. In a job run through our regression test harness I see that the four read/write direct I/O tests are all failing on Solaris. The initTests() methods in these tests check on Solaris whether the file system type supports direct I/O and return but this verification is not propagated to the main() which continues. Probably you should instead call DirectIOTest.isDirectIOSupportedByFS() in each of these four tests and check the return value and if false the test should return as if it had succeeded. Also in the regression run it looks as if all four read/write tests passed on Windows but the harness flagged them as failed with the message " failed to clean up files after test" but I think that this could be a problem in the harness as I do not see this in my local VM tests. In the regression run DirectIOTest fails on all the Unix platforms but this is because the nativepath is not set on jtreg. I do not as yet know how to make this work in the automatic test runs but that is not your problem. Thanks, Brian On Sep 15, 2017, at 11:00 AM, Lu, Yingqi > wrote: Please find version 14 of the patch is available athttp://cr.openjdk.java.net/~kkharbas/8164900/webrev.14/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Fri Sep 15 22:45:01 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Fri, 15 Sep 2017 22:45:01 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 References: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9! c7a47b7e577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> <1422ea82-d6ae-a5e2-2d06-5bfb35eb1557@oracl! e.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1207BDA@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD12083EB@ORSMSX113.amr.corp.intel.com> Hi Brian, For the first issue, I do not see it on my CentOS box. However, I see other people discuss the issue online. Could you please replace the line 78 in jdk/test/java/nio/channels/FileChannel/directio/libDirectIO.c with the following line and give it a try on your Ubuntu box? I tested it on my CentOS system, it does not complain. int fd = open(path, 'r', 0600); At the same time, I am working on the rest of the issues. Thanks, Lucy From: Lu, Yingqi Sent: Friday, September 15, 2017 3:29 PM To: 'Brian Burkhalter' Cc: Alan Bateman ; nio-dev at openjdk.java.net Subject: RE: Proposal for adding O_DIRECT support into JDK 9 Hi Brian, I will look into all the issues. Thank you very much running all these tests. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Friday, September 15, 2017 3:19 PM To: Lu, Yingqi > Cc: Alan Bateman >; nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, I've not re-examined the patch yet but I ran some tests. I observed some problems on my system. When I do 'make test-image-jdk-jtreg-native' in the repo root directory in my Ubuntu 16.04 64-bit VM, I get this error: In file included from /usr/include/fcntl.h:289:0, from /home/bpb/Work/jdk10-dev/jdk/test/java/nio/channels/FileChannel/directio/libDirectIO.c:28: In function 'open', inlined from 'Java_DirectIOTest_isFileInCache0' at /home/bpb/Work/jdk10-dev/jdk/test/java/nio/channels/FileChannel/directio/libDirectIO.c:74:9: /usr/include/x86_64-linux-gnu/bits/fcntl2.h:50:4: error: call to '__open_missing_mode' declared with attribute error: open with O_CREAT or O_TMPFILE in second argument needs 3 arguments __open_missing_mode (); On macOS native I get a failure for ReadDirect: java.lang.NumberFormatException: For input string: "aa" at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.base/java.lang.Integer.parseInt(Integer.java:652) at java.base/java.lang.Integer.parseInt(Integer.java:770) at ReadDirect.testWithArrayOfBuffer(ReadDirect.java:203) at ReadDirect.main(ReadDirect.java:262) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at com.sun.javatest.regtest.agent.MainActionHelper$SameVMRunnable.run(MainActionHelper.java:230) at java.base/java.lang.Thread.run(Thread.java:844) In my Windows 7 64-bit VM all four read/write direct I/O tests pass. The foregoing results were on my own system, as mentioned. In a job run through our regression test harness I see that the four read/write direct I/O tests are all failing on Solaris. The initTests() methods in these tests check on Solaris whether the file system type supports direct I/O and return but this verification is not propagated to the main() which continues. Probably you should instead call DirectIOTest.isDirectIOSupportedByFS() in each of these four tests and check the return value and if false the test should return as if it had succeeded. Also in the regression run it looks as if all four read/write tests passed on Windows but the harness flagged them as failed with the message " failed to clean up files after test" but I think that this could be a problem in the harness as I do not see this in my local VM tests. In the regression run DirectIOTest fails on all the Unix platforms but this is because the nativepath is not set on jtreg. I do not as yet know how to make this work in the automatic test runs but that is not your problem. Thanks, Brian On Sep 15, 2017, at 11:00 AM, Lu, Yingqi > wrote: Please find version 14 of the patch is available athttp://cr.openjdk.java.net/~kkharbas/8164900/webrev.14/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Fri Sep 15 22:53:10 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 15 Sep 2017 15:53:10 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD12083EB@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9! c7a47b7e577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> <1422ea82-d6ae-a5e2-2d06-5bfb35eb1557@oracl! ! e.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1207BDA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12083EB@ORSMSX113.amr.corp.intel.com> Message-ID: <8E172C93-DC2C-4E7D-BDA0-EA0E51825348@oracle.com> Hi Lucy, This change works: --- a/test/java/nio/channels/FileChannel/directio/libDirectIO.c +++ b/test/java/nio/channels/FileChannel/directio/libDirectIO.c @@ -71,7 +71,7 @@ const char* path = (*env)->GetStringUTFChars(env, file_path, JNI_FALSE); - int fd = open(path, 'r'); + int fd = open(path, 'r', 0600); (*env)->ReleaseStringUTFChars(env, file_path, path); All five tests pass in my Ubuntu 16.04 64-bit VM. On my macOS system I also observed this failure: java.lang.RuntimeException: DirectIO is not working properly with write. File still exists in cache! at DirectIOTest.main(DirectIOTest.java:117) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at com.sun.javatest.regtest.agent.MainActionHelper$SameVMRunnable.run(MainActionHelper.java:230) at java.base/java.lang.Thread.run(Thread.java:844) Neither this nor the other failure on macOS occurs every time, i.e., they appear to be intermittent. Thanks, Brian On Sep 15, 2017, at 3:45 PM, Lu, Yingqi wrote: > Could you please replace the line 78 in jdk/test/java/nio/channels/FileChannel/directio/libDirectIO.c with the following line and give it a try on your Ubuntu box? I tested it on my CentOS system, it does not complain. > > int fd = open(path, 'r', 0600); -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Fri Sep 15 23:01:36 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Fri, 15 Sep 2017 23:01:36 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <8E172C93-DC2C-4E7D-BDA0-EA0E51825348@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9! c7a47b7e577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> <1422ea82-d6ae-a5e2-2d06-5bfb35eb1557@oracl! ! e.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1207BDA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12083EB@ORSMSX113.amr.corp.intel.com> <8E172C93-DC2C-4E7D-BDA0-EA0E51825348@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD12084E8@ORSMSX113.amr.corp.intel.com> Hmm, let me take a look into this MacOS issue. I am wondering if we will need to do "purge" before the testing. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Friday, September 15, 2017 3:53 PM To: Lu, Yingqi Cc: Alan Bateman ; nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, This change works: --- a/test/java/nio/channels/FileChannel/directio/libDirectIO.c +++ b/test/java/nio/channels/FileChannel/directio/libDirectIO.c @@ -71,7 +71,7 @@ const char* path = (*env)->GetStringUTFChars(env, file_path, JNI_FALSE); - int fd = open(path, 'r'); + int fd = open(path, 'r', 0600); (*env)->ReleaseStringUTFChars(env, file_path, path); All five tests pass in my Ubuntu 16.04 64-bit VM. On my macOS system I also observed this failure: java.lang.RuntimeException: DirectIO is not working properly with write. File still exists in cache! at DirectIOTest.main(DirectIOTest.java:117) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at com.sun.javatest.regtest.agent.MainActionHelper$SameVMRunnable.run(MainActionHelper.java:230) at java.base/java.lang.Thread.run(Thread.java:844) Neither this nor the other failure on macOS occurs every time, i.e., they appear to be intermittent. Thanks, Brian On Sep 15, 2017, at 3:45 PM, Lu, Yingqi > wrote: Could you please replace the line 78 in jdk/test/java/nio/channels/FileChannel/directio/libDirectIO.c with the following line and give it a try on your Ubuntu box? I tested it on my CentOS system, it does not complain. int fd = open(path, 'r', 0600); -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Sat Sep 16 17:20:57 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Sat, 16 Sep 2017 17:20:57 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD12084E8@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9! c7a47b7e577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> <1422ea82-d6ae-a5e2-2d06-5bfb35eb1557@oracl! ! e.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1207BDA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12083EB@ORSMSX113.amr.corp.intel.com> <8E172C93-DC2C-4E7D-BDA0-EA0E51825348@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12084E8@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD1208BE3@ORSMSX113.amr.corp.intel.com> Hi Brian, I tested the current DirectIOTest on my Linux CentOS system for 200,000 iterations, they all passed. I tested the same test on MacOS VM for 20 iteration, some of them failed with the "file in cache" error while the rest passed. I searched online, and found this thread https://lists.apple.com/archives/filesystem-dev/2007/Sep/msg00012.html. If F_NOCACHE flag is set, an IO block might exist in the file system cache for a very short amount of time and then will be evicted. Then, I tried to add a 200 milliseconds delay between testWrite() and isFileInCache() as well as testRead() and isFileInCache(). I again did 100,000 iterations of DirectIOTest, they all passed. 200,000 iterations take too much time on my VM which is hosted on a tiny desktop :) I will test DirectIOTest on Solaris VM as well for 100,000 iterations and see how it behaves with and without the delay. My thinking is to add the delay for the platforms who need it. By the way, I already fixed the DirectIOTest.isDirectIOSupportedByFS() issue on Solaris. I will submit another version of the patch by Monday. Thanks, Lucy From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of Lu, Yingqi Sent: Friday, September 15, 2017 4:02 PM To: Brian Burkhalter Cc: nio-dev at openjdk.java.net Subject: RE: Proposal for adding O_DIRECT support into JDK 9 Hmm, let me take a look into this MacOS issue. I am wondering if we will need to do "purge" before the testing. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Friday, September 15, 2017 3:53 PM To: Lu, Yingqi > Cc: Alan Bateman >; nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, This change works: --- a/test/java/nio/channels/FileChannel/directio/libDirectIO.c +++ b/test/java/nio/channels/FileChannel/directio/libDirectIO.c @@ -71,7 +71,7 @@ const char* path = (*env)->GetStringUTFChars(env, file_path, JNI_FALSE); - int fd = open(path, 'r'); + int fd = open(path, 'r', 0600); (*env)->ReleaseStringUTFChars(env, file_path, path); All five tests pass in my Ubuntu 16.04 64-bit VM. On my macOS system I also observed this failure: java.lang.RuntimeException: DirectIO is not working properly with write. File still exists in cache! at DirectIOTest.main(DirectIOTest.java:117) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at com.sun.javatest.regtest.agent.MainActionHelper$SameVMRunnable.run(MainActionHelper.java:230) at java.base/java.lang.Thread.run(Thread.java:844) Neither this nor the other failure on macOS occurs every time, i.e., they appear to be intermittent. Thanks, Brian On Sep 15, 2017, at 3:45 PM, Lu, Yingqi > wrote: Could you please replace the line 78 in jdk/test/java/nio/channels/FileChannel/directio/libDirectIO.c with the following line and give it a try on your Ubuntu box? I tested it on my CentOS system, it does not complain. int fd = open(path, 'r', 0600); -------------- next part -------------- An HTML attachment was scrubbed... URL: From volker.simonis at gmail.com Mon Sep 18 08:28:02 2017 From: volker.simonis at gmail.com (Volker Simonis) Date: Mon, 18 Sep 2017 10:28:02 +0200 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <8E172C93-DC2C-4E7D-BDA0-EA0E51825348@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1207BDA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12083EB@ORSMSX113.amr.corp.intel.com> <8E172C93-DC2C-4E7D-BDA0-EA0E51825348@oracle.com> Message-ID: Sorry if I'm missing something here, but I think "open()" has the following signature: int open(const char *pathname, int flags); int open(const char *pathname, int flags, mode_t mode); So I don't quite understand what: open(path, 'r') should stand for. Maybe you've mixed it up with "fopen()" FILE *fopen(const char *path, const char *mode); which takes a character as second argument? "open()" simply takes one of the following constants O_RDONLY, O_WRONLY, or O_RDWR as 'flags' argument. If you use that you shouldn't need the third 'mode' argument. And if you want to use a 'mode' argument, please use one of the predefined constants instead of a plain number. Regards, Volker On Sat, Sep 16, 2017 at 12:53 AM, Brian Burkhalter wrote: > Hi Lucy, > > This change works: > > --- a/test/java/nio/channels/FileChannel/directio/libDirectIO.c > +++ b/test/java/nio/channels/FileChannel/directio/libDirectIO.c > @@ -71,7 +71,7 @@ > > const char* path = (*env)->GetStringUTFChars(env, file_path, > JNI_FALSE); > > - int fd = open(path, 'r'); > + int fd = open(path, 'r', 0600); > > (*env)->ReleaseStringUTFChars(env, file_path, path); > > All five tests pass in my Ubuntu 16.04 64-bit VM. > > On my macOS system I also observed this failure: > > java.lang.RuntimeException: DirectIO is not working properly with write. > File still exists in cache! > at DirectIOTest.main(DirectIOTest.java:117) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:564) > at > com.sun.javatest.regtest.agent.MainActionHelper$SameVMRunnable.run(MainActionHelper.java:230) > at java.base/java.lang.Thread.run(Thread.java:844) > > Neither this nor the other failure on macOS occurs every time, i.e., they > appear to be intermittent. > > Thanks, > > Brian > > On Sep 15, 2017, at 3:45 PM, Lu, Yingqi wrote: > > Could you please replace the line 78 in > jdk/test/java/nio/channels/FileChannel/directio/libDirectIO.c with the > following line and give it a try on your Ubuntu box? I tested it on my > CentOS system, it does not complain. > > int fd = open(path, 'r', 0600); > > From brian.burkhalter at oracle.com Mon Sep 18 19:02:04 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 18 Sep 2017 12:02:04 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD1208BE3@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9! c7a47b7e577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> <1422ea82-d6ae-a5e2-2d06-5bfb35eb1557@oracl! ! ! e.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1207BDA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12083EB@ORSMSX113.amr.corp.intel.com> <8E172C93-DC2C-4E7D-BDA0-EA0E51825348@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12084E8@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1208BE3@ORSMSX113.amr.corp.intel.com> Message-ID: <9D28C97D-652B-457D-9E97-C052CFB94843@oracle.com> Hi Lucy, On Sep 16, 2017, at 10:20 AM, Lu, Yingqi wrote: > I tested the current DirectIOTest on my Linux CentOS system for 200,000 iterations, they all passed. I tested the same test on MacOS VM for 20 iteration, some of them failed with the ?file in cache? error while the rest passed. > > I searched online, and found this thread https://lists.apple.com/archives/filesystem-dev/2007/Sep/msg00012.html. If F_NOCACHE flag is set, an IO block might exist in the file system cache for a very short amount of time and then will be evicted. > > Then, I tried to add a 200 milliseconds delay between testWrite() and isFileInCache() as well as testRead() and isFileInCache(). I again did 100,000 iterations of DirectIOTest, they all passed. 200,000 iterations take too much time on my VM which is hosted on a tiny desktop J > > I will test DirectIOTest on Solaris VM as well for 100,000 iterations and see how it behaves with and without the delay. My thinking is to add the delay for the platforms who need it. We can try that but often having a sleep call eventually results in the test?s intermittently failing nonetheless. If that situation is observed then eventually we might just restrict this test further to the platforms on which it passes consistently. As Linux is the real target here that would not be tragic but it would be better to have it work on the other supported Unix flavors if possible. > By the way, I already fixed the DirectIOTest.isDirectIOSupportedByFS() issue on Solaris. I will submit another version of the patch by Monday. OK, good. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Mon Sep 18 19:03:17 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 18 Sep 2017 12:03:17 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1207BDA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29! CF9AD12083EB@ORSMSX113.amr.corp.intel.com> <8E172C93-DC2C-4E7D-BDA0-EA0E51825348@oracle.com> Message-ID: <964E56C2-7D30-41E1-B545-759DA26B87F4@oracle.com> Hi Volker, Thanks for pointing this out. On Sep 18, 2017, at 1:28 AM, Volker Simonis wrote: > "open()" simply takes one of the following constants O_RDONLY, > O_WRONLY, or O_RDWR as 'flags' argument. If you use that you shouldn't > need the third 'mode' argument. And if you want to use a 'mode' > argument, please use one of the predefined constants instead of a > plain number. Yes, definitely no hard coding of the mode argument. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Mon Sep 18 19:27:41 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Mon, 18 Sep 2017 19:27:41 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9D28C97D-652B-457D-9E97-C052CFB94843@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9! c7a47b7e577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> <1422ea82-d6ae-a5e2-2d06-5bfb35eb1557@oracl! ! ! e.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1207BDA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12083EB@ORSMSX113.amr.corp.intel.com> <8E172C93-DC2C-4E7D-BDA0-EA0E51825348@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12084E8@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1208BE3@ORSMSX113.amr.corp.intel.com> <9D28C97D-652B-457D-9E97-C052CFB94843@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD120A6FC@ORSMSX113.amr.corp.intel.com> Hi Brian, I will remove MacOS from this test for now. Could you please help test DirectIOTest on Solaris for many iterations? My VM crashed and it will take me some time to recover. Hi Volker, Thank you for pointing out the issue with open() call. I will take a look and modify accordingly. At the meantime, it would be very helpful if you can please help test DirectIOTest.java on AIX for a big number of iterations as well. I do not have the test environment at all. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Monday, September 18, 2017 12:02 PM To: Lu, Yingqi Cc: nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, On Sep 16, 2017, at 10:20 AM, Lu, Yingqi > wrote: I tested the current DirectIOTest on my Linux CentOS system for 200,000 iterations, they all passed. I tested the same test on MacOS VM for 20 iteration, some of them failed with the "file in cache" error while the rest passed. I searched online, and found this thread https://lists.apple.com/archives/filesystem-dev/2007/Sep/msg00012.html. If F_NOCACHE flag is set, an IO block might exist in the file system cache for a very short amount of time and then will be evicted. Then, I tried to add a 200 milliseconds delay between testWrite() and isFileInCache() as well as testRead() and isFileInCache(). I again did 100,000 iterations of DirectIOTest, they all passed. 200,000 iterations take too much time on my VM which is hosted on a tiny desktop :) I will test DirectIOTest on Solaris VM as well for 100,000 iterations and see how it behaves with and without the delay. My thinking is to add the delay for the platforms who need it. We can try that but often having a sleep call eventually results in the test's intermittently failing nonetheless. If that situation is observed then eventually we might just restrict this test further to the platforms on which it passes consistently. As Linux is the real target here that would not be tragic but it would be better to have it work on the other supported Unix flavors if possible. By the way, I already fixed the DirectIOTest.isDirectIOSupportedByFS() issue on Solaris. I will submit another version of the patch by Monday. OK, good. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Mon Sep 18 20:57:06 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 18 Sep 2017 13:57:06 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD120A6FC@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9! c7a47b7e577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> <1422ea82-d6ae-a5e2-2d06-5bfb35eb1557@oracl! ! ! ! e.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1207BDA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12083EB@ORSMSX113.amr.corp.intel.com> <8E172C93-DC2C-4E7D-BDA0-EA0E51825348@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12084E8@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1208BE3@ORSMSX113.amr.corp.intel.com> <9D28C97D-652B-457D-9E97-C052CFB94843@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120A6FC@ORSMSX113.amr.corp.intel.com> Message-ID: <0D1076BC-E261-48D0-AAF2-15C05FD33577@oracle.com> Hi Lucy, Do you have a wrapper script or something that you use to iterate over the test? Thanks, Brian On Sep 18, 2017, at 12:27 PM, Lu, Yingqi wrote: > I will remove MacOS from this test for now. Could you please help test DirectIOTest on Solaris for many iterations? My VM crashed and it will take me some time to recover. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Mon Sep 18 20:57:40 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Mon, 18 Sep 2017 20:57:40 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD120A6FC@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9! c7a47b7e577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> <1422ea82-d6ae-a5e2-2d06-5bfb35eb1557@oracl! ! ! e.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1207BDA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12083EB@ORSMSX113.amr.corp.intel.com> <8E172C93-DC2C-4E7D-BDA0-EA0E51825348@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12084E8@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1208BE3@ORSMSX113.amr.corp.intel.com> <9D28C97D-652B-457D-9E97-C052CFB94843@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120A6FC@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD120AD24@ORSMSX113.amr.corp.intel.com> Hi Brian, Here is the updated the patch: http://cr.openjdk.java.net/~kkharbas/8164900/webrev.15/ In this version, I did following changes: 1. In libDirectIO.c file, changed open(). Please let me know if you see any issues on Ubuntu system. 2. For ReadDirect, PreadDirect, WriteDirect, and PwriteDirect, I changed initTests() from void to boolean. On Solaris with non-supported file systems, initTests will return false so that the rest of the functions in main () will not be executed. 3. I removed MacOS from DirectIOTest. Please let me know if you see any issues on Solaris. Thanks, Lucy From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of Lu, Yingqi Sent: Monday, September 18, 2017 12:28 PM To: Brian Burkhalter Cc: nio-dev at openjdk.java.net Subject: RE: Proposal for adding O_DIRECT support into JDK 9 Hi Brian, I will remove MacOS from this test for now. Could you please help test DirectIOTest on Solaris for many iterations? My VM crashed and it will take me some time to recover. Hi Volker, Thank you for pointing out the issue with open() call. I will take a look and modify accordingly. At the meantime, it would be very helpful if you can please help test DirectIOTest.java on AIX for a big number of iterations as well. I do not have the test environment at all. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Monday, September 18, 2017 12:02 PM To: Lu, Yingqi > Cc: nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, On Sep 16, 2017, at 10:20 AM, Lu, Yingqi > wrote: I tested the current DirectIOTest on my Linux CentOS system for 200,000 iterations, they all passed. I tested the same test on MacOS VM for 20 iteration, some of them failed with the "file in cache" error while the rest passed. I searched online, and found this thread https://lists.apple.com/archives/filesystem-dev/2007/Sep/msg00012.html. If F_NOCACHE flag is set, an IO block might exist in the file system cache for a very short amount of time and then will be evicted. Then, I tried to add a 200 milliseconds delay between testWrite() and isFileInCache() as well as testRead() and isFileInCache(). I again did 100,000 iterations of DirectIOTest, they all passed. 200,000 iterations take too much time on my VM which is hosted on a tiny desktop :) I will test DirectIOTest on Solaris VM as well for 100,000 iterations and see how it behaves with and without the delay. My thinking is to add the delay for the platforms who need it. We can try that but often having a sleep call eventually results in the test's intermittently failing nonetheless. If that situation is observed then eventually we might just restrict this test further to the platforms on which it passes consistently. As Linux is the real target here that would not be tragic but it would be better to have it work on the other supported Unix flavors if possible. By the way, I already fixed the DirectIOTest.isDirectIOSupportedByFS() issue on Solaris. I will submit another version of the patch by Monday. OK, good. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Mon Sep 18 21:17:56 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Mon, 18 Sep 2017 21:17:56 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <0D1076BC-E261-48D0-AAF2-15C05FD33577@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9! c7a47b7e577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> <1422ea82-d6ae-a5e2-2d06-5bfb35eb1557@oracl! ! ! ! e.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1207BDA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12083EB@ORSMSX113.amr.corp.intel.com> <8E172C93-DC2C-4E7D-BDA0-EA0E51825348@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12084E8@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1208BE3@ORSMSX113.amr.corp.intel.com> <9D28C97D-652B-457D-9E97-C052CFB94843@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120A6FC@ORSMSX113.amr.corp.intel.com> <0D1076BC-E261-48D0-AAF2-15C05FD33577@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD120ADC0@ORSMSX113.amr.corp.intel.com> Hi Brian, Our email crossed :) I used a VERY simple way to test. I removed the all the dependencies to jdk.test.lib.Platform so that I can compile and run it independently. Then, I just compiled DirectIOTest.java use the following script to test. #!/bin/bash for i in {1..200000} do echo $i java -Djava.library.path=/home/JDK9-O_DIRECT/jdk10-o_direct-v15/build/linux-x86_64-normal-server-release/support/test/jdk/jtreg/native/lib DirectIOTest done exit 0 Since I just sent a newer version of the patch, could you please use that for testing? Please let me know if you have any question. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Monday, September 18, 2017 1:57 PM To: Lu, Yingqi Cc: nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, Do you have a wrapper script or something that you use to iterate over the test? Thanks, Brian On Sep 18, 2017, at 12:27 PM, Lu, Yingqi > wrote: I will remove MacOS from this test for now. Could you please help test DirectIOTest on Solaris for many iterations? My VM crashed and it will take me some time to recover. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Mon Sep 18 21:19:56 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 18 Sep 2017 14:19:56 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD120ADC0@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9! c7a47b7e577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> <1422ea82-d6ae-a5e2-2d06-5bfb35eb1557@oracl! ! ! ! ! e.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1207BDA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12083EB@ORSMSX113.amr.corp.intel.com> <8E172C93-DC2C-4E7D-BDA0-EA0E51825348@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12084E8@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1208BE3@ORSMSX113.amr.corp.intel.com> <9D28C97D-652B-457D-9E97-C052CFB94843@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120A6FC@ORSMSX113.amr.corp.intel.com> <0D1076BC-E261-48D0-AAF2-15C05FD33577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120ADC0@ORSMSX113.amr.corp.intel.com> Message-ID: <8561AA3C-1A42-4F51-90CA-340226EFA75B@oracle.com> Hi Lucy, On Sep 18, 2017, at 2:17 PM, Lu, Yingqi wrote: > I used a VERY simple way to test. [?] Thanks. > Since I just sent a newer version of the patch, could you please use that for testing? Please let me know if you have any question. Yes, of course. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Sep 19 01:08:17 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 18 Sep 2017 18:08:17 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <8561AA3C-1A42-4F51-90CA-340226EFA75B@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9! c7a47b7e577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> <1422ea82-d6ae-a5e2-2d06-5bfb35eb1557@oracl! ! ! ! ! ! e.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1207BDA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12083EB@ORSMSX113.amr.corp.intel.com> <8E172C93-DC2C-4E7D-BDA0-EA0E51825348@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12084E8@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1208BE3@ORSMSX113.amr.corp.intel.com> <9D28C97D-652B-457D-9E97-C052CFB94843@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120A6FC@ORSMSX113.amr.corp.intel.com> <0D1076BC-E261-48D0-AAF2-15C05FD33577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120ADC0@ORSMSX113.amr.corp.intel.com> <8561AA3C-1A42-4F51-90CA-340226EFA75B@oracle.com> Message-ID: <25DE662F-2D28-40B9-AA90-B75F50B07E3D@oracle.com> Hi Lucy, I ran the tests manually on Linux and macOS and did not see any problems. I ran your script for 5000 iterations of DirectIOTest on Solaris and did not see any problems. I still need to run the other four tests on Solaris as they do not really execute in the test harness which is on ZFS. I created a small patch http://cr.openjdk.java.net/~bpb/8164900/webrev.15-fix/ which has a couple of minor cleanups in DirectIOTest.java and trailing whitespace fixes in the other files in the this patch. You should be able to apply it directly. It looks like you fixed the things Alan commented on, viz., the need to do the position check in FileChannelImpl with the position lock held, and the use of rem instead of dst.remaining() in IOUtil::read. I don?t myself see any other problems in the code but we shall see what Alan and others think. I intend to look over the read/write tests one more time. Thanks, Brian From yingqi.lu at intel.com Tue Sep 19 01:34:55 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Tue, 19 Sep 2017 01:34:55 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <25DE662F-2D28-40B9-AA90-B75F50B07E3D@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9! c7a47b7e577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> <1422ea82-d6ae-a5e2-2d06-5bfb35eb1557@oracl! ! ! ! ! ! e.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1207BDA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12083EB@ORSMSX113.amr.corp.intel.com> <8E172C93-DC2C-4E7D-BDA0-EA0E51825348@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12084E8@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1208BE3@ORSMSX113.amr.corp.intel.com> <9D28C97D-652B-457D-9E97-C052CFB94843@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120A6FC@ORSMSX113.amr.corp.intel.com> <0D1076BC-E261-48D0-AAF2-15C05FD33577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120ADC0@ORSMSX113.amr.corp.intel.com> <8561AA3C-1A42-4F51-90CA-340226EFA75B@oracle.com> <25DE662F-2D28-40B9-AA90-B75F50B07E3D@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD120B26D@ORSMSX113.amr.corp.intel.com> Hi Brian/Alan, Thank you very much for all the help! I will apply your patch and send another version tomorrow. At the meantime, please let me know if you see any additional issues. All, please let us know if you have any comments/feedback in this patch. Thank you! Lucy >-----Original Message----- >From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] >Sent: Monday, September 18, 2017 6:08 PM >To: Lu, Yingqi >Cc: nio-dev at openjdk.java.net >Subject: Re: Proposal for adding O_DIRECT support into JDK 9 > >Hi Lucy, > >I ran the tests manually on Linux and macOS and did not see any problems. I ran >your script for 5000 iterations of DirectIOTest on Solaris and did not see any >problems. I still need to run the other four tests on Solaris as they do not really >execute in the test harness which is on ZFS. > >I created a small patch http://cr.openjdk.java.net/~bpb/8164900/webrev.15-fix/ >which has a couple of minor cleanups in DirectIOTest.java and trailing whitespace >fixes in the other files in the this patch. You should be able to apply it directly. > >It looks like you fixed the things Alan commented on, viz., the need to do the >position check in FileChannelImpl with the position lock held, and the use of rem >instead of dst.remaining() in IOUtil::read. I don't myself see any other problems in >the code but we shall see what Alan and others think. > >I intend to look over the read/write tests one more time. > >Thanks, > >Brian From yingqi.lu at intel.com Tue Sep 19 17:08:55 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Tue, 19 Sep 2017 17:08:55 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 References: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9! c7a47b7e577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> <1422ea82-d6ae-a5e2-2d06-5bfb35eb1557@oracl! ! ! ! ! ! e.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1207BDA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12083EB@ORSMSX113.amr.corp.intel.com> <8E172C93-DC2C-4E7D-BDA0-EA0E51825348@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12084E8@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1208BE3@ORSMSX113.amr.corp.intel.com> <9D28C97D-652B-457D-9E97-C052CFB94843@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120A6FC@ORSMSX113.amr.corp.intel.com> <0D1076BC-E261-48D0-AAF2-15C05FD33577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120ADC0@ORSMSX113.amr.corp.intel.com> <8561AA3C-1A42-4F51-90CA-340226EFA75B@oracle.com> <25DE662F-2D28-40B9-AA90-B75F50B07E3D@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD120BD39@ORSMSX113.amr.corp.intel.com> Hi Brian, Here is the updated version of the patch: http://cr.openjdk.java.net/~kkharbas/8164900/webrev.16/ In this version, I applied your clean-up fix. Please let me know if there is anything missing. Thanks, Lucy >-----Original Message----- >From: Lu, Yingqi >Sent: Monday, September 18, 2017 6:35 PM >To: 'Brian Burkhalter' >Cc: nio-dev at openjdk.java.net >Subject: RE: Proposal for adding O_DIRECT support into JDK 9 > >Hi Brian/Alan, > >Thank you very much for all the help! I will apply your patch and send another >version tomorrow. At the meantime, please let me know if you see any additional >issues. > >All, please let us know if you have any comments/feedback in this patch. > >Thank you! >Lucy > >>-----Original Message----- >>From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] >>Sent: Monday, September 18, 2017 6:08 PM >>To: Lu, Yingqi >>Cc: nio-dev at openjdk.java.net >>Subject: Re: Proposal for adding O_DIRECT support into JDK 9 >> >>Hi Lucy, >> >>I ran the tests manually on Linux and macOS and did not see any >>problems. I ran your script for 5000 iterations of DirectIOTest on >>Solaris and did not see any problems. I still need to run the other >>four tests on Solaris as they do not really execute in the test harness which is on >ZFS. >> >>I created a small patch >>http://cr.openjdk.java.net/~bpb/8164900/webrev.15-fix/ >>which has a couple of minor cleanups in DirectIOTest.java and trailing >>whitespace fixes in the other files in the this patch. You should be able to apply it >directly. >> >>It looks like you fixed the things Alan commented on, viz., the need to >>do the position check in FileChannelImpl with the position lock held, >>and the use of rem instead of dst.remaining() in IOUtil::read. I don't >>myself see any other problems in the code but we shall see what Alan and others >think. >> >>I intend to look over the read/write tests one more time. >> >>Thanks, >> >>Brian From brian.burkhalter at oracle.com Wed Sep 20 17:28:13 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 20 Sep 2017 10:28:13 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD120BD39@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9! c7a47b7e577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> <1422ea82-d6ae-a5e2-2d06-5bfb35eb1557@oracl! ! ! ! ! ! ! e.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1207BDA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12083EB@ORSMSX113.amr.corp.intel.com> <8E172C93-DC2C-4E7D-BDA0-EA0E51825348@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12084E8@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1208BE3@ORSMSX113.amr.corp.intel.com> <9D28C97D-652B-457D-9E97-C052CFB94843@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120A6FC@ORSMSX113.amr.corp.intel.com> <0D1076BC-E261-48D0-AAF2-15C05FD33577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120ADC0@ORSMSX113.amr.corp.intel.com> <8561AA3C-1A42-4F51-90CA-340226EFA75B@oracle.com> <25DE662F-2D28-40B9-AA90-B75F50B07E3D@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120BD39@ORSMSX113.amr.corp.intel.com> Message-ID: Hi Lucy, The jcheck Mercurial extension [1] indicates a trailing whitespace at line 468 of src/java.base/windows/native/libnio/ch/FileDispatcherImpl.c. No need to generate a new webrev for this alone but before the final version of the patch you might want to run jcheck and fix any problems using the Perl normalizer script [2]. Thanks, Brian [1] http://openjdk.java.net/projects/code-tools/jcheck/ [2] http://hg.openjdk.java.net/jdk10/jdk10/file/62306e615de1/make/scripts/normalizer.pl On Sep 19, 2017, at 10:08 AM, Lu, Yingqi wrote: > Here is the updated version of the patch:http://cr.openjdk.java.net/~kkharbas/8164900/webrev.16/ > > In this version, I applied your clean-up fix. > > Please let me know if there is anything missing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Wed Sep 20 20:24:03 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Wed, 20 Sep 2017 20:24:03 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9! c7a47b7e577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> <1422ea82-d6ae-a5e2-2d06-5bfb35eb1557@oracl! ! ! ! ! ! ! e.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1207BDA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12083EB@ORSMSX113.amr.corp.intel.com> <8E172C93-DC2C-4E7D-BDA0-EA0E51825348@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12084E8@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1208BE3@ORSMSX113.amr.corp.intel.com> <9D28C97D-652B-457D-9E97-C052CFB94843@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120A6FC@ORSMSX113.amr.corp.intel.com> <0D1076BC-E261-48D0-AAF2-15C05FD33577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120ADC0@ORSMSX113.amr.corp.intel.com> <8561AA3C-1A42-4F51-90CA-340226EFA75B@oracle.com> <25DE662F-2D28-40B9-AA90-B75F50B07E3D@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120BD39@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD120DB12@ORSMSX113.amr.corp.intel.com> Thank you! I will use the tool for the future versions. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Wednesday, September 20, 2017 10:28 AM To: Lu, Yingqi Cc: nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, The jcheck Mercurial extension [1] indicates a trailing whitespace at line 468 of src/java.base/windows/native/libnio/ch/FileDispatcherImpl.c. No need to generate a new webrev for this alone but before the final version of the patch you might want to run jcheck and fix any problems using the Perl normalizer script [2]. Thanks, Brian [1] http://openjdk.java.net/projects/code-tools/jcheck/ [2] http://hg.openjdk.java.net/jdk10/jdk10/file/62306e615de1/make/scripts/normalizer.pl On Sep 19, 2017, at 10:08 AM, Lu, Yingqi > wrote: Here is the updated version of the patch:http://cr.openjdk.java.net/~kkharbas/8164900/webrev.16/ In this version, I applied your clean-up fix. Please let me know if there is anything missing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vyom.tewari at oracle.com Fri Sep 22 09:42:52 2017 From: vyom.tewari at oracle.com (vyom tewari) Date: Fri, 22 Sep 2017 15:12:52 +0530 Subject: RFR 8145635 : Add TCP_QUICKACK socket option In-Reply-To: <0b0989b5-c791-a476-a82f-6c686b19c9d3@oracle.com> References: <56CD74E1.60309@oracle.com> <56CD7C0A.2080402@oracle.com> <0b0989b5-c791-a476-a82f-6c686b19c9d3@oracle.com> Message-ID: ping!! Vyom On Monday 11 September 2017 09:08 PM, vyom tewari wrote: > > Hi All, > > As jdk.net API already moved out of java.base, Please review the below > code change for jdk10. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8145635 > > Webrev: http://cr.openjdk.java.net/~vtewari/8145635/webrev0.1/index.html > > Thanks, > > Vyom > > > On Wednesday 24 February 2016 03:16 PM, Alan Bateman wrote: >> >> On 24/02/2016 09:16, vyom wrote: >>> Hi All, >>> >>> Please review my code changes for the below issue. >>> >>> Bug: JDK-8145635 : Add TCP_QUICKACK socket option >>> >>> Webrev: >>> http://cr.openjdk.java.net/~vtewari/8145635/webrev0.0/index.html >>> >>> Currently TCP_QUICKACK is only supported on Linux( since Linux >>> 2.4.4) so i implemented is as "ExtendedSocketOption". >> >> Is there any urgency on this one? I'm just wondering if we should try >> to the jdk.net API moved out of java.base before we add more socket >> options. This is currently tracked as JDK-8044773 and important to >> get into JDK 9. >> >> -Alan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Sep 26 22:09:56 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 26 Sep 2017 15:09:56 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD120DB12@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9! c7a47b7e577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> <1422ea82-d6ae-a5e2-2d06-5bfb35eb1557@oracl! ! ! ! ! ! ! ! e.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1207BDA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12083EB@ORSMSX113.amr.corp.intel.com> <8E172C93-DC2C-4E7D-BDA0-EA0E51825348@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12084E8@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1208BE3@ORSMSX113.amr.corp.intel.com> <9D28C97D-652B-457D-9E97-C052CFB94843@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120A6FC@ORSMSX113.amr.corp.intel.com> <0D1076BC-E261-48D0-AAF2-15C05FD33577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120ADC0@ORSMSX113.amr.corp.intel.com> <8561AA3C-1A42-4F51-90CA-340226EFA75B@oracle.com> <25DE662F-2D28-40B9-AA90-B75F50B07E3D@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120BD39@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120DB12@ORSMSX113.amr.corp.intel.com> Message-ID: <7835146B-2FFC-44BB-BE7A-64181273029D@oracle.com> Hi Lucy, Per the JDK 10 repository consolidation [1], the Direct I/O patch needed some minor reshuffling. I have created a patch re-based to the new JDK 10 master [2]. Unless I missed something, the differences are: make/test/JtregNative.gmk -> make/test/JtregNativeJdk.gmk with some content update test/java/nio -> test/jdk/java/nio removal of some trailing white spaces in the Windows version of FileDispatcherImpl.c I tested this locally on macOS and am running it presently through our test harness. Thanks, Brian [1] http://mail.openjdk.java.net/pipermail/jdk10-dev/2017-September/000499.html [2] http://cr.openjdk.java.net/~bpb/8164900/webrev.16-bpb/ On Sep 20, 2017, at 1:24 PM, Lu, Yingqi wrote: > Thank you! I will use the tool for the future versions. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Tue Sep 26 23:04:18 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Tue, 26 Sep 2017 23:04:18 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <7835146B-2FFC-44BB-BE7A-64181273029D@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9! c7a47b7e577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> <1422ea82-d6ae-a5e2-2d06-5bfb35eb1557@oracl! ! ! ! ! ! ! ! e.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1207BDA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12083EB@ORSMSX113.amr.corp.intel.com> <8E172C93-DC2C-4E7D-BDA0-EA0E51825348@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12084E8@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1208BE3@ORSMSX113.amr.corp.intel.com> <9D28C97D-652B-457D-9E97-C052CFB94843@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120A6FC@ORSMSX113.amr.corp.intel.com> <0D1076BC-E261-48D0-AAF2-15C05FD33577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120ADC0@ORSMSX113.amr.corp.intel.com> <8561AA3C-1A42-4F51-90CA-340226EFA75B@oracle.com> <25DE662F-2D28-40B9-AA90-B75F50B07E3D@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120BD39@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120DB12@ORSMSX113.amr.corp.intel.com> <7835146B-2FFC-44BB-BE7A-64181273029D@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD1214A0A@ORSMSX113.amr.corp.intel.com> Hi Brian, Thank you very much for the help! Please let me know if you see any errors on your test harness. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Tuesday, September 26, 2017 3:10 PM To: Lu, Yingqi Cc: nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, Per the JDK 10 repository consolidation [1], the Direct I/O patch needed some minor reshuffling. I have created a patch re-based to the new JDK 10 master [2]. Unless I missed something, the differences are: * make/test/JtregNative.gmk -> make/test/JtregNativeJdk.gmk with some content update * test/java/nio -> test/jdk/java/nio * removal of some trailing white spaces in the Windows version of FileDispatcherImpl.c I tested this locally on macOS and am running it presently through our test harness. Thanks, Brian [1] http://mail.openjdk.java.net/pipermail/jdk10-dev/2017-September/000499.html [2] http://cr.openjdk.java.net/~bpb/8164900/webrev.16-bpb/ On Sep 20, 2017, at 1:24 PM, Lu, Yingqi > wrote: Thank you! I will use the tool for the future versions. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Sep 26 23:08:44 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 26 Sep 2017 16:08:44 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD1214A0A@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD12003BF@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1200642@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1201FE8@ORSMSX113.amr.corp.intel.com> <8921C32C-EF69-4F64-8757-8C0B3E92C95E@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12026DA@ORSMSX113.amr.corp.intel.com> <8B047ED4-C243-4997-8182-7F2A2C998FA1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120320F@ORSMSX113.amr.corp.intel.com> <5b6d7a90-5564-d24a-aa67-9! c7a47b7e577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120419E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12042CC@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120625E@ORSMSX113.amr.corp.intel.com> <1422ea82-d6ae-a5e2-2d06-5bfb35eb1557@oracl! ! ! ! ! ! ! ! ! e.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1207BDA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12083EB@ORSMSX113.amr.corp.intel.com> <8E172C93-DC2C-4E7D-BDA0-EA0E51825348@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12084E8@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1208BE3@ORSMSX113.amr.corp.intel.com> <9D28C97D-652B-457D-9E97-C052CFB94843@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120A6FC@ORSMSX113.amr.corp.intel.com> <0D1076BC-E261-48D0-AAF2-15C05FD33577@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120ADC0@ORSMSX113.amr.corp.intel.com> <8561AA3C-1A42-4F51-90CA-340226EFA75B@oracle.com> <25DE662F-2D28-40B9-AA90-B75F50B07E3D@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120BD39@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD120DB12@ORSMSX113.amr.corp.intel.com> <7835146B-2FFC-44BB-BE7A-64181273029D@o! racle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1214A0A@ORSMSX113.amr.corp.intel.com> Message-ID: Hi Lucy, On Sep 26, 2017, at 4:04 PM, Lu, Yingqi wrote: > Thank you very much for the help! Please note that the webrev I uploaded contains a complete patch, i.e., it is not an overlay. > Please let me know if you see any errors on your test harness. Will do. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From fridrich.strba at suse.com Wed Sep 27 12:31:13 2017 From: fridrich.strba at suse.com (Fridrich Strba) Date: Wed, 27 Sep 2017 14:31:13 +0200 Subject: A possible fix for JDK-8165852: Mount point not found for a file which is present in overlayfs Message-ID: Hello, good people, I think I fixed the overlayfs and btrfs problems in LinuxFileStore. The attached patch is what makes it work for us. Basically, when we come to the device-id boundary, we check whether it corresponds to a mount point in the files and if so, we return. If not, we go down the file system looking for another boundary. If we come to the root of the file-system without finding other mount-point and if the / is in the files, we return the entry that corresponds to it. If not, we throw the exception. It is not elegant, but it works. I would love to have someone to look at it and consider whether it could be up-streamed. Thanks in advance Fridrich -------------- next part -------------- A non-text attachment was scrubbed... Name: java-10-openjdk-linuxfilestore.patch Type: text/x-patch Size: 949 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 195 bytes Desc: OpenPGP digital signature URL: From brian.burkhalter at oracle.com Wed Sep 27 14:50:25 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 27 Sep 2017 07:50:25 -0700 Subject: A possible fix for JDK-8165852: Mount point not found for a file which is present in overlayfs In-Reply-To: References: Message-ID: Hello, Fridrich, Excellent! This would be good if it works albeit inelegantly: function over form. I?ll add it to my list to review. To test it I will need to dust off my Linux VM which has all this stuff (Docker, btrfs, etc.) installed. Thanks, Brian On Sep 27, 2017, at 5:31 AM, Fridrich Strba wrote: > Hello, good people, > > I think I fixed the overlayfs and btrfs problems in LinuxFileStore. The > attached patch is what makes it work for us. Basically, when we come to > the device-id boundary, we check whether it corresponds to a mount point > in the files and if so, we return. If not, we go down the file system > looking for another boundary. If we come to the root of the file-system > without finding other mount-point and if the / is in the files, we > return the entry that corresponds to it. If not, we throw the exception. > > It is not elegant, but it works. I would love to have someone to look at > it and consider whether it could be up-streamed. > > Thanks in advance > > Fridrich > > >