From brian.burkhalter at oracle.com Mon Oct 3 23:58:30 2016 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 3 Oct 2016 16:58:30 -0700 Subject: RFC on 8165823: (se) EPollArrayWrapper throws NPE if limits.conf set to 65536 and fd=65536 Message-ID: Either my reasoning based on code inspection is faulty or there is something fishy about this bug report [1]. If the maximum number of open file descriptors is set to 65536 by ulimit -n, in limits.conf, or some other means it looks like the range of allowed file descriptors should be [0,65535]. In EPollArrayWrapper [2] the array of fds has size MAX_UPDATE_ARRAY_SIZE = min(OPEN_MAX,64*1024) (lines 83-84) where OPEN_MAX is the value of the rlim_max member of the struct rlimit set by a call to getrlimit() [3] (line 73). The getrlimit() system call for resource RLIMIT_NOFILE is specified to return ?one greater than the maximum file descriptor number that can be opened by this process.? In this case OPEN_MAX should be 65536 which is 64*1024 so that MAX_UPDATE_ARRAY_SIZE is 65536 hence equal to OPEN_MAX and the eventsHigh HashMap in EPollArrayWrapper is not initialized (line 138). Then if the maximum allowed fd for the process should not exceed 65535, the fd parameter of EPollArrayWrapper.setUpdateEvents() (line 183) should not exceed 65535 which is less than MAX_UPDATE_ARRAY_SIZE (65536) so the branch which uses the eventsHigh map (line 189) should never be entered. What seems to be a more likely scenario for the NPE failure is that the resource limit on the number of open file descriptors is being changed to a value greater than MAX_UPDATE_ARRAY_SIZE *after* the EPollArrayWrapper instance is constructed which would mean that setUpdateEvents() could in fact receive an fd parameter greater than MAX_UPDATE_ARRAY_SIZE and an NPE due to a null eventsHigh would ensue. Comments welcome. Thanks, Brian [1] https://bugs.openjdk.java.net/browse/JDK-8165823 [2] http://hg.openjdk.java.net/jdk9/dev/jdk/file/2a474d0ba36d/src/java.base/linux/classes/sun/nio/ch/EPollArrayWrapper.java [3] http://manpages.ubuntu.com/manpages/xenial/man2/getrlimit.2.html From brian.burkhalter at oracle.com Tue Oct 4 00:13:25 2016 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 3 Oct 2016 17:13:25 -0700 Subject: JDK 9 RFR of 8167058: UnixDirectoryIterator::stream unused Message-ID: Please review this minor change at your convenience. Issue: https://bugs.openjdk.java.net/browse/JDK-8167058 Patch: http://cr.openjdk.java.net/~bpb/8167058/ Thanks, Brian From claes.redestad at oracle.com Tue Oct 4 05:58:35 2016 From: claes.redestad at oracle.com (Claes Redestad) Date: Tue, 4 Oct 2016 07:58:35 +0200 Subject: JDK 9 RFR of 8167058: UnixDirectoryIterator::stream unused In-Reply-To: References: Message-ID: <57F3450B.3080602@oracle.com> Looks good, thanks! /Claes On 2016-10-04 02:13, Brian Burkhalter wrote: > Please review this minor change at your convenience. > > Issue: https://bugs.openjdk.java.net/browse/JDK-8167058 > Patch: http://cr.openjdk.java.net/~bpb/8167058/ > > Thanks, > > Brian > From christoph.langer at sap.com Tue Oct 4 07:44:41 2016 From: christoph.langer at sap.com (Langer, Christoph) Date: Tue, 4 Oct 2016 07:44:41 +0000 Subject: RFC on 8165823: (se) EPollArrayWrapper throws NPE if limits.conf set to 65536 and fd=65536 In-Reply-To: References: Message-ID: <783e980056154a39a9381d8daa446aa3@DEWDFE13DE11.global.corp.sap> Hi Brian, I agree to your code analysis, that for a ulimit value of 65536 still the NPE should not occur, assuming the fd number can't be higher than 65535. However, is there a guarantee about the file descriptor numbers themselves being in the range between 0 and OPEN_MAX - 1? Maybe just the count of open file descriptors has to be in the OPEN_MAX range but single number values could well be out of that range? At least on certain operating systems? I remember Thomas has just recently done some work in the area of file descriptors: @Thomas: Do you have anything to add here? Best regards Christoph > -----Original Message----- > From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of Brian > Burkhalter > Sent: Dienstag, 4. Oktober 2016 01:59 > To: nio-dev > Subject: RFC on 8165823: (se) EPollArrayWrapper throws NPE if limits.conf set > to 65536 and fd=65536 > > Either my reasoning based on code inspection is faulty or there is something > fishy about this bug report [1]. > > If the maximum number of open file descriptors is set to 65536 by ulimit -n, in > limits.conf, or some other means it looks like the range of allowed file > descriptors should be [0,65535]. In EPollArrayWrapper [2] the array of fds has > size > > MAX_UPDATE_ARRAY_SIZE = min(OPEN_MAX,64*1024) > > (lines 83-84) where OPEN_MAX is the value of the rlim_max member of the > struct rlimit set by a call to getrlimit() [3] (line 73). The getrlimit() system call > for resource RLIMIT_NOFILE is specified to return "one greater than the > maximum file descriptor number that can be opened by this process." In this > case OPEN_MAX should be 65536 which is 64*1024 so that > MAX_UPDATE_ARRAY_SIZE is 65536 hence equal to OPEN_MAX and the > eventsHigh HashMap in EPollArrayWrapper is not initialized (line 138). Then if > the maximum allowed fd for the process should not exceed 65535, the fd > parameter of EPollArrayWrapper.setUpdateEvents() (line 183) should not > exceed 65535 which is less than MAX_UPDATE_ARRAY_SIZE (65536) so the > branch which uses the eventsHigh map (line 189) should never be entered. > > What seems to be a more likely scenario for the NPE failure is that the > resource limit on the number of open file descriptors is being changed to a > value greater than MAX_UPDATE_ARRAY_SIZE *after* the EPollArrayWrapper > instance is constructed which would mean that setUpdateEvents() could in fact > receive an fd parameter greater than MAX_UPDATE_ARRAY_SIZE and an NPE > due to a null eventsHigh would ensue. > > Comments welcome. > > Thanks, > > Brian > > [1] https://bugs.openjdk.java.net/browse/JDK-8165823 > [2] > http://hg.openjdk.java.net/jdk9/dev/jdk/file/2a474d0ba36d/src/java.base/linux > /classes/sun/nio/ch/EPollArrayWrapper.java > [3] http://manpages.ubuntu.com/manpages/xenial/man2/getrlimit.2.html From Alan.Bateman at oracle.com Tue Oct 4 07:47:42 2016 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 4 Oct 2016 08:47:42 +0100 Subject: JDK 9 RFR of 8167058: UnixDirectoryIterator::stream unused In-Reply-To: References: Message-ID: <0b044a44-d6b5-160a-c55e-94b1b9653f7b@oracle.com> On 04/10/2016 01:13, Brian Burkhalter wrote: > Please review this minor change at your convenience. > > Issue: https://bugs.openjdk.java.net/browse/JDK-8167058 > Patch: http://cr.openjdk.java.net/~bpb/8167058/ > Looks fine. From christoph.langer at sap.com Tue Oct 4 08:33:19 2016 From: christoph.langer at sap.com (Langer, Christoph) Date: Tue, 4 Oct 2016 08:33:19 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5134F66@ORSMSX113.amr.corp.intel.com> <062c589c-fa54-1652-683a-4e779816391d@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> Message-ID: Hi Lucy, FWIW: I ran a build on AIX and this looks ok. I also assume in your final version you'll update all copyright years where it's not 2016 yet? Other than that the changes look ok to me - but I'm neither a reviewer nor a deep expert in the area of your changes. Best regards Christoph > -----Original Message----- > From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of Lu, > Yingqi > Sent: Freitag, 30. September 2016 18:55 > To: Lu, Yingqi ; Alan Bateman > ; core-libs-dev at openjdk.java.net > Cc: nio-dev at openjdk.java.net; Kaczmarek, Eric ; > Kharbas, Kishor > Subject: RE: Proposal for adding O_DIRECT support into JDK 9 > > Hi All, > > Please find the most recent version of the patch available at > http://cr.openjdk.java.net/~igraves/8164900-2/ > > In this version, we have following two changes: > > 1. Move O_DIRECT flag from StandardOpenOption to ExtendedOpenOption > 2. Move the checks of O_DIRECT from native code to Java code. > > Please let us know your feedback. > > Thanks, > Lucy > > -----Original Message----- > From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of Lu, > Yingqi > Sent: Tuesday, September 27, 2016 9:57 AM > To: Alan Bateman ; core-libs- > dev at openjdk.java.net > Cc: nio-dev at openjdk.java.net > Subject: RE: Proposal for adding O_DIRECT support into JDK 9 > > Alan, > > Thank you for the explanation, we will modify the code accordingly and send it > out soon for review. > > Thanks, > Lucy > > -----Original Message----- > From: Alan Bateman [mailto:Alan.Bateman at oracle.com] > Sent: Tuesday, September 27, 2016 8:45 AM > To: Lu, Yingqi ; core-libs-dev at openjdk.java.net > Cc: nio-dev at openjdk.java.net > Subject: Re: Proposal for adding O_DIRECT support into JDK 9 > > On 26/09/2016 19:50, Lu, Yingqi wrote: > > > Alan, you mean readv0/write0 or read0/write0? I just want to make sure > > :-) > Apologies, I meant each of the native methods where the decision to use direct > I/O or not would be a lot more maintainable in Java. You'll see that there are > already code paths for direct vs. heap buffers. > > > > > > Anyone else has other opinions on where is the best home for O_DIRECT flag? > The flags under jdk.unsupported will eventually be removed in the future JDK > release? > > > > If we agree ExtendedOpenOpen is the best home for O_DIRECT, we can > modify that for sure. > > > I think ExtendedOpenOption is the right place. It's still TDB as to whether to put > these extensions but that should be transparent to anyone using this when on > the class path. > > -Alan From brian.burkhalter at oracle.com Tue Oct 4 15:26:06 2016 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 4 Oct 2016 08:26:06 -0700 Subject: RFC on 8165823: (se) EPollArrayWrapper throws NPE if limits.conf set to 65536 and fd=65536 In-Reply-To: <783e980056154a39a9381d8daa446aa3@DEWDFE13DE11.global.corp.sap> References: <783e980056154a39a9381d8daa446aa3@DEWDFE13DE11.global.corp.sap> Message-ID: Hi Christoph, Thanks for your comments. On Oct 4, 2016, at 12:44 AM, Langer, Christoph wrote: > I agree to your code analysis, that for a ulimit value of 65536 still the NPE should not occur, assuming the fd number can't be higher than 65535. However, is there a guarantee about the file descriptor numbers themselves being in the range between 0 and OPEN_MAX - 1? Maybe just the count of open file descriptors has to be in the OPEN_MAX range but single number values could well be out of that range? At least on certain operating systems? Based on my reading of [1, 2] it seems like the fd number must be in the range [0,OPEN_MAX), i.e., 0 <= fd < OPEN_MAX. RLIMIT_NOFILE Specifies a value one greater than the maximum file descriptor number that can be opened by this process. It is possible of course that some operating systems interpret this as the cardinality rather than the exclusive least upper bound. > I remember Thomas has just recently done some work in the area of file descriptors: @Thomas: Do you have anything to add here? I would be curious to hear about it. In the worse case the code could be made more defensive for this case with for example a getEventsHigh() method which would lazily initialize the eventsHigh HashMap. The penalty of course would be in performance. Best regards, Brian [1] https://linux.die.net/man/2/getrlimit [2] http://manpages.ubuntu.com/manpages/xenial/man2/getrlimit.2.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Oct 4 16:27:39 2016 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 4 Oct 2016 17:27:39 +0100 Subject: RFC on 8165823: (se) EPollArrayWrapper throws NPE if limits.conf set to 65536 and fd=65536 In-Reply-To: References: Message-ID: On 04/10/2016 00:58, Brian Burkhalter wrote: > Either my reasoning based on code inspection is faulty or there is something fishy about this bug report [1]. > Can you check the jdk7u forest, the bug report seems to be 7u rather than 8 or 9. -Alan From yingqi.lu at intel.com Tue Oct 4 17:24:21 2016 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Tue, 4 Oct 2016 17:24:21 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5134F66@ORSMSX113.amr.corp.intel.com> <062c589c-fa54-1652-683a-4e779816391d@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> Hi Christoph, Thank you very much for trying our patch. We are still seeking the feedback from the community. When we get closer to the final version of the patch, we will modify the copyright years. Thank you for reminding us! Anyone else has any feedback/comments? Thanks, Lucy -----Original Message----- From: Langer, Christoph [mailto:christoph.langer at sap.com] Sent: Tuesday, October 04, 2016 1:33 AM To: Lu, Yingqi ; Alan Bateman ; core-libs-dev at openjdk.java.net Cc: nio-dev at openjdk.java.net; Kaczmarek, Eric ; Kharbas, Kishor Subject: RE: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, FWIW: I ran a build on AIX and this looks ok. I also assume in your final version you'll update all copyright years where it's not 2016 yet? Other than that the changes look ok to me - but I'm neither a reviewer nor a deep expert in the area of your changes. Best regards Christoph > -----Original Message----- > From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of > Lu, Yingqi > Sent: Freitag, 30. September 2016 18:55 > To: Lu, Yingqi ; Alan Bateman > ; core-libs-dev at openjdk.java.net > Cc: nio-dev at openjdk.java.net; Kaczmarek, Eric > ; Kharbas, Kishor > Subject: RE: Proposal for adding O_DIRECT support into JDK 9 > > Hi All, > > Please find the most recent version of the patch available at > http://cr.openjdk.java.net/~igraves/8164900-2/ > > In this version, we have following two changes: > > 1. Move O_DIRECT flag from StandardOpenOption to ExtendedOpenOption 2. > Move the checks of O_DIRECT from native code to Java code. > > Please let us know your feedback. > > Thanks, > Lucy > > -----Original Message----- > From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of > Lu, Yingqi > Sent: Tuesday, September 27, 2016 9:57 AM > To: Alan Bateman ; core-libs- > dev at openjdk.java.net > Cc: nio-dev at openjdk.java.net > Subject: RE: Proposal for adding O_DIRECT support into JDK 9 > > Alan, > > Thank you for the explanation, we will modify the code accordingly and > send it out soon for review. > > Thanks, > Lucy > > -----Original Message----- > From: Alan Bateman [mailto:Alan.Bateman at oracle.com] > Sent: Tuesday, September 27, 2016 8:45 AM > To: Lu, Yingqi ; core-libs-dev at openjdk.java.net > Cc: nio-dev at openjdk.java.net > Subject: Re: Proposal for adding O_DIRECT support into JDK 9 > > On 26/09/2016 19:50, Lu, Yingqi wrote: > > > Alan, you mean readv0/write0 or read0/write0? I just want to make > > sure > > :-) > Apologies, I meant each of the native methods where the decision to > use direct I/O or not would be a lot more maintainable in Java. You'll > see that there are already code paths for direct vs. heap buffers. > > > > > > Anyone else has other opinions on where is the best home for O_DIRECT flag? > The flags under jdk.unsupported will eventually be removed in the > future JDK release? > > > > If we agree ExtendedOpenOpen is the best home for O_DIRECT, we can > modify that for sure. > > > I think ExtendedOpenOption is the right place. It's still TDB as to > whether to put these extensions but that should be transparent to > anyone using this when on the class path. > > -Alan From brian.burkhalter at oracle.com Tue Oct 4 18:04:05 2016 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 4 Oct 2016 11:04:05 -0700 Subject: RFC on 8165823: (se) EPollArrayWrapper throws NPE if limits.conf set to 65536 and fd=65536 In-Reply-To: References: Message-ID: <562286F8-1828-4823-8EB1-BA9717E2BE32@oracle.com> On Oct 4, 2016, at 9:27 AM, Alan Bateman wrote: > On 04/10/2016 00:58, Brian Burkhalter wrote: > >> Either my reasoning based on code inspection is faulty or there is something fishy about this bug report [1]. >> > Can you check the jdk7u forest, the bug report seems to be 7u rather than 8 or 9. The jdk7u and jdk9-dev implementatinos of EPollArrayWrapper.java are identical aside from the more recent copyright year in the latter. Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Wed Oct 5 00:06:06 2016 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 4 Oct 2016 17:06:06 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5134F66@ORSMSX113.amr.corp.intel.com> <062c589c-fa54-1652-683a-4e779816391d@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> Message-ID: <45C5D272-E70A-40D9-8166-48ADD43763D9@oracle.com> Hi Lucy, On Oct 4, 2016, at 10:24 AM, Lu, Yingqi wrote: > Anyone else has any feedback/comments? I looked over the 8164900-2 patch and did not see any conceptual problems per se. I did not however scrutinize all the detailed changes, especially the more extensive ones in FileDispatcherImpl.c. I did however run the patch through our regression tests for IO and NIO and encountered some problems. Firstly the patch breaks the build on OS X [1], Solaris [2], and Windows [3]. Note that the code under src/java.base/unix is built on all Unix systems including Linux, OS X, and Solaris. Secondly but of lesser note there were some warnings during the Linux build [4]. Otherwise the IO and NIO tests succeeded on Linux, but that only suggests that pre-existing functionality is preserved as those tests of course do not exercise the new code for which I suppose tests will be provided in the final version of the patch. Regards, Brian [1] OS X build/macosx-x64/support/gensrc/java.base/sun/nio/fs/UnixConstants.java:41: error: self-reference in initializer static final int O_DIRECT = O_DIRECT; ^ [2] Solaris build/solaris-x64/support/gensrc/java.base/sun/nio/fs/UnixConstants.java:41: error: self-reference in initializer static final int O_DIRECT = O_DIRECT; ^ [3] Windows jdk\src\java.base\windows\classes\java\io\FileDescriptor.java:76: error: is not abstract and does not override abstract method getDirect(FileDescriptor) in JavaIOFileDescriptorAccess new JavaIOFileDescriptorAccess() { ^ [4] Linux (Ubuntu 16.04 amd64 desktop) jdk/src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c: In function ?readDirect?: jdk/src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c:104:5: warning: ignoring return value of ?posix_memalign?, declared with attribute warn_unused_result [-Wunused-result] posix_memalign(&bytes, pageSize, newLen); ^ jdk/src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c: In function ?writeDirect?: jdk/src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c:131:5: warning: ignoring return value of ?posix_memalign?, declared with attribute warn_unused_result [-Wunused-result] posix_memalign(&bytes, pageSize, len); ^ jdk/src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c: In function ?Java_sun_nio_ch_FileDispatcherImpl_readvDirect0?: jdk/src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c:250:5: warning: ignoring return value of ?posix_memalign?, declared with attribute warn_unused_result [-Wunused-result] posix_memalign(&bytes, pageSize, newLen); ^ jdk/src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c: In function ?Java_sun_nio_ch_FileDispatcherImpl_writevDirect0?: jdk/src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c:393:5: warning: ignoring return value of ?posix_memalign?, declared with attribute warn_unused_result [-Wunused-result] posix_memalign(&bytes, pageSize, totalLen) -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Wed Oct 5 00:35:04 2016 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Wed, 5 Oct 2016 00:35:04 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <45C5D272-E70A-40D9-8166-48ADD43763D9@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5134F66@ORSMSX113.amr.corp.intel.com> <062c589c-fa54-1652-683a-4e779816391d@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <45C5D272-E70A-40D9-8166-48ADD43763D9@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AA513D886@ORSMSX113.amr.corp.intel.com> Hi Brain, Thank you very much for reviewing the patch and providing the detailed error/warning messages. They are very helpful! We will look into the issues and solve them in the next version of the patch. Test cases for O_DIRECT and modified copyright year will be added in next version too. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Tuesday, October 04, 2016 5:06 PM To: Lu, Yingqi Cc: Langer, Christoph ; Alan Bateman ; core-libs-dev at openjdk.java.net; nio-dev at openjdk.java.net; Kharbas, Kishor ; Kaczmarek, Eric Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, On Oct 4, 2016, at 10:24 AM, Lu, Yingqi > wrote: Anyone else has any feedback/comments? I looked over the 8164900-2 patch and did not see any conceptual problems per se. I did not however scrutinize all the detailed changes, especially the more extensive ones in FileDispatcherImpl.c. I did however run the patch through our regression tests for IO and NIO and encountered some problems. Firstly the patch breaks the build on OS X [1], Solaris [2], and Windows [3]. Note that the code under src/java.base/unix is built on all Unix systems including Linux, OS X, and Solaris. Secondly but of lesser note there were some warnings during the Linux build [4]. Otherwise the IO and NIO tests succeeded on Linux, but that only suggests that pre-existing functionality is preserved as those tests of course do not exercise the new code for which I suppose tests will be provided in the final version of the patch. Regards, Brian [1] OS X build/macosx-x64/support/gensrc/java.base/sun/nio/fs/UnixConstants.java:41: error: self-reference in initializer static final int O_DIRECT = O_DIRECT; ^ [2] Solaris build/solaris-x64/support/gensrc/java.base/sun/nio/fs/UnixConstants.java:41: error: self-reference in initializer static final int O_DIRECT = O_DIRECT; ^ [3] Windows jdk\src\java.base\windows\classes\java\io\FileDescriptor.java:76: error: is not abstract and does not override abstract method getDirect(FileDescriptor) in JavaIOFileDescriptorAccess new JavaIOFileDescriptorAccess() { ^ [4] Linux (Ubuntu 16.04 amd64 desktop) jdk/src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c: In function 'readDirect': jdk/src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c:104:5: warning: ignoring return value of 'posix_memalign', declared with attribute warn_unused_result [-Wunused-result] posix_memalign(&bytes, pageSize, newLen); ^ jdk/src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c: In function 'writeDirect': jdk/src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c:131:5: warning: ignoring return value of 'posix_memalign', declared with attribute warn_unused_result [-Wunused-result] posix_memalign(&bytes, pageSize, len); ^ jdk/src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c: In function 'Java_sun_nio_ch_FileDispatcherImpl_readvDirect0': jdk/src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c:250:5: warning: ignoring return value of 'posix_memalign', declared with attribute warn_unused_result [-Wunused-result] posix_memalign(&bytes, pageSize, newLen); ^ jdk/src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c: In function 'Java_sun_nio_ch_FileDispatcherImpl_writevDirect0': jdk/src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c:393:5: warning: ignoring return value of 'posix_memalign', declared with attribute warn_unused_result [-Wunused-result] posix_memalign(&bytes, pageSize, totalLen) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Wed Oct 5 10:35:58 2016 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 5 Oct 2016 11:35:58 +0100 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5134F66@ORSMSX113.amr.corp.intel.com> <062c589c-fa54-1652-683a-4e779816391d@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> Message-ID: <767dc2e2-900a-b26f-9da9-b2dbc0f1eb71@oracle.com> On 04/10/2016 18:24, Lu, Yingqi wrote: > Hi Christoph, > > Thank you very much for trying our patch. > > We are still seeking the feedback from the community. When we get closer to the final version of the patch, we will modify the copyright years. Thank you for reminding us! > > Anyone else has any feedback/comments? > I have this on my list to review too, hopefully very soon. -Alan From Alan.Bateman at oracle.com Wed Oct 5 13:50:54 2016 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 5 Oct 2016 14:50:54 +0100 Subject: RFC on 8165823: (se) EPollArrayWrapper throws NPE if limits.conf set to 65536 and fd=65536 In-Reply-To: <562286F8-1828-4823-8EB1-BA9717E2BE32@oracle.com> References: <562286F8-1828-4823-8EB1-BA9717E2BE32@oracle.com> Message-ID: <680eab58-4cbb-5cd7-d33a-c79b0d552643@oracle.com> On 04/10/2016 19:04, Brian Burkhalter wrote: > The jdk7u and jdk9-dev implementatinos of EPollArrayWrapper.java are > identical aside from the more recent copyright year in the latter. > Thanks for checking, I didn't realize all the changes in this area had been back-ported. I've looked at the bug report and it is a bit confusing. It was originally reported as being observed with 8u102 but the java -version output is a build from jdk7u-dev I think. In any case, if ulimit -Hn is 64k then the highest file descriptor will be 65535. You may be right that they are change the limit dynamic, I can't otherwise explain the exception. -Alan. From brian.burkhalter at oracle.com Wed Oct 5 15:37:23 2016 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 5 Oct 2016 08:37:23 -0700 Subject: RFC on 8165823: (se) EPollArrayWrapper throws NPE if limits.conf set to 65536 and fd=65536 In-Reply-To: <680eab58-4cbb-5cd7-d33a-c79b0d552643@oracle.com> References: <562286F8-1828-4823-8EB1-BA9717E2BE32@oracle.com> <680eab58-4cbb-5cd7-d33a-c79b0d552643@oracle.com> Message-ID: On Oct 5, 2016, at 6:50 AM, Alan Bateman wrote: > On 04/10/2016 19:04, Brian Burkhalter wrote: >> The jdk7u and jdk9-dev implementatinos of EPollArrayWrapper.java are identical aside from the more recent copyright year in the latter. >> > Thanks for checking, I didn't realize all the changes in this area had been back-ported. I've looked at the bug report and it is a bit confusing. It was originally reported as being observed with 8u102 but the java -version output is a build from jdk7u-dev I think. Indeed it is confusing. It was also originally reported against OS X which it is clearly not. > In any case, if ulimit -Hn is 64k then the highest file descriptor will be 65535. You may be right that they are change the limit dynamic, I can't otherwise explain the exception. I?m not sure what to do about it other than something like this: http://cr.openjdk.java.net/~bpb/8165823/webrev.00/ Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Wed Oct 5 23:31:33 2016 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 5 Oct 2016 16:31:33 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AA513D886@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5134F66@ORSMSX113.amr.corp.intel.com> <062c589c-fa54-1652-683a-4e779816391d@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <45C5D272-E70A-40D9-8166-48ADD43763D9@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513D886@ORSMSX113.amr.corp.intel.com> Message-ID: Hi Lucy, On Oct 4, 2016, at 5:35 PM, Lu, Yingqi wrote: > Thank you very much for reviewing the patch and providing the detailed error/warning messages. They are very helpful! You are welcome. > We will look into the issues and solve them in the next version of the patch. Test cases for O_DIRECT and modified copyright year will be added in next version too. Given that the functionality of O_DIRECT on Linux appears to be supported by other interfaces on OS X, Solaris, and Windows, I wonder whether the patch will need to be refactored in some way to accommodate these other operating systems? For reference it looks as if direct I/O on OS X uses the F_NOCACHE command of fcntl(2) [1] (although per some online comments this might have some problems), Solaris uses the advice argument of directio(3c) [2], and Windows uses a combination of flags passed to CreateFile() [3, 4]. Thanks, Brian [1] https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man2/fcntl.2.html [2] https://docs.oracle.com/cd/E26505_01/html/816-5168/directio-3c.html [3] https://support.microsoft.com/en-us/kb/99794 [4] https://msdn.microsoft.com/en-us/library/windows/desktop/aa363858(v=vs.85).aspx -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Oct 6 13:41:22 2016 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 6 Oct 2016 14:41:22 +0100 Subject: RFC on 8165823: (se) EPollArrayWrapper throws NPE if limits.conf set to 65536 and fd=65536 In-Reply-To: References: <562286F8-1828-4823-8EB1-BA9717E2BE32@oracle.com> <680eab58-4cbb-5cd7-d33a-c79b0d552643@oracle.com> Message-ID: <73d756cd-81f9-e8e8-d96b-0f84e87195f8@oracle.com> On 05/10/2016 16:37, Brian Burkhalter wrote: > On Oct 5, 2016, at 6:50 AM, Alan Bateman > wrote: > >> On 04/10/2016 19:04, Brian Burkhalter wrote: >>> The jdk7u and jdk9-dev implementatinos of EPollArrayWrapper.java are >>> identical aside from the more recent copyright year in the latter. >>> >> Thanks for checking, I didn't realize all the changes in this area >> had been back-ported. I've looked at the bug report and it is a bit >> confusing. It was originally reported as being observed with 8u102 >> but the java -version output is a build from jdk7u-dev I think. > > Indeed it is confusing. It was also originally reported against OS X > which it is clearly not. Maybe the simplest thing is to reach out to the original submitter to see if they are changing the limit while the VM is running. That would at least confirm whether this is the issue or not. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Thu Oct 6 15:16:55 2016 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 6 Oct 2016 08:16:55 -0700 Subject: RFC on 8165823: (se) EPollArrayWrapper throws NPE if limits.conf set to 65536 and fd=65536 In-Reply-To: <73d756cd-81f9-e8e8-d96b-0f84e87195f8@oracle.com> References: <562286F8-1828-4823-8EB1-BA9717E2BE32@oracle.com> <680eab58-4cbb-5cd7-d33a-c79b0d552643@oracle.com> <73d756cd-81f9-e8e8-d96b-0f84e87195f8@oracle.com> Message-ID: On Oct 6, 2016, at 6:41 AM, Alan Bateman wrote: > Maybe the simplest thing is to reach out to the original submitter to see if they are changing the limit while the VM is running. That would at least confirm whether this is the issue or not. OK I will contact them. Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From christoph.langer at sap.com Thu Oct 6 15:43:37 2016 From: christoph.langer at sap.com (Langer, Christoph) Date: Thu, 6 Oct 2016 15:43:37 +0000 Subject: RFR(L): 8167295: Contribute further changes from SAP to native parts of libnet/libnio Message-ID: <39b854eaa42a4417a401ff52633d3d1b@DEWDFE13DE11.global.corp.sap> Hi, I'm looking to contribute a few of our diffs in the network area to OpenJDK. This is the bug: https://bugs.openjdk.java.net/browse/JDK-8167295 This is the webrev: http://cr.openjdk.java.net/~clanger/webrevs/8167295.0/ Besides minor cleanups, initializations and fixes, the main thing that I'm proposing is the unification of the union datatype SOCKADDR (unix) and SOCKETADDRESS (windows). I think the definition for all platforms should basically look like the following, of course depending on IPv6 support and the local datatypes: typedef union { struct sockaddr him; struct sockaddr_in him4; struct sockaddr_in6 him6; } SOCKETADDRESS; The type 'SOCKADDR' is already defined on Windows so we should consistently use 'SOCKETADDRESS'. This move would allow for better writing of shared code dealing with sockaddr structures, which we do at SAP especially for some tracing code. I don't know yet if it's a good idea to get rid of the definitions SOCKADDR_LEN resp. SOCKETADDRESS_LEN(x) and fully rely on sizeof(SOCKETADDRESS). I've done this in my webrev and it builds. But I'm not sure if especially some Windows APIs would behave strangely if passed in a longer length values for an AF_INET socket address. Maybe you have some comments on that point. Apart from that, I think it is a good idea to get rid of 'NET_AllocSockaddr' as there is no need to malloc SOCKETADDRESS fields at the few places where it was used (libnio unix only). A stack variable would probably be better and less error prone. And the declaration of NET_Wait can move to the common net_util.h header as the function is available everywhere with the same signature. Right now I'm just at a stage where my change builds on all platforms and I need to do some further testing. But I'm hoping for some early comments :) Thanks and best regards Christoph -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Fri Oct 7 00:18:37 2016 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 6 Oct 2016 17:18:37 -0700 Subject: RFC on FileStore and /proc/mounts on Linux in a chroot jail, Docker overlayfs and with btrfs Message-ID: I would be interested in any ideas anyone would like to share on this topic. There are three related issues [1, 2, 3] which have been filed. The most generic issue is [3] which has been observed in three different contexts: A) chroot environment [1] B) Docker container with overlay and overlay2 storage drivers [2] C) btrfs file system with an unmounted sub-volume [2] In all cases the failure was the IOException with message ?Mount point not found? at line 91 in LinuxFileStore::findMountEntry [4]. This occurred because the directory identified in step 2 as the mount point of the path was not found in /proc/mounts in step 3. The mount point is either the root (/) of the path or the longest sub-path which has a different device ID from its parent. The device ID is obtained from the st_dev member of the struct stat populated by stat(2) [5]. In the chroot environment the contents of /proc/mounts was just none /proc proc rw,relatime 0 0 and the mount point was determined to be ?/? which is not contained in /proc/mounts so the code falls through to the IOException. In the overlayfs cases, when step 2 breaks out of the parent traversal loop, the device ID of the given path is valid in the operating system file system, but the device ID of the parent is not and apparently corresponds instead to a device ID in the overlayfs itself. As there is no entry in /proc/mounts for the identified mount point we get the IOException. In the btrfs unmounted sub-volume case, for example, a btrfs file system is mounted on /mnt and contains a sub-volume /mnt/new_subvol which is not mounted. If an attempt is made to obtain the FileStore for example corresponding to /mnt/new_subvol/junk/junk.txt, step 2 traverses upwards and breaks when it finds that the device IDs of /mnt and /mnt/new_subvol differ. As /mnt/new_subvol is not in /proc/mounts the exception ensues. In thinking about this I was wondering whether it could be solved by creating some ?spoofed? mount entry not in /proc/mounts. This however does not seem like an inherently great idea and would probably create problems elsewhere. Viewing it from a different angle, one could instead say that the chroot jail and btrfs sub-volume problems are actually due to incorrect or incomplete configurations and this is not an issue with findMountEntry() at all. The overlayfs problem could be viewed as a bug in the overlay and overlay2 storage driver implementations. This problem for example does not manifest when the aufs storage driver is used with Docker. Comments and suggestions welcome. Thanks, Brian [1] https://bugs.openjdk.java.net/browse/JDK-8165323 - cannot get FileStore in chroot environment (*) [2] https://bugs.openjdk.java.net/browse/JDK-8165852 - cannot get FileStore for a file in overlayfs in Docker [3] https://bugs.openjdk.java.net/browse/JDK-8166162 - cannot get FileStore if device of path is not in /proc/mounts [4] http://hg.openjdk.java.net/jdk9/dev/jdk/file/65042b713b12/src/java.base/linux/classes/sun/nio/fs/LinuxFileStore.java [5] https://linux.die.net/man/2/stat (*) The immediate problem caused by [1] was fixed and [3] was filed to track the underlying cause. From christoph.langer at sap.com Fri Oct 7 15:17:05 2016 From: christoph.langer at sap.com (Langer, Christoph) Date: Fri, 7 Oct 2016 15:17:05 +0000 Subject: RFR(L): 8167295: Further cleanup to the native parts of libnet/libnio Message-ID: <5d9ce3b78da9480490e36bcea3159cb2@DEWDFE13DE11.global.corp.sap> Hi again, I have respun my patch a little bit: http://cr.openjdk.java.net/~clanger/webrevs/8167295.1/ The reason is that the naming of the members of SOCKETADDRESS should be changed to 'sa' instead of 'him' as the fields are of type 'struct sockaddr...'. I also did a careful inspection of the places where SOCKADDR_LEN resp. SOCKETADDRESS_LEN(x) used to be used and found that it should really be ok if this define is completely dropped and always sizeof(SOCKETADDRESS) at these places. Basic testing showed that the changes were ok - I'll add the patch to the queue for our nightly build/test runs and check for regressions... Thanks Christoph From: Langer, Christoph Sent: Donnerstag, 6. Oktober 2016 17:44 To: net-dev at openjdk.java.net Cc: nio-dev at openjdk.java.net Subject: RFR(L): 8167295: Contribute further changes from SAP to native parts of libnet/libnio Hi, I'm looking to contribute a few of our diffs in the network area to OpenJDK. This is the bug: https://bugs.openjdk.java.net/browse/JDK-8167295 This is the webrev: http://cr.openjdk.java.net/~clanger/webrevs/8167295.0/ Besides minor cleanups, initializations and fixes, the main thing that I'm proposing is the unification of the union datatype SOCKADDR (unix) and SOCKETADDRESS (windows). I think the definition for all platforms should basically look like the following, of course depending on IPv6 support and the local datatypes: typedef union { struct sockaddr him; struct sockaddr_in him4; struct sockaddr_in6 him6; } SOCKETADDRESS; The type 'SOCKADDR' is already defined on Windows so we should consistently use 'SOCKETADDRESS'. This move would allow for better writing of shared code dealing with sockaddr structures, which we do at SAP especially for some tracing code. I don't know yet if it's a good idea to get rid of the definitions SOCKADDR_LEN resp. SOCKETADDRESS_LEN(x) and fully rely on sizeof(SOCKETADDRESS). I've done this in my webrev and it builds. But I'm not sure if especially some Windows APIs would behave strangely if passed in a longer length values for an AF_INET socket address. Maybe you have some comments on that point. Apart from that, I think it is a good idea to get rid of 'NET_AllocSockaddr' as there is no need to malloc SOCKETADDRESS fields at the few places where it was used (libnio unix only). A stack variable would probably be better and less error prone. And the declaration of NET_Wait can move to the common net_util.h header as the function is available everywhere with the same signature. Right now I'm just at a stage where my change builds on all platforms and I need to do some further testing. But I'm hoping for some early comments :) Thanks and best regards Christoph -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.hegarty at oracle.com Mon Oct 10 12:07:51 2016 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Mon, 10 Oct 2016 13:07:51 +0100 Subject: RFR(L): 8167295: Further cleanup to the native parts of libnet/libnio In-Reply-To: <5d9ce3b78da9480490e36bcea3159cb2@DEWDFE13DE11.global.corp.sap> References: <5d9ce3b78da9480490e36bcea3159cb2@DEWDFE13DE11.global.corp.sap> Message-ID: <53a7ee60-341c-60b1-3910-4e1bb0dd2798@oracle.com> Hi Christoph, On 07/10/16 16:17, Langer, Christoph wrote: > Hi again, > > I have respun my patch a little bit: > http://cr.openjdk.java.net/~clanger/webrevs/8167295.1/ This is a nice cleanup and an improvement to the code. Specifically, adding 'struct sockaddr sa' to SOCKETADDRESS allows for the removal of many cast operations. One minor double space I noticed when reviewing the changes. Net.c 339 sa.sa4.sin_len = sizeof(struct sockaddr_in); ^^ My eyes hurt from this kind of review, e.g. addrs->addr.sa6.sin6_addr.s6_addr!!! so I put your patch through our internal build and test system too. All looks good. -Chris. > The reason is that the naming of the members of SOCKETADDRESS should be > changed to ?sa? instead of ?him? as the fields are of type ?struct > sockaddr??. I also did a careful inspection of the places where > SOCKADDR_LEN resp. SOCKETADDRESS_LEN(x) used to be used and found that > it should really be ok if this define is completely dropped and always > sizeof(SOCKETADDRESS) at these places. Basic testing showed that the > changes were ok ? I?ll add the patch to the queue for our nightly > build/test runs and check for regressions? > > > > Thanks > > Christoph > > > > > > *From:*Langer, Christoph > *Sent:* Donnerstag, 6. Oktober 2016 17:44 > *To:* net-dev at openjdk.java.net > *Cc:* nio-dev at openjdk.java.net > *Subject:* RFR(L): 8167295: Contribute further changes from SAP to > native parts of libnet/libnio > > > > Hi, > > > > I?m looking to contribute a few of our diffs in the network area to OpenJDK. > > > > This is the bug: https://bugs.openjdk.java.net/browse/JDK-8167295 > > This is the webrev: http://cr.openjdk.java.net/~clanger/webrevs/8167295.0/ > > > > Besides minor cleanups, initializations and fixes, the main thing that > I?m proposing is the unification of the union datatype SOCKADDR (unix) > and SOCKETADDRESS (windows). > > > > I think the definition for all platforms should basically look like the > following, of course depending on IPv6 support and the local datatypes: > > > > typedef union { > > struct sockaddr him; > > struct sockaddr_in him4; > > struct sockaddr_in6 him6; > > } SOCKETADDRESS; > > > > The type ?SOCKADDR? is already defined on Windows so we should > consistently use ?SOCKETADDRESS?. This move would allow for better > writing of shared code dealing with sockaddr structures, which we do at > SAP especially for some tracing code. > > > > I don?t know yet if it?s a good idea to get rid of the definitions > SOCKADDR_LEN resp. SOCKETADDRESS_LEN(x) and fully rely on > sizeof(SOCKETADDRESS). I?ve done this in my webrev and it builds. But > I?m not sure if especially some Windows APIs would behave strangely if > passed in a longer length values for an AF_INET socket address. Maybe > you have some comments on that point. > > > > Apart from that, I think it is a good idea to get rid of > ?NET_AllocSockaddr? as there is no need to malloc SOCKETADDRESS fields > at the few places where it was used (libnio unix only). A stack variable > would probably be better and less error prone. And the declaration of > NET_Wait can move to the common net_util.h header as the function is > available everywhere with the same signature. > > > > Right now I?m just at a stage where my change builds on all platforms > and I need to do some further testing. But I?m hoping for some early > comments J > > > > Thanks and best regards > > Christoph > > > From christoph.langer at sap.com Mon Oct 10 20:51:48 2016 From: christoph.langer at sap.com (Langer, Christoph) Date: Mon, 10 Oct 2016 20:51:48 +0000 Subject: RFR(L): 8167295: Further cleanup to the native parts of libnet/libnio In-Reply-To: <53a7ee60-341c-60b1-3910-4e1bb0dd2798@oracle.com> References: <5d9ce3b78da9480490e36bcea3159cb2@DEWDFE13DE11.global.corp.sap> <53a7ee60-341c-60b1-3910-4e1bb0dd2798@oracle.com> Message-ID: <1e35891dffd040c9ba582a6c5bef7e01@DEWDFE13DE11.global.corp.sap> Hi Chris, thanks for the review. We couldn't observe issues in our OpenJDK test systems and most of the change was part of the SAP JVM already for quite some time. So I pushed it just now: http://hg.openjdk.java.net/jdk9/dev/jdk/rev/d4f70e7859c7 Some more cleanup is still to come... :) Best regards Christoph > -----Original Message----- > From: Chris Hegarty [mailto:chris.hegarty at oracle.com] > Sent: Montag, 10. Oktober 2016 14:08 > To: Langer, Christoph ; net-dev at openjdk.java.net > Cc: nio-dev at openjdk.java.net > Subject: Re: RFR(L): 8167295: Further cleanup to the native parts of > libnet/libnio > > Hi Christoph, > > On 07/10/16 16:17, Langer, Christoph wrote: > > Hi again, > > > > I have respun my patch a little bit: > > http://cr.openjdk.java.net/~clanger/webrevs/8167295.1/ > > This is a nice cleanup and an improvement to the code. Specifically, > adding 'struct sockaddr sa' to SOCKETADDRESS allows for the removal > of many cast operations. > > One minor double space I noticed when reviewing the changes. > > Net.c > 339 sa.sa4.sin_len = sizeof(struct sockaddr_in); > ^^ > > My eyes hurt from this kind of review, e.g. > addrs->addr.sa6.sin6_addr.s6_addr!!! so I put your patch through > our internal build and test system too. All looks good. > > -Chris. > > > > > The reason is that the naming of the members of SOCKETADDRESS should be > > changed to 'sa' instead of 'him' as the fields are of type 'struct > > sockaddr...'. I also did a careful inspection of the places where > > SOCKADDR_LEN resp. SOCKETADDRESS_LEN(x) used to be used and found > that > > it should really be ok if this define is completely dropped and always > > sizeof(SOCKETADDRESS) at these places. Basic testing showed that the > > changes were ok - I'll add the patch to the queue for our nightly > > build/test runs and check for regressions... > > > > > > > > Thanks > > > > Christoph > > > > > > > > > > > > *From:*Langer, Christoph > > *Sent:* Donnerstag, 6. Oktober 2016 17:44 > > *To:* net-dev at openjdk.java.net > > *Cc:* nio-dev at openjdk.java.net > > *Subject:* RFR(L): 8167295: Contribute further changes from SAP to > > native parts of libnet/libnio > > > > > > > > Hi, > > > > > > > > I'm looking to contribute a few of our diffs in the network area to OpenJDK. > > > > > > > > This is the bug: https://bugs.openjdk.java.net/browse/JDK-8167295 > > > > This is the webrev: http://cr.openjdk.java.net/~clanger/webrevs/8167295.0/ > > > > > > > > Besides minor cleanups, initializations and fixes, the main thing that > > I'm proposing is the unification of the union datatype SOCKADDR (unix) > > and SOCKETADDRESS (windows). > > > > > > > > I think the definition for all platforms should basically look like the > > following, of course depending on IPv6 support and the local datatypes: > > > > > > > > typedef union { > > > > struct sockaddr him; > > > > struct sockaddr_in him4; > > > > struct sockaddr_in6 him6; > > > > } SOCKETADDRESS; > > > > > > > > The type 'SOCKADDR' is already defined on Windows so we should > > consistently use 'SOCKETADDRESS'. This move would allow for better > > writing of shared code dealing with sockaddr structures, which we do at > > SAP especially for some tracing code. > > > > > > > > I don't know yet if it's a good idea to get rid of the definitions > > SOCKADDR_LEN resp. SOCKETADDRESS_LEN(x) and fully rely on > > sizeof(SOCKETADDRESS). I've done this in my webrev and it builds. But > > I'm not sure if especially some Windows APIs would behave strangely if > > passed in a longer length values for an AF_INET socket address. Maybe > > you have some comments on that point. > > > > > > > > Apart from that, I think it is a good idea to get rid of > > 'NET_AllocSockaddr' as there is no need to malloc SOCKETADDRESS fields > > at the few places where it was used (libnio unix only). A stack variable > > would probably be better and less error prone. And the declaration of > > NET_Wait can move to the common net_util.h header as the function is > > available everywhere with the same signature. > > > > > > > > Right now I'm just at a stage where my change builds on all platforms > > and I need to do some further testing. But I'm hoping for some early > > comments J > > > > > > > > Thanks and best regards > > > > Christoph > > > > > > From brian.burkhalter at oracle.com Tue Oct 11 18:22:35 2016 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 11 Oct 2016 11:22:35 -0700 Subject: RFC on 8165823: (se) EPollArrayWrapper throws NPE if limits.conf set to 65536 and fd=65536 In-Reply-To: References: <783e980056154a39a9381d8daa446aa3@DEWDFE13DE11.global.corp.sap> Message-ID: <2CAD7459-E810-4BA1-B493-64C6292FB911@oracle.com> Hello Thomas, Christoph, On Oct 10, 2016, at 12:34 AM, Thomas St?fe wrote: > The only input I could give is that on some platforms getrlimit() may return RLIM_INFINITY for NOFILE, which has to be handled. But I do not think this is the case on Linux. The RLIM_INFINITY case is handled in the native layer [1] therefore this should not be a problem in this situation: JNIEXPORT jint JNICALL Java_sun_nio_ch_IOUtil_fdLimit(JNIEnv *env, jclass this) { struct rlimit rlp; if (getrlimit(RLIMIT_NOFILE, &rlp) < 0) { JNU_ThrowIOExceptionWithLastError(env, "getrlimit failed"); return -1; } if (rlp.rlim_max == RLIM_INFINITY || rlp.rlim_max > (rlim_t)java_lang_Integer_MAX_VALUE) { return java_lang_Integer_MAX_VALUE; } else { return (jint)rlp.rlim_max; } } Thanks, Brian [1] http://hg.openjdk.java.net/jdk9/dev/jdk/file/b909daf8fdbc/src/java.base/unix/native/libnio/ch/IOUtil.c, lines 124-138 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mkanat at google.com Tue Oct 11 23:30:21 2016 From: mkanat at google.com (Max Kanat-Alexander) Date: Tue, 11 Oct 2016 16:30:21 -0700 Subject: Files.copy does userspace copies but FileChannel uses sendfile Message-ID: java.nio.file.Files.copy uses this code, ultimately, to copy files: http://hg.openjdk.java.net/jdk9/dev/jdk/file/tip/src/ java.base/unix/native/libnio/fs/UnixCopyFile.c I was surprised that that didn't use sendfile when it's available. I was even more surprised that there is already another copying implementation in the JDK that _does_ use sendfile: http://hg.openjdk.java.net/jdk9/dev/jdk/file/b909daf8fdbc/src/java.base/ unix/native/libnio/ch/FileChannelImpl.c#l164 Should these be merged? Or should the Java-side implementation of Files.copy be changed to use FileChannel underneath? -Max -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Wed Oct 12 00:41:35 2016 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 11 Oct 2016 17:41:35 -0700 Subject: Files.copy does userspace copies but FileChannel uses sendfile In-Reply-To: References: Message-ID: <3C609808-05BB-4B8F-85EA-6F8733CCC273@oracle.com> Hi Max, I don?t know the historical context for this design, but after looking over the code a bit it seems that you have a point, at least for the case where Files.copy() is a non-interruptible file copy. For a non-interruptible file copy I do not see why sendfile() or its equivalent on the Unix platform in question could not be used. Whether a file copy is interruptible is set by an ExtendedOption in the CopyOption passed to [1]. If the foregoing is correct, the refactoring might better be done by moving the native transferTo0() functionality to UnixNativeDispatcher and having FileChannelImpl.transferToDirectlyInternal() and the non-interruptible case of Files.copy() call it. Thanks, Brian [1] http://download.java.net/java/jdk9/docs/api/java/nio/file/Files.html#copy-java.nio.file.Path-java.nio.file.Path-java.nio.file.CopyOption...- On Oct 11, 2016, at 4:30 PM, Max Kanat-Alexander wrote: > java.nio.file.Files.copy uses this code, ultimately, to copy files: > > http://hg.openjdk.java.net/jdk9/dev/jdk/file/tip/src/java.base/unix/native/libnio/fs/UnixCopyFile.c > > I was surprised that that didn't use sendfile when it's available. I was even more surprised that there is already another copying implementation in the JDK that _does_ use sendfile: > > http://hg.openjdk.java.net/jdk9/dev/jdk/file/b909daf8fdbc/src/java.base/unix/native/libnio/ch/FileChannelImpl.c#l164 > > Should these be merged? Or should the Java-side implementation of Files.copy be changed to use FileChannel underneath? > > -Max -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Wed Oct 12 05:41:53 2016 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 12 Oct 2016 06:41:53 +0100 Subject: Files.copy does userspace copies but FileChannel uses sendfile In-Reply-To: References: Message-ID: On 12/10/2016 00:30, Max Kanat-Alexander wrote: > : > > Should these be merged? Or should the Java-side implementation of > Files.copy be changed to use FileChannel underneath? > > It's been a copy while the performance was checked but the last time then sendfile was great for file <--> socket but we weren't see any benefit for file <--> file. Kernels improve and there are many different file systems so it probably should be remeasured. If there are benefits to the FileSystemProvider to use sendfile then it could do that. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Wed Oct 12 13:50:47 2016 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 12 Oct 2016 14:50:47 +0100 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5134F66@ORSMSX113.amr.corp.intel.com> <062c589c-fa54-1652-683a-4e779816391d@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> Message-ID: On 04/10/2016 18:24, Lu, Yingqi wrote: > Hi Christoph, > > Thank you very much for trying our patch. > > We are still seeking the feedback from the community. When we get closer to the final version of the patch, we will modify the copyright years. Thank you for reminding us! > > Anyone else has any feedback/comments? > Lucy, I've looked through the changes and have a number of concerns. The main concern is that I/O with files opened for O_DIRECT will usually require the buffer to be page aligned and the buffer sized to be a multiple of the file system block size. The patch deals with the alignment by doing a posix_memalign + copy + free for every I/O operation. I don't know if you've tried doing a write with a ByteBuffer backed with an array in the heap but if I read the patch correctly then you'll end up copying it twice, once into a thread local direct buffer and then a second time into the temporary allocated buffer. Ditto for read where it will initially read it into a temporary allocated buffer, then into a thread local direct buffer, and finally into the user's buffer backed by an array in the heap. In the past when we've looked at this issue then the conclusion was that there needed to be a way to get a ByteBuffer that is appropriate aligned and also a way to get information about the file system to size I/O operations too. It is possible to do it transparently (like you are doing) but it requires the thread local buffer cache support aligned buffers. Aside from that then the other concern is addding a direct flag to FileDescriptor. I would think it should be simpler to capture this in FileChannelImpl constructor and use that to send the I/O through the appropriate path (this avoids have the methods in IOUtils fan out). It may have been mentioned in other mails but there aren't any tests in the patch and I think this will require a good set of tests in order to be confident with all the cases. -Alan From martinrb at google.com Wed Oct 12 16:22:03 2016 From: martinrb at google.com (Martin Buchholz) Date: Wed, 12 Oct 2016 09:22:03 -0700 Subject: Files.copy does userspace copies but FileChannel uses sendfile In-Reply-To: References: Message-ID: sendfile(2) says sendfile() copies data between one file descriptor and another. Because this copying is done within the kernel, sendfile() is more efficient than the combination of read(2) and write(2), which would require transferring data to and from user space. The advantage of read/write is portability, but we've already paid the price and are already using sendfile. The savings with a particular pair of fds may not be great, but in principle we should always be using sendfile when we can. On Tue, Oct 11, 2016 at 10:41 PM, Alan Bateman wrote: > On 12/10/2016 00:30, Max Kanat-Alexander wrote: > > : > > Should these be merged? Or should the Java-side implementation of > Files.copy be changed to use FileChannel underneath? > > > It's been a copy while the performance was checked but the last time then > sendfile was great for file <--> socket but we weren't see any benefit for > file <--> file. Kernels improve and there are many different file systems > so it probably should be remeasured. If there are benefits to the > FileSystemProvider to use sendfile then it could do that. > > -Alan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Wed Oct 12 16:30:19 2016 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 12 Oct 2016 09:30:19 -0700 Subject: Files.copy does userspace copies but FileChannel uses sendfile In-Reply-To: References: Message-ID: It seems as if this is a good candidate for some JMH testing from the Java layer. Brian On Oct 12, 2016, at 9:22 AM, Martin Buchholz wrote: > sendfile(2) says > > sendfile() copies data between one file descriptor and another. > Because this copying is done within the kernel, sendfile() is more > efficient than the combination of read(2) and write(2), which would > require transferring data to and from user space. > > The advantage of read/write is portability, but we've already paid the price and are already using sendfile. The savings with a particular pair of fds may not be great, but in principle we should always be using sendfile when we can. > > > On Tue, Oct 11, 2016 at 10:41 PM, Alan Bateman wrote: > On 12/10/2016 00:30, Max Kanat-Alexander wrote: >> : >> >> Should these be merged? Or should the Java-side implementation of Files.copy be changed to use FileChannel underneath? >> >> > It's been a copy while the performance was checked but the last time then sendfile was great for file <--> socket but we weren't see any benefit for file <--> file. Kernels improve and there are many different file systems so it probably should be remeasured. If there are benefits to the FileSystemProvider to use sendfile then it could do that. > > -Alan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Wed Oct 12 16:31:39 2016 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 12 Oct 2016 17:31:39 +0100 Subject: Files.copy does userspace copies but FileChannel uses sendfile In-Reply-To: References: Message-ID: <264e6f6d-9ffe-a6e2-a254-5c3068aa92e6@oracle.com> On 12/10/2016 17:22, Martin Buchholz wrote: > sendfile(2) says > > sendfile() copies data between one file descriptor and another. > Because this copying is done within the kernel, sendfile() is more > efficient than the combination of read(2) and write(2), which would > require transferring data to and from user space. > > The advantage of read/write is portability, but we've already paid the > price and are already using sendfile. The savings with a particular > pair of fds may not be great, but in principle we should always be > using sendfile when we can. > I don't disagree, it' just that historically sendfile was for file <-> socket and there wasn't support for file <-> file on all platforms. In this case, it shouldn't be hard to try out using sendfile for the regular file copy and then get some data on performance for a number of different platforms and file systems. -Alan From Alan.Bateman at oracle.com Wed Oct 12 16:39:59 2016 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 12 Oct 2016 17:39:59 +0100 Subject: Files.copy does userspace copies but FileChannel uses sendfile In-Reply-To: References: Message-ID: <5ea31616-89ba-9d38-e3f7-2cfe799f716c@oracle.com> On 12/10/2016 17:30, Brian Burkhalter wrote: > It seems as if this is a good candidate for some JMH testing from the > Java layer. Yes although you also have to measure the copy when the source isn't in the file system cache so that means dropping caches between copies. -Alan From yingqi.lu at intel.com Wed Oct 12 16:41:38 2016 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Wed, 12 Oct 2016 16:41:38 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5134F66@ORSMSX113.amr.corp.intel.com> <062c589c-fa54-1652-683a-4e779816391d@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AA51447D9@ORSMSX113.amr.corp.intel.com> Hi Alan, Thank you for your feedback. The reason we added "direct" flag to the FileDescriptor class is to follow the example of "append" flag. We can surely change the "direct" check to FileChannelImpl constructor, should be easy to do. You are correct about the "extra copy" with DirectIO. Will it be acceptable if we add a function "Util.getAlignedTemporaryDirectBuffer" and use that for the DirectIO operation? In this case, I think we should be able to avoid the additional copy? Thanks, Lucy -----Original Message----- From: Alan Bateman [mailto:Alan.Bateman at oracle.com] Sent: Wednesday, October 12, 2016 6:51 AM To: Lu, Yingqi Cc: nio-dev at openjdk.java.net; Kaczmarek, Eric ; Kharbas, Kishor Subject: Re: Proposal for adding O_DIRECT support into JDK 9 On 04/10/2016 18:24, Lu, Yingqi wrote: > Hi Christoph, > > Thank you very much for trying our patch. > > We are still seeking the feedback from the community. When we get closer to the final version of the patch, we will modify the copyright years. Thank you for reminding us! > > Anyone else has any feedback/comments? > Lucy, I've looked through the changes and have a number of concerns. The main concern is that I/O with files opened for O_DIRECT will usually require the buffer to be page aligned and the buffer sized to be a multiple of the file system block size. The patch deals with the alignment by doing a posix_memalign + copy + free for every I/O operation. I don't know if you've tried doing a write with a ByteBuffer backed with an array in the heap but if I read the patch correctly then you'll end up copying it twice, once into a thread local direct buffer and then a second time into the temporary allocated buffer. Ditto for read where it will initially read it into a temporary allocated buffer, then into a thread local direct buffer, and finally into the user's buffer backed by an array in the heap. In the past when we've looked at this issue then the conclusion was that there needed to be a way to get a ByteBuffer that is appropriate aligned and also a way to get information about the file system to size I/O operations too. It is possible to do it transparently (like you are doing) but it requires the thread local buffer cache support aligned buffers. Aside from that then the other concern is addding a direct flag to FileDescriptor. I would think it should be simpler to capture this in FileChannelImpl constructor and use that to send the I/O through the appropriate path (this avoids have the methods in IOUtils fan out). It may have been mentioned in other mails but there aren't any tests in the patch and I think this will require a good set of tests in order to be confident with all the cases. -Alan From brian.burkhalter at oracle.com Wed Oct 12 17:56:35 2016 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 12 Oct 2016 10:56:35 -0700 Subject: Files.copy does userspace copies but FileChannel uses sendfile In-Reply-To: <5ea31616-89ba-9d38-e3f7-2cfe799f716c@oracle.com> References: <5ea31616-89ba-9d38-e3f7-2cfe799f716c@oracle.com> Message-ID: <004B903D-0344-47EE-95C5-E7D85EC90788@oracle.com> On Oct 12, 2016, at 9:39 AM, Alan Bateman wrote: > On 12/10/2016 17:30, Brian Burkhalter wrote: > >> It seems as if this is a good candidate for some JMH testing from the Java layer. > Yes although you also have to measure the copy when the source isn't in the file system cache so that means dropping caches between copies. That should be feasible given the control available within JMH. Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From christoph.langer at sap.com Thu Oct 13 06:19:23 2016 From: christoph.langer at sap.com (Langer, Christoph) Date: Thu, 13 Oct 2016 06:19:23 +0000 Subject: AF_INET6 support Message-ID: <964200b6d1bf43f6be476d531f00d3fb@DEWDFE13DE11.global.corp.sap> Hi networking experts, I've got a question regarding AF_INET6. In jdk native code you'll still find lots of code guarded by "#ifdef AF_INET6". I'm wondering if there are still supported build environments where AF_INET6 is not defined. Or is it time now to assume AF_INET6 and remove this guarding? Here at SAP we don't support non AF_INET6 build environments for quite a long time already. But probably there are scenarios out in the Java world where only IPv4 builds are done?? Maybe you can shed some light on this and/or give your opinion? Thanks Christoph -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.hegarty at oracle.com Thu Oct 13 08:28:12 2016 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 13 Oct 2016 09:28:12 +0100 Subject: AF_INET6 support In-Reply-To: <964200b6d1bf43f6be476d531f00d3fb@DEWDFE13DE11.global.corp.sap> References: <964200b6d1bf43f6be476d531f00d3fb@DEWDFE13DE11.global.corp.sap> Message-ID: <098BBC2F-4535-4A66-BB05-BF67E5E3977F@oracle.com> Christoph, > On 13 Oct 2016, at 07:19, Langer, Christoph wrote: > > Hi networking experts, > > I?ve got a question regarding AF_INET6. > > In jdk native code you?ll still find lots of code guarded by ?#ifdef AF_INET6?. I?m wondering if there are still supported build environments where AF_INET6 is not defined. Or is it time now to assume AF_INET6 and remove this guarding? Here at SAP we don?t support non AF_INET6 build environments for quite a long time already. But probably there are scenarios out in the Java world where only IPv4 builds are done?? > > Maybe you can shed some light on this and/or give your opinion? A while back we did consider removing #ifdef AF_INET6, so that the code could be cleaned up and made more readable, but we never got around to it ( it was just lower priority than other tasks ). I do remember fixing, or sponsoring, a change in the last year or so, where an #ifdef AF_INET6 was missing, that make me think that it was good that we did not remove these ( i.e. the person that filed the bug had a good use-case for building without IPv6 support ). I?ll see if I can jog my memory by looking through history. -Chris. From Alan.Bateman at oracle.com Thu Oct 13 08:37:47 2016 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 13 Oct 2016 09:37:47 +0100 Subject: AF_INET6 support In-Reply-To: <964200b6d1bf43f6be476d531f00d3fb@DEWDFE13DE11.global.corp.sap> References: <964200b6d1bf43f6be476d531f00d3fb@DEWDFE13DE11.global.corp.sap> Message-ID: <881f96c1-cb2c-49f9-a0fc-91bea00dc509@oracle.com> On 13/10/2016 07:19, Langer, Christoph wrote: > > Hi networking experts, > > I?ve got a question regarding AF_INET6. > > In jdk native code you?ll still find lots of code guarded by ?#ifdef > AF_INET6?. I?m wondering if there are still supported build > environments where AF_INET6 is not defined. Or is it time now to > assume AF_INET6 and remove this guarding? Here at SAP we don?t support > non AF_INET6 build environments for quite a long time already. But > probably there are scenarios out in the Java world where only IPv4 > builds are done?? > > Maybe you can shed some light on this and/or give your opinion? > > It dates back to when IPv6 support was initially added and when there wasn't support on all platforms. Also there was a long period where it was supported to cross build to older (mostly embedded) platform that didn't have IPv6. I'm not aware of any now and don't see any reason not to clean this up. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From christoph.langer at sap.com Thu Oct 13 08:46:59 2016 From: christoph.langer at sap.com (Langer, Christoph) Date: Thu, 13 Oct 2016 08:46:59 +0000 Subject: AF_INET6 support In-Reply-To: <098BBC2F-4535-4A66-BB05-BF67E5E3977F@oracle.com> References: <964200b6d1bf43f6be476d531f00d3fb@DEWDFE13DE11.global.corp.sap> <098BBC2F-4535-4A66-BB05-BF67E5E3977F@oracle.com> Message-ID: <9e5d1e66416a4ac895457cff65bd4b49@DEWDFE13DE11.global.corp.sap> Hi Chris, > > I?ve got a question regarding AF_INET6. > > > > In jdk native code you?ll still find lots of code guarded by ?#ifdef AF_INET6?. > I?m wondering if there are still supported build environments where AF_INET6 > is not defined. Or is it time now to assume AF_INET6 and remove this guarding? > Here at SAP we don?t support non AF_INET6 build environments for quite a long > time already. But probably there are scenarios out in the Java world where only > IPv4 builds are done?? > > > > Maybe you can shed some light on this and/or give your opinion? > > A while back we did consider removing #ifdef AF_INET6, so that the > code could be cleaned up and made more readable, but we never got > around to it ( it was just lower priority than other tasks ). I do remember > fixing, or sponsoring, a change in the last year or so, where an #ifdef > AF_INET6 was missing, that make me think that it was good that we > did not remove these ( i.e. the person that filed the bug had a good > use-case for building without IPv6 support ). I?ll see if I can jog my > memory by looking through history. Ok, it would really be great if you could find more information about that case where #ifdef AF_INET6 was missing and it lead to problems. But if nobody really does IPv4 only builds and tests the results, I doubt that the IPv4 only scenario would work or even build. Then I think it was likely that somehow unguarded IPv6 code sneaks in or has already done so... In case it can be removed it would really make the code more readable in several places. So, as I'm still doing cleanups, I could also take care of removing those #ifdefs at the places where I'm going. Just let me know. Thanks Christoph From Alan.Burlison at oracle.com Thu Oct 13 12:19:43 2016 From: Alan.Burlison at oracle.com (Alan Burlison) Date: Thu, 13 Oct 2016 13:19:43 +0100 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5134F66@ORSMSX113.amr.corp.intel.com> <062c589c-fa54-1652-683a-4e779816391d@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <45C5D272-E70A-40D9-8166-48ADD43763D9@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513D886@ORSMSX113.amr.corp.intel.com> Message-ID: On 06/10/2016 00:31, Brian Burkhalter wrote: > Given that the functionality of O_DIRECT on Linux appears to be > supported by other interfaces on OS X, Solaris, and Windows, I wonder > whether the patch will need to be refactored in some way to > accommodate these other operating systems? For reference it looks as > if direct I/O on OS X uses the F_NOCACHE command of fcntl(2) [1] > (although per some online comments this might have some problems), > Solaris uses the advice argument of directio(3c) [2], and Windows > uses a combination of flags passed to CreateFile() [3, 4]. The Linux open(2) manpage contains a long list of warnings about O_DIRECT, including: ---------- In Linux alignment restrictions vary by filesystem and kernel version and might be absent entirely. However there is currently no filesystem-independent interface for an application to discover these restrictions for a given file or filesystem. ---------- ---------- O_DIRECT I/Os should never be run concurrently with the fork(2) system call, if the memory buffer is a private mapping (i.e., any mapping created with the mmap(2) MAP_PRIVATE flag; this includes memory allocated on the heap and statically allocated buffers). Any such I/Os, whether submitted via an asynchronous I/O interface or from another thread in the process, should be completed before fork(2) is called. Failure to do so can result in data corruption and undefined behavior in parent and child processes. ---------- ---------- Applications should avoid mixing O_DIRECT and normal I/O to the same file, and especially to overlapping byte regions in the same file. Even when the filesystem correctly handles the coherency issues in this situation, overall I/O throughput is likely to be slower than using either mode alone. Likewise, applications should avoid mixing mmap(2) of files with direct I/O to the same files. ---------- ---------- "The thing that has always disturbed me about O_DIRECT is that the whole interface is just stupid, and was probably designed by a deranged monkey on some serious mind-controlling substances." - Linus ---------- Adding support for O_DIRECT has a far wider impact than adding just another IO handle flag. As such I'm opposed to this change as it seems to be prone to cause hard-to-diagnose failures on Linux and it is also specific to just Linux. -- Alan Burlison -- From yingqi.lu at intel.com Thu Oct 13 22:37:07 2016 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Thu, 13 Oct 2016 22:37:07 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5134F66@ORSMSX113.amr.corp.intel.com> <062c589c-fa54-1652-683a-4e779816391d@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <45C5D272-E70A-40D9-8166-48ADD43763D9@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513D886@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AA5146652@ORSMSX113.amr.corp.intel.com> O_DIRECT is widely adopted in applications designed for high IO throughput, such as webservers and Databases. It bypasses filesystem cache and Linux readahead buffer which some time slow down the throughput and cause unpredictable IO performance. As an example, we recently measured on our Apache Cassandra database (one of the most popular distributed database systems written in Java) on default buffered IO, lowering readahead buffer from 128KB (default) to 8KB improves throughout by up to 4X. This is a typical usage for DirectIO and we expect to see even greater gains by doing that. Another example is we have enabled Hadoop Distributed File System (HDFS) with O_DIRECT through native calls and also measure significant performance gains running a cloud workload. It would be really important to have O_DIRECT supported by Java so that all Java based applications can take advantage of it. We agree on the other hand, O_DIRECT has certain limitations. That is why it is normally recommended to be used as a performance option. Our purpose is to enable it inside Java and provide the application writers a uniform and secure way to use it. All the limitations apply to native applications as well. 1. Regarding to alignment restriction, I cross checked with our kernel experts, and we think it is safe to align the memory buffer used by DirectIO to the kernel page size. 2. In term of having O_DIRECT I/O running concurrently with fork(2) on privately mapped memory buffer, the issue can be solved by creating DirectIO memory buffer using shmat(2) or mmap(2) with the MAP_SHARED flag. Alternatively, MADV_DONTFORK with madvise(2) can be used on the memory buffer to avoid the issue as well. Details can be find http://man7.org/linux/man-pages/man2/open.2.html 3. Combination of DirectIO and BufferedIO on the same file is not recommended for performance reason. However, we think application writers should be familiar with all of these before using it. Now, we are planning to do the following changes to the existing patch, hope it will be structured a little better this way :-) 1. Create a function to allocate aligned DirectByteBuffer and use it for Direct I/O (default is to use DirectByteBuffer as well, but not aligned). The buffer will be aligned to page boundary. There are some existing code in Direct-X-Buffer.java.template for VM.isDirectMemoryPageAligned. We will follow this as an example. We think this will address the "extra copy" issue in the last version of the patch. 2. Move all the changes to FileDispatcherImpl.c to Java level. 3. Remove the changes to FileDescriptor and do the DirectIO check inside FileChannelImpl. Thanks, Lucy >-----Original Message----- >From: Alan Burlison [mailto:Alan.Burlison at oracle.com] >Sent: Thursday, October 13, 2016 5:20 AM >To: Brian Burkhalter ; Lu, Yingqi > >Cc: Kharbas, Kishor ; nio-dev at openjdk.java.net; >core-libs-dev at openjdk.java.net; Kaczmarek, Eric >Subject: Re: Proposal for adding O_DIRECT support into JDK 9 > >On 06/10/2016 00:31, Brian Burkhalter wrote: > >> Given that the functionality of O_DIRECT on Linux appears to be >> supported by other interfaces on OS X, Solaris, and Windows, I wonder >> whether the patch will need to be refactored in some way to >> accommodate these other operating systems? For reference it looks as >> if direct I/O on OS X uses the F_NOCACHE command of fcntl(2) [1] >> (although per some online comments this might have some problems), >> Solaris uses the advice argument of directio(3c) [2], and Windows uses >> a combination of flags passed to CreateFile() [3, 4]. > >The Linux open(2) manpage contains a long list of warnings about O_DIRECT, >including: > >---------- >In Linux alignment restrictions vary by filesystem and kernel version and might be >absent entirely. However there is currently no filesystem-independent interface >for an application to discover these restrictions for a given file or filesystem. >---------- > >---------- >O_DIRECT I/Os should never be run concurrently with the fork(2) system call, if >the memory buffer is a private mapping (i.e., any mapping created with the >mmap(2) MAP_PRIVATE flag; this includes memory allocated on the heap and >statically allocated buffers). Any such I/Os, whether submitted via an >asynchronous I/O interface or from another thread in the process, should be >completed before >fork(2) is called. Failure to do so can result in data corruption and undefined >behavior in parent and child processes. >---------- > >---------- >Applications should avoid mixing O_DIRECT and normal I/O to the same file, and >especially to overlapping byte regions in the same file. >Even when the filesystem correctly handles the coherency issues in this situation, >overall I/O throughput is likely to be slower than using either mode alone. >Likewise, applications should avoid mixing >mmap(2) of files with direct I/O to the same files. >---------- > >---------- > "The thing that has always disturbed me about O_DIRECT is that the whole >interface is just stupid, and was probably designed by a deranged monkey on >some serious mind-controlling substances." - Linus >---------- > >Adding support for O_DIRECT has a far wider impact than adding just another IO >handle flag. As such I'm opposed to this change as it seems to be prone to cause >hard-to-diagnose failures on Linux and it is also specific to just Linux. > >-- >Alan Burlison >-- From brian.burkhalter at oracle.com Thu Oct 13 23:37:34 2016 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 13 Oct 2016 16:37:34 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <5800061e.423cc20a.87625.39eb@mx.google.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5134F66@ORSMSX113.amr.corp.intel.com> <062c589c-fa54-1652-683a-4e779816391d@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <45C5D272-E70A-40D9-8166-48ADD43763D9@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513D886@ORSMSX113.amr.corp.intel.com> <5800061e.423cc20a.87625.39eb@mx.google.com> Message-ID: <6977846B-882C-480B-9F55-776E3DA31EB0@oracle.com> An interesting comparison of approaches on a Linux server and an OS X client is given here: http://adityaramesh.com/io_benchmark/ Brian On Oct 13, 2016, at 3:09 PM, ecki at zusammenkunft.net wrote: > The question is, if support for fadvice() would be more flexible (and somewhat saner). > > And of course real async fio .) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Burlison at oracle.com Fri Oct 14 08:28:12 2016 From: Alan.Burlison at oracle.com (Alan Burlison) Date: Fri, 14 Oct 2016 09:28:12 +0100 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AA5146652@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5134F66@ORSMSX113.amr.corp.intel.com> <062c589c-fa54-1652-683a-4e779816391d@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <45C5D272-E70A-40D9-8166-48ADD43763D9@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513D886@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5146652@ORSMSX113.amr.corp.intel.com> Message-ID: <715e7654-8037-8688-73b2-35b002307a23@oracle.com> As has been pointed out already, platforms other than Linux provide similar functionality, yet this proposal does not provide any support for them. Even if such support isn't added initially, the APIs added for the Linux implementation should be generic enough that direct IO support could be added for other platforms in the future, and that Java code using direct IO on Linux would work on other direct IO enabled platforms as well. My concern is that the proposed APIs are too Linux-specific to make that possible. I'm well aware of the issues in things like Hadoop and Apache Spark, their over-reliance on platform-specific functionality and native code has made it extremely difficult to get them to work well on platforms other than Linux/x86. We should not make the same mistake in Java itself. -- Alan Burlison -- From yingqi.lu at intel.com Fri Oct 14 17:16:21 2016 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Fri, 14 Oct 2016 17:16:21 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <715e7654-8037-8688-73b2-35b002307a23@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5134F66@ORSMSX113.amr.corp.intel.com> <062c589c-fa54-1652-683a-4e779816391d@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <45C5D272-E70A-40D9-8166-48ADD43763D9@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513D886@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5146652@ORSMSX113.amr.corp.intel.com> <715e7654-8037-8688-73b2-35b002307a23@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AA514738B@ORSMSX113.amr.corp.intel.com> Alan, Thank you for the feedback. We will come up with a more generic API that is flexible for supporting all the platforms that have DirectIO. Will send it here for review soon. Thanks, Lucy >-----Original Message----- >From: Alan Burlison [mailto:Alan.Burlison at oracle.com] >Sent: Friday, October 14, 2016 1:28 AM >To: Lu, Yingqi ; Brian Burkhalter > >Cc: Kharbas, Kishor ; nio-dev at openjdk.java.net; >core-libs-dev at openjdk.java.net; Kaczmarek, Eric >Subject: Re: Proposal for adding O_DIRECT support into JDK 9 > >As has been pointed out already, platforms other than Linux provide similar >functionality, yet this proposal does not provide any support for them. Even if >such support isn't added initially, the APIs added for the Linux implementation >should be generic enough that direct IO support could be added for other >platforms in the future, and that Java code using direct IO on Linux would work >on other direct IO enabled platforms as well. My concern is that the proposed >APIs are too Linux-specific to make that possible. > >I'm well aware of the issues in things like Hadoop and Apache Spark, their over- >reliance on platform-specific functionality and native code has made it extremely >difficult to get them to work well on platforms other than Linux/x86. We should >not make the same mistake in Java itself. > >-- >Alan Burlison >-- From Alan.Bateman at oracle.com Mon Oct 17 14:59:03 2016 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 17 Oct 2016 15:59:03 +0100 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AA51447D9@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5134F66@ORSMSX113.amr.corp.intel.com> <062c589c-fa54-1652-683a-4e779816391d@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51447D9@ORSMSX113.amr.corp.intel.com> Message-ID: <006c75c4-1404-d66b-737d-e07ea2b493fd@oracle.com> On 12/10/2016 17:41, Lu, Yingqi wrote: > : > > You are correct about the "extra copy" with DirectIO. Will it be acceptable if we add a function "Util.getAlignedTemporaryDirectBuffer" and use that for the DirectIO operation? In this case, I think we should be able to avoid the additional copy? > Yes, that should work but it still lacks a way for the user to get an aligned buffer and so you will always be copying in and out of an aligned buffer. The other thing is the sizing of the I/O operation where I think you will also need a way to expose the multiple (or block size) to the user. -Alan. From chris.hegarty at oracle.com Tue Oct 18 10:57:01 2016 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 18 Oct 2016 11:57:01 +0100 Subject: AF_INET6 support In-Reply-To: <9e5d1e66416a4ac895457cff65bd4b49@DEWDFE13DE11.global.corp.sap> References: <964200b6d1bf43f6be476d531f00d3fb@DEWDFE13DE11.global.corp.sap> <098BBC2F-4535-4A66-BB05-BF67E5E3977F@oracle.com> <9e5d1e66416a4ac895457cff65bd4b49@DEWDFE13DE11.global.corp.sap> Message-ID: <48656747-9151-47F2-875D-A211AB0806FA@oracle.com> On 13 Oct 2016, at 09:46, Langer, Christoph wrote: > > Hi Chris, > >>> I?ve got a question regarding AF_INET6. >>> >>> In jdk native code you?ll still find lots of code guarded by ?#ifdef AF_INET6?. >> I?m wondering if there are still supported build environments where AF_INET6 >> is not defined. Or is it time now to assume AF_INET6 and remove this guarding? >> Here at SAP we don?t support non AF_INET6 build environments for quite a long >> time already. But probably there are scenarios out in the Java world where only >> IPv4 builds are done?? >>> >>> Maybe you can shed some light on this and/or give your opinion? >> >> A while back we did consider removing #ifdef AF_INET6, so that the >> code could be cleaned up and made more readable, but we never got >> around to it ( it was just lower priority than other tasks ). I do remember >> fixing, or sponsoring, a change in the last year or so, where an #ifdef >> AF_INET6 was missing, that make me think that it was good that we >> did not remove these ( i.e. the person that filed the bug had a good >> use-case for building without IPv6 support ). I?ll see if I can jog my >> memory by looking through history. > > Ok, it would really be great if you could find more information about that case where #ifdef AF_INET6 was missing and it lead to problems. > > But if nobody really does IPv4 only builds and tests the results, I doubt that the IPv4 only scenario would work or even build. Then I think it was likely that somehow unguarded IPv6 code sneaks in or has already done so... > > In case it can be removed it would really make the code more readable in several places. So, as I'm still doing cleanups, I could also take care of removing those #ifdefs at the places where I'm going. Just let me know. It appears that we do not have a requirement to be able to build IPv4 only. So, it should be possible to do the clean up that you are suggesting. If possible, can the change be kept as a single separate JIRA issue, so that we can easily identify the change in the future, if required. -Chris. From brian.burkhalter at oracle.com Tue Oct 18 15:06:21 2016 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 18 Oct 2016 08:06:21 -0700 Subject: Files.copy does userspace copies but FileChannel uses sendfile In-Reply-To: <004B903D-0344-47EE-95C5-E7D85EC90788@oracle.com> References: <5ea31616-89ba-9d38-e3f7-2cfe799f716c@oracle.com> <004B903D-0344-47EE-95C5-E7D85EC90788@oracle.com> Message-ID: <4C5A502D-ABE1-4864-92AC-EB6C0EF3DFF6@oracle.com> On Oct 12, 2016, at 10:56 AM, Brian Burkhalter wrote: > On Oct 12, 2016, at 9:39 AM, Alan Bateman wrote: > >> On 12/10/2016 17:30, Brian Burkhalter wrote: >> >>> It seems as if this is a good candidate for some JMH testing from the Java layer. >> Yes although you also have to measure the copy when the source isn't in the file system cache so that means dropping caches between copies. > > That should be feasible given the control available within JMH. It?s not the best environment for performance testing, but preliminary JMH benchmark runs on an Ubuntu 16.04 64-bit guest VM on an OS X host show approximately a 6-10% improvement of sendfile(2) over read(2) + write(2) for file-to-file copies. This is with an fsync(2) invocation inserted before the return from Java_sun_nio_fs_UnixCopyFile_transfer() --- a/src/java.base/unix/native/libnio/fs/UnixCopyFile.c +++ b/src/java.base/unix/native/libnio/fs/UnixCopyFile.c @@ -82,4 +82,5 @@ len -= n; } while (len > 0); } + fsync((int)dst); } to ensure that the modified buffer cache pages are flushed to disk. Without the contrived insertion of fsync() the improvement is about 5%. If this seems promising enough I can look into running the benchmark on some other more realistic systems. As an aside, sendfile(2) on the version of OS X I am running (10.9.5 Mavericks) does not support file-to-file transfers: The sendfile() system call sends a regular file specified by descriptor fd out a stream socket specified by descriptor s. On Solaris however file-to-file appears to be supported [2]. Brian [1] http://hg.openjdk.java.net/jdk9/dev/jdk/file/tip/src/java.base/unix/native/libnio/fs/UnixCopyFile.c [2] https://docs.oracle.com/cd/E36784_01/html/E36876/sendfile-3ext.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From weijun.wang at oracle.com Tue Oct 18 15:07:40 2016 From: weijun.wang at oracle.com (Wang Weijun) Date: Tue, 18 Oct 2016 23:07:40 +0800 Subject: NIO Path method to get canonicalized pathname of a non-existing file Message-ID: <0411D370-770E-4BFB-8AC6-747204E30D86@oracle.com> With java.io.File, one can call getCanonicalPath() to resolve symlinks (of a directory) even if the file does not exist. With the NIO Path, it seems toRealPath() only works for an existing file, is there a similar method to do this? Thanks Max From pavel.rappo at oracle.com Thu Oct 20 18:40:48 2016 From: pavel.rappo at oracle.com (Pavel Rappo) Date: Thu, 20 Oct 2016 19:40:48 +0100 Subject: RFR 8168417: Pending exceptions in java.base/windows/native/libnio Message-ID: <20266D7F-6908-4566-BC0D-DD3D65CB39F7@oracle.com> Hello, Could you please review the following change for [1]? http://cr.openjdk.java.net/~prappo/8168417/webrev/ This change addresses some code paths in the native NIO code for Windows operating system where thrown exceptions might be left unnoticed. Thanks, -Pavel -------------------------------------------------------------------------------- [1] https://bugs.openjdk.java.net/browse/JDK-8168417 From chris.hegarty at oracle.com Thu Oct 20 18:41:41 2016 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 20 Oct 2016 19:41:41 +0100 Subject: RFR 8168417: Pending exceptions in java.base/windows/native/libnio In-Reply-To: <20266D7F-6908-4566-BC0D-DD3D65CB39F7@oracle.com> References: <20266D7F-6908-4566-BC0D-DD3D65CB39F7@oracle.com> Message-ID: > On 20 Oct 2016, at 19:40, Pavel Rappo wrote: > > Hello, > > Could you please review the following change for [1]? > > http://cr.openjdk.java.net/~prappo/8168417/webrev/ Thank you Pavel, looks good. -Chris. > This change addresses some code paths in the native NIO code for Windows > operating system where thrown exceptions might be left unnoticed. > > Thanks, > -Pavel > > -------------------------------------------------------------------------------- > [1] https://bugs.openjdk.java.net/browse/JDK-8168417 > From Alan.Bateman at oracle.com Fri Oct 21 06:46:56 2016 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 21 Oct 2016 07:46:56 +0100 Subject: NIO Path method to get canonicalized pathname of a non-existing file In-Reply-To: <0411D370-770E-4BFB-8AC6-747204E30D86@oracle.com> References: <0411D370-770E-4BFB-8AC6-747204E30D86@oracle.com> Message-ID: <8c2e801a-88d9-84bd-d1ef-7063032737d1@oracle.com> On 18/10/2016 16:07, Wang Weijun wrote: > With java.io.File, one can call getCanonicalPath() to resolve symlinks (of a directory) even if the file does not exist. With the NIO Path, it seems toRealPath() only works for an existing file, is there a similar method to do this? There isn't an equivalent, mostly because there isn't a real path for a file that does not exist. You can build on toRealPath to do what you want but as per legacy getCanonicalPath then you cannot predict what the path will be when the file is created. -Alan From Alan.Bateman at oracle.com Fri Oct 21 06:58:08 2016 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 21 Oct 2016 07:58:08 +0100 Subject: RFR 8168417: Pending exceptions in java.base/windows/native/libnio In-Reply-To: <20266D7F-6908-4566-BC0D-DD3D65CB39F7@oracle.com> References: <20266D7F-6908-4566-BC0D-DD3D65CB39F7@oracle.com> Message-ID: <3ab34b9d-37b8-ebd5-ae48-4e1e53ab7cce@oracle.com> On 20/10/2016 19:40, Pavel Rappo wrote: > Hello, > > Could you please review the following change for [1]? > > http://cr.openjdk.java.net/~prappo/8168417/webrev/ > > This change addresses some code paths in the native NIO code for Windows > operating system where thrown exceptions might be left unnoticed. > This looks fine. -Alan From brian.burkhalter at oracle.com Fri Oct 21 22:46:57 2016 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 21 Oct 2016 15:46:57 -0700 Subject: RFC on 8165823: (se) EPollArrayWrapper throws NPE if limits.conf set to 65536 and fd=65536 In-Reply-To: <2CAD7459-E810-4BA1-B493-64C6292FB911@oracle.com> References: <783e980056154a39a9381d8daa446aa3@DEWDFE13DE11.global.corp.sap> <2CAD7459-E810-4BA1-B493-64C6292FB911@oracle.com> Message-ID: I contacted the original submitter of this issue more than two weeks ago about whether a dynamically changing resource limit on file descriptor number could be the cause of the problem in their workload. Thus far I have received no response and have marked the issue [1] as Resolved -> Incomplete. Prior to that I posted a webrev [2] in which the eventsHigh HashMap would be lazily initialized and hopefully avert this problem defensively. As an alternative, and probably this idea will be rejected, but instead of adding relatively complex lazy initialization could we not just unconditionally allocate eventsHigh at line 121 of [3]? private Map eventsHigh = new HashMap<>(); Yes, there would be a footprint and performance penalty but perhaps it is so small as to be in the ?noise? category. According to JOL [4] for JDK 9 the footprint of the default HashMap is 48 bytes (see below) and according to JMH [5] the time to construct the instance is about 20 nanoseconds on my MacBookPro11,1. Also I noticed that this issue [1] was observed in another use case which is discussed in [6]. Thanks, Brian T Tha $ java -jar jol-cli/target/jol-cli.jar footprint java.util.HashMap Running 64-bit HotSpot VM. Using compressed references with 3-bit shift. Objects are 8 bytes aligned. Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes] Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes] java.util.HashMap at 2893de87d footprint: COUNT AVG SUM DESCRIPTION 1 48 48 java.util.HashMap 1 48 (total) [1] https://bugs.openjdk.java.net/browse/JDK-8165823 [2] http://cr.openjdk.java.net/~bpb/8165823/webrev.00/ [3] http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/7b0b28ceca62/src/java.base/linux/classes/sun/nio/ch/EPollArrayWrapper.java [4] http://openjdk.java.net/projects/code-tools/jol/ [5] http://openjdk.java.net/projects/code-tools/jmh/ [6] https://github.com/elastic/elasticsearch/issues/11706 -------------- next part -------------- An HTML attachment was scrubbed... URL: From brunoaiss at gmail.com Fri Oct 28 07:54:00 2016 From: brunoaiss at gmail.com (Brunoais) Date: Fri, 28 Oct 2016 08:54:00 +0100 Subject: Why doesn't NIO provide an async-singlethreaded I/O? In-Reply-To: References: <077bf6e8-14ae-5588-b1a4-03fb4333c15b@gmail.com> <00d24fd4-3c8f-9e9a-2419-91c7c37c91e5@gmail.com> <8AB1350F-F3EE-4756-923C-4178263BEBF5@oracle.com> <48cd729d-7922-3de5-3983-cc7486b5a9d3@gmail.com> <4bf67d65-7b2d-59a7-cd4a-19d84a062799@gmail.com> <10318b41-0383-d68a-3145-bdc57ae78450@gmail.com> <7f977d1d-3aa7-ec2e-6df5-51e8ac6473e5@gmail.com> Message-ID: <88caffab-5d56-5984-2d8a-98a9a3f8f392@gmail.com> Please read the quotes. They are from the thread "Re: Request/discussion: BufferedReader reading using async API while providing sync API" from the mailing list core-libs-dev at openjdk.java.net . The main question I have can be summarized to: Why isn't there (or was there) any non-block read-ahead I/O in java's NIO (more question information in the quotes)? Thanks in advance. On 27/10/2016 23:59, David Holmes wrote: > Sorry nio-dev ... > > On 28/10/2016 7:09 AM, David Holmes wrote: >> You might try discussing on net-dev rather than core-libs-dev, to get >> additional historical info related to the io and nio file APIs. >> >> David >> >> On 28/10/2016 5:08 AM, Brunoais wrote: >>> You are right. Even in windows it does not set the flags for async >>> reads. It seems like it is windows itself that does the decision to >>> buffer the contents based on its own heuristics. >>> >>> But... Why? Why won't it be? Why is there no API for it? How am I >>> getting 100% HDD use and faster times when I fake work to delay getting >>> more data and I only have a fluctuating 60-90% (always going up and >>> down) when I use an InputStream? >>> Is it related to how both classes cache and how frequently and how much >>> each one asks for data? >>> >>> I really would prefer not having to read the source code because it >>> takes a real long time T.T. >>> >>> I end up reinstating... And wondering... >>> >>> Why doesn't java provide a single-threaded non-block API for file reads >>> for all OS that support it? I simply cannot find that information no >>> matter how much I search on google, bing, duck duck go... Can any of >>> you >>> point me to whomever knows? >>> From Alan.Bateman at oracle.com Fri Oct 28 08:46:03 2016 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 28 Oct 2016 09:46:03 +0100 Subject: Why doesn't NIO provide an async-singlethreaded I/O? In-Reply-To: <88caffab-5d56-5984-2d8a-98a9a3f8f392@gmail.com> References: <077bf6e8-14ae-5588-b1a4-03fb4333c15b@gmail.com> <8AB1350F-F3EE-4756-923C-4178263BEBF5@oracle.com> <48cd729d-7922-3de5-3983-cc7486b5a9d3@gmail.com> <4bf67d65-7b2d-59a7-cd4a-19d84a062799@gmail.com> <10318b41-0383-d68a-3145-bdc57ae78450@gmail.com> <7f977d1d-3aa7-ec2e-6df5-51e8ac6473e5@gmail.com> <88caffab-5d56-5984-2d8a-98a9a3f8f392@gmail.com> Message-ID: On 28/10/2016 08:54, Brunoais wrote: > Please read the quotes. > > They are from the thread "Re: Request/discussion: BufferedReader > reading using async API while providing sync API" from the mailing > list core-libs-dev at openjdk.java.net . > > The main question I have can be summarized to: > > Why isn't there (or was there) any non-block read-ahead I/O in java's > NIO (more question information in the quotes)? If you looking to read ahead then a thread + synchronous I/O should work just fine, you don't need any additions to the Java SE APIs to do that. As always, do your performance tests to see if helps or not. I see Peter has provided some JMH examples, include dropping the file system cache between runs. One other thing to add to your list to try is memory mapping the file with FileChannel::map and MappedByteBuffer. In particular, see if MappedByteBuffer::load helps. (Aside from MappedByteBuffer::load, none of the existing APIs map to fadvise or madvise, in particular there is no way to advise that the access is expected to be sequential) As regards AsynchronousFileChannel (which is the only non-blocking file I/O API in the platform) then it's not for sequential access. You can make it do sequential access of course but it will be a bit of work to do right. -Alan From brunoaiss at gmail.com Fri Oct 28 10:00:54 2016 From: brunoaiss at gmail.com (Brunoais) Date: Fri, 28 Oct 2016 11:00:54 +0100 Subject: Why doesn't NIO provide an async-singlethreaded I/O? In-Reply-To: References: <077bf6e8-14ae-5588-b1a4-03fb4333c15b@gmail.com> <48cd729d-7922-3de5-3983-cc7486b5a9d3@gmail.com> <4bf67d65-7b2d-59a7-cd4a-19d84a062799@gmail.com> <10318b41-0383-d68a-3145-bdc57ae78450@gmail.com> <7f977d1d-3aa7-ec2e-6df5-51e8ac6473e5@gmail.com> <88caffab-5d56-5984-2d8a-98a9a3f8f392@gmail.com> Message-ID: Thank you! That helped me deciding a possible direction to go to. I'll checkout that FileChannel::map method as well as Asynchonous. I still do not know about why there's no non-blocking I/O in java but that's fin, I guess. I'll see what I can do with this information. On 28/10/2016 09:46, Alan Bateman wrote: > On 28/10/2016 08:54, Brunoais wrote: > >> Please read the quotes. >> >> They are from the thread "Re: Request/discussion: BufferedReader >> reading using async API while providing sync API" from the mailing >> list core-libs-dev at openjdk.java.net . >> >> The main question I have can be summarized to: >> >> Why isn't there (or was there) any non-block read-ahead I/O in java's >> NIO (more question information in the quotes)? > If you looking to read ahead then a thread + synchronous I/O should > work just fine, you don't need any additions to the Java SE APIs to do > that. As always, do your performance tests to see if helps or not. I > see Peter has provided some JMH examples, include dropping the file > system cache between runs. > > One other thing to add to your list to try is memory mapping the file > with FileChannel::map and MappedByteBuffer. In particular, see if > MappedByteBuffer::load helps. > > (Aside from MappedByteBuffer::load, none of the existing APIs map to > fadvise or madvise, in particular there is no way to advise that the > access is expected to be sequential) > > As regards AsynchronousFileChannel (which is the only non-blocking > file I/O API in the platform) then it's not for sequential access. You > can make it do sequential access of course but it will be a bit of > work to do right. > > -Alan >