From yingqi.lu at intel.com Wed Nov 2 23:34:49 2016 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Wed, 2 Nov 2016 23:34:49 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <006c75c4-1404-d66b-737d-e07ea2b493fd@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5134F66@ORSMSX113.amr.corp.intel.com> <062c589c-fa54-1652-683a-4e779816391d@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51447D9@ORSMSX113.amr.corp.intel.com> <006c75c4-1404-d66b-737d-e07ea2b493fd@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AA5166194@ORSMSX113.amr.corp.intel.com> Hi All, Our most recent DirectIO patch is available at http://cr.openjdk.java.net/~igraves/8164900-3/ In this version, we made following changes: 1. Removed the flag "direct" from FileDescriptor class. Instead, moved it to the FileChannelImpl class. 2. Provided a way for user to allocate a page aligned direct ByteBuffer. 1) Added a constructor DirectByteBuffer(int cap, boolean direct) to allocate a direct ByteBuffer that is aligned to the page size. 2) Added Util.getTemporaryAlignedDirectBuffer(int size) 3) Added DirectByteBuffer.isAligned(int pos) to check if the buffer is aligned before doing native IO with it. 3. Moved all the alignment check from C code to Java code (mainly FileChannelImpl and IOUtil.java). 4. Made the DirectIO functionality consistent between read and write operations. With current version of the patch, user would be responsible for the alignment with file offset and IO size. 5. Made the API for DirectIO more extensible to all the supporting platforms. 1) Unix OS specific code are done through UnixConstants.java.template and FileChannelImpl.c. 2) Coded and tested for Linux and OS X (OS X testing is done through VirtualBox with OS X VM on top of Linux OS). 3) Coded for Solaris. We do not have environment to test it so that we commented the changes out. 6. We added 4 test cases following the existing non-direct IO examples. 7. We did jtreg test for the entire nio package and no errors were found due to our changes. Please let us know your feedback and comment. Thank you very much for your time and consideration! Thanks, Lucy >-----Original Message----- >From: Alan Bateman [mailto:Alan.Bateman at oracle.com] >Sent: Monday, October 17, 2016 7:59 AM >To: Lu, Yingqi >Cc: nio-dev at openjdk.java.net; Kaczmarek, Eric ; >Kharbas, Kishor >Subject: Re: Proposal for adding O_DIRECT support into JDK 9 > >On 12/10/2016 17:41, Lu, Yingqi wrote: > >> : >> >> You are correct about the "extra copy" with DirectIO. Will it be acceptable if we >add a function "Util.getAlignedTemporaryDirectBuffer" and use that for the >DirectIO operation? In this case, I think we should be able to avoid the additional >copy? >> >Yes, that should work but it still lacks a way for the user to get an aligned buffer >and so you will always be copying in and out of an aligned buffer. The other thing >is the sizing of the I/O operation where I think you will also need a way to expose >the multiple (or block size) to the user. > >-Alan. From david.lloyd at redhat.com Thu Nov 3 12:14:42 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 3 Nov 2016 07:14:42 -0500 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AA5166194@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5134F66@ORSMSX113.amr.corp.intel.com> <062c589c-fa54-1652-683a-4e779816391d@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51447D9@ORSMSX113.amr.corp.intel.com> <006c75c4-1404-d66b-737d-e07ea2b493fd@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5166194@ORSMSX113.amr.corp.intel.com> Message-ID: What happens if you're using large pages? Are the direct buffers always allocated off of small pages? On 11/02/2016 06:34 PM, Lu, Yingqi wrote: > Hi All, > > Our most recent DirectIO patch is available at http://cr.openjdk.java.net/~igraves/8164900-3/ > > In this version, we made following changes: > > 1. Removed the flag "direct" from FileDescriptor class. Instead, moved it to the FileChannelImpl class. > > 2. Provided a way for user to allocate a page aligned direct ByteBuffer. > 1) Added a constructor DirectByteBuffer(int cap, boolean direct) to allocate a direct ByteBuffer that is aligned to the page size. > 2) Added Util.getTemporaryAlignedDirectBuffer(int size) > 3) Added DirectByteBuffer.isAligned(int pos) to check if the buffer is aligned before doing native IO with it. > > 3. Moved all the alignment check from C code to Java code (mainly FileChannelImpl and IOUtil.java). > > 4. Made the DirectIO functionality consistent between read and write operations. With current version of the patch, user would be responsible for the alignment with file offset and IO size. > > 5. Made the API for DirectIO more extensible to all the supporting platforms. > 1) Unix OS specific code are done through UnixConstants.java.template and FileChannelImpl.c. > 2) Coded and tested for Linux and OS X (OS X testing is done through VirtualBox with OS X VM on top of Linux OS). > 3) Coded for Solaris. We do not have environment to test it so that we commented the changes out. > > 6. We added 4 test cases following the existing non-direct IO examples. > > 7. We did jtreg test for the entire nio package and no errors were found due to our changes. > > Please let us know your feedback and comment. Thank you very much for your time and consideration! > > Thanks, > Lucy > >> -----Original Message----- >> From: Alan Bateman [mailto:Alan.Bateman at oracle.com] >> Sent: Monday, October 17, 2016 7:59 AM >> To: Lu, Yingqi >> Cc: nio-dev at openjdk.java.net; Kaczmarek, Eric ; >> Kharbas, Kishor >> Subject: Re: Proposal for adding O_DIRECT support into JDK 9 >> >> On 12/10/2016 17:41, Lu, Yingqi wrote: >> >>> : >>> >>> You are correct about the "extra copy" with DirectIO. Will it be acceptable if we >> add a function "Util.getAlignedTemporaryDirectBuffer" and use that for the >> DirectIO operation? In this case, I think we should be able to avoid the additional >> copy? >>> >> Yes, that should work but it still lacks a way for the user to get an aligned buffer >> and so you will always be copying in and out of an aligned buffer. The other thing >> is the sizing of the I/O operation where I think you will also need a way to expose >> the multiple (or block size) to the user. >> >> -Alan. -- - DML From sybersnake at gmail.com Thu Nov 3 12:22:54 2016 From: sybersnake at gmail.com (Jon V.) Date: Thu, 3 Nov 2016 08:22:54 -0400 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5134F66@ORSMSX113.amr.corp.intel.com> <062c589c-fa54-1652-683a-4e779816391d@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51447D9@ORSMSX113.amr.corp.intel.com> <006c75c4-1404-d66b-737d-e07ea2b493fd@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5166194@ORSMSX113.amr.corp.intel.com> Message-ID: Why not call aligned_alloc for doing page alignment? It should waste much less memory. http://en.cppreference.com/w/c/memory/aligned_alloc On Thu, Nov 3, 2016 at 8:14 AM, David M. Lloyd wrote: > What happens if you're using large pages? Are the direct buffers always > allocated off of small pages? > > > On 11/02/2016 06:34 PM, Lu, Yingqi wrote: > >> Hi All, >> >> Our most recent DirectIO patch is available at >> http://cr.openjdk.java.net/~igraves/8164900-3/ >> >> In this version, we made following changes: >> >> 1. Removed the flag "direct" from FileDescriptor class. Instead, moved it >> to the FileChannelImpl class. >> >> 2. Provided a way for user to allocate a page aligned direct ByteBuffer. >> 1) Added a constructor DirectByteBuffer(int cap, boolean direct) to >> allocate a direct ByteBuffer that is aligned to the page size. >> 2) Added Util.getTemporaryAlignedDirectBuffer(int size) >> 3) Added DirectByteBuffer.isAligned(int pos) to check if the buffer >> is aligned before doing native IO with it. >> >> 3. Moved all the alignment check from C code to Java code (mainly >> FileChannelImpl and IOUtil.java). >> >> 4. Made the DirectIO functionality consistent between read and write >> operations. With current version of the patch, user would be responsible >> for the alignment with file offset and IO size. >> >> 5. Made the API for DirectIO more extensible to all the supporting >> platforms. >> 1) Unix OS specific code are done through UnixConstants.java.template >> and FileChannelImpl.c. >> 2) Coded and tested for Linux and OS X (OS X testing is done through >> VirtualBox with OS X VM on top of Linux OS). >> 3) Coded for Solaris. We do not have environment to test it so that >> we commented the changes out. >> >> 6. We added 4 test cases following the existing non-direct IO examples. >> >> 7. We did jtreg test for the entire nio package and no errors were found >> due to our changes. >> >> Please let us know your feedback and comment. Thank you very much for >> your time and consideration! >> >> Thanks, >> Lucy >> >> -----Original Message----- >>> From: Alan Bateman [mailto:Alan.Bateman at oracle.com] >>> Sent: Monday, October 17, 2016 7:59 AM >>> To: Lu, Yingqi >>> Cc: nio-dev at openjdk.java.net; Kaczmarek, Eric >> >; >>> Kharbas, Kishor >>> Subject: Re: Proposal for adding O_DIRECT support into JDK 9 >>> >>> On 12/10/2016 17:41, Lu, Yingqi wrote: >>> >>> : >>>> >>>> You are correct about the "extra copy" with DirectIO. Will it be >>>> acceptable if we >>>> >>> add a function "Util.getAlignedTemporaryDirectBuffer" and use that for >>> the >>> DirectIO operation? In this case, I think we should be able to avoid the >>> additional >>> copy? >>> >>>> >>>> Yes, that should work but it still lacks a way for the user to get an >>> aligned buffer >>> and so you will always be copying in and out of an aligned buffer. The >>> other thing >>> is the sizing of the I/O operation where I think you will also need a >>> way to expose >>> the multiple (or block size) to the user. >>> >>> -Alan. >>> >> > -- > - DML > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brunoaiss at gmail.com Thu Nov 3 14:47:42 2016 From: brunoaiss at gmail.com (Brunoais) Date: Thu, 3 Nov 2016 14:47:42 +0000 Subject: Why doesn't NIO provide an async-singlethreaded I/O? In-Reply-To: References: <077bf6e8-14ae-5588-b1a4-03fb4333c15b@gmail.com> <4bf67d65-7b2d-59a7-cd4a-19d84a062799@gmail.com> <10318b41-0383-d68a-3145-bdc57ae78450@gmail.com> <7f977d1d-3aa7-ec2e-6df5-51e8ac6473e5@gmail.com> <88caffab-5d56-5984-2d8a-98a9a3f8f392@gmail.com> Message-ID: <922e9f81-e8de-2ec0-9ea8-11b5e69d09b7@gmail.com> Hey Alan. I ended up using AsynchronousFileChannel to do the reading work for the implementation of the class. I've submitted it for peer reviewing but I don't know if anything is being done about it. After reading the C code, it seems like there's no real advantage between FileChannel::map and AsynchronousFileChannel for reads although there are hints that there can exist some advantages with AsynchronousFileChannel if there's a lot of processing between reads. If you can, I'd like you to review the speed tests I made to see if I missed or entered in a pitfall anything in the process Finally, I'll just reinstate that, in my local tests on a windows 10 machine, I got ~2.8s of improvement reading a 1GB file that takes ~20-23s to read. I don't have any tests for when the OS is concurrently scheduling reads but I think it is an great improvement for situations when a server is the only thing reading and writing to/from disk where the seconds stack up as the operations also stack on each other. On 28/10/2016 11:00, Brunoais wrote: > Thank you! > > That helped me deciding a possible direction to go to. I'll checkout > that FileChannel::map method as well as Asynchonous. > > I still do not know about why there's no non-blocking I/O in java but > that's fin, I guess. > > I'll see what I can do with this information. > > > On 28/10/2016 09:46, Alan Bateman wrote: >> On 28/10/2016 08:54, Brunoais wrote: >> >>> Please read the quotes. >>> >>> They are from the thread "Re: Request/discussion: BufferedReader >>> reading using async API while providing sync API" from the mailing >>> list core-libs-dev at openjdk.java.net . >>> >>> The main question I have can be summarized to: >>> >>> Why isn't there (or was there) any non-block read-ahead I/O in >>> java's NIO (more question information in the quotes)? >> If you looking to read ahead then a thread + synchronous I/O should >> work just fine, you don't need any additions to the Java SE APIs to >> do that. As always, do your performance tests to see if helps or not. >> I see Peter has provided some JMH examples, include dropping the file >> system cache between runs. >> >> One other thing to add to your list to try is memory mapping the file >> with FileChannel::map and MappedByteBuffer. In particular, see if >> MappedByteBuffer::load helps. >> >> (Aside from MappedByteBuffer::load, none of the existing APIs map to >> fadvise or madvise, in particular there is no way to advise that the >> access is expected to be sequential) >> >> As regards AsynchronousFileChannel (which is the only non-blocking >> file I/O API in the platform) then it's not for sequential access. >> You can make it do sequential access of course but it will be a bit >> of work to do right. >> >> -Alan >> > From yingqi.lu at intel.com Thu Nov 3 18:10:57 2016 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Thu, 3 Nov 2016 18:10:57 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5134F66@ORSMSX113.amr.corp.intel.com> <062c589c-fa54-1652-683a-4e779816391d@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51447D9@ORSMSX113.amr.corp.intel.com> <006c75c4-1404-d66b-737d-e07ea2b493fd@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5166194@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AA51675D0@ORSMSX113.amr.corp.intel.com> We tried to minimize the changes to native C code and put alignment at Java level. We borrow the same approach being used for VM.isDirectMemoryPageAligned here for DirectByteBuffer alignment. Thanks, Lucy From: Jon V. [mailto:sybersnake at gmail.com] Sent: Thursday, November 03, 2016 5:23 AM To: David M. Lloyd Cc: Lu, Yingqi ; Alan Bateman ; nio-dev at openjdk.java.net; Kaczmarek, Eric ; core-libs-dev at openjdk.java.net; Kharbas, Kishor Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Why not call aligned_alloc for doing page alignment? It should waste much less memory. http://en.cppreference.com/w/c/memory/aligned_alloc On Thu, Nov 3, 2016 at 8:14 AM, David M. Lloyd > wrote: What happens if you're using large pages? Are the direct buffers always allocated off of small pages? On 11/02/2016 06:34 PM, Lu, Yingqi wrote: Hi All, Our most recent DirectIO patch is available at http://cr.openjdk.java.net/~igraves/8164900-3/ In this version, we made following changes: 1. Removed the flag "direct" from FileDescriptor class. Instead, moved it to the FileChannelImpl class. 2. Provided a way for user to allocate a page aligned direct ByteBuffer. 1) Added a constructor DirectByteBuffer(int cap, boolean direct) to allocate a direct ByteBuffer that is aligned to the page size. 2) Added Util.getTemporaryAlignedDirectBuffer(int size) 3) Added DirectByteBuffer.isAligned(int pos) to check if the buffer is aligned before doing native IO with it. 3. Moved all the alignment check from C code to Java code (mainly FileChannelImpl and IOUtil.java). 4. Made the DirectIO functionality consistent between read and write operations. With current version of the patch, user would be responsible for the alignment with file offset and IO size. 5. Made the API for DirectIO more extensible to all the supporting platforms. 1) Unix OS specific code are done through UnixConstants.java.template and FileChannelImpl.c. 2) Coded and tested for Linux and OS X (OS X testing is done through VirtualBox with OS X VM on top of Linux OS). 3) Coded for Solaris. We do not have environment to test it so that we commented the changes out. 6. We added 4 test cases following the existing non-direct IO examples. 7. We did jtreg test for the entire nio package and no errors were found due to our changes. Please let us know your feedback and comment. Thank you very much for your time and consideration! Thanks, Lucy -----Original Message----- From: Alan Bateman [mailto:Alan.Bateman at oracle.com] Sent: Monday, October 17, 2016 7:59 AM To: Lu, Yingqi > Cc: nio-dev at openjdk.java.net; Kaczmarek, Eric >; Kharbas, Kishor > Subject: Re: Proposal for adding O_DIRECT support into JDK 9 On 12/10/2016 17:41, Lu, Yingqi wrote: : You are correct about the "extra copy" with DirectIO. Will it be acceptable if we add a function "Util.getAlignedTemporaryDirectBuffer" and use that for the DirectIO operation? In this case, I think we should be able to avoid the additional copy? Yes, that should work but it still lacks a way for the user to get an aligned buffer and so you will always be copying in and out of an aligned buffer. The other thing is the sizing of the I/O operation where I think you will also need a way to expose the multiple (or block size) to the user. -Alan. -- - DML -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Thu Nov 3 18:14:21 2016 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Thu, 3 Nov 2016 18:14:21 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5134F66@ORSMSX113.amr.corp.intel.com> <062c589c-fa54-1652-683a-4e779816391d@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51447D9@ORSMSX113.amr.corp.intel.com> <006c75c4-1404-d66b-737d-e07ea2b493fd@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5166194@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AA51675E9@ORSMSX113.amr.corp.intel.com> I think DirectByteBuffer do not use large page. Please refer to https://bugs.openjdk.java.net/browse/JDK-8021829 for more details. Our code changes are done for alignment. I think they should not impact the usage of either small pages or large pages. Thanks, Lucy >-----Original Message----- >From: David M. Lloyd [mailto:david.lloyd at redhat.com] >Sent: Thursday, November 03, 2016 5:15 AM >To: Lu, Yingqi ; Alan Bateman >Cc: nio-dev at openjdk.java.net; Kharbas, Kishor ; >core-libs-dev at openjdk.java.net; Kaczmarek, Eric >Subject: Re: Proposal for adding O_DIRECT support into JDK 9 > >What happens if you're using large pages? Are the direct buffers always >allocated off of small pages? > >On 11/02/2016 06:34 PM, Lu, Yingqi wrote: >> Hi All, >> >> Our most recent DirectIO patch is available at >> http://cr.openjdk.java.net/~igraves/8164900-3/ >> >> In this version, we made following changes: >> >> 1. Removed the flag "direct" from FileDescriptor class. Instead, moved it to the >FileChannelImpl class. >> >> 2. Provided a way for user to allocate a page aligned direct ByteBuffer. >> 1) Added a constructor DirectByteBuffer(int cap, boolean direct) to allocate a >direct ByteBuffer that is aligned to the page size. >> 2) Added Util.getTemporaryAlignedDirectBuffer(int size) >> 3) Added DirectByteBuffer.isAligned(int pos) to check if the buffer is aligned >before doing native IO with it. >> >> 3. Moved all the alignment check from C code to Java code (mainly >FileChannelImpl and IOUtil.java). >> >> 4. Made the DirectIO functionality consistent between read and write >operations. With current version of the patch, user would be responsible for the >alignment with file offset and IO size. >> >> 5. Made the API for DirectIO more extensible to all the supporting platforms. >> 1) Unix OS specific code are done through UnixConstants.java.template and >FileChannelImpl.c. >> 2) Coded and tested for Linux and OS X (OS X testing is done through >VirtualBox with OS X VM on top of Linux OS). >> 3) Coded for Solaris. We do not have environment to test it so that we >commented the changes out. >> >> 6. We added 4 test cases following the existing non-direct IO examples. >> >> 7. We did jtreg test for the entire nio package and no errors were found due to >our changes. >> >> Please let us know your feedback and comment. Thank you very much for your >time and consideration! >> >> Thanks, >> Lucy >> >>> -----Original Message----- >>> From: Alan Bateman [mailto:Alan.Bateman at oracle.com] >>> Sent: Monday, October 17, 2016 7:59 AM >>> To: Lu, Yingqi >>> Cc: nio-dev at openjdk.java.net; Kaczmarek, Eric >>> ; Kharbas, Kishor >>> >>> Subject: Re: Proposal for adding O_DIRECT support into JDK 9 >>> >>> On 12/10/2016 17:41, Lu, Yingqi wrote: >>> >>>> : >>>> >>>> You are correct about the "extra copy" with DirectIO. Will it be >>>> acceptable if we >>> add a function "Util.getAlignedTemporaryDirectBuffer" and use that >>> for the DirectIO operation? In this case, I think we should be able >>> to avoid the additional copy? >>>> >>> Yes, that should work but it still lacks a way for the user to get an >>> aligned buffer and so you will always be copying in and out of an >>> aligned buffer. The other thing is the sizing of the I/O operation >>> where I think you will also need a way to expose the multiple (or block size) to >the user. >>> >>> -Alan. > >-- >- DML From peter.levart at gmail.com Thu Nov 3 19:38:54 2016 From: peter.levart at gmail.com (Peter Levart) Date: Thu, 3 Nov 2016 20:38:54 +0100 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AA5166194@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5134F66@ORSMSX113.amr.corp.intel.com> <062c589c-fa54-1652-683a-4e779816391d@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51447D9@ORSMSX113.amr.corp.intel.com> <006c75c4-1404-d66b-737d-e07ea2b493fd@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5166194@ORSMSX113.amr.corp.intel.com> Message-ID: <4d4c5ae5-06f9-40a9-f29c-56d0a44e671d@gmail.com> Hi Lucy, Did you know that in JDK 9 the following two methods have been added to java.nio.ByteBuffer: /** @return The indexed byte's memory address modulus the unit size */ public final int alignmentOffset(int index, int unitSize); /** @return new byte buffer whose content is a shared and aligned subsequence of this buffer's content */ public final ByteBuffer alignedSlice(int unitSize); So you could express your proposed methods ByteBuffer.allocateDirectAligned() and ByteBuffer.isAligned() with the above two methods in the following way: instead of: ByteBuffer bb = ByteBuffer.allocateDirectAligned(capacity) you could do: int pageSize = ... ByteBuffer bb = ByteBuffer.allocateDirect(capacity + pageSize - 1).alignedSlice(pageSize); And instead of: bb.isAligned() you could do: bb.alignmentOffset(pageSize) == 0 The only thing that needs to be added is a public API to return the pageSize (i.e. Bits.pageSize()). What do you think? Regards, Peter On 11/03/2016 12:34 AM, Lu, Yingqi wrote: > Hi All, > > Our most recent DirectIO patch is available at http://cr.openjdk.java.net/~igraves/8164900-3/ > > In this version, we made following changes: > > 1. Removed the flag "direct" from FileDescriptor class. Instead, moved it to the FileChannelImpl class. > > 2. Provided a way for user to allocate a page aligned direct ByteBuffer. > 1) Added a constructor DirectByteBuffer(int cap, boolean direct) to allocate a direct ByteBuffer that is aligned to the page size. > 2) Added Util.getTemporaryAlignedDirectBuffer(int size) > 3) Added DirectByteBuffer.isAligned(int pos) to check if the buffer is aligned before doing native IO with it. > > 3. Moved all the alignment check from C code to Java code (mainly FileChannelImpl and IOUtil.java). > > 4. Made the DirectIO functionality consistent between read and write operations. With current version of the patch, user would be responsible for the alignment with file offset and IO size. > > 5. Made the API for DirectIO more extensible to all the supporting platforms. > 1) Unix OS specific code are done through UnixConstants.java.template and FileChannelImpl.c. > 2) Coded and tested for Linux and OS X (OS X testing is done through VirtualBox with OS X VM on top of Linux OS). > 3) Coded for Solaris. We do not have environment to test it so that we commented the changes out. > > 6. We added 4 test cases following the existing non-direct IO examples. > > 7. We did jtreg test for the entire nio package and no errors were found due to our changes. > > Please let us know your feedback and comment. Thank you very much for your time and consideration! > > Thanks, > Lucy > >> -----Original Message----- >> From: Alan Bateman [mailto:Alan.Bateman at oracle.com] >> Sent: Monday, October 17, 2016 7:59 AM >> To: Lu, Yingqi >> Cc: nio-dev at openjdk.java.net; Kaczmarek, Eric ; >> Kharbas, Kishor >> Subject: Re: Proposal for adding O_DIRECT support into JDK 9 >> >> On 12/10/2016 17:41, Lu, Yingqi wrote: >> >>> : >>> >>> You are correct about the "extra copy" with DirectIO. Will it be acceptable if we >> add a function "Util.getAlignedTemporaryDirectBuffer" and use that for the >> DirectIO operation? In this case, I think we should be able to avoid the additional >> copy? >> Yes, that should work but it still lacks a way for the user to get an aligned buffer >> and so you will always be copying in and out of an aligned buffer. The other thing >> is the sizing of the I/O operation where I think you will also need a way to expose >> the multiple (or block size) to the user. >> >> -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.sandoz at oracle.com Thu Nov 3 19:45:48 2016 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 3 Nov 2016 12:45:48 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <4d4c5ae5-06f9-40a9-f29c-56d0a44e671d@gmail.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5134F66@ORSMSX113.amr.corp.intel.com> <062c589c-fa54-1652-683a-4e779816391d@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51447D9@ORSMSX113.amr.corp.intel.com> <006c75c4-1404-d66b-737d-e07ea2b493fd@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5166194@ORSMSX113.amr.corp.intel.com> <4d4c5ae5-06f9-40a9-f29c-56d0a44e671d@! gmail.com> Message-ID: Hi Peter, Thanks for pointing this out, you beat me to it :-) Note that the alignment for a unit size corresponding to page size will only be stable for direct byte buffers, since for heap byte buffers the GC can move the underlying byte array and the alignment properties may change. Paul. > On 3 Nov 2016, at 12:38, Peter Levart wrote: > > Hi Lucy, > > Did you know that in JDK 9 the following two methods have been added to java.nio.ByteBuffer: > > /** @return The indexed byte's memory address modulus the unit size > */ > public final int alignmentOffset(int index, int unitSize); > > /** @return new byte buffer whose content is a shared and aligned subsequence of this buffer's content > */ > public final ByteBuffer alignedSlice(int unitSize); > > > So you could express your proposed methods ByteBuffer.allocateDirectAligned() and ByteBuffer.isAligned() with the above two methods in the following way: > > instead of: > > ByteBuffer bb = ByteBuffer.allocateDirectAligned(capacity) > > you could do: > > int pageSize = ... > ByteBuffer bb = ByteBuffer.allocateDirect(capacity + pageSize - 1).alignedSlice(pageSize); > > And instead of: > > bb.isAligned() > > you could do: > > bb.alignmentOffset(pageSize) == 0 > > > > The only thing that needs to be added is a public API to return the pageSize (i.e. Bits.pageSize()). > > > What do you think? > > Regards, Peter > > On 11/03/2016 12:34 AM, Lu, Yingqi wrote: >> Hi All, >> >> Our most recent DirectIO patch is available at http://cr.openjdk.java.net/~igraves/8164900-3/ >> >> In this version, we made following changes: >> >> 1. Removed the flag "direct" from FileDescriptor class. Instead, moved it to the FileChannelImpl class. >> >> 2. Provided a way for user to allocate a page aligned direct ByteBuffer. >> 1) Added a constructor DirectByteBuffer(int cap, boolean direct) to allocate a direct ByteBuffer that is aligned to the page size. >> 2) Added Util.getTemporaryAlignedDirectBuffer(int size) >> 3) Added DirectByteBuffer.isAligned(int pos) to check if the buffer is aligned before doing native IO with it. >> >> 3. Moved all the alignment check from C code to Java code (mainly FileChannelImpl and IOUtil.java). >> >> 4. Made the DirectIO functionality consistent between read and write operations. With current version of the patch, user would be responsible for the alignment with file offset and IO size. >> >> 5. Made the API for DirectIO more extensible to all the supporting platforms. >> 1) Unix OS specific code are done through UnixConstants.java.template and FileChannelImpl.c. >> 2) Coded and tested for Linux and OS X (OS X testing is done through VirtualBox with OS X VM on top of Linux OS). >> 3) Coded for Solaris. We do not have environment to test it so that we commented the changes out. >> >> 6. We added 4 test cases following the existing non-direct IO examples. >> >> 7. We did jtreg test for the entire nio package and no errors were found due to our changes. >> >> Please let us know your feedback and comment. Thank you very much for your time and consideration! >> >> Thanks, >> Lucy >> >>> -----Original Message----- >>> From: Alan Bateman [mailto:Alan.Bateman at oracle.com ] >>> Sent: Monday, October 17, 2016 7:59 AM >>> To: Lu, Yingqi >>> Cc: nio-dev at openjdk.java.net ; Kaczmarek, Eric ; >>> Kharbas, Kishor >>> Subject: Re: Proposal for adding O_DIRECT support into JDK 9 >>> >>> On 12/10/2016 17:41, Lu, Yingqi wrote: >>> >>>> : >>>> >>>> You are correct about the "extra copy" with DirectIO. Will it be acceptable if we >>> add a function "Util.getAlignedTemporaryDirectBuffer" and use that for the >>> DirectIO operation? In this case, I think we should be able to avoid the additional >>> copy? >>> Yes, that should work but it still lacks a way for the user to get an aligned buffer >>> and so you will always be copying in and out of an aligned buffer. The other thing >>> is the sizing of the I/O operation where I think you will also need a way to expose >>> the multiple (or block size) to the user. >>> >>> -Alan. > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From yingqi.lu at intel.com Thu Nov 3 19:49:57 2016 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Thu, 3 Nov 2016 19:49:57 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5134F66@ORSMSX113.amr.corp.intel.com> <062c589c-fa54-1652-683a-4e779816391d@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51447D9@ORSMSX113.amr.corp.intel.com> <006c75c4-1404-d66b-737d-e07ea2b493fd@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5166194@ORSMSX113.amr.corp.intel.com> <4d4c5ae5-06f9-40a9-f29c-56d0a44e671d@! gmail.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AA516782E@ORSMSX113.amr.corp.intel.com> Hi Paul and Peter, Thank you very much for pointing this out. We did not know about it :) We will modify the patch accordingly and will send it here for review. Thanks, Lucy From: Paul Sandoz [mailto:paul.sandoz at oracle.com] Sent: Thursday, November 03, 2016 12:46 PM To: Peter Levart Cc: Lu, Yingqi ; Alan Bateman ; nio-dev at openjdk.java.net; Kaczmarek, Eric ; core-libs-dev at openjdk.java.net; Kharbas, Kishor Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Peter, Thanks for pointing this out, you beat me to it :-) Note that the alignment for a unit size corresponding to page size will only be stable for direct byte buffers, since for heap byte buffers the GC can move the underlying byte array and the alignment properties may change. Paul. On 3 Nov 2016, at 12:38, Peter Levart > wrote: Hi Lucy, Did you know that in JDK 9 the following two methods have been added to java.nio.ByteBuffer: /** @return The indexed byte's memory address modulus the unit size */ public final int alignmentOffset(int index, int unitSize); /** @return new byte buffer whose content is a shared and aligned subsequence of this buffer's content */ public final ByteBuffer alignedSlice(int unitSize); So you could express your proposed methods ByteBuffer.allocateDirectAligned() and ByteBuffer.isAligned() with the above two methods in the following way: instead of: ByteBuffer bb = ByteBuffer.allocateDirectAligned(capacity) you could do: int pageSize = ... ByteBuffer bb = ByteBuffer.allocateDirect(capacity + pageSize - 1).alignedSlice(pageSize); And instead of: bb.isAligned() you could do: bb.alignmentOffset(pageSize) == 0 The only thing that needs to be added is a public API to return the pageSize (i.e. Bits.pageSize()). What do you think? Regards, Peter On 11/03/2016 12:34 AM, Lu, Yingqi wrote: Hi All, Our most recent DirectIO patch is available at http://cr.openjdk.java.net/~igraves/8164900-3/ In this version, we made following changes: 1. Removed the flag "direct" from FileDescriptor class. Instead, moved it to the FileChannelImpl class. 2. Provided a way for user to allocate a page aligned direct ByteBuffer. 1) Added a constructor DirectByteBuffer(int cap, boolean direct) to allocate a direct ByteBuffer that is aligned to the page size. 2) Added Util.getTemporaryAlignedDirectBuffer(int size) 3) Added DirectByteBuffer.isAligned(int pos) to check if the buffer is aligned before doing native IO with it. 3. Moved all the alignment check from C code to Java code (mainly FileChannelImpl and IOUtil.java). 4. Made the DirectIO functionality consistent between read and write operations. With current version of the patch, user would be responsible for the alignment with file offset and IO size. 5. Made the API for DirectIO more extensible to all the supporting platforms. 1) Unix OS specific code are done through UnixConstants.java.template and FileChannelImpl.c. 2) Coded and tested for Linux and OS X (OS X testing is done through VirtualBox with OS X VM on top of Linux OS). 3) Coded for Solaris. We do not have environment to test it so that we commented the changes out. 6. We added 4 test cases following the existing non-direct IO examples. 7. We did jtreg test for the entire nio package and no errors were found due to our changes. Please let us know your feedback and comment. Thank you very much for your time and consideration! Thanks, Lucy -----Original Message----- From: Alan Bateman [mailto:Alan.Bateman at oracle.com] Sent: Monday, October 17, 2016 7:59 AM To: Lu, Yingqi Cc: nio-dev at openjdk.java.net; Kaczmarek, Eric ; Kharbas, Kishor Subject: Re: Proposal for adding O_DIRECT support into JDK 9 On 12/10/2016 17:41, Lu, Yingqi wrote: : You are correct about the "extra copy" with DirectIO. Will it be acceptable if we add a function "Util.getAlignedTemporaryDirectBuffer" and use that for the DirectIO operation? In this case, I think we should be able to avoid the additional copy? Yes, that should work but it still lacks a way for the user to get an aligned buffer and so you will always be copying in and out of an aligned buffer. The other thing is the sizing of the I/O operation where I think you will also need a way to expose the multiple (or block size) to the user. -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.sandoz at oracle.com Fri Nov 4 16:58:21 2016 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Fri, 4 Nov 2016 09:58:21 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AA516782E@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5134F66@ORSMSX113.amr.corp.intel.com> <062c589c-fa54-1652-683a-4e779816391d@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51447D9@ORSMSX113.amr.corp.intel.com> <006c75c4-1404-d66b-737d-e07ea2b493fd@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5166194@ORSMSX113.amr.corp.intel.com> <4d4c5ae5-06f9-40a9-f29c-56d0a44e671d@! ! gmail.com> <9ACD5B67AAC5594CB6268234CF29CF9AA516782E@ORSMSX113.amr.corp.intel.com> Message-ID: <36CE90D5-B86C-4E0A-A2BD-AC812D4262D1@oracle.com> Hi Lucy, I am struggling to find an entirely satisfactory location to place a method such as getPageSize, but java.lang.Runtime seems less worse than other places :-) Paul. > On 3 Nov 2016, at 12:49, Lu, Yingqi wrote: > > Hi Paul and Peter, > > Thank you very much for pointing this out. We did not know about it J > > We will modify the patch accordingly and will send it here for review. > > Thanks, > Lucy > > From: Paul Sandoz [mailto:paul.sandoz at oracle.com] > Sent: Thursday, November 03, 2016 12:46 PM > To: Peter Levart > Cc: Lu, Yingqi ; Alan Bateman ; nio-dev at openjdk.java.net; Kaczmarek, Eric ; core-libs-dev at openjdk.java.net; Kharbas, Kishor > Subject: Re: Proposal for adding O_DIRECT support into JDK 9 > > Hi Peter, > > Thanks for pointing this out, you beat me to it :-) > > Note that the alignment for a unit size corresponding to page size will only be stable for direct byte buffers, since for heap byte buffers the GC can move the underlying byte array and the alignment properties may change. > > Paul. > > On 3 Nov 2016, at 12:38, Peter Levart wrote: > > Hi Lucy, > > Did you know that in JDK 9 the following two methods have been added to java.nio.ByteBuffer: > > /** @return The indexed byte's memory address modulus the unit size > */ > public final int alignmentOffset(int index, int unitSize); > > /** @return new byte buffer whose content is a shared and aligned subsequence of this buffer's content > */ > public final ByteBuffer alignedSlice(int unitSize); > > > So you could express your proposed methods ByteBuffer.allocateDirectAligned() and ByteBuffer.isAligned() with the above two methods in the following way: > > instead of: > > ByteBuffer bb = ByteBuffer.allocateDirectAligned(capacity) > > you could do: > > int pageSize = ... > ByteBuffer bb = ByteBuffer.allocateDirect(capacity + pageSize - 1).alignedSlice(pageSize); > > And instead of: > > bb.isAligned() > > you could do: > > bb.alignmentOffset(pageSize) == 0 > > > > The only thing that needs to be added is a public API to return the pageSize (i.e. Bits.pageSize()). > > > What do you think? > > Regards, Peter > > On 11/03/2016 12:34 AM, Lu, Yingqi wrote: > Hi All, > > Our most recent DirectIO patch is available at http://cr.openjdk.java.net/~igraves/8164900-3/ > > In this version, we made following changes: > > 1. Removed the flag "direct" from FileDescriptor class. Instead, moved it to the FileChannelImpl class. > > 2. Provided a way for user to allocate a page aligned direct ByteBuffer. > 1) Added a constructor DirectByteBuffer(int cap, boolean direct) to allocate a direct ByteBuffer that is aligned to the page size. > 2) Added Util.getTemporaryAlignedDirectBuffer(int size) > 3) Added DirectByteBuffer.isAligned(int pos) to check if the buffer is aligned before doing native IO with it. > > 3. Moved all the alignment check from C code to Java code (mainly FileChannelImpl and IOUtil.java). > > 4. Made the DirectIO functionality consistent between read and write operations. With current version of the patch, user would be responsible for the alignment with file offset and IO size. > > 5. Made the API for DirectIO more extensible to all the supporting platforms. > 1) Unix OS specific code are done through UnixConstants.java.template and FileChannelImpl.c. > 2) Coded and tested for Linux and OS X (OS X testing is done through VirtualBox with OS X VM on top of Linux OS). > 3) Coded for Solaris. We do not have environment to test it so that we commented the changes out. > > 6. We added 4 test cases following the existing non-direct IO examples. > > 7. We did jtreg test for the entire nio package and no errors were found due to our changes. > > Please let us know your feedback and comment. Thank you very much for your time and consideration! > > Thanks, > Lucy > > -----Original Message----- > From: Alan Bateman [mailto:Alan.Bateman at oracle.com] > Sent: Monday, October 17, 2016 7:59 AM > To: Lu, Yingqi > Cc: nio-dev at openjdk.java.net; Kaczmarek, Eric ; > Kharbas, Kishor > Subject: Re: Proposal for adding O_DIRECT support into JDK 9 > > On 12/10/2016 17:41, Lu, Yingqi wrote: > > : > > You are correct about the "extra copy" with DirectIO. Will it be acceptable if we > add a function "Util.getAlignedTemporaryDirectBuffer" and use that for the > DirectIO operation? In this case, I think we should be able to avoid the additional > copy? > Yes, that should work but it still lacks a way for the user to get an aligned buffer > and so you will always be copying in and out of an aligned buffer. The other thing > is the sizing of the I/O operation where I think you will also need a way to expose > the multiple (or block size) to the user. > > -Alan. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From yingqi.lu at intel.com Fri Nov 4 17:08:59 2016 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Fri, 4 Nov 2016 17:08:59 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <36CE90D5-B86C-4E0A-A2BD-AC812D4262D1@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5134F66@ORSMSX113.amr.corp.intel.com> <062c589c-fa54-1652-683a-4e779816391d@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51447D9@ORSMSX113.amr.corp.intel.com> <006c75c4-1404-d66b-737d-e07ea2b493fd@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5166194@ORSMSX113.amr.corp.intel.com> <4d4c5ae5-06f9-40a9-f29c-56d0a44e671d@! ! gmail.com> <9ACD5B67AAC5594CB6268234CF29CF9AA516782E@ORSMSX113.amr.corp.intel.com> <36CE90D5-B86C-4E0A-A2BD-AC812D4262D1@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AA516893D@ORSMSX113.amr.corp.intel.com> Hi Paul, Thank you very much for all your suggestions. Currently, we add getAlignment() function to FileChannel and override it in FileChannelImpl with details. Basically, it just calls sun.nio.ch.Util.pageSize(). However, if you think exposing pageSize() from java.lang.Runtime is a better approach, we can certainly do that. Please let us know! Thanks, Lucy >-----Original Message----- >From: Paul Sandoz [mailto:paul.sandoz at oracle.com] >Sent: Friday, November 04, 2016 9:58 AM >To: Lu, Yingqi >Cc: Peter Levart ; Alan Bateman >; nio-dev at openjdk.java.net; Kaczmarek, Eric >; core-libs-dev at openjdk.java.net; Kharbas, Kishor > >Subject: Re: Proposal for adding O_DIRECT support into JDK 9 > >Hi Lucy, > >I am struggling to find an entirely satisfactory location to place a method such as >getPageSize, but java.lang.Runtime seems less worse than other places :-) > >Paul. > >> On 3 Nov 2016, at 12:49, Lu, Yingqi wrote: >> >> Hi Paul and Peter, >> >> Thank you very much for pointing this out. We did not know about it J >> >> We will modify the patch accordingly and will send it here for review. >> >> Thanks, >> Lucy >> >> From: Paul Sandoz [mailto:paul.sandoz at oracle.com] >> Sent: Thursday, November 03, 2016 12:46 PM >> To: Peter Levart >> Cc: Lu, Yingqi ; Alan Bateman >> ; nio-dev at openjdk.java.net; Kaczmarek, Eric >> ; core-libs-dev at openjdk.java.net; Kharbas, >> Kishor >> Subject: Re: Proposal for adding O_DIRECT support into JDK 9 >> >> Hi Peter, >> >> Thanks for pointing this out, you beat me to it :-) >> >> Note that the alignment for a unit size corresponding to page size will only be >stable for direct byte buffers, since for heap byte buffers the GC can move the >underlying byte array and the alignment properties may change. >> >> Paul. >> >> On 3 Nov 2016, at 12:38, Peter Levart wrote: >> >> Hi Lucy, >> >> Did you know that in JDK 9 the following two methods have been added to >java.nio.ByteBuffer: >> >> /** @return The indexed byte's memory address modulus the unit size >> */ >> public final int alignmentOffset(int index, int unitSize); >> >> /** @return new byte buffer whose content is a shared and aligned >subsequence of this buffer's content >> */ >> public final ByteBuffer alignedSlice(int unitSize); >> >> >> So you could express your proposed methods ByteBuffer.allocateDirectAligned() >and ByteBuffer.isAligned() with the above two methods in the following way: >> >> instead of: >> >> ByteBuffer bb = ByteBuffer.allocateDirectAligned(capacity) >> >> you could do: >> >> int pageSize = ... >> ByteBuffer bb = ByteBuffer.allocateDirect(capacity + pageSize - >> 1).alignedSlice(pageSize); >> >> And instead of: >> >> bb.isAligned() >> >> you could do: >> >> bb.alignmentOffset(pageSize) == 0 >> >> >> >> The only thing that needs to be added is a public API to return the pageSize (i.e. >Bits.pageSize()). >> >> >> What do you think? >> >> Regards, Peter >> >> On 11/03/2016 12:34 AM, Lu, Yingqi wrote: >> Hi All, >> >> Our most recent DirectIO patch is available at >> http://cr.openjdk.java.net/~igraves/8164900-3/ >> >> In this version, we made following changes: >> >> 1. Removed the flag "direct" from FileDescriptor class. Instead, moved it to the >FileChannelImpl class. >> >> 2. Provided a way for user to allocate a page aligned direct ByteBuffer. >> 1) Added a constructor DirectByteBuffer(int cap, boolean direct) to allocate a >direct ByteBuffer that is aligned to the page size. >> 2) Added Util.getTemporaryAlignedDirectBuffer(int size) >> 3) Added DirectByteBuffer.isAligned(int pos) to check if the buffer is aligned >before doing native IO with it. >> >> 3. Moved all the alignment check from C code to Java code (mainly >FileChannelImpl and IOUtil.java). >> >> 4. Made the DirectIO functionality consistent between read and write >operations. With current version of the patch, user would be responsible for the >alignment with file offset and IO size. >> >> 5. Made the API for DirectIO more extensible to all the supporting platforms. >> 1) Unix OS specific code are done through UnixConstants.java.template and >FileChannelImpl.c. >> 2) Coded and tested for Linux and OS X (OS X testing is done through >VirtualBox with OS X VM on top of Linux OS). >> 3) Coded for Solaris. We do not have environment to test it so that we >commented the changes out. >> >> 6. We added 4 test cases following the existing non-direct IO examples. >> >> 7. We did jtreg test for the entire nio package and no errors were found due to >our changes. >> >> Please let us know your feedback and comment. Thank you very much for your >time and consideration! >> >> Thanks, >> Lucy >> >> -----Original Message----- >> From: Alan Bateman [mailto:Alan.Bateman at oracle.com] >> Sent: Monday, October 17, 2016 7:59 AM >> To: Lu, Yingqi >> Cc: nio-dev at openjdk.java.net; Kaczmarek, Eric >> ; Kharbas, Kishor >> Subject: Re: Proposal for adding O_DIRECT support into JDK 9 >> >> On 12/10/2016 17:41, Lu, Yingqi wrote: >> >> : >> >> You are correct about the "extra copy" with DirectIO. Will it be >> acceptable if we add a function "Util.getAlignedTemporaryDirectBuffer" >> and use that for the DirectIO operation? In this case, I think we >> should be able to avoid the additional copy? >> Yes, that should work but it still lacks a way for the user to get an >> aligned buffer and so you will always be copying in and out of an >> aligned buffer. The other thing is the sizing of the I/O operation >> where I think you will also need a way to expose the multiple (or block size) to >the user. >> >> -Alan. From david.lloyd at redhat.com Fri Nov 4 18:32:01 2016 From: david.lloyd at redhat.com (David M. Lloyd) Date: Fri, 4 Nov 2016 13:32:01 -0500 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <36CE90D5-B86C-4E0A-A2BD-AC812D4262D1@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51447D9@ORSMSX113.amr.corp.intel.com> <006c75c4-1404-d66b-737d-e07ea2b493fd@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5166194@ORSMSX113.amr.corp.intel.com> <4d4c5ae5-06f9-40a9-f29c-56d0a44e671d@! ! gmail.com> <9ACD5B67AAC5594CB6268234CF29CF9AA516782E@ORSMSX113.amr.corp.intel.com> <36CE90D5-B86C-4E0A-A2BD-AC812D4262D1@oracle.com> Message-ID: <82b85038-36a9-bd93-7131-2d9a2607e60a@redhat.com> I still feel like it could be a problem to have just one simple getPageSize method, given how many CPU architectures and OSes support multiple page sizes. On 11/04/2016 11:58 AM, Paul Sandoz wrote: > Hi Lucy, > > I am struggling to find an entirely satisfactory location to place a method such as getPageSize, but java.lang.Runtime seems less worse than other places :-) > > Paul. > >> On 3 Nov 2016, at 12:49, Lu, Yingqi wrote: >> >> Hi Paul and Peter, >> >> Thank you very much for pointing this out. We did not know about it J >> >> We will modify the patch accordingly and will send it here for review. >> >> Thanks, >> Lucy >> >> From: Paul Sandoz [mailto:paul.sandoz at oracle.com] >> Sent: Thursday, November 03, 2016 12:46 PM >> To: Peter Levart >> Cc: Lu, Yingqi ; Alan Bateman ; nio-dev at openjdk.java.net; Kaczmarek, Eric ; core-libs-dev at openjdk.java.net; Kharbas, Kishor >> Subject: Re: Proposal for adding O_DIRECT support into JDK 9 >> >> Hi Peter, >> >> Thanks for pointing this out, you beat me to it :-) >> >> Note that the alignment for a unit size corresponding to page size will only be stable for direct byte buffers, since for heap byte buffers the GC can move the underlying byte array and the alignment properties may change. >> >> Paul. >> >> On 3 Nov 2016, at 12:38, Peter Levart wrote: >> >> Hi Lucy, >> >> Did you know that in JDK 9 the following two methods have been added to java.nio.ByteBuffer: >> >> /** @return The indexed byte's memory address modulus the unit size >> */ >> public final int alignmentOffset(int index, int unitSize); >> >> /** @return new byte buffer whose content is a shared and aligned subsequence of this buffer's content >> */ >> public final ByteBuffer alignedSlice(int unitSize); >> >> >> So you could express your proposed methods ByteBuffer.allocateDirectAligned() and ByteBuffer.isAligned() with the above two methods in the following way: >> >> instead of: >> >> ByteBuffer bb = ByteBuffer.allocateDirectAligned(capacity) >> >> you could do: >> >> int pageSize = ... >> ByteBuffer bb = ByteBuffer.allocateDirect(capacity + pageSize - 1).alignedSlice(pageSize); >> >> And instead of: >> >> bb.isAligned() >> >> you could do: >> >> bb.alignmentOffset(pageSize) == 0 >> >> >> >> The only thing that needs to be added is a public API to return the pageSize (i.e. Bits.pageSize()). >> >> >> What do you think? >> >> Regards, Peter >> >> On 11/03/2016 12:34 AM, Lu, Yingqi wrote: >> Hi All, >> >> Our most recent DirectIO patch is available at http://cr.openjdk.java.net/~igraves/8164900-3/ >> >> In this version, we made following changes: >> >> 1. Removed the flag "direct" from FileDescriptor class. Instead, moved it to the FileChannelImpl class. >> >> 2. Provided a way for user to allocate a page aligned direct ByteBuffer. >> 1) Added a constructor DirectByteBuffer(int cap, boolean direct) to allocate a direct ByteBuffer that is aligned to the page size. >> 2) Added Util.getTemporaryAlignedDirectBuffer(int size) >> 3) Added DirectByteBuffer.isAligned(int pos) to check if the buffer is aligned before doing native IO with it. >> >> 3. Moved all the alignment check from C code to Java code (mainly FileChannelImpl and IOUtil.java). >> >> 4. Made the DirectIO functionality consistent between read and write operations. With current version of the patch, user would be responsible for the alignment with file offset and IO size. >> >> 5. Made the API for DirectIO more extensible to all the supporting platforms. >> 1) Unix OS specific code are done through UnixConstants.java.template and FileChannelImpl.c. >> 2) Coded and tested for Linux and OS X (OS X testing is done through VirtualBox with OS X VM on top of Linux OS). >> 3) Coded for Solaris. We do not have environment to test it so that we commented the changes out. >> >> 6. We added 4 test cases following the existing non-direct IO examples. >> >> 7. We did jtreg test for the entire nio package and no errors were found due to our changes. >> >> Please let us know your feedback and comment. Thank you very much for your time and consideration! >> >> Thanks, >> Lucy >> >> -----Original Message----- >> From: Alan Bateman [mailto:Alan.Bateman at oracle.com] >> Sent: Monday, October 17, 2016 7:59 AM >> To: Lu, Yingqi >> Cc: nio-dev at openjdk.java.net; Kaczmarek, Eric ; >> Kharbas, Kishor >> Subject: Re: Proposal for adding O_DIRECT support into JDK 9 >> >> On 12/10/2016 17:41, Lu, Yingqi wrote: >> >> : >> >> You are correct about the "extra copy" with DirectIO. Will it be acceptable if we >> add a function "Util.getAlignedTemporaryDirectBuffer" and use that for the >> DirectIO operation? In this case, I think we should be able to avoid the additional >> copy? >> Yes, that should work but it still lacks a way for the user to get an aligned buffer >> and so you will always be copying in and out of an aligned buffer. The other thing >> is the sizing of the I/O operation where I think you will also need a way to expose >> the multiple (or block size) to the user. >> >> -Alan. > -- - DML From Alan.Burlison at oracle.com Sat Nov 5 08:13:18 2016 From: Alan.Burlison at oracle.com (Alan Burlison) Date: Sat, 5 Nov 2016 08:13:18 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <82b85038-36a9-bd93-7131-2d9a2607e60a@redhat.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51447D9@ORSMSX113.amr.corp.intel.com> <006c75c4-1404-d66b-737d-e07ea2b493fd@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5166194@ORSMSX113.amr.corp.intel.com> <4d4c5ae5-06f9-40a9-f29c-56d0a44e671d@! ! gmail.com> <9ACD5B67AAC5594CB6268234CF29CF9AA516782E@ORSMSX113.amr.corp.intel.com> <36CE90D5-B86C-4E0A-A2BD-AC812D4262D1@oracle.com> <82b85038-36a9-bd93-7131-2d9a2607e60a@redhat.com> Message-ID: <706b2cc6-6fb1-a006-5549-bea484edb5fb@oracle.com> On 04/11/2016 18:32, David M. Lloyd wrote: > I still feel like it could be a problem to have just one simple > getPageSize method, given how many CPU architectures and OSes support > multiple page sizes. Yes. $ pagesize -a 8192 65536 4194304 268435456 2147483648 -- Alan Burlison -- From david.holmes at oracle.com Sat Nov 5 17:42:15 2016 From: david.holmes at oracle.com (David Holmes) Date: Sun, 6 Nov 2016 03:42:15 +1000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <36CE90D5-B86C-4E0A-A2BD-AC812D4262D1@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51447D9@ORSMSX113.amr.corp.intel.com> <006c75c4-1404-d66b-737d-e07ea2b493fd@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5166194@ORSMSX113.amr.corp.intel.com> <4d4c5ae5-06f9-40a9-f29c-56d0a44e671d@! ! gmail.com> <9ACD5B67AAC5594CB6268234CF29CF9AA516782E@ORSMSX113.amr.corp.intel.com> <36CE90D5-B86C-4E0A-A2BD-AC812D4262D1@oracle.com> Message-ID: <66b40d54-fcf7-4258-6499-082742727e66@oracle.com> Hi Paul, On 5/11/2016 2:58 AM, Paul Sandoz wrote: > Hi Lucy, > > I am struggling to find an entirely satisfactory location to place a method such as getPageSize, but java.lang.Runtime seems less worse than other places :-) This does not seem an appropriate method to expose in this form at the java.lang.Runtime level. If we want a public, platform agnostic API for exposing virtual memory page sizes (plural) then we should design such an API IMHO. David > Paul. > >> On 3 Nov 2016, at 12:49, Lu, Yingqi wrote: >> >> Hi Paul and Peter, >> >> Thank you very much for pointing this out. We did not know about it J >> >> We will modify the patch accordingly and will send it here for review. >> >> Thanks, >> Lucy >> >> From: Paul Sandoz [mailto:paul.sandoz at oracle.com] >> Sent: Thursday, November 03, 2016 12:46 PM >> To: Peter Levart >> Cc: Lu, Yingqi ; Alan Bateman ; nio-dev at openjdk.java.net; Kaczmarek, Eric ; core-libs-dev at openjdk.java.net; Kharbas, Kishor >> Subject: Re: Proposal for adding O_DIRECT support into JDK 9 >> >> Hi Peter, >> >> Thanks for pointing this out, you beat me to it :-) >> >> Note that the alignment for a unit size corresponding to page size will only be stable for direct byte buffers, since for heap byte buffers the GC can move the underlying byte array and the alignment properties may change. >> >> Paul. >> >> On 3 Nov 2016, at 12:38, Peter Levart wrote: >> >> Hi Lucy, >> >> Did you know that in JDK 9 the following two methods have been added to java.nio.ByteBuffer: >> >> /** @return The indexed byte's memory address modulus the unit size >> */ >> public final int alignmentOffset(int index, int unitSize); >> >> /** @return new byte buffer whose content is a shared and aligned subsequence of this buffer's content >> */ >> public final ByteBuffer alignedSlice(int unitSize); >> >> >> So you could express your proposed methods ByteBuffer.allocateDirectAligned() and ByteBuffer.isAligned() with the above two methods in the following way: >> >> instead of: >> >> ByteBuffer bb = ByteBuffer.allocateDirectAligned(capacity) >> >> you could do: >> >> int pageSize = ... >> ByteBuffer bb = ByteBuffer.allocateDirect(capacity + pageSize - 1).alignedSlice(pageSize); >> >> And instead of: >> >> bb.isAligned() >> >> you could do: >> >> bb.alignmentOffset(pageSize) == 0 >> >> >> >> The only thing that needs to be added is a public API to return the pageSize (i.e. Bits.pageSize()). >> >> >> What do you think? >> >> Regards, Peter >> >> On 11/03/2016 12:34 AM, Lu, Yingqi wrote: >> Hi All, >> >> Our most recent DirectIO patch is available at http://cr.openjdk.java.net/~igraves/8164900-3/ >> >> In this version, we made following changes: >> >> 1. Removed the flag "direct" from FileDescriptor class. Instead, moved it to the FileChannelImpl class. >> >> 2. Provided a way for user to allocate a page aligned direct ByteBuffer. >> 1) Added a constructor DirectByteBuffer(int cap, boolean direct) to allocate a direct ByteBuffer that is aligned to the page size. >> 2) Added Util.getTemporaryAlignedDirectBuffer(int size) >> 3) Added DirectByteBuffer.isAligned(int pos) to check if the buffer is aligned before doing native IO with it. >> >> 3. Moved all the alignment check from C code to Java code (mainly FileChannelImpl and IOUtil.java). >> >> 4. Made the DirectIO functionality consistent between read and write operations. With current version of the patch, user would be responsible for the alignment with file offset and IO size. >> >> 5. Made the API for DirectIO more extensible to all the supporting platforms. >> 1) Unix OS specific code are done through UnixConstants.java.template and FileChannelImpl.c. >> 2) Coded and tested for Linux and OS X (OS X testing is done through VirtualBox with OS X VM on top of Linux OS). >> 3) Coded for Solaris. We do not have environment to test it so that we commented the changes out. >> >> 6. We added 4 test cases following the existing non-direct IO examples. >> >> 7. We did jtreg test for the entire nio package and no errors were found due to our changes. >> >> Please let us know your feedback and comment. Thank you very much for your time and consideration! >> >> Thanks, >> Lucy >> >> -----Original Message----- >> From: Alan Bateman [mailto:Alan.Bateman at oracle.com] >> Sent: Monday, October 17, 2016 7:59 AM >> To: Lu, Yingqi >> Cc: nio-dev at openjdk.java.net; Kaczmarek, Eric ; >> Kharbas, Kishor >> Subject: Re: Proposal for adding O_DIRECT support into JDK 9 >> >> On 12/10/2016 17:41, Lu, Yingqi wrote: >> >> : >> >> You are correct about the "extra copy" with DirectIO. Will it be acceptable if we >> add a function "Util.getAlignedTemporaryDirectBuffer" and use that for the >> DirectIO operation? In this case, I think we should be able to avoid the additional >> copy? >> Yes, that should work but it still lacks a way for the user to get an aligned buffer >> and so you will always be copying in and out of an aligned buffer. The other thing >> is the sizing of the I/O operation where I think you will also need a way to expose >> the multiple (or block size) to the user. >> >> -Alan. > From yingqi.lu at intel.com Sat Nov 5 18:33:12 2016 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Sat, 5 Nov 2016 18:33:12 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <706b2cc6-6fb1-a006-5549-bea484edb5fb@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51447D9@ORSMSX113.amr.corp.intel.com> <006c75c4-1404-d66b-737d-e07ea2b493fd@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5166194@ORSMSX113.amr.corp.intel.com> <4d4c5ae5-06f9-40a9-f29c-56d0a44e671d@! ! gmail.com> <9ACD5B67AAC5594CB6268234CF29CF9AA516782E@ORSMSX113.amr.corp.intel.com> <36CE90D5-B86C-4E0A-A2BD-AC812D4262D1@oracle.com> <82b85038-36a9-bd93-7131-2d9a2607e60a@redhat.com> <706b2cc6-6fb1-a006-5549-bea484edb5fb@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AA51697D2@ORSMSX113.amr.corp.intel.com> Hi Alan and David, Thank you for your email. I totally agree that OS kernel supports multiple page sizes. For example, my Linux box supports both 4K and 2M pages. Currently, our getAlignment() function inside FileChannel/FileChannelImpl calls an existing function sun.nio.ch.Util.pageSize() which eventually calls unsafe.pageSize(). I use a small Java program to test unsafe.pageSize() with and without the flag " -XX:+UseLargePages" on my Linux box, both testing return 4096. To make DirectIO work, IO size and file position need to be aligned to the disk block size. Native memory page size (in our case, 4096) is always a multiple or equal of disk block size. That is why we try to use this existing JDK function (sun.nio.ch.Util.pageSize()) for the alignment value. To make it simpler, we use getAlignment as the function name and put it inside the FileChannel/FileChannelImpl. Please let us know if you have any questions, comments and concerns. Thank you very much for your consideration! Thanks, Lucy >-----Original Message----- >From: Alan Burlison [mailto:Alan.Burlison at oracle.com] >Sent: Saturday, November 05, 2016 1:13 AM >To: David M. Lloyd ; Paul Sandoz >; Lu, Yingqi >Cc: nio-dev at openjdk.java.net; Kaczmarek, Eric ; >core-libs-dev at openjdk.java.net; Kharbas, Kishor >Subject: Re: Proposal for adding O_DIRECT support into JDK 9 > >On 04/11/2016 18:32, David M. Lloyd wrote: > >> I still feel like it could be a problem to have just one simple >> getPageSize method, given how many CPU architectures and OSes support >> multiple page sizes. > >Yes. > >$ pagesize -a >8192 >65536 >4194304 >268435456 >2147483648 > >-- >Alan Burlison >-- From brian.burkhalter at oracle.com Sat Nov 5 19:14:58 2016 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Sat, 5 Nov 2016 12:14:58 -0700 Subject: [9] RFR 8166253: (ch) FileLock object can get GC'd and result in unexpected release of file lock Message-ID: <1D32417F-2DAA-4C83-9D63-71D58C2028DE@oracle.com> This review request supersedes the previous .00 version [1] for this issue. The current .01 version of the patch as proposed here is incomplete as it does not address AsynchronousFileChannels. I wanted to post it at this time however as it seems overcomplicated and I do not wish to expend more effort on it unless it is actually headed in the right direction. Issue: https://bugs.openjdk.java.net/browse/JDK-8166253 Patch: http://cr.openjdk.java.net/~bpb/8166253/webrev.01/ The FileLocks for a given file are maintained via an internal table wherein the values are lists of WeakReferences to the FileLock instances corresponding to a given channel (file). No hard references to the FileLocks are maintained as this could prevent garbage collection of the FileChannel associated with the FileLock as the latter contains a hard reference to the former. The problem is that if the calling code which requests the lock does not maintain a hard reference to the FileLock, then it could be GCed without actually releasing the underlying native lock. In this situation it appears to the Java layer that the lock no longer exists although it has not been released by the native layer. This could result in unexpected behavior both within the Java process itself and between it and other processes using locks on the file. The approach here is to keep track of the state of valid FileLocks which have become unreachable and have been removed from the associated ReferenceQueue. The FileLockState maintains the position, size, and validity of the lock. The shared status of the lock is irrelevant as the state if used is only to release the lock which does not require specifying shared or exclusive. A reference to the FileLockState is maintained in the FileLock instance, by the FileChannel on which the lock was obtained, and in the FileLockReference. The FileLockState is invalidated if the FileLock is invalidated or if the channel is closed, whichever occurs first. In FileLockTable.removeStaleEntries(), if it is determined that the FileLockState of the lock removed from the ReferenceQueue is still valid, the FileLockState is added to a global table under the same key as that used for the lock in the global table of lists of FileLockReferences. If the FileLockState is invalid, then the state instance is removed from that table. In FileLockTable.checkList(), the list of FileLockStates is checked for the file key to see whether any valid orphaned states overlap the requested region and if so an exception is thrown. Any invalid states encountered are removed from the list and if the list becomes empty it and it?s key are removed from the table. The handling of the contents of the table of states was mirrored from that of the table of lists of FileLockReferences. In FileChannelImpl, a list of FileLockStates of locks acquired on the channel is maintained. In implCloseChannel() this list is traversed and any regions associated with valid states are released and the state instance invalidated. This patch passes the core NIO regression tests on all the usual platforms. The test included in the patch fails on all platforms if the implementation portion of the patch is omitted. Thanks, Brian [1] http://mail.openjdk.java.net/pipermail/nio-dev/2016-September/003881.html From brian.burkhalter at oracle.com Sat Nov 5 19:15:01 2016 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Sat, 5 Nov 2016 12:15:01 -0700 Subject: [9] PING: RFC on 8165823 & 8168500: (se) EPollArrayWrapper optimization for large file descriptors Message-ID: <46855902-E839-49CF-9D44-58B14C18C5E5@oracle.com> This post follows from the incomplete thread [1] regarding issue [2]. A request for more information was sent on Oct. 6 to the original submitter of [2] and the issue was marked as Resolved-Incomplete. The information requested was whether the file descriptor number resource limit could change dynamically in the submitter?s execution environment. As of now, no response has yet been received. Another issue [3] was filed so as not to lose track of the problem in case [2] remains incomplete. Two solutions [4, 5] were proposed. Solution [4] removes initialization of the eventsHigh map based on resource limits in the EPollArrayWrapper constructor in favor of creating the map lazily if it is actually needed. Solution [5] always initializes the eventsHigh map during EPollArrayWrapper instantiation on the basis that this is simpler and has extremely low impact on performance and footprint. My question here is whether either of the proposed solutions [4, 5] would be acceptable, an alternative solution should be investigated, or the topic abandoned until some future date? Thanks, Brian [1] http://mail.openjdk.java.net/pipermail/nio-dev/2016-October/003896.html [2] https://bugs.openjdk.java.net/browse/JDK-8165823 [3] https://bugs.openjdk.java.net/browse/JDK-8168500 [4] http://cr.openjdk.java.net/~bpb/8165823/webrev.00/ [5] http://mail.openjdk.java.net/pipermail/nio-dev/2016-October/003947.html From Alan.Bateman at oracle.com Sat Nov 5 20:31:37 2016 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 5 Nov 2016 20:31:37 +0000 Subject: [9] PING: RFC on 8165823 & 8168500: (se) EPollArrayWrapper optimization for large file descriptors In-Reply-To: <46855902-E839-49CF-9D44-58B14C18C5E5@oracle.com> References: <46855902-E839-49CF-9D44-58B14C18C5E5@oracle.com> Message-ID: On 05/11/2016 19:15, Brian Burkhalter wrote: > This post follows from the incomplete thread [1] regarding issue [2]. > > A request for more information was sent on Oct. 6 to the original submitter of [2] and the issue was marked as Resolved-Incomplete. The information requested was whether the file descriptor number resource limit could change dynamically in the submitter?s execution environment. As of now, no response has yet been received. Another issue [3] was filed so as not to lose track of the problem in case [2] remains incomplete. > > Two solutions [4, 5] were proposed. Solution [4] removes initialization of the eventsHigh map based on resource limits in the EPollArrayWrapper constructor in favor of creating the map lazily if it is actually needed. Solution [5] always initializes the eventsHigh map during EPollArrayWrapper instantiation on the basis that this is simpler and has extremely low impact on performance and footprint. > > My question here is whether either of the proposed solutions [4, 5] would be acceptable, an alternative solution should be investigated, or the topic abandoned until some future date? Changing eventsHigh to final and always initializing it to an empty HashMap should be fine. However I do think we are just guessing that the nfiles is changing dynamically, hopefully the original submitter will come back to say more about the environment. -Alan From yingqi.lu at intel.com Sun Nov 6 15:51:14 2016 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Sun, 6 Nov 2016 15:51:14 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513500C@ORSMSX113.amr.corp.intel.com> <542288c7-b0b1-5b41-4c45-d07730e3cddf@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51447D9@ORSMSX113.amr.corp.intel.com> <006c75c4-1404-d66b-737d-e07ea2b493fd@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5166194@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA516782E@ORSMSX113.amr.corp.intel.com> <36CE90D5-B86C-4E0A-A2BD-AC812D4262D1@oracle.com> <66b40d54-fcf7-4258-6499-082742727e66@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AA5169EFD@ORSMSX113.amr.corp.intel.com> Another approach might be use ioctl to get the block size and use that as the alignment value. I think ioctl is available on UNIX based OSes (please chime in if I miss any OS here) and has a similar call for Windows named DeviceIoControl. The request code of ioctl is different from OS to OS, but we could address that in the native code. What do you all think on this? Thanks, Lucy From: Thomas St?fe [mailto:thomas.stuefe at gmail.com] Sent: Sunday, November 06, 2016 3:22 AM To: David Holmes Cc: Paul Sandoz ; Lu, Yingqi ; nio-dev at openjdk.java.net; Kaczmarek, Eric ; core-libs-dev at openjdk.java.net; Kharbas, Kishor Subject: Re: Proposal for adding O_DIRECT support into JDK 9 On Sat, Nov 5, 2016 at 6:42 PM, David Holmes > wrote: Hi Paul, On 5/11/2016 2:58 AM, Paul Sandoz wrote: Hi Lucy, I am struggling to find an entirely satisfactory location to place a method such as getPageSize, but java.lang.Runtime seems less worse than other places :-) This does not seem an appropriate method to expose in this form at the java.lang.Runtime level. If we want a public, platform agnostic API for exposing virtual memory page sizes (plural) then we should design such an API IMHO. I absolutely agree. Different memory regions can have different page sizes following different rules. The fact that in hotspot coding os::vm_page_size() is used in many places as "the pagesize" gave us lots of headaches in the AIX port, where we can have up to four different page sizes for heap, stack, text and different instances of shared memory. I consider the fact that unsafe.pageSize() exists sloppy API design. Even on Linux we can have two page sizes within one process, or? If we really need a generic "pageSize" API this should be designed to cover a reasonable large array of OS breeds. IMHO a good generic API should allow me to query, given a memory range, its page-size-composition, because within a region of memory one may have different page sizes. Thomas David Paul. On 3 Nov 2016, at 12:49, Lu, Yingqi > wrote: Hi Paul and Peter, Thank you very much for pointing this out. We did not know about it J We will modify the patch accordingly and will send it here for review. Thanks, Lucy From: Paul Sandoz [mailto:paul.sandoz at oracle.com] Sent: Thursday, November 03, 2016 12:46 PM To: Peter Levart > Cc: Lu, Yingqi >; Alan Bateman >; nio-dev at openjdk.java.net; Kaczmarek, Eric >; core-libs-dev at openjdk.java.net; Kharbas, Kishor > Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Peter, Thanks for pointing this out, you beat me to it :-) Note that the alignment for a unit size corresponding to page size will only be stable for direct byte buffers, since for heap byte buffers the GC can move the underlying byte array and the alignment properties may change. Paul. On 3 Nov 2016, at 12:38, Peter Levart > wrote: Hi Lucy, Did you know that in JDK 9 the following two methods have been added to java.nio.ByteBuffer: /** @return The indexed byte's memory address modulus the unit size */ public final int alignmentOffset(int index, int unitSize); /** @return new byte buffer whose content is a shared and aligned subsequence of this buffer's content */ public final ByteBuffer alignedSlice(int unitSize); So you could express your proposed methods ByteBuffer.allocateDirectAligned() and ByteBuffer.isAligned() with the above two methods in the following way: instead of: ByteBuffer bb = ByteBuffer.allocateDirectAligned(capacity) you could do: int pageSize = ... ByteBuffer bb = ByteBuffer.allocateDirect(capacity + pageSize - 1).alignedSlice(pageSize); And instead of: bb.isAligned() you could do: bb.alignmentOffset(pageSize) == 0 The only thing that needs to be added is a public API to return the pageSize (i.e. Bits.pageSize()). What do you think? Regards, Peter On 11/03/2016 12:34 AM, Lu, Yingqi wrote: Hi All, Our most recent DirectIO patch is available at http://cr.openjdk.java.net/~igraves/8164900-3/ In this version, we made following changes: 1. Removed the flag "direct" from FileDescriptor class. Instead, moved it to the FileChannelImpl class. 2. Provided a way for user to allocate a page aligned direct ByteBuffer. 1) Added a constructor DirectByteBuffer(int cap, boolean direct) to allocate a direct ByteBuffer that is aligned to the page size. 2) Added Util.getTemporaryAlignedDirectBuffer(int size) 3) Added DirectByteBuffer.isAligned(int pos) to check if the buffer is aligned before doing native IO with it. 3. Moved all the alignment check from C code to Java code (mainly FileChannelImpl and IOUtil.java). 4. Made the DirectIO functionality consistent between read and write operations. With current version of the patch, user would be responsible for the alignment with file offset and IO size. 5. Made the API for DirectIO more extensible to all the supporting platforms. 1) Unix OS specific code are done through UnixConstants.java.template and FileChannelImpl.c. 2) Coded and tested for Linux and OS X (OS X testing is done through VirtualBox with OS X VM on top of Linux OS). 3) Coded for Solaris. We do not have environment to test it so that we commented the changes out. 6. We added 4 test cases following the existing non-direct IO examples. 7. We did jtreg test for the entire nio package and no errors were found due to our changes. Please let us know your feedback and comment. Thank you very much for your time and consideration! Thanks, Lucy -----Original Message----- From: Alan Bateman [mailto:Alan.Bateman at oracle.com] Sent: Monday, October 17, 2016 7:59 AM To: Lu, Yingqi > Cc: nio-dev at openjdk.java.net; Kaczmarek, Eric >; Kharbas, Kishor > Subject: Re: Proposal for adding O_DIRECT support into JDK 9 On 12/10/2016 17:41, Lu, Yingqi wrote: : You are correct about the "extra copy" with DirectIO. Will it be acceptable if we add a function "Util.getAlignedTemporaryDirectBuffer" and use that for the DirectIO operation? In this case, I think we should be able to avoid the additional copy? Yes, that should work but it still lacks a way for the user to get an aligned buffer and so you will always be copying in and out of an aligned buffer. The other thing is the sizing of the I/O operation where I think you will also need a way to expose the multiple (or block size) to the user. -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Sun Nov 6 17:04:22 2016 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sun, 6 Nov 2016 17:04:22 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AA5169EFD@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51447D9@ORSMSX113.amr.corp.intel.com> <006c75c4-1404-d66b-737d-e07ea2b493fd@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5166194@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA516782E@ORSMSX113.amr.corp.intel.com> <36CE90D5-B86C-4E0A-A2BD-AC812D4262D1@oracle.com> <66b40d54-fcf7-4258-6499-082742727e66@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5169EFD@ORSMSX113.amr.corp.intel.com> Message-ID: <482dd41a-89c8-fe56-c9ab-e662a2f957e8@oracle.com> On 06/11/2016 15:51, Lu, Yingqi wrote: > Another approach might be use ioctl to get the block size and use that > as the alignment value. I think ioctl is available on UNIX based OSes > (please chime in if I miss any OS here) and has a similar call for > Windows named DeviceIoControl. The request code of ioctl is different > from OS to OS, but we could address that in the native code. > > A possible approach (I think I mentioned this in one of the early mails) is to have FileStore expose the block size that can be use to align and size the buffer. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Sun Nov 6 17:36:57 2016 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Sun, 6 Nov 2016 17:36:57 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <482dd41a-89c8-fe56-c9ab-e662a2f957e8@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51447D9@ORSMSX113.amr.corp.intel.com> <006c75c4-1404-d66b-737d-e07ea2b493fd@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5166194@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA516782E@ORSMSX113.amr.corp.intel.com> <36CE90D5-B86C-4E0A-A2BD-AC812D4262D1@oracle.com> <66b40d54-fcf7-4258-6499-082742727e66@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5169EFD@ORSMSX113.amr.corp.intel.com>, <482dd41a-89c8-fe56-c9ab-e662a2f957e8@oracle.com> Message-ID: <77D7708E-07FE-4E39-BF7A-30799EB8CEA7@intel.com> Hi Alan, Thank you very much for your suggestion. FileStore is a better home for the alignment value. We will use It in our next version of the patch and update here for review soon. Thanks, Lucy Sent from my iPhone On Nov 6, 2016, at 9:04 AM, Alan Bateman > wrote: On 06/11/2016 15:51, Lu, Yingqi wrote: Another approach might be use ioctl to get the block size and use that as the alignment value. I think ioctl is available on UNIX based OSes (please chime in if I miss any OS here) and has a similar call for Windows named DeviceIoControl. The request code of ioctl is different from OS to OS, but we could address that in the native code. A possible approach (I think I mentioned this in one of the early mails) is to have FileStore expose the block size that can be use to align and size the buffer. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.fuchs at oracle.com Mon Nov 7 16:10:19 2016 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Mon, 7 Nov 2016 16:10:19 +0000 Subject: [9] RFR 8166253: (ch) FileLock object can get GC'd and result in unexpected release of file lock In-Reply-To: <1D32417F-2DAA-4C83-9D63-71D58C2028DE@oracle.com> References: <1D32417F-2DAA-4C83-9D63-71D58C2028DE@oracle.com> Message-ID: <14b4340f-cf69-efcd-53ea-8bb956e6105e@oracle.com> Hi Brian, NIO is not my area of expertise, but I have some questions WRT to MT-safety of the proposed changes: FileChannelImpl.java: - is it safe for fileLockState to be a simple ArrayList? It seems to be modified outside of any critical section (lock/tryLock), and it is also iterated over from outside any critical section (line 137), so should this be a CopyOnWriteArrayList instead? FileLockImpl.java: - is there any reason while state is not final here? if it were final then maybe 'invalid' would no longer be needed - as it seems to be a duplicate of state.invalid FileLockTable.java: - The 'states' list is a simple ArrayList, which means that removeKeyIfStatesEmpty(fileKey, states) should be called while holding a lock on 'states', to make sure that nothing gets added to states while the method is running. This is good for line 295, but I believe line 348 should be moved up just after line 346 in order to be within the critical section - shouldn't it? best regards, -- daniel On 05/11/16 19:14, Brian Burkhalter wrote: > This review request supersedes the previous .00 version [1] for this issue. The current .01 version of the patch as proposed here is incomplete as it does not address AsynchronousFileChannels. I wanted to post it at this time however as it seems overcomplicated and I do not wish to expend more effort on it unless it is actually headed in the right direction. > > Issue: https://bugs.openjdk.java.net/browse/JDK-8166253 > Patch: http://cr.openjdk.java.net/~bpb/8166253/webrev.01/ > > The FileLocks for a given file are maintained via an internal table wherein the values are lists of WeakReferences to the FileLock instances corresponding to a given channel (file). No hard references to the FileLocks are maintained as this could prevent garbage collection of the FileChannel associated with the FileLock as the latter contains a hard reference to the former. The problem is that if the calling code which requests the lock does not maintain a hard reference to the FileLock, then it could be GCed without actually releasing the underlying native lock. In this situation it appears to the Java layer that the lock no longer exists although it has not been released by the native layer. This could result in unexpected behavior both within the Java process itself and between it and other processes using locks on the file. > > The approach here is to keep track of the state of valid FileLocks which have become unreachable and have been removed from the associated ReferenceQueue. The FileLockState maintains the position, size, and validity of the lock. The shared status of the lock is irrelevant as the state if used is only to release the lock which does not require specifying shared or exclusive. A reference to the FileLockState is maintained in the FileLock instance, by the FileChannel on which the lock was obtained, and in the FileLockReference. The FileLockState is invalidated if the FileLock is invalidated or if the channel is closed, whichever occurs first. > > In FileLockTable.removeStaleEntries(), if it is determined that the FileLockState of the lock removed from the ReferenceQueue is still valid, the FileLockState is added to a global table under the same key as that used for the lock in the global table of lists of FileLockReferences. If the FileLockState is invalid, then the state instance is removed from that table. > > In FileLockTable.checkList(), the list of FileLockStates is checked for the file key to see whether any valid orphaned states overlap the requested region and if so an exception is thrown. Any invalid states encountered are removed from the list and if the list becomes empty it and it?s key are removed from the table. The handling of the contents of the table of states was mirrored from that of the table of lists of FileLockReferences. > > In FileChannelImpl, a list of FileLockStates of locks acquired on the channel is maintained. In implCloseChannel() this list is traversed and any regions associated with valid states are released and the state instance invalidated. > > This patch passes the core NIO regression tests on all the usual platforms. The test included in the patch fails on all platforms if the implementation portion of the patch is omitted. > > Thanks, > > Brian > > [1] http://mail.openjdk.java.net/pipermail/nio-dev/2016-September/003881.html > From brian.burkhalter at oracle.com Mon Nov 7 20:18:37 2016 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 7 Nov 2016 12:18:37 -0800 Subject: [9] RFR 8166253: (ch) FileLock object can get GC'd and result in unexpected release of file lock In-Reply-To: <14b4340f-cf69-efcd-53ea-8bb956e6105e@oracle.com> References: <1D32417F-2DAA-4C83-9D63-71D58C2028DE@oracle.com> <14b4340f-cf69-efcd-53ea-8bb956e6105e@oracle.com> Message-ID: Hi Daniel, Thank you for your useful comments. On Nov 7, 2016, at 8:10 AM, Daniel Fuchs wrote: > NIO is not my area of expertise, but I have some questions WRT > to MT-safety of the proposed changes: I am not surprised that there are problems of this nature. I made some of the modifications to my prototype quickly with the objective of publishing the review request sooner as I am not certain whether this is conceptually even the best way to address the issue. > FileChannelImpl.java: > > - is it safe for fileLockState to be a simple ArrayList? > It seems to be modified outside of any critical section > (lock/tryLock), and it is also iterated over from outside > any critical section (line 137), so should this be a > CopyOnWriteArrayList instead? Indeed that looks to be a better choice. > FileLockImpl.java: > > - is there any reason while state is not final here? > if it were final then maybe 'invalid' would no longer > be needed - as it seems to be a duplicate of state.invalid Good point. Probably it would be better always to initialize the state in the FileLockImpl constructor and remove the latter?s ?invalid? instance variable. > FileLockTable.java: > > - The 'states' list is a simple ArrayList, which means > that removeKeyIfStatesEmpty(fileKey, states) should > be called while holding a lock on 'states', to make > sure that nothing gets added to states while the > method is running. > This is good for line 295, but I believe line 348 > should be moved up just after line 346 in order > to be within the critical section - shouldn't it? You are correct. I?ll make the suggested changes. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Mon Nov 7 22:13:20 2016 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 7 Nov 2016 14:13:20 -0800 Subject: [9] RFR 8166253: (ch) FileLock object can get GC'd and result in unexpected release of file lock In-Reply-To: References: <1D32417F-2DAA-4C83-9D63-71D58C2028DE@oracle.com> <14b4340f-cf69-efcd-53ea-8bb956e6105e@oracle.com> Message-ID: I have updated the patch to include the suggested changes: http://cr.openjdk.java.net/~bpb/8166253/webrev.02/ The pertinent regression tests still pass. Thanks, Brian On Nov 7, 2016, at 12:18 PM, Brian Burkhalter wrote: > I?ll make the suggested changes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.fuchs at oracle.com Tue Nov 8 14:44:53 2016 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Tue, 8 Nov 2016 14:44:53 +0000 Subject: [9] RFR 8166253: (ch) FileLock object can get GC'd and result in unexpected release of file lock In-Reply-To: References: <1D32417F-2DAA-4C83-9D63-71D58C2028DE@oracle.com> <14b4340f-cf69-efcd-53ea-8bb956e6105e@oracle.com> Message-ID: <6b81d498-0bf9-6ad7-53e1-3cc28a379cbb@oracle.com> Hi Brian, The updates look good to me. As for whether that's the most appropriate fix for the issue, then maybe the regular nio Reviewers will chime in :-) best regards, -- daniel On 07/11/16 22:13, Brian Burkhalter wrote: > I have updated the patch to include the suggested changes: > > http://cr.openjdk.java.net/~bpb/8166253/webrev.02/ > > The pertinent regression tests still pass. > > Thanks, > > Brian > > On Nov 7, 2016, at 12:18 PM, Brian Burkhalter > > wrote: > >> I?ll make the suggested changes. > From brian.burkhalter at oracle.com Fri Nov 11 01:28:38 2016 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 10 Nov 2016 17:28:38 -0800 Subject: [9] PING: RFC on 8165823 & 8168500: (se) EPollArrayWrapper optimization for large file descriptors In-Reply-To: References: <46855902-E839-49CF-9D44-58B14C18C5E5@oracle.com> Message-ID: On Nov 5, 2016, at 1:31 PM, Alan Bateman wrote: > Changing eventsHigh to final and always initializing it to an empty HashMap should be fine. However I do think we are just guessing that the nfiles is changing dynamically, hopefully the original submitter will come back to say more about the environment. I have re-contacted the original submitter but as yet have had no response. I was however able to reproduce the problem by modifying the LotsOfChannels test and the EPollArrayWrapper constructor to sleep at two particular places during which I executed the prlimit command as $ prlimit --nofile=512:512 --pid PID // before EPollArrayWrapper constructed $ sudo prlimit --nofile=2048:2048 --pid PID // at end of EPollArrayWrapper constructor after eventsHigh NOT initialized where PID is the process ID of the Java process executing LotsOfChannels. While this does not confirm that the RLIMIT_NOFILE resource changed dynamically in the submitter?s use case, it does verify that the scenario is plausible. It would seem therefore that always initializing a final eventsHigh to an empty HashMap is a reasonable defensive solution. Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From sybersnake at gmail.com Fri Nov 11 01:45:02 2016 From: sybersnake at gmail.com (Jon V.) Date: Thu, 10 Nov 2016 20:45:02 -0500 Subject: [9] PING: RFC on 8165823 & 8168500: (se) EPollArrayWrapper optimization for large file descriptors In-Reply-To: References: <46855902-E839-49CF-9D44-58B14C18C5E5@oracle.com> Message-ID: Does anyone know why the OPEN_MAX ?optimization? was added to begin with? Java must already be able to handle failed file opens already. This seems unnecessary. Maybe I?m missing something. On Thu, Nov 10, 2016 at 8:28 PM, Brian Burkhalter < brian.burkhalter at oracle.com> wrote: > On Nov 5, 2016, at 1:31 PM, Alan Bateman wrote: > > Changing eventsHigh to final and always initializing it to an empty > HashMap should be fine. However I do think we are just guessing that the > nfiles is changing dynamically, hopefully the original submitter will come > back to say more about the environment. > > > I have re-contacted the original submitter but as yet have had no > response. I was however able to reproduce the problem by modifying the > LotsOfChannels test and the EPollArrayWrapper constructor to sleep at two > particular places during which I executed the prlimit command as > > $ prlimit --nofile=512:512 --pid PID // before EPollArrayWrapper > constructed > $ sudo prlimit --nofile=2048:2048 --pid PID // at end of EPollArrayWrapper > constructor after eventsHigh NOT initialized > > where PID is the process ID of the Java process executing LotsOfChannels. > While this does not confirm that the RLIMIT_NOFILE resource changed > dynamically in the submitter?s use case, it does verify that the scenario > is plausible. It would seem therefore that always initializing a final > eventsHigh to an empty HashMap is a reasonable defensive solution. > > Brian > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Fri Nov 11 08:29:18 2016 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 11 Nov 2016 08:29:18 +0000 Subject: [9] PING: RFC on 8165823 & 8168500: (se) EPollArrayWrapper optimization for large file descriptors In-Reply-To: References: <46855902-E839-49CF-9D44-58B14C18C5E5@oracle.com> Message-ID: On 11/11/2016 01:28, Brian Burkhalter wrote: > I was however able to reproduce the problem by modifying the > LotsOfChannels test and the EPollArrayWrapper constructor to sleep at > two particular places during which I executed the prlimit command as > > $ prlimit --nofile=512:512 --pid PID // before EPollArrayWrapper > constructed > $ sudo prlimit --nofile=2048:2048 --pid PID // at end of > EPollArrayWrapper constructor after eventsHigh NOT initialized > > where PID is the process ID of the Java process executing > LotsOfChannels. While this does not confirm that the RLIMIT_NOFILE > resource changed dynamically in the submitter?s use case, it does > verify that the scenario is plausible. It would seem therefore that > always initializing a final eventsHigh to an empty HashMap is a > reasonable defensive solution. > That seems fine, got for it. -Alan. From brian.burkhalter at oracle.com Mon Nov 14 19:55:58 2016 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 14 Nov 2016 11:55:58 -0800 Subject: [9] PING: RFC on 8165823 & 8168500: (se) EPollArrayWrapper optimization for large file descriptors In-Reply-To: References: <46855902-E839-49CF-9D44-58B14C18C5E5@oracle.com> Message-ID: <18B9C3CE-790C-400D-B469-F3D5E933CC6F@oracle.com> On Nov 11, 2016, at 12:29 AM, Alan Bateman wrote: >> It would seem therefore that always initializing a final eventsHigh to an empty HashMap is a reasonable defensive solution. >> > That seems fine, got for it. Patch for final review: http://cr.openjdk.java.net/~bpb/8168500/webrev.00/ Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Nov 15 07:43:46 2016 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 15 Nov 2016 07:43:46 +0000 Subject: [9] PING: RFC on 8165823 & 8168500: (se) EPollArrayWrapper optimization for large file descriptors In-Reply-To: <18B9C3CE-790C-400D-B469-F3D5E933CC6F@oracle.com> References: <46855902-E839-49CF-9D44-58B14C18C5E5@oracle.com> <18B9C3CE-790C-400D-B469-F3D5E933CC6F@oracle.com> Message-ID: <9ff04b85-82cc-6473-4308-7ecfce2e0914@oracle.com> On 14/11/2016 19:55, Brian Burkhalter wrote: > On Nov 11, 2016, at 12:29 AM, Alan Bateman > wrote: > >>> It would seem therefore that always initializing a final eventsHigh >>> to an empty HashMap is a reasonable defensive solution. >>> >> That seems fine, got for it. > > Patch for final review: > > http://cr.openjdk.java.net/~bpb/8168500/webrev.00/ > > Looks fine. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergei.kovalev at oracle.com Tue Nov 15 11:21:16 2016 From: sergei.kovalev at oracle.com (Sergei Kovalev) Date: Tue, 15 Nov 2016 14:21:16 +0300 Subject: RFR(s): 8169710: com/sun/nio/sctp tests has undeclared dependency on jdk.sctp module Message-ID: Hello team, Please review very small fix for test suite from com/sun/nio/sctp folder. BugID: https://bugs.openjdk.java.net/browse/JDK-8169710 Web review: http://cr.openjdk.java.net/~skovalev/8169710/webrev.00/ Issue: The tests failing in case of usage "--limit modules java.base" command line options. Solution: add module declaration for tests. Because all tests from the appropriate folders are affected it is better to add "TEST.properties" file that contains dependency declaration to each folder. In this case we have no need to modify each test separately. -- With best regards, Sergei From daniel.fuchs at oracle.com Tue Nov 15 11:26:28 2016 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Tue, 15 Nov 2016 11:26:28 +0000 Subject: RFR(s): 8169710: com/sun/nio/sctp tests has undeclared dependency on jdk.sctp module In-Reply-To: References: Message-ID: <990b3eb5-d1d2-1008-6841-49bd6f06b1d5@oracle.com> Hi Sergey, I'm not an expert in TEST.properties but this looks bizarre to me. Why not simply put a single TEST.properties file in com/sun/nio/sctp? best regards -- daniel On 15/11/16 11:21, Sergei Kovalev wrote: > Hello team, > > Please review very small fix for test suite from com/sun/nio/sctp folder. > > BugID: https://bugs.openjdk.java.net/browse/JDK-8169710 > Web review: http://cr.openjdk.java.net/~skovalev/8169710/webrev.00/ > > Issue: The tests failing in case of usage "--limit modules java.base" > command line options. > Solution: add module declaration for tests. Because all tests from the > appropriate folders are affected it is better to add "TEST.properties" > file that contains dependency declaration to each folder. In this case > we have no need to modify each test separately. > From sergei.kovalev at oracle.com Tue Nov 15 11:59:34 2016 From: sergei.kovalev at oracle.com (Sergei Kovalev) Date: Tue, 15 Nov 2016 14:59:34 +0300 Subject: RFR(s): 8169710: com/sun/nio/sctp tests has undeclared dependency on jdk.sctp module In-Reply-To: <990b3eb5-d1d2-1008-6841-49bd6f06b1d5@oracle.com> References: <990b3eb5-d1d2-1008-6841-49bd6f06b1d5@oracle.com> Message-ID: Hi Daniel, Good catch. Thank you. I've wrongly thought that TEST.properties affects only test of single level and not affects sub-folders. I checked your recommendation and found that it also works fine. See web review: http://cr.openjdk.java.net/~skovalev/8169710/webrev.01/ -- With best regards, Sergei 15.11.16 14:26, Daniel Fuchs wrote: > Hi Sergey, > > I'm not an expert in TEST.properties but this looks bizarre > to me. Why not simply put a single TEST.properties file in > com/sun/nio/sctp? > > best regards > > -- daniel > > On 15/11/16 11:21, Sergei Kovalev wrote: >> Hello team, >> >> Please review very small fix for test suite from com/sun/nio/sctp >> folder. >> >> BugID: https://bugs.openjdk.java.net/browse/JDK-8169710 >> Web review: http://cr.openjdk.java.net/~skovalev/8169710/webrev.00/ >> >> Issue: The tests failing in case of usage "--limit modules java.base" >> command line options. >> Solution: add module declaration for tests. Because all tests from the >> appropriate folders are affected it is better to add "TEST.properties" >> file that contains dependency declaration to each folder. In this case >> we have no need to modify each test separately. From daniel.fuchs at oracle.com Tue Nov 15 12:02:26 2016 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Tue, 15 Nov 2016 12:02:26 +0000 Subject: RFR(s): 8169710: com/sun/nio/sctp tests has undeclared dependency on jdk.sctp module In-Reply-To: References: <990b3eb5-d1d2-1008-6841-49bd6f06b1d5@oracle.com> Message-ID: <83472b14-55b9-a113-955a-8a88d8febadc@oracle.com> On 15/11/16 11:59, Sergei Kovalev wrote: > Hi Daniel, > > Good catch. Thank you. I've wrongly thought that TEST.properties affects > only test of single level and not affects sub-folders. I checked your > recommendation and found that it also works fine. > > See web review: http://cr.openjdk.java.net/~skovalev/8169710/webrev.01/ looks good to me. best regards, -- daniel From chris.hegarty at oracle.com Tue Nov 15 12:22:26 2016 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 15 Nov 2016 12:22:26 +0000 Subject: RFR(s): 8169710: com/sun/nio/sctp tests has undeclared dependency on jdk.sctp module In-Reply-To: <83472b14-55b9-a113-955a-8a88d8febadc@oracle.com> References: <990b3eb5-d1d2-1008-6841-49bd6f06b1d5@oracle.com> <83472b14-55b9-a113-955a-8a88d8febadc@oracle.com> Message-ID: <0645cf46-9db2-8453-30bb-752ddac41a03@oracle.com> On 15/11/16 12:02, Daniel Fuchs wrote: > On 15/11/16 11:59, Sergei Kovalev wrote: >> Hi Daniel, >> >> Good catch. Thank you. I've wrongly thought that TEST.properties affects >> only test of single level and not affects sub-folders. I checked your >> recommendation and found that it also works fine. >> >> See web review: http://cr.openjdk.java.net/~skovalev/8169710/webrev.01/ > > looks good to me. +1 -Chris. From huaming.li at oracle.com Wed Nov 16 09:03:19 2016 From: huaming.li at oracle.com (Hamlin Li) Date: Wed, 16 Nov 2016 17:03:19 +0800 Subject: RFR of JDK-8156505: output more information when java/nio/channels/AsynchronousSocketChannel/Basic.java fails Message-ID: <4c5b953f-bfed-70d3-9caf-de9560210a3d@oracle.com> Would you please review below patch? There are 2 kinds of failures reported by JDK-8156505 in test java/nio/channels/AsynchronousSocketChannel/Basic.java, both say that instead of expected exception some unknown exception is caught. need to add some debugging code to get the exact unknown exception. bug: https://bugs.openjdk.java.net/browse/JDK-8169764 webrev: http://cr.openjdk.java.net/~mli/8169764/webrev.00/ Thank you -Hamlin From Alan.Bateman at oracle.com Wed Nov 16 10:04:25 2016 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 16 Nov 2016 10:04:25 +0000 Subject: RFR of JDK-8156505: output more information when java/nio/channels/AsynchronousSocketChannel/Basic.java fails In-Reply-To: <4c5b953f-bfed-70d3-9caf-de9560210a3d@oracle.com> References: <4c5b953f-bfed-70d3-9caf-de9560210a3d@oracle.com> Message-ID: On 16/11/2016 09:03, Hamlin Li wrote: > Would you please review below patch? > > There are 2 kinds of failures reported by JDK-8156505 in test > java/nio/channels/AsynchronousSocketChannel/Basic.java, both say that > instead of expected exception some unknown exception is caught. need > to add some debugging code to get the exact unknown exception. > > bug: https://bugs.openjdk.java.net/browse/JDK-8169764 > webrev: http://cr.openjdk.java.net/~mli/8169764/webrev.00/ Looks okay but it would be good to keep the line length consistent (really long lines are annoying when doing side-by-side diffs). -Alan From yingqi.lu at intel.com Wed Nov 16 22:14:36 2016 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Wed, 16 Nov 2016 22:14:36 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <77D7708E-07FE-4E39-BF7A-30799EB8CEA7@intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51447D9@ORSMSX113.amr.corp.intel.com> <006c75c4-1404-d66b-737d-e07ea2b493fd@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5166194@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA516782E@ORSMSX113.amr.corp.intel.com> <36CE90D5-B86C-4E0A-A2BD-AC812D4262D1@oracle.com> <66b40d54-fcf7-4258-6499-082742727e66@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5169EFD@ORSMSX113.amr.corp.intel.com>, <482dd41a-89c8-fe56-c9ab-e662a2f957e8@oracle.com> <77D7708E-07FE-4E39-BF7A-30799EB8CEA7@intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AA518165F@ORSMSX113.amr.corp.intel.com> Hi All, Our most recent version of our DirectIO patch is available at http://cr.openjdk.java.net/~kkharbas/8164900/webrev.04/ In this version, we have modified the following items: 1. Remove ByteBuffer.allocateDirectAligned() and ByteBuffer.isAligned(). Instead, using existing APIs alignmentOffset() and alignedSlice() 2. Use Block Size as the alignment value instead of OS page size 3. Create a function named getBlockSize from FileStore class 4. Implement DirectIO for Windows 5. JTreg test passed for Windows, Linux and OS X with regarding to the DirectIO changes we made Please let us know your comments and feedback. Thank you very much for your consideration! Thanks, Lucy From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of Lu, Yingqi Sent: Sunday, November 06, 2016 9:37 AM To: Alan Bateman Cc: Kharbas, Kishor ; David Holmes ; core-libs-dev at openjdk.java.net; nio-dev at openjdk.java.net; Thomas St?fe ; Kaczmarek, Eric Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Alan, Thank you very much for your suggestion. FileStore is a better home for the alignment value. We will use It in our next version of the patch and update here for review soon. Thanks, Lucy Sent from my iPhone On Nov 6, 2016, at 9:04 AM, Alan Bateman > wrote: On 06/11/2016 15:51, Lu, Yingqi wrote: Another approach might be use ioctl to get the block size and use that as the alignment value. I think ioctl is available on UNIX based OSes (please chime in if I miss any OS here) and has a similar call for Windows named DeviceIoControl. The request code of ioctl is different from OS to OS, but we could address that in the native code. A possible approach (I think I mentioned this in one of the early mails) is to have FileStore expose the block size that can be use to align and size the buffer. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From huaming.li at oracle.com Thu Nov 17 01:31:08 2016 From: huaming.li at oracle.com (Hamlin Li) Date: Thu, 17 Nov 2016 09:31:08 +0800 Subject: RFR of JDK-8169764: output more information when java/nio/channels/AsynchronousSocketChannel/Basic.java fails In-Reply-To: References: <4c5b953f-bfed-70d3-9caf-de9560210a3d@oracle.com> Message-ID: <37530fff-f857-79d4-1e54-94bbbf448c16@oracle.com> Hi Alan, Thank you for reviewing. Just fixed the long lines and pushed. -Hamlin On 2016/11/16 18:04, Alan Bateman wrote: > > > On 16/11/2016 09:03, Hamlin Li wrote: >> Would you please review below patch? >> >> There are 2 kinds of failures reported by JDK-8156505 in test >> java/nio/channels/AsynchronousSocketChannel/Basic.java, both say that >> instead of expected exception some unknown exception is caught. need >> to add some debugging code to get the exact unknown exception. >> >> bug: https://bugs.openjdk.java.net/browse/JDK-8169764 >> webrev: http://cr.openjdk.java.net/~mli/8169764/webrev.00/ > Looks okay but it would be good to keep the line length consistent > (really long lines are annoying when doing side-by-side diffs). > > -Alan From philippe.marschall at gmail.com Thu Nov 17 14:01:40 2016 From: philippe.marschall at gmail.com (Philippe Marschall) Date: Thu, 17 Nov 2016 15:01:40 +0100 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AA518165F@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51447D9@ORSMSX113.amr.corp.intel.com> <006c75c4-1404-d66b-737d-e07ea2b493fd@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5166194@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA516782E@ORSMSX113.amr.corp.intel.com> <36CE90D5-B86C-4E0A-A2BD-AC812D4262D1@oracle.com> <66b40d54-fcf7-4258-6499-082742727e66@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5169EFD@ORSMSX113.amr.corp.intel.com> <482dd41a-89c8-fe56-c9ab-e662a2f957e8@oracle.com> <77D7708E-07FE-4E39-BF7A-30799EB8CEA7@intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA518165F@ORSMSX113.amr.corp.intel.com> Message-ID: Hi I'm just a lurker so excuse me when overstep some boundaries here. Assuming FileStore#getBlockSize stays: - Could that be changed into a default method? That would be a help for any file system implementer (eg myself) and would also simplify JrtFileStore and ZipFileStore. - Could the Javadoc be updated to state that throwing UnsupportedOperationException is valid? Cheers Philippe On Wed, Nov 16, 2016 at 11:14 PM, Lu, Yingqi wrote: > Hi All, > > > > Our most recent version of our DirectIO patch is available at > http://cr.openjdk.java.net/~kkharbas/8164900/webrev.04/ > > In this version, we have modified the following items: > > 1. Remove ByteBuffer.allocateDirectAligned() and > ByteBuffer.isAligned(). Instead, using existing APIs alignmentOffset() and > alignedSlice() > > 2. Use Block Size as the alignment value instead of OS page size > > 3. Create a function named getBlockSize from FileStore class > > 4. Implement DirectIO for Windows > > 5. JTreg test passed for Windows, Linux and OS X with regarding to the > DirectIO changes we made > > > > Please let us know your comments and feedback. Thank you very much for your > consideration! > > > > Thanks, > > Lucy > > > > From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of Lu, > Yingqi > Sent: Sunday, November 06, 2016 9:37 AM > To: Alan Bateman > Cc: Kharbas, Kishor ; David Holmes > ; core-libs-dev at openjdk.java.net; > nio-dev at openjdk.java.net; Thomas St?fe ; Kaczmarek, > Eric > Subject: Re: Proposal for adding O_DIRECT support into JDK 9 > > > > Hi Alan, > > > > Thank you very much for your suggestion. FileStore is a better home for the > alignment value. We will use It in our next version of the patch and update > here for review soon. > > Thanks, > > Lucy > > Sent from my iPhone > > On Nov 6, 2016, at 9:04 AM, Alan Bateman wrote: > > On 06/11/2016 15:51, Lu, Yingqi wrote: > > Another approach might be use ioctl to get the block size and use that as > the alignment value. I think ioctl is available on UNIX based OSes (please > chime in if I miss any OS here) and has a similar call for Windows named > DeviceIoControl. The request code of ioctl is different from OS to OS, but > we could address that in the native code. > > > > A possible approach (I think I mentioned this in one of the early mails) is > to have FileStore expose the block size that can be use to align and size > the buffer. > > -Alan From yingqi.lu at intel.com Thu Nov 17 17:05:46 2016 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Thu, 17 Nov 2016 17:05:46 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51447D9@ORSMSX113.amr.corp.intel.com> <006c75c4-1404-d66b-737d-e07ea2b493fd@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5166194@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA516782E@ORSMSX113.amr.corp.intel.com> <36CE90D5-B86C-4E0A-A2BD-AC812D4262D1@oracle.com> <66b40d54-fcf7-4258-6499-082742727e66@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5169EFD@ORSMSX113.amr.corp.intel.com> <482dd41a-89c8-fe56-c9ab-e662a2f957e8@oracle.com> <77D7708E-07FE-4E39-BF7A-30799EB8CEA7@intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA518165F@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AA51835F9@ORSMSX113.amr.corp.intel.com> Hi Philippe, Thank you very much for your feedback. We can certainly change FileStore.getBlockSize to default method. We will make this modification into our next version of the patch. All, Besides this change, do you have any other feedback and comments? Thanks, Lucy >-----Original Message----- >From: Philippe Marschall [mailto:philippe.marschall at gmail.com] >Sent: Thursday, November 17, 2016 6:02 AM >To: nio-dev at openjdk.java.net >Cc: Alan Bateman ; Lu, Yingqi >; Kharbas, Kishor ; David >Holmes ; Thomas St?fe >; Kaczmarek, Eric >Subject: Re: Proposal for adding O_DIRECT support into JDK 9 > >Hi > >I'm just a lurker so excuse me when overstep some boundaries here. >Assuming FileStore#getBlockSize stays: >- Could that be changed into a default method? That would be a help for any file >system implementer (eg myself) and would also simplify JrtFileStore and >ZipFileStore. >- Could the Javadoc be updated to state that throwing >UnsupportedOperationException is valid? > >Cheers >Philippe > >On Wed, Nov 16, 2016 at 11:14 PM, Lu, Yingqi wrote: >> Hi All, >> >> >> >> Our most recent version of our DirectIO patch is available at >> http://cr.openjdk.java.net/~kkharbas/8164900/webrev.04/ >> >> In this version, we have modified the following items: >> >> 1. Remove ByteBuffer.allocateDirectAligned() and >> ByteBuffer.isAligned(). Instead, using existing APIs alignmentOffset() >> and >> alignedSlice() >> >> 2. Use Block Size as the alignment value instead of OS page size >> >> 3. Create a function named getBlockSize from FileStore class >> >> 4. Implement DirectIO for Windows >> >> 5. JTreg test passed for Windows, Linux and OS X with regarding to the >> DirectIO changes we made >> >> >> >> Please let us know your comments and feedback. Thank you very much for >> your consideration! >> >> >> >> Thanks, >> >> Lucy >> >> >> >> From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of >> Lu, Yingqi >> Sent: Sunday, November 06, 2016 9:37 AM >> To: Alan Bateman >> Cc: Kharbas, Kishor ; David Holmes >> ; core-libs-dev at openjdk.java.net; >> nio-dev at openjdk.java.net; Thomas St?fe ; >> Kaczmarek, Eric >> Subject: Re: Proposal for adding O_DIRECT support into JDK 9 >> >> >> >> Hi Alan, >> >> >> >> Thank you very much for your suggestion. FileStore is a better home >> for the alignment value. We will use It in our next version of the >> patch and update here for review soon. >> >> Thanks, >> >> Lucy >> >> Sent from my iPhone >> >> On Nov 6, 2016, at 9:04 AM, Alan Bateman >wrote: >> >> On 06/11/2016 15:51, Lu, Yingqi wrote: >> >> Another approach might be use ioctl to get the block size and use that >> as the alignment value. I think ioctl is available on UNIX based OSes >> (please chime in if I miss any OS here) and has a similar call for >> Windows named DeviceIoControl. The request code of ioctl is different >> from OS to OS, but we could address that in the native code. >> >> >> >> A possible approach (I think I mentioned this in one of the early >> mails) is to have FileStore expose the block size that can be use to >> align and size the buffer. >> >> -Alan From yingqi.lu at intel.com Mon Nov 21 19:50:37 2016 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Mon, 21 Nov 2016 19:50:37 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AA51835F9@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51447D9@ORSMSX113.amr.corp.intel.com> <006c75c4-1404-d66b-737d-e07ea2b493fd@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5166194@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA516782E@ORSMSX113.amr.corp.intel.com> <36CE90D5-B86C-4E0A-A2BD-AC812D4262D1@oracle.com> <66b40d54-fcf7-4258-6499-082742727e66@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5169EFD@ORSMSX113.amr.corp.intel.com> <482dd41a-89c8-fe56-c9ab-e662a2f957e8@oracle.com> <77D7708E-07FE-4E39-BF7A-30799EB8CEA7@intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA518165F@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51835F9@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AA51884AF@ORSMSX113.amr.corp.intel.com> Hi All, Just want to check if there is any other feedback on the most recent DirectIO patch which is available at http://cr.openjdk.java.net/~kkharbas/8164900/webrev.04/ Thanks, Lucy >-----Original Message----- >From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of Lu, >Yingqi >Sent: Thursday, November 17, 2016 9:06 AM >To: Philippe Marschall ; nio- >dev at openjdk.java.net >Cc: David Holmes ; Kaczmarek, Eric >; Thomas St?fe ; >Kharbas, Kishor >Subject: RE: Proposal for adding O_DIRECT support into JDK 9 > >Hi Philippe, > >Thank you very much for your feedback. We can certainly change >FileStore.getBlockSize to default method. We will make this modification into our >next version of the patch. > >All, > >Besides this change, do you have any other feedback and comments? > >Thanks, >Lucy > >>-----Original Message----- >>From: Philippe Marschall [mailto:philippe.marschall at gmail.com] >>Sent: Thursday, November 17, 2016 6:02 AM >>To: nio-dev at openjdk.java.net >>Cc: Alan Bateman ; Lu, Yingqi >>; Kharbas, Kishor ; >>David Holmes ; Thomas St?fe >>; Kaczmarek, Eric >>Subject: Re: Proposal for adding O_DIRECT support into JDK 9 >> >>Hi >> >>I'm just a lurker so excuse me when overstep some boundaries here. >>Assuming FileStore#getBlockSize stays: >>- Could that be changed into a default method? That would be a help for >>any file system implementer (eg myself) and would also simplify >>JrtFileStore and ZipFileStore. >>- Could the Javadoc be updated to state that throwing >>UnsupportedOperationException is valid? >> >>Cheers >>Philippe >> >>On Wed, Nov 16, 2016 at 11:14 PM, Lu, Yingqi wrote: >>> Hi All, >>> >>> >>> >>> Our most recent version of our DirectIO patch is available at >>> http://cr.openjdk.java.net/~kkharbas/8164900/webrev.04/ >>> >>> In this version, we have modified the following items: >>> >>> 1. Remove ByteBuffer.allocateDirectAligned() and >>> ByteBuffer.isAligned(). Instead, using existing APIs >>> alignmentOffset() and >>> alignedSlice() >>> >>> 2. Use Block Size as the alignment value instead of OS page size >>> >>> 3. Create a function named getBlockSize from FileStore class >>> >>> 4. Implement DirectIO for Windows >>> >>> 5. JTreg test passed for Windows, Linux and OS X with regarding to the >>> DirectIO changes we made >>> >>> >>> >>> Please let us know your comments and feedback. Thank you very much >>> for your consideration! >>> >>> >>> >>> Thanks, >>> >>> Lucy >>> >>> >>> >>> From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of >>> Lu, Yingqi >>> Sent: Sunday, November 06, 2016 9:37 AM >>> To: Alan Bateman >>> Cc: Kharbas, Kishor ; David Holmes >>> ; core-libs-dev at openjdk.java.net; >>> nio-dev at openjdk.java.net; Thomas St?fe ; >>> Kaczmarek, Eric >>> Subject: Re: Proposal for adding O_DIRECT support into JDK 9 >>> >>> >>> >>> Hi Alan, >>> >>> >>> >>> Thank you very much for your suggestion. FileStore is a better home >>> for the alignment value. We will use It in our next version of the >>> patch and update here for review soon. >>> >>> Thanks, >>> >>> Lucy >>> >>> Sent from my iPhone >>> >>> On Nov 6, 2016, at 9:04 AM, Alan Bateman >>wrote: >>> >>> On 06/11/2016 15:51, Lu, Yingqi wrote: >>> >>> Another approach might be use ioctl to get the block size and use >>> that as the alignment value. I think ioctl is available on UNIX based >>> OSes (please chime in if I miss any OS here) and has a similar call >>> for Windows named DeviceIoControl. The request code of ioctl is >>> different from OS to OS, but we could address that in the native code. >>> >>> >>> >>> A possible approach (I think I mentioned this in one of the early >>> mails) is to have FileStore expose the block size that can be use to >>> align and size the buffer. >>> >>> -Alan From yingqi.lu at intel.com Tue Nov 29 00:17:07 2016 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Tue, 29 Nov 2016 00:17:07 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AA51884AF@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AA510A5DA@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5135D27@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5139A26@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA513CE96@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51447D9@ORSMSX113.amr.corp.intel.com> <006c75c4-1404-d66b-737d-e07ea2b493fd@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5166194@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA516782E@ORSMSX113.amr.corp.intel.com> <36CE90D5-B86C-4E0A-A2BD-AC812D4262D1@oracle.com> <66b40d54-fcf7-4258-6499-082742727e66@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AA5169EFD@ORSMSX113.amr.corp.intel.com> <482dd41a-89c8-fe56-c9ab-e662a2f957e8@oracle.com> <77D7708E-07FE-4E39-BF7A-30799EB8CEA7@intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA518165F@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51835F9@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AA51884AF@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AA51903B7@ORSMSX113.amr.corp.intel.com> Hi All, We are checking if you get any extra feedback on the DirectIO patch listed below. Just a FYI. I will be on vacation from Nov. 30th to Dec. 8th and will be back to office on Dec. 9th. Thanks, Lucy >-----Original Message----- >From: Lu, Yingqi >Sent: Monday, November 21, 2016 11:51 AM >To: Lu, Yingqi ; Philippe Marschall >; nio-dev at openjdk.java.net; Alan Bateman >; Brian Burkhalter >Cc: David Holmes ; Kaczmarek, Eric >; Thomas St?fe ; >Kharbas, Kishor >Subject: RE: Proposal for adding O_DIRECT support into JDK 9 > >Hi All, > >Just want to check if there is any other feedback on the most recent DirectIO >patch which is available at >http://cr.openjdk.java.net/~kkharbas/8164900/webrev.04/ > >Thanks, >Lucy > >>-----Original Message----- >>From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of >>Lu, Yingqi >>Sent: Thursday, November 17, 2016 9:06 AM >>To: Philippe Marschall ; nio- >>dev at openjdk.java.net >>Cc: David Holmes ; Kaczmarek, Eric >>; Thomas St?fe ; >>Kharbas, Kishor >>Subject: RE: Proposal for adding O_DIRECT support into JDK 9 >> >>Hi Philippe, >> >>Thank you very much for your feedback. We can certainly change >>FileStore.getBlockSize to default method. We will make this >>modification into our next version of the patch. >> >>All, >> >>Besides this change, do you have any other feedback and comments? >> >>Thanks, >>Lucy >> >>>-----Original Message----- >>>From: Philippe Marschall [mailto:philippe.marschall at gmail.com] >>>Sent: Thursday, November 17, 2016 6:02 AM >>>To: nio-dev at openjdk.java.net >>>Cc: Alan Bateman ; Lu, Yingqi >>>; Kharbas, Kishor ; >>>David Holmes ; Thomas St?fe >>>; Kaczmarek, Eric >>>Subject: Re: Proposal for adding O_DIRECT support into JDK 9 >>> >>>Hi >>> >>>I'm just a lurker so excuse me when overstep some boundaries here. >>>Assuming FileStore#getBlockSize stays: >>>- Could that be changed into a default method? That would be a help >>>for any file system implementer (eg myself) and would also simplify >>>JrtFileStore and ZipFileStore. >>>- Could the Javadoc be updated to state that throwing >>>UnsupportedOperationException is valid? >>> >>>Cheers >>>Philippe >>> >>>On Wed, Nov 16, 2016 at 11:14 PM, Lu, Yingqi wrote: >>>> Hi All, >>>> >>>> >>>> >>>> Our most recent version of our DirectIO patch is available at >>>> http://cr.openjdk.java.net/~kkharbas/8164900/webrev.04/ >>>> >>>> In this version, we have modified the following items: >>>> >>>> 1. Remove ByteBuffer.allocateDirectAligned() and >>>> ByteBuffer.isAligned(). Instead, using existing APIs >>>> alignmentOffset() and >>>> alignedSlice() >>>> >>>> 2. Use Block Size as the alignment value instead of OS page size >>>> >>>> 3. Create a function named getBlockSize from FileStore class >>>> >>>> 4. Implement DirectIO for Windows >>>> >>>> 5. JTreg test passed for Windows, Linux and OS X with regarding to the >>>> DirectIO changes we made >>>> >>>> >>>> >>>> Please let us know your comments and feedback. Thank you very much >>>> for your consideration! >>>> >>>> >>>> >>>> Thanks, >>>> >>>> Lucy >>>> >>>> >>>> >>>> From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of >>>> Lu, Yingqi >>>> Sent: Sunday, November 06, 2016 9:37 AM >>>> To: Alan Bateman >>>> Cc: Kharbas, Kishor ; David Holmes >>>> ; core-libs-dev at openjdk.java.net; >>>> nio-dev at openjdk.java.net; Thomas St?fe ; >>>> Kaczmarek, Eric >>>> Subject: Re: Proposal for adding O_DIRECT support into JDK 9 >>>> >>>> >>>> >>>> Hi Alan, >>>> >>>> >>>> >>>> Thank you very much for your suggestion. FileStore is a better home >>>> for the alignment value. We will use It in our next version of the >>>> patch and update here for review soon. >>>> >>>> Thanks, >>>> >>>> Lucy >>>> >>>> Sent from my iPhone >>>> >>>> On Nov 6, 2016, at 9:04 AM, Alan Bateman >>>wrote: >>>> >>>> On 06/11/2016 15:51, Lu, Yingqi wrote: >>>> >>>> Another approach might be use ioctl to get the block size and use >>>> that as the alignment value. I think ioctl is available on UNIX >>>> based OSes (please chime in if I miss any OS here) and has a similar >>>> call for Windows named DeviceIoControl. The request code of ioctl is >>>> different from OS to OS, but we could address that in the native code. >>>> >>>> >>>> >>>> A possible approach (I think I mentioned this in one of the early >>>> mails) is to have FileStore expose the block size that can be use to >>>> align and size the buffer. >>>> >>>> -Alan From huaming.li at oracle.com Tue Nov 29 09:23:31 2016 From: huaming.li at oracle.com (Hamlin Li) Date: Tue, 29 Nov 2016 17:23:31 +0800 Subject: RFR of JDK-8049316: TEST_BUG: java/nio/channels/Selector/Wakeup.java fails in same binary run b19 Message-ID: <55b7c230-6f1a-57fd-847b-6ee79728f67c@oracle.com> Would you please review the below patch? bug: https://bugs.openjdk.java.net/browse/JDK-8049316 webrev: http://cr.openjdk.java.net/~mli/8049316/webrev.00/ Root cause: 1. it depends on sleeping time to check failure, which is not reliable in some extreme situation 2. it mix several tests together with a loop in Sleeper Solution: 1. synchronize between threads. 2. isolate tests. Thank you Hamlin From Roger.Riggs at Oracle.com Tue Nov 29 16:03:08 2016 From: Roger.Riggs at Oracle.com (Roger Riggs) Date: Tue, 29 Nov 2016 11:03:08 -0500 Subject: RFR of JDK-8049316: TEST_BUG: java/nio/channels/Selector/Wakeup.java fails in same binary run b19 In-Reply-To: <55b7c230-6f1a-57fd-847b-6ee79728f67c@oracle.com> References: <55b7c230-6f1a-57fd-847b-6ee79728f67c@oracle.com> Message-ID: <0d58f7a4-aec3-5190-2240-ef75d9733944@Oracle.com> Hi Hamlin, Wakeup.java: - Some refactoring may make it easier to understand. Perhaps moving waitSleeper to be a method on Sleeper. The use of cyclicBarrier seems fine but could use a comment somewhere saying what threads/functions are being coordinated; putting them both in Sleeper would make it easier to see. - line 126,127: Add a Sleeper constructor to initialize these as needed. - Some of the test stimulus is buried in the newSleeper methods that was previously in line in main. It seemed clearer what case was being tested in the original. They are buried in static factory 'newSleeper' functions with uninformative names. lines 112, 116, 120. - If I read it right, some cases where events don't happen that used to be reported as exceptions ("Interrupt never delivered") will now be reported as the test timing out. (infinite loop at line 85). Thanks, Roger On 11/29/2016 4:23 AM, Hamlin Li wrote: > Would you please review the below patch? > > bug: https://bugs.openjdk.java.net/browse/JDK-8049316 > webrev: http://cr.openjdk.java.net/~mli/8049316/webrev.00/ > > Root cause: > 1. it depends on sleeping time to check failure, which is not reliable > in some extreme situation > 2. it mix several tests together with a loop in Sleeper > > Solution: > 1. synchronize between threads. > 2. isolate tests. > > Thank you > Hamlin From huaming.li at oracle.com Wed Nov 30 02:09:49 2016 From: huaming.li at oracle.com (Hamlin Li) Date: Wed, 30 Nov 2016 10:09:49 +0800 Subject: RFR of JDK-8049316: TEST_BUG: java/nio/channels/Selector/Wakeup.java fails in same binary run b19 In-Reply-To: <0d58f7a4-aec3-5190-2240-ef75d9733944@Oracle.com> References: <55b7c230-6f1a-57fd-847b-6ee79728f67c@oracle.com> <0d58f7a4-aec3-5190-2240-ef75d9733944@Oracle.com> Message-ID: Hi Roger, Thank you for reviewing, please check comments in line. webrev: http://cr.openjdk.java.net/~mli/8049316/webrev.01/ On 2016/11/30 0:03, Roger Riggs wrote: > Hi Hamlin, > > Wakeup.java: > > - Some refactoring may make it easier to understand. > Perhaps moving waitSleeper to be a method on Sleeper. > The use of cyclicBarrier seems fine but could use a comment > somewhere saying what > threads/functions are being coordinated; putting them both in Sleeper > would make it easier to see. > > - line 126,127: Add a Sleeper constructor to initialize these as needed. > > - Some of the test stimulus is buried in the newSleeper methods that > was previously > in line in main. It seemed clearer what case was being tested in > the original. > They are buried in static factory 'newSleeper' functions with > uninformative names. > lines 112, 116, 120. All above fixed. > > - If I read it right, some cases where events don't happen that used > to be reported as exceptions > ("Interrupt never delivered") will now be reported as the test > timing out. (infinite loop at line 85). No, it will finally checked by a time sleeper.finish(0). At first, I don't want to depends on a specific time, because I don't know what exact timeout we should use, but as you asked, I think it's ok to finish(20_000), it should be long enough. Thank you -Hamlin > > Thanks, Roger > > On 11/29/2016 4:23 AM, Hamlin Li wrote: >> Would you please review the below patch? >> >> bug: https://bugs.openjdk.java.net/browse/JDK-8049316 >> webrev: http://cr.openjdk.java.net/~mli/8049316/webrev.00/ >> >> Root cause: >> 1. it depends on sleeping time to check failure, which is not >> reliable in some extreme situation >> 2. it mix several tests together with a loop in Sleeper >> >> Solution: >> 1. synchronize between threads. >> 2. isolate tests. >> >> Thank you >> Hamlin >