From Alan.Bateman at oracle.com Sun Dec 2 16:11:32 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sun, 2 Dec 2018 16:11:32 +0000 Subject: adding rsockets support into JDK In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> References: <120160AC-6250-4933-AE93-CAF36861A153@oracle.com> <4b5d8793-5a1d-2ca9-f012-6b26812fc8b1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159263B@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159353B@ORSMSX113.amr.corp.intel.com> <046bdfb4-cfdf-451a-74fe-435a698057af@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1593F08@ORSMSX113.amr.corp.intel.com> <2F1BEBF0-C79C-4135-8FC8-15B02D5DB0E9@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15941F6@ORSMSX113.amr.corp.intel.com> <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> Message-ID: <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> Lucy, I checked the CSR and it looks good but in the class description it has the wording that we discussed in mail: "The selector provider does not support datagram channels and pipes. The The openDatagramChannel and openPipe methods throw UOE." I assume you meant to change "UOE" to "UnsupportedOperationException", and I think we can link the openXXX methods to their javadoc. For the implementation then it looks like RmdaPollSelectorProvider extends SelectorProviderImpl without overriding these openXXX methods. This means it will create instances of DatagramChannelImpl and PipeImpl associated with the RDMA provider which isn't right. It might be simpler to just extend SelectorProvider and that will force you to override the abstract methods. While looking into that I see that the Linux version of RdmaPollSelectorProvider is still checking the value of os.name. It doesn't need to do that now and probably left over from an early iteration. -Alan From mikael.vidstedt at oracle.com Mon Dec 3 00:41:43 2018 From: mikael.vidstedt at oracle.com (Mikael Vidstedt) Date: Mon, 3 Dec 2018 10:41:43 +1000 Subject: [8u] RFR 8141491: Unaligned memory access in Bits.c In-Reply-To: <8cf98c3b-20ac-f65e-081e-57fe1be2c27b@oracle.com> References: <136455d0-53d0-25df-6dbe-c3f1cb0f47a9@oracle.com> <900e425b-8e2d-300f-64a1-1ee1b90ba8a0@oracle.com> <8cf98c3b-20ac-f65e-081e-57fe1be2c27b@oracle.com> Message-ID: <8D5BFC84-372C-4FE0-AEF2-484A58D2D062@oracle.com> This seems reasonable to me in order to avoid adding to the API surface. I slightly dislike the duplication of the MAX_OBJECT_SIZE/field_offset_to_byte_offset/assert_field_offset_sane from unsafe.cpp, but it?s small enough that I?m okay with it. All in all, thumbs up from me. Cheers, Mikael > On Nov 30, 2018, at 7:10 PM, Alan Bateman wrote: > > Adding hotspot-runtime-dev as this will require discussion there. > > As context, David Buck is back porting Mikael's changes for JDK-8141491 to 8u. The backport is trying to avoid adding public methods to sun.misc.Unsafe as those methods are not present in sun.misc.Unsafe of newer releases (we don't want to add methods to JDK 8's Unsafe, have people find them, use them, and then show up with pitchforks because they do not exist in newer releases). The approach on the table now is to add the JVM_CopySwapMemory and have it be called by a JNI method. JVM_* functions are JDK internal/undocumented/unsupported but it needs to be discussed and reviewed here. > > -Alan > > > On 30/11/2018 06:58, David Buck wrote: >> Hi Alan and Mikael! >> >> Thank you for the review. I'm sorry it took so long to respond. >> >> Based on advice received from Alan off-line, I have reworked the backport: >> >> http://cr.openjdk.java.net/~dbuck/8141491_jdk8_ver01/ >> >> The native portion of the copySwapMemory implementation has been moved from Unsafe into jvm.cpp (JNI_* and friends). A few minor inline helper functions and a macro have been copied from the Unsafe implementation to facilitate. Normally I consider redundancy a code smell, but any attempt to factor that out of Unsafe felt unnatural and inelegant. The Java portion of copySwapMemory's implementation has been included in java.nio.Bits as that is the only place it is used. Otherwise, there should not be anything too surprising. >> >> Any feedback would be greatly appreciated. >> >> Cheers, >> -Buck > From yingqi.lu at intel.com Mon Dec 3 05:48:54 2018 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Mon, 3 Dec 2018 05:48:54 +0000 Subject: adding rsockets support into JDK In-Reply-To: <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> References: <120160AC-6250-4933-AE93-CAF36861A153@oracle.com> <4b5d8793-5a1d-2ca9-f012-6b26812fc8b1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159263B@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159353B@ORSMSX113.amr.corp.intel.com> <046bdfb4-cfdf-451a-74fe-435a698057af@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1593F08@ORSMSX113.amr.corp.intel.com> <2F1BEBF0-C79C-4135-8FC8-15B02D5DB0E9@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15941F6@ORSMSX113.amr.corp.intel.com> <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> Hi Alan, Please find version 22 of the patch is available at http://cr.openjdk.java.net/~ylu/8195160.22 The updated Javadoc is available at http://cr.openjdk.java.net/~ylu/rsocket_docs/api/jdk.net/jdk/net/package-summary.html Same Javadoc zip file has also been updated in the CSR at https://bugs.openjdk.java.net/browse/JDK-8205186 In this version, I have changed following items: 1. In the class description of RdmaSockets.java, I changed UOE to UnsupportedOperationException, in addition, I added the Javadoc link for openXXX methods 2. Made RdmaPollSelectorProvider directly extends SelectorProvider 3. Merged LinuxRdmaPollSelectorProvider with Linux version of RdmaPollSelectorProvider since they are both only available to Linux Platform 4. Similar to 3, I merged LinuxRdmaSocketDispatcher with RdmaSocketDispatcher 5. Removed os.name check in RdmaSocketImpl.java as well since the code is only available to Linux platform Thanks, Lucy >-----Original Message----- >From: Alan Bateman [mailto:Alan.Bateman at oracle.com] >Sent: Sunday, December 2, 2018 8:12 AM >To: Lu, Yingqi >Cc: Aundhe, Shirish ; Viswanathan, Sandhya >; nio-dev at openjdk.java.net; Kaczmarek, >Eric >Subject: Re: adding rsockets support into JDK > >Lucy, > >I checked the CSR and it looks good but in the class description it has the >wording that we discussed in mail: > >"The selector provider does not support datagram channels and pipes. The >The openDatagramChannel and openPipe methods throw UOE." > >I assume you meant to change "UOE" to "UnsupportedOperationException", >and I think we can link the openXXX methods to their javadoc. > >For the implementation then it looks like RmdaPollSelectorProvider extends >SelectorProviderImpl without overriding these openXXX methods. >This means it will create instances of DatagramChannelImpl and PipeImpl >associated with the RDMA provider which isn't right. It might be simpler to just >extend SelectorProvider and that will force you to override the abstract >methods. > >While looking into that I see that the Linux version of >RdmaPollSelectorProvider is still checking the value of os.name. It doesn't >need to do that now and probably left over from an early iteration. > >-Alan From david.holmes at oracle.com Mon Dec 3 06:09:57 2018 From: david.holmes at oracle.com (David Holmes) Date: Mon, 3 Dec 2018 16:09:57 +1000 Subject: [8u] RFR 8141491: Unaligned memory access in Bits.c In-Reply-To: <8D5BFC84-372C-4FE0-AEF2-484A58D2D062@oracle.com> References: <136455d0-53d0-25df-6dbe-c3f1cb0f47a9@oracle.com> <900e425b-8e2d-300f-64a1-1ee1b90ba8a0@oracle.com> <8cf98c3b-20ac-f65e-081e-57fe1be2c27b@oracle.com> <8D5BFC84-372C-4FE0-AEF2-484A58D2D062@oracle.com> Message-ID: <32ac58fc-ce67-0f8b-f1f8-6dc97e71de7f@oracle.com> I also see no concerns with making this a JVM_* API rather than an Unsafe one. I can't comment on the actual bit copying bits. Cheers, David On 3/12/2018 10:41 am, Mikael Vidstedt wrote: > > This seems reasonable to me in order to avoid adding to the API surface. > > I slightly dislike the duplication of the MAX_OBJECT_SIZE/field_offset_to_byte_offset/assert_field_offset_sane from unsafe.cpp, but it?s small enough that I?m okay with it. > > All in all, thumbs up from me. > > Cheers, > Mikael > >> On Nov 30, 2018, at 7:10 PM, Alan Bateman wrote: >> >> Adding hotspot-runtime-dev as this will require discussion there. >> >> As context, David Buck is back porting Mikael's changes for JDK-8141491 to 8u. The backport is trying to avoid adding public methods to sun.misc.Unsafe as those methods are not present in sun.misc.Unsafe of newer releases (we don't want to add methods to JDK 8's Unsafe, have people find them, use them, and then show up with pitchforks because they do not exist in newer releases). The approach on the table now is to add the JVM_CopySwapMemory and have it be called by a JNI method. JVM_* functions are JDK internal/undocumented/unsupported but it needs to be discussed and reviewed here. >> >> -Alan >> >> >> On 30/11/2018 06:58, David Buck wrote: >>> Hi Alan and Mikael! >>> >>> Thank you for the review. I'm sorry it took so long to respond. >>> >>> Based on advice received from Alan off-line, I have reworked the backport: >>> >>> http://cr.openjdk.java.net/~dbuck/8141491_jdk8_ver01/ >>> >>> The native portion of the copySwapMemory implementation has been moved from Unsafe into jvm.cpp (JNI_* and friends). A few minor inline helper functions and a macro have been copied from the Unsafe implementation to facilitate. Normally I consider redundancy a code smell, but any attempt to factor that out of Unsafe felt unnatural and inelegant. The Java portion of copySwapMemory's implementation has been included in java.nio.Bits as that is the only place it is used. Otherwise, there should not be anything too surprising. >>> >>> Any feedback would be greatly appreciated. >>> >>> Cheers, >>> -Buck >> > From deepak.kejriwal at oracle.com Mon Dec 3 14:25:35 2018 From: deepak.kejriwal at oracle.com (Deepak Kejriwal) Date: Mon, 3 Dec 2018 06:25:35 -0800 (PST) Subject: RFR: JDK-8197398 (zipfs) OutOfMemoryError when talking contents of empty JAR file In-Reply-To: References: <84f22c6c-66dd-4c1e-a9a2-881f631d9eb1@default> <1b64c50d-6a1a-cfef-e225-43502078d7af@oracle.com> Message-ID: <79c98e06-8fbb-486f-8a85-c775dc341cf5@default> Hi Roger, ? Thanks for review. I have modified the code to avoid creation of empty array. ? Please find the below the updated webrev for fix. http://cr.openjdk.java.net/~rpatil/8199776/webrev.03/ ? Regards, Deepak ? From: Roger Riggs Sent: Thursday, November 29, 2018 10:16 PM To: Deepak Kejriwal ; Lance Andersen ; nio-dev at openjdk.java.net Cc: core-libs-dev Subject: Re: RFR: JDK-8197398 (zipfs) OutOfMemoryError when talking contents of empty JAR file ? Hi Deepak, A small change could avoid creating the array if it would be empty. Move the array creation inside the if and compare (startIndex < endIndex). Thanks, Roger On 11/29/2018 05:10 AM, Deepak Kejriwal wrote: Hi Roger / Lance, ? Thanks for reviewing the fix. I have modified the fix as the review comments given. ? Please find the below the updated webrev for fix. HYPERLINK "http://cr.openjdk.java.net/%7Erpatil/8199776/webrev.02/"http://cr.openjdk.java.net/~rpatil/8199776/webrev.02/ ? Regards, Deepak ? ? From: Roger Riggs Sent: Wednesday, November 28, 2018 3:15 AM To: Deepak Kejriwal HYPERLINK "mailto:deepak.kejriwal at oracle.com"; HYPERLINK "mailto:nio-dev at openjdk.java.net"nio-dev at openjdk.java.net Cc: core-libs-dev HYPERLINK "mailto:core-libs-dev at openjdk.java.net" Subject: Re: RFR: JDK-8197398 (zipfs) OutOfMemoryError when talking contents of empty JAR file ? Hi, In ZipFileSystem.java, at 1071. ?- The code will be more readable if you use character literal '/' instead of 47. ?- Save making an extra copy of the name array by testing for '/' directly in the cen array ?? and create either the full name or the name without the '/'. ?- 1073: The coding style should put the '{' on the same line as the 'if'. ?? Use the file's coding style. Thanks, Roger On 11/27/2018 04:51 AM, Deepak Kejriwal wrote: Hi nio-dev team, ? ? Gentle reminder for review of below fix. ? ? Regards, ? Deepak ? ? From: Deepak Kejriwal HYPERLINK "mailto:deepak.kejriwal at oracle.com" Sent: Thursday, November 8, 2018 12:17 AM To: 'HYPERLINK "mailto:nio-dev at openjdk.java.net"nio-dev at openjdk.java.net' HYPERLINK "mailto:nio-dev at openjdk.java.net" Subject: RFR: JDK-8197398 (zipfs) OutOfMemoryError when talking contents of empty JAR file ? ? Hi all, ? ? Please help review the changes for JDK-8199776 to JDK8u:- ? ? Master Bug: https://bugs.openjdk.java.net/browse/JDK-8199776 ? Webrev: HYPERLINK "http://cr.openjdk.java.net/%7Erpatil/8199776/webrev.00/"http://cr.openjdk.java.net/~rpatil/8199776/webrev.00/ ? ? ? Background:- ? The Files.walkFileTree walk indefinitely while processing JAR file with "/" as a directory inside. ? ? Proposed fix: ? ? Files.walkFileTree api for jar internally uses ZipfileSystem. While iterating over the zip entries we have added a check to see if entry name is equal to "/" (absolute path). If the entry name is equal to "/" we don't add it to "inodes" map so that we don't end up with infinite loop. ? ? We have similar bug HYPERLINK HYPERLINK "https://bugs.openjdk.java.net/browse/JDK-8197398""https://bugs.openjdk.java.net/browse/JDK-8197398"JDK-8197398 fixed in jdk 12. Below is the difference between fix for jdk 12 and jdk 8:- ? The fix for above issue for jdk 12 handle a case where we can update an existing entry in jar with "/" (absolute path).? As in case of jdk 8 we cannot update the existing zipentry as doing so we get "FileAlreadyExistsException". Therefore, fix for jdk 8u does not consider this case. ? ? Regards, ? Deepak ? ? ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From Roger.Riggs at oracle.com Mon Dec 3 18:55:26 2018 From: Roger.Riggs at oracle.com (Roger Riggs) Date: Mon, 3 Dec 2018 13:55:26 -0500 Subject: RFR: JDK-8197398 (zipfs) OutOfMemoryError when talking contents of empty JAR file In-Reply-To: <79c98e06-8fbb-486f-8a85-c775dc341cf5@default> References: <84f22c6c-66dd-4c1e-a9a2-881f631d9eb1@default> <1b64c50d-6a1a-cfef-e225-43502078d7af@oracle.com> <79c98e06-8fbb-486f-8a85-c775dc341cf5@default> Message-ID: Hi Deepak, Looks fine.? :) Thanks, Roger On 12/03/2018 09:25 AM, Deepak Kejriwal wrote: > > Hi Roger, > > Thanks for review. I have modified the code to avoid creation of empty > array. > > Please find the below the updated webrev for fix. > > http://cr.openjdk.java.net/~rpatil/8199776/webrev.03/ > > > Regards, > > Deepak > > *From:*Roger Riggs > *Sent:* Thursday, November 29, 2018 10:16 PM > *To:* Deepak Kejriwal ; Lance Andersen > ; nio-dev at openjdk.java.net > *Cc:* core-libs-dev > *Subject:* Re: RFR: JDK-8197398 (zipfs) OutOfMemoryError when talking > contents of empty JAR file > > Hi Deepak, > > A small change could avoid creating the array if it would be empty. > Move the array creation inside the if and compare (startIndex < endIndex). > > Thanks, Roger > > > > On 11/29/2018 05:10 AM, Deepak Kejriwal wrote: > > Hi Roger / Lance, > > Thanks for reviewing the fix. I have modified the fix as the > review comments given. > > Please find the below the updated webrev for fix. > > http://cr.openjdk.java.net/~rpatil/8199776/webrev.02/ > > > Regards, > > Deepak > > *From:*Roger Riggs > *Sent:* Wednesday, November 28, 2018 3:15 AM > *To:* Deepak Kejriwal > ; nio-dev at openjdk.java.net > > *Cc:* core-libs-dev > > *Subject:* Re: RFR: JDK-8197398 (zipfs) OutOfMemoryError when > talking contents of empty JAR file > > Hi, > > In ZipFileSystem.java, at 1071. > > ?- The code will be more readable if you use character literal '/' > instead of 47. > > ?- Save making an extra copy of the name array by testing for '/' > directly in the cen array > ?? and create either the full name or the name without the '/'. > > ?- 1073: The coding style should put the '{' on the same line as > the 'if'. > ?? Use the file's coding style. > > Thanks, Roger > > On 11/27/2018 04:51 AM, Deepak Kejriwal wrote: > > Hi nio-dev team, > > > > > > > > Gentle reminder for review of below fix. > > > > > > > > Regards, > > > > Deepak > > > > > > > > From: Deepak Kejriwal > > Sent: Thursday, November 8, 2018 12:17 AM > > To: 'nio-dev at openjdk.java.net ' > > Subject: RFR: JDK-8197398 (zipfs) OutOfMemoryError when talking contents of empty JAR file > > > > > > > > Hi all, > > > > > > > > Please help review the changes for JDK-8199776 to JDK8u:- > > > > > > > > Master Bug:https://bugs.openjdk.java.net/browse/JDK-8199776 > > > > Webrev:http://cr.openjdk.java.net/~rpatil/8199776/webrev.00/ > > > > > > > > > > > > > Background:- > > > > The Files.walkFileTree walk indefinitely while processing JAR file with "/" as a directory inside. > > > > > > > > Proposed fix: > > > > > > > > Files.walkFileTree api for jar internally uses ZipfileSystem. While iterating over the zip entries we have added a check to see if entry name is equal to "/" (absolute path). If the entry name is equal to "/" we don't add it to "inodes" map so that we don't end up with infinite loop. > > > > > > > > We have similar bug HYPERLINK"https://bugs.openjdk.java.net/browse/JDK-8197398" > JDK-8197398 fixed in jdk 12. Below is the difference between fix for jdk 12 and jdk 8:- > > > > The fix for above issue for jdk 12 handle a case where we can update an existing entry in jar with "/" (absolute path).? As in case of jdk 8 we cannot update the existing zipentry as doing so we get "FileAlreadyExistsException". Therefore, fix for jdk 8u does not consider this case. > > > > > > > > Regards, > > > > Deepak > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lance.andersen at oracle.com Mon Dec 3 19:58:59 2018 From: lance.andersen at oracle.com (Lance Andersen) Date: Mon, 3 Dec 2018 14:58:59 -0500 Subject: RFR: JDK-8197398 (zipfs) OutOfMemoryError when talking contents of empty JAR file In-Reply-To: <79c98e06-8fbb-486f-8a85-c775dc341cf5@default> References: <84f22c6c-66dd-4c1e-a9a2-881f631d9eb1@default> <1b64c50d-6a1a-cfef-e225-43502078d7af@oracle.com> <79c98e06-8fbb-486f-8a85-c775dc341cf5@default> Message-ID: Hi Deepak Looks OK. Thank you for your efforts > On Dec 3, 2018, at 9:25 AM, Deepak Kejriwal wrote: > > Hi Roger, > > Thanks for review. I have modified the code to avoid creation of empty array. > > Please find the below the updated webrev for fix. > http://cr.openjdk.java.net/~rpatil/8199776/webrev.03/ > > Regards, > Deepak > > From: Roger Riggs > Sent: Thursday, November 29, 2018 10:16 PM > To: Deepak Kejriwal ; Lance Andersen ; nio-dev at openjdk.java.net > Cc: core-libs-dev > Subject: Re: RFR: JDK-8197398 (zipfs) OutOfMemoryError when talking contents of empty JAR file > > Hi Deepak, > > A small change could avoid creating the array if it would be empty. > Move the array creation inside the if and compare (startIndex < endIndex). > > Thanks, Roger > > > > > On 11/29/2018 05:10 AM, Deepak Kejriwal wrote: > Hi Roger / Lance, > > Thanks for reviewing the fix. I have modified the fix as the review comments given. > > Please find the below the updated webrev for fix. > http://cr.openjdk.java.net/~rpatil/8199776/webrev.02/ > > Regards, > Deepak > > > From: Roger Riggs > Sent: Wednesday, November 28, 2018 3:15 AM > To: Deepak Kejriwal ; nio-dev at openjdk.java.net > Cc: core-libs-dev > Subject: Re: RFR: JDK-8197398 (zipfs) OutOfMemoryError when talking contents of empty JAR file > > Hi, > > In ZipFileSystem.java, at 1071. > > - The code will be more readable if you use character literal '/' instead of 47. > > - Save making an extra copy of the name array by testing for '/' directly in the cen array > and create either the full name or the name without the '/'. > > - 1073: The coding style should put the '{' on the same line as the 'if'. > Use the file's coding style. > > Thanks, Roger > > On 11/27/2018 04:51 AM, Deepak Kejriwal wrote: > Hi nio-dev team, > > > > Gentle reminder for review of below fix. > > > > Regards, > > Deepak > > > > From: Deepak Kejriwal > Sent: Thursday, November 8, 2018 12:17 AM > To: 'nio-dev at openjdk.java.net ' > Subject: RFR: JDK-8197398 (zipfs) OutOfMemoryError when talking contents of empty JAR file > > > > Hi all, > > > > Please help review the changes for JDK-8199776 to JDK8u:- > > > > Master Bug: https://bugs.openjdk.java.net/browse/JDK-8199776 > > Webrev: http://cr.openjdk.java.net/~rpatil/8199776/webrev.00/ > > > > > > Background:- > > The Files.walkFileTree walk indefinitely while processing JAR file with "/" as a directory inside. > > > > Proposed fix: > > > > Files.walkFileTree api for jar internally uses ZipfileSystem. While iterating over the zip entries we have added a check to see if entry name is equal to "/" (absolute path). If the entry name is equal to "/" we don't add it to "inodes" map so that we don't end up with infinite loop. > > > > We have similar bug HYPERLINK "https://bugs.openjdk.java.net/browse/JDK-8197398" JDK-8197398 fixed in jdk 12. Below is the difference between fix for jdk 12 and jdk 8:- > > The fix for above issue for jdk 12 handle a case where we can update an existing entry in jar with "/" (absolute path). As in case of jdk 8 we cannot update the existing zipentry as doing so we get "FileAlreadyExistsException". Therefore, fix for jdk 8u does not consider this case. > > > > Regards, > > Deepak > > > > Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: oracle_sig_logo.gif Type: image/gif Size: 658 bytes Desc: not available URL: From yingqi.lu at intel.com Tue Dec 4 01:25:58 2018 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Tue, 4 Dec 2018 01:25:58 +0000 Subject: adding rsockets support into JDK In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> References: <120160AC-6250-4933-AE93-CAF36861A153@oracle.com> <4b5d8793-5a1d-2ca9-f012-6b26812fc8b1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159263B@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159353B@ORSMSX113.amr.corp.intel.com> <046bdfb4-cfdf-451a-74fe-435a698057af@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1593F08@ORSMSX113.amr.corp.intel.com> <2F1BEBF0-C79C-4135-8FC8-15B02D5DB0E9@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15941F6@ORSMSX113.amr.corp.intel.com> <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> Hi All, Here is the link to the version 23 of the patch: http://cr.openjdk.java.net/~ylu/8195160.23/ In this version, I did following items: 1. Further cleaned up the patch. I merged LinuxRdmaSocketImpl into RdmaSocketImpl and modified the native code accordingly 2. ioctl with FIONREAD is not implemented with rsocket, I changed RdmaSocketInputStream.available() returns UOE 3. dup2 system call is also not available with rsocket. I checked with Linux rdma kernel developer regarding to what happens when there is data remaining on the rsocket during close. The response I got is rsocket would send out the remaining data before being closed. Based on this information, I simply removed useDeferredClose flag for close call in RdmaSocketImpl. Please review and let me know your feedback and comments. Thanks, Lucy >-----Original Message----- >From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of Lu, >Yingqi >Sent: Sunday, December 2, 2018 9:49 PM >To: Alan Bateman >Cc: Viswanathan, Sandhya ; nio- >dev at openjdk.java.net; Aundhe, Shirish ; >Kaczmarek, Eric >Subject: RE: adding rsockets support into JDK > >Hi Alan, > >Please find version 22 of the patch is available at >http://cr.openjdk.java.net/~ylu/8195160.22 > >The updated Javadoc is available at >http://cr.openjdk.java.net/~ylu/rsocket_docs/api/jdk.net/jdk/net/package- >summary.html > >Same Javadoc zip file has also been updated in the CSR at >https://bugs.openjdk.java.net/browse/JDK-8205186 > >In this version, I have changed following items: > >1. In the class description of RdmaSockets.java, I changed UOE to >UnsupportedOperationException, in addition, I added the Javadoc link for >openXXX methods > >2. Made RdmaPollSelectorProvider directly extends SelectorProvider > >3. Merged LinuxRdmaPollSelectorProvider with Linux version of >RdmaPollSelectorProvider since they are both only available to Linux Platform > >4. Similar to 3, I merged LinuxRdmaSocketDispatcher with >RdmaSocketDispatcher > >5. Removed os.name check in RdmaSocketImpl.java as well since the code is >only available to Linux platform > >Thanks, >Lucy > >>-----Original Message----- >>From: Alan Bateman [mailto:Alan.Bateman at oracle.com] >>Sent: Sunday, December 2, 2018 8:12 AM >>To: Lu, Yingqi >>Cc: Aundhe, Shirish ; Viswanathan, Sandhya >>; nio-dev at openjdk.java.net; Kaczmarek, >>Eric >>Subject: Re: adding rsockets support into JDK >> >>Lucy, >> >>I checked the CSR and it looks good but in the class description it has >>the wording that we discussed in mail: >> >>"The selector provider does not support datagram channels and pipes. >>The The openDatagramChannel and openPipe methods throw UOE." >> >>I assume you meant to change "UOE" to "UnsupportedOperationException", >>and I think we can link the openXXX methods to their javadoc. >> >>For the implementation then it looks like RmdaPollSelectorProvider >>extends SelectorProviderImpl without overriding these openXXX methods. >>This means it will create instances of DatagramChannelImpl and PipeImpl >>associated with the RDMA provider which isn't right. It might be >>simpler to just extend SelectorProvider and that will force you to >>override the abstract methods. >> >>While looking into that I see that the Linux version of >>RdmaPollSelectorProvider is still checking the value of os.name. It >>doesn't need to do that now and probably left over from an early iteration. >> >>-Alan From brian.burkhalter at oracle.com Tue Dec 4 01:34:36 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 3 Dec 2018 17:34:36 -0800 Subject: adding rsockets support into JDK In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> References: <120160AC-6250-4933-AE93-CAF36861A153@oracle.com> <4b5d8793-5a1d-2ca9-f012-6b26812fc8b1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159263B@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159353B@ORSMSX113.amr.corp.intel.com> <046bdfb4-cfdf-451a-74fe-435a698057af@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1593F08@ORSMSX113.amr.corp.intel.com> <2F1BEBF0-C79C-4135-8FC8-15B02D5DB0E9@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15941F6@ORSMSX113.amr.corp.intel.com> <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> Message-ID: <23AA9886-A09F-47A8-8F44-2A024B483B25@oracle.com> Hi Lucy, I was just about to post something about version 22 when you posted version 23! I have some javadoc suggestions which are given in the form of a patch versus version 22: http://cr.openjdk.java.net/~bpb/8195160/webrev-22-delta/ These are my changes only so you might want to wait until someone else reviews them before changing your version. I also ran version 22 through the usual testing and have not seen any problems yet (Windows is still in progress). I?ll submit a version 23 test to run during the evening. Thanks, Brian > On Dec 2, 2018, at 9:48 PM, Lu, Yingqi wrote: > > Please find version 22 of the patch is available at http://cr.openjdk.java.net/~ylu/8195160.22 > > The updated Javadoc is available at http://cr.openjdk.java.net/~ylu/rsocket_docs/api/jdk.net/jdk/net/package-summary.html > > Same Javadoc zip file has also been updated in the CSR at https://bugs.openjdk.java.net/browse/JDK-8205186 -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Dec 4 01:49:34 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 3 Dec 2018 17:49:34 -0800 Subject: adding rsockets support into JDK In-Reply-To: <23AA9886-A09F-47A8-8F44-2A024B483B25@oracle.com> References: <120160AC-6250-4933-AE93-CAF36861A153@oracle.com> <4b5d8793-5a1d-2ca9-f012-6b26812fc8b1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159263B@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159353B@ORSMSX113.amr.corp.intel.com> <046bdfb4-cfdf-451a-74fe-435a698057af@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1593F08@ORSMSX113.amr.corp.intel.com> <2F1BEBF0-C79C-4135-8FC8-15B02D5DB0E9@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15941F6@ORSMSX113.amr.corp.intel.com> <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> <23AA9886-A09F-47A8-8F44-2A024B483B25@oracle.com> Message-ID: <20444046-11A8-4B61-9B4B-FE608AE7D393@oracle.com> I think it would be clearer if I provided some annotation for this by new version line number. 61: ?The? -> ?Its? 78: ?A RDMA? -> ?An RDMA? and ?@code? -> ?@link? 82: Add ?@code? for connect 92 and elsewhere: ?null? -> ?{@code null}? 95-98 and elsewhere: Modified verbiage 109: ?A RDMA? -> ?An RDMA? and ?@code? -> ?@link? 113: ?bind? -> ?{@code bind}? 138 & 143: ?a RDMA? -> ?an RDMA? 174: ?a RDMA? -> ?an RDMA? 178: deleted extra space before comma 179: ?bind? -> ?{@code bind}? Thanks, Brian > On Dec 3, 2018, at 5:34 PM, Brian Burkhalter wrote: > > I have some javadoc suggestions which are given in the form of a patch versus version 22: > > http://cr.openjdk.java.net/~bpb/8195160/webrev-22-delta/ > > These are my changes only so you might want to wait until someone else reviews them before changing your version. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Dec 4 02:02:11 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 3 Dec 2018 18:02:11 -0800 Subject: adding rsockets support into JDK In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> References: <120160AC-6250-4933-AE93-CAF36861A153@oracle.com> <4b5d8793-5a1d-2ca9-f012-6b26812fc8b1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159263B@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159353B@ORSMSX113.amr.corp.intel.com> <046bdfb4-cfdf-451a-74fe-435a698057af@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1593F08@ORSMSX113.amr.corp.intel.com> <2F1BEBF0-C79C-4135-8FC8-15B02D5DB0E9@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15941F6@ORSMSX113.amr.corp.intel.com> <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> Message-ID: <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> Hi Lucy, I am seeing a build failure on Linux with this version: workspace/build/linux-x64/support/native/jdk.net/libextnet/RdmaSocketInputStream.o: In function `Java_jdk_internal_net_rdma_RdmaSocketInputStream_init': workspace/open/src/jdk.net/linux/native/libextnet/RdmaSocketInputStream.c:40: undefined reference to `NET_GetFileDescriptorID' workspace/build/linux-x64/support/native/jdk.net/libextnet/RdmaSocketOutputStream.o: In function `Java_jdk_internal_net_rdma_RdmaSocketOutputStream_init': workspace/open/src/jdk.net/linux/native/libextnet/RdmaSocketOutputStream.c:42: undefined reference to `NET_GetFileDescriptorID' collect2: error: ld returned 1 exit status Lib-jdk.net.gmk:32: recipe for target 'workspace/build/linux-x64/support/modules_libs/jdk.net/libextnet.so' failed make[3]: *** [workspace/build/linux-x64/support/modules_libs/jdk.net/libextnet.so] Error 1 make/Main.gmk:215: recipe for target 'jdk.net-libs' failed Thanks, Brian > On Dec 3, 2018, at 5:25 PM, Lu, Yingqi wrote: > > Here is the link to the version 23 of the patch: http://cr.openjdk.java.net/~ylu/8195160.23/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Tue Dec 4 03:05:54 2018 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Tue, 4 Dec 2018 03:05:54 +0000 Subject: adding rsockets support into JDK In-Reply-To: <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> References: <120160AC-6250-4933-AE93-CAF36861A153@oracle.com> <4b5d8793-5a1d-2ca9-f012-6b26812fc8b1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159263B@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159353B@ORSMSX113.amr.corp.intel.com> <046bdfb4-cfdf-451a-74fe-435a698057af@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1593F08@ORSMSX113.amr.corp.intel.com> <2F1BEBF0-C79C-4135-8FC8-15B02D5DB0E9@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15941F6@ORSMSX113.amr.corp.intel.com> <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> Hi Brian, Thank you for the quick check. I forgot to add RdmaSocketImpl.c into the original patch file. The patch is updated at the same link: http://cr.openjdk.java.net/~ylu/8195160.23/. Please let me know if you see any issues with the updated version. I also took a look at the wording changes you made at http://cr.openjdk.java.net/~bpb/8195160/webrev-22-delta/. They look pretty good to me:). Unless others have more feedback on the wording, I will incorporate exactly the changes into version 24. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Monday, December 3, 2018 6:02 PM To: Lu, Yingqi Cc: Alan Bateman ; Viswanathan, Sandhya ; nio-dev at openjdk.java.net; Aundhe, Shirish ; Kaczmarek, Eric Subject: Re: adding rsockets support into JDK Hi Lucy, I am seeing a build failure on Linux with this version: workspace/build/linux-x64/support/native/jdk.net/libextnet/RdmaSocketInputStream.o: In function `Java_jdk_internal_net_rdma_RdmaSocketInputStream_init': workspace/open/src/jdk.net/linux/native/libextnet/RdmaSocketInputStream.c:40: undefined reference to `NET_GetFileDescriptorID' workspace/build/linux-x64/support/native/jdk.net/libextnet/RdmaSocketOutputStream.o: In function `Java_jdk_internal_net_rdma_RdmaSocketOutputStream_init': workspace/open/src/jdk.net/linux/native/libextnet/RdmaSocketOutputStream.c:42: undefined reference to `NET_GetFileDescriptorID' collect2: error: ld returned 1 exit status Lib-jdk.net.gmk:32: recipe for target 'workspace/build/linux-x64/support/modules_libs/jdk.net/libextnet.so' failed make[3]: *** [workspace/build/linux-x64/support/modules_libs/jdk.net/libextnet.so] Error 1 make/Main.gmk:215: recipe for target 'jdk.net-libs' failed Thanks, Brian On Dec 3, 2018, at 5:25 PM, Lu, Yingqi > wrote: Here is the link to the version 23 of the patch: http://cr.openjdk.java.net/~ylu/8195160.23/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Dec 4 18:05:16 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 4 Dec 2018 10:05:16 -0800 Subject: adding rsockets support into JDK In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> References: <120160AC-6250-4933-AE93-CAF36861A153@oracle.com> <4b5d8793-5a1d-2ca9-f012-6b26812fc8b1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159263B@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159353B@ORSMSX113.amr.corp.intel.com> <046bdfb4-cfdf-451a-74fe-435a698057af@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1593F08@ORSMSX113.amr.corp.intel.com> <2F1BEBF0-C79C-4135-8FC8-15B02D5DB0E9@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15941F6@ORSMSX113.amr.corp.intel.com> <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> Message-ID: <9AC5EB38-941D-460F-8C95-FDF8D4F93A54@oracle.com> Hi Lucy, > On Dec 3, 2018, at 7:05 PM, Lu, Yingqi wrote: > > Thank you for the quick check. I forgot to add RdmaSocketImpl.c into the original patch file. > > The patch is updated at the same link:http://cr.openjdk.java.net/~ylu/8195160.23/ . Please let me know if you see any issues with the updated version. Build and test succeeded with this version. > I also took a look at the wording changes you made athttp://cr.openjdk.java.net/~bpb/8195160/webrev-22-delta/ . They look pretty good to meJ. Unless others have more feedback on the wording, I will incorporate exactly the changes into version 24. Sounds good. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.hegarty at oracle.com Tue Dec 4 20:22:47 2018 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 4 Dec 2018 20:22:47 +0000 Subject: adding rsockets support into JDK In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> References: <120160AC-6250-4933-AE93-CAF36861A153@oracle.com> <4b5d8793-5a1d-2ca9-f012-6b26812fc8b1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159263B@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159353B@ORSMSX113.amr.corp.intel.com> <046bdfb4-cfdf-451a-74fe-435a698057af@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1593F08@ORSMSX113.amr.corp.intel.com> <2F1BEBF0-C79C-4135-8FC8-15B02D5DB0E9@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15941F6@ORSMSX113.amr.corp.intel.com> <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> Message-ID: <8C5ED530-8C5B-4A99-8471-B43B82B8E5E3@oracle.com> Lucy, > On 4 Dec 2018, at 03:05, Lu, Yingqi wrote: > > Hi Brian, > > Thank you for the quick check. I forgot to add RdmaSocketImpl.c into the original patch file. > > The patch is updated at the same link: http://cr.openjdk.java.net/~ylu/8195160.23/ . Please let me know if you see any issues with the updated version. Finally, I have an environment where I can build and run the tests for this code. I need to circle back and document this, but for now I can successfully create an rsocket and run the java tests. I am continuing to review the code, but here are a few comments so far: 1) BasicSocketChannelTest.java There is a race in the test code. Sometimes the test fails with an exception when it tries to shutdown the input/output of the client socket/channel. The reason is that the main thread may have already closed the socket/channel. While fixing the race I took the liberty of refactoring the test a little. It passes successfully, reliably, now in my environment. https://cr.openjdk.java.net/~chegar/rsocket/webrev.23.1/ Note: I've left a TODO comment in the test. Read to EOF is not working as I would expect ( the original version of the test did not exercise this). I'll revisit this later. 2) Test consistency of API exception, across all platforms. I added a test to ensure that NPE and other API exceptions are being thrown as expected. This test can be run on all platforms, so it is not Linux specific. I used testng, very nice for test kinda tests. https://cr.openjdk.java.net/~chegar/rsocket/webrev.23.2/ 3) Handling of null binding, automatically assigned address. Handling of null binding is a bit tricky now that we have protocol family support. The correct InetXAddress "any local address" needs to be created. I added a test and a little refactoring. If you run the test without any source changes you will be able to see the issues. https://cr.openjdk.java.net/~chegar/rsocket/webrev.23.3/ Note: There is still an open issue, ToDo, with java.net.Socket. You can see this by a small comment in the test. I'll revisit this later. -Chris. P.S. The above webrevs were created from patches in my patch queue, and should apply cleanly to version 23 if applied in order. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Tue Dec 4 20:25:16 2018 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Tue, 4 Dec 2018 20:25:16 +0000 Subject: adding rsockets support into JDK In-Reply-To: <9AC5EB38-941D-460F-8C95-FDF8D4F93A54@oracle.com> References: <120160AC-6250-4933-AE93-CAF36861A153@oracle.com> <4b5d8793-5a1d-2ca9-f012-6b26812fc8b1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159263B@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159353B@ORSMSX113.amr.corp.intel.com> <046bdfb4-cfdf-451a-74fe-435a698057af@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1593F08@ORSMSX113.amr.corp.intel.com> <2F1BEBF0-C79C-4135-8FC8-15B02D5DB0E9@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15941F6@ORSMSX113.amr.corp.intel.com> <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> <9AC5EB38-941D-460F-8C95-FDF8D4F93A54@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD159A375@ORSMSX113.amr.corp.intel.com> Hi All, I attached a presentation "Programing Modern Devices with JDK10 and Beyond" to JEP 337: https://bugs.openjdk.java.net/browse/JDK-8203434 Slide 15 to 24 covers some preliminary data we collected on the RDMA work here. Please note: all the public APIs we talked in the presentation was not up to date. Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Tuesday, December 4, 2018 10:05 AM To: Lu, Yingqi Cc: Alan Bateman ; Viswanathan, Sandhya ; nio-dev at openjdk.java.net; Aundhe, Shirish ; Kaczmarek, Eric Subject: Re: adding rsockets support into JDK Hi Lucy, On Dec 3, 2018, at 7:05 PM, Lu, Yingqi > wrote: Thank you for the quick check. I forgot to add RdmaSocketImpl.c into the original patch file. The patch is updated at the same link:http://cr.openjdk.java.net/~ylu/8195160.23/. Please let me know if you see any issues with the updated version. Build and test succeeded with this version. I also took a look at the wording changes you made athttp://cr.openjdk.java.net/~bpb/8195160/webrev-22-delta/. They look pretty good to me:). Unless others have more feedback on the wording, I will incorporate exactly the changes into version 24. Sounds good. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Wed Dec 5 05:15:29 2018 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Wed, 5 Dec 2018 05:15:29 +0000 Subject: adding rsockets support into JDK In-Reply-To: <8C5ED530-8C5B-4A99-8471-B43B82B8E5E3@oracle.com> References: <120160AC-6250-4933-AE93-CAF36861A153@oracle.com> <4b5d8793-5a1d-2ca9-f012-6b26812fc8b1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159263B@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159353B@ORSMSX113.amr.corp.intel.com> <046bdfb4-cfdf-451a-74fe-435a698057af@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1593F08@ORSMSX113.amr.corp.intel.com> <2F1BEBF0-C79C-4135-8FC8-15B02D5DB0E9@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15941F6@ORSMSX113.amr.corp.intel.com> <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> <8C5ED530-8C5B-4A99-8471-B43B82B8E5E3@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD159AE8E@ORSMSX113.amr.corp.intel.com> Hi Chris, Thank you for your feedback. Here is the version 24 of the patch: http://cr.openjdk.java.net/~ylu/8195160.24 In this version, 1. I first applied Javadoc wording changes following Brian's suggestion from http://cr.openjdk.java.net/~bpb/8195160/webrev-22-delta/openjdk.patch I then regenerated the Javadoc at http://cr.openjdk.java.net/~ylu/rsocket_docs/api/jdk.net/jdk/net/package-summary.html and attached the updated zip file at CSR at https://bugs.openjdk.java.net/browse/JDK-8205186. 2. Then, I applied the three patches from you in the order of 23.1, 23.2 and 23.3. 3. I took a look at the first issue you found regarding to socket channel read does not return EOF at the end. I found that issue might not related to the current patch. I tested regular TCP socket channel with both JDK11 and current trunk and I saw the exact same issue. 4. I am looking into the second issue you found with null binding. Thanks, Lucy From: Chris Hegarty [mailto:chris.hegarty at oracle.com] Sent: Tuesday, December 4, 2018 12:23 PM To: Lu, Yingqi Cc: Brian Burkhalter ; Aundhe, Shirish ; Viswanathan, Sandhya ; nio-dev at openjdk.java.net; Kaczmarek, Eric Subject: Re: adding rsockets support into JDK Lucy, On 4 Dec 2018, at 03:05, Lu, Yingqi > wrote: Hi Brian, Thank you for the quick check. I forgot to add RdmaSocketImpl.c into the original patch file. The patch is updated at the same link: http://cr.openjdk.java.net/~ylu/8195160.23/. Please let me know if you see any issues with the updated version. Finally, I have an environment where I can build and run the tests for this code. I need to circle back and document this, but for now I can successfully create an rsocket and run the java tests. I am continuing to review the code, but here are a few comments so far: 1) BasicSocketChannelTest.java There is a race in the test code. Sometimes the test fails with an exception when it tries to shutdown the input/output of the client socket/channel. The reason is that the main thread may have already closed the socket/channel. While fixing the race I took the liberty of refactoring the test a little. It passes successfully, reliably, now in my environment. https://cr.openjdk.java.net/~chegar/rsocket/webrev.23.1/ Note: I've left a TODO comment in the test. Read to EOF is not working as I would expect ( the original version of the test did not exercise this). I'll revisit this later. 2) Test consistency of API exception, across all platforms. I added a test to ensure that NPE and other API exceptions are being thrown as expected. This test can be run on all platforms, so it is not Linux specific. I used testng, very nice for test kinda tests. https://cr.openjdk.java.net/~chegar/rsocket/webrev.23.2/ 3) Handling of null binding, automatically assigned address. Handling of null binding is a bit tricky now that we have protocol family support. The correct InetXAddress "any local address" needs to be created. I added a test and a little refactoring. If you run the test without any source changes you will be able to see the issues. https://cr.openjdk.java.net/~chegar/rsocket/webrev.23.3/ Note: There is still an open issue, ToDo, with java.net.Socket. You can see this by a small comment in the test. I'll revisit this later. -Chris. P.S. The above webrevs were created from patches in my patch queue, and should apply cleanly to version 23 if applied in order. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Wed Dec 5 06:44:36 2018 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Wed, 5 Dec 2018 06:44:36 +0000 Subject: adding rsockets support into JDK In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD159AE8E@ORSMSX113.amr.corp.intel.com> References: <120160AC-6250-4933-AE93-CAF36861A153@oracle.com> <4b5d8793-5a1d-2ca9-f012-6b26812fc8b1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159263B@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159353B@ORSMSX113.amr.corp.intel.com> <046bdfb4-cfdf-451a-74fe-435a698057af@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1593F08@ORSMSX113.amr.corp.intel.com> <2F1BEBF0-C79C-4135-8FC8-15B02D5DB0E9@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15941F6@ORSMSX113.amr.corp.intel.com> <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> <8C5ED530-8C5B-4A99-8471-B43B82B8E5E3@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159AE8E@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD159AFCA@ORSMSX113.amr.corp.intel.com> Hi Chris, For the null binding issue, I took a quick look. Depends on the value of InetAddress.preferIPv6Address, Inet6AddressImpl.anyLocalAddress() will return either IPv6 address or IPv4 address. However, value of InetAddress.preferIPv6Address does not relate to protocol family. I think this might be why we get IPv4 address when we expect to have IPv6 address during null binding. Since both java.net.InetAddress and java.net.Socket are part of public JDK APIs, I am looking forward to your suggestions and guidance to solve the issue :) Thank you for your help! Lucy From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of Lu, Yingqi Sent: Tuesday, December 4, 2018 9:15 PM To: Chris Hegarty Cc: Viswanathan, Sandhya ; nio-dev at openjdk.java.net; Aundhe, Shirish ; Kaczmarek, Eric Subject: RE: adding rsockets support into JDK Hi Chris, Thank you for your feedback. Here is the version 24 of the patch: http://cr.openjdk.java.net/~ylu/8195160.24 In this version, 1. I first applied Javadoc wording changes following Brian's suggestion from http://cr.openjdk.java.net/~bpb/8195160/webrev-22-delta/openjdk.patch I then regenerated the Javadoc at http://cr.openjdk.java.net/~ylu/rsocket_docs/api/jdk.net/jdk/net/package-summary.html and attached the updated zip file at CSR at https://bugs.openjdk.java.net/browse/JDK-8205186. 2. Then, I applied the three patches from you in the order of 23.1, 23.2 and 23.3. 3. I took a look at the first issue you found regarding to socket channel read does not return EOF at the end. I found that issue might not related to the current patch. I tested regular TCP socket channel with both JDK11 and current trunk and I saw the exact same issue. 4. I am looking into the second issue you found with null binding. Thanks, Lucy From: Chris Hegarty [mailto:chris.hegarty at oracle.com] Sent: Tuesday, December 4, 2018 12:23 PM To: Lu, Yingqi > Cc: Brian Burkhalter >; Aundhe, Shirish >; Viswanathan, Sandhya >; nio-dev at openjdk.java.net; Kaczmarek, Eric > Subject: Re: adding rsockets support into JDK Lucy, On 4 Dec 2018, at 03:05, Lu, Yingqi > wrote: Hi Brian, Thank you for the quick check. I forgot to add RdmaSocketImpl.c into the original patch file. The patch is updated at the same link: http://cr.openjdk.java.net/~ylu/8195160.23/. Please let me know if you see any issues with the updated version. Finally, I have an environment where I can build and run the tests for this code. I need to circle back and document this, but for now I can successfully create an rsocket and run the java tests. I am continuing to review the code, but here are a few comments so far: 1) BasicSocketChannelTest.java There is a race in the test code. Sometimes the test fails with an exception when it tries to shutdown the input/output of the client socket/channel. The reason is that the main thread may have already closed the socket/channel. While fixing the race I took the liberty of refactoring the test a little. It passes successfully, reliably, now in my environment. https://cr.openjdk.java.net/~chegar/rsocket/webrev.23.1/ Note: I've left a TODO comment in the test. Read to EOF is not working as I would expect ( the original version of the test did not exercise this). I'll revisit this later. 2) Test consistency of API exception, across all platforms. I added a test to ensure that NPE and other API exceptions are being thrown as expected. This test can be run on all platforms, so it is not Linux specific. I used testng, very nice for test kinda tests. https://cr.openjdk.java.net/~chegar/rsocket/webrev.23.2/ 3) Handling of null binding, automatically assigned address. Handling of null binding is a bit tricky now that we have protocol family support. The correct InetXAddress "any local address" needs to be created. I added a test and a little refactoring. If you run the test without any source changes you will be able to see the issues. https://cr.openjdk.java.net/~chegar/rsocket/webrev.23.3/ Note: There is still an open issue, ToDo, with java.net.Socket. You can see this by a small comment in the test. I'll revisit this later. -Chris. P.S. The above webrevs were created from patches in my patch queue, and should apply cleanly to version 23 if applied in order. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.hegarty at oracle.com Wed Dec 5 14:39:03 2018 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 5 Dec 2018 14:39:03 +0000 Subject: adding rsockets support into JDK In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD159AE8E@ORSMSX113.amr.corp.intel.com> References: <120160AC-6250-4933-AE93-CAF36861A153@oracle.com> <4b5d8793-5a1d-2ca9-f012-6b26812fc8b1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159263B@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159353B@ORSMSX113.amr.corp.intel.com> <046bdfb4-cfdf-451a-74fe-435a698057af@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1593F08@ORSMSX113.amr.corp.intel.com> <2F1BEBF0-C79C-4135-8FC8-15B02D5DB0E9@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15941F6@ORSMSX113.amr.corp.intel.com> <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> <8C5ED530-8C5B-4A99-8471-B43B82B8E5E3@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159AE8E@ORSMSX113.amr.corp.intel.com> Message-ID: Lucy, > On 5 Dec 2018, at 05:15, Lu, Yingqi wrote: > ... > > 3. I took a look at the first issue you found regarding to socket channel read does not return EOF at the end. I found that issue might not related to the current patch. I tested regular TCP socket channel with both JDK11 and current trunk and I saw the exact same issue. I should have spotted this earlier! The test creates a buffer with just enough space for the expected message. When the complete message has been received, the read method will return 0 since there is not enough space in the buffer, even though the only remaining data to read is EOF. I fixed this in the test and added a few more assertions for the shutdown states. https://cr.openjdk.java.net/~chegar/rsocket/webrev.24.1/ -Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Wed Dec 5 17:03:34 2018 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Wed, 5 Dec 2018 17:03:34 +0000 Subject: adding rsockets support into JDK In-Reply-To: References: <120160AC-6250-4933-AE93-CAF36861A153@oracle.com> <4b5d8793-5a1d-2ca9-f012-6b26812fc8b1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159263B@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159353B@ORSMSX113.amr.corp.intel.com> <046bdfb4-cfdf-451a-74fe-435a698057af@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1593F08@ORSMSX113.amr.corp.intel.com> <2F1BEBF0-C79C-4135-8FC8-15B02D5DB0E9@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15941F6@ORSMSX113.amr.corp.intel.com> <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> <8C5ED530-8C5B-4A99-8471-B43B82B8E5E3@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159AE8E@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD159B83F@ORSMSX113.amr.corp.intel.com> Hi Chris, That totally explains it! Thank you for the help! I already applied the patch locally. I am waiting for more feedback before doing version 25 of the patch. Thanks, Lucy From: Chris Hegarty [mailto:chris.hegarty at oracle.com] Sent: Wednesday, December 5, 2018 6:39 AM To: Lu, Yingqi Cc: Brian Burkhalter ; Aundhe, Shirish ; Viswanathan, Sandhya ; nio-dev at openjdk.java.net; Kaczmarek, Eric Subject: Re: adding rsockets support into JDK Lucy, On 5 Dec 2018, at 05:15, Lu, Yingqi > wrote: ... 3. I took a look at the first issue you found regarding to socket channel read does not return EOF at the end. I found that issue might not related to the current patch. I tested regular TCP socket channel with both JDK11 and current trunk and I saw the exact same issue. I should have spotted this earlier! The test creates a buffer with just enough space for the expected message. When the complete message has been received, the read method will return 0 since there is not enough space in the buffer, even though the only remaining data to read is EOF. I fixed this in the test and added a few more assertions for the shutdown states. https://cr.openjdk.java.net/~chegar/rsocket/webrev.24.1/ -Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.hegarty at oracle.com Sat Dec 8 09:39:23 2018 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Sat, 8 Dec 2018 09:39:23 +0000 Subject: adding rsockets support into JDK In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD159B83F@ORSMSX113.amr.corp.intel.com> References: <120160AC-6250-4933-AE93-CAF36861A153@oracle.com> <4b5d8793-5a1d-2ca9-f012-6b26812fc8b1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159263B@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159353B@ORSMSX113.amr.corp.intel.com> <046bdfb4-cfdf-451a-74fe-435a698057af@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1593F08@ORSMSX113.amr.corp.intel.com> <2F1BEBF0-C79C-4135-8FC8-15B02D5DB0E9@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15941F6@ORSMSX113.amr.corp.intel.com> <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> <8C5ED530-8C5B-4A99-8471-B43B82B8E5E3@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159AE8E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159B83F@ORSMSX113.amr.corp.intel.com> Message-ID: <5C96DD75-48C1-474F-9A91-5671256353F1@oracle.com> I would like to summarise a few issues that have been encountered when reviewing and testing. - An issue in the Linux rdma-core implementation has been found and reported [1][2]. It is not immediately clear how to reasonably workaround this issue, or if/when it will be acknowledged and/or fixed. - It has become apparent that mixing blocking and non-blocking connect/accept operations, in the same thread, may cause issues. For example, attempting to setup a connected-socket on the same host by issuing a non-blocking connect followed by a blocking accept, will just hang and not make progress [3]. Upon further enquiries it appears that the programming model for rsocket is a subtly different than that of the regular Berkeley sockets ( at least for the connection handshake ). It is not immediately clear how to reasonably workaround this issue ( it's not a bug in rdma-core, but more a fundamental part of its thread-less operation ). -Chris [1] https://marc.info/?l=linux-rdma&m=154418100500796&w=2 [2] https://cr.openjdk.java.net/~chegar/rsocket/testNonBlocking_raccept.c [3] https://cr.openjdk.java.net/~chegar/rsocket/testBlocking_raccept.c > On 5 Dec 2018, at 17:03, Lu, Yingqi wrote: > > Hi Chris, > > That totally explains it! Thank you for the help! I already applied the patch locally. > > I am waiting for more feedback before doing version 25 of the patch. > > Thanks, > Lucy > ? <> > <>From: Chris Hegarty [mailto:chris.hegarty at oracle.com] > Sent: Wednesday, December 5, 2018 6:39 AM > To: Lu, Yingqi > Cc: Brian Burkhalter ; Aundhe, Shirish ; Viswanathan, Sandhya ; nio-dev at openjdk.java.net; Kaczmarek, Eric > Subject: Re: adding rsockets support into JDK > > Lucy, > > > On 5 Dec 2018, at 05:15, Lu, Yingqi > wrote: > ... > > 3. I took a look at the first issue you found regarding to socket channel read does not return EOF at the end. I found that issue might not related to the current patch. I tested regular TCP socket channel with both JDK11 and current trunk and I saw the exact same issue. > > I should have spotted this earlier! The test creates a buffer with just > enough space for the expected message. When the complete message has > been received, the read method will return 0 since there is not enough > space in the buffer, even though the only remaining data to read is > EOF. I fixed this in the test and added a few more assertions for the > shutdown states. > > https://cr.openjdk.java.net/~chegar/rsocket/webrev.24.1/ > > -Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Sat Dec 8 16:09:56 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 8 Dec 2018 16:09:56 +0000 Subject: adding rsockets support into JDK In-Reply-To: <5C96DD75-48C1-474F-9A91-5671256353F1@oracle.com> References: <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> <8C5ED530-8C5B-4A99-8471-B43B82B8E5E3@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159AE8E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159B83F@ORSMSX113.amr.corp.intel.com> <5C96DD75-48C1-474F-9A91-5671256353F1@oracle.com> Message-ID: On 08/12/2018 09:39, Chris Hegarty wrote: > : > > - It has become apparent that mixing blocking and non-blocking > ? connect/accept operations, in the same thread, may cause issues. For > ? example, attempting to setup a connected-socket on the same host by > ? issuing a non-blocking connect followed by a blocking accept, will > ? just hang and not make progress [3]. Upon further enquiries it appears > ? that the programming model for rsocket is a subtly different than that > ? of the regular Berkeley sockets ( at least for the connection > ? handshake ). It is not immediately clear how to reasonably workaround > ? this issue ( it's not a bug in rdma-core, but more a fundamental part > ? of its thread-less operation ). > Would it be possible to expand on this to say whether the same issues arises when the non-blocking connect is initiated on a different thread, or in a different process, or even a different machine on the fabric. That is, if the socket is non-blocking and I do a rconnect and then delay before doing anything else on the socket then will the peer doing accept be blocked/hung in the mean-time? -Alan From yingqi.lu at intel.com Tue Dec 11 00:36:13 2018 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Tue, 11 Dec 2018 00:36:13 +0000 Subject: adding rsockets support into JDK In-Reply-To: References: <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> <8C5ED530-8C5B-4A99-8471-B43B82B8E5E3@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159AE8E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159B83F@ORSMSX113.amr.corp.intel.com> <5C96DD75-48C1-474F-9A91-5671256353F1@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD15B5E36@ORSMSX113.amr.corp.intel.com> Hi Alan/Chris, I was able to confirm that connecting on non-blocking socket causes issues. It happens when connect/accept occurs in the same thread or different threads in the same process. Then, I did a small tweak in Chris's sample application by spawning a thread doing rpoll on the connection_fd. Now, the connect/accept works in both of the cases above. Please let me know if this is a valid workaround for the issue. Performance wise, this workaround should not impact send/receive at all. It might only add a small overhead to the connection setup phase only with non-blocking RDMA socket. The modified app code is available at For connect/accept occur in the same thread: https://cr.openjdk.java.net/~ylu/testNonBlocking_raccept_modified.c For connect/accept occur in two different threads: https://cr.openjdk.java.net/~ylu/testNonBlocking_raccept_modified_2threads.c Thanks, Lucy >-----Original Message----- >From: Alan Bateman [mailto:Alan.Bateman at oracle.com] >Sent: Saturday, December 8, 2018 8:10 AM >To: Chris Hegarty ; Lu, Yingqi >; nio-dev at openjdk.java.net >Cc: Viswanathan, Sandhya ; Aundhe, >Shirish ; Kaczmarek, Eric > >Subject: Re: adding rsockets support into JDK > >On 08/12/2018 09:39, Chris Hegarty wrote: >> : >> >> - It has become apparent that mixing blocking and non-blocking >> ? connect/accept operations, in the same thread, may cause issues. For >> ? example, attempting to setup a connected-socket on the same host by >> ? issuing a non-blocking connect followed by a blocking accept, will >> ? just hang and not make progress [3]. Upon further enquiries it >> appears >> ? that the programming model for rsocket is a subtly different than >> that >> ? of the regular Berkeley sockets ( at least for the connection >> ? handshake ). It is not immediately clear how to reasonably >> workaround >> ? this issue ( it's not a bug in rdma-core, but more a fundamental >> part >> ? of its thread-less operation ). >> >Would it be possible to expand on this to say whether the same issues arises >when the non-blocking connect is initiated on a different thread, or in a >different process, or even a different machine on the fabric. >That is, if the socket is non-blocking and I do a rconnect and then delay before >doing anything else on the socket then will the peer doing accept be >blocked/hung in the mean-time? > >-Alan From yingqi.lu at intel.com Tue Dec 11 00:48:01 2018 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Tue, 11 Dec 2018 00:48:01 +0000 Subject: adding rsockets support into JDK References: <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> <8C5ED530-8C5B-4A99-8471-B43B82B8E5E3@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159AE8E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159B83F@ORSMSX113.amr.corp.intel.com> <5C96DD75-48C1-474F-9A91-5671256353F1@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD15B5E99@ORSMSX113.amr.corp.intel.com> Rename the sample code filenames: For connect/accept occur in the same thread: http://cr.openjdk.java.net/~ylu/testNonBlocking_rconnect_modified.c For connect/accept occur in two different threads: http://cr.openjdk.java.net/~ylu/testNonBlocking_rconnect_modified_2threads.c Thanks, Lucy >-----Original Message----- >From: Lu, Yingqi >Sent: Monday, December 10, 2018 4:36 PM >To: 'Alan Bateman' ; Chris Hegarty >; nio-dev at openjdk.java.net >Cc: Viswanathan, Sandhya ; Aundhe, >Shirish ; Kaczmarek, Eric > >Subject: RE: adding rsockets support into JDK > >Hi Alan/Chris, > >I was able to confirm that connecting on non-blocking socket causes issues. It >happens when connect/accept occurs in the same thread or different threads >in the same process. > >Then, I did a small tweak in Chris's sample application by spawning a thread >doing rpoll on the connection_fd. Now, the connect/accept works in both of >the cases above. Please let me know if this is a valid workaround for the issue. > >Performance wise, this workaround should not impact send/receive at all. It >might only add a small overhead to the connection setup phase only with non- >blocking RDMA socket. > >The modified app code is available at > >For connect/accept occur in the same thread: >https://cr.openjdk.java.net/~ylu/testNonBlocking_raccept_modified.c > >For connect/accept occur in two different threads: >https://cr.openjdk.java.net/~ylu/testNonBlocking_raccept_modified_2threa >ds.c > >Thanks, >Lucy > >>-----Original Message----- >>From: Alan Bateman [mailto:Alan.Bateman at oracle.com] >>Sent: Saturday, December 8, 2018 8:10 AM >>To: Chris Hegarty ; Lu, Yingqi >>; nio-dev at openjdk.java.net >>Cc: Viswanathan, Sandhya ; Aundhe, >>Shirish ; Kaczmarek, Eric >> >>Subject: Re: adding rsockets support into JDK >> >>On 08/12/2018 09:39, Chris Hegarty wrote: >>> : >>> >>> - It has become apparent that mixing blocking and non-blocking >>> ? connect/accept operations, in the same thread, may cause issues. >>> For >>> ? example, attempting to setup a connected-socket on the same host by >>> ? issuing a non-blocking connect followed by a blocking accept, will >>> ? just hang and not make progress [3]. Upon further enquiries it >>> appears >>> ? that the programming model for rsocket is a subtly different than >>> that >>> ? of the regular Berkeley sockets ( at least for the connection >>> ? handshake ). It is not immediately clear how to reasonably >>> workaround >>> ? this issue ( it's not a bug in rdma-core, but more a fundamental >>> part >>> ? of its thread-less operation ). >>> >>Would it be possible to expand on this to say whether the same issues >>arises when the non-blocking connect is initiated on a different >>thread, or in a different process, or even a different machine on the fabric. >>That is, if the socket is non-blocking and I do a rconnect and then >>delay before doing anything else on the socket then will the peer doing >>accept be blocked/hung in the mean-time? >> >>-Alan From yingqi.lu at intel.com Tue Dec 11 05:44:59 2018 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Tue, 11 Dec 2018 05:44:59 +0000 Subject: adding rsockets support into JDK In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD15B5E99@ORSMSX113.amr.corp.intel.com> References: <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> <8C5ED530-8C5B-4A99-8471-B43B82B8E5E3@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159AE8E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159B83F@ORSMSX113.amr.corp.intel.com> <5C96DD75-48C1-474F-9A91-5671256353F1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B5E99@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD15B626F@ORSMSX113.amr.corp.intel.com> Hi Alan/Chris, For the first issue mentioned in Chris's summary (rread returns EAGAIN when listening socket is non-blocking and accepted socket is blocking), would an rpoll call on the accepted socket workaround the issue? The EAGAIN error code is only expected to be seen for once. rpoll blocks till the resource is available. The modified code is available at: http://cr.openjdk.java.net/~ylu/testNonBlocking_raccept_modified.c Please let me know if this is a reasonable workaround. Thanks, Lucy >-----Original Message----- >From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of Lu, >Yingqi >Sent: Monday, December 10, 2018 4:48 PM >To: Alan Bateman ; Chris Hegarty >; nio-dev at openjdk.java.net >Cc: Viswanathan, Sandhya ; Aundhe, >Shirish ; Kaczmarek, Eric > >Subject: RE: adding rsockets support into JDK > >Rename the sample code filenames: > >For connect/accept occur in the same thread: >http://cr.openjdk.java.net/~ylu/testNonBlocking_rconnect_modified.c > >For connect/accept occur in two different threads: >http://cr.openjdk.java.net/~ylu/testNonBlocking_rconnect_modified_2threa >ds.c > >Thanks, >Lucy > >>-----Original Message----- >>From: Lu, Yingqi >>Sent: Monday, December 10, 2018 4:36 PM >>To: 'Alan Bateman' ; Chris Hegarty >>; nio-dev at openjdk.java.net >>Cc: Viswanathan, Sandhya ; Aundhe, >>Shirish ; Kaczmarek, Eric >> >>Subject: RE: adding rsockets support into JDK >> >>Hi Alan/Chris, >> >>I was able to confirm that connecting on non-blocking socket causes >>issues. It happens when connect/accept occurs in the same thread or >>different threads in the same process. >> >>Then, I did a small tweak in Chris's sample application by spawning a >>thread doing rpoll on the connection_fd. Now, the connect/accept works >>in both of the cases above. Please let me know if this is a valid workaround >for the issue. >> >>Performance wise, this workaround should not impact send/receive at >>all. It might only add a small overhead to the connection setup phase >>only with non- blocking RDMA socket. >> >>The modified app code is available at >> >>For connect/accept occur in the same thread: >>https://cr.openjdk.java.net/~ylu/testNonBlocking_raccept_modified.c >> >>For connect/accept occur in two different threads: >>https://cr.openjdk.java.net/~ylu/testNonBlocking_raccept_modified_2thre >>a >>ds.c >> >>Thanks, >>Lucy >> >>>-----Original Message----- >>>From: Alan Bateman [mailto:Alan.Bateman at oracle.com] >>>Sent: Saturday, December 8, 2018 8:10 AM >>>To: Chris Hegarty ; Lu, Yingqi >>>; nio-dev at openjdk.java.net >>>Cc: Viswanathan, Sandhya ; Aundhe, >>>Shirish ; Kaczmarek, Eric >>> >>>Subject: Re: adding rsockets support into JDK >>> >>>On 08/12/2018 09:39, Chris Hegarty wrote: >>>> : >>>> >>>> - It has become apparent that mixing blocking and non-blocking >>>> ? connect/accept operations, in the same thread, may cause issues. >>>> For >>>> ? example, attempting to setup a connected-socket on the same host >>>> by >>>> ? issuing a non-blocking connect followed by a blocking accept, will >>>> ? just hang and not make progress [3]. Upon further enquiries it >>>> appears >>>> ? that the programming model for rsocket is a subtly different than >>>> that >>>> ? of the regular Berkeley sockets ( at least for the connection >>>> ? handshake ). It is not immediately clear how to reasonably >>>> workaround >>>> ? this issue ( it's not a bug in rdma-core, but more a fundamental >>>> part >>>> ? of its thread-less operation ). >>>> >>>Would it be possible to expand on this to say whether the same issues >>>arises when the non-blocking connect is initiated on a different >>>thread, or in a different process, or even a different machine on the fabric. >>>That is, if the socket is non-blocking and I do a rconnect and then >>>delay before doing anything else on the socket then will the peer >>>doing accept be blocked/hung in the mean-time? >>> >>>-Alan From yingqi.lu at intel.com Tue Dec 11 05:51:23 2018 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Tue, 11 Dec 2018 05:51:23 +0000 Subject: adding rsockets support into JDK References: <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> <8C5ED530-8C5B-4A99-8471-B43B82B8E5E3@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159AE8E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159B83F@ORSMSX113.amr.corp.intel.com> <5C96DD75-48C1-474F-9A91-5671256353F1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B5E99@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD15B62D6@ORSMSX113.amr.corp.intel.com> Forgot to mention that the changes are based on Chris's sample code at https://cr.openjdk.java.net/~chegar/rsocket/testNonBlocking_raccept.c Thanks, Lucy >-----Original Message----- >From: Lu, Yingqi >Sent: Monday, December 10, 2018 9:43 PM >To: 'Lu, Yingqi' ; Alan Bateman >; Chris Hegarty ; >nio-dev at openjdk.java.net >Cc: Viswanathan, Sandhya ; Aundhe, >Shirish ; Kaczmarek, Eric > >Subject: RE: adding rsockets support into JDK > >Hi Alan/Chris, > >For the first issue mentioned in Chris's summary (rread returns EAGAIN when >listening socket is non-blocking and accepted socket is blocking), would an >rpoll call on the accepted socket workaround the issue? The EAGAIN error >code is only expected to be seen for once. rpoll blocks till the resource is >available. > >The modified code is available at: >http://cr.openjdk.java.net/~ylu/testNonBlocking_raccept_modified.c > >Please let me know if this is a reasonable workaround. > >Thanks, >Lucy > >>-----Original Message----- >>From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of >>Lu, Yingqi >>Sent: Monday, December 10, 2018 4:48 PM >>To: Alan Bateman ; Chris Hegarty >>; nio-dev at openjdk.java.net >>Cc: Viswanathan, Sandhya ; Aundhe, >>Shirish ; Kaczmarek, Eric >> >>Subject: RE: adding rsockets support into JDK >> >>Rename the sample code filenames: >> >>For connect/accept occur in the same thread: >>http://cr.openjdk.java.net/~ylu/testNonBlocking_rconnect_modified.c >> >>For connect/accept occur in two different threads: >>http://cr.openjdk.java.net/~ylu/testNonBlocking_rconnect_modified_2thre >>a >>ds.c >> >>Thanks, >>Lucy >> >>>-----Original Message----- >>>From: Lu, Yingqi >>>Sent: Monday, December 10, 2018 4:36 PM >>>To: 'Alan Bateman' ; Chris Hegarty >>>; nio-dev at openjdk.java.net >>>Cc: Viswanathan, Sandhya ; Aundhe, >>>Shirish ; Kaczmarek, Eric >>> >>>Subject: RE: adding rsockets support into JDK >>> >>>Hi Alan/Chris, >>> >>>I was able to confirm that connecting on non-blocking socket causes >>>issues. It happens when connect/accept occurs in the same thread or >>>different threads in the same process. >>> >>>Then, I did a small tweak in Chris's sample application by spawning a >>>thread doing rpoll on the connection_fd. Now, the connect/accept works >>>in both of the cases above. Please let me know if this is a valid >>>workaround >>for the issue. >>> >>>Performance wise, this workaround should not impact send/receive at >>>all. It might only add a small overhead to the connection setup phase >>>only with non- blocking RDMA socket. >>> >>>The modified app code is available at >>> >>>For connect/accept occur in the same thread: >>>https://cr.openjdk.java.net/~ylu/testNonBlocking_raccept_modified.c >>> >>>For connect/accept occur in two different threads: >>>https://cr.openjdk.java.net/~ylu/testNonBlocking_raccept_modified_2thr >>>e >>>a >>>ds.c >>> >>>Thanks, >>>Lucy >>> >>>>-----Original Message----- >>>>From: Alan Bateman [mailto:Alan.Bateman at oracle.com] >>>>Sent: Saturday, December 8, 2018 8:10 AM >>>>To: Chris Hegarty ; Lu, Yingqi >>>>; nio-dev at openjdk.java.net >>>>Cc: Viswanathan, Sandhya ; Aundhe, >>>>Shirish ; Kaczmarek, Eric >>>> >>>>Subject: Re: adding rsockets support into JDK >>>> >>>>On 08/12/2018 09:39, Chris Hegarty wrote: >>>>> : >>>>> >>>>> - It has become apparent that mixing blocking and non-blocking >>>>> ? connect/accept operations, in the same thread, may cause issues. >>>>> For >>>>> ? example, attempting to setup a connected-socket on the same host >>>>> by >>>>> ? issuing a non-blocking connect followed by a blocking accept, >>>>> will >>>>> ? just hang and not make progress [3]. Upon further enquiries it >>>>> appears >>>>> ? that the programming model for rsocket is a subtly different than >>>>> that >>>>> ? of the regular Berkeley sockets ( at least for the connection >>>>> ? handshake ). It is not immediately clear how to reasonably >>>>> workaround >>>>> ? this issue ( it's not a bug in rdma-core, but more a fundamental >>>>> part >>>>> ? of its thread-less operation ). >>>>> >>>>Would it be possible to expand on this to say whether the same issues >>>>arises when the non-blocking connect is initiated on a different >>>>thread, or in a different process, or even a different machine on the fabric. >>>>That is, if the socket is non-blocking and I do a rconnect and then >>>>delay before doing anything else on the socket then will the peer >>>>doing accept be blocked/hung in the mean-time? >>>> >>>>-Alan From Nick.Gasson at arm.com Tue Dec 11 08:05:21 2018 From: Nick.Gasson at arm.com (Nick Gasson (Arm Technology China)) Date: Tue, 11 Dec 2018 08:05:21 +0000 Subject: RFR: 8215202: AArch64: jtreg test test/jdk/sun/nio/cs/FindEncoderBugs.java fails Message-ID: Hi, Please help me review this patch to fix an issue with the ISO-8859-1 encoder intrinsic on AArch64 (nio-dev CC-ed due to test change). Webrev: http://cr.openjdk.java.net/~njian/8215202/webrev.0/ Bug: https://bugs.openjdk.java.net/browse/JDK-8215202 The jtreg test test/jdk/sun/nio/cs/FindEncoderBugs.java fails with the following error on AArch64 with compilation: Buffer overrun: ISO-8859-1 "\u8a8e"[0/1] => UNMAPPABLE[1] ""[0/19] -114 failures=1 Passed = 223410156, failed = 10000 STDERR: java.lang.AssertionError: Some tests failed at FindEncoderBugs.main(FindEncoderBugs.java:526) The problem is caused by the encoder function for the ISO-8859-1 character set writing one extra character to the output array when it encounters a character it can't encode. This is implemented as an intrinsic on AArch64, but the intrinsic doesn't exactly match the original Java code in ISO_8859_1.java: @HotSpotIntrinsicCandidate private static int implEncodeISOArray(char[] sa, int sp, byte[] da, int dp, int len) { int i = 0; for (; i < len; i++) { char c = sa[sp++]; if (c > '\u00FF') // (1) break; da[dp++] = (byte)c; // (2) } return i; } In the intrinsic code statements (1) and (2) are swapped. Also Extended the FindEncoderBugs test so that it hits all cases in the intrinsic assembly. Currently only the final NEXT_1 section is covered by this test. Thanks, Nick From chris.hegarty at oracle.com Tue Dec 11 19:30:45 2018 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 11 Dec 2018 19:30:45 +0000 Subject: adding rsockets support into JDK In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD15B5E36@ORSMSX113.amr.corp.intel.com> References: <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> <8C5ED530-8C5B-4A99-8471-B43B82B8E5E3@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159AE8E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159B83F@ORSMSX113.amr.corp.intel.com> <5C96DD75-48C1-474F-9A91-5671256353F1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B5E36@ORSMSX113.amr.corp.intel.com> Message-ID: <3B5379DE-5D47-4B50-8DFC-6B4CEF93A977@oracle.com> Lucy, Sure, the small test scenarios can be modified to make them "work". The bigger question is how the proposed JDK-RDMA implementation code can be modified to provide the semantics of the `SocketChannel` API. Issue #1: rread returns EAGAIN. One possible solution could be that the blocking code path in RdmaSocketChannelImpl could fallback back into a blocking rpoll POLLOUT if the IOUtil.read method invocation returns IOStatus.UNAVAILABLE. I think this should work, and not have too much of a negative impact since the fallback will only occur infrequently. Q: can rwrite return EAGAIN? I have not checked yet. Issue #2: This issue is likely to be encountered mainly during testing, since a non-blocking connect followed by an accept, on the same thread, is not all the common in non-test code. That said, the semantics of the SocketChannel API would lead one to expect it to work. ( I get that rsocket is not asynchronous, but the semantics of non-blocking channels implies some asynchronousity ). I wonder if the JDK-RDMA implementation should have a dedicated thread that "pulls" on unfinished non-blocking connects that are not subsequently registered with a Selector? Maybe accepts too? I'm not sure yet. -Chris. > On 11 Dec 2018, at 00:36, Lu, Yingqi wrote: > > Hi Alan/Chris, > > I was able to confirm that connecting on non-blocking socket causes issues. It happens when connect/accept occurs in the same thread or different threads in the same process. > > Then, I did a small tweak in Chris's sample application by spawning a thread doing rpoll on the connection_fd. Now, the connect/accept works in both of the cases above. Please let me know if this is a valid workaround for the issue. > > Performance wise, this workaround should not impact send/receive at all. It might only add a small overhead to the connection setup phase only with non-blocking RDMA socket. > > The modified app code is available at > > For connect/accept occur in the same thread: https://cr.openjdk.java.net/~ylu/testNonBlocking_raccept_modified.c > > For connect/accept occur in two different threads: https://cr.openjdk.java.net/~ylu/testNonBlocking_raccept_modified_2threads.c > > Thanks, > Lucy > >> -----Original Message----- >> From: Alan Bateman [mailto:Alan.Bateman at oracle.com] >> Sent: Saturday, December 8, 2018 8:10 AM >> To: Chris Hegarty ; Lu, Yingqi >> ; nio-dev at openjdk.java.net >> Cc: Viswanathan, Sandhya ; Aundhe, >> Shirish ; Kaczmarek, Eric >> >> Subject: Re: adding rsockets support into JDK >> >> On 08/12/2018 09:39, Chris Hegarty wrote: >>> : >>> >>> - It has become apparent that mixing blocking and non-blocking >>> connect/accept operations, in the same thread, may cause issues. For >>> example, attempting to setup a connected-socket on the same host by >>> issuing a non-blocking connect followed by a blocking accept, will >>> just hang and not make progress [3]. Upon further enquiries it >>> appears >>> that the programming model for rsocket is a subtly different than >>> that >>> of the regular Berkeley sockets ( at least for the connection >>> handshake ). It is not immediately clear how to reasonably >>> workaround >>> this issue ( it's not a bug in rdma-core, but more a fundamental >>> part >>> of its thread-less operation ). >>> >> Would it be possible to expand on this to say whether the same issues arises >> when the non-blocking connect is initiated on a different thread, or in a >> different process, or even a different machine on the fabric. >> That is, if the socket is non-blocking and I do a rconnect and then delay before >> doing anything else on the socket then will the peer doing accept be >> blocked/hung in the mean-time? >> >> -Alan From yingqi.lu at intel.com Tue Dec 11 20:05:13 2018 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Tue, 11 Dec 2018 20:05:13 +0000 Subject: adding rsockets support into JDK In-Reply-To: <3B5379DE-5D47-4B50-8DFC-6B4CEF93A977@oracle.com> References: <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> <8C5ED530-8C5B-4A99-8471-B43B82B8E5E3@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159AE8E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159B83F@ORSMSX113.amr.corp.intel.com> <5C96DD75-48C1-474F-9A91-5671256353F1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B5E36@ORSMSX113.amr.corp.intel.com> <3B5379DE-5D47-4B50-8DFC-6B4CEF93A977@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD15B7562@ORSMSX113.amr.corp.intel.com> Hi Chris, On issue #1: Q: can rwrite return EAGAIN? I have not checked yet. I checked with the testing app. Yes, it can return EAGAIN when the resource is not available. In this case, we need to do an rpoll on POLLOUT. I can quickly try a patch to address it. On issue #2: I discussed if we need to poll on the unfinished accepts with kernel RDMA developer yesterday. The answer I got was we do not really need to (we can of course). Reason is that connect side is the initiator of the handshake. To start a connection (rconnect being called), it sends out a "connection request message" to the accept side. If the message has not been sent over, raccept will simply return. If the message has arrived, rdma_accept be called and "connection response message" will be sent to connection side. The connect side will read the message and then send "connection ready message" to the accept side. rconnect only sends the first message, that is why we need a thread on the connect side to make sure the rest of steps are being taken care of. Thanks, Lucy >-----Original Message----- >From: Chris Hegarty [mailto:chris.hegarty at oracle.com] >Sent: Tuesday, December 11, 2018 11:31 AM >To: Lu, Yingqi >Cc: Alan Bateman ; nio-dev at openjdk.java.net; >Viswanathan, Sandhya ; Aundhe, Shirish >; Kaczmarek, Eric >Subject: Re: adding rsockets support into JDK > >Lucy, > >Sure, the small test scenarios can be modified to make them "work". The >bigger question is how the proposed JDK-RDMA implementation code can be >modified to provide the semantics of the `SocketChannel` API. > >Issue #1: rread returns EAGAIN. One possible solution could be that the >blocking code path in RdmaSocketChannelImpl could fallback back into a >blocking rpoll POLLOUT if the IOUtil.read method invocation returns >IOStatus.UNAVAILABLE. I think this should work, and not have too much of a >negative impact since the fallback will only occur infrequently. > Q: can rwrite return EAGAIN? I have not checked yet. > >Issue #2: This issue is likely to be encountered mainly during testing, since a >non-blocking connect followed by an accept, on the same thread, is not all the >common in non-test code. That said, the semantics of the SocketChannel API >would lead one to expect it to work. ( I get that rsocket is not asynchronous, >but the semantics of non-blocking channels implies some asynchronousity ). I >wonder if the JDK-RDMA implementation should have a dedicated thread that >"pulls" on unfinished non-blocking connects that are not subsequently >registered with a Selector? Maybe accepts too? I'm not sure yet. > >-Chris. > >> On 11 Dec 2018, at 00:36, Lu, Yingqi wrote: >> >> Hi Alan/Chris, >> >> I was able to confirm that connecting on non-blocking socket causes issues. >It happens when connect/accept occurs in the same thread or different >threads in the same process. >> >> Then, I did a small tweak in Chris's sample application by spawning a thread >doing rpoll on the connection_fd. Now, the connect/accept works in both of >the cases above. Please let me know if this is a valid workaround for the issue. >> >> Performance wise, this workaround should not impact send/receive at all. It >might only add a small overhead to the connection setup phase only with non- >blocking RDMA socket. >> >> The modified app code is available at >> >> For connect/accept occur in the same thread: >> https://cr.openjdk.java.net/~ylu/testNonBlocking_raccept_modified.c >> >> For connect/accept occur in two different threads: >> https://cr.openjdk.java.net/~ylu/testNonBlocking_raccept_modified_2thr >> eads.c >> >> Thanks, >> Lucy >> >>> -----Original Message----- >>> From: Alan Bateman [mailto:Alan.Bateman at oracle.com] >>> Sent: Saturday, December 8, 2018 8:10 AM >>> To: Chris Hegarty ; Lu, Yingqi >>> ; nio-dev at openjdk.java.net >>> Cc: Viswanathan, Sandhya ; Aundhe, >>> Shirish ; Kaczmarek, Eric >>> >>> Subject: Re: adding rsockets support into JDK >>> >>> On 08/12/2018 09:39, Chris Hegarty wrote: >>>> : >>>> >>>> - It has become apparent that mixing blocking and non-blocking >>>> connect/accept operations, in the same thread, may cause issues. For >>>> example, attempting to setup a connected-socket on the same host by >>>> issuing a non-blocking connect followed by a blocking accept, will >>>> just hang and not make progress [3]. Upon further enquiries it >>>> appears >>>> that the programming model for rsocket is a subtly different than >>>> that >>>> of the regular Berkeley sockets ( at least for the connection >>>> handshake ). It is not immediately clear how to reasonably >>>> workaround >>>> this issue ( it's not a bug in rdma-core, but more a fundamental >>>> part >>>> of its thread-less operation ). >>>> >>> Would it be possible to expand on this to say whether the same issues >>> arises when the non-blocking connect is initiated on a different >>> thread, or in a different process, or even a different machine on the fabric. >>> That is, if the socket is non-blocking and I do a rconnect and then >>> delay before doing anything else on the socket then will the peer >>> doing accept be blocked/hung in the mean-time? >>> >>> -Alan From Nick.Gasson at arm.com Wed Dec 12 01:13:29 2018 From: Nick.Gasson at arm.com (Nick Gasson (Arm Technology China)) Date: Wed, 12 Dec 2018 01:13:29 +0000 Subject: [aarch64-port-dev ] RFR: 8215202: AArch64: jtreg test test/jdk/sun/nio/cs/FindEncoderBugs.java fails In-Reply-To: <414ab5fa-33cd-47f3-e76f-2ee95291bea6@redhat.com> References: <414ab5fa-33cd-47f3-e76f-2ee95291bea6@redhat.com> Message-ID: Hi Andrew, > One thing: is randomChar() fully reproducible? That is to say, are > we sure that we always run the same tests? It uses jdk.test.lib.RandomFactory which seeds the RNG from the "seed" system property. So if this is set when running jtreg the results are fully reproducible. Thanks, Nick > -----Original Message----- > From: Andrew Haley > Sent: 12 December 2018 00:23 > To: Nick Gasson (Arm Technology China) ; hotspot- > compiler-dev at openjdk.java.net > Cc: nio-dev at openjdk.java.net; nd ; aarch64-port- > dev at openjdk.java.net > Subject: Re: [aarch64-port-dev ] RFR: 8215202: AArch64: jtreg test > test/jdk/sun/nio/cs/FindEncoderBugs.java fails > > Good catch, again. > > On 12/11/18 8:05 AM, Nick Gasson (Arm Technology China) wrote: > > Also Extended the FindEncoderBugs test so that it hits all cases in > > the intrinsic assembly. Currently only the final NEXT_1 section is > > covered by this test. > > One thing: is randomChar() fully reproducible? That is to say, are > we sure that we always run the same tests? > > -- > Andrew Haley > Java Platform Lead Engineer > Red Hat UK Ltd. > EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From yingqi.lu at intel.com Thu Dec 13 17:59:23 2018 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Thu, 13 Dec 2018 17:59:23 +0000 Subject: adding rsockets support into JDK In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD15B7562@ORSMSX113.amr.corp.intel.com> References: <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> <8C5ED530-8C5B-4A99-8471-B43B82B8E5E3@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159AE8E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159B83F@ORSMSX113.amr.corp.intel.com> <5C96DD75-48C1-474F-9A91-5671256353F1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B5E36@ORSMSX113.amr.corp.intel.com> <3B5379DE-5D47-4B50-8DFC-6B4CEF93A977@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B7562@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD15BB6D5@ORSMSX113.amr.corp.intel.com> Hi Chris/Alan, Here is the version 25 of the patch: http://cr.openjdk.java.net/~ylu/8195160.25/ In this version, I have modified following items: 1. I applied Javadoc wording change suggestions from Brian http://cr.openjdk.java.net/~bpb/8195160/webrev-22-delta/ 2. I applied the suggested changes from Chris http://cr.openjdk.java.net/~chegar/rsocket/webrev.23.1/ https://cr.openjdk.java.net/~chegar/rsocket/webrev.23.2/ http://cr.openjdk.java.net/~chegar/rsocket/webrev.23.3/ https://cr.openjdk.java.net/~chegar/rsocket/webrev.24.1/ 3. I made changes to address the issue #1 (non-blocking connect/blocking accept). Instead of making changes inside RdmaSocketChannelImpl.java, I think it might be easier to directly change the native implementation at src/jdk.net/linux/native/libextnet/LinuxRdmaSocketDispatcherImpl.c Please let me know your feedback. At the same time, I am working on addressing issue #2. Thanks, Lucy >-----Original Message----- >From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of Lu, >Yingqi >Sent: Tuesday, December 11, 2018 12:05 PM >To: Chris Hegarty >Cc: Aundhe, Shirish ; nio-dev at openjdk.java.net; >Viswanathan, Sandhya ; Kaczmarek, Eric > >Subject: RE: adding rsockets support into JDK > >Hi Chris, > >On issue #1: >Q: can rwrite return EAGAIN? I have not checked yet. >I checked with the testing app. Yes, it can return EAGAIN when the resource is >not available. In this case, we need to do an rpoll on POLLOUT. I can quickly try >a patch to address it. > >On issue #2: >I discussed if we need to poll on the unfinished accepts with kernel RDMA >developer yesterday. The answer I got was we do not really need to (we can >of course). Reason is that connect side is the initiator of the handshake. To >start a connection (rconnect being called), it sends out a "connection request >message" to the accept side. If the message has not been sent over, raccept >will simply return. If the message has arrived, rdma_accept be called and >"connection response message" will be sent to connection side. The connect >side will read the message and then send "connection ready message" to the >accept side. rconnect only sends the first message, that is why we need a >thread on the connect side to make sure the rest of steps are being taken >care of. > >Thanks, >Lucy > >>-----Original Message----- >>From: Chris Hegarty [mailto:chris.hegarty at oracle.com] >>Sent: Tuesday, December 11, 2018 11:31 AM >>To: Lu, Yingqi >>Cc: Alan Bateman ; nio-dev at openjdk.java.net; >>Viswanathan, Sandhya ; Aundhe, Shirish >>; Kaczmarek, Eric >>Subject: Re: adding rsockets support into JDK >> >>Lucy, >> >>Sure, the small test scenarios can be modified to make them "work". The >>bigger question is how the proposed JDK-RDMA implementation code can >be >>modified to provide the semantics of the `SocketChannel` API. >> >>Issue #1: rread returns EAGAIN. One possible solution could be that the >>blocking code path in RdmaSocketChannelImpl could fallback back into a >>blocking rpoll POLLOUT if the IOUtil.read method invocation returns >>IOStatus.UNAVAILABLE. I think this should work, and not have too much >>of a negative impact since the fallback will only occur infrequently. >> Q: can rwrite return EAGAIN? I have not checked yet. >> >>Issue #2: This issue is likely to be encountered mainly during testing, >>since a non-blocking connect followed by an accept, on the same thread, >>is not all the common in non-test code. That said, the semantics of the >>SocketChannel API would lead one to expect it to work. ( I get that >>rsocket is not asynchronous, but the semantics of non-blocking channels >>implies some asynchronousity ). I wonder if the JDK-RDMA >>implementation should have a dedicated thread that "pulls" on >>unfinished non-blocking connects that are not subsequently registered with >a Selector? Maybe accepts too? I'm not sure yet. >> >>-Chris. >> >>> On 11 Dec 2018, at 00:36, Lu, Yingqi wrote: >>> >>> Hi Alan/Chris, >>> >>> I was able to confirm that connecting on non-blocking socket causes issues. >>It happens when connect/accept occurs in the same thread or different >>threads in the same process. >>> >>> Then, I did a small tweak in Chris's sample application by spawning a >>> thread >>doing rpoll on the connection_fd. Now, the connect/accept works in both >>of the cases above. Please let me know if this is a valid workaround for the >issue. >>> >>> Performance wise, this workaround should not impact send/receive at >>> all. It >>might only add a small overhead to the connection setup phase only with >>non- blocking RDMA socket. >>> >>> The modified app code is available at >>> >>> For connect/accept occur in the same thread: >>> https://cr.openjdk.java.net/~ylu/testNonBlocking_raccept_modified.c >>> >>> For connect/accept occur in two different threads: >>> https://cr.openjdk.java.net/~ylu/testNonBlocking_raccept_modified_2th >>> r >>> eads.c >>> >>> Thanks, >>> Lucy >>> >>>> -----Original Message----- >>>> From: Alan Bateman [mailto:Alan.Bateman at oracle.com] >>>> Sent: Saturday, December 8, 2018 8:10 AM >>>> To: Chris Hegarty ; Lu, Yingqi >>>> ; nio-dev at openjdk.java.net >>>> Cc: Viswanathan, Sandhya ; Aundhe, >>>> Shirish ; Kaczmarek, Eric >>>> >>>> Subject: Re: adding rsockets support into JDK >>>> >>>> On 08/12/2018 09:39, Chris Hegarty wrote: >>>>> : >>>>> >>>>> - It has become apparent that mixing blocking and non-blocking >>>>> connect/accept operations, in the same thread, may cause issues. For >>>>> example, attempting to setup a connected-socket on the same host by >>>>> issuing a non-blocking connect followed by a blocking accept, will >>>>> just hang and not make progress [3]. Upon further enquiries it >>>>> appears >>>>> that the programming model for rsocket is a subtly different than >>>>> that >>>>> of the regular Berkeley sockets ( at least for the connection >>>>> handshake ). It is not immediately clear how to reasonably >>>>> workaround >>>>> this issue ( it's not a bug in rdma-core, but more a fundamental >>>>> part >>>>> of its thread-less operation ). >>>>> >>>> Would it be possible to expand on this to say whether the same >>>> issues arises when the non-blocking connect is initiated on a >>>> different thread, or in a different process, or even a different machine on >the fabric. >>>> That is, if the socket is non-blocking and I do a rconnect and then >>>> delay before doing anything else on the socket then will the peer >>>> doing accept be blocked/hung in the mean-time? >>>> >>>> -Alan From chris.hegarty at oracle.com Thu Dec 13 19:56:52 2018 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 13 Dec 2018 19:56:52 +0000 Subject: adding rsockets support into JDK In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD15BB6D5@ORSMSX113.amr.corp.intel.com> References: <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> <8C5ED530-8C5B-4A99-8471-B43B82B8E5E3@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159AE8E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159B83F@ORSMSX113.amr.corp.intel.com> <5C96DD75-48C1-474F-9A91-5671256353F1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B5E36@ORSMSX113.amr.corp.intel.com> <3B5379DE-5D47-4B50-8DFC-6B4CEF93A977@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B7562@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15BB6D5@ORSMSX113.amr.corp.intel.com> Message-ID: Lucy, I will take a look at this latest version. In the meantime, I?ve put together a test that exercises a number of connecting scenarios exercising both blocking and non-blocking accept, connect, and I/O operations. Twenty four scenarios in total. The test can be configured to either use either RDMA sockets or regular TCP sockets ( there is a static final field useRDMA ). The test passes reliably using TCP sockets, not so with RDMA ( I have yet to test webrev 25, but I will do it ). http://cr.openjdk.java.net/~chegar/rsocket/IOExchanges/ The test uses testng as it provides a number of useful mechanisms that allow the writing of reasonably self contained and compact code. The test scenarios are described within, and while there is a small amount of duplication, it is reasonable to maintain readability. -Chris. > On 13 Dec 2018, at 17:59, Lu, Yingqi wrote: > > Hi Chris/Alan, > > Here is the version 25 of the patch: http://cr.openjdk.java.net/~ylu/8195160.25/ > > In this version, I have modified following items: > > 1. I applied Javadoc wording change suggestions from Brian http://cr.openjdk.java.net/~bpb/8195160/webrev-22-delta/ > > 2. I applied the suggested changes from Chris > http://cr.openjdk.java.net/~chegar/rsocket/webrev.23.1/ > https://cr.openjdk.java.net/~chegar/rsocket/webrev.23.2/ > http://cr.openjdk.java.net/~chegar/rsocket/webrev.23.3/ > https://cr.openjdk.java.net/~chegar/rsocket/webrev.24.1/ > > 3. I made changes to address the issue #1 (non-blocking connect/blocking accept). Instead of making changes inside RdmaSocketChannelImpl.java, I think it might be easier to directly change the native implementation at src/jdk.net/linux/native/libextnet/LinuxRdmaSocketDispatcherImpl.c > > Please let me know your feedback. At the same time, I am working on addressing issue #2. > > Thanks, > Lucy > >> -----Original Message----- >> From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of Lu, >> Yingqi >> Sent: Tuesday, December 11, 2018 12:05 PM >> To: Chris Hegarty >> Cc: Aundhe, Shirish ; nio-dev at openjdk.java.net; >> Viswanathan, Sandhya ; Kaczmarek, Eric >> >> Subject: RE: adding rsockets support into JDK >> >> Hi Chris, >> >> On issue #1: >> Q: can rwrite return EAGAIN? I have not checked yet. >> I checked with the testing app. Yes, it can return EAGAIN when the resource is >> not available. In this case, we need to do an rpoll on POLLOUT. I can quickly try >> a patch to address it. >> >> On issue #2: >> I discussed if we need to poll on the unfinished accepts with kernel RDMA >> developer yesterday. The answer I got was we do not really need to (we can >> of course). Reason is that connect side is the initiator of the handshake. To >> start a connection (rconnect being called), it sends out a "connection request >> message" to the accept side. If the message has not been sent over, raccept >> will simply return. If the message has arrived, rdma_accept be called and >> "connection response message" will be sent to connection side. The connect >> side will read the message and then send "connection ready message" to the >> accept side. rconnect only sends the first message, that is why we need a >> thread on the connect side to make sure the rest of steps are being taken >> care of. >> >> Thanks, >> Lucy >> >>> -----Original Message----- >>> From: Chris Hegarty [mailto:chris.hegarty at oracle.com] >>> Sent: Tuesday, December 11, 2018 11:31 AM >>> To: Lu, Yingqi >>> Cc: Alan Bateman ; nio-dev at openjdk.java.net; >>> Viswanathan, Sandhya ; Aundhe, Shirish >>> ; Kaczmarek, Eric >>> Subject: Re: adding rsockets support into JDK >>> >>> Lucy, >>> >>> Sure, the small test scenarios can be modified to make them "work". The >>> bigger question is how the proposed JDK-RDMA implementation code can >> be >>> modified to provide the semantics of the `SocketChannel` API. >>> >>> Issue #1: rread returns EAGAIN. One possible solution could be that the >>> blocking code path in RdmaSocketChannelImpl could fallback back into a >>> blocking rpoll POLLOUT if the IOUtil.read method invocation returns >>> IOStatus.UNAVAILABLE. I think this should work, and not have too much >>> of a negative impact since the fallback will only occur infrequently. >>> Q: can rwrite return EAGAIN? I have not checked yet. >>> >>> Issue #2: This issue is likely to be encountered mainly during testing, >>> since a non-blocking connect followed by an accept, on the same thread, >>> is not all the common in non-test code. That said, the semantics of the >>> SocketChannel API would lead one to expect it to work. ( I get that >>> rsocket is not asynchronous, but the semantics of non-blocking channels >>> implies some asynchronousity ). I wonder if the JDK-RDMA >>> implementation should have a dedicated thread that "pulls" on >>> unfinished non-blocking connects that are not subsequently registered with >> a Selector? Maybe accepts too? I'm not sure yet. >>> >>> -Chris. >>> >>>> On 11 Dec 2018, at 00:36, Lu, Yingqi wrote: >>>> >>>> Hi Alan/Chris, >>>> >>>> I was able to confirm that connecting on non-blocking socket causes issues. >>> It happens when connect/accept occurs in the same thread or different >>> threads in the same process. >>>> >>>> Then, I did a small tweak in Chris's sample application by spawning a >>>> thread >>> doing rpoll on the connection_fd. Now, the connect/accept works in both >>> of the cases above. Please let me know if this is a valid workaround for the >> issue. >>>> >>>> Performance wise, this workaround should not impact send/receive at >>>> all. It >>> might only add a small overhead to the connection setup phase only with >>> non- blocking RDMA socket. >>>> >>>> The modified app code is available at >>>> >>>> For connect/accept occur in the same thread: >>>> https://cr.openjdk.java.net/~ylu/testNonBlocking_raccept_modified.c >>>> >>>> For connect/accept occur in two different threads: >>>> https://cr.openjdk.java.net/~ylu/testNonBlocking_raccept_modified_2th >>>> r >>>> eads.c >>>> >>>> Thanks, >>>> Lucy >>>> >>>>> -----Original Message----- >>>>> From: Alan Bateman [mailto:Alan.Bateman at oracle.com] >>>>> Sent: Saturday, December 8, 2018 8:10 AM >>>>> To: Chris Hegarty ; Lu, Yingqi >>>>> ; nio-dev at openjdk.java.net >>>>> Cc: Viswanathan, Sandhya ; Aundhe, >>>>> Shirish ; Kaczmarek, Eric >>>>> >>>>> Subject: Re: adding rsockets support into JDK >>>>> >>>>> On 08/12/2018 09:39, Chris Hegarty wrote: >>>>>> : >>>>>> >>>>>> - It has become apparent that mixing blocking and non-blocking >>>>>> connect/accept operations, in the same thread, may cause issues. For >>>>>> example, attempting to setup a connected-socket on the same host by >>>>>> issuing a non-blocking connect followed by a blocking accept, will >>>>>> just hang and not make progress [3]. Upon further enquiries it >>>>>> appears >>>>>> that the programming model for rsocket is a subtly different than >>>>>> that >>>>>> of the regular Berkeley sockets ( at least for the connection >>>>>> handshake ). It is not immediately clear how to reasonably >>>>>> workaround >>>>>> this issue ( it's not a bug in rdma-core, but more a fundamental >>>>>> part >>>>>> of its thread-less operation ). >>>>>> >>>>> Would it be possible to expand on this to say whether the same >>>>> issues arises when the non-blocking connect is initiated on a >>>>> different thread, or in a different process, or even a different machine on >> the fabric. >>>>> That is, if the socket is non-blocking and I do a rconnect and then >>>>> delay before doing anything else on the socket then will the peer >>>>> doing accept be blocked/hung in the mean-time? >>>>> >>>>> -Alan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Thu Dec 13 20:02:47 2018 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Thu, 13 Dec 2018 20:02:47 +0000 Subject: adding rsockets support into JDK In-Reply-To: References: <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> <8C5ED530-8C5B-4A99-8471-B43B82B8E5E3@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159AE8E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159B83F@ORSMSX113.amr.corp.intel.com> <5C96DD75-48C1-474F-9A91-5671256353F1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B5E36@ORSMSX113.amr.corp.intel.com> <3B5379DE-5D47-4B50-8DFC-6B4CEF93A977@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B7562@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15BB6D5@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD15BBB7D@ORSMSX113.amr.corp.intel.com> Chris, I will take a look at the new test and also try it with version 25 as well. Thanks, Lucy From: Chris Hegarty [mailto:chris.hegarty at oracle.com] Sent: Thursday, December 13, 2018 11:57 AM To: Lu, Yingqi Cc: Aundhe, Shirish ; nio-dev at openjdk.java.net; Viswanathan, Sandhya ; Kaczmarek, Eric Subject: Re: adding rsockets support into JDK Lucy, I will take a look at this latest version. In the meantime, I?ve put together a test that exercises a number of connecting scenarios exercising both blocking and non-blocking accept, connect, and I/O operations. Twenty four scenarios in total. The test can be configured to either use either RDMA sockets or regular TCP sockets ( there is a static final field useRDMA ). The test passes reliably using TCP sockets, not so with RDMA ( I have yet to test webrev 25, but I will do it ). http://cr.openjdk.java.net/~chegar/rsocket/IOExchanges/ The test uses testng as it provides a number of useful mechanisms that allow the writing of reasonably self contained and compact code. The test scenarios are described within, and while there is a small amount of duplication, it is reasonable to maintain readability. -Chris. On 13 Dec 2018, at 17:59, Lu, Yingqi > wrote: Hi Chris/Alan, Here is the version 25 of the patch: http://cr.openjdk.java.net/~ylu/8195160.25/ In this version, I have modified following items: 1. I applied Javadoc wording change suggestions from Brian http://cr.openjdk.java.net/~bpb/8195160/webrev-22-delta/ 2. I applied the suggested changes from Chris http://cr.openjdk.java.net/~chegar/rsocket/webrev.23.1/ https://cr.openjdk.java.net/~chegar/rsocket/webrev.23.2/ http://cr.openjdk.java.net/~chegar/rsocket/webrev.23.3/ https://cr.openjdk.java.net/~chegar/rsocket/webrev.24.1/ 3. I made changes to address the issue #1 (non-blocking connect/blocking accept). Instead of making changes inside RdmaSocketChannelImpl.java, I think it might be easier to directly change the native implementation at src/jdk.net/linux/native/libextnet/LinuxRdmaSocketDispatcherImpl.c Please let me know your feedback. At the same time, I am working on addressing issue #2. Thanks, Lucy -----Original Message----- From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of Lu, Yingqi Sent: Tuesday, December 11, 2018 12:05 PM To: Chris Hegarty > Cc: Aundhe, Shirish >; nio-dev at openjdk.java.net; Viswanathan, Sandhya >; Kaczmarek, Eric > Subject: RE: adding rsockets support into JDK Hi Chris, On issue #1: Q: can rwrite return EAGAIN? I have not checked yet. I checked with the testing app. Yes, it can return EAGAIN when the resource is not available. In this case, we need to do an rpoll on POLLOUT. I can quickly try a patch to address it. On issue #2: I discussed if we need to poll on the unfinished accepts with kernel RDMA developer yesterday. The answer I got was we do not really need to (we can of course). Reason is that connect side is the initiator of the handshake. To start a connection (rconnect being called), it sends out a "connection request message" to the accept side. If the message has not been sent over, raccept will simply return. If the message has arrived, rdma_accept be called and "connection response message" will be sent to connection side. The connect side will read the message and then send "connection ready message" to the accept side. rconnect only sends the first message, that is why we need a thread on the connect side to make sure the rest of steps are being taken care of. Thanks, Lucy -----Original Message----- From: Chris Hegarty [mailto:chris.hegarty at oracle.com] Sent: Tuesday, December 11, 2018 11:31 AM To: Lu, Yingqi > Cc: Alan Bateman >; nio-dev at openjdk.java.net; Viswanathan, Sandhya >; Aundhe, Shirish >; Kaczmarek, Eric > Subject: Re: adding rsockets support into JDK Lucy, Sure, the small test scenarios can be modified to make them "work". The bigger question is how the proposed JDK-RDMA implementation code can be modified to provide the semantics of the `SocketChannel` API. Issue #1: rread returns EAGAIN. One possible solution could be that the blocking code path in RdmaSocketChannelImpl could fallback back into a blocking rpoll POLLOUT if the IOUtil.read method invocation returns IOStatus.UNAVAILABLE. I think this should work, and not have too much of a negative impact since the fallback will only occur infrequently. Q: can rwrite return EAGAIN? I have not checked yet. Issue #2: This issue is likely to be encountered mainly during testing, since a non-blocking connect followed by an accept, on the same thread, is not all the common in non-test code. That said, the semantics of the SocketChannel API would lead one to expect it to work. ( I get that rsocket is not asynchronous, but the semantics of non-blocking channels implies some asynchronousity ). I wonder if the JDK-RDMA implementation should have a dedicated thread that "pulls" on unfinished non-blocking connects that are not subsequently registered with a Selector? Maybe accepts too? I'm not sure yet. -Chris. On 11 Dec 2018, at 00:36, Lu, Yingqi > wrote: Hi Alan/Chris, I was able to confirm that connecting on non-blocking socket causes issues. It happens when connect/accept occurs in the same thread or different threads in the same process. Then, I did a small tweak in Chris's sample application by spawning a thread doing rpoll on the connection_fd. Now, the connect/accept works in both of the cases above. Please let me know if this is a valid workaround for the issue. Performance wise, this workaround should not impact send/receive at all. It might only add a small overhead to the connection setup phase only with non- blocking RDMA socket. The modified app code is available at For connect/accept occur in the same thread: https://cr.openjdk.java.net/~ylu/testNonBlocking_raccept_modified.c For connect/accept occur in two different threads: https://cr.openjdk.java.net/~ylu/testNonBlocking_raccept_modified_2th r eads.c Thanks, Lucy -----Original Message----- From: Alan Bateman [mailto:Alan.Bateman at oracle.com] Sent: Saturday, December 8, 2018 8:10 AM To: Chris Hegarty >; Lu, Yingqi >; nio-dev at openjdk.java.net Cc: Viswanathan, Sandhya >; Aundhe, Shirish >; Kaczmarek, Eric > Subject: Re: adding rsockets support into JDK On 08/12/2018 09:39, Chris Hegarty wrote: : - It has become apparent that mixing blocking and non-blocking connect/accept operations, in the same thread, may cause issues. For example, attempting to setup a connected-socket on the same host by issuing a non-blocking connect followed by a blocking accept, will just hang and not make progress [3]. Upon further enquiries it appears that the programming model for rsocket is a subtly different than that of the regular Berkeley sockets ( at least for the connection handshake ). It is not immediately clear how to reasonably workaround this issue ( it's not a bug in rdma-core, but more a fundamental part of its thread-less operation ). Would it be possible to expand on this to say whether the same issues arises when the non-blocking connect is initiated on a different thread, or in a different process, or even a different machine on the fabric. That is, if the socket is non-blocking and I do a rconnect and then delay before doing anything else on the socket then will the peer doing accept be blocked/hung in the mean-time? -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.hegarty at oracle.com Fri Dec 14 17:47:05 2018 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Fri, 14 Dec 2018 17:47:05 +0000 Subject: adding rsockets support into JDK In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD15BB6D5@ORSMSX113.amr.corp.intel.com> References: <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> <8C5ED530-8C5B-4A99-8471-B43B82B8E5E3@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159AE8E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159B83F@ORSMSX113.amr.corp.intel.com> <5C96DD75-48C1-474F-9A91-5671256353F1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B5E36@ORSMSX113.amr.corp.intel.com> <3B5379DE-5D47-4B50-8DFC-6B4CEF93A977@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B7562@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15BB6D5@ORSMSX113.amr.corp.intel.com> Message-ID: <6C2EEE6F-B9D4-478D-8CE7-E471773943F1@oracle.com> Lucy, > On 13 Dec 2018, at 17:59, Lu, Yingqi wrote: > > Hi Chris/Alan, > > Here is the version 25 of the patch: http://cr.openjdk.java.net/~ylu/8195160.25/ Some further comments. 1) test/jdk/jdk/net/RdmaSockets/rsocket/Selector/Connect.java This test unnecessarily creates a thread for each connection. http://cr.openjdk.java.net/~chegar/rsocket/webrev.25.1/ 2) Additional test coverage for basic switching of blocking state. http://cr.openjdk.java.net/~chegar/rsocket/webrev.25.2/ 3) I think your work-around for Issue #1 is probably ok. I refined it a little so avoid accessing the flags unless necessary. I think this may be slightly more efficient, since the typical non-blocking usage will be preceded by a select/rpoll. src/jdk.net/linux/native/libextnet/LinuxRdmaSocketDispatcherImpl.c http://cr.openjdk.java.net/~chegar/rsocket/webrev.25.3/ 4) LinuxRdmaSocketDispatcherImpl.java While reading the code I noticed an odd/stray unused `close` method. I think it should be removed to avoid confusion ( it confused me ). http://cr.openjdk.java.net/~chegar/rsocket/webrev.25.4/ 5) test/jdk/jdk/net/RdmaSockets/rsocket/SocketChannel/IOExchanges.java I updated this test to ensure that all selectors are now closed ( they were not all closed consistently in the previous version ). http://cr.openjdk.java.net/~chegar/rsocket/webrev.25.5/ What is nice about this test is that the *_BConn_* method verify the changes in 3 above ( for Issue #1 ). I still wanna add scatter/gather variants ( its on my TODO list ). For now, until Issue #2 is resolved, then just comment out the `@Test` from the twelve *_NBConn_* methods ( they currently fail, since they verify the yet-to-be-fixed Issue #2 ). It is on my list to look into fix this issue too. 6) test/jdk/jdk/net/RdmaSockets/rsocket/SocketChannel/CloseDuringWrite.java Just some cleanup in this test. http://cr.openjdk.java.net/~chegar/rsocket/webrev.25.6/ The reason I'm doing some of this cleanup is that I see infrequent failures in a few of these tests ( about 1 failure in ever 30 or so runs ). That's my patch queue empty, for now. I hope that applying these patches is not too onerous. They are mostly review comments, and I hope to be getting to the end of this soon. -Chris. From yingqi.lu at intel.com Fri Dec 14 17:58:03 2018 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Fri, 14 Dec 2018 17:58:03 +0000 Subject: adding rsockets support into JDK In-Reply-To: <6C2EEE6F-B9D4-478D-8CE7-E471773943F1@oracle.com> References: <3F3A331C-861C-4289-A9ED-666AD7793341@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594318@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15945D6@ORSMSX113.amr.corp.intel.com> <1644588D-69D5-4567-97F9-59B8F07B8339@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1594787@ORSMSX113.amr.corp.intel.com> <47f6dbae-9db2-4489-f5ec-d614d5c8ec3c@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> <8C5ED530-8C5B-4A99-8471-B43B82B8E5E3@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159AE8E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159B83F@ORSMSX113.amr.corp.intel.com> <5C96DD75-48C1-474F-9A91-5671256353F1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B5E36@ORSMSX113.amr.corp.intel.com> <3B5379DE-5D47-4B50-8DFC-6B4CEF93A977@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B7562@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15BB6D5@ORSMSX113.amr.corp.intel.com> <6C2EEE6F-B9D4-478D-8CE7-E471773943F1@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD15BCC8F@ORSMSX113.amr.corp.intel.com> Hi Chris, Thank you so much for the help on this work. Really appreciate!! Just to let you know. I also tried your IOExchanges.java test on webrev.25. I commented out the test 7-12 and 7a-12a as they relate to issue #2. I ran the test for 10000 loops and I do not see any failures or errors. I will apply all your patches to my local repository now. I remember there is one more open item from your comment earlier in addition to issue #2 -- the null binding issue. Details are listed below. To resolve this issue, we might need to change some JDK public APIs either at java.net.Socket or java.netInetXAddress. I am looking forward to your guidance here as well :-) ---------------------------------------------------------------------------------------------- 3) Handling of null binding, automatically assigned address. Handling of null binding is a bit tricky now that we have protocol family support. The correct InetXAddress "any local address" needs to be created. I added a test and a little refactoring. If you run the test without any source changes you will be able to see the issues. https://cr.openjdk.java.net/~chegar/rsocket/webrev.23.3/ Note: There is still an open issue, ToDo, with java.net.Socket. You can see this by a small comment in the test. I'll revisit this later. --------------------------------------------------------------------------------------------- Thanks, Lucy >-----Original Message----- >From: Chris Hegarty [mailto:chris.hegarty at oracle.com] >Sent: Friday, December 14, 2018 9:47 AM >To: Lu, Yingqi >Cc: Aundhe, Shirish ; nio-dev at openjdk.java.net; >Viswanathan, Sandhya ; Kaczmarek, Eric > >Subject: Re: adding rsockets support into JDK > >Lucy, > >> On 13 Dec 2018, at 17:59, Lu, Yingqi wrote: >> >> Hi Chris/Alan, >> >> Here is the version 25 of the patch: >> http://cr.openjdk.java.net/~ylu/8195160.25/ > >Some further comments. > >1) test/jdk/jdk/net/RdmaSockets/rsocket/Selector/Connect.java > > This test unnecessarily creates a thread for each connection. > http://cr.openjdk.java.net/~chegar/rsocket/webrev.25.1/ > >2) Additional test coverage for basic switching of blocking state. > > http://cr.openjdk.java.net/~chegar/rsocket/webrev.25.2/ > >3) I think your work-around for Issue #1 is probably ok. I refined it a > little so avoid accessing the flags unless necessary. I think this may > be slightly more efficient, since the typical non-blocking usage will > be preceded by a select/rpoll. > > src/jdk.net/linux/native/libextnet/LinuxRdmaSocketDispatcherImpl.c > http://cr.openjdk.java.net/~chegar/rsocket/webrev.25.3/ > >4) LinuxRdmaSocketDispatcherImpl.java > > While reading the code I noticed an odd/stray unused `close` method. I > think it should be removed to avoid confusion ( it confused me ). > > http://cr.openjdk.java.net/~chegar/rsocket/webrev.25.4/ > >5) test/jdk/jdk/net/RdmaSockets/rsocket/SocketChannel/IOExchanges.java > > I updated this test to ensure that all selectors are now closed ( they > were not all closed consistently in the previous version ). > > http://cr.openjdk.java.net/~chegar/rsocket/webrev.25.5/ > > What is nice about this test is that the *_BConn_* method verify the > changes in 3 above ( for Issue #1 ). I still wanna add scatter/gather > variants ( its on my TODO list ). > > For now, until Issue #2 is resolved, then just comment out the `@Test` > from the twelve *_NBConn_* methods ( they currently fail, since they > verify the yet-to-be-fixed Issue #2 ). It is on my list to look into > fix this issue too. > >6) >test/jdk/jdk/net/RdmaSockets/rsocket/SocketChannel/CloseDuringWrite.jav >a > > Just some cleanup in this test. > http://cr.openjdk.java.net/~chegar/rsocket/webrev.25.6/ > > The reason I'm doing some of this cleanup is that I see infrequent > failures in a few of these tests ( about 1 failure in ever 30 or so > runs ). > > >That's my patch queue empty, for now. I hope that applying these patches is >not too onerous. They are mostly review comments, and I hope to be getting >to the end of this soon. > >-Chris. From Alan.Bateman at oracle.com Fri Dec 14 18:26:27 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 14 Dec 2018 18:26:27 +0000 Subject: adding rsockets support into JDK In-Reply-To: <6C2EEE6F-B9D4-478D-8CE7-E471773943F1@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> <8C5ED530-8C5B-4A99-8471-B43B82B8E5E3@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159AE8E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159B83F@ORSMSX113.amr.corp.intel.com> <5C96DD75-48C1-474F-9A91-5671256353F1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B5E36@ORSMSX113.amr.corp.intel.com> <3B5379DE-5D47-4B50-8DFC-6B4CEF93A977@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B7562@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15BB6D5@ORSMSX113.amr.corp.intel.com> <6C2EEE6F-B9D4-478D-8CE7-E471773943F1@oracle.com> Message-ID: On 14/12/2018 17:47, Chris Hegarty wrote: > : > 3) I think your work-around for Issue #1 is probably ok. I refined it a > little so avoid accessing the flags unless necessary. I think this may > be slightly more efficient, since the typical non-blocking usage will > be preceded by a select/rpoll. > > src/jdk.net/linux/native/libextnet/LinuxRdmaSocketDispatcherImpl.c > http://cr.openjdk.java.net/~chegar/rsocket/webrev.25.3/ > I don't think we should have the workaround at this level. The native methods should be just simple wrappers on rs_read/rs_write. If there is no hope of getting the RDMA library fixed then we should configure the rsocket to be non-blocking at create and emulate blocking in the RdmaSocketChannelImpl. -Alan. From christoph.langer at sap.com Mon Dec 17 08:28:43 2018 From: christoph.langer at sap.com (Langer, Christoph) Date: Mon, 17 Dec 2018 08:28:43 +0000 Subject: RFR (S): 8215472: Cleanups in implementation classes of jdk.zipfs and tests Message-ID: Hi, when working with jdk.zipfs, I found some opportunity for cleanups. I'd like to contribute this independently from my other thread regarding Posix permissions in zip files (http://mail.openjdk.java.net/pipermail/core-libs-dev/2018-November/056426.html). So, please help to review the cleanup. Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8215472.0/ Bug: https://bugs.openjdk.java.net/browse/JDK-8215472 Thanks Christoph -------------- next part -------------- An HTML attachment was scrubbed... URL: From claes.redestad at oracle.com Mon Dec 17 10:38:07 2018 From: claes.redestad at oracle.com (Claes Redestad) Date: Mon, 17 Dec 2018 11:38:07 +0100 Subject: RFR (S): 8215472: Cleanups in implementation classes of jdk.zipfs and tests In-Reply-To: References: Message-ID: Hi Christoph, I believe the convention in the OpenJDK sources is to sort static imports after non-static, so changing to the other way around in a few places adds inconsistencies. In ZipFileSystem you remove the unused method releaseDeflater - to me this indicates the resource pooling is actually broken? I.e., shouldn't EntryOutputStreamDef return its Deflater to the cache when it's closed, similar to how the anonymous InflaterInputStream in getInputStream does it? As it's currently implemented the deflaters list will always be empty and no Deflater returned, so could go the other way and remove that cache if caching Deflaters isn't useful. Otherwise looks good to me. /Claes On 2018-12-17 09:28, Langer, Christoph wrote: > Hi, > > when working with jdk.zipfs, I found some opportunity for cleanups. I'd like to contribute this independently from my other thread regarding Posix permissions in zip files (http://mail.openjdk.java.net/pipermail/core-libs-dev/2018-November/056426.html). So, please help to review the cleanup. > > Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8215472.0/ > Bug: https://bugs.openjdk.java.net/browse/JDK-8215472 > > Thanks > Christoph > From yingqi.lu at intel.com Mon Dec 17 18:18:46 2018 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Mon, 17 Dec 2018 18:18:46 +0000 Subject: adding rsockets support into JDK In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AD15973EE@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598B1E@ORSMSX113.amr.corp.intel.com> <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> <8C5ED530-8C5B-4A99-8471-B43B82B8E5E3@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159AE8E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159B83F@ORSMSX113.amr.corp.intel.com> <5C96DD75-48C1-474F-9A91-5671256353F1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B5E36@ORSMSX113.amr.corp.intel.com> <3B5379DE-5D47-4B50-8DFC-6B4CEF93A977@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B7562@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15BB6D5@ORSMSX113.amr.corp.intel.com> <6C2EEE6F-B9D4-478D-8CE7-E471773943F1@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD15C0F33@ORSMSX113.amr.corp.intel.com> Hi Alan, Thank you for your suggestion. I just want to confirm that you meant all the RDMA based socket channels and server socket channels will be created as non-blocking initially? Later, if application sets them to be blocking, we then do the emulation. If this is the case, do we need to specify it in the Javadoc since the tcp based channels are created to be blocking? Thanks, Lucy >-----Original Message----- >From: Alan Bateman [mailto:Alan.Bateman at oracle.com] >Sent: Friday, December 14, 2018 10:26 AM >To: Chris Hegarty ; Lu, Yingqi > >Cc: nio-dev at openjdk.java.net; Viswanathan, Sandhya >; Aundhe, Shirish >; Kaczmarek, Eric >Subject: Re: adding rsockets support into JDK > >On 14/12/2018 17:47, Chris Hegarty wrote: >> : >> 3) I think your work-around for Issue #1 is probably ok. I refined it a >> little so avoid accessing the flags unless necessary. I think this may >> be slightly more efficient, since the typical non-blocking usage will >> be preceded by a select/rpoll. >> >> src/jdk.net/linux/native/libextnet/LinuxRdmaSocketDispatcherImpl.c >> http://cr.openjdk.java.net/~chegar/rsocket/webrev.25.3/ >> >I don't think we should have the workaround at this level. The native methods >should be just simple wrappers on rs_read/rs_write. If there is no hope of >getting the RDMA library fixed then we should configure the rsocket to be >non-blocking at create and emulate blocking in the RdmaSocketChannelImpl. > >-Alan. From Alan.Bateman at oracle.com Mon Dec 17 18:54:00 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 17 Dec 2018 18:54:00 +0000 Subject: adding rsockets support into JDK In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD15C0F33@ORSMSX113.amr.corp.intel.com> References: <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> <8C5ED530-8C5B-4A99-8471-B43B82B8E5E3@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159AE8E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159B83F@ORSMSX113.amr.corp.intel.com> <5C96DD75-48C1-474F-9A91-5671256353F1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B5E36@ORSMSX113.amr.corp.intel.com> <3B5379DE-5D47-4B50-8DFC-6B4CEF93A977@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B7562@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15BB6D5@ORSMSX113.amr.corp.intel.com> <6C2EEE6F-B9D4-478D-8CE7-E471773943F1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15C0F33@ORSMSX113.amr.corp.intel.com> Message-ID: <59ecdb38-8d20-3416-602c-47198fac6e41@oracle.com> On 17/12/2018 18:18, Lu, Yingqi wrote: > Hi Alan, > > Thank you for your suggestion. I just want to confirm that you meant all the RDMA based socket channels and server socket channels will be created as non-blocking initially? Later, if application sets them to be blocking, we then do the emulation. > > If this is the case, do we need to specify it in the Javadoc since the tcp based channels are created to be blocking? > This doesn't impact anyone using the API, this is just about emulating blocking behavior with an rsocket that is non-blocking. The rational is to keep the native code as simple as possible so that the native methods are a single syscall rather than many. Whether we need this of course depends on whether the bug in the Linux RDMA library will be fixed or not. Maybe we need to see if Sean can help progress the bug there before spending time on this. -Alan From yingqi.lu at intel.com Mon Dec 17 19:07:54 2018 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Mon, 17 Dec 2018 19:07:54 +0000 Subject: adding rsockets support into JDK In-Reply-To: <59ecdb38-8d20-3416-602c-47198fac6e41@oracle.com> References: <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> <8C5ED530-8C5B-4A99-8471-B43B82B8E5E3@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159AE8E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159B83F@ORSMSX113.amr.corp.intel.com> <5C96DD75-48C1-474F-9A91-5671256353F1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B5E36@ORSMSX113.amr.corp.intel.com> <3B5379DE-5D47-4B50-8DFC-6B4CEF93A977@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B7562@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15BB6D5@ORSMSX113.amr.corp.intel.com> <6C2EEE6F-B9D4-478D-8CE7-E471773943F1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15C0F33@ORSMSX113.amr.corp.intel.com> <59ecdb38-8d20-3416-602c-47198fac6e41@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD15C116B@ORSMSX113.amr.corp.intel.com> I already discussed with Sean on this specific issue. He mentioned that even he can propose a fix to the kernel librdmacm module, it would take years to get into majority of the distros. I would recommend to emulate the behavior inside JDK and make use of it. I will quickly draft a patch and send here for review. Thanks, Lucy >-----Original Message----- >From: Alan Bateman [mailto:Alan.Bateman at oracle.com] >Sent: Monday, December 17, 2018 10:54 AM >To: Lu, Yingqi ; Chris Hegarty > >Cc: nio-dev at openjdk.java.net; Viswanathan, Sandhya >; Aundhe, Shirish >; Kaczmarek, Eric >Subject: Re: adding rsockets support into JDK > >On 17/12/2018 18:18, Lu, Yingqi wrote: >> Hi Alan, >> >> Thank you for your suggestion. I just want to confirm that you meant all the >RDMA based socket channels and server socket channels will be created as >non-blocking initially? Later, if application sets them to be blocking, we then >do the emulation. >> >> If this is the case, do we need to specify it in the Javadoc since the tcp based >channels are created to be blocking? >> >This doesn't impact anyone using the API, this is just about emulating blocking >behavior with an rsocket that is non-blocking. The rational is to keep the >native code as simple as possible so that the native methods are a single >syscall rather than many. Whether we need this of course depends on >whether the bug in the Linux RDMA library will be fixed or not. Maybe we >need to see if Sean can help progress the bug there before spending time on >this. > >-Alan From lance.andersen at oracle.com Mon Dec 17 20:03:27 2018 From: lance.andersen at oracle.com (Lance Andersen) Date: Mon, 17 Dec 2018 15:03:27 -0500 Subject: RFR (S): 8215472: Cleanups in implementation classes of jdk.zipfs and tests In-Reply-To: References: Message-ID: <247E396D-3C1B-4C4C-897D-54A749AEB6D6@oracle.com> HI Christoph > On Dec 17, 2018, at 5:38 AM, Claes Redestad wrote: > > Hi Christoph, > > I believe the convention in the OpenJDK sources is to sort static > imports after non-static, so changing to the other way around in a few > places adds inconsistencies. > > In ZipFileSystem you remove the unused method releaseDeflater - to me > this indicates the resource pooling is actually broken? I.e., shouldn't > EntryOutputStreamDef return its Deflater to the cache when it's closed, > similar to how the anonymous InflaterInputStream in getInputStream does > it? As it's currently implemented the deflaters list will always be > empty and no Deflater returned, so could go the other way and remove > that cache if caching Deflaters isn't useful. Yes I would leave this for now, make a note to look at this further and deal with it separate from this change Also in ZipFileSystem, I would instead of removing line 2115, I would either keep it and remove the other references to version() in writeLOC, or make the change in writeCEN so that the usage of version is consistent. Best Lance > > Otherwise looks good to me. > > /Claes > > On 2018-12-17 09:28, Langer, Christoph wrote: >> Hi, >> when working with jdk.zipfs, I found some opportunity for cleanups. I'd like to contribute this independently from my other thread regarding Posix permissions in zip files (http://mail.openjdk.java.net/pipermail/core-libs-dev/2018-November/056426.html). So, please help to review the cleanup. >> Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8215472.0/ >> Bug: https://bugs.openjdk.java.net/browse/JDK-8215472 >> Thanks >> Christoph Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: oracle_sig_logo.gif Type: image/gif Size: 658 bytes Desc: not available URL: From brian.burkhalter at oracle.com Tue Dec 18 00:47:17 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 17 Dec 2018 16:47:17 -0800 Subject: Adding Path-based constructors to various classes In-Reply-To: References: Message-ID: (looping in nio-dev) Hi Andrew, The NIO APIs (Java 1.4) were intended to supplement the pre-existing Java I/O APIs and this effort was continued in the NIO.2 APIs (Java 7). The Path interface is part of the latter. My impression is that the intent was more to supersede the older APIs than to enhance them to coexist better with the new ones. The addition for example of the constructors you suggest therefore would not be encouraged despite the convenience they might afford in some situations. There are others on these mailing lists however who know the historical context of this area better than I do and who I expect will chime in with a better answer. Thanks, Brian > On Dec 17, 2018, at 4:12 PM, Andrew Luo wrote: > > Many classes such as: > > https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/io/RandomAccessFile.html > https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/zip/ZipFile.html > > have constructors that use the File API or String but no constructor that takes Path. Is there any interest in adding these? The reason I ask this is because we now encourage new code to use Path instead of File, so having to do .toFile() in many places can seems unnecessary. Then again, this is a minor annoyance, but I think it is a useful addition. What do you guys think? -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Dec 18 01:00:28 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 17 Dec 2018 17:00:28 -0800 Subject: Adding Path-based constructors to various classes In-Reply-To: References: Message-ID: <9B8DC831-8468-4782-AA9C-CF4EE1CF0067@oracle.com> Hi Andrew, java.nio.channels.FileChannel pretty much covers RandomAccessFile, I think. The ZipFile case is something else, not really in I/O. Thanks, Brian > On Dec 17, 2018, at 4:56 PM, Andrew Luo wrote: > > Hi Brian, > > Thanks for your thoughts on this. I agree that if a class is superseded by a newer NIO.2 class with Path APIs, we should not add the Path-based APIs to the older class, but I do not believe there are NIO.2 replacements that supersede ZipFile. I also don?t believe that there are any NIO.2 APIs that supersede RandomAccessFile either, but I could be wrong. > > Thanks, > > -Andrew > > From: Brian Burkhalter > > Sent: Monday, December 17, 2018 4:47 PM > To: Andrew Luo >; nio-dev > > Cc: Core-Libs-Dev > > Subject: Re: Adding Path-based constructors to various classes > > (looping in nio-dev) > > Hi Andrew, > > The NIO APIs (Java 1.4) were intended to supplement the pre-existing Java I/O APIs and this effort was continued in the NIO.2 APIs (Java 7). The Path interface is part of the latter. My impression is that the intent was more to supersede the older APIs than to enhance them to coexist better with the new ones. The addition for example of the constructors you suggest therefore would not be encouraged despite the convenience they might afford in some situations. There are others on these mailing lists however who know the historical context of this area better than I do and who I expect will chime in with a better answer. > > Thanks, > > Brian > > > On Dec 17, 2018, at 4:12 PM, Andrew Luo > wrote: > > Many classes such as: > > https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/io/RandomAccessFile.html > https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/zip/ZipFile.html > > have constructors that use the File API or String but no constructor that takes Path. Is there any interest in adding these? The reason I ask this is because we now encourage new code to use Path instead of File, so having to do .toFile() in many places can seems unnecessary. Then again, this is a minor annoyance, but I think it is a useful addition. What do you guys think? -------------- next part -------------- An HTML attachment was scrubbed... URL: From christoph.langer at sap.com Tue Dec 18 13:44:56 2018 From: christoph.langer at sap.com (Langer, Christoph) Date: Tue, 18 Dec 2018 13:44:56 +0000 Subject: RFR (S): 8215472: Cleanups in implementation classes of jdk.zipfs and tests In-Reply-To: <247E396D-3C1B-4C4C-897D-54A749AEB6D6@oracle.com> References: <247E396D-3C1B-4C4C-897D-54A749AEB6D6@oracle.com> Message-ID: <846e3c3c8a3546379c00103f08a1b8a6@sap.com> Hi Claes, Hi Lance, thanks for reviewing my patch. > I believe the convention in the OpenJDK sources is to sort static > imports after non-static, so changing to the other way around in a few > places adds inconsistencies. I was unaware of this but it seems that's the way most files are structured (though you'll find exceptions from that rule). I adapted my change. > In ZipFileSystem you remove the unused method releaseDeflater - to me > this indicates the resource pooling is actually broken? I.e., shouldn't > EntryOutputStreamDef return its Deflater to the cache when it's closed, > similar to how the anonymous InflaterInputStream in getInputStream does > it? As it's currently implemented the deflaters list will always be > empty and no Deflater returned, so could go the other way and remove > that cache if caching Deflaters isn't useful. You are right. I think this is a flaw of the change for 8034802 [1] [2]. I added the call to releaseDeflater() in the close method of EntryOutputStreamDef. > Also in ZipFileSystem, ?I would instead of removing line 2115, I would either > keep it and remove the other references to version() in writeLOC, or make > the change in ?writeCEN so that the usage of version is consistent. Good catch, I've updated this for consistency. This is the new webrev: http://cr.openjdk.java.net/~clanger/webrevs/8215472.1/ Are you good with it? I'll run jtreg tests before pushing... Thanks Christoph [1] http://hg.openjdk.java.net/jdk/jdk/rev/ba51515b64e5 [2] https://bugs.openjdk.java.net/browse/JDK-8034802 From chris.hegarty at oracle.com Tue Dec 18 14:14:33 2018 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 18 Dec 2018 14:14:33 +0000 Subject: adding rsockets support into JDK In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD15C116B@ORSMSX113.amr.corp.intel.com> References: <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> <8C5ED530-8C5B-4A99-8471-B43B82B8E5E3@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159AE8E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159B83F@ORSMSX113.amr.corp.intel.com> <5C96DD75-48C1-474F-9A91-5671256353F1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B5E36@ORSMSX113.amr.corp.intel.com> <3B5379DE-5D47-4B50-8DFC-6B4CEF93A977@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B7562@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15BB6D5@ORSMSX113.amr.corp.intel.com> <6C2EEE6F-B9D4-478D-8CE7-E471773943F1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15C0F33@ORSMSX113.amr.corp.intel.com> <59ecdb38-8d20-3416-602c-47198fac6e41@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15C116B@ORSMSX113.amr.corp.intel.com> Message-ID: <503229EF-747A-4783-B985-0DB0D647194E@oracle.com> > On 17 Dec 2018, at 19:07, Lu, Yingqi wrote: > > I already discussed with Sean on this specific issue. He mentioned that even he can propose a fix to the kernel librdmacm module, it would take years to get into majority of the distros. I would recommend to emulate the behavior inside JDK and make use of it. Sure, but it would be good to have some forward movement on this, so that the Java implementation can remove the workaround at some future point. -Chris. From jonathan.gibbons at oracle.com Tue Dec 18 15:17:47 2018 From: jonathan.gibbons at oracle.com (Jonathan Gibbons) Date: Tue, 18 Dec 2018 07:17:47 -0800 Subject: Adding Path-based constructors to various classes In-Reply-To: <9B8DC831-8468-4782-AA9C-CF4EE1CF0067@oracle.com> References: <9B8DC831-8468-4782-AA9C-CF4EE1CF0067@oracle.com> Message-ID: <61b98500-a7e4-b9b4-b1e7-105aef6e91cc@oracle.com> Zip files are now accessible through the zipfs NIO FileSystem provider. -- Jon On 12/17/18 5:00 PM, Brian Burkhalter wrote: > Hi Andrew, > > java.nio.channels.FileChannel pretty much covers RandomAccessFile, I think. The ZipFile case is something else, not really in I/O. > > Thanks, > > Brian > >> On Dec 17, 2018, at 4:56 PM, Andrew Luo wrote: >> >> Hi Brian, >> >> Thanks for your thoughts on this. I agree that if a class is superseded by a newer NIO.2 class with Path APIs, we should not add the Path-based APIs to the older class, but I do not believe there are NIO.2 replacements that supersede ZipFile. I also don?t believe that there are any NIO.2 APIs that supersede RandomAccessFile either, but I could be wrong. >> >> Thanks, >> >> -Andrew >> >> From: Brian Burkhalter > >> Sent: Monday, December 17, 2018 4:47 PM >> To: Andrew Luo >; nio-dev > >> Cc: Core-Libs-Dev > >> Subject: Re: Adding Path-based constructors to various classes >> >> (looping in nio-dev) >> >> Hi Andrew, >> >> The NIO APIs (Java 1.4) were intended to supplement the pre-existing Java I/O APIs and this effort was continued in the NIO.2 APIs (Java 7). The Path interface is part of the latter. My impression is that the intent was more to supersede the older APIs than to enhance them to coexist better with the new ones. The addition for example of the constructors you suggest therefore would not be encouraged despite the convenience they might afford in some situations. There are others on these mailing lists however who know the historical context of this area better than I do and who I expect will chime in with a better answer. >> >> Thanks, >> >> Brian >> >> >> On Dec 17, 2018, at 4:12 PM, Andrew Luo > wrote: >> >> Many classes such as: >> >> https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/io/RandomAccessFile.html >> https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/zip/ZipFile.html >> >> have constructors that use the File API or String but no constructor that takes Path. Is there any interest in adding these? The reason I ask this is because we now encourage new code to use Path instead of File, so having to do .toFile() in many places can seems unnecessary. Then again, this is a minor annoyance, but I think it is a useful addition. What do you guys think? From yingqi.lu at intel.com Tue Dec 18 19:44:13 2018 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Tue, 18 Dec 2018 19:44:13 +0000 Subject: adding rsockets support into JDK In-Reply-To: <503229EF-747A-4783-B985-0DB0D647194E@oracle.com> References: <768888A4-FCED-4FB2-AF1B-38E2D99F4814@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1598DFF@ORSMSX113.amr.corp.intel.com> <8C5ED530-8C5B-4A99-8471-B43B82B8E5E3@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159AE8E@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD159B83F@ORSMSX113.amr.corp.intel.com> <5C96DD75-48C1-474F-9A91-5671256353F1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B5E36@ORSMSX113.amr.corp.intel.com> <3B5379DE-5D47-4B50-8DFC-6B4CEF93A977@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15B7562@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15BB6D5@ORSMSX113.amr.corp.intel.com> <6C2EEE6F-B9D4-478D-8CE7-E471773943F1@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15C0F33@ORSMSX113.amr.corp.intel.com> <59ecdb38-8d20-3416-602c-47198fac6e41@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD15C116B@ORSMSX113.amr.corp.intel.com> <503229EF-747A-4783-B985-0DB0D647194E@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD15C2B69@ORSMSX113.amr.corp.intel.com> Hi Alan/Chris, Here is the version 26 of the patch: http://cr.openjdk.java.net/~ylu/8195160.26/ In this version, I first applied the following patches from Chris: http://cr.openjdk.java.net/~chegar/rsocket/webrev.25.1/ http://cr.openjdk.java.net/~chegar/rsocket/webrev.25.2/ http://cr.openjdk.java.net/~chegar/rsocket/webrev.25.3/ http://cr.openjdk.java.net/~chegar/rsocket/webrev.25.4/ http://cr.openjdk.java.net/~chegar/rsocket/webrev.25.5/ http://cr.openjdk.java.net/~chegar/rsocket/webrev.25.6/ Then, following Alan's suggestion, I made the rsocket to be non-blocking at the creation time for socket channels. I modified accept function from RdmaServerSocketChannelImpl, and connect/read/write functions from RdmaSocketChannelImpl to simulate blocking. Please let me know if this is a reasonable approach. I tested the patch with all 23 JTreg test cases and they all pass (I commented out all 12 tests for non-blocking connect cases in IOExchanges.java given issue #2 is still open) I also sent an email to Sean on the Linux kernel fix for issue #1. Once I get a reply from him, I will let you both know. Thanks, Lucy >-----Original Message----- >From: Chris Hegarty [mailto:chris.hegarty at oracle.com] >Sent: Tuesday, December 18, 2018 6:15 AM >To: Lu, Yingqi >Cc: Alan Bateman ; nio-dev at openjdk.java.net; >Viswanathan, Sandhya ; Aundhe, Shirish >; Kaczmarek, Eric >Subject: Re: adding rsockets support into JDK > > > >> On 17 Dec 2018, at 19:07, Lu, Yingqi wrote: >> >> I already discussed with Sean on this specific issue. He mentioned that even he >can propose a fix to the kernel librdmacm module, it would take years to get into >majority of the distros. I would recommend to emulate the behavior inside JDK >and make use of it. > >Sure, but it would be good to have some forward movement on this, so that the >Java implementation can remove the workaround at some future point. > >-Chris. From christoph.langer at sap.com Wed Dec 19 10:14:51 2018 From: christoph.langer at sap.com (Langer, Christoph) Date: Wed, 19 Dec 2018 10:14:51 +0000 Subject: RFR (S): 8215472: Cleanups in implementation classes of jdk.zipfs and tests In-Reply-To: <6b71deaa-671f-242d-d837-9792b97d140b@gmail.com> References: <247E396D-3C1B-4C4C-897D-54A749AEB6D6@oracle.com> <846e3c3c8a3546379c00103f08a1b8a6@sap.com> <6b71deaa-671f-242d-d837-9792b97d140b@gmail.com> Message-ID: <3a8081a2187f44a580a885c7ba125775@sap.com> Hi Sherman, thanks for the confirmation. Further refactoring regarding Deflator caching can be done with a separate issue. So, hearing no objections I'll push the patch once my testing is done. Best regards Christoph > -----Original Message----- > From: core-libs-dev On Behalf > Of Xueming Shen > Sent: Mittwoch, 19. Dezember 2018 08:15 > To: core-libs-dev at openjdk.java.net > Subject: Re: RFR (S): 8215472: Cleanups in implementation classes of jdk.zipfs > and tests > > > On 12/18/18 5:44 AM, Langer, Christoph wrote: > > > >> In ZipFileSystem you remove the unused method releaseDeflater - to me > >> this indicates the resource pooling is actually broken? I.e., shouldn't > >> EntryOutputStreamDef return its Deflater to the cache when it's closed, > >> similar to how the anonymous InflaterInputStream in getInputStream > does > >> it? As it's currently implemented the deflaters list will always be > >> empty and no Deflater returned, so could go the other way and remove > >> that cache if caching Deflaters isn't useful. > > You are right. I think this is a flaw of the change for 8034802 [1] [2]. I added > the call to releaseDeflater() in the close method of EntryOutputStreamDef. > > > > > My bad. Adding the logic back looks fine. In latest implementation the > EntryOutputStreamDef > > is only created/used in sync() to write out any updated/compressed > entry, which means there > > is actually only one deflater is being used at a time during the sync(), > so you can just have one > > deflater and then reset it before passing on to the next entry write. > And in fact even the EOSDef > > is really not necessary. I was working on that ... but somehow I dropped > the ball during copy/ > > paste :-( ended up using the deflater from the cache but failed to > return it back. > > > Anyway. It's fine to keep current deflater cache mechanism, but it might > be worth considering > > to "inline" the EntryOutputStreamDef logic and/or remove the deflater > cache at all in the future. > > > -Sherman From christoph.langer at sap.com Fri Dec 21 10:41:18 2018 From: christoph.langer at sap.com (Langer, Christoph) Date: Fri, 21 Dec 2018 10:41:18 +0000 Subject: RFR - CSR: 8213082: (zipfs) Add support for POSIX file permissions (was: Re: RFR 8213031: (zipfs) Add support for POSIX file permissions) Message-ID: <6b4b78c731e24ef8aa45b6c048d017da@sap.com> Hi folks, getting back to the topic of adding POSIX file permission support to jdk.zipfs... I think as we are now in the early stages of JDK13, it's a good point in time to get some (hopefully final) activity on that one. In the last review discussions you were asking me to provide some write-up of the proposal. Therefore I updated the CSR. It should now be a valid document for discussing the whole proposal, comprising the problem to solve, the proposed solution and its specification as well as addressing some concerns. And to get it clear: This item is only about jdk.zipfs. It is really independent of potential enhancements for java.util.zip or the jartool. So, I gently ask you to review the CSR. As for the implementation: I've worked on it together with Volker and will post an update soon. Thanks and Best regards Christoph > -----Original Message----- > From: Chris Hegarty > Sent: Montag, 5. November 2018 17:19 > To: Langer, Christoph ; core-libs-dev dev at openjdk.java.net>; security-dev at openjdk.java.net; Xueming Shen > > Cc: nio-dev > Subject: Re: RFR 8213031: (zipfs) Add support for POSIX file permissions > > > On 05/11/18 15:59, Alan Bateman wrote: > > On 05/11/2018 13:05, Langer, Christoph wrote: > >> > >>... > >> > > I think you'll need to do a write-up of the overall proposal so that > > folks can jump in and point out the implications. It's not easy to do > > this in a code review of a small piece of the solution. > > Right, that's the main motivation for my previous questions. It is good > to get a *complete* view of what is intended before reviewing the code. > ( Sorry, if I've missed some of the previous context ). > > -Chris. From Alan.Bateman at oracle.com Fri Dec 21 11:16:44 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 21 Dec 2018 11:16:44 +0000 Subject: RFR - CSR: 8213082: (zipfs) Add support for POSIX file permissions (was: Re: RFR 8213031: (zipfs) Add support for POSIX file permissions) In-Reply-To: <6b4b78c731e24ef8aa45b6c048d017da@sap.com> References: <6b4b78c731e24ef8aa45b6c048d017da@sap.com> Message-ID: <6dc8281e-b4ea-e27e-acdb-984612d91565@oracle.com> On 21/12/2018 10:41, Langer, Christoph wrote: > Hi folks, > > getting back to the topic of adding POSIX file permission support to jdk.zipfs... I think as we are now in the early stages of JDK13, it's a good point in time to get some (hopefully final) activity on that one. > > In the last review discussions you were asking me to provide some write-up of the proposal. > Therefore I updated the CSR. It should now be a valid document for discussing the whole proposal, comprising the problem to solve, the proposed solution and its specification as well as addressing some concerns. > > And to get it clear: This item is only about jdk.zipfs. It is really independent of potential enhancements for java.util.zip or the jartool. So, I gently ask you to review the CSR. > > As for the implementation: I've worked on it together with Volker and will post an update soon. > Adding support for POSIX file permissions to the zip APIs is problematic as we've been discussing here. There are security concerns and also concerns that how it interacts with JAR files and signed JAR in particular. I don't disagree that we can come to agreement on zipfs supporting a solution but I think we need to get the bigger picture on where this is going first. If the piece to change the java.util.zip APIs is dropped then it would make these discussions a lot simpler as it removes most of the security issues from the table. -Alan From christoph.langer at sap.com Fri Dec 21 13:43:13 2018 From: christoph.langer at sap.com (Langer, Christoph) Date: Fri, 21 Dec 2018 13:43:13 +0000 Subject: RFR - CSR: 8213082: (zipfs) Add support for POSIX file permissions (was: Re: RFR 8213031: (zipfs) Add support for POSIX file permissions) In-Reply-To: <6dc8281e-b4ea-e27e-acdb-984612d91565@oracle.com> References: <6b4b78c731e24ef8aa45b6c048d017da@sap.com> <6dc8281e-b4ea-e27e-acdb-984612d91565@oracle.com> Message-ID: Hi Alan, > Adding support for POSIX file permissions to the zip APIs is problematic > as we've been discussing here. There are security concerns and also > concerns that how it interacts with JAR files and signed JAR in > particular. I don't disagree that we can come to agreement on zipfs > supporting a solution but I think we need to get the bigger picture on > where this is going first. If the piece to change the java.util.zip APIs > is dropped then it would make these discussions a lot simpler as it > removes most of the security issues from the table. Yes, please consider changes to java.util.zip APIs as dropped. At least for the moment. I'm not saying I won't ever get back to that topic but maybe an enhancement of jdk.zipfs is already sufficient to provide the required Posix permission support for the Java platform. Best regards Christoph From christoph.langer at sap.com Fri Dec 21 14:31:11 2018 From: christoph.langer at sap.com (Langer, Christoph) Date: Fri, 21 Dec 2018 14:31:11 +0000 Subject: RFR 8213031: (zipfs) Add support for POSIX file permissions In-Reply-To: References: Message-ID: Hi all, here comes the updated webrev: http://cr.openjdk.java.net/~clanger/webrevs/8213031.3/ I've rebased the change to the current state of the JDK depot. Thanks to Volker, the test has been enhanced and now also tests more copy operations (from zip file system to zip file system and from zip file system to default file system and back). The points below were also addressed: > ZipFileAttributeView.java > - can you please throw an AssertionError() for the default branch in > the switch expression in the "ZipFileAttributeView.name()". The default branch will return "basic". Looking at the code it should not be jumped to anyway. > ZipFileSystem.java > - please also put @Override annotations on the method implementations > from ZipFileAttributes (e.g. "compressedSize()", "crc()", etc...) and > a comment at the place where the implementation of the > "PosixFileAttributes" methods starts. Done, I also reordered the methods - first "basic" then "posix" then "zip". > ZipUtils.java > - just a question: isn't it possible to reuse > sun.nio.fs.UnixFileModeAttribute.toUnixMode() and the corresponding > constants from sun.nio.fs.UnixConstants instead of redefining them > here? You could export them from java.base to jdk.zipfs but the > problem is probably that at least sun.nio.fs.UnixConstants is a > generated class which only gets generated on Unix systems, right ? You've already answered yourself ?? These classes only exist on Unix type JDKs. > ZipFileAttributes.java > - it seems a little odd that you've added "setPermissions()" to > ZipFileAttributes. All the attribute classes (i.e. > BasicFileAttributes, PosixFileAttributes) have no setters. Isn't it > possible to implement "ZipFileAttributeView.setPermissions()" by > calling "path.setPermissions()" (as this is done in > "ZipFileAttributeView.setTimes()") and handle everything in > ZipFileSyste (as this is done with "setTimes()"). Good catch & thanks for providing the better implementation. I think this version is quite final now and thoroughly tested. I hope to get some valid reviews soon... Thanks Christoph From volker.simonis at gmail.com Fri Dec 21 16:34:22 2018 From: volker.simonis at gmail.com (Volker Simonis) Date: Fri, 21 Dec 2018 17:34:22 +0100 Subject: RFR 8213031: (zipfs) Add support for POSIX file permissions In-Reply-To: References: Message-ID: Hi Christoph, thanks for updating the change. I think it is in a good state now and ready to go! Also the documentation in the CSR for this issue ( https://bugs.openjdk.java.net/browse/JDK-8213082) is greatly appreciated and answers all the questions which have been raised so far. So if there are still any open questions I'd recommend that any potential reviewer consults the CSR at https://bugs.openjdk.java.net/browse/JDK-8213082 first. Thank you and best regards, Volker On Fri, Dec 21, 2018 at 3:31 PM Langer, Christoph wrote: > Hi all, > > here comes the updated webrev: > http://cr.openjdk.java.net/~clanger/webrevs/8213031.3/ > > I've rebased the change to the current state of the JDK depot. Thanks to > Volker, the test has been enhanced and now also tests more copy operations > (from zip file system to zip file system and from zip file system to > default file system and back). > > The points below were also addressed: > > > ZipFileAttributeView.java > > - can you please throw an AssertionError() for the default branch in > > the switch expression in the "ZipFileAttributeView.name()". > > The default branch will return "basic". Looking at the code it should not > be jumped to anyway. > > > ZipFileSystem.java > > - please also put @Override annotations on the method implementations > > from ZipFileAttributes (e.g. "compressedSize()", "crc()", etc...) and > > a comment at the place where the implementation of the > > "PosixFileAttributes" methods starts. > > Done, I also reordered the methods - first "basic" then "posix" then "zip". > > > ZipUtils.java > > - just a question: isn't it possible to reuse > > sun.nio.fs.UnixFileModeAttribute.toUnixMode() and the corresponding > > constants from sun.nio.fs.UnixConstants instead of redefining them > > here? You could export them from java.base to jdk.zipfs but the > > problem is probably that at least sun.nio.fs.UnixConstants is a > > generated class which only gets generated on Unix systems, right ? > > You've already answered yourself ?? These classes only exist on Unix type > JDKs. > > > ZipFileAttributes.java > > - it seems a little odd that you've added "setPermissions()" to > > ZipFileAttributes. All the attribute classes (i.e. > > BasicFileAttributes, PosixFileAttributes) have no setters. Isn't it > > possible to implement "ZipFileAttributeView.setPermissions()" by > > calling "path.setPermissions()" (as this is done in > > "ZipFileAttributeView.setTimes()") and handle everything in > > ZipFileSyste (as this is done with "setTimes()"). > > Good catch & thanks for providing the better implementation. > > > I think this version is quite final now and thoroughly tested. I hope to > get some valid reviews soon... > > Thanks > Christoph > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From volker.simonis at gmail.com Fri Dec 21 16:43:14 2018 From: volker.simonis at gmail.com (Volker Simonis) Date: Fri, 21 Dec 2018 17:43:14 +0100 Subject: RFR - CSR: 8213082: (zipfs) Add support for POSIX file permissions (was: Re: RFR 8213031: (zipfs) Add support for POSIX file permissions) In-Reply-To: References: <6b4b78c731e24ef8aa45b6c048d017da@sap.com> <6dc8281e-b4ea-e27e-acdb-984612d91565@oracle.com> Message-ID: Hi Alan, thanks for looking at this issue. I've dived into the ZipFS implementation during the last weeks and together with Christoph we've extended and improved both the implementation the test coverage. As Christoph already emphasized, this change is only for improving jdk.nio.zipfs without any side effects on java.util.zip and java.util.jar. Please find my comments for the CSR below (for some reason I couldn't add them to the CSR directly). I'd really appreciate if you could reconsider reviewing Christoph's change ("RFR 8213031: (zipfs) Add support for POSIX file permissions") and CSR. Thank you, Volker ============================================= I've thoroughly looked at this proposal and honestly speaking, I don't think that we even need a CSR for the proposed enhancement. Only the implementation of some classes in the `jdk.nio.zipfs` package are changed but no publicly exported API's. As Christoph emphasized, the proposed changes **only** affect the Zip File System as implemented in the `jdk.nio.zipfs` package. It doesn't touch neither the implementation nor the behavior of any of the classes in the `java.util.zip` or `java.util.jar` packages. Previous reviewers of this CSR and of the corresponding change JDK-8213031 raised some concerns regarding the security implication of this change - especially in the context of signed jars. I've looked at these concerns but I couldn't find any evidence for problems because: - no exposed JAR functionality is affected by these changes (because that is handled by the implementation in `java.util.zip` and `java.util.jar`) - Jar signing doesn't take file attributes into account (even not the basic attributes like the modification time). It only hashes the file contents. You can already now use external zip tools to update the attributes of files in a signed archive without affecting the validity of the signature. For all these reasons I've reviewed this CSR and I kindly ask you to approve it. On Fri, Dec 21, 2018 at 2:43 PM Langer, Christoph wrote: > > Hi Alan, > > > Adding support for POSIX file permissions to the zip APIs is problematic > > as we've been discussing here. There are security concerns and also > > concerns that how it interacts with JAR files and signed JAR in > > particular. I don't disagree that we can come to agreement on zipfs > > supporting a solution but I think we need to get the bigger picture on > > where this is going first. If the piece to change the java.util.zip APIs > > is dropped then it would make these discussions a lot simpler as it > > removes most of the security issues from the table. > > Yes, please consider changes to java.util.zip APIs as dropped. At least for the moment. I'm not saying I won't ever get back to that topic but maybe an enhancement of jdk.zipfs is already sufficient to provide the required Posix permission support for the Java platform. > > Best regards > Christoph > From joe.darcy at oracle.com Fri Dec 21 17:45:48 2018 From: joe.darcy at oracle.com (Joe Darcy) Date: Fri, 21 Dec 2018 09:45:48 -0800 Subject: RFR - CSR: 8213082: (zipfs) Add support for POSIX file permissions In-Reply-To: References: <6b4b78c731e24ef8aa45b6c048d017da@sap.com> <6dc8281e-b4ea-e27e-acdb-984612d91565@oracle.com> Message-ID: <04f9501d-4a4d-be0e-74ff-0c9dd4978dc6@oracle.com> Hello, On 12/21/2018 8:43 AM, Volker Simonis wrote: > Hi Alan, > > thanks for looking at this issue. I've dived into the ZipFS > implementation during the last weeks and together with Christoph we've > extended and improved both the implementation the test coverage. As > Christoph already emphasized, this change is only for improving > jdk.nio.zipfs without any side effects on java.util.zip and > java.util.jar. > > Please find my comments for the CSR below (for some reason I couldn't > add them to the CSR directly). > To leave a comment on a CSR, using the "Comment" button at the top of the page or hit "Edit", rather than the "Comment" button at the bottom of the page (known bug). HTH, -Joe From Nick.Gasson at arm.com Fri Dec 28 03:35:27 2018 From: Nick.Gasson at arm.com (Nick Gasson (Arm Technology China)) Date: Fri, 28 Dec 2018 03:35:27 +0000 Subject: RFR: 8215202: AArch64: jtreg test test/jdk/sun/nio/cs/FindEncoderBugs.java fails In-Reply-To: References: <414ab5fa-33cd-47f3-e76f-2ee95291bea6@redhat.com> Message-ID: <29dd416e-6eca-e956-4668-410fd592492f@arm.com> Hi Andrew / Dmitrij, Is there anything else you want me to do with this one before it can be committed? Thanks, Nick On 12/12/2018 09:13, Nick Gasson (Arm Technology China) wrote: > Hi Andrew, > >> One thing: is randomChar() fully reproducible? That is to say, are >> we sure that we always run the same tests? > > It uses jdk.test.lib.RandomFactory which seeds the RNG from the "seed" > system property. So if this is set when running jtreg the results are > fully reproducible. > > Thanks, > Nick > >> -----Original Message----- >> From: Andrew Haley >> Sent: 12 December 2018 00:23 >> To: Nick Gasson (Arm Technology China) ; hotspot- >> compiler-dev at openjdk.java.net >> Cc: nio-dev at openjdk.java.net; nd ; aarch64-port- >> dev at openjdk.java.net >> Subject: Re: [aarch64-port-dev ] RFR: 8215202: AArch64: jtreg test >> test/jdk/sun/nio/cs/FindEncoderBugs.java fails >> >> Good catch, again. >> >> On 12/11/18 8:05 AM, Nick Gasson (Arm Technology China) wrote: >>> Also Extended the FindEncoderBugs test so that it hits all cases in >>> the intrinsic assembly. Currently only the final NEXT_1 section is >>> covered by this test. >> >> One thing: is randomChar() fully reproducible? That is to say, are >> we sure that we always run the same tests? >> >> -- >> Andrew Haley >> Java Platform Lead Engineer >> Red Hat UK Ltd. >> EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 From Nick.Gasson at arm.com Fri Dec 28 03:49:45 2018 From: Nick.Gasson at arm.com (Nick Gasson (Arm Technology China)) Date: Fri, 28 Dec 2018 03:49:45 +0000 Subject: RFR: 8215202: AArch64: jtreg test test/jdk/sun/nio/cs/FindEncoderBugs.java fails In-Reply-To: <29dd416e-6eca-e956-4668-410fd592492f@arm.com> References: <414ab5fa-33cd-47f3-e76f-2ee95291bea6@redhat.com> <29dd416e-6eca-e956-4668-410fd592492f@arm.com> Message-ID: <0eaf2cc7-e199-7041-f222-1fce83b68d55@arm.com> Sorry I just found it in jdk/jdk12, please ignore. Thanks Andrew! Nick On 28/12/2018 11:35, Nick Gasson (Arm Technology China) wrote: > Hi Andrew / Dmitrij, > > Is there anything else you want me to do with this one before it can be > committed? > > Thanks, > Nick > > On 12/12/2018 09:13, Nick Gasson (Arm Technology China) wrote: >> Hi Andrew, >> >>> One thing: is randomChar() fully reproducible? That is to say, are >>> we sure that we always run the same tests? >> >> It uses jdk.test.lib.RandomFactory which seeds the RNG from the "seed" >> system property. So if this is set when running jtreg the results are >> fully reproducible. >> >> Thanks, >> Nick >> >>> -----Original Message----- >>> From: Andrew Haley >>> Sent: 12 December 2018 00:23 >>> To: Nick Gasson (Arm Technology China) ; hotspot- >>> compiler-dev at openjdk.java.net >>> Cc: nio-dev at openjdk.java.net; nd ; aarch64-port- >>> dev at openjdk.java.net >>> Subject: Re: [aarch64-port-dev ] RFR: 8215202: AArch64: jtreg test >>> test/jdk/sun/nio/cs/FindEncoderBugs.java fails >>> >>> Good catch, again. >>> >>> On 12/11/18 8:05 AM, Nick Gasson (Arm Technology China) wrote: >>>> Also Extended the FindEncoderBugs test so that it hits all cases in >>>> the intrinsic assembly. Currently only the final NEXT_1 section is >>>> covered by this test. >>> >>> One thing: is randomChar() fully reproducible? That is to say, are >>> we sure that we always run the same tests? >>> >>> -- >>> Andrew Haley >>> Java Platform Lead Engineer >>> Red Hat UK Ltd. >>> EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671