From Alan.Bateman at oracle.com Sat Aug 21 08:17:11 2010 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 21 Aug 2010 16:17:11 +0100 Subject: Two small fixes Message-ID: <4C6FEDF7.3050202@oracle.com> I'd like to get fixes for two small fixes into b108. The first is 6431344 where FileChannel's transferTo falls back to mmap when sendfile/equivalent can't be used and the target is a "trusted channel". The implementation attempts to mmap the entire region to be transferred which is problematic for large transfers, esp. 32-bit where the address space may not be available. Same issue with transferFrom where mmap is used when the source is also a FileChannel. The proposed change just maps the file in chunks, up to 8MB at a time. I don't propose to include a regression test as the code is well exercised by existing tests, and also tests that transfer several GBs take too long for the regression test suite. The webrev is here: http://cr.openjdk.java.net/~alanb/6431344/webrev The other one is 6978511, an oversight in the implementation to Path.toRealPath. The method should fail if the file does not exist. The fix is to trivially check that it does actually exist. The webrev is here: http://cr.openjdk.java.net/~alanb/6978511/webrev/ Thanks, Alan. From forax at univ-mlv.fr Sun Aug 22 07:22:54 2010 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Sun, 22 Aug 2010 16:22:54 +0200 Subject: Two small fixes In-Reply-To: <4C6FEDF7.3050202@oracle.com> References: <4C6FEDF7.3050202@oracle.com> Message-ID: <4C7132BE.5070004@univ-mlv.fr> Le 21/08/2010 17:17, Alan Bateman a ?crit : > I'd like to get fixes for two small fixes into b108. > > The first is 6431344 where FileChannel's transferTo falls back to mmap > when sendfile/equivalent can't be used and the target is a "trusted > channel". The implementation attempts to mmap the entire region to be > transferred which is problematic for large transfers, esp. 32-bit > where the address space may not be available. Same issue with > transferFrom where mmap is used when the source is also a FileChannel. > The proposed change just maps the file in chunks, up to 8MB at a time. > I don't propose to include a regression test as the code is well > exercised by existing tests, and also tests that transfer several GBs > take too long for the regression test suite. The webrev is here: > http://cr.openjdk.java.net/~alanb/6431344/webrev a minor nit: line 468 (end of transferToTrustedChannel) should be : return count - remaining; instead of return count-remaining; I wonder if the 8 megs is enough. I've played some time ago with Windows XP mmap using Java API and found that it was more efficient to use read() if the file length was less than few hundred megabytes. It was ok for linux, and i have not tested with more recent version of Windows. > > The other one is 6978511, an oversight in the implementation to > Path.toRealPath. The method should fail if the file does not exist. > The fix is to trivially check that it does actually exist. The webrev > is here: > http://cr.openjdk.java.net/~alanb/6978511/webrev/ fine, even if the patch is named ScatterGather :) > > Thanks, > Alan. R?mi From chris.hegarty at oracle.com Mon Aug 23 06:17:21 2010 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Mon, 23 Aug 2010 14:17:21 +0100 Subject: Two small fixes In-Reply-To: <4C6FEDF7.3050202@oracle.com> References: <4C6FEDF7.3050202@oracle.com> Message-ID: <4C7274E1.8080900@oracle.com> On 21/08/2010 16:17, Alan Bateman wrote: > I'd like to get fixes for two small fixes into b108. > > The first is 6431344 where FileChannel's transferTo falls back to mmap > when sendfile/equivalent can't be used and the target is a "trusted > channel". The implementation attempts to mmap the entire region to be > transferred which is problematic for large transfers, esp. 32-bit where > the address space may not be available. Same issue with transferFrom > where mmap is used when the source is also a FileChannel. The proposed > change just maps the file in chunks, up to 8MB at a time. I don't > propose to include a regression test as the code is well exercised by > existing tests, and also tests that transfer several GBs take too long > for the regression test suite. The webrev is here: > http://cr.openjdk.java.net/~alanb/6431344/webrev This change mainly looks fine. One minor comment; do we really need the nested try statements in transferFromFileChannel? Can the catch not simply match the first try and remove the second? > The other one is 6978511, an oversight in the implementation to > Path.toRealPath. The method should fail if the file does not exist. The > fix is to trivially check that it does actually exist. The webrev is here: > http://cr.openjdk.java.net/~alanb/6978511/webrev/ This change looks fine. -Chris. > Thanks, > Alan. From Alan.Bateman at oracle.com Mon Aug 23 07:36:59 2010 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 23 Aug 2010 15:36:59 +0100 Subject: Two small fixes In-Reply-To: <4C7132BE.5070004@univ-mlv.fr> References: <4C6FEDF7.3050202@oracle.com> <4C7132BE.5070004@univ-mlv.fr> Message-ID: <4C72878B.1010004@oracle.com> R?mi Forax wrote: > : > a minor nit: > line 468 (end of transferToTrustedChannel) should be : > > return count - remaining; > > instead of > > return count-remaining; Thanks R?mi, I'll fix that and refresh the webrev. > > I wonder if the 8 megs is enough. I've played some time ago with > Windows XP > mmap using Java API and found that it was more efficient to use read() > if the file length was less than few hundred megabytes. > It was ok for linux, and i have not tested with more recent version of > Windows. This is definitely worth looking at. Memory mapping is usually a win on Solaris/Linux although it might be more expensive for very small transfers. Solaris cp(1), for example, only maps if the file is larger than 32k. Windows is different and I haven't observed any real benefit on various editions of Windows, even with large files. There probably isn't one answer that is right for all platforms and we might have to change FileChannel.transferXXX so that the transfer method uses the right transfer strategy for the platform. So if it's okay with you, I'd prefer to keep the current bugID focused on removing mapping of the entire file and use a different bugID to "tune" the transfer methods. One thing I should say is the most important case for the transferTo method is where the target is a SocketChannel. Solaris/Linux will uses sendfile as before and not the code in this patch. The 8MB limit shouldn't be an issue on Windows because the socket buffer will limit the size anyway. The changes in the patch really only impact the FileChannel to FileChannel case when the transfer size is >8MB. I've run tests with transfer sizes of 16, 32, 64, 128, 512MB... and I don't any real difference compared to jdk7-b106 (this includes Windows XP and Windows Server 2008). > > : > fine, even if the patch is named ScatterGather :) Yeah, that's webrev as it names the patch based on the repo name and I re-used a repository that I had been using recently. Thanks for spending time looking at these changes. -Alan. From Alan.Bateman at oracle.com Mon Aug 23 07:48:32 2010 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 23 Aug 2010 15:48:32 +0100 Subject: Two small fixes In-Reply-To: <4C7274E1.8080900@oracle.com> References: <4C6FEDF7.3050202@oracle.com> <4C7274E1.8080900@oracle.com> Message-ID: <4C728A40.8030004@oracle.com> Chris Hegarty wrote: > : >> The webrev is here: >> http://cr.openjdk.java.net/~alanb/6431344/webrev > > This change mainly looks fine. One minor comment; do we really need > the nested try statements in transferFromFileChannel? Can the catch > not simply match the first try and remove the second? That would be better - thanks! I've updated the webrev. -Alan From martinrb at google.com Mon Aug 30 11:53:36 2010 From: martinrb at google.com (Martin Buchholz) Date: Mon, 30 Aug 2010 11:53:36 -0700 Subject: Small improvements to nio native code (unix) Message-ID: Hi Alan, I'd like you to do a code review. Avoid problematic JNI *Critical functions. Fix rawtypes compile warnings. Make IOUtil native code more readable. Rename initPipe to makePipe. Rename "block" argument to "blocking" for consistency. http://cr.openjdk.java.net/~martin/webrevs/openjdk7/IOUtil/ Thanks, Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100830/c2c6270f/attachment.html From martinrb at google.com Mon Aug 30 12:41:34 2010 From: martinrb at google.com (Martin Buchholz) Date: Mon, 30 Aug 2010 12:41:34 -0700 Subject: DevPollSelector on Linux Message-ID: Hi nio maintainers, It appears that the intent of make/java/nio/Makefile that DevPollSelector* is only built/included on the Solaris platform, but ... it it is built/included on Linux as well, because ... DefaultSelectorProvider has an explicit reference to it: ./src/solaris/classes/sun/nio/ch/DefaultSelectorProvider.java:51: return new sun.nio.ch.DevPollSelectorProvider(); Is that a bug? Should DefaultSelectorProvider use reflection to find its default provider? Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100830/a6c62716/attachment.html From Alan.Bateman at oracle.com Mon Aug 30 12:50:52 2010 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 30 Aug 2010 20:50:52 +0100 Subject: DevPollSelector on Linux In-Reply-To: References: Message-ID: <4C7C0B9C.2090800@oracle.com> Martin Buchholz wrote: > Hi nio maintainers, > > It appears that the intent of make/java/nio/Makefile that > DevPollSelector* is only built/included on the Solaris platform, > but ... it it is built/included on Linux as well, > because ... DefaultSelectorProvider has an explicit reference to it: > > ./src/solaris/classes/sun/nio/ch/DefaultSelectorProvider.java:51: > return new sun.nio.ch.DevPollSelectorProvider(); > > Is that a bug? Should DefaultSelectorProvider use reflection to find > its default provider? > > Martin Yes, it's a bug. At one point I replaced the explicit reference with reflection but I didn't push the patch at the time (it was part of other work to move the DevPolll classes out of src/share/classes). -Alan. From Alan.Bateman at oracle.com Tue Aug 31 02:30:28 2010 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 31 Aug 2010 10:30:28 +0100 Subject: Small improvements to nio native code (unix) In-Reply-To: References: Message-ID: <4C7CCBB4.9050208@oracle.com> Martin Buchholz wrote: > Hi Alan, > > I'd like you to do a code review. > > Avoid problematic JNI *Critical functions. > Fix rawtypes compile warnings. > Make IOUtil native code more readable. > Rename initPipe to makePipe. > Rename "block" argument to "blocking" for consistency. > > http://cr.openjdk.java.net/~martin/webrevs/openjdk7/IOUtil/ > > > Thanks, > > Martin It makes sense to return the two file descriptors in a jlong rather than using a JNI critical section (although they aren't as bad as they used to be in that a thread requiring a GC because an allocation cannot be satisfied will now stall rather than throw OOME). Anyway, the changes look okay to me. I've created this bug: 6981145: (se) Eliminate JNI*Critical when creating pipes and other cleanups Minor nit but at line IOUtil.c L140 you've got "const char * msg" (space on either side of the asterisk). In passing, I see in Java_sun_nio_ch_IOUtil_makePipe doesn't close the pipe in the highly unlikely event that the configureBlocking fails. It might be good to fix that while you are there (if you don't mind of course). -Alan. From martinrb at google.com Tue Aug 31 12:36:55 2010 From: martinrb at google.com (Martin Buchholz) Date: Tue, 31 Aug 2010 12:36:55 -0700 Subject: Small improvements to nio native code (unix) In-Reply-To: <4C7CCBB4.9050208@oracle.com> References: <4C7CCBB4.9050208@oracle.com> Message-ID: On Tue, Aug 31, 2010 at 02:30, Alan Bateman wrote: > Martin Buchholz wrote: > >> Hi Alan, >> >> I'd like you to do a code review. >> >> Avoid problematic JNI *Critical functions. >> Fix rawtypes compile warnings. >> Make IOUtil native code more readable. >> Rename initPipe to makePipe. >> Rename "block" argument to "blocking" for consistency. >> >> http://cr.openjdk.java.net/~martin/webrevs/openjdk7/IOUtil/< >> http://cr.openjdk.java.net/%7Emartin/webrevs/openjdk7/IOUtil/> >> >> Thanks, >> >> Martin >> > It makes sense to return the two file descriptors in a jlong rather than > using a JNI critical section (although they aren't as bad as they used to be > in that a thread requiring a GC because an allocation cannot be satisfied > will now stall rather than throw OOME). > > Anyway, the changes look okay to me. I've created this bug: > 6981145: (se) Eliminate JNI*Critical when creating pipes and other > cleanups > > Minor nit but at line IOUtil.c L140 you've got "const char * msg" (space on > either side of the asterisk). > > Done, I've changed it to char *msg which seems to be winning the whitespace style popularity contest. > In passing, I see in Java_sun_nio_ch_IOUtil_makePipe doesn't close the pipe > in the highly unlikely event that the configureBlocking fails. It might be > good to fix that while you are there (if you don't mind of course). > > Done. Webrev regenerated. Martin > -Alan. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100831/f29ab1c3/attachment.html From Alan.Bateman at oracle.com Tue Aug 31 13:37:46 2010 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 31 Aug 2010 21:37:46 +0100 Subject: Small improvements to nio native code (unix) In-Reply-To: References: <4C7CCBB4.9050208@oracle.com> Message-ID: <4C7D681A.6040800@oracle.com> Martin Buchholz wrote: > > > : > > > > In passing, I see in Java_sun_nio_ch_IOUtil_makePipe doesn't close > the pipe in the highly unlikely event that the configureBlocking > fails. It might be good to fix that while you are there (if you > don't mind of course). > > > Done. > > Webrev regenerated. > Thanks for doing that. It's probably to close the pipe after JNU_ThrowIOExceptionWithLastError (in case one of the close calls fails, overriding errno). -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100831/885e42c3/attachment.html From martinrb at google.com Tue Aug 31 14:10:48 2010 From: martinrb at google.com (Martin Buchholz) Date: Tue, 31 Aug 2010 14:10:48 -0700 Subject: Small improvements to nio native code (unix) In-Reply-To: <4C7D681A.6040800@oracle.com> References: <4C7CCBB4.9050208@oracle.com> <4C7D681A.6040800@oracle.com> Message-ID: On Tue, Aug 31, 2010 at 13:37, Alan Bateman wrote: > Martin Buchholz wrote: > Thanks for doing that. It's probably to close the pipe after > JNU_ThrowIOExceptionWithLastError (in case one of the close calls fails, > overriding errno). > > Done. I appreciate the attention to detail, even if this bug would never occur in the wild. Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100831/662d61b5/attachment.html