From shirishk at linux.vnet.ibm.com Tue Aug 7 02:34:23 2012 From: shirishk at linux.vnet.ibm.com (Shirish Kuncolienkar) Date: Tue, 07 Aug 2012 15:04:23 +0530 Subject: Code review request 7132889: Possible race conditions in AbstractSelectableChannel Message-ID: <5020E11F.8020007@linux.vnet.ibm.com> Could I get this change reviewed please code inspection of the AbstractSelectableChannel class shows that a timing window exists between the channel close and cancellation of the key. Which can lead to following consequences 1. Selector registered on this channel can select the key while the channel is closed or is being closed 2. A new key can be added to the channel while the it is closed or is being closed 3. configureBlocking() can be invoked while the channel is closed or is being closed One might be able to construct a testcase using a test-specific subclass of AbstractSelectableChannel, which has a synchronization point in its implementation of implCloseSelectableChannel() so that tests can be performed there. However, it would be quite an involved thing to write with clarity. I have created a patch under http://cr.openjdk.java.net/~luchsh/7132889/ From Alan.Bateman at oracle.com Tue Aug 7 02:58:31 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 07 Aug 2012 10:58:31 +0100 Subject: Code review request 7132889: Possible race conditions in AbstractSelectableChannel In-Reply-To: <5020E11F.8020007@linux.vnet.ibm.com> References: <5020E11F.8020007@linux.vnet.ibm.com> Message-ID: <5020E6C7.90501@oracle.com> On 07/08/2012 10:34, Shirish Kuncolienkar wrote: > Could I get this change reviewed please > > code inspection of the AbstractSelectableChannel class shows that a > timing window exists between the channel close and cancellation of the > key. Which can lead to following consequences > 1. Selector registered on this channel can select the key while the > channel is closed or is being closed > 2. A new key can be added to the channel while the it is closed or is > being closed > 3. configureBlocking() can be invoked while the channel is closed or > is being closed > > One might be able to construct a testcase using a test-specific > subclass of AbstractSelectableChannel, which has a synchronization > point in its implementation of implCloseSelectableChannel() so that > tests can be performed there. However, it would be quite an involved > thing to write with clarity. > > I have created a patch under http://cr.openjdk.java.net/~luchsh/7132889/ > The main issue in 7132889 has already been fixed in jdk8 and 7u6 via 6346658. Would it be possible to test with a build of one of these releases and tell us if you still run into the original problem? On the regLock vs. keyLock issue then I would like to review this but I don't have cycles just at the moment as you may be right that the synchronization isn't consistent (the synchronization is covered in various places in the Selector spec so it requires reading that, hopefully I will get time soon but it won't be this week). -Alan From shirishk at linux.vnet.ibm.com Tue Aug 7 06:17:37 2012 From: shirishk at linux.vnet.ibm.com (Shirish Kuncolienkar) Date: Tue, 07 Aug 2012 18:47:37 +0530 Subject: Code review request 7132889: Possible race conditions in AbstractSelectableChannel In-Reply-To: <5020E6C7.90501@oracle.com> References: <5020E11F.8020007@linux.vnet.ibm.com> <5020E6C7.90501@oracle.com> Message-ID: <50211571.8050900@linux.vnet.ibm.com> On 8/7/2012 3:28 PM, Alan Bateman wrote: > On 07/08/2012 10:34, Shirish Kuncolienkar wrote: >> Could I get this change reviewed please >> >> code inspection of the AbstractSelectableChannel class shows that a >> timing window exists between the channel close and cancellation of >> the key. Which can lead to following consequences >> 1. Selector registered on this channel can select the key while the >> channel is closed or is being closed >> 2. A new key can be added to the channel while the it is closed or is >> being closed >> 3. configureBlocking() can be invoked while the channel is closed or >> is being closed >> >> One might be able to construct a testcase using a test-specific >> subclass of AbstractSelectableChannel, which has a synchronization >> point in its implementation of implCloseSelectableChannel() so that >> tests can be performed there. However, it would be quite an involved >> thing to write with clarity. >> >> I have created a patch under http://cr.openjdk.java.net/~luchsh/7132889/ >> > The main issue in 7132889 has already been fixed in jdk8 and 7u6 via > 6346658. Would it be possible to test with a build of one of these > releases and tell us if you still run into the original problem? > > On the regLock vs. keyLock issue then I would like to review this but > I don't have cycles just at the moment as you may be right that the > synchronization isn't consistent (the synchronization is covered in > various places in the Selector spec so it requires reading that, > hopefully I will get time soon but it won't be this week). > > -Alan > I looked at the fix for 6346658 and that will fix the looping issue as reported. However I feel that there are still issues with regLock vs. keyLock as you pointed out that the synchronization is inconsistent. So I suggest that you look at the change set once you have some free time. -Shirish From Alan.Bateman at oracle.com Wed Aug 8 03:30:42 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 08 Aug 2012 11:30:42 +0100 Subject: Code review request 7132889: Possible race conditions in AbstractSelectableChannel In-Reply-To: <50211571.8050900@linux.vnet.ibm.com> References: <5020E11F.8020007@linux.vnet.ibm.com> <5020E6C7.90501@oracle.com> <50211571.8050900@linux.vnet.ibm.com> Message-ID: <50223FD2.1090401@oracle.com> On 07/08/2012 14:17, Shirish Kuncolienkar wrote: > > I looked at the fix for 6346658 and that will fix the looping issue as > reported. However I feel that there are still issues with regLock vs. > keyLock as you pointed out that the synchronization is inconsistent. > So I suggest that you look at the change set once you have some free > time. > -Shirish > I think that expanding the scope of regLock in the register and configureBlocking methods is right. Changing implCloseChannel requires a bit of consideration and I will try to look at it next week. I expect that the proposed specification change will not be necessary and that it will be solely an implementation change to synchronize on regLock to avoid the case that someone registers a channel and gets a valid SelectionKey at around the time that the channel is closed. -Alan. From amy.lu at oracle.com Wed Aug 8 04:25:16 2012 From: amy.lu at oracle.com (Amy Lu) Date: Wed, 08 Aug 2012 19:25:16 +0800 Subject: Code review request 7189886: improve AsynchronousChannelGroup tests to cover withThreadPool method Message-ID: <50224C9C.5080000@oracle.com> While going through the missing coverage APIs, withThreadPool method of AsynchronousChannelGroup is not covered by any test. This is improvement for AsynchronousChannelGroup tests to cover this method. Thanks Alan for help posting the webrev. Please review: http://cr.openjdk.java.net/~alanb/7189886/webrev.00/ Thanks, Amy -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120808/99a8ec5c/attachment.html From shirishk at linux.vnet.ibm.com Wed Aug 8 05:40:37 2012 From: shirishk at linux.vnet.ibm.com (Shirish Kuncolienkar) Date: Wed, 08 Aug 2012 18:10:37 +0530 Subject: Code review request 7132889: Possible race conditions in AbstractSelectableChannel In-Reply-To: <50223FD2.1090401@oracle.com> References: <5020E11F.8020007@linux.vnet.ibm.com> <5020E6C7.90501@oracle.com> <50211571.8050900@linux.vnet.ibm.com> <50223FD2.1090401@oracle.com> Message-ID: <50225E45.3090305@linux.vnet.ibm.com> On 8/8/2012 4:00 PM, Alan Bateman wrote: > On 07/08/2012 14:17, Shirish Kuncolienkar wrote: >> >> I looked at the fix for 6346658 and that will fix the looping issue >> as reported. However I feel that there are still issues with regLock >> vs. keyLock as you pointed out that the synchronization is >> inconsistent. So I suggest that you look at the change set once you >> have some free time. >> -Shirish >> > I think that expanding the scope of regLock in the register and > configureBlocking methods is right. Changing implCloseChannel requires > a bit of consideration and I will try to look at it next week. I > expect that the proposed specification change will not be necessary > and that it will be solely an implementation change to synchronize on > regLock to avoid the case that someone registers a channel and gets a > valid SelectionKey at around the time that the channel is closed. > > -Alan. > I agree that the specification change is not necessary. The change in implCloseChannel could be limited to eliminating the possibility of getting a valid selection key while the channel is being closed. -Shirish From Alan.Bateman at oracle.com Wed Aug 8 06:28:05 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 08 Aug 2012 14:28:05 +0100 Subject: Code review request 7189886: improve AsynchronousChannelGroup tests to cover withThreadPool method In-Reply-To: <50224C9C.5080000@oracle.com> References: <50224C9C.5080000@oracle.com> Message-ID: <50226965.7020808@oracle.com> On 08/08/2012 12:25, Amy Lu wrote: > > While going through the missing coverage APIs, withThreadPool method > of AsynchronousChannelGroup is not covered by any test. > This is improvement for AsynchronousChannelGroup tests to cover this > method. > > Thanks Alan for help posting the webrev. Please review: > http://cr.openjdk.java.net/~alanb/7189886/webrev.00/ > > Thanks, > Amy > Thanks Amy, it's good to add coverage for this method. The changes mostly look good. The only thing is the Executors.newFixedThreadPool(rand.nextInt(5),factory) in Restart.java where it might be possible for nextInt to return 0 and so newFixedThreadPool will fail. I can fix this before pushing the change for you. -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120808/8da8c150/attachment.html From amy.lu at oracle.com Wed Aug 8 19:41:45 2012 From: amy.lu at oracle.com (Amy Lu) Date: Thu, 09 Aug 2012 10:41:45 +0800 Subject: Code review request 7189886: improve AsynchronousChannelGroup tests to cover withThreadPool method In-Reply-To: <50226965.7020808@oracle.com> References: <50224C9C.5080000@oracle.com> <50226965.7020808@oracle.com> Message-ID: <50232369.1020104@oracle.com> Thank you Alan ! Thanks, Amy On 8/8/12 9:28 PM, Alan Bateman wrote: > On 08/08/2012 12:25, Amy Lu wrote: >> >> While going through the missing coverage APIs, withThreadPool method >> of AsynchronousChannelGroup is not covered by any test. >> This is improvement for AsynchronousChannelGroup tests to cover this >> method. >> >> Thanks Alan for help posting the webrev. Please review: >> http://cr.openjdk.java.net/~alanb/7189886/webrev.00/ >> >> Thanks, >> Amy >> > Thanks Amy, it's good to add coverage for this method. > > The changes mostly look good. The only thing is the > Executors.newFixedThreadPool(rand.nextInt(5),factory) in Restart.java > where it might be possible for nextInt to return 0 and so > newFixedThreadPool will fail. I can fix this before pushing the change > for you. > > -Alan. > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120809/06575096/attachment.html From Alan.Bateman at oracle.com Thu Aug 9 06:28:44 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 09 Aug 2012 14:28:44 +0100 Subject: Code review request 7190219 CharBuffer position changed after BufferOverflowException in put() In-Reply-To: <5023AA2B.9030502@linux.vnet.ibm.com> References: <5023AA2B.9030502@linux.vnet.ibm.com> Message-ID: <5023BB0C.2080406@oracle.com> On 09/08/2012 13:16, Jonathan Lu wrote: > Hi folks, > > Here's a patch for bug 7190219, could you please help to have a look? > http://cr.openjdk.java.net/~luchsh/7190219/ > > According to the specification, in the method > java.nio.CharBuffer.put(String src, int start, int end). > If there are more chars to be copied from the string than remain in > this buffer, that is, if end - start > remaining(), then no chars are > transferred and a|BufferOverflowException||| > is thrown. > > But actually the test case from that webrev proves that the buffer was > modified even after BufferOverflowException, so I suggest to add > additional check after checkBounds() call, in the same way as > java.nio.CharBuffer.put(char[] src, int offset, int length). You're right, it's missing a check to make sure that there is remaining space and the proposed change looks right to me. Just to keep things locally consistent you can remove the braces around the throw new BufferOverflowException. I think the test could be improved. In particular it could duplicate the buffer and then check that it is equals to the duplicate after the put fails (in addition to checking that that the position hasn't changed). That way you really check that the buffer content haven't been modified. Do you mind seeing if you can add this to the existing unit test for the buffer classes rather than adding a new test? You'll see the existing tests in Basic.java and Basic-X.java.template. I see you've submitted an incident via bugs.sun.com for this, I've moved to the right place as: 7190219: (bf) CharBuffer.put(String,int,int) modifies position even if BufferOverflowException thrown -Alan. From david.lloyd at redhat.com Fri Aug 10 16:36:39 2012 From: david.lloyd at redhat.com (David M. Lloyd) Date: Fri, 10 Aug 2012 18:36:39 -0500 Subject: Problems persist in KQueueSelectorProvider (Mac) in 7u6 ea Message-ID: <50259B07.1080306@redhat.com> We're consistently seeing issues under load on Mac with KQueueSelectorProvider. There are two possibly related symptoms: the KQueueSelectorImpl is going into a mode where select() does not block, despite the continued emptiness of the selected key set; and FileDispatcherImpl#preClose0 is hanging, presumably in dup2(), trying to close a socket. My current hypothesis that some evil race condition exists and is being tripped between kqueue and dup2 (a relatively rare way to close a socket, at least until NIO came along I guess). My thought though is that sockets should not be preclosed this way: instead it would be better to use shutdown(fd, SHUT_RDWR), which would effectively preclose the socket and hopefully dodge this issue. I'm hopefully going to have time to try out a patch which does this, but I'm taking a couple weeks off starting tonight so I may not have time, so we shall see. -- - DML From david.lloyd at redhat.com Fri Aug 10 17:29:52 2012 From: david.lloyd at redhat.com (David M. Lloyd) Date: Fri, 10 Aug 2012 19:29:52 -0500 Subject: Problems persist in KQueueSelectorProvider (Mac) in 7u6 ea In-Reply-To: <50259B07.1080306@redhat.com> References: <50259B07.1080306@redhat.com> Message-ID: <5025A780.6010003@redhat.com> On 08/10/2012 06:36 PM, David M. Lloyd wrote: > We're consistently seeing issues under load on Mac with > KQueueSelectorProvider. > > There are two possibly related symptoms: the KQueueSelectorImpl is going > into a mode where select() does not block, despite the continued > emptiness of the selected key set; and FileDispatcherImpl#preClose0 is > hanging, presumably in dup2(), trying to close a socket. > > My current hypothesis that some evil race condition exists and is being > tripped between kqueue and dup2 (a relatively rare way to close a > socket, at least until NIO came along I guess). My thought though is > that sockets should not be preclosed this way: instead it would be > better to use shutdown(fd, SHUT_RDWR), which would effectively preclose > the socket and hopefully dodge this issue. > > I'm hopefully going to have time to try out a patch which does this, but > I'm taking a couple weeks off starting tonight so I may not have time, > so we shall see. Just took a sec and did up this patch: diff --git a/src/solaris/classes/sun/nio/ch/SocketDispatcher.java b/src/solaris/classes/sun/nio/ch/SocketDispatcher.java index 1c5f8ef..d7933a8 100644 --- a/src/solaris/classes/sun/nio/ch/SocketDispatcher.java +++ b/src/solaris/classes/sun/nio/ch/SocketDispatcher.java @@ -56,6 +56,6 @@ class SocketDispatcher extends NativeDispatcher } void preClose(FileDescriptor fd) throws IOException { - FileDispatcherImpl.preClose0(fd); + Net.shutdown(fd, Net.SHUT_RDWR); } } The good news is that it doesn't hang in preClose0 anymore (of course!)... the bad news is, now it hangs in close0, which seems to imply the problem is occuring when the socket is closed in the kernel. This, BTW, is the problem which seems to be related to the process being permanently frozen in "E" (exiting) state. -- - DML From jason.greene at redhat.com Sat Aug 11 21:50:54 2012 From: jason.greene at redhat.com (Jason T. Greene) Date: Sat, 11 Aug 2012 23:50:54 -0500 Subject: Problems persist in KQueueSelectorProvider (Mac) in 7u6 ea In-Reply-To: <50259B07.1080306@redhat.com> References: <50259B07.1080306@redhat.com> Message-ID: <5027362E.50006@redhat.com> I have been pretty short on time with paternity leave, but I was able to debug this a little bit further. One of the main issues is that interest ops is zero on keys that kqueue is returning, hence the tight loop. There appears to be thread safety issues with EV_ADD/EV_DELETE from multiple threads. Synchronizing KQueueArrayWrapper.setInterest(int,int) seems to resolve both issues, at least so far. I do wonder if an alternative approach using EV_ENABLE/EV_DISABLE might eliminate the need for extra locking. On 8/10/12 6:36 PM, David M. Lloyd wrote: > We're consistently seeing issues under load on Mac with > KQueueSelectorProvider. > > There are two possibly related symptoms: the KQueueSelectorImpl is going > into a mode where select() does not block, despite the continued > emptiness of the selected key set; and FileDispatcherImpl#preClose0 is > hanging, presumably in dup2(), trying to close a socket. > > My current hypothesis that some evil race condition exists and is being > tripped between kqueue and dup2 (a relatively rare way to close a > socket, at least until NIO came along I guess). My thought though is > that sockets should not be preclosed this way: instead it would be > better to use shutdown(fd, SHUT_RDWR), which would effectively preclose > the socket and hopefully dodge this issue. > > I'm hopefully going to have time to try out a patch which does this, but > I'm taking a couple weeks off starting tonight so I may not have time, > so we shall see. -- Jason T. Greene JBoss AS Lead / EAP Platform Architect JBoss, a division of Red Hat From jason.greene at redhat.com Sat Aug 11 22:27:38 2012 From: jason.greene at redhat.com (Jason T. Greene) Date: Sun, 12 Aug 2012 00:27:38 -0500 Subject: Problems persist in KQueueSelectorProvider (Mac) in 7u6 ea In-Reply-To: <5027362E.50006@redhat.com> References: <50259B07.1080306@redhat.com> <5027362E.50006@redhat.com> Message-ID: <50273ECA.8020002@redhat.com> I spoke too soon. This change does not work on all combinations of VM and OSX version. On 8/11/12 11:50 PM, Jason T. Greene wrote: > I have been pretty short on time with paternity leave, but I was able to > debug this a little bit further. One of the main issues is that interest > ops is zero on keys that kqueue is returning, hence the tight loop. > > There appears to be thread safety issues with EV_ADD/EV_DELETE from > multiple threads. Synchronizing KQueueArrayWrapper.setInterest(int,int) > seems to resolve both issues, at least so far. > > I do wonder if an alternative approach using EV_ENABLE/EV_DISABLE might > eliminate the need for extra locking. > > > On 8/10/12 6:36 PM, David M. Lloyd wrote: >> We're consistently seeing issues under load on Mac with >> KQueueSelectorProvider. >> >> There are two possibly related symptoms: the KQueueSelectorImpl is going >> into a mode where select() does not block, despite the continued >> emptiness of the selected key set; and FileDispatcherImpl#preClose0 is >> hanging, presumably in dup2(), trying to close a socket. >> >> My current hypothesis that some evil race condition exists and is being >> tripped between kqueue and dup2 (a relatively rare way to close a >> socket, at least until NIO came along I guess). My thought though is >> that sockets should not be preclosed this way: instead it would be >> better to use shutdown(fd, SHUT_RDWR), which would effectively preclose >> the socket and hopefully dodge this issue. >> >> I'm hopefully going to have time to try out a patch which does this, but >> I'm taking a couple weeks off starting tonight so I may not have time, >> so we shall see. > > -- Jason T. Greene JBoss AS Lead / EAP Platform Architect JBoss, a division of Red Hat From Alan.Bateman at oracle.com Mon Aug 13 01:11:11 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 13 Aug 2012 09:11:11 +0100 Subject: Problems persist in KQueueSelectorProvider (Mac) in 7u6 ea In-Reply-To: <50259B07.1080306@redhat.com> References: <50259B07.1080306@redhat.com> Message-ID: <5028B69F.5060907@oracle.com> On 11/08/2012 00:36, David M. Lloyd wrote: > We're consistently seeing issues under load on Mac with > KQueueSelectorProvider. > > There are two possibly related symptoms: the KQueueSelectorImpl is > going into a mode where select() does not block, despite the continued > emptiness of the selected key set; and FileDispatcherImpl#preClose0 is > hanging, presumably in dup2(), trying to close a socket. > > My current hypothesis that some evil race condition exists and is > being tripped between kqueue and dup2 (a relatively rare way to close > a socket, at least until NIO came along I guess). My thought though > is that sockets should not be preclosed this way: instead it would be > better to use shutdown(fd, SHUT_RDWR), which would effectively > preclose the socket and hopefully dodge this issue. > > I'm hopefully going to have time to try out a patch which does this, > but I'm taking a couple weeks off starting tonight so I may not have > time, so we shall see. The kqueue Selector that is in 7u6 was contributed by Apple via the macosx-port project. I believe (but can't be sure) that it's the same as what they have in their jdk6 and jdk5. It would be interesting to know if run into this issue with jdk6 (I don't know if it is possible for you to try that). I'm also interested to know where you tried 7u4 or 7u5. While we included the kqueue Selector in those releases, it wasn't used by default because the kqueue Selector was failing several of the OpenJDK Selector tests. We fixed those issues in 8 and 7u6 as part of restoring it to be the default Selector. To get another thing out of way, do you have any native code in this server that might be using file descriptors? (mentioning it on the off chance that somehow a non-socket has been registered). I think the most interesting issue from the above is the hang in preClose0. That is a dup2 so it's the same as close (and by the way, dup2 is used because it is not safe to close a socket and release the file descriptor in multi-threaded environments like this, we have been doing the same in classic networking since jdk1.4). I think we need to get a stack trace to know why preClose0 is hanging, I assume hanging in the kernel. We are aware of problems closing files when there is concurrent I/O, these cause a hang in the kernel in either dup2 or close. This is why a number of tests are currently excluded on Mac (same issues are observed with Apple's jdk6 too). If we can understand why dup2 is hanging in the kernel then it may explain the spin too. I see Jason's reply on setInterest and there is indeed a problem there. The specification is that changing the interest set is effective at the next select operation but this Selector is doing it asynchronously. This needs to be changed to batch the changes to the next select as is done in the other Selector implementations. I will create a bug for that. -Alan From luchsh at linux.vnet.ibm.com Mon Aug 13 01:26:52 2012 From: luchsh at linux.vnet.ibm.com (Jonathan Lu) Date: Mon, 13 Aug 2012 16:26:52 +0800 Subject: Code review request 7190219 CharBuffer position changed after BufferOverflowException in put() In-Reply-To: <5023BB0C.2080406@oracle.com> References: <5023AA2B.9030502@linux.vnet.ibm.com> <5023BB0C.2080406@oracle.com> Message-ID: <5028BA4C.2030405@linux.vnet.ibm.com> Hello Alan, Thanks for reviewing, I've updated the webrev, could you please take a look? http://cr.openjdk.java.net/~luchsh/7190219_2/ On 08/09/2012 09:28 PM, Alan Bateman wrote: > On 09/08/2012 13:16, Jonathan Lu wrote: >> Hi folks, >> >> Here's a patch for bug 7190219, could you please help to have a look? >> http://cr.openjdk.java.net/~luchsh/7190219/ >> >> According to the specification, in the method >> java.nio.CharBuffer.put(String src, int start, int end). >> If there are more chars to be copied from the string than remain in >> this buffer, that is, if end - start > remaining(), then no chars are >> transferred and a|BufferOverflowException||| >> is thrown. >> >> But actually the test case from that webrev proves that the buffer >> was modified even after BufferOverflowException, so I suggest to add >> additional check after checkBounds() call, in the same way as >> java.nio.CharBuffer.put(char[] src, int offset, int length). > You're right, it's missing a check to make sure that there is > remaining space and the proposed change looks right to me. Just to > keep things locally consistent you can remove the braces around the > throw new BufferOverflowException. > > I think the test could be improved. In particular it could duplicate > the buffer and then check that it is equals to the duplicate after the > put fails (in addition to checking that that the position hasn't > changed). That way you really check that the buffer content haven't > been modified. Do you mind seeing if you can add this to the existing > unit test for the buffer classes rather than adding a new test? You'll > see the existing tests in Basic.java and Basic-X.java.template. In the updated webrev, I'm using relGet() to perform the content checking after put(). And I also updated all the generated Basic.java files using genBasic.sh. > > I see you've submitted an incident via bugs.sun.com for this, I've > moved to the right place as: > > 7190219: (bf) CharBuffer.put(String,int,int) modifies position even if > BufferOverflowException thrown > > -Alan. > Thanks & regards Jonathan From Alan.Bateman at oracle.com Mon Aug 13 03:41:26 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 13 Aug 2012 11:41:26 +0100 Subject: Code review request 7190219 CharBuffer position changed after BufferOverflowException in put() In-Reply-To: <5028BA4C.2030405@linux.vnet.ibm.com> References: <5023AA2B.9030502@linux.vnet.ibm.com> <5023BB0C.2080406@oracle.com> <5028BA4C.2030405@linux.vnet.ibm.com> Message-ID: <5028D9D6.8070901@oracle.com> On 13/08/2012 09:26, Jonathan Lu wrote: > Hello Alan, > > Thanks for reviewing, I've updated the webrev, could you please take a > look? > > http://cr.openjdk.java.net/~luchsh/7190219_2/ > > : > In the updated webrev, I'm using relGet() to perform the content > checking after put(). > And I also updated all the generated Basic.java files using > genBasic.sh. > This looks good to me and thanks for adding the test to to the main unit test for this area. Do you mind adding the bugID to the list in Basic.java before you push this? Also, minor nit, can add a space "//String ops" then it will keep the comments locally consistent (no need to re-generate the webrev on these points). -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120813/9b5ce01c/attachment.html From luchsh at linux.vnet.ibm.com Mon Aug 13 04:55:50 2012 From: luchsh at linux.vnet.ibm.com (Jonathan Lu) Date: Mon, 13 Aug 2012 19:55:50 +0800 Subject: Code review request 7190219 CharBuffer position changed after BufferOverflowException in put() In-Reply-To: <5028D9D6.8070901@oracle.com> References: <5023AA2B.9030502@linux.vnet.ibm.com> <5023BB0C.2080406@oracle.com> <5028BA4C.2030405@linux.vnet.ibm.com> <5028D9D6.8070901@oracle.com> Message-ID: <5028EB46.1060500@linux.vnet.ibm.com> On 08/13/2012 06:41 PM, Alan Bateman wrote: > On 13/08/2012 09:26, Jonathan Lu wrote: >> Hello Alan, >> >> Thanks for reviewing, I've updated the webrev, could you please take >> a look? >> >> http://cr.openjdk.java.net/~luchsh/7190219_2/ >> >> : >> In the updated webrev, I'm using relGet() to perform the content >> checking after put(). >> And I also updated all the generated Basic.java files using >> genBasic.sh. >> > This looks good to me and thanks for adding the test to to the main > unit test for this area. Do you mind adding the bugID to the list in > Basic.java before you push this? Also, minor nit, can add a space > "//String ops" then it will keep the comments locally consistent (no > need to re-generate the webrev on these points). > > -Alan > Thanks, Alan. I've pushed the changes. Best regards Jonathan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120813/7a5aea6e/attachment.html From Alan.Bateman at oracle.com Mon Aug 13 06:01:59 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 13 Aug 2012 14:01:59 +0100 Subject: Code review request 7132889: Possible race conditions in AbstractSelectableChannel In-Reply-To: <50225E45.3090305@linux.vnet.ibm.com> References: <5020E11F.8020007@linux.vnet.ibm.com> <5020E6C7.90501@oracle.com> <50211571.8050900@linux.vnet.ibm.com> <50223FD2.1090401@oracle.com> <50225E45.3090305@linux.vnet.ibm.com> Message-ID: <5028FAC7.7080708@oracle.com> On 08/08/2012 13:40, Shirish Kuncolienkar wrote: > > I agree that the specification change is not necessary. The change in > implCloseChannel could be limited to eliminating the possibility of > getting a valid selection key while the channel is being closed. > -Shirish > I think the remaining changes are correct. Can you re-generate the webrev so that we have the final patch and I will sponsor this one? -Alan. From david.lloyd at redhat.com Mon Aug 13 07:06:01 2012 From: david.lloyd at redhat.com (David M. Lloyd) Date: Mon, 13 Aug 2012 10:06:01 -0400 Subject: Problems persist in KQueueSelectorProvider (Mac) in 7u6 ea In-Reply-To: <5028B69F.5060907@oracle.com> References: <50259B07.1080306@redhat.com> <5028B69F.5060907@oracle.com> Message-ID: <502909C9.2040107@redhat.com> Thanks for replying, responses inline. On 8/13/12 4:11 AM, Alan Bateman wrote: > On 11/08/2012 00:36, David M. Lloyd wrote: >> We're consistently seeing issues under load on Mac with >> KQueueSelectorProvider. >> >> There are two possibly related symptoms: the KQueueSelectorImpl is >> going into a mode where select() does not block, despite the continued >> emptiness of the selected key set; and FileDispatcherImpl#preClose0 is >> hanging, presumably in dup2(), trying to close a socket. >> >> My current hypothesis that some evil race condition exists and is >> being tripped between kqueue and dup2 (a relatively rare way to close >> a socket, at least until NIO came along I guess). My thought though is >> that sockets should not be preclosed this way: instead it would be >> better to use shutdown(fd, SHUT_RDWR), which would effectively >> preclose the socket and hopefully dodge this issue. >> >> I'm hopefully going to have time to try out a patch which does this, >> but I'm taking a couple weeks off starting tonight so I may not have >> time, so we shall see. > The kqueue Selector that is in 7u6 was contributed by Apple via the > macosx-port project. I believe (but can't be sure) that it's the same as > what they have in their jdk6 and jdk5. It would be interesting to know > if run into this issue with jdk6 (I don't know if it is possible for you > to try that). We did test on 6 with similar results. > I'm also interested to know where you tried 7u4 or 7u5. While we > included the kqueue Selector in those releases, it wasn't used by > default because the kqueue Selector was failing several of the OpenJDK > Selector tests. We fixed those issues in 8 and 7u6 as part of restoring > it to be the default Selector. We've tested on the u6 tip as well as on previous 7 versions (we explicitly choose the selector provider), with similar results. > To get another thing out of way, do you have any native code in this > server that might be using file descriptors? (mentioning it on the off > chance that somehow a non-socket has been registered). No, though we do plan on using pipes in the future (though if NIO implements pipes as half-duplex socketpairs that'd be fine, I imagine; I'll have to review that code). I do believe that the kqueue provider is using pipe() for wakeup implementation though... perhaps that is an issue if non-sockets are no good. > I think the most interesting issue from the above is the hang in > preClose0. That is a dup2 so it's the same as close (and by the way, > dup2 is used because it is not safe to close a socket and release the > file descriptor in multi-threaded environments like this, we have been > doing the same in classic networking since jdk1.4). I think we need to > get a stack trace to know why preClose0 is hanging, I assume hanging in > the kernel. It is definitely hanging in dup2(), as dup2() is the only thing that preClose0 actually does. As I said I implemented a test to use shutdown() instead of dup2() for preClose0 but that just moved the problem to the real close(). > We are aware of problems closing files when there is > concurrent I/O, these cause a hang in the kernel in either dup2 or > close. This is why a number of tests are currently excluded on Mac (same > issues are observed with Apple's jdk6 too). If we can understand why > dup2 is hanging in the kernel then it may explain the spin too. I suspect that while both problems are related to kqueue bugs, the spin is specifically related to how setInterest works, whereas the dup2 hang/stuck process is something separate. One workaround I plan to test when I return from vacation is to avoid closing the socket until after it has been deregistered from all Selectors. > I see Jason's reply on setInterest and there is indeed a problem there. > The specification is that changing the interest set is effective at the > next select operation but this Selector is doing it asynchronously. This > needs to be changed to batch the changes to the next select as is done > in the other Selector implementations. I will create a bug for that. Yes, I believe this is a separate problem which may account for the spin. -- - DML From Alan.Bateman at oracle.com Mon Aug 13 07:51:57 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 13 Aug 2012 15:51:57 +0100 Subject: Problems persist in KQueueSelectorProvider (Mac) in 7u6 ea In-Reply-To: <502909C9.2040107@redhat.com> References: <50259B07.1080306@redhat.com> <5028B69F.5060907@oracle.com> <502909C9.2040107@redhat.com> Message-ID: <5029148C.1000507@oracle.com> On 13/08/2012 15:06, David M. Lloyd wrote: > : > We did test on 6 with similar results. > > We've tested on the u6 tip as well as on previous 7 versions (we > explicitly choose the selector provider), with similar results. Thanks for checking, at least this shows that we aren't dealing with something new. > : > > It is definitely hanging in dup2(), as dup2() is the only thing that > preClose0 actually does. As I said I implemented a test to use > shutdown() instead of dup2() for preClose0 but that just moved the > problem to the real close(). One thing that would be useful is a thread dump. A stack trace with the kernel frames would be useful too but may not be possible with stackshot, maybe need to come up with a dtrace script. At least with the fifo case, where we have also observed close or dup2 to hang, the issue is that it blocks waiting for an I/O on the fifo to complete. > > I suspect that while both problems are related to kqueue bugs, the > spin is specifically related to how setInterest works, whereas the > dup2 hang/stuck process is something separate. They may be related too. In particular if dup2 has caused the file descriptor to reference the pre-close socket then it maye cause any Selector that it is registered with to spin until the key has been canceled. -Alan. From Alan.Bateman at oracle.com Mon Aug 13 08:23:42 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 13 Aug 2012 16:23:42 +0100 Subject: Code review request 7132889: Possible race conditions in AbstractSelectableChannel In-Reply-To: <5028FAC7.7080708@oracle.com> References: <5020E11F.8020007@linux.vnet.ibm.com> <5020E6C7.90501@oracle.com> <50211571.8050900@linux.vnet.ibm.com> <50223FD2.1090401@oracle.com> <50225E45.3090305@linux.vnet.ibm.com> <5028FAC7.7080708@oracle.com> Message-ID: <50291BFE.4030102@oracle.com> On 13/08/2012 14:01, Alan Bateman wrote: > On 08/08/2012 13:40, Shirish Kuncolienkar wrote: >> >> I agree that the specification change is not necessary. The change in >> implCloseChannel could be limited to eliminating the possibility of >> getting a valid selection key while the channel is being closed. >> -Shirish >> > I think the remaining changes are correct. Can you re-generate the > webrev so that we have the final patch and I will sponsor this one? On second thoughts, as the regLock is the blockingLock then I don't think it can be acquired by implCloseChannel so the change will need to limited to register and configure. One thing that would be good to do is to run the tests (in test/java/nio/channels) with your changes to make sure that there aren't any issues. -Alan. From jason.greene at redhat.com Mon Aug 13 12:13:49 2012 From: jason.greene at redhat.com (Jason T. Greene) Date: Mon, 13 Aug 2012 14:13:49 -0500 Subject: Problems persist in KQueueSelectorProvider (Mac) in 7u6 ea In-Reply-To: <5028B69F.5060907@oracle.com> References: <50259B07.1080306@redhat.com> <5028B69F.5060907@oracle.com> Message-ID: <502951ED.4060704@redhat.com> On 8/13/12 3:11 AM, Alan Bateman wrote: -snip- > I see Jason's reply on setInterest and there is indeed a problem there. > The specification is that changing the interest set is effective at the > next select operation but this Selector is doing it asynchronously. This > needs to be changed to batch the changes to the next select as is done > in the other Selector implementations. I will create a bug for that. Ah that should have been obvious to me! Thanks for that. I threw together a quick patch (attached below against 7u-dev), and it appears to resolve the issues on 6 and 7. Although it needs more testing. The preClose problem disappears as well with this change. My hunch is that the tight spin loop on kqueue makes it impossible for close/dup2 to ever complete. -- Jason T. Greene JBoss AS Lead / EAP Platform Architect JBoss, a division of Red Hat -------------- next part -------------- diff -r 043839897cd9 src/macosx/classes/sun/nio/ch/KQueueArrayWrapper.java --- a/src/macosx/classes/sun/nio/ch/KQueueArrayWrapper.java Thu Aug 09 20:46:39 2012 -0700 +++ b/src/macosx/classes/sun/nio/ch/KQueueArrayWrapper.java Mon Aug 13 14:04:08 2012 -0500 @@ -34,7 +34,8 @@ import sun.misc.*; import java.io.IOException; import java.io.FileDescriptor; - +import java.util.Iterator; +import java.util.LinkedList; /* * struct kevent { // 32-bit 64-bit @@ -99,6 +100,18 @@ kq = init(); } + // Used to update file description registrations + private static class Update { + SelChImpl channel; + int events; + Update(SelChImpl channel, int events) { + this.channel = channel; + this.events = events; + } + } + + private LinkedList updateList = new LinkedList(); + void initInterrupt(int fd0, int fd1) { outgoingInterruptFD = fd1; incomingInterruptFD = fd0; @@ -136,14 +149,41 @@ } } - void setInterest(int fd, int events) { - register0(kq, fd, events & POLLIN, events & POLLOUT); + void setInterest(SelChImpl channel, int events) { + synchronized (updateList) { + // update existing registration + updateList.add(new Update(channel, events)); + } } - void release(int fd) { - register0(kq, fd, 0, 0); + void release(SelChImpl channel) { + synchronized (updateList) { + // flush any pending updates + for (Iterator it = updateList.iterator(); it.hasNext();) { + if (it.next().channel == channel) { + it.remove(); + } + } + + // remove + register0(kq, channel.getFDVal(), 0, 0); + } } + void updateRegistrations() { + synchronized (updateList) { + Update u = null; + while ((u = updateList.poll()) != null) { + SelChImpl ch = u.channel; + if (!ch.isOpen()) + continue; + + register0(kq, ch.getFDVal(), u.events & POLLIN, u.events & POLLOUT); + } + } + } + + void close() throws IOException { if (keventArray != null) { keventArray.free(); @@ -156,6 +196,7 @@ } int poll(long timeout) { + updateRegistrations(); int updated = kevent0(kq, keventArrayAddress, NUM_KEVENTS, timeout); return updated; } diff -r 043839897cd9 src/macosx/classes/sun/nio/ch/KQueueSelectorImpl.java --- a/src/macosx/classes/sun/nio/ch/KQueueSelectorImpl.java Thu Aug 09 20:46:39 2012 -0700 +++ b/src/macosx/classes/sun/nio/ch/KQueueSelectorImpl.java Mon Aug 13 14:04:08 2012 -0500 @@ -86,7 +86,7 @@ fd1 = (int)fds; kqueueWrapper = new KQueueArrayWrapper(); kqueueWrapper.initInterrupt(fd0, fd1); - fdMap = new HashMap<>(); + fdMap = new HashMap(); totalChannels = 1; } @@ -184,7 +184,6 @@ FileDispatcherImpl.closeIntFD(fd0); FileDispatcherImpl.closeIntFD(fd1); if (kqueueWrapper != null) { - kqueueWrapper.release(fd0); kqueueWrapper.close(); kqueueWrapper = null; selectedKeys = null; @@ -220,7 +219,7 @@ protected void implDereg(SelectionKeyImpl ski) throws IOException { int fd = ski.channel.getFDVal(); fdMap.remove(Integer.valueOf(fd)); - kqueueWrapper.release(fd); + kqueueWrapper.release(ski.channel); totalChannels--; keys.remove(ski); selectedKeys.remove(ski); @@ -234,8 +233,7 @@ public void putEventOps(SelectionKeyImpl ski, int ops) { if (closed) throw new ClosedSelectorException(); - int fd = IOUtil.fdVal(ski.channel.getFD()); - kqueueWrapper.setInterest(fd, ops); + kqueueWrapper.setInterest(ski.channel, ops); } From schlosna at gmail.com Mon Aug 13 14:24:59 2012 From: schlosna at gmail.com (David Schlosnagle) Date: Mon, 13 Aug 2012 17:24:59 -0400 Subject: Problems persist in KQueueSelectorProvider (Mac) in 7u6 ea In-Reply-To: <502951ED.4060704@redhat.com> References: <50259B07.1080306@redhat.com> <5028B69F.5060907@oracle.com> <502951ED.4060704@redhat.com> Message-ID: On Mon, Aug 13, 2012 at 3:13 PM, Jason T. Greene wrote: > I threw together a quick patch (attached below against 7u-dev), and it > appears to resolve the issues on 6 and 7. Although it needs more testing. > > The preClose problem disappears as well with this change. My hunch is that > the tight spin loop on kqueue makes it impossible for close/dup2 to ever > complete. Hi Jason, A few minor comments/questions in KQueueArrayWrapper.java: Should the updateList and Update instance fields all be final? Is there any reason not to avoid the continue in updateRegistrations()? while ((u = updateList.poll()) != null) { SelChImpl ch = u.channel; if (ch.isOpen()) { register0(kq, ch.getFDVal(), u.events & POLLIN, u.events & POLLOUT); } } Do you foresee any lock contention issues with the synchronization on updateList? Would using a java.util.concurrent.LinkedBlockingQueue rather than the LinkedList with synchronization violate the API semantics of batching the updates? I was thinking something along the lines of the KQueueArrayWrapper.java diff below would work (but I haven't tested at all). Thanks, Dave diff --git a/KQueueArrayWrapper.java b/KQueueArrayWrapper.java index 5a7020c..936f086 100644 --- a/KQueueArrayWrapper.java +++ b/KQueueArrayWrapper.java @@ -34,7 +34,9 @@ package sun.nio.ch; import sun.misc.*; import java.io.IOException; import java.io.FileDescriptor; - +import java.util.Iterator; +import java.util.Queue; +import java.util.concurrent.LinkedBlockingQueue; /* * struct kevent { // 32-bit 64-bit @@ -85,6 +87,9 @@ class KQueueArrayWrapper { // The fd of the interrupt line coming in private int incomingInterruptFD; + // The queue of file descriptor registration updates to process + private final Queue updateQueue = new LinkedBlockingQueue<>(); + static { initStructSizes(); String datamodel = java.security.AccessController.doPrivileged( @@ -100,6 +105,16 @@ class KQueueArrayWrapper { kq = init(); } + // Used to update file description registrations + private static class Update { + final SelChImpl channel; + final int events; + Update(SelChImpl channel, int events) { + this.channel = channel; + this.events = events; + } + } + void initInterrupt(int fd0, int fd1) { outgoingInterruptFD = fd1; incomingInterruptFD = fd0; @@ -137,12 +152,31 @@ class KQueueArrayWrapper { } } - void setInterest(int fd, int events) { - register0(kq, fd, events & POLLIN, events & POLLOUT); + void setInterest(SelChImpl channel, int events) { + // update existing registration + updateQueue.add(new Update(channel, events)); } - void release(int fd) { - register0(kq, fd, 0, 0); + void release(SelChImpl channel) { + // flush any pending updates + for (Iterator it = updateQueue.iterator(); it.hasNext();) { + if (it.next().channel == channel) { + it.remove(); + } + } + + // remove + register0(kq, channel.getFDVal(), 0, 0); + } + + void updateRegistrations() { + Update u = null; + while ((u = updateQueue.poll()) != null) { + SelChImpl ch = u.channel; + if (ch.isOpen()) { + register0(kq, ch.getFDVal(), u.events & POLLIN, u.events & POLLOUT); + } + } } void close() throws IOException { @@ -157,6 +191,7 @@ class KQueueArrayWrapper { } int poll(long timeout) { + updateRegistrations(); int updated = kevent0(kq, keventArrayAddress, NUM_KEVENTS, timeout); return updated; } From Alan.Bateman at oracle.com Mon Aug 13 14:42:16 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 13 Aug 2012 22:42:16 +0100 Subject: Problems persist in KQueueSelectorProvider (Mac) in 7u6 ea In-Reply-To: <502951ED.4060704@redhat.com> References: <50259B07.1080306@redhat.com> <5028B69F.5060907@oracle.com> <502951ED.4060704@redhat.com> Message-ID: <502974B8.1080206@oracle.com> On 13/08/2012 20:13, Jason T. Greene wrote: > On 8/13/12 3:11 AM, Alan Bateman wrote: > -snip- >> I see Jason's reply on setInterest and there is indeed a problem there. >> The specification is that changing the interest set is effective at the >> next select operation but this Selector is doing it asynchronously. This >> needs to be changed to batch the changes to the next select as is done >> in the other Selector implementations. I will create a bug for that. > > Ah that should have been obvious to me! Thanks for that. > > I threw together a quick patch (attached below against 7u-dev), and it > appears to resolve the issues on 6 and 7. Although it needs more testing. > > The preClose problem disappears as well with this change. My hunch is > that the tight spin loop on kqueue makes it impossible for close/dup2 > to ever complete. > Thanks for trying that out, it's good to hear that the problems have gone away. On the tight loop theory then can you check if a small sleep, or may a Thread.yield, causes the dup2 hang to go away too? On the patch, are you submitting it here as a contribution (once you are done with your testing)? I haven't looked at it closely yet but I think it is close to what we have in the epoll Selector. -Alan. From jgreene at redhat.com Mon Aug 13 14:50:31 2012 From: jgreene at redhat.com (Jason Greene) Date: Mon, 13 Aug 2012 17:50:31 -0400 (EDT) Subject: Problems persist in KQueueSelectorProvider (Mac) in 7u6 ea In-Reply-To: <502974B8.1080206@oracle.com> References: <50259B07.1080306@redhat.com> <5028B69F.5060907@oracle.com> <502951ED.4060704@redhat.com> <502974B8.1080206@oracle.com> Message-ID: <46F92297-591C-4A6E-85CF-2943A4A3F099@redhat.com> On Aug 13, 2012, at 4:42 PM, Alan Bateman wrote: > On 13/08/2012 20:13, Jason T. Greene wrote: >> On 8/13/12 3:11 AM, Alan Bateman wrote: >> -snip- >>> I see Jason's reply on setInterest and there is indeed a problem there. >>> The specification is that changing the interest set is effective at the >>> next select operation but this Selector is doing it asynchronously. This >>> needs to be changed to batch the changes to the next select as is done >>> in the other Selector implementations. I will create a bug for that. >> >> Ah that should have been obvious to me! Thanks for that. >> >> I threw together a quick patch (attached below against 7u-dev), and it appears to resolve the issues on 6 and 7. Although it needs more testing. >> >> The preClose problem disappears as well with this change. My hunch is that the tight spin loop on kqueue makes it impossible for close/dup2 to ever complete. >> > Thanks for trying that out, it's good to hear that the problems have gone away. On the tight loop theory then can you check if a small sleep, or may a Thread.yield, causes the dup2 hang to go away too? Sure I can give that a try. I'll try to get the dump you requested earlier if not. > > On the patch, are you submitting it here as a contribution (once you are done with your testing)? I haven't looked at it closely yet but I think it is close to what we have in the epoll Selector. Yes I just reused the same approach that epoll is using with slight alterations, since it seems to work well. The patch if useful is a contribution, although as you noticed its mostly a derivative. > > -Alan. > > From shirishk at linux.vnet.ibm.com Mon Aug 13 19:19:03 2012 From: shirishk at linux.vnet.ibm.com (Shirish Kuncolienkar) Date: Tue, 14 Aug 2012 07:49:03 +0530 Subject: Code review request 7132889: Possible race conditions in AbstractSelectableChannel In-Reply-To: <50291BFE.4030102@oracle.com> References: <5020E11F.8020007@linux.vnet.ibm.com> <5020E6C7.90501@oracle.com> <50211571.8050900@linux.vnet.ibm.com> <50223FD2.1090401@oracle.com> <50225E45.3090305@linux.vnet.ibm.com> <5028FAC7.7080708@oracle.com> <50291BFE.4030102@oracle.com> Message-ID: <5029B597.4080709@linux.vnet.ibm.com> On 8/13/2012 8:53 PM, Alan Bateman wrote: > On 13/08/2012 14:01, Alan Bateman wrote: >> On 08/08/2012 13:40, Shirish Kuncolienkar wrote: >>> >>> I agree that the specification change is not necessary. The change >>> in implCloseChannel could be limited to eliminating the possibility >>> of getting a valid selection key while the channel is being closed. >>> -Shirish >>> >> I think the remaining changes are correct. Can you re-generate the >> webrev so that we have the final patch and I will sponsor this one? > On second thoughts, as the regLock is the blockingLock then I don't > think it can be acquired by implCloseChannel so the change will need > to limited to register and configure. One thing that would be good to > do is to run the tests (in test/java/nio/channels) with your changes > to make sure that there aren't any issues. > > -Alan. > I see that regLock is not limited to blocking mode. Many places the keyLock is acquired while the regLock is held. If we do not guard the implCloseSelectableChannel() and key cancellation within the regLock we would not be able to avoid a key being added while a channel is closed. Do you think such a behavior should be documented ? I will run the channel tests as suggested by you. -Shirish From Alan.Bateman at oracle.com Tue Aug 14 01:31:24 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 14 Aug 2012 09:31:24 +0100 Subject: Code review request 7132889: Possible race conditions in AbstractSelectableChannel In-Reply-To: <5029B597.4080709@linux.vnet.ibm.com> References: <5020E11F.8020007@linux.vnet.ibm.com> <5020E6C7.90501@oracle.com> <50211571.8050900@linux.vnet.ibm.com> <50223FD2.1090401@oracle.com> <50225E45.3090305@linux.vnet.ibm.com> <5028FAC7.7080708@oracle.com> <50291BFE.4030102@oracle.com> <5029B597.4080709@linux.vnet.ibm.com> Message-ID: <502A0CDC.4020604@oracle.com> On 14/08/2012 03:19, Shirish Kuncolienkar wrote: > > I see that regLock is not limited to blocking mode. Many places the > keyLock is acquired while the regLock is held. It guards the blocking mode, and may have been clearer if it were named blockingLock. Locking regLock and keyLock (in that order) should be fine, just not in implCloseSelectableChannel because you need to be able to invoke close when there is a blocking operation in progress. > If we do not guard the implCloseSelectableChannel() and key > cancellation within the regLock we would not be able to avoid a key > being added while a channel is closed. Do you think such a behavior > should be documented ? > I will run the channel tests as suggested by you. Yes, it would be great if you could run the tests with your patch as I am concerned that it will break asynchronous close causing several tests will either hang or deadlock. I think the residual issue with register returning a valid key can be solved by checking the channel status when holding the keyLock, eg: synchronized (keyLock) { if (!isOpen()) throw new ClosedChannelException(); addKey(k); } and change addKey to assert Thread.holdsLock(keyLock); So I'm curious as to the background to this issue, is this a patch that you have in the IBM JDK? -Alan. From Alan.Bateman at oracle.com Wed Aug 15 04:20:14 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 15 Aug 2012 12:20:14 +0100 Subject: 7191467: (fs) WatchService periodically fails to queue ENTRY_DELETE event for short lived file [sol11] Message-ID: <502B85EE.9090909@oracle.com> MayFlies is one of the tests for the WatchService, it's a stress test that ensures that there are corresponding ENTRY_DELETE events for each ENTRY_CREATE event when the files are very short lived. This test fails periodically on Solaris 11 where it appears that FILE_REMOVE events aren't being queued by the kernel. There are a couple of workarounds in this code to deal with a kernel issues in this area and I've decided it to be best to just remove these and change the implementation so that it detects file deletions when scanning the directory (which it has to do anyway when the directory changes). The webrev with the changes is here: http://cr.openjdk.java.net/~alanb/7191467/webrev/ The main change is that entries in directories are only registered with the kernel when the directory is registered for ENTRY_MODIFY events. ENTRY_CREATE and ENTRY_DELETE events are now completely triggered on changes to the directory. One other change in here, and it stemmed from a discussion on this mail a few months ago, is that if there is an error encountered when iterating over the directory then an OVERFLOW event is queued. That is the trigger to the user application to re-scan the directory to update its view of the directory. -Alan From Alan.Bateman at oracle.com Wed Aug 15 04:43:58 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 15 Aug 2012 12:43:58 +0100 Subject: 7191556: (fs) UnixNativeDispatcher.getextmntent should be moved into platform specific code Message-ID: <502B8B7E.2030600@oracle.com> This one has been in my patch queue for a long time, I'd actually forgotten about it. It's also just a clean-up covering two issues: 1. UnixNativeDispatcher defines getextmntent to read an entry from the mount table but it's not very portable with lots of #ifdef for each platform. The change moves this method down to SolarisNativeDispatcher and adds a getmntent to LinuxNativeDispatcher so it's much cleaner. 2. When the Mac port was added then it included the GNOME FileTypeDetector which I don't think makes sense because it will never be used. The clean-up here is to not build this FileTypeDetector on Mac. The DefaultFileTypeDetector that is compiled when building on Solaris/Linux/Mac is changed so that it delegates to the FileSystemProvider to get the file type detector and this ensures that the proper implementation is included. As I say, this is solely clean-up so there isn't a test. The webrev with the changes is here: http://cr.openjdk.java.net/~alanb/7191556/webrev/ -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120815/5ad10099/attachment.html From Alan.Bateman at oracle.com Wed Aug 15 05:05:55 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 15 Aug 2012 13:05:55 +0100 Subject: 7050570: (fs) FileSysteProvider fails to initializes if run with file.encoding set to Cp037 Message-ID: <502B90A3.2020601@oracle.com> This is another one that has been in my patch queue for a while, actually from before jdk7 shipped as it arose with someone trying tests with file.encoding set on the command line (something that has never been supported but sometimes useful for tests). It's also an issue that came up recently on the core-libs list [1] where someone was trying to do something similar. The changes here just change the file system provider to use sun.jnu.encoding property as the charset to use when encoding/decoding file names. I think eventually we should be use this consistently and leaving file.encoding to the file content. The assumption remains in the code that there aren't any locales that map to UTF-16 or other encoding that require shift in/out. If there were such a locale then more further changes would be required in order to deal with multi-byte slash a few others used in the code. Sherman - I think you would be the best person to review this. The webrev with the changes is here: http://cr.openjdk.java.net/~alanb/7050570/webrev/ -Alan. [1] http://mail.openjdk.java.net/pipermail/core-libs-dev/2012-July/010758.html -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120815/bb3e6540/attachment.html From vitalyd at gmail.com Wed Aug 15 05:50:39 2012 From: vitalyd at gmail.com (Vitaly Davidovich) Date: Wed, 15 Aug 2012 08:50:39 -0400 Subject: 7191467: (fs) WatchService periodically fails to queue ENTRY_DELETE event for short lived file [sol11] In-Reply-To: <502B85EE.9090909@oracle.com> References: <502B85EE.9090909@oracle.com> Message-ID: Hi Alan, SolarisWatchKey.getChild() doesn't need the null check anymore. Thanks Sent from my phone On Aug 15, 2012 7:20 AM, "Alan Bateman" wrote: > > MayFlies is one of the tests for the WatchService, it's a stress test that > ensures that there are corresponding ENTRY_DELETE events for each > ENTRY_CREATE event when the files are very short lived. This test fails > periodically on Solaris 11 where it appears that FILE_REMOVE events aren't > being queued by the kernel. There are a couple of workarounds in this code > to deal with a kernel issues in this area and I've decided it to be best to > just remove these and change the implementation so that it detects file > deletions when scanning the directory (which it has to do anyway when the > directory changes). > > The webrev with the changes is here: > > http://cr.openjdk.java.net/~**alanb/7191467/webrev/ > > The main change is that entries in directories are only registered with > the kernel when the directory is registered for ENTRY_MODIFY events. > ENTRY_CREATE and ENTRY_DELETE events are now completely triggered on > changes to the directory. > > One other change in here, and it stemmed from a discussion on this mail a > few months ago, is that if there is an error encountered when iterating > over the directory then an OVERFLOW event is queued. That is the trigger to > the user application to re-scan the directory to update its view of the > directory. > > -Alan > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120815/291cc57b/attachment.html From Alan.Bateman at oracle.com Wed Aug 15 06:42:50 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 15 Aug 2012 14:42:50 +0100 Subject: Problems persist in KQueueSelectorProvider (Mac) in 7u6 ea In-Reply-To: <46F92297-591C-4A6E-85CF-2943A4A3F099@redhat.com> References: <50259B07.1080306@redhat.com> <5028B69F.5060907@oracle.com> <502951ED.4060704@redhat.com> <502974B8.1080206@oracle.com> <46F92297-591C-4A6E-85CF-2943A4A3F099@redhat.com> Message-ID: <502BA75A.9020101@oracle.com> On 13/08/2012 22:50, Jason Greene wrote: > On Aug 13, 2012, at 4:42 PM, Alan Bateman wrote: > >> : >> Thanks for trying that out, it's good to hear that the problems have gone away. On the tight loop theory then can you check if a small sleep, or may a Thread.yield, causes the dup2 hang to go away too? > Sure I can give that a try. I'll try to get the dump you requested earlier if not. That would be good. > : >> On the patch, are you submitting it here as a contribution (once you are done with your testing)? I haven't looked at it closely yet but I think it is close to what we have in the epoll Selector. > Yes I just reused the same approach that epoll is using with slight alterations, since it seems to work well. The patch if useful is a contribution, although as you noticed its mostly a derivative. > It make sense too. I've created a bug to track this: 7191587: (se) SelectionKey.interestOps does not defer changing the interest set to the next select [macosx] and once you are done with your testing we will look to get this into jdk8. Once we are satisfied they we can seek approval for 7u8. -Alan From Alan.Bateman at oracle.com Wed Aug 15 06:43:40 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 15 Aug 2012 14:43:40 +0100 Subject: 7191467: (fs) WatchService periodically fails to queue ENTRY_DELETE event for short lived file [sol11] In-Reply-To: References: <502B85EE.9090909@oracle.com> Message-ID: <502BA78C.5040209@oracle.com> On 15/08/2012 13:50, Vitaly Davidovich wrote: > > Hi Alan, > > SolarisWatchKey.getChild() doesn't need the null check anymore. > > Thanks > You're right, it can no longer ever be null so this check can be removed. Thanks for looking at the changes so closely. -Alan From ahughes at redhat.com Wed Aug 15 07:18:02 2012 From: ahughes at redhat.com (Andrew Hughes) Date: Wed, 15 Aug 2012 10:18:02 -0400 (EDT) Subject: 7191556: (fs) UnixNativeDispatcher.getextmntent should be moved into platform specific code In-Reply-To: <502B8B7E.2030600@oracle.com> Message-ID: <1936535749.5322045.1345040282195.JavaMail.root@redhat.com> ----- Original Message ----- > > This one has been in my patch queue for a long time, I'd actually > forgotten about it. It's also just a clean-up covering two issues: > > 1. UnixNativeDispatcher defines getextmntent to read an entry from > the mount table but it's not very portable with lots of #ifdef for > each platform. The change moves this method down to > SolarisNativeDispatcher and adds a getmntent to > LinuxNativeDispatcher so it's much cleaner. > > 2. When the Mac port was added then it included the GNOME > FileTypeDetector which I don't think makes sense because it will > never be used. The clean-up here is to not build this > FileTypeDetector on Mac. The DefaultFileTypeDetector that is > compiled when building on Solaris/Linux/Mac is changed so that it > delegates to the FileSystemProvider to get the file type detector > and this ensures that the proper implementation is included. > > As I say, this is solely clean-up so there isn't a test. The webrev > with the changes is here: > > http://cr.openjdk.java.net/~alanb/7191556/webrev/ > > -Alan. > This looks good to me and built ok here. My only concern is that this now means that Macs now always get a null file type detector, whereas there might have been a working one if glib was installed. The name "GNOMEFileTypeDetector" is a bit of a misnomer as GIO is part of glib, which is intentionally multi-platform (even Windows, I believe). Maybe worth asking on the BSD and Mac lists? It might motivate them to provide a solution using the native Mac API (assuming there is one) ;-) Thanks, -- Andrew :) Free Java Software Engineer Red Hat, Inc. (http://www.redhat.com) PGP Key: 248BDC07 (https://keys.indymedia.org/) Fingerprint = EC5A 1F5E C0AD 1D15 8F1F 8F91 3B96 A578 248B DC07 From Alan.Bateman at oracle.com Wed Aug 15 07:36:17 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 15 Aug 2012 15:36:17 +0100 Subject: 7191556: (fs) UnixNativeDispatcher.getextmntent should be moved into platform specific code In-Reply-To: <1936535749.5322045.1345040282195.JavaMail.root@redhat.com> References: <1936535749.5322045.1345040282195.JavaMail.root@redhat.com> Message-ID: <502BB3E1.9060707@oracle.com> On 15/08/2012 15:18, Andrew Hughes wrote: > : > This looks good to me and built ok here. > > My only concern is that this now means that Macs now always get a null > file type detector, whereas there might have been a working one if glib > was installed. The name "GNOMEFileTypeDetector" is a bit of a misnomer > as GIO is part of glib, which is intentionally multi-platform (even Windows, > I believe). > > Maybe worth asking on the BSD and Mac lists? It might motivate them to provide > a solution using the native Mac API (assuming there is one) ;-) > > Thanks, Thanks Andrew. There are a few things still missing on Mac, a FileTypeDetector is one. I don't know common it is to have GIO on the Mac but I would prefer that there would be separate change-set to add support for file type detector on the Mac (if that's okay with you). Once there is an implementation then it would plug-in by overriding getFileTypeDetector in MacOSXFileSystemProvider. On GnomeFileTypeDetector then it is so named because it started out by using GNOME VFS, it got GIO support later. It should probably be separated out now. Maybe this is something you would like to take on? -Alan From ahughes at redhat.com Wed Aug 15 11:29:28 2012 From: ahughes at redhat.com (Andrew Hughes) Date: Wed, 15 Aug 2012 14:29:28 -0400 (EDT) Subject: 7191556: (fs) UnixNativeDispatcher.getextmntent should be moved into platform specific code In-Reply-To: <502BB3E1.9060707@oracle.com> Message-ID: <1429211087.5419822.1345055368448.JavaMail.root@redhat.com> ----- Original Message ----- > On 15/08/2012 15:18, Andrew Hughes wrote: > > : > > This looks good to me and built ok here. > > > > My only concern is that this now means that Macs now always get a > > null > > file type detector, whereas there might have been a working one if > > glib > > was installed. The name "GNOMEFileTypeDetector" is a bit of a > > misnomer > > as GIO is part of glib, which is intentionally multi-platform (even > > Windows, > > I believe). > > > > Maybe worth asking on the BSD and Mac lists? It might motivate > > them to provide > > a solution using the native Mac API (assuming there is one) ;-) > > > > Thanks, > Thanks Andrew. > > There are a few things still missing on Mac, a FileTypeDetector is > one. > I don't know common it is to have GIO on the Mac but I would prefer > that > there would be separate change-set to add support for file type > detector > on the Mac (if that's okay with you). Once there is an implementation > then it would plug-in by overriding getFileTypeDetector in > MacOSXFileSystemProvider. It's ok by me. I'm not sure how common it is either (fairly rare I would have thought). I would guess it's much more common on *BSD (it's certainly in FreeBSD's ports, as is GNOME), but that platform doesn't seem to be supported anyway. FileSystemProvider's create method will throw an AssertionError rather than returning the provider, which seems to be there just as a base for the MacOS X one. > > On GnomeFileTypeDetector then it is so named because it started out > by > using GNOME VFS, it got GIO support later. It should probably be > separated out now. Maybe this is something you would like to take on? > Sure. It's not so much of an issue with NIO, but GIO/Glib will also need to be used to replace GConf in sun.net.spi.DefaultProxySelector, so perhaps it's worth seeing the two could be unified in some way, rather than having a search for GIO at two different places in the codebase. > -Alan > -- Andrew :) Free Java Software Engineer Red Hat, Inc. (http://www.redhat.com) PGP Key: 248BDC07 (https://keys.indymedia.org/) Fingerprint = EC5A 1F5E C0AD 1D15 8F1F 8F91 3B96 A578 248B DC07 From rob.mckenna at oracle.com Wed Aug 15 14:09:56 2012 From: rob.mckenna at oracle.com (Rob McKenna) Date: Wed, 15 Aug 2012 22:09:56 +0100 Subject: request for review: 7184932: Remove the temporary Selector usage in the NIO socket adapters Message-ID: <502C1024.5010403@oracle.com> Hi folks, Looking for a codereview of this change. Basically we're looking to avoid the use of temporary selectors in socket adapters. This should result in resource usage improvements for users who make extensive use of these adapters in multiple threads. http://cr.openjdk.java.net/~robm/7184932/webrev.02/ Thanks, -Rob -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120815/e6805cc9/attachment.html From Alan.Bateman at oracle.com Thu Aug 16 00:54:50 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 16 Aug 2012 08:54:50 +0100 Subject: 7191556: (fs) UnixNativeDispatcher.getextmntent should be moved into platform specific code In-Reply-To: <1429211087.5419822.1345055368448.JavaMail.root@redhat.com> References: <1429211087.5419822.1345055368448.JavaMail.root@redhat.com> Message-ID: <502CA74A.5000704@oracle.com> On 15/08/2012 19:29, Andrew Hughes wrote: > : > It's ok by me. I'm not sure how common it is either (fairly rare I would > have thought). I would guess it's much more common on *BSD (it's certainly > in FreeBSD's ports, as is GNOME), but that platform doesn't seem to be supported > anyway. FileSystemProvider's create method will throw an AssertionError rather > than returning the provider, which seems to be there just as a base for the MacOS X > one. I think most of the BSD port came into the jdk8 and jdk7u forests as part of the Mac port but there is a lot of inconsistencies. To my knowledge, BSD isn't supported in the main forests and that there are still patches and changes in the down-stream bsd porting project. In this specific case I think the right thing to do is to add src/bsd/classes/sun/nio/fs/DefaultFileSystemprovider.java to create the BSD provider. At least the assertion error "Platform not recognized" will serve as a reminder to porters. Of course this also brings up the topic of having code for several platforms in src/solaris. We've been kicking that can down the road for several years but that's a bigger topic. > Sure. It's not so much of an issue with NIO, but GIO/Glib will also need to be used > to replace GConf in sun.net.spi.DefaultProxySelector, so perhaps it's worth seeing > the two could be unified in some way, rather than having a search for GIO at two > different places in the codebase. > I agree we shouldn't have to duplicate code to search for GIO but we also have to be careful with dependencies between given parts of the JDK code. I would suggest coming up with a proposal and we can discuss it in more detail. On net-dev there are periodic issues with the default proxy selector code and I completely agree this needs attention too. -Alan. From Alan.Bateman at oracle.com Thu Aug 16 04:53:08 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 16 Aug 2012 12:53:08 +0100 Subject: request for review: 7184932: Remove the temporary Selector usage in the NIO socket adapters In-Reply-To: <502C1024.5010403@oracle.com> References: <502C1024.5010403@oracle.com> Message-ID: <502CDF24.7010505@oracle.com> On 15/08/2012 22:09, Rob McKenna wrote: > Hi folks, > > Looking for a codereview of this change. Basically we're looking to > avoid the use of temporary selectors in socket adapters. This should > result in resource usage improvements for users who make extensive use > of these adapters in multiple threads. > > http://cr.openjdk.java.net/~robm/7184932/webrev.02/ > > > Thanks, > > -Rob This looks like good progress since the last round. DatagramSocketAdaptor.receive - I think this needs to check for POLLIN rather than dc.isConnected. DatagramChannelImpl.poll - are you missing the reset of readerThread? Same thing in ServerSocketChannelImpl.poll. ServerSocketAdaptor.accept - shouldn't this be POLLIN rather than POLLCONN? Minor nit in src/windows/native/sun/nio/ch/Net.c where the indenting should be 4-spaces. In src/solaris/native/sun/nio/ch/Net.c then the changes are okay although given the existing assumptions on the values of POLL* then you could return pfd.revents. I haven't gone through the test yet but I will get to it when I get time. -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120816/013e3490/attachment.html From ahughes at redhat.com Thu Aug 16 09:32:11 2012 From: ahughes at redhat.com (Andrew Hughes) Date: Thu, 16 Aug 2012 12:32:11 -0400 (EDT) Subject: 7191556: (fs) UnixNativeDispatcher.getextmntent should be moved into platform specific code In-Reply-To: <502CA74A.5000704@oracle.com> Message-ID: <1504683718.6128203.1345134731913.JavaMail.root@redhat.com> ----- Original Message ----- > On 15/08/2012 19:29, Andrew Hughes wrote: > > : > > It's ok by me. I'm not sure how common it is either (fairly rare I > > would > > have thought). I would guess it's much more common on *BSD (it's > > certainly > > in FreeBSD's ports, as is GNOME), but that platform doesn't seem to > > be supported > > anyway. FileSystemProvider's create method will throw an > > AssertionError rather > > than returning the provider, which seems to be there just as a base > > for the MacOS X > > one. > I think most of the BSD port came into the jdk8 and jdk7u forests as > part of the Mac port but there is a lot of inconsistencies. To my > knowledge, BSD isn't supported in the main forests and that there are > still patches and changes in the down-stream bsd porting project. In > this specific case I think the right thing to do is to add > src/bsd/classes/sun/nio/fs/DefaultFileSystemprovider.java to create > the > BSD provider. At least the assertion error "Platform not recognized" > will serve as a reminder to porters. Yeah, that's what I gathered from studying the code. The BSD bits that have actually been added are just enough to make the MacOS ones work. > Of course this also brings up > the > topic of having code for several platforms in src/solaris. We've been > kicking that can down the road for several years but that's a bigger > topic. I don't think this is a huge issue. I tend to just read solaris in that context as posix. It seems we now have a macosx one with plenty of code in, but then they have all this proprietary GUI stuff. The POSIX APIs shouldn't differ that much. From what I could see, the PPC-AIX folks have managed to get the JDK building on AIX just by altering the solaris code slightly. > > > Sure. It's not so much of an issue with NIO, but GIO/Glib will > > also need to be used > > to replace GConf in sun.net.spi.DefaultProxySelector, so perhaps > > it's worth seeing > > the two could be unified in some way, rather than having a search > > for GIO at two > > different places in the codebase. > > > I agree we shouldn't have to duplicate code to search for GIO but we > also have to be careful with dependencies between given parts of the > JDK > code. I would suggest coming up with a proposal and we can discuss it > in > more detail. On net-dev there are periodic issues with the default > proxy > selector code and I completely agree this needs attention too. Ok, what I was thinking was something like the following. I can draft some code if it helps, but basically the current situation is: Current situation: java.awt.Desktop --> GNOME-VFS sun.net.spi.DefaultProxySelector --> GConf sun.nio.fs.GnomeFileTypeDetector --> GIO or GNOME-VFS Proposed solution: Above classes all depend on a common SystemServiceProvider which determines whether to use GIO or GNOME2 (GConf/GNOME-VFS) to provide the required functionality (GIO in newer versions of GLib replaces all three). We could possibly extend this to other platforms, but cleaning up the scattered POSIX implementations would be a start. I'm not sure of an appropriate package; perhaps sun.misc which already includes OSEnvironment (a pointless class at present on Solaris & Linux)? Going through these also highlights the issue that MacOS will also presumably try to use GConf for proxy functionality at present; I don't see anything overriding it. For java.awt.Desktop, support is provided in the Mac AWT implementation. > > -Alan. > -- Andrew :) Free Java Software Engineer Red Hat, Inc. (http://www.redhat.com) PGP Key: 248BDC07 (https://keys.indymedia.org/) Fingerprint = EC5A 1F5E C0AD 1D15 8F1F 8F91 3B96 A578 248B DC07 From chris.hegarty at oracle.com Thu Aug 16 13:11:11 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 16 Aug 2012 21:11:11 +0100 Subject: 7191467: (fs) WatchService periodically fails to queue ENTRY_DELETE event for short lived file [sol11] In-Reply-To: <502B85EE.9090909@oracle.com> References: <502B85EE.9090909@oracle.com> Message-ID: <502D53DF.5060707@oracle.com> The change looks ok to me. -Chris. On 15/08/12 12:20, Alan Bateman wrote: > > MayFlies is one of the tests for the WatchService, it's a stress test > that ensures that there are corresponding ENTRY_DELETE events for each > ENTRY_CREATE event when the files are very short lived. This test fails > periodically on Solaris 11 where it appears that FILE_REMOVE events > aren't being queued by the kernel. There are a couple of workarounds in > this code to deal with a kernel issues in this area and I've decided it > to be best to just remove these and change the implementation so that it > detects file deletions when scanning the directory (which it has to do > anyway when the directory changes). > > The webrev with the changes is here: > > http://cr.openjdk.java.net/~alanb/7191467/webrev/ > > The main change is that entries in directories are only registered with > the kernel when the directory is registered for ENTRY_MODIFY events. > ENTRY_CREATE and ENTRY_DELETE events are now completely triggered on > changes to the directory. > > One other change in here, and it stemmed from a discussion on this mail > a few months ago, is that if there is an error encountered when > iterating over the directory then an OVERFLOW event is queued. That is > the trigger to the user application to re-scan the directory to update > its view of the directory. > > -Alan From Alan.Bateman at oracle.com Fri Aug 17 02:00:15 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 17 Aug 2012 10:00:15 +0100 Subject: request for review: 7184932: Remove the temporary Selector usage in the NIO socket adapters In-Reply-To: <502C1024.5010403@oracle.com> References: <502C1024.5010403@oracle.com> Message-ID: <502E081F.1030807@oracle.com> On 15/08/2012 22:09, Rob McKenna wrote: > Hi folks, > > Looking for a codereview of this change. Basically we're looking to > avoid the use of temporary selectors in socket adapters. This should > result in resource usage improvements for users who make extensive use > of these adapters in multiple threads. > > http://cr.openjdk.java.net/~robm/7184932/webrev.02/ > > > Thanks, > > -Rob One other thing, now that this is extended to DatagramSocketAdpator and ServerSocketChannelAdaptor then it you should be able to remove getTemporarySelector and friends from sun.nio.ch.Util as I don't think they are used anywhere else. -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120817/ac9eab20/attachment.html From jason.greene at redhat.com Fri Aug 17 14:43:44 2012 From: jason.greene at redhat.com (Jason T. Greene) Date: Fri, 17 Aug 2012 16:43:44 -0500 Subject: Problems persist in KQueueSelectorProvider (Mac) in 7u6 ea In-Reply-To: <502BA75A.9020101@oracle.com> References: <50259B07.1080306@redhat.com> <5028B69F.5060907@oracle.com> <502951ED.4060704@redhat.com> <502974B8.1080206@oracle.com> <46F92297-591C-4A6E-85CF-2943A4A3F099@redhat.com> <502BA75A.9020101@oracle.com> Message-ID: <502EBB10.3040308@redhat.com> On 8/15/12 8:42 AM, Alan Bateman wrote: > On 13/08/2012 22:50, Jason Greene wrote: >> On Aug 13, 2012, at 4:42 PM, Alan Bateman >> wrote: >> >>> : >>> Thanks for trying that out, it's good to hear that the problems have >>> gone away. On the tight loop theory then can you check if a small >>> sleep, or may a Thread.yield, causes the dup2 hang to go away too? >> Sure I can give that a try. I'll try to get the dump you requested >> earlier if not. > That would be good. Sleep and yield don't seem to prevent the dup2 issue, so there must be something more complex going on. For whatever reason I have yet to trigger it with the setInterest patch Here is the kernel stack: Process ID 65053 is a shambling zombie. PID: 65053 Process: java Thread ID: 0x71660 Thread state: 0x9 == TH_WAIT |TH_UNINT Thread wait_event: 0xffffff800c3bdda8 Kernel stack: machine_switch_context (in mach_kernel) + 361 (0xffffff80002c2939) thread_continue (in mach_kernel) + 1661 (0xffffff800022f1ad) thread_block_reason (in mach_kernel) + 299 (0xffffff800022f42b) lck_mtx_sleep (in mach_kernel) + 74 (0xffffff8000227efa) wakeup (in mach_kernel) + 267 (0xffffff8000554d3b) msleep (in mach_kernel) + 119 (0xffffff8000555397) fileproc_drain (in mach_kernel) + 249 (0xffffff8000534c59) fstat_extended (in mach_kernel) + 300 (0xffffff800053675c) dup2 (in mach_kernel) + 272 (0xffffff80005369c0) unix_syscall64 (in mach_kernel) + 507 (0xffffff80005cd61b) hndl_unix_scall64 (in mach_kernel) + 19 (0xffffff80002daa13) > >> : >>> On the patch, are you submitting it here as a contribution (once you >>> are done with your testing)? I haven't looked at it closely yet but I >>> think it is close to what we have in the epoll Selector. >> Yes I just reused the same approach that epoll is using with slight >> alterations, since it seems to work well. The patch if useful is a >> contribution, although as you noticed its mostly a derivative. >> > It make sense too. I've created a bug to track this: > > 7191587: (se) SelectionKey.interestOps does not defer changing the > interest set to the next select [macosx] > > and once you are done with your testing we will look to get this into > jdk8. Once we are satisfied they we can seek approval for 7u8. Ok I have tested this very thoroughly, and I think it's ready for submission. This includes jdk 6 (snow leopard, lion, mountain lion) and 7 (lion, mountain lion). I also ran through the nio selector tests, all pass. What process should I follow? A potential change that could be made, that I think should be left for the future should perf testing ever show it's importance, is that the kqueue change list could be combined into one syscall, instead of a list of N syscalls. -- Jason T. Greene JBoss AS Lead / EAP Platform Architect JBoss, a division of Red Hat From jason.greene at redhat.com Sat Aug 18 13:43:38 2012 From: jason.greene at redhat.com (Jason T. Greene) Date: Sat, 18 Aug 2012 15:43:38 -0500 Subject: Problems persist in KQueueSelectorProvider (Mac) in 7u6 ea In-Reply-To: References: <50259B07.1080306@redhat.com> <5028B69F.5060907@oracle.com> <502951ED.4060704@redhat.com> Message-ID: <502FFE7A.6070208@redhat.com> On 8/13/12 4:24 PM, David Schlosnagle wrote: > On Mon, Aug 13, 2012 at 3:13 PM, Jason T. Greene > wrote: >> I threw together a quick patch (attached below against 7u-dev), and it >> appears to resolve the issues on 6 and 7. Although it needs more testing. >> >> The preClose problem disappears as well with this change. My hunch is that >> the tight spin loop on kqueue makes it impossible for close/dup2 to ever >> complete. > > Hi Jason, > > A few minor comments/questions in KQueueArrayWrapper.java: Thanks! > Should the updateList and Update instance fields all be final? Sure they could be. > Is there any reason not to avoid the continue in updateRegistrations()? > while ((u = updateList.poll()) != null) { > SelChImpl ch = u.channel; > if (ch.isOpen()) { > register0(kq, ch.getFDVal(), u.events & POLLIN, u.events & POLLOUT); > } > } No reason, but then again there is also no reason to avoid it either. It's just a stylistic difference. > Do you foresee any lock contention issues with the synchronization on > updateList? The NIO design is such that this is necessary, you need to ensure atomic and consistent changes to the underlying queue/poll. The lock is only held for a short period of time. > Would using a java.util.concurrent.LinkedBlockingQueue > rather than the LinkedList with synchronization violate the API > semantics of batching the updates? Before the selector is ran, you need to process the entire queue. Else there is no guarantee the poll implementation's state is consistent with the NIO API's state. So you would have to use the drainTo operation to get proper locking semantics, and that requires a needless copy into another collection. Even if you didn't have the copy the memory overhead of LinkedBlockingQueue is quite a bit more than a LinkedList. There's also a small difference in that JDK locks (synchronized) have optimization paths in openjdk that do not exist for ReentrantLock (and they use far less memory). An ArrayDeque on the other hand would likely be more efficient (memory + cpu) than the LinkedList. I was planning to change the patch to use it instead, although at the expected queue size, I doubt there will be much of a real world significance. I was thinking something along the > lines of the KQueueArrayWrapper.java diff below would work (but I > haven't tested at all). > > Thanks, > Dave > > > diff --git a/KQueueArrayWrapper.java b/KQueueArrayWrapper.java > index 5a7020c..936f086 100644 > --- a/KQueueArrayWrapper.java > +++ b/KQueueArrayWrapper.java > @@ -34,7 +34,9 @@ package sun.nio.ch; > import sun.misc.*; > import java.io.IOException; > import java.io.FileDescriptor; > - > +import java.util.Iterator; > +import java.util.Queue; > +import java.util.concurrent.LinkedBlockingQueue; > > /* > * struct kevent { // 32-bit 64-bit > @@ -85,6 +87,9 @@ class KQueueArrayWrapper { > // The fd of the interrupt line coming in > private int incomingInterruptFD; > > + // The queue of file descriptor registration updates to process > + private final Queue updateQueue = new LinkedBlockingQueue<>(); > + > static { > initStructSizes(); > String datamodel = java.security.AccessController.doPrivileged( > @@ -100,6 +105,16 @@ class KQueueArrayWrapper { > kq = init(); > } > > + // Used to update file description registrations > + private static class Update { > + final SelChImpl channel; > + final int events; > + Update(SelChImpl channel, int events) { > + this.channel = channel; > + this.events = events; > + } > + } > + > void initInterrupt(int fd0, int fd1) { > outgoingInterruptFD = fd1; > incomingInterruptFD = fd0; > @@ -137,12 +152,31 @@ class KQueueArrayWrapper { > } > } > > - void setInterest(int fd, int events) { > - register0(kq, fd, events & POLLIN, events & POLLOUT); > + void setInterest(SelChImpl channel, int events) { > + // update existing registration > + updateQueue.add(new Update(channel, events)); > } > > - void release(int fd) { > - register0(kq, fd, 0, 0); > + void release(SelChImpl channel) { > + // flush any pending updates > + for (Iterator it = updateQueue.iterator(); it.hasNext();) { > + if (it.next().channel == channel) { > + it.remove(); > + } > + } > + > + // remove > + register0(kq, channel.getFDVal(), 0, 0); > + } > + > + void updateRegistrations() { > + Update u = null; > + while ((u = updateQueue.poll()) != null) { > + SelChImpl ch = u.channel; > + if (ch.isOpen()) { > + register0(kq, ch.getFDVal(), u.events & POLLIN, > u.events & POLLOUT); > + } > + } > } > > void close() throws IOException { > @@ -157,6 +191,7 @@ class KQueueArrayWrapper { > } > > int poll(long timeout) { > + updateRegistrations(); > int updated = kevent0(kq, keventArrayAddress, NUM_KEVENTS, timeout); > return updated; > } > -- Jason T. Greene JBoss AS Lead / EAP Platform Architect JBoss, a division of Red Hat From shirishk at linux.vnet.ibm.com Mon Aug 20 07:20:52 2012 From: shirishk at linux.vnet.ibm.com (Shirish Kuncolienkar) Date: Mon, 20 Aug 2012 19:50:52 +0530 Subject: Code review request 7132889: Possible race conditions in AbstractSelectableChannel In-Reply-To: <502A0CDC.4020604@oracle.com> References: <5020E11F.8020007@linux.vnet.ibm.com> <5020E6C7.90501@oracle.com> <50211571.8050900@linux.vnet.ibm.com> <50223FD2.1090401@oracle.com> <50225E45.3090305@linux.vnet.ibm.com> <5028FAC7.7080708@oracle.com> <50291BFE.4030102@oracle.com> <5029B597.4080709@linux.vnet.ibm.com> <502A0CDC.4020604@oracle.com> Message-ID: <503247C4.1050604@linux.vnet.ibm.com> On 8/14/2012 2:01 PM, Alan Bateman wrote: > On 14/08/2012 03:19, Shirish Kuncolienkar wrote: >> >> I see that regLock is not limited to blocking mode. Many places the >> keyLock is acquired while the regLock is held. > It guards the blocking mode, and may have been clearer if it were > named blockingLock. Locking regLock and keyLock (in that order) should > be fine, just not in implCloseSelectableChannel because you need to be > able to invoke close when there is a blocking operation in progress. > >> If we do not guard the implCloseSelectableChannel() and key >> cancellation within the regLock we would not be able to avoid a key >> being added while a channel is closed. Do you think such a behavior >> should be documented ? >> I will run the channel tests as suggested by you. > Yes, it would be great if you could run the tests with your patch as I > am concerned that it will break asynchronous close causing several > tests will either hang or deadlock. > > I think the residual issue with register returning a valid key can be > solved by checking the channel status when holding the keyLock, eg: > > synchronized (keyLock) { > if (!isOpen()) > throw new ClosedChannelException(); > addKey(k); > } > > and change addKey to assert Thread.holdsLock(keyLock); > > So I'm curious as to the background to this issue, is this a patch > that you have in the IBM JDK? > > -Alan. Alan, I created a new patch http://cr.openjdk.java.net/~luchsh/webrev.000/ based on your suggestions and tested the same against java/nio/channels tests I see no new failures as compared to java version "1.8.0-ea" Java(TM) SE Runtime Environment (build 1.8.0-ea-b49) Java HotSpot(TM) 64-Bit Server VM (build 24.0-b16, mixed mode) Detailed Test results passed: 159; failed: 1; error: 1 Error: java/nio/channels/DatagramChannel/AdaptDatagramSocket.java FAILED: java/nio/channels/DatagramChannel/MulticastSendReceiveTests.java The timing window was seen during fixing of the select loop issue and the changes are being held as a patch. Thanks -Shirish From Alan.Bateman at oracle.com Mon Aug 20 12:19:25 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 20 Aug 2012 20:19:25 +0100 Subject: Problems persist in KQueueSelectorProvider (Mac) in 7u6 ea In-Reply-To: <502EBB10.3040308@redhat.com> References: <50259B07.1080306@redhat.com> <5028B69F.5060907@oracle.com> <502951ED.4060704@redhat.com> <502974B8.1080206@oracle.com> <46F92297-591C-4A6E-85CF-2943A4A3F099@redhat.com> <502BA75A.9020101@oracle.com> <502EBB10.3040308@redhat.com> Message-ID: <50328DBD.5040401@oracle.com> On 17/08/2012 22:43, Jason T. Greene wrote: > > Sleep and yield don't seem to prevent the dup2 issue, so there must be > something more complex going on. For whatever reason I have yet to > trigger it with the setInterest patch > > Here is the kernel stack: Thanks for checking, it does appear that something else is going on. It would be really useful if you could create a standalone test case that duplicates it (don't know if that is possible as I assume you are seeing this in conjunction with lots of middleware). > > Ok I have tested this very thoroughly, and I think it's ready for > submission. This includes jdk 6 (snow leopard, lion, mountain lion) > and 7 (lion, mountain lion). I also ran through the nio selector > tests, all pass. > > What process should I follow? Thanks for testing in all environments (I'm surprised that it works with 6 as we have made several changes to this code since the the original Apple contribution). For process then you need a sponsor to commit it for you so I will do that for you, hopefully this week, I just need to spend a bit of time to look at it more closely. > > A potential change that could be made, that I think should be left for > the future should perf testing ever show it's importance, is that the > kqueue change list could be combined into one syscall, instead of a > list of N syscalls. > Yes, I think that would be a good improvement but it does not need to be in this patch. We do in /dev/poll Selector so you might get some ideas there. So thanks for the bug report and patch, I will get back to you soon. -Alan From Alan.Bateman at oracle.com Mon Aug 20 12:21:25 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 20 Aug 2012 20:21:25 +0100 Subject: Code review request 7132889: Possible race conditions in AbstractSelectableChannel In-Reply-To: <503247C4.1050604@linux.vnet.ibm.com> References: <5020E11F.8020007@linux.vnet.ibm.com> <5020E6C7.90501@oracle.com> <50211571.8050900@linux.vnet.ibm.com> <50223FD2.1090401@oracle.com> <50225E45.3090305@linux.vnet.ibm.com> <5028FAC7.7080708@oracle.com> <50291BFE.4030102@oracle.com> <5029B597.4080709@linux.vnet.ibm.com> <502A0CDC.4020604@oracle.com> <503247C4.1050604@linux.vnet.ibm.com> Message-ID: <50328E35.6000904@oracle.com> On 20/08/2012 15:20, Shirish Kuncolienkar wrote: > > Alan, > > I created a new patch http://cr.openjdk.java.net/~luchsh/webrev.000/ > based on your suggestions and tested the same against > java/nio/channels tests > > I see no new failures as compared to > java version "1.8.0-ea" > Java(TM) SE Runtime Environment (build 1.8.0-ea-b49) > Java HotSpot(TM) 64-Bit Server VM (build 24.0-b16, mixed mode) > > Detailed Test results > passed: 159; failed: 1; error: 1 > Error: java/nio/channels/DatagramChannel/AdaptDatagramSocket.java > FAILED: java/nio/channels/DatagramChannel/MulticastSendReceiveTests.java > > The timing window was seen during fixing of the select loop issue and > the changes are being held as a patch. Formatting aside, the updated patch looks good. I will sponsor this patch. On the tests then I'm not sure why MulticastSendReceiveTests.java is failing for you. Can you forward the .jtr (output) file? It it possible that firewall or VPN settings are causing muticast packets to be dropped? -Alan. From jgreene at redhat.com Mon Aug 20 15:41:05 2012 From: jgreene at redhat.com (Jason Greene) Date: Mon, 20 Aug 2012 18:41:05 -0400 (EDT) Subject: Problems persist in KQueueSelectorProvider (Mac) in 7u6 ea In-Reply-To: <50328DBD.5040401@oracle.com> References: <50259B07.1080306@redhat.com> <5028B69F.5060907@oracle.com> <502951ED.4060704@redhat.com> <502974B8.1080206@oracle.com> <46F92297-591C-4A6E-85CF-2943A4A3F099@redhat.com> <502BA75A.9020101@oracle.com> <502EBB10.3040308@redhat.com> <50328DBD.5040401@oracle.com> Message-ID: On Aug 20, 2012, at 2:19 PM, Alan Bateman wrote: > On 17/08/2012 22:43, Jason T. Greene wrote: >> >> Sleep and yield don't seem to prevent the dup2 issue, so there must be something more complex going on. For whatever reason I have yet to trigger it with the setInterest patch >> >> Here is the kernel stack: > Thanks for checking, it does appear that something else is going on. It would be really useful if you could create a standalone test case that duplicates it (don't know if that is possible as I assume you are seeing this in conjunction with lots of middleware). Unfortunately all my efforts to reproduce it in a simple test failed. I even attempted a very aggressive native kqueue stress test, but it worked as expected.Looking at the kernel source, this could only happen if the pending I/O refcount is off, or there is a lost signal. A blocking operation or pending select will keep the count up, but those are suppose to be interrupted. The really odd thing is that the process will always become a zombie, and when killed all pending ops should have been cancelled. At this point the only way I can think of to debug this further is to add custom kernel debugging to track the io counts. I probably won't have time for that unless we still see the issue again from some other scenario. > >> >> Ok I have tested this very thoroughly, and I think it's ready for submission. This includes jdk 6 (snow leopard, lion, mountain lion) and 7 (lion, mountain lion). I also ran through the nio selector tests, all pass. >> >> What process should I follow? > Thanks for testing in all environments (I'm surprised that it works with 6 as we have made several changes to this code since the the original Apple contribution). I backported the patched JDK 7 impl to test 6. It only requires a few changes (makePipe change, and methods moving from FileDispatcher into FDImpl). > For process then you need a sponsor to commit it for you so I will do that for you, hopefully this week, I just need to spend a bit of time to look at it more closely. Great thanks! > >> >> A potential change that could be made, that I think should be left for the future should perf testing ever show it's importance, is that the kqueue change list could be combined into one syscall, instead of a list of N syscalls. >> > Yes, I think that would be a good improvement but it does not need to be in this patch. We do in /dev/poll Selector so you might get some ideas there. > > So thanks for the bug report and patch, I will get back to you soon. Ok thanks From Alan.Bateman at oracle.com Tue Aug 21 03:11:01 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 21 Aug 2012 11:11:01 +0100 Subject: Code review request 7132889: Possible race conditions in AbstractSelectableChannel In-Reply-To: <50328E35.6000904@oracle.com> References: <5020E11F.8020007@linux.vnet.ibm.com> <5020E6C7.90501@oracle.com> <50211571.8050900@linux.vnet.ibm.com> <50223FD2.1090401@oracle.com> <50225E45.3090305@linux.vnet.ibm.com> <5028FAC7.7080708@oracle.com> <50291BFE.4030102@oracle.com> <5029B597.4080709@linux.vnet.ibm.com> <502A0CDC.4020604@oracle.com> <503247C4.1050604@linux.vnet.ibm.com> <50328E35.6000904@oracle.com> Message-ID: <50335EB5.80707@oracle.com> I've put a webrev with what I hope is the final patch for this: http://cr.openjdk.java.net/~alanb/7132889/webrev/ This is your patch with tabs removed and the aligned fixed. I've also added a test as it is turns out to be easy to create the scenario where you get a valid SelectionKey even though the channel is now closed. If you are happy with this then I will push it today, attributing you in the Contributed-by of course. -Alan From ahughes at redhat.com Wed Aug 22 01:34:57 2012 From: ahughes at redhat.com (Andrew Hughes) Date: Wed, 22 Aug 2012 04:34:57 -0400 (EDT) Subject: 7191556: (fs) UnixNativeDispatcher.getextmntent should be moved into platform specific code In-Reply-To: <1504683718.6128203.1345134731913.JavaMail.root@redhat.com> Message-ID: <1585504951.8874527.1345624497900.JavaMail.root@redhat.com> Any thoughts on this? ----- Original Message ----- > ----- Original Message ----- > > On 15/08/2012 19:29, Andrew Hughes wrote: > > > : > > > It's ok by me. I'm not sure how common it is either (fairly rare > > > I > > > would > > > have thought). I would guess it's much more common on *BSD (it's > > > certainly > > > in FreeBSD's ports, as is GNOME), but that platform doesn't seem > > > to > > > be supported > > > anyway. FileSystemProvider's create method will throw an > > > AssertionError rather > > > than returning the provider, which seems to be there just as a > > > base > > > for the MacOS X > > > one. > > I think most of the BSD port came into the jdk8 and jdk7u forests > > as > > part of the Mac port but there is a lot of inconsistencies. To my > > knowledge, BSD isn't supported in the main forests and that there > > are > > still patches and changes in the down-stream bsd porting project. > > In > > this specific case I think the right thing to do is to add > > src/bsd/classes/sun/nio/fs/DefaultFileSystemprovider.java to create > > the > > BSD provider. At least the assertion error "Platform not > > recognized" > > will serve as a reminder to porters. > > Yeah, that's what I gathered from studying the code. The BSD bits > that > have actually been added are just enough to make the MacOS ones work. > > > Of course this also brings up > > the > > topic of having code for several platforms in src/solaris. We've > > been > > kicking that can down the road for several years but that's a > > bigger > > topic. > > I don't think this is a huge issue. I tend to just read solaris in > that > context as posix. It seems we now have a macosx one with plenty of > code > in, but then they have all this proprietary GUI stuff. The POSIX > APIs > shouldn't differ that much. From what I could see, the PPC-AIX folks > have managed to get the JDK building on AIX just by altering the > solaris > code slightly. > > > > > > Sure. It's not so much of an issue with NIO, but GIO/Glib will > > > also need to be used > > > to replace GConf in sun.net.spi.DefaultProxySelector, so perhaps > > > it's worth seeing > > > the two could be unified in some way, rather than having a search > > > for GIO at two > > > different places in the codebase. > > > > > I agree we shouldn't have to duplicate code to search for GIO but > > we > > also have to be careful with dependencies between given parts of > > the > > JDK > > code. I would suggest coming up with a proposal and we can discuss > > it > > in > > more detail. On net-dev there are periodic issues with the default > > proxy > > selector code and I completely agree this needs attention too. > > Ok, what I was thinking was something like the following. I can > draft > some code if it helps, but basically the current situation is: > > Current situation: > > java.awt.Desktop --> GNOME-VFS > > sun.net.spi.DefaultProxySelector --> GConf > > sun.nio.fs.GnomeFileTypeDetector --> GIO or GNOME-VFS > > Proposed solution: > > Above classes all depend on a common SystemServiceProvider which > determines > whether to use GIO or GNOME2 (GConf/GNOME-VFS) to provide the > required functionality (GIO > in newer versions of GLib replaces all three). We could possibly > extend > this to other platforms, but cleaning up the scattered POSIX > implementations > would be a start. > > I'm not sure of an appropriate package; perhaps sun.misc which > already includes > OSEnvironment (a pointless class at present on Solaris & Linux)? > > Going through these also highlights the issue that MacOS will also > presumably > try to use GConf for proxy functionality at present; I don't see > anything overriding > it. For java.awt.Desktop, support is provided in the Mac AWT > implementation. > > > > > -Alan. > > > > -- > Andrew :) > > Free Java Software Engineer > Red Hat, Inc. (http://www.redhat.com) > > PGP Key: 248BDC07 (https://keys.indymedia.org/) > Fingerprint = EC5A 1F5E C0AD 1D15 8F1F 8F91 3B96 A578 248B DC07 > > -- Andrew :) Free Java Software Engineer Red Hat, Inc. (http://www.redhat.com) PGP Key: 248BDC07 (https://keys.indymedia.org/) Fingerprint = EC5A 1F5E C0AD 1D15 8F1F 8F91 3B96 A578 248B DC07 From Alan.Bateman at oracle.com Wed Aug 22 05:46:07 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 22 Aug 2012 13:46:07 +0100 Subject: 7191556: (fs) UnixNativeDispatcher.getextmntent should be moved into platform specific code In-Reply-To: <1504683718.6128203.1345134731913.JavaMail.root@redhat.com> References: <1504683718.6128203.1345134731913.JavaMail.root@redhat.com> Message-ID: <5034D48F.7090709@oracle.com> (apologies for the delay replying to you, lots of things on my plate at the moment). On 16/08/2012 17:32, Andrew Hughes wrote: > : >> Of course this also brings up >> the >> topic of having code for several platforms in src/solaris. We've been >> kicking that can down the road for several years but that's a bigger >> topic. > I don't think this is a huge issue. I tend to just read solaris in that > context as posix. It seems we now have a macosx one with plenty of code > in, but then they have all this proprietary GUI stuff. The POSIX APIs > shouldn't differ that much. From what I could see, the PPC-AIX folks > have managed to get the JDK building on AIX just by altering the solaris > code slightly. It's a not major issue but it is one that comes up regularly, mostly because src/solaris requires a lot more than POSIX and because it also contains both Solaris and Linux specific code. If OpenJDK is going to take on more platforms then I do think some clean-up is required although it's hard to spend time on such things, and of course there is never a right time. > : > Ok, what I was thinking was something like the following. I can draft > some code if it helps, but basically the current situation is: > > Current situation: > > java.awt.Desktop --> GNOME-VFS > > sun.net.spi.DefaultProxySelector --> GConf > > sun.nio.fs.GnomeFileTypeDetector --> GIO or GNOME-VFS > > Proposed solution: > > Above classes all depend on a common SystemServiceProvider which determines > whether to use GIO or GNOME2 (GConf/GNOME-VFS) to provide the required functionality (GIO > in newer versions of GLib replaces all three). We could possibly extend > this to other platforms, but cleaning up the scattered POSIX implementations > would be a start. > > I'm not sure of an appropriate package; perhaps sun.misc which already includes > OSEnvironment (a pointless class at present on Solaris& Linux)? So are you thinking about something relatively high level that hides whether the underlying implementation is GIO or GConf or whatever? Or maybe you are thinking about a simple shim over all the above that the classes in the JDK could use? I think once you have some code to discuss then it will become clearer as to possible naming and location (my preference would be to not add it to sun.misc, mostly because so many developer think it's fair game to use anything in that package, it's really hard to actually remove or change things in sun.misc as a result). > > Going through these also highlights the issue that MacOS will also presumably > try to use GConf for proxy functionality at present; I don't see anything overriding > it. For java.awt.Desktop, support is provided in the Mac AWT implementation. > The Mac port picks up the proxy settings from the system (java_props_macosx.c). It's a bit different to the other platforms in that "use system proxy settings" is essentially enabled by default (which on the surface is great but also causes problems, particular for tests that make local connections). -Alan. From kurchi.subhra.hazra at oracle.com Wed Aug 22 11:38:56 2012 From: kurchi.subhra.hazra at oracle.com (Kurchi Hazra) Date: Wed, 22 Aug 2012 11:38:56 -0700 Subject: Code Review Request: 7168172: (fs) Files.isReadable slow on Windows Message-ID: <50352740.8060302@oracle.com> Hi, Files.isReadable() is slow on Windows, since it involves reading the DACL and computing the effective access to the file. [1] However, it is faster to simply open the file being checked and close it. In the case of a directory then, a DirectoryStream can be opened and closed to check the same. Bug: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7168172 Webrev: http://cr.openjdk.java.net/~khazra/7168172/webrev.00/ Thanks to Alan for helping out a lot with this fix. - Kurchi [1] http://mail.openjdk.java.net/pipermail/nio-dev/2012-May/001667.html From Ulf.Zibis at CoSoCo.de Wed Aug 22 16:33:23 2012 From: Ulf.Zibis at CoSoCo.de (Ulf Zibis) Date: Thu, 23 Aug 2012 01:33:23 +0200 Subject: Code Review Request: 7168172: (fs) Files.isReadable slow on Windows In-Reply-To: <50352740.8060302@oracle.com> References: <50352740.8060302@oracle.com> Message-ID: <50356C43.1080706@CoSoCo.de> Am 22.08.2012 20:38, schrieb Kurchi Hazra: > However, it is faster to simply open the file being checked and close it. In the case of a > directory then, a DirectoryStream can be opened and closed to check the same. I would rename checkReadAccess(WindowsPath file) to tryReadAccess(WindowsPath file) > > Bug: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7168172 > Webrev: http://cr.openjdk.java.net/~khazra/7168172/webrev.00/ I'm really wondering, why we need 3 types of file access options and have to translate up and down between these: - FileOption - AccessMode - boolean At least the boolean one could be saved easily and (modes.length == 0) is superfluous: public void checkAccess(Path obj, AccessMode... modes) throws IOException { AccessMode r = null; AccessMode w = null; AccessMode x = null; for (AccessMode mode: modes) { switch (mode) { case READ : r = READ; break; case WRITE : w = WRITE; break; case EXECUTE : x = EXECUTE; break; default: throw new AssertionError("Should not get here"); } } WindowsPath file = WindowsPath.toWindowsPath(obj); // special-case read access to avoid needing to determine effective // access to file if (w == null && x == null) { tryReadAccess(file); return; } ... } -Ulf From zhong.j.yu at gmail.com Wed Aug 22 16:50:23 2012 From: zhong.j.yu at gmail.com (Zhong Yu) Date: Wed, 22 Aug 2012 18:50:23 -0500 Subject: Code Review Request: 7168172: (fs) Files.isReadable slow on Windows In-Reply-To: <50356C43.1080706@CoSoCo.de> References: <50352740.8060302@oracle.com> <50356C43.1080706@CoSoCo.de> Message-ID: On Wed, Aug 22, 2012 at 6:33 PM, Ulf Zibis wrote: > Am 22.08.2012 20:38, schrieb Kurchi Hazra: > >> However, it is faster to simply open the file being checked and close it. >> In the case of a directory then, a DirectoryStream can be opened and closed >> to check the same. > > > I would rename checkReadAccess(WindowsPath file) to > tryReadAccess(WindowsPath file) > > >> >> Bug: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7168172 >> Webrev: http://cr.openjdk.java.net/~khazra/7168172/webrev.00/ > > > I'm really wondering, why we need 3 types of file access options and have to > translate up and down between these: > - FileOption > - AccessMode > - boolean > > At least the boolean one could be saved easily and (modes.length == 0) is > superfluous: > > public void checkAccess(Path obj, AccessMode... modes) throws > IOException { > AccessMode r = null; > AccessMode w = null; > AccessMode x = null; > for (AccessMode mode: modes) { > switch (mode) { > case READ : r = READ; break; > case WRITE : w = WRITE; break; > case EXECUTE : x = EXECUTE; break; > default: throw new AssertionError("Should not get here"); > } > } That'll leave r==null if `modes` is empty. But an empty `modes` is supposed to be equivalent to [READ]. > WindowsPath file = WindowsPath.toWindowsPath(obj); > > // special-case read access to avoid needing to determine effective > // access to file > if (w == null && x == null) { > tryReadAccess(file); > return; > } > > ... > } > > -Ulf > From Alan.Bateman at oracle.com Thu Aug 23 01:54:09 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 23 Aug 2012 09:54:09 +0100 Subject: Code Review Request: 7168172: (fs) Files.isReadable slow on Windows In-Reply-To: <50352740.8060302@oracle.com> References: <50352740.8060302@oracle.com> Message-ID: <5035EFB1.8030904@oracle.com> On 22/08/2012 19:38, Kurchi Hazra wrote: > Hi, > > Files.isReadable() is slow on Windows, since it involves reading > the DACL and computing the effective access to the file. [1] > However, it is faster to simply open the file being checked and close > it. In the case of a directory then, a DirectoryStream can > be opened and closed to check the same. > > Bug: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7168172 > Webrev: http://cr.openjdk.java.net/~khazra/7168172/webrev.00/ > > Thanks to Alan for helping out a lot with this fix. Thanks for taking this one, the changes looks fine to me. Just to put more context on this, this change improves the reliability and performance of the checkAccess() or checkAccess(READ) case, further work is required to test access when it includes WRITE or EXECUTE. The changes for this will be more extensive as it requires changing the win32 calls that this code is using. -Alan. From rob.mckenna at oracle.com Thu Aug 23 03:46:01 2012 From: rob.mckenna at oracle.com (Rob McKenna) Date: Thu, 23 Aug 2012 11:46:01 +0100 Subject: request for review: 7184932: Remove the temporary Selector usage in the NIO socket adapters In-Reply-To: <502E081F.1030807@oracle.com> References: <502C1024.5010403@oracle.com> <502E081F.1030807@oracle.com> Message-ID: <503609E9.7080508@oracle.com> Cheers Alan, Latest iteration at: http://cr.openjdk.java.net/~robm/7184932/webrev.03/ -Rob On 17/08/12 10:00, Alan Bateman wrote: > On 15/08/2012 22:09, Rob McKenna wrote: >> Hi folks, >> >> Looking for a codereview of this change. Basically we're looking to >> avoid the use of temporary selectors in socket adapters. This should >> result in resource usage improvements for users who make extensive >> use of these adapters in multiple threads. >> >> http://cr.openjdk.java.net/~robm/7184932/webrev.02/ >> >> >> Thanks, >> >> -Rob > One other thing, now that this is extended to DatagramSocketAdpator > and ServerSocketChannelAdaptor then it you should be able to remove > getTemporarySelector and friends from sun.nio.ch.Util as I don't think > they are used anywhere else. > > -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120823/96ce6c3c/attachment.html From Alan.Bateman at oracle.com Thu Aug 23 04:52:35 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 23 Aug 2012 12:52:35 +0100 Subject: request for review: 7184932: Remove the temporary Selector usage in the NIO socket adapters In-Reply-To: <503609E9.7080508@oracle.com> References: <502C1024.5010403@oracle.com> <502E081F.1030807@oracle.com> <503609E9.7080508@oracle.com> Message-ID: <50361983.5070207@oracle.com> On 23/08/2012 11:46, Rob McKenna wrote: > Cheers Alan, > > Latest iteration at: > > http://cr.openjdk.java.net/~robm/7184932/webrev.03/ > > > -Rob > Thanks Rob, you've fixed all the issues that I was concerned about so I think the implementation is looking good now. I have not reviewed the test yet, I hope to get to it soon. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120823/cc089480/attachment.html From Alan.Bateman at oracle.com Thu Aug 23 05:13:28 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 23 Aug 2012 13:13:28 +0100 Subject: Problems persist in KQueueSelectorProvider (Mac) in 7u6 ea In-Reply-To: References: <50259B07.1080306@redhat.com> <5028B69F.5060907@oracle.com> <502951ED.4060704@redhat.com> <502974B8.1080206@oracle.com> <46F92297-591C-4A6E-85CF-2943A4A3F099@redhat.com> <502BA75A.9020101@oracle.com> <502EBB10.3040308@redhat.com> <50328DBD.5040401@oracle.com> Message-ID: <50361E68.2070402@oracle.com> Jason, I've pushed your patch to jdk8/tl/jdk: http://hg.openjdk.java.net/jdk8/tl/jdk/rev/de5a85353f4d Thanks for taking the time to work on it, trying it out on different MacOSX releases, etc. From the thread the it sounds like there is lurking issue here so we'll need to keep an eye our for any other sightings. Also there are other potential improvements that have been mentioned here, they can be follow-up change-sets if needed. One other thing, the above repository is jdk8 and I think it's best to let it bake there for a bit before proposing it for jdk7u. 7u8 is open for changes at this time and is probably the more suitable release to target. -Alan From Ulf.Zibis at CoSoCo.de Thu Aug 23 05:59:42 2012 From: Ulf.Zibis at CoSoCo.de (Ulf Zibis) Date: Thu, 23 Aug 2012 14:59:42 +0200 Subject: Code Review Request: 7168172: (fs) Files.isReadable slow on Windows In-Reply-To: References: <50352740.8060302@oracle.com> <50356C43.1080706@CoSoCo.de> Message-ID: <5036293E.2010002@CoSoCo.de> Am 23.08.2012 01:50, schrieb Zhong Yu: >> public void checkAccess(Path obj, AccessMode... modes) throws >> IOException { >> AccessMode r = null; >> AccessMode w = null; >> AccessMode x = null; >> for (AccessMode mode: modes) { >> switch (mode) { >> case READ : r = READ; break; >> case WRITE : w = WRITE; break; >> case EXECUTE : x = EXECUTE; break; >> default: throw new AssertionError("Should not get here"); >> } >> } > That'll leave r==null if `modes` is empty. But an empty `modes` is > supposed to be equivalent to [READ]. Yes, but in that case w and x are also null, so the "special-case read access" will be invoked anyway, and the null-ness of r wouldn't matter. >> WindowsPath file = WindowsPath.toWindowsPath(obj); >> >> // special-case read access to avoid needing to determine effective >> // access to file >> if (w == null && x == null) { >> tryReadAccess(file); >> return; >> } >> >> ... >> } >> >> -Ulf >> From ahughes at redhat.com Thu Aug 23 07:35:20 2012 From: ahughes at redhat.com (Andrew Hughes) Date: Thu, 23 Aug 2012 10:35:20 -0400 (EDT) Subject: 7191556: (fs) UnixNativeDispatcher.getextmntent should be moved into platform specific code In-Reply-To: <5034D48F.7090709@oracle.com> Message-ID: <724055071.9755134.1345732520495.JavaMail.root@redhat.com> ----- Original Message ----- > (apologies for the delay replying to you, lots of things on my plate > at > the moment). > It's ok. There's no rush. I was just wondering if it had got lost in the tubes. > On 16/08/2012 17:32, Andrew Hughes wrote: > > : > >> Of course this also brings up > >> the > >> topic of having code for several platforms in src/solaris. We've > >> been > >> kicking that can down the road for several years but that's a > >> bigger > >> topic. > > I don't think this is a huge issue. I tend to just read solaris in > > that > > context as posix. It seems we now have a macosx one with plenty of > > code > > in, but then they have all this proprietary GUI stuff. The POSIX > > APIs > > shouldn't differ that much. From what I could see, the PPC-AIX > > folks > > have managed to get the JDK building on AIX just by altering the > > solaris > > code slightly. > It's a not major issue but it is one that comes up regularly, mostly > because src/solaris requires a lot more than POSIX and because it > also > contains both Solaris and Linux specific code. If OpenJDK is going to > take on more platforms then I do think some clean-up is required > although it's hard to spend time on such things, and of course there > is > never a right time. True. This actually sounds very much like cleanup work that could be a good introduction for new OpenJDK developers. There's not any real code that's got to be written. It just needs someone to find the Solaris/Linux/etc.-specific code, break it out into new files and test the result heavily. At least, that's the impression I get. > > > : > > Ok, what I was thinking was something like the following. I can > > draft > > some code if it helps, but basically the current situation is: > > > > Current situation: > > > > java.awt.Desktop --> GNOME-VFS > > > > sun.net.spi.DefaultProxySelector --> GConf > > > > sun.nio.fs.GnomeFileTypeDetector --> GIO or GNOME-VFS > > > > Proposed solution: > > > > Above classes all depend on a common SystemServiceProvider which > > determines > > whether to use GIO or GNOME2 (GConf/GNOME-VFS) to provide the > > required functionality (GIO > > in newer versions of GLib replaces all three). We could possibly > > extend > > this to other platforms, but cleaning up the scattered POSIX > > implementations > > would be a start. > > > > I'm not sure of an appropriate package; perhaps sun.misc which > > already includes > > OSEnvironment (a pointless class at present on Solaris& Linux)? > So are you thinking about something relatively high level that hides > whether the underlying implementation is GIO or GConf or whatever? > Or > maybe you are thinking about a simple shim over all the above that > the > classes in the JDK could use? I think once you have some code to > discuss > then it will become clearer as to possible naming and location (my > preference would be to not add it to sun.misc, mostly because so many > developer think it's fair game to use anything in that package, it's > really hard to actually remove or change things in sun.misc as a > result). Sure, I can write something up. I'm pretty much in the dark as the best package name to choose though (I wasn't aware of the significance of sun.misc either way). I'm looking for something that would be usable by code in the NIO, net and AWT packages. Any ideas? > > > > Going through these also highlights the issue that MacOS will also > > presumably > > try to use GConf for proxy functionality at present; I don't see > > anything overriding > > it. For java.awt.Desktop, support is provided in the Mac AWT > > implementation. > > > The Mac port picks up the proxy settings from the system > (java_props_macosx.c). It's a bit different to the other platforms in > that "use system proxy settings" is essentially enabled by default > (which on the surface is great but also causes problems, particular > for > tests that make local connections). > Ah, right. I didn't see that. > -Alan. > > -- Andrew :) Free Java Software Engineer Red Hat, Inc. (http://www.redhat.com) PGP Key: 248BDC07 (https://keys.indymedia.org/) Fingerprint = EC5A 1F5E C0AD 1D15 8F1F 8F91 3B96 A578 248B DC07 From zhong.j.yu at gmail.com Thu Aug 23 08:08:51 2012 From: zhong.j.yu at gmail.com (Zhong Yu) Date: Thu, 23 Aug 2012 10:08:51 -0500 Subject: Code Review Request: 7168172: (fs) Files.isReadable slow on Windows In-Reply-To: <503627B7.7050604@gmx.de> References: <50352740.8060302@oracle.com> <50356C43.1080706@CoSoCo.de> <503627B7.7050604@gmx.de> Message-ID: Ulf you are right. Question about release schedule: will this fix be included in a jdk7 release? On Thu, Aug 23, 2012 at 7:53 AM, Ulf Zibis wrote: > Am 23.08.2012 01:50, schrieb Zhong Yu: >>> >>> public void checkAccess(Path obj, AccessMode... modes) throws >>> >>> IOException { >>> AccessMode r = null; >>> AccessMode w = null; >>> AccessMode x = null; >>> for (AccessMode mode: modes) { >>> switch (mode) { >>> case READ : r = READ; break; >>> case WRITE : w = WRITE; break; >>> case EXECUTE : x = EXECUTE; break; >>> default: throw new AssertionError("Should not get >>> here"); >>> } >>> } >> >> That'll leave r==null if `modes` is empty. But an empty `modes` is >> supposed to be equivalent to [READ]. > > > Yes, but in that case w and x are also null, so the "special-case read > access" will be invoked anyway, and the null-ness of r wouldn't matter. > >>> WindowsPath file = WindowsPath.toWindowsPath(obj); >>> >>> // special-case read access to avoid needing to determine >>> effective >>> // access to file >>> if (w == null && x == null) { >>> tryReadAccess(file); >>> return; >>> } >>> >>> ... >>> } >>> >>> -Ulf >>> > From kurchi.subhra.hazra at oracle.com Thu Aug 23 09:11:14 2012 From: kurchi.subhra.hazra at oracle.com (Kurchi Subhra Hazra) Date: Thu, 23 Aug 2012 09:11:14 -0700 Subject: Code Review Request: 7168172: (fs) Files.isReadable slow on Windows In-Reply-To: References: <50352740.8060302@oracle.com> <50356C43.1080706@CoSoCo.de> <503627B7.7050604@gmx.de> Message-ID: <50365622.5040407@oracle.com> Yes I plan to backport it to 7u8. - Kurchi On 8/23/12 8:08 AM, Zhong Yu wrote: > Ulf you are right. > > Question about release schedule: will this fix be included in a jdk7 release? > > On Thu, Aug 23, 2012 at 7:53 AM, Ulf Zibis wrote: >> Am 23.08.2012 01:50, schrieb Zhong Yu: >>>> public void checkAccess(Path obj, AccessMode... modes) throws >>>> >>>> IOException { >>>> AccessMode r = null; >>>> AccessMode w = null; >>>> AccessMode x = null; >>>> for (AccessMode mode: modes) { >>>> switch (mode) { >>>> case READ : r = READ; break; >>>> case WRITE : w = WRITE; break; >>>> case EXECUTE : x = EXECUTE; break; >>>> default: throw new AssertionError("Should not get >>>> here"); >>>> } >>>> } >>> That'll leave r==null if `modes` is empty. But an empty `modes` is >>> supposed to be equivalent to [READ]. >> >> Yes, but in that case w and x are also null, so the "special-case read >> access" will be invoked anyway, and the null-ness of r wouldn't matter. >> >>>> WindowsPath file = WindowsPath.toWindowsPath(obj); >>>> >>>> // special-case read access to avoid needing to determine >>>> effective >>>> // access to file >>>> if (w == null&& x == null) { >>>> tryReadAccess(file); >>>> return; >>>> } >>>> >>>> ... >>>> } >>>> >>>> -Ulf >>>> From Alan.Bateman at oracle.com Thu Aug 23 12:07:23 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 23 Aug 2012 20:07:23 +0100 Subject: Code Review Request: 7168172: (fs) Files.isReadable slow on Windows In-Reply-To: <50356C43.1080706@CoSoCo.de> References: <50352740.8060302@oracle.com> <50356C43.1080706@CoSoCo.de> Message-ID: <50367F6B.2050506@oracle.com> On 23/08/2012 00:33, Ulf Zibis wrote: > Am 22.08.2012 20:38, schrieb Kurchi Hazra: >> However, it is faster to simply open the file being checked and close >> it. In the case of a directory then, a DirectoryStream can be opened >> and closed to check the same. > > I would rename checkReadAccess(WindowsPath file) to > tryReadAccess(WindowsPath file) The API that the provider defines is checkAccess so I think it's okay to have this private specialized method for read be named checkReadAccess. > : > > > At least the boolean one could be saved easily and (modes.length == 0) > is superfluous: > > public void checkAccess(Path obj, AccessMode... modes) throws > IOException { > AccessMode r = null; > AccessMode w = null; > AccessMode x = null; > for (AccessMode mode: modes) { > switch (mode) { > case READ : r = READ; break; > case WRITE : w = WRITE; break; > case EXECUTE : x = EXECUTE; break; > default: throw new AssertionError("Should not get here"); > } > } > > WindowsPath file = WindowsPath.toWindowsPath(obj); > > // special-case read access to avoid needing to determine > effective > // access to file > if (w == null && x == null) { > tryReadAccess(file); > return; > } You're right that the modes.length == 0 check could be skipped and then it becomes "if (!w && !x) { ... }". I'll leave it to Kurchi as to whether she wants to change this. I'm okay either way. -Alan From kurchi.subhra.hazra at oracle.com Thu Aug 23 14:17:45 2012 From: kurchi.subhra.hazra at oracle.com (Kurchi Hazra) Date: Thu, 23 Aug 2012 14:17:45 -0700 Subject: Code Review Request: 7168172: (fs) Files.isReadable slow on Windows In-Reply-To: <50367F6B.2050506@oracle.com> References: <50352740.8060302@oracle.com> <50356C43.1080706@CoSoCo.de> <50367F6B.2050506@oracle.com> Message-ID: <50369DF9.7080908@oracle.com> Here is an updated webrev: http://cr.openjdk.java.net/~khazra/7168172/webrev.01/ I left the booleans as it is, but removed the modes.length() check. Thanks, Kurchi On 8/23/2012 12:07 PM, Alan Bateman wrote: > On 23/08/2012 00:33, Ulf Zibis wrote: >> Am 22.08.2012 20:38, schrieb Kurchi Hazra: >>> However, it is faster to simply open the file being checked and >>> close it. In the case of a directory then, a DirectoryStream can be >>> opened and closed to check the same. >> >> I would rename checkReadAccess(WindowsPath file) to >> tryReadAccess(WindowsPath file) > The API that the provider defines is checkAccess so I think it's okay > to have this private specialized method for read be named > checkReadAccess. > > >> : >> >> >> At least the boolean one could be saved easily and (modes.length == >> 0) is superfluous: >> >> public void checkAccess(Path obj, AccessMode... modes) throws >> IOException { >> AccessMode r = null; >> AccessMode w = null; >> AccessMode x = null; >> for (AccessMode mode: modes) { >> switch (mode) { >> case READ : r = READ; break; >> case WRITE : w = WRITE; break; >> case EXECUTE : x = EXECUTE; break; >> default: throw new AssertionError("Should not get >> here"); >> } >> } >> >> WindowsPath file = WindowsPath.toWindowsPath(obj); >> >> // special-case read access to avoid needing to determine >> effective >> // access to file >> if (w == null && x == null) { >> tryReadAccess(file); >> return; >> } > You're right that the modes.length == 0 check could be skipped and > then it becomes "if (!w && !x) { ... }". I'll leave it to Kurchi as to > whether she wants to change this. I'm okay either way. > > -Alan > -- -Kurchi From Alan.Bateman at oracle.com Fri Aug 24 01:04:16 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 24 Aug 2012 09:04:16 +0100 Subject: Code Review Request: 7168172: (fs) Files.isReadable slow on Windows In-Reply-To: <50369DF9.7080908@oracle.com> References: <50352740.8060302@oracle.com> <50356C43.1080706@CoSoCo.de> <50367F6B.2050506@oracle.com> <50369DF9.7080908@oracle.com> Message-ID: <50373580.1050503@oracle.com> On 23/08/2012 22:17, Kurchi Hazra wrote: > > Here is an updated webrev: > http://cr.openjdk.java.net/~khazra/7168172/webrev.01/ > > I left the booleans as it is, but removed the modes.length() check. > This looks good to me and thanks for taking Ulf's comment to avoid needing to check is modes has 0 elements. -Alan. From Ulf.Zibis at CoSoCo.de Fri Aug 24 04:00:55 2012 From: Ulf.Zibis at CoSoCo.de (Ulf Zibis) Date: Fri, 24 Aug 2012 13:00:55 +0200 Subject: Code Review Request: 7168172: (fs) Files.isReadable slow on Windows In-Reply-To: <50369DF9.7080908@oracle.com> References: <50352740.8060302@oracle.com> <50356C43.1080706@CoSoCo.de> <50367F6B.2050506@oracle.com> <50369DF9.7080908@oracle.com> Message-ID: <50375EE7.5010604@CoSoCo.de> Am 23.08.2012 23:17, schrieb Kurchi Hazra: > > Here is an updated webrev: http://cr.openjdk.java.net/~khazra/7168172/webrev.01/ > > I left the booleans as it is, but removed the modes.length() check. By error I had removed your former comment: // default if modes not specified Maybe you better integrate it in: // special-case read access to avoid needing to determine effective // access to file ... so people will better understand the trick. To my taste the line: WindowsPath file = WindowsPath.toWindowsPath(obj); ... should be moved to the place, where it is actually needed. -Ulf >> On 23/08/2012 00:33, Ulf Zibis wrote: >> >>> At least the boolean one could be saved easily and (modes.length == 0) is superfluous: >>> >>> public void checkAccess(Path obj, AccessMode... modes) throws IOException { >>> AccessMode r = null; >>> AccessMode w = null; >>> AccessMode x = null; >>> for (AccessMode mode: modes) { >>> switch (mode) { >>> case READ : r = READ; break; >>> case WRITE : w = WRITE; break; >>> case EXECUTE : x = EXECUTE; break; >>> default: throw new AssertionError("Should not get here"); >>> } >>> } >>> >>> WindowsPath file = WindowsPath.toWindowsPath(obj); >>> >>> // special-case read access to avoid needing to determine effective >>> // access to file >>> if (w == null && x == null) { >>> tryReadAccess(file); >>> return; >>> } > From Alan.Bateman at oracle.com Fri Aug 24 04:17:15 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 24 Aug 2012 12:17:15 +0100 Subject: Code Review Request: 7168172: (fs) Files.isReadable slow on Windows In-Reply-To: <50375EE7.5010604@CoSoCo.de> References: <50352740.8060302@oracle.com> <50356C43.1080706@CoSoCo.de> <50367F6B.2050506@oracle.com> <50369DF9.7080908@oracle.com> <50375EE7.5010604@CoSoCo.de> Message-ID: <503762BB.5010107@oracle.com> On 24/08/2012 12:00, Ulf Zibis wrote: > : > > > To my taste the line: > WindowsPath file = WindowsPath.toWindowsPath(obj); > ... should be moved to the place, where it is actually needed. > > -Ulf > It's just that in the original implementation that the parameters are checked in order, in this case the path parameter is checked for null and that it is associated with the right provider before it checks the modes parameter. -Alan From kurchi.subhra.hazra at oracle.com Fri Aug 24 11:00:11 2012 From: kurchi.subhra.hazra at oracle.com (Kurchi Hazra) Date: Fri, 24 Aug 2012 11:00:11 -0700 Subject: Code Review Request: 7168172: (fs) Files.isReadable slow on Windows In-Reply-To: <50375EE7.5010604@CoSoCo.de> References: <50352740.8060302@oracle.com> <50356C43.1080706@CoSoCo.de> <50367F6B.2050506@oracle.com> <50369DF9.7080908@oracle.com> <50375EE7.5010604@CoSoCo.de> Message-ID: <5037C12B.70509@oracle.com> Thanks for the reviews. I'll integrate the comments and push this fix then. - Kurchi On 8/24/2012 4:00 AM, Ulf Zibis wrote: > > Am 23.08.2012 23:17, schrieb Kurchi Hazra: >> >> Here is an updated webrev: >> http://cr.openjdk.java.net/~khazra/7168172/webrev.01/ >> >> I left the booleans as it is, but removed the modes.length() check. > > By error I had removed your former comment: > // default if modes not specified > > Maybe you better integrate it in: > // special-case read access to avoid needing to determine > effective > // access to file > > ... so people will better understand the trick. > > To my taste the line: > WindowsPath file = WindowsPath.toWindowsPath(obj); > ... should be moved to the place, where it is actually needed. > > -Ulf > >>> On 23/08/2012 00:33, Ulf Zibis wrote: >>> >>>> At least the boolean one could be saved easily and (modes.length == >>>> 0) is superfluous: >>>> >>>> public void checkAccess(Path obj, AccessMode... modes) throws >>>> IOException { >>>> AccessMode r = null; >>>> AccessMode w = null; >>>> AccessMode x = null; >>>> for (AccessMode mode: modes) { >>>> switch (mode) { >>>> case READ : r = READ; break; >>>> case WRITE : w = WRITE; break; >>>> case EXECUTE : x = EXECUTE; break; >>>> default: throw new AssertionError("Should not get >>>> here"); >>>> } >>>> } >>>> >>>> WindowsPath file = WindowsPath.toWindowsPath(obj); >>>> >>>> // special-case read access to avoid needing to determine >>>> effective >>>> // access to file >>>> if (w == null && x == null) { >>>> tryReadAccess(file); >>>> return; >>>> } >> > -- -Kurchi From Ulf.Zibis at CoSoCo.de Fri Aug 24 12:38:23 2012 From: Ulf.Zibis at CoSoCo.de (Ulf Zibis) Date: Fri, 24 Aug 2012 21:38:23 +0200 Subject: hg: jdk8/tl/jdk: 7168172: (fs) Files.isReadable slow on Windows In-Reply-To: <20120824184950.4B799476ED@hg.openjdk.java.net> References: <20120824184950.4B799476ED@hg.openjdk.java.net> Message-ID: <5037D82F.4040004@CoSoCo.de> Kurchi, thanks for listing me. BTW, my official openJDK ID is ulfzibis -Ulf Am 24.08.2012 20:49, schrieb kurchi.subhra.hazra at oracle.com: > Changeset: bd91a601265c > Author: khazra > Date: 2012-08-24 11:48 -0700 > URL: http://hg.openjdk.java.net/jdk8/tl/jdk/rev/bd91a601265c > > 7168172: (fs) Files.isReadable slow on Windows > Summary: Remove DACL checking for read access, also reviewed by Ulf.Zibis at CoSoCo.de, zhong.j.yu at gmail.com > Reviewed-by: alanb > > ! src/windows/classes/sun/nio/fs/WindowsFileSystemProvider.java > > From Ulf.Zibis at CoSoCo.de Fri Aug 24 12:42:03 2012 From: Ulf.Zibis at CoSoCo.de (Ulf Zibis) Date: Fri, 24 Aug 2012 21:42:03 +0200 Subject: hg: jdk8/tl/jdk: 7168172: (fs) Files.isReadable slow on Windows In-Reply-To: <5037D8C9.5090800@oracle.com> References: <20120824184950.4B799476ED@hg.openjdk.java.net> <5037D40C.3050302@CoSoCo.de> <5037D8C9.5090800@oracle.com> Message-ID: <5037D90B.7070704@CoSoCo.de> Oops, thanks for your correction, Phil. Sorry, I wasn't aware about that difference. -Ulf Am 24.08.2012 21:40, schrieb Phil Race: > It may not be enforced by tools, but I don't think you can be listed as > a reviewer there unless you have the JDK 8 project reviewer attribute :- > > Alan does - http://openjdk.java.net/census#alanb > > but you don't - http://openjdk.java.net/census#ulfzibis > > Now maybe you should ask if you can be .. > > -phil. > > On 8/24/2012 12:20 PM, Ulf Zibis wrote: >> Kurchi, >> >> thanks for listing me. >> >> BTW, my official openJDK ID is ulfzibis >> >> -Ulf >> >> Am 24.08.2012 20:49, schrieb kurchi.subhra.hazra at oracle.com: >>> Changeset: bd91a601265c >>> Author: khazra >>> Date: 2012-08-24 11:48 -0700 >>> URL: http://hg.openjdk.java.net/jdk8/tl/jdk/rev/bd91a601265c >>> >>> 7168172: (fs) Files.isReadable slow on Windows >>> Summary: Remove DACL checking for read access, also reviewed by Ulf.Zibis at CoSoCo.de, >>> zhong.j.yu at gmail.com >>> Reviewed-by: alanb >>> >>> ! src/windows/classes/sun/nio/fs/WindowsFileSystemProvider.java >>> >>> >> > > From Alan.Bateman at oracle.com Sun Aug 26 03:26:53 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sun, 26 Aug 2012 11:26:53 +0100 Subject: 7191556: (fs) UnixNativeDispatcher.getextmntent should be moved into platform specific code In-Reply-To: <724055071.9755134.1345732520495.JavaMail.root@redhat.com> References: <724055071.9755134.1345732520495.JavaMail.root@redhat.com> Message-ID: <5039F9ED.2010500@oracle.com> On 23/08/2012 15:35, Andrew Hughes wrote: > : > True. This actually sounds very much like cleanup work that could be a > good introduction for new OpenJDK developers. There's not any real > code that's got to be written. It just needs someone to find the > Solaris/Linux/etc.-specific code, break it out into new files and > test the result heavily. At least, that's the impression I get. Implementation-wise then some of it would be clean-up and refactoring but it's a direction that would require broader discussion and agreement first. The topic has been touched on various lists and not always clear to me that there is stomach for it (everyone is busy, it would be disruptive, etc.). Once we get over the transition to the new build system then it may be a good topic to bring up again. AIX or other ports wanting to come into the main line is probably going to force a discussion too. > : > Sure, I can write something up. I'm pretty much in the dark as the best > package name to choose though (I wasn't aware of the significance of sun.misc > either way). I'm looking for something that would be usable by code in > the NIO, net and AWT packages. Any ideas? I'm not sure about the package either but I don't think it matters too much at this point as long as it's not sun.awt (from a modularity perspective it is okay to have AWT -> core dependencies but core -> AWT is problematic). A possible starting point might be sun.util.gio or sun.util.gnomeservices and we figure the package name out later. It really depends on whether you are thinking about internal APIs for each of these or whether you are thinking of a facade. -Alan. From xueming.shen at oracle.com Sun Aug 26 15:05:44 2012 From: xueming.shen at oracle.com (Xueming Shen) Date: Sun, 26 Aug 2012 15:05:44 -0700 Subject: Fwd: 7050570: (fs) FileSysteProvider fails to initializes if run with file.encoding set to Cp037 In-Reply-To: <50376B9F.10705@oracle.com> References: <502B90A3.2020601@oracle.com> <50376B9F.10705@oracle.com> Message-ID: <503A9DB8.8080509@oracle.com> Alan, change looks fine. Is there possibility that SolarisUserDefinedFileAttributeView might be used/compiled for a non-ascii based, for example ebcdic, platform. It might be a potential problem if the source code is compiled with encoding flag -ascii and be targeted for a non-ascii platform. Otherwise, not an issue. -Sherman > Subject: 7050570: (fs) FileSysteProvider fails to initializes if run > with file.encoding set to Cp037 > Date: Wed, 15 Aug 2012 13:05:55 +0100 > From: Alan Bateman > To: nio-dev > > > > > This is another one that has been in my patch queue for a while, > actually from before jdk7 shipped as it arose with someone trying > tests with file.encoding set on the command line (something that has > never been supported but sometimes useful for tests). It's also an > issue that came up recently on the core-libs list [1] where someone > was trying to do something similar. > > The changes here just change the file system provider to use > sun.jnu.encoding property as the charset to use when encoding/decoding > file names. I think eventually we should be use this consistently and > leaving file.encoding to the file content. The assumption remains in > the code that there aren't any locales that map to UTF-16 or other > encoding that require shift in/out. If there were such a locale then > more further changes would be required in order to deal with > multi-byte slash a few others used in the code. > > Sherman - I think you would be the best person to review this. The > webrev with the changes is here: > http://cr.openjdk.java.net/~alanb/7050570/webrev/ > > -Alan. > > [1] > http://mail.openjdk.java.net/pipermail/core-libs-dev/2012-July/010758.html -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120826/f39578c7/attachment.html From Ulf.Zibis at CoSoCo.de Mon Aug 27 01:47:11 2012 From: Ulf.Zibis at CoSoCo.de (Ulf Zibis) Date: Mon, 27 Aug 2012 10:47:11 +0200 Subject: A bug in filesystem bootstrap (unix/ linux) prevents In-Reply-To: <4FF547DB.10400@oracle.com> References: <4FF443F1.40502@CoSoCo.de> <4FF4E958.2020103@oracle.com> <4FF547DB.10400@oracle.com> Message-ID: <503B340F.6050007@CoSoCo.de> Hi Sherman, what is "jnu" standing for? You have touched on 3 "classes" of system encodings: - default encoding of the file content - the encoding of the file path - the "text" encoding when use the platform APIs Are there more? > so in theory file.encoding should be used to only for the encoding of "file content", and > the sun.jnu.encoding should be used when you need the encoding to talk to those platform APIs Which property is used for the encoding of the file path? In Charset.defaultCharset() it is not specified, on which of those 3+? "classes" this method refers. IMHO this should be done! Thanks, -Ulf Am 05.07.2012 09:52, schrieb Xueming Shen: > The code cited is a little shortcut, if there is locale over there is indeed using > utf-16, or any encoding that needs to switch/shift into ASCII (or its single byte > charset area) with a shift/in/out character.. So far I'm not aware of any such > a locale on any our supported platform. Historically, this kind of assumption > might run into trouble when being ported to other platform, such as ebcdic > based system (but I don't think it's a problem in this case). Ideally, the code > probably should be coded to be able to deal with a mb type of "/", but obviously > it was decided to take the short-cut for better performance here. > > "We" have been taking the stand that file.encoding is an informative/read-only > system property for a long time, mainly because of two reasons. First this > property is really defined/implemented/used as the default encoding that the jvm > uses to communicated with the underlying platform for local/encoding sensitive > stuff, the default encoding of the file content, the encoding of the file path and > the "text" encoding when use the platform APIs, for example. It's like a "contract" > between the jvm and the underlying platform, it needs to be understood by both > and agreed on by both. So it needs to be set based on what your underlying system > is using, not something you want to set via either -D or System.setProperty. If > your underlying locale is not UTF-16, I don't think you should expect the jvm > could work correctly if it keeps "talking" in UTF-16 to the underlying system, > for example, pass in a file name in utf-16, when your are running on a utf-8 > locale (it is more complicated on a windows platform, when you have system > locale and user locale, and historically file.encoding was used for both, consider > if your system locale and user locale are set differently...). > > The property sun.jnu.encoding introduced in jdk6 (this is mainly > to address the issue we have with file.encoding on windows platform though) > somehow helps remove some "pressure" from the file.encoding, so in theory > file.encoding should be used to only for the encoding of "file content", and > the sun.jnu.encoding should be used when you need the encoding to talk to > those platform APIs, so something might be done here (currently file.encoding > and sun.jnu.encoding are set to the same thing on non-Windows platform). > > The other reason is the timing of how the file.encoding is being initialized and > how it is being used during the "complicated" system initialization stage, almost > everyone touched System. initializeSystemClass() got burned here and there > in the past:-) So sometime you want to ask if it is worth the risk to change > something work for a use scenario that is not "supported". That said, as > I said above, something might be done to address this issue, but obviously > not a priority for now. > > -Sherman > > if you want to do -Dfile.encoding=xyz, you > are on your own, it might work, it might not work. > From Ulf.Zibis at CoSoCo.de Mon Aug 27 02:53:09 2012 From: Ulf.Zibis at CoSoCo.de (Ulf Zibis) Date: Mon, 27 Aug 2012 11:53:09 +0200 Subject: Fwd: 7050570: (fs) FileSysteProvider fails to initializes if run with file.encoding set to Cp037 In-Reply-To: <503A9DB8.8080509@oracle.com> References: <502B90A3.2020601@oracle.com> <50376B9F.10705@oracle.com> <503A9DB8.8080509@oracle.com> Message-ID: <503B4385.6080809@CoSoCo.de> Hi, shouldn't we better code? : -53 if (this.defaultDirectory[0] != '/') { +53 if (this.defaultDirectory[0] != (byte)'/') { It would make it more clear to the reader, that only single bytes are compared here, and maybe would prevent from the problem when "source code encoding != platform encoding". At least there should be an explaining comment. But why not simply? : 51 this.provider = provider; 52 String normalizedDir = UnixPath.normalizeAndCheck(dir); 53 if (normalizedDir.charAt(0) != '/') { 54 throw new RuntimeException("default directory must be absolute"); 55 } 52 this.defaultDirectory = Util.toBytes(normalizedDir); Anyway there already is UnixPath.isAbsolute() to use. Similar in SolarisUserDefinedFileAttributeView. But anyway better code: 45 private static final byte[] HERE = Util.toBytes("."); 46 47 private byte[] nameAsBytes(UnixPath file, String name) throws IOException { 48 byte[] bytes = Util.toBytes(name); 49 // "", "." and ".." not allowed 50 if (bytes.length == 0 || (bytes[0] == '.' && 51 (bytes.length == 1 || (bytes.length == 2 && bytes[1] == '.'))) { 52 .... -Ulf Am 27.08.2012 00:05, schrieb Xueming Shen: > Alan, change looks fine. > > Is there possibility that SolarisUserDefinedFileAttributeView might be used/compiled > for a non-ascii based, for example ebcdic, platform. It might be a potential problem > if the source code is compiled with encoding flag -ascii and be targeted for a non-ascii > platform. Otherwise, not an issue. > > -Sherman > >> Subject: 7050570: (fs) FileSysteProvider fails to initializes if run with file.encoding set to >> Cp037 >> Date: Wed, 15 Aug 2012 13:05:55 +0100 >> From: Alan Bateman >> To: nio-dev >> >> >> >> >> This is another one that has been in my patch queue for a while, actually from before jdk7 >> shipped as it arose with someone trying tests with file.encoding set on the command line >> (something that has never been supported but sometimes useful for tests). It's also an issue that >> came up recently on the core-libs list [1] where someone was trying to do something similar. >> >> The changes here just change the file system provider to use sun.jnu.encoding property as the >> charset to use when encoding/decoding file names. I think eventually we should be use this >> consistently and leaving file.encoding to the file content. The assumption remains in the code >> that there aren't any locales that map to UTF-16 or other encoding that require shift in/out. If >> there were such a locale then more further changes would be required in order to deal with >> multi-byte slash a few others used in the code. >> >> Sherman - I think you would be the best person to review this. The webrev with the changes is here: >> http://cr.openjdk.java.net/~alanb/7050570/webrev/ >> >> -Alan. >> >> [1] http://mail.openjdk.java.net/pipermail/core-libs-dev/2012-July/010758.html > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120827/47ccbc8b/attachment.html From Alan.Bateman at oracle.com Mon Aug 27 03:58:37 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 27 Aug 2012 11:58:37 +0100 Subject: Fwd: 7050570: (fs) FileSysteProvider fails to initializes if run with file.encoding set to Cp037 In-Reply-To: <503B4385.6080809@CoSoCo.de> References: <502B90A3.2020601@oracle.com> <50376B9F.10705@oracle.com> <503A9DB8.8080509@oracle.com> <503B4385.6080809@CoSoCo.de> Message-ID: <503B52DD.5030807@oracle.com> On 27/08/2012 10:53, Ulf Zibis wrote: > Hi, > > shouldn't we better code? : > -53 if (this.defaultDirectory[0] != '/') { > +53 if (this.defaultDirectory[0] != (byte)'/') { > It would make it more clear to the reader, that only single bytes are > compared here, and maybe would prevent from the problem when "source > code encoding != platform encoding". > At least there should be an explaining comment. > > But why not simply? : > 51 this.provider = provider; > 52 String normalizedDir = UnixPath.normalizeAndCheck(dir); > 53 if (normalizedDir.charAt(0) != '/') { > 54 throw new RuntimeException("default directory must be > absolute"); > 55 } > 52 this.defaultDirectory = Util.toBytes(normalizedDir); > > Anyway there already is UnixPath.isAbsolute() to use. Thanks for helping to review this. One thing to say that this is FileSystem creation time so we need to be careful about invoking instance methods on UnixPath until initialization it complete. Also given the assumptions then I don't think it matters here whether it's byte 0 or char 0 but it's the bytes that are used for the syscalls and so if byte 0 is '/' then it's considered an absolute path by the operating system. However a comment to explain what is going on seems necessary so we should do. > > Similar in SolarisUserDefinedFileAttributeView. > But anyway better code: > > 45 private static final byte[] HERE = Util.toBytes("."); > 46 > 47 private byte[] nameAsBytes(UnixPath file, String name) throws > IOException { > 48 byte[] bytes = Util.toBytes(name); > 49 // "", "." and ".." not allowed > 50 if (bytes.length == 0 || (bytes[0] == '.' && > 51 (bytes.length == 1 || (bytes.length == 2 && > bytes[1] == '.'))) { > 52 .... This hasn't been changed but I think you are saying that the nested "if" isn't necessary. This seems a good clean-up to me too - thanks! -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120827/fc0272da/attachment-0001.html From Ulf.Zibis at CoSoCo.de Mon Aug 27 04:57:42 2012 From: Ulf.Zibis at CoSoCo.de (Ulf Zibis) Date: Mon, 27 Aug 2012 13:57:42 +0200 Subject: Fwd: 7050570: (fs) FileSysteProvider fails to initializes if run with file.encoding set to Cp037 In-Reply-To: <503B52DD.5030807@oracle.com> References: <502B90A3.2020601@oracle.com> <50376B9F.10705@oracle.com> <503A9DB8.8080509@oracle.com> <503B4385.6080809@CoSoCo.de> <503B52DD.5030807@oracle.com> Message-ID: <503B60B6.7050905@CoSoCo.de> Am 27.08.2012 12:58, schrieb Alan Bateman: > On 27/08/2012 10:53, Ulf Zibis wrote: >> Hi, >> >> shouldn't we better code? : >> -53 if (this.defaultDirectory[0] != '/') { >> +53 if (this.defaultDirectory[0] != (byte)'/') { >> It would make it more clear to the reader, that only single bytes are compared here, and maybe >> would prevent from the problem when "source code encoding != platform encoding". >> At least there should be an explaining comment. >> >> But why not simply? : >> 51 this.provider = provider; >> 52 String normalizedDir = UnixPath.normalizeAndCheck(dir); >> 53 if (normalizedDir.charAt(0) != '/') { >> 54 throw new RuntimeException("default directory must be absolute"); >> 55 } >> 52 this.defaultDirectory = Util.toBytes(normalizedDir); >> >> Anyway there already is UnixPath.isAbsolute() to use. > Thanks for helping to review this. > > One thing to say that this is FileSystem creation time so we need to be careful about invoking > instance methods on UnixPath until initialization it complete. You are right, we would need a static isAbsolute(String path), which could be used from both classes, like it is already the case for normalizeAndCheck(dir). > Also given the assumptions then I don't think it matters here whether it's byte 0 or char 0 It matters, if sun.jnu.encoding is a double-byte coding. ... and compared to the anyway processed normalizeAndCheck(dir), performance doesn't matter here. > but it's the bytes that are used for the syscalls and so if byte 0 is '/' then it's considered an > absolute path by the operating system. However a comment to explain what is going on seems > necessary so we should do. :-) >> Similar in SolarisUserDefinedFileAttributeView. >> But anyway better code: >> >> 45 private static final byte[] HERE = Util.toBytes("."); >> 46 >> 47 private byte[] nameAsBytes(UnixPath file, String name) throws IOException { >> 48 byte[] bytes = Util.toBytes(name); >> 49 // "", "." and ".." not allowed >> 50 if (bytes.length == 0 || (bytes[0] == '.' && >> 51 (bytes.length == 1 || (bytes.length == 2 && bytes[1] == '.'))) { >> 52 .... > This hasn't been changed but I think you are saying that the nested "if" isn't necessary. This > seems a good clean-up to me too - thanks! Yes, and additionally I wanted to say, that constant HERE should be initialized double-byte encoding proof. I still do not see any reasonable why this UnixFileSystem-API is partly processing chars vs. bytes. E.G compare UnixPath.normalize(...) vs. UnixPath.initOffsets(). A last thought: I don't like the naming: Util.toString(...). toString has a dedicated meaning in the Java world, which is not meant here. What about - String jnuDecode(byte[]) - byte[] jnuEncode(String) -Ulf -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120827/42235749/attachment.html From Alan.Bateman at oracle.com Mon Aug 27 05:12:06 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 27 Aug 2012 13:12:06 +0100 Subject: Fwd: 7050570: (fs) FileSysteProvider fails to initializes if run with file.encoding set to Cp037 In-Reply-To: <503B60B6.7050905@CoSoCo.de> References: <502B90A3.2020601@oracle.com> <50376B9F.10705@oracle.com> <503A9DB8.8080509@oracle.com> <503B4385.6080809@CoSoCo.de> <503B52DD.5030807@oracle.com> <503B60B6.7050905@CoSoCo.de> Message-ID: <503B6416.1000607@oracle.com> On 27/08/2012 12:57, Ulf Zibis wrote: > > It matters, if sun.jnu.encoding is a double-byte coding. > ... and compared to the anyway processed normalizeAndCheck(dir), > performance doesn't matter here. > : > > Yes, and additionally I wanted to say, that constant HERE should be > initialized double-byte encoding proof. > As I said in the original mail, this is beyond the scope of this patch. Yes, we know this is kicking the can down the road and maybe some day there will be a locale on one of the supported platforms where this becomes an issue. The other point is that sun.jnu.encoding is completely internal to JDK, nobody should ever be setting it (same thing goes for file.encoding too but clearly people do seem to try that). -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120827/f4cea2da/attachment.html From Alan.Bateman at oracle.com Mon Aug 27 05:43:41 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 27 Aug 2012 13:43:41 +0100 Subject: Fwd: 7050570: (fs) FileSysteProvider fails to initializes if run with file.encoding set to Cp037 In-Reply-To: <503B60B6.7050905@CoSoCo.de> References: <502B90A3.2020601@oracle.com> <50376B9F.10705@oracle.com> <503A9DB8.8080509@oracle.com> <503B4385.6080809@CoSoCo.de> <503B52DD.5030807@oracle.com> <503B60B6.7050905@CoSoCo.de> Message-ID: <503B6B7D.3070102@oracle.com> On 27/08/2012 12:57, Ulf Zibis wrote: > : > > I still do not see any reasonable why this UnixFileSystem-API is > partly processing chars vs. bytes. > E.G compare UnixPath.normalize(...) vs. UnixPath.initOffsets(). This is the initialization of the file system and so is somewhat special, it basically needs the default directory so that it can compare it it to the working directory of the process. The syscall for the later give it to us in bytes. You are right that the absolute path check could be done with chars but I don't think it makes any difference unless there are wider changes to support multi-byte encodings. -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120827/93ba3e7d/attachment.html From Ulf.Zibis at CoSoCo.de Mon Aug 27 05:46:22 2012 From: Ulf.Zibis at CoSoCo.de (Ulf Zibis) Date: Mon, 27 Aug 2012 14:46:22 +0200 Subject: Fwd: 7050570: (fs) FileSysteProvider fails to initializes if run with file.encoding set to Cp037 In-Reply-To: <503B6416.1000607@oracle.com> References: <502B90A3.2020601@oracle.com> <50376B9F.10705@oracle.com> <503A9DB8.8080509@oracle.com> <503B4385.6080809@CoSoCo.de> <503B52DD.5030807@oracle.com> <503B60B6.7050905@CoSoCo.de> <503B6416.1000607@oracle.com> Message-ID: <503B6C1E.7060909@CoSoCo.de> Am 27.08.2012 14:12, schrieb Alan Bateman: > On 27/08/2012 12:57, Ulf Zibis wrote: >> >> It matters, if sun.jnu.encoding is a double-byte coding. >> ... and compared to the anyway processed normalizeAndCheck(dir), performance doesn't matter here. >> : >> >> Yes, and additionally I wanted to say, that constant HERE should be initialized double-byte >> encoding proof. >> Hm, before your change, lines using "."getBytes() were double-byte encoding proof, so HERE = { '.' } is a kind of regression. > As I said in the original mail, this is beyond the scope of this patch. Yes, we know this is > kicking the can down the road and maybe some day there will be a locale on one of the supported > platforms where this becomes an issue. The other point is that sun.jnu.encoding is completely > internal to JDK, nobody should ever be setting it (same thing goes for file.encoding too but > clearly people do seem to try that). Ah, I see. If you only want to "allow" file.enconding to be set, maybe you could include fix https://bugs.openjdk.java.net/show_bug.cgi?id=100091 -Ulf -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120827/8b7cd2ed/attachment.html From Alan.Bateman at oracle.com Mon Aug 27 06:05:26 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 27 Aug 2012 14:05:26 +0100 Subject: Fwd: 7050570: (fs) FileSysteProvider fails to initializes if run with file.encoding set to Cp037 In-Reply-To: <503B6C1E.7060909@CoSoCo.de> References: <502B90A3.2020601@oracle.com> <50376B9F.10705@oracle.com> <503A9DB8.8080509@oracle.com> <503B4385.6080809@CoSoCo.de> <503B52DD.5030807@oracle.com> <503B60B6.7050905@CoSoCo.de> <503B6416.1000607@oracle.com> <503B6C1E.7060909@CoSoCo.de> Message-ID: <503B7096.2010402@oracle.com> On 27/08/2012 13:46, Ulf Zibis wrote: > > Hm, before your change, lines using "."getBytes() were double-byte > encoding proof, so HERE = { '.' } is a kind of regression. Maybe but this code is Solaris specific, I don't think it is possible to configure a Solaris system in a way that causes a problem. > > Ah, I see. > If you only want to "allow" file.enconding to be set, maybe you could > include fix https://bugs.openjdk.java.net/show_bug.cgi?id=100091 Sorry, but that this is way beyond the scope of what is being proposed here. -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120827/e7a1044c/attachment.html From Ulf.Zibis at CoSoCo.de Mon Aug 27 06:09:27 2012 From: Ulf.Zibis at CoSoCo.de (Ulf Zibis) Date: Mon, 27 Aug 2012 15:09:27 +0200 Subject: Fwd: 7050570: (fs) FileSysteProvider fails to initializes if run with file.encoding set to Cp037 In-Reply-To: <503B6B7D.3070102@oracle.com> References: <502B90A3.2020601@oracle.com> <50376B9F.10705@oracle.com> <503A9DB8.8080509@oracle.com> <503B4385.6080809@CoSoCo.de> <503B52DD.5030807@oracle.com> <503B60B6.7050905@CoSoCo.de> <503B6B7D.3070102@oracle.com> Message-ID: <503B7187.6060001@CoSoCo.de> Am 27.08.2012 14:43, schrieb Alan Bateman: > On 27/08/2012 12:57, Ulf Zibis wrote: >> : >> >> I still do not see any reasonable why this UnixFileSystem-API is partly processing chars vs. bytes. >> E.G compare UnixPath.normalize(...) vs. UnixPath.initOffsets(). > This is the initialization of the file system and so is somewhat special, it basically needs the > default directory so that it can compare it it to the working directory of the process. The > syscall for the later give it to us in bytes. You are right that the absolute path check could be > done with chars but I don't think it makes any difference unless there are wider changes to > support multi-byte encodings. Yes, you are right, there should be more changes to support multi-byte encodings. My first assumption was, that Cp037 is a multi-byte charset, as you referred to [1] which is about UTF-16, and you wanted to support this. [1] http://mail.openjdk.java.net/pipermail/core-libs-dev/2012-July/010758.html Other arguments had been, that processing bytes would perform better than chars, so I was wondering why of all things UnixPath.normalize() processes on chars, as it should run on every new file path being opened. -Ulf -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120827/18bbf211/attachment-0001.html From Alan.Bateman at oracle.com Mon Aug 27 09:56:43 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 27 Aug 2012 17:56:43 +0100 Subject: Fwd: 7050570: (fs) FileSysteProvider fails to initializes if run with file.encoding set to Cp037 In-Reply-To: <503B7187.6060001@CoSoCo.de> References: <502B90A3.2020601@oracle.com> <50376B9F.10705@oracle.com> <503A9DB8.8080509@oracle.com> <503B4385.6080809@CoSoCo.de> <503B52DD.5030807@oracle.com> <503B60B6.7050905@CoSoCo.de> <503B6B7D.3070102@oracle.com> <503B7187.6060001@CoSoCo.de> Message-ID: <503BA6CB.8070707@oracle.com> On 27/08/2012 14:09, Ulf Zibis wrote: > > Yes, you are right, there should be more changes to support multi-byte > encodings. > > My first assumption was, that Cp037 is a multi-byte charset, as you > referred to [1] which is about UTF-16, and you wanted to support this. > > [1] > http://mail.openjdk.java.net/pipermail/core-libs-dev/2012-July/010758.html > > Other arguments had been, that processing bytes would perform better > than chars, so I was wondering why of all things UnixPath.normalize() > processes on chars, as it should run on every new file path being opened. It's only necessary to do the normalization when creating a file-path from a String that the user (or application) provides. If the file-path is coming from the operating system, iterating over the contents of a directory for example, then it's in bytes and also does not require to be normalized. I'm not aware of any locales on the supported platforms that would give us Cp037 as the default charset. The changes here are really just to us exercise some conformance tests with this encoding (despite the fact that file.encoding is meant to be a read-only property and should never be set on the command line). -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120827/e8826a66/attachment.html From kurchi.subhra.hazra at oracle.com Mon Aug 27 11:38:56 2012 From: kurchi.subhra.hazra at oracle.com (Kurchi Hazra) Date: Mon, 27 Aug 2012 11:38:56 -0700 Subject: [7u8] Request for approval: 7168172: (fs) Files.isReadable slow on Windows In-Reply-To: <4FE6555E.1040703@oracle.com> References: <4FE6555E.1040703@oracle.com> Message-ID: <503BBEC0.20105@oracle.com> This is a request for approval to backport the fix for 7168172 from 8 to 7u8. Bug:http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7168172 Webrev:http://cr.openjdk.java.net/~khazra/7168172/7u8/webrev.00/ This had been reviewed by Alan Bateman, Ulf Zibis and Zhong Yu(zhong.j.yu at gmail.com). [1] This fix has been pushed into jdk8 [2]. The fix for 7u8 is identical. Thanks, Kurchi [1]http://mail.openjdk.java.net/pipermail/nio-dev/2012-August/001847.html [2]http://hg.openjdk.java.net/jdk8/tl/jdk/rev/bd91a601265c From Ulf.Zibis at CoSoCo.de Mon Aug 27 11:57:32 2012 From: Ulf.Zibis at CoSoCo.de (Ulf Zibis) Date: Mon, 27 Aug 2012 20:57:32 +0200 Subject: [7u8] Request for approval: 7168172: (fs) Files.isReadable slow on Windows In-Reply-To: <503BBEC0.20105@oracle.com> References: <4FE6555E.1040703@oracle.com> <503BBEC0.20105@oracle.com> Message-ID: <503BC31C.8070305@CoSoCo.de> I have no objections. -Ulf Am 27.08.2012 20:38, schrieb Kurchi Hazra: > > > This is a request for approval to backport the fix for 7168172 from 8 to 7u8. > > Bug:http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7168172 > Webrev:http://cr.openjdk.java.net/~khazra/7168172/7u8/webrev.00/ > > This had been reviewed by Alan Bateman, Ulf Zibis and Zhong Yu(zhong.j.yu at gmail.com). [1] > > This fix has been pushed into jdk8 [2]. The fix for 7u8 is identical. > > Thanks, > Kurchi > > [1]http://mail.openjdk.java.net/pipermail/nio-dev/2012-August/001847.html > [2]http://hg.openjdk.java.net/jdk8/tl/jdk/rev/bd91a601265c > > > From sean.coffey at oracle.com Mon Aug 27 14:56:49 2012 From: sean.coffey at oracle.com (=?ISO-8859-1?Q?Se=E1n_Coffey?=) Date: Mon, 27 Aug 2012 22:56:49 +0100 Subject: [7u8] Request for approval: 7168172: (fs) Files.isReadable slow on Windows In-Reply-To: <503BBEC0.20105@oracle.com> References: <4FE6555E.1040703@oracle.com> <503BBEC0.20105@oracle.com> Message-ID: <503BED21.1020403@oracle.com> Approved. regards, Sean. On 27/08/2012 19:38, Kurchi Hazra wrote: > > > This is a request for approval to backport the fix for 7168172 from 8 > to 7u8. > > Bug:http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7168172 > Webrev:http://cr.openjdk.java.net/~khazra/7168172/7u8/webrev.00/ > > This had been reviewed by Alan Bateman, Ulf Zibis and Zhong > Yu(zhong.j.yu at gmail.com). [1] > > This fix has been pushed into jdk8 [2]. The fix for 7u8 is identical. > > Thanks, > Kurchi > > [1]http://mail.openjdk.java.net/pipermail/nio-dev/2012-August/001847.html > [2]http://hg.openjdk.java.net/jdk8/tl/jdk/rev/bd91a601265c > > From david.lloyd at redhat.com Tue Aug 28 08:14:40 2012 From: david.lloyd at redhat.com (David M. Lloyd) Date: Tue, 28 Aug 2012 10:14:40 -0500 Subject: request for review: 7184932: Remove the temporary Selector usage in the NIO socket adapters In-Reply-To: <502C1024.5010403@oracle.com> References: <502C1024.5010403@oracle.com> Message-ID: <503CE060.1090509@redhat.com> On 08/15/2012 04:09 PM, Rob McKenna wrote: > Hi folks, > > Looking for a codereview of this change. Basically we're looking to > avoid the use of temporary selectors in socket adapters. This should > result in resource usage improvements for users who make extensive use > of these adapters in multiple threads. > > http://cr.openjdk.java.net/~robm/7184932/webrev.02/ > What about generalizing that API a bit, and rolling it up to SelectableChannel with methods like: public abstract void await(int ops); public abstract void await(int ops, long time, TimeUnit unit); ...where ops must conform to validOps(). This way it becomes trivial for anyone to mix blocking and non-blocking APIs which is definitely a common API usage. -- - DML From Alan.Bateman at oracle.com Tue Aug 28 08:23:23 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 28 Aug 2012 16:23:23 +0100 Subject: request for review: 7184932: Remove the temporary Selector usage in the NIO socket adapters In-Reply-To: <503CE060.1090509@redhat.com> References: <502C1024.5010403@oracle.com> <503CE060.1090509@redhat.com> Message-ID: <503CE26B.7050501@oracle.com> On 28/08/2012 16:14, David M. Lloyd wrote: > > What about generalizing that API a bit, and rolling it up to > SelectableChannel with methods like: > > public abstract void await(int ops); > public abstract void await(int ops, long time, TimeUnit unit); > > ...where ops must conform to validOps(). > > This way it becomes trivial for anyone to mix blocking and > non-blocking APIs which is definitely a common API usage. > I've chatted with Rob about a potential step along these lines. First step though is to eliminate the temporary Selector usage in the adapters. -Alan From Alan.Bateman at oracle.com Fri Aug 31 02:32:55 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 31 Aug 2012 10:32:55 +0100 Subject: 7033824: TEST_BUG: java/nio/file/Files/CopyAndMove.java fails intermittently Message-ID: <504084C7.3050400@oracle.com> This is small change to a test that fails if the test working directory is on a different file system to temporary file system. The test exercises copy and move operations on the same file system and different file systems if it can. When testing a move of a symbolic link from one file system to another then the test incorrectly checks that the last modified time is copied. The proposed patch is below. I should mention taht the spec is clear that when copying a symbolic link that there is no guarantee that its attributes are copied, this isn't made clear in the move specification so this is something we need to clarify at some point. -Alan diff --git a/test/java/nio/file/Files/CopyAndMove.java b/test/java/nio/file/Files/CopyAndMove.java --- a/test/java/nio/file/Files/CopyAndMove.java +++ b/test/java/nio/file/Files/CopyAndMove.java @@ -86,10 +86,12 @@ public class CopyAndMove { assertTrue(attrs1.isSymbolicLink() == attrs2.isSymbolicLink()); assertTrue(attrs1.isOther() == attrs2.isOther()); - // check last modified time - long time1 = attrs1.lastModifiedTime().toMillis(); - long time2 = attrs2.lastModifiedTime().toMillis(); - assertTrue(time1 == time2); + // check last modified time if not a symbolic link + if (!attrs1.isSymbolicLink()) { + long time1 = attrs1.lastModifiedTime().toMillis(); + long time2 = attrs2.lastModifiedTime().toMillis(); + assertTrue(time1 == time2); + } // check size if (attrs1.isRegularFile()) From chris.hegarty at oracle.com Fri Aug 31 04:12:15 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Fri, 31 Aug 2012 12:12:15 +0100 Subject: 7033824: TEST_BUG: java/nio/file/Files/CopyAndMove.java fails intermittently In-Reply-To: <504084C7.3050400@oracle.com> References: <504084C7.3050400@oracle.com> Message-ID: <50409C0F.6020808@oracle.com> Seems reasonable Alan. -Chris. On 31/08/12 10:32, Alan Bateman wrote: > > This is small change to a test that fails if the test working directory > is on a different file system to temporary file system. The test > exercises copy and move operations on the same file system and different > file systems if it can. When testing a move of a symbolic link from one > file system to another then the test incorrectly checks that the last > modified time is copied. The proposed patch is below. I should mention > taht the spec is clear that when copying a symbolic link that there is > no guarantee that its attributes are copied, this isn't made clear in > the move specification so this is something we need to clarify at some > point. > > -Alan > > diff --git a/test/java/nio/file/Files/CopyAndMove.java > b/test/java/nio/file/Files/CopyAndMove.java > --- a/test/java/nio/file/Files/CopyAndMove.java > +++ b/test/java/nio/file/Files/CopyAndMove.java > @@ -86,10 +86,12 @@ public class CopyAndMove { > assertTrue(attrs1.isSymbolicLink() == attrs2.isSymbolicLink()); > assertTrue(attrs1.isOther() == attrs2.isOther()); > > - // check last modified time > - long time1 = attrs1.lastModifiedTime().toMillis(); > - long time2 = attrs2.lastModifiedTime().toMillis(); > - assertTrue(time1 == time2); > + // check last modified time if not a symbolic link > + if (!attrs1.isSymbolicLink()) { > + long time1 = attrs1.lastModifiedTime().toMillis(); > + long time2 = attrs2.lastModifiedTime().toMillis(); > + assertTrue(time1 == time2); > + } > > // check size > if (attrs1.isRegularFile()) >