From Alan.Bateman at oracle.com Fri Jun 1 08:32:14 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 01 Jun 2012 16:32:14 +0100 Subject: 7173515: (se) Selector.open fails with OOME on Solaris when unlimited file descriptors Message-ID: <4FC8E07E.2060002@oracle.com> This is an update to the changes to the /dev/poll Selector from a few weeks ago [1]. The changes in 7169050 fixed a long standing performance issue that stemmed from the insertion of a POLLREMOVE prior to each update. Unfortunately the changes introduced an issue on systems configured with an unlimited number of file descriptors. The issue that the update events are indexed by file descriptor. To fix this I've changed it so that it updates to file descriptors >64k spill into into a Map. One other improvement is the addition of a bit set to track which file descriptors are registered as this ensures that the updates to the registrations are accurate. The final improvement is to use the same poll array for both dpwrite and ioctl(DP_POLL). The webrev is here: http://cr.openjdk.java.net/~alanb/7173515/webrev/ BTW: I hope to post a webrev soon with a completely new Selector implementation for Solaris. The new Selector will use the Solaris event ports facilities than the /dev/poll driver. -Alan [1] http://hg.openjdk.java.net/jdk8/tl/jdk/rev/21703d431217 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120601/0f77f271/attachment.html From sean.coffey at oracle.com Sat Jun 2 05:23:00 2012 From: sean.coffey at oracle.com (Sean Coffey) Date: Sat, 02 Jun 2012 13:23:00 +0100 Subject: 7173515: (se) Selector.open fails with OOME on Solaris when unlimited file descriptors In-Reply-To: <4FC8E07E.2060002@oracle.com> References: <4FC8E07E.2060002@oracle.com> Message-ID: <4FCA05A4.5030401@oracle.com> Looks good Alan - I might create a local test binary and run some tests also. In the release method - line 206, can you explain why you're setting a bit there ? 206 registered.set(fd); would it not be a clear ? regards, Sean. On 01/06/2012 16:32, Alan Bateman wrote: > > This is an update to the changes to the /dev/poll Selector from a few > weeks ago [1]. > > The changes in 7169050 fixed a long standing performance issue that > stemmed from the insertion of a POLLREMOVE prior to each update. > Unfortunately the changes introduced an issue on systems configured > with an unlimited number of file descriptors. The issue that the > update events are indexed by file descriptor. To fix this I've changed > it so that it updates to file descriptors >64k spill into into a Map. > One other improvement is the addition of a bit set to track which file > descriptors are registered as this ensures that the updates to the > registrations are accurate. The final improvement is to use the same > poll array for both dpwrite and ioctl(DP_POLL). The webrev is here: > > http://cr.openjdk.java.net/~alanb/7173515/webrev/ > > BTW: I hope to post a webrev soon with a completely new Selector > implementation for Solaris. The new Selector will use the Solaris > event ports facilities than the /dev/poll driver. > > -Alan > > [1] http://hg.openjdk.java.net/jdk8/tl/jdk/rev/21703d431217 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120602/07e6e4ea/attachment.html From Alan.Bateman at oracle.com Sat Jun 2 12:47:34 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 02 Jun 2012 20:47:34 +0100 Subject: 7173515: (se) Selector.open fails with OOME on Solaris when unlimited file descriptors In-Reply-To: <4FCA05A4.5030401@oracle.com> References: <4FC8E07E.2060002@oracle.com> <4FCA05A4.5030401@oracle.com> Message-ID: <4FCA6DD6.2030207@oracle.com> On 02/06/2012 13:23, Sean Coffey wrote: > Looks good Alan - I might create a local test binary and run some > tests also. > > In the release method - line 206, can you explain why you're setting a > bit there ? > > 206 registered.set(fd); > > would it not be a clear ? Yes it should, well spotted. I missed that because it doesn't actually matter if the bit is cleared or not after the key is canceled. I've re-generated the webrev and re-test before pushing the changes. Thanks for reviewing in such detail. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120602/79e3b0a0/attachment.html From chris.hegarty at oracle.com Sun Jun 3 00:12:55 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Sun, 03 Jun 2012 08:12:55 +0100 Subject: 7173515: (se) Selector.open fails with OOME on Solaris when unlimited file descriptors In-Reply-To: <4FCA6DD6.2030207@oracle.com> References: <4FC8E07E.2060002@oracle.com> <4FCA05A4.5030401@oracle.com> <4FCA6DD6.2030207@oracle.com> Message-ID: <4FCB0E77.2080603@oracle.com> On 02/06/12 20:47, Alan Bateman wrote: > On 02/06/2012 13:23, Sean Coffey wrote: >> Looks good Alan - I might create a local test binary and run some >> tests also. >> >> In the release method - line 206, can you explain why you're setting a >> bit there ? >> >> 206 registered.set(fd); >> >> would it not be a clear ? > Yes it should, well spotted. I missed that because it doesn't actually > matter if the bit is cleared or not after the key is canceled. I've > re-generated the webrev and re-test before pushing the changes. Thanks > for reviewing in such detail. The changes look good. Trivially, the map could be created lazily (i guess it won't be used in many cases), but not a big deal. -Chris. > > -Alan From Alan.Bateman at oracle.com Sun Jun 3 08:41:23 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sun, 03 Jun 2012 16:41:23 +0100 Subject: 7173515: (se) Selector.open fails with OOME on Solaris when unlimited file descriptors In-Reply-To: <4FCB0E77.2080603@oracle.com> References: <4FC8E07E.2060002@oracle.com> <4FCA05A4.5030401@oracle.com> <4FCA6DD6.2030207@oracle.com> <4FCB0E77.2080603@oracle.com> Message-ID: <4FCB85A3.4060005@oracle.com> On 03/06/2012 08:12, Chris Hegarty wrote: > : > > Trivially, the map could be created lazily (i guess it won't be used > in many cases), but not a big deal. Yes it could because it's only going to be used when there are more than 64k file descriptors in use. -Alan. From Alan.Bateman at oracle.com Sun Jun 3 09:20:00 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sun, 03 Jun 2012 17:20:00 +0100 Subject: Mac specific file meta information using nio.2 attributes In-Reply-To: References: Message-ID: <4FCB8EB0.2050501@oracle.com> On 03/06/2012 04:05, Michael Hall wrote: > I have put together something to provide some Mac platform api meta data using nio.2 and a little jni. Possibly it's to the point where there might be some feedback. > The NIO file system API has many extension points, one of which is to interpose on the default provider, as you are doing, to add additional features. I looked briefly at your zip file and I would say that this is a good approach for prototyping and to help converge on a useful set of additional attributes that should be supported on this platform. For OpenJDK then we will need add better support for HFS+ and Mac specific file system features. If you look at what has been done for Solaris, Linux and Windows already then you'll see that the default provider on these platforms supports several file system and platform specific features beyond what is required by the specification. For Mac and HFS+ then things that come to mind are: - support for Unciode Normalization Form D and case insensitive file name comparison - a kqueue based WatchService (the Mac port is using the brain dead polling implementation that we have for platforms that don't have support for events from the file system). - a default FileSystemDetector for file content type detection - support for HFS+ file attributes like birth time and user flags. There is also extended attributes and ACLs. This seems to be the area that you are looking into now and it would be great to get the support for these attributes into the default provider (assuming they make sense of course). If you look at the existing code then you'll see how the Unix* provider is extended. -Alan. From mik3hall at gmail.com Sun Jun 3 10:51:06 2012 From: mik3hall at gmail.com (Michael Hall) Date: Sun, 3 Jun 2012 12:51:06 -0500 Subject: Mac specific file meta information using nio.2 attributes In-Reply-To: <4FCB8EB0.2050501@oracle.com> References: <4FCB8EB0.2050501@oracle.com> Message-ID: <02D46885-491F-45B2-A50F-1D548E149817@gmail.com> On Jun 3, 2012, at 11:20 AM, Alan Bateman wrote: > On 03/06/2012 04:05, Michael Hall wrote: >> I have put together something to provide some Mac platform api meta data using nio.2 and a little jni. Possibly it's to the point where there might be some feedback. >> > The NIO file system API has many extension points, one of which is to interpose on the default provider, as you are doing, to add additional features. I looked briefly at your zip file and I would say that this is a good approach for prototyping and to help converge on a useful set of additional attributes that should be supported on this platform. > > For OpenJDK then we will need add better support for HFS+ and Mac specific file system features. If you look at what has been done for Solaris, Linux and Windows already then you'll see that the default provider on these platforms supports several file system and platform specific features beyond what is required by the specification. For Mac and HFS+ then things that come to mind are: > > - support for Unciode Normalization Form D and case insensitive file name comparison Not familiar but I'll look into this. > > - a kqueue based WatchService (the Mac port is using the brain dead polling implementation that we have for platforms that don't have support for events from the file system). > Not an area I was into yet. Strictly attributes and related meta data. One advantage being as far as I knew it's not a project to-do item yet so I wouldn't be duplicating anyone's effort. > - a default FileSystemDetector for file content type detection > Not sure I've seen anything on this in what I've looked at so far. Aren't the providers and filesystems pretty much bound? I'll do some searching to see what I find on this. > - support for HFS+ file attributes like birth time and user flags. There is also extended attributes and ACLs. This seems to be the area that you are looking into now and it would be great to get the support for these attributes into the default provider (assuming they make sense of course). If you look at the existing code then you'll see how the Unix* provider is extended. The api's I've looked at so far and included meta data for are Finder and Launch Services. Still planning on also looking into NSFileManager related meta. You seem to consider HFS+ to involve different meta these don't pick up. I will have to look into that as well. I have been looking at the project nio source for the BSD/Unix providers and zip demo as reference and will continue to do this. > > -Alan. Thanks From Alan.Bateman at oracle.com Sun Jun 3 13:53:35 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sun, 03 Jun 2012 21:53:35 +0100 Subject: 7172826: (se) Selector based on the Solaris event port mechanism Message-ID: <4FCBCECF.9070609@oracle.com> I mentioned in a recent mail that I have a new Selector implementation for Solaris that uses the event port mechanism rather than the /dev/poll driver. Joy Xiong from the Solaris performance team in Oracle has also been running various workloads with this Selector and is reporting good results, better in some cases. The patch proposed here is to add this Selector, but without changing the default at this time. The default will continue to be the /dev/poll Selector and the java.nio.channels.spi.SelectorProvider system property can be used to select the event port Selector where needed. Once the new Selector has baked and we have more performance data, then we may consider changing the default. Another motivation to changing the default is that the /dev/poll is legacy and may go away in the future. The Selector API was originally modeled on the poll interface and so doesn't map too well to I/O polling facilities that aren't level triggered. The way this implementation works is that after each poll (port_getn to retrieve multiple events) then the polled file descriptors are candidates for re-registration for the next poll. Changes to the interest ops will override this so that the file descriptor may be re-registered for different events, or not re-registered for cases where the interest ops are changed to 0 or the selection key is canceled. This seems to work better than optimistically re-registering all polled file descriptors immediately. One other difference compared to the other Selector implementations is the wakeup mechanism (Selector.wakeup) is implemented by sending a user event to the port rather than using the socket pair mechanism. A couple of notes on the implementation: 1. For now sun.nio.ch.SolarisEventPort is changed to expose the event port API for the Selector implementation. Ideally the event port API would be factored out for use by the Selector and Asynchronous I/O implementations but that has ripple effects that would require renaming some classes. This can be done via a refactoring patch later. 2. The fdLimit method has moved from the Selectors to IOUtil so this requires minor adjustments in a few places. 3. The changes to the tests in the webrev are to allow the java.nio.channels.spi.SelectorProvider property be passed to tests via jtreg's -vmoption. Another idea is to add a script as a test that runs several of the existing tests with this Selector. We have alternative Selector implementations already, like the poll Selector that is our backup on Linux, Solaris and Mac and that could be tested in a similar way. I think that's mostly it, the webrev with patch is here: http://cr.openjdk.java.net/~alanb/7172826/webrev/ -Alan. From youdwei at linux.vnet.ibm.com Tue Jun 5 00:09:38 2012 From: youdwei at linux.vnet.ibm.com (Deven You) Date: Tue, 05 Jun 2012 15:09:38 +0800 Subject: Using OP_CONNECT with Selector.select causes selector to fire repeatedly In-Reply-To: <4FC72201.7090307@linux.vnet.ibm.com> References: <4FAA1C40.6040709@linux.vnet.ibm.com> <4FB2BF30.9060902@oracle.com> <4FC72201.7090307@linux.vnet.ibm.com> Message-ID: <4FCDB0B2.1040902@linux.vnet.ibm.com> Hi Alan, What is your opinion about the javadoc clarification? Thanks alot! On 05/31/2012 03:47 PM, Deven You wrote: > Hi Alan, > > Thanks for your comments. How do you feel the javadoc change should > be, from my knowledge, for the Selector.select(), we may add a > sentence like "For OP_CONNECT, once a channel is already connected, an > application should not check OP_CONNECT again, in this situation, > Selector will ignore OP_CONNECT and return immediately." > > On 05/16/2012 04:40 AM, Alan Bateman wrote: >> Devon, >> >> Just to follow up from my previous mail. I looked at the changes in >> the patch and also tried it out to see the side effects. One issue is >> that the changes mean that OP_CONNECT will be "automatically" removed >> from the interest ops, minimally that would require a spec change. >> When I ran the Selector tests (in the jdk repository) then I noticed >> a few failures so there are other side effects too. In any case, I >> think the one is really just a mis-use of the API and that we should >> instead adding wording to the javadoc rather than change the >> implementation. >> >> -Alan >> > > -- Best Regards, Deven From forax at univ-mlv.fr Tue Jun 5 03:00:36 2012 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Tue, 05 Jun 2012 12:00:36 +0200 Subject: non-blocking channel Infinite loop in java.util.Scanner Message-ID: <4FCDD8C4.9070106@univ-mlv.fr> One of my student find a bug in the implementation of Scanner, that allows you to use a non blocking channel as input of a Scanner. The Scanner uses Channels.newReader() to create a Reader from a channel which itself create a StreamDecoder. In that case, StreamDecoder.impReader() goes into an infinite loop because impReader() calls readBytes() that does nothing if channel.read() returns zero. The javadoc of Channels.newReader() clearly states that it should throw a IllegalBlockingModeException but there is no code that checks that. I think a way to solve the problem is to insert a code that check the blocking state in Channels.newWriter(). if (ch instanceof SelectableChannel) { SelectableChannel sc = (SelectableChannel)ch; if (!sc.isBlocking()) throw new IllegalBlockingModeException(); } } and to document the exception in the constructor of Scanner that takes a channel. If someone provide me a bug id, it will provide a patch :) cheers, R?mi PS: The code below is a simple test to reproduce the infinite loop. ---------------------------------------------------- import java.io.IOException; import java.net.InetSocketAddress; import java.nio.ByteBuffer; import java.nio.channels.ServerSocketChannel; import java.nio.channels.SocketChannel; import java.util.Scanner; public class Main { public static void main(String[] args) throws IOException { ServerSocketChannel serverSocketChannel = ServerSocketChannel.open(); serverSocketChannel.bind(new InetSocketAddress(2332)); new Thread(new Runnable() { @Override public void run() { try { SocketChannel channel = SocketChannel.open(new InetSocketAddress("localhost", 2332)); do { channel.write(ByteBuffer.wrap(new byte[] {'A'})); Thread.sleep(1000); } while(true); } catch (IOException | InterruptedException e) { throw new AssertionError(e); } } }).start(); SocketChannel socketChannel = serverSocketChannel.accept(); socketChannel.configureBlocking(false); Scanner scanner = new Scanner(socketChannel); while(scanner.hasNextLine()) { System.out.println(scanner.nextLine()); } } } From Alan.Bateman at oracle.com Tue Jun 5 03:40:05 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 05 Jun 2012 11:40:05 +0100 Subject: non-blocking channel Infinite loop in java.util.Scanner In-Reply-To: <4FCDD8C4.9070106@univ-mlv.fr> References: <4FCDD8C4.9070106@univ-mlv.fr> Message-ID: <4FCDE205.3030808@oracle.com> On 05/06/2012 11:00, R?mi Forax wrote: > One of my student find a bug in the implementation of Scanner, > that allows you to use a non blocking channel as input of a Scanner. > > The Scanner uses Channels.newReader() to create a Reader > from a channel which itself create a StreamDecoder. > > In that case, StreamDecoder.impReader() goes into an infinite loop > because impReader() calls readBytes() that does nothing > if channel.read() returns zero. > > The javadoc of Channels.newReader() clearly states that > it should throw a IllegalBlockingModeException but > there is no code that checks that. > > I think a way to solve the problem is to insert a code > that check the blocking state in Channels.newWriter(). > > if (ch instanceof SelectableChannel) { > SelectableChannel sc = (SelectableChannel)ch; > if (!sc.isBlocking()) > throw new IllegalBlockingModeException(); > } > } This could be fixed in Channels.newReader or in StreamDecoder, the former would be consistent with Channels.newWriter. I guess you know this already, but you will need to synchronize on the selectable channel's blockingLock to ensure that the blocking mode doesn't change. I've created a bug for this: 7174305: (ch) Channels.newReader doesn't throw IllegalBlockingMode if channel is configured non-blocking -Alan. From forax at univ-mlv.fr Tue Jun 5 16:12:16 2012 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Wed, 06 Jun 2012 01:12:16 +0200 Subject: non-blocking channel Infinite loop in java.util.Scanner In-Reply-To: <4FCDE205.3030808@oracle.com> References: <4FCDD8C4.9070106@univ-mlv.fr> <4FCDE205.3030808@oracle.com> Message-ID: <4FCE9250.3010105@univ-mlv.fr> On 06/05/2012 12:40 PM, Alan Bateman wrote: > On 05/06/2012 11:00, R?mi Forax wrote: >> One of my student find a bug in the implementation of Scanner, >> that allows you to use a non blocking channel as input of a Scanner. >> >> The Scanner uses Channels.newReader() to create a Reader >> from a channel which itself create a StreamDecoder. >> >> In that case, StreamDecoder.impReader() goes into an infinite loop >> because impReader() calls readBytes() that does nothing >> if channel.read() returns zero. >> >> The javadoc of Channels.newReader() clearly states that >> it should throw a IllegalBlockingModeException but >> there is no code that checks that. >> >> I think a way to solve the problem is to insert a code >> that check the blocking state in Channels.newWriter(). >> >> if (ch instanceof SelectableChannel) { >> SelectableChannel sc = (SelectableChannel)ch; >> if (!sc.isBlocking()) >> throw new IllegalBlockingModeException(); >> } >> } > This could be fixed in Channels.newReader or in StreamDecoder, the > former would be consistent with Channels.newWriter. I guess you know > this already, but you will need to synchronize on the selectable > channel's blockingLock to ensure that the blocking mode doesn't change. > > I've created a bug for this: > > 7174305: (ch) Channels.newReader doesn't throw IllegalBlockingMode if > channel is configured non-blocking > > -Alan. Thanks Alan, I can't ensure that the blocking mode will not change by synchronizing on the channel's blockingLock because I will have to take the lock before creating the reader and releasing it when the reader is closed. An easier solution is to throw an IllegalBlockingModeException if the channel.read() returns 0, in that case I think I don't have to even check the configured mode (the question is what to do if the configured mode is changed by another thread between the call to read() and the call to isBlocking()). and BTW isBlocking() already takes the lock, may be the doc of isBlocking() should more clear about that. R?mi From Alan.Bateman at oracle.com Wed Jun 6 03:08:00 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 06 Jun 2012 11:08:00 +0100 Subject: non-blocking channel Infinite loop in java.util.Scanner In-Reply-To: <4FCE9250.3010105@univ-mlv.fr> References: <4FCDD8C4.9070106@univ-mlv.fr> <4FCDE205.3030808@oracle.com> <4FCE9250.3010105@univ-mlv.fr> Message-ID: <4FCF2C00.8030905@oracle.com> On 06/06/2012 00:12, R?mi Forax wrote: > > Thanks Alan, > > I can't ensure that the blocking mode will not change by synchronizing on > the channel's blockingLock because I will have to take the lock > before creating the reader and releasing it when the reader is closed. Looking at now, it would need to be done in StreamDecoder.readBytes. > > An easier solution is to throw an IllegalBlockingModeException > if the channel.read() returns 0, in that case I think I don't have to > even check the configured mode > (the question is what to do if the configured mode is changed by > another thread between the call to read() > and the call to isBlocking()). It's possible that read may return bytes immediately when configured non-blocking so it means you will need to check the blocking mode to ensure that the expected IllegalBlockingModeException is thrown. I guess you could also check for 0 to avoid grabbing the blocking lock. -Alan. From chris.hegarty at oracle.com Wed Jun 6 07:03:17 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 06 Jun 2012 15:03:17 +0100 Subject: 7172826: (se) Selector based on the Solaris event port mechanism In-Reply-To: <4FCBCECF.9070609@oracle.com> References: <4FCBCECF.9070609@oracle.com> Message-ID: <4FCF6325.50504@oracle.com> Nice work! Looks good to me. -Chris. On 03/06/2012 21:53, Alan Bateman wrote: > > I mentioned in a recent mail that I have a new Selector implementation > for Solaris that uses the event port mechanism rather than the /dev/poll > driver. Joy Xiong from the Solaris performance team in Oracle has also > been running various workloads with this Selector and is reporting good > results, better in some cases. > > The patch proposed here is to add this Selector, but without changing > the default at this time. The default will continue to be the /dev/poll > Selector and the java.nio.channels.spi.SelectorProvider system property > can be used to select the event port Selector where needed. Once the new > Selector has baked and we have more performance data, then we may > consider changing the default. Another motivation to changing the > default is that the /dev/poll is legacy and may go away in the future. > > The Selector API was originally modeled on the poll interface and so > doesn't map too well to I/O polling facilities that aren't level > triggered. The way this implementation works is that after each poll > (port_getn to retrieve multiple events) then the polled file descriptors > are candidates for re-registration for the next poll. Changes to the > interest ops will override this so that the file descriptor may be > re-registered for different events, or not re-registered for cases where > the interest ops are changed to 0 or the selection key is canceled. This > seems to work better than optimistically re-registering all polled file > descriptors immediately. One other difference compared to the other > Selector implementations is the wakeup mechanism (Selector.wakeup) is > implemented by sending a user event to the port rather than using the > socket pair mechanism. > > A couple of notes on the implementation: > > 1. For now sun.nio.ch.SolarisEventPort is changed to expose the event > port API for the Selector implementation. Ideally the event port API > would be factored out for use by the Selector and Asynchronous I/O > implementations but that has ripple effects that would require renaming > some classes. This can be done via a refactoring patch later. > > 2. The fdLimit method has moved from the Selectors to IOUtil so this > requires minor adjustments in a few places. > > 3. The changes to the tests in the webrev are to allow the > java.nio.channels.spi.SelectorProvider property be passed to tests via > jtreg's -vmoption. Another idea is to add a script as a test that runs > several of the existing tests with this Selector. We have alternative > Selector implementations already, like the poll Selector that is our > backup on Linux, Solaris and Mac and that could be tested in a similar way. > > I think that's mostly it, the webrev with patch is here: > > http://cr.openjdk.java.net/~alanb/7172826/webrev/ > > -Alan. From sean.coffey at oracle.com Wed Jun 6 07:36:59 2012 From: sean.coffey at oracle.com (=?ISO-8859-1?Q?Se=E1n_Coffey?=) Date: Wed, 06 Jun 2012 15:36:59 +0100 Subject: 7172826: (se) Selector based on the Solaris event port mechanism In-Reply-To: <4FCF6325.50504@oracle.com> References: <4FCBCECF.9070609@oracle.com> <4FCF6325.50504@oracle.com> Message-ID: <4FCF6B0B.3060801@oracle.com> Meant to get back on this also Alan. Look goods to me. Will be interesting to see how performance compares. Minor format comment on EventPortSelectorImpl.doSelect(long) method (line 64-66). Perhaps some javadoc style comments could be used in EventPortWrapper.java also. (again only minor) thanks for looking after this. regards, Sean. On 06/06/12 15:03, Chris Hegarty wrote: > Nice work! Looks good to me. > > -Chris. > > On 03/06/2012 21:53, Alan Bateman wrote: >> >> I mentioned in a recent mail that I have a new Selector implementation >> for Solaris that uses the event port mechanism rather than the /dev/poll >> driver. Joy Xiong from the Solaris performance team in Oracle has also >> been running various workloads with this Selector and is reporting good >> results, better in some cases. >> >> The patch proposed here is to add this Selector, but without changing >> the default at this time. The default will continue to be the /dev/poll >> Selector and the java.nio.channels.spi.SelectorProvider system property >> can be used to select the event port Selector where needed. Once the new >> Selector has baked and we have more performance data, then we may >> consider changing the default. Another motivation to changing the >> default is that the /dev/poll is legacy and may go away in the future. >> >> The Selector API was originally modeled on the poll interface and so >> doesn't map too well to I/O polling facilities that aren't level >> triggered. The way this implementation works is that after each poll >> (port_getn to retrieve multiple events) then the polled file descriptors >> are candidates for re-registration for the next poll. Changes to the >> interest ops will override this so that the file descriptor may be >> re-registered for different events, or not re-registered for cases where >> the interest ops are changed to 0 or the selection key is canceled. This >> seems to work better than optimistically re-registering all polled file >> descriptors immediately. One other difference compared to the other >> Selector implementations is the wakeup mechanism (Selector.wakeup) is >> implemented by sending a user event to the port rather than using the >> socket pair mechanism. >> >> A couple of notes on the implementation: >> >> 1. For now sun.nio.ch.SolarisEventPort is changed to expose the event >> port API for the Selector implementation. Ideally the event port API >> would be factored out for use by the Selector and Asynchronous I/O >> implementations but that has ripple effects that would require renaming >> some classes. This can be done via a refactoring patch later. >> >> 2. The fdLimit method has moved from the Selectors to IOUtil so this >> requires minor adjustments in a few places. >> >> 3. The changes to the tests in the webrev are to allow the >> java.nio.channels.spi.SelectorProvider property be passed to tests via >> jtreg's -vmoption. Another idea is to add a script as a test that runs >> several of the existing tests with this Selector. We have alternative >> Selector implementations already, like the poll Selector that is our >> backup on Linux, Solaris and Mac and that could be tested in a >> similar way. >> >> I think that's mostly it, the webrev with patch is here: >> >> http://cr.openjdk.java.net/~alanb/7172826/webrev/ >> >> -Alan. From rob.mckenna at oracle.com Wed Jun 6 13:47:04 2012 From: rob.mckenna at oracle.com (Rob McKenna) Date: Wed, 06 Jun 2012 21:47:04 +0100 Subject: 7161881: (dc) DatagramChannel.bind(null) fails if IPv4 socket and running with preferIPv6Addresses=true Message-ID: <4FCFC1C8.40007@oracle.com> Hi folks, Small bug with in DatagramChannel where Java.net.preferIPv6Addresses=true is causing bind(null) to use an ipv6 address even though we're connected to an ipv4 socket. Additional context: http://mail.openjdk.java.net/pipermail/nio-dev/2012-April/001632.html Webrev: http://cr.openjdk.java.net/~robm/7161881/webrev.01/ -Rob From Alan.Bateman at oracle.com Wed Jun 6 14:02:11 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 06 Jun 2012 22:02:11 +0100 Subject: 7161881: (dc) DatagramChannel.bind(null) fails if IPv4 socket and running with preferIPv6Addresses=true In-Reply-To: <4FCFC1C8.40007@oracle.com> References: <4FCFC1C8.40007@oracle.com> Message-ID: <4FCFC553.40302@oracle.com> On 06/06/2012 21:47, Rob McKenna wrote: > Hi folks, > > Small bug with in DatagramChannel where > Java.net.preferIPv6Addresses=true is causing bind(null) to use an ipv6 > address even though we're connected to an ipv4 socket. > > Additional context: > http://mail.openjdk.java.net/pipermail/nio-dev/2012-April/001632.html > > Webrev: > http://cr.openjdk.java.net/~robm/7161881/webrev.01/ > > -Rob Thanks Rob, this looks fine except for the outer try-catch in the test which I think should be removed (meaning no need to catch IOException and re-throw as RuntimeException). -Alan. From chris.hegarty at oracle.com Wed Jun 6 14:12:27 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 06 Jun 2012 22:12:27 +0100 Subject: 7161881: (dc) DatagramChannel.bind(null) fails if IPv4 socket and running with preferIPv6Addresses=true In-Reply-To: <4FCFC1C8.40007@oracle.com> References: <4FCFC1C8.40007@oracle.com> Message-ID: <4FCFC7BB.5010802@oracle.com> Interesting bug. Changes look ok to me. -Chris. On 06/06/12 21:47, Rob McKenna wrote: > Hi folks, > > Small bug with in DatagramChannel where > Java.net.preferIPv6Addresses=true is causing bind(null) to use an ipv6 > address even though we're connected to an ipv4 socket. > > Additional context: > http://mail.openjdk.java.net/pipermail/nio-dev/2012-April/001632.html > > Webrev: > http://cr.openjdk.java.net/~robm/7161881/webrev.01/ > > -Rob From rob.mckenna at oracle.com Wed Jun 6 14:23:00 2012 From: rob.mckenna at oracle.com (Rob McKenna) Date: Wed, 06 Jun 2012 22:23:00 +0100 Subject: 7161881: (dc) DatagramChannel.bind(null) fails if IPv4 socket and running with preferIPv6Addresses=true In-Reply-To: <4FCFC553.40302@oracle.com> References: <4FCFC1C8.40007@oracle.com> <4FCFC553.40302@oracle.com> Message-ID: <4FCFCA34.10600@oracle.com> Done and briefly tested. I'll run it through jprt before integration. http://cr.openjdk.java.net/~robm/7161881/webrev.02/ -Rob On 06/06/12 22:02, Alan Bateman wrote: > On 06/06/2012 21:47, Rob McKenna wrote: >> Hi folks, >> >> Small bug with in DatagramChannel where >> Java.net.preferIPv6Addresses=true is causing bind(null) to use an >> ipv6 address even though we're connected to an ipv4 socket. >> >> Additional context: >> http://mail.openjdk.java.net/pipermail/nio-dev/2012-April/001632.html >> >> Webrev: >> http://cr.openjdk.java.net/~robm/7161881/webrev.01/ >> >> -Rob > Thanks Rob, this looks fine except for the outer try-catch in the test > which I think should be removed (meaning no need to catch IOException > and re-throw as RuntimeException). > > -Alan. From Alan.Bateman at oracle.com Thu Jun 7 00:18:21 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 07 Jun 2012 08:18:21 +0100 Subject: 7161881: (dc) DatagramChannel.bind(null) fails if IPv4 socket and running with preferIPv6Addresses=true In-Reply-To: <4FCFCA34.10600@oracle.com> References: <4FCFC1C8.40007@oracle.com> <4FCFC553.40302@oracle.com> <4FCFCA34.10600@oracle.com> Message-ID: <4FD055BD.3050206@oracle.com> On 06/06/2012 22:23, Rob McKenna wrote: > Done and briefly tested. I'll run it through jprt before integration. > > http://cr.openjdk.java.net/~robm/7161881/webrev.02/ > > -Rob Looks fine, thanks for taking it. -Alan From Alan.Bateman at oracle.com Wed Jun 13 06:54:23 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 13 Jun 2012 14:54:23 +0100 Subject: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] Message-ID: <4FD89B8F.3000704@oracle.com> I need a reviewer for a small change to the SocketChannel.write implementation on Windows. The issue on Windows is that we limit the buffer size to 128k-1 because asynchronous close doesn't work on Windows for sizes later than that (the long standing Microsoft recommendation is to use <=64k). The issue with limiting the size of the I/O operation is that the write method may return a value less than what is expected when the channel is configured blocking and this can cause problem for applications that don't check the return value. The changes just convert the large write into multiple calls to WSASend. The webrev with the changes is here: http://cr.openjdk.java.net/~alanb/7176630/webrev/ Thanks, Alan. From wolfgang.baltes at laposte.net Wed Jun 13 09:01:56 2012 From: wolfgang.baltes at laposte.net (Wolfgang Baltes) Date: Wed, 13 Jun 2012 09:01:56 -0700 Subject: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] In-Reply-To: <4FD89B8F.3000704@oracle.com> References: <4FD89B8F.3000704@oracle.com> Message-ID: <4FD8B974.7060606@laposte.net> Looks ok to me. Wolfgang. On 2012-06-13 06:54, Alan Bateman wrote: > > I need a reviewer for a small change to the SocketChannel.write > implementation on Windows. The issue on Windows is that we limit the > buffer size to 128k-1 because asynchronous close doesn't work on > Windows for sizes later than that (the long standing Microsoft > recommendation is to use <=64k). The issue with limiting the size of > the I/O operation is that the write method may return a value less > than what is expected when the channel is configured blocking and this > can cause problem for applications that don't check the return value. > The changes just convert the large write into multiple calls to > WSASend. The webrev with the changes is here: > > http://cr.openjdk.java.net/~alanb/7176630/webrev/ > > Thanks, > Alan. > > From kurchi.subhra.hazra at oracle.com Wed Jun 13 09:24:14 2012 From: kurchi.subhra.hazra at oracle.com (Kurchi Subhra Hazra) Date: Wed, 13 Jun 2012 09:24:14 -0700 Subject: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] In-Reply-To: <4FD8B974.7060606@laposte.net> References: <4FD89B8F.3000704@oracle.com> <4FD8B974.7060606@laposte.net> Message-ID: <4FD8BEAE.9030103@oracle.com> Just out of curiosity - why are you doing multiple calls to WSASend as opposed to setting up and sending multiple buffers with one call? - Kurchi On 6/13/12 9:01 AM, Wolfgang Baltes wrote: > Looks ok to me. > Wolfgang. > > On 2012-06-13 06:54, Alan Bateman wrote: >> >> I need a reviewer for a small change to the SocketChannel.write >> implementation on Windows. The issue on Windows is that we limit the >> buffer size to 128k-1 because asynchronous close doesn't work on >> Windows for sizes later than that (the long standing Microsoft >> recommendation is to use <=64k). The issue with limiting the size of >> the I/O operation is that the write method may return a value less >> than what is expected when the channel is configured blocking and >> this can cause problem for applications that don't check the return >> value. The changes just convert the large write into multiple calls >> to WSASend. The webrev with the changes is here: >> >> http://cr.openjdk.java.net/~alanb/7176630/webrev/ >> >> Thanks, >> Alan. >> >> > > From Alan.Bateman at oracle.com Wed Jun 13 09:27:34 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 13 Jun 2012 17:27:34 +0100 Subject: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] In-Reply-To: <4FD8B974.7060606@laposte.net> References: <4FD89B8F.3000704@oracle.com> <4FD8B974.7060606@laposte.net> Message-ID: <4FD8BF76.80503@oracle.com> On 13/06/2012 17:01, Wolfgang Baltes wrote: > Looks ok to me. > Wolfgang. Thanks Wolfgang but for the change-set I need to get an official reviewer. -Alan. From Alan.Bateman at oracle.com Wed Jun 13 09:46:58 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 13 Jun 2012 17:46:58 +0100 Subject: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] In-Reply-To: <4FD8BEAE.9030103@oracle.com> References: <4FD89B8F.3000704@oracle.com> <4FD8B974.7060606@laposte.net> <4FD8BEAE.9030103@oracle.com> Message-ID: <4FD8C402.4040303@oracle.com> On 13/06/2012 17:24, Kurchi Subhra Hazra wrote: > Just out of curiosity - why are you doing multiple calls to WSASend as > opposed to setting up > and sending multiple buffers with one call? > > - Kurchi The issue is the asynchronous close when the total size (over all buffers) is >= 128k. -Alan From kurchi.subhra.hazra at oracle.com Wed Jun 13 10:40:26 2012 From: kurchi.subhra.hazra at oracle.com (Kurchi Hazra) Date: Wed, 13 Jun 2012 10:40:26 -0700 Subject: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] In-Reply-To: <4FD8C402.4040303@oracle.com> References: <4FD89B8F.3000704@oracle.com> <4FD8B974.7060606@laposte.net> <4FD8BEAE.9030103@oracle.com> <4FD8C402.4040303@oracle.com> Message-ID: <4FD8D08A.8010106@oracle.com> On 6/13/2012 9:46 AM, Alan Bateman wrote: > On 13/06/2012 17:24, Kurchi Subhra Hazra wrote: >> Just out of curiosity - why are you doing multiple calls to WSASend >> as opposed to setting up >> and sending multiple buffers with one call? >> >> - Kurchi > The issue is the asynchronous close when the total size (over all > buffers) is >= 128k. I see. Thanks! - Kurchi From dbelfer at gmail.com Wed Jun 13 12:14:05 2012 From: dbelfer at gmail.com (Diego Belfer) Date: Wed, 13 Jun 2012 16:14:05 -0300 Subject: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] In-Reply-To: <4FD89B8F.3000704@oracle.com> References: <4FD89B8F.3000704@oracle.com> Message-ID: Hi Alan, Do you know if the WSASend function may return SOCKET_ERROR and still have written some bytes to the underlying socket? (I read the doc for WSASend function and it is not clear for me if it is possible or not) If that is possible, you may need to "add" written to count before the "if" in order to return the count. Best, Diego On Wed, Jun 13, 2012 at 10:54 AM, Alan Bateman wrote: > > I need a reviewer for a small change to the SocketChannel.write > implementation on Windows. The issue on Windows is that we limit the buffer > size to 128k-1 because asynchronous close doesn't work on Windows for sizes > later than that (the long standing Microsoft recommendation is to use > <=64k). The issue with limiting the size of the I/O operation is that the > write method may return a value less than what is expected when the channel > is configured blocking and this can cause problem for applications that > don't check the return value. The changes just convert the large write into > multiple calls to WSASend. The webrev with the changes is here: > > http://cr.openjdk.java.net/~**alanb/7176630/webrev/ > > Thanks, > Alan. > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120613/972f4654/attachment.html From Alan.Bateman at oracle.com Wed Jun 13 12:19:36 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 13 Jun 2012 20:19:36 +0100 Subject: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] In-Reply-To: References: <4FD89B8F.3000704@oracle.com> Message-ID: <4FD8E7C8.8030807@oracle.com> On 13/06/2012 20:14, Diego Belfer wrote: > Hi Alan, > > Do you know if the WSASend function may return SOCKET_ERROR and still > have written some bytes to the underlying socket? (I read the doc for > WSASend function and it is not clear for me if it is possible or not) > If that is possible, you may need to "add" written to count before the > "if" in order to return the count. That would be a bad bug in Windows if that were to happen. -Alan. From chris.hegarty at oracle.com Wed Jun 13 14:39:23 2012 From: chris.hegarty at oracle.com (chris hegarty) Date: Wed, 13 Jun 2012 22:39:23 +0100 Subject: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] In-Reply-To: <4FD89B8F.3000704@oracle.com> References: <4FD89B8F.3000704@oracle.com> Message-ID: <4FD9088B.7030806@oracle.com> Source changes look fine to me. Are gathering operations effected? -Chris. On 13/06/2012 14:54, Alan Bateman wrote: > > I need a reviewer for a small change to the SocketChannel.write > implementation on Windows. The issue on Windows is that we limit the > buffer size to 128k-1 because asynchronous close doesn't work on Windows > for sizes later than that (the long standing Microsoft recommendation is > to use <=64k). The issue with limiting the size of the I/O operation is > that the write method may return a value less than what is expected when > the channel is configured blocking and this can cause problem for > applications that don't check the return value. The changes just convert > the large write into multiple calls to WSASend. The webrev with the > changes is here: > > http://cr.openjdk.java.net/~alanb/7176630/webrev/ > > Thanks, > Alan. From esmond.pitt at bigpond.com Wed Jun 13 16:28:47 2012 From: esmond.pitt at bigpond.com (Esmond Pitt) Date: Thu, 14 Jun 2012 09:28:47 +1000 Subject: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] In-Reply-To: References: Message-ID: <667720003EBD41BA9B352ACD21D5B977@HP6910P> Alan I suggest the loop should only happen in blocking mode, if at all. Non-blocking mode programs must already cope with short writes, and shouldn't be held up dispatching writes, on fairness grounds. I'm not convinced about the need for the loop in blocking mode either, as applications must be written correctly, and forums are already full of war stories about checking write (and read) return values in java.nio and indeed java.net and java.io. EJP -----Original Message----- From: nio-dev-bounces at openjdk.java.net [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of nio-dev-request at openjdk.java.net Sent: Thursday, 14 June 2012 5:00 AM To: nio-dev at openjdk.java.net Subject: nio-dev Digest, Vol 51, Issue 8 Send nio-dev mailing list submissions to nio-dev at openjdk.java.net To subscribe or unsubscribe via the World Wide Web, visit http://mail.openjdk.java.net/mailman/listinfo/nio-dev or, via email, send a message with subject or body 'help' to nio-dev-request at openjdk.java.net You can reach the person managing the list at nio-dev-owner at openjdk.java.net When replying, please edit your Subject line so it is more specific than "Re: Contents of nio-dev digest..." Today's Topics: 1. 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] (Alan Bateman) 2. Re: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] (Wolfgang Baltes) 3. Re: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] (Kurchi Subhra Hazra) 4. Re: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] (Alan Bateman) 5. Re: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] (Alan Bateman) 6. Re: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] (Kurchi Hazra) ---------------------------------------------------------------------- Message: 1 Date: Wed, 13 Jun 2012 14:54:23 +0100 From: Alan Bateman Subject: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] To: nio-dev Message-ID: <4FD89B8F.3000704 at oracle.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed I need a reviewer for a small change to the SocketChannel.write implementation on Windows. The issue on Windows is that we limit the buffer size to 128k-1 because asynchronous close doesn't work on Windows for sizes later than that (the long standing Microsoft recommendation is to use <=64k). The issue with limiting the size of the I/O operation is that the write method may return a value less than what is expected when the channel is configured blocking and this can cause problem for applications that don't check the return value. The changes just convert the large write into multiple calls to WSASend. The webrev with the changes is here: http://cr.openjdk.java.net/~alanb/7176630/webrev/ Thanks, Alan. ------------------------------ Message: 2 Date: Wed, 13 Jun 2012 09:01:56 -0700 From: Wolfgang Baltes Subject: Re: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] To: Alan Bateman Cc: nio-dev Message-ID: <4FD8B974.7060606 at laposte.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Looks ok to me. Wolfgang. On 2012-06-13 06:54, Alan Bateman wrote: > > I need a reviewer for a small change to the SocketChannel.write > implementation on Windows. The issue on Windows is that we limit the > buffer size to 128k-1 because asynchronous close doesn't work on > Windows for sizes later than that (the long standing Microsoft > recommendation is to use <=64k). The issue with limiting the size of > the I/O operation is that the write method may return a value less > than what is expected when the channel is configured blocking and this > can cause problem for applications that don't check the return value. > The changes just convert the large write into multiple calls to > WSASend. The webrev with the changes is here: > > http://cr.openjdk.java.net/~alanb/7176630/webrev/ > > Thanks, > Alan. > > ------------------------------ Message: 3 Date: Wed, 13 Jun 2012 09:24:14 -0700 From: Kurchi Subhra Hazra Subject: Re: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] To: nio-dev at openjdk.java.net Message-ID: <4FD8BEAE.9030103 at oracle.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Just out of curiosity - why are you doing multiple calls to WSASend as opposed to setting up and sending multiple buffers with one call? - Kurchi On 6/13/12 9:01 AM, Wolfgang Baltes wrote: > Looks ok to me. > Wolfgang. > > On 2012-06-13 06:54, Alan Bateman wrote: >> >> I need a reviewer for a small change to the SocketChannel.write >> implementation on Windows. The issue on Windows is that we limit the >> buffer size to 128k-1 because asynchronous close doesn't work on >> Windows for sizes later than that (the long standing Microsoft >> recommendation is to use <=64k). The issue with limiting the size of >> the I/O operation is that the write method may return a value less >> than what is expected when the channel is configured blocking and >> this can cause problem for applications that don't check the return >> value. The changes just convert the large write into multiple calls >> to WSASend. The webrev with the changes is here: >> >> http://cr.openjdk.java.net/~alanb/7176630/webrev/ >> >> Thanks, >> Alan. >> >> > > ------------------------------ Message: 4 Date: Wed, 13 Jun 2012 17:27:34 +0100 From: Alan Bateman Subject: Re: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] To: Wolfgang Baltes Cc: nio-dev Message-ID: <4FD8BF76.80503 at oracle.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed On 13/06/2012 17:01, Wolfgang Baltes wrote: > Looks ok to me. > Wolfgang. Thanks Wolfgang but for the change-set I need to get an official reviewer. -Alan. ------------------------------ Message: 5 Date: Wed, 13 Jun 2012 17:46:58 +0100 From: Alan Bateman Subject: Re: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] To: Kurchi Subhra Hazra Cc: nio-dev at openjdk.java.net Message-ID: <4FD8C402.4040303 at oracle.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed On 13/06/2012 17:24, Kurchi Subhra Hazra wrote: > Just out of curiosity - why are you doing multiple calls to WSASend as > opposed to setting up and sending multiple buffers with one call? > > - Kurchi The issue is the asynchronous close when the total size (over all buffers) is >= 128k. -Alan ------------------------------ Message: 6 Date: Wed, 13 Jun 2012 10:40:26 -0700 From: Kurchi Hazra Subject: Re: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] To: Alan Bateman Cc: nio-dev at openjdk.java.net Message-ID: <4FD8D08A.8010106 at oracle.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed On 6/13/2012 9:46 AM, Alan Bateman wrote: > On 13/06/2012 17:24, Kurchi Subhra Hazra wrote: >> Just out of curiosity - why are you doing multiple calls to WSASend >> as opposed to setting up and sending multiple buffers with one call? >> >> - Kurchi > The issue is the asynchronous close when the total size (over all > buffers) is >= 128k. I see. Thanks! - Kurchi End of nio-dev Digest, Vol 51, Issue 8 ************************************** From Alan.Bateman at oracle.com Thu Jun 14 01:12:25 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 14 Jun 2012 09:12:25 +0100 Subject: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] In-Reply-To: <4FD9088B.7030806@oracle.com> References: <4FD89B8F.3000704@oracle.com> <4FD9088B.7030806@oracle.com> Message-ID: <4FD99CE9.1010700@oracle.com> On 13/06/2012 22:39, chris hegarty wrote: > Source changes look fine to me. > > Are gathering operations effected? > > -Chris. Thanks Chris. Yes, gathering write needs changes too, including a specification clarification so it has its own bug. The changes are this patch sort out the common usage and are so risk so that they can go into 7u too. -Alan. From Alan.Bateman at oracle.com Thu Jun 14 01:45:16 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 14 Jun 2012 09:45:16 +0100 Subject: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] In-Reply-To: <667720003EBD41BA9B352ACD21D5B977@HP6910P> References: <667720003EBD41BA9B352ACD21D5B977@HP6910P> Message-ID: <4FD9A49C.7080402@oracle.com> On 14/06/2012 00:28, Esmond Pitt wrote: > Alan > > I suggest the loop should only happen in blocking mode, if at all. > Non-blocking mode programs must already cope with short writes, and > shouldn't be held up dispatching writes, on fairness grounds. I'm not > convinced about the need for the loop in blocking mode either, as > applications must be written correctly, and forums are already full of war > stories about checking write (and read) return values in java.nio and indeed > java.net and java.io. > > EJP > This patch does not change anything when doing writes with <= MAX_BUFFER_SIZE so there isn't any loop in either blocking or non-blocking mode (a short write in non-blocking mode is as before, it just cause the function to return with the number of bytes written). For > MAX_BUFFER_SIZE case this write(ByteBuffer) currently only writes up to MAX_BUFFER_SIZE. The changes means that the blocking write will attempt to write all remaining bytes, in the non-blocking case it will do the same except that a short write will cause the loop to terminate early. At some point I'd like to split the blocking and non-blocking code paths in NIO but not in this patch. As regards people not checking the return value, yes, we see this all the time too. The specification sets the expectation that all remaining bytes will be written in blocking mode so this is why it is viewed as a regression in jdk7. As part of a follow-on patch we will need to adjust the specification slightly as there will be cases in non-blocking mode where all remaining bytes cannot be written, write(ByteBuffer[]) with an array larger than IOV_MAX is one example that comes to mind. -Alan From zhong.j.yu at gmail.com Thu Jun 14 08:08:41 2012 From: zhong.j.yu at gmail.com (Zhong Yu) Date: Thu, 14 Jun 2012 10:08:41 -0500 Subject: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] In-Reply-To: <667720003EBD41BA9B352ACD21D5B977@HP6910P> References: <667720003EBD41BA9B352ACD21D5B977@HP6910P> Message-ID: On Wed, Jun 13, 2012 at 6:28 PM, Esmond Pitt wrote: > Alan > > I suggest the loop should only happen in blocking mode, if at all. > Non-blocking mode programs must already cope with short writes, and > shouldn't be held up dispatching writes, on fairness grounds. I'm not > convinced about the need for the loop in blocking mode either, as > applications must be written correctly, and forums are already full of war > stories about checking write (and read) return values in java.nio and indeed > java.net and java.io. Esmond, can you send some links to these discussions? Alan, if a short write occurs in blocking mode, what is the caller supposed to do? Apparently it's not an error condition, so the caller can immediately try write() again on the remaining bytes? Thanks, Zhong Yu From Alan.Bateman at oracle.com Thu Jun 14 09:24:07 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 14 Jun 2012 17:24:07 +0100 Subject: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] In-Reply-To: References: <667720003EBD41BA9B352ACD21D5B977@HP6910P> Message-ID: <4FDA1027.4020208@oracle.com> On 14/06/2012 16:08, Zhong Yu wrote: > : > > Alan, if a short write occurs in blocking mode, what is the caller > supposed to do? Apparently it's not an error condition, so the caller > can immediately try write() again on the remaining bytes? > Normally you won't see short writes in blocking mode. The only "normal" case, that I'm aware of anyway, is on Windows where you you attempt to write with >= 128k bytes remaining in the buffer and that is the case that this patch attempts to smooth over. There are one or two cases odd cases, say where someone does write(ByteBuffer[]) with an array of hundreds of buffers, more than IOV_MAX. I think the only thing we can do is set expectations in the javadoc and that is something I plan to do. -Alan From zhong.j.yu at gmail.com Thu Jun 14 10:28:04 2012 From: zhong.j.yu at gmail.com (Zhong Yu) Date: Thu, 14 Jun 2012 12:28:04 -0500 Subject: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] In-Reply-To: <4FDA1027.4020208@oracle.com> References: <667720003EBD41BA9B352ACD21D5B977@HP6910P> <4FDA1027.4020208@oracle.com> Message-ID: On Thu, Jun 14, 2012 at 11:24 AM, Alan Bateman wrote: >> Alan, if a short write occurs in blocking mode, what is the caller >> supposed to do? Apparently it's not an error condition, so the caller >> can immediately try write() again on the remaining bytes? >> > Normally you won't see short writes in blocking mode. The only "normal" > case, that I'm aware of anyway, is on Windows where you you attempt to write > with >= 128k bytes remaining in the buffer and that is the case that this > patch attempts to smooth over. There are one or two cases odd cases, say > where someone does write(ByteBuffer[]) with an array of hundreds of buffers, > more than IOV_MAX. I think the only thing we can do is set expectations in > the javadoc and that is something I plan to do. I must be missing something - but in blocking mode, whatever the reason for short write is, why can't write() loop internally, return only when all bytes are written? Why is it necessary to report the short write to caller, who then has to loop externally? From Alan.Bateman at oracle.com Thu Jun 14 11:19:14 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 14 Jun 2012 19:19:14 +0100 Subject: 7176630: (sc) SocketChannel.write does not write more than 128k when channel configured blocking [win] In-Reply-To: References: <667720003EBD41BA9B352ACD21D5B977@HP6910P> <4FDA1027.4020208@oracle.com> Message-ID: <4FDA2B22.3020600@oracle.com> On 14/06/2012 18:28, Zhong Yu wrote: > : > I must be missing something - but in blocking mode, whatever the > reason for short write is, why can't write() loop internally, return > only when all bytes are written? Just so we're clear, this is what this patch does, and only on Windows because that is the only place where we have this issue. -Alan From Alan.Bateman at oracle.com Fri Jun 15 04:32:40 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 15 Jun 2012 12:32:40 +0100 Subject: 7176485: (bf) Allow temporary buffer cache to grow to IOV_MAX Message-ID: <4FDB1D58.8080509@oracle.com> This one relates to the temporary buffer cache, used to cache direct direct buffers for applications that do I/O with buffers that are backed by byte arrays in the heap. Currently the temporary buffer cache can cache up to 8 buffer and this should be sufficient for most scattering read and gathering write operations. There are however extreme cases where the number of buffers in the array is higher and those cases take a big malloc/free hit once they go beyond 8 buffers. The patch proposed here changes the limit so that it matches IOV_MAX. Note that there isn't a regression test with this because this area is covered by existing tests and the changes are just to help the performance for this scenario. http://cr.openjdk.java.net/~alanb/7176485/webrev/ -Alan. From chris.hegarty at oracle.com Fri Jun 15 05:20:10 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Fri, 15 Jun 2012 13:20:10 +0100 Subject: 7176485: (bf) Allow temporary buffer cache to grow to IOV_MAX In-Reply-To: <4FDB1D58.8080509@oracle.com> References: <4FDB1D58.8080509@oracle.com> Message-ID: <4FDB287A.8010305@oracle.com> Looks good to me Alan. -Chris. On 15/06/2012 12:32, Alan Bateman wrote: > > This one relates to the temporary buffer cache, used to cache direct > direct buffers for applications that do I/O with buffers that are backed > by byte arrays in the heap. Currently the temporary buffer cache can > cache up to 8 buffer and this should be sufficient for most scattering > read and gathering write operations. There are however extreme cases > where the number of buffers in the array is higher and those cases take > a big malloc/free hit once they go beyond 8 buffers. The patch proposed > here changes the limit so that it matches IOV_MAX. Note that there isn't > a regression test with this because this area is covered by existing > tests and the changes are just to help the performance for this scenario. > > http://cr.openjdk.java.net/~alanb/7176485/webrev/ > > -Alan. From sean.coffey at oracle.com Fri Jun 15 05:39:45 2012 From: sean.coffey at oracle.com (=?ISO-8859-1?Q?Se=E1n_Coffey?=) Date: Fri, 15 Jun 2012 13:39:45 +0100 Subject: 7176485: (bf) Allow temporary buffer cache to grow to IOV_MAX In-Reply-To: <4FDB1D58.8080509@oracle.com> References: <4FDB1D58.8080509@oracle.com> Message-ID: <4FDB2D11.2070409@oracle.com> Also had a look Alan - looks fine. regards, Sean. On 15/06/12 12:32, Alan Bateman wrote: > > This one relates to the temporary buffer cache, used to cache direct > direct buffers for applications that do I/O with buffers that are > backed by byte arrays in the heap. Currently the temporary buffer > cache can cache up to 8 buffer and this should be sufficient for most > scattering read and gathering write operations. There are however > extreme cases where the number of buffers in the array is higher and > those cases take a big malloc/free hit once they go beyond 8 buffers. > The patch proposed here changes the limit so that it matches IOV_MAX. > Note that there isn't a regression test with this because this area is > covered by existing tests and the changes are just to help the > performance for this scenario. > > http://cr.openjdk.java.net/~alanb/7176485/webrev/ > > -Alan. From Alan.Bateman at oracle.com Sun Jun 17 13:03:17 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sun, 17 Jun 2012 21:03:17 +0100 Subject: 7177617: java/nio/channels/AsyncCloseAndInterrupt.java failing on Windows Message-ID: <4FDE3805.3070606@oracle.com> This one is an amendment to the changes for 7176630 to sort out the asynchronous close case. I unfortunately missed the test failure because it was masked by a completely different issue, the long standing issue that closing a FileChannel when doing a transfer* operation. The proposed patch fix the issue and also fixes the skips this in the test as it was only previously passing by chance. I've also added an additional test so that we have more test coverage for the close during blocking write case. The only minor changes are because I removed TestUtil.onME, not needed now as the JDK hasn't run on Windows ME for several releases. The webrev with the patch is here: http://cr.openjdk.java.net/~alanb/7177617/webrev/ Thanks, Alan. From chris.hegarty at oracle.com Sun Jun 17 14:47:10 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Sun, 17 Jun 2012 22:47:10 +0100 Subject: 7177617: java/nio/channels/AsyncCloseAndInterrupt.java failing on Windows In-Reply-To: <4FDE3805.3070606@oracle.com> References: <4FDE3805.3070606@oracle.com> Message-ID: <001B56DF-DB14-4C40-85B2-AC20AA2C2B52@oracle.com> Changes look fine to me. -Chris On 17 Jun 2012, at 21:03, Alan Bateman wrote: > > This one is an amendment to the changes for 7176630 to sort out the asynchronous close case. I unfortunately missed the test failure because it was masked by a completely different issue, the long standing issue that closing a FileChannel when doing a transfer* operation. The proposed patch fix the issue and also fixes the skips this in the test as it was only previously passing by chance. I've also added an additional test so that we have more test coverage for the close during blocking write case. The only minor changes are because I removed TestUtil.onME, not needed now as the JDK hasn't run on Windows ME for several releases. The webrev with the patch is here: > > http://cr.openjdk.java.net/~alanb/7177617/webrev/ > > Thanks, > Alan. From youdwei at linux.vnet.ibm.com Mon Jun 18 00:37:59 2012 From: youdwei at linux.vnet.ibm.com (Deven You) Date: Mon, 18 Jun 2012 15:37:59 +0800 Subject: Using OP_CONNECT with Selector.select causes selector to fire repeatedly In-Reply-To: <4FCDB0B2.1040902@linux.vnet.ibm.com> References: <4FAA1C40.6040709@linux.vnet.ibm.com> <4FB2BF30.9060902@oracle.com> <4FC72201.7090307@linux.vnet.ibm.com> <4FCDB0B2.1040902@linux.vnet.ibm.com> Message-ID: <4FDEDAD7.60908@linux.vnet.ibm.com> Hi All, Could any one takes a look at this problem, I think we need clarify this situation so that user or customer will avoid keep selecting the OP_CONNECT! Thanks a lot! On 06/05/2012 03:09 PM, Deven You wrote: > Hi Alan, > > What is your opinion about the javadoc clarification? > > Thanks alot! > On 05/31/2012 03:47 PM, Deven You wrote: >> Hi Alan, >> >> Thanks for your comments. How do you feel the javadoc change should >> be, from my knowledge, for the Selector.select(), we may add a >> sentence like "For OP_CONNECT, once a channel is already connected, >> an application should not check OP_CONNECT again, in this situation, >> Selector will ignore OP_CONNECT and return immediately." >> >> On 05/16/2012 04:40 AM, Alan Bateman wrote: >>> Devon, >>> >>> Just to follow up from my previous mail. I looked at the changes in >>> the patch and also tried it out to see the side effects. One issue >>> is that the changes mean that OP_CONNECT will be "automatically" >>> removed from the interest ops, minimally that would require a spec >>> change. When I ran the Selector tests (in the jdk repository) then I >>> noticed a few failures so there are other side effects too. In any >>> case, I think the one is really just a mis-use of the API and that >>> we should instead adding wording to the javadoc rather than change >>> the implementation. >>> >>> -Alan >>> >> >> > > -- Best Regards, Deven From sean.coffey at oracle.com Mon Jun 18 01:34:14 2012 From: sean.coffey at oracle.com (=?ISO-8859-1?Q?Se=E1n_Coffey?=) Date: Mon, 18 Jun 2012 09:34:14 +0100 Subject: 7177617: java/nio/channels/AsyncCloseAndInterrupt.java failing on Windows In-Reply-To: <001B56DF-DB14-4C40-85B2-AC20AA2C2B52@oracle.com> References: <4FDE3805.3070606@oracle.com> <001B56DF-DB14-4C40-85B2-AC20AA2C2B52@oracle.com> Message-ID: <4FDEE806.6040809@oracle.com> + 1. regards, Sean. On 17/06/2012 22:47, Chris Hegarty wrote: > Changes look fine to me. > > -Chris > > On 17 Jun 2012, at 21:03, Alan Bateman wrote: > >> This one is an amendment to the changes for 7176630 to sort out the asynchronous close case. I unfortunately missed the test failure because it was masked by a completely different issue, the long standing issue that closing a FileChannel when doing a transfer* operation. The proposed patch fix the issue and also fixes the skips this in the test as it was only previously passing by chance. I've also added an additional test so that we have more test coverage for the close during blocking write case. The only minor changes are because I removed TestUtil.onME, not needed now as the JDK hasn't run on Windows ME for several releases. The webrev with the patch is here: >> >> http://cr.openjdk.java.net/~alanb/7177617/webrev/ >> >> Thanks, >> Alan. From Alan.Bateman at oracle.com Mon Jun 18 04:27:49 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 18 Jun 2012 12:27:49 +0100 Subject: Using OP_CONNECT with Selector.select causes selector to fire repeatedly In-Reply-To: <4FDEDAD7.60908@linux.vnet.ibm.com> References: <4FAA1C40.6040709@linux.vnet.ibm.com> <4FB2BF30.9060902@oracle.com> <4FC72201.7090307@linux.vnet.ibm.com> <4FCDB0B2.1040902@linux.vnet.ibm.com> <4FDEDAD7.60908@linux.vnet.ibm.com> Message-ID: <4FDF10B5.9070703@oracle.com> On 18/06/2012 08:37, Deven You wrote: > Hi All, > > Could any one takes a look at this problem, I think we need clarify > this situation so that user or customer will avoid keep selecting the > OP_CONNECT! Sorry, I haven't had a chance to look at this. However, I definitely agree we should clarify the javadoc in JDK8. -Alan. From luchsh at linux.vnet.ibm.com Wed Jun 20 00:18:16 2012 From: luchsh at linux.vnet.ibm.com (Jonathan Lu) Date: Wed, 20 Jun 2012 15:18:16 +0800 Subject: Using OP_CONNECT with Selector.select causes selector to fire repeatedly In-Reply-To: <4FDEDAD7.60908@linux.vnet.ibm.com> References: <4FAA1C40.6040709@linux.vnet.ibm.com> <4FB2BF30.9060902@oracle.com> <4FC72201.7090307@linux.vnet.ibm.com> <4FCDB0B2.1040902@linux.vnet.ibm.com> <4FDEDAD7.60908@linux.vnet.ibm.com> Message-ID: <4FE17938.6020702@linux.vnet.ibm.com> Hi Deven, What don't you make a webrev with changes of the javadoc? it is clearer for review. Regards Jonathan On 06/18/2012 03:37 PM, Deven You wrote: > Hi All, > > Could any one takes a look at this problem, I think we need clarify > this situation so that user or customer will avoid keep selecting the > OP_CONNECT! > > Thanks a lot! > On 06/05/2012 03:09 PM, Deven You wrote: >> Hi Alan, >> >> What is your opinion about the javadoc clarification? >> >> Thanks alot! >> On 05/31/2012 03:47 PM, Deven You wrote: >>> Hi Alan, >>> >>> Thanks for your comments. How do you feel the javadoc change should >>> be, from my knowledge, for the Selector.select(), we may add a >>> sentence like "For OP_CONNECT, once a channel is already connected, >>> an application should not check OP_CONNECT again, in this >>> situation, Selector will ignore OP_CONNECT and return immediately." >>> >>> On 05/16/2012 04:40 AM, Alan Bateman wrote: >>>> Devon, >>>> >>>> Just to follow up from my previous mail. I looked at the changes in >>>> the patch and also tried it out to see the side effects. One issue >>>> is that the changes mean that OP_CONNECT will be "automatically" >>>> removed from the interest ops, minimally that would require a spec >>>> change. When I ran the Selector tests (in the jdk repository) then >>>> I noticed a few failures so there are other side effects too. In >>>> any case, I think the one is really just a mis-use of the API and >>>> that we should instead adding wording to the javadoc rather than >>>> change the implementation. >>>> >>>> -Alan >>>> >>> >>> >> >> > > From youdwei at linux.vnet.ibm.com Wed Jun 20 00:21:44 2012 From: youdwei at linux.vnet.ibm.com (Deven You) Date: Wed, 20 Jun 2012 15:21:44 +0800 Subject: Using OP_CONNECT with Selector.select causes selector to fire repeatedly In-Reply-To: <4FDF10B5.9070703@oracle.com> References: <4FAA1C40.6040709@linux.vnet.ibm.com> <4FB2BF30.9060902@oracle.com> <4FC72201.7090307@linux.vnet.ibm.com> <4FCDB0B2.1040902@linux.vnet.ibm.com> <4FDEDAD7.60908@linux.vnet.ibm.com> <4FDF10B5.9070703@oracle.com> Message-ID: <4FE17A08.7010801@linux.vnet.ibm.com> Hi Alan and Jonathan, I have tried modifying the Selector.select() javadoc[1] so that users can get clear information about the behavior of dealing with OP_CONNECT. Could you or any one else take a look at review it. Thanks a lot! [1] http://cr.openjdk.java.net/~littlee/ojdk-317/webrev.01/ On 06/18/2012 07:27 PM, Alan Bateman wrote: > On 18/06/2012 08:37, Deven You wrote: >> Hi All, >> >> Could any one takes a look at this problem, I think we need clarify >> this situation so that user or customer will avoid keep selecting the >> OP_CONNECT! > Sorry, I haven't had a chance to look at this. However, I definitely > agree we should clarify the javadoc in JDK8. > > -Alan. > -- Best Regards, Deven -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120620/0df17e96/attachment.html From Alan.Bateman at oracle.com Wed Jun 20 00:41:49 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 20 Jun 2012 08:41:49 +0100 Subject: Using OP_CONNECT with Selector.select causes selector to fire repeatedly In-Reply-To: <4FE17A08.7010801@linux.vnet.ibm.com> References: <4FAA1C40.6040709@linux.vnet.ibm.com> <4FB2BF30.9060902@oracle.com> <4FC72201.7090307@linux.vnet.ibm.com> <4FCDB0B2.1040902@linux.vnet.ibm.com> <4FDEDAD7.60908@linux.vnet.ibm.com> <4FDF10B5.9070703@oracle.com> <4FE17A08.7010801@linux.vnet.ibm.com> Message-ID: <4FE17EBD.6060704@oracle.com> On 20/06/2012 08:21, Deven You wrote: > Hi Alan and Jonathan, > > I have tried modifying the Selector.select() javadoc[1] so that users > can get clear information about the behavior of dealing with OP_CONNECT. > > Could you or any one else take a look at review it. I'm really busy at the moment and do not have time to spend on this. It will mostly likely require updates to Selector and SelectionKey's class description, a note in SocketChannel connect might be useful too. -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120620/8fc95c05/attachment.html From xueming.shen at oracle.com Fri Jun 22 10:01:10 2012 From: xueming.shen at oracle.com (Xueming Shen) Date: Fri, 22 Jun 2012 10:01:10 -0700 Subject: Codereview request for 7130915: File.equals does not give expected results when path contains Non-English characters on Mac OS X Message-ID: <4FE4A4D6.4070100@oracle.com> Hi Here is the proposed change to support Unicode nfd/nfc and case insensitive file path on MacOSX file system. 7130915: File.equals does not give expected results when path contains Non-English characters on Mac OS X 7168427: FileInputStream cannot open file where the file path contains asian characters [macosx] While these two bug reports are only against java.io, we have the same issue in javax.nio.file. Here is the webrev http://cr.openjdk.java.net/~sherman/7130915_7168427/webrev/ Here is the brief summary of the changes java.io.File: (1) removed nfc->nfd conversion in io_util.h/WITH_PLATFORM_STRING, which means we are now passing nfc/composite characters directly into macosx file system APIs without normalize them to nfd. It appears macosx fs APIs do take nfc, though it uses nfd. (2) normalize the resulting file name from macosx fs APIs from nfd->nfd before passing back to java.io.File (File.list() and canonicalize()), so we deal with nfdc file name (as "usual") for java.io classes/APIs. (3) fs.compare()/hashCode() was updated to be case insensitive (4) hasCode() was updated to use the new String.hash32(). java.nio.file: (5) added a setof MacOSXFile... on top of existing BsdFile... classes. An alternative is to update those BsdFile... classes directly to address the macosx specific issues. But given there might be developers over there might work on open jdk BSD port and have dependency on these classes, it might be desirable to have another separate layer of MacOSXFile... classes. So now the default FileSystem/Provider is MacOSXFileSystemProvider and MacOSXFileSystem. (6) the "main" changes are in MacOSXFileSystem, in which the corresponding methods were added to handle, case insensitive and nfd<=>nfc normalization, including the pathmatcher. (7) compare is now are case-insensitive (8) hashCode is now murmur3_32(), this is true for all Solaris/Unix/Linux and maxosx. Though lots of files have been touched, but the line of changes are actually relatively small. The proposed change only deals with the default case-sensitiveness seting, which is case insensitive. On MaxOSX, you actually can configure the HFS+ file system or the mounted vol to be case-sensitive. A possible approach is to have some extra FileStore attributes, such as a isCaseSensitive and to use case sensitive compare/equal on such fs, but this can be dealt with separately later. Thanks, -Sherman From mike.duigou at oracle.com Fri Jun 22 11:02:28 2012 From: mike.duigou at oracle.com (Mike Duigou) Date: Fri, 22 Jun 2012 11:02:28 -0700 Subject: Codereview request for 7130915: File.equals does not give expected results when path contains Non-English characters on Mac OS X In-Reply-To: <4FE4A4D6.4070100@oracle.com> References: <4FE4A4D6.4070100@oracle.com> Message-ID: On Jun 22 2012, at 10:01 , Xueming Shen wrote: > Hi > > Here is the proposed change to support Unicode nfd/nfc and case insensitive > file path on MacOSX file system. > > 7130915: File.equals does not give expected results when path contains Non-English characters on Mac OS X > 7168427: FileInputStream cannot open file where the file path contains asian characters [macosx] > > While these two bug reports are only against java.io, we have the same issue in javax.nio.file. > Here is the webrev > > (3) fs.compare()/hashCode() was updated to be case insensitive Won't this cause problems on case sensitive file systems? The MacOSX filesystem is by default case insensitive but case sensitive file systems are not entirely uncommon. > (4) hasCode() was updated to use the new String.hash32(). It's possible that this interface may not make it into Java 8. Doug Lea has an alternate proposal for hash based maps that would make this interface unnecessary. > (7) compare is now are case-insensitive Repeated concern about implications for case sensitive file systems. Mike From xueming.shen at oracle.com Fri Jun 22 21:09:48 2012 From: xueming.shen at oracle.com (Xueming Shen) Date: Fri, 22 Jun 2012 21:09:48 -0700 Subject: Codereview request for 7130915: File.equals does not give expected results when path contains Non-English characters on Mac OS X In-Reply-To: References: <4FE4A4D6.4070100@oracle.com> Message-ID: <4FE5418C.1040707@oracle.com> On 6/22/12 11:02 AM, Mike Duigou wrote: > On Jun 22 2012, at 10:01 , Xueming Shen wrote: > >> Hi >> >> Here is the proposed change to support Unicode nfd/nfc and case insensitive >> file path on MacOSX file system. >> >> 7130915: File.equals does not give expected results when path contains Non-English characters on Mac OS X >> 7168427: FileInputStream cannot open file where the file path contains asian characters [macosx] >> >> While these two bug reports are only against java.io, we have the same issue in javax.nio.file. >> Here is the webrev >> >> (3) fs.compare()/hashCode() was updated to be case insensitive > Won't this cause problems on case sensitive file systems? The MacOSX filesystem is by default case insensitive but case sensitive file systems are not entirely uncommon. > > Yes, it might/will cause problem on case sensitive hfs+ file system, but this use scenario is not this patch tries to address. On MacOSX you can format one of your disks to be case sensitive (create a new disk image and set the format to be case sensitive, via the Disk Utility, for example) or you might be able to configure your whole HFS+ file system to be case sensitive, which means the case sensitiveness is actually one of the attributes of the volume (FileStore, in JSR203's term), not the whole file system. But the file system has its default setting regarding the path case sensitiveness. On HFS+ it's case insensitive. This is actually not a unique problem of MacOS file system, you can mount a Windows FAT32 drive on LInux or vise versa, it's a difficult issue. The JSR-203's solution is to use the Path + FileSystem to "modle and be consistent with the platform's virtual file system, not the specific underlying file system", so this means on Unix/Linux, the path matching is case sensitive, on Windows it's case insensitive and on MacOSX, we go with the default case_insensitive. That said, an alternative is to set the default case sensitiveness behavior bases on the setting of the volume that the default file system is mounted on, if the root is on a volume that has case sensitive, then the MaxOSXFileSystem is case sensitive. The code to detect the volume's case sensitive setting is currently committed out. Alan and I chatted about this, we agreed that this is out of the scope of this patch, we can leave that for a future enhancement. -Sherman From Alan.Bateman at oracle.com Sat Jun 23 04:30:22 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 23 Jun 2012 12:30:22 +0100 Subject: Codereview request for 7130915: File.equals does not give expected results when path contains Non-English characters on Mac OS X In-Reply-To: References: <4FE4A4D6.4070100@oracle.com> Message-ID: <4FE5A8CE.4030900@oracle.com> On 22/06/2012 19:02, Mike Duigou wrote: > : > > Won't this cause problems on case sensitive file systems? The MacOSX filesystem is by default case insensitive but case sensitive file systems are not entirely uncommon. > It shouldn't cause any issues accessing files, this is really just about equals, sorting, and path matching. I think it requires a bit of thought as to whether to change this because Apple's JDK6 and older releases does not appear to have changed File#equals. In any case, just to put some context on Sherman's changes, this really just another installation of the port to Mac as this area was not completely ported. In that context then the changes to fix the normalization issues are very welcome. Other missing pieces in this area included the watch service, and also a FileTypeDetector implementation. -Alan. From Alan.Bateman at oracle.com Sat Jun 23 11:52:47 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 23 Jun 2012 19:52:47 +0100 Subject: method name In-Reply-To: <4FE60A48.8030207@oracle.com> References: <4FE60A48.8030207@oracle.com> Message-ID: <4FE6107F.5090604@oracle.com> I've created a bug so that this isn't forgotten: 7179305: (fs) Method name sun.nio.fs.UnixPath.getPathForExecptionMessage is misspelled -Alan From Alan.Bateman at oracle.com Sat Jun 23 13:44:36 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 23 Jun 2012 21:44:36 +0100 Subject: method name In-Reply-To: References: <4FE60A48.8030207@oracle.com> Message-ID: <4FE62AB4.5010809@oracle.com> On 23/06/2012 21:04, Michael Hall wrote: > : > One thing on this. With these nio.2 API's going in a more extensible direction. Wouldn't it make sense to have some of these classes more accessible. > For example, it looks like the watch service api's follow a similar pattern of including a Runnable Poller class to handle the threaded concerns. These extend a inaccessible, non-public, internal AbstractPoller class. Getting the threaded part of the code right for performance and scalability concerns would be an important concern in attempting a 3rd party watch services implementation. I would and am imagining. Modeling the existing implementations as much as possible would be helpful? cc'ing nio-dev as that is the right place for this discussion. The extensions points are via the provider mechanism, see java.nio.file.spi.FileSystemProvider. I can't quite tell what you are doing but if you are looking to "extend" the default provider without changing it then you will need to create your own FileSystemProvider that sits on what would otherwise be the default provider - see the javadoc in FileSystems.getDefault for the details. Also if you rummage around in the tests then you'll find a skeleton implementation that might get your started. The AbstractWatch* and AbstractPoller classes in sun.nio.fs provide the base classes for the 4 implementations that we currently have in OpenJDK. I would be very slow about moving them to java.nio.file.spi (which I think is what you are suggesting) because it is a relatively niche area and likely only interesting when porting to a new platform that has such a facility (many operating systems do not have anything and for those platforms we have the portable PollingWatchService). -Alan. From mik3hall at gmail.com Sat Jun 23 14:14:03 2012 From: mik3hall at gmail.com (Michael Hall) Date: Sat, 23 Jun 2012 16:14:03 -0500 Subject: method name In-Reply-To: <4FE62AB4.5010809@oracle.com> References: <4FE60A48.8030207@oracle.com> <4FE62AB4.5010809@oracle.com> Message-ID: On Jun 23, 2012, at 3:44 PM, Alan Bateman wrote: > On 23/06/2012 21:04, Michael Hall wrote: >> : >> One thing on this. With these nio.2 API's going in a more extensible direction. Wouldn't it make sense to have some of these classes more accessible. >> For example, it looks like the watch service api's follow a similar pattern of including a Runnable Poller class to handle the threaded concerns. These extend a inaccessible, non-public, internal AbstractPoller class. Getting the threaded part of the code right for performance and scalability concerns would be an important concern in attempting a 3rd party watch services implementation. I would and am imagining. Modeling the existing implementations as much as possible would be helpful? > cc'ing nio-dev as that is the right place for this discussion. Fine. I probably should of put it there. I just haven't frequented there as much as OS X port. > > The extensions points are via the provider mechanism, see java.nio.file.spi.FileSystemProvider. I can't quite tell what you are doing but if you are looking to "extend" the default provider without changing it then you will need to create your own FileSystemProvider that sits on what would otherwise be the default provider - see the javadoc in FileSystems.getDefault for the details. Also if you rummage around in the tests then you'll find a skeleton implementation that might get your started. I have the FileSystemProvider mechanism (fwiw [1]), that I used to provide some extended functionality for Mac specific file attributes (Current api's - Finder, Launch Services, NSFileManager derived, and xattr type extended attributes). I was thinking about a possibly custom provider where sync type functionality would be nice. I immediately thought - well this is where watch services is nice now isn't it? I remembered you had mentioned that OS X could use something better than the current polling api for doing this. kqueue I think you mentioned although I've seen it also said that for OS X FSEvents might be a alternative worth considering. I thought I might take a shot at a 'naive' implementation of these two. A learning experience for the api. > > The AbstractWatch* and AbstractPoller classes in sun.nio.fs provide the base classes for the 4 implementations that we currently have in OpenJDK. I would be very slow about moving them to java.nio.file.spi (which I think is what you are suggesting) because it is a relatively niche area and likely only interesting when porting to a new platform that has such a facility (many operating systems do not have anything and for those platforms we have the portable PollingWatchService). Yes, my purpose is for the OS X platform. Although it did occur to me that both kqueue and possibly FSEvents could be cross-platform at least to Unix type systems. > > -Alan. [1] http://www195.pair.com/mik3hall/index.html#macattr From mik3hall at gmail.com Sun Jun 24 06:42:12 2012 From: mik3hall at gmail.com (Michael Hall) Date: Sun, 24 Jun 2012 08:42:12 -0500 Subject: SolarisWatchKey Message-ID: As I think was mentioned previously I am trying to implement some WatchService code. I am for now sort picking SolarisWatchService to model how I am framing my code. Something was causing me a little trouble and I thought I'd verify it is intended and isn't at least a little confusing in the Solaris implementation. My source might be non-current and this already changed, my apologies if that is the case. AbstractWatchKey has the field // pending events private List> events; I can't test exactly this having no Solaris access but I believe this means for SolarisWatchKey which doesn't override pollEvents that this AbstractWatchKey method will return the List field. SolarisWatchKey (in SolarisWatchService.java) which extends AbstractWatchKey has the field // events (may be changed). set to null when watch key is invalid private volatile Set> events; Any references to events in the subclass will I believe reference this field. And if say it was decided to override the pollEvents method any references to events would now be to the subclass 's WatchEvent.Kind Set? Is there not some possibility this might cause someone else some confusion in the future having these different type fields have the same name? From xueming.shen at oracle.com Sun Jun 24 09:58:01 2012 From: xueming.shen at oracle.com (Xueming Shen) Date: Sun, 24 Jun 2012 09:58:01 -0700 Subject: Codereview request for 7130915: File.equals does not give expected results when path contains Non-English characters on Mac OS X In-Reply-To: <191321F8-8360-44A2-AA61-1504B074E264@sudo.ch> References: <4FE4A4D6.4070100@oracle.com> <191321F8-8360-44A2-AA61-1504B074E264@sudo.ch> Message-ID: <4FE74719.8080709@oracle.com> Yes, I believe the issue described in MACOSX_PORT-165 is the same issue this patch is trying to solve. Btw, it appears there are typos in the note(2), my mini keyboard obviously is too sticky:-) (2) normalize the resulting file name from macosx fs APIs from nfd->nfc before passing back to java.io.File (File.list() and canonicalize()), so we deal with nfc file name (as "usual") for java.io classes/APIs. -sherman On 6/24/12 7:50 AM, David Kocher wrote: > Will this address issue MACOSX_PORT-165 [1]? > > [1] http://java.net/jira/browse/MACOSX_PORT-165 > > -- David > > On 22.06.2012, at 19:01, Xueming Shen wrote: > >> Hi >> >> Here is the proposed change to support Unicode nfd/nfc and case insensitive >> file path on MacOSX file system. >> >> 7130915: File.equals does not give expected results when path contains Non-English characters on Mac OS X >> 7168427: FileInputStream cannot open file where the file path contains asian characters [macosx] >> >> While these two bug reports are only against java.io, we have the same issue in javax.nio.file. >> Here is the webrev >> >> http://cr.openjdk.java.net/~sherman/7130915_7168427/webrev/ >> >> Here is the brief summary of the changes >> >> java.io.File: >> (1) removed nfc->nfd conversion in io_util.h/WITH_PLATFORM_STRING, which means >> we are now passing nfc/composite characters directly into macosx file system APIs >> without normalize them to nfd. It appears macosx fs APIs do take nfc, though it uses >> nfd. >> >> (2) normalize the resulting file name from macosx fs APIs from nfd->nfd before passing >> back to java.io.File (File.list() and canonicalize()), so we deal with nfdc file name >> (as "usual") for java.io classes/APIs. >> >> (3) fs.compare()/hashCode() was updated to be case insensitive >> >> (4) hasCode() was updated to use the new String.hash32(). >> >> java.nio.file: >> >> (5) added a setof MacOSXFile... on top of existing BsdFile... classes. An alternative is to >> update those BsdFile... classes directly to address the macosx specific issues. But given >> there might be developers over there might work on open jdk BSD port and have dependency >> on these classes, it might be desirable to have another separate layer of MacOSXFile... >> classes. So now the default FileSystem/Provider is MacOSXFileSystemProvider and >> MacOSXFileSystem. >> >> (6) the "main" changes are in MacOSXFileSystem, in which the corresponding methods >> were added to handle, case insensitive and nfd<=>nfc normalization, including the >> pathmatcher. >> >> (7) compare is now are case-insensitive >> >> (8) hashCode is now murmur3_32(), this is true for all Solaris/Unix/Linux and maxosx. >> >> >> Though lots of files have been touched, but the line of changes are actually relatively >> small. >> >> The proposed change only deals with the default case-sensitiveness seting, which is >> case insensitive. On MaxOSX, you actually can configure the HFS+ file system or the >> mounted vol to be case-sensitive. A possible approach is to have some extra FileStore >> attributes, such as a isCaseSensitive and to use case sensitive compare/equal on >> such fs, but this can be dealt with separately later. >> >> Thanks, >> -Sherman >> From jeffhain at rocketmail.com Sun Jun 24 11:08:54 2012 From: jeffhain at rocketmail.com (Jeff Hain) Date: Sun, 24 Jun 2012 19:08:54 +0100 (BST) Subject: Bitwise ByteBuffer Message-ID: <1340561334.87205.YahooMailNeo@web132102.mail.ird.yahoo.com> Hello. Does anyone know of a sort of bitwise version of ByteBuffer, which would allow to put and get integer values at specified bit positions, and over specified numbers of bits, in big or little endian? And is there any plan to introduce one in the JDK? Thanks. -Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120624/f2a98e9f/attachment.html From Alan.Bateman at oracle.com Mon Jun 25 02:00:19 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 25 Jun 2012 10:00:19 +0100 Subject: SolarisWatchKey In-Reply-To: References: Message-ID: <4FE828A3.9030703@oracle.com> On 24/06/2012 14:42, Michael Hall wrote: > As I think was mentioned previously I am trying to implement some WatchService code. > I am for now sort picking SolarisWatchService to model how I am framing my code. > Something was causing me a little trouble and I thought I'd verify it is intended and isn't at least a little confusing in the Solaris implementation. > My source might be non-current and this already changed, my apologies if that is the case. > > AbstractWatchKey has the field > > // pending events > private List> events; > > I can't test exactly this having no Solaris access but I believe this means for SolarisWatchKey which doesn't override pollEvents that this AbstractWatchKey method will return the List field. It might be better to rename this field to pendingEvents to make it clearer. The methods that access it (pollEvents, signalEvent and reset) are final. Note that the Solaris 11 WatchService implementation is significantly more complicated that the implementations for the other platforms so you might be better looking at the other implementations first to get a feel for how they work. -Alan. From mik3hall at gmail.com Mon Jun 25 02:41:38 2012 From: mik3hall at gmail.com (Michael Hall) Date: Mon, 25 Jun 2012 04:41:38 -0500 Subject: SolarisWatchKey In-Reply-To: <4FE828A3.9030703@oracle.com> References: <4FE828A3.9030703@oracle.com> Message-ID: <135B0FEE-10E1-47F9-AFE5-83282734C2EB@gmail.com> On Jun 25, 2012, at 4:00 AM, Alan Bateman wrote: > On 24/06/2012 14:42, Michael Hall wrote: >> As I think was mentioned previously I am trying to implement some WatchService code. >> I am for now sort picking SolarisWatchService to model how I am framing my code. >> Something was causing me a little trouble and I thought I'd verify it is intended and isn't at least a little confusing in the Solaris implementation. >> My source might be non-current and this already changed, my apologies if that is the case. >> >> AbstractWatchKey has the field >> >> // pending events >> private List> events; >> >> I can't test exactly this having no Solaris access but I believe this means for SolarisWatchKey which doesn't override pollEvents that this AbstractWatchKey method will return the List field. > It might be better to rename this field to pendingEvents to make it clearer. The methods that access it (pollEvents, signalEvent and reset) are final. I ran the abstract superclass and and subclass together in this case since I didn't have access to the abstract. I ran into problems with the field doing double duty attempting that. Otherwise it probably would not usually matter as long the abstract methods are final for access to the private field. Although different names for the fields for anyone even just trying to understand the code might be clearer. > > Note that the Solaris 11 WatchService implementation is significantly more complicated that the implementations for the other platforms so you might be better looking at the other implementations first to get a feel for how they work. > > -Alan. It wasn't immediately clear what the Solaris implementation was doing but then none of the watch services were simple at a quick glance . This is definitely more involved than accessing api's for file attributes. I was hoping to get something in place modeling the process and then start removing it's original functionality and replace that with kqueue/fsevents. I will look at the others as you suggest if that simplifies understanding things I'm all for it. From t.lasseter at comcast.net Mon Jun 25 20:18:23 2012 From: t.lasseter at comcast.net (Tom Lasseter) Date: Mon, 25 Jun 2012 20:18:23 -0700 Subject: FileSystemProvider example.... Message-ID: <033f01cd534a$58b33fc0$0a19bf40$@comcast.net> The new Java 7 FileSystem is very cool..just what I need.but for the life of me I can't create a custom FileSystemProvider. There must be an example somewhere! There is only the ZipFileSystemProvider. I want to create a new FileSystem on either my local system or on a remote system. The simplest case is given the local system as the DefaultFileSystemProvider, I want a new FileSystem from it rooted at some arbitrary path, like c:/FileSystems/myFileSystem. What could be easier than that given the technology being provided. I would think that this would do it: Map env = new HashMap<>(); env.put("create", "true"); Path path = Paths.get("c:/FileSystems/myFileSystem"); FileSystem fileSystem = null; try { fileSystem = fileSystemProvider.newFileSystem(path, env); } catch(Exception e) { e.printStackTrace(); } Help would be appreciated. Thanks and regards, Tom -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120625/801d1366/attachment.html From xueming.shen at oracle.com Mon Jun 25 22:58:07 2012 From: xueming.shen at oracle.com (Xueming Shen) Date: Mon, 25 Jun 2012 22:58:07 -0700 Subject: Codereview request for 7130915: File.equals does not give expected results when path contains Non-English characters on Mac OS X In-Reply-To: <4FE4A4D6.4070100@oracle.com> References: <4FE4A4D6.4070100@oracle.com> Message-ID: <4FE94F6F.2060002@oracle.com> Hi, While I still believe that case-insensitive is the right choice for File/Path on MacOSX, it is suggested that we might want to be a little conservative in this patch, with the assumption that this patch will be backport to 7u release after being baked in jdk8 for a while (given Apple's JDK6 is case sensitive for File, it might be too much for a update releases to go with two in-compatible changes, case sensitive and hash32). So here is the webrev for a strip-down version from the original patch, in which it only targets the nfd/nfc issue raised in 7130915 and 7168427. The proposed changes for case insensitive compare and hash32 for both java.io.File and java.nio.file.Path are removed. http://cr.openjdk.java.net/~sherman/7130915_7168427/webrev (The webrev for the "full" version has been moved to http://cr.openjdk.java.net/~sherman/7130915_7168427/webrev_full) Thanks, -Sherman On 6/22/12 10:01 AM, Xueming Shen wrote: > Hi > > Here is the proposed change to support Unicode nfd/nfc and case > insensitive > file path on MacOSX file system. > > 7130915: File.equals does not give expected results when path contains > Non-English characters on Mac OS X > 7168427: FileInputStream cannot open file where the file path contains > asian characters [macosx] > > While these two bug reports are only against java.io, we have the same > issue in javax.nio.file. > Here is the webrev > > http://cr.openjdk.java.net/~sherman/7130915_7168427/webrev/ > > Here is the brief summary of the changes > > java.io.File: > (1) removed nfc->nfd conversion in io_util.h/WITH_PLATFORM_STRING, > which means > we are now passing nfc/composite characters directly into macosx > file system APIs > without normalize them to nfd. It appears macosx fs APIs do take > nfc, though it uses > nfd. > > (2) normalize the resulting file name from macosx fs APIs from > nfd->nfc before passing > back to java.io.File (File.list() and canonicalize()), so we deal > with nfc file name > (as "usual") for java.io classes/APIs. > > (3) fs.compare()/hashCode() was updated to be case insensitive > > (4) hasCode() was updated to use the new String.hash32(). > > java.nio.file: > > (5) added a setof MacOSXFile... on top of existing BsdFile... classes. > An alternative is to > update those BsdFile... classes directly to address the macosx > specific issues. But given > there might be developers over there might work on open jdk BSD port > and have dependency > on these classes, it might be desirable to have another separate layer > of MacOSXFile... > classes. So now the default FileSystem/Provider is > MacOSXFileSystemProvider and > MacOSXFileSystem. > > (6) the "main" changes are in MacOSXFileSystem, in which the > corresponding methods > were added to handle, case insensitive and nfd<=>nfc normalization, > including the > pathmatcher. > > (7) compare is now are case-insensitive > > (8) hashCode is now murmur3_32(), this is true for all > Solaris/Unix/Linux and maxosx. > > > Though lots of files have been touched, but the line of changes are > actually relatively > small. > > The proposed change only deals with the default case-sensitiveness > seting, which is > case insensitive. On MaxOSX, you actually can configure the HFS+ file > system or the > mounted vol to be case-sensitive. A possible approach is to have some > extra FileStore > attributes, such as a isCaseSensitive and to use case sensitive > compare/equal on > such fs, but this can be dealt with separately later. > > Thanks, > -Sherman From Alan.Bateman at oracle.com Tue Jun 26 02:07:39 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 26 Jun 2012 10:07:39 +0100 Subject: FileSystemProvider example.... In-Reply-To: <033f01cd534a$58b33fc0$0a19bf40$@comcast.net> References: <033f01cd534a$58b33fc0$0a19bf40$@comcast.net> Message-ID: <4FE97BDB.3010508@oracle.com> On 26/06/2012 04:18, Tom Lasseter wrote: > > The new Java 7 FileSystem is very cool....just what I need...but for > the life of me I can't create a custom FileSystemProvider. There must > be an example somewhere! There is only the ZipFileSystemProvider. I > want to create a new FileSystem on either my local system or on a > remote system. > > The simplest case is given the local system as the > DefaultFileSystemProvider, I want a new FileSystem from it rooted at > some arbitrary path, like c:/FileSystems/myFileSystem. What could be > easier than that given the technology being provided. I would think > that this would do it: > > Map env = new HashMap<>(); > > env.put("create", "true"); > > Path path = Paths.get("c:/FileSystems/myFileSystem"); > > FileSystem fileSystem = null; > > try > > { > > fileSystem = fileSystemProvider.newFileSystem(path, env); > > } > > catch(Exception e) > > { > > e.printStackTrace(); > > } > > Help would be appreciated. > > Thanks and regards, > > Tom > Developing your own file system implementation is non-trivial, there is quite a bit of work to do it right. The zip provider is the example provider that we include in the JDK, you'll find the source code in the $JDK_HOME/demo/nio/zipfs directory. Sorry, we don't include any others at this time but there are folks on this list that may be able to share their experiences. In this API then FileSystems.getDefault() will give you a reference to the FileSystem for your local file system. This can be used to access of the volumes that you have mounted, be they local or remote file systems. If I understand your mail correctly then you're looking to create a chroot like environment. One idea is to grab the PassThroughFileSystem that you'll find in the test sources and then hack on it so that you get to a relative path that can be resolved against whatever root you want. You'll also need to get the eventual real path to deal with sym links and other cases that would escape into other areas of the default file system. A fun project. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120626/627258af/attachment.html From Alan.Bateman at oracle.com Tue Jun 26 02:21:02 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 26 Jun 2012 10:21:02 +0100 Subject: Bitwise ByteBuffer In-Reply-To: <1340561334.87205.YahooMailNeo@web132102.mail.ird.yahoo.com> References: <1340561334.87205.YahooMailNeo@web132102.mail.ird.yahoo.com> Message-ID: <4FE97EFE.2090905@oracle.com> On 24/06/2012 19:08, Jeff Hain wrote: > Hello. > > Does anyone know of a sort of bitwise version of ByteBuffer, > which would allow to put and get integer values at specified > bit positions, and over specified numbers of bits, in big or > little endian? > > And is there any plan to introduce one in the JDK? > I'm not aware of anyone working on something like this. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120626/ca2ac884/attachment.html From mik3hall at gmail.com Tue Jun 26 03:57:29 2012 From: mik3hall at gmail.com (Michael Hall) Date: Tue, 26 Jun 2012 05:57:29 -0500 Subject: FileSystemProvider example.... In-Reply-To: <4FE97BDB.3010508@oracle.com> References: <033f01cd534a$58b33fc0$0a19bf40$@comcast.net> <4FE97BDB.3010508@oracle.com> Message-ID: <6285AA93-EB09-46B3-8AA2-94EA3C23FE7E@gmail.com> On Jun 26, 2012, at 4:07 AM, Alan Bateman wrote: > On 26/06/2012 04:18, Tom Lasseter wrote: >> >> >> >> The new Java 7 FileSystem is very cool?.just what I need?but for the life of me I can?t create a custom FileSystemProvider. There must be an example somewhere! There is only the ZipFileSystemProvider. I want to create a new FileSystem on either my local system or on a remote system. >> >> The simplest case is given the local system as the DefaultFileSystemProvider, I want a new FileSystem from it rooted at some arbitrary path, like c:/FileSystems/myFileSystem. What could be easier than that given the technology being provided. I would think that this would do it: >> >> Map env = new HashMap<>(); >> env.put("create", "true"); >> Path path = Paths.get(?c:/FileSystems/myFileSystem?); >> FileSystem fileSystem = null; >> try >> { >> fileSystem = fileSystemProvider.newFileSystem(path, env); >> } >> catch(Exception e) >> { >> e.printStackTrace(); >> } >> >> Help would be appreciated. >> >> Thanks and regards, >> >> Tom >> >> Possibly helpful. Developing a Custom File System Provider http://docs.oracle.com/javase/7/docs/technotes/guides/io/fsp/filesystemprovider.html > Developing your own file system implementation is non-trivial, there is quite a bit of work to do it right. The zip provider is the example provider that we include in the JDK, you'll find the source code in the $JDK_HOME/demo/nio/zipfs directory. Sorry, we don't include any others at this time but there are folks on this list that may be able to share their experiences. > > In this API then FileSystems.getDefault() will give you a reference to the FileSystem for your local file system. This can be used to access of the volumes that you have mounted, be they local or remote file systems. > > If I understand your mail correctly then you're looking to create a chroot like environment. One idea is to grab the PassThroughFileSystem that you'll find in the test sources and then hack on it so that you get to a relative path that can be resolved against whatever root you want. You'll also need to get the eventual real path to deal with sym links and other cases that would escape into other areas of the default file system. A fun project. Starting with a PassThroughFileSystem implementation allows a incremental or more piecemeal approach.You don't have to change everything the zip demo does. You can change only the parts of interest. Then later improve that support or add support for something else. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120626/c735271c/attachment-0001.html From Alan.Bateman at oracle.com Tue Jun 26 08:02:11 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 26 Jun 2012 16:02:11 +0100 Subject: Fwd: Codereview request for 7130915: File.equals does not give expected results when path contains Non-English characters on Mac OS X In-Reply-To: <4FE94FED.6050406@oracle.com> References: <4FE4A4D6.4070100@oracle.com> <4FE94F6F.2060002@oracle.com> <4FE94FED.6050406@oracle.com> Message-ID: <4FE9CEF3.8080905@oracle.com> On 26/06/2012 07:00, Xueming Shen wrote: > On 6/25/12 10:58 PM, Xueming Shen wrote: >> Hi, >> >> While I still believe that case-insensitive is the right choice for >> File/Path on MacOSX, it is >> suggested that we might want to be a little conservative in this >> patch, with the assumption >> that this patch will be backport to 7u release after being baked in >> jdk8 for a while (given >> Apple's JDK6 is case sensitive for File, it might be too much for a >> update releases to go >> with two in-compatible changes, case sensitive and hash32). >> >> So here is the webrev for a strip-down version from the original >> patch, in which it only >> targets the nfd/nfc issue raised in 7130915 and 7168427. The proposed >> changes for >> case insensitive compare and hash32 for both java.io.File and >> java.nio.file.Path are >> removed. >> >> http://cr.openjdk.java.net/~sherman/7130915_7168427/webrev >> >> (The webrev for the "full" version has been moved to >> http://cr.openjdk.java.net/~sherman/7130915_7168427/webrev_full) >> >> Thanks, >> -Sherman I had to dig up the File Systems, Unicode, and Normalization presentation [1] before reviewing this, it's been a while. I think the changes for java.io for fine, I just worry that there may be a few odd cases where it will be different to Apple JDK6. I looked through the macosx-port/macosx-port/jdk forest and there is one patch from Apple that does the NFC->NFD conversation. I don't get that patch because you say that the syscalls handle NFC okay. In your changes then you normalize when decoding the file names to Strings, which seems right but that seems to be completely missing from the changes that Apple brought over. The only minor comment is that we probably need to check the return from core foundation functions such as CFStringCreateMutable (this goes for the other native code too). Adding the FileSystemProvider for MacOSX is great to see. The approach is fine for but is somewhat inefficient when ->NFD or ->NFC is needed. One inefficiency that can be fixed is in sun.nio.fs.MacOSXFileSystem. normalizeJavaPath then you can eliminate the second call to toCharArray. I think the new methods on sun.no.fs.UnixFileSystem should be given comments so that it's clear whether they should be overridden (the Unix* provider tends to be starting point for most ports). A minor comment is that MacOSXNativeDispatcher has a bunch of blank lines at the end, also I think "static final" is more normal than "final static". At some point I think we should re-write MacOSXNativeDispatcher.normalizepath so that it deosn't require the critical section or malloc as in this area we try to keep the native methods to a bare minimum. Do the tests assume they are run on HFS? Just wondering if you you need to look at the FileStore name/type to check. Some of variable namesa bit non-standard, very C like. Minor nit is that the copyright in the header isn't right in the tests. Also the tests list 2 CRs whereas I assume this is one CR (with the other closed as a dup). On case sensitive vs. insensitive then I think it should be insensitive as per the first round but that would be a significant change given that Apple's JDK does not appear to have changed File.equals. Maybe we should think about this again once these changes are in 8. -Alan [1] http://developers.sun.com/global/products_platforms/solaris/reference/presentations/IUC29-FileSystems.pdf -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120626/3bd52767/attachment.html From xueming.shen at oracle.com Tue Jun 26 20:33:50 2012 From: xueming.shen at oracle.com (Xueming Shen) Date: Tue, 26 Jun 2012 20:33:50 -0700 Subject: Fwd: Codereview request for 7130915: File.equals does not give expected results when path contains Non-English characters on Mac OS X In-Reply-To: <4FE9CEF3.8080905@oracle.com> References: <4FE4A4D6.4070100@oracle.com> <4FE94F6F.2060002@oracle.com> <4FE94FED.6050406@oracle.com> <4FE9CEF3.8080905@oracle.com> Message-ID: <4FEA7F1E.4070500@oracle.com> Alan, Webrev has been updated accordingly at http://cr.openjdk.java.net/~sherman/7130915/webrev with changes (1) added a CFStringCreateMutable(...) != null check in both io and nio native, though it is unlikely to fail here because we are passing a NULL and 0 length, like new StringBuilder() invocation, if it fails the system probably goes very wrong. Both FStringAppendCharacters and CFStringNormalize are void return type. (2) The first line of path.toCharArray in normalizeJavaPath is a typo, it is supposed to be deleted. The implementation only goes toCharArray when it needs to go native. Currently it uses >0x80 as the fast path criteria, it is possible to expose some sun.text.normalizer's internal methods to have a "quick nfc" check, but I'm not sure how much the performance gain would be, but might worth some investigation later. (3) Comments have been added for those override-able methods in UnixFileSystem. (4) blank lines have been removed from dispatcher.c (5) I don't think we need to do the HFS check, given we are only doing nfc/nfd stuff now, as long as it is a MacOSX, I believe its nfc/nfd layer will be there. Copyright has been re-copy/ pasted and we now only use only bugid. -Sherman On 6/26/12 8:02 AM, Alan Bateman wrote: > On 26/06/2012 07:00, Xueming Shen wrote: >> On 6/25/12 10:58 PM, Xueming Shen wrote: >>> Hi, >>> >>> While I still believe that case-insensitive is the right choice for >>> File/Path on MacOSX, it is >>> suggested that we might want to be a little conservative in this >>> patch, with the assumption >>> that this patch will be backport to 7u release after being baked in >>> jdk8 for a while (given >>> Apple's JDK6 is case sensitive for File, it might be too much for a >>> update releases to go >>> with two in-compatible changes, case sensitive and hash32). >>> >>> So here is the webrev for a strip-down version from the original >>> patch, in which it only >>> targets the nfd/nfc issue raised in 7130915 and 7168427. The >>> proposed changes for >>> case insensitive compare and hash32 for both java.io.File and >>> java.nio.file.Path are >>> removed. >>> >>> http://cr.openjdk.java.net/~sherman/7130915_7168427/webrev >>> >>> (The webrev for the "full" version has been moved to >>> http://cr.openjdk.java.net/~sherman/7130915_7168427/webrev_full) >>> >>> Thanks, >>> -Sherman > I had to dig up the File Systems, Unicode, and Normalization > presentation [1] before reviewing this, it's been a while. > > I think the changes for java.io for fine, I just worry that there may > be a few odd cases where it will be different to Apple JDK6. I looked > through the macosx-port/macosx-port/jdk forest and there is one patch > from Apple that does the NFC->NFD conversation. I don't get that patch > because you say that the syscalls handle NFC okay. In your changes > then you normalize when decoding the file names to Strings, which > seems right but that seems to be completely missing from the changes > that Apple brought over. The only minor comment is that we probably > need to check the return from core foundation functions such as > CFStringCreateMutable (this goes for the other native code too). > > Adding the FileSystemProvider for MacOSX is great to see. The approach > is fine for but is somewhat inefficient when ->NFD or ->NFC is needed. > One inefficiency that can be fixed is in sun.nio.fs.MacOSXFileSystem. > normalizeJavaPath then you can eliminate the second call to > toCharArray. I think the new methods on sun.no.fs.UnixFileSystem > should be given comments so that it's clear whether they should be > overridden (the Unix* provider tends to be starting point for most > ports). A minor comment is that MacOSXNativeDispatcher has a bunch of > blank lines at the end, also I think "static final" is more normal > than "final static". At some point I think we should re-write > MacOSXNativeDispatcher.normalizepath so that it deosn't require the > critical section or malloc as in this area we try to keep the native > methods to a bare minimum. > > Do the tests assume they are run on HFS? Just wondering if you you > need to look at the FileStore name/type to check. Some of variable > namesa bit non-standard, very C like. Minor nit is that the copyright > in the header isn't right in the tests. Also the tests list 2 CRs > whereas I assume this is one CR (with the other closed as a dup). > > On case sensitive vs. insensitive then I think it should be > insensitive as per the first round but that would be a significant > change given that Apple's JDK does not appear to have changed > File.equals. Maybe we should think about this again once these changes > are in 8. > > -Alan > > [1] > http://developers.sun.com/global/products_platforms/solaris/reference/presentations/IUC29-FileSystems.pdf -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120626/afea832b/attachment.html From t.lasseter at comcast.net Tue Jun 26 22:49:49 2012 From: t.lasseter at comcast.net (Tom Lasseter) Date: Tue, 26 Jun 2012 22:49:49 -0700 Subject: FileSystemProvider example.... In-Reply-To: <033f01cd534a$58b33fc0$0a19bf40$@comcast.net> References: <033f01cd534a$58b33fc0$0a19bf40$@comcast.net> Message-ID: <000701cd5428$aaf83270$00e89750$@comcast.net> We can very easily use the existing local FileSystemProvider to generate a new local FileSystem rooted at an arbitrary point in the code below. The somewhat bizarre "trick" here is using the USER_DIR property to set the root location for the FileSystem ultimately provided: System.setProperty(USER_DIR, partition + rootPathname + args[0]); // partition: "driveLetter:", rootPathname: root for all FileSystems created, arg[0]: relative path to the root subdirectory desired and then creating the new FileSystem: uri = new URI("file", "", "/" , null); // Anything other than "/" is NOT ALLOWED (see checkURI(uri) method in WindowsFileSystemProvider) FileSystemProvider fileSystemProvider = FileSystemProvider.installedProviders().get(0); // assumption here is that first one is the "default" FileSystem fileSystem = fileSystemProvider.getFileSystem(uri); The output when you run the code below is: User dir: G:/FileSystems/projectA File path: \test.txt Deleted file: \test.txt Writing... String written: Hello! Reading... String read: Hello! I've tried everything else, but this is the only way I've found to use the local FileSystemProvider to generate local FileSystem(s). Ideally, you would think you would not have to use this System.setProperty hack and could provide the root path in the URI constructor above as: uri = new URI("file", "", "g:/FileSystems/ProjectA" , null); This doesn't work as the FileSystemProvider insists on having the URI defined effectively as just "/" (above). You almost might ask: "Why am I being asked in the constructor to specify the URI if the only one it will take is "/" ? Running on Windows, the WindowsFileSystemProvider is in package sun.nio.fs. the newFileSystem and getFileSystem methods both call the checkURI method which must return true for the FileSystem to be created/fetched: private void checkUri(URI uri) { if (!uri.getScheme().equalsIgnoreCase(getScheme())) throw new IllegalArgumentException("URI does not match this provider"); if (uri.getAuthority() != null) throw new IllegalArgumentException("Authority component present"); if (uri.getPath() == null) throw new IllegalArgumentException("Path component is undefined"); if (!uri.getPath().equals("/")) throw new IllegalArgumentException("Path component should be '/'"); if (uri.getQuery() != null) throw new IllegalArgumentException("Query component present"); if (uri.getFragment() != null) throw new IllegalArgumentException("Fragment component present"); } So that: if(!uri.getPath().equals("/")) you are hosed. Thus you need to set the root in some other sneaky fashion such as the USER_DIR trick above. Seems like either (1) there is a more flexible constructor; or (2) there isn't and the URI path argument should be removed from the constructor, and thus the only allowed URI is "scheme:/" where scheme must match the provider. I'm SURE I must be missing something. I can appreciate that if FileSystem(s) are created but they are unknown effectively to the defaultFileSystem, then they can't be created higgily-piggily (arcane technical term). So by forcing this very restrictive URI, we are limiting where they actually can be created. Seems like there should be a somewhat less restrictive capability..which is why I'm sure that I am REALLY missing something. Regards, Tom Code: public class GcFileSystemProvider { static public String partition = "G:/"; static public String rootPathname = "FileSystems/"; private static final String USER_DIR = "user.dir"; GcFileSystemProvider() { } static public void main(String[] args) { System.setProperty(USER_DIR, partition + rootPathname + args[0]); String userDir = System.getProperty(USER_DIR); System.out.println("User dir: " + userDir); URI uri = null; Map env = new HashMap<>(); env.put("create", "true"); FileSystem fileSystem = null; FileSystemProvider fileSystemProvider = null; try { uri = new URI("file", "", "/" , null); fileSystemProvider = FileSystemProvider.installedProviders().get(0); // assumption here is that first one is the "standard" fileSystem = fileSystemProvider.getFileSystem(uri); if(fileSystem == null) fileSystem = fileSystemProvider.newFileSystem(uri, env); } catch(Exception e) { e.printStackTrace(); } Path path = fileSystem.getPath("/test.txt"); System.out.println("File path: " + path.toString()); String text = "Hello!"; byte[] bytes = text.getBytes(); try { if(Files.deleteIfExists(path)) System.out.println("Deleted file: " + path.toString()); Set writeOptions = new HashSet(); writeOptions.add(StandardOpenOption.CREATE); writeOptions.add(StandardOpenOption.TRUNCATE_EXISTING); writeOptions.add(StandardOpenOption.WRITE); Files.createFile(path); SeekableByteChannel seekableByteChannel = fileSystemProvider.newByteChannel(path, writeOptions); ByteBuffer writeByteBuffer = ByteBuffer.wrap(bytes); System.out.println("Writing..."); System.out.println("String written: " + text); seekableByteChannel.write(writeByteBuffer); seekableByteChannel.close(); } catch(Exception e) { e.printStackTrace(); } System.out.println("Reading..."); try { Set readOptions = new HashSet(); readOptions.add(StandardOpenOption.READ); SeekableByteChannel seekableReadByteChannel = fileSystemProvider.newByteChannel(path, readOptions); ByteBuffer readByteBuffer = ByteBuffer.allocate(100); int bytesRead = seekableReadByteChannel.read(readByteBuffer); byte[] readBytes = readByteBuffer.array(); String readString = new String(readBytes); System.out.println("String read: " + readString); } catch (IOException e) { e.printStackTrace(); } } } From: Tom Lasseter [mailto:t.lasseter at comcast.net] Sent: Monday, June 25, 2012 8:18 PM To: nio-dev at openjdk.java.net Subject: FileSystemProvider example.... The new Java 7 FileSystem is very cool..just what I need.but for the life of me I can't create a custom FileSystemProvider. There must be an example somewhere! There is only the ZipFileSystemProvider. I want to create a new FileSystem on either my local system or on a remote system. The simplest case is given the local system as the DefaultFileSystemProvider, I want a new FileSystem from it rooted at some arbitrary path, like c:/FileSystems/myFileSystem. What could be easier than that given the technology being provided. I would think that this would do it: Map env = new HashMap<>(); env.put("create", "true"); Path path = Paths.get("c:/FileSystems/myFileSystem"); FileSystem fileSystem = null; try { fileSystem = fileSystemProvider.newFileSystem(path, env); } catch(Exception e) { e.printStackTrace(); } Help would be appreciated. Thanks and regards, Tom -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120626/f06f1efc/attachment-0001.html From Alan.Bateman at oracle.com Tue Jun 26 23:41:37 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 27 Jun 2012 07:41:37 +0100 Subject: Fwd: Codereview request for 7130915: File.equals does not give expected results when path contains Non-English characters on Mac OS X In-Reply-To: <4FEA7F1E.4070500@oracle.com> References: <4FE4A4D6.4070100@oracle.com> <4FE94F6F.2060002@oracle.com> <4FE94FED.6050406@oracle.com> <4FE9CEF3.8080905@oracle.com> <4FEA7F1E.4070500@oracle.com> Message-ID: <4FEAAB21.9050205@oracle.com> On 27/06/2012 04:33, Xueming Shen wrote: > Alan, > > Webrev has been updated accordingly at > > http://cr.openjdk.java.net/~sherman/7130915/webrev > > with changes > > (1) added a CFStringCreateMutable(...) != null check in both io and > nio native, though it is > unlikely to fail here because we are passing a NULL and 0 > length, like new StringBuilder() > invocation, if it fails the system probably goes very wrong. Both > FStringAppendCharacters > and CFStringNormalize are void return type. > > (2) The first line of path.toCharArray in normalizeJavaPath is a typo, > it is supposed to be > deleted. The implementation only goes toCharArray when it needs > to go native. Currently > it uses >0x80 as the fast path criteria, it is possible to expose > some sun.text.normalizer's > internal methods to have a "quick nfc" check, but I'm not sure > how much the performance > gain would be, but might worth some investigation later. > > (3) Comments have been added for those override-able methods in > UnixFileSystem. > > (4) blank lines have been removed from dispatcher.c > > (5) I don't think we need to do the HFS check, given we are only doing > nfc/nfd stuff now, as > long as it is a MacOSX, I believe its nfc/nfd layer will be > there. Copyright has been re-copy/ > pasted and we now only use only bugid. > > -Sherman I'm heading away on vacation now and only quickly scanned the updated webrev. All looks okay to me. On the calls to the CF functions then one thing is that if they fail (and this is unlikely I know) then you won't have an exception pending so may need to add code to call one of the JNU* functions to throw OOME, otherwise it might cause a NPE in the caller rather than throwing an exception as you would expect. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120627/11f8776a/attachment.html From Alan.Bateman at oracle.com Tue Jun 26 23:55:54 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 27 Jun 2012 07:55:54 +0100 Subject: FileSystemProvider example.... In-Reply-To: <000701cd5428$aaf83270$00e89750$@comcast.net> References: <033f01cd534a$58b33fc0$0a19bf40$@comcast.net> <000701cd5428$aaf83270$00e89750$@comcast.net> Message-ID: <4FEAAE7A.4040102@oracle.com> Tom - I don't have time to send a detailed reply now but the user.dir property is used by the default provider in order to determine the working directory, it's not the root. The default provider does not support a way to prevent someone "escaping" to other areas of the file system and I think, as per my first reply, that you'll need to create another provider that layers over the default provider to do that. On the URI business then this is because the default provider is a singleton, it's not meant to be instantiated with different URIs. The normal way to get a reference to it is via FileSystems.getDefault(). -Alan. On 27/06/2012 06:49, Tom Lasseter wrote: > > We can very easily use the existing local FileSystemProvider to > generate a new local FileSystem rooted at an arbitrary point in the > code below. > > The somewhat bizarre "trick" here is using the USER_DIR property to > set the root location for the FileSystem ultimately provided: > > /System.setProperty(USER_DIR, partition + rootPathname + args[0]); // > partition: "driveLetter:", rootPathname: root for all FileSystems > created, arg[0]: relative path to the root subdirectory desired/ > > // > > and then creating the new FileSystem: > > /uri = new URI("file", "", "/" , null); // Anything other than "/" is > NOT ALLOWED (see checkURI(uri) method in WindowsFileSystemProvider)/ > > /FileSystemProvider fileSystemProvider = > FileSystemProvider.installedProviders().get(0); // assumption here is > that first one is the "default"/ > > /FileSystem fileSystem = fileSystemProvider.getFileSystem(uri);/// > > The output when you run the code below is: > > *User dir: G:/FileSystems/projectA* > > *File path: \test.txt* > > *Deleted file: \test.txt* > > *Writing...* > > *String written: Hello!* > > *Reading...* > > *String read: Hello!* > > ** > > I've tried everything else, but this is the only way I've found to use > the local FileSystemProvider to generate local FileSystem(s). > > Ideally, you would think you would not have to use this > System.setProperty hack and could provide the root path in the URI > constructor above as: > > /uri = new URI("file", "", "g:/FileSystems/ProjectA" , null);/ > > This doesn't work as the FileSystemProvider insists on having the URI > defined effectively as just "/" (above). You almost might ask: "Why > am I being asked in the constructor to specify the URI if the only one > it will take is "/" ? > > Running on Windows, the WindowsFileSystemProvider is in package > sun.nio.fs. the newFileSystem and getFileSystem methods both call the > /checkURI/ method which must return true for the FileSystem to be > created/fetched: > > /private void checkUri(URI uri) / > > /{/ > > / if (!uri.getScheme().equalsIgnoreCase(getScheme()))/ > > / throw new IllegalArgumentException("URI does not match > this provider");/ > > / if (uri.getAuthority() != null)/ > > / throw new IllegalArgumentException("Authority > component present");/ > > / if (uri.getPath() == null)/ > > / throw new IllegalArgumentException("Path component is > undefined");/ > > / if (!uri.getPath().equals("/"))/ > > / throw new IllegalArgumentException("Path component > should be '/'");/ > > / if (uri.getQuery() != null)/ > > / throw new IllegalArgumentException("Query component > present");/ > > / if (uri.getFragment() != null)/ > > / throw new IllegalArgumentException("Fragment component > present");/ > > /}/ > > // > > So that: /if(!uri.getPath().equals("/")) /you are hosed. Thus you > need to set the root in some other sneaky fashion such as the USER_DIR > trick above. > > Seems like either (1) there is a more flexible constructor; or (2) > there isn't and the URI path argument should be removed from the > constructor, and thus the only allowed URI is "scheme:/" where scheme > must match the provider. > > I'm SURE I must be missing something. I can appreciate that if > FileSystem(s) are created but they are unknown effectively to the > defaultFileSystem, then they can't be created higgily-piggily (arcane > technical term). So by forcing this very restrictive URI, we are > limiting where they actually can be created. Seems like there should > be a somewhat less restrictive capability....which is why I'm sure > that I am REALLY missing something. > > Regards, > > Tom > > Code: > > /public class GcFileSystemProvider/ > > /{/ > > / static public String partition = "G:/";/ > > / static public String rootPathname = "FileSystems/";/ > > // > > / private static final String USER_DIR = "user.dir";/ > > // > > / GcFileSystemProvider()/ > > / {/ > > / }/ > > // > > / static public void main(String[] args)/ > > / {/ > > / System.setProperty(USER_DIR, partition > + rootPathname + args[0]);/ > > / String userDir = > System.getProperty(USER_DIR);/ > > / System.out.println("User dir: " + > userDir);/ > > / URI uri = null;/ > > / Map env = new HashMap<>();/ > > / env.put("create", "true");/ > > / FileSystem fileSystem = null;/ > > / FileSystemProvider fileSystemProvider = > null;/ > > / try/ > > / {/ > > / uri = new URI("file", > "", "/" , null);/ > > / fileSystemProvider = > FileSystemProvider.installedProviders().get(0); // assumption here is > that first one is the "standard"/ > > / fileSystem = > fileSystemProvider.getFileSystem(uri);/ > > / if(fileSystem == null) > fileSystem = fileSystemProvider.newFileSystem(uri, env);/ > > / }/ > > / catch(Exception e)/ > > / {/ > > / e.printStackTrace();/ > > / }/ > > // > > / Path path = > fileSystem.getPath("/test.txt");/ > > / System.out.println("File path: " + > path.toString());/ > > // > > / String text = "Hello!";/ > > / byte[] bytes = text.getBytes();/ > > / try/ > > / {/ > > / > if(Files.deleteIfExists(path))/ > > / > System.out.println("Deleted file: " + path.toString());/ > > // > > / Set > writeOptions = new HashSet();/ > > / > writeOptions.add(StandardOpenOption.CREATE);/ > > / > writeOptions.add(StandardOpenOption.TRUNCATE_EXISTING);/ > > / > writeOptions.add(StandardOpenOption.WRITE);/ > > / Files.createFile(path);/ > > / SeekableByteChannel > seekableByteChannel = fileSystemProvider.newByteChannel(path, > writeOptions);/ > > / ByteBuffer > writeByteBuffer = ByteBuffer.wrap(bytes);/ > > / > System.out.println("Writing...");/ > > / > System.out.println("String written: " + text);/ > > / > seekableByteChannel.write(writeByteBuffer);/ > > / > seekableByteChannel.close();/ > > / }/ > > / catch(Exception e)/ > > / {/ > > / e.printStackTrace();/ > > / }/ > > / System.out.println("Reading...");/ > > // > > / try/ > > / {/ > > / Set > readOptions = new HashSet();/ > > / > readOptions.add(StandardOpenOption.READ);/ > > / SeekableByteChannel > seekableReadByteChannel = fileSystemProvider.newByteChannel(path, > readOptions);/ > > / ByteBuffer > readByteBuffer = ByteBuffer.allocate(100);/ > > / int bytesRead = > seekableReadByteChannel.read(readByteBuffer);/ > > / byte[] readBytes = > readByteBuffer.array();/ > > / String readString = new > String(readBytes);/ > > / > System.out.println("String read: " + readString);/ > > / }/ > > / catch (IOException e)/ > > / {/ > > / e.printStackTrace();/ > > / }/ > > / }/ > > /}/ > > *From:*Tom Lasseter [mailto:t.lasseter at comcast.net] > *Sent:* Monday, June 25, 2012 8:18 PM > *To:* nio-dev at openjdk.java.net > *Subject:* FileSystemProvider example.... > > The new Java 7 FileSystem is very cool....just what I need...but for > the life of me I can't create a custom FileSystemProvider. There must > be an example somewhere! There is only the ZipFileSystemProvider. I > want to create a new FileSystem on either my local system or on a > remote system. > > The simplest case is given the local system as the > DefaultFileSystemProvider, I want a new FileSystem from it rooted at > some arbitrary path, like c:/FileSystems/myFileSystem. What could be > easier than that given the technology being provided. I would think > that this would do it: > > Map env = new HashMap<>(); > > env.put("create", "true"); > > Path path = Paths.get("c:/FileSystems/myFileSystem"); > > FileSystem fileSystem = null; > > try > > { > > fileSystem = fileSystemProvider.newFileSystem(path, env); > > } > > catch(Exception e) > > { > > e.printStackTrace(); > > } > > Help would be appreciated. > > Thanks and regards, > > Tom > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120627/5b7cba9a/attachment-0001.html From xueming.shen at oracle.com Wed Jun 27 00:03:13 2012 From: xueming.shen at oracle.com (Xueming Shen) Date: Wed, 27 Jun 2012 00:03:13 -0700 Subject: Fwd: Codereview request for 7130915: File.equals does not give expected results when path contains Non-English characters on Mac OS X In-Reply-To: <4FEAAB21.9050205@oracle.com> References: <4FE4A4D6.4070100@oracle.com> <4FE94F6F.2060002@oracle.com> <4FE94FED.6050406@oracle.com> <4FE9CEF3.8080905@oracle.com> <4FEA7F1E.4070500@oracle.com> <4FEAAB21.9050205@oracle.com> Message-ID: <4FEAB031.4080107@oracle.com> Thanks Alan! The webrev has been updated to throw OOME as your other nio native dispatcher does. http://cr.openjdk.java.net/~sherman/7130915/webrev. I can wait for your back from the vacation:-) -Sherman On 6/26/12 11:41 PM, Alan Bateman wrote: > On 27/06/2012 04:33, Xueming Shen wrote: >> Alan, >> >> Webrev has been updated accordingly at >> >> http://cr.openjdk.java.net/~sherman/7130915/webrev >> >> with changes >> >> (1) added a CFStringCreateMutable(...) != null check in both io and >> nio native, though it is >> unlikely to fail here because we are passing a NULL and 0 >> length, like new StringBuilder() >> invocation, if it fails the system probably goes very wrong. >> Both FStringAppendCharacters >> and CFStringNormalize are void return type. >> >> (2) The first line of path.toCharArray in normalizeJavaPath is a >> typo, it is supposed to be >> deleted. The implementation only goes toCharArray when it needs >> to go native. Currently >> it uses >0x80 as the fast path criteria, it is possible to >> expose some sun.text.normalizer's >> internal methods to have a "quick nfc" check, but I'm not sure >> how much the performance >> gain would be, but might worth some investigation later. >> >> (3) Comments have been added for those override-able methods in >> UnixFileSystem. >> >> (4) blank lines have been removed from dispatcher.c >> >> (5) I don't think we need to do the HFS check, given we are only >> doing nfc/nfd stuff now, as >> long as it is a MacOSX, I believe its nfc/nfd layer will be >> there. Copyright has been re-copy/ >> pasted and we now only use only bugid. >> >> -Sherman > I'm heading away on vacation now and only quickly scanned the updated > webrev. All looks okay to me. On the calls to the CF functions then > one thing is that if they fail (and this is unlikely I know) then you > won't have an exception pending so may need to add code to call one of > the JNU* functions to throw OOME, otherwise it might cause a NPE in > the caller rather than throwing an exception as you would expect. > > -Alan > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20120627/f44a0a2a/attachment.html From david.holmes at oracle.com Thu Jun 28 18:24:59 2012 From: david.holmes at oracle.com (David Holmes) Date: Fri, 29 Jun 2012 11:24:59 +1000 Subject: RFR 7179383 (was Re: -XX:MaxDirectMemorySize argument parsing) In-Reply-To: <4FECC6FD.5090504@oracle.com> References: <4FECC2B0.5040704@oracle.com> <4FECC6FD.5090504@oracle.com> Message-ID: <4FED03EB.3050901@oracle.com> Chris, The suggested patch removes the ability to set MaxDirectMemorySize to zero, thereby preventing use of direct memory completely. I don't think this has any practical significance (a value of 1 will disable it just as effectively), but it does impact a NIO test and so I want to run this past the NIO folks - now cc'ed. NIO folk: the main issue was that use of intx prevented setting this >2GB on 32-bit. So the proposed change is to use uintx, which means we can no longer use -1 as the "use the default" setting. Zero was chosen instead, but that now prevents setting an actual limit of zero. Note the VM still sets the property to -1 when it sees zero, so the JDK side of this is unchanged. Perhaps the VM could use "uintx_max" instead? There is also an issue of whether this needs to go through CCC and if we need to document this in the release notes (assuming it goes ahead). This is not a supported option and there is nothing I can find that documents the "use the default" setting, other than the source code. Aside: in the process I discovered that the test java/nio/Buffer/LimitDirectMemory.sh actually assumes that -1 is an illegal value. But the script is written incorrectly and so fails for the wrong reason (and the fail equates to success). I've filed 7180649 for this. Thanks, David On 29/06/2012 7:05 AM, David Holmes wrote: > > > Hi Bengt, > > Thanks for taking a detailed look at this contribution. I now have a > large group of reviewers :) > > On 28/06/2012 11:08 PM, Bengt Rutisson wrote: >> I have a question regarding this change. When I look at >> VM.saveAndRemoveProperties(), which parses the >> "sun.nio.MaxDirectMemorySize" option that hotspot sets, it looks to me >> like it is intended to have three cases: >> >> (1) option not set - use default value of 64 MB >> (2) option is set to "-1" - use all available memory for native >> allocation >> (3) option is set to "an actual value" - use the value as a limit for >> native allocation >> >> See: >> http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/b92353a01aa0/src/share/classes/sun/misc/VM.java >> >> >> >> So, it seems to me that before your change it would be possible for a >> user to set -XX:MaxDirectMemorySize=-1 to make the JDK use all available >> memory for native allocations. But after your change it will not be >> possible set this value. > > With the change the user will set -XX:MaxDirectMemorySize=0 on the > command line, which is then used to set the property to -1 and so > instruct use of all available memory. So the only difference in that > regard is what value gets specified on the command-line to get the > default behaviour of "all available memory". > >> On the other hand hotspot always sets the "sun.nio.MaxDirectMemorySize" >> property, which means that case (1) above can't really happen. To me >> this is strange and I sent a question to Alan Bateman and Chris Hegarty >> about that. This means that there is really no need for a user to have >> -XX:MaxDirectMemorySize=-1 on the command line. > > I agree there is no need for the user to ever have to specify -1 as that > is the current default. Case (1) is the 64MB "default" but it isn't > really a default. The comments before it state: > > // The initial value of this field is arbitrary; during JRE initialization > // it will be reset to the value specified on the command line, if any, > // otherwise to Runtime.getRuntime().maxMemory(). > // > private static long directMemory = 64 * 1024 * 1024; > > So there are only two expected cases as hotspot always sets the > property: either to -1 (which says use all available memory), else the > set value. > > The 64MB will only be used if either the property is not set (doesn't > happen for hotspot) or else it was set to a negative value < -1. The > change prevents that second error case from occurring. > > However I have just realized that there is now no way to set the value > to zero. Not that this is useful (no small value is at all > useful/practical) but there may be a test somewhere that checks boundary > values. > >> I think your change is fine since it was already impossible to get to >> (1) and thus the default was in practice (2), which will also be the >> default after your change. Hopefully there are not many customers with >> -XX:MaxDirectMemorySize=-1 on their command line. > > I was concerned about this too. But I could not find anything documented > about the value -1 having special meaning, so no reason for anyone to > use that. It would also be redundant as that is the internal default set > by the VM. Anyone using that will now get an error and realize something > has changed. Which would suggest we may need something in the release > notes. > >> So, to conclude, I think your change is fine, but it looks to me like >> VM.java and jvm.cpp are not really in agreement of what the default >> value for the maximum direct buffer limit should be. > > Thanks again for the detailed review. > > David > ----- > >> Bengt >> >> >> On 2012-06-28 07:42, David Holmes wrote: >>> Can I please get someone else to take a look at this community >>> contribution - _pretty pleeze_ ;-) >>> >>> Thanks, >>> David >>> >>> On 25/06/2012 3:20 PM, David Holmes wrote: >>>> Sorry for the delay on this Chris. I've filed 7179383 and generated a >>>> webrev: >>>> >>>> http://cr.openjdk.java.net/~dholmes/7179383/webrev/ >>>> >>>> Still need an additional runtime reviewer. >>>> >>>> Thanks, >>>> David >>>> >>>> On 11/06/2012 11:18 PM, Chris Dennis wrote: >>>>> On Jun 7, 2012, at 9:41 PM, David Holmes wrote: >>>>> >>>>>> On 8/06/2012 12:20 AM, Alan Bateman wrote: >>>>>>> On 07/06/2012 14:28, Chris Dennis wrote: >>>>>>>> Yes, I'm listed under "Terracotta Inc. (Christopher Dennis)". >>>>>>>> >>>>>>>> There is one additional complication to this in that the >>>>>>>> LimitDirectMemory test in the jdk sources is currently broken. The >>>>>>>> patch below "fixes" the test - but leaves two open questions: >>>>>>>> >>>>>>>> What should the grep be looking for? This is JDK test asserting on >>>>>>>> output generated by Hotspot - that seems a little screwed up to me, >>>>>>>> right? >>>>>>>> >>>>>>>> Chris >>>>>>> it might be nicer to just check the exit code and not depend on the >>>>>>> error message. >>>>>> >>>>>> Agreed. Seems cleaner. >>>>> Okay, I'll prepare a second jdk patch that modifies this test to use >>>>> the exit value of the JVM as the indicator of startup failure. Once we >>>>> have a bug-id for this issue I'll propose the test patch on the >>>>> relevant mailing list referencing the upcoming behavior change and our >>>>> desire for a more hotspot-neutral test assertion. >>>>> >>>>> Alan: What would the correct forest to provide and patch against and >>>>> which mailing list should I post it to? >>>>> >>>>>> >>>>>>> Just on logistics, as hotspot and jdk changes take a >>>>>>> different route into master it means that we'll need to wait until >>>>>>> the >>>>>>> hotspot changes get to jdk8/jdk8 (and probably down to jdk8/tl) >>>>>>> before >>>>>>> pushing a change to the LimitDirectMemory.sh test. >>>>>> >>>>>> Also is the hotspot fix targeted for 8 and 7u, or just 8? >>>>> I'm not sure if this question was intended for me, but as far as I'm >>>>> aware currently this change doesn't even have a bug-id. Personally, I >>>>> don't see the pressing need to have it merged back to 7u, and not >>>>> doing so would help mitigate the backwards-compatibility issue of the >>>>> subtle changes it makes in the behavior of the switch. >>>>> >>>>> >>>>>> >>>>>> Still need additional reviewer from runtime - thanks. >>>>>> >>>>>> David >>>>>> >>>>>>> -Alan >>>>> >> >> From david.holmes at oracle.com Thu Jun 28 22:44:10 2012 From: david.holmes at oracle.com (David Holmes) Date: Fri, 29 Jun 2012 15:44:10 +1000 Subject: RFR 7179383 (was Re: -XX:MaxDirectMemorySize argument parsing) In-Reply-To: References: <4FECC2B0.5040704@oracle.com> <4FECC6FD.5090504@oracle.com> <4FED03EB.3050901@oracle.com> Message-ID: <4FED40AA.1010306@oracle.com> Hi Chris, On 29/06/2012 1:27 PM, Chris Dennis wrote: > Hi All, > > I'm not sure this analysis is correct. If I understand how this all works correctly then the default value specified for MaxDirectMemorySize in globals.hpp is really not important. The command line parsing understands both the origin of a flag and it's value. "FLAG_IS_DEFAULT(MaxDirectMemorySize)" doesn't translate to "is the current value of the MaxDirectMemorySize flag the same as it's default value" but instead to "is the origin of the current value of the MaxDirectMemorySize flag not the default" (the other options being command-line or ergonomic). I ran the following test class: You are right, I was misinterpreting what FLAG_IS_DEFAULT meant. So all is well with the fix and we have a plethora of reviewers. I'd also forgotten about the earlier discussion regarding the test, but at least now there is a CR for it. :) I'm just doing a fresh test build as it turned out my testing the other day didn't include the fix. I'm hoping the bugs in the test program mean the new functionality will still "pass" on the broken test. Thanks, David ----- > public class TestMaxDirectMemory { > public static void main(String[] args) { > System.out.println(sun.misc.VM.maxDirectMemory()); > } > } > > on a JVM with and without my changes: > > java -Xmx128m TestMaxDirectMemory > current: 119341056 > patched: 119341056 > java -XX:MaxDirectMemorySize=-2 -Xmx128m TestMaxDirectMemory > current: 67108864 > patched: Invalid maximum direct memory size: -XX:MaxDirectMemorySize=-2 > java -XX:MaxDirectMemorySize=-1 -Xmx128m TestMaxDirectMemory > current: 119341056 > patched: Invalid maximum direct memory size: -XX:MaxDirectMemorySize=-1 > java -XX:MaxDirectMemorySize=0 -Xmx128m TestMaxDirectMemory > current: 0 > patched: 0 > java -XX:MaxDirectMemorySize=1 -Xmx128m TestMaxDirectMemory > current: 1 > patched: 1 > java -XX:MaxDirectMemorySize=96m -Xmx128m TestMaxDirectMemory > current: 100663296 > patched: 100663296 > > Everything seems to be working fine as far as I can tell. The only changes in behavior that might be concerning for someone here are the changes for negative values, you can still specify zero if you so wish (although as David points out this would be a pretty strange thing to do). > > As for the broken test, back when we originally discussed it I think Alan Bateman suggested that we change the test for this to assert not on the textual output of the failing process but instead on the return value (since a JDK test asserting on Hotspot generated output seemed wrong to everyone involved). I actually have the changes to fix the test prepped and ready to submit but I imagine it will be simpler for us to wait until this patch has been integrated in to the JDK8 forest before submitting the test fix to avoid causing confusion (or test failures) in the JDK8 forest if the fix were to get merged too soon. > > Thanks, > > Chris > > On Jun 28, 2012, at 9:24 PM, David Holmes wrote: > >> Chris, >> >> The suggested patch removes the ability to set MaxDirectMemorySize to zero, thereby preventing use of direct memory completely. I don't think this has any practical significance (a value of 1 will disable it just as effectively), but it does impact a NIO test and so I want to run this past the NIO folks - now cc'ed. >> >> NIO folk: the main issue was that use of intx prevented setting this>2GB on 32-bit. So the proposed change is to use uintx, which means we can no longer use -1 as the "use the default" setting. Zero was chosen instead, but that now prevents setting an actual limit of zero. Note the VM still sets the property to -1 when it sees zero, so the JDK side of this is unchanged. Perhaps the VM could use "uintx_max" instead? >> >> There is also an issue of whether this needs to go through CCC and if we need to document this in the release notes (assuming it goes ahead). This is not a supported option and there is nothing I can find that documents the "use the default" setting, other than the source code. >> >> Aside: in the process I discovered that the test java/nio/Buffer/LimitDirectMemory.sh actually assumes that -1 is an illegal value. But the script is written incorrectly and so fails for the wrong reason (and the fail equates to success). I've filed 7180649 for this. >> >> Thanks, >> David >> >> On 29/06/2012 7:05 AM, David Holmes wrote: >>> >>> >>> Hi Bengt, >>> >>> Thanks for taking a detailed look at this contribution. I now have a >>> large group of reviewers :) >>> >>> On 28/06/2012 11:08 PM, Bengt Rutisson wrote: >>>> I have a question regarding this change. When I look at >>>> VM.saveAndRemoveProperties(), which parses the >>>> "sun.nio.MaxDirectMemorySize" option that hotspot sets, it looks to me >>>> like it is intended to have three cases: >>>> >>>> (1) option not set - use default value of 64 MB >>>> (2) option is set to "-1" - use all available memory for native >>>> allocation >>>> (3) option is set to "an actual value" - use the value as a limit for >>>> native allocation >>>> >>>> See: >>>> http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/b92353a01aa0/src/share/classes/sun/misc/VM.java >>>> >>>> >>>> >>>> So, it seems to me that before your change it would be possible for a >>>> user to set -XX:MaxDirectMemorySize=-1 to make the JDK use all available >>>> memory for native allocations. But after your change it will not be >>>> possible set this value. >>> >>> With the change the user will set -XX:MaxDirectMemorySize=0 on the >>> command line, which is then used to set the property to -1 and so >>> instruct use of all available memory. So the only difference in that >>> regard is what value gets specified on the command-line to get the >>> default behaviour of "all available memory". >>> >>>> On the other hand hotspot always sets the "sun.nio.MaxDirectMemorySize" >>>> property, which means that case (1) above can't really happen. To me >>>> this is strange and I sent a question to Alan Bateman and Chris Hegarty >>>> about that. This means that there is really no need for a user to have >>>> -XX:MaxDirectMemorySize=-1 on the command line. >>> >>> I agree there is no need for the user to ever have to specify -1 as that >>> is the current default. Case (1) is the 64MB "default" but it isn't >>> really a default. The comments before it state: >>> >>> // The initial value of this field is arbitrary; during JRE initialization >>> // it will be reset to the value specified on the command line, if any, >>> // otherwise to Runtime.getRuntime().maxMemory(). >>> // >>> private static long directMemory = 64 * 1024 * 1024; >>> >>> So there are only two expected cases as hotspot always sets the >>> property: either to -1 (which says use all available memory), else the >>> set value. >>> >>> The 64MB will only be used if either the property is not set (doesn't >>> happen for hotspot) or else it was set to a negative value< -1. The >>> change prevents that second error case from occurring. >>> >>> However I have just realized that there is now no way to set the value >>> to zero. Not that this is useful (no small value is at all >>> useful/practical) but there may be a test somewhere that checks boundary >>> values. >>> >>>> I think your change is fine since it was already impossible to get to >>>> (1) and thus the default was in practice (2), which will also be the >>>> default after your change. Hopefully there are not many customers with >>>> -XX:MaxDirectMemorySize=-1 on their command line. >>> >>> I was concerned about this too. But I could not find anything documented >>> about the value -1 having special meaning, so no reason for anyone to >>> use that. It would also be redundant as that is the internal default set >>> by the VM. Anyone using that will now get an error and realize something >>> has changed. Which would suggest we may need something in the release >>> notes. >>> >>>> So, to conclude, I think your change is fine, but it looks to me like >>>> VM.java and jvm.cpp are not really in agreement of what the default >>>> value for the maximum direct buffer limit should be. >>> >>> Thanks again for the detailed review. >>> >>> David >>> ----- >>> >>>> Bengt >>>> >>>> >>>> On 2012-06-28 07:42, David Holmes wrote: >>>>> Can I please get someone else to take a look at this community >>>>> contribution - _pretty pleeze_ ;-) >>>>> >>>>> Thanks, >>>>> David >>>>> >>>>> On 25/06/2012 3:20 PM, David Holmes wrote: >>>>>> Sorry for the delay on this Chris. I've filed 7179383 and generated a >>>>>> webrev: >>>>>> >>>>>> http://cr.openjdk.java.net/~dholmes/7179383/webrev/ >>>>>> >>>>>> Still need an additional runtime reviewer. >>>>>> >>>>>> Thanks, >>>>>> David >>>>>> >>>>>> On 11/06/2012 11:18 PM, Chris Dennis wrote: >>>>>>> On Jun 7, 2012, at 9:41 PM, David Holmes wrote: >>>>>>> >>>>>>>> On 8/06/2012 12:20 AM, Alan Bateman wrote: >>>>>>>>> On 07/06/2012 14:28, Chris Dennis wrote: >>>>>>>>>> Yes, I'm listed under "Terracotta Inc. (Christopher Dennis)". >>>>>>>>>> >>>>>>>>>> There is one additional complication to this in that the >>>>>>>>>> LimitDirectMemory test in the jdk sources is currently broken. The >>>>>>>>>> patch below "fixes" the test - but leaves two open questions: >>>>>>>>>> >>>>>>>>>> What should the grep be looking for? This is JDK test asserting on >>>>>>>>>> output generated by Hotspot - that seems a little screwed up to me, >>>>>>>>>> right? >>>>>>>>>> >>>>>>>>>> Chris >>>>>>>>> it might be nicer to just check the exit code and not depend on the >>>>>>>>> error message. >>>>>>>> >>>>>>>> Agreed. Seems cleaner. >>>>>>> Okay, I'll prepare a second jdk patch that modifies this test to use >>>>>>> the exit value of the JVM as the indicator of startup failure. Once we >>>>>>> have a bug-id for this issue I'll propose the test patch on the >>>>>>> relevant mailing list referencing the upcoming behavior change and our >>>>>>> desire for a more hotspot-neutral test assertion. >>>>>>> >>>>>>> Alan: What would the correct forest to provide and patch against and >>>>>>> which mailing list should I post it to? >>>>>>> >>>>>>>> >>>>>>>>> Just on logistics, as hotspot and jdk changes take a >>>>>>>>> different route into master it means that we'll need to wait until >>>>>>>>> the >>>>>>>>> hotspot changes get to jdk8/jdk8 (and probably down to jdk8/tl) >>>>>>>>> before >>>>>>>>> pushing a change to the LimitDirectMemory.sh test. >>>>>>>> >>>>>>>> Also is the hotspot fix targeted for 8 and 7u, or just 8? >>>>>>> I'm not sure if this question was intended for me, but as far as I'm >>>>>>> aware currently this change doesn't even have a bug-id. Personally, I >>>>>>> don't see the pressing need to have it merged back to 7u, and not >>>>>>> doing so would help mitigate the backwards-compatibility issue of the >>>>>>> subtle changes it makes in the behavior of the switch. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Still need additional reviewer from runtime - thanks. >>>>>>>> >>>>>>>> David >>>>>>>> >>>>>>>>> -Alan >>>>>>> >>>> >>>> >