From Alan.Bateman at oracle.com Wed Sep 1 01:38:32 2010 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 01 Sep 2010 09:38:32 +0100 Subject: Small improvements to nio native code (unix) In-Reply-To: References: <4C7CCBB4.9050208@oracle.com> <4C7D681A.6040800@oracle.com> Message-ID: <4C7E1108.8010905@oracle.com> Martin Buchholz wrote: > > : > > > > Done. I appreciate the attention to detail, even if this bug would > never occur in the wild. The updated webrev looks good to me. -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100901/bc436ad4/attachment.html From fudmail at gmail.com Wed Sep 1 07:31:50 2010 From: fudmail at gmail.com (f m) Date: Wed, 1 Sep 2010 07:31:50 -0700 Subject: WatchService - Exposing more of the Inotify Event Model Message-ID: Per Alan B.'s request, I'm moving a discussion here that was started at http://forums.java.net/jive/thread.jspa?threadID=152825&tstart=0 I'll try to summarize. A useful goal of the JDK7 WatchService is to provide a uniform file system event facility across multiple operating systems. However, in some cases, it may be more desirable to give up the portability in order to tap into the full expressiveness of the event model of the native OS. For example, Inotify on the Linux platform provides additional events not currently used by the LinuxWatchService implementation such as: IN_ACCESS, IN_CLOSE_WRITE, IN_CLOSE_NO_WRITE, and IN_OPEN. I gave one example at the above URI where an event like IN_CLOSE_WRITE could be used to detect and process a large file only after it has been completely copied into a watch directory. After looking over the internal LinuxWatchService.java source code, it appears that the implementation could be readily used (modified or resused?) to also provide access to these additional event types. Another event class representation like LinuxWatchEventKinds would need to be added and used alternatively to the StandardWatchEventKinds. My question. Would extending the LinuxWatchService capability to optionally expose the greater set of Inotify event types be in scope for JDK 7 development, or will client developers who desire this capability need to find 3rd party implementations (or develop it themselves)? Regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100901/6d2b78ba/attachment.html From Alan.Bateman at oracle.com Thu Sep 2 08:00:03 2010 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 02 Sep 2010 16:00:03 +0100 Subject: WatchService - Exposing more of the Inotify Event Model In-Reply-To: References: Message-ID: <4C7FBBF3.7020501@oracle.com> f m wrote: > Per Alan B.'s request, I'm moving a discussion here that was started at > http://forums.java.net/jive/thread.jspa?threadID=152825&tstart=0 > > > I'll try to summarize. > > A useful goal of the JDK7 WatchService is to provide a uniform file > system event facility across multiple operating systems. However, in > some cases, it may be more desirable to give up the portability in > order to tap into the full expressiveness of the event model of the > native OS. For example, Inotify on the Linux platform provides > additional events not currently used by the LinuxWatchService > implementation such as: IN_ACCESS, IN_CLOSE_WRITE, IN_CLOSE_NO_WRITE, > and IN_OPEN. > > I gave one example at the above URI where an event like IN_CLOSE_WRITE > could be used to detect and process a large file only after it has > been completely copied into a watch directory. > > After looking over the internal LinuxWatchService.java source code, it > appears that the implementation could be readily used (modified or > resused?) to also provide access to these additional event types. > Another event class representation like LinuxWatchEventKinds would > need to be added and used alternatively to the StandardWatchEventKinds. > > My question. Would extending the LinuxWatchService capability to > optionally expose the greater set of Inotify event types be in scope > for JDK 7 development, or will client developers who desire this > capability need to find 3rd party implementations (or develop it > themselves)? > > Regards. > The API is designed to allow for events and modifiers that are defined beyond the javadoc so provider implementations can include support for other events if they wish. We haven't included any support for other events in the JDK because our primary motive has been to help the performance of applications that are forced to poll the file system today. If you are looking for pointers then there is one implementation specific modifier (com.nio.file.ExtendedWatchEvent) that might help show you how additional support is included. You are welcome to hack on the code and try out additional events but you have to be a contributer to contribute patches [1]. A small comment on the "close after write" event is that I don't think you are guaranteed that the file isn't being modified by other entities. That might not be a concern in your environment but just mentioning it. -Alan. [1] http://openjdk.java.net/contribute/ From forax at univ-mlv.fr Thu Sep 2 09:03:14 2010 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Thu, 02 Sep 2010 18:03:14 +0200 Subject: WatchService - Exposing more of the Inotify Event Model In-Reply-To: <4C7FBBF3.7020501@oracle.com> References: <4C7FBBF3.7020501@oracle.com> Message-ID: <4C7FCAC2.6010703@univ-mlv.fr> Le 02/09/2010 17:00, Alan Bateman a ?crit : > f m wrote: >> Per Alan B.'s request, I'm moving a discussion here that was started at >> http://forums.java.net/jive/thread.jspa?threadID=152825&tstart=0 >> >> >> I'll try to summarize. >> >> A useful goal of the JDK7 WatchService is to provide a uniform file >> system event facility across multiple operating systems. However, in >> some cases, it may be more desirable to give up the portability in >> order to tap into the full expressiveness of the event model of the >> native OS. For example, Inotify on the Linux platform provides >> additional events not currently used by the LinuxWatchService >> implementation such as: IN_ACCESS, IN_CLOSE_WRITE, IN_CLOSE_NO_WRITE, >> and IN_OPEN. >> >> I gave one example at the above URI where an event like >> IN_CLOSE_WRITE could be used to detect and process a large file only >> after it has been completely copied into a watch directory. >> >> After looking over the internal LinuxWatchService.java source code, >> it appears that the implementation could be readily used (modified or >> resused?) to also provide access to these additional event types. >> Another event class representation like LinuxWatchEventKinds would >> need to be added and used alternatively to the StandardWatchEventKinds. >> >> My question. Would extending the LinuxWatchService capability to >> optionally expose the greater set of Inotify event types be in scope >> for JDK 7 development, or will client developers who desire this >> capability need to find 3rd party implementations (or develop it >> themselves)? >> >> Regards. >> > The API is designed to allow for events and modifiers that are defined > beyond the javadoc so provider implementations can include support for > other events if they wish. We haven't included any support for other > events in the JDK because our primary motive has been to help the > performance of applications that are forced to poll the file system > today. If you are looking for pointers then there is one > implementation specific modifier (com.nio.file.ExtendedWatchEvent) > that might help show you how additional support is included. You are > welcome to hack on the code and try out additional events but you have > to be a contributer to contribute patches [1]. A small comment on the > "close after write" event is that I don't think you are guaranteed > that the file isn't being modified by other entities. That might not > be a concern in your environment but just mentioning it. Alan, yes, you're right, see inotify FAQ: http://inotify.aiken.cz/?section=inotify&page=faq&lang=en > > -Alan. > > [1] http://openjdk.java.net/contribute/ R?mi From fudmail at gmail.com Thu Sep 2 10:59:54 2010 From: fudmail at gmail.com (fm) Date: Thu, 2 Sep 2010 10:59:54 -0700 Subject: WatchService - Exposing more of the Inotify Event Model In-Reply-To: <4C7FBBF3.7020501@oracle.com> References: <4C7FBBF3.7020501@oracle.com> Message-ID: Thank you for the response. This is the type of info I was looking for. Reading the WatchService related API documentation I got the sense it was designed for extension. After skimming over some of the related source code, it wasn't yet clear to me the best way to go about accessing the additional Linux specific filesystem events. After looking at the JDK7 source code it was clear to me a lot of hard work went into the design of the WatchService notification capability and it would be a shame not to be able to leverage it for accessing additional event tyeps. I will look into you suggestions. I am also aware that Inotify's IN_CLOSE_WRITE event is not the silver bullet to my particular project--but as the link R?mi provided points out, it may be more suitable that IN_MODIFY (in my case as well as others). But I believe you are correct also that one cannot assume something else does not have a file open for writing...it is just much more unlikely in my particular case. Regards On Thu, Sep 2, 2010 at 8:00 AM, Alan Bateman wrote: > f m wrote: >> >> Per Alan B.'s request, I'm moving a discussion here that was started at >> http://forums.java.net/jive/thread.jspa?threadID=152825&tstart=0 >> >> >> I'll try to summarize. >> >> A useful goal of the JDK7 WatchService is to provide a uniform file system >> event facility across multiple operating systems. However, in some cases, it >> may be more desirable to give up the portability in order to tap into the >> full expressiveness of the event model of the native OS. For example, >> Inotify on the Linux platform provides additional events not currently used >> by the LinuxWatchService implementation such as: IN_ACCESS, IN_CLOSE_WRITE, >> IN_CLOSE_NO_WRITE, and IN_OPEN. >> >> I gave one example at the above URI where an event like IN_CLOSE_WRITE >> could be used to detect and process a large file only after it has been >> completely copied into a watch directory. >> >> After looking over the internal LinuxWatchService.java source code, it >> appears that the implementation could be readily used (modified or resused?) >> to also provide access to these additional event types. Another event class >> representation like LinuxWatchEventKinds would need to be added and used >> alternatively to the StandardWatchEventKinds. >> >> My question. Would extending the LinuxWatchService capability to >> optionally expose the greater set of Inotify event types be in scope for JDK >> 7 development, or will client developers who desire this capability need to >> find 3rd party implementations (or develop it themselves)? >> >> Regards. >> > The API is designed to allow for events and modifiers that are defined > beyond the javadoc so provider implementations can include support for other > events if they wish. We haven't included any support for other events in the > JDK because our primary motive has been to help the performance of > applications that are forced to poll the file system today. If you are > looking for pointers then there is one implementation specific modifier > (com.nio.file.ExtendedWatchEvent) ?that might help show you how additional > support is included. You are welcome to hack on the code and try out > additional events but you have to be a contributer to contribute patches > [1]. ?A small comment on the "close after write" event is that I don't think > you are guaranteed that the file isn't being modified by other entities. > That might not be a concern in your environment but just mentioning it. > > -Alan. > > [1] http://openjdk.java.net/contribute/ > From Alan.Bateman at oracle.com Fri Sep 10 03:03:01 2010 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 10 Sep 2010 11:03:01 +0100 Subject: 6881498: (file) Re-examine DirectoryStream exception handling Message-ID: <4C8A0255.3050705@oracle.com> The exception handling in java.nio.file.DirectoryStream is currently painful, at least if you want to recover from I/O errors. Joe, Doug, and I have been talking about improving this. The changes here introduce a more specific exception, DirectoryIteratorException, that is thrown if an I/O error is encountered. That's a bit more obvious than catching ConcurrentModificationException and looking at the cause to distinguish I/O errors from other causes. The closed stream case is changed so that the iterator behaves as if end of stream has been reached. The remove operation was originally specified to attempt to delete the directory. That was probably a bad idea and is now changed to be unsupported. The webrev with the changes is here: http://cr.openjdk.java.net/~alanb/6881498/webrev/ Thanks, Alan. From Alan.Bateman at oracle.com Fri Sep 10 03:17:08 2010 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 10 Sep 2010 11:17:08 +0100 Subject: 6983794: TEST_BUG: test/java/nio/channels/Selector/ConnectWrite.java failing Message-ID: <4C8A05A4.50301@oracle.com> This test connects to the daytime and discard service on the configured test host. It's the only test we have that connects to the discard port (to my knowledge anyway). I'd like to remove that dependency with the following patch. It will continue to test by connecting to the daytime service. Thanks, Alan. diff -r cecc431cd78a test/java/nio/channels/Selector/ConnectWrite.java --- a/test/java/nio/channels/Selector/ConnectWrite.java Tue Sep 07 08:36:10 2010 +0100 +++ b/test/java/nio/channels/Selector/ConnectWrite.java Fri Sep 10 11:15:30 2010 +0100 @@ -38,7 +38,6 @@ public class ConnectWrite { public static void main(String[] args) throws Exception { test1(13); - test1(9); } public static void test1(int port) throws Exception { From forax at univ-mlv.fr Fri Sep 10 03:23:48 2010 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Fri, 10 Sep 2010 12:23:48 +0200 Subject: 6881498: (file) Re-examine DirectoryStream exception handling In-Reply-To: <4C8A0255.3050705@oracle.com> References: <4C8A0255.3050705@oracle.com> Message-ID: <4C8A0734.7050603@univ-mlv.fr> Le 10/09/2010 12:03, Alan Bateman a ?crit : > > The exception handling in java.nio.file.DirectoryStream is currently > painful, at least if you want to recover from I/O errors. Joe, Doug, > and I have been talking about improving this. The changes here > introduce a more specific exception, DirectoryIteratorException, that > is thrown if an I/O error is encountered. That's a bit more obvious > than catching ConcurrentModificationException and looking at the cause > to distinguish I/O errors from other causes. The closed stream case is > changed so that the iterator behaves as if end of stream has been > reached. The remove operation was originally specified to attempt to > delete the directory. That was probably a bad idea and is now changed > to be unsupported. > > The webrev with the changes is here: > http://cr.openjdk.java.net/~alanb/6881498/webrev/ > > Thanks, > Alan. Looks good. Only one comment, DirectoryIteratorException should have a constructor that takes an IOException and a message. Some implementations may want to provide a specific message. R?mi From chris.hegarty at oracle.com Fri Sep 10 03:30:32 2010 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Fri, 10 Sep 2010 11:30:32 +0100 Subject: 6983794: TEST_BUG: test/java/nio/channels/Selector/ConnectWrite.java failing In-Reply-To: <4C8A05A4.50301@oracle.com> References: <4C8A05A4.50301@oracle.com> Message-ID: <4C8A08C8.9040809@oracle.com> I have no problem with this. -Chris. On 10/09/2010 11:17, Alan Bateman wrote: > > This test connects to the daytime and discard service on the configured > test host. It's the only test we have that connects to the discard port > (to my knowledge anyway). I'd like to remove that dependency with the > following patch. It will continue to test by connecting to the daytime > service. > > Thanks, > Alan. > > > diff -r cecc431cd78a test/java/nio/channels/Selector/ConnectWrite.java > --- a/test/java/nio/channels/Selector/ConnectWrite.java Tue Sep 07 > 08:36:10 2010 +0100 > +++ b/test/java/nio/channels/Selector/ConnectWrite.java Fri Sep 10 > 11:15:30 2010 +0100 > @@ -38,7 +38,6 @@ public class ConnectWrite { > > public static void main(String[] args) throws Exception { > test1(13); > - test1(9); > } > > public static void test1(int port) throws Exception { > > From Alan.Bateman at oracle.com Fri Sep 10 07:22:42 2010 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 10 Sep 2010 15:22:42 +0100 Subject: 6907737: (file) FileVisitor and Files.walkFileTree issues Message-ID: <4C8A3F32.8010200@oracle.com> A few weeks ago I mentioned that we needed to fix a several issues with Files.walkFileTree/FileVisitor. The list is here: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6907737 I'd like to fix these as a batch to avoid too much disruption. The webrev with the proposed changes is here: http://cr.openjdk.java.net/~alanb/6907737/webrev/ In addition to this, there is also a one-line change in the langtools repo (to add the new parameter to preVisitDirectory). I'll push that at the same time. Thanks, -Alan. From Alan.Bateman at oracle.com Fri Sep 10 08:29:47 2010 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 10 Sep 2010 16:29:47 +0100 Subject: 6881498: (file) Re-examine DirectoryStream exception handling In-Reply-To: <4C8A0734.7050603@univ-mlv.fr> References: <4C8A0255.3050705@oracle.com> <4C8A0734.7050603@univ-mlv.fr> Message-ID: <4C8A4EEB.2020206@oracle.com> R?mi Forax wrote: > : > Looks good. > Only one comment, DirectoryIteratorException should have a constructor > that > takes an IOException and a message. > Some implementations may want to provide a specific message. > > R?mi Thanks for reviewing this. I didn't include a second constructor as this is a very specific exception and the detail is in the required cause. Maybe for consistency it should be added. If it's okay with you, I'll push what we have now and create a bug to track thinking about this one. -Alan. From forax at univ-mlv.fr Fri Sep 10 08:40:05 2010 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Fri, 10 Sep 2010 17:40:05 +0200 Subject: 6881498: (file) Re-examine DirectoryStream exception handling In-Reply-To: <4C8A4EEB.2020206@oracle.com> References: <4C8A0255.3050705@oracle.com> <4C8A0734.7050603@univ-mlv.fr> <4C8A4EEB.2020206@oracle.com> Message-ID: <4C8A5155.4010801@univ-mlv.fr> Le 10/09/2010 17:29, Alan Bateman a ?crit : > R?mi Forax wrote: >> : >> Looks good. >> Only one comment, DirectoryIteratorException should have a >> constructor that >> takes an IOException and a message. >> Some implementations may want to provide a specific message. >> >> R?mi > Thanks for reviewing this. I didn't include a second constructor as > this is a very specific exception and the detail is in the required > cause. Maybe for consistency it should be added. If it's okay with > you, I'll push what we have now and create a bug to track thinking > about this one. Okay > > -Alan. R?mi From Alan.Bateman at oracle.com Tue Sep 14 15:45:15 2010 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 14 Sep 2010 15:45:15 -0700 Subject: 6984545: (fc) transferFrom does not throw NonReadableChannelException when target is size 0 and non-readable Message-ID: <4C8FFAFB.7010102@oracle.com> The changes for 6431344 introduced a small regression. If FileChannel's transferFrom is invoked with a source that is non-readable FileChannel then NonReadableChannelException is not thrown when there are no bytes to transfer. The fix is to trivially check if the source is readable: http://cr.openjdk.java.net/~alanb/6984545/webrev/ Thanks, Alan. [1] http://hg.openjdk.java.net/jdk7/jdk7/jdk/rev/f4a2b4e7a831 From forax at univ-mlv.fr Tue Sep 14 16:15:47 2010 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Wed, 15 Sep 2010 01:15:47 +0200 Subject: 6984545: (fc) transferFrom does not throw NonReadableChannelException when target is size 0 and non-readable In-Reply-To: <4C8FFAFB.7010102@oracle.com> References: <4C8FFAFB.7010102@oracle.com> Message-ID: <4C900223.6020309@univ-mlv.fr> Le 15/09/2010 00:45, Alan Bateman a ?crit : > > The changes for 6431344 introduced a small regression. If > FileChannel's transferFrom is invoked with a source that is > non-readable FileChannel then NonReadableChannelException is not > thrown when there are no bytes to transfer. The fix is to trivially > check if the source is readable: > http://cr.openjdk.java.net/~alanb/6984545/webrev/ > > Thanks, > Alan. > > [1] http://hg.openjdk.java.net/jdk7/jdk7/jdk/rev/f4a2b4e7a831 what if transferFromArbitraryChannel is used instead of transferFromFileChannel ? R?mi PS: I hope this sentence is correct in English From Alan.Bateman at oracle.com Tue Sep 14 20:38:04 2010 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 14 Sep 2010 20:38:04 -0700 Subject: 6984545: (fc) transferFrom does not throw NonReadableChannelException when target is size 0 and non-readable In-Reply-To: <4C900223.6020309@univ-mlv.fr> References: <4C8FFAFB.7010102@oracle.com> <4C900223.6020309@univ-mlv.fr> Message-ID: <4C903F9C.9050703@oracle.com> R?mi Forax wrote: > : > what if transferFromArbitraryChannel is used instead of > transferFromFileChannel ? > > R?mi > PS: I hope this sentence is correct in English Your English is fine and it's a good question. In the arbitrary channel (not a FileChannel) case then the transferFrom/transferTo methods would need to attempt a read or write for the count == 0 case as they can't tell from the channel type. The long standing behavior has been to do nothing and so would require a bit of consideration before changing it. In implementation terms this would mean reading with a buffer that has 0 bytes remaining. I'll create a bug as a reminder. It may be that the spec needs to be clarified for this corner case. For now, I'd like to restore the behavior for the FileChannel case as it is impacted by the zero length file case. -Alan. From forax at univ-mlv.fr Wed Sep 15 01:58:27 2010 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Wed, 15 Sep 2010 10:58:27 +0200 Subject: 6984545: (fc) transferFrom does not throw NonReadableChannelException when target is size 0 and non-readable In-Reply-To: <4C903F9C.9050703@oracle.com> References: <4C8FFAFB.7010102@oracle.com> <4C900223.6020309@univ-mlv.fr> <4C903F9C.9050703@oracle.com> Message-ID: <4C908AB3.1020202@univ-mlv.fr> Le 15/09/2010 05:38, Alan Bateman a ?crit : > R?mi Forax wrote: >> : >> what if transferFromArbitraryChannel is used instead of >> transferFromFileChannel ? >> >> R?mi >> PS: I hope this sentence is correct in English > Your English is fine and it's a good question. In the arbitrary > channel (not a FileChannel) case then the transferFrom/transferTo > methods would need to attempt a read or write for the count == 0 case > as they can't tell from the channel type. The long standing behavior > has been to do nothing and so would require a bit of consideration > before changing it. In implementation terms this would mean reading > with a buffer that has 0 bytes remaining. I'll create a bug as a > reminder. It may be that the spec needs to be clarified for this > corner case. For now, I'd like to restore the behavior for the > FileChannel case as it is impacted by the zero length file case. > > -Alan. > Ok, fix for 6984545 is good. R?mi From Alan.Bateman at oracle.com Wed Sep 15 05:53:59 2010 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 15 Sep 2010 05:53:59 -0700 Subject: 6984545: (fc) transferFrom does not throw NonReadableChannelException when target is size 0 and non-readable In-Reply-To: <4C908AB3.1020202@univ-mlv.fr> References: <4C8FFAFB.7010102@oracle.com> <4C900223.6020309@univ-mlv.fr> <4C903F9C.9050703@oracle.com> <4C908AB3.1020202@univ-mlv.fr> Message-ID: <4C90C1E7.2040703@oracle.com> R?mi Forax wrote: > : > Ok, > fix for 6984545 is good. Thanks, and thanks for bringing up the issue or other channel types when count==0. Although it has always been there then the possibility is likely to increase with jdk7 because of custom SeekableByteChannel implementations. -Alan From xueming.shen at oracle.com Wed Sep 15 11:52:39 2010 From: xueming.shen at oracle.com (Xueming Shen) Date: Wed, 15 Sep 2010 11:52:39 -0700 Subject: nio-dev Digest, Vol 30, Issue 3 In-Reply-To: References: Message-ID: <4C9115F7.6000602@oracle.com> Looks fine. -Sherman > Date: Fri, 10 Sep 2010 15:22:42 +0100 > From: Alan Bateman > Subject: 6907737: (file) FileVisitor and Files.walkFileTree issues > To: nio-dev > Message-ID:<4C8A3F32.8010200 at oracle.com> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > > A few weeks ago I mentioned that we needed to fix a several issues with > Files.walkFileTree/FileVisitor. The list is here: > http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6907737 > > I'd like to fix these as a batch to avoid too much disruption. The > webrev with the proposed changes is here: > http://cr.openjdk.java.net/~alanb/6907737/webrev/ > > In addition to this, there is also a one-line change in the langtools > repo (to add the new parameter to preVisitDirectory). I'll push that at > the same time. > > Thanks, > > -Alan. > From fudmail at gmail.com Wed Sep 22 23:41:51 2010 From: fudmail at gmail.com (fm) Date: Wed, 22 Sep 2010 23:41:51 -0700 Subject: WatchService - Exposing more of the Inotify Event Model In-Reply-To: References: <4C7FBBF3.7020501@oracle.com> Message-ID: I've modified a few of the WatchService implementation classes to allow for the registration of additional WatchEvent.Kind instances that represent the native events of the underlying OS platform. I believe the changes add flexibility to the WatchService API, with minimal disruptive code changes to the existing implementation. There should be a negligible impact on the existing performance. I have altered the implementation of the LinuxWatchService to support the additional event model; however, the SolarisWatchService and WindowsWatchService implementations are effectively unchanged and should work the same as before. Similar modifications could be made to these WatchService implementations to also expose their native event models. I am completely open to hearing comments on if this type of added flexibility to the WatchService API is appropriate for the JDK 7 or better left to an ex ternal project. I will gladly provide the source code files for evaluation if there is interest. Modified files are AbstractWatchKey.java, AbstractPoller.java, LinuxWatchService.java, and one new file LinuxWatchEventKind.java. Regards On Thu, Sep 2, 2010 at 10:59 AM, fm wrote: > Thank you for the response. This is the type of info I was looking > for. Reading the WatchService related API documentation I got the > sense it was designed for extension. After skimming over some of the > related source code, it wasn't yet clear to me the best way to go > about accessing the additional Linux specific filesystem events. After > looking at the JDK7 source code it was clear to me a lot of hard work > went into the design of the WatchService notification capability and > it would be a shame not to be able to leverage it for accessing > additional event tyeps. ?I will look into you suggestions. > > I am also aware that Inotify's IN_CLOSE_WRITE event is not the silver > bullet to my particular project--but as the link R?mi provided points > out, it may be more suitable that IN_MODIFY (in my case as well as > others). But I believe you are correct also that one cannot assume > something else does not have a file open for writing...it is just much > more unlikely in my particular case. > > Regards > > > On Thu, Sep 2, 2010 at 8:00 AM, Alan Bateman wrote: >> f m wrote: >>> >>> Per Alan B.'s request, I'm moving a discussion here that was started at >>> http://forums.java.net/jive/thread.jspa?threadID=152825&tstart=0 >>> >>> >>> I'll try to summarize. >>> >>> A useful goal of the JDK7 WatchService is to provide a uniform file system >>> event facility across multiple operating systems. However, in some cases, it >>> may be more desirable to give up the portability in order to tap into the >>> full expressiveness of the event model of the native OS. For example, >>> Inotify on the Linux platform provides additional events not currently used >>> by the LinuxWatchService implementation such as: IN_ACCESS, IN_CLOSE_WRITE, >>> IN_CLOSE_NO_WRITE, and IN_OPEN. >>> >>> I gave one example at the above URI where an event like IN_CLOSE_WRITE >>> could be used to detect and process a large file only after it has been >>> completely copied into a watch directory. >>> >>> After looking over the internal LinuxWatchService.java source code, it >>> appears that the implementation could be readily used (modified or resused?) >>> to also provide access to these additional event types. Another event class >>> representation like LinuxWatchEventKinds would need to be added and used >>> alternatively to the StandardWatchEventKinds. >>> >>> My question. Would extending the LinuxWatchService capability to >>> optionally expose the greater set of Inotify event types be in scope for JDK >>> 7 development, or will client developers who desire this capability need to >>> find 3rd party implementations (or develop it themselves)? >>> >>> Regards. >>> >> The API is designed to allow for events and modifiers that are defined >> beyond the javadoc so provider implementations can include support for other >> events if they wish. We haven't included any support for other events in the >> JDK because our primary motive has been to help the performance of >> applications that are forced to poll the file system today. If you are >> looking for pointers then there is one implementation specific modifier >> (com.nio.file.ExtendedWatchEvent) ?that might help show you how additional >> support is included. You are welcome to hack on the code and try out >> additional events but you have to be a contributer to contribute patches >> [1]. ?A small comment on the "close after write" event is that I don't think >> you are guaranteed that the file isn't being modified by other entities. >> That might not be a concern in your environment but just mentioning it. >> >> -Alan. >> >> [1] http://openjdk.java.net/contribute/ >> > From lists at laerad.com Thu Sep 23 07:43:14 2010 From: lists at laerad.com (Benedict Elliott Smith) Date: Thu, 23 Sep 2010 15:43:14 +0100 Subject: WatchService - Exposing more of the Inotify Event Model In-Reply-To: References: <4C7FBBF3.7020501@oracle.com> Message-ID: While on the topic of the WatchService API, I wonder if there are any plans to modify the WatchKey to include a method for retrieving the object/path it is watching. The fact that it does not introduces a race condition, whereby the WatchService can return a WatchKey before the thread registering the key has a chance to put the key in a map containing this information. On 23 September 2010 07:41, fm wrote: > I've modified a few of the WatchService implementation classes to > allow for the registration of additional WatchEvent.Kind instances > that represent the native events of the underlying OS platform. I > believe the changes add flexibility to the WatchService API, with > minimal disruptive code changes to the existing implementation. There > should be a negligible impact on the existing performance. I have > altered the implementation of the LinuxWatchService to support the > additional event model; however, the SolarisWatchService and > WindowsWatchService implementations are effectively unchanged and > should work the same as before. Similar modifications could be made to > these WatchService implementations to also expose their native event > models. > > I am completely open to hearing comments on if this type of added > flexibility to the WatchService API is appropriate for the JDK 7 or > better left to an ex > ternal project. > > I will gladly provide the source code files for evaluation if there is > interest. Modified files are AbstractWatchKey.java, > AbstractPoller.java, LinuxWatchService.java, and one new file > LinuxWatchEventKind.java. > > Regards > > On Thu, Sep 2, 2010 at 10:59 AM, fm wrote: > > Thank you for the response. This is the type of info I was looking > > for. Reading the WatchService related API documentation I got the > > sense it was designed for extension. After skimming over some of the > > related source code, it wasn't yet clear to me the best way to go > > about accessing the additional Linux specific filesystem events. After > > looking at the JDK7 source code it was clear to me a lot of hard work > > went into the design of the WatchService notification capability and > > it would be a shame not to be able to leverage it for accessing > > additional event tyeps. I will look into you suggestions. > > > > I am also aware that Inotify's IN_CLOSE_WRITE event is not the silver > > bullet to my particular project--but as the link R?mi provided points > > out, it may be more suitable that IN_MODIFY (in my case as well as > > others). But I believe you are correct also that one cannot assume > > something else does not have a file open for writing...it is just much > > more unlikely in my particular case. > > > > Regards > > > > > > On Thu, Sep 2, 2010 at 8:00 AM, Alan Bateman > wrote: > >> f m wrote: > >>> > >>> Per Alan B.'s request, I'm moving a discussion here that was started at > >>> http://forums.java.net/jive/thread.jspa?threadID=152825&tstart=0 > >>> > >>> > >>> I'll try to summarize. > >>> > >>> A useful goal of the JDK7 WatchService is to provide a uniform file > system > >>> event facility across multiple operating systems. However, in some > cases, it > >>> may be more desirable to give up the portability in order to tap into > the > >>> full expressiveness of the event model of the native OS. For example, > >>> Inotify on the Linux platform provides additional events not currently > used > >>> by the LinuxWatchService implementation such as: IN_ACCESS, > IN_CLOSE_WRITE, > >>> IN_CLOSE_NO_WRITE, and IN_OPEN. > >>> > >>> I gave one example at the above URI where an event like IN_CLOSE_WRITE > >>> could be used to detect and process a large file only after it has been > >>> completely copied into a watch directory. > >>> > >>> After looking over the internal LinuxWatchService.java source code, it > >>> appears that the implementation could be readily used (modified or > resused?) > >>> to also provide access to these additional event types. Another event > class > >>> representation like LinuxWatchEventKinds would need to be added and > used > >>> alternatively to the StandardWatchEventKinds. > >>> > >>> My question. Would extending the LinuxWatchService capability to > >>> optionally expose the greater set of Inotify event types be in scope > for JDK > >>> 7 development, or will client developers who desire this capability > need to > >>> find 3rd party implementations (or develop it themselves)? > >>> > >>> Regards. > >>> > >> The API is designed to allow for events and modifiers that are defined > >> beyond the javadoc so provider implementations can include support for > other > >> events if they wish. We haven't included any support for other events in > the > >> JDK because our primary motive has been to help the performance of > >> applications that are forced to poll the file system today. If you are > >> looking for pointers then there is one implementation specific modifier > >> (com.nio.file.ExtendedWatchEvent) that might help show you how > additional > >> support is included. You are welcome to hack on the code and try out > >> additional events but you have to be a contributer to contribute patches > >> [1]. A small comment on the "close after write" event is that I don't > think > >> you are guaranteed that the file isn't being modified by other entities. > >> That might not be a concern in your environment but just mentioning it. > >> > >> -Alan. > >> > >> [1] http://openjdk.java.net/contribute/ > >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100923/457f599b/attachment.html From forax at univ-mlv.fr Thu Sep 23 08:40:35 2010 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Thu, 23 Sep 2010 17:40:35 +0200 Subject: WatchService - Exposing more of the Inotify Event Model In-Reply-To: References: <4C7FBBF3.7020501@oracle.com> Message-ID: <4C9B74F3.5080102@univ-mlv.fr> Le 23/09/2010 16:43, Benedict Elliott Smith a ?crit : > While on the topic of the WatchService API, I wonder if there are any > plans to modify the WatchKey to include a method for retrieving the > object/path it is watching. The Watchable ? Why ? > The fact that it does not introduces a race condition, whereby the > WatchService can return a WatchKey before the thread registering the > key has a chance to put the key in a map containing this information. I think I don't understand. It's easy to store the watchable in a field of the watch key. I wonder if instead of storing a watchable, it's perhaps better to allow the user to choose what he want to store by adding a kind of attachment object similar to the one used for the Selector/SelectionKey mechanism. R?mi > > > > On 23 September 2010 07:41, fm > wrote: > > I've modified a few of the WatchService implementation classes to > allow for the registration of additional WatchEvent.Kind instances > that represent the native events of the underlying OS platform. I > believe the changes add flexibility to the WatchService API, with > minimal disruptive code changes to the existing implementation. There > should be a negligible impact on the existing performance. I have > altered the implementation of the LinuxWatchService to support the > additional event model; however, the SolarisWatchService and > WindowsWatchService implementations are effectively unchanged and > should work the same as before. Similar modifications could be made to > these WatchService implementations to also expose their native event > models. > > I am completely open to hearing comments on if this type of added > flexibility to the WatchService API is appropriate for the JDK 7 or > better left to an ex > ternal project. > > I will gladly provide the source code files for evaluation if there is > interest. Modified files are AbstractWatchKey.java, > AbstractPoller.java, LinuxWatchService.java, and one new file > LinuxWatchEventKind.java. > > Regards > > On Thu, Sep 2, 2010 at 10:59 AM, fm > wrote: > > Thank you for the response. This is the type of info I was looking > > for. Reading the WatchService related API documentation I got the > > sense it was designed for extension. After skimming over some of the > > related source code, it wasn't yet clear to me the best way to go > > about accessing the additional Linux specific filesystem events. > After > > looking at the JDK7 source code it was clear to me a lot of hard > work > > went into the design of the WatchService notification capability and > > it would be a shame not to be able to leverage it for accessing > > additional event tyeps. I will look into you suggestions. > > > > I am also aware that Inotify's IN_CLOSE_WRITE event is not the > silver > > bullet to my particular project--but as the link R?mi provided > points > > out, it may be more suitable that IN_MODIFY (in my case as well as > > others). But I believe you are correct also that one cannot assume > > something else does not have a file open for writing...it is > just much > > more unlikely in my particular case. > > > > Regards > > > > > > On Thu, Sep 2, 2010 at 8:00 AM, Alan Bateman > > wrote: > >> f m wrote: > >>> > >>> Per Alan B.'s request, I'm moving a discussion here that was > started at > >>> > http://forums.java.net/jive/thread.jspa?threadID=152825&tstart=0 > > >>> > > > >>> > >>> I'll try to summarize. > >>> > >>> A useful goal of the JDK7 WatchService is to provide a uniform > file system > >>> event facility across multiple operating systems. However, in > some cases, it > >>> may be more desirable to give up the portability in order to > tap into the > >>> full expressiveness of the event model of the native OS. For > example, > >>> Inotify on the Linux platform provides additional events not > currently used > >>> by the LinuxWatchService implementation such as: IN_ACCESS, > IN_CLOSE_WRITE, > >>> IN_CLOSE_NO_WRITE, and IN_OPEN. > >>> > >>> I gave one example at the above URI where an event like > IN_CLOSE_WRITE > >>> could be used to detect and process a large file only after it > has been > >>> completely copied into a watch directory. > >>> > >>> After looking over the internal LinuxWatchService.java source > code, it > >>> appears that the implementation could be readily used > (modified or resused?) > >>> to also provide access to these additional event types. > Another event class > >>> representation like LinuxWatchEventKinds would need to be > added and used > >>> alternatively to the StandardWatchEventKinds. > >>> > >>> My question. Would extending the LinuxWatchService capability to > >>> optionally expose the greater set of Inotify event types be in > scope for JDK > >>> 7 development, or will client developers who desire this > capability need to > >>> find 3rd party implementations (or develop it themselves)? > >>> > >>> Regards. > >>> > >> The API is designed to allow for events and modifiers that are > defined > >> beyond the javadoc so provider implementations can include > support for other > >> events if they wish. We haven't included any support for other > events in the > >> JDK because our primary motive has been to help the performance of > >> applications that are forced to poll the file system today. If > you are > >> looking for pointers then there is one implementation specific > modifier > >> (com.nio.file.ExtendedWatchEvent) that might help show you how > additional > >> support is included. You are welcome to hack on the code and > try out > >> additional events but you have to be a contributer to > contribute patches > >> [1]. A small comment on the "close after write" event is that > I don't think > >> you are guaranteed that the file isn't being modified by other > entities. > >> That might not be a concern in your environment but just > mentioning it. > >> > >> -Alan. > >> > >> [1] http://openjdk.java.net/contribute/ > >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100923/4676c215/attachment-0001.html From lists at laerad.com Thu Sep 23 08:45:15 2010 From: lists at laerad.com (Benedict Elliott Smith) Date: Thu, 23 Sep 2010 16:45:15 +0100 Subject: WatchService - Exposing more of the Inotify Event Model In-Reply-To: <4C9B74F3.5080102@univ-mlv.fr> References: <4C7FBBF3.7020501@oracle.com> <4C9B74F3.5080102@univ-mlv.fr> Message-ID: It *is* fairly easy to store the Watchable as a field in the WatchKey, except that the WatchKey is created by the WatchService, over which the user typically has no control - and the default implementations return a WatchKey that cannot tell you what it is watching (despite obviously having to know this). As such, currently (and in the example usages) the user has to insert the WatchKey that is returned by the WatchService into a map which stores the Watchable. The problem is that if a different thread registers the Watchable to the one consuming events from the WatchService, it is possible for a WatchKey to be returned to the consumer before the map contains it, and hence the consumer does not know what Watchable it is for. This seems bad to me. Even if this weren't the case though, it seems a bit unnecessary to expect the user to store the Watchable themselves - as you say, it should be easy to store it in the WatchKey, so I'm not sure why it isn't exposed to the user? Or am I missing something obvious? On 23 September 2010 16:40, R?mi Forax wrote: > Le 23/09/2010 16:43, Benedict Elliott Smith a ?crit : > > While on the topic of the WatchService API, I wonder if there are any plans > to modify the WatchKey to include a method for retrieving the object/path it > is watching. > > > The Watchable ? Why ? > > > The fact that it does not introduces a race condition, whereby the > WatchService can return a WatchKey before the thread registering the key has > a chance to put the key in a map containing this information. > > > I think I don't understand. > It's easy to store the watchable in a field of the watch key. > > I wonder if instead of storing a watchable, it's perhaps better to allow > the user > to choose what he want to store by adding a kind of attachment object > similar to > the one used for the Selector/SelectionKey mechanism. > > R?mi > > > > > > On 23 September 2010 07:41, fm wrote: > >> I've modified a few of the WatchService implementation classes to >> allow for the registration of additional WatchEvent.Kind instances >> that represent the native events of the underlying OS platform. I >> believe the changes add flexibility to the WatchService API, with >> minimal disruptive code changes to the existing implementation. There >> should be a negligible impact on the existing performance. I have >> altered the implementation of the LinuxWatchService to support the >> additional event model; however, the SolarisWatchService and >> WindowsWatchService implementations are effectively unchanged and >> should work the same as before. Similar modifications could be made to >> these WatchService implementations to also expose their native event >> models. >> >> I am completely open to hearing comments on if this type of added >> flexibility to the WatchService API is appropriate for the JDK 7 or >> better left to an ex >> ternal project. >> >> I will gladly provide the source code files for evaluation if there is >> interest. Modified files are AbstractWatchKey.java, >> AbstractPoller.java, LinuxWatchService.java, and one new file >> LinuxWatchEventKind.java. >> >> Regards >> >> On Thu, Sep 2, 2010 at 10:59 AM, fm wrote: >> > Thank you for the response. This is the type of info I was looking >> > for. Reading the WatchService related API documentation I got the >> > sense it was designed for extension. After skimming over some of the >> > related source code, it wasn't yet clear to me the best way to go >> > about accessing the additional Linux specific filesystem events. After >> > looking at the JDK7 source code it was clear to me a lot of hard work >> > went into the design of the WatchService notification capability and >> > it would be a shame not to be able to leverage it for accessing >> > additional event tyeps. I will look into you suggestions. >> > >> > I am also aware that Inotify's IN_CLOSE_WRITE event is not the silver >> > bullet to my particular project--but as the link R?mi provided points >> > out, it may be more suitable that IN_MODIFY (in my case as well as >> > others). But I believe you are correct also that one cannot assume >> > something else does not have a file open for writing...it is just much >> > more unlikely in my particular case. >> > >> > Regards >> > >> > >> > On Thu, Sep 2, 2010 at 8:00 AM, Alan Bateman >> wrote: >> >> f m wrote: >> >>> >> >>> Per Alan B.'s request, I'm moving a discussion here that was started >> at >> >>> http://forums.java.net/jive/thread.jspa?threadID=152825&tstart=0 >> >>> >> >>> >> >>> I'll try to summarize. >> >>> >> >>> A useful goal of the JDK7 WatchService is to provide a uniform file >> system >> >>> event facility across multiple operating systems. However, in some >> cases, it >> >>> may be more desirable to give up the portability in order to tap into >> the >> >>> full expressiveness of the event model of the native OS. For example, >> >>> Inotify on the Linux platform provides additional events not currently >> used >> >>> by the LinuxWatchService implementation such as: IN_ACCESS, >> IN_CLOSE_WRITE, >> >>> IN_CLOSE_NO_WRITE, and IN_OPEN. >> >>> >> >>> I gave one example at the above URI where an event like IN_CLOSE_WRITE >> >>> could be used to detect and process a large file only after it has >> been >> >>> completely copied into a watch directory. >> >>> >> >>> After looking over the internal LinuxWatchService.java source code, it >> >>> appears that the implementation could be readily used (modified or >> resused?) >> >>> to also provide access to these additional event types. Another event >> class >> >>> representation like LinuxWatchEventKinds would need to be added and >> used >> >>> alternatively to the StandardWatchEventKinds. >> >>> >> >>> My question. Would extending the LinuxWatchService capability to >> >>> optionally expose the greater set of Inotify event types be in scope >> for JDK >> >>> 7 development, or will client developers who desire this capability >> need to >> >>> find 3rd party implementations (or develop it themselves)? >> >>> >> >>> Regards. >> >>> >> >> The API is designed to allow for events and modifiers that are defined >> >> beyond the javadoc so provider implementations can include support for >> other >> >> events if they wish. We haven't included any support for other events >> in the >> >> JDK because our primary motive has been to help the performance of >> >> applications that are forced to poll the file system today. If you are >> >> looking for pointers then there is one implementation specific modifier >> >> (com.nio.file.ExtendedWatchEvent) that might help show you how >> additional >> >> support is included. You are welcome to hack on the code and try out >> >> additional events but you have to be a contributer to contribute >> patches >> >> [1]. A small comment on the "close after write" event is that I don't >> think >> >> you are guaranteed that the file isn't being modified by other >> entities. >> >> That might not be a concern in your environment but just mentioning it. >> >> >> >> -Alan. >> >> >> >> [1] http://openjdk.java.net/contribute/ >> >> >> > >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100923/222088f8/attachment.html From forax at univ-mlv.fr Thu Sep 23 10:11:32 2010 From: forax at univ-mlv.fr (=?UTF-8?B?UsOpbWkgRm9yYXg=?=) Date: Thu, 23 Sep 2010 19:11:32 +0200 Subject: WatchService - Exposing more of the Inotify Event Model In-Reply-To: References: <4C7FBBF3.7020501@oracle.com> <4C9B74F3.5080102@univ-mlv.fr> Message-ID: <4C9B8A44.5090509@univ-mlv.fr> Le 23/09/2010 17:45, Benedict Elliott Smith a ?crit : > It /is/ fairly easy to store the Watchable as a field in the WatchKey, > except that the WatchKey is created by the WatchService, over which > the user typically has no control - and the default implementations > return a WatchKey that cannot tell you what it is watching (despite > obviously having to know this). Ok. As I said, I think a more general attached Object is a better solution in my opinion. > > As such, currently (and in the example usages) the user has to insert > the WatchKey that is returned by the WatchService into a map which > stores the Watchable. The problem is that if a different thread > registers the Watchable to the one consuming events from the > WatchService, it is possible for a WatchKey to be returned to the > consumer before the map contains it, and hence the consumer does not > know what Watchable it is for. If you store/retreive the path in a synchronized block or use a ConcurrentHashMap and call put() before registering, I don't see why there is a problem ? > > This seems bad to me. > > Even if this weren't the case though, it seems a bit unnecessary to > expect the user to store the Watchable themselves - as you say, it > should be easy to store it in the WatchKey, so I'm not sure why it > isn't exposed to the user? Or am I missing something obvious? It's not exposed because the current implementation doesn't store it :) On Linux the file descriptor is stored not the corresponding Path. R?mi > > > > On 23 September 2010 16:40, R?mi Forax > wrote: > > Le 23/09/2010 16:43, Benedict Elliott Smith a ?crit : >> While on the topic of the WatchService API, I wonder if there are >> any plans to modify the WatchKey to include a method for >> retrieving the object/path it is watching. > > The Watchable ? Why ? > > >> The fact that it does not introduces a race condition, whereby >> the WatchService can return a WatchKey before the thread >> registering the key has a chance to put the key in a map >> containing this information. > > I think I don't understand. > It's easy to store the watchable in a field of the watch key. > > I wonder if instead of storing a watchable, it's perhaps better to > allow the user > to choose what he want to store by adding a kind of attachment > object similar to > the one used for the Selector/SelectionKey mechanism. > > R?mi > > >> >> >> >> On 23 September 2010 07:41, fm > > wrote: >> >> I've modified a few of the WatchService implementation classes to >> allow for the registration of additional WatchEvent.Kind >> instances >> that represent the native events of the underlying OS platform. I >> believe the changes add flexibility to the WatchService API, with >> minimal disruptive code changes to the existing >> implementation. There >> should be a negligible impact on the existing performance. I have >> altered the implementation of the LinuxWatchService to >> support the >> additional event model; however, the SolarisWatchService and >> WindowsWatchService implementations are effectively unchanged and >> should work the same as before. Similar modifications could >> be made to >> these WatchService implementations to also expose their >> native event >> models. >> >> I am completely open to hearing comments on if this type of added >> flexibility to the WatchService API is appropriate for the >> JDK 7 or >> better left to an ex >> ternal project. >> >> I will gladly provide the source code files for evaluation if >> there is >> interest. Modified files are AbstractWatchKey.java, >> AbstractPoller.java, LinuxWatchService.java, and one new file >> LinuxWatchEventKind.java. >> >> Regards >> >> On Thu, Sep 2, 2010 at 10:59 AM, fm > > wrote: >> > Thank you for the response. This is the type of info I was >> looking >> > for. Reading the WatchService related API documentation I >> got the >> > sense it was designed for extension. After skimming over >> some of the >> > related source code, it wasn't yet clear to me the best way >> to go >> > about accessing the additional Linux specific filesystem >> events. After >> > looking at the JDK7 source code it was clear to me a lot of >> hard work >> > went into the design of the WatchService notification >> capability and >> > it would be a shame not to be able to leverage it for accessing >> > additional event tyeps. I will look into you suggestions. >> > >> > I am also aware that Inotify's IN_CLOSE_WRITE event is not >> the silver >> > bullet to my particular project--but as the link R?mi >> provided points >> > out, it may be more suitable that IN_MODIFY (in my case as >> well as >> > others). But I believe you are correct also that one cannot >> assume >> > something else does not have a file open for writing...it >> is just much >> > more unlikely in my particular case. >> > >> > Regards >> > >> > >> > On Thu, Sep 2, 2010 at 8:00 AM, Alan Bateman >> > wrote: >> >> f m wrote: >> >>> >> >>> Per Alan B.'s request, I'm moving a discussion here that >> was started at >> >>> >> http://forums.java.net/jive/thread.jspa?threadID=152825&tstart=0 >> >> >>> >> > > >> >>> >> >>> I'll try to summarize. >> >>> >> >>> A useful goal of the JDK7 WatchService is to provide a >> uniform file system >> >>> event facility across multiple operating systems. >> However, in some cases, it >> >>> may be more desirable to give up the portability in order >> to tap into the >> >>> full expressiveness of the event model of the native OS. >> For example, >> >>> Inotify on the Linux platform provides additional events >> not currently used >> >>> by the LinuxWatchService implementation such as: >> IN_ACCESS, IN_CLOSE_WRITE, >> >>> IN_CLOSE_NO_WRITE, and IN_OPEN. >> >>> >> >>> I gave one example at the above URI where an event like >> IN_CLOSE_WRITE >> >>> could be used to detect and process a large file only >> after it has been >> >>> completely copied into a watch directory. >> >>> >> >>> After looking over the internal LinuxWatchService.java >> source code, it >> >>> appears that the implementation could be readily used >> (modified or resused?) >> >>> to also provide access to these additional event types. >> Another event class >> >>> representation like LinuxWatchEventKinds would need to be >> added and used >> >>> alternatively to the StandardWatchEventKinds. >> >>> >> >>> My question. Would extending the LinuxWatchService >> capability to >> >>> optionally expose the greater set of Inotify event types >> be in scope for JDK >> >>> 7 development, or will client developers who desire this >> capability need to >> >>> find 3rd party implementations (or develop it themselves)? >> >>> >> >>> Regards. >> >>> >> >> The API is designed to allow for events and modifiers that >> are defined >> >> beyond the javadoc so provider implementations can include >> support for other >> >> events if they wish. We haven't included any support for >> other events in the >> >> JDK because our primary motive has been to help the >> performance of >> >> applications that are forced to poll the file system >> today. If you are >> >> looking for pointers then there is one implementation >> specific modifier >> >> (com.nio.file.ExtendedWatchEvent) that might help show >> you how additional >> >> support is included. You are welcome to hack on the code >> and try out >> >> additional events but you have to be a contributer to >> contribute patches >> >> [1]. A small comment on the "close after write" event is >> that I don't think >> >> you are guaranteed that the file isn't being modified by >> other entities. >> >> That might not be a concern in your environment but just >> mentioning it. >> >> >> >> -Alan. >> >> >> >> [1] http://openjdk.java.net/contribute/ >> >> >> > >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100923/95602461/attachment-0001.html From lists at laerad.com Thu Sep 23 10:15:06 2010 From: lists at laerad.com (Benedict Elliott Smith) Date: Thu, 23 Sep 2010 18:15:06 +0100 Subject: WatchService - Exposing more of the Inotify Event Model In-Reply-To: <4C9B8A44.5090509@univ-mlv.fr> References: <4C7FBBF3.7020501@oracle.com> <4C9B74F3.5080102@univ-mlv.fr> <4C9B8A44.5090509@univ-mlv.fr> Message-ID: On 23 September 2010 18:11, R?mi Forax wrote: > Le 23/09/2010 17:45, Benedict Elliott Smith a ?crit : > > It *is* fairly easy to store the Watchable as a field in the WatchKey, > except that the WatchKey is created by the WatchService, over which the user > typically has no control - and the default implementations return > a WatchKey that cannot tell you what it is watching (despite obviously > having to know this). > > > Ok. > As I said, I think a more general attached Object is a better solution in > my opinion. > > If you would rather provide the ability to annotate the information WatchKey with arbitrary information this would be a perfectly acceptable solution IMHO. > > As such, currently (and in the example usages) the user has to insert the > WatchKey that is returned by the WatchService into a map which stores the > Watchable. The problem is that if a different thread registers the Watchable > to the one consuming events from the WatchService, it is possible for a > WatchKey to be returned to the consumer before the map contains it, and > hence the consumer does not know what Watchable it is for. > > > If you store/retreive the path in a synchronized block or use a > ConcurrentHashMap > and call put() before registering, I don't see why there is a problem ? > > Using a ConcurrentHashMap only enforces a happens-before relation, it does not address race conditions. Consider the following pseudo-code: Thread1 { watchkey = watchservice.register(mypath) sharedmap.put(watchkey, mypath) } Thread2 { watchkey = watchservice.poll() path = sharedmap.get(watchkey) } While making sharedmap either synchronized or a ConcurrentHashMap does guarantee that the map is consistent and that the mapping eventually shows up for Thread2, it does nothing to address the fact that Thread1 may be swapped out after executing the first instruction, and not execute its second instruction until after Thread2 reads from the map, getting no result. You could synchronize around both steps, but this is really not a very elegant solution. Also, it is a pitfall many people will not even notice, and hence not code to avoid. It seems best to address it with an obvious and simply solution that is built into the standard examples of usage. i.e. that the path is obtainable from the WatchKey. It seems likely that this information will required more often than not, surely? > This seems bad to me. > > Even if this weren't the case though, it seems a bit unnecessary to > expect the user to store the Watchable themselves - as you say, it should be > easy to store it in the WatchKey, so I'm not sure why it isn't exposed to > the user? Or am I missing something obvious? > > > It's not exposed because the current implementation doesn't store it :) > On Linux the file descriptor is stored not the corresponding Path. > > Fair enough, I can see why you would have chosen to avoid putting it in (since Path is not a particularly lightweight object) - but it seems like it might be a necessary evil to help prevent this problem biting users randomly without their knowing why or what to do about it. > R?mi > > > > > On 23 September 2010 16:40, R?mi Forax wrote: > >> Le 23/09/2010 16:43, Benedict Elliott Smith a ?crit : >> >> While on the topic of the WatchService API, I wonder if there are any >> plans to modify the WatchKey to include a method for retrieving the >> object/path it is watching. >> >> >> The Watchable ? Why ? >> >> >> The fact that it does not introduces a race condition, whereby the >> WatchService can return a WatchKey before the thread registering the key has >> a chance to put the key in a map containing this information. >> >> >> I think I don't understand. >> It's easy to store the watchable in a field of the watch key. >> >> I wonder if instead of storing a watchable, it's perhaps better to allow >> the user >> to choose what he want to store by adding a kind of attachment object >> similar to >> the one used for the Selector/SelectionKey mechanism. >> >> R?mi >> >> >> >> >> >> On 23 September 2010 07:41, fm wrote: >> >>> I've modified a few of the WatchService implementation classes to >>> allow for the registration of additional WatchEvent.Kind instances >>> that represent the native events of the underlying OS platform. I >>> believe the changes add flexibility to the WatchService API, with >>> minimal disruptive code changes to the existing implementation. There >>> should be a negligible impact on the existing performance. I have >>> altered the implementation of the LinuxWatchService to support the >>> additional event model; however, the SolarisWatchService and >>> WindowsWatchService implementations are effectively unchanged and >>> should work the same as before. Similar modifications could be made to >>> these WatchService implementations to also expose their native event >>> models. >>> >>> I am completely open to hearing comments on if this type of added >>> flexibility to the WatchService API is appropriate for the JDK 7 or >>> better left to an ex >>> ternal project. >>> >>> I will gladly provide the source code files for evaluation if there is >>> interest. Modified files are AbstractWatchKey.java, >>> AbstractPoller.java, LinuxWatchService.java, and one new file >>> LinuxWatchEventKind.java. >>> >>> Regards >>> >>> On Thu, Sep 2, 2010 at 10:59 AM, fm wrote: >>> > Thank you for the response. This is the type of info I was looking >>> > for. Reading the WatchService related API documentation I got the >>> > sense it was designed for extension. After skimming over some of the >>> > related source code, it wasn't yet clear to me the best way to go >>> > about accessing the additional Linux specific filesystem events. After >>> > looking at the JDK7 source code it was clear to me a lot of hard work >>> > went into the design of the WatchService notification capability and >>> > it would be a shame not to be able to leverage it for accessing >>> > additional event tyeps. I will look into you suggestions. >>> > >>> > I am also aware that Inotify's IN_CLOSE_WRITE event is not the silver >>> > bullet to my particular project--but as the link R?mi provided points >>> > out, it may be more suitable that IN_MODIFY (in my case as well as >>> > others). But I believe you are correct also that one cannot assume >>> > something else does not have a file open for writing...it is just much >>> > more unlikely in my particular case. >>> > >>> > Regards >>> > >>> > >>> > On Thu, Sep 2, 2010 at 8:00 AM, Alan Bateman >>> wrote: >>> >> f m wrote: >>> >>> >>> >>> Per Alan B.'s request, I'm moving a discussion here that was started >>> at >>> >>> http://forums.java.net/jive/thread.jspa?threadID=152825&tstart=0 >>> >>> >>> >>> >>> >>> I'll try to summarize. >>> >>> >>> >>> A useful goal of the JDK7 WatchService is to provide a uniform file >>> system >>> >>> event facility across multiple operating systems. However, in some >>> cases, it >>> >>> may be more desirable to give up the portability in order to tap into >>> the >>> >>> full expressiveness of the event model of the native OS. For example, >>> >>> Inotify on the Linux platform provides additional events not >>> currently used >>> >>> by the LinuxWatchService implementation such as: IN_ACCESS, >>> IN_CLOSE_WRITE, >>> >>> IN_CLOSE_NO_WRITE, and IN_OPEN. >>> >>> >>> >>> I gave one example at the above URI where an event like >>> IN_CLOSE_WRITE >>> >>> could be used to detect and process a large file only after it has >>> been >>> >>> completely copied into a watch directory. >>> >>> >>> >>> After looking over the internal LinuxWatchService.java source code, >>> it >>> >>> appears that the implementation could be readily used (modified or >>> resused?) >>> >>> to also provide access to these additional event types. Another event >>> class >>> >>> representation like LinuxWatchEventKinds would need to be added and >>> used >>> >>> alternatively to the StandardWatchEventKinds. >>> >>> >>> >>> My question. Would extending the LinuxWatchService capability to >>> >>> optionally expose the greater set of Inotify event types be in scope >>> for JDK >>> >>> 7 development, or will client developers who desire this capability >>> need to >>> >>> find 3rd party implementations (or develop it themselves)? >>> >>> >>> >>> Regards. >>> >>> >>> >> The API is designed to allow for events and modifiers that are defined >>> >> beyond the javadoc so provider implementations can include support for >>> other >>> >> events if they wish. We haven't included any support for other events >>> in the >>> >> JDK because our primary motive has been to help the performance of >>> >> applications that are forced to poll the file system today. If you are >>> >> looking for pointers then there is one implementation specific >>> modifier >>> >> (com.nio.file.ExtendedWatchEvent) that might help show you how >>> additional >>> >> support is included. You are welcome to hack on the code and try out >>> >> additional events but you have to be a contributer to contribute >>> patches >>> >> [1]. A small comment on the "close after write" event is that I don't >>> think >>> >> you are guaranteed that the file isn't being modified by other >>> entities. >>> >> That might not be a concern in your environment but just mentioning >>> it. >>> >> >>> >> -Alan. >>> >> >>> >> [1] http://openjdk.java.net/contribute/ >>> >> >>> > >>> >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100923/01626296/attachment.html From forax at univ-mlv.fr Thu Sep 23 11:12:13 2010 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Thu, 23 Sep 2010 20:12:13 +0200 Subject: WatchService - Exposing more of the Inotify Event Model In-Reply-To: References: <4C7FBBF3.7020501@oracle.com> <4C9B74F3.5080102@univ-mlv.fr> <4C9B8A44.5090509@univ-mlv.fr> Message-ID: <4C9B987D.2000907@univ-mlv.fr> Le 23/09/2010 19:15, Benedict Elliott Smith a ?crit : > On 23 September 2010 18:11, R?mi Forax > wrote: > > Le 23/09/2010 17:45, Benedict Elliott Smith a ?crit : >> It /is/ fairly easy to store the Watchable as a field in the >> WatchKey, except that the WatchKey is created by the >> WatchService, over which the user typically has no control - and >> the default implementations return a WatchKey that cannot tell >> you what it is watching (despite obviously having to know this). > > Ok. > As I said, I think a more general attached Object is a better > solution in my opinion. > > If you would rather provide the ability to annotate the information > WatchKey with arbitrary information this would be a perfectly > acceptable solution IMHO. > >> >> As such, currently (and in the example usages) the user has to >> insert the WatchKey that is returned by the WatchService into a >> map which stores the Watchable. The problem is that if a >> different thread registers the Watchable to the one consuming >> events from the WatchService, it is possible for a WatchKey to be >> returned to the consumer before the map contains it, and hence >> the consumer does not know what Watchable it is for. > > If you store/retreive the path in a synchronized block or use a > ConcurrentHashMap > and call put() before registering, I don't see why there is a > problem ? > > Using a ConcurrentHashMap only enforces a happens-before relation, it > does not address race conditions. Consider the following pseudo-code: > > Thread1 { > watchkey = watchservice.register(mypath) > sharedmap.put(watchkey, mypath) > } > > Thread2 { > watchkey = watchservice.poll() > path = sharedmap.get(watchkey) > } > > While making sharedmap either synchronized or a ConcurrentHashMap does > guarantee that the map is consistent and that the mapping eventually > shows up for Thread2, it does nothing to address the fact that Thread1 > may be swapped out after executing the first instruction, and not > execute its second instruction until after Thread2 reads from the map, > getting no result. > > You could synchronize around both steps, but this is really not a very > elegant solution. Also, it is a pitfall many people will not even > notice, and hence not code to avoid. It seems best to address it with > an obvious and simply solution that is built into the standard > examples of usage. i.e. that the path is obtainable from the WatchKey. > It seems likely that this information will required more often than > not, surely? There is another possible implementation, using a blocking queue and interrupt: Thread { queue.put(mypath); watchserviceThread.interrupt(); } WatchserviceThread { for(;;) { try { watchkey = watchservice.take() } catch(InterruptedException e) { while ((path = queue.poll()) != null) { path.register(watchservice); sharedmap.put(watchey, path); } continue; } path = sharedmap.get(watchkey); ... } } In that case, the registration and the watchkey retreival are confined in one thread. > >> >> This seems bad to me. >> >> Even if this weren't the case though, it seems a bit unnecessary >> to expect the user to store the Watchable themselves - as you >> say, it should be easy to store it in the WatchKey, so I'm not >> sure why it isn't exposed to the user? Or am I missing something >> obvious? > > It's not exposed because the current implementation doesn't store > it :) > On Linux the file descriptor is stored not the corresponding Path. > > > Fair enough, I can see why you would have chosen to avoid putting it > in (since Path is not a particularly lightweight object) - but it > seems like it might be a necessary evil to help prevent this problem > biting users randomly without their knowing why or what to do about it. > > > R?mi > R?mi -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100923/cab3cc90/attachment-0001.html From fudmail at gmail.com Thu Sep 23 11:50:00 2010 From: fudmail at gmail.com (fm) Date: Thu, 23 Sep 2010 11:50:00 -0700 Subject: WatchService - Exposing more of the Inotify Event Model In-Reply-To: References: <4C7FBBF3.7020501@oracle.com> <4C9B74F3.5080102@univ-mlv.fr> Message-ID: I agree. It seems to make more sense to me that what is registered with the WatchService should be retrievable from the WatchKey where you retrieve your WatchEvents from. It is not obvious to me why the user should have to create and manage a separate Map in order to reconstruct/resolve the original full context (e.g., absolute path that was registered). I asked this question a while back before I was aware of this mailing list but never received a response. Good discussion. On Thu, Sep 23, 2010 at 8:45 AM, Benedict Elliott Smith wrote: > It is fairly easy to store the Watchable as a field in the WatchKey, except > that the WatchKey is created by the WatchService, over which the user > typically has no control - and the default?implementations return a?WatchKey > that cannot tell you what it is watching (despite obviously having to know > this). > As such, currently (and in the example usages) the user has to insert the > WatchKey that is returned by the WatchService into a map which stores the > Watchable.?The problem is that if a different thread registers the Watchable > to the one consuming events from the WatchService, it is possible for a > WatchKey to be returned to the consumer before the map contains it, and > hence the consumer does not know what Watchable it is for. > This seems bad to me. > Even if this weren't the case though, it seems a bit unnecessary to expect > the user to store the Watchable themselves - as you say, it should be easy > to store it in the WatchKey, so I'm not sure why it isn't exposed to the > user? Or am I missing something obvious? > > > On 23 September 2010 16:40, R?mi Forax wrote: >> >> Le 23/09/2010 16:43, Benedict Elliott Smith a ?crit?: >> >> While on the topic of the WatchService API, I wonder if there are any >> plans to modify the WatchKey to include a method for retrieving the >> object/path it is watching. >> >> The Watchable ? Why ? >> >> The fact that it does not introduces a race condition, whereby the >> WatchService can return a WatchKey before the thread registering the key has >> a chance to put the key in a map containing this information. >> >> I think I don't understand. >> It's easy to store the watchable in a field of the watch key. >> >> I wonder if instead of storing a watchable, it's perhaps better to allow >> the user >> to choose what he want to store by adding a kind of attachment object >> similar to >> the one used for the Selector/SelectionKey mechanism. >> >> R?mi >> >> >> >> >> On 23 September 2010 07:41, fm wrote: >>> >>> I've modified a few of the WatchService implementation classes to >>> allow for the registration of additional WatchEvent.Kind instances >>> that represent the native events of the underlying OS platform. I >>> believe the changes add flexibility to the WatchService API, with >>> minimal disruptive code changes to the existing implementation. There >>> should be a negligible impact on the existing performance. I have >>> altered the implementation of the LinuxWatchService to support the >>> additional event model; however, the SolarisWatchService and >>> WindowsWatchService implementations are effectively unchanged and >>> should work the same as before. Similar modifications could be made to >>> these WatchService implementations to also expose their native event >>> models. >>> >>> I am completely open to hearing comments on if this type of added >>> flexibility to the WatchService API is appropriate for the JDK 7 or >>> better left to an ex >>> ternal project. >>> >>> I will gladly provide the source code files for evaluation if there is >>> interest. Modified files are AbstractWatchKey.java, >>> AbstractPoller.java, LinuxWatchService.java, and one new file >>> LinuxWatchEventKind.java. >>> >>> Regards >>> >>> On Thu, Sep 2, 2010 at 10:59 AM, fm wrote: >>> > Thank you for the response. This is the type of info I was looking >>> > for. Reading the WatchService related API documentation I got the >>> > sense it was designed for extension. After skimming over some of the >>> > related source code, it wasn't yet clear to me the best way to go >>> > about accessing the additional Linux specific filesystem events. After >>> > looking at the JDK7 source code it was clear to me a lot of hard work >>> > went into the design of the WatchService notification capability and >>> > it would be a shame not to be able to leverage it for accessing >>> > additional event tyeps. ?I will look into you suggestions. >>> > >>> > I am also aware that Inotify's IN_CLOSE_WRITE event is not the silver >>> > bullet to my particular project--but as the link R?mi provided points >>> > out, it may be more suitable that IN_MODIFY (in my case as well as >>> > others). But I believe you are correct also that one cannot assume >>> > something else does not have a file open for writing...it is just much >>> > more unlikely in my particular case. >>> > >>> > Regards >>> > >>> > >>> > On Thu, Sep 2, 2010 at 8:00 AM, Alan Bateman >>> > wrote: >>> >> f m wrote: >>> >>> >>> >>> Per Alan B.'s request, I'm moving a discussion here that was started >>> >>> at >>> >>> http://forums.java.net/jive/thread.jspa?threadID=152825&tstart=0 >>> >>> >>> >>> >>> >>> I'll try to summarize. >>> >>> >>> >>> A useful goal of the JDK7 WatchService is to provide a uniform file >>> >>> system >>> >>> event facility across multiple operating systems. However, in some >>> >>> cases, it >>> >>> may be more desirable to give up the portability in order to tap into >>> >>> the >>> >>> full expressiveness of the event model of the native OS. For example, >>> >>> Inotify on the Linux platform provides additional events not >>> >>> currently used >>> >>> by the LinuxWatchService implementation such as: IN_ACCESS, >>> >>> IN_CLOSE_WRITE, >>> >>> IN_CLOSE_NO_WRITE, and IN_OPEN. >>> >>> >>> >>> I gave one example at the above URI where an event like >>> >>> IN_CLOSE_WRITE >>> >>> could be used to detect and process a large file only after it has >>> >>> been >>> >>> completely copied into a watch directory. >>> >>> >>> >>> After looking over the internal LinuxWatchService.java source code, >>> >>> it >>> >>> appears that the implementation could be readily used (modified or >>> >>> resused?) >>> >>> to also provide access to these additional event types. Another event >>> >>> class >>> >>> representation like LinuxWatchEventKinds would need to be added and >>> >>> used >>> >>> alternatively to the StandardWatchEventKinds. >>> >>> >>> >>> My question. Would extending the LinuxWatchService capability to >>> >>> optionally expose the greater set of Inotify event types be in scope >>> >>> for JDK >>> >>> 7 development, or will client developers who desire this capability >>> >>> need to >>> >>> find 3rd party implementations (or develop it themselves)? >>> >>> >>> >>> Regards. >>> >>> >>> >> The API is designed to allow for events and modifiers that are defined >>> >> beyond the javadoc so provider implementations can include support for >>> >> other >>> >> events if they wish. We haven't included any support for other events >>> >> in the >>> >> JDK because our primary motive has been to help the performance of >>> >> applications that are forced to poll the file system today. If you are >>> >> looking for pointers then there is one implementation specific >>> >> modifier >>> >> (com.nio.file.ExtendedWatchEvent) ?that might help show you how >>> >> additional >>> >> support is included. You are welcome to hack on the code and try out >>> >> additional events but you have to be a contributer to contribute >>> >> patches >>> >> [1]. ?A small comment on the "close after write" event is that I don't >>> >> think >>> >> you are guaranteed that the file isn't being modified by other >>> >> entities. >>> >> That might not be a concern in your environment but just mentioning >>> >> it. >>> >> >>> >> -Alan. >>> >> >>> >> [1] http://openjdk.java.net/contribute/ >>> >> >>> > >> >> > > From lists at laerad.com Thu Sep 23 15:52:25 2010 From: lists at laerad.com (Benedict Elliott Smith) Date: Thu, 23 Sep 2010 23:52:25 +0100 Subject: WatchService - Exposing more of the Inotify Event Model In-Reply-To: <4C9B987D.2000907@univ-mlv.fr> References: <4C7FBBF3.7020501@oracle.com> <4C9B74F3.5080102@univ-mlv.fr> <4C9B8A44.5090509@univ-mlv.fr> <4C9B987D.2000907@univ-mlv.fr> Message-ID: The problem with this solution is that it is not intuitive and its necessity and added complexity can be completely removed by a simple change to the API to make it more intuitive (and hence generally safer). IMHO the number one aim of public APIs like this should be simplicity and safety of use. With all APIs there turn out to be common mistakes/pitfalls in using them that become apparent over time, but in this case there is a chance to mitigate this particular potential pitfall by a simple modification to the API itself, prior to it being widely used. This seems to me like it can only be a good thing. On 23 September 2010 19:12, R?mi Forax wrote: > Le 23/09/2010 19:15, Benedict Elliott Smith a ?crit : > > On 23 September 2010 18:11, R?mi Forax wrote: > >> Le 23/09/2010 17:45, Benedict Elliott Smith a ?crit : >> >> It *is* fairly easy to store the Watchable as a field in the WatchKey, >> except that the WatchKey is created by the WatchService, over which the user >> typically has no control - and the default implementations return >> a WatchKey that cannot tell you what it is watching (despite obviously >> having to know this). >> >> >> Ok. >> As I said, I think a more general attached Object is a better solution in >> my opinion. >> >> > If you would rather provide the ability to annotate the information > WatchKey with arbitrary information this would be a perfectly acceptable > solution IMHO. > >> >> As such, currently (and in the example usages) the user has to insert >> the WatchKey that is returned by the WatchService into a map which stores >> the Watchable. The problem is that if a different thread registers the >> Watchable to the one consuming events from the WatchService, it is possible >> for a WatchKey to be returned to the consumer before the map contains it, >> and hence the consumer does not know what Watchable it is for. >> >> >> If you store/retreive the path in a synchronized block or use a >> ConcurrentHashMap >> and call put() before registering, I don't see why there is a problem ? >> >> > Using a ConcurrentHashMap only enforces a happens-before relation, it does > not address race conditions. Consider the following pseudo-code: > > Thread1 { > watchkey = watchservice.register(mypath) > sharedmap.put(watchkey, mypath) > } > > Thread2 { > watchkey = watchservice.poll() > path = sharedmap.get(watchkey) > } > > While making sharedmap either synchronized or a ConcurrentHashMap does > guarantee that the map is consistent and that the mapping eventually shows > up for Thread2, it does nothing to address the fact that Thread1 may be > swapped out after executing the first instruction, and not execute its > second instruction until after Thread2 reads from the map, getting no > result. > > You could synchronize around both steps, but this is really not a very > elegant solution. Also, it is a pitfall many people will not even notice, > and hence not code to avoid. It seems best to address it with an obvious and > simply solution that is built into the standard examples of usage. i.e. that > the path is obtainable from the WatchKey. It seems likely that this > information will required more often than not, surely? > > > There is another possible implementation, using a blocking queue and > interrupt: > > Thread { > queue.put(mypath); > watchserviceThread.interrupt(); > } > > WatchserviceThread { > for(;;) { > try { > watchkey = watchservice.take() > } catch(InterruptedException e) { > while ((path = queue.poll()) != null) { > path.register(watchservice); > sharedmap.put(watchey, path); > } > continue; > } > path = sharedmap.get(watchkey); > ... > } > } > > In that case, the registration and the watchkey retreival are > confined in one thread. > > > > >> This seems bad to me. >> >> Even if this weren't the case though, it seems a bit unnecessary to >> expect the user to store the Watchable themselves - as you say, it should be >> easy to store it in the WatchKey, so I'm not sure why it isn't exposed to >> the user? Or am I missing something obvious? >> >> >> It's not exposed because the current implementation doesn't store it :) >> On Linux the file descriptor is stored not the corresponding Path. >> >> > Fair enough, I can see why you would have chosen to avoid putting it in > (since Path is not a particularly lightweight object) - but it seems like it > might be a necessary evil to help prevent this problem biting users randomly > without their knowing why or what to do about it. > > >> R?mi >> > > R?mi > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100923/6ec79159/attachment.html From Alan.Bateman at oracle.com Thu Sep 23 17:15:44 2010 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 23 Sep 2010 17:15:44 -0700 Subject: WatchService - Exposing more of the Inotify Event Model In-Reply-To: References: <4C7FBBF3.7020501@oracle.com> Message-ID: <4C9BEDB0.1070904@oracle.com> Benedict Elliott Smith wrote: > While on the topic of the WatchService API, I wonder if there are any > plans to modify the WatchKey to include a method for retrieving the > object/path it is watching. I've often wondered when this would come up :-) It was left out because of concern that the watch may be on the file (think inode). It may also be confusing where you attempt to watch a directory that is already being watched, but originally registered with a completely different path. It's clearly something that needs to be looked at again, now that the API has been used for some time. If WatchKey does get a method to return the Watchable then it could only reasonably be specified to returned the original Watchable. -Alan. From hjohn at xs4all.nl Thu Sep 23 23:58:13 2010 From: hjohn at xs4all.nl (John Hendrikx) Date: Fri, 24 Sep 2010 08:58:13 +0200 Subject: WatchService - Exposing more of the Inotify Event Model In-Reply-To: <4C9BEDB0.1070904@oracle.com> References: <4C7FBBF3.7020501@oracle.com> <4C9BEDB0.1070904@oracle.com> Message-ID: <4C9C4C05.2040103@xs4all.nl> Alan Bateman wrote: > Benedict Elliott Smith wrote: >> While on the topic of the WatchService API, I wonder if there are any >> plans to modify the WatchKey to include a method for retrieving the >> object/path it is watching. > I've often wondered when this would come up :-) It was left out > because of concern that the watch may be on the file (think inode). It > may also be confusing where you attempt to watch a directory that is > already being watched, but originally registered with a completely > different path. It's clearly something that needs to be looked at > again, now that the API has been used for some time. If WatchKey does > get a method to return the Watchable then it could only reasonably be > specified to returned the original Watchable. Well, I brought it up before... I too had to use a map to keep track of the original path :-) > > -Alan. > From lists at laerad.com Fri Sep 24 01:16:22 2010 From: lists at laerad.com (Benedict Elliott Smith) Date: Fri, 24 Sep 2010 09:16:22 +0100 Subject: WatchService - Exposing more of the Inotify Event Model In-Reply-To: <4C9BEDB0.1070904@oracle.com> References: <4C7FBBF3.7020501@oracle.com> <4C9BEDB0.1070904@oracle.com> Message-ID: I was thinking about this very thing on my commute this morning. The contract of WatchService is necessarily different on different OSes, and I'm not sure how you could mitigate that, so what you suggest seems reasonable. I think a satisfactory alternative might be to return what the OS thinks is the path of the file descriptor (which you currently store), in order to keep memory requirements low. This would also mitigate the problem that might occur from getting events for a moved directory, since the Watchable/Path returned would match the directory's current location. Without this feature, using events from directories after they have moved would be quite tricky in fact. It should probably be documented that some WatchService implementations will track events for directories after they have moved, and that others do not though. The PollingWatchService will invalidate a key (as if the directory were deleted) in this case, and this is (unfortunately) still the default implementation for us Solaris users. On 24 September 2010 01:15, Alan Bateman wrote: > Benedict Elliott Smith wrote: > >> While on the topic of the WatchService API, I wonder if there are any >> plans to modify the WatchKey to include a method for retrieving the >> object/path it is watching. >> > I've often wondered when this would come up :-) It was left out because of > concern that the watch may be on the file (think inode). It may also be > confusing where you attempt to watch a directory that is already being > watched, but originally registered with a completely different path. It's > clearly something that needs to be looked at again, now that the API has > been used for some time. If WatchKey does get a method to return the > Watchable then it could only reasonably be specified to returned the > original Watchable. > > -Alan. > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100924/3a8088ad/attachment.html