From Alan.Bateman at Sun.COM Sat Jan 2 03:18:13 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Sat, 02 Jan 2010 11:18:13 +0000 Subject: GetQueueCompletionStatusEx In-Reply-To: References: <4B3CD418.8000406@sun.com> Message-ID: <4B3F2B75.7000206@sun.com> John Davis wrote: > Forgot the CC > > On Thu, Dec 31, 2009 at 11:01 AM, John Davis > wrote: > > I'm not an expert on the nio internals. My thinking is it would > be nice if everything could be done with a single thread (ie all > reads/writes on all handles are async and pumped thru IOCP). > Because the kernel context switch overhead is pretty much > eliminated with this call, there is no longer a need to have a > thread pool to handle the context switch overhead. We also > wouldn't have the batching dilema with the thread pool gone. Only > problem with this model is that there's no way to guarantee a > completion handler won't block. > You can use a single thread if you wish but aren't you concerned about scaling to multiple sockets and cores? However you are right, if the channel group's thread pool has a single thread, then the completion results can be retrieved in batch to avoid a call to GetQueueCompletionStatus per I/O. > : > SetFileCompletionNotificationModes, allows the callback to be > invoked immediately, rather than pumping thru IOCP and thread pool. > We can certainly try the FILE_SKIP_SET_EVENT_ON_HANDLE mode to see if it makes a difference but for now we can't really make use of the FILE_SKIP_COMPLETION_PORT_ON_SUCCESS mode. Our implementation can actually immediate case we have it disabled at this time. The reason is that under load conditions, for loopback connections at least, the number of bytes transferred, as returned by WSARecv/WSASend is not reliable. We therefore require the completion result so that we can update the buffer position and notify the completion handler with the correct result. -Alan. From Alan.Bateman at Sun.COM Mon Jan 4 07:10:49 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Mon, 04 Jan 2010 15:10:49 +0000 Subject: Need reviewer for 6907760: (file) OVERFLOW event should cause pending events to be purged Message-ID: <4B4204F9.6020800@sun.com> This is a follow-up to the discussion on the handling of WatchService overflow events that John Hendrikx brought up on this mailing list in December. A suggestion to come from that discussion is that the implementation can drop pending events when queuing an overflow event. That will save the application from needlessly processing these events. Similarly, there isn't any point queuing events while there is an overflow event pending. The webrev with the proposed changes is here: http://cr.openjdk.java.net/~alanb/6907760/webrev.00/ Thanks, Alan. From Alan.Bateman at Sun.COM Mon Jan 4 08:31:55 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Mon, 04 Jan 2010 16:31:55 +0000 Subject: Need reviewer for 6913877: (fs) AsynchronousFileChannel.write can return wrong result under load [win] Message-ID: <4B4217FB.5060403@sun.com> This one is a Windows specific reliability issue with AsynchronousFileChannel.write. It's the issue that Joel Buckley brought up on nio-discuss a few days ago. Under load conditions (duplicates for me intermittently on a 8-core server) it is possible for AsynchronousFileChannel.write (or read) to complete with the incorrect result. The issue is similar to 6834246 in that the implementation is using the number of bytes transferred, as returned by WriteFile/ReadFile, as the result. This result is only reliable for synchronous operations so this patch changes the implementation so that result is always handled via the completion notification. The webrev is here: http://cr.openjdk.java.net/~alanb/6913877/webrev.00/ Thanks, Alan. From sebastian.sickelmann at gmx.de Tue Jan 5 01:52:18 2010 From: sebastian.sickelmann at gmx.de (Sebastian Sickelmann) Date: Tue, 05 Jan 2010 10:52:18 +0100 Subject: WatchService questions In-Reply-To: <4B1BED7B.40103@sun.com> References: <4B1A4F80.2070106@xs4all.nl> <4B1AC735.50404@sun.com> <4B1B9B28.9090007@xs4all.nl> <4B1BED7B.40103@sun.com> Message-ID: <20100105095218.78140@gmx.net> Alan Bateman wrote: > John Hendrikx wrote: > > : > > > > What I hadn't realized is that after you receive an OVERFLOW that the > > subsequent events will form a cohesive stream of events again which > > can be used to update a directory after re-reading it. It does make > > sense to me though to "scan" for the overflow so to speak -- either > > that or I would probably expect an implementation of WatchService to > > clear all pending events before adding the OVERFLOW. Not only are the > > pending events useless if an overflow occurs, but clearing them would > > make room for subsequent events again (if not, you'd be in a permanent > > state of overflow...?) > This is a good point as the watch service could drop all pending events > when the limit is reached. Furthermore it can drop all subsequent events > while the head is an OVERFLOW event. I'll create a bug to track this - > thanks! > > > My original assessment was wrong. The Swing Event thread was not > > starved for CPU, but instead was busy handling the WatchService events > > in the invokeAndWait part. This made it seem that the Event thread > > was starved for CPU (as other parts of the UI weren't updating > > anymore) while it really was consuming the CPU all by itself. > > > > In effect, in the time it took me to handle 60 WatchService events, > > 513 (512 + Overflow?) more had accumulated, resulting in second long > > delays for the UI to become responsive again. > Thanks for the update. I did a quick test that watches a directory and > retrieves all events in a timely manner. Another program creates and > deletes 10000 files in the watched directory which I think is what you > are doing. The test was on Windows (as I think this is where your UI is > at, right?). As expected the watcher didn't consume too many CPU cycles > and is barely noticeable. Also, the list of events typically only > contain a single event as it can easily keep up with programs creating > lots of files (which is a very slow in comparison). I changed the test > to sleep periodically, thus delaying the retrieval of events. In that > case, 512 events are queued very quickly, and subsequent events are > dropped. Again, the CPU usage is barely noticeable so this concurs with > your observation that the time is spent on processing the events. > > Anyway, let us know how your solution goes - your war story may be > useful to others. > > -Alan. While reviewing 6907760 i think i found one problem why John is facing so many Events. If the Copy of the many files is not done in one Thread (ex. multiple Applications writing to the same directory), AbstractWatchKey will only increase modification count if the context that is changing is the same as the last added to the events-lists. If i am right, the order of the events is not guaranteed. So AbstractWatchKey could check the whole events-list and increase the modification count. This can only be done if we can find a quick implementation. I got some ideas but not tried it yet. I add a Testcase that shows the Problem with multiple Threads. You can change SEQUENCIAL to true to see how it works in an non-threaded Environment. Kind regards Sebastian -- GRATIS f?r alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01 -------------- next part -------------- A non-text attachment was scrubbed... Name: MultipleWriterThreadShouldNotOverflowWatchService.java Type: text/x-java Size: 5072 bytes Desc: not available Url : http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100105/46ce845e/attachment.bin From Christopher.Hegarty at Sun.COM Tue Jan 5 02:39:13 2010 From: Christopher.Hegarty at Sun.COM (Christopher Hegarty -Sun Microsystems Ireland) Date: Tue, 05 Jan 2010 10:39:13 +0000 Subject: Need reviewer for 6913877: (fs) AsynchronousFileChannel.write can return wrong result under load [win] In-Reply-To: <4B4217FB.5060403@sun.com> References: <4B4217FB.5060403@sun.com> Message-ID: <4B4316D1.9070605@sun.com> Alan, This change looks good. I have just one question about the requirement to call Invoker.invoke(result) at the end of the run method of WriteTask. I can see that in the ReadTask case this may be an optimization for EOF, but I don't think it is necessary for write. If you run with system assertions enabled and write throws an Exception then the assertion in Invoker.invoke will be triggered, right? Otherwise looks good. -Chris. Alan Bateman wrote: > > This one is a Windows specific reliability issue with > AsynchronousFileChannel.write. It's the issue that Joel Buckley brought > up on nio-discuss a few days ago. Under load conditions (duplicates for > me intermittently on a 8-core server) it is possible for > AsynchronousFileChannel.write (or read) to complete with the incorrect > result. The issue is similar to 6834246 in that the implementation is > using the number of bytes transferred, as returned by > WriteFile/ReadFile, as the result. This result is only reliable for > synchronous operations so this patch changes the implementation so that > result is always handled via the completion notification. The webrev is > here: > http://cr.openjdk.java.net/~alanb/6913877/webrev.00/ > > Thanks, > Alan. From jdavis at pcprogramming.com Tue Jan 5 02:46:25 2010 From: jdavis at pcprogramming.com (John Davis) Date: Tue, 5 Jan 2010 04:46:25 -0600 Subject: Repository Message-ID: Is there a git repo where I can see the source? Looking at the GetQueuedCompletionsStatusEx stuff ... Thanks, JD -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100105/fe638a0b/attachment.html From Alan.Bateman at Sun.COM Tue Jan 5 03:01:50 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Tue, 05 Jan 2010 11:01:50 +0000 Subject: WatchService questions In-Reply-To: <20100105095218.78140@gmx.net> References: <4B1A4F80.2070106@xs4all.nl> <4B1AC735.50404@sun.com> <4B1B9B28.9090007@xs4all.nl> <4B1BED7B.40103@sun.com> <20100105095218.78140@gmx.net> Message-ID: <4B431C1E.8020304@sun.com> Sebastian Sickelmann wrote: > : > While reviewing 6907760 i think i found one problem why John is facing so many Events. > > If the Copy of the many files is not done in one Thread (ex. multiple Applications writing to the same directory), AbstractWatchKey will only increase modification count if the context that is changing is the same as the last added to the events-lists. > > If i am right, the order of the events is not guaranteed. So AbstractWatchKey could check the whole events-list and increase the modification count. This can only be done if we can find a quick implementation. I got some ideas but not tried it yet. > > I add a Testcase that shows the Problem with multiple Threads. You can change SEQUENCIAL to true to see how it works in an non-threaded Environment. > > Kind regards > Sebastian > The watch service implementations that makes use of kernel support always process and queues the events in the order that they are received. The polling implementation (used when there isn't kernel support) processes snapshots of the directory and so cannot infer the order of changes between snapshots. It's okay for several entities (maybe threads in the same VM, maybe other programs or processes) to be creating or changing files at the same time. It does reduce the likelihood of repeated modification events to the same file and so the representation will be less efficient. You can't re-order create or delete events as otherwise the application's view of the directory will get out of sync. It may be useful in some cases to know when a file was modified relative to other operations - I think you are proposing to not do this, is that right? I looked at the test you attached but it doesn't seem to demonstrate a problem (or at least I don't see it on the first glance). -Alan. From Alan.Bateman at Sun.COM Tue Jan 5 03:03:32 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Tue, 05 Jan 2010 11:03:32 +0000 Subject: Need reviewer for 6913877: (fs) AsynchronousFileChannel.write can return wrong result under load [win] In-Reply-To: <4B4316D1.9070605@sun.com> References: <4B4217FB.5060403@sun.com> <4B4316D1.9070605@sun.com> Message-ID: <4B431C84.5060007@sun.com> Christopher Hegarty -Sun Microsystems Ireland wrote: > Alan, > > This change looks good. I have just one question about the requirement > to call Invoker.invoke(result) at the end of the run method of > WriteTask. I can see that in the ReadTask case this may be an > optimization for EOF, but I don't think it is necessary for write. If > you run with system assertions enabled and write throws an Exception > then the assertion in Invoker.invoke will be triggered, right? > > Otherwise looks good. > > -Chris. Thanks Chris. If a write operation is initiated specifying a completion handler, and the write operation cannot be initiated then the completion handler's failed method must be invoked. That Invoker.invoke(result) does that (directly or indirectly depending on if it's a pooled-thread or not). -Alan. From Christopher.Hegarty at Sun.COM Tue Jan 5 03:03:28 2010 From: Christopher.Hegarty at Sun.COM (Christopher Hegarty -Sun Microsystems Ireland) Date: Tue, 05 Jan 2010 11:03:28 +0000 Subject: Need reviewer for 6913877: (fs) AsynchronousFileChannel.write can return wrong result under load [win] In-Reply-To: <4B431C84.5060007@sun.com> References: <4B4217FB.5060403@sun.com> <4B4316D1.9070605@sun.com> <4B431C84.5060007@sun.com> Message-ID: <4B431C80.2030409@sun.com> Alan Bateman wrote: > Thanks Chris. If a write operation is initiated specifying a completion > handler, and the write operation cannot be initiated then the completion > handler's failed method must be invoked. That Invoker.invoke(result) > does that (directly or indirectly depending on if it's a pooled-thread > or not). Ah yes, my mistake. I see this now. The change looks fine so. -Chris. > > -Alan. From Alan.Bateman at Sun.COM Tue Jan 5 03:16:38 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Tue, 05 Jan 2010 11:16:38 +0000 Subject: Repository In-Reply-To: References: Message-ID: <4B431F96.6090702@sun.com> John Davis wrote: > Is there a git repo where I can see the source? Looking at the > GetQueuedCompletionsStatusEx stuff ... > > Thanks, > JD > We use Mercurial rather than Git. The jdk master repository is here: http://hg.openjdk.java.net/jdk7/jdk7/jdk For the "GetQueuedCompletionsStatusEx stuff..." you'll probably want to look at http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/tip/src/windows/classes/sun/nio/ch/Iocp.java as that is the code that handles I/O completion events. -Alan. From Christopher.Hegarty at Sun.COM Tue Jan 5 03:17:00 2010 From: Christopher.Hegarty at Sun.COM (Christopher Hegarty -Sun Microsystems Ireland) Date: Tue, 05 Jan 2010 11:17:00 +0000 Subject: Need reviewer for 6907760: (file) OVERFLOW event should cause pending events to be purged In-Reply-To: <4B4204F9.6020800@sun.com> References: <4B4204F9.6020800@sun.com> Message-ID: <4B431FAC.5050203@sun.com> Alan, This change looks like a good optimization. One thing I noticed is that if there are multiple events after an OVERFLOW, then the OVERFLOW count get incremented. I don't think this is a problem, does the count even make sense in this situation? Just a observation. I also noticed the nice use of Objects.equals. Is the file system API backported to 1.6? Will this require a patch to compile/run on 1.6? I think the use of Objects.equals simplifies the logic and I like it. Just another observation. -Chris. Alan Bateman wrote: > This is a follow-up to the discussion on the handling of WatchService > overflow events that John Hendrikx brought up on this mailing list in > December. A suggestion to come from that discussion is that the > implementation can drop pending events when queuing an overflow event. > That will save the application from needlessly processing these events. > Similarly, there isn't any point queuing events while there is an > overflow event pending. The webrev with the proposed changes is here: > http://cr.openjdk.java.net/~alanb/6907760/webrev.00/ > > Thanks, > Alan. From Alan.Bateman at Sun.COM Tue Jan 5 03:38:32 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Tue, 05 Jan 2010 11:38:32 +0000 Subject: Need reviewer for 6907760: (file) OVERFLOW event should cause pending events to be purged In-Reply-To: <4B431FAC.5050203@sun.com> References: <4B4204F9.6020800@sun.com> <4B431FAC.5050203@sun.com> Message-ID: <4B4324B8.9040109@sun.com> Christopher Hegarty -Sun Microsystems Ireland wrote: > Alan, > > This change looks like a good optimization. > > One thing I noticed is that if there are multiple events after an > OVERFLOW, then the OVERFLOW count get incremented. I don't think this > is a problem, does the count even make sense in this situation? Just a > observation. Thanks for reviewing. The incrementing of the overflow count is intentional but not necessary. For debugging purposes it may give a hint that many events are being dropped. > > I also noticed the nice use of Objects.equals. Is the file system API > backported to 1.6? Will this require a patch to compile/run on 1.6? I > think the use of Objects.equals simplifies the logic and I like it. > Just another observation. It's not a goal to run with jdk6 or older releases (and can't of course, without work, because it's a new API and also updates existing APIs). There are other jdk7 dependencies too so I'm not worried about it. -Alan. From sebastian.sickelmann at gmx.de Tue Jan 5 11:43:57 2010 From: sebastian.sickelmann at gmx.de (Sebastian Sickelmann) Date: Tue, 05 Jan 2010 20:43:57 +0100 Subject: WatchService questions In-Reply-To: <4B431C1E.8020304@sun.com> References: <4B1A4F80.2070106@xs4all.nl> <4B1AC735.50404@sun.com> <4B1B9B28.9090007@xs4all.nl> <4B1BED7B.40103@sun.com> <20100105095218.78140@gmx.net> <4B431C1E.8020304@sun.com> Message-ID: <20100105194357.168390@gmx.net> Alan Bateman wrote: > Sebastian Sickelmann wrote: > > : > > While reviewing 6907760 i think i found one problem why John is facing > so many Events. > > > > If the Copy of the many files is not done in one Thread (ex. multiple > Applications writing to the same directory), AbstractWatchKey will only > increase modification count if the context that is changing is the same as the > last added to the events-lists. > > > > If i am right, the order of the events is not guaranteed. So > AbstractWatchKey could check the whole events-list and increase the modification > count. This can only be done if we can find a quick implementation. I got some > ideas but not tried it yet. > > > > I add a Testcase that shows the Problem with multiple Threads. You can > change SEQUENCIAL to true to see how it works in an non-threaded > Environment. > > > > Kind regards > > Sebastian > > > The watch service implementations that makes use of kernel support > always process and queues the events in the order that they are > received. The polling implementation (used when there isn't kernel > support) processes snapshots of the directory and so cannot infer the > order of changes between snapshots. So a user of the WatchService-API cannot count in the order of the events. Or is there a way to ask the FileSystem or the Watch-Service if it makes use of kernel support. Maybe this would be a usefull method(isPolling() or isNative()) in WatchService. Maybe a Method(isOrderGuaranteed) in WatchService to evaluate if the order is guaranteed is usefull too. > It's okay for several entities > (maybe threads in the same VM, maybe other programs or processes) to be > creating or changing files at the same time. It does reduce the > likelihood of repeated modification events to the same file and so the > representation will be less efficient. You can't re-order create or > delete events as otherwise the application's view of the directory will > get out of sync. I have overseen this fact and i think you are right. While searching for the last MODIFY the implementation should stop searching after Events like CREATE and DELETE. Here an example: newest to oldest events. 9- MODIFY File3 8- MODIFY File2 7- CREATE File2 6- CREATE File1 5- DELETE File1 4- MODIFY File1 3- CREATE File3 2- CREATE File2 1- CREATE File1 An Update to File3 should increase the modification count of Event 9 An Update to File2 should increase the modification count of Event 8 An Update to File1 should not increase the modification count of Event 4 because there are CREATE and DELETE Events at Event 5,6 and 7. A Event 10(MODIFY File1) should be inserted instead. > It may be useful in some cases to know when a file was > modified relative to other operations - I think you are proposing to not > do this, is that right? I think that you understand my proposal correct. See Example above. I think also that the relative Position of MODIFY to other operations is interesting, but not to other Events (in particular other MODIFY-Events)on other Files. I can imagine a case where the order of concurrent Modification of different files would be interessting. But only if i get more information(for example the size of change the change, or which attribute has changed) in the modify-event. But i only get the modification count. Example: 4- MODIFY File2 ModCount=3 3- MODIFY File1 ModCount=2 2- MODIFY File2 ModCount=5 1- MODIFY File1 ModCount=7 gives me the same information as 2- MODIFY File2 ModCount=8 1- MODIFY File1 ModCount=9 But 5- MODIFY File2 ModCount=3 4- MODIFY File1 ModCount=2 3- CREATE File3 ModCount=1 2- MODIFY File2 ModCount=5 1- MODIFY File1 ModCount=7 is not the same as 3- CREATE File3 ModCount=1 2- MODIFY File2 ModCount=8 1- MODIFY File1 ModCount=9 > I looked at the test you attached but it doesn't > seem to demonstrate a problem (or at least I don't see it on the first > glance). The test writes 64 Files in parallel(each in 16 Steps). If the order of MODIFY Events on different Files is not relevant(see above), than there should be 128 Events in the Event List (64 CREATE with ModCount 1 and 64 MODIFY with ModCount 16). But because in the actual implementation the order of MODIFY Events on different Files seems to be relevant, it generates many more "unnecessary" Events. This leeds to earlier OVERFLOW Events which forces me to rescan the Directory again. > > -Alan. - Sebastian -- Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 - sicherer, schneller und einfacher! http://portal.gmx.net/de/go/atbrowser From Alan.Bateman at Sun.COM Wed Jan 6 03:51:52 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Wed, 06 Jan 2010 11:51:52 +0000 Subject: WatchService questions In-Reply-To: <20100105194357.168390@gmx.net> References: <4B1A4F80.2070106@xs4all.nl> <4B1AC735.50404@sun.com> <4B1B9B28.9090007@xs4all.nl> <4B1BED7B.40103@sun.com> <20100105095218.78140@gmx.net> <4B431C1E.8020304@sun.com> <20100105194357.168390@gmx.net> Message-ID: <4B447958.5080101@sun.com> Sebastian Sickelmann wrote: > : > So a user of the WatchService-API cannot count in the order of the events. The sequence of events needs to keep the user's view of the directory consistent. This implies an ordering for at least the create and delete events (irrespective of implementation). Your mails do raise the question on if modification events need to be ordered. Clearly there are cases where they should be ordered with respect to other event kinds. If a file is modified after it is created then you should expect the modification event after the create event. There are other examples like this. If several files are modified at around the same time then the ordering may or may not be useful but clearly cannot be guaranteed everywhere. Where it is important then the user can examine the file's modification time if that helps. So I think you have a worthy point and we should think about this case further to see if we should change the implementation. If it is changed then there are several possible approaches - for example modification events could be merged towards the head if the pending list reaches its limit. Alternatively modification events could initially be stored in a map (for fast lookup) and only added to the pending list when an event of a different kind needs to be queued (or the user retrieves the pending events). -Alan. From sebastian.sickelmann at gmx.de Thu Jan 7 11:35:22 2010 From: sebastian.sickelmann at gmx.de (Sebastian Sickelmann) Date: Thu, 07 Jan 2010 20:35:22 +0100 Subject: WatchService questions In-Reply-To: <4B447958.5080101@sun.com> References: <4B1A4F80.2070106@xs4all.nl> <4B1AC735.50404@sun.com> <4B1B9B28.9090007@xs4all.nl> <4B1BED7B.40103@sun.com> <20100105095218.78140@gmx.net> <4B431C1E.8020304@sun.com> <20100105194357.168390@gmx.net> <4B447958.5080101@sun.com> Message-ID: <20100107193522.304800@gmx.net> Alan Bateman wrote: > Sebastian Sickelmann wrote: > > : > > So a user of the WatchService-API cannot count in the order of the > events. > The sequence of events needs to keep the user's view of the directory > consistent. This implies an ordering for at least the create and delete > events (irrespective of implementation). Your mails do raise the > question on if modification events need to be ordered. Clearly there are > cases where they should be ordered with respect to other event kinds. If > a file is modified after it is created then you should expect the > modification event after the create event. There are other examples like > this. If several files are modified at around the same time then the > ordering may or may not be useful but clearly cannot be guaranteed > everywhere. Where it is important then the user can examine the file's > modification time if that helps. So I think you have a worthy point and > we should think about this case further to see if we should change the > implementation. I think so to. When someone gets the following Events that happend at time t the user has no change to exermine the status (ex. Modification-Time, Size) for the Event 2. Because the Event raises at t=20 at he receives this Event-List sometime after t=40. 4- MODIDY F2 modCount=5 at t=40 3- MODIFY F1 modCount=4 at t=30 2- MODIDY F2 modCount=3 at t=20 1- MODIDY F1 modCount=2 at t=10 So you gain no more information that in this example: 4- MODIDY F2 modCount=8 at t=40 t=20 3- MODIFY F1 modCount=6 at t=30 t=10 Naturally t=xyz should not be an information of the events. >If it is changed then there are several possible > approaches - for example modification events could be merged towards the > head if the pending list reaches its limit. Alternatively modification > events could initially be stored in a map (for fast lookup) and only > added to the pending list when an event of a different kind needs to be > queued (or the user retrieves the pending events). This are exactly the 2 Mainimplemetatios i see too. 1. Merging MODIFY-Events and move to Head 2. Merging MODIFY-Events and leave them in there Position. (Searching can be done via HashMap) Implementation 1. Merging MODIFY-Events ======================================= Searching the List in linear Order can lead to fast results, but only if the list is reorderd. The question is: How much the reorderung in the Event-List hurts. In single Threaded modifiying of the watched directory there is a slightly good change to find older MODIFY-Events to the some context very fast at the head of the list. But in Multithreaded uses of the Directory there will only a few hits at the head of the list. The order of the MODIFY-EVENTS of different contexts are than relative to the ==last== Modification of the contexts. Implementation 2. Merging MODIFY-Events and leave them in there Position. (Searching can be done via HashMap) ============================================================================================================= Searching a MODIFY-EVENT in an HashMap should be pretty fast. Incrementation could be done in the found MODIFY-Event. If the top-most Event is a MODIFY-EVENT to the same context, lookup at the head of the list would be faster. At EVENTS like CREATE and DELETE the HashMap should be cleared(Implementation 2A). Alternativ only the MODIFY-EVENTS of the same context can be removed from the Map (Implementation 2B). I think both solutions (Implementation 2A and 2B) are only realy good in multithreaded environments. When there is only one other programm or thread that writes to the watched Directory Implementation 1 will be faster. The order of the MODIFY-EVENTS of different contexts are than relative to the ==first== Modification of the contexts. But what should be done with other Events than StdWatchEventKind? Are there implementation specific EventKinds? And must there be a way(overrideable protected Method) so that implementations can override Merging and Reordering of Events? I have made two(2A,2B) Test-Implemetations and it seems to work and reduce MODIFY-Events. - Sebastian > > -Alan. > -- Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 - sicherer, schneller und einfacher! http://portal.gmx.net/de/go/atbrowser From Alan.Bateman at Sun.COM Thu Jan 7 13:30:46 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Thu, 07 Jan 2010 21:30:46 +0000 Subject: WatchService questions In-Reply-To: <20100107193522.304800@gmx.net> References: <4B1A4F80.2070106@xs4all.nl> <4B1AC735.50404@sun.com> <4B1B9B28.9090007@xs4all.nl> <4B1BED7B.40103@sun.com> <20100105095218.78140@gmx.net> <4B431C1E.8020304@sun.com> <20100105194357.168390@gmx.net> <4B447958.5080101@sun.com> <20100107193522.304800@gmx.net> Message-ID: <4B465286.8040309@sun.com> Sebastian Sickelmann wrote: > I think both solutions (Implementation 2A and 2B) are only realy good in multithreaded environments. When there is only > one other programm or thread that writes to the watched Directory Implementation 1 will be faster. > The order of the MODIFY-EVENTS of different contexts are than relative to the ==first== Modification of the contexts. > If you have cycles to experiment and send results that would be great. Also it's good to think about different scenarios where there is one or several programs changing files. One thing to mention about our implementations is that the 3 native implementations use a single poller thread per watch service. So if you are hacking on AbstractWatchKey.signalEvent then there will be only one producer. The polling implementation could potentially have concurrent threads polling directories but they will be different directories (never the same directory) and events are queued on per watch key basis so again only one producer. > But what should be done with other Events than StdWatchEventKind? > Our implementation does support a non-standard modifier when registering events but it doesn't support any additional events at this time. For now, it is okay to focus only on ENTRY_MODIFY events and assume that you can't reorder other events. -Alan. From java.net at twistedprotein.com Fri Jan 8 05:38:32 2010 From: java.net at twistedprotein.com (Brendon McLean) Date: Fri, 8 Jan 2010 15:38:32 +0200 Subject: Resolving empty paths. Message-ID: <816916EC-F126-4F26-9EE9-FE70923D852A@twistedprotein.com> Hi, I apologise if this has already been discussed, but I could not find a way to search the mailing list, or properly use Google to check for that. I am already using the Paths API in production code (www.mindsilver.com) as I was reluctant to create my own Jetbrains-like VirtualFile type abstraction when a JDK substitute lies around the corner. I pulled the whole API out and have been using about 50% of it very successfully on Mac, Unix and Linux. I plan to revert to your API when JDK7 is released. One issue I run into a lot though, is exceptions caused by an "Empty path". Mostly, this happens when resolving the empty string against an absolute path. I see that subsequent to my branching, there is a null guard which will doubtless save many people, but lots of GUIs, whether web or Swing, will use the empty string instead of a null. Would there be any adverse effects of adding a check for the empty string and simply returning "this" from the resolve method in such a case? i.e. Surely Paths.get("/usr/local").resolve("") should return Paths.get("/usr/local")? Regards, Brendon McLean. Mindsilver Ltd. From Alan.Bateman at Sun.COM Fri Jan 8 06:28:37 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Fri, 08 Jan 2010 14:28:37 +0000 Subject: Resolving empty paths. In-Reply-To: <816916EC-F126-4F26-9EE9-FE70923D852A@twistedprotein.com> References: <816916EC-F126-4F26-9EE9-FE70923D852A@twistedprotein.com> Message-ID: <4B474115.8000807@sun.com> Brendon McLean wrote: > Hi, > > I apologise if this has already been discussed, but I could not find a way to search the mailing list, or properly use Google to check for that. > > I am already using the Paths API in production code (www.mindsilver.com) as I was reluctant to create my own Jetbrains-like VirtualFile type abstraction when a JDK substitute lies around the corner. I pulled the whole API out and have been using about 50% of it very successfully on Mac, Unix and Linux. I plan to revert to your API when JDK7 is released. > Interesting, can you say more about this? Are you using the API and implementation or just the API? > One issue I run into a lot though, is exceptions caused by an "Empty path". Mostly, this happens when resolving the empty string against an absolute path. I see that subsequent to my branching, there is a null guard which will doubtless save many people, but lots of GUIs, whether web or Swing, will use the empty string instead of a null. Would there be any adverse effects of adding a check for the empty string and simply returning "this" from the resolve method in such a case? > > i.e. Surely Paths.get("/usr/local").resolve("") should return Paths.get("/usr/local")? > This probably make sense. We would need to change the spec slightly to allow for this as it is currently specified to convert the given path-string to a Path. There are a bunch of small API changes accumulating so I'll add this one to the list. Just on null. If you invoke resolve(String) with null then it should throw NPE. I suspect your "null guard" comment might relate to resolve(Path) where it should return "this" if invoked with null. This might seem inconsistent but it's to make it easy to use resolve(Path) in conjunction with relativize(Path). -Alan. From java.net at twistedprotein.com Fri Jan 8 06:46:55 2010 From: java.net at twistedprotein.com (Brendon McLean) Date: Fri, 8 Jan 2010 16:46:55 +0200 Subject: Resolving empty paths. In-Reply-To: <4B474115.8000807@sun.com> References: <816916EC-F126-4F26-9EE9-FE70923D852A@twistedprotein.com> <4B474115.8000807@sun.com> Message-ID: <1ACFD075-71EB-4D27-87FE-7B6F69AEA515@twistedprotein.com> Hi Alan, On 8 Jan 2010, at 16:28 , Alan Bateman wrote: > Brendon McLean wrote: >> Hi, >> >> I apologise if this has already been discussed, but I could not find a way to search the mailing list, or properly use Google to check for that. >> >> I am already using the Paths API in production code (www.mindsilver.com) as I was reluctant to create my own Jetbrains-like VirtualFile type abstraction when a JDK substitute lies around the corner. I pulled the whole API out and have been using about 50% of it very successfully on Mac, Unix and Linux. I plan to revert to your API when JDK7 is released. >> > Interesting, can you say more about this? Are you using the API and implementation or just the API? I'm using as much of the implementation that doesn't require native calls. I'm using a polling system instead of the new IO API for the same reason. My primary use of this is to manage to user specified paths, relative or absolute. The API is quite pleasant to use. > >> One issue I run into a lot though, is exceptions caused by an "Empty path". Mostly, this happens when resolving the empty string against an absolute path. I see that subsequent to my branching, there is a null guard which will doubtless save many people, but lots of GUIs, whether web or Swing, will use the empty string instead of a null. Would there be any adverse effects of adding a check for the empty string and simply returning "this" from the resolve method in such a case? >> >> i.e. Surely Paths.get("/usr/local").resolve("") should return Paths.get("/usr/local")? >> > This probably make sense. We would need to change the spec slightly to allow for this as it is currently specified to convert the given path-string to a Path. There are a bunch of small API changes accumulating so I'll add this one to the list. Thanks. It's just one of those use cases I need so often. For example, if a user needs to create a sub path relative some "project path" then I end up code like this: File f = projectPath.resolve(packageDir).resolve(file).toFile(); The empty package strikes again! So obviously, I have to guard against it whenever I find this pattern emerging. > Just on null. If you invoke resolve(String) with null then it should throw NPE. I suspect your "null guard" comment might relate to resolve(Path) where it should return "this" if invoked with null. This might seem inconsistent but it's to make it easy to use resolve(Path) in conjunction with relativize(Path). You are correct. That will teach me to compile code in my head. Brendon. From Alan.Bateman at Sun.COM Fri Jan 8 07:53:35 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Fri, 08 Jan 2010 15:53:35 +0000 Subject: Resolving empty paths. In-Reply-To: <1ACFD075-71EB-4D27-87FE-7B6F69AEA515@twistedprotein.com> References: <816916EC-F126-4F26-9EE9-FE70923D852A@twistedprotein.com> <4B474115.8000807@sun.com> <1ACFD075-71EB-4D27-87FE-7B6F69AEA515@twistedprotein.com> Message-ID: <4B4754FF.5050204@sun.com> Brendon McLean wrote: > : > Thanks. It's just one of those use cases I need so often. For example, if a user needs to create a sub path relative some "project path" then I end up code like this: > > File f = projectPath.resolve(packageDir).resolve(file).toFile(); > > The empty package strikes again! So obviously, I have to guard against it whenever I find this pattern emerging. > Right, in the above then each invocation of resolve could throw InvalidPathException if the input is garbage. Out of curiosity, is toFile() a method that you have added? This doesn't exist in our API as it wouldn't always make sense (for example the Path might locate a file in a custom file system). -Alan. From java.net at twistedprotein.com Sun Jan 10 02:03:21 2010 From: java.net at twistedprotein.com (Brendon McLean) Date: Sun, 10 Jan 2010 12:03:21 +0200 Subject: Resolving empty paths. In-Reply-To: <4B4754FF.5050204@sun.com> References: <816916EC-F126-4F26-9EE9-FE70923D852A@twistedprotein.com> <4B474115.8000807@sun.com> <1ACFD075-71EB-4D27-87FE-7B6F69AEA515@twistedprotein.com> <4B4754FF.5050204@sun.com> Message-ID: <9FD7F18A-63C5-4956-9C7B-ADC13A183083@twistedprotein.com> Alan, I actually don't have a toFile method - I just made it up for the purposes of the example because the way I normally do that is more difficult to read and detracts from the usage of resolve that I was trying to highlight. Sorry for the confusion. I copied this stuff before you added most of the file operation equivalents to the Path class, so I internally convert to File whenever I need to do anything useful. I look forward to not having to do that. Regards, Brendon. On 8 Jan 2010, at 17:53 , Alan Bateman wrote: > Brendon McLean wrote: >> : >> Thanks. It's just one of those use cases I need so often. For example, if a user needs to create a sub path relative some "project path" then I end up code like this: >> >> File f = projectPath.resolve(packageDir).resolve(file).toFile(); >> >> The empty package strikes again! So obviously, I have to guard against it whenever I find this pattern emerging. >> > Right, in the above then each invocation of resolve could throw InvalidPathException if the input is garbage. > > Out of curiosity, is toFile() a method that you have added? This doesn't exist in our API as it wouldn't always make sense (for example the Path might locate a file in a custom file system). > > -Alan. From sebastian.sickelmann at gmx.de Mon Jan 11 13:51:49 2010 From: sebastian.sickelmann at gmx.de (Sebastian Sickelmann) Date: Mon, 11 Jan 2010 22:51:49 +0100 Subject: WatchService questions In-Reply-To: <4B465286.8040309@sun.com> References: <4B1A4F80.2070106@xs4all.nl> <4B1AC735.50404@sun.com> <4B1B9B28.9090007@xs4all.nl> <4B1BED7B.40103@sun.com> <20100105095218.78140@gmx.net> <4B431C1E.8020304@sun.com> <20100105194357.168390@gmx.net> <4B447958.5080101@sun.com> <20100107193522.304800@gmx.net> <4B465286.8040309@sun.com> Message-ID: <20100111215149.181530@gmx.net> Alan Bateman wrote: > Sebastian Sickelmann wrote: > > I think both solutions (Implementation 2A and 2B) are only realy good in > multithreaded environments. When there is only > > one other programm or thread that writes to the watched Directory > Implementation 1 will be faster. > > The order of the MODIFY-EVENTS of different contexts are than relative > to the ==first== Modification of the contexts. > > > If you have cycles to experiment and send results that would be great. > Also it's good to think about different scenarios where there is one or > several programs changing files. One thing to mention about our > implementations is that the 3 native implementations use a single poller > thread per watch service. So if you are hacking on > AbstractWatchKey.signalEvent then there will be only one producer. The > polling implementation could potentially have concurrent threads polling > directories but they will be different directories (never the same > directory) and events are queued on per watch key basis so again only > one producer. I have done the Implementations of 2A and 2B. It based on your suggested Change (6907760) http://cr.openjdk.java.net/~alanb/6907760/webrev.00/raw_files/new/src/share/classes/sun/nio/fs/AbstractWatchKey.java Should i send it via mailing-list? Maybe it is better to append the source to an enhancement-ticket in bugzilla? Btw. is there a way to faster Compile changes in NIO? Calling make in the top-level openjdk-dir leads my to a long wait time. Is there an good document for this. I cant find one!! > > > But what should be done with other Events than StdWatchEventKind? > > > Our implementation does support a non-standard modifier when registering > events but it doesn't support any additional events at this time. For > now, it is okay to focus only on ENTRY_MODIFY events and assume that you > can't reorder other events. OK, so thinks are easier. > > -Alan. > > -- GRATIS f?r alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01 From Alan.Bateman at Sun.COM Mon Jan 11 14:22:51 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Mon, 11 Jan 2010 22:22:51 +0000 Subject: WatchService questions In-Reply-To: <20100111215149.181530@gmx.net> References: <4B1A4F80.2070106@xs4all.nl> <4B1AC735.50404@sun.com> <4B1B9B28.9090007@xs4all.nl> <4B1BED7B.40103@sun.com> <20100105095218.78140@gmx.net> <4B431C1E.8020304@sun.com> <20100105194357.168390@gmx.net> <4B447958.5080101@sun.com> <20100107193522.304800@gmx.net> <4B465286.8040309@sun.com> <20100111215149.181530@gmx.net> Message-ID: <4B4BA4BB.8020601@sun.com> Sebastian Sickelmann wrote: > : > I have done the Implementations of 2A and 2B. It based on your suggested Change (6907760) http://cr.openjdk.java.net/~alanb/6907760/webrev.00/raw_files/new/src/share/classes/sun/nio/fs/AbstractWatchKey.java > > Should i send it via mailing-list? Maybe it is better to append the source to an enhancement-ticket in bugzilla? > Are you signed up as a contributer yet? If not, then the "How to contribute" page [1] is a good start. Normally I would say create a bug in the bugzilla, but I think you are going to propose alternative approaches, to append patch/diffs is okay. > Btw. is there a way to faster Compile changes in NIO? Calling make in > the top-level openjdk-dir leads my to a long wait time. Is there an good document for this. I cant find one If do you a partial build, meaning only the jdk repository, then it is possible to do incremental builds in make/java/nio. -Alan. [1] http://openjdk.java.net/contribute/ From sebastian.sickelmann at gmx.de Tue Jan 12 15:19:12 2010 From: sebastian.sickelmann at gmx.de (Sebastian Sickelmann) Date: Wed, 13 Jan 2010 00:19:12 +0100 Subject: WatchService questions Message-ID: <20100112231912.24670@gmx.net> Alan Bateman wrote: > Sebastian Sickelmann wrote: > > : > > I have done the Implementations of 2A and 2B. It based on your suggested > Change (6907760) > http://cr.openjdk.java.net/~alanb/6907760/webrev.00/raw_files/new/src/share/classes/sun/nio/fs/AbstractWatchKey.java > > > > Should i send it via mailing-list? Maybe it is better to append the > source to an enhancement-ticket in bugzilla? > > > Are you signed up as a contributer yet? If not, then the "How to > contribute" page [1] is a good start. > > Normally I would say create a bug in the bugzilla, but I think you are > going to propose alternative approaches, to append patch/diffs is okay. I add the two diffs to this mail. Tomorrow i will try using nice looking webrev-output. The diffs are based on: [2] > > > Btw. is there a way to faster Compile changes in NIO? Calling make in > > the top-level openjdk-dir leads my to a long wait time. Is there an good > document for this. I cant find one > If do you a partial build, meaning only the jdk repository, then it is > possible to do incremental builds in make/java/nio. I am using JDK_IMPORT_PATH but got problems while building nio only. > > -Alan. > > > [1] http://openjdk.java.net/contribute/ [2] http://cr.openjdk.java.net/~alanb/6907760/webrev.00/raw_files/new/src/share/classes/sun/nio/fs/AbstractWatchKey.java -- Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 - sicherer, schneller und einfacher! http://portal.gmx.net/de/go/atbrowser -------------- next part -------------- A non-text attachment was scrubbed... Name: 2A.diff Type: text/x-patch Size: 1651 bytes Desc: not available Url : http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100113/297f0b4a/attachment.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: 2B.diff Type: text/x-patch Size: 1744 bytes Desc: not available Url : http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100113/297f0b4a/attachment-0001.bin From sebastian.sickelmann at gmx.de Tue Jan 12 15:24:16 2010 From: sebastian.sickelmann at gmx.de (Sebastian Sickelmann) Date: Wed, 13 Jan 2010 00:24:16 +0100 Subject: Reorder modify-events was:WatchService questions Message-ID: <20100112232416.24670@gmx.net> Sebastian Sickelmann wrote: > Alan Bateman wrote: > > Sebastian Sickelmann wrote: > > > : > > > I have done the Implementations of 2A and 2B. It based on your > suggested > > Change (6907760) > > > http://cr.openjdk.java.net/~alanb/6907760/webrev.00/raw_files/new/src/share/classes/sun/nio/fs/AbstractWatchKey.java > > > > > > Should i send it via mailing-list? Maybe it is better to append the > > source to an enhancement-ticket in bugzilla? > > > > > Are you signed up as a contributer yet? If not, then the "How to > > contribute" page [1] is a good start. > > > > Normally I would say create a bug in the bugzilla, but I think you are > > going to propose alternative approaches, to append patch/diffs is okay. > I add the two diffs to this mail. Tomorrow i will try using nice looking > webrev-output. > > The diffs are based on: [2] > > > > > > Btw. is there a way to faster Compile changes in NIO? Calling make in > > > the top-level openjdk-dir leads my to a long wait time. Is there an > good > > document for this. I cant find one > > If do you a partial build, meaning only the jdk repository, then it is > > possible to do incremental builds in make/java/nio. > I am using JDK_IMPORT_PATH but got problems while building nio only. > > > > -Alan. > > > > > > [1] http://openjdk.java.net/contribute/ > [2] > http://cr.openjdk.java.net/~alanb/6907760/webrev.00/raw_files/new/src/share/classes/sun/nio/fs/AbstractWatchKey.java > -- Here are the test-cases. -- GRATIS f?r alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01 -------------- next part -------------- A non-text attachment was scrubbed... Name: NoReorderungAfterCreateOrDeleteEvent2A.java Type: text/x-java Size: 5843 bytes Desc: not available Url : http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100113/adb0dd30/attachment.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: NoReorderungAfterCreateOrDeleteEvent2B.java Type: text/x-java Size: 6360 bytes Desc: not available Url : http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100113/adb0dd30/attachment-0001.bin From Alan.Bateman at Sun.COM Wed Jan 13 10:02:32 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Wed, 13 Jan 2010 18:02:32 +0000 Subject: WatchService questions In-Reply-To: <20100112231912.24670@gmx.net> References: <20100112231912.24670@gmx.net> Message-ID: <4B4E0AB8.2050003@sun.com> Sebastian Sickelmann wrote: > : > I add the two diffs to this mail. Tomorrow i will try using nice looking webrev-output. > > The diffs are based on: [2] > I skimmed through both patches. Looks like they are close with the only difference that the first patch (2A.diff) preserves the ordering of modify events relative to non-modify events for other files. In practical terms there probably isn't too much of a difference but 2B.diff may be slightly more efficient if the user falls way behind and there are lots of files being of changes going on at the same time that lots of files are being created and deleted. One thing that doesn't look right is that pollEvents doesn't seem to be clear the map of modify events - that would be important to avoid incrementing the count on events that have already been retrieved by the user. > : > I am using JDK_IMPORT_PATH but got problems while building nio only. > That's used when you aren't building the entire JDK. Even though you are doing a partial build you will still need to build the entire jdk repository. Once you've got a build then you can do into the make/java/nio directory to do incremental builds of just this area. -Alan. From Thomas.Salter at unisys.com Thu Jan 14 08:46:41 2010 From: Thomas.Salter at unisys.com (Salter, Thomas A) Date: Thu, 14 Jan 2010 10:46:41 -0600 Subject: copyTo/moveTo overwrite Message-ID: When doing a copyTo or moveTo with different providers, it looks like AbstractPath.copyToForeignTarget will overwrite the target file even if REPLACE_EXISTING is not set. It correctly deletes the target only if REPLACE_EXISTING is set but then later in copyRegularFileToForeignTarget calls newByteChannel with CREATE, so any existing file is overwritten. It should have used CREATE_NEW instead of CREATE. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100114/52cb771e/attachment.html From Alan.Bateman at Sun.COM Thu Jan 14 12:00:12 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Thu, 14 Jan 2010 20:00:12 +0000 Subject: copyTo/moveTo overwrite In-Reply-To: References: Message-ID: <4B4F77CC.8050303@sun.com> Salter, Thomas A wrote: > When doing a copyTo or moveTo with different providers, it looks like > AbstractPath.copyToForeignTarget will overwrite the target file even > if REPLACE_EXISTING is not set. It correctly deletes the target only > if REPLACE_EXISTING is set but then later in > copyRegularFileToForeignTarget calls newByteChannel with CREATE, so > any existing file is overwritten. It should have used CREATE_NEW > instead of CREATE. > You are right! When copying or moving a file to a location that is associated with some other provider then it shouldn't replace an existing file unless the REPLACE_EXISTING option is present. Thanks for finding this. Our of curiosity, did you find this by inspection or were you testing with your own provider? Our existing tests doesn't exercise this scenario (not yet anyway). -Alan. From Thomas.Salter at unisys.com Thu Jan 14 12:19:37 2010 From: Thomas.Salter at unisys.com (Salter, Thomas A) Date: Thu, 14 Jan 2010 14:19:37 -0600 Subject: copyTo/moveTo overwrite In-Reply-To: <4B4F77CC.8050303@sun.com> References: <4B4F77CC.8050303@sun.com> Message-ID: I found it through testing. We're porting to a system with where the file system is essentially two independent file systems (one Windows, one legacy). Copying within a file system works with the platform's CopyFile/MoveFile functions. Copying between the file systems is equivalent to copying between providers, so I borrowed the methods in AbstractPath. This would be much easier, by the way, if copyToForeignTarget and convertMoveToCopyOptions were protected rather than private in AbstractPath. -----Original Message----- From: Alan.Bateman at Sun.COM [mailto:Alan.Bateman at Sun.COM] Sent: Thursday, January 14, 2010 3:00 PM To: Salter, Thomas A Cc: nio-dev at openjdk.java.net Subject: Re: copyTo/moveTo overwrite Salter, Thomas A wrote: > When doing a copyTo or moveTo with different providers, it looks like > AbstractPath.copyToForeignTarget will overwrite the target file even > if REPLACE_EXISTING is not set. It correctly deletes the target only > if REPLACE_EXISTING is set but then later in > copyRegularFileToForeignTarget calls newByteChannel with CREATE, so > any existing file is overwritten. It should have used CREATE_NEW > instead of CREATE. > You are right! When copying or moving a file to a location that is associated with some other provider then it shouldn't replace an existing file unless the REPLACE_EXISTING option is present. Thanks for finding this. Our of curiosity, did you find this by inspection or were you testing with your own provider? Our existing tests doesn't exercise this scenario (not yet anyway). -Alan. From sebastian.sickelmann at gmx.de Sat Jan 16 12:47:51 2010 From: sebastian.sickelmann at gmx.de (Sebastian Sickelmann) Date: Sat, 16 Jan 2010 21:47:51 +0100 Subject: Which forest to clone? Message-ID: <20100116204751.113770@gmx.net> Hi, while reading: - http://openjdk.java.net/guide/index.html i found some repositories at: - http://hg.openjdk.java.net at the NIO project-page: - http://openjdk.java.net/projects/nio/ there is a link to the source-repository: - http://hg.openjdk.java.net/nio/nio but it is 6 weeks old. (subproject jdk) In the repository: - http://hg.openjdk.java.net/jdk7/tl (subproject jdk) i found a new (7 days) NIO-changesets (changeset 2050 755dd6bdccca) i want to build on my changesets. should i now clone from forest: - http://hg.openjdk.java.net/nio/nio or - http://hg.openjdk.java.net/jdk7/tl to create changesets? Kind regards Sebastian -- GRATIS f?r alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01 From Alan.Bateman at Sun.COM Sat Jan 16 13:27:03 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Sat, 16 Jan 2010 21:27:03 +0000 Subject: Which forest to clone? In-Reply-To: <20100116204751.113770@gmx.net> References: <20100116204751.113770@gmx.net> Message-ID: <4B522F27.3050300@sun.com> Sebastian Sickelmann wrote: > Hi, > > while reading: > - http://openjdk.java.net/guide/index.html > > i found some repositories at: > - http://hg.openjdk.java.net > > at the NIO project-page: > - http://openjdk.java.net/projects/nio/ > > there is a link to the source-repository: > - http://hg.openjdk.java.net/nio/nio > > but it is 6 weeks old. (subproject jdk) > > In the repository: > - http://hg.openjdk.java.net/jdk7/tl (subproject jdk) > > i found a new (7 days) NIO-changesets (changeset 2050 755dd6bdccca) > i want to build on my changesets. > > should i now clone from forest: > - http://hg.openjdk.java.net/nio/nio > or > - http://hg.openjdk.java.net/jdk7/tl > > to create changesets? > > Kind regards > Sebastian > The nio/nio/jdk repository is where the development was originally hosted before it was integrated into jdk7. It's not used much now which is why it isn't sync'ed too often. The reason we keep it around is because it has a number of experimental changes that aren't in jdk7. I've just edited the page to hopefully make that a bit clearer. It's probably best if you clone jdk7/tl - that's the forest where all fixes/changes to the core libraries are pushed. That's the reason why you see 755dd6bdccca in that repository. It will be pushed to the jdk7 master at the next integration slot. -Alan. From Alan.Bateman at Sun.COM Sun Jan 17 04:32:18 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Sun, 17 Jan 2010 12:32:18 +0000 Subject: Need reviewer for 6917021: (file) copyTo/moveTo can overrwrite existing file when target associated with custom provider Message-ID: <4B530352.2050708@sun.com> When the copyTo/moveTo methods are invoked with a target that is associated with a custom file system then a simple copy/move implementation is used rather than the implementation for the platform. Thomas Salter pointed out a bug in this code whereby it could override a target file even when the REPLACE_EXISTING option is not used. The bug highlights that the current copy/move tests don't exercise the foreign provider case. The patch proposed extends the existing test for this case. In addition to the issue that Thomas ran into, the tests exposed two minor issues where the wrong exception is thrown. The webrev with the changes is here: http://cr.openjdk.java.net/~alanb/6917021/webrev/ Thanks, Alan. From Christopher.Hegarty at Sun.COM Mon Jan 18 06:51:06 2010 From: Christopher.Hegarty at Sun.COM (Christopher Hegarty -Sun Microsystems Ireland) Date: Mon, 18 Jan 2010 14:51:06 +0000 Subject: Need reviewer for 6917021: (file) copyTo/moveTo can overrwrite existing file when target associated with custom provider In-Reply-To: <4B530352.2050708@sun.com> References: <4B530352.2050708@sun.com> Message-ID: <4B54755A.4070009@sun.com> Alan Bateman wrote: > When the copyTo/moveTo methods are invoked with a target that is > associated with a custom file system then a simple copy/move > implementation is used rather than the implementation for the platform. > Thomas Salter pointed out a bug in this code whereby it could override a > target file even when the REPLACE_EXISTING option is not used. The bug > highlights that the current copy/move tests don't exercise the foreign > provider case. The patch proposed extends the existing test for this > case. In addition to the issue that Thomas ran into, the tests exposed > two minor issues where the wrong exception is thrown. > > The webrev with the changes is here: > http://cr.openjdk.java.net/~alanb/6917021/webrev/ These changes look fine. Although, I'll admit I just scanned the test changes. -Chris. > > Thanks, > Alan. From Alan.Bateman at Sun.COM Mon Jan 18 07:33:36 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Mon, 18 Jan 2010 15:33:36 +0000 Subject: Need reviewer for 6917021: (file) copyTo/moveTo can overrwrite existing file when target associated with custom provider In-Reply-To: <4B54755A.4070009@sun.com> References: <4B530352.2050708@sun.com> <4B54755A.4070009@sun.com> Message-ID: <4B547F50.2090803@sun.com> Christopher Hegarty -Sun Microsystems Ireland wrote: > : > These changes look fine. Although, I'll admit I just scanned the test > changes. Thanks Chris - the test update might seem large but it's just extending the existing test so that the tests are re-run with the target associated with a custom provider. -Alan. From sebastian.sickelmann at gmx.de Tue Jan 19 11:02:00 2010 From: sebastian.sickelmann at gmx.de (Sebastian Sickelmann) Date: Tue, 19 Jan 2010 20:02:00 +0100 Subject: WatchService questions Message-ID: <20100119190200.117690@gmx.net> Hi, in the attachment you can find the two Mercurial Exports. (Version 2A and Version2B) > Alan Bateman wrote: > I skimmed through both patches. Looks like they are close with the only > difference that the first patch (2A.diff) preserves the ordering of > modify events relative to non-modify events for other files. In > practical terms there probably isn't too much of a difference but > 2B.diff may be slightly more efficient if the user falls way behind and > there are lots of files being of changes going on at the same time that > lots of files are being created and deleted. I have found an additional interesting thing in WatchService. What are those Modifiers by the registering for. Should there be a Modifier to turn on the merging of MODIFY-Events? Maybe the Level of Merging (Version2A and Version2B)? It looks to me that the SensitivityModfies does some comparable feature than merging(reducing) MODIFY-Events. > > One thing that doesn't look right is that pollEvents doesn't seem to be > clear the map of modify events - that would be important to avoid > incrementing the count on events that have already been retrieved by the > user. Thanks, you are totally right. This is corrected in the diffs and the testcases. -- GRATIS f?r alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01 -------------- next part -------------- A non-text attachment was scrubbed... Name: hg.WatchServer.2A.export Type: application/octet-stream Size: 19527 bytes Desc: not available Url : http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100119/f7c1a3d2/attachment-0002.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: hg.WatchServer.2B.export Type: application/octet-stream Size: 20235 bytes Desc: not available Url : http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100119/f7c1a3d2/attachment-0003.obj From Alan.Bateman at Sun.COM Wed Jan 20 12:44:45 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Wed, 20 Jan 2010 20:44:45 +0000 Subject: WatchService questions In-Reply-To: <20100119190200.117690@gmx.net> References: <20100119190200.117690@gmx.net> Message-ID: <4B576B3D.1020508@sun.com> Sebastian Sickelmann wrote: > Hi, > > in the attachment you can find the two Mercurial Exports. (Version 2A and Version2B) > Thanks for updating the patches to ensure that the count isn't updated after the events have been retrieved. The approach in Version2B is probably better - as I mentioned in the previous mail, it will likely be a bit more efficient for the case that the user falls very behind, and there is a lot of activity, meaning many files being modified at around the same time that files are being created and deleted. BTW, did you look at that link I sent you on the contributor agreement? : > I have found an additional interesting thing in WatchService. What are those Modifiers by the registering for. Should there be a Modifier to turn on the merging of MODIFY-Events? > Maybe the Level of Merging (Version2A and Version2B)? > It looks to me that the SensitivityModfies does some comparable feature than merging(reducing) MODIFY-Events. > Modifiers are just an extension for the future so that we have a way to change how objects are registered. The API doesn't currently define any standard modifiers but you'll find two provider specific modifiers in the implementation. The modifier for the "sensitivity level" that you found is only interesting to the polling implementation (it is ignored by the other watch service implementations). If directories aren't polled very frequently then it's a bit like merging events although I'm not sure that it's worth adding a new modifier for this. -Alan. From uckelman at nomic.net Mon Jan 25 12:10:29 2010 From: uckelman at nomic.net (Joel Uckelman) Date: Mon, 25 Jan 2010 21:10:29 +0100 Subject: zipfs and write support Message-ID: <20100125201029.452291007A@charybdis.ellipsis.cx> I have what I believe is a functioning thread-safe read-write zipfs implementation now---at least it passes all the tests I wrote--- subclassed from the existing zipfs classes. My question now is whether it would be better to fold that code into the original read-only zipfs classes, or to keep the read-only and read-write versions separate. One thing I don't know is whether there will be much of a performance hit in using the read-write version for reading only. I implemented locking in the simplest possible way, with a ReadWriteLock, so it supports shared reads but exclusive writes. As a result, if you're only reading, there should never be any lock contention, so I don't expect this to cause much overhead. (The way which would permit for greater concurrency, locking on subtrees instead of having one lock for the whole ZIP archive, is a bit more work, so I held off on that for now.) Thoughts? -- J. From uckelman at nomic.net Mon Jan 25 12:36:26 2010 From: uckelman at nomic.net (Joel Uckelman) Date: Mon, 25 Jan 2010 21:36:26 +0100 Subject: getting known keys from an AttributeView Message-ID: <20100125203626.29ED31007A@charybdis.ellipsis.cx> I've noticed that there seems to be no way to implement Path.readAttributes() without hardcoding all of the known attribute keys. For example, if you receive "foo:*" as an argument, then the docs require that you return a Map containing all of the FooAttributeView keys and their values. I see no programmatic way to find out what keys FooAttributeView supports, so the only thing I can do is build the map by hand, pair by pair. This is rather inconvenient, and generates a lot of boilerplate code. Have I missed something? It would be helpful if AttributeView had a keys() method which returned a String[] of the keys it supports via Path.getAttribute(). -- J. From Alan.Bateman at Sun.COM Wed Jan 27 02:01:03 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Wed, 27 Jan 2010 10:01:03 +0000 Subject: zipfs and write support In-Reply-To: <20100125201029.452291007A@charybdis.ellipsis.cx> References: <20100125201029.452291007A@charybdis.ellipsis.cx> Message-ID: <4B600EDF.40701@sun.com> Joel Uckelman wrote: > I have what I believe is a functioning thread-safe read-write zipfs > implementation now---at least it passes all the tests I wrote--- > subclassed from the existing zipfs classes. My question now is whether > it would be better to fold that code into the original read-only zipfs > classes, or to keep the read-only and read-write versions separate. > > One thing I don't know is whether there will be much of a performance > hit in using the read-write version for reading only. I implemented > locking in the simplest possible way, with a ReadWriteLock, so it > supports shared reads but exclusive writes. As a result, if you're > only reading, there should never be any lock contention, so I don't > expect this to cause much overhead. (The way which would permit for > greater concurrency, locking on subtrees instead of having one lock > for the whole ZIP archive, is a bit more work, so I held off on that > for now.) > > Thoughts? > It sounds like you've made good progress. If it extends the existing zip provider then I assume it must still be based on java.util.zip, right? Have you given any thought to access from others VM or applications? It's hard to comment on the locking without more information - if there a file opened for writing then does it block all other (read) access to the "file system"? -Alan. From Alan.Bateman at Sun.COM Wed Jan 27 02:19:18 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Wed, 27 Jan 2010 10:19:18 +0000 Subject: getting known keys from an AttributeView In-Reply-To: <20100125203626.29ED31007A@charybdis.ellipsis.cx> References: <20100125203626.29ED31007A@charybdis.ellipsis.cx> Message-ID: <4B601326.4080605@sun.com> Joel Uckelman wrote: > I've noticed that there seems to be no way to implement Path.readAttributes() > without hardcoding all of the known attribute keys. For example, if you > receive "foo:*" as an argument, then the docs require that you return a > Map containing all of the FooAttributeView keys and their values. I see no > programmatic way to find out what keys FooAttributeView supports, so the > only thing I can do is build the map by hand, pair by pair. This is rather > inconvenient, and generates a lot of boilerplate code. Have I missed > something? > > It would be helpful if AttributeView had a keys() method which returned a > String[] of the keys it supports via Path.getAttribute(). > > Right, if a provider supports FooAttributeView then it must have semantic knowledge of all attributes that the view defines (the exception being UserDeifnedFileAttributeView where the values are opaque). For dynamic access, where the user chooses to identity the attributes by name, then it does mean that the provider have some way to map the names to specific attributes (mundane, but not hard). I'm curious how the suggested keys method improves things. It just avoids helps the provider map "*" to the list of attributes names but the provider must still have semantic knowledge of each of the attributes, and know each of their names. -Alan. From uckelman at nomic.net Wed Jan 27 03:13:25 2010 From: uckelman at nomic.net (Joel Uckelman) Date: Wed, 27 Jan 2010 12:13:25 +0100 Subject: zipfs and write support In-Reply-To: <4B600EDF.40701@sun.com> References: <20100125201029.452291007A@charybdis.ellipsis.cx> <4B600EDF.40701@sun.com> Message-ID: <20100127111325.C26A21007A@charybdis.ellipsis.cx> Thus spake Alan Bateman: > > It sounds like you've made good progress. If it extends the existing zip > provider then I assume it must still be based on java.util.zip, right? Yes, it is. I'm not terribly interested in rewriting the low-level ZIP file twiddling stuff myself. > Have you given any thought to access from others VM or applications? I don't know of any way to stop other proceses from corrupting the underlying ZIP file. Do you? I thought file locks were advistory on many OSes. I'm not familiar with what would need to be done to coordinate across VMs. Is there a compelling reason to do this? It's an unusual requirement. > It's hard to comment on the locking without more information - if there > a file opened for writing then does it block all other (read) access to > the "file system"? Yes, but certain kinds of writes will be nearly instant because they don't involve writing any data, e.g., deleting files or moving files which were not originally in the ZIP archive. You could tie up the whole archive by getting an OutputStream and hanging on onto it. It would be nicer to have subtree locking instead. -- J. From uckelman at nomic.net Wed Jan 27 03:20:03 2010 From: uckelman at nomic.net (Joel Uckelman) Date: Wed, 27 Jan 2010 12:20:03 +0100 Subject: getting known keys from an AttributeView In-Reply-To: <4B601326.4080605@sun.com> References: <20100125203626.29ED31007A@charybdis.ellipsis.cx> <4B601326.4080605@sun.com> Message-ID: <20100127112004.4EA0D1007A@charybdis.ellipsis.cx> Thus spake Alan Bateman: > > Right, if a provider supports FooAttributeView then it must have > semantic knowledge of all attributes that the view defines (the > exception being UserDeifnedFileAttributeView where the values are > opaque). For dynamic access, where the user chooses to identity the > attributes by name, then it does mean that the provider have some way to > map the names to specific attributes (mundane, but not hard). I'm > curious how the suggested keys method improves things. It just avoids > helps the provider map "*" to the list of attributes names but the > provider must still have semantic knowledge of each of the attributes, > and know each of their names. > > -Alan. If you have a way to get the attrib names programatically, then Path.readAttributes() only needs contain code for parsing the attribute string, and can be basically the same across all Path implementations. Without it, you end up having to list out all of the possible attributes, which means that you're now maintaining this list in multiple places. (For example, the names from BasicFileAttributeView will be repeated in every readAttributes() implementation.) This is error-prone. -- J. From Alan.Bateman at Sun.COM Wed Jan 27 06:04:03 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Wed, 27 Jan 2010 14:04:03 +0000 Subject: getting known keys from an AttributeView In-Reply-To: <20100127112004.4EA0D1007A@charybdis.ellipsis.cx> References: <20100125203626.29ED31007A@charybdis.ellipsis.cx> <4B601326.4080605@sun.com> <20100127112004.4EA0D1007A@charybdis.ellipsis.cx> Message-ID: <4B6047D3.50208@sun.com> Joel Uckelman wrote: > : > If you have a way to get the attrib names programatically, then > Path.readAttributes() only needs contain code for parsing the attribute > string, and can be basically the same across all Path implementations. > Without it, you end up having to list out all of the possible attributes, > which means that you're now maintaining this list in multiple places. > (For example, the names from BasicFileAttributeView will be repeated in > every readAttributes() implementation.) This is error-prone > For now, we don't have any base classes for provider implementers. It is something we could examine in the future once there is more experience developing custom providers. BasicFileAttributeView is required to be supported by all providers so a case could be made to define a base implementation class in the spi package that concrete providers could extend. That base class could implement set/get/readAttributes, which I think it what you are suggesting. In our default provider we have sun.nio.fs.AbstractBasicFileAttribtueView for this purpose but that isn't useful to other providers. -Alan. From sebastian.sickelmann at gmx.de Wed Jan 27 12:36:02 2010 From: sebastian.sickelmann at gmx.de (Sebastian Sickelmann) Date: Wed, 27 Jan 2010 21:36:02 +0100 Subject: WatchService questions In-Reply-To: <4B576B3D.1020508@sun.com> References: <20100119190200.117690@gmx.net> <4B576B3D.1020508@sun.com> Message-ID: <20100127203602.147290@gmx.net> > Alan Bateman wrote: > Sebastian Sickelmann wrote: > > Hi, > > > > in the attachment you can find the two Mercurial Exports. (Version 2A > and Version2B) > > > Thanks for updating the patches to ensure that the count isn't updated > after the events have been retrieved. The approach in Version2B is > probably better - as I mentioned in the previous mail, it will likely be > a bit more efficient for the case that the user falls very behind, and > there is a lot of activity, meaning many files being modified at around > the same time that files are being created and deleted. > > BTW, did you look at that link I sent you on the contributor agreement? Yes, i have signed it and send it in. But did no response on that yet. Should i get a response? > > : > > I have found an additional interesting thing in WatchService. What are > those Modifiers by the registering for. Should there be a Modifier to turn > on the merging of MODIFY-Events? > > Maybe the Level of Merging (Version2A and Version2B)? > > It looks to me that the SensitivityModfies does some comparable feature > than merging(reducing) MODIFY-Events. > > > Modifiers are just an extension for the future so that we have a way to > change how objects are registered. The API doesn't currently define any > standard modifiers but you'll find two provider specific modifiers in > the implementation. The modifier for the "sensitivity level" that you > found is only interesting to the polling implementation (it is ignored > by the other watch service implementations). If directories aren't > polled very frequently then it's a bit like merging events although I'm > not sure that it's worth adding a new modifier for this. I am trying to create an implementation that lets you choose between Version 2A and 2B or just the "old" one. But this will take some time, i think am be back on this at the middle of February. But for now what would be the next step to make progess for the actual send in patches? Public Review? - Sebastian -- Nur noch bis 31.01.2010: DSL-Komplettpaket f?r 16,99 Euro/mtl.!* http://portal.gmx.net/de/go/dsl02 From Alan.Bateman at Sun.COM Thu Jan 28 00:46:05 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Thu, 28 Jan 2010 08:46:05 +0000 Subject: zipfs and write support In-Reply-To: <20100127111325.C26A21007A@charybdis.ellipsis.cx> References: <20100125201029.452291007A@charybdis.ellipsis.cx> <4B600EDF.40701@sun.com> <20100127111325.C26A21007A@charybdis.ellipsis.cx> Message-ID: <4B614ECD.1090808@sun.com> Joel Uckelman wrote: > : > I don't know of any way to stop other proceses from corrupting the > underlying ZIP file. Do you? I thought file locks were advistory on > many OSes. I'm not familiar with what would need to be done to > coordinate across VMs. Is there a compelling reason to do this? It's > an unusual requirement. > Yes, to be portable, it is best to assume the file locking is advisory. But, file locking isn't going to help when you are replacing the zip file (which I assume is going to happen as it is based on java.util.zip). An approach that I think I've suggested before is to use a "server" to coordinate access. That would allow applications in different VMs to access the "file system" concurrently. It could uses the default file system as a temporary location for updates and re-build the the zip file file when the file system is closed or the server is shutdown. Clearly this is would be a more ambitious project and I'm just mentioning it as a potential solution that someone might like to pursue. -Alan. From Alan.Bateman at Sun.COM Thu Jan 28 01:17:52 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Thu, 28 Jan 2010 09:17:52 +0000 Subject: WatchService questions In-Reply-To: <20100127203602.147290@gmx.net> References: <20100119190200.117690@gmx.net> <4B576B3D.1020508@sun.com> <20100127203602.147290@gmx.net> Message-ID: <4B615640.5060606@sun.com> Sebastian Sickelmann wrote: > : > Yes, i have signed it and send it in. But did no response on that yet. Should i get a response? > I checked, and it has been received , but just hasn't been "processed" yet. > I am trying to create an implementation that lets you choose between Version 2A and 2B or just the "old" one. But this will take some time, i think am be back on this at the middle of February. > > But for now what would be the next step to make progess for the actual send in patches? Public Review? > Sending patches to this mail list is fine. I can sponsor it once we've agreed the patch. -Alan. From uckelman at nomic.net Thu Jan 28 01:27:49 2010 From: uckelman at nomic.net (Joel Uckelman) Date: Thu, 28 Jan 2010 10:27:49 +0100 Subject: zipfs and write support In-Reply-To: <4B600EDF.40701@sun.com> References: <20100125201029.452291007A@charybdis.ellipsis.cx> <4B600EDF.40701@sun.com> Message-ID: <20100128092749.C8BB610086@charybdis.ellipsis.cx> Thus spake Alan Bateman: > > It sounds like you've made good progress. If it extends the existing zip > provider then I assume it must still be based on java.util.zip, right? > Have you given any thought to access from others VM or applications? > It's hard to comment on the locking without more information - if there > a file opened for writing then does it block all other (read) access to > the "file system"? > > -Alan. It dawned on me just now that subtree locking isn't necessary for handling nonoverlapping writes here, because you can neither delete nor move nonempty directories with the Path API---anything whch can be written to is a leaf. Per-Path locking will be easy to do, I might even finish it this evening. -- J. From gnu_andrew at member.fsf.org Thu Jan 28 14:46:50 2010 From: gnu_andrew at member.fsf.org (Andrew John Hughes) Date: Thu, 28 Jan 2010 22:46:50 +0000 Subject: -Werror in make/java/sun_nio/Makefile Message-ID: <17c6771e1001281446i6bf5e070w332d5ea1942f72@mail.gmail.com> I decided to do a build with JAVAC_MAX_WARNINGS=true just to see the lay of the land, and it seems that this causes the sun_nio build to bail out. The Makefile adds -Werror itself locally: OTHER_JAVACFLAGS += -Xlint:serial -Werror which works fine with the default settings but causes to build to fail when -Xlint:all is added: /usr/lib/jvm/icedtea6/bin/java -XX:-PrintVMOptions -XX:+UnlockDiagnosticVMOptions -XX:-LogVMOutput -Xmx896m -Xms128m -XX:PermSize=32m -XX:MaxPermSize=160m -Xbootclasspath/p:/mnt/builder/icedtea/langtools/dist/bootstrap/lib/javac.jar -jar /mnt/builder/icedtea/langtools/dist/bootstrap/lib/javac.jar -Xlint:all -source 7 -target 7 -encoding ascii -Xbootclasspath:/mnt/builder/icedtea/classes -Xlint:serial -Werror -sourcepath /mnt/builder/icedtea/gensrc:../../../src/solaris/classes:../../../src/share/classes -d /mnt/builder/icedtea/classes @/mnt/builder/icedtea/tmp/sun/sun.nio/.classes.list.filtered There is a similar flag for -Werror (JAVAC_WARNINGS_FATAL) and this should be used when -Werror builds are required. The warnings that cause the build to fail are all to do with deprecated code e.g.: ../../../src/share/classes/sun/io/ByteToCharASCII.java:33: warning: [deprecation] ByteToCharConverter in sun.io has been deprecated public class ByteToCharASCII extends ByteToCharConverter { ^ ../../../src/share/classes/sun/io/ByteToCharASCII.java:52: warning: [deprecation] ConversionBufferFullException in sun.io has been deprecated throws ConversionBufferFullException, UnknownCharacterException We hit the 100 warning limit so I don't know how many there are in total. It seems that ByteToCharConverter was marked as deprecated with the intention of removing it in JDK6: * @deprecated Replaced by {@link java.nio.charset}. THIS API WILL BE * REMOVED IN J2SE 1.6. This still being present in OpenJDK7 suggests that never happened. I'd suggest we either drop the local -Werror or add -deprecation after serial to make the build with maximum warnings complete. Long term, we can look at getting rid of that code if appropriate, though I doubt we'll ever be able to do a build with the deprecation warning and -Werror on simply because of legacy code. -- Andrew :-) Free Java Software Engineer Red Hat, Inc. (http://www.redhat.com) Support Free Java! Contribute to GNU Classpath and the OpenJDK http://www.gnu.org/software/classpath http://openjdk.java.net PGP Key: 94EFD9D8 (http://subkeys.pgp.net) Fingerprint: F8EF F1EA 401E 2E60 15FA 7927 142C 2591 94EF D9D8 From Alan.Bateman at Sun.COM Thu Jan 28 15:15:12 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Thu, 28 Jan 2010 23:15:12 +0000 Subject: -Werror in make/java/sun_nio/Makefile In-Reply-To: <17c6771e1001281446i6bf5e070w332d5ea1942f72@mail.gmail.com> References: <17c6771e1001281446i6bf5e070w332d5ea1942f72@mail.gmail.com> Message-ID: <4B621A80.9060402@sun.com> Andrew John Hughes wrote: > : > > It seems that ByteToCharConverter was marked as deprecated with the > intention of removing it in JDK6: > > * @deprecated Replaced by {@link java.nio.charset}. THIS API WILL BE > * REMOVED IN J2SE 1.6. > > This still being present in OpenJDK7 suggests that never happened. > Unfortunately we haven't been able to remove the legacy converters as planned. JDBC drivers are the main problem. Sherman and Martin may want to chime in with more history on this. > I'd suggest we either drop the local -Werror or add -deprecation after > serial to make the build with maximum warnings complete. Long term, > we can look at getting rid of that code if appropriate, though I doubt > we'll ever be able to do a build with the deprecation warning and > -Werror on simply because of legacy code. > I'd prefer not drop -Werror as that would let warnings creep in. Add -deprecation is probably better. -Alan. From uckelman at nomic.net Fri Jan 29 05:11:16 2010 From: uckelman at nomic.net (Joel Uckelman) Date: Fri, 29 Jan 2010 14:11:16 +0100 Subject: zipfs and write support In-Reply-To: <4B614ECD.1090808@sun.com> References: <20100125201029.452291007A@charybdis.ellipsis.cx> <4B600EDF.40701@sun.com> <20100127111325.C26A21007A@charybdis.ellipsis.cx> <4B614ECD.1090808@sun.com> Message-ID: <20100129131117.0D06710076@charybdis.ellipsis.cx> Thus spake Alan Bateman: > > Yes, to be portable, it is best to assume the file locking is advisory. > But, file locking isn't going to help when you are replacing the zip > file (which I assume is going to happen as it is based on > java.util.zip). An approach that I think I've suggested before is to use > a "server" to coordinate access. That would allow applications in > different VMs to access the "file system" concurrently. It could uses > the default file system as a temporary location for updates and re-build > the the zip file file when the file system is closed or the server is > shutdown. Clearly this is would be a more ambitious project and I'm just > mentioning it as a potential solution that someone might like to pursue. > This sounds like a reasonable approach. But: That's quite some way beyond what my project needs right now, so most likely it won't be me implementing this. -- J. From uckelman at nomic.net Fri Jan 29 11:37:39 2010 From: uckelman at nomic.net (Joel Uckelman) Date: Fri, 29 Jan 2010 20:37:39 +0100 Subject: zipfs patches Message-ID: <20100129193739.D2F4010076@charybdis.ellipsis.cx> Two more zipfs patches: * Implemented readAttributes(). * copyTo() should return target, not this. -------------- next part -------------- # HG changeset patch # User Joel Uckelman # Date 1264792812 -3600 # Node ID c057cc149682d4ff84a9aed8e85c0fd8e74b0637 # Parent 36841187ed0717208ca2862bd60d4519d18a9b17 Implemented readAttributes(). diff --git a/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFilePath.java b/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFilePath.java --- a/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFilePath.java +++ b/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFilePath.java @@ -944,10 +944,72 @@ } @Override - public Map readAttributes(String attribute, LinkOption... options) - throws IOException - { - throw new RuntimeException("not implemented"); + public Map readAttributes(String attributes, + LinkOption... options) + throws IOException { + return readAttributes(attributes.split(","), options); + } + + protected Map readAttributes(String[] attributes, + LinkOption... options) + throws IOException { + final Map map = new HashMap(); + + for (String attr : attributes) { + final int colon = attr.indexOf(':'); + + final String view; + final String name; + + if (colon == -1) { + view = "basic"; + name = attr; + } + else { + view = attr.substring(0, colon); + name = attr.substring(colon+1, attr.length()); + } + + if ("*".equals(name)) { + if ("basic".equals(view)) { + map.putAll(readAttributes(new String[] { + "basic:lastModifiedTime", + "basic:lastAccessTime", + "basic:creationTime", + "basic:size", + "basic:isRegularFile", + "basic:isDirectory", + "basic:isSymbolicLink", + "basic:isOther", + "basic:fileKey" + })); + } + else if ("zip".equals(view)) { + map.putAll(readAttributes(new String[] { + "zip:comment", + "zip:compressedSize", + "zip:crc", + "zip:extra", + "zip:method", + "zip:name", + "zip:isArchiveFile", + "zip:versionMadeBy", + "zip:extAttrs" + })); + } + else if ("jar".equals(view)) { + map.putAll(readAttributes(new String[] { + "jar:manifestAttributes", + "jar:entryAttributes" + })); + } + } + else { + map.put(attr, getAttribute(attr)); + } + } + + return map; } @Override -------------- next part -------------- # HG changeset patch # User Joel Uckelman # Date 1264793497 -3600 # Node ID 8709878d43e6b051d6b7280c2cd51ead5de63eaa # Parent c057cc149682d4ff84a9aed8e85c0fd8e74b0637 copyTo() should return target, not this. diff --git a/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFilePath.java b/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFilePath.java --- a/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFilePath.java +++ b/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFilePath.java @@ -1343,7 +1343,7 @@ } } - return this; + return target; } private void copyToTarget(Path target) throws IOException { From gnu_andrew at member.fsf.org Fri Jan 29 18:19:55 2010 From: gnu_andrew at member.fsf.org (Andrew John Hughes) Date: Sat, 30 Jan 2010 02:19:55 +0000 Subject: -Werror in make/java/sun_nio/Makefile In-Reply-To: <4B621A80.9060402@sun.com> References: <17c6771e1001281446i6bf5e070w332d5ea1942f72@mail.gmail.com> <4B621A80.9060402@sun.com> Message-ID: <17c6771e1001291819v4abb2fabkb4bc731664c6df03@mail.gmail.com> On 28 January 2010 23:15, Alan Bateman wrote: > Andrew John Hughes wrote: >> >> : >> >> It seems that ByteToCharConverter was marked as deprecated with the >> intention of removing it in JDK6: >> >> ?* @deprecated Replaced by {@link java.nio.charset}. ?THIS API WILL BE >> ?* REMOVED IN J2SE 1.6. >> >> This still being present in OpenJDK7 suggests that never happened. >> > > Unfortunately we haven't been able to remove the legacy converters as > planned. JDBC drivers are the main problem. Sherman and Martin may want to > chime in with more history on this. > >> I'd suggest we either drop the local -Werror or add -deprecation after >> serial to make the build with maximum warnings complete. ?Long term, >> we can look at getting rid of that code if appropriate, though I doubt >> we'll ever be able to do a build with the deprecation warning and >> -Werror on simply because of legacy code. >> > > I'd prefer not drop -Werror as that would let warnings creep in. Add > -deprecation is probably better. > Ok, that's what I've done in this webrev: http://cr.openjdk.java.net/~andrew/warnings/webrev.01/ It also fixes a number of unchecked and redundant cast warnings that occur when building with JAVAC_MAX_WARNINGS=true. With this patch, sun_io now compiles when JAVAC_MAX_WARNINGS=true. Does this look ok? > -Alan. > -- Andrew :-) Free Java Software Engineer Red Hat, Inc. (http://www.redhat.com) Support Free Java! Contribute to GNU Classpath and the OpenJDK http://www.gnu.org/software/classpath http://openjdk.java.net PGP Key: 94EFD9D8 (http://subkeys.pgp.net) Fingerprint: F8EF F1EA 401E 2E60 15FA 7927 142C 2591 94EF D9D8 From Alan.Bateman at Sun.COM Sat Jan 30 11:51:59 2010 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Sat, 30 Jan 2010 19:51:59 +0000 Subject: zipfs patches In-Reply-To: <20100129193739.D2F4010076@charybdis.ellipsis.cx> References: <20100129193739.D2F4010076@charybdis.ellipsis.cx> Message-ID: <4B648DDF.9030703@sun.com> Joel Uckelman wrote: > Two more zipfs patches: > > * Implemented readAttributes(). > * copyTo() should return target, not this. > > Thanks for taking these bugs. The map key is the attribute name rather than view:attribute and so I assume map.put(attr, ..) should be map.put(name, ...). Another comment is that you've implemented it to use getAttribute, and if I read the patch correctly, it means that a bulk read turns into a sequence of gets where each reads the zip entry. With caching this might not be noticeable but I'm interested to know if you've compared it to reading the zip entry once and then populating the map with the requested attributes. -Alan.