From Alan.Bateman at Sun.COM  Sat Jan  2 03:18:13 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Sat, 02 Jan 2010 11:18:13 +0000
Subject: GetQueueCompletionStatusEx
In-Reply-To: <fb36a6ef0912310903u694a6e79va3811274d28101c9@mail.gmail.com>
References: <fb36a6ef0912290232n7f1fa8e0ge4357475bfaafb15@mail.gmail.com>
	<4B3CD418.8000406@sun.com>
	<fb36a6ef0912310901u24636c92j96cbd23aeddca3a8@mail.gmail.com>
	<fb36a6ef0912310903u694a6e79va3811274d28101c9@mail.gmail.com>
Message-ID: <4B3F2B75.7000206@sun.com>

John Davis wrote:
> Forgot the CC
>
> On Thu, Dec 31, 2009 at 11:01 AM, John Davis <jdavis at pcprogramming.com 
> <mailto:jdavis at pcprogramming.com>> wrote:
>
>     I'm not an expert on the nio internals.  My thinking is it would
>     be nice if everything could be done with a single thread (ie all
>     reads/writes on all handles are async and pumped thru IOCP). 
>     Because the kernel context switch overhead is pretty much
>     eliminated with this call, there is no longer a need to have a
>     thread pool to handle the context switch overhead.  We also
>     wouldn't have the batching dilema with the thread pool gone.  Only
>     problem with this model is that there's no way to guarantee a
>     completion handler won't block.
>
You can use a single thread if you wish but aren't you concerned about 
scaling to multiple sockets and cores? However you are right, if the 
channel group's thread pool  has a single thread, then the completion 
results can be retrieved in batch to avoid a call to 
GetQueueCompletionStatus per I/O.

>     :
>     SetFileCompletionNotificationModes, allows the callback to be
>     invoked immediately, rather than pumping thru IOCP and thread pool.
>
We can certainly try the FILE_SKIP_SET_EVENT_ON_HANDLE mode to see if it 
makes a difference but for now we can't really make use of the 
FILE_SKIP_COMPLETION_PORT_ON_SUCCESS mode. Our implementation can 
actually immediate case we have it disabled at this time. The reason is 
that under load conditions, for loopback connections at least, the 
number of bytes transferred, as returned by WSARecv/WSASend is not 
reliable. We therefore require the completion result so that we can 
update the buffer position and notify the completion handler with the 
correct result.

-Alan.

From Alan.Bateman at Sun.COM  Mon Jan  4 07:10:49 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Mon, 04 Jan 2010 15:10:49 +0000
Subject: Need reviewer for 6907760: (file) OVERFLOW event should cause pending
	events to be purged
Message-ID: <4B4204F9.6020800@sun.com>

This is a follow-up to the discussion on the handling of WatchService 
overflow events that John Hendrikx brought up on this mailing list in 
December. A suggestion to come from that discussion is that the 
implementation can drop pending events when queuing an overflow event. 
That will save the application from needlessly processing these events. 
Similarly, there isn't any point queuing events while there is an 
overflow event pending.  The webrev with the proposed changes is here:
  http://cr.openjdk.java.net/~alanb/6907760/webrev.00/

Thanks,
Alan.

From Alan.Bateman at Sun.COM  Mon Jan  4 08:31:55 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Mon, 04 Jan 2010 16:31:55 +0000
Subject: Need reviewer for 6913877: (fs) AsynchronousFileChannel.write can
	return wrong result under load [win]
Message-ID: <4B4217FB.5060403@sun.com>


This one is a Windows specific reliability issue with 
AsynchronousFileChannel.write. It's the issue that Joel Buckley brought 
up on nio-discuss a few days ago. Under load conditions (duplicates for 
me intermittently on a 8-core server) it is possible for 
AsynchronousFileChannel.write (or read) to complete with the incorrect 
result. The issue is similar to 6834246 in that the implementation is 
using the number of bytes transferred, as returned by 
WriteFile/ReadFile, as the result. This result is only reliable for 
synchronous operations so this patch changes the implementation so that 
result is always handled via the completion notification. The webrev is 
here:
  http://cr.openjdk.java.net/~alanb/6913877/webrev.00/

Thanks,
Alan.

From sebastian.sickelmann at gmx.de  Tue Jan  5 01:52:18 2010
From: sebastian.sickelmann at gmx.de (Sebastian Sickelmann)
Date: Tue, 05 Jan 2010 10:52:18 +0100
Subject: WatchService questions
In-Reply-To: <4B1BED7B.40103@sun.com>
References: <4B1A4F80.2070106@xs4all.nl> <4B1AC735.50404@sun.com>
	<4B1B9B28.9090007@xs4all.nl> <4B1BED7B.40103@sun.com>
Message-ID: <20100105095218.78140@gmx.net>

Alan Bateman wrote:
> John Hendrikx wrote:
> > :
> >
> > What I hadn't realized is that after you receive an OVERFLOW that the 
> > subsequent events will form a cohesive stream of events again which 
> > can be used to update a directory after re-reading it.  It does make 
> > sense to me though to "scan" for the overflow so to speak -- either 
> > that or I would probably expect an implementation of WatchService to 
> > clear all pending events before adding the OVERFLOW.  Not only are the 
> > pending events useless if an overflow occurs, but clearing them would 
> > make room for subsequent events again (if not, you'd be in a permanent 
> > state of overflow...?)
> This is a good point as the watch service could drop all pending events 
> when the limit is reached. Furthermore it can drop all subsequent events 
> while the head is an OVERFLOW event. I'll create a bug to track this - 
> thanks!
> 
> > My original assessment was wrong. The Swing Event thread was not 
> > starved for CPU, but instead was busy handling the WatchService events 
> > in the invokeAndWait part.  This made it seem that the Event thread 
> > was starved for CPU (as other parts of the UI weren't updating 
> > anymore) while it really was consuming the CPU all by itself.
> >
> > In effect, in the time it took me to handle 60 WatchService events, 
> > 513 (512 + Overflow?) more had accumulated, resulting in second long 
> > delays for the UI to become responsive again.
> Thanks for the update. I did a quick test that watches a directory and 
> retrieves all events in a timely manner. Another program creates and 
> deletes 10000 files in the watched directory which I think is what you 
> are doing. The test was on Windows (as I think this is where your UI is 
> at, right?). As expected the watcher didn't consume too many CPU cycles 
> and is barely noticeable. Also, the list of events typically only 
> contain a single event as it can easily keep up with programs creating 
> lots of files (which is a very slow in comparison).  I changed the test 
> to sleep periodically, thus delaying the retrieval of events. In that 
> case, 512 events are queued very quickly, and subsequent events are 
> dropped. Again, the CPU usage is barely noticeable so this concurs with 
> your observation that the time is spent on processing the events.
> 
> Anyway, let us know how your solution goes - your war story may be 
> useful to others.
> 
> -Alan.

While reviewing 6907760 i think i found one problem why John is facing so many Events.

If the Copy of the many files is not done in one Thread (ex. multiple Applications writing to the same directory), AbstractWatchKey will only increase modification count if the context that is changing is the same as the last added to the events-lists.

If i am right, the order of the events is not guaranteed. So AbstractWatchKey could check the whole events-list and increase the modification count. This can only be done if we can find a quick implementation. I got some ideas but not tried it yet.

I add a Testcase that shows the Problem with multiple Threads. You can change SEQUENCIAL to true to see how it works in an non-threaded Environment.

Kind regards
Sebastian
-- 
GRATIS f?r alle GMX-Mitglieder: Die maxdome Movie-FLAT!
Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01
-------------- next part --------------
A non-text attachment was scrubbed...
Name: MultipleWriterThreadShouldNotOverflowWatchService.java
Type: text/x-java
Size: 5072 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100105/46ce845e/attachment.bin 

From Christopher.Hegarty at Sun.COM  Tue Jan  5 02:39:13 2010
From: Christopher.Hegarty at Sun.COM (Christopher Hegarty -Sun Microsystems Ireland)
Date: Tue, 05 Jan 2010 10:39:13 +0000
Subject: Need reviewer for 6913877: (fs) AsynchronousFileChannel.write can
	return wrong result under load [win]
In-Reply-To: <4B4217FB.5060403@sun.com>
References: <4B4217FB.5060403@sun.com>
Message-ID: <4B4316D1.9070605@sun.com>

Alan,

This change looks good. I have just one question about the requirement 
to call Invoker.invoke(result) at the end of the run method of 
WriteTask. I can see that in the ReadTask case this may be an 
optimization for EOF, but I don't think it is necessary for write. If 
you run with system assertions enabled and write throws an Exception 
then the assertion in Invoker.invoke will be triggered, right?

Otherwise looks good.

-Chris.

Alan Bateman wrote:
> 
> This one is a Windows specific reliability issue with 
> AsynchronousFileChannel.write. It's the issue that Joel Buckley brought 
> up on nio-discuss a few days ago. Under load conditions (duplicates for 
> me intermittently on a 8-core server) it is possible for 
> AsynchronousFileChannel.write (or read) to complete with the incorrect 
> result. The issue is similar to 6834246 in that the implementation is 
> using the number of bytes transferred, as returned by 
> WriteFile/ReadFile, as the result. This result is only reliable for 
> synchronous operations so this patch changes the implementation so that 
> result is always handled via the completion notification. The webrev is 
> here:
>  http://cr.openjdk.java.net/~alanb/6913877/webrev.00/
> 
> Thanks,
> Alan.

From jdavis at pcprogramming.com  Tue Jan  5 02:46:25 2010
From: jdavis at pcprogramming.com (John Davis)
Date: Tue, 5 Jan 2010 04:46:25 -0600
Subject: Repository
Message-ID: <fb36a6ef1001050246s5e7eead5ua03d9e03b1bcb996@mail.gmail.com>

Is there a git repo where I can see the source?  Looking at the
GetQueuedCompletionsStatusEx stuff ...

Thanks,
JD
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100105/fe638a0b/attachment.html 

From Alan.Bateman at Sun.COM  Tue Jan  5 03:01:50 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Tue, 05 Jan 2010 11:01:50 +0000
Subject: WatchService questions
In-Reply-To: <20100105095218.78140@gmx.net>
References: <4B1A4F80.2070106@xs4all.nl> <4B1AC735.50404@sun.com>
	<4B1B9B28.9090007@xs4all.nl> <4B1BED7B.40103@sun.com>
	<20100105095218.78140@gmx.net>
Message-ID: <4B431C1E.8020304@sun.com>

Sebastian Sickelmann wrote:
> :
> While reviewing 6907760 i think i found one problem why John is facing so many Events.
>
> If the Copy of the many files is not done in one Thread (ex. multiple Applications writing to the same directory), AbstractWatchKey will only increase modification count if the context that is changing is the same as the last added to the events-lists.
>
> If i am right, the order of the events is not guaranteed. So AbstractWatchKey could check the whole events-list and increase the modification count. This can only be done if we can find a quick implementation. I got some ideas but not tried it yet.
>
> I add a Testcase that shows the Problem with multiple Threads. You can change SEQUENCIAL to true to see how it works in an non-threaded Environment.
>
> Kind regards
> Sebastian
>   
The watch service implementations that makes use of kernel support 
always process and queues the events in the order that they are 
received. The polling implementation (used when there isn't kernel 
support) processes snapshots of the directory and so cannot infer the 
order of changes between snapshots. It's okay for several entities 
(maybe threads in the same VM, maybe other programs or processes) to be 
creating or changing files at the same time. It does reduce the 
likelihood of repeated modification events to the same file and so the 
representation will be less efficient. You can't re-order create or 
delete events as otherwise the application's view of the directory will 
get out of sync. It may be useful in some cases to know when a file was 
modified relative to other operations - I think you are proposing to not 
do this, is that right? I looked at the test you attached but it doesn't 
seem to demonstrate a problem (or at least I don't see it on the first 
glance).

-Alan.

From Alan.Bateman at Sun.COM  Tue Jan  5 03:03:32 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Tue, 05 Jan 2010 11:03:32 +0000
Subject: Need reviewer for 6913877: (fs) AsynchronousFileChannel.write can
	return wrong result under load [win]
In-Reply-To: <4B4316D1.9070605@sun.com>
References: <4B4217FB.5060403@sun.com> <4B4316D1.9070605@sun.com>
Message-ID: <4B431C84.5060007@sun.com>

Christopher Hegarty -Sun Microsystems Ireland wrote:
> Alan,
>
> This change looks good. I have just one question about the requirement 
> to call Invoker.invoke(result) at the end of the run method of 
> WriteTask. I can see that in the ReadTask case this may be an 
> optimization for EOF, but I don't think it is necessary for write. If 
> you run with system assertions enabled and write throws an Exception 
> then the assertion in Invoker.invoke will be triggered, right?
>
> Otherwise looks good.
>
> -Chris.
Thanks Chris. If a write operation is initiated specifying a completion 
handler, and the write operation cannot be initiated then the completion 
handler's failed method must be invoked. That Invoker.invoke(result) 
does that (directly or indirectly depending on if it's a pooled-thread 
or not).

-Alan.

From Christopher.Hegarty at Sun.COM  Tue Jan  5 03:03:28 2010
From: Christopher.Hegarty at Sun.COM (Christopher Hegarty -Sun Microsystems Ireland)
Date: Tue, 05 Jan 2010 11:03:28 +0000
Subject: Need reviewer for 6913877: (fs) AsynchronousFileChannel.write can
	return wrong result under load [win]
In-Reply-To: <4B431C84.5060007@sun.com>
References: <4B4217FB.5060403@sun.com> <4B4316D1.9070605@sun.com>
	<4B431C84.5060007@sun.com>
Message-ID: <4B431C80.2030409@sun.com>

Alan Bateman wrote:
> Thanks Chris. If a write operation is initiated specifying a completion 
> handler, and the write operation cannot be initiated then the completion 
> handler's failed method must be invoked. That Invoker.invoke(result) 
> does that (directly or indirectly depending on if it's a pooled-thread 
> or not).

Ah yes, my mistake. I see this now. The change looks fine so.

-Chris.

> 
> -Alan.

From Alan.Bateman at Sun.COM  Tue Jan  5 03:16:38 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Tue, 05 Jan 2010 11:16:38 +0000
Subject: Repository
In-Reply-To: <fb36a6ef1001050246s5e7eead5ua03d9e03b1bcb996@mail.gmail.com>
References: <fb36a6ef1001050246s5e7eead5ua03d9e03b1bcb996@mail.gmail.com>
Message-ID: <4B431F96.6090702@sun.com>

John Davis wrote:
> Is there a git repo where I can see the source?  Looking at the 
> GetQueuedCompletionsStatusEx stuff ...
>  
> Thanks,
> JD
>  
We use Mercurial rather than Git. The jdk master repository is here:
    http://hg.openjdk.java.net/jdk7/jdk7/jdk

For the "GetQueuedCompletionsStatusEx  stuff..." you'll probably want to 
look at
  
http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/tip/src/windows/classes/sun/nio/ch/Iocp.java
as that is the code that handles I/O completion events.

-Alan.


From Christopher.Hegarty at Sun.COM  Tue Jan  5 03:17:00 2010
From: Christopher.Hegarty at Sun.COM (Christopher Hegarty -Sun Microsystems Ireland)
Date: Tue, 05 Jan 2010 11:17:00 +0000
Subject: Need reviewer for 6907760: (file) OVERFLOW event should cause
	pending	events to be purged
In-Reply-To: <4B4204F9.6020800@sun.com>
References: <4B4204F9.6020800@sun.com>
Message-ID: <4B431FAC.5050203@sun.com>

Alan,

This change looks like a good optimization.

One thing I noticed is that if there are multiple events after an 
OVERFLOW, then the OVERFLOW count get incremented. I don't think this is 
a problem, does the count even make sense in this situation? Just a 
observation.

I also noticed the nice use of Objects.equals. Is the file system API 
backported to 1.6? Will this require a patch to compile/run on 1.6? I 
think the use of Objects.equals simplifies the logic and I like it. Just 
another observation.

-Chris.

Alan Bateman wrote:
> This is a follow-up to the discussion on the handling of WatchService 
> overflow events that John Hendrikx brought up on this mailing list in 
> December. A suggestion to come from that discussion is that the 
> implementation can drop pending events when queuing an overflow event. 
> That will save the application from needlessly processing these events. 
> Similarly, there isn't any point queuing events while there is an 
> overflow event pending.  The webrev with the proposed changes is here:
>  http://cr.openjdk.java.net/~alanb/6907760/webrev.00/
> 
> Thanks,
> Alan.

From Alan.Bateman at Sun.COM  Tue Jan  5 03:38:32 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Tue, 05 Jan 2010 11:38:32 +0000
Subject: Need reviewer for 6907760: (file) OVERFLOW event should cause
	pending	events to be purged
In-Reply-To: <4B431FAC.5050203@sun.com>
References: <4B4204F9.6020800@sun.com> <4B431FAC.5050203@sun.com>
Message-ID: <4B4324B8.9040109@sun.com>

Christopher Hegarty -Sun Microsystems Ireland wrote:
> Alan,
>
> This change looks like a good optimization.
>
> One thing I noticed is that if there are multiple events after an 
> OVERFLOW, then the OVERFLOW count get incremented. I don't think this 
> is a problem, does the count even make sense in this situation? Just a 
> observation.
Thanks for reviewing. The incrementing of the overflow count is 
intentional but not necessary. For debugging purposes it may give a hint 
that many events are being dropped.
>
> I also noticed the nice use of Objects.equals. Is the file system API 
> backported to 1.6? Will this require a patch to compile/run on 1.6? I 
> think the use of Objects.equals simplifies the logic and I like it. 
> Just another observation.
It's not a goal to run with jdk6 or older releases (and can't of course, 
without work, because it's a new API and also updates existing APIs). 
There are other jdk7 dependencies too so I'm not worried about it.

-Alan.

From sebastian.sickelmann at gmx.de  Tue Jan  5 11:43:57 2010
From: sebastian.sickelmann at gmx.de (Sebastian Sickelmann)
Date: Tue, 05 Jan 2010 20:43:57 +0100
Subject: WatchService questions
In-Reply-To: <4B431C1E.8020304@sun.com>
References: <4B1A4F80.2070106@xs4all.nl> <4B1AC735.50404@sun.com>
	<4B1B9B28.9090007@xs4all.nl> <4B1BED7B.40103@sun.com>
	<20100105095218.78140@gmx.net> <4B431C1E.8020304@sun.com>
Message-ID: <20100105194357.168390@gmx.net>

Alan Bateman wrote:
> Sebastian Sickelmann wrote:
> > :
> > While reviewing 6907760 i think i found one problem why John is facing
> so many Events.
> >
> > If the Copy of the many files is not done in one Thread (ex. multiple
> Applications writing to the same directory), AbstractWatchKey will only
> increase modification count if the context that is changing is the same as the
> last added to the events-lists.
> >
> > If i am right, the order of the events is not guaranteed. So
> AbstractWatchKey could check the whole events-list and increase the modification
> count. This can only be done if we can find a quick implementation. I got some
> ideas but not tried it yet.
> >
> > I add a Testcase that shows the Problem with multiple Threads. You can
> change SEQUENCIAL to true to see how it works in an non-threaded
> Environment.
> >
> > Kind regards
> > Sebastian
> >   
> The watch service implementations that makes use of kernel support 
> always process and queues the events in the order that they are 
> received. The polling implementation (used when there isn't kernel 
> support) processes snapshots of the directory and so cannot infer the 
> order of changes between snapshots.
So a user of the WatchService-API cannot count in the order of the events. Or is there a way to ask the FileSystem or the Watch-Service if it makes use of kernel support. Maybe this would be a usefull method(isPolling() or isNative()) in WatchService. Maybe a Method(isOrderGuaranteed) in WatchService to evaluate if the order is guaranteed is usefull too.

> It's okay for several entities 
> (maybe threads in the same VM, maybe other programs or processes) to be 
> creating or changing files at the same time. It does reduce the 
> likelihood of repeated modification events to the same file and so the 
> representation will be less efficient. You can't re-order create or 
> delete events as otherwise the application's view of the directory will 
> get out of sync.
I have overseen this fact and i think you are right.
While searching for the last MODIFY the implementation should stop searching after Events like CREATE and DELETE.
Here an example: newest to oldest events.
9- MODIFY File3
8- MODIFY File2
7- CREATE File2
6- CREATE File1
5- DELETE File1
4- MODIFY File1
3- CREATE File3
2- CREATE File2
1- CREATE File1

An Update to File3 should increase the modification count of Event 9
An Update to File2 should increase the modification count of Event 8
An Update to File1 should not increase the modification count of Event 4 because there are CREATE and DELETE Events at Event 5,6 and 7. A Event 10(MODIFY File1) should be inserted instead.

> It may be useful in some cases to know when a file was 
> modified relative to other operations - I think you are proposing to not 
> do this, is that right?
I think that you understand my proposal correct. See Example above.
I think also that the relative Position of MODIFY to other operations is interesting, but not to other Events (in particular other MODIFY-Events)on other Files.

I can imagine a case where the order of concurrent Modification of different files would be interessting. But only if i get more information(for example the size of change the change, or which attribute has changed) in the modify-event. But i only get the modification count. Example:

4- MODIFY File2 ModCount=3 
3- MODIFY File1 ModCount=2
2- MODIFY File2 ModCount=5
1- MODIFY File1 ModCount=7

gives me the same information as

2- MODIFY File2 ModCount=8
1- MODIFY File1 ModCount=9

But 

5- MODIFY File2 ModCount=3 
4- MODIFY File1 ModCount=2
3- CREATE File3 ModCount=1
2- MODIFY File2 ModCount=5
1- MODIFY File1 ModCount=7

is not the same as

3- CREATE File3 ModCount=1
2- MODIFY File2 ModCount=8
1- MODIFY File1 ModCount=9


> I looked at the test you attached but it doesn't 
> seem to demonstrate a problem (or at least I don't see it on the first 
> glance).
The test writes 64 Files in parallel(each in 16 Steps). If the order of MODIFY Events on different Files is not relevant(see above), than there should be 128 Events in the Event List (64 CREATE with ModCount 1 and 64 MODIFY with ModCount 16).

But because in the actual implementation the order of MODIFY Events on different Files seems to be relevant, it generates many more "unnecessary" Events. This leeds to earlier OVERFLOW Events which forces me to rescan the Directory again. 
> 
> -Alan.
- Sebastian
-- 
Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 -
sicherer, schneller und einfacher! http://portal.gmx.net/de/go/atbrowser

From Alan.Bateman at Sun.COM  Wed Jan  6 03:51:52 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Wed, 06 Jan 2010 11:51:52 +0000
Subject: WatchService questions
In-Reply-To: <20100105194357.168390@gmx.net>
References: <4B1A4F80.2070106@xs4all.nl> <4B1AC735.50404@sun.com>
	<4B1B9B28.9090007@xs4all.nl> <4B1BED7B.40103@sun.com>
	<20100105095218.78140@gmx.net> <4B431C1E.8020304@sun.com>
	<20100105194357.168390@gmx.net>
Message-ID: <4B447958.5080101@sun.com>

Sebastian Sickelmann wrote:
> :
> So a user of the WatchService-API cannot count in the order of the events. 
The sequence of events needs to keep the user's view of the directory 
consistent. This implies an ordering for at least the create and delete 
events (irrespective of implementation). Your mails do raise the 
question on if modification events need to be ordered. Clearly there are 
cases where they should be ordered with respect to other event kinds. If 
a file is modified after it is created then you should expect the 
modification event after the create event. There are other examples like 
this. If several files are modified at around the same time then the 
ordering may or may not be useful but clearly cannot be guaranteed 
everywhere. Where it is important then the user can examine the file's 
modification time if that helps. So I think you have a worthy point and 
we should think about this case further to see if we should change the 
implementation. If it is changed then there are several possible 
approaches - for example modification events could be merged towards the 
head if the pending list reaches its limit. Alternatively modification 
events could initially be stored in a map (for fast lookup) and only 
added to the pending list when an event of a different kind needs to be 
queued (or the user retrieves the pending events).

-Alan.



From sebastian.sickelmann at gmx.de  Thu Jan  7 11:35:22 2010
From: sebastian.sickelmann at gmx.de (Sebastian Sickelmann)
Date: Thu, 07 Jan 2010 20:35:22 +0100
Subject: WatchService questions
In-Reply-To: <4B447958.5080101@sun.com>
References: <4B1A4F80.2070106@xs4all.nl> <4B1AC735.50404@sun.com>
	<4B1B9B28.9090007@xs4all.nl> <4B1BED7B.40103@sun.com>
	<20100105095218.78140@gmx.net> <4B431C1E.8020304@sun.com>
	<20100105194357.168390@gmx.net> <4B447958.5080101@sun.com>
Message-ID: <20100107193522.304800@gmx.net>

Alan Bateman wrote:
> Sebastian Sickelmann wrote:
> > :
> > So a user of the WatchService-API cannot count in the order of the
> events. 
> The sequence of events needs to keep the user's view of the directory 
> consistent. This implies an ordering for at least the create and delete 
> events (irrespective of implementation). Your mails do raise the 
> question on if modification events need to be ordered. Clearly there are 
> cases where they should be ordered with respect to other event kinds. If 
> a file is modified after it is created then you should expect the 
> modification event after the create event. There are other examples like 
> this. If several files are modified at around the same time then the 
> ordering may or may not be useful but clearly cannot be guaranteed 
> everywhere. Where it is important then the user can examine the file's 
> modification time if that helps. So I think you have a worthy point and 
> we should think about this case further to see if we should change the 
> implementation.
I think so to. When someone gets the following Events that happend at time t
the user has no change to exermine the status (ex. Modification-Time, Size) for 
the Event 2. Because the Event raises at t=20 at he receives this Event-List 
sometime after t=40.

4- MODIDY F2 modCount=5 at t=40
3- MODIFY F1 modCount=4 at t=30
2- MODIDY F2 modCount=3 at t=20
1- MODIDY F1 modCount=2 at t=10

So you gain no more information that in this example:

4- MODIDY F2 modCount=8 at t=40 t=20
3- MODIFY F1 modCount=6 at t=30 t=10

Naturally t=xyz should not be an information of the events. 


>If it is changed then there are several possible 
> approaches - for example modification events could be merged towards the 
> head if the pending list reaches its limit. Alternatively modification 
> events could initially be stored in a map (for fast lookup) and only 
> added to the pending list when an event of a different kind needs to be 
> queued (or the user retrieves the pending events).
This are exactly the 2 Mainimplemetatios i see too. 
1. Merging MODIFY-Events and move to Head
2. Merging MODIFY-Events and leave them in there Position. (Searching can be done via HashMap)

Implementation 1. Merging MODIFY-Events
=======================================
Searching the List in linear Order can lead to fast results, but only if the list is reorderd.
The question is: How much the reorderung in the Event-List hurts.
In single Threaded modifiying of the watched directory there is a slightly good change to find
older MODIFY-Events to the some context very fast at the head of the list. But in Multithreaded
uses of the Directory there will only a few hits at the head of the list.
The order of the MODIFY-EVENTS of different contexts are than relative to the ==last== Modification of the contexts.

Implementation 2. Merging MODIFY-Events and leave them in there Position. (Searching can be done via HashMap)
=============================================================================================================
Searching a MODIFY-EVENT in an HashMap should be pretty fast. Incrementation could be done in the found MODIFY-Event.
If the top-most Event is a MODIFY-EVENT to the same context, lookup at the head of the list would be faster.
At EVENTS like CREATE and DELETE the HashMap should be cleared(Implementation 2A). Alternativ only the MODIFY-EVENTS
of the same context can be removed from the Map (Implementation 2B).
I think both solutions (Implementation 2A and 2B) are only realy good in multithreaded environments. When there is only 
one other programm or thread that writes to the watched Directory Implementation 1 will be faster.
The order of the MODIFY-EVENTS of different contexts are than relative to the ==first== Modification of the contexts.

But what should be done with other Events than StdWatchEventKind?

Are there implementation specific EventKinds? And must there be a way(overrideable protected Method) so that implementations can override Merging and Reordering of Events?

I have made two(2A,2B) Test-Implemetations and it seems to work and reduce MODIFY-Events.

- Sebastian
> 
> -Alan.
> 

-- 
Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 -
sicherer, schneller und einfacher! http://portal.gmx.net/de/go/atbrowser

From Alan.Bateman at Sun.COM  Thu Jan  7 13:30:46 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Thu, 07 Jan 2010 21:30:46 +0000
Subject: WatchService questions
In-Reply-To: <20100107193522.304800@gmx.net>
References: <4B1A4F80.2070106@xs4all.nl> <4B1AC735.50404@sun.com>
	<4B1B9B28.9090007@xs4all.nl> <4B1BED7B.40103@sun.com>
	<20100105095218.78140@gmx.net> <4B431C1E.8020304@sun.com>
	<20100105194357.168390@gmx.net> <4B447958.5080101@sun.com>
	<20100107193522.304800@gmx.net>
Message-ID: <4B465286.8040309@sun.com>

Sebastian Sickelmann wrote:
> I think both solutions (Implementation 2A and 2B) are only realy good in multithreaded environments. When there is only 
> one other programm or thread that writes to the watched Directory Implementation 1 will be faster.
> The order of the MODIFY-EVENTS of different contexts are than relative to the ==first== Modification of the contexts.
>   
If you have cycles to experiment and send results that would be great. 
Also it's good to think about different scenarios where there is one or 
several programs changing files. One thing to mention about our 
implementations is that the 3 native implementations use a single poller 
thread per watch service. So if you are hacking on 
AbstractWatchKey.signalEvent then there will be only one producer. The 
polling implementation could potentially have concurrent threads polling 
directories but they will be different directories (never the same 
directory) and events are queued on per watch key basis so again only 
one producer.

> But what should be done with other Events than StdWatchEventKind?
>   
Our implementation does support a non-standard modifier when registering 
events but it doesn't support any additional events at this time. For 
now, it is okay to focus only on ENTRY_MODIFY events and assume that you 
can't reorder other events.

-Alan.




From java.net at twistedprotein.com  Fri Jan  8 05:38:32 2010
From: java.net at twistedprotein.com (Brendon McLean)
Date: Fri, 8 Jan 2010 15:38:32 +0200
Subject: Resolving empty paths.
Message-ID: <816916EC-F126-4F26-9EE9-FE70923D852A@twistedprotein.com>

Hi,

I apologise if this has already been discussed, but I could not find a way to search the mailing list, or properly use Google to check for that.

I am already using the Paths API in production code (www.mindsilver.com) as I was reluctant to create my own Jetbrains-like VirtualFile type abstraction when a JDK substitute lies around the corner.  I pulled the whole API out and have been using about 50% of it very successfully on Mac, Unix and Linux.  I plan to revert to your API when JDK7 is released.

One issue I run into a lot though, is exceptions caused by an "Empty path".  Mostly, this happens when resolving the empty string against an absolute path.  I see that subsequent to my branching, there is a null guard which will doubtless save many people, but lots of GUIs, whether web or Swing, will use the empty string instead of a null.  Would there be any adverse effects of adding a check for the empty string and simply returning "this" from the resolve method in such a case?

i.e.  Surely Paths.get("/usr/local").resolve("") should return Paths.get("/usr/local")?

Regards,
Brendon McLean.

Mindsilver Ltd.

From Alan.Bateman at Sun.COM  Fri Jan  8 06:28:37 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Fri, 08 Jan 2010 14:28:37 +0000
Subject: Resolving empty paths.
In-Reply-To: <816916EC-F126-4F26-9EE9-FE70923D852A@twistedprotein.com>
References: <816916EC-F126-4F26-9EE9-FE70923D852A@twistedprotein.com>
Message-ID: <4B474115.8000807@sun.com>

Brendon McLean wrote:
> Hi,
>
> I apologise if this has already been discussed, but I could not find a way to search the mailing list, or properly use Google to check for that.
>
> I am already using the Paths API in production code (www.mindsilver.com) as I was reluctant to create my own Jetbrains-like VirtualFile type abstraction when a JDK substitute lies around the corner.  I pulled the whole API out and have been using about 50% of it very successfully on Mac, Unix and Linux.  I plan to revert to your API when JDK7 is released.
>   
Interesting, can you say more about this? Are you using the API and 
implementation or just the API?

> One issue I run into a lot though, is exceptions caused by an "Empty path".  Mostly, this happens when resolving the empty string against an absolute path.  I see that subsequent to my branching, there is a null guard which will doubtless save many people, but lots of GUIs, whether web or Swing, will use the empty string instead of a null.  Would there be any adverse effects of adding a check for the empty string and simply returning "this" from the resolve method in such a case?
>
> i.e.  Surely Paths.get("/usr/local").resolve("") should return Paths.get("/usr/local")?
>   
This probably make sense. We would need to change the spec slightly to 
allow for this as it is currently specified to convert the given 
path-string to a Path. There are a bunch of small API changes 
accumulating so I'll add this one to the list.

Just on null. If you invoke resolve(String) with null then it should 
throw NPE. I suspect your "null guard" comment might relate to 
resolve(Path) where it should return "this" if invoked with null. This 
might seem inconsistent but it's to make it easy to use resolve(Path) in 
conjunction with relativize(Path).

-Alan.





From java.net at twistedprotein.com  Fri Jan  8 06:46:55 2010
From: java.net at twistedprotein.com (Brendon McLean)
Date: Fri, 8 Jan 2010 16:46:55 +0200
Subject: Resolving empty paths.
In-Reply-To: <4B474115.8000807@sun.com>
References: <816916EC-F126-4F26-9EE9-FE70923D852A@twistedprotein.com>
	<4B474115.8000807@sun.com>
Message-ID: <1ACFD075-71EB-4D27-87FE-7B6F69AEA515@twistedprotein.com>

Hi Alan,

On 8 Jan 2010, at 16:28 , Alan Bateman wrote:

> Brendon McLean wrote:
>> Hi,
>> 
>> I apologise if this has already been discussed, but I could not find a way to search the mailing list, or properly use Google to check for that.
>> 
>> I am already using the Paths API in production code (www.mindsilver.com) as I was reluctant to create my own Jetbrains-like VirtualFile type abstraction when a JDK substitute lies around the corner.  I pulled the whole API out and have been using about 50% of it very successfully on Mac, Unix and Linux.  I plan to revert to your API when JDK7 is released.
>>  
> Interesting, can you say more about this? Are you using the API and implementation or just the API?

I'm using as much of the implementation that doesn't require native calls.  I'm using a polling system instead of the new IO API for the same reason.

My primary use of this is to manage to user specified paths, relative or absolute.  The API is quite pleasant to use.

> 
>> One issue I run into a lot though, is exceptions caused by an "Empty path".  Mostly, this happens when resolving the empty string against an absolute path.  I see that subsequent to my branching, there is a null guard which will doubtless save many people, but lots of GUIs, whether web or Swing, will use the empty string instead of a null.  Would there be any adverse effects of adding a check for the empty string and simply returning "this" from the resolve method in such a case?
>> 
>> i.e.  Surely Paths.get("/usr/local").resolve("") should return Paths.get("/usr/local")?
>>  
> This probably make sense. We would need to change the spec slightly to allow for this as it is currently specified to convert the given path-string to a Path. There are a bunch of small API changes accumulating so I'll add this one to the list.

Thanks.  It's just one of those use cases I need so often.  For example, if a user needs to create a sub path relative some "project path" then I end up code like this:

File f = projectPath.resolve(packageDir).resolve(file).toFile();

The empty package strikes again!  So obviously, I have to guard against it whenever I find this pattern emerging.

> Just on null. If you invoke resolve(String) with null then it should throw NPE. I suspect your "null guard" comment might relate to resolve(Path) where it should return "this" if invoked with null. This might seem inconsistent but it's to make it easy to use resolve(Path) in conjunction with relativize(Path).

You are correct.  That will teach me to compile code in my head.

Brendon.


From Alan.Bateman at Sun.COM  Fri Jan  8 07:53:35 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Fri, 08 Jan 2010 15:53:35 +0000
Subject: Resolving empty paths.
In-Reply-To: <1ACFD075-71EB-4D27-87FE-7B6F69AEA515@twistedprotein.com>
References: <816916EC-F126-4F26-9EE9-FE70923D852A@twistedprotein.com>
	<4B474115.8000807@sun.com>
	<1ACFD075-71EB-4D27-87FE-7B6F69AEA515@twistedprotein.com>
Message-ID: <4B4754FF.5050204@sun.com>

Brendon McLean wrote:
> :
> Thanks.  It's just one of those use cases I need so often.  For example, if a user needs to create a sub path relative some "project path" then I end up code like this:
>
> File f = projectPath.resolve(packageDir).resolve(file).toFile();
>
> The empty package strikes again!  So obviously, I have to guard against it whenever I find this pattern emerging.
>   
Right, in the above then each invocation of resolve could throw 
InvalidPathException if the input is garbage.

Out of curiosity, is toFile() a method that you have added? This doesn't 
exist in our API as it wouldn't always make sense (for example the Path 
might locate a file in a custom file system).

-Alan.

From java.net at twistedprotein.com  Sun Jan 10 02:03:21 2010
From: java.net at twistedprotein.com (Brendon McLean)
Date: Sun, 10 Jan 2010 12:03:21 +0200
Subject: Resolving empty paths.
In-Reply-To: <4B4754FF.5050204@sun.com>
References: <816916EC-F126-4F26-9EE9-FE70923D852A@twistedprotein.com>
	<4B474115.8000807@sun.com>
	<1ACFD075-71EB-4D27-87FE-7B6F69AEA515@twistedprotein.com>
	<4B4754FF.5050204@sun.com>
Message-ID: <9FD7F18A-63C5-4956-9C7B-ADC13A183083@twistedprotein.com>

Alan,

I actually don't have a toFile method - I just made it up for the purposes of the example because the way I normally do that is more difficult to read and detracts from the usage of resolve that I was trying to highlight.  

Sorry for the confusion.  I copied this stuff before you added most of the file operation equivalents to the Path class, so I internally convert to File whenever I need to do anything useful.  I look forward to not having to do that.

Regards,
Brendon.

On 8 Jan 2010, at 17:53 , Alan Bateman wrote:

> Brendon McLean wrote:
>> :
>> Thanks.  It's just one of those use cases I need so often.  For example, if a user needs to create a sub path relative some "project path" then I end up code like this:
>> 
>> File f = projectPath.resolve(packageDir).resolve(file).toFile();
>> 
>> The empty package strikes again!  So obviously, I have to guard against it whenever I find this pattern emerging.
>>  
> Right, in the above then each invocation of resolve could throw InvalidPathException if the input is garbage.
> 
> Out of curiosity, is toFile() a method that you have added? This doesn't exist in our API as it wouldn't always make sense (for example the Path might locate a file in a custom file system).
> 
> -Alan.


From sebastian.sickelmann at gmx.de  Mon Jan 11 13:51:49 2010
From: sebastian.sickelmann at gmx.de (Sebastian Sickelmann)
Date: Mon, 11 Jan 2010 22:51:49 +0100
Subject: WatchService questions
In-Reply-To: <4B465286.8040309@sun.com>
References: <4B1A4F80.2070106@xs4all.nl> <4B1AC735.50404@sun.com>
	<4B1B9B28.9090007@xs4all.nl> <4B1BED7B.40103@sun.com>
	<20100105095218.78140@gmx.net> <4B431C1E.8020304@sun.com>
	<20100105194357.168390@gmx.net> <4B447958.5080101@sun.com>
	<20100107193522.304800@gmx.net> <4B465286.8040309@sun.com>
Message-ID: <20100111215149.181530@gmx.net>

Alan Bateman wrote:
> Sebastian Sickelmann wrote:
> > I think both solutions (Implementation 2A and 2B) are only realy good in
> multithreaded environments. When there is only
> > one other programm or thread that writes to the watched Directory
> Implementation 1 will be faster.
> > The order of the MODIFY-EVENTS of different contexts are than relative
> to the ==first== Modification of the contexts.
> >   
> If you have cycles to experiment and send results that would be great.
> Also it's good to think about different scenarios where there is one or
> several programs changing files. One thing to mention about our
> implementations is that the 3 native implementations use a single poller
> thread per watch service. So if you are hacking on
> AbstractWatchKey.signalEvent then there will be only one producer. The
> polling implementation could potentially have concurrent threads polling
> directories but they will be different directories (never the same
> directory) and events are queued on per watch key basis so again only
> one producer.
I have done the Implementations of 2A and 2B. It based on your suggested Change (6907760) http://cr.openjdk.java.net/~alanb/6907760/webrev.00/raw_files/new/src/share/classes/sun/nio/fs/AbstractWatchKey.java

Should i send it via mailing-list? Maybe it is better to append the source to an enhancement-ticket in bugzilla?

Btw. is there a way to faster Compile changes in NIO? Calling make in
the top-level openjdk-dir leads my to a long wait time. Is there an good document for this. I cant find one!!

>
> > But what should be done with other Events than StdWatchEventKind?
> >   
> Our implementation does support a non-standard modifier when registering
> events but it doesn't support any additional events at this time. For
> now, it is okay to focus only on ENTRY_MODIFY events and assume that you
> can't reorder other events.

OK, so thinks are easier.
>
> -Alan.
>
> 
-- 
GRATIS f?r alle GMX-Mitglieder: Die maxdome Movie-FLAT!
Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01

From Alan.Bateman at Sun.COM  Mon Jan 11 14:22:51 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Mon, 11 Jan 2010 22:22:51 +0000
Subject: WatchService questions
In-Reply-To: <20100111215149.181530@gmx.net>
References: <4B1A4F80.2070106@xs4all.nl> <4B1AC735.50404@sun.com>
	<4B1B9B28.9090007@xs4all.nl> <4B1BED7B.40103@sun.com>
	<20100105095218.78140@gmx.net> <4B431C1E.8020304@sun.com>
	<20100105194357.168390@gmx.net> <4B447958.5080101@sun.com>
	<20100107193522.304800@gmx.net> <4B465286.8040309@sun.com>
	<20100111215149.181530@gmx.net>
Message-ID: <4B4BA4BB.8020601@sun.com>

Sebastian Sickelmann wrote:
> :
> I have done the Implementations of 2A and 2B. It based on your suggested Change (6907760) http://cr.openjdk.java.net/~alanb/6907760/webrev.00/raw_files/new/src/share/classes/sun/nio/fs/AbstractWatchKey.java
>
> Should i send it via mailing-list? Maybe it is better to append the source to an enhancement-ticket in bugzilla?
>   
Are you signed up as a contributer yet? If not, then the "How to 
contribute" page [1] is a good start.

Normally I would say create a bug in the bugzilla, but I think you are 
going to propose alternative approaches, to append patch/diffs is okay.

> Btw. is there a way to faster Compile changes in NIO? Calling make in
> the top-level openjdk-dir leads my to a long wait time. Is there an good document for this. I cant find one
If do you a partial build, meaning only the jdk repository, then it is 
possible to do incremental builds in make/java/nio.

-Alan.


[1] http://openjdk.java.net/contribute/

From sebastian.sickelmann at gmx.de  Tue Jan 12 15:19:12 2010
From: sebastian.sickelmann at gmx.de (Sebastian Sickelmann)
Date: Wed, 13 Jan 2010 00:19:12 +0100
Subject: WatchService questions
Message-ID: <20100112231912.24670@gmx.net>

Alan Bateman wrote:
> Sebastian Sickelmann wrote:
> > :
> > I have done the Implementations of 2A and 2B. It based on your suggested
> Change (6907760)
> http://cr.openjdk.java.net/~alanb/6907760/webrev.00/raw_files/new/src/share/classes/sun/nio/fs/AbstractWatchKey.java
> >
> > Should i send it via mailing-list? Maybe it is better to append the
> source to an enhancement-ticket in bugzilla?
> >   
> Are you signed up as a contributer yet? If not, then the "How to 
> contribute" page [1] is a good start.
> 
> Normally I would say create a bug in the bugzilla, but I think you are 
> going to propose alternative approaches, to append patch/diffs is okay.
I add the two diffs to this mail. Tomorrow i will try using nice looking webrev-output.

The diffs are based on: [2]

> 
> > Btw. is there a way to faster Compile changes in NIO? Calling make in
> > the top-level openjdk-dir leads my to a long wait time. Is there an good
> document for this. I cant find one
> If do you a partial build, meaning only the jdk repository, then it is 
> possible to do incremental builds in make/java/nio.
I am using JDK_IMPORT_PATH but got problems while building nio only.
> 
> -Alan.
> 
> 
> [1] http://openjdk.java.net/contribute/
[2] http://cr.openjdk.java.net/~alanb/6907760/webrev.00/raw_files/new/src/share/classes/sun/nio/fs/AbstractWatchKey.java
-- 
Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 -
sicherer, schneller und einfacher! http://portal.gmx.net/de/go/atbrowser
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2A.diff
Type: text/x-patch
Size: 1651 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100113/297f0b4a/attachment.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2B.diff
Type: text/x-patch
Size: 1744 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100113/297f0b4a/attachment-0001.bin 

From sebastian.sickelmann at gmx.de  Tue Jan 12 15:24:16 2010
From: sebastian.sickelmann at gmx.de (Sebastian Sickelmann)
Date: Wed, 13 Jan 2010 00:24:16 +0100
Subject: Reorder modify-events was:WatchService questions
Message-ID: <20100112232416.24670@gmx.net>

Sebastian Sickelmann wrote:
> Alan Bateman wrote:
> > Sebastian Sickelmann wrote:
> > > :
> > > I have done the Implementations of 2A and 2B. It based on your
> suggested
> > Change (6907760)
> >
> http://cr.openjdk.java.net/~alanb/6907760/webrev.00/raw_files/new/src/share/classes/sun/nio/fs/AbstractWatchKey.java
> > >
> > > Should i send it via mailing-list? Maybe it is better to append the
> > source to an enhancement-ticket in bugzilla?
> > >   
> > Are you signed up as a contributer yet? If not, then the "How to 
> > contribute" page [1] is a good start.
> > 
> > Normally I would say create a bug in the bugzilla, but I think you are 
> > going to propose alternative approaches, to append patch/diffs is okay.
> I add the two diffs to this mail. Tomorrow i will try using nice looking
> webrev-output.
> 
> The diffs are based on: [2]
> 
> > 
> > > Btw. is there a way to faster Compile changes in NIO? Calling make in
> > > the top-level openjdk-dir leads my to a long wait time. Is there an
> good
> > document for this. I cant find one
> > If do you a partial build, meaning only the jdk repository, then it is 
> > possible to do incremental builds in make/java/nio.
> I am using JDK_IMPORT_PATH but got problems while building nio only.
> > 
> > -Alan.
> > 
> > 
> > [1] http://openjdk.java.net/contribute/
> [2]
> http://cr.openjdk.java.net/~alanb/6907760/webrev.00/raw_files/new/src/share/classes/sun/nio/fs/AbstractWatchKey.java
> -- 
Here are the test-cases.



-- 
GRATIS f?r alle GMX-Mitglieder: Die maxdome Movie-FLAT!
Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01
-------------- next part --------------
A non-text attachment was scrubbed...
Name: NoReorderungAfterCreateOrDeleteEvent2A.java
Type: text/x-java
Size: 5843 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100113/adb0dd30/attachment.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: NoReorderungAfterCreateOrDeleteEvent2B.java
Type: text/x-java
Size: 6360 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100113/adb0dd30/attachment-0001.bin 

From Alan.Bateman at Sun.COM  Wed Jan 13 10:02:32 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Wed, 13 Jan 2010 18:02:32 +0000
Subject: WatchService questions
In-Reply-To: <20100112231912.24670@gmx.net>
References: <20100112231912.24670@gmx.net>
Message-ID: <4B4E0AB8.2050003@sun.com>

Sebastian Sickelmann wrote:
> :
> I add the two diffs to this mail. Tomorrow i will try using nice looking webrev-output.
>
> The diffs are based on: [2]
>   
I skimmed through both patches. Looks like they are close with the only 
difference that the first patch (2A.diff) preserves the ordering of 
modify events relative to non-modify events for other files. In 
practical terms there probably isn't too much of a difference but 
2B.diff may be slightly more efficient if the user falls way behind and 
there are lots of files being of changes going on at the same time that 
lots of files are being created and deleted.

One thing that doesn't look right is that pollEvents doesn't seem to be 
clear the map of modify events - that would be important to avoid 
incrementing the count on events that have already been retrieved by the 
user.

> :
> I am using JDK_IMPORT_PATH but got problems while building nio only.
>   
That's used when you aren't building the entire JDK. Even though you are 
doing a partial build you will still need to build the entire jdk 
repository. Once you've got a build then you can do into the 
make/java/nio directory to do incremental builds of just this area.

-Alan.

From Thomas.Salter at unisys.com  Thu Jan 14 08:46:41 2010
From: Thomas.Salter at unisys.com (Salter, Thomas A)
Date: Thu, 14 Jan 2010 10:46:41 -0600
Subject: copyTo/moveTo overwrite
Message-ID: <F089913638199841958E00B22EF169882A4B129980@USEA-EXCH7.na.uis.unisys.com>

When doing a copyTo or moveTo with different providers, it looks like AbstractPath.copyToForeignTarget will overwrite the target file even if REPLACE_EXISTING is not set.  It correctly deletes the target only if REPLACE_EXISTING is set but then later in copyRegularFileToForeignTarget calls newByteChannel with CREATE, so any existing file is overwritten.  It should have used CREATE_NEW instead of CREATE.







-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100114/52cb771e/attachment.html 

From Alan.Bateman at Sun.COM  Thu Jan 14 12:00:12 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Thu, 14 Jan 2010 20:00:12 +0000
Subject: copyTo/moveTo overwrite
In-Reply-To: <F089913638199841958E00B22EF169882A4B129980@USEA-EXCH7.na.uis.unisys.com>
References: <F089913638199841958E00B22EF169882A4B129980@USEA-EXCH7.na.uis.unisys.com>
Message-ID: <4B4F77CC.8050303@sun.com>

Salter, Thomas A wrote:
> When doing a copyTo or moveTo with different providers, it looks like 
> AbstractPath.copyToForeignTarget will overwrite the target file even 
> if REPLACE_EXISTING is not set.  It correctly deletes the target only 
> if REPLACE_EXISTING is set but then later in 
> copyRegularFileToForeignTarget calls newByteChannel with CREATE, so 
> any existing file is overwritten.  It should have used CREATE_NEW 
> instead of CREATE.
>  
You are right! When copying or moving a file to a location that is 
associated with some other provider then it shouldn't replace an 
existing file unless the REPLACE_EXISTING option is present. Thanks for 
finding this. Our of curiosity, did you find this by inspection or were 
you testing with your own provider? Our existing tests doesn't exercise 
this scenario (not yet anyway).

-Alan.

From Thomas.Salter at unisys.com  Thu Jan 14 12:19:37 2010
From: Thomas.Salter at unisys.com (Salter, Thomas A)
Date: Thu, 14 Jan 2010 14:19:37 -0600
Subject: copyTo/moveTo overwrite
In-Reply-To: <4B4F77CC.8050303@sun.com>
References: <F089913638199841958E00B22EF169882A4B129980@USEA-EXCH7.na.uis.unisys.com>
	<4B4F77CC.8050303@sun.com>
Message-ID: <F089913638199841958E00B22EF169882A4B1AE3B7@USEA-EXCH7.na.uis.unisys.com>

I found it through testing.  We're porting to a system with where the file system is essentially two independent file systems (one Windows, one legacy).  Copying within a file system works with the platform's CopyFile/MoveFile functions.  Copying between the file systems is equivalent to copying between providers, so I borrowed the methods in AbstractPath.  This would be much easier, by the way, if copyToForeignTarget and convertMoveToCopyOptions were protected rather than private in AbstractPath.


-----Original Message-----
From: Alan.Bateman at Sun.COM [mailto:Alan.Bateman at Sun.COM]
Sent: Thursday, January 14, 2010 3:00 PM
To: Salter, Thomas A
Cc: nio-dev at openjdk.java.net
Subject: Re: copyTo/moveTo overwrite

Salter, Thomas A wrote:
> When doing a copyTo or moveTo with different providers, it looks like
> AbstractPath.copyToForeignTarget will overwrite the target file even
> if REPLACE_EXISTING is not set.  It correctly deletes the target only
> if REPLACE_EXISTING is set but then later in
> copyRegularFileToForeignTarget calls newByteChannel with CREATE, so
> any existing file is overwritten.  It should have used CREATE_NEW
> instead of CREATE.
>
You are right! When copying or moving a file to a location that is
associated with some other provider then it shouldn't replace an
existing file unless the REPLACE_EXISTING option is present. Thanks for
finding this. Our of curiosity, did you find this by inspection or were
you testing with your own provider? Our existing tests doesn't exercise
this scenario (not yet anyway).

-Alan.

From sebastian.sickelmann at gmx.de  Sat Jan 16 12:47:51 2010
From: sebastian.sickelmann at gmx.de (Sebastian Sickelmann)
Date: Sat, 16 Jan 2010 21:47:51 +0100
Subject: Which forest to clone?
Message-ID: <20100116204751.113770@gmx.net>

Hi,

while reading: 
 - http://openjdk.java.net/guide/index.html

i found some repositories at:
 - http://hg.openjdk.java.net

at the NIO project-page:
 - http://openjdk.java.net/projects/nio/

there is a link to the source-repository:
 - http://hg.openjdk.java.net/nio/nio

but it is 6 weeks old. (subproject jdk)

In the repository:
 - http://hg.openjdk.java.net/jdk7/tl (subproject jdk)

i found a new (7 days) NIO-changesets (changeset 2050 755dd6bdccca)
i want to build on my changesets.

should i now clone from forest:
 - http://hg.openjdk.java.net/nio/nio
or
 - http://hg.openjdk.java.net/jdk7/tl

to create changesets? 

Kind regards
Sebastian
-- 
GRATIS f?r alle GMX-Mitglieder: Die maxdome Movie-FLAT!
Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01

From Alan.Bateman at Sun.COM  Sat Jan 16 13:27:03 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Sat, 16 Jan 2010 21:27:03 +0000
Subject: Which forest to clone?
In-Reply-To: <20100116204751.113770@gmx.net>
References: <20100116204751.113770@gmx.net>
Message-ID: <4B522F27.3050300@sun.com>

Sebastian Sickelmann wrote:
> Hi,
>
> while reading: 
>  - http://openjdk.java.net/guide/index.html
>
> i found some repositories at:
>  - http://hg.openjdk.java.net
>
> at the NIO project-page:
>  - http://openjdk.java.net/projects/nio/
>
> there is a link to the source-repository:
>  - http://hg.openjdk.java.net/nio/nio
>
> but it is 6 weeks old. (subproject jdk)
>
> In the repository:
>  - http://hg.openjdk.java.net/jdk7/tl (subproject jdk)
>
> i found a new (7 days) NIO-changesets (changeset 2050 755dd6bdccca)
> i want to build on my changesets.
>
> should i now clone from forest:
>  - http://hg.openjdk.java.net/nio/nio
> or
>  - http://hg.openjdk.java.net/jdk7/tl
>
> to create changesets? 
>
> Kind regards
> Sebastian
>   
The nio/nio/jdk repository is where the development was originally 
hosted before it was integrated into jdk7. It's not used much now which 
is why it isn't sync'ed too often. The reason we keep it around is 
because it has a number of experimental changes that aren't in jdk7. 
I've just edited the page to hopefully make that a bit clearer.

It's probably best if you clone jdk7/tl - that's the forest where all 
fixes/changes to the core libraries are pushed. That's the reason why 
you see 755dd6bdccca in that repository. It will be pushed to the jdk7 
master at the next integration slot.

-Alan.


From Alan.Bateman at Sun.COM  Sun Jan 17 04:32:18 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Sun, 17 Jan 2010 12:32:18 +0000
Subject: Need reviewer for 6917021: (file) copyTo/moveTo can overrwrite
	existing file when target associated with custom provider
Message-ID: <4B530352.2050708@sun.com>

When the copyTo/moveTo methods are invoked with a target that is 
associated with a custom file system then a simple copy/move 
implementation is used rather than the implementation for the platform. 
Thomas Salter pointed out a bug in this code whereby it could override a 
target file even when the REPLACE_EXISTING option is not used. The bug 
highlights that the current copy/move tests don't exercise the foreign 
provider case. The patch proposed extends the existing test for this 
case. In addition to the issue that Thomas ran into, the tests exposed 
two minor issues where the wrong exception is thrown.

The webrev with the changes is here:
  http://cr.openjdk.java.net/~alanb/6917021/webrev/

Thanks,
Alan.

From Christopher.Hegarty at Sun.COM  Mon Jan 18 06:51:06 2010
From: Christopher.Hegarty at Sun.COM (Christopher Hegarty -Sun Microsystems Ireland)
Date: Mon, 18 Jan 2010 14:51:06 +0000
Subject: Need reviewer for 6917021: (file) copyTo/moveTo can overrwrite
	existing file when target associated with custom provider
In-Reply-To: <4B530352.2050708@sun.com>
References: <4B530352.2050708@sun.com>
Message-ID: <4B54755A.4070009@sun.com>

Alan Bateman wrote:
> When the copyTo/moveTo methods are invoked with a target that is 
> associated with a custom file system then a simple copy/move 
> implementation is used rather than the implementation for the platform. 
> Thomas Salter pointed out a bug in this code whereby it could override a 
> target file even when the REPLACE_EXISTING option is not used. The bug 
> highlights that the current copy/move tests don't exercise the foreign 
> provider case. The patch proposed extends the existing test for this 
> case. In addition to the issue that Thomas ran into, the tests exposed 
> two minor issues where the wrong exception is thrown.
> 
> The webrev with the changes is here:
>  http://cr.openjdk.java.net/~alanb/6917021/webrev/

These changes look fine. Although, I'll admit I just scanned the test 
changes.

-Chris.

> 
> Thanks,
> Alan.

From Alan.Bateman at Sun.COM  Mon Jan 18 07:33:36 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Mon, 18 Jan 2010 15:33:36 +0000
Subject: Need reviewer for 6917021: (file) copyTo/moveTo can overrwrite
	existing file when target associated with custom provider
In-Reply-To: <4B54755A.4070009@sun.com>
References: <4B530352.2050708@sun.com> <4B54755A.4070009@sun.com>
Message-ID: <4B547F50.2090803@sun.com>

Christopher Hegarty -Sun Microsystems Ireland wrote:
> :
> These changes look fine. Although, I'll admit I just scanned the test 
> changes.
Thanks Chris  - the test update might seem large but it's just extending 
the existing test so that the tests are re-run with the target 
associated with a custom provider.

-Alan.

From sebastian.sickelmann at gmx.de  Tue Jan 19 11:02:00 2010
From: sebastian.sickelmann at gmx.de (Sebastian Sickelmann)
Date: Tue, 19 Jan 2010 20:02:00 +0100
Subject: WatchService questions
Message-ID: <20100119190200.117690@gmx.net>

Hi,

in the attachment you can find the two Mercurial Exports. (Version 2A and Version2B)

> Alan Bateman wrote:
> I skimmed through both patches. Looks like they are close with the only 
> difference that the first patch (2A.diff) preserves the ordering of 
> modify events relative to non-modify events for other files. In 
> practical terms there probably isn't too much of a difference but 
> 2B.diff may be slightly more efficient if the user falls way behind and 
> there are lots of files being of changes going on at the same time that 
> lots of files are being created and deleted.
I have found an additional interesting thing in WatchService. What are those Modifiers by the registering for. Should there be a Modifier to turn on the merging of MODIFY-Events?
Maybe the Level of Merging (Version2A and Version2B)?
It looks to me that the SensitivityModfies does some comparable feature than merging(reducing) MODIFY-Events.

> 
> One thing that doesn't look right is that pollEvents doesn't seem to be 
> clear the map of modify events - that would be important to avoid 
> incrementing the count on events that have already been retrieved by the 
> user.
Thanks, you are totally right. This is corrected in the diffs and the testcases.
-- 
GRATIS f?r alle GMX-Mitglieder: Die maxdome Movie-FLAT!
Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hg.WatchServer.2A.export
Type: application/octet-stream
Size: 19527 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100119/f7c1a3d2/attachment-0002.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hg.WatchServer.2B.export
Type: application/octet-stream
Size: 20235 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20100119/f7c1a3d2/attachment-0003.obj 

From Alan.Bateman at Sun.COM  Wed Jan 20 12:44:45 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Wed, 20 Jan 2010 20:44:45 +0000
Subject: WatchService questions
In-Reply-To: <20100119190200.117690@gmx.net>
References: <20100119190200.117690@gmx.net>
Message-ID: <4B576B3D.1020508@sun.com>

Sebastian Sickelmann wrote:
> Hi,
>
> in the attachment you can find the two Mercurial Exports. (Version 2A and Version2B)
>   
Thanks for updating the patches to ensure that the count isn't updated 
after the events have been retrieved. The approach in Version2B is 
probably better - as I mentioned in the previous mail, it will likely be 
a bit more efficient for the case that the user falls very behind, and 
there is a lot of activity, meaning many files being modified at around 
the same time that files are being created and deleted.

BTW, did you look at that link I sent you on the contributor agreement?

:
> I have found an additional interesting thing in WatchService. What are those Modifiers by the registering for. Should there be a Modifier to turn on the merging of MODIFY-Events?
> Maybe the Level of Merging (Version2A and Version2B)?
> It looks to me that the SensitivityModfies does some comparable feature than merging(reducing) MODIFY-Events.
>   
Modifiers are just an extension for the future so that we have a way to 
change how objects are registered. The API doesn't currently define any 
standard modifiers but you'll find two provider specific modifiers in 
the implementation. The modifier for the "sensitivity level" that you 
found is only interesting to the polling implementation (it is ignored 
by the other watch service implementations). If directories aren't 
polled very frequently then it's a bit like merging events although I'm 
not sure that it's worth adding a new modifier for this.

-Alan.

From uckelman at nomic.net  Mon Jan 25 12:10:29 2010
From: uckelman at nomic.net (Joel Uckelman)
Date: Mon, 25 Jan 2010 21:10:29 +0100
Subject: zipfs and write support
Message-ID: <20100125201029.452291007A@charybdis.ellipsis.cx>

I have what I believe is a functioning thread-safe read-write zipfs
implementation now---at least it passes all the tests I wrote---
subclassed from the existing zipfs classes. My question now is whether
it would be better to fold that code into the original read-only zipfs 
classes, or to keep the read-only and read-write versions separate.

One thing I don't know is whether there will be much of a performance
hit in using the read-write version for reading only. I implemented 
locking in the simplest possible way, with a ReadWriteLock, so it 
supports shared reads but exclusive writes. As a result, if you're
only reading, there should never be any lock contention, so I don't
expect this to cause much overhead. (The way which would permit for
greater concurrency, locking on subtrees instead of having one lock
for the whole ZIP archive, is a bit more work, so I held off on that
for now.)

Thoughts?

-- 
J.

From uckelman at nomic.net  Mon Jan 25 12:36:26 2010
From: uckelman at nomic.net (Joel Uckelman)
Date: Mon, 25 Jan 2010 21:36:26 +0100
Subject: getting known keys from an AttributeView
Message-ID: <20100125203626.29ED31007A@charybdis.ellipsis.cx>

I've noticed that there seems to be no way to implement Path.readAttributes()
without hardcoding all of the known attribute keys. For example, if you
receive "foo:*" as an argument, then the docs require that you return a
Map containing all of the FooAttributeView keys and their values. I see no
programmatic way to find out what keys FooAttributeView supports, so the
only thing I can do is build the map by hand, pair by pair. This is rather
inconvenient, and generates a lot of boilerplate code. Have I missed
something?

It would be helpful if AttributeView had a keys() method which returned a
String[] of the keys it supports via Path.getAttribute().

-- 
J.

From Alan.Bateman at Sun.COM  Wed Jan 27 02:01:03 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Wed, 27 Jan 2010 10:01:03 +0000
Subject: zipfs and write support
In-Reply-To: <20100125201029.452291007A@charybdis.ellipsis.cx>
References: <20100125201029.452291007A@charybdis.ellipsis.cx>
Message-ID: <4B600EDF.40701@sun.com>

Joel Uckelman wrote:
> I have what I believe is a functioning thread-safe read-write zipfs
> implementation now---at least it passes all the tests I wrote---
> subclassed from the existing zipfs classes. My question now is whether
> it would be better to fold that code into the original read-only zipfs 
> classes, or to keep the read-only and read-write versions separate.
>
> One thing I don't know is whether there will be much of a performance
> hit in using the read-write version for reading only. I implemented 
> locking in the simplest possible way, with a ReadWriteLock, so it 
> supports shared reads but exclusive writes. As a result, if you're
> only reading, there should never be any lock contention, so I don't
> expect this to cause much overhead. (The way which would permit for
> greater concurrency, locking on subtrees instead of having one lock
> for the whole ZIP archive, is a bit more work, so I held off on that
> for now.)
>
> Thoughts?
>   
It sounds like you've made good progress. If it extends the existing zip 
provider then I assume it must still be based on java.util.zip, right? 
Have you given any thought to access from others VM or applications? 
It's hard to comment on the locking without more information - if there 
a file opened for writing then does it block all other (read) access to 
the "file system"?

-Alan.

From Alan.Bateman at Sun.COM  Wed Jan 27 02:19:18 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Wed, 27 Jan 2010 10:19:18 +0000
Subject: getting known keys from an AttributeView
In-Reply-To: <20100125203626.29ED31007A@charybdis.ellipsis.cx>
References: <20100125203626.29ED31007A@charybdis.ellipsis.cx>
Message-ID: <4B601326.4080605@sun.com>

Joel Uckelman wrote:
> I've noticed that there seems to be no way to implement Path.readAttributes()
> without hardcoding all of the known attribute keys. For example, if you
> receive "foo:*" as an argument, then the docs require that you return a
> Map containing all of the FooAttributeView keys and their values. I see no
> programmatic way to find out what keys FooAttributeView supports, so the
> only thing I can do is build the map by hand, pair by pair. This is rather
> inconvenient, and generates a lot of boilerplate code. Have I missed
> something?
>
> It would be helpful if AttributeView had a keys() method which returned a
> String[] of the keys it supports via Path.getAttribute().
>
>   
Right, if a provider supports FooAttributeView then it must have 
semantic knowledge of all attributes that the view defines (the 
exception being UserDeifnedFileAttributeView where the values are 
opaque). For dynamic access, where the user chooses to identity the 
attributes by name, then it does mean that the provider have some way to 
map the names to specific attributes (mundane, but not hard). I'm 
curious how the suggested keys method improves things. It just avoids 
helps the provider map "*" to the list of attributes names but the 
provider must still have semantic knowledge of each of the attributes, 
and know each of their names.

-Alan.

From uckelman at nomic.net  Wed Jan 27 03:13:25 2010
From: uckelman at nomic.net (Joel Uckelman)
Date: Wed, 27 Jan 2010 12:13:25 +0100
Subject: zipfs and write support
In-Reply-To: <4B600EDF.40701@sun.com>
References: <20100125201029.452291007A@charybdis.ellipsis.cx>
	<4B600EDF.40701@sun.com>
Message-ID: <20100127111325.C26A21007A@charybdis.ellipsis.cx>

Thus spake Alan Bateman:
>   
> It sounds like you've made good progress. If it extends the existing zip 
> provider then I assume it must still be based on java.util.zip, right? 

Yes, it is. I'm not terribly interested in rewriting the low-level ZIP
file twiddling stuff myself.

> Have you given any thought to access from others VM or applications?

I don't know of any way to stop other proceses from corrupting the
underlying ZIP file. Do you? I thought file locks were advistory on
many OSes. I'm not familiar with what would need to be done to
coordinate across VMs. Is there a compelling reason to do this? It's
an unusual requirement.

> It's hard to comment on the locking without more information - if there 
> a file opened for writing then does it block all other (read) access to 
> the "file system"?

Yes, but certain kinds of writes will be nearly instant because they
don't involve writing any data, e.g., deleting files or moving files
which were not originally in the ZIP archive. You could tie up the
whole archive by getting an OutputStream and hanging on onto it. It
would be nicer to have subtree locking instead.

-- 
J.

From uckelman at nomic.net  Wed Jan 27 03:20:03 2010
From: uckelman at nomic.net (Joel Uckelman)
Date: Wed, 27 Jan 2010 12:20:03 +0100
Subject: getting known keys from an AttributeView
In-Reply-To: <4B601326.4080605@sun.com>
References: <20100125203626.29ED31007A@charybdis.ellipsis.cx>
	<4B601326.4080605@sun.com>
Message-ID: <20100127112004.4EA0D1007A@charybdis.ellipsis.cx>

Thus spake Alan Bateman:
>
> Right, if a provider supports FooAttributeView then it must have 
> semantic knowledge of all attributes that the view defines (the 
> exception being UserDeifnedFileAttributeView where the values are 
> opaque). For dynamic access, where the user chooses to identity the 
> attributes by name, then it does mean that the provider have some way to 
> map the names to specific attributes (mundane, but not hard). I'm 
> curious how the suggested keys method improves things. It just avoids 
> helps the provider map "*" to the list of attributes names but the 
> provider must still have semantic knowledge of each of the attributes, 
> and know each of their names.
> 
> -Alan.

If you have a way to get the attrib names programatically, then
Path.readAttributes() only needs contain code for parsing the attribute
string, and can be basically the same across all Path implementations.
Without it, you end up having to list out all of the possible attributes,
which means that you're now maintaining this list in multiple places.
(For example, the names from BasicFileAttributeView will be repeated in
every readAttributes() implementation.) This is error-prone.
 
-- 
J.

From Alan.Bateman at Sun.COM  Wed Jan 27 06:04:03 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Wed, 27 Jan 2010 14:04:03 +0000
Subject: getting known keys from an AttributeView
In-Reply-To: <20100127112004.4EA0D1007A@charybdis.ellipsis.cx>
References: <20100125203626.29ED31007A@charybdis.ellipsis.cx>
	<4B601326.4080605@sun.com>
	<20100127112004.4EA0D1007A@charybdis.ellipsis.cx>
Message-ID: <4B6047D3.50208@sun.com>

Joel Uckelman wrote:
> :
> If you have a way to get the attrib names programatically, then
> Path.readAttributes() only needs contain code for parsing the attribute
> string, and can be basically the same across all Path implementations.
> Without it, you end up having to list out all of the possible attributes,
> which means that you're now maintaining this list in multiple places.
> (For example, the names from BasicFileAttributeView will be repeated in
> every readAttributes() implementation.) This is error-prone
>   
For now, we don't have any base classes for provider implementers. It is 
something we could examine in the future once there is more experience 
developing custom providers. BasicFileAttributeView is required to be 
supported by all providers so a case could be made to define a base 
implementation class in the spi package that concrete providers could 
extend. That base class could implement set/get/readAttributes, which I 
think it what you are suggesting. In our default provider we have 
sun.nio.fs.AbstractBasicFileAttribtueView for this purpose but that 
isn't useful to other providers.

-Alan.

From sebastian.sickelmann at gmx.de  Wed Jan 27 12:36:02 2010
From: sebastian.sickelmann at gmx.de (Sebastian Sickelmann)
Date: Wed, 27 Jan 2010 21:36:02 +0100
Subject: WatchService questions
In-Reply-To: <4B576B3D.1020508@sun.com>
References: <20100119190200.117690@gmx.net> <4B576B3D.1020508@sun.com>
Message-ID: <20100127203602.147290@gmx.net>

> Alan Bateman wrote:
> Sebastian Sickelmann wrote:
> > Hi,
> >
> > in the attachment you can find the two Mercurial Exports. (Version 2A
> and Version2B)
> >   
> Thanks for updating the patches to ensure that the count isn't updated
> after the events have been retrieved. The approach in Version2B is
> probably better - as I mentioned in the previous mail, it will likely be
> a bit more efficient for the case that the user falls very behind, and
> there is a lot of activity, meaning many files being modified at around
> the same time that files are being created and deleted.
>
> BTW, did you look at that link I sent you on the contributor agreement?
Yes, i have signed it and send it in. But did no response on that yet. Should i get a response?
>
> :
> > I have found an additional interesting thing in WatchService. What are
> those Modifiers by the registering for. Should there be a Modifier to turn
> on the merging of MODIFY-Events?
> > Maybe the Level of Merging (Version2A and Version2B)?
> > It looks to me that the SensitivityModfies does some comparable feature
> than merging(reducing) MODIFY-Events.
> >   
> Modifiers are just an extension for the future so that we have a way to
> change how objects are registered. The API doesn't currently define any
> standard modifiers but you'll find two provider specific modifiers in
> the implementation. The modifier for the "sensitivity level" that you
> found is only interesting to the polling implementation (it is ignored
> by the other watch service implementations). If directories aren't
> polled very frequently then it's a bit like merging events although I'm
> not sure that it's worth adding a new modifier for this.
I am trying to create an implementation that lets you choose between Version 2A and 2B or just the "old" one. But this will take some time, i think am be back on this at the middle of February.

But for now what would be the next step to make progess for the actual send in patches? Public Review?

- Sebastian
-- 
Nur noch bis 31.01.2010: DSL-Komplettpaket f?r 16,99 Euro/mtl.!*
http://portal.gmx.net/de/go/dsl02

From Alan.Bateman at Sun.COM  Thu Jan 28 00:46:05 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Thu, 28 Jan 2010 08:46:05 +0000
Subject: zipfs and write support
In-Reply-To: <20100127111325.C26A21007A@charybdis.ellipsis.cx>
References: <20100125201029.452291007A@charybdis.ellipsis.cx>
	<4B600EDF.40701@sun.com>
	<20100127111325.C26A21007A@charybdis.ellipsis.cx>
Message-ID: <4B614ECD.1090808@sun.com>

Joel Uckelman wrote:
> :
> I don't know of any way to stop other proceses from corrupting the
> underlying ZIP file. Do you? I thought file locks were advistory on
> many OSes. I'm not familiar with what would need to be done to
> coordinate across VMs. Is there a compelling reason to do this? It's
> an unusual requirement.
>   
Yes, to be portable, it is best to assume the file locking is advisory. 
But, file locking isn't going to help when you are replacing the zip 
file (which I assume is going to happen as it is based on 
java.util.zip). An approach that I think I've suggested before is to use 
a "server" to coordinate access. That would allow applications in 
different VMs to access the "file system" concurrently. It could uses 
the default file system as a temporary location for updates and re-build 
the the zip file file when the file system is closed or the server is 
shutdown. Clearly this is would be a more ambitious project and I'm just 
mentioning it as a potential solution that someone might like to pursue.

-Alan.

From Alan.Bateman at Sun.COM  Thu Jan 28 01:17:52 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Thu, 28 Jan 2010 09:17:52 +0000
Subject: WatchService questions
In-Reply-To: <20100127203602.147290@gmx.net>
References: <20100119190200.117690@gmx.net> <4B576B3D.1020508@sun.com>
	<20100127203602.147290@gmx.net>
Message-ID: <4B615640.5060606@sun.com>

Sebastian Sickelmann wrote:
> :
> Yes, i have signed it and send it in. But did no response on that yet. Should i get a response?
>   
I checked, and it has been received , but just hasn't been "processed" yet.

> I am trying to create an implementation that lets you choose between Version 2A and 2B or just the "old" one. But this will take some time, i think am be back on this at the middle of February.
>
> But for now what would be the next step to make progess for the actual send in patches? Public Review?
>   
Sending patches to this mail list is fine. I can sponsor it once we've 
agreed the patch.

-Alan.

From uckelman at nomic.net  Thu Jan 28 01:27:49 2010
From: uckelman at nomic.net (Joel Uckelman)
Date: Thu, 28 Jan 2010 10:27:49 +0100
Subject: zipfs and write support
In-Reply-To: <4B600EDF.40701@sun.com>
References: <20100125201029.452291007A@charybdis.ellipsis.cx>
	<4B600EDF.40701@sun.com>
Message-ID: <20100128092749.C8BB610086@charybdis.ellipsis.cx>

Thus spake Alan Bateman:
>    
> It sounds like you've made good progress. If it extends the existing zip 
> provider then I assume it must still be based on java.util.zip, right? 
> Have you given any thought to access from others VM or applications? 
> It's hard to comment on the locking without more information - if there 
> a file opened for writing then does it block all other (read) access to 
> the "file system"?
> 
> -Alan.

It dawned on me just now that subtree locking isn't necessary for handling
nonoverlapping writes here, because you can neither delete nor move nonempty
directories with the Path API---anything whch can be written to is a leaf.
Per-Path locking will be easy to do, I might even finish it this evening.

-- 
J.

From gnu_andrew at member.fsf.org  Thu Jan 28 14:46:50 2010
From: gnu_andrew at member.fsf.org (Andrew John Hughes)
Date: Thu, 28 Jan 2010 22:46:50 +0000
Subject: -Werror in make/java/sun_nio/Makefile
Message-ID: <17c6771e1001281446i6bf5e070w332d5ea1942f72@mail.gmail.com>

I decided to do a build with JAVAC_MAX_WARNINGS=true just to see the
lay of the land, and it seems that this causes the sun_nio build to
bail out.  The Makefile adds -Werror itself locally:

OTHER_JAVACFLAGS += -Xlint:serial -Werror

which works fine with the default settings but causes to build to fail
when -Xlint:all is added:

/usr/lib/jvm/icedtea6/bin/java -XX:-PrintVMOptions
-XX:+UnlockDiagnosticVMOptions -XX:-LogVMOutput -Xmx896m -Xms128m
-XX:PermSize=32m -XX:MaxPermSize=160m
-Xbootclasspath/p:/mnt/builder/icedtea/langtools/dist/bootstrap/lib/javac.jar
-jar /mnt/builder/icedtea/langtools/dist/bootstrap/lib/javac.jar
-Xlint:all -source 7 -target 7 -encoding ascii
-Xbootclasspath:/mnt/builder/icedtea/classes -Xlint:serial -Werror
-sourcepath /mnt/builder/icedtea/gensrc:../../../src/solaris/classes:../../../src/share/classes
-d /mnt/builder/icedtea/classes
@/mnt/builder/icedtea/tmp/sun/sun.nio/.classes.list.filtered

There is a similar flag for -Werror (JAVAC_WARNINGS_FATAL) and this
should be used when -Werror builds are required.

The warnings that cause the build to fail are all to do with
deprecated code e.g.:

../../../src/share/classes/sun/io/ByteToCharASCII.java:33: warning:
[deprecation] ByteToCharConverter in sun.io has been deprecated
public class ByteToCharASCII extends ByteToCharConverter {
                                     ^
../../../src/share/classes/sun/io/ByteToCharASCII.java:52: warning:
[deprecation] ConversionBufferFullException in sun.io has been
deprecated
        throws ConversionBufferFullException, UnknownCharacterException

We hit the 100 warning limit so I don't know how many there are in total.

It seems that ByteToCharConverter was marked as deprecated with the
intention of removing it in JDK6:

 * @deprecated Replaced by {@link java.nio.charset}.  THIS API WILL BE
 * REMOVED IN J2SE 1.6.

This still being present in OpenJDK7 suggests that never happened.

I'd suggest we either drop the local -Werror or add -deprecation after
serial to make the build with maximum warnings complete.  Long term,
we can look at getting rid of that code if appropriate, though I doubt
we'll ever be able to do a build with the deprecation warning and
-Werror on simply because of legacy code.
-- 
Andrew :-)

Free Java Software Engineer
Red Hat, Inc. (http://www.redhat.com)

Support Free Java!
Contribute to GNU Classpath and the OpenJDK
http://www.gnu.org/software/classpath
http://openjdk.java.net

PGP Key: 94EFD9D8 (http://subkeys.pgp.net)
Fingerprint: F8EF F1EA 401E 2E60 15FA  7927 142C 2591 94EF D9D8

From Alan.Bateman at Sun.COM  Thu Jan 28 15:15:12 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Thu, 28 Jan 2010 23:15:12 +0000
Subject: -Werror in make/java/sun_nio/Makefile
In-Reply-To: <17c6771e1001281446i6bf5e070w332d5ea1942f72@mail.gmail.com>
References: <17c6771e1001281446i6bf5e070w332d5ea1942f72@mail.gmail.com>
Message-ID: <4B621A80.9060402@sun.com>

Andrew John Hughes wrote:
> :
>
> It seems that ByteToCharConverter was marked as deprecated with the
> intention of removing it in JDK6:
>
>  * @deprecated Replaced by {@link java.nio.charset}.  THIS API WILL BE
>  * REMOVED IN J2SE 1.6.
>
> This still being present in OpenJDK7 suggests that never happened.
>   
Unfortunately we haven't been able to remove the legacy converters as 
planned. JDBC drivers are the main problem. Sherman and Martin may want 
to chime in with more history on this.

> I'd suggest we either drop the local -Werror or add -deprecation after
> serial to make the build with maximum warnings complete.  Long term,
> we can look at getting rid of that code if appropriate, though I doubt
> we'll ever be able to do a build with the deprecation warning and
> -Werror on simply because of legacy code.
>   

I'd prefer not drop -Werror as that would let warnings creep in. Add 
-deprecation is probably better.

-Alan.

From uckelman at nomic.net  Fri Jan 29 05:11:16 2010
From: uckelman at nomic.net (Joel Uckelman)
Date: Fri, 29 Jan 2010 14:11:16 +0100
Subject: zipfs and write support
In-Reply-To: <4B614ECD.1090808@sun.com>
References: <20100125201029.452291007A@charybdis.ellipsis.cx>
	<4B600EDF.40701@sun.com>
	<20100127111325.C26A21007A@charybdis.ellipsis.cx>
	<4B614ECD.1090808@sun.com>
Message-ID: <20100129131117.0D06710076@charybdis.ellipsis.cx>

Thus spake Alan Bateman:
>   
> Yes, to be portable, it is best to assume the file locking is advisory. 
> But, file locking isn't going to help when you are replacing the zip 
> file (which I assume is going to happen as it is based on 
> java.util.zip). An approach that I think I've suggested before is to use 
> a "server" to coordinate access. That would allow applications in 
> different VMs to access the "file system" concurrently. It could uses 
> the default file system as a temporary location for updates and re-build 
> the the zip file file when the file system is closed or the server is 
> shutdown. Clearly this is would be a more ambitious project and I'm just 
> mentioning it as a potential solution that someone might like to pursue.
> 

This sounds like a reasonable approach. But: That's quite some way beyond
what my project needs right now, so most likely it won't be me implementing
this.

-- 
J.

From uckelman at nomic.net  Fri Jan 29 11:37:39 2010
From: uckelman at nomic.net (Joel Uckelman)
Date: Fri, 29 Jan 2010 20:37:39 +0100
Subject: zipfs patches
Message-ID: <20100129193739.D2F4010076@charybdis.ellipsis.cx>

Two more zipfs patches:

* Implemented readAttributes().
* copyTo() should return target, not this.

-------------- next part --------------
# HG changeset patch
# User Joel Uckelman <uckelman at nomic.net>
# Date 1264792812 -3600
# Node ID c057cc149682d4ff84a9aed8e85c0fd8e74b0637
# Parent  36841187ed0717208ca2862bd60d4519d18a9b17
Implemented readAttributes().

diff --git a/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFilePath.java b/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFilePath.java
--- a/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFilePath.java
+++ b/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFilePath.java
@@ -944,10 +944,72 @@
     }
 
     @Override
-    public Map<String,?> readAttributes(String attribute, LinkOption... options)
-        throws IOException
-    {
-        throw new RuntimeException("not implemented");
+    public Map<String,?> readAttributes(String attributes,
+                                        LinkOption... options)
+                                                           throws IOException {
+        return readAttributes(attributes.split(","), options);
+    }
+
+    protected Map<String,?> readAttributes(String[] attributes,
+                                           LinkOption... options)
+                                                           throws IOException {
+        final Map<String,Object> map = new HashMap<String,Object>();
+
+        for (String attr : attributes) {
+            final int colon = attr.indexOf(':');
+
+            final String view;
+            final String name;
+
+            if (colon == -1) {
+                view = "basic";
+                name = attr;
+            }
+            else {
+                view = attr.substring(0, colon);
+                name = attr.substring(colon+1, attr.length());
+            }
+
+            if ("*".equals(name)) {
+                if ("basic".equals(view)) {
+                    map.putAll(readAttributes(new String[] {
+                        "basic:lastModifiedTime",
+                        "basic:lastAccessTime",
+                        "basic:creationTime",
+                        "basic:size",
+                        "basic:isRegularFile",
+                        "basic:isDirectory",
+                        "basic:isSymbolicLink",
+                        "basic:isOther",
+                        "basic:fileKey"
+                    }));
+                }
+                else if ("zip".equals(view)) {
+                    map.putAll(readAttributes(new String[] {
+                        "zip:comment",
+                        "zip:compressedSize",
+                        "zip:crc",
+                        "zip:extra",
+                        "zip:method",
+                        "zip:name",
+                        "zip:isArchiveFile",
+                        "zip:versionMadeBy",
+                        "zip:extAttrs"
+                    }));
+                }
+                else if ("jar".equals(view)) {
+                    map.putAll(readAttributes(new String[] {
+                        "jar:manifestAttributes",
+                        "jar:entryAttributes"
+                      }));
+                }
+            }
+            else {
+                map.put(attr, getAttribute(attr));
+            }
+        }
+    
+        return map;
     }
 
     @Override
-------------- next part --------------
# HG changeset patch
# User Joel Uckelman <uckelman at nomic.net>
# Date 1264793497 -3600
# Node ID 8709878d43e6b051d6b7280c2cd51ead5de63eaa
# Parent  c057cc149682d4ff84a9aed8e85c0fd8e74b0637
copyTo() should return target, not this.

diff --git a/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFilePath.java b/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFilePath.java
--- a/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFilePath.java
+++ b/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFilePath.java
@@ -1343,7 +1343,7 @@
             }
         }
 
-        return this;
+        return target;
     }
 
     private void copyToTarget(Path target) throws IOException {

From gnu_andrew at member.fsf.org  Fri Jan 29 18:19:55 2010
From: gnu_andrew at member.fsf.org (Andrew John Hughes)
Date: Sat, 30 Jan 2010 02:19:55 +0000
Subject: -Werror in make/java/sun_nio/Makefile
In-Reply-To: <4B621A80.9060402@sun.com>
References: <17c6771e1001281446i6bf5e070w332d5ea1942f72@mail.gmail.com>
	<4B621A80.9060402@sun.com>
Message-ID: <17c6771e1001291819v4abb2fabkb4bc731664c6df03@mail.gmail.com>

On 28 January 2010 23:15, Alan Bateman <Alan.Bateman at sun.com> wrote:
> Andrew John Hughes wrote:
>>
>> :
>>
>> It seems that ByteToCharConverter was marked as deprecated with the
>> intention of removing it in JDK6:
>>
>> ?* @deprecated Replaced by {@link java.nio.charset}. ?THIS API WILL BE
>> ?* REMOVED IN J2SE 1.6.
>>
>> This still being present in OpenJDK7 suggests that never happened.
>>
>
> Unfortunately we haven't been able to remove the legacy converters as
> planned. JDBC drivers are the main problem. Sherman and Martin may want to
> chime in with more history on this.
>
>> I'd suggest we either drop the local -Werror or add -deprecation after
>> serial to make the build with maximum warnings complete. ?Long term,
>> we can look at getting rid of that code if appropriate, though I doubt
>> we'll ever be able to do a build with the deprecation warning and
>> -Werror on simply because of legacy code.
>>
>
> I'd prefer not drop -Werror as that would let warnings creep in. Add
> -deprecation is probably better.
>

Ok, that's what I've done in this webrev:

http://cr.openjdk.java.net/~andrew/warnings/webrev.01/

It also fixes a number of unchecked and redundant cast warnings that
occur when building with JAVAC_MAX_WARNINGS=true.  With this patch,
sun_io now compiles when JAVAC_MAX_WARNINGS=true.

Does this look ok?

> -Alan.
>



-- 
Andrew :-)

Free Java Software Engineer
Red Hat, Inc. (http://www.redhat.com)

Support Free Java!
Contribute to GNU Classpath and the OpenJDK
http://www.gnu.org/software/classpath
http://openjdk.java.net

PGP Key: 94EFD9D8 (http://subkeys.pgp.net)
Fingerprint: F8EF F1EA 401E 2E60 15FA  7927 142C 2591 94EF D9D8

From Alan.Bateman at Sun.COM  Sat Jan 30 11:51:59 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Sat, 30 Jan 2010 19:51:59 +0000
Subject: zipfs patches
In-Reply-To: <20100129193739.D2F4010076@charybdis.ellipsis.cx>
References: <20100129193739.D2F4010076@charybdis.ellipsis.cx>
Message-ID: <4B648DDF.9030703@sun.com>

Joel Uckelman wrote:
> Two more zipfs patches:
>
> * Implemented readAttributes().
> * copyTo() should return target, not this.
>
>   
Thanks for taking these bugs. The map key is the attribute name rather 
than view:attribute and so I assume map.put(attr, ..) should be 
map.put(name, ...). Another comment is that you've implemented it to use 
getAttribute, and if I read the patch correctly, it means that a bulk 
read turns into a sequence of gets where each reads the zip entry. With 
caching this might not be noticeable but I'm interested to know if 
you've compared it to reading the zip entry once and then populating the 
map with the requested attributes.

-Alan.