From Alan.Bateman at Sun.COM Wed Oct 14 04:45:14 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Wed, 14 Oct 2009 12:45:14 +0100 Subject: Need reviewer for 6891404: (fs) ACL tests fail with "Invalid argument" on ZFS (sol) Message-ID: <4AD5B9CA.2020205@sun.com> This one is Solaris specific and has been bugging Kelly as it causes a number of tests, that update ACLs, to fail. The bug is that we don't set the ACE_IDENTIFIER_GROUP flag on group@ entries (group@ is one of the special "who" fields defined by RFC 3530 for servers that support both ACLs and Unix permissions). The fix is trivial, and I've used the opportunity to change the code to check references rather than the principal name (as there is only one instance of each). The webrev is here: http://cr.openjdk.java.net/~alanb/6891404/webrev.00/ Thanks, Alan. From hjohn at xs4all.nl Wed Oct 14 09:22:38 2009 From: hjohn at xs4all.nl (John Hendrikx) Date: Wed, 14 Oct 2009 18:22:38 +0200 Subject: NIO feedback Message-ID: <4AD5FACE.9070908@xs4all.nl> I hope you don't mind me sending to the list, but I figured I'd share some things while working with NIO2 and perhaps gain a bit more insight in its workings. The project I'm working on may be a slightly unusual use-case for NIO2 in that it isn't just an application that wants to access the filesystem. It is more of a file management system. Reading Attributes: I found there is no way to just get all attributes that a filesystem can provide, or find out what the "most specific" AttributeView is that a filesystem provides. Something like: path.getAttributes("*:*"); is not allowed. That's ok, so I went looking for a way to figure out what system I'm running on... should I get "dos:*" or "posix:*" or even something else entirely.. I don't know. The Linux system I use here gives me the following possible views when I do: path.getFileSystem().supportedFileAttributeViews(); It results in [basic, owner, user, dos, unix, posix] which confuses me. Why is "dos" in there? So I tried querying it. I get a FileSystemException (/temp: Operation not supported). For now in my application I simply query several views. Any that throw an exception or result in an empty map are ignored. I'm left wondering if there is a better way. Writing Attributes: Above I managed to read the attributes in some fashion. Now I want to set them. I discovered some of the attributes (like "attributes" in dos) are read only. So instead I query "readOnly,hidden,archive,system", not really optimal. Especially because I cannot find a way to set multiple attributes at once. I'm left with: for(String key : attributes.keySet()) { destinationPath.setAttribute(type + ":" + key, attributes.get(key), LinkOption.NOFOLLOW_LINKS); } Again, is there a better way? What I'm trying to achieve is to simply make an exact copy of a file, including most attributes, lastModifiedTime, ACL's and owner/group information. Setting half a dozen attributes in a row seems akward. ACL: I cannot read ACL's at all under Linux (ext3), I get an UnsupportedOperationException. I hope support for this is still forthcoming. copyTo/moveTo: These are not very useful for me, I'd like to give feedback during this process (progress indicator). Also, I really miss a rename function on the Path class. Currently it is important for my application to know when a moveTo will simply be a quick rename, or when it involves actually doing a copy (in which case I want to do it manually to give progress information). I have to compare FileStores to see if it will be a copy or just a plain rename (not sure how much of a guarantee that is though). The ATOMIC_MOVE option is unclear whether this will mean "just a rename" and/or "yes we'll do an-hour-long-copy transactionally if the filesystem supports it". Still, NIO2 is definitely allowing me to do what I want with the various filesystems out there. Sometimes it feels a little akward, but I've not found anything yet that I couldn't achieve with it. Any tips are appreciated :) --John From Alan.Bateman at Sun.COM Wed Oct 14 10:21:37 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Wed, 14 Oct 2009 18:21:37 +0100 Subject: NIO feedback In-Reply-To: <4AD5FACE.9070908@xs4all.nl> References: <4AD5FACE.9070908@xs4all.nl> Message-ID: <4AD608A1.2070905@sun.com> John Hendrikx wrote: > : > > Reading Attributes: > > I found there is no way to just get all attributes that a filesystem > can provide, or find out what the "most specific" AttributeView is > that a filesystem provides. Something like: > > path.getAttributes("*:*"); > > is not allowed. I think this is what you are looking for: Path file = ... FileStore store = file.getFileStore(); for (String view: file.getFileSystem().supportedFileAttributeViews()) { if (store.supportsFileAttributeView(view)) { Map attrs = file.readAttributes(view + ":*"); } } This iterates over the supported views and queries the FileStore (the underlying file system in Unix/Linux terms) to see if it supported. If supported then it reads all attributes that the view supports. As should become clear, the exact list of attributes depends on the underlying file system and so can/will vary if you have different types of file system mounted. > : > > It results in [basic, owner, user, dos, unix, posix] which confuses > me. Why is "dos" in there? So I tried querying it. I get a > FileSystemException (/temp: Operation not supported). > The "dos" view is supported when you have extended attributes enabled. I think in one of your other mails you mentioned that you were using Samba and if you have this configured to store dos attributes then you can you use the dos view to read/write the DOS attributes that the clients see. The "Operation not supported" is probably because it's not mounted with the user_xattr mount (use "mount -o remount,user_xattr ..." to resolve that). Once the file system is mounted with this option then supportsFileAttributeView("dos") will return true (I assume it is returning false now). > > Writing Attributes: > > Above I managed to read the attributes in some fashion. Now I want to > set them. I discovered some of the attributes (like "attributes" in > dos) are read only. So instead I query > "readOnly,hidden,archive,system", not really optimal. Especially > because I cannot find a way to set multiple attributes at once. I'm > left with: > > for(String key : attributes.keySet()) { > destinationPath.setAttribute(type + ":" + key, > attributes.get(key), LinkOption.NOFOLLOW_LINKS); > } > > Again, is there a better way? What I'm trying to achieve is to simply > make an exact copy of a file, including most attributes, > lastModifiedTime, ACL's and owner/group information. Setting half a > dozen attributes in a row seems akward. The simplest way to do this is: source.copyTo(target, COPY_ATTRIBUTES); This will attempt to copy all the attributes from the source to the target file (including basic attributes, permissions, DOS, ACLs, extended attributes, etc.). (I might not be understanding the full context here and maybe you are rolling your own file copy??). > > ACL: > > I cannot read ACL's at all under Linux (ext3), I get an > UnsupportedOperationException. I hope support for this is still > forthcoming. This API is based on the NFSv4 ACL model. I don't think any of the Linux distributions ship with support for this yet. When they do then we can add support for the "acl" view. > > copyTo/moveTo: > > These are not very useful for me, I'd like to give feedback during > this process (progress indicator). Our default provider doesn't have any hooks or notifications for this. It's definitely something that a "desktop" provider would need to support. > Also, I really miss a rename function on the Path class. Currently it > is important for my application to know when a moveTo will simply be a > quick rename, or when it involves actually doing a copy (in which case > I want to do it manually to give progress information). I have to > compare FileStores to see if it will be a copy or just a plain rename > (not sure how much of a guarantee that is though). The ATOMIC_MOVE > option is unclear whether this will mean "just a rename" and/or "yes > we'll do an-hour-long-copy transactionally if the filesystem supports > it". The assumption has been that source.moveTo(target) or source.moveTo(target, REPLACE_EXISTING) will satisfy most needs. The ATOMIC_MOVE option is the advanced option but it's essentially a rename. The difference is that a "simple rename" can check if the target file exists and so is not atomic. We have an RFE for this: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6878283 In the mean-time, you can do this: if (target.notExists()) source.moveTo(target, ATOMIC_MOVE); -Alan. From hjohn at xs4all.nl Thu Oct 15 04:16:41 2009 From: hjohn at xs4all.nl (John Hendrikx) Date: Thu, 15 Oct 2009 13:16:41 +0200 Subject: NIO feedback In-Reply-To: <4AD608A1.2070905@sun.com> References: <4AD5FACE.9070908@xs4all.nl> <4AD608A1.2070905@sun.com> Message-ID: <4AD70499.1040400@xs4all.nl> Alan Bateman wrote: > John Hendrikx wrote: >> >> It results in [basic, owner, user, dos, unix, posix] which confuses >> me. Why is "dos" in there? So I tried querying it. I get a >> FileSystemException (/temp: Operation not supported). >> > The "dos" view is supported when you have extended attributes enabled. > I think in one of your other mails you mentioned that you were using > Samba and if you have this configured to store dos attributes then you > can you use the dos view to read/write the DOS attributes that the > clients see. The "Operation not supported" is probably because it's > not mounted with the user_xattr mount (use "mount -o > remount,user_xattr ..." to resolve that). Once the file system is > mounted with this option then supportsFileAttributeView("dos") will > return true (I assume it is returning false now). It was not a Samba mount in this case, it was an ext3 filesystem mounted on /raid1 -- I have not checked what FileStore's supportsFileAttributeView will return, but if I understand you correctly it is possible that FileSystem's supportedFileAttributeViews() can return views which are in turn returning false for FileStore's supportsFileAttributeView(xxx) ? I will check this when I get home. > >> >> Writing Attributes: >> >> Above I managed to read the attributes in some fashion. Now I want >> to set them. I discovered some of the attributes (like "attributes" >> in dos) are read only. So instead I query >> "readOnly,hidden,archive,system", not really optimal. Especially >> because I cannot find a way to set multiple attributes at once. I'm >> left with: >> >> for(String key : attributes.keySet()) { >> destinationPath.setAttribute(type + ":" + key, >> attributes.get(key), LinkOption.NOFOLLOW_LINKS); >> } >> >> Again, is there a better way? What I'm trying to achieve is to >> simply make an exact copy of a file, including most attributes, >> lastModifiedTime, ACL's and owner/group information. Setting half a >> dozen attributes in a row seems akward. > The simplest way to do this is: > source.copyTo(target, COPY_ATTRIBUTES); > This will attempt to copy all the attributes from the source to the > target file (including basic attributes, permissions, DOS, ACLs, > extended attributes, etc.). > > (I might not be understanding the full context here and maybe you are > rolling your own file copy??). I'm rolling my own copy to be able to give progress information and was looking for a way to make a nearly exact copy. I did not see a more framework supported way of doing it (although you gave a hint below about a desktop provider). I'm using Path's newByteChannel(Set options, FileAttribute... attrs) for the copy, however I saw no good way of providing all the FileAttribute's. There seems to be only one method that even creates these FileAttribute instances, and it only does so for posix permissions. > > This API is based on the NFSv4 ACL model. I don't think any of the > Linux distributions ship with support for this yet. When they do then > we can add support for the "acl" view. Yeah, it doesn't look like the support for these is present in the latest kernel's yet. > The assumption has been that source.moveTo(target) or > source.moveTo(target, REPLACE_EXISTING) will satisfy most needs. The > ATOMIC_MOVE option is the advanced option but it's essentially a > rename. The difference is that a "simple rename" can check if the > target file exists and so is not atomic. We have an RFE for this: > http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6878283 > > In the mean-time, you can do this: > if (target.notExists()) source.moveTo(target, ATOMIC_MOVE); The RFE would solve this. My only concern was that highly advanced implementations might try to do the move wrapped in a transaction somehow (which would satisfy the ATOMIC_MOVE requirement) but still take a long time as they're doing a full copy in the background. Thanks, John From Alan.Bateman at Sun.COM Thu Oct 15 05:47:23 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Thu, 15 Oct 2009 13:47:23 +0100 Subject: NIO feedback In-Reply-To: <4AD70499.1040400@xs4all.nl> References: <4AD5FACE.9070908@xs4all.nl> <4AD608A1.2070905@sun.com> <4AD70499.1040400@xs4all.nl> Message-ID: <4AD719DB.400@sun.com> John Hendrikx wrote: > It was not a Samba mount in this case, it was an ext3 filesystem > mounted on /raid1 -- I have not checked what FileStore's > supportsFileAttributeView will return, but if I understand you > correctly it is possible that FileSystem's > supportedFileAttributeViews() can return views which are in turn > returning false for FileStore's supportsFileAttributeView(xxx) ? Yes, that's right. A provider/FileSystem implementation supports a set of views but a specific underlying file system/volume might not support all of them. A simple example to explain this is where you are on Windows and some volumes are NTFS and others are FAT32. If a FileStore represents one of the NTFS volumes then it will return true if you ask it if supports AclFileAttributeView and UserDefinedFileAttributeView. Ask a FileStore representing a FAT32 volume if it supports these view and it will return "false". > I'm rolling my own copy to be able to give progress information and > was looking for a way to make a nearly exact copy. I did not see a > more framework supported way of doing it (although you gave a hint > below about a desktop provider). > > I'm using Path's newByteChannel(Set options, > FileAttribute... attrs) for the copy, however I saw no good way of > providing all the FileAttribute's. There seems to be only one method > that even creates these FileAttribute instances, and it only does so > for posix permissions. There isn't a "framework" for doing your own file-copy. The assumption is that the copyTo defines copy options to support most needs and the provider can support additional provider-specific options where required. So this means you need to copy the attributes in a loop with code like this: FileStore sourceStore = source.getFileStore(); FileStore targetStore = target.getFileStore(); for (String view: source.getFileSystem().supportedFileAttributeViews()) { if (sourceStore.supportsFileAttributeView(view) && targetStore.supportsFileAttributeView(view)) { Map attrs = source.readAttributes(view + ":*"); for (Map.Entry attr: attrs.entrySet()) { try { target.setAttribute(view + ":" + attr.getKey(), attr.getValue()); } catch (UnsupportedOperationException x) { } } } } This is clearly not very efficient in that it will copy some attributes more than once (because attribute views can extend or overlap with other views) but its probably not too bad because most of the time will be spent copying the file contents. In any case, I'll create a bug for tracking progress. The other one that has come up is cancellation (our implementation has support for this already). Another one is the ability to be selective as to which attributes are copied and to indicate if the method should fail/continue if specific attributes cannot be copied (our implementation has this capability but it isn't exposed as a CopyOption). As regards the newByteChannel method (and other methods that allow initial attributes to be specified as an array of FileAttributes). The use-case for this is file permissions and other security attributes (including ACLs) that you want to set when creating the file to avoid the timing window that would arise if you created the file and set the permissions afterwards. In other words, it's for security use-cases and is not intended as a way to set all the initial attributes (as that is just not feasible). Note that FileAttribute just encapsulates a name/value pair and you need to eliminate it in order to provider an instance to the newByteChannel method. The PosixFilePermissions.asFileAttribute method is just for the common-case where you want to set the file permissions when creating a file. Hope this helps, -Alan. From Xueming.Shen at Sun.COM Thu Oct 15 12:51:35 2009 From: Xueming.Shen at Sun.COM (Xueming Shen) Date: Thu, 15 Oct 2009 12:51:35 -0700 Subject: [Fwd: Need reviewer for 6891404: (fs) ACL tests fail with "Invalid argument" on ZFS (sol)] In-Reply-To: <4AD633AD.5010500@sun.com> References: <4AD633AD.5010500@sun.com> Message-ID: <4AD77D47.4070003@sun.com> approved. Alan Bateman wrote: > Will you be able to review this one? > > ------------------------------------------------------------------------ > > Subject: > Need reviewer for 6891404: (fs) ACL tests fail with "Invalid argument" > on ZFS (sol) > From: > Alan Bateman > Date: > Wed, 14 Oct 2009 12:45:14 +0100 > To: > nio-dev , Kelly O'Hair > > To: > nio-dev , Kelly O'Hair > > > > This one is Solaris specific and has been bugging Kelly as it causes a > number of tests, that update ACLs, to fail. The bug is that we don't > set the ACE_IDENTIFIER_GROUP flag on group@ entries (group@ is one of > the special "who" fields defined by RFC 3530 for servers that support > both ACLs and Unix permissions). The fix is trivial, and I've used the > opportunity to change the code to check references rather than the > principal name (as there is only one instance of each). The webrev is > here: > http://cr.openjdk.java.net/~alanb/6891404/webrev.00/ > > Thanks, > > Alan. From Alan.Bateman at Sun.COM Sat Oct 17 14:22:16 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Sat, 17 Oct 2009 22:22:16 +0100 Subject: Need reviewer for a few test updates Message-ID: <4ADA3588.5000406@sun.com> I need a reviewer for two test bugs that have been bugging Kelly. 6892710 is tracking two tests that fail when run in jtreg samevm mode. Neither test can run in this mode and need the @run tag to force the run to be in othervm mode. The CheckPermissions test fails because it needs to set a security manager. ContentType fails because the FileTypeDetector used by the test can't be located. The changes are trivial: --- a/test/java/nio/file/Files/ContentType.java Thu Oct 15 22:47:56 2009 -0700 +++ b/test/java/nio/file/Files/ContentType.java Sat Oct 17 22:15:04 2009 +0100 @@ -26,7 +26,7 @@ * @summary Unit test for probeContentType method * @library .. * @build ContentType SimpleFileTypeDetector - * @run main ContentType + * @run main/othervm ContentType */ import java.nio.file.*; diff -r 368bb282464a test/java/nio/file/Path/CheckPermissions.java --- a/test/java/nio/file/Path/CheckPermissions.java Thu Oct 15 22:47:56 2009 -0700 +++ b/test/java/nio/file/Path/CheckPermissions.java Sat Oct 17 22:15:04 2009 +0100 @@ -25,6 +25,8 @@ * @bug 6866804 * @summary Unit test for java.nio.file.Path * @library .. + * @build CheckPermissions + * @run main/othervm CheckPermissions */ 6892711 is that test/java/nio/file/Path/CopyAndMove.java is failing on JPRT windows-i586 clients. The failure only duplicates reliably on Windows 2000 and while that edition is ancient and unsupported it does highlight that the test doesn't take into account that the DOS archive attribute may be set automatically. The right thing is to simply remove the check from the test. diff -r 368bb282464a test/java/nio/file/Path/CopyAndMove.java --- a/test/java/nio/file/Path/CopyAndMove.java Thu Oct 15 22:47:56 2009 -0700 +++ b/test/java/nio/file/Path/CopyAndMove.java Sat Oct 17 22:15:04 2009 +0100 @@ -92,7 +92,6 @@ public class CopyAndMove { { assertTrue(attrs1.isReadOnly() == attrs2.isReadOnly()); assertTrue(attrs1.isHidden() == attrs2.isHidden()); - assertTrue(attrs1.isArchive() == attrs2.isArchive()); assertTrue(attrs1.isSystem() == attrs2.isSystem()); } Thanks, Alan. From Kelly.Ohair at Sun.COM Sat Oct 17 14:47:30 2009 From: Kelly.Ohair at Sun.COM (Kelly O'Hair) Date: Sat, 17 Oct 2009 14:47:30 -0700 Subject: Need reviewer for a few test updates In-Reply-To: <4ADA3588.5000406@sun.com> References: <4ADA3588.5000406@sun.com> Message-ID: <4ADA3B72.1040907@sun.com> They look fine to me. But I suspect a nio person should review it. ;^) -kto Alan Bateman wrote: > I need a reviewer for two test bugs that have been bugging Kelly. > > 6892710 is tracking two tests that fail when run in jtreg samevm mode. > Neither test can run in this mode and need the @run tag to force the run > to be in othervm mode. The CheckPermissions test fails because it needs > to set a security manager. ContentType fails because the > FileTypeDetector used by the test can't be located. The changes are > trivial: > > --- a/test/java/nio/file/Files/ContentType.java Thu Oct 15 22:47:56 2009 > -0700 > +++ b/test/java/nio/file/Files/ContentType.java Sat Oct 17 22:15:04 2009 > +0100 > @@ -26,7 +26,7 @@ > * @summary Unit test for probeContentType method > * @library .. > * @build ContentType SimpleFileTypeDetector > - * @run main ContentType > + * @run main/othervm ContentType > */ > > import java.nio.file.*; > diff -r 368bb282464a test/java/nio/file/Path/CheckPermissions.java > --- a/test/java/nio/file/Path/CheckPermissions.java Thu Oct 15 > 22:47:56 2009 -0700 > +++ b/test/java/nio/file/Path/CheckPermissions.java Sat Oct 17 > 22:15:04 2009 +0100 > @@ -25,6 +25,8 @@ > * @bug 6866804 > * @summary Unit test for java.nio.file.Path > * @library .. > + * @build CheckPermissions > + * @run main/othervm CheckPermissions > */ > > 6892711 is that test/java/nio/file/Path/CopyAndMove.java is failing on > JPRT windows-i586 clients. The failure only duplicates reliably on > Windows 2000 and while that edition is ancient and unsupported it does > highlight that the test doesn't take into account that the DOS archive > attribute may be set automatically. The right thing is to simply remove > the check from the test. > > diff -r 368bb282464a test/java/nio/file/Path/CopyAndMove.java > --- a/test/java/nio/file/Path/CopyAndMove.java Thu Oct 15 22:47:56 2009 > -0700 > +++ b/test/java/nio/file/Path/CopyAndMove.java Sat Oct 17 22:15:04 2009 > +0100 > @@ -92,7 +92,6 @@ public class CopyAndMove { > { > assertTrue(attrs1.isReadOnly() == attrs2.isReadOnly()); > assertTrue(attrs1.isHidden() == attrs2.isHidden()); > - assertTrue(attrs1.isArchive() == attrs2.isArchive()); > assertTrue(attrs1.isSystem() == attrs2.isSystem()); > } > > Thanks, > > Alan. > > > From uckelman at nomic.net Sun Oct 18 10:35:20 2009 From: uckelman at nomic.net (Joel Uckelman) Date: Sun, 18 Oct 2009 19:35:20 +0200 Subject: zipfs problems Message-ID: <20091018173520.F0AE91008B@charybdis.ellipsis.cx> Hi, I've been looking at the zipfs demo and have noticed some problems: * The HashSet ZipFileSystem.closeableObjects could be modified from two different threads if each thread calls ZipFilePath.newInputStream() or newByteChannel() at about the same time. * Closeables are only removed from ZipFileSystem.closeableObjects when the ZipFileSystem is closed. This means that closed streams and channels will stick around long after they could otherwise be gc'd. * ZipFileSystemProvider.fileSystems has concurrency problems similar to ZipFileSystem.closeableObjects. * Should ZipFileSystemProvider.newFileSystem() really return two distinct FileSystem objects when called with two different env Maps? -- J. From Alan.Bateman at Sun.COM Sun Oct 18 12:26:56 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Sun, 18 Oct 2009 20:26:56 +0100 Subject: zipfs problems In-Reply-To: <20091018173520.F0AE91008B@charybdis.ellipsis.cx> References: <20091018173520.F0AE91008B@charybdis.ellipsis.cx> Message-ID: <4ADB6C00.7050806@sun.com> Joel Uckelman wrote: > Hi, > > I've been looking at the zipfs demo and have noticed some problems: > > * The HashSet ZipFileSystem.closeableObjects could be modified from two > different threads if each thread calls ZipFilePath.newInputStream() or > newByteChannel() at about the same time. > > * Closeables are only removed from ZipFileSystem.closeableObjects when > the ZipFileSystem is closed. This means that closed streams and channels > will stick around long after they could otherwise be gc'd. > > * ZipFileSystemProvider.fileSystems has concurrency problems similar to > ZipFileSystem.closeableObjects. > You are right and there are concurrency issues here. Also all returned closeables should be wrapped so that close method removes them from the set. Rajendra, do you have cycles to fix this? (Joel - as background, the zip provider is a demo provider that was developed to exercise/test the provider mechanism. It needs a bit of work to turn it into a robust provider) > * Should ZipFileSystemProvider.newFileSystem() really return two distinct > FileSystem objects when called with two different env Maps? > I don't see a problem here as the newFileSystem methods are required to return a new file system (irrespective of the entries in the map). Note that it's okay for several FileSystem instances to share access to the underlying zip file and this can be transparent to the user of the API. -Alan. From Alan.Bateman at Sun.COM Sun Oct 18 12:31:25 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Sun, 18 Oct 2009 20:31:25 +0100 Subject: Need reviewer for a few test updates In-Reply-To: <4ADA3B72.1040907@sun.com> References: <4ADA3588.5000406@sun.com> <4ADA3B72.1040907@sun.com> Message-ID: <4ADB6D0D.4000307@sun.com> Kelly O'Hair wrote: > > They look fine to me. But I suspect a nio person should review it. ;^) > > -kto Thanks Kelly. I'll get these into b75. -Alan. From uckelman at nomic.net Sun Oct 18 12:52:15 2009 From: uckelman at nomic.net (Joel Uckelman) Date: Sun, 18 Oct 2009 21:52:15 +0200 Subject: zipfs problems In-Reply-To: <4ADB6C00.7050806@sun.com> References: <20091018173520.F0AE91008B@charybdis.ellipsis.cx> <4ADB6C00.7050806@sun.com> Message-ID: <20091018195216.5D6071008B@charybdis.ellipsis.cx> Thus spake Alan Bateman: > Joel Uckelman wrote: > > Hi, > > > > I've been looking at the zipfs demo and have noticed some problems: > > > > * The HashSet ZipFileSystem.closeableObjects could be modified from two > > different threads if each thread calls ZipFilePath.newInputStream() or > > newByteChannel() at about the same time. > > > > * Closeables are only removed from ZipFileSystem.closeableObjects when > > the ZipFileSystem is closed. This means that closed streams and channels > > will stick around long after they could otherwise be gc'd. > > > > * ZipFileSystemProvider.fileSystems has concurrency problems similar to > > ZipFileSystem.closeableObjects. > > > You are right and there are concurrency issues here. Also all returned > closeables should be wrapped so that close method removes them from the > set. Rajendra, do you have cycles to fix this? Actually, I've already done it for closeableObjects. The only issue I'm wondering aobut is whether to use Collections.synchronizedSet() or to build a Set wrapper for a ConcurrentHashMap. I'll be sending in the contributor agreement when I get to the office tomorrow (I can't print at home), so I'll be happy to start sumitting patches then. > > * Should ZipFileSystemProvider.newFileSystem() really return two distinct > > FileSystem objects when called with two different env Maps? > > > I don't see a problem here as the newFileSystem methods are required to > return a new file system (irrespective of the entries in the map). Note > that it's okay for several FileSystem instances to share access to the > underlying zip file and this can be transparent to the user of the API. I'm viewing this with the intention of adding the ability to write ZIP archives also. Having multiple FileSystem instances for the same archive and doing some writes with each one might lead to trouble. OTOH, maybe the place for ensuring that the state of the archive remains consistent is ZipFileStore---every FileSystem for the same archive could share one ZipFileStore. It's not clear to me what should happen if someone tries to open multiple streams or channels on the same archive entry. With a disk filesystem, there's nothing stopping you from opening a FileInputStream and a FileOutputStream on the same file at the same time. In some limited cases, it might not even be wrong, e.g., if you're reading from and writing to non-overlapping parts of a file. Various other pathological things are possible... If you close() a read-write archive, that should cause whatever writing was done (by writing to the temporary file cache) to be written back to the archive---which could mean that if there are other ZipFileSystem instances for the same archive, then the archive will have changed out from under them. Whether this is a problem, I'm unsure at present. Maybe preventing this should just be the responsibility of the user? Hmm. -- J. From Xueming.Shen at Sun.COM Sun Oct 18 12:50:11 2009 From: Xueming.Shen at Sun.COM (Xueming Shen) Date: Sun, 18 Oct 2009 12:50:11 -0700 Subject: 6879463 In-Reply-To: <4AD4E3CD.7030505@sun.com> References: <4AD4E3CD.7030505@sun.com> Message-ID: <4ADB7173.5010100@sun.com> looks fine. Alan Bateman wrote: > Sherman, > > Do you mind reviewing this JCK failure? The openjdk servers have been > down so I couldn't push it to cr.openjdk.java.net. The bug is that > the iterator methods are specific to throw a > ConcurrentModificationException with cause > ClosedDirectoryStreamException if invoked when the stream is closed. > Unfortunately I didn't have this quite right in a few places. > > Thanks, > > > -Alan. > > diff -r f6770138c0fa > src/solaris/classes/sun/nio/fs/UnixDirectoryStream.java > --- a/src/solaris/classes/sun/nio/fs/UnixDirectoryStream.java Sat > Oct 10 10:14:51 2009 +0100 > +++ b/src/solaris/classes/sun/nio/fs/UnixDirectoryStream.java Tue > Oct 13 21:28:04 2009 +0100 > @@ -235,7 +235,8 @@ class UnixDirectoryStream > @Override > public void remove() { > if (isClosed) { > - throw new ClosedDirectoryStreamException(); > + throwAsConcurrentModificationException(new > + ClosedDirectoryStreamException()); > } > Path entry; > synchronized (this) { > diff -r f6770138c0fa > src/windows/classes/sun/nio/fs/WindowsDirectoryStream.java > --- > a/src/windows/classes/sun/nio/fs/WindowsDirectoryStream.java > Sat Oct 10 10:14:51 2009 +0100 > +++ > b/src/windows/classes/sun/nio/fs/WindowsDirectoryStream.java > Tue Oct 13 21:28:04 2009 +0100 > @@ -179,7 +179,7 @@ class WindowsDirectoryStream > synchronized (closeLock) { > if (!isOpen) > throwAsConcurrentModificationException(new > - IllegalStateException("Directory stream > is closed")); > + ClosedDirectoryStreamException()); > try { > name = FindNextFile(handle, > findDataBuffer.address()); > if (name == null) { > @@ -236,7 +236,8 @@ class WindowsDirectoryStream > @Override > public void remove() { > if (!isOpen) { > - throw new IllegalStateException("Directory stream is > closed"); > + throwAsConcurrentModificationException(new > + ClosedDirectoryStreamException()); > } > Path entry; > synchronized (this) { > diff -r f6770138c0fa test/java/nio/file/DirectoryStream/Basic.java > --- a/test/java/nio/file/DirectoryStream/Basic.java Sat Oct 10 > 10:14:51 2009 +0100 > +++ b/test/java/nio/file/DirectoryStream/Basic.java Tue Oct 13 > 21:28:04 2009 +0100 > @@ -154,8 +154,10 @@ public class Basic { > stream.close(); > > // test IllegalStateException > + dir.resolve(foo).createFile(); > stream = dir.newDirectoryStream(); > i = stream.iterator(); > + i.next(); > try { > stream.iterator(); > throw new RuntimeException("IllegalStateException not > thrown as expected"); > @@ -172,17 +174,26 @@ public class Basic { > throw new > RuntimeException("ConcurrentModificationException not thrown as > expected"); > } catch (ConcurrentModificationException x) { > Throwable t = x.getCause(); > - if (!(t instanceof IllegalStateException)) > - throw new RuntimeException("Cause is not > IllegalStateException as expected"); > + if (!(t instanceof ClosedDirectoryStreamException)) > + throw new RuntimeException("Cause is not > ClosedDirectoryStreamException as expected"); > } > try { > i.next(); > - throw new RuntimeException("IllegalStateException not > thrown as expected"); > - } catch (ConcurrentModificationException x) { > - Throwable t = x.getCause(); > - if (!(t instanceof IllegalStateException)) > - throw new RuntimeException("Cause is not > IllegalStateException as expected"); > - } > + throw new > RuntimeException("ConcurrentModificationException not thrown as > expected"); > + } catch (ConcurrentModificationException x) { > + Throwable t = x.getCause(); > + if (!(t instanceof ClosedDirectoryStreamException)) > + throw new RuntimeException("Cause is not > ClosedDirectoryStreamException as expected"); > + } > + try { > + i.remove(); > + throw new > RuntimeException("ConcurrentModificationException not thrown as > expected"); > + } catch (ConcurrentModificationException x) { > + Throwable t = x.getCause(); > + if (!(t instanceof ClosedDirectoryStreamException)) > + throw new RuntimeException("Cause is not > ClosedDirectoryStreamException as expected"); > + } > + > } > > public static void main(String[] args) throws IOException { > > > > > > From uckelman at nomic.net Sun Oct 18 12:54:12 2009 From: uckelman at nomic.net (Joel Uckelman) Date: Sun, 18 Oct 2009 21:54:12 +0200 Subject: zipfs problems In-Reply-To: <4ADB6C00.7050806@sun.com> References: <20091018173520.F0AE91008B@charybdis.ellipsis.cx> <4ADB6C00.7050806@sun.com> Message-ID: <20091018195412.6AC1D1008B@charybdis.ellipsis.cx> Thus spake Alan Bateman: > > (Joel - as background, the zip provider is a demo provider that was > developed to exercise/test the provider mechanism. It needs a bit of > work to turn it into a robust provider) > I'd love to hear about the areas where it's weak. I might put some time into fixing them, if someone points them out for me. -- J. From Alan.Bateman at Sun.COM Mon Oct 19 04:23:06 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Mon, 19 Oct 2009 12:23:06 +0100 Subject: zipfs problems In-Reply-To: <20091018195216.5D6071008B@charybdis.ellipsis.cx> References: <20091018173520.F0AE91008B@charybdis.ellipsis.cx> <4ADB6C00.7050806@sun.com> <20091018195216.5D6071008B@charybdis.ellipsis.cx> Message-ID: <4ADC4C1A.1060009@sun.com> Joel Uckelman wrote: > : > Actually, I've already done it for closeableObjects. The only issue I'm > wondering aobut is whether to use Collections.synchronizedSet() or to > build a Set wrapper for a ConcurrentHashMap. > A synchronizedSet should be fine. The other part to the fix is each returned InputStream, SeekableByteChannel, DirectoryStream, etc. needs its close method to remove the closeable from the set. > I'll be sending in the contributor agreement when I get to the office > tomorrow (I can't print at home), so I'll be happy to start sumitting > patches then. > Great! > : > I'm viewing this with the intention of adding the ability to write ZIP > archives also. Having multiple FileSystem instances for the same archive > and doing some writes with each one might lead to trouble. OTOH, maybe > the place for ensuring that the state of the archive remains consistent > is ZipFileStore---every FileSystem for the same archive could share one > ZipFileStore. > > It's not clear to me what should happen if someone tries to open multiple > streams or channels on the same archive entry. With a disk filesystem, > there's nothing stopping you from opening a FileInputStream and a > FileOutputStream on the same file at the same time. In some limited cases, > it might not even be wrong, e.g., if you're reading from and writing to > non-overlapping parts of a file. Various other pathological things are > possible... > > If you close() a read-write archive, that should cause whatever writing > was done (by writing to the temporary file cache) to be written back to > the archive---which could mean that if there are other ZipFileSystem > instances for the same archive, then the archive will have changed out > from under them. Whether this is a problem, I'm unsure at present. Maybe > preventing this should just be the responsibility of the user? Hmm. > The zip provider is read-only because it is based on java.util.zip and so don't have support for updating zip/JAR files (it would require re-writing the entire zip file). I assume, if you are looking to add update support then you're probably basing it on something else - is that right? As regards concurrency, then the simplest approach is opening a zip entry for writing requires exclusive access to that entry. That would be good start anyway. Asynchronously closing the FileSystem requires that you have an asynchronous close solution for each of the channel, streams, etc. that might be open. The required behavior for channel is well defined but not so for input/output streams. Closing a FileSystem should not impact anything that is accessing the zip/JAR file via another FileSystem instance. You'll likely need to use file-locking to coordinate access with other entities on the system. -Alan. From Alan.Bateman at Sun.COM Mon Oct 19 04:24:13 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Mon, 19 Oct 2009 12:24:13 +0100 Subject: Need reviewer for 6884480: (file) Path.relativize has typo in parameter description Message-ID: <4ADC4C5D.8090609@sun.com> I need a reviewer for a small typo in Path#relativize's javadoc that Jon noticed. The description of the "other" param isn't correct. While fixing that, I noticed that Path#isSameFile has several references to FilePath that should be Path so I'll fix that while I'm there. The diffs are attached. Thanks, Alan. diff --git a/src/share/classes/java/nio/file/Path.java b/src/share/classes/java/nio/file/Path.java --- a/src/share/classes/java/nio/file/Path.java +++ b/src/share/classes/java/nio/file/Path.java @@ -412,7 +412,7 @@ public abstract class Path * dependent if {@code "a/b/../x"} would locate the same file as {@code "/a/x"}. * * @param other - * the resulting path + * the path to be relativized against this path * * @return the resulting relative path, or {@code null} if both paths are * equal @@ -1615,23 +1615,23 @@ public abstract class Path * Tests if the file referenced by this object is the same file referenced * by another object. * - *

If this {@code FileRef} and the given {@code FileRef} are {@link + *

If this {@code Path} and the given {@code Path} are {@link * #equals(Object) equal} then this method returns {@code true} without checking - * if the file exists. If the {@code FileRef} and the given {@code FileRef} - * are associated with different providers, or the given {@code FileRef} is + * if the file exists. If the {@code Path} and the given {@code Path} + * are associated with different providers, or the given {@code Path} is * {@code null} then this method returns {@code false}. Otherwise, this method - * checks if both {@code FileRefs} locate the same file, and depending on the + * checks if both {@code Paths} locate the same file, and depending on the * implementation, may require to open or access both files. * *

If the file system and files remain static, then this method implements - * an equivalence relation for non-null {@code FileRefs}. + * an equivalence relation for non-null {@code Paths}. *

    - *
  • It is reflexive: for a non-null {@code FileRef} {@code f}, + *
  • It is reflexive: for a non-null {@code Path} {@code f}, * {@code f.isSameFile(f)} should return {@code true}. - *
  • It is symmetric: for two non-null {@code FileRefs} + *
  • It is symmetric: for two non-null {@code Path} * {@code f} and {@code g}, {@code f.isSameFile(g)} will equal * {@code g.isSameFile(f)}. - *
  • It is transitive: for three {@code FileRefs} + *
  • It is transitive: for three {@code Paths} * {@code f}, {@code g}, and {@code h}, if {@code f.isSameFile(g)} returns * {@code true} and {@code g.isSameFile(h)} returns {@code true}, then * {@code f.isSameFile(h)} will return return {@code true}. From uckelman at nomic.net Mon Oct 19 05:24:33 2009 From: uckelman at nomic.net (Joel Uckelman) Date: Mon, 19 Oct 2009 14:24:33 +0200 Subject: zipfs problems In-Reply-To: <4ADC4C1A.1060009@sun.com> References: <20091018173520.F0AE91008B@charybdis.ellipsis.cx> <4ADB6C00.7050806@sun.com> <20091018195216.5D6071008B@charybdis.ellipsis.cx> <4ADC4C1A.1060009@sun.com> Message-ID: <20091019122433.7B27D10074@charybdis.ellipsis.cx> Thus spake Alan Bateman: > > The zip provider is read-only because it is based on java.util.zip and > so don't have support for updating zip/JAR files (it would require > re-writing the entire zip file). I assume, if you are looking to add > update support then you're probably basing it on something else - is > that right? > No, what I've already written (separately from any fs work) is based on java.util.zip, because I didn't want to write the ZIP code from scratch. When an entry is opened for writing, it's copied out to a temporary file (much like the zipfs implementation does when you request a SeekableByteChannel). When a modified archive is closed (or flushed), I use ZipOutputStream to create a new archive, copying the modified files from their temporary locations and the unmodified files from the original archive, and finally replace the original archive with the updated one. I suppose this could be done more efficiently with some custom code, as there's no point in uncompressing and then recompressing entries that weren't modified, but writes happen infrequently enough in VASSAL (the project where we're using this) that I haven't bothered with it yet. > As regards concurrency, then the simplest approach is opening a zip > entry for writing requires exclusive access to that entry. That would be > good start anyway. I already have code for it in my old implementation which I could contribute, but see below... > Asynchronously closing the FileSystem requires that you have an > asynchronous close solution for each of the channel, streams, etc. that > might be open. The required behavior for channel is well defined but not > so for input/output streams. Closing a FileSystem should not impact > anything that is accessing the zip/JAR file via another FileSystem > instance. You'll likely need to use file-locking to coordinate access > with other entities on the system. I don't follow. What is the problem with asynchronously closing streams that you're referring to? Is it just that not all streams are thread safe? I'm a bit hazy on whether locking belongs in an FS implementation at all (given the current design). E.g, the default FS doesn't provide any way of locking itself, though it happens that you can lock files by getting FileLocks from the FileChannels returned by the default FS. For zipfs, the Channels aren't FileChannels, so you can't lock ZIP entries. On the other hand, if zipfs doesn't provide any kind of locking mechanism, then it's going to be hard to use it in a multithreaded app. Maybe what I'm moving toward is that Path should have methods for acquiring read and write locks. -- J. From Christopher.Hegarty at Sun.COM Mon Oct 19 08:28:02 2009 From: Christopher.Hegarty at Sun.COM (Christopher Hegarty - Sun Microsystems Ireland) Date: Mon, 19 Oct 2009 16:28:02 +0100 Subject: Need reviewer for 6884480: (file) Path.relativize has typo in parameter description In-Reply-To: <4ADC4C5D.8090609@sun.com> References: <4ADC4C5D.8090609@sun.com> Message-ID: <4ADC8582.2000007@Sun.COM> These changes look fine. Once minor comment about the tense of updated description of 'other' > + * the path to be relativized against this path To be consistent with the other param descriptions, like for the resolve method, maybe you should change this to: 'the path to relativize against this path' This is a very minor nit, I'm happy with the changes either way. -Chris. On 19/10/2009 12:24, Alan Bateman wrote: > > I need a reviewer for a small typo in Path#relativize's javadoc that Jon > noticed. The description of the "other" param isn't correct. While > fixing that, I noticed that Path#isSameFile has several references to > FilePath that should be Path so I'll fix that while I'm there. The diffs > are attached. > > Thanks, > > Alan. > > diff --git a/src/share/classes/java/nio/file/Path.java > b/src/share/classes/java/nio/file/Path.java > --- a/src/share/classes/java/nio/file/Path.java > +++ b/src/share/classes/java/nio/file/Path.java > @@ -412,7 +412,7 @@ public abstract class Path > * dependent if {@code "a/b/../x"} would locate the same file as > {@code "/a/x"}. > * > * @param other > - * the resulting path > + * the path to be relativized against this path > * > * @return the resulting relative path, or {@code null} if both > paths are > * equal > @@ -1615,23 +1615,23 @@ public abstract class Path > * Tests if the file referenced by this object is the same file > referenced > * by another object. > * > - *

    If this {@code FileRef} and the given {@code FileRef} are > {@link > + *

    If this {@code Path} and the given {@code Path} are {@link > * #equals(Object) equal} then this method returns {@code true} > without checking > - * if the file exists. If the {@code FileRef} and the given {@code > FileRef} > - * are associated with different providers, or the given {@code > FileRef} is > + * if the file exists. If the {@code Path} and the given {@code Path} > + * are associated with different providers, or the given {@code > Path} is > * {@code null} then this method returns {@code false}. Otherwise, > this method > - * checks if both {@code FileRefs} locate the same file, and > depending on the > + * checks if both {@code Paths} locate the same file, and depending > on the > * implementation, may require to open or access both files. > * > *

    If the file system and files remain static, then this method > implements > - * an equivalence relation for non-null {@code FileRefs}. > + * an equivalence relation for non-null {@code Paths}. > *

      > - *
    • It is reflexive: for a non-null {@code FileRef} > {@code f}, > + *
    • It is reflexive: for a non-null {@code Path} {@code f}, > * {@code f.isSameFile(f)} should return {@code true}. > - *
    • It is symmetric: for two non-null {@code FileRefs} > + *
    • It is symmetric: for two non-null {@code Path} > * {@code f} and {@code g}, {@code f.isSameFile(g)} will equal > * {@code g.isSameFile(f)}. > - *
    • It is transitive: for three {@code FileRefs} > + *
    • It is transitive: for three {@code Paths} > * {@code f}, {@code g}, and {@code h}, if {@code > f.isSameFile(g)} returns > * {@code true} and {@code g.isSameFile(h)} returns {@code > true}, then > * {@code f.isSameFile(h)} will return return {@code true}. > From uckelman at nomic.net Mon Oct 19 11:06:00 2009 From: uckelman at nomic.net (Joel Uckelman) Date: Mon, 19 Oct 2009 20:06:00 +0200 Subject: zipfs problems In-Reply-To: <4ADB6C00.7050806@sun.com> References: <20091018173520.F0AE91008B@charybdis.ellipsis.cx> <4ADB6C00.7050806@sun.com> Message-ID: <20091019180600.DFC7410076@charybdis.ellipsis.cx> Thus spake Alan Bateman: > > You are right and there are concurrency issues here. Also all returned > closeables should be wrapped so that close method removes them from the > set. Rajendra, do you have cycles to fix this? > BTW, what's the preferred way for me to submit patches? (Is it the one here? http://openjdk.java.net/contribute/) -- J. From Alan.Bateman at Sun.COM Mon Oct 19 12:20:06 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Mon, 19 Oct 2009 20:20:06 +0100 Subject: Need reviewer for 6884480: (file) Path.relativize has typo in parameter description In-Reply-To: <4ADC8582.2000007@Sun.COM> References: <4ADC4C5D.8090609@sun.com> <4ADC8582.2000007@Sun.COM> Message-ID: <4ADCBBE6.6050404@sun.com> Christopher Hegarty - Sun Microsystems Ireland wrote: > These changes look fine. > > Once minor comment about the tense of updated description of 'other' > > > + * the path to be relativized against this path > > To be consistent with the other param descriptions, like for the > resolve method, maybe you should change this to: > > 'the path to relativize against this path' > > This is a very minor nit, I'm happy with the changes either way. > > -Chris. You are right - it's a bit inconsistent. I'll go with your suggested wording - thanks! -Alan. From Alan.Bateman at Sun.COM Mon Oct 19 13:00:31 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Mon, 19 Oct 2009 21:00:31 +0100 Subject: zipfs problems In-Reply-To: <20091019122433.7B27D10074@charybdis.ellipsis.cx> References: <20091018173520.F0AE91008B@charybdis.ellipsis.cx> <4ADB6C00.7050806@sun.com> <20091018195216.5D6071008B@charybdis.ellipsis.cx> <4ADC4C1A.1060009@sun.com> <20091019122433.7B27D10074@charybdis.ellipsis.cx> Message-ID: <4ADCC55F.3010705@sun.com> Joel Uckelman wrote: > : > I don't follow. What is the problem with asynchronously closing streams > that you're referring to? Is it just that not all streams are thread safe? > > I'm a bit hazy on whether locking belongs in an FS implementation at all > (given the current design). E.g, the default FS doesn't provide any way of > locking itself, though it happens that you can lock files by getting > FileLocks from the FileChannels returned by the default FS. For zipfs, the > Channels aren't FileChannels, so you can't lock ZIP entries. > > On the other hand, if zipfs doesn't provide any kind of locking mechanism, > then it's going to be hard to use it in a multithreaded app. Maybe what > I'm moving toward is that Path should have methods for acquiring read and > write locks. > On one level you'll need to coordinate the re-writing of the zip file with other VMs or entities that are accessing that zip file. For example, our zip implementation mmaps the central directory on some platforms so if one VM re-writes the zip file then it's very possible that it will cause access in other VMs to SIGBUS. File locking (via FileChannel.lock) might help but I suspect you'll need more - perhaps even a shared server to coordinates access. Within a VM, the provider could coordinate access where you have several FileSystem instances representing different file systems on the same zip file. For zip entries, the simplest starting point is that write access requires exclusive access. Later you could define provider specific OpenOptions to support other locking modes. As regards asynchronous close - always tricky. My point was that asynchronous close of streams it not well defined. Without checking, I can't say how robust the zip streams are when closed asynchronously but you'll likely have bigger fish to fly before you come to this. -Alan. From uckelman at nomic.net Mon Oct 19 13:25:08 2009 From: uckelman at nomic.net (Joel Uckelman) Date: Mon, 19 Oct 2009 22:25:08 +0200 Subject: zipfs problems In-Reply-To: <4ADCC55F.3010705@sun.com> References: <20091018173520.F0AE91008B@charybdis.ellipsis.cx> <4ADB6C00.7050806@sun.com> <20091018195216.5D6071008B@charybdis.ellipsis.cx> <4ADC4C1A.1060009@sun.com> <20091019122433.7B27D10074@charybdis.ellipsis.cx> <4ADCC55F.3010705@sun.com> Message-ID: <20091019202508.DB85810076@charybdis.ellipsis.cx> Thus spake Alan Bateman: > > On one level you'll need to coordinate the re-writing of the zip file > with other VMs or entities that are accessing that zip file. For > example, our zip implementation mmaps the central directory on some > platforms so if one VM re-writes the zip file then it's very possible > that it will cause access in other VMs to SIGBUS. File locking (via > FileChannel.lock) might help but I suspect you'll need more - perhaps > even a shared server to coordinates access. I'm not so interested in locking ZIP entries outside of a single VM. If other processes are trying to read or write to a ZIP archive while we're using it and this is expected behavior rather than a screw-up, then we're way beyond the usual use case anyway. > Within a VM, the provider could coordinate access where you have several > FileSystem instances representing different file systems on the same zip > file. For zip entries, the simplest starting point is that write access > requires exclusive access. Later you could define provider specific > OpenOptions to support other locking modes. Ah! That's what I was looking for. This should be done with OpenOptions. Thanks. -- J. From Alan.Bateman at Sun.COM Tue Oct 20 01:54:41 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Tue, 20 Oct 2009 09:54:41 +0100 Subject: zipfs problems In-Reply-To: <20091019180600.DFC7410076@charybdis.ellipsis.cx> References: <20091018173520.F0AE91008B@charybdis.ellipsis.cx> <4ADB6C00.7050806@sun.com> <20091019180600.DFC7410076@charybdis.ellipsis.cx> Message-ID: <4ADD7AD1.9070904@sun.com> Joel Uckelman wrote: > Thus spake Alan Bateman: > >> >> You are right and there are concurrency issues here. Also all returned >> closeables should be wrapped so that close method removes them from the >> set. Rajendra, do you have cycles to fix this? >> >> > > BTW, what's the preferred way for me to submit patches? (Is it the one > here? http://openjdk.java.net/contribute/) > For now, patches are fine. If/when the demo goes into the jdk7 repository then we'll need to create a bug for each change, do proper code reviews, etc. -Alan. From Alan.Bateman at Sun.COM Tue Oct 20 01:56:46 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Tue, 20 Oct 2009 09:56:46 +0100 Subject: zipfs problems In-Reply-To: <20091019202508.DB85810076@charybdis.ellipsis.cx> References: <20091018173520.F0AE91008B@charybdis.ellipsis.cx> <4ADB6C00.7050806@sun.com> <20091018195216.5D6071008B@charybdis.ellipsis.cx> <4ADC4C1A.1060009@sun.com> <20091019122433.7B27D10074@charybdis.ellipsis.cx> <4ADCC55F.3010705@sun.com> <20091019202508.DB85810076@charybdis.ellipsis.cx> Message-ID: <4ADD7B4E.2000803@sun.com> Joel Uckelman wrote: > : > > Ah! That's what I was looking for. This should be done with OpenOptions. > Yes, and if you look our default provider on Windows you'll see an example of this extensibility where it supports open options that map to the legacy DOS sharing modes. -Alan. From uckelman at nomic.net Tue Oct 20 13:07:40 2009 From: uckelman at nomic.net (Joel Uckelman) Date: Tue, 20 Oct 2009 22:07:40 +0200 Subject: zipfs problems In-Reply-To: <20091018195216.5D6071008B@charybdis.ellipsis.cx> References: <20091018173520.F0AE91008B@charybdis.ellipsis.cx> <4ADB6C00.7050806@sun.com> <20091018195216.5D6071008B@charybdis.ellipsis.cx> Message-ID: <20091020200740.5BA2A1008B@charybdis.ellipsis.cx> Thus spake Joel Uckelman: > Thus spake Alan Bateman: > > > > You are right and there are concurrency issues here. Also all returned > > closeables should be wrapped so that close method removes them from the > > set. Rajendra, do you have cycles to fix this? > > Actually, I've already done it for closeableObjects. The only issue I'm > wondering aobut is whether to use Collections.synchronizedSet() or to > build a Set wrapper for a ConcurrentHashMap. > Here's a patch for this issue. -------------- next part -------------- # HG changeset patch # User uckelman at adsl-208-39.dsl.uva.nl # Date 1256066426 -7200 # Node ID 707295cce2384eb8083246a7910b862f997bfa8d # Parent 9b426af16c0dae4b85a53e9766c6fe92345a31a5 * Renamed closeableObjects to closeables (Objects is redundant). * closeables is now a synchronized Set to prevent concurrent modification * InputStreams and SeekableByteChannels now register themselves with ZipFileSystem on creation and unregister themselves on closure. diff --git a/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFilePath.java b/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFilePath.java --- a/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFilePath.java +++ b/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFilePath.java @@ -31,6 +31,7 @@ package com.sun.nio.zipfs; import java.io.File; +import java.io.FilterInputStream; import java.io.IOException; import java.io.InputStream; import java.io.OutputStream; @@ -727,6 +728,73 @@ return pathToZip; } + /** + * An {@link InputStream} which registers itself with the + * {@link ZipFileSystem} on creation and unregisters itself on + * closure. + */ + private class ZipFilePathInputStream extends FilterInputStream { + public ZipFilePathInputStream(InputStream in) { + super(in); + fileSystem.registerCloseable(this); + } + + @Override + public void close() throws IOException { + in.close(); + fileSystem.unregisterCloseable(this); + } + } + + /** + * A {@link SeekableByteChannel} which registers itself with the + * {@link ZipFileSystem} on creation and unregisters itself on + * closure. + */ + private class ZipFilePathChannel implements SeekableByteChannel { + private final SeekableByteChannel ch; + + public ZipFilePathChannel(SeekableByteChannel ch) { + this.ch = ch; + fileSystem.registerCloseable(this); + } + + public long position() throws IOException { + return ch.position(); + } + + public SeekableByteChannel position(long newPos) throws IOException { + ch.position(newPos); + return this; + } + + public int read(ByteBuffer dst) throws IOException { + return ch.read(dst); + } + + public long size() throws IOException { + return ch.size(); + } + + public SeekableByteChannel truncate(long size) throws IOException { + ch.truncate(size); + return this; + } + + public int write(ByteBuffer src) throws IOException { + return ch.write(src); + } + + public boolean isOpen() { + return ch.isOpen(); + } + + public void close() throws IOException { + ch.close(); + fileSystem.unregisterCloseable(this); + } + } + @Override public InputStream newInputStream(OpenOption... options) throws IOException { @@ -751,9 +819,8 @@ zfile.close(); throw new IOException("entry not found" + entryStr); } - InputStream is = zfile.getInputStream(entry); - fileSystem.addCloseableObjects(is); - return is; + + return new ZipFilePathInputStream(zfile.getInputStream(entry)); } } finally { end(); @@ -1009,9 +1076,10 @@ InputStream in = zfile.getInputStream(entry); Path pathtoZip = Paths.get(ZipUtils.readFileInZip(in)); zfile.close(); - SeekableByteChannel sbc = FileChannel.open(pathtoZip, options); - fileSystem.addCloseableObjects(sbc); - return sbc; + + return new ZipFilePathChannel( + FileChannel.open(pathtoZip, options) + ); } } finally { end(); diff --git a/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFileSystem.java b/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFileSystem.java --- a/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFileSystem.java +++ b/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFileSystem.java @@ -51,7 +51,9 @@ private final String defaultdir; private final ReadWriteLock closeLock = new ReentrantReadWriteLock(); private boolean open = true; - private Set closeableObjects = new HashSet(); + + private final Set closeables = + Collections.synchronizedSet(new HashSet()); ZipFileSystem(ZipFileSystemProvider provider, FileRef fref) { @@ -117,7 +119,7 @@ private void implClose(URI root) throws IOException { ZipUtils.remove(root); // remove cached filesystem provider.removeFileSystem(root); - Iterator itr = closeableObjects.iterator(); + Iterator itr = closeables.iterator(); while (itr.hasNext()) { try { itr.next().close(); @@ -128,8 +130,12 @@ } } - boolean addCloseableObjects(Closeable obj) { - return closeableObjects.add(obj); + boolean registerCloseable(Closeable c) { + return closeables.add(c); + } + + boolean unregisterCloseable(Closeable c) { + return closeables.remove(c); } @Override # HG changeset patch # User uckelman at adsl-208-39.dsl.uva.nl # Date 1256067699 -7200 # Node ID 716fe8ce64d09f69265c46466f5d53eca296a35a # Parent 707295cce2384eb8083246a7910b862f997bfa8d Added missing import. diff --git a/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFilePath.java b/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFilePath.java --- a/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFilePath.java +++ b/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFilePath.java @@ -35,6 +35,7 @@ import java.io.IOException; import java.io.InputStream; import java.io.OutputStream; +import java.nio.ByteBuffer; import java.nio.channels.SeekableByteChannel; import java.nio.file.*; import java.nio.file.DirectoryStream.Filter; From uckelman at nomic.net Tue Oct 20 13:22:45 2009 From: uckelman at nomic.net (Joel Uckelman) Date: Tue, 20 Oct 2009 22:22:45 +0200 Subject: zipfs problems In-Reply-To: <20091018173520.F0AE91008B@charybdis.ellipsis.cx> References: <20091018173520.F0AE91008B@charybdis.ellipsis.cx> Message-ID: <20091020202245.9BAF71008B@charybdis.ellipsis.cx> Thus spake Joel Uckelman: > Hi, > > I've been looking at the zipfs demo and have noticed some problems: > > * ZipFileSystemProvider.fileSystems has concurrency problems similar to > ZipFileSystem.closeableObjects. > Attached are patches to fix this problem, as well as a patch to add a missing type parameter. -------------- next part -------------- # HG changeset patch # User uckelman at adsl-208-39.dsl.uva.nl # Date 1256069972 -7200 # Node ID da6c4e140924f13a4e63f11b859ca8acf972e2cb # Parent 6062b4852112a09a4cf59720a59cc61d40380da7 ZipFileSystemProvider.fileSystems synchronized to prevent concurrent modification. diff --git a/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFileSystemProvider.java b/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFileSystemProvider.java --- a/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFileSystemProvider.java +++ b/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFileSystemProvider.java @@ -38,6 +38,7 @@ import java.io.IOException; import java.net.URISyntaxException; import java.nio.channels.FileChannel; +import java.util.Collections; import java.util.HashMap; import java.util.Map; import java.util.Set; @@ -46,7 +47,8 @@ extends FileSystemProvider { private String scheme = "zip"; - private Map fileSystems = new HashMap(); + private Map fileSystems = + Collections.synchronizedMap(new HashMap()); public ZipFileSystemProvider() { } -------------- next part -------------- # HG changeset patch # User uckelman at adsl-208-39.dsl.uva.nl # Date 1256069552 -7200 # Node ID 6062b4852112a09a4cf59720a59cc61d40380da7 # Parent 716fe8ce64d09f69265c46466f5d53eca296a35a Specified type parameter for HashSet. diff --git a/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFileSystem.java b/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFileSystem.java --- a/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFileSystem.java +++ b/src/share/demo/nio/ZipFileSystem/com/sun/nio/zipfs/ZipFileSystem.java @@ -258,7 +258,8 @@ } } private static final Set supportedFileAttributeViews = - Collections.unmodifiableSet(new HashSet(Arrays.asList("basic", "zip", "jar"))); + Collections.unmodifiableSet( + new HashSet(Arrays.asList("basic", "zip", "jar"))); @Override public Set supportedFileAttributeViews() { From Alan.Bateman at Sun.COM Wed Oct 21 00:35:54 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Wed, 21 Oct 2009 08:35:54 +0100 Subject: zipfs problems In-Reply-To: <20091020200740.5BA2A1008B@charybdis.ellipsis.cx> References: <20091018173520.F0AE91008B@charybdis.ellipsis.cx> <4ADB6C00.7050806@sun.com> <20091018195216.5D6071008B@charybdis.ellipsis.cx> <20091020200740.5BA2A1008B@charybdis.ellipsis.cx> Message-ID: <4ADEB9DA.3050205@sun.com> Joel Uckelman wrote: > Thus spake Joel Uckelman: > >> Thus spake Alan Bateman: >> >>> >>> You are right and there are concurrency issues here. Also all returned >>> closeables should be wrapped so that close method removes them from the >>> set. Rajendra, do you have cycles to fix this? >>> >> Actually, I've already done it for closeableObjects. The only issue I'm >> wondering aobut is whether to use Collections.synchronizedSet() or to >> build a Set wrapper for a ConcurrentHashMap. >> >> > > Here's a patch for this issue. > > Thanks for the patches. Once the list of SCA signatories is refreshed and I can find you, then I'll push these patches to nio/nio/jdk, listing you are contributor. -Alan. From uckelman at nomic.net Thu Oct 22 03:22:30 2009 From: uckelman at nomic.net (Joel Uckelman) Date: Thu, 22 Oct 2009 12:22:30 +0200 Subject: Path.createFile() and createDirectory() and nonexistent parents Message-ID: <20091022102230.A2A0310070@charybdis.ellipsis.cx> I can't tell from the Javadoc for Path.createFile() and createDirectory() what the expected behavior is if some element in the path (other than the name) doesn't exist. For example, if my Path is /foo/bar, but /foo doesn't exist, will calling Path.createFile() also create the directory /foo or will it throw an IOException? The docs appear to leave this open; it would be nice if they made a clear statement one way or the other, since the current state makes it hard to know how to handle failure. -- J. From Alan.Bateman at Sun.COM Thu Oct 22 04:29:40 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Thu, 22 Oct 2009 12:29:40 +0100 Subject: Path.createFile() and createDirectory() and nonexistent parents In-Reply-To: <20091022102230.A2A0310070@charybdis.ellipsis.cx> References: <20091022102230.A2A0310070@charybdis.ellipsis.cx> Message-ID: <4AE04224.60402@sun.com> Joel Uckelman wrote: > I can't tell from the Javadoc for Path.createFile() and createDirectory() > what the expected behavior is if some element in the path (other than > the name) doesn't exist. For example, if my Path is /foo/bar, but /foo > doesn't exist, will calling Path.createFile() also create the directory > /foo or will it throw an IOException? The docs appear to leave this open; > it would be nice if they made a clear statement one way or the other, > since the current state makes it hard to know how to handle failure. > No, these methods do not create any missing parent directories (has cannot do so as they are created to be atomic). However, point taken this isn't as clear as it should - there is a specification bug tracking this: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6876000 -Alan. From Alan.Bateman at Sun.COM Thu Oct 22 04:34:47 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Thu, 22 Oct 2009 12:34:47 +0100 Subject: Path.createFile() and createDirectory() and nonexistent parents In-Reply-To: <4AE04224.60402@sun.com> References: <20091022102230.A2A0310070@charybdis.ellipsis.cx> <4AE04224.60402@sun.com> Message-ID: <4AE04357.2020306@sun.com> Alan Bateman wrote: > (has cannot do so as they are created to be atomic). This seems to be garbled - that should say that they are specified to be atomic. -Alan. From uckelman at nomic.net Thu Oct 22 05:22:32 2009 From: uckelman at nomic.net (Joel Uckelman) Date: Thu, 22 Oct 2009 14:22:32 +0200 Subject: Path.createFile() and createDirectory() and nonexistent parents In-Reply-To: <4AE04224.60402@sun.com> References: <20091022102230.A2A0310070@charybdis.ellipsis.cx> <4AE04224.60402@sun.com> Message-ID: <20091022122233.47ADB10071@charybdis.ellipsis.cx> Thus spake Alan Bateman: > Joel Uckelman wrote: > > I can't tell from the Javadoc for Path.createFile() and createDirectory() > > what the expected behavior is if some element in the path (other than > > the name) doesn't exist. For example, if my Path is /foo/bar, but /foo > > doesn't exist, will calling Path.createFile() also create the directory > > /foo or will it throw an IOException? The docs appear to leave this open; > > it would be nice if they made a clear statement one way or the other, > > since the current state makes it hard to know how to handle failure. > > > No, these methods do not create any missing parent directories (has > cannot do so as they are created to be atomic). However, point taken > this isn't as clear as it should - there is a specification bug tracking > this: > http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6876000 > That was my suscipcion (given also the existence of Files.createDirectories(), which would have no point otherwsie). Thanks for clearing that up and filing the bug report. -- J. From Alan.Bateman at Sun.COM Sun Oct 25 14:26:23 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Sun, 25 Oct 2009 21:26:23 +0000 Subject: NIO feedback In-Reply-To: <4ADEF64C.6070007@xs4all.nl> References: <4AD5FACE.9070908@xs4all.nl> <4AD608A1.2070905@sun.com> <4AD70499.1040400@xs4all.nl> <4ADAFF39.1090905@sun.com> <4ADC3378.2070101@xs4all.nl> <4ADC6B44.4020609@sun.com> <4ADCA823.10909@sun.com> <4ADEF64C.6070007@xs4all.nl> Message-ID: <4AE4C27F.5090907@sun.com> [ John and I exchanged a few mails offline on this and John has being trying out a quick prototype [1] that I hacked up to check the usefulness of this. ] John Hendrikx wrote: > : > > I've been running it against jdk7 b74, freshly downloaded, and it > seems to work as expected. My observations: > > - I get progress feedback roughly twice a second The interval is currently 200ms so it could be up to 5 times a second. Clearly this is something that could be configurable but it is a fixed interval for now to keep it simple. > > - Sometimes the feedback has the same values for several calls in a > row (due to some OS filesystem contention I'm pretty sure). This is > good as it allows me to abort at any time. Right, this is likely wait for I/O or the network (or in theory a sampling glitch as the transfer progress is sampled without synchronization - if an out-of-range value is sampled then it is dropped and the previous "good" value is reported to the progress method). > > - After aborting a Copy by returning false, I still get several > feedback calls (10+ on large files) with constant 'bytetransferred' > values. I assume this is due to the deletion occuring. I keep > returning "false" for these, don't know what would happen if I > "change" my mind so to speak :) The cancellation is done cooperatively (for feasibility reasons) and might not be instantaneous. Also, as you noted, it does attempt to recover and remove the target if partially created. The copyTo method does not complete until this is done and progress events will continue to be reported. Another approach would have been to discontinue events after "false" has been returned. > > - File was properly removed after an aborted Copy. Yes but cannot be guaranteed - say you are copying to a remote file system and the network connection is lost. > > - After integrating it in my project, I tested copying a few 1000's of > files and it behaved as expected (the code actually does a verify > after copying and no anomalies were detected). Good. > > The only thing I couldn't do that my own version could was throttling > and pausing of the copy process. Those are very specific needs though > and I doubt they'd be of much value as standard features -- currently > I'd lean to just removing those features in favor of using a standard > copy function (which handles the nasty details of copying attributes > as well). I did however test what happened if you slowed down the > ProgressMonitor callback considerably, but the copy continued > unimpeded (as I would have expected). Throttling and pausing are a bit beyond what the default provider should be expected to support. A "desktop" provider or a provider that supports access to files over http or very remote file systems. As regards the observation that the copy is not impacted by the progress method - you are right, as they copy runs in the background when this option is used. -Alan. [1] http://cr.openjdk.java.net/~alanb/6892923/ExtendedCopyOption.ProgressMonitor.html From hjohn at xs4all.nl Mon Oct 26 05:07:00 2009 From: hjohn at xs4all.nl (John Hendrikx) Date: Mon, 26 Oct 2009 13:07:00 +0100 Subject: NIO feedback In-Reply-To: <4AE4C27F.5090907@sun.com> References: <4AD5FACE.9070908@xs4all.nl> <4AD608A1.2070905@sun.com> <4AD70499.1040400@xs4all.nl> <4ADAFF39.1090905@sun.com> <4ADC3378.2070101@xs4all.nl> <4ADC6B44.4020609@sun.com> <4ADCA823.10909@sun.com> <4ADEF64C.6070007@xs4all.nl> <4AE4C27F.5090907@sun.com> Message-ID: <4AE590E4.7060901@xs4all.nl> Alan Bateman wrote: >> - Sometimes the feedback has the same values for several calls in a >> row (due to some OS filesystem contention I'm pretty sure). This is >> good as it allows me to abort at any time. > Right, this is likely wait for I/O or the network (or in theory a > sampling glitch as the transfer progress is sampled without > synchronization - if an out-of-range value is sampled then it is > dropped and the previous "good" value is reported to the progress > method). You mean a long is read without synchronization? :) >> - After aborting a Copy by returning false, I still get several >> feedback calls (10+ on large files) with constant 'bytetransferred' >> values. I assume this is due to the deletion occuring. I keep >> returning "false" for these, don't know what would happen if I >> "change" my mind so to speak :) > The cancellation is done cooperatively (for feasibility reasons) and > might not be instantaneous. Also, as you noted, it does attempt to > recover and remove the target if partially created. The copyTo method > does not complete until this is done and progress events will continue > to be reported. Another approach would have been to discontinue events > after "false" has been returned. Something I forgot to mention, the cancellation results in a FileSystemException being thrown. I think a more specific exception or just a true/false return might be helpful. Currently the caller code just catches this exception, and then checks if it was an abort that caused it (in which case it is ignored) and otherwise rethrows it. I can't however really be 100% sure if it was an abort (ie, whether the abort triggered from the callback was actually processed) or if some other kind of abnormal termination occured. I gave the discontinuing of the events after "false" has been returned some more thought, and I think it is probably better to keep sending events so even during the clean-up phase it is possible to give feedback. My own specific implementation doesn't require it, but other implementations may find it useful. Another thing I found while I was testing this code is that it doesn't eliminate the need for a function to copy attributes. Even though copyTo can do this for files, I still need a custom implementation for softlinks and directories (unless of course, copyTo also works for these kinds of objects... I'll admit that I haven't even considered testing this yet, but the javadoc seems to indicate it isn't allowed). A suggestion could be to make copyTo work with directories/softlinks as well (not recursively), or expose a function to copy attributes. I will find time to test this under Linux soon as well. Thanks, --John From Alan.Bateman at Sun.COM Mon Oct 26 06:00:25 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Mon, 26 Oct 2009 13:00:25 +0000 Subject: NIO feedback In-Reply-To: <4AE590E4.7060901@xs4all.nl> References: <4AD5FACE.9070908@xs4all.nl> <4AD608A1.2070905@sun.com> <4AD70499.1040400@xs4all.nl> <4ADAFF39.1090905@sun.com> <4ADC3378.2070101@xs4all.nl> <4ADC6B44.4020609@sun.com> <4ADCA823.10909@sun.com> <4ADEF64C.6070007@xs4all.nl> <4AE4C27F.5090907@sun.com> <4AE590E4.7060901@xs4all.nl> Message-ID: <4AE59D69.2040600@sun.com> John Hendrikx wrote: > Alan Bateman wrote: >>> - Sometimes the feedback has the same values for several calls in a >>> row (due to some OS filesystem contention I'm pretty sure). This is >>> good as it allows me to abort at any time. >> Right, this is likely wait for I/O or the network (or in theory a >> sampling glitch as the transfer progress is sampled without >> synchronization - if an out-of-range value is sampled then it is >> dropped and the previous "good" value is reported to the progress >> method). > You mean a long is read without synchronization? :) Yep - it's just a prototype and there isn't any synchronization between the native code and the thread that is sampling and reporting events (the sampler simply checks that the value hasn't decreased and is in range so in theory it might jump forward more that it should - probably not a big deal for this usage). : > Something I forgot to mention, the cancellation results in a > FileSystemException being thrown. I think a more specific exception > or just a true/false return might be helpful. Currently the caller > code just catches this exception, and then checks if it was an abort > that caused it (in which case it is ignored) and otherwise rethrows it. > I can't however really be 100% sure if it was an abort (ie, whether > the abort triggered from the callback was actually processed) or if > some other kind of abnormal termination occured. An exception during recovery doesn't suppress the original abort/cancel exception but a more specific exception would make this clearer. > > I gave the discontinuing of the events after "false" has been returned > some more thought, and I think it is probably better to keep sending > events so even during the clean-up phase it is possible to give > feedback. My own specific implementation doesn't require it, but > other implementations may find it useful. > > Another thing I found while I was testing this code is that it doesn't > eliminate the need for a function to copy attributes. Even though > copyTo can do this for files, I still need a custom implementation for > softlinks and directories (unless of course, copyTo also works for > these kinds of objects... I'll admit that I haven't even considered > testing this yet, but the javadoc seems to indicate it isn't > allowed). A suggestion could be to make copyTo work with > directories/softlinks as well (not recursively), or expose a function > to copy attributes. The copyTo method follow sym links by default but it will copy a sym link if you specify the NOFOLLOW_LINKS option. The file attributes of sym links aren't usually interesting but our implementation does whatever is appropriate for the platform. The copyTo method will copy a directory too. That's for playing with this. -Alan. From hjohn at xs4all.nl Fri Oct 30 19:56:12 2009 From: hjohn at xs4all.nl (John Hendrikx) Date: Sat, 31 Oct 2009 03:56:12 +0100 Subject: More NIO feedback Message-ID: <4AEBA74C.8080307@xs4all.nl> As I'm working with NIO2 pretty extensively, here's some more feedback on the framework: 1) Other than parsing exception message text, there's no way to distinguish a "disk full" condition and other FileSystemExceptions. Disk Full conditions are something a user can solve in the background allowing the offending operation to be retried and so I would like to inform the user of this condition in a more specific way. 2) Will there be any new way of obtaining an icon for a Path? It is a kind of Attribute. Currently obtaining these involves using javax.swing.filechooser.FileSystemView. This works well for Windows. For Linux however it is very bland, resulting in the need to customize this. Files.probeContentType's mimetype is already a big help for doing this manually though. 3) I was having some trouble automagically handling retries of operations that throw AccessDeniedException. For example, Path.delete() can throw this, and what I'm trying to achieve is to see if the Attributes can be altered in such a way (with the current privileges) that would result in the delete succeeding (a "forced" delete). This is somewhat harder than I expected, although not impossible. I'm writing a static class to handle these nasty details. 4) Related to 3, I'm wondering if there's any way Java could temporarily get higher privileges (like obtaining root privileges by prompting for a password). Is such a thing possible at all? 5) Is there any way to obtain FileStore specific parameters, like: - case sensitivity - limitations (max name length, max file size) - charset - soft/hard link support available - sparse file creation Thanks --John