From jeffhain at rocketmail.com Thu Nov 1 18:29:53 2012 From: jeffhain at rocketmail.com (Jeff Hain) Date: Fri, 2 Nov 2012 01:29:53 +0000 (GMT) Subject: more about - System.arraycopy(...) equivalents for ByteBuffer/FileChannel Message-ID: <1351819793.63782.YahooMailNeo@web132102.mail.ird.yahoo.com> Hello. ? Resurrecting this subject, for I'm now quite done with my implementation (*) and have more (and less approximative) remarks. (*) code.google.com/p/jodk/source/browse/src/net/jodk/io/ByteCopyUtils.java (contains some no-longer used code (mapped ByteBuffers), but I let it around for experimentation purpose) 1) I said in previous mail: >With the current API, you can do whatever you want, >with only native copies (and just the required amount), >both with ByteBuffers and FileChannels contents. ? That's modulo the fact that MappedByteBuffers can't be explicitly and portably unmapped (Bug ID: 4724038), so you can't always use them. ? Maybe having a FileChannel.unmap(MappedByteBuffer) method wouldn't hurt more than its absence does, due to either the performance hit of not using MBBs, or the non-portability and hackiness of the workarounds people have to elaborate. ===> ? As a result, even when using a MBB is appropriate, i.e. when src is a FileChannel, and dst a heap ByteBuffer or another FileChannel, and the number of bytes to copy above some threshold, portable code must still have an alternative, typically involving temporary direct ByteBuffers, first doing FileChannel.read(tmpBB), and then native copy from tmpBB to dstBB, or FileChannel.write(tmpBB). ? That's also modulo the fact that, for BB to FC copies, if the FileChannel is not readable, you can't use MBB (no WRITE_ONLY mapping mode). A FileChannel.isReadable() method would allow to use MBB for readable (and writable) channels, without risk of an exception being thrown. ===> ? As a result, when src is not direct, an intermediary temporary direct ByteBuffer is required. ? That's also modulo the fact that when you copy between a MappedByteBuffer and a FileChannel, src and dst might share memory, but there is no API to figure out if and how, thus you don't know if you have to copy forward or backward (to avoid erasing bytes to copy with copied bytes). ===> ? As a result, for BB to FC and FC to BB copies, in case of overlapping, the result is undefined. ? That's also modulo the fact that, depending on OS or architecture, FileChannel.write methods don't use overlapping-proof treatments, but either forward or backward loops on bytes, erasing bytes to copy with copied bytes. ? On a T2300/WinXP, it only occurred for copies below a certain threshold (around 256+ bytes), and was always a forward copy. ? On a 980X/Win7, it was more messy, I couldn't figure out a simple logic. ===> ? As a result, for FC to FC copies, if wanting to support overlapping cases, one can't use MBBs, and an intermediary temporary direct ByteBuffer is required. 2) Performance remarks. ? I found FileChannel.write(ByteBuffer,long) to be very slow on WinXP/7, by a factor 15, when the ByteBuffer is "large" (like 500Ko), even if it is direct (mapped or not), in which case it resolves to sun.nio.ch.IOUtils.writeFromNativeBuffer (which I supposed to always be fast)). ? On Linux there was no slow down. ===> ? This problem also hurts FileChannel.transferTo/transferFrom, which I stopped using. ? Writing the direct BB by chunks of 32Kio makes up for that. ? If the ByteBuffer is not direct, FileChannel.write methods causes to use an AS BIG temporary direct ByteBuffer (see sun.nio.ch.IOUtils), and might also be slow due to the direct ByteBuffer being large (previous problem). ===> ? A work-around is to copy by chunks, using your own temporary direct ByteBuffer to avoid creation of multiple ones (especially if you have some local temporary instances available already). ? Intensive benches involving MBBs (namely FC to heap BB copies) were hanging from time to time, up to nearly a second, and then resumed at usual speed (slightly faster than temporary direct ByteBuffer approaches). ===> ? As a result, I completely disabled MBBs usage for my copies. 3) Non-performance remarks: ? When copying between direct ByteBuffers, the efficient way is to use ByteBuffer.put(ByteBuffer), but its spec says that it's about equivalent to "while (src.hasRemaining()) dst.put(src.get());", i.e. that it is not suited if memory is shared (which no API allows to figure out) and srcPos < dstPos (as raw memmory positions), since then it could erase bytes to copy with copied bytes. ? Fortunately, in that case the implementation doesn't follow the spec, and instead uses Unsafe.copyMemory(long,long,long), which seems to handle overlapping. ? It is also the case for the heap-to-heap case, where the implementation uses System.arraycopy(...) (but I don't use this put method for heap-to-heap cases). ===> ? I hope that Unsafe.copyMemory(long,long,long) effectively handles overlapping, as my tests show, and that the put method spec will be relaxed (its excessive precision just looks like an unfortunate way to explain what it does), and possibly aligned with its implementation for direct-to-direct and heap-to-heap cases, as I'm relying on it to handle overlapping for BB to BB copies. ? FileChannelImpl.truncate(long): ? The bug we already talked about (the early return if size > size(), which masks the writability check, and the fact that size >= size() could be done instead) prevents reworking position whether truncation occurs or not, as the spec says (even if size > size(), position should be set to size if it is superior, but it isn't - or is it a spec bug?). ? FileChannelImpl.map(...): ? If mode is null, and assertions enabled, "assert (imode >= 0)" fails (I have somewhere in my head the idea that in JDK assertions should only check private code - well in a sense here it does :). ? If mode is null, and channel non-writable, and assertions disabled, NonWritableChannelException is thrown, even though the spec says it can only be thrown if mode is READ_WRITE or PRIVATE. ? The spec says that NonReadableChannelException is thrown if the channel is not readable AND mode is READ_ONLY. The part about the mode could be removed, to align the spec on the implementation, for which a non-readable channel is enough for this exception to be thrown. ? FileChannelImpl.transferFrom(...): ? This method can grow destination channel, but if the specified position (in dst) is > dst.size(), it just returns 0. It looks like a bug, as the spec says nothing about this surprising behavior. ? FileChannel.write(ByteBuffer,long): ? The Javadoc says that position is not updated, but if the channel is in append mode it might be (since then we have position = size, and this method can grow the file). -Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20121102/d8118528/attachment.html From Alan.Bateman at oracle.com Fri Nov 2 03:24:24 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 02 Nov 2012 10:24:24 +0000 Subject: more about - System.arraycopy(...) equivalents for ByteBuffer/FileChannel In-Reply-To: <1351819793.63782.YahooMailNeo@web132102.mail.ird.yahoo.com> References: <1351819793.63782.YahooMailNeo@web132102.mail.ird.yahoo.com> Message-ID: <50939F58.5060706@oracle.com> On 02/11/2012 01:29, Jeff Hain wrote: > : > > That's modulo the fact that MappedByteBuffers can't be > explicitly and portably unmapped (Bug ID: 4724038), so > you can't always use them. Yes, a very difficult issue that many people have looked at but didn't come up a solution that addresses all the concerns, see: http://bugs.sun.com/view_bug.do?bug_id=4724038 > > > That's also modulo the fact that, for BB to FC copies, > if the FileChannel is not readable, you can't use MBB > (no WRITE_ONLY mapping mode). A FileChannel.isReadable() > method would allow to use MBB for readable (and writable) > channels, without risk of an exception being thrown. It would although if code is running into then it suggests that it might not be using the API as intended. > : > > That's also modulo the fact that when you copy between > a MappedByteBuffer and a FileChannel, src and dst might > share memory, but there is no API to figure out if and how, > thus you don't know if you have to copy forward or backward > (to avoid erasing bytes to copy with copied bytes). Right, there isn't anything in this API to know when buffers or mappings overlap. > : > > I found FileChannel.write(ByteBuffer,long) to be very slow > on WinXP/7, by a factor 15, when the ByteBuffer is "large" > (like 500Ko), even if it is direct (mapped or not), in which > case it resolves to sun.nio.ch.IOUtils.writeFromNativeBuffer > (which I supposed to always be fast)). > On Linux there was no slow down. This is another long standing issue, see: http://bugs.sun.com/view_bug.do?bug_id=6265734 It comes down to Windows not having the equivalent of a pread/pwrite that doesn't change the global position. There are ideas on how to solve this but it requires significant refactoring, hopefully some day. > : > > Intensive benches involving MBBs (namely FC to heap BB > copies) were hanging from time to time, up to nearly a second, > and then resumed at usual speed (slightly faster than temporary > direct ByteBuffer approaches). > ===> > As a result, I completely disabled MBBs usage for my copies. Was this Windows 32-bit? We've seen a lot of issues with memory management on Windows where it takes time to write-out dirty pages and unmap the memory. I don't think we can do anything about this, assuming this is what you are running into. Windows 64-bit appears to be better although it can be a problem too when memory is over committed. > > > > > 3) Non-performance remarks: > > > When copying between direct ByteBuffers, the efficient way > is to use ByteBuffer.put(ByteBuffer), but its spec says that > it's about equivalent to > "while (src.hasRemaining()) dst.put(src.get());", > i.e. that it is not suited if memory is shared (which no API > allows to figure out) and srcPos < dstPos (as raw memmory > positions), since then it could erase bytes to copy with > copied bytes. We could potentially clarify the spec here although warning about overlapping regions may require changes in several other places too. > : > > FileChannelImpl.truncate(long): > The bug we already talked about (the early return if > size > size(), which masks the writability check, and the > fact that size >= size() could be done instead) > prevents reworking position whether truncation occurs or not, > as the spec says (even if size > size(), position should be > set to size if it is superior, but it isn't - or is it a > spec bug?). This is a corner case which is probably why it was never reported before. We have a bug for it since you brought it up: http://bugs.sun.com/view_bug.do?bug_id=8000330 > > > FileChannelImpl.map(...): > If mode is null, and assertions enabled, "assert (imode >= 0)" > fails (I have somewhere in my head the idea that in JDK assertions > should only check private code - well in a sense here it does :). If mode is passed as null then NullPointerException should be thrown. Another 10+ year issue that hasn't been noticed. I'll create a bug for this - thanks! > > > FileChannelImpl.transferFrom(...): > This method can grow destination channel, but if the specified > position (in dst) is > dst.size(), it just returns 0. It looks > like a bug, as the spec says nothing about this surprising behavior. This is specified: " If the given position is greater than the file's current size then no bytes are transferred" > > > FileChannel.write(ByteBuffer,long): > The Javadoc says that position is not updated, but if the channel > is in append mode it might be (since then we have position = size, > and this method can grow the file). There is a long standing bug tracking updating the spec on this issue, see: http://bugs.sun.com/view_bug.do?bug_id=6924219 Thanks for bug reports, you've gone in corner cases that our tests for this area clearly haven't gone into. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20121102/47ccd343/attachment-0001.html From zhong.j.yu at gmail.com Fri Nov 2 19:59:39 2012 From: zhong.j.yu at gmail.com (Zhong Yu) Date: Fri, 2 Nov 2012 21:59:39 -0500 Subject: Unmapping MappedByteBuffer Message-ID: (cc Doug: a non-trivial use case for ReentrantLock.isHeldByCurrentThread()) On Fri, Nov 2, 2012 at 5:24 AM, Alan Bateman wrote: > On 02/11/2012 01:29, Jeff Hain wrote: > > : > > That's modulo the fact that MappedByteBuffers can't be > explicitly and portably unmapped (Bug ID: 4724038), so > you can't always use them. > > Yes, a very difficult issue that many people have looked at but didn't come > up a solution that addresses all the concerns, see: > > http://bugs.sun.com/view_bug.do?bug_id=4724038 (I just saw it questioned on SO: http://stackoverflow.com/questions/13204656 ) I think we can solve the problem by enforcing that any thread accessing the buffer must "own" it first, and there can only be one owner: bb.own(); bb.get(0); .... bb.get(n); bb.disown(); The access methods check first that the current thread is the owner thread. This check can be very cheap, e.g. ReentrantLock.isHeldByCurrentThread(). After explicit disposal, no thread can obtain the ownership anymore, therefore no thread can access the buffer anymore. I tested this strategy (source attached below), for tight loops of get(index), the speed is 90% of bare ByteBuffer.get(index). Ideally, VM optimization should be able to completely remove the check of ownership, making the 2 versions equally fast. Zhong Yu https://gist.github.com/4005639 SingleOwnerByteBuffer.java ========================================= import java.nio.ByteBuffer; import java.util.concurrent.locks.ReentrantLock; public class SingleOwnerByteBuffer { final ReentrantLock lock = new ReentrantLock(); ByteBuffer bb; boolean disposed; public SingleOwnerByteBuffer(ByteBuffer bb) { this.bb = bb; } public int length() { assertOwnedByCurrentThread(); return bb.remaining(); } public byte get(int index) { assertOwnedByCurrentThread(); return bb.get(index); } public void dispose() { lock.lock(); try { if(disposed) return; /* unmap or dealloc bb here */ disposed = true; } finally { lock.unlock(); } } public void own() { lock.lock(); if(disposed) { lock.unlock(); throw new IllegalStateException("disposed"); } } void assertOwnedByCurrentThread() { if(!lock.isHeldByCurrentThread()) throw new IllegalStateException("not owned by current thread"); } public void disown() { lock.unlock(); } public static void main(String[] args) throws Exception { //test1(100_000, 1000_000); test2(100_000, 1000_000); } static public void test1(int cap, int repeat) throws Exception { int sum=0; long time = System.currentTimeMillis(); ByteBuffer bb = ByteBuffer.allocateDirect(cap); for(int r=0; r References: Message-ID: <50949D55.6090207@uni-konstanz.de> On 11/03/2012 03:59 AM, Zhong Yu wrote:> static public void test1(int cap, int repeat) throws Exception > { > int sum=0; > long time = System.currentTimeMillis(); > ByteBuffer bb = ByteBuffer.allocateDirect(cap); > for(int r=0; r { > for(int i=0; i { > sum += bb.get(i); > } > } > time = System.currentTimeMillis() - time; > System.out.printf("time=%,d, sum=%,d %n", time, sum); > } > static public void test2(int cap, int repeat) throws Exception > { > int sum=0; > long time = System.currentTimeMillis(); > ByteBuffer bb0 = ByteBuffer.allocateDirect(cap); > SingleOwnerByteBuffer bb = new SingleOwnerByteBuffer(bb0); > bb.own(); > for(int r=0; r { > for(int i=0; i { > sum += bb.get(i); > } > } > bb.disown(); > time = System.currentTimeMillis() - time; > System.out.printf("time=%,d, sum=%,d %n", time, sum); > } > } Is the use of System.currentTimeMills() instead of System.nanoTime() deliberately used? I'd use nanoTime() as always suggested (because currentTimeMills() depends on wallclock time which is sometimes corrected). Otherwise it might even cause negative values in some cases... kind regards, Johannes From Johannes.Lichtenberger at uni-konstanz.de Fri Nov 2 21:32:35 2012 From: Johannes.Lichtenberger at uni-konstanz.de (Johannes.Lichtenberger) Date: Sat, 03 Nov 2012 05:32:35 +0100 Subject: Unmapping MappedByteBuffer In-Reply-To: <50949D55.6090207@uni-konstanz.de> References: <50949D55.6090207@uni-konstanz.de> Message-ID: <50949E63.7030000@uni-konstanz.de> On 11/03/2012 05:28 AM, Johannes.Lichtenberger wrote: > On 11/03/2012 03:59 AM, Zhong Yu wrote:> static public void > test1(int cap, int repeat) throws Exception > > { > > int sum=0; > > long time = System.currentTimeMillis(); > > ByteBuffer bb = ByteBuffer.allocateDirect(cap); > > for(int r=0; r > { > > for(int i=0; i > { > > sum += bb.get(i); > > } > > } > > time = System.currentTimeMillis() - time; > > System.out.printf("time=%,d, sum=%,d %n", time, sum); > > } > > static public void test2(int cap, int repeat) throws Exception > > { > > int sum=0; > > long time = System.currentTimeMillis(); > > ByteBuffer bb0 = ByteBuffer.allocateDirect(cap); > > SingleOwnerByteBuffer bb = new SingleOwnerByteBuffer(bb0); > > bb.own(); > > for(int r=0; r > { > > for(int i=0; i > { > > sum += bb.get(i); > > } > > } > > bb.disown(); > > time = System.currentTimeMillis() - time; > > System.out.printf("time=%,d, sum=%,d %n", time, sum); > > } > > } > > Is the use of System.currentTimeMills() instead of System.nanoTime() > deliberately used? I'd use nanoTime() as always suggested (because > currentTimeMills() depends on wallclock time which is sometimes > corrected). Otherwise it might even cause negative values in some cases... Ok, in this case probably doesn't make a lot of difference ;-) kind regards, Johannes From mthornton at optrak.com Sat Nov 3 02:21:29 2012 From: mthornton at optrak.com (Mark Thornton) Date: Sat, 03 Nov 2012 09:21:29 +0000 Subject: Unmapping MappedByteBuffer In-Reply-To: References: Message-ID: <5094E219.8000205@optrak.com> On 03/11/12 02:59, Zhong Yu wrote: > (cc Doug: a non-trivial use case for ReentrantLock.isHeldByCurrentThread()) > > On Fri, Nov 2, 2012 at 5:24 AM, Alan Bateman wrote: >> On 02/11/2012 01:29, Jeff Hain wrote: >> >> : >> >> That's modulo the fact that MappedByteBuffers can't be >> explicitly and portably unmapped (Bug ID: 4724038), so >> you can't always use them. >> >> Yes, a very difficult issue that many people have looked at but didn't come >> up a solution that addresses all the concerns, see: >> >> http://bugs.sun.com/view_bug.do?bug_id=4724038 > (I just saw it questioned on SO: http://stackoverflow.com/questions/13204656 ) > > I think we can solve the problem by enforcing that any thread > accessing the buffer must "own" it first, and there can only be one > owner: > > bb.own(); > > bb.get(0); > .... > bb.get(n); > > bb.disown(); > That wouldn't be compatible with existing use of MappedByteBuffer. However you could perhaps apply that strategy to a new class (possibly a subclass of MappedByteBuffer). You could do without the disown() method --- legal usage would still be legal, only faulty use might get an exception in a different place. Mark From Alan.Bateman at oracle.com Sat Nov 3 05:27:10 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 03 Nov 2012 12:27:10 +0000 Subject: Unmapping MappedByteBuffer In-Reply-To: <5094E219.8000205@optrak.com> References: <5094E219.8000205@optrak.com> Message-ID: <50950D9E.4050808@oracle.com> On 03/11/2012 09:21, Mark Thornton wrote: > : > > That wouldn't be compatible with existing use of MappedByteBuffer. > However you could perhaps apply that strategy to a new class (possibly > a subclass of MappedByteBuffer). You could do without the disown() > method --- legal usage would still be legal, only faulty use might get > an exception in a different place. > > Mark > That's right, plus slices, duplicates and views would need to be considered too. One prototype that we had a few years ago was to add a modifier to MapMode so that the map method could request an unmappable buffer. The resulting mapped byte buffer (and resulting slices/duplicates/views) were generated with code to coordinate with unmap. It didn't of course deal with the access from native code case because that doesn't go through the java API. Introducing a notion of locked or owned buffer could require updates to JNI for these cases. -Alan From zhong.j.yu at gmail.com Sat Nov 3 08:28:54 2012 From: zhong.j.yu at gmail.com (Zhong Yu) Date: Sat, 3 Nov 2012 10:28:54 -0500 Subject: Unmapping MappedByteBuffer In-Reply-To: <5094E219.8000205@optrak.com> References: <5094E219.8000205@optrak.com> Message-ID: On Sat, Nov 3, 2012 at 4:21 AM, Mark Thornton wrote: > On 03/11/12 02:59, Zhong Yu wrote: >> >> (cc Doug: a non-trivial use case for >> ReentrantLock.isHeldByCurrentThread()) >> >> On Fri, Nov 2, 2012 at 5:24 AM, Alan Bateman >> wrote: >>> >>> On 02/11/2012 01:29, Jeff Hain wrote: >>> >>> : >>> >>> That's modulo the fact that MappedByteBuffers can't be >>> explicitly and portably unmapped (Bug ID: 4724038), so >>> you can't always use them. >>> >>> Yes, a very difficult issue that many people have looked at but didn't >>> come >>> up a solution that addresses all the concerns, see: >>> >>> http://bugs.sun.com/view_bug.do?bug_id=4724038 >> >> (I just saw it questioned on SO: >> http://stackoverflow.com/questions/13204656 ) >> >> I think we can solve the problem by enforcing that any thread >> accessing the buffer must "own" it first, and there can only be one >> owner: >> >> bb.own(); >> >> bb.get(0); >> .... >> bb.get(n); >> >> bb.disown(); >> > That wouldn't be compatible with existing use of MappedByteBuffer. However > you could perhaps apply that strategy to a new class (possibly a subclass of > MappedByteBuffer). You could do without the disown() method --- legal usage > would still be legal, only faulty use might get an exception in a different > place. > > Mark > (change disown() to Closeable.close() method) We can extend the idea to allow concurrent owner threads; each owner will have to explicitly call close(). Probably something like try( CloseableByteBuffer cbb = mbb.newThreadLocalView() ) { // access cbb } public class CloseableByteBuffer implements Closeable { Thread ownerThread = Thread.currentThread(); void assertOwning() { if(ownerThread==null) throw new IllegalStateException("closed"); if(ownerThread!=Thread.currentThread()) throw new IllegalAccessError("not owner"); } @Override public void close() { assertOwning(); ownerThread = null; /* ref counting business */ } public byte get(int index) { assertOwning(); return bb.get(index); } ... } From jeffhain at rocketmail.com Sat Nov 3 10:06:44 2012 From: jeffhain at rocketmail.com (Jeff Hain) Date: Sat, 3 Nov 2012 17:06:44 +0000 (GMT) Subject: more about - System.arraycopy(...) equivalents for ByteBuffer/FileChannel In-Reply-To: <50939F58.5060706@oracle.com> References: <1351819793.63782.YahooMailNeo@web132102.mail.ird.yahoo.com> <50939F58.5060706@oracle.com> Message-ID: <1351962404.70447.YahooMailNeo@web132101.mail.ird.yahoo.com> >>?? That's also modulo the fact that, for BB to FC copies, >> if the FileChannel is not readable, you can't use MBB >> (no WRITE_ONLY mapping mode). A FileChannel.isReadable() >> method would allow to use MBB for readable (and writable) >> channels, without risk of an exception being thrown. > >It would although if code is running into then it suggests >that it might not be using the API as intended. ? I mean, if dst.isReadable() is true, we try to map in READ_WRITE mode (and it throws if dst is not writable: fine), and if dst.isReadable() is false, we don't try to map (as there is no WRITE_ONLY mode (which would require specific MBBs)), and just use dst.write methods. ? That said, I don't know if writable-but-not-readable channels can even exist (being new to IO stuffs). >>?? Intensive benches involving MBBs (namely FC to heap BB >> copies) were hanging from time to time, up to nearly a second, >> and then resumed at usual speed (slightly faster than temporary >> direct ByteBuffer approaches). >> ===> >>?? As a result, I completely disabled MBBs usage for my copies. > >Was this Windows 32-bit? We've seen a lot of issues with memory >management on Windows where it takes time to write-out dirty pages >and unmap the memory. I don't think we can do anything about this, >assuming this is what you are running into. Windows 64-bit appears >to be better although it can be a problem too when memory is over >committed. Copying from FC to heap BB with MBBs, 64 times in a row (same FC, same BB), from a same file (and -verbosegc to make sure hangs are not GCs), using SSDs: - on T2300/WinXP-32 (1Go RAM and not much left), ? - with a 40Mo file, each copy took about 60ms, ??? with one hang of 200ms (260ms measured minus copy time). - on 980X/Win7-64 (6Go RAM), ? - with a 40Mo file, each copy took about 17ms, with ??? one hang of 60ms. ? - with a 200Mo file, each copy took about 80ms, with ??? 3 hangs of about half a second. ? - with a 400Mo file, each copy took about 170ms, with ??? 8 hangs of 250-600ms, twice next to each other. ? - with a 800Mo file, each copy took about 270ms, with ??? 11 hangs of 200-700ms. For that last 800Mo bench, copy without MBB was steady 330ms. >>?? FileChannelImpl.transferFrom(...): >>?? This method can grow destination channel, but if the specified >> position (in dst) is > dst.size(), it just returns 0. It looks >> like a bug, as the spec says nothing about this surprising behavior. > >This is specified: " If the given position is greater than the file's >current size then no bytes are transferred" My bad. My surprise made me confident it was a problem, instead of making me read the spec more carefully. NB: In the code I linked (ByteCopyUtils), I had a bug in case of concurrent src truncation or no space left on device, during a backward copy, where the position was moved by the amount of bytes copied, i.e. as if the copy had been done forward. It is now corrected and I throw whenever the copy stops earlier than could be expected from initial src size and dst capacity. -Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20121103/08a6e3f9/attachment-0001.html From Alan.Bateman at oracle.com Sat Nov 3 14:20:01 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 03 Nov 2012 21:20:01 +0000 Subject: 8000330: (fc) FileChannel.truncate issues when given size > file size Message-ID: <50958A81.4080104@oracle.com> This mail concerns a couple of the issues that Jeff Hain has brought up on the list in the last few days. 8000330 concerns the issues with FileChannel's truncate method when invoked with a size that is greater than the current size. This case doesn't change the file of course but it should change the channel's position when it is beyond the given size. This case has an additional corner case when the file is opened for reading that isn't handled properly in the current implementation. 8002180 is just that FileChannel.map doesn't throw NullPointerException when invoked with a MapMode of null. The webrev with the proposed changes is here: http://cr.openjdk.java.net/~alanb/8000330+8002180/webrev/ The changes are trivial, with most of the changes just adding additional tests. They aren't issues that most applications are likely to run into but they are issues that should have been caught by tests back in jdk1.4. Note that the changes to MapTest are less than what might seem initially, this is because I updated the existing test to use try-with-resources. I should also note that FileChannelImpl.map is deliberately not using Objects.requireNonNull, this is because there are several pre-conditions to check and I'm trying to keep them consistent. Thanks, Alan. From jeffhain at rocketmail.com Sat Nov 3 15:09:00 2012 From: jeffhain at rocketmail.com (Jeff Hain) Date: Sat, 3 Nov 2012 22:09:00 +0000 (GMT) Subject: 8000330: (fc) FileChannel.truncate issues when given size > file size In-Reply-To: <50958A81.4080104@oracle.com> References: <50958A81.4080104@oracle.com> Message-ID: <1351980540.97795.YahooMailNeo@web132101.mail.ird.yahoo.com> >The webrev with the proposed changes is here: > >http://cr.openjdk.java.net/~alanb/8000330+8002180/webrev/ position0(fd, p) is called even if position didn't change (was already the case before). Also the exception message could be "Negative size" for homogeneity with map method. -Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20121103/0dd9ddde/attachment.html From chris.hegarty at oracle.com Sun Nov 4 00:52:49 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Sun, 04 Nov 2012 07:52:49 +0000 Subject: 8000330: (fc) FileChannel.truncate issues when given size > file size In-Reply-To: <1351980540.97795.YahooMailNeo@web132101.mail.ird.yahoo.com> References: <50958A81.4080104@oracle.com> <1351980540.97795.YahooMailNeo@web132101.mail.ird.yahoo.com> Message-ID: <50961ED1.1040105@oracle.com> On 11/03/2012 10:09 PM, Jeff Hain wrote: >>The webrev with the proposed changes is here: >> >>http://cr.openjdk.java.net/~alanb/8000330+8002180/webrev/ > > position0(fd, p) is called even if position didn't change > (was already the case before). Agreed. I think it is deliberate on your part, but if there is a concern about maintaining existing behavior, you could move the size check after the writable check. Otherwise looks fine. -Chris. > > Also the exception message could be "Negative size" > for homogeneity with map method. > > -Jeff > From Alan.Bateman at oracle.com Sun Nov 4 02:57:28 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sun, 04 Nov 2012 10:57:28 +0000 Subject: 8000330: (fc) FileChannel.truncate issues when given size > file size In-Reply-To: <1351980540.97795.YahooMailNeo@web132101.mail.ird.yahoo.com> References: <50958A81.4080104@oracle.com> <1351980540.97795.YahooMailNeo@web132101.mail.ird.yahoo.com> Message-ID: <50964A18.6040906@oracle.com> On 03/11/2012 22:09, Jeff Hain wrote: > >The webrev with the proposed changes is here: > > > >http://cr.openjdk.java.net/~alanb/8000330+8002180/webrev/ > > > position0(fd, p) is called even if position didn't change > (was already the case before). This is necessary because truncating the file can change the position, Windows in particular. It could re-factored to deal with platform specific behavior or the platform implementations could be changed to reset the position but it hardly seems worth it. > > Also the exception message could be "Negative size" > for homogeneity with map method. This make sense - thanks. -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20121104/d1624693/attachment.html From Alan.Bateman at oracle.com Mon Nov 5 04:23:20 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 05 Nov 2012 12:23:20 +0000 Subject: Review Request - JDK-6720349: (ch) Channels tests depending on hosts inside Sun In-Reply-To: <50901D36.3060602@oracle.com> References: <50901D36.3060602@oracle.com> Message-ID: <5097AFB8.4030006@oracle.com> On 30/10/2012 18:32, Daniel Fuchs wrote: > Hi, > > I'm looking for reviewers for: > > JDK-6720349: (ch) Channels tests depending on hosts inside Sun > http://cr.openjdk.java.net/~dfuchs/JDK-6720349/webrev.00/ > > This is to fix a test issue. > > Several tests in the jdk_nio test suites depends on remote services > (echo, daytime, discard, to cite a few) being available on remote > hosts inside Oracle network. > > The issue is that these tests cannot succeed outside of oracle network > without modification, and were often known to fail even inside oracle > networks depending on the availability of these hosts. > > This patch removes the dependencies on remote hosts by making the tests > instantiate and start small TCP or UDP servers within the test VM, thus > emulating the services that used to be provided by the internal > remote hosts. > > There are however - a few side effects on the tests themselves (read: on > what the tests are actually testing). > > Given that the servers accessed by the tests are now co-located on the > same machine - some assumptions made by the test code had to > be revisited. For instance, some of the tests assumed that by > setting the non blocking flag of the channel to true, and then > calling connect(), they would always obtain a channel in a > connection pending state. With servers on the same hosts, this is > no longer always the case (for instance on Solaris you end up > with a channel already connected). > > As a consequence some of the tests, on some architectures, no > longer exactly test what they used to be testing. I have identified > these parts of the test code that may not always be activated > and added comments to outline this. > > Tests performed: > > I ran the jdk_nio tests using JPRT (hotspotwest). > > best regards, > > -- daniel Thanks for doing this, this is long overdue. Also nice that there aren't too many changes to the tests themselves. My preference would be to put the test servers into a new class TestClass, to sit along aside TestThread and TestUtil. I suggest this because TestUtil was originally intended for utility methods for these tests. The only other general observation is that some of the indenting when statements or source lines slip into a second line is a bit inconsistent, I can't tell what is doing the formatting but it's just a bit inconsistent with the style that is used in this area (we don't have an JDK-wide coding convention/style guidelines). A couple of specific comments: - In Alias then the comment about Solaris should probably be replaced with a comment to say that the connection is established immediately. I also suggest dropping the message to System.err as this is inside the while loop and so will be printed LIMIT times. - BasicConnect.java (and many other tests) then the message that the connection is established immediately is printed to System.err, which suggests its an error or warning but it is normal so I'd suggest dropping it or printing it to System.out. - SocketChannel/Connect.java - the comment at L40-41 will likely confusing future maintainers, I don't think it is needed. - SocketChannel/ConnectState.java - I assume there isn't any need to check for exceptions==null or exceptions[0]==null in expectedExceptions. Otherwise the additionally handling of ST_PENDING_OR_CONNECT looks fine to me. So overall I think this will be great to get in. -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20121105/7fc17095/attachment.html From daniel.fuchs at oracle.com Mon Nov 5 05:47:41 2012 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Mon, 05 Nov 2012 14:47:41 +0100 Subject: Review Request - JDK-6720349: (ch) Channels tests depending on hosts inside Sun In-Reply-To: <5097AFB8.4030006@oracle.com> References: <50901D36.3060602@oracle.com> <5097AFB8.4030006@oracle.com> Message-ID: <5097C37D.3040106@oracle.com> Hi Alan, Thanks for reviewing. I will make the changes and send another webrev when they are ready. Do I need a second reviewer? Or is one reviewer enough? See some comments inline: On 11/5/12 1:23 PM, Alan Bateman wrote: > On 30/10/2012 18:32, Daniel Fuchs wrote: >> Hi, >> >> I'm looking for reviewers for: >> >> JDK-6720349: (ch) Channels tests depending on hosts inside Sun >> http://cr.openjdk.java.net/~dfuchs/JDK-6720349/webrev.00/ >> >> This is to fix a test issue. [...] > > My preference would be to put the test servers into a new class > TestClass, to sit along aside TestThread and TestUtil. I suggest this > because TestUtil was originally intended for utility methods for these > tests. Will do. If I add a new file next to TestUtil, I won't have to modify any makefile right? > The only other general observation is that some of the indenting when > statements or source lines slip into a second line is a bit > inconsistent, I can't tell what is doing the formatting but it's just a > bit inconsistent with the style that is used in this area (we don't have > an JDK-wide coding convention/style guidelines). I think I let NetBeans do the formatting in most cases - except when it looked confusing. I'd be happy to reformat - or to configure NetBeans with the appropriate options - I think what looks strange is the indentation in the try () { } with resource statements, when the try () line is split in two. Is that what surprised you? > A couple of specific comments: > > - In Alias then the comment about Solaris should probably be replaced > with a comment to say that the connection is established immediately. It looks as if this is platform specific. On Mac OS - (10.7.5) - the behavior was as the text expected. On Solaris - the connection was established immediately. On windows I don't remember what I observed. I can replace "On Solaris" with "On some platforms". > I also suggest dropping the message to System.err as this is inside the > while loop and so will be printed LIMIT times. OK. > - BasicConnect.java (and many other tests) then the message that the > connection is established immediately is printed to System.err, which > suggests its an error or warning but it is normal so I'd suggest > dropping it or printing it to System.out. OK. > - SocketChannel/Connect.java - the comment at L40-41 will likely > confusing future maintainers, I don't think it is needed. OK. > - SocketChannel/ConnectState.java - I assume there isn't any need to > check for exceptions==null or exceptions[0]==null in expectedExceptions. Yes. I changed my mind between the time I wrote the method and the time I called it. Will fix :-) > Otherwise the additionally handling of ST_PENDING_OR_CONNECT looks fine > to me. > > So overall I think this will be great to get in. > > -Alan. From Alan.Bateman at oracle.com Mon Nov 5 07:46:13 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 05 Nov 2012 15:46:13 +0000 Subject: Review Request - JDK-6720349: (ch) Channels tests depending on hosts inside Sun In-Reply-To: <5097C37D.3040106@oracle.com> References: <50901D36.3060602@oracle.com> <5097AFB8.4030006@oracle.com> <5097C37D.3040106@oracle.com> Message-ID: <5097DF45.2010403@oracle.com> On 05/11/2012 13:47, Daniel Fuchs wrote: > Hi Alan, > > Thanks for reviewing. > I will make the changes and send another webrev when they are ready. > Do I need a second reviewer? Or is one reviewer enough? For the jdk repo then one reviewer is fine. > : > > Will do. If I add a new file next to TestUtil, I won't have to > modify any makefile right? No need to change any make files. >> >> - In Alias then the comment about Solaris should probably be replaced >> with a comment to say that the connection is established immediately. > > It looks as if this is platform specific. On Mac OS - (10.7.5) - the > behavior was as the text expected. On Solaris - the connection was > established immediately. On windows I don't remember what I observed. > I can replace "On Solaris" with "On some platforms". Thanks. -Alan From daniel.fuchs at oracle.com Tue Nov 6 08:10:41 2012 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Tue, 06 Nov 2012 17:10:41 +0100 Subject: Review Request - JDK-6720349: (ch) Channels tests depending on hosts inside Sun In-Reply-To: <5097AFB8.4030006@oracle.com> References: <50901D36.3060602@oracle.com> <5097AFB8.4030006@oracle.com> Message-ID: <50993681.3080101@oracle.com> Hi, Here is a new webrev incorporating Alan's feedback. I fixed the indenting issues by looking at the original code in the modified file (or in other files in the same directory), and trying to guess how the author might have indented the lines in question. That's just guesses on my part but hopefully it's better now than before. http://cr.openjdk.java.net/~dfuchs/JDK-6720349/webrev.01/ best regards, -- daniel On 11/5/12 1:23 PM, Alan Bateman wrote: > On 30/10/2012 18:32, Daniel Fuchs wrote: >> Hi, >> >> I'm looking for reviewers for: >> >> JDK-6720349: (ch) Channels tests depending on hosts inside Sun >> http://cr.openjdk.java.net/~dfuchs/JDK-6720349/webrev.00/ >> >> This is to fix a test issue. >> >> Several tests in the jdk_nio test suites depends on remote services >> (echo, daytime, discard, to cite a few) being available on remote >> hosts inside Oracle network. >> >> The issue is that these tests cannot succeed outside of oracle network >> without modification, and were often known to fail even inside oracle >> networks depending on the availability of these hosts. >> >> This patch removes the dependencies on remote hosts by making the tests >> instantiate and start small TCP or UDP servers within the test VM, thus >> emulating the services that used to be provided by the internal >> remote hosts. >> >> There are however - a few side effects on the tests themselves (read: on >> what the tests are actually testing). >> >> Given that the servers accessed by the tests are now co-located on the >> same machine - some assumptions made by the test code had to >> be revisited. For instance, some of the tests assumed that by >> setting the non blocking flag of the channel to true, and then >> calling connect(), they would always obtain a channel in a >> connection pending state. With servers on the same hosts, this is >> no longer always the case (for instance on Solaris you end up >> with a channel already connected). >> >> As a consequence some of the tests, on some architectures, no >> longer exactly test what they used to be testing. I have identified >> these parts of the test code that may not always be activated >> and added comments to outline this. >> >> Tests performed: >> >> I ran the jdk_nio tests using JPRT (hotspotwest). >> >> best regards, >> >> -- daniel > Thanks for doing this, this is long overdue. Also nice that there aren't > too many changes to the tests themselves. > > My preference would be to put the test servers into a new class > TestClass, to sit along aside TestThread and TestUtil. I suggest this > because TestUtil was originally intended for utility methods for these > tests. > > The only other general observation is that some of the indenting when > statements or source lines slip into a second line is a bit > inconsistent, I can't tell what is doing the formatting but it's just a > bit inconsistent with the style that is used in this area (we don't have > an JDK-wide coding convention/style guidelines). > > A couple of specific comments: > > - In Alias then the comment about Solaris should probably be replaced > with a comment to say that the connection is established immediately. I > also suggest dropping the message to System.err as this is inside the > while loop and so will be printed LIMIT times. > > - BasicConnect.java (and many other tests) then the message that the > connection is established immediately is printed to System.err, which > suggests its an error or warning but it is normal so I'd suggest > dropping it or printing it to System.out. > > - SocketChannel/Connect.java - the comment at L40-41 will likely > confusing future maintainers, I don't think it is needed. > > - SocketChannel/ConnectState.java - I assume there isn't any need to > check for exceptions==null or exceptions[0]==null in expectedExceptions. > Otherwise the additionally handling of ST_PENDING_OR_CONNECT looks fine > to me. > > So overall I think this will be great to get in. > > -Alan. > > > > > From Alan.Bateman at oracle.com Tue Nov 6 08:49:56 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 06 Nov 2012 16:49:56 +0000 Subject: Review Request - JDK-6720349: (ch) Channels tests depending on hosts inside Sun In-Reply-To: <50993681.3080101@oracle.com> References: <50901D36.3060602@oracle.com> <5097AFB8.4030006@oracle.com> <50993681.3080101@oracle.com> Message-ID: <50993FB4.9040102@oracle.com> On 06/11/2012 16:10, Daniel Fuchs wrote: > Hi, > > Here is a new webrev incorporating Alan's feedback. > > I fixed the indenting issues by looking at the original code > in the modified file (or in other files in the same directory), > and trying to guess how the author might have indented the lines > in question. That's just guesses on my part but hopefully it's > better now than before. > > http://cr.openjdk.java.net/~dfuchs/JDK-6720349/webrev.01/ > > best regards, > > -- daniel I think this looks very good. The indenting was a bit of a side issue anything, I think it's fine now. On "TestClass", I just realized I suggested this to you in the first mail. My brain was thinking "TestServers", my fingers wrote "TestClass" for some reason and I see you've pulled it out of TestUtil with that name. Minor nits: - in LocalAddress the comment reads "... hopefully it doesn't matter for the test purposes". I think this can be removed as an echo server is fine here. - typo in the comment in ConnectState.java (line 114). It reads " We may get a an AlreadyConnected exception", I assume this should be "We may get AlreadyConnectedException". Otherwise, thumbs up from me. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20121106/de9cd496/attachment.html From chris.hegarty at oracle.com Wed Nov 7 04:50:38 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 07 Nov 2012 12:50:38 +0000 Subject: Review Request - JDK-6720349: (ch) Channels tests depending on hosts inside Sun In-Reply-To: <50993681.3080101@oracle.com> References: <50901D36.3060602@oracle.com> <5097AFB8.4030006@oracle.com> <50993681.3080101@oracle.com> Message-ID: <509A591E.4040608@oracle.com> This looks really good Daniel. * Trivially, RefusingServer. startNewServer L84 could use try-with-resources. * AbstractTcpServer.newServerSocket seems unnecessary -Chris. On 06/11/2012 16:10, Daniel Fuchs wrote: > Hi, > > Here is a new webrev incorporating Alan's feedback. > > I fixed the indenting issues by looking at the original code > in the modified file (or in other files in the same directory), > and trying to guess how the author might have indented the lines > in question. That's just guesses on my part but hopefully it's > better now than before. > > http://cr.openjdk.java.net/~dfuchs/JDK-6720349/webrev.01/ > > best regards, > > -- daniel > > > On 11/5/12 1:23 PM, Alan Bateman wrote: >> On 30/10/2012 18:32, Daniel Fuchs wrote: >>> Hi, >>> >>> I'm looking for reviewers for: >>> >>> JDK-6720349: (ch) Channels tests depending on hosts inside Sun >>> http://cr.openjdk.java.net/~dfuchs/JDK-6720349/webrev.00/ >>> >>> This is to fix a test issue. >>> >>> Several tests in the jdk_nio test suites depends on remote services >>> (echo, daytime, discard, to cite a few) being available on remote >>> hosts inside Oracle network. >>> >>> The issue is that these tests cannot succeed outside of oracle network >>> without modification, and were often known to fail even inside oracle >>> networks depending on the availability of these hosts. >>> >>> This patch removes the dependencies on remote hosts by making the tests >>> instantiate and start small TCP or UDP servers within the test VM, thus >>> emulating the services that used to be provided by the internal >>> remote hosts. >>> >>> There are however - a few side effects on the tests themselves (read: on >>> what the tests are actually testing). >>> >>> Given that the servers accessed by the tests are now co-located on the >>> same machine - some assumptions made by the test code had to >>> be revisited. For instance, some of the tests assumed that by >>> setting the non blocking flag of the channel to true, and then >>> calling connect(), they would always obtain a channel in a >>> connection pending state. With servers on the same hosts, this is >>> no longer always the case (for instance on Solaris you end up >>> with a channel already connected). >>> >>> As a consequence some of the tests, on some architectures, no >>> longer exactly test what they used to be testing. I have identified >>> these parts of the test code that may not always be activated >>> and added comments to outline this. >>> >>> Tests performed: >>> >>> I ran the jdk_nio tests using JPRT (hotspotwest). >>> >>> best regards, >>> >>> -- daniel >> Thanks for doing this, this is long overdue. Also nice that there aren't >> too many changes to the tests themselves. >> >> My preference would be to put the test servers into a new class >> TestClass, to sit along aside TestThread and TestUtil. I suggest this >> because TestUtil was originally intended for utility methods for these >> tests. >> >> The only other general observation is that some of the indenting when >> statements or source lines slip into a second line is a bit >> inconsistent, I can't tell what is doing the formatting but it's just a >> bit inconsistent with the style that is used in this area (we don't have >> an JDK-wide coding convention/style guidelines). >> >> A couple of specific comments: >> >> - In Alias then the comment about Solaris should probably be replaced >> with a comment to say that the connection is established immediately. I >> also suggest dropping the message to System.err as this is inside the >> while loop and so will be printed LIMIT times. >> >> - BasicConnect.java (and many other tests) then the message that the >> connection is established immediately is printed to System.err, which >> suggests its an error or warning but it is normal so I'd suggest >> dropping it or printing it to System.out. >> >> - SocketChannel/Connect.java - the comment at L40-41 will likely >> confusing future maintainers, I don't think it is needed. >> >> - SocketChannel/ConnectState.java - I assume there isn't any need to >> check for exceptions==null or exceptions[0]==null in expectedExceptions. >> Otherwise the additionally handling of ST_PENDING_OR_CONNECT looks fine >> to me. >> >> So overall I think this will be great to get in. >> >> -Alan. >> >> >> >> >> > From daniel.fuchs at oracle.com Wed Nov 7 05:08:34 2012 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Wed, 07 Nov 2012 14:08:34 +0100 Subject: Review Request - JDK-6720349: (ch) Channels tests depending on hosts inside Sun In-Reply-To: <509A591E.4040608@oracle.com> References: <50901D36.3060602@oracle.com> <5097AFB8.4030006@oracle.com> <50993681.3080101@oracle.com> <509A591E.4040608@oracle.com> Message-ID: <509A5D52.5060007@oracle.com> On 11/7/12 1:50 PM, Chris Hegarty wrote: > This looks really good Daniel. Thanks. > * Trivially, RefusingServer. startNewServer L84 could use > try-with-resources. Refusing server is not connected - and thus not closeable. It's just a bean that wrap an address on which no one is listening. I could add a dummy noop close() for consistency but why bother? > * AbstractTcpServer.newServerSocket seems unnecessary I naively thought I might be able to use a subclass of ServerSocket to trigger connection timeout with delays in the accept() method but it did not work. The method remained. Do you wish me to remove it? I was almost ready to push :-) -- daniel > -Chris. > > On 06/11/2012 16:10, Daniel Fuchs wrote: >> Hi, >> >> Here is a new webrev incorporating Alan's feedback. >> >> I fixed the indenting issues by looking at the original code >> in the modified file (or in other files in the same directory), >> and trying to guess how the author might have indented the lines >> in question. That's just guesses on my part but hopefully it's >> better now than before. >> >> http://cr.openjdk.java.net/~dfuchs/JDK-6720349/webrev.01/ >> >> best regards, >> >> -- daniel >> >> >> On 11/5/12 1:23 PM, Alan Bateman wrote: >>> On 30/10/2012 18:32, Daniel Fuchs wrote: >>>> Hi, >>>> >>>> I'm looking for reviewers for: >>>> >>>> JDK-6720349: (ch) Channels tests depending on hosts inside Sun >>>> http://cr.openjdk.java.net/~dfuchs/JDK-6720349/webrev.00/ >>>> >>>> This is to fix a test issue. >>>> >>>> Several tests in the jdk_nio test suites depends on remote services >>>> (echo, daytime, discard, to cite a few) being available on remote >>>> hosts inside Oracle network. >>>> >>>> The issue is that these tests cannot succeed outside of oracle network >>>> without modification, and were often known to fail even inside oracle >>>> networks depending on the availability of these hosts. >>>> >>>> This patch removes the dependencies on remote hosts by making the tests >>>> instantiate and start small TCP or UDP servers within the test VM, thus >>>> emulating the services that used to be provided by the internal >>>> remote hosts. >>>> >>>> There are however - a few side effects on the tests themselves >>>> (read: on >>>> what the tests are actually testing). >>>> >>>> Given that the servers accessed by the tests are now co-located on the >>>> same machine - some assumptions made by the test code had to >>>> be revisited. For instance, some of the tests assumed that by >>>> setting the non blocking flag of the channel to true, and then >>>> calling connect(), they would always obtain a channel in a >>>> connection pending state. With servers on the same hosts, this is >>>> no longer always the case (for instance on Solaris you end up >>>> with a channel already connected). >>>> >>>> As a consequence some of the tests, on some architectures, no >>>> longer exactly test what they used to be testing. I have identified >>>> these parts of the test code that may not always be activated >>>> and added comments to outline this. >>>> >>>> Tests performed: >>>> >>>> I ran the jdk_nio tests using JPRT (hotspotwest). >>>> >>>> best regards, >>>> >>>> -- daniel >>> Thanks for doing this, this is long overdue. Also nice that there aren't >>> too many changes to the tests themselves. >>> >>> My preference would be to put the test servers into a new class >>> TestClass, to sit along aside TestThread and TestUtil. I suggest this >>> because TestUtil was originally intended for utility methods for these >>> tests. >>> >>> The only other general observation is that some of the indenting when >>> statements or source lines slip into a second line is a bit >>> inconsistent, I can't tell what is doing the formatting but it's just a >>> bit inconsistent with the style that is used in this area (we don't have >>> an JDK-wide coding convention/style guidelines). >>> >>> A couple of specific comments: >>> >>> - In Alias then the comment about Solaris should probably be replaced >>> with a comment to say that the connection is established immediately. I >>> also suggest dropping the message to System.err as this is inside the >>> while loop and so will be printed LIMIT times. >>> >>> - BasicConnect.java (and many other tests) then the message that the >>> connection is established immediately is printed to System.err, which >>> suggests its an error or warning but it is normal so I'd suggest >>> dropping it or printing it to System.out. >>> >>> - SocketChannel/Connect.java - the comment at L40-41 will likely >>> confusing future maintainers, I don't think it is needed. >>> >>> - SocketChannel/ConnectState.java - I assume there isn't any need to >>> check for exceptions==null or exceptions[0]==null in expectedExceptions. >>> Otherwise the additionally handling of ST_PENDING_OR_CONNECT looks fine >>> to me. >>> >>> So overall I think this will be great to get in. >>> >>> -Alan. >>> >>> >>> >>> >>> >> From chris.hegarty at oracle.com Wed Nov 7 05:56:37 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 07 Nov 2012 13:56:37 +0000 Subject: Review Request - JDK-6720349: (ch) Channels tests depending on hosts inside Sun In-Reply-To: <509A5D52.5060007@oracle.com> References: <50901D36.3060602@oracle.com> <5097AFB8.4030006@oracle.com> <50993681.3080101@oracle.com> <509A591E.4040608@oracle.com> <509A5D52.5060007@oracle.com> Message-ID: <509A6895.5030904@oracle.com> On 07/11/2012 13:08, Daniel Fuchs wrote: > On 11/7/12 1:50 PM, Chris Hegarty wrote: >> This looks really good Daniel. > > Thanks. > >> * Trivially, RefusingServer. startNewServer L84 could use >> try-with-resources. > > Refusing server is not connected - and thus not closeable. > It's just a bean that wrap an address on which no one is listening. > I could add a dummy noop close() for consistency but why bother? I was thinking of the ServerSocket that you later explicitly close. It is not really not worth spending time on. >> * AbstractTcpServer.newServerSocket seems unnecessary > > I naively thought I might be able to use a subclass of > ServerSocket to trigger connection timeout with delays in > the accept() method but it did not work. The method remained. > Do you wish me to remove it? I was almost ready to push :-) It is not necessary to make any changes. My comments were just trivial things I noticed. -Chris. > > -- daniel > >> -Chris. >> >> On 06/11/2012 16:10, Daniel Fuchs wrote: >>> Hi, >>> >>> Here is a new webrev incorporating Alan's feedback. >>> >>> I fixed the indenting issues by looking at the original code >>> in the modified file (or in other files in the same directory), >>> and trying to guess how the author might have indented the lines >>> in question. That's just guesses on my part but hopefully it's >>> better now than before. >>> >>> http://cr.openjdk.java.net/~dfuchs/JDK-6720349/webrev.01/ >>> >>> best regards, >>> >>> -- daniel >>> >>> >>> On 11/5/12 1:23 PM, Alan Bateman wrote: >>>> On 30/10/2012 18:32, Daniel Fuchs wrote: >>>>> Hi, >>>>> >>>>> I'm looking for reviewers for: >>>>> >>>>> JDK-6720349: (ch) Channels tests depending on hosts inside Sun >>>>> http://cr.openjdk.java.net/~dfuchs/JDK-6720349/webrev.00/ >>>>> >>>>> This is to fix a test issue. >>>>> >>>>> Several tests in the jdk_nio test suites depends on remote services >>>>> (echo, daytime, discard, to cite a few) being available on remote >>>>> hosts inside Oracle network. >>>>> >>>>> The issue is that these tests cannot succeed outside of oracle network >>>>> without modification, and were often known to fail even inside oracle >>>>> networks depending on the availability of these hosts. >>>>> >>>>> This patch removes the dependencies on remote hosts by making the >>>>> tests >>>>> instantiate and start small TCP or UDP servers within the test VM, >>>>> thus >>>>> emulating the services that used to be provided by the internal >>>>> remote hosts. >>>>> >>>>> There are however - a few side effects on the tests themselves >>>>> (read: on >>>>> what the tests are actually testing). >>>>> >>>>> Given that the servers accessed by the tests are now co-located on the >>>>> same machine - some assumptions made by the test code had to >>>>> be revisited. For instance, some of the tests assumed that by >>>>> setting the non blocking flag of the channel to true, and then >>>>> calling connect(), they would always obtain a channel in a >>>>> connection pending state. With servers on the same hosts, this is >>>>> no longer always the case (for instance on Solaris you end up >>>>> with a channel already connected). >>>>> >>>>> As a consequence some of the tests, on some architectures, no >>>>> longer exactly test what they used to be testing. I have identified >>>>> these parts of the test code that may not always be activated >>>>> and added comments to outline this. >>>>> >>>>> Tests performed: >>>>> >>>>> I ran the jdk_nio tests using JPRT (hotspotwest). >>>>> >>>>> best regards, >>>>> >>>>> -- daniel >>>> Thanks for doing this, this is long overdue. Also nice that there >>>> aren't >>>> too many changes to the tests themselves. >>>> >>>> My preference would be to put the test servers into a new class >>>> TestClass, to sit along aside TestThread and TestUtil. I suggest this >>>> because TestUtil was originally intended for utility methods for these >>>> tests. >>>> >>>> The only other general observation is that some of the indenting when >>>> statements or source lines slip into a second line is a bit >>>> inconsistent, I can't tell what is doing the formatting but it's just a >>>> bit inconsistent with the style that is used in this area (we don't >>>> have >>>> an JDK-wide coding convention/style guidelines). >>>> >>>> A couple of specific comments: >>>> >>>> - In Alias then the comment about Solaris should probably be replaced >>>> with a comment to say that the connection is established immediately. I >>>> also suggest dropping the message to System.err as this is inside the >>>> while loop and so will be printed LIMIT times. >>>> >>>> - BasicConnect.java (and many other tests) then the message that the >>>> connection is established immediately is printed to System.err, which >>>> suggests its an error or warning but it is normal so I'd suggest >>>> dropping it or printing it to System.out. >>>> >>>> - SocketChannel/Connect.java - the comment at L40-41 will likely >>>> confusing future maintainers, I don't think it is needed. >>>> >>>> - SocketChannel/ConnectState.java - I assume there isn't any need to >>>> check for exceptions==null or exceptions[0]==null in >>>> expectedExceptions. >>>> Otherwise the additionally handling of ST_PENDING_OR_CONNECT looks fine >>>> to me. >>>> >>>> So overall I think this will be great to get in. >>>> >>>> -Alan. >>>> >>>> >>>> >>>> >>>> >>> > From Alan.Bateman at oracle.com Sat Nov 10 14:07:16 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 10 Nov 2012 22:07:16 +0000 Subject: 8003253: TEST_BUG: java/nio/channels/AsynchronousChannelGroup/Unbounded.java hang intermittently [win] Message-ID: <509ED014.1080609@oracle.com> There have been a couple of reports of this test hanging on Windows. The test checks that the default channel group uses an unbounded thread pool. When a read operation completes then the completion handle awaits on a barrier and the test completes when there are 256 threads (plus the main thread) get to the barrier. The problem is that the test completely ignores the case where I/O operations fail and this happens periodically on Windows because the close may be abrupt. The 1-line fix to this is to do a shutdownOutput so that the connection is closed gracefully. The only changes are just to add additional diagnostic output and to ensure that a failure causes the test to terminate in a timely manner. The webrev with the changes is here: http://cr.openjdk.java.net/~alanb/8003253/webrev/ -Alan. From chris.hegarty at oracle.com Sun Nov 11 01:22:37 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Sun, 11 Nov 2012 09:22:37 +0000 Subject: 8003253: TEST_BUG: java/nio/channels/AsynchronousChannelGroup/Unbounded.java hang intermittently [win] In-Reply-To: <509ED014.1080609@oracle.com> References: <509ED014.1080609@oracle.com> Message-ID: I've noticed this failure too, thanks for fixing it. Changes look fine to me. -Chris On 10 Nov 2012, at 22:07, Alan Bateman wrote: > > There have been a couple of reports of this test hanging on Windows. > > The test checks that the default channel group uses an unbounded thread pool. When a read operation completes then the completion handle awaits on a barrier and the test completes when there are 256 threads (plus the main thread) get to the barrier. The problem is that the test completely ignores the case where I/O operations fail and this happens periodically on Windows because the close may be abrupt. The 1-line fix to this is to do a shutdownOutput so that the connection is closed gracefully. The only changes are just to add additional diagnostic output and to ensure that a failure causes the test to terminate in a timely manner. > > The webrev with the changes is here: > > http://cr.openjdk.java.net/~alanb/8003253/webrev/ > > -Alan. From Alan.Bateman at oracle.com Mon Nov 12 11:55:45 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 12 Nov 2012 19:55:45 +0000 Subject: 8003253: TEST_BUG: java/nio/channels/AsynchronousChannelGroup/Unbounded.java hang intermittently [win] In-Reply-To: References: <509ED014.1080609@oracle.com> Message-ID: <50A15441.1010309@oracle.com> Daniel Fuchs pinged me to say that this started failing intermittently him on Mac as soon as I pushed the change for 8003253. Mea culpa, the patch does bring a problem. At the end of the test the listener is closely and this the outstanding accept to fail with an AsycnhronousCloseException and so sets "failed" to true. This creates a race in that the main thread may see failed as true and fails the test. I've created a new bug for this, 8003285, and the patch to fix this is attached. If someone can review then we can sweep this one under the carpet before anyone else notices. -Alan diff --git a/test/java/nio/channels/AsynchronousChannelGroup/Unbounded.java b/test/java/nio/channels/AsynchronousChannelGroup/Unbounded.java --- a/test/java/nio/channels/AsynchronousChannelGroup/Unbounded.java +++ b/test/java/nio/channels/AsynchronousChannelGroup/Unbounded.java @@ -39,6 +39,9 @@ // set to true if an I/O operation fails static volatile boolean failed; + // set to true when the test is done + static volatile boolean finished; + public static void main(String[] args) throws Exception { // all accepted connections are added to a queue final ArrayBlockingQueue queue = @@ -54,8 +57,10 @@ listener.accept((Void)null, this); } public void failed(Throwable exc, Void att) { - failed = true; - System.err.println("accept failed: " + exc); + if (!finished) { + failed = true; + System.err.println("accept failed: " + exc); + } } }); System.out.println("Listener created."); @@ -120,8 +125,11 @@ // wait for all threads to reach the barrier System.out.println("Waiting for all threads to reach barrier"); barrier.await(); + + // finish up + finished = true; listener.close(); if (failed) - throw new RuntimeException("I/O failed failed, see log for details"); + throw new RuntimeException("I/O operation failed, see log for details"); } } From chris.hegarty at oracle.com Tue Nov 13 01:09:33 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 13 Nov 2012 09:09:33 +0000 Subject: 8003253: TEST_BUG: java/nio/channels/AsynchronousChannelGroup/Unbounded.java hang intermittently [win] In-Reply-To: <50A15441.1010309@oracle.com> References: <509ED014.1080609@oracle.com> <50A15441.1010309@oracle.com> Message-ID: <50A20E4D.2040401@oracle.com> Looks fine to me Alan. Nasty race, good to catch and fix it early. -Chris. On 11/12/2012 07:55 PM, Alan Bateman wrote: > > Daniel Fuchs pinged me to say that this started failing intermittently > him on Mac as soon as I pushed the change for 8003253. > > Mea culpa, the patch does bring a problem. At the end of the test the > listener is closely and this the outstanding accept to fail with an > AsycnhronousCloseException and so sets "failed" to true. This creates a > race in that the main thread may see failed as true and fails the test. > > I've created a new bug for this, 8003285, and the patch to fix this is > attached. If someone can review then we can sweep this one under the > carpet before anyone else notices. > > -Alan > > diff --git > a/test/java/nio/channels/AsynchronousChannelGroup/Unbounded.java > b/test/java/nio/channels/AsynchronousChannelGroup/Unbounded.java > --- a/test/java/nio/channels/AsynchronousChannelGroup/Unbounded.java > +++ b/test/java/nio/channels/AsynchronousChannelGroup/Unbounded.java > @@ -39,6 +39,9 @@ > // set to true if an I/O operation fails > static volatile boolean failed; > > + // set to true when the test is done > + static volatile boolean finished; > + > public static void main(String[] args) throws Exception { > // all accepted connections are added to a queue > final ArrayBlockingQueue queue = > @@ -54,8 +57,10 @@ > listener.accept((Void)null, this); > } > public void failed(Throwable exc, Void att) { > - failed = true; > - System.err.println("accept failed: " + exc); > + if (!finished) { > + failed = true; > + System.err.println("accept failed: " + exc); > + } > } > }); > System.out.println("Listener created."); > @@ -120,8 +125,11 @@ > // wait for all threads to reach the barrier > System.out.println("Waiting for all threads to reach barrier"); > barrier.await(); > + > + // finish up > + finished = true; > listener.close(); > if (failed) > - throw new RuntimeException("I/O failed failed, see log for > details"); > + throw new RuntimeException("I/O operation failed, see log > for details"); > } > } > From jim.gish at oracle.com Thu Nov 15 07:20:52 2012 From: jim.gish at oracle.com (Jim Gish) Date: Thu, 15 Nov 2012 10:20:52 -0500 Subject: ENOTDIR - NotDirectoryException or FileSystemException ? In-Reply-To: <50A4D6AA.4030801@oracle.com> References: <509D777E.9020603@oracle.com> <509E3277.7010607@oracle.com> <50A184C7.4050706@oracle.com> <50A23838.9020501@oracle.com>, <50A29097.5020607@oracle.com> <50A2B316.3050805@oracle.com> <50A2BBEA.1080101@oracle.com> <50A37BA9.2000605@oracle.com> <50A4057B.3020006@oracle.com> <50A40F4E.4000807@oracle.com> <50A41328.7040503@oracle.com> <50A4D6AA.4030801@oracle.com> Message-ID: <50A50854.9010702@oracle.com> Moving a discussion between Alan and me here that was on core-libs-dev. In short, I'm suggesting that UnixException.translateToIOException(File,String) check for UnixConstants.ENOTDIR and throw a NotDirectoryException instead of the more general FileSystemException. On 11/15/2012 06:48 AM, Alan Bateman wrote: > >> >> Exactly -- that's my point. This is one of those cases. You're >> trying to create a new file in a directory, but the file you >> specified isn't a directory - it's a plain file. The error code >> coming back is ENOTDIR and so what you get is the more general >> FileSystemException with "Not a directory" as the error message, when >> in fact, we could throw NotDirectoryException if we'd simply check >> for errno of ENOTDIR in the first place along with the other checks >> being done in UnixException translateToIOException(File,String). > This isn't an obvious as it might seem because having ENOTDIR always > map to DirectoryNotEmptyException may cause that exception to be > thrown in other cases too. Additionally, this specific exception was > intended for cases where you attempt to do something on a directory > but it turns out the file is not a directory, this is subtly different > to the case here. So we need to separate this one, I think the > FileSystemException that you are seeing now is reasonable. > I assume you meant to say NotDirectoryException, above. I realize that it might be thrown in other cases, but seems appropriate. I'm fine with it staying as FileSystemException, but would like to understand this thoroughly. The only cases I can see breaking are those that expect the more general FileSystemException and are counting on the message being "Not a directory". Those would break /unless /NotDirectoryException also carried the message "Not a directory", which we can't do since there are currently other messages being used, such as the path in error. You say "this specific exception was intended for cases where you attempt to do something on a directory but it turns out the file is not a directory". Isn't that what we're doing in this case? We're attempting to create a file in the path specified (which is not, as it turns out, a directory). Thanks, Jim > -Alan -- Jim Gish | Consulting Member of Technical Staff | +1.781.442.0304 Oracle Java Platform Group | Core Libraries Team 35 Network Drive Burlington, MA 01803 jim.gish at oracle.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20121115/401263df/attachment.html From david.lloyd at redhat.com Thu Nov 15 07:31:33 2012 From: david.lloyd at redhat.com (David M. Lloyd) Date: Thu, 15 Nov 2012 09:31:33 -0600 Subject: ENOTDIR - NotDirectoryException or FileSystemException ? In-Reply-To: <50A50854.9010702@oracle.com> References: <509D777E.9020603@oracle.com> <509E3277.7010607@oracle.com> <50A184C7.4050706@oracle.com> <50A23838.9020501@oracle.com>, <50A29097.5020607@oracle.com> <50A2B316.3050805@oracle.com> <50A2BBEA.1080101@oracle.com> <50A37BA9.2000605@oracle.com> <50A4057B.3020006@oracle.com> <50A40F4E.4000807@oracle.com> <50A41328.7040503@oracle.com> <50A4D6AA.4030801@oracle.com> <50A50854.9010702@oracle.com> Message-ID: <50A50AD5.8050303@redhat.com> For what my opinion is worth, I agree. I think it is perfectly reasonable to expand the definition of NotDirectoryException to include this case, even though I do acknowledge Alan's point that there is a subtle difference. On 11/15/2012 09:20 AM, Jim Gish wrote: > Moving a discussion between Alan and me here that was on core-libs-dev. > In short, I'm suggesting that > UnixException.translateToIOException(File,String) check for > UnixConstants.ENOTDIR and throw a NotDirectoryException instead of the > more general FileSystemException. > > > On 11/15/2012 06:48 AM, Alan Bateman wrote: >> >>> >>> Exactly -- that's my point. This is one of those cases. You're >>> trying to create a new file in a directory, but the file you >>> specified isn't a directory - it's a plain file. The error code >>> coming back is ENOTDIR and so what you get is the more general >>> FileSystemException with "Not a directory" as the error message, when >>> in fact, we could throw NotDirectoryException if we'd simply check >>> for errno of ENOTDIR in the first place along with the other checks >>> being done in UnixException translateToIOException(File,String). >> This isn't an obvious as it might seem because having ENOTDIR always >> map to DirectoryNotEmptyException may cause that exception to be >> thrown in other cases too. Additionally, this specific exception was >> intended for cases where you attempt to do something on a directory >> but it turns out the file is not a directory, this is subtly different >> to the case here. So we need to separate this one, I think the >> FileSystemException that you are seeing now is reasonable. >> > I assume you meant to say NotDirectoryException, above. I realize that > it might be thrown in other cases, but seems appropriate. I'm fine with > it staying as FileSystemException, but would like to understand this > thoroughly. The only cases I can see breaking are those that expect the > more general FileSystemException and are counting on the message being > "Not a directory". Those would break /unless /NotDirectoryException > also carried the message "Not a directory", which we can't do since > there are currently other messages being used, such as the path in error. > > You say "this specific exception was intended for cases where you > attempt to do something on a directory but it turns out the file is not > a directory". Isn't that what we're doing in this case? We're > attempting to create a file in the path specified (which is not, as it > turns out, a directory). > > Thanks, > Jim > >> -Alan > > -- > Jim Gish | Consulting Member of Technical Staff | +1.781.442.0304 > Oracle Java Platform Group | Core Libraries Team > 35 Network Drive > Burlington, MA 01803 > jim.gish at oracle.com > -- - DML From forax at univ-mlv.fr Thu Nov 15 07:51:22 2012 From: forax at univ-mlv.fr (Remi Forax) Date: Thu, 15 Nov 2012 16:51:22 +0100 Subject: ENOTDIR - NotDirectoryException or FileSystemException ? In-Reply-To: <50A50AD5.8050303@redhat.com> References: <509D777E.9020603@oracle.com> <509E3277.7010607@oracle.com> <50A184C7.4050706@oracle.com> <50A23838.9020501@oracle.com>, <50A29097.5020607@oracle.com> <50A2B316.3050805@oracle.com> <50A2BBEA.1080101@oracle.com> <50A37BA9.2000605@oracle.com> <50A4057B.3020006@oracle.com> <50A40F4E.4000807@oracle.com> <50A41328.7040503@oracle.com> <50A4D6AA.4030801@oracle.com> <50A50854.9010702@oracle.com> <50A50AD5.8050303@redhat.com> Message-ID: <50A50F7A.60903@univ-mlv.fr> On 11/15/2012 04:31 PM, David M. Lloyd wrote: > For what my opinion is worth, I agree. I think it is perfectly > reasonable to expand the definition of NotDirectoryException to > include this case, even though I do acknowledge Alan's point that > there is a subtle difference. +1 R?mi > > On 11/15/2012 09:20 AM, Jim Gish wrote: >> Moving a discussion between Alan and me here that was on core-libs-dev. >> In short, I'm suggesting that >> UnixException.translateToIOException(File,String) check for >> UnixConstants.ENOTDIR and throw a NotDirectoryException instead of the >> more general FileSystemException. >> >> >> On 11/15/2012 06:48 AM, Alan Bateman wrote: >>> >>>> >>>> Exactly -- that's my point. This is one of those cases. You're >>>> trying to create a new file in a directory, but the file you >>>> specified isn't a directory - it's a plain file. The error code >>>> coming back is ENOTDIR and so what you get is the more general >>>> FileSystemException with "Not a directory" as the error message, when >>>> in fact, we could throw NotDirectoryException if we'd simply check >>>> for errno of ENOTDIR in the first place along with the other checks >>>> being done in UnixException translateToIOException(File,String). >>> This isn't an obvious as it might seem because having ENOTDIR always >>> map to DirectoryNotEmptyException may cause that exception to be >>> thrown in other cases too. Additionally, this specific exception was >>> intended for cases where you attempt to do something on a directory >>> but it turns out the file is not a directory, this is subtly different >>> to the case here. So we need to separate this one, I think the >>> FileSystemException that you are seeing now is reasonable. >>> >> I assume you meant to say NotDirectoryException, above. I realize that >> it might be thrown in other cases, but seems appropriate. I'm fine with >> it staying as FileSystemException, but would like to understand this >> thoroughly. The only cases I can see breaking are those that expect the >> more general FileSystemException and are counting on the message being >> "Not a directory". Those would break /unless /NotDirectoryException >> also carried the message "Not a directory", which we can't do since >> there are currently other messages being used, such as the path in >> error. >> >> You say "this specific exception was intended for cases where you >> attempt to do something on a directory but it turns out the file is not >> a directory". Isn't that what we're doing in this case? We're >> attempting to create a file in the path specified (which is not, as it >> turns out, a directory). >> >> Thanks, >> Jim >> >>> -Alan >> >> -- >> Jim Gish | Consulting Member of Technical Staff | +1.781.442.0304 >> Oracle Java Platform Group | Core Libraries Team >> 35 Network Drive >> Burlington, MA 01803 >> jim.gish at oracle.com >> > > From Alan.Bateman at oracle.com Thu Nov 15 08:56:43 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 15 Nov 2012 16:56:43 +0000 Subject: ENOTDIR - NotDirectoryException or FileSystemException ? In-Reply-To: <50A50854.9010702@oracle.com> References: <509D777E.9020603@oracle.com> <509E3277.7010607@oracle.com> <50A184C7.4050706@oracle.com> <50A23838.9020501@oracle.com>, <50A29097.5020607@oracle.com> <50A2B316.3050805@oracle.com> <50A2BBEA.1080101@oracle.com> <50A37BA9.2000605@oracle.com> <50A4057B.3020006@oracle.com> <50A40F4E.4000807@oracle.com> <50A41328.7040503@oracle.com> <50A4D6AA.4030801@oracle.com> <50A50854.9010702@oracle.com> Message-ID: <50A51ECB.1030702@oracle.com> On 15/11/2012 15:20, Jim Gish wrote: > Moving a discussion between Alan and me here that was on > core-libs-dev. In short, I'm suggesting that > UnixException.translateToIOException(File,String) check for > UnixConstants.ENOTDIR and throw a NotDirectoryException instead of the > more general FileSystemException. I'm not against but it would require a small adjustment to NotDirectoryException's spec and also a consideration of which APIs that might optionally specify it - for example the create* methods might specify it as an optional specific exception for the case where you try to create a file, directory, sym link, etc. and the parent path does not resolve to a directory. -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20121115/b564f12e/attachment.html From rob.mckenna at oracle.com Thu Nov 15 10:54:13 2012 From: rob.mckenna at oracle.com (Rob McKenna) Date: Thu, 15 Nov 2012 18:54:13 +0000 Subject: request for review: 6720349: (ch) Channels tests depending on hosts inside Sun Message-ID: <50A53A55.2000808@oracle.com> Hi folks, Looking for a review for: 6720349: (ch) Channels tests depending on hosts inside Sun http://cr.openjdk.java.net/~robm/6720349/webrev.01/ A straight backport from 8, and a very helpful fix. -Rob -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20121115/91851ed0/attachment.html From chris.hegarty at oracle.com Fri Nov 16 01:08:09 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Fri, 16 Nov 2012 09:08:09 +0000 Subject: request for review: 6720349: (ch) Channels tests depending on hosts inside Sun In-Reply-To: <50A53A55.2000808@oracle.com> References: <50A53A55.2000808@oracle.com> Message-ID: <50A60279.20903@oracle.com> Targeting 7u-dev, if the same jdk8 patch ( hg export/import) , then I'm fine with change. -Chris. On 15/11/2012 18:54, Rob McKenna wrote: > Hi folks, > > Looking for a review for: > > 6720349: (ch) Channels tests depending on hosts inside Sun > > http://cr.openjdk.java.net/~robm/6720349/webrev.01/ > > A straight backport from 8, and a very helpful fix. > > -Rob From rob.mckenna at oracle.com Fri Nov 16 06:36:24 2012 From: rob.mckenna at oracle.com (Rob McKenna) Date: Fri, 16 Nov 2012 14:36:24 +0000 Subject: request for review: 6720349: (ch) Channels tests depending on hosts inside Sun In-Reply-To: <50A60279.20903@oracle.com> References: <50A53A55.2000808@oracle.com> <50A60279.20903@oracle.com> Message-ID: <50A64F68.2060302@oracle.com> Yup, sorry, neglected to mention I was aiming for 7u-dev. -Rob On 16/11/12 09:08, Chris Hegarty wrote: > Targeting 7u-dev, if the same jdk8 patch ( hg export/import) , then > I'm fine with change. > > -Chris. > > On 15/11/2012 18:54, Rob McKenna wrote: >> Hi folks, >> >> Looking for a review for: >> >> 6720349: (ch) Channels tests depending on hosts inside Sun >> >> http://cr.openjdk.java.net/~robm/6720349/webrev.01/ >> >> >> A straight backport from 8, and a very helpful fix. >> >> -Rob From Alan.Bateman at oracle.com Mon Nov 19 05:57:38 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 19 Nov 2012 13:57:38 +0000 Subject: request for review: 6720349: (ch) Channels tests depending on hosts inside Sun In-Reply-To: <50A53A55.2000808@oracle.com> References: <50A53A55.2000808@oracle.com> Message-ID: <50AA3AD2.5010203@oracle.com> On 15/11/2012 18:54, Rob McKenna wrote: > Hi folks, > > Looking for a review for: > > 6720349: (ch) Channels tests depending on hosts inside Sun > > http://cr.openjdk.java.net/~robm/6720349/webrev.01/ > > A straight backport from 8, and a very helpful fix. > > -Rob Thanks for bringing this to 7u. So just to confirm, you're going to do a hg export + hg import so you'll just bringing Daniel's changes to jdk7u-dev without any changes? If so then thumbs up from me. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20121119/663b18d4/attachment.html From rob.mckenna at oracle.com Mon Nov 19 07:02:06 2012 From: rob.mckenna at oracle.com (Rob McKenna) Date: Mon, 19 Nov 2012 15:02:06 +0000 Subject: request for review: 6720349: (ch) Channels tests depending on hosts inside Sun In-Reply-To: <50AA3AD2.5010203@oracle.com> References: <50A53A55.2000808@oracle.com> <50AA3AD2.5010203@oracle.com> Message-ID: <50AA49EE.1080805@oracle.com> Absolutely. -Rob On 19/11/12 13:57, Alan Bateman wrote: > On 15/11/2012 18:54, Rob McKenna wrote: >> Hi folks, >> >> Looking for a review for: >> >> 6720349: (ch) Channels tests depending on hosts inside Sun >> >> http://cr.openjdk.java.net/~robm/6720349/webrev.01/ >> >> A straight backport from 8, and a very helpful fix. >> >> -Rob > Thanks for bringing this to 7u. So just to confirm, you're going to do > a hg export + hg import so you'll just bringing Daniel's changes to > jdk7u-dev without any changes? If so then thumbs up from me. > > -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20121119/383d50b8/attachment.html From youdwei at linux.vnet.ibm.com Fri Nov 23 00:05:42 2012 From: youdwei at linux.vnet.ibm.com (Deven You) Date: Fri, 23 Nov 2012 16:05:42 +0800 Subject: Using OP_CONNECT with Selector.select causes selector to fire repeatedly In-Reply-To: <4FE17EBD.6060704@oracle.com> References: <4FAA1C40.6040709@linux.vnet.ibm.com> <4FB2BF30.9060902@oracle.com> <4FC72201.7090307@linux.vnet.ibm.com> <4FCDB0B2.1040902@linux.vnet.ibm.com> <4FDEDAD7.60908@linux.vnet.ibm.com> <4FDF10B5.9070703@oracle.com> <4FE17A08.7010801@linux.vnet.ibm.com> <4FE17EBD.6060704@oracle.com> Message-ID: <50AF2E56.1010205@linux.vnet.ibm.com> Hi All, I just made a new java doc change[1] for this problem, could you take a look and give your suggestions so we may improve it. [1] http://cr.openjdk.java.net/~youdwei/ojdk-317/webrev.02/ Thanks a lot! On 06/20/2012 03:41 PM, Alan Bateman wrote: > On 20/06/2012 08:21, Deven You wrote: >> Hi Alan and Jonathan, >> >> I have tried modifying the Selector.select() javadoc[1] so that users >> can get clear information about the behavior of dealing with OP_CONNECT. >> >> Could you or any one else take a look at review it. > I'm really busy at the moment and do not have time to spend on this. > It will mostly likely require updates to Selector and SelectionKey's > class description, a note in SocketChannel connect might be useful too. > > -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20121123/9ac1c325/attachment.html From rob.mckenna at oracle.com Fri Nov 23 20:08:20 2012 From: rob.mckenna at oracle.com (Rob McKenna) Date: Sat, 24 Nov 2012 04:08:20 +0000 Subject: request for review: 6720349: (ch) Channels tests depending on hosts inside Sun In-Reply-To: <50A60279.20903@oracle.com> References: <50A53A55.2000808@oracle.com> <50A60279.20903@oracle.com> Message-ID: <50B04834.6000208@oracle.com> Almost, but not quite. TestServers.java thinks it was moved from TestUtil.java when I do a straight hg export/import. Not completely sure why. (see the webrev) A test/java/nio/channels/TestServers.java ? test/java/nio/channels/TestServers.java.rej The .rej file: /* - * Copyright (c) 2000, 2012, Oracle and/or its affiliates. All rights reserved. + * Copyright (c) 2012, Oracle and/or its affiliates. All rights reserved. * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. * * This code is free software; you can redistribute it and/or modify it Hence the request for this review really. The only change I've made was to the copyright line. -Rob On 16/11/12 09:08, Chris Hegarty wrote: > Targeting 7u-dev, if the same jdk8 patch ( hg export/import) , then > I'm fine with change. > > -Chris. > > On 15/11/2012 18:54, Rob McKenna wrote: >> Hi folks, >> >> Looking for a review for: >> >> 6720349: (ch) Channels tests depending on hosts inside Sun >> >> http://cr.openjdk.java.net/~robm/6720349/webrev.01/ >> >> >> A straight backport from 8, and a very helpful fix. >> >> -Rob From rob.mckenna at oracle.com Fri Nov 23 20:37:28 2012 From: rob.mckenna at oracle.com (Rob McKenna) Date: Sat, 24 Nov 2012 04:37:28 +0000 Subject: request for review: 6720349: (ch) Channels tests depending on hosts inside Sun In-Reply-To: <50B04834.6000208@oracle.com> References: <50A53A55.2000808@oracle.com> <50A60279.20903@oracle.com> <50B04834.6000208@oracle.com> Message-ID: <50B04F08.8040307@oracle.com> Ah, apologies, false alarm. Looks like an old version of mercurial at fault. It is indeed a straight backport. -Rob On 24/11/12 04:08, Rob McKenna wrote: > Almost, but not quite. TestServers.java thinks it was moved from > TestUtil.java when I do a straight hg export/import. Not completely > sure why. (see the webrev) > > A test/java/nio/channels/TestServers.java > ? test/java/nio/channels/TestServers.java.rej > > The .rej file: > /* > - * Copyright (c) 2000, 2012, Oracle and/or its affiliates. All rights > reserved. > + * Copyright (c) 2012, Oracle and/or its affiliates. All rights > reserved. > * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. > * > * This code is free software; you can redistribute it and/or modify it > > Hence the request for this review really. The only change I've made > was to the copyright line. > > -Rob > > > On 16/11/12 09:08, Chris Hegarty wrote: >> Targeting 7u-dev, if the same jdk8 patch ( hg export/import) , then >> I'm fine with change. >> >> -Chris. >> >> On 15/11/2012 18:54, Rob McKenna wrote: >>> Hi folks, >>> >>> Looking for a review for: >>> >>> 6720349: (ch) Channels tests depending on hosts inside Sun >>> >>> http://cr.openjdk.java.net/~robm/6720349/webrev.01/ >>> >>> >>> A straight backport from 8, and a very helpful fix. >>> >>> -Rob > From daniel.fuchs at oracle.com Sat Nov 24 04:06:20 2012 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Sat, 24 Nov 2012 13:06:20 +0100 Subject: request for review: 6720349: (ch) Channels tests depending on hosts inside Sun In-Reply-To: <50B04F08.8040307@oracle.com> References: <50A53A55.2000808@oracle.com> <50A60279.20903@oracle.com> <50B04834.6000208@oracle.com> <50B04F08.8040307@oracle.com> Message-ID: <50B0B83C.1050805@oracle.com> On 11/24/12 5:37 AM, Rob McKenna wrote: > On 24/11/12 04:08, Rob McKenna wrote: >> Almost, but not quite. TestServers.java thinks it was moved from >> TestUtil.java when I do a straight hg export/import. Not completely >> sure why. (see the webrev) >> Sorry about that - I had started by coding all the TestServers as inner classes of TestUtil, and alan suggested it would be better to put them in a separate class. So I copied TestUtil.java from within NetBeans and pasted it as TestServers.java - keeping only the Servers stuff in the new file and removing the rest. I think NetBeans must have added TestServers.java using some mercurial command that remembered this history... I don't think it should matter - hopefully it won't! -- daniel From rob.mckenna at oracle.com Sat Nov 24 06:50:14 2012 From: rob.mckenna at oracle.com (Rob McKenna) Date: Sat, 24 Nov 2012 14:50:14 +0000 Subject: request for review: 6720349: (ch) Channels tests depending on hosts inside Sun In-Reply-To: <50B0B83C.1050805@oracle.com> References: <50A53A55.2000808@oracle.com> <50A60279.20903@oracle.com> <50B04834.6000208@oracle.com> <50B04F08.8040307@oracle.com> <50B0B83C.1050805@oracle.com> Message-ID: <50B0DEA6.6050702@oracle.com> No problem Daniel. As per my follow up mail I thought it was odd so I checked a more recent version of mercurial and the patch applied cleanly. Thanks for the fix! -Rob On 24/11/12 12:06, Daniel Fuchs wrote: > On 11/24/12 5:37 AM, Rob McKenna wrote: >> On 24/11/12 04:08, Rob McKenna wrote: >>> Almost, but not quite. TestServers.java thinks it was moved from >>> TestUtil.java when I do a straight hg export/import. Not completely >>> sure why. (see the webrev) >>> > > Sorry about that - I had started by coding all the TestServers as > inner classes of TestUtil, and alan suggested it would be better to > put them in a separate class. So I copied TestUtil.java from within > NetBeans and pasted it as TestServers.java - keeping only the Servers > stuff in the new file and removing the rest. I think NetBeans must > have added TestServers.java using some mercurial command that > remembered this history... > I don't think it should matter - hopefully it won't! > > -- daniel From dan.xu at oracle.com Sun Nov 25 22:40:02 2012 From: dan.xu at oracle.com (Dan Xu) Date: Sun, 25 Nov 2012 22:40:02 -0800 Subject: Review Request: JDK-7142921,(fs) Files.probeContentType reports a MIME type of "text/plain" on Ubuntu 11.04 Message-ID: <50B30EC2.4060202@oracle.com> Hi, Please help review the code changes for CR 7142921 and CR 7144997. The webrev is uploaded to, http://cr.openjdk.java.net/~dxu/7142921/webrev.00/. In the fix, I added two new file type detectors, one is MimeTypesFileTypeDetector, and the other is MagicFileTypeDetector.MimeTypesFileTypeDetector is using a file name extension to look up its MIME type by checking mime.types files, which can be used in Linux, Solaris, and Mac.MagicFileTypeDetector is using libmagic to detect a file type in Linux. The library, libmagic, is provided in the same package of file command, which is currently only available in Linux platform. Thanks, -Dan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20121125/a9cff863/attachment.html From Alan.Bateman at oracle.com Mon Nov 26 05:46:28 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 26 Nov 2012 13:46:28 +0000 Subject: Review Request: JDK-7142921,(fs) Files.probeContentType reports a MIME type of "text/plain" on Ubuntu 11.04 In-Reply-To: <50B30EC2.4060202@oracle.com> References: <50B30EC2.4060202@oracle.com> Message-ID: <50B372B4.3000401@oracle.com> On 26/11/2012 06:40, Dan Xu wrote: > Hi, > > Please help review the code changes for CR 7142921 and CR 7144997. The > webrev is uploaded to, http://cr.openjdk.java.net/~dxu/7142921/webrev.00/. > > In the fix, I added two new file type detectors, one is > MimeTypesFileTypeDetector, and the other is > MagicFileTypeDetector.MimeTypesFileTypeDetector is using a file name > extension to look up its MIME type by checking mime.types files, which > can be used in Linux, Solaris, and Mac.MagicFileTypeDetector is using > libmagic to detect a file type in Linux. The library, libmagic, is > provided in the same package of file command, which is currently only > available in Linux platform. > > Thanks, > > -Dan Thanks for taking this on. I don't have time to do a detailed review of the two new FileTypeDetector implementations but just some initial comments on the the other changes: 1. I don't think AbstractFileTypeDetector should be changed to check if the file exists and is a directory, maybe you intended this check to be in the libmagic based FileTypeDetector? 2. If we are using /etc/mime.types then we might also have to look at ${user.home}/mime.types as I think that can be used to override the mapping in /etc. 3. It would be good to add a comment to UnixFileSystemProvider.chain. Also if the line length is a problem when you can probably drop protected as the usage is package-private anyway. Otherwise the make file updates looks fine. I will get back to you soon on the magic and mimetypes detectors, just can't spend time on it today. The other big thing with this proposal is to agree the priority of the detectors, we also need to consider the diagnosability on each platform (needs to be easy for the user to understand how the file type is determined). Finally, we should double check that /private/etc/apache2/mime.types is the right place to look on Mac. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20121126/ff7c8d75/attachment.html From mike.duigou at oracle.com Mon Nov 26 10:27:08 2012 From: mike.duigou at oracle.com (Mike Duigou) Date: Mon, 26 Nov 2012 10:27:08 -0800 Subject: Review Request: JDK-7142921, (fs) Files.probeContentType reports a MIME type of "text/plain" on Ubuntu 11.04 In-Reply-To: <50B30EC2.4060202@oracle.com> References: <50B30EC2.4060202@oracle.com> Message-ID: <87C996B1-17D5-4A4A-BDE3-40DB15378980@oracle.com> I am concerned about relying on the location of the mime.types file on mac : /private/etc/apache2/mime.types The Apache installation included in MacOS X was partially disabled in Mountain Lion (10.8) and it seems possible that they will eventually remove it or replace it with a different server. Mike On Nov 25 2012, at 22:40 , Dan Xu wrote: > Hi, > > Please help review the code changes for CR 7142921 and CR 7144997. The webrev is uploaded to, http://cr.openjdk.java.net/~dxu/7142921/webrev.00/. > > In the fix, I added two new file type detectors, one is MimeTypesFileTypeDetector, and the other is MagicFileTypeDetector. MimeTypesFileTypeDetector is using a file name extension to look up its MIME type by checking mime.types files, which can be used in Linux, Solaris, and Mac. MagicFileTypeDetector is using libmagic to detect a file type in Linux. The library, libmagic, is provided in the same package of file command, which is currently only available in Linux platform. > > Thanks, > > -Dan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20121126/58d64772/attachment.html From dan.xu at oracle.com Mon Nov 26 11:14:43 2012 From: dan.xu at oracle.com (Dan Xu) Date: Mon, 26 Nov 2012 11:14:43 -0800 Subject: Review Request: JDK-7142921, (fs) Files.probeContentType reports a MIME type of "text/plain" on Ubuntu 11.04 In-Reply-To: <87C996B1-17D5-4A4A-BDE3-40DB15378980@oracle.com> References: <50B30EC2.4060202@oracle.com> <87C996B1-17D5-4A4A-BDE3-40DB15378980@oracle.com> Message-ID: <50B3BFA3.60002@oracle.com> Hi Mike, Currently, this is the only mime.types file that I find in Mac platform. And according to its comment as below, it is also used by other software. # This file maps Internet media types to unique file extension(s). # Although created for httpd, this file is used by many software systems # and has been placed in the public domain for unlimited redisribution. If this file is not present in some versions of Mac, say Mountain Lion, the code can still proceed by loading other mime.types files and then do the look-up. This detector will load mime.types files from four locations. The first is the file specified by the jvm system property, mime.types. The second is the .mime.types file in the user home directory. The third is the platform-dependent MIME types file, like this file in Mac platform. The last is the resource file, META-INF/mimetypes.default, in resource.jar. Therefore, this detector can at least load META-INF/mimetypes.default from resource.jar file. -Dan On Mon 26 Nov 2012 10:27:08 AM PST, Mike Duigou wrote: > > I am concerned about relying on the location of the mime.types file on > mac : > > /private/etc/apache2/mime.types > > The Apache installation included in MacOS X was partially disabled in > Mountain Lion (10.8) and it seems possible that they will eventually > remove it or replace it with a different server. > > Mike > > > On Nov 25 2012, at 22:40 , Dan Xu wrote: > >> >> Hi, >> >> Please help review the code changes for CR 7142921 and CR 7144997. >> The webrev is uploaded to, >> http://cr.openjdk.java.net/~dxu/7142921/webrev.00/. >> >> In the fix, I added two new file type detectors, one is >> MimeTypesFileTypeDetector, and the other is >> MagicFileTypeDetector.MimeTypesFileTypeDetector is using a file name >> extension to look up its MIME type by checking mime.types files, >> which can be used in Linux, Solaris, and Mac.MagicFileTypeDetector is >> using libmagic to detect a file type in Linux. The library, libmagic, >> is provided in the same package of file command, which is currently >> only available in Linux platform. >> >> Thanks, >> >> -Dan > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20121126/aaed4a53/attachment.html From dan.xu at oracle.com Mon Nov 26 11:29:40 2012 From: dan.xu at oracle.com (Dan Xu) Date: Mon, 26 Nov 2012 11:29:40 -0800 Subject: Review Request: JDK-7142921,(fs) Files.probeContentType reports a MIME type of "text/plain" on Ubuntu 11.04 In-Reply-To: <50B372B4.3000401@oracle.com> References: <50B30EC2.4060202@oracle.com> <50B372B4.3000401@oracle.com> Message-ID: <50B3C324.8070408@oracle.com> Hi Alan, Thanks for your comments. Please see my answers inline. On 11/26/2012 05:46 AM, Alan Bateman wrote: > On 26/11/2012 06:40, Dan Xu wrote: >> Hi, >> >> Please help review the code changes for CR 7142921 and CR 7144997. >> The webrev is uploaded to, >> http://cr.openjdk.java.net/~dxu/7142921/webrev.00/. >> >> In the fix, I added two new file type detectors, one is >> MimeTypesFileTypeDetector, and the other is >> MagicFileTypeDetector.MimeTypesFileTypeDetector is using a file name >> extension to look up its MIME type by checking mime.types files, >> which can be used in Linux, Solaris, and Mac.MagicFileTypeDetector is >> using libmagic to detect a file type in Linux. The library, libmagic, >> is provided in the same package of file command, which is currently >> only available in Linux platform. >> >> Thanks, >> >> -Dan > Thanks for taking this on. I don't have time to do a detailed review > of the two new FileTypeDetector implementations but just some initial > comments on the the other changes: > > 1. I don't think AbstractFileTypeDetector should be changed to check > if the file exists and is a directory, maybe you intended this check > to be in the libmagic based FileTypeDetector? This is a shortcut for known MIME types for non-exist files and directories for all detectors. GnomeFileTypeDetector and MagicFileTypeDetector can detect correctly for those special cases. But MimeTypesFileTypeDetector may return wrong results. I can move the special check into MimeTypesFileTypeDetector if you think it is a better way. > > 2. If we are using /etc/mime.types then we might also have to look at > ${user.home}/mime.types as I think that can be used to override the > mapping in /etc. /etc/mime.types says "Users can add their own types if they wish by creating a ".mime.types" file in their home directory. Definitions included there will take precedence over those listed here." So I try to load ${user.home}/.mime.types before /etc/mime.types. And all files loaded earlier take precedence over files loaded later. > > 3. It would be good to add a comment to UnixFileSystemProvider.chain. > Also if the line length is a problem when you can probably drop > protected as the usage is package-private anyway. I am going to add the comment and remove "protected". > > Otherwise the make file updates looks fine. I will get back to you > soon on the magic and mimetypes detectors, just can't spend time on it > today. The other big thing with this proposal is to agree the priority > of the detectors, we also need to consider the diagnosability on each > platform (needs to be easy for the user to understand how the file > type is determined). Finally, we should double check that > /private/etc/apache2/mime.types is the right place to look on Mac. > > -Alan > > Thanks, -Dan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20121126/f8c9ce30/attachment-0001.html From dan.xu at oracle.com Mon Nov 26 16:53:17 2012 From: dan.xu at oracle.com (Dan Xu) Date: Mon, 26 Nov 2012 16:53:17 -0800 Subject: Review Request: JDK-7142921,(fs) Files.probeContentType reports a MIME type of "text/plain" on Ubuntu 11.04 In-Reply-To: <50B3C324.8070408@oracle.com> References: <50B30EC2.4060202@oracle.com> <50B372B4.3000401@oracle.com> <50B3C324.8070408@oracle.com> Message-ID: <50B40EFD.7020409@oracle.com> Hi, I have added a comment to UnixFileSystemProvider.chain and remove its "protected" keyword. The new webrev is at, http://cr.openjdk.java.net/~dxu/7142921/webrev.01/. Thanks! -Dan On Mon 26 Nov 2012 11:29:40 AM PST, Dan Xu wrote: > Hi Alan, > > Thanks for your comments. Please see my answers inline. > > > On 11/26/2012 05:46 AM, Alan Bateman wrote: >> On 26/11/2012 06:40, Dan Xu wrote: >>> Hi, >>> >>> Please help review the code changes for CR 7142921 and CR 7144997. >>> The webrev is uploaded to, >>> http://cr.openjdk.java.net/~dxu/7142921/webrev.00/. >>> >>> In the fix, I added two new file type detectors, one is >>> MimeTypesFileTypeDetector, and the other is >>> MagicFileTypeDetector.MimeTypesFileTypeDetector is using a file name >>> extension to look up its MIME type by checking mime.types files, >>> which can be used in Linux, Solaris, and Mac.MagicFileTypeDetector >>> is using libmagic to detect a file type in Linux. The library, >>> libmagic, is provided in the same package of file command, which is >>> currently only available in Linux platform. >>> >>> Thanks, >>> >>> -Dan >> Thanks for taking this on. I don't have time to do a detailed review >> of the two new FileTypeDetector implementations but just some initial >> comments on the the other changes: >> >> 1. I don't think AbstractFileTypeDetector should be changed to check >> if the file exists and is a directory, maybe you intended this check >> to be in the libmagic based FileTypeDetector? > This is a shortcut for known MIME types for non-exist files and > directories for all detectors. GnomeFileTypeDetector and > MagicFileTypeDetector can detect correctly for those special cases. > But MimeTypesFileTypeDetector may return wrong results. I can move the > special check into MimeTypesFileTypeDetector if you think it is a > better way. >> >> 2. If we are using /etc/mime.types then we might also have to look at >> ${user.home}/mime.types as I think that can be used to override the >> mapping in /etc. > /etc/mime.types says "Users can add their own types if they wish by > creating a ".mime.types" file in their home directory. Definitions > included there will take precedence over those listed here." So I try > to load ${user.home}/.mime.types before /etc/mime.types. And all files > loaded earlier take precedence over files loaded later. > >> >> 3. It would be good to add a comment to UnixFileSystemProvider.chain. >> Also if the line length is a problem when you can probably drop >> protected as the usage is package-private anyway. > I am going to add the comment and remove "protected". >> >> Otherwise the make file updates looks fine. I will get back to you >> soon on the magic and mimetypes detectors, just can't spend time on >> it today. The other big thing with this proposal is to agree the >> priority of the detectors, we also need to consider the >> diagnosability on each platform (needs to be easy for the user to >> understand how the file type is determined). Finally, we should >> double check that /private/etc/apache2/mime.types is the right place >> to look on Mac. >> >> -Alan >> >> > Thanks, > > -Dan From Alan.Bateman at oracle.com Tue Nov 27 04:06:28 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 27 Nov 2012 12:06:28 +0000 Subject: Review Request: JDK-7142921,(fs) Files.probeContentType reports a MIME type of "text/plain" on Ubuntu 11.04 In-Reply-To: <50B3C324.8070408@oracle.com> References: <50B30EC2.4060202@oracle.com> <50B372B4.3000401@oracle.com> <50B3C324.8070408@oracle.com> Message-ID: <50B4ACC4.9080109@oracle.com> On 26/11/2012 19:29, Dan Xu wrote: >> >> 1. I don't think AbstractFileTypeDetector should be changed to check >> if the file exists and is a directory, maybe you intended this check >> to be in the libmagic based FileTypeDetector? > This is a shortcut for known MIME types for non-exist files and > directories for all detectors. GnomeFileTypeDetector and > MagicFileTypeDetector can detect correctly for those special cases. > But MimeTypesFileTypeDetector may return wrong results. I can move the > special check into MimeTypesFileTypeDetector if you think it is a > better way. A FileTypeDetector does not require that the file exists so if you put this check in AbstractFileTypeDetector then it essentially disable all file type detectors for the case that the file doesn't exist. So I think this needs to be moved down, I suspect only the libmagic based FileTypeDetector needs to know if the file exists or not. -Alan From Alan.Bateman at oracle.com Tue Nov 27 04:39:24 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 27 Nov 2012 12:39:24 +0000 Subject: Review Request: JDK-7142921,(fs) Files.probeContentType reports a MIME type of "text/plain" on Ubuntu 11.04 In-Reply-To: <50B40EFD.7020409@oracle.com> References: <50B30EC2.4060202@oracle.com> <50B372B4.3000401@oracle.com> <50B3C324.8070408@oracle.com> <50B40EFD.7020409@oracle.com> Message-ID: <50B4B47C.9080201@oracle.com> On 27/11/2012 00:53, Dan Xu wrote: > Hi, > > I have added a comment to UnixFileSystemProvider.chain and remove its > "protected" keyword. The new webrev is at, > http://cr.openjdk.java.net/~dxu/7142921/webrev.01/. Thanks! > > -Dan I've looked at the libmagic based FileTypeDetector and it looks good. From the man page, it looks like magic_file will return NULL if it can't open the file, in which case you might not need to test if the file exists. A couple of minor comments: - the function prototype for magic_open uses "flag" but "flags" would be more correct. - in initialize0 you can move the second magic_handle == NULL into the previous statement to avoid checking it twice. - in MagicFileTypeDetector then you shouldn't need to call super explicitly. I still have on my list to review the mime.types FileTypeDetector, hopefully find time soon. I'm sure others can help review that one too. -Alan. From Alan.Bateman at oracle.com Tue Nov 27 06:35:08 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 27 Nov 2012 14:35:08 +0000 Subject: Review Request: JDK-7142921,(fs) Files.probeContentType reports a MIME type of "text/plain" on Ubuntu 11.04 In-Reply-To: <50B40EFD.7020409@oracle.com> References: <50B30EC2.4060202@oracle.com> <50B372B4.3000401@oracle.com> <50B3C324.8070408@oracle.com> <50B40EFD.7020409@oracle.com> Message-ID: <50B4CF9C.5000800@oracle.com> On 27/11/2012 00:53, Dan Xu wrote: > Hi, > > I have added a comment to UnixFileSystemProvider.chain and remove its > "protected" keyword. The new webrev is at, > http://cr.openjdk.java.net/~dxu/7142921/webrev.01/. Thanks! > > -Dan > I found time to do a very quick pass over MimeTypesFileTypeDetector.java. In terms of diagnosability then I think it would be hard for developers to understand how the file type is obtained when there are so many ways to do. To that end it's probably best if MimeTypesFileTypeDetector didn't uses META-INF/mimetypes.default, that resource file is included in rt.jar because it is used by the Java Activation Framework, itself used by JAX-WS and is essentially invisible to users. On ~/.mime.types then we have choice, use it as a fallback in MimeTypesFileTypeDetector so let the provider create two MimeTypesFileTypeDetector instances, one for the system mime.types files, one for the user file. I don't have a preference but the latter would make the FileTypeDetector a bit simpler. In the implProbeContentType method it only support UnixPath but this FileTypeDetector should be independent of the Path type. It looks like it tries a case-insensitive search after a case-sensitive search but file extensions are case sensitive on Solaris and Linux, I realize this will need special handing on Mac. loadFile/parse are a bit messy in that try-with-resources is being combined with an explicit call to close. Given that the mime.types file is likely to be quite small then a possible choice is Files.readAllLines and that would save you some of this. That's all I have for now. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20121127/83a767a3/attachment.html From zhong.j.yu at gmail.com Tue Nov 27 07:20:51 2012 From: zhong.j.yu at gmail.com (Zhong Yu) Date: Tue, 27 Nov 2012 09:20:51 -0600 Subject: Review Request: JDK-7142921, (fs) Files.probeContentType reports a MIME type of "text/plain" on Ubuntu 11.04 In-Reply-To: <50B4CF9C.5000800@oracle.com> References: <50B30EC2.4060202@oracle.com> <50B372B4.3000401@oracle.com> <50B3C324.8070408@oracle.com> <50B40EFD.7020409@oracle.com> <50B4CF9C.5000800@oracle.com> Message-ID: I wish MimeTypesFileTypeDetector is a public utility class, so that an applications can use it directly against a fixed "mime.types" file. Some applications couldn't care less about what OS thinks about file types, since the applications are supposed to be OS-independent. They need predictable and stable file type mapping. (Of course, they can always implement their own MimeTypesFileTypeDetectors) On Tue, Nov 27, 2012 at 8:35 AM, Alan Bateman wrote: > On 27/11/2012 00:53, Dan Xu wrote: > > Hi, > > I have added a comment to UnixFileSystemProvider.chain and remove its > "protected" keyword. The new webrev is at, > http://cr.openjdk.java.net/~dxu/7142921/webrev.01/. Thanks! > > -Dan > > I found time to do a very quick pass over MimeTypesFileTypeDetector.java. > > In terms of diagnosability then I think it would be hard for developers to > understand how the file type is obtained when there are so many ways to do. > To that end it's probably best if MimeTypesFileTypeDetector didn't uses > META-INF/mimetypes.default, that resource file is included in rt.jar because > it is used by the Java Activation Framework, itself used by JAX-WS and is > essentially invisible to users. > > On ~/.mime.types then we have choice, use it as a fallback in > MimeTypesFileTypeDetector so let the provider create two > MimeTypesFileTypeDetector instances, one for the system mime.types files, > one for the user file. I don't have a preference but the latter would make > the FileTypeDetector a bit simpler. > > In the implProbeContentType method it only support UnixPath but this > FileTypeDetector should be independent of the Path type. > > It looks like it tries a case-insensitive search after a case-sensitive > search but file extensions are case sensitive on Solaris and Linux, I > realize this will need special handing on Mac. > > loadFile/parse are a bit messy in that try-with-resources is being combined > with an explicit call to close. Given that the mime.types file is likely to > be quite small then a possible choice is Files.readAllLines and that would > save you some of this. > > That's all I have for now. > > -Alan > > > > > > > > > > From Alan.Bateman at oracle.com Tue Nov 27 07:25:17 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 27 Nov 2012 15:25:17 +0000 Subject: Review Request: JDK-7142921,(fs) Files.probeContentType reports a MIME type of "text/plain" on Ubuntu 11.04 In-Reply-To: References: <50B30EC2.4060202@oracle.com> <50B372B4.3000401@oracle.com> <50B3C324.8070408@oracle.com> <50B40EFD.7020409@oracle.com> <50B4CF9C.5000800@oracle.com> Message-ID: <50B4DB5D.4080705@oracle.com> On 27/11/2012 15:20, Zhong Yu wrote: > I wish MimeTypesFileTypeDetector is a public utility class, so that an > applications can use it directly against a fixed "mime.types" file. > > Some applications couldn't care less about what OS thinks about file > types, since the applications are supposed to be OS-independent. They > need predictable and stable file type mapping. > > (Of course, they can always implement their own MimeTypesFileTypeDetectors) > Are you looking for javax.activiation.MimetypesFileTypeMap? -Alan From zhong.j.yu at gmail.com Tue Nov 27 07:36:16 2012 From: zhong.j.yu at gmail.com (Zhong Yu) Date: Tue, 27 Nov 2012 09:36:16 -0600 Subject: Review Request: JDK-7142921, (fs) Files.probeContentType reports a MIME type of "text/plain" on Ubuntu 11.04 In-Reply-To: <50B4DB5D.4080705@oracle.com> References: <50B30EC2.4060202@oracle.com> <50B372B4.3000401@oracle.com> <50B3C324.8070408@oracle.com> <50B40EFD.7020409@oracle.com> <50B4CF9C.5000800@oracle.com> <50B4DB5D.4080705@oracle.com> Message-ID: On Tue, Nov 27, 2012 at 9:25 AM, Alan Bateman wrote: > On 27/11/2012 15:20, Zhong Yu wrote: >> >> I wish MimeTypesFileTypeDetector is a public utility class, so that an >> applications can use it directly against a fixed "mime.types" file. >> >> Some applications couldn't care less about what OS thinks about file >> types, since the applications are supposed to be OS-independent. They >> need predictable and stable file type mapping. >> >> (Of course, they can always implement their own >> MimeTypesFileTypeDetectors) >> > Are you looking for javax.activiation.MimetypesFileTypeMap? Not exactly, that one still depends on factors outside app's control. I meant a file type mapping based on a specific mime.types file. > -Alan From Alan.Bateman at oracle.com Tue Nov 27 07:37:48 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 27 Nov 2012 15:37:48 +0000 Subject: Review Request: JDK-7142921,(fs) Files.probeContentType reports a MIME type of "text/plain" on Ubuntu 11.04 In-Reply-To: References: <50B30EC2.4060202@oracle.com> <50B372B4.3000401@oracle.com> <50B3C324.8070408@oracle.com> <50B40EFD.7020409@oracle.com> <50B4CF9C.5000800@oracle.com> <50B4DB5D.4080705@oracle.com> Message-ID: <50B4DE4C.308@oracle.com> On 27/11/2012 15:36, Zhong Yu wrote: > On Tue, Nov 27, 2012 at 9:25 AM, Alan Bateman wrote: >> On 27/11/2012 15:20, Zhong Yu wrote: >>> I wish MimeTypesFileTypeDetector is a public utility class, so that an >>> applications can use it directly against a fixed "mime.types" file. >>> >>> Some applications couldn't care less about what OS thinks about file >>> types, since the applications are supposed to be OS-independent. They >>> need predictable and stable file type mapping. >>> >>> (Of course, they can always implement their own >>> MimeTypesFileTypeDetectors) >>> >> Are you looking for javax.activiation.MimetypesFileTypeMap? > Not exactly, that one still depends on factors outside app's control. > I meant a file type mapping based on a specific mime.types file. > I haven't used JAF but doesn't it have a constructor that allows you to specify the mime.types files? -Alan From zhong.j.yu at gmail.com Tue Nov 27 08:08:39 2012 From: zhong.j.yu at gmail.com (Zhong Yu) Date: Tue, 27 Nov 2012 10:08:39 -0600 Subject: Review Request: JDK-7142921, (fs) Files.probeContentType reports a MIME type of "text/plain" on Ubuntu 11.04 In-Reply-To: <50B4DE4C.308@oracle.com> References: <50B30EC2.4060202@oracle.com> <50B372B4.3000401@oracle.com> <50B3C324.8070408@oracle.com> <50B40EFD.7020409@oracle.com> <50B4CF9C.5000800@oracle.com> <50B4DB5D.4080705@oracle.com> <50B4DE4C.308@oracle.com> Message-ID: On Tue, Nov 27, 2012 at 9:37 AM, Alan Bateman wrote: > On 27/11/2012 15:36, Zhong Yu wrote: >> >> On Tue, Nov 27, 2012 at 9:25 AM, Alan Bateman >> wrote: >>> >>> On 27/11/2012 15:20, Zhong Yu wrote: >>>> >>>> I wish MimeTypesFileTypeDetector is a public utility class, so that an >>>> applications can use it directly against a fixed "mime.types" file. >>>> >>>> Some applications couldn't care less about what OS thinks about file >>>> types, since the applications are supposed to be OS-independent. They >>>> need predictable and stable file type mapping. >>>> >>>> (Of course, they can always implement their own >>>> MimeTypesFileTypeDetectors) >>>> >>> Are you looking for javax.activiation.MimetypesFileTypeMap? >> >> Not exactly, that one still depends on factors outside app's control. >> I meant a file type mapping based on a specific mime.types file. >> > I haven't used JAF but doesn't it have a constructor that allows you to > specify the mime.types files? Yes, but the constructor calls the default constructor first, loading other mime.types files > -Alan From dan.xu at oracle.com Tue Nov 27 11:27:54 2012 From: dan.xu at oracle.com (Dan Xu) Date: Tue, 27 Nov 2012 11:27:54 -0800 Subject: Review Request: JDK-7142921,(fs) Files.probeContentType reports a MIME type of "text/plain" on Ubuntu 11.04 In-Reply-To: <50B4B47C.9080201@oracle.com> References: <50B30EC2.4060202@oracle.com> <50B372B4.3000401@oracle.com> <50B3C324.8070408@oracle.com> <50B40EFD.7020409@oracle.com> <50B4B47C.9080201@oracle.com> Message-ID: <50B5143A.3080802@oracle.com> Thanks for your comments. I have updated MagicFileTypeDetector. As you said, libmagic can handle non-existent file. But if we think the MIME type of non-existent file is null, MimeTypesFileTypeDetector will return wrong type because it only checks the file name extension. Shall I add the check to MimeTypesFileTypeDetector? -Dan On 11/27/2012 04:39 AM, Alan Bateman wrote: > On 27/11/2012 00:53, Dan Xu wrote: >> Hi, >> >> I have added a comment to UnixFileSystemProvider.chain and remove its >> "protected" keyword. The new webrev is at, >> http://cr.openjdk.java.net/~dxu/7142921/webrev.01/. Thanks! >> >> -Dan > I've looked at the libmagic based FileTypeDetector and it looks good. > From the man page, it looks like magic_file will return NULL if it > can't open the file, in which case you might not need to test if the > file exists. A couple of minor comments: > > - the function prototype for magic_open uses "flag" but "flags" would > be more correct. > > - in initialize0 you can move the second magic_handle == NULL into the > previous statement to avoid checking it twice. > > - in MagicFileTypeDetector then you shouldn't need to call super > explicitly. > > I still have on my list to review the mime.types FileTypeDetector, > hopefully find time soon. I'm sure others can help review that one too. > > -Alan. From Alan.Bateman at oracle.com Tue Nov 27 11:31:08 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 27 Nov 2012 19:31:08 +0000 Subject: Review Request: JDK-7142921,(fs) Files.probeContentType reports a MIME type of "text/plain" on Ubuntu 11.04 In-Reply-To: References: <50B30EC2.4060202@oracle.com> <50B372B4.3000401@oracle.com> <50B3C324.8070408@oracle.com> <50B40EFD.7020409@oracle.com> <50B4CF9C.5000800@oracle.com> <50B4DB5D.4080705@oracle.com> <50B4DE4C.308@oracle.com> Message-ID: <50B514FC.7060208@oracle.com> On 27/11/2012 16:08, Zhong Yu wrote: > : > Yes, but the constructor calls the default constructor first, loading > other mime.types files > In any case, if you have your own mime.types file then just create a FileTypeDetector for it, package it as a service provider as per the javadoc, and Files.probeContentType will use it. -Alan From Alan.Bateman at oracle.com Tue Nov 27 11:39:06 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 27 Nov 2012 19:39:06 +0000 Subject: Review Request: JDK-7142921,(fs) Files.probeContentType reports a MIME type of "text/plain" on Ubuntu 11.04 In-Reply-To: <50B5143A.3080802@oracle.com> References: <50B30EC2.4060202@oracle.com> <50B372B4.3000401@oracle.com> <50B3C324.8070408@oracle.com> <50B40EFD.7020409@oracle.com> <50B4B47C.9080201@oracle.com> <50B5143A.3080802@oracle.com> Message-ID: <50B516DA.6070003@oracle.com> On 27/11/2012 19:27, Dan Xu wrote: > Thanks for your comments. I have updated MagicFileTypeDetector. As you > said, libmagic can handle non-existent file. But if we think the MIME > type of non-existent file is null, MimeTypesFileTypeDetector will > return wrong type because it only checks the file name extension. > Shall I add the check to MimeTypesFileTypeDetector? > > -Dan MagicFileTypeDetector can't detect the file type when the file doesn't exist so returning null is fine. MimeTypesFileTypeDetector is just based off the file extension so it just needs a file name. MimeTypesFileTypeDetector shouldn't need to check if the file exists or not. -Alan From dan.xu at oracle.com Tue Nov 27 12:14:21 2012 From: dan.xu at oracle.com (Dan Xu) Date: Tue, 27 Nov 2012 12:14:21 -0800 Subject: Review Request: JDK-7142921,(fs) Files.probeContentType reports a MIME type of "text/plain" on Ubuntu 11.04 In-Reply-To: <50B516DA.6070003@oracle.com> References: <50B30EC2.4060202@oracle.com> <50B372B4.3000401@oracle.com> <50B3C324.8070408@oracle.com> <50B40EFD.7020409@oracle.com> <50B4B47C.9080201@oracle.com> <50B5143A.3080802@oracle.com> <50B516DA.6070003@oracle.com> Message-ID: <50B51F1D.90502@oracle.com> I see. Different detectors hold different principles to detect file types. Gnome detector and Magic detector will open the file and decide the type based on the file content. But MimeTypes detector will only use the file extension. -Dan On Tue 27 Nov 2012 11:39:06 AM PST, Alan Bateman wrote: > On 27/11/2012 19:27, Dan Xu wrote: >> Thanks for your comments. I have updated MagicFileTypeDetector. As >> you said, libmagic can handle non-existent file. But if we think the >> MIME type of non-existent file is null, MimeTypesFileTypeDetector >> will return wrong type because it only checks the file name >> extension. Shall I add the check to MimeTypesFileTypeDetector? >> >> -Dan > MagicFileTypeDetector can't detect the file type when the file doesn't > exist so returning null is fine. MimeTypesFileTypeDetector is just > based off the file extension so it just needs a file name. > MimeTypesFileTypeDetector shouldn't need to check if the file exists > or not. > > -Alan From dan.xu at oracle.com Tue Nov 27 17:26:34 2012 From: dan.xu at oracle.com (Dan Xu) Date: Tue, 27 Nov 2012 17:26:34 -0800 Subject: Review Request: JDK-7142921,(fs) Files.probeContentType reports a MIME type of "text/plain" on Ubuntu 11.04 In-Reply-To: <50B4CF9C.5000800@oracle.com> References: <50B30EC2.4060202@oracle.com> <50B372B4.3000401@oracle.com> <50B3C324.8070408@oracle.com> <50B40EFD.7020409@oracle.com> <50B4CF9C.5000800@oracle.com> Message-ID: <50B5684A.1030309@oracle.com> Alan, Thanks for your comments on MimeTypesFileTypeDetector. One concern about not using META-INF/mimetypes.default in resources.jar is that we may not be able to support Mac platform. As Mike mentioned yesterday, the system mime types file, /private/etc/apache2/mime.types, was not present in Mountain Lion (10.8). Then, we have to use the fallback mime types file. And the .mime.types in user's home directory is not a sure thing. I am not sure whether we can rely on that. Thanks! -Dan On 11/27/2012 06:35 AM, Alan Bateman wrote: > On 27/11/2012 00:53, Dan Xu wrote: >> Hi, >> >> I have added a comment to UnixFileSystemProvider.chain and remove its >> "protected" keyword. The new webrev is at, >> http://cr.openjdk.java.net/~dxu/7142921/webrev.01/. Thanks! >> >> -Dan >> > I found time to do a very quick pass over MimeTypesFileTypeDetector.java. > > In terms of diagnosability then I think it would be hard for > developers to understand how the file type is obtained when there are > so many ways to do. To that end it's probably best if > MimeTypesFileTypeDetector didn't uses META-INF/mimetypes.default, that > resource file is included in rt.jar because it is used by the Java > Activation Framework, itself used by JAX-WS and is essentially > invisible to users. > > On ~/.mime.types then we have choice, use it as a fallback in > MimeTypesFileTypeDetector so let the provider create two > MimeTypesFileTypeDetector instances, one for the system mime.types > files, one for the user file. I don't have a preference but the latter > would make the FileTypeDetector a bit simpler. > > In the implProbeContentType method it only support UnixPath but this > FileTypeDetector should be independent of the Path type. > > It looks like it tries a case-insensitive search after a > case-sensitive search but file extensions are case sensitive on > Solaris and Linux, I realize this will need special handing on Mac. > > loadFile/parse are a bit messy in that try-with-resources is being > combined with an explicit call to close. Given that the mime.types > file is likely to be quite small then a possible choice is > Files.readAllLines and that would save you some of this. > > That's all I have for now. > > -Alan > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20121127/7d48d93e/attachment.html From Alan.Bateman at oracle.com Wed Nov 28 03:27:20 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 28 Nov 2012 11:27:20 +0000 Subject: Review Request: JDK-7142921,(fs) Files.probeContentType reports a MIME type of "text/plain" on Ubuntu 11.04 In-Reply-To: <50B5684A.1030309@oracle.com> References: <50B30EC2.4060202@oracle.com> <50B372B4.3000401@oracle.com> <50B3C324.8070408@oracle.com> <50B40EFD.7020409@oracle.com> <50B4CF9C.5000800@oracle.com> <50B5684A.1030309@oracle.com> Message-ID: <50B5F518.3080509@oracle.com> On 28/11/2012 01:26, Dan Xu wrote: > Alan, > > Thanks for your comments on MimeTypesFileTypeDetector. > > One concern about not using META-INF/mimetypes.default in > resources.jar is that we may not be able to support Mac platform. As > Mike mentioned yesterday, the system mime types file, > /private/etc/apache2/mime.types, was not present in Mountain Lion > (10.8). Then, we have to use the fallback mime types file. And the > .mime.types in user's home directory is not a sure thing. I am not > sure whether we can rely on that. Thanks! > > -Dan > I think the Mac needs a bit more investigation to determine if /private/etc/apache2/mime.types is really the right source on the system (I suspect not). A starting point to just support ${user.home}/.mime.types might be okay as at least that it editable by developers to get the right mapping. I understand the desire to use META-INF/mimetypes.default but I'm just concerned that it is invisible and not configurable by users. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20121128/176ca7d2/attachment.html From dan.xu at oracle.com Wed Nov 28 23:45:00 2012 From: dan.xu at oracle.com (Dan Xu) Date: Wed, 28 Nov 2012 23:45:00 -0800 Subject: Review Request: JDK-7142921,(fs) Files.probeContentType reports a MIME type of "text/plain" on Ubuntu 11.04 In-Reply-To: <50B5F518.3080509@oracle.com> References: <50B30EC2.4060202@oracle.com> <50B372B4.3000401@oracle.com> <50B3C324.8070408@oracle.com> <50B40EFD.7020409@oracle.com> <50B4CF9C.5000800@oracle.com> <50B5684A.1030309@oracle.com> <50B5F518.3080509@oracle.com> Message-ID: <50B7127C.808@oracle.com> I have updated my fix accordingly and uploaded it to webrev at http://cr.openjdk.java.net/~dxu/7142921/webrev.02/. Please take a review. Thanks! -Dan On Wed 28 Nov 2012 03:27:20 AM PST, Alan Bateman wrote: > On 28/11/2012 01:26, Dan Xu wrote: >> Alan, >> >> Thanks for your comments on MimeTypesFileTypeDetector. >> >> One concern about not using META-INF/mimetypes.default in >> resources.jar is that we may not be able to support Mac platform. As >> Mike mentioned yesterday, the system mime types file, >> /private/etc/apache2/mime.types, was not present in Mountain Lion >> (10.8). Then, we have to use the fallback mime types file. And the >> .mime.types in user's home directory is not a sure thing. I am not >> sure whether we can rely on that. Thanks! >> >> -Dan >> > I think the Mac needs a bit more investigation to determine if > /private/etc/apache2/mime.types is really the right source on the > system (I suspect not). A starting point to just support > ${user.home}/.mime.types might be okay as at least that it editable by > developers to get the right mapping. I understand the desire to use > META-INF/mimetypes.default but I'm just concerned that it is invisible > and not configurable by users. > > -Alan > > > From Alan.Bateman at oracle.com Thu Nov 29 09:54:54 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 29 Nov 2012 17:54:54 +0000 Subject: Review Request: JDK-7142921,(fs) Files.probeContentType reports a MIME type of "text/plain" on Ubuntu 11.04 In-Reply-To: <50B7127C.808@oracle.com> References: <50B30EC2.4060202@oracle.com> <50B372B4.3000401@oracle.com> <50B3C324.8070408@oracle.com> <50B40EFD.7020409@oracle.com> <50B4CF9C.5000800@oracle.com> <50B5684A.1030309@oracle.com> <50B5F518.3080509@oracle.com> <50B7127C.808@oracle.com> Message-ID: <50B7A16E.3070900@oracle.com> On 29/11/2012 07:45, Dan Xu wrote: > I have updated my fix accordingly and uploaded it to webrev at > http://cr.openjdk.java.net/~dxu/7142921/webrev.02/. Please take a > review. Thanks! > > -Dan Thanks for the update. I see you've removed the protected constructors from Abstract* and the public constructors from *FileSystemProvider, I assume you didn't mean to do that. In the getFileSystemDetector methods then I assume that System.getProperty("user.name") needs to be done in a privileged block, otherwise Files.probeContentType will not work with a security manager. I think the MagicFileTypeDetector looks fine. MimeTypesFileTypeDetector looks better now that it encapsulates just one mime.types files. I think it needs a bit of clean-up though, the main issues seems to be that loadMimeTypes is not synchronized and there are several potential problems if implProbeContentType is invoked by more than one thread at around the same time. Also point is that the previous mails I thought that testing the extension would be case sensitive only on Linux and Solaris. So overall I think we are a step closer on this. -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20121129/51e9161d/attachment.html From Alan.Bateman at oracle.com Fri Nov 30 05:40:22 2012 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 30 Nov 2012 13:40:22 +0000 Subject: 7165762: (aio) Default thread pool should be configured so that threads terminated after a timeout period Message-ID: <50B8B746.40109@oracle.com> This is something that we discussed here back in April [1]. To re-cap, the thread pool for the default channel group is a cached thread pool where the threads don't terminate after an idle time. The reason for that is because of support for Windows editions that didn't support thread agnostic I/O (so we have to keep the threads alive, otherwise any I/O operations that they initiated would abort when the thread terminated). It was an oversight that this went into shared code to be used for all platforms (including newer version of Windows where this is not a concern). As the older editions of Windows are dying out (Windows Server 2003 is now in extended support, long superseded by Windows Server 2008 and newer) then it's probably not worth refactoring this now and that the simplest thing is to just change it to use Executors.newCachedThreadPool so that all platforms get a thread pool where the threads to terminate after an idle time. The patch with the trivial change to do this is attached. I've decided not to include a test case, mostly because it would take a long time (>1min) and also would be a bit fragile when with residual thread left behind by tests that were run previously in the same VM -- this is an issue that we've had in other areas where the thread count changes for reasons attributable to other tests. Thanks, Alan. diff --git a/src/share/classes/sun/nio/ch/ThreadPool.java b/src/share/classes/sun/nio/ch/ThreadPool.java --- a/src/share/classes/sun/nio/ch/ThreadPool.java +++ b/src/share/classes/sun/nio/ch/ThreadPool.java @@ -102,11 +102,7 @@ public class ThreadPool { if (threadFactory == null) threadFactory = defaultThreadFactory; // create thread pool - ExecutorService executor = - new ThreadPoolExecutor(0, Integer.MAX_VALUE, - Long.MAX_VALUE, TimeUnit.MILLISECONDS, - new SynchronousQueue(), - threadFactory); + ExecutorService executor = Executors.newCachedThreadPool(threadFactory); return new ThreadPool(executor, false, initialSize); } [1] http://mail.openjdk.java.net/pipermail/nio-dev/2012-April/001629.html From chris.hegarty at oracle.com Fri Nov 30 07:47:11 2012 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Fri, 30 Nov 2012 15:47:11 +0000 Subject: 7165762: (aio) Default thread pool should be configured so that threads terminated after a timeout period In-Reply-To: <50B8B746.40109@oracle.com> References: <50B8B746.40109@oracle.com> Message-ID: <50B8D4FF.4040500@oracle.com> Given your detailed description, it makes reviewing the trivial patch easy. Looks fine to me. -Chris. On 30/11/2012 13:40, Alan Bateman wrote: > > This is something that we discussed here back in April [1]. > > To re-cap, the thread pool for the default channel group is a cached > thread pool where the threads don't terminate after an idle time. The > reason for that is because of support for Windows editions that didn't > support thread agnostic I/O (so we have to keep the threads alive, > otherwise any I/O operations that they initiated would abort when the > thread terminated). It was an oversight that this went into shared code > to be used for all platforms (including newer version of Windows where > this is not a concern). > > As the older editions of Windows are dying out (Windows Server 2003 is > now in extended support, long superseded by Windows Server 2008 and > newer) then it's probably not worth refactoring this now and that the > simplest thing is to just change it to use Executors.newCachedThreadPool > so that all platforms get a thread pool where the threads to terminate > after an idle time. The patch with the trivial change to do this is > attached. > > I've decided not to include a test case, mostly because it would take a > long time (>1min) and also would be a bit fragile when with residual > thread left behind by tests that were run previously in the same VM -- > this is an issue that we've had in other areas where the thread count > changes for reasons attributable to other tests. > > Thanks, > Alan. > > diff --git a/src/share/classes/sun/nio/ch/ThreadPool.java > b/src/share/classes/sun/nio/ch/ThreadPool.java > --- a/src/share/classes/sun/nio/ch/ThreadPool.java > +++ b/src/share/classes/sun/nio/ch/ThreadPool.java > @@ -102,11 +102,7 @@ public class ThreadPool { > if (threadFactory == null) > threadFactory = defaultThreadFactory; > // create thread pool > - ExecutorService executor = > - new ThreadPoolExecutor(0, Integer.MAX_VALUE, > - Long.MAX_VALUE, TimeUnit.MILLISECONDS, > - new SynchronousQueue(), > - threadFactory); > + ExecutorService executor = Executors.newCachedThreadPool(threadFactory); > return new ThreadPool(executor, false, initialSize); > } > > > [1] http://mail.openjdk.java.net/pipermail/nio-dev/2012-April/001629.html > > > From dan.xu at oracle.com Fri Nov 30 08:28:56 2012 From: dan.xu at oracle.com (Dan Xu) Date: Fri, 30 Nov 2012 08:28:56 -0800 Subject: Review Request: JDK-7142921,(fs) Files.probeContentType reports a MIME type of "text/plain" on Ubuntu 11.04 In-Reply-To: <50B7A16E.3070900@oracle.com> References: <50B30EC2.4060202@oracle.com> <50B372B4.3000401@oracle.com> <50B3C324.8070408@oracle.com> <50B40EFD.7020409@oracle.com> <50B4CF9C.5000800@oracle.com> <50B5684A.1030309@oracle.com> <50B5F518.3080509@oracle.com> <50B7127C.808@oracle.com> <50B7A16E.3070900@oracle.com> Message-ID: <50B8DEC8.7040001@oracle.com> Thanks for your comments. Here is the updated patch, http://cr.openjdk.java.net/~dxu/7142921/webrev.03/ . Please review it. -Dan On 11/29/2012 09:54 AM, Alan Bateman wrote: > On 29/11/2012 07:45, Dan Xu wrote: >> I have updated my fix accordingly and uploaded it to webrev at >> http://cr.openjdk.java.net/~dxu/7142921/webrev.02/. Please take a >> review. Thanks! >> >> -Dan > Thanks for the update. > > I see you've removed the protected constructors from Abstract* and the > public constructors from *FileSystemProvider, I assume you didn't mean > to do that. > > In the getFileSystemDetector methods then I assume that > System.getProperty("user.name") needs to be done in a privileged > block, otherwise Files.probeContentType will not work with a security > manager. > > I think the MagicFileTypeDetector looks fine. > > MimeTypesFileTypeDetector looks better now that it encapsulates just > one mime.types files. I think it needs a bit of clean-up though, the > main issues seems to be that loadMimeTypes is not synchronized and > there are several potential problems if implProbeContentType is > invoked by more than one thread at around the same time. Also point is > that the previous mails I thought that testing the extension would be > case sensitive only on Linux and Solaris. > > So overall I think we are a step closer on this. > > -Alan. > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20121130/38ce02fe/attachment.html