From Alan.Bateman at oracle.com Fri Jun 3 09:54:08 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 03 Jun 2011 17:54:08 +0100 Subject: 7050358: (fs spec) Path.toUri doesn't allow custom providers to use opaque URIs. Message-ID: <4DE911B0.8080500@oracle.com> This is a spec bug that came up the nio-discuss list in the last week. Path's toUri method currently specifies that the method returns a hierarchical URI but this prevents custom providers from using opaque URIs. It should have specified that the URI is absolute. The demo zip provider that we have in jdk7 falls foul of this because it uses the legacy JAR URL syntax which are opaque. The webrev with the change to fix this is here: http://cr.openjdk.java.net/~alanb/7050358/webrev/ I have a reviewer, and I hope to get this into jdk7 soon. -Alan From Alan.Bateman at oracle.com Wed Jun 8 05:35:11 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 08 Jun 2011 13:35:11 +0100 Subject: 7052272: (aio) Definition of ConnectEx doesn't specify PASCAL calling convention Message-ID: <4DEF6C7F.5030808@oracle.com> The asynchronous I/O implementation on Windows defines the ConnectEx extension without specifying the calling convention and so doesn't match the actual calling convention. This creates potential issues although we haven't had any reports. Easily fixed with the attached patch. This one is for jdk8 (too late for jdk7). -Alan diff --git a/src/windows/native/sun/nio/ch/WindowsAsynchronousSocketChannelImpl.c b/src/windows/native/sun/nio/ch/WindowsAsynchronousSocketChannelImpl.c --- a/src/windows/native/sun/nio/ch/WindowsAsynchronousSocketChannelImpl.c +++ b/src/windows/native/sun/nio/ch/WindowsAsynchronousSocketChannelImpl.c @@ -44,7 +44,7 @@ #define SO_UPDATE_CONNECT_CONTEXT 0x7010 #endif -typedef BOOL (*ConnectEx_t) +typedef BOOL (PASCAL *ConnectEx_t) ( SOCKET s, const struct sockaddr* name, From martijnverburg at gmail.com Wed Jun 8 05:38:34 2011 From: martijnverburg at gmail.com (Martijn Verburg) Date: Wed, 8 Jun 2011 13:38:34 +0100 Subject: 7052272: (aio) Definition of ConnectEx doesn't specify PASCAL calling convention In-Reply-To: <4DEF6C7F.5030808@oracle.com> References: <4DEF6C7F.5030808@oracle.com> Message-ID: Hi Alan, Is this sort of change back portable to a 7 point release? Cheers, Martijn On Wednesday, 8 June 2011, Alan Bateman wrote: > > The asynchronous I/O implementation on Windows defines the ConnectEx extension without specifying the calling convention and so doesn't match the actual calling convention. This creates potential issues although we haven't had any reports. Easily fixed with the attached patch. This one is for jdk8 (too late for jdk7). > > -Alan > > > diff --git a/src/windows/native/sun/nio/ch/WindowsAsynchronousSocketChannelImpl.c b/src/windows/native/sun/nio/ch/WindowsAsynchronousSocketChannelImpl.c > --- a/src/windows/native/sun/nio/ch/WindowsAsynchronousSocketChannelImpl.c > +++ b/src/windows/native/sun/nio/ch/WindowsAsynchronousSocketChannelImpl.c > @@ -44,7 +44,7 @@ > #define SO_UPDATE_CONNECT_CONTEXT 0x7010 > #endif > > -typedef BOOL (*ConnectEx_t) > +typedef BOOL (PASCAL *ConnectEx_t) > ( > ? ?SOCKET s, > ? ?const struct sockaddr* name, > > From Alan.Bateman at oracle.com Wed Jun 8 05:50:13 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 08 Jun 2011 13:50:13 +0100 Subject: 7052272: (aio) Definition of ConnectEx doesn't specify PASCAL calling convention In-Reply-To: References: <4DEF6C7F.5030808@oracle.com> Message-ID: <4DEF7005.60806@oracle.com> Martijn Verburg wrote: > Hi Alan, > > Is this sort of change back portable to a 7 point release? > Yes it is, but there isn't a 7 update project yet. I'm sure it will be proposed soon but I don't know the who and when. -Alan From zhong.j.yu at gmail.com Fri Jun 10 16:41:48 2011 From: zhong.j.yu at gmail.com (Zhong Yu) Date: Fri, 10 Jun 2011 18:41:48 -0500 Subject: Differentiate synchronous and asynchronous error of an AIO operation Message-ID: Greetings NIO team, kudos for your hard work and excellent results! I have a problem. I was very surprised by the fact that a completion handler may be invoked directly by the initiating thread. This is quite strange, and it's not trivial to write a thread safe handler to handle it properly. I think it would be much better that, if an IO operation completes immediately, the operation throws an exception, instead of invoking the handler directly. I know it's very late and there's no much chance for API change, but I'll record my thoughts here for reference. When an AIO operation is issued at OS level, it may complete synchronously (due to immediate error), or it may complete asynchronously (error or success). The Java methods however don't syntactically differentiate the two cases. One handler is required to handle both possibilities. void operation( handler ) However there's a difference in execution: in the sync case, the handler may be invoked immediately inside operation() frame, before the method returns. This is confusing on many levels, let's examine how concurrency is affected. handler1 onEvent() ... operation( handler2 ) ... In any nontrivial application, handler1 and handler2 must share some mutable variables. Access to these variables must be properly synchronized, because Java AIO doesn't specify any stronger concurrency semantics(unlike e.g. Swing event model). Here's a seemingly innocent example: newConnection() { init(); read( readHandler ); } The read() is the last statement of the method. It would seem that synchronization is not needed - certainly the readHandler will be invoked after init(). But without synchronization, there is no "happens-before" guarantee - when readHandler is invoked, it may see none of the writes from init(). So there must be adequate synchronizations in any Java AIO program, which deserves more public awareness. That does not require much work though, usually just associate a lock with each channel. (For fully duplex applications more locks are needed.) Now, back to handler1 and handler2. Their methods are now protected by the lock. That's still not enough, because, surprise surprise, handler2 may be invoked within the frame of handler1. It's natural for handler1 to think that it owns the lock therefore the exclusive access to system state, overlooking the case that handler2 sneaks in the same frame, messing with the same mutables. handler2 may invoke handler3 and so on, in the same frame - or not. If we don't want to drive ourselves insane by tracking all the intermediate states, we must follow this principle: before calling any operation(), always bring system(wrt a channel) to a consistent state; after calling the operation(), we must abandon previous knowledge of the system state. So in principle we should always code in this pattern: lock .... // invariant must hold here operation( handler2 ) // invariant may be updated ... unlock That pattern is very unfamiliar and inconvenient. Ordinarily we only need to preserve invariant at the time of unlock, while inside lock-unlock block we can put system in an inconsistent state, comfortably knowing that others are not accessing it. So we don't want to following that coding pattern, it is too excessive. Actually we know the only time that handler2 can be invoked in handler1's frame is when operation() returns an immediate/synchronous error. So let's adopt a different and simpler principle - whenever handler2 is invoked for such reason, handler2 doesn't read/write any system state; instead it somehow notifies handler1 about the error, and let handler1 handle it. lock ... operation( handler2 ) if there_was_a_sync_error ... ... unlock This is simple, familiar, safe and correct. On top of the existing Java AIO API, we can implement wrapper methods that detect attempt of synchronous handler invocation for error, and throw that error to the caller. The handler will only be invoked for asynchronous completions, in a new frame. void w_operation( handler ) throws SynchronousError // usage lock ... try{ w_operation( handler ); }catch(SynchronousError e){ error = e.getCause()); ... } ... unlock Now, let's look back, and see how we got here. For all the reasons listed above, it's clear that we have to always use the w_operation() version. The original operation() is extremely difficult to use correctly. (Or maybe it's just me - if you know a safe and simple way to use operation() directly, do tell.) If I'm correct, then we should really change the signature of operation() to that of w_operation(). It appears to me that, the current design is out of some economic concern - why not abstract the two error cases into one kind, and let the one handler deal with it? But the abstraction leaks badly - you have no choice but to differentiate them by different modes of execution, and this difference must be known by the programmer, and the programmer then must translate this difference back to 2 different error cases, and finally de-abstract the abstraction. The economic motive turns out to be very costly for everybody! If we simply throw the synchronous error instead of calling the handler in the same frame, it'll drastically simplify both the JDK code and the client code. Most importantly, it's much easier to understand. - Zhong Yu From Alan.Bateman at oracle.com Sat Jun 11 01:27:51 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 11 Jun 2011 09:27:51 +0100 Subject: Differentiate synchronous and asynchronous error of an AIO operation In-Reply-To: References: Message-ID: <4DF32707.3060906@oracle.com> Zhong Yu wrote: > Greetings NIO team, kudos for your hard work and excellent results! > > I have a problem. I was very surprised by the fact that a completion > handler may be invoked directly by the initiating thread. This is > quite strange, and it's not trivial to write a thread safe handler to > handle it properly. I think it would be much better that, if an IO > operation completes immediately, the operation throws an exception, > instead of invoking the handler directly. > > I don't have time to reply to this now but just to say that I/O operations can also completely successfully on the current thread. It depends on the implementation of course but it's not just the error case. There is an implementation specific property (sun.nio.ch.maxCompletionHandlersOnStack) that may be useful as it can be used to limit the number of completion handler frames on the stack. Setting it to 1 will give you the behavior you are looking for. -Alan. From jdavis at pcprogramming.com Sat Jun 11 06:51:05 2011 From: jdavis at pcprogramming.com (John Davis) Date: Sat, 11 Jun 2011 08:51:05 -0500 Subject: Differentiate synchronous and asynchronous error of an AIO operation In-Reply-To: References: Message-ID: Changing this would affect performance considerably. The ideal case is to have the callback fire on the same thread that initiated, this eliminates context switch overhead and cache flushing. In fact, it is this same reason that projects like nodejs strongly outperform multithreaded servers which don't center on epoll or IOCP. The whole point is to keep everything running on the same core as much as possible. On Fri, Jun 10, 2011 at 6:41 PM, Zhong Yu wrote: > Greetings NIO team, kudos for your hard work and excellent results! > > I have a problem. I was very surprised by the fact that a completion > handler may be invoked directly by the initiating thread. This is > quite strange, and it's not trivial to write a thread safe handler to > handle it properly. I think it would be much better that, if an IO > operation completes immediately, the operation throws an exception, > instead of invoking the handler directly. > > I know it's very late and there's no much chance for API change, but > I'll record my thoughts here for reference. > > When an AIO operation is issued at OS level, it may complete > synchronously (due to immediate error), or it may complete > asynchronously (error or success). The Java methods however don't > syntactically differentiate the two cases. One handler is required to > handle both possibilities. > > void operation( handler ) > > However there's a difference in execution: in the sync case, the > handler may be invoked immediately inside operation() frame, before > the method returns. > > This is confusing on many levels, let's examine how concurrency is > affected. > > handler1 > onEvent() > ... > operation( handler2 ) > ... > > In any nontrivial application, handler1 and handler2 must share some > mutable variables. Access to these variables must be properly > synchronized, because Java AIO doesn't specify any stronger > concurrency semantics(unlike e.g. Swing event model). Here's a > seemingly innocent example: > > newConnection() { > init(); > read( readHandler ); > } > > The read() is the last statement of the method. It would seem that > synchronization is not needed - certainly the readHandler will be > invoked after init(). But without synchronization, there is no > "happens-before" guarantee - when readHandler is invoked, it may see > none of the writes from init(). > > So there must be adequate synchronizations in any Java AIO program, > which deserves more public awareness. That does not require much work > though, usually just associate a lock with each channel. (For fully > duplex applications more locks are needed.) > > Now, back to handler1 and handler2. Their methods are now protected by > the lock. That's still not enough, because, surprise surprise, > handler2 may be invoked within the frame of handler1. It's natural for > handler1 to think that it owns the lock therefore the exclusive access > to system state, overlooking the case that handler2 sneaks in the same > frame, messing with the same mutables. > > handler2 may invoke handler3 and so on, in the same frame - or not. If > we don't want to drive ourselves insane by tracking all the > intermediate states, we must follow this principle: before calling any > operation(), always bring system(wrt a channel) to a consistent state; > after calling the operation(), we must abandon previous knowledge of > the system state. So in principle we should always code in this > pattern: > > lock > .... > // invariant must hold here > operation( handler2 ) > // invariant may be updated > ... > unlock > > That pattern is very unfamiliar and inconvenient. Ordinarily we only > need to preserve invariant at the time of unlock, while inside > lock-unlock block we can put system in an inconsistent state, > comfortably knowing that others are not accessing it. > > So we don't want to following that coding pattern, it is too > excessive. Actually we know the only time that handler2 can be invoked > in handler1's frame is when operation() returns an > immediate/synchronous error. So let's adopt a different and simpler > principle - whenever handler2 is invoked for such reason, handler2 > doesn't read/write any system state; instead it somehow notifies > handler1 about the error, and let handler1 handle it. > > lock > ... > operation( handler2 ) > if there_was_a_sync_error > ... > ... > unlock > > This is simple, familiar, safe and correct. > > On top of the existing Java AIO API, we can implement wrapper methods > that detect attempt of synchronous handler invocation for error, and > throw that error to the caller. The handler will only be invoked for > asynchronous completions, in a new frame. > > void w_operation( handler ) throws SynchronousError > > // usage > lock > ... > try{ > w_operation( handler ); > }catch(SynchronousError e){ > error = e.getCause()); > ... > } > ... > unlock > > Now, let's look back, and see how we got here. For all the reasons > listed above, it's clear that we have to always use the w_operation() > version. The original operation() is extremely difficult to use > correctly. (Or maybe it's just me - if you know a safe and simple way > to use operation() directly, do tell.) > > If I'm correct, then we should really change the signature of > operation() to that of w_operation(). > > It appears to me that, the current design is out of some economic > concern - why not abstract the two error cases into one kind, and let > the one handler deal with it? But the abstraction leaks badly - you > have no choice but to differentiate them by different modes of > execution, and this difference must be known by the programmer, and > the programmer then must translate this difference back to 2 different > error cases, and finally de-abstract the abstraction. The economic > motive turns out to be very costly for everybody! > > If we simply throw the synchronous error instead of calling the > handler in the same frame, it'll drastically simplify both the JDK > code and the client code. Most importantly, it's much easier to > understand. > > - Zhong Yu > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20110611/9bbd9712/attachment.html From zhong.j.yu at gmail.com Sat Jun 11 09:20:52 2011 From: zhong.j.yu at gmail.com (Zhong Yu) Date: Sat, 11 Jun 2011 11:20:52 -0500 Subject: Differentiate synchronous and asynchronous error of an AIO operation In-Reply-To: <4DF32707.3060906@oracle.com> References: <4DF32707.3060906@oracle.com> Message-ID: Alan and John, I do not want to turn a synchronous event into an asynchronous one. I still want the event to be processed promptly by the same thread - but by exception/return, not by calling the completion handler. I'm with you on the idea that it's bad to dispatch a synchronous event to a new thread. However my reason is not performance, but semantics. If we do that, there's no way to know whether an event was sync or async. Maybe that doesn't matter to most developers, but for those who do care, we are depriving them of this information. Setting maxCompletionHandlersOnStack=1 will turn all sync events into async, that is not the behavior I'm looking for. I want to keep sync events sync. Actually, since maxCompletionHandlersOnStack has to be a small number, It won't be rare that the max is reached, and the next sync event becomes async. For example, a channel has 64K bytes ready for immediate read; the memory conscious application uses a 1K buffer to drain it. The first 16 reads occur on the same thread, with nested frames. (There will be 16N call stacks, and N is not small in typical Java apps) The 17th read however begin on a new thread. If instead the sync event is returned to the caller, there won't be nested calls; the caller will have an iteration of arbitrary length, on the same thread and with zero increase of call stack. The API could look like this Integer read(buffer, handler) throws SyncError or void read(buffer, syncHandler, asyncHandler) If we keep the current API, it would be nice to be provided a utility method to determine whether the event is sync or async void read(buffer, new CompletionHandler(){ completed() if( eventIsSync() ) S1 else S2 failed() if( eventIsSync() ) S3 else S4 Finally, in real world applications, does it really matter that an event is sync or async? What would real code look like? I admit my previous concurrency arguments are too abstract and may be much more general than reality. I suspect that an AIO operation is usually the last call lock ... operation( handler ); // nothing in between unlock If this pattern is consistently followed by a developer, the handlers don't have to worry about sync/async, all event processing executions are serialized, and the order is what it appears to be in source code. We'll have to see some real code examples to judge. I'll port a selector based HTTP server to AIO, and see how the experience is like. Zhong Yu From Alan.Bateman at oracle.com Sat Jun 11 12:13:55 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 11 Jun 2011 20:13:55 +0100 Subject: Differentiate synchronous and asynchronous error of an AIO operation In-Reply-To: References: <4DF32707.3060906@oracle.com> Message-ID: <4DF3BE73.2040805@oracle.com> Zhong Yu wrote: > I suspect that an AIO operation is usually the > last call > > lock > ... > operation( handler ); > // nothing in between > unlock > > Yes, I expect we will see this pattern quite often and probably with handler replaced by "this", meaning the same completion handler instance will be invoked to deal with the result. -Alan From cowwoc at bbs.darktech.org Mon Jun 13 12:51:11 2011 From: cowwoc at bbs.darktech.org (cowwoc) Date: Mon, 13 Jun 2011 12:51:11 -0700 (PDT) Subject: AsynchronousSocketChannel still throws unspecified exception Message-ID: <1307994671919-6471557.post@n2.nabble.com> Hi Alan, Remember in http://web.archiveorange.com/archive/v/gpYtBJmUMNrbuF39jKmu we discussed that read() should throw a specific exception (IllegalStateException) instead of "an unspecified runtime exception"? You mentioned you would resolve this but it doesn't seem to have happened in the past couple of months. Now, with the JDK7 API being officially "frozen" I'm worried this will never make it in. Is this change going in? Gili -- View this message in context: http://nio-dev.3157472.n2.nabble.com/AsynchronousSocketChannel-still-throws-unspecified-exception-tp6471557p6471557.html Sent from the nio-dev mailing list archive at Nabble.com. From Alan.Bateman at oracle.com Mon Jun 13 13:27:27 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 13 Jun 2011 21:27:27 +0100 Subject: AsynchronousSocketChannel still throws unspecified exception In-Reply-To: <1307994671919-6471557.post@n2.nabble.com> References: <1307994671919-6471557.post@n2.nabble.com> Message-ID: <4DF672AF.30809@oracle.com> cowwoc wrote: > Hi Alan, > > Remember in http://web.archiveorange.com/archive/v/gpYtBJmUMNrbuF39jKmu we > discussed that read() should throw a specific exception > (IllegalStateException) instead of "an unspecified runtime exception"? You > mentioned you would resolve this but it doesn't seem to have happened in the > past couple of months. Now, with the JDK7 API being officially "frozen" I'm > worried this will never make it in. > > Is this change going in? > Ah boo, I did say we should change this to an IllegalStateException but it didn't happen. My fault. As it's not a critical/showstopper issue then there is little chance that we have fix this now. This means we'll have to re-examine it in 8. It helps that the implementation already throws IllegalStateException for this case and so compatibility risk is confined to other AsynchronousChannel implementations (likely to be few). As I recall you ran into this with a serial port channel where data loss was acceptable or you could guarantee that no data would be loss, in which case you may already have specified your channel differently. -Alan. From cowwoc at bbs.darktech.org Mon Jun 13 14:00:15 2011 From: cowwoc at bbs.darktech.org (cowwoc) Date: Mon, 13 Jun 2011 14:00:15 -0700 (PDT) Subject: AsynchronousSocketChannel still throws unspecified exception In-Reply-To: <4DF672AF.30809@oracle.com> References: <1307994671919-6471557.post@n2.nabble.com> <4DF672AF.30809@oracle.com> Message-ID: <1307998815455-6471787.post@n2.nabble.com> Alan Bateman-2 wrote: > > cowwoc wrote: >> Hi Alan, >> >> Remember in http://web.archiveorange.com/archive/v/gpYtBJmUMNrbuF39jKmu >> we >> discussed that read() should throw a specific exception >> (IllegalStateException) instead of "an unspecified runtime exception"? >> You >> mentioned you would resolve this but it doesn't seem to have happened in >> the >> past couple of months. Now, with the JDK7 API being officially "frozen" >> I'm >> worried this will never make it in. >> >> Is this change going in? >> > Ah boo, I did say we should change this to an IllegalStateException but > it didn't happen. My fault. As it's not a critical/showstopper issue > then there is little chance that we have fix this now. This means we'll > have to re-examine it in 8. It helps that the implementation already > throws IllegalStateException for this case and so compatibility risk is > confined to other AsynchronousChannel implementations (likely to be > few). As I recall you ran into this with a serial port channel where > data loss was acceptable or you could guarantee that no data would be > loss, in which case you may already have specified your channel > differently. > > -Alan. > Can you please provide me with a BugParade number so I can track when this gets fixed? :) Thanks, Gili -- View this message in context: http://nio-dev.3157472.n2.nabble.com/AsynchronousSocketChannel-still-throws-unspecified-exception-tp6471557p6471787.html Sent from the nio-dev mailing list archive at Nabble.com. From zhong.j.yu at gmail.com Mon Jun 13 14:44:53 2011 From: zhong.j.yu at gmail.com (Zhong Yu) Date: Mon, 13 Jun 2011 16:44:53 -0500 Subject: Differentiate synchronous and asynchronous error of an AIO operation In-Reply-To: References: Message-ID: On Sat, Jun 11, 2011 at 8:51 AM, John Davis wrote: > Changing this would affect performance considerably. ?The ideal case is to > have the callback fire on the same thread that initiated, this eliminates > context switch overhead and cache flushing. ?In fact, it is this same reason > that projects like nodejs strongly outperform multithreaded servers which > don't center on epoll or IOCP. ?The whole point is to keep everything > running on the same core as much as possible. With Selector API, it is possible to bind a channel to a thread, i.e. all actions related to the channel happen on the same thread. This is possible because threading is the sole responsibility of the user code. With AIO API, this seems impossible, even with the fact that user can provide a customized executor. If the executor is used only for completion handlers, we can write a thin executor that executes tasks immediately on the calling thread, giving the handlers the responsibility to switch to other threads. Handlers can arrange to always dispatch the same channel to the same thread. However the javadoc implies that the executor will also be used by AIO library for other tasks. Then it's impossible to the executor to do the above trick, for it cannot distinguish completion handlers from other tasks, and it cannot risk to execute any given task immediately. Would it be better, if "other" tasks are dispatched by a separate, internal executor, leaving the user provided executor solely for completion handlers? - Zhong Yu From cowwoc at bbs.darktech.org Mon Jun 13 21:21:43 2011 From: cowwoc at bbs.darktech.org (cowwoc) Date: Mon, 13 Jun 2011 21:21:43 -0700 (PDT) Subject: AsynchronousSocketChannel still throws unspecified exception In-Reply-To: <1307998815455-6471787.post@n2.nabble.com> References: <1307994671919-6471557.post@n2.nabble.com> <4DF672AF.30809@oracle.com> <1307998815455-6471787.post@n2.nabble.com> Message-ID: <1308025303088-6472724.post@n2.nabble.com> cowwoc wrote: > > > Alan Bateman-2 wrote: >> >> cowwoc wrote: >>> Hi Alan, >>> >>> Remember in http://web.archiveorange.com/archive/v/gpYtBJmUMNrbuF39jKmu >>> we >>> discussed that read() should throw a specific exception >>> (IllegalStateException) instead of "an unspecified runtime exception"? >>> You >>> mentioned you would resolve this but it doesn't seem to have happened in >>> the >>> past couple of months. Now, with the JDK7 API being officially "frozen" >>> I'm >>> worried this will never make it in. >>> >>> Is this change going in? >>> >> Ah boo, I did say we should change this to an IllegalStateException but >> it didn't happen. My fault. As it's not a critical/showstopper issue >> then there is little chance that we have fix this now. This means we'll >> have to re-examine it in 8. It helps that the implementation already >> throws IllegalStateException for this case and so compatibility risk is >> confined to other AsynchronousChannel implementations (likely to be >> few). As I recall you ran into this with a serial port channel where >> data loss was acceptable or you could guarantee that no data would be >> loss, in which case you may already have specified your channel >> differently. >> >> -Alan. >> > > Can you please provide me with a BugParade number so I can track when this > gets fixed? :) > > Thanks, > Gili > Also, I think you should push very hard to get this fixed in JDK7. I'd be very surprised if Oracle lets you change this part of the specification in JDK8 because doing so would break backwards-compatibility for existing implementations. Better to fix it now before the final spec is published. Gili -- View this message in context: http://nio-dev.3157472.n2.nabble.com/AsynchronousSocketChannel-still-throws-unspecified-exception-tp6471557p6472724.html Sent from the nio-dev mailing list archive at Nabble.com. From Alan.Bateman at oracle.com Tue Jun 14 00:50:40 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 14 Jun 2011 08:50:40 +0100 Subject: Differentiate synchronous and asynchronous error of an AIO operation In-Reply-To: References: Message-ID: <4DF712D0.3010907@oracle.com> Zhong Yu wrote: > : > With Selector API, it is possible to bind a channel to a thread, i.e. > all actions related to the channel happen on the same thread. This is > possible because threading is the sole responsibility of the user > code. > > With AIO API, this seems impossible, even with the fact that user can > provide a customized executor. > > If the executor is used only for completion handlers, we can write a > thin executor that executes tasks immediately on the calling thread, > giving the handlers the responsibility to switch to other threads. > Handlers can arrange to always dispatch the same channel to the same > thread. > > However the javadoc implies that the executor will also be used by AIO > library for other tasks. Then it's impossible to the executor to do > the above trick, for it cannot distinguish completion handlers from > other tasks, and it cannot risk to execute any given task immediately. > > Would it be better, if "other" tasks are dispatched by a separate, > internal executor, leaving the user provided executor solely for > completion handlers? > > We have a page with some implementation details [1] that may be useful to this discussion. With a fixed thread pool then the tasks execute a loop that dequeue events and dispatch directly to the completion handler. These are long running tasks with no hand-off per I/O operation. When not using a fixed thread pool then it works a bit like what you are suggesting. A small pool of internal threads handles the I/O events then executes tasks that invoke the completion handler (so a task per I/O operation). The issue with calling the completion handler directly is that you will call it on an internal thread and so it won't have the expected identity. Also the size of this internal thread pool is only configurable via an implementation specific property. -Alan. [1] http://openjdk.java.net/projects/nio/resources/AsynchronousIo.html From Alan.Bateman at oracle.com Tue Jun 14 01:45:00 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 14 Jun 2011 09:45:00 +0100 Subject: AsynchronousSocketChannel still throws unspecified exception In-Reply-To: <1308025303088-6472724.post@n2.nabble.com> References: <1307994671919-6471557.post@n2.nabble.com> <4DF672AF.30809@oracle.com> <1307998815455-6471787.post@n2.nabble.com> <1308025303088-6472724.post@n2.nabble.com> Message-ID: <4DF71F8C.7040800@oracle.com> cowwoc wrote: > : > Also, I think you should push very hard to get this fixed in JDK7. I'd be > very surprised if Oracle lets you change this part of the specification in > JDK8 because doing so would break backwards-compatibility for existing > implementations. Better to fix it now before the final spec is published. > > I've created this bug to track it: 7054403: (aio spec) IllegalStateExcepiton should be thrown if I/O operation attempted after timeout It's unfortunate that this got forgotten but I think we can fix this early in jdk8. It helps that the current implementation already throws IllegalStateException. Furthermore, AsynchronousSocketChannel cannot be implemented in isolation as it is tied to an AsynchronousChannelProvider. This means the number of implementations will likely be very few. Again apologies about this, it was my fault that this got forgotten. -Alan. From zhong.j.yu at gmail.com Tue Jun 14 08:05:15 2011 From: zhong.j.yu at gmail.com (Zhong Yu) Date: Tue, 14 Jun 2011 10:05:15 -0500 Subject: Differentiate synchronous and asynchronous error of an AIO operation In-Reply-To: <4DF712D0.3010907@oracle.com> References: <4DF712D0.3010907@oracle.com> Message-ID: On Tue, Jun 14, 2011 at 2:50 AM, Alan Bateman wrote: > We have a page with some implementation details [1] that may be useful to > this discussion. With a fixed thread pool then the tasks execute a loop that > dequeue events and dispatch directly to the completion handler. ?These are > long running tasks with no hand-off per I/O operation. When not using a > fixed thread pool then it works a bit like what you are suggesting. A small > pool of internal threads handles the I/O events then executes tasks that > invoke the completion handler (so a task per I/O operation). The issue with That means the user provided executor service will only be used for completion handlers? That's great. However this is not guaranteed by the javadoc, which actually hints otherwise: http://download.java.net/jdk7/docs/api/java/nio/channels/AsynchronousChannelGroup.html "In addition to handling I/O events, the pooled threads may also execute other tasks required to support the execution of asynchronous I/O operations." Should the javadoc be clear in that the user provided executor service will *not* be used for such "other tasks"? > calling the completion handler directly is that you will call it on an > internal thread and so it won't have the expected identity. Also the size of The little trick I described requires that the executor and the handlers cooperate. The responsibility of switching thread is moved from the executor to the handlers, and the handlers must switch thread immediately. However since the handlers have more information about tasks, they can do smarter threading, e.g. dispatch all tasks of the same channel to the same thread. The executor doesn't have such knowledge about tasks, which all look the same the the executor. - Zhong Yu From Alan.Bateman at oracle.com Tue Jun 14 12:01:10 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 14 Jun 2011 20:01:10 +0100 Subject: Differentiate synchronous and asynchronous error of an AIO operation In-Reply-To: References: <4DF712D0.3010907@oracle.com> Message-ID: <4DF7AFF6.8040705@oracle.com> Zhong Yu wrote: > : > That means the user provided executor service will only be used for > completion handlers? That's great. However this is not guaranteed by > the javadoc It's not guaranteed by the javadoc as it has to allow for different implementations and configurations. As I mentioned, when a group is created with a fixed thread pool then the tasks are long running tasks that run a loop that dequeues events and dispatches to completion handlers. When created with other thread pools then the tasks are simple tasks that mostly just invoke the completion handler. There are a couple of exceptions though. For example on editions of Windows that don't support thread agnostic I/O then we have to initiate the I/O operations on pooled threads and so a task may be executed to initiate the I/O operation. I will guess that tasks like this will complicate your approach. -Alan. From zhong.j.yu at gmail.com Tue Jun 14 16:51:35 2011 From: zhong.j.yu at gmail.com (Zhong Yu) Date: Tue, 14 Jun 2011 18:51:35 -0500 Subject: Differentiate synchronous and asynchronous error of an AIO operation In-Reply-To: <4DF7AFF6.8040705@oracle.com> References: <4DF712D0.3010907@oracle.com> <4DF7AFF6.8040705@oracle.com> Message-ID: Thanks Alan, I think I have a better understanding of fixed thread pool configuration. If a handler is potentially blocking, it has to dispatch the real work to another thread, but that's not hard. Under this configuration it's easy to add flexible threading strategies. I still don't have a clear understanding of the official behavior of a group with user-supplied executor. Reading the implementation doesn't help - it got me more confused. Well I'll steer clear from them. And the default group too. On Tue, Jun 14, 2011 at 2:01 PM, Alan Bateman wrote: > For example on editions of Windows that don't support thread agnostic I/O > then we have to initiate the I/O operations on pooled threads and so a task > may be executed to initiate the I/O operation. Any reason there can't be a fixed internal thread for these I/O requests? - Zhong Yu From Alan.Bateman at oracle.com Wed Jun 15 05:08:59 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 15 Jun 2011 13:08:59 +0100 Subject: Differentiate synchronous and asynchronous error of an AIO operation In-Reply-To: References: <4DF712D0.3010907@oracle.com> <4DF7AFF6.8040705@oracle.com> Message-ID: <4DF8A0DB.80401@oracle.com> Zhong Yu wrote: > : > >> For example on editions of Windows that don't support thread agnostic I/O >> then we have to initiate the I/O operations on pooled threads and so a task >> may be executed to initiate the I/O operation. >> > > Any reason there can't be a fixed internal thread for these I/O requests? > For this specific case then it could be done with an internal thread (or a small pool). Something for the next time we are re-visiting that code. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20110615/d8799dc0/attachment.html From chris.hegarty at oracle.com Fri Jun 17 00:50:33 2011 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Fri, 17 Jun 2011 08:50:33 +0100 Subject: 7052272: (aio) Definition of ConnectEx doesn't specify PASCAL calling convention In-Reply-To: <4DEF6C7F.5030808@oracle.com> References: <4DEF6C7F.5030808@oracle.com> Message-ID: <4DFB0749.2050406@oracle.com> Look fine to me. -Chris. On 06/ 8/11 01:35 PM, Alan Bateman wrote: > > The asynchronous I/O implementation on Windows defines the ConnectEx > extension without specifying the calling convention and so doesn't match > the actual calling convention. This creates potential issues although we > haven't had any reports. Easily fixed with the attached patch. This one > is for jdk8 (too late for jdk7). > > -Alan > > > diff --git > a/src/windows/native/sun/nio/ch/WindowsAsynchronousSocketChannelImpl.c > b/src/windows/native/sun/nio/ch/WindowsAsynchronousSocketChannelImpl.c > --- a/src/windows/native/sun/nio/ch/WindowsAsynchronousSocketChannelImpl.c > +++ b/src/windows/native/sun/nio/ch/WindowsAsynchronousSocketChannelImpl.c > @@ -44,7 +44,7 @@ > #define SO_UPDATE_CONNECT_CONTEXT 0x7010 > #endif > > -typedef BOOL (*ConnectEx_t) > +typedef BOOL (PASCAL *ConnectEx_t) > ( > SOCKET s, > const struct sockaddr* name, > From Alan.Bateman at oracle.com Fri Jun 17 03:17:23 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 17 Jun 2011 11:17:23 +0100 Subject: 7052272: (aio) Definition of ConnectEx doesn't specify PASCAL calling convention In-Reply-To: <4DFB0749.2050406@oracle.com> References: <4DEF6C7F.5030808@oracle.com> <4DFB0749.2050406@oracle.com> Message-ID: <4DFB29B2.3000107@oracle.com> Chris Hegarty wrote: > Look fine to me. > > -Chris. Thanks Chris. As it happens the JCK folks have run into this so it looks like we might have to fix in 7 after all. All amusing to have a zero day bug show up at the last minute. -Alan From Alan.Bateman at oracle.com Mon Jun 20 09:41:06 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 20 Jun 2011 17:41:06 +0100 Subject: 6965150: TEST_BUG: java/nio/channels/AsynchronousSocketChannel/Basic.java takes too long Message-ID: <4DFF7822.8030102@oracle.com> I brought this up about a year ago [1]. This test takes about 4 mins, most of which is spent in a sub-test that attempts to connect to a host that probably does not exist. I'm looking at this test again as we need to speed it up. Here's the updated webrev with a change that adds the option -skipSlowConnectTest to the test. The @run tag is changed to pass this option so it skips the "slow connect" test during normal test runs: http://cr.openjdk.java.net/~alanb/6965150/webrev/ The alternative is to just get rid of this sub-test as originally suggested. This test predates try-with-resources so I've retrofitted to a few places. Not everywhere, and more could be done if anyone has cycles. -Alan. [1] http://mail.openjdk.java.net/pipermail/nio-dev/2010-June/001003.html From gfz87 at hotmail.com Tue Jun 21 15:01:41 2011 From: gfz87 at hotmail.com (Gian Franco Zabarino) Date: Tue, 21 Jun 2011 22:01:41 +0000 Subject: AsynchronousSocketChannel operation cancelling on b145 Message-ID: I've been testing this on b145, and after I cancel a read request by cancelling the Future, I can no longer use the AsynchronousSocketChannel for reading. On NIO I always had the opportunity to remove the key from the Selector, and then to register the SocketChannel with another selector... Is there anyway to do this using NIO.2? Sometimes I need to read for detecting if a Socket was disconnected, and sometimes I need to read for user input. Both reads have different CompletionHandler classes, and different timeout parameters. I've been looking around what the cancel(boolean) method does, and it seems to call a onCancel method on the AsynchronousChannel, which on both Windows and Unix implementations, cancels the operation for calling it again later, like I'm experimenting: @Override 167 public void onCancel(PendingFuture task) { 168 if (task.getContext() instanceof ConnectTask) 169 killConnect(); 170 if (task.getContext() instanceof ReadTask) 171 killReading(); 172 if (task.getContext() instanceof WriteTask) 173 killWriting(); 174 } Is there other way to accomplish what I'm trying to do? Greetings Gian Franco Zabarino -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20110621/d14e0688/attachment.html From Alan.Bateman at oracle.com Wed Jun 22 00:46:38 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 22 Jun 2011 08:46:38 +0100 Subject: AsynchronousSocketChannel operation cancelling on b145 In-Reply-To: References: Message-ID: <4E019DDE.502@oracle.com> Gian Franco Zabarino wrote: > I've been testing this on b145, and after I cancel a read request by > cancelling the Future, I can no longer use the > AsynchronousSocketChannel for reading. On NIO I always had the > opportunity to remove the key from the Selector, and then to register > the SocketChannel with another selector... Is there anyway to do this > using NIO.2? Sometimes I need to read for detecting if a Socket was > disconnected, and sometimes I need to read for user input. Both reads > have different CompletionHandler classes, and different timeout > parameters > If I understand correctly then you are looking to separate the "completed" and "failed" cases? Or maybe you mean if the connection reaches end-of-stream (the peer has closed the connection gracefully), or there is an I/O error (the peer has closed the connection abruptly for example), then you want one completion handler invoked. If bytes are read then you want a different completion handler invoked. There isn't any way in this API to separate these cases so it means you will need to specify a completion handler that handles both cases or else specify a completion handler that dispatches to other handlers depending on the result. -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20110622/53e42ec9/attachment.html From chris.hegarty at oracle.com Wed Jun 22 07:14:43 2011 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 22 Jun 2011 15:14:43 +0100 Subject: 7057935: com/sun/nio/sctp tests should be moved out of jdk_nio and into their own target, jdk_sctp Message-ID: <4E01F8D3.9020600@oracle.com> Kelly, Alan, The com/sun/nio/sctp tests are run when running the jdk regression tests with the makefile in the jdk/test directory. They are currently part of the jdk_nio3 target. SCTP is a niche area and there is no need to have these tests run with the nio tests. Also most of them are on the ProblemList.txt since there are many kernel issues ( most of which are resolved in the latest Solaris 10 update release, but there sill remains a few that will be addressed in S10u11 ). The bottom line is that these tests are not suitable to be run in JPRT, and give little value to developers not actively developing SCTP. Best to provide a specific jdk_sctp target that can be used when the sctp tests are required to be run. http://cr.openjdk.java.net/~chegar/7057935/jdk8.webrev.00/webrev/ -Chris. From Alan.Bateman at oracle.com Wed Jun 22 08:10:16 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 22 Jun 2011 16:10:16 +0100 Subject: 7057935: com/sun/nio/sctp tests should be moved out of jdk_nio and into their own target, jdk_sctp In-Reply-To: <4E01F8D3.9020600@oracle.com> References: <4E01F8D3.9020600@oracle.com> Message-ID: <4E0205D8.8030408@oracle.com> Chris Hegarty wrote: > Kelly, Alan, > > The com/sun/nio/sctp tests are run when running the jdk regression > tests with the makefile in the jdk/test directory. They are currently > part of the jdk_nio3 target. > > SCTP is a niche area and there is no need to have these tests run with > the nio tests. Also most of them are on the ProblemList.txt since > there are many kernel issues ( most of which are resolved in the > latest Solaris 10 update release, but there sill remains a few that > will be addressed in S10u11 ). The bottom line is that these tests are > not suitable to be run in JPRT, and give little value to developers > not actively developing SCTP. > > Best to provide a specific jdk_sctp target that can be used when the > sctp tests are required to be run. > > http://cr.openjdk.java.net/~chegar/7057935/jdk8.webrev.00/webrev/ > > -Chris. Separating these out into their own target make sense to me, at least for the short term. One coment is that you've replaced com/sun/nio in the jdk_nio3 target with com/sun/nio/file but there aren't any tests there so I assume you should just remove it. There is some additional plumbing in the top level repository's make/jprt.properties file that Kelly might want to comment on. My guess is that the new batch shouldn't need to be list there but Kelly might want JPRT be capable of running it. A general comment is that Jon added agentvm mode to jtreg recently so for jdk8 I think this make file should be re-worked to take advantage of it. Also there are batches such as jdk_misc that are unusual mix of tests. My guess is that most people aren't familiar with the names of the test batches and would prefer to specify the tests to run (or just to "make" to run all tests or a reasonable default set). -Alan. From kelly.ohair at oracle.com Wed Jun 22 10:26:01 2011 From: kelly.ohair at oracle.com (Kelly O'Hair) Date: Wed, 22 Jun 2011 10:26:01 -0700 Subject: 7057935: com/sun/nio/sctp tests should be moved out of jdk_nio and into their own target, jdk_sctp In-Reply-To: <4E0205D8.8030408@oracle.com> References: <4E01F8D3.9020600@oracle.com> <4E0205D8.8030408@oracle.com> Message-ID: <3A12FE96-4B99-407E-BDF0-DF4045E7AC7A@oracle.com> On Jun 22, 2011, at 8:10 AM, Alan Bateman wrote: > Chris Hegarty wrote: >> Kelly, Alan, >> >> The com/sun/nio/sctp tests are run when running the jdk regression tests with the makefile in the jdk/test directory. They are currently part of the jdk_nio3 target. >> >> SCTP is a niche area and there is no need to have these tests run with the nio tests. Also most of them are on the ProblemList.txt since there are many kernel issues ( most of which are resolved in the latest Solaris 10 update release, but there sill remains a few that will be addressed in S10u11 ). The bottom line is that these tests are not suitable to be run in JPRT, and give little value to developers not actively developing SCTP. >> >> Best to provide a specific jdk_sctp target that can be used when the sctp tests are required to be run. >> >> http://cr.openjdk.java.net/~chegar/7057935/jdk8.webrev.00/webrev/ >> >> -Chris. > Separating these out into their own target make sense to me, at least for the short term. One coment is that you've replaced com/sun/nio in the jdk_nio3 target with com/sun/nio/file but there aren't any tests there so I assume you should just remove it. You guys could also consider merging all the jdk_nio[123] into one test run if you get it working well enough. > > There is some additional plumbing in the top level repository's make/jprt.properties file that Kelly might want to comment on. My guess is that the new batch shouldn't need to be list there but Kelly might want JPRT be capable of running it. > Yeah, we'll need to add any new jdk_* test targets to the make/jprt.properties file and the jdk/make/jprt_properties file. > A general comment is that Jon added agentvm mode to jtreg recently so for jdk8 I think this make file should be re-worked to take advantage of it. Also there are batches such as jdk_misc that are unusual mix of tests. My guess is that most people aren't familiar with the names of the test batches and would prefer to specify the tests to run (or just to "make" to run all tests or a reasonable default set). > The default 'make test' should run the default or required tests. Developers should not need to know the test batch names, but I suspect if they are using JPRT they will. -kto > -Alan. > > > > From chris.hegarty at oracle.com Thu Jun 23 04:31:12 2011 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 23 Jun 2011 12:31:12 +0100 Subject: 6965150: TEST_BUG: java/nio/channels/AsynchronousSocketChannel/Basic.java takes too long In-Reply-To: <4DFF7822.8030102@oracle.com> References: <4DFF7822.8030102@oracle.com> Message-ID: <4E032400.60605@oracle.com> The change looks fine to me. -Chris. On 06/20/11 05:41 PM, Alan Bateman wrote: > > I brought this up about a year ago [1]. This test takes about 4 mins, > most of which is spent in a sub-test that attempts to connect to a host > that probably does not exist. I'm looking at this test again as we need > to speed it up. Here's the updated webrev with a change that adds the > option -skipSlowConnectTest to the test. The @run tag is changed to pass > this option so it skips the "slow connect" test during normal test runs: > http://cr.openjdk.java.net/~alanb/6965150/webrev/ > > The alternative is to just get rid of this sub-test as originally > suggested. This test predates try-with-resources so I've retrofitted to > a few places. Not everywhere, and more could be done if anyone has cycles. > > -Alan. > > [1] http://mail.openjdk.java.net/pipermail/nio-dev/2010-June/001003.html From Alan.Bateman at oracle.com Sat Jun 25 04:23:39 2011 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 25 Jun 2011 12:23:39 +0100 Subject: Proposal Final Draft Message-ID: <4E05C53B.4010404@oracle.com> As an FYI, JSR 203 now has a proposal final draft [1]. It's also archived on the project page [2] for easier browsing. It's the same as what is in jdk7-b146, except that the docs for the latest jdk7 build have the new CSS. -Alan. [1] http://jcp.org/aboutJava/communityprocess/pfd/jsr203/index.html [2] http://openjdk.java.net/projects/nio/pfd/docs/