From Thomas.Salter at unisys.com Mon Apr 6 11:50:38 2009 From: Thomas.Salter at unisys.com (Salter, Thomas A) Date: Mon, 6 Apr 2009 13:50:38 -0500 Subject: Sun.misc.Unsafe usage in WindowsNativeDisppatcher Message-ID: I was surprised that the implementation of WindowsFileAttributes.get uses Unsafe functions to directly access fields in the Windows structure returned by GetFileAttributesEx. Does this give a large performance gain over a series of SetField calls in the C code? -- If not, why add the maintenance issues of keeping Java code in sync with future Windows releases? -- If so, why not do the same thing in the Unix implementation? From Alan.Bateman at Sun.COM Mon Apr 6 12:17:08 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Mon, 06 Apr 2009 20:17:08 +0100 Subject: Sun.misc.Unsafe usage in WindowsNativeDisppatcher In-Reply-To: References: Message-ID: <49DA5534.4000703@sun.com> Salter, Thomas A wrote: > I was surprised that the implementation of WindowsFileAttributes.get uses Unsafe functions to directly access fields in the Windows structure returned by GetFileAttributesEx. Does this give a large performance gain over a series of SetField calls in the C code? > -- If not, why add the maintenance issues of keeping Java code in sync with future Windows releases? > -- If so, why not do the same thing in the Unix implementation? > The Unix/Linux implementation will likely be ported to other architectures and Unix implementations so having it reasonably portable is useful. Windows is very unique and I wouldn't expect the code to be used as a porting base to other operating systems. In terms of performance, WindowsFileAttributes.get is dominated by the Win32 call so it could have been implemented in native code as you suggest. However, maintenance-wise it's a bit of coin toss because a native implementation could have a dependency on field type/names like the Unix implementation. -Alan. From jjfraney at gmail.com Sat Apr 11 18:41:36 2009 From: jjfraney at gmail.com (John Franey) Date: Sat, 11 Apr 2009 21:41:36 -0400 Subject: NIO.2 and GC activity Message-ID: Folks, I only just became interested in nio 2, found this entry (Re: NIO.2 and GC activity ). +1 for design decisions that can keep memory througput low. I've been load testing an RTP based media server. I have found that thread pauses are directly correlated to media quality. Thread pauses cause jitter (variance in delay between packets). When threads are active, packets can be sent at a constant rate with low jitter. But when threads are paused, there is a delay in one or more of the packets. For example, in a stream of packets at rate of once every 20ms, a gc pause of 50ms would delay the immediate packet by 30-50ms, and the following packet by up to 10ms. These are tolerable, AFAICT, but could be only a component of a larger jitter detected at the receiving end. I've verified the above by correlating the thread pauses (from the gc log) with actual traffic using tcpdump/wireshark. I think memory allocated per send/receive can eat up the heap pretty quickly. At 250 udp 'connections', exchanging 50 packets each way per second, will add up to 25000 read and write requests per second. But, perhaps jvm will allocate Futures (retun type from read and write) on the thread stack? Thanks, John -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20090411/14e5500f/attachment.html From Alan.Bateman at Sun.COM Tue Apr 14 08:32:22 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Tue, 14 Apr 2009 16:32:22 +0100 Subject: NIO.2 and GC activity In-Reply-To: References: Message-ID: <49E4AC86.6080607@sun.com> John Franey wrote: > > Folks, > > I only just became interested in nio 2, found this entry (Re: NIO.2 > and GC activity ). > > +1 for design decisions that can keep memory througput low. > > I've been load testing an RTP based media server. I have found that > thread pauses are directly correlated to media quality. Thread pauses > cause jitter (variance in delay between packets). > > When threads are active, packets can be sent at a constant rate with > low jitter. But when threads are paused, there is a delay in one or > more of the packets. For example, in a stream of packets at rate of > once every 20ms, a gc pause of 50ms would delay the immediate packet > by 30-50ms, and the following packet by up to 10ms. These are > tolerable, AFAICT, but could be only a component of a larger jitter > detected at the receiving end. I've verified the above by correlating > the thread pauses (from the gc log) with actual traffic using > tcpdump/wireshark. > > I think memory allocated per send/receive can eat up the heap pretty > quickly. At 250 udp 'connections', exchanging 50 packets each way per > second, will add up to 25000 read and write requests per second. > > But, perhaps jvm will allocate Futures (retun type from read and > write) on the thread stack? Changing the methods that have a completion handler as parameter to return void is probably the right thing to do. However, as I said in a previous rely, this doesn't eliminate all object allocation as there are still cases where some objects are required (timers for example). A while back I did a number of tests but didn't observe any noticeable difference because the scavenging of the new generation was so fast. So are your tests with AsynchronousDatagramChannel? Just curious because this is the odd one out in that it doesn't have a high performance implementation yet. Hopefully this will be in the repository soon. -Alan. From leonfin at optonline.net Tue Apr 14 10:17:30 2009 From: leonfin at optonline.net (Leon Finker) Date: Tue, 14 Apr 2009 13:17:30 -0400 Subject: NIO.2 and GC activity In-Reply-To: <49E4AC86.6080607@sun.com> References: <49E4AC86.6080607@sun.com> Message-ID: <000c01c9bd24$e4865150$ad92f3f0$@net> But creating objects (i.e. timers) is current implementation detail. Implementation can change to not create objects per request. Implementation can allow objects to be reused. When you are talking about new generation scavenging are you only referring to actively streaming connections where flow of data is never delayed? Object will be promoted as soon as data availability is delayed (whatever the reason will be). With thousands of connections this has to cause a delay. In addition the recommended way (at least on windows iocp) is to have more than one outstanding socket operation pending per socket (btw does NIO.2 allow for this? To have more than one asynchronous pending socket receives for example). This increases the number of objects that will get promoted per connection per socket operation. I didn't do any stress testing yet myself. -----Original Message----- From: nio-dev-bounces at openjdk.java.net [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of Alan Bateman Sent: Tuesday, April 14, 2009 11:32 AM To: John Franey Cc: nio-dev at openjdk.java.net Subject: Re: NIO.2 and GC activity >Changing the methods that have a completion handler as parameter to >return void is probably the right thing to do. However, as I said in a >previous reply, this doesn't eliminate all object allocation as there are >still cases where some objects are required (timers for example). A >while back I did a number of tests but didn't observe any noticeable >difference because the scavenging of the new generation was so fast. ... From Alan.Bateman at Sun.COM Tue Apr 14 10:32:04 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Tue, 14 Apr 2009 18:32:04 +0100 Subject: NIO.2 and GC activity In-Reply-To: <000c01c9bd24$e4865150$ad92f3f0$@net> References: <49E4AC86.6080607@sun.com> <000c01c9bd24$e4865150$ad92f3f0$@net> Message-ID: <49E4C894.4090305@sun.com> Leon Finker wrote: > But creating objects (i.e. timers) is current implementation detail. > Implementation can change to not create objects per request. Implementation > can allow objects to be reused. > > When you are talking about new generation scavenging are you only referring > to actively streaming connections where flow of data is never delayed? > Object will be promoted as soon as data availability is delayed (whatever > the reason will be). With thousands of connections this has to cause a > delay. > Right, as I said in the original reply, the real concern is where the objects survive just long enough to be promoted. > In addition the recommended way (at least on windows iocp) is to have more > than one outstanding socket operation pending per socket (btw does NIO.2 > allow for this? To have more than one asynchronous pending socket receives > for example). This increases the number of objects that will get promoted > per connection per socket operation. I didn't do any stress testing yet > myself. > For stream based protocols you don't want to have several read operations outstanding on the same channel. No problem for datagram based protocols. The API does enforces this (in AsynchronousSocketChannel) for stream protocols. This has already helped several people to find bugs that would otherwise have caused bytes to be read or written out of order. -Alan. From alan.bateman at sun.com Tue Apr 14 10:48:12 2009 From: alan.bateman at sun.com (alan.bateman at sun.com) Date: Tue, 14 Apr 2009 17:48:12 +0000 Subject: hg: nio/nio/hotspot: 34 new changesets Message-ID: <20090414174918.DBB48EA1B@hg.openjdk.java.net> Changeset: 6e56a851ccaa Author: xdono Date: 2009-03-27 14:11 -0700 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/6e56a851ccaa Added tag jdk7-b52 for changeset 1b1e8f1a4fe8 ! .hgtags Changeset: ec1a6dc46005 Author: iveresov Date: 2009-03-12 14:01 -0700 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/ec1a6dc46005 6816433: Test G1 and ParOld in JPRT Reviewed-by: jmasa, never, ysr ! make/jprt.properties Changeset: 4018e98c778a Author: tonyp Date: 2009-03-13 16:10 -0400 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/4018e98c778a Merge Changeset: 09f82af55c3e Author: ysr Date: 2009-03-13 13:56 -0700 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/09f82af55c3e 6808322: ParNew, CMS, G1: ParGCAllocBuffer overflow Summary: Correct the overflow check in ParGCAllocBuffer::allocate(); simplify ParGCAllocBuffer::undo_allocation(). Reviewed-by: apetrusenko, jcoomes, jmasa, minqi, phh, tonyp ! src/share/vm/gc_implementation/parNew/parGCAllocBuffer.hpp Changeset: fe2441500281 Author: ysr Date: 2009-03-13 17:06 -0700 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/fe2441500281 Merge Changeset: 6c4cea9bfa11 Author: tonyp Date: 2009-03-15 22:03 -0400 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/6c4cea9bfa11 6604422: G1: re-use half-promoted regions 6728271: G1: Cleanup G1CollectedHeap::get_gc_alloc_regions() Summary: It allows the last half-full region to be allocated to during a GC to be reused during the next GC. Reviewed-by: apetrusenko, jcoomes ! src/share/vm/gc_implementation/g1/g1CollectedHeap.cpp ! src/share/vm/gc_implementation/g1/g1CollectedHeap.hpp ! src/share/vm/gc_implementation/g1/g1CollectorPolicy.cpp Changeset: 25e146966e7c Author: iveresov Date: 2009-03-16 08:01 -0700 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/25e146966e7c 6817419: G1: Enable extensive verification for humongous regions Summary: Enabled full verification for humongous regions. Also made sure that the VerifyAfterGC works with deferred updates and G1HRRSFlushLogBuffersOnVerify. Reviewed-by: tonyp ! src/share/vm/gc_implementation/g1/concurrentG1Refine.cpp ! src/share/vm/gc_implementation/g1/g1CollectedHeap.cpp ! src/share/vm/gc_implementation/g1/g1RemSet.cpp Changeset: 2a5da27ccae9 Author: tonyp Date: 2009-03-16 10:52 -0400 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/2a5da27ccae9 6816154: G1: introduce flags to enable/disable RSet updating and scanning Summary: Introduces two flags, -XX:-/+G1EnableParallelRSetUpdating and -XX:-/+G1EnableParallelRSetScanning, to turn on/off the "band aid" fix that serializes RSet updating / scanning during GCs. Reviewed-by: iveresov ! src/share/vm/gc_implementation/g1/g1RemSet.cpp ! src/share/vm/gc_implementation/g1/g1_globals.hpp Changeset: 922c573ea67d Author: iveresov Date: 2009-03-16 17:48 -0700 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/922c573ea67d 6815683: G1: SEGV during marking Summary: We should mark the regions that continue humongous regions as live if the first region is live. Reviewed-by: tonyp ! src/share/vm/gc_implementation/g1/concurrentMark.cpp Changeset: ba50942c8138 Author: tonyp Date: 2009-03-18 11:37 -0400 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/ba50942c8138 Merge ! src/share/vm/gc_implementation/g1/g1CollectedHeap.cpp ! src/share/vm/gc_implementation/g1/g1CollectedHeap.hpp ! src/share/vm/gc_implementation/g1/g1CollectorPolicy.cpp ! src/share/vm/gc_implementation/g1/g1RemSet.cpp ! src/share/vm/gc_implementation/g1/g1_globals.hpp Changeset: 6af0a709d52b Author: twisti Date: 2009-03-11 14:16 -0700 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/6af0a709d52b 6812587: Use auxv to determine SPARC hardware features on Solaris Summary: A similar function to getisax(2) should be used to determine all possible instruction set extensions. Reviewed-by: never, kvn ! src/cpu/sparc/vm/vm_version_sparc.cpp ! src/cpu/sparc/vm/vm_version_sparc.hpp ! src/os/solaris/vm/os_solaris.cpp ! src/os/solaris/vm/os_solaris.hpp ! src/os_cpu/solaris_sparc/vm/vm_version_solaris_sparc.cpp ! src/share/vm/includeDB_core Changeset: 660978a2a31a Author: kvn Date: 2009-03-12 10:37 -0700 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/660978a2a31a 6791178: Specialize for zero as the compressed oop vm heap base Summary: Use zero based compressed oops if java heap is below 32gb and unscaled compressed oops if java heap is below 4gb. Reviewed-by: never, twisti, jcoomes, coleenp ! agent/src/share/classes/sun/jvm/hotspot/debugger/Debugger.java ! agent/src/share/classes/sun/jvm/hotspot/debugger/DebuggerBase.java ! agent/src/share/classes/sun/jvm/hotspot/debugger/JVMDebugger.java ! agent/src/share/classes/sun/jvm/hotspot/debugger/remote/RemoteDebugger.java ! agent/src/share/classes/sun/jvm/hotspot/debugger/remote/RemoteDebuggerClient.java ! agent/src/share/classes/sun/jvm/hotspot/debugger/remote/RemoteDebuggerServer.java ! agent/src/share/classes/sun/jvm/hotspot/memory/Universe.java ! agent/src/share/classes/sun/jvm/hotspot/runtime/VM.java ! src/cpu/sparc/vm/assembler_sparc.cpp ! src/cpu/sparc/vm/sparc.ad ! src/cpu/sparc/vm/stubGenerator_sparc.cpp ! src/cpu/sparc/vm/vm_version_sparc.cpp ! src/cpu/sparc/vm/vtableStubs_sparc.cpp ! src/cpu/x86/vm/assembler_x86.cpp ! src/cpu/x86/vm/assembler_x86.hpp ! src/cpu/x86/vm/x86_64.ad ! src/os/linux/vm/os_linux.cpp ! src/os/solaris/dtrace/generateJvmOffsets.cpp ! src/os/solaris/dtrace/jhelper.d ! src/os/solaris/dtrace/libjvm_db.c ! src/os/solaris/vm/os_solaris.cpp ! src/os/windows/vm/os_windows.cpp ! src/os_cpu/linux_sparc/vm/globals_linux_sparc.hpp ! src/os_cpu/linux_x86/vm/globals_linux_x86.hpp ! src/os_cpu/solaris_sparc/vm/globals_solaris_sparc.hpp ! src/os_cpu/solaris_x86/vm/globals_solaris_x86.hpp ! src/os_cpu/windows_x86/vm/globals_windows_x86.hpp ! src/share/vm/asm/assembler.cpp ! src/share/vm/gc_implementation/g1/g1CollectedHeap.cpp ! src/share/vm/gc_implementation/parallelScavenge/parallelScavengeHeap.cpp ! src/share/vm/memory/genCollectedHeap.cpp ! src/share/vm/memory/universe.cpp ! src/share/vm/memory/universe.hpp ! src/share/vm/oops/oop.inline.hpp ! src/share/vm/opto/addnode.cpp ! src/share/vm/opto/compile.cpp ! src/share/vm/opto/connode.cpp ! src/share/vm/opto/lcm.cpp ! src/share/vm/opto/matcher.cpp ! src/share/vm/runtime/arguments.cpp ! src/share/vm/runtime/globals.hpp ! src/share/vm/runtime/os.hpp ! src/share/vm/runtime/virtualspace.cpp ! src/share/vm/runtime/virtualspace.hpp ! src/share/vm/runtime/vmStructs.cpp Changeset: c771b7f43bbf Author: twisti Date: 2009-03-13 11:35 -0700 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/c771b7f43bbf 6378821: bitCount() should use POPC on SPARC processors and AMD+10h Summary: bitCount() should use POPC on SPARC processors where POPC is implemented directly in hardware. Reviewed-by: kvn, never ! src/cpu/sparc/vm/sparc.ad ! src/cpu/sparc/vm/vm_version_sparc.cpp ! src/cpu/sparc/vm/vm_version_sparc.hpp ! src/cpu/x86/vm/assembler_x86.cpp ! src/cpu/x86/vm/assembler_x86.hpp ! src/cpu/x86/vm/vm_version_x86.cpp ! src/cpu/x86/vm/vm_version_x86.hpp ! src/cpu/x86/vm/x86_32.ad ! src/cpu/x86/vm/x86_64.ad ! src/os_cpu/solaris_sparc/vm/vm_version_solaris_sparc.cpp ! src/share/vm/classfile/vmSymbols.hpp ! src/share/vm/opto/classes.hpp ! src/share/vm/opto/connode.hpp ! src/share/vm/opto/library_call.cpp ! src/share/vm/runtime/globals.hpp + test/compiler/6378821/Test6378821.java Changeset: c517646eef23 Author: jrose Date: 2009-03-13 18:39 -0700 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/c517646eef23 6813212: factor duplicated assembly code for general subclass check (for 6655638) Summary: Code in interp_masm, stubGenerator, c1_LIRAssembler, and AD files moved into MacroAssembler. Reviewed-by: kvn ! src/cpu/sparc/vm/assembler_sparc.cpp ! src/cpu/sparc/vm/assembler_sparc.hpp ! src/cpu/sparc/vm/c1_LIRAssembler_sparc.cpp ! src/cpu/sparc/vm/c1_Runtime1_sparc.cpp ! src/cpu/sparc/vm/interp_masm_sparc.cpp ! src/cpu/sparc/vm/stubGenerator_sparc.cpp ! src/cpu/x86/vm/assembler_x86.cpp ! src/cpu/x86/vm/assembler_x86.hpp ! src/cpu/x86/vm/c1_LIRAssembler_x86.cpp ! src/cpu/x86/vm/c1_Runtime1_x86.cpp ! src/cpu/x86/vm/interp_masm_x86_32.cpp ! src/cpu/x86/vm/interp_masm_x86_64.cpp ! src/cpu/x86/vm/stubGenerator_x86_32.cpp ! src/cpu/x86/vm/stubGenerator_x86_64.cpp ! src/cpu/x86/vm/x86_32.ad ! src/cpu/x86/vm/x86_64.ad ! src/share/vm/opto/graphKit.cpp Changeset: c3a720eefe82 Author: kvn Date: 2009-03-16 15:06 -0700 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/c3a720eefe82 6816308: Changes to allow builds with latest Windows SDK 6.1 on 64bit Windows 2003 Summary: Allow Hotspot builds with latest Windows SDK 6.1 on 64bit Windows 2003 Reviewed-by: ohair, tbell, jcoomes ! make/windows/get_msc_ver.sh ! make/windows/makefiles/compile.make ! make/windows/makefiles/sa.make ! make/windows/makefiles/sanity.make ! src/cpu/x86/vm/interpreterRT_x86_64.cpp ! src/os_cpu/windows_x86/vm/unwind_windows_x86.hpp ! src/share/vm/adlc/adlc.hpp ! src/share/vm/gc_implementation/g1/concurrentMark.cpp ! src/share/vm/gc_implementation/g1/heapRegionRemSet.cpp ! src/share/vm/memory/blockOffsetTable.hpp ! src/share/vm/runtime/vm_version.cpp ! src/share/vm/utilities/globalDefinitions_visCPP.hpp Changeset: 039a914095f4 Author: kvn Date: 2009-03-18 13:25 -0700 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/039a914095f4 6772368: REGRESSION:tomcat crashed twice with JDK 7 Summary: Call make_block_at() with the original handler limits. Reviewed-by: never ! src/share/vm/ci/ciMethodBlocks.cpp Changeset: bd441136a5ce Author: kvn Date: 2009-03-19 09:13 -0700 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/bd441136a5ce Merge ! src/cpu/sparc/vm/sparc.ad ! src/cpu/x86/vm/assembler_x86.cpp ! src/cpu/x86/vm/c1_LIRAssembler_x86.cpp ! src/cpu/x86/vm/c1_Runtime1_x86.cpp ! src/cpu/x86/vm/interp_masm_x86_32.cpp ! src/cpu/x86/vm/interp_masm_x86_64.cpp ! src/cpu/x86/vm/stubGenerator_x86_32.cpp ! src/cpu/x86/vm/stubGenerator_x86_64.cpp ! src/cpu/x86/vm/x86_32.ad ! src/cpu/x86/vm/x86_64.ad ! src/os/linux/vm/os_linux.cpp ! src/os/solaris/vm/os_solaris.cpp ! src/os/windows/vm/os_windows.cpp ! src/share/vm/classfile/vmSymbols.hpp ! src/share/vm/gc_implementation/g1/concurrentMark.cpp ! src/share/vm/gc_implementation/g1/g1CollectedHeap.cpp ! src/share/vm/gc_implementation/g1/heapRegionRemSet.cpp ! src/share/vm/includeDB_core ! src/share/vm/memory/genCollectedHeap.cpp ! src/share/vm/memory/universe.cpp ! src/share/vm/memory/universe.hpp ! src/share/vm/oops/oop.inline.hpp ! src/share/vm/opto/classes.hpp ! src/share/vm/opto/compile.cpp ! src/share/vm/opto/graphKit.cpp ! src/share/vm/opto/lcm.cpp ! src/share/vm/opto/matcher.cpp ! src/share/vm/runtime/arguments.cpp ! src/share/vm/runtime/globals.hpp ! src/share/vm/runtime/os.hpp Changeset: 2314b7336582 Author: tonyp Date: 2009-03-21 22:53 -0400 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/2314b7336582 6820321: G1: Error: guarantee(check_nums(total, n, parts), "all seq lengths should match") Summary: Small fixes to sort out some verbosegc-related incorrectness and a failure Reviewed-by: apetrusenko ! src/share/vm/gc_implementation/g1/g1CollectorPolicy.cpp ! src/share/vm/gc_implementation/g1/g1RemSet.cpp Changeset: 59f139e8a8d1 Author: tonyp Date: 2009-03-25 10:36 -0400 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/59f139e8a8d1 Merge Changeset: 54782a4cd321 Author: poonam Date: 2009-03-15 18:11 -0700 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/54782a4cd321 6812971: SA: re-attaching to process fails Summary: After attaching, de-attaching SA from a process, the second time attach() call fails. This happens because in VM.initialize(), Universe does not get re-initialized before it is accessed. Reviewed-by: swamyv ! agent/src/share/classes/sun/jvm/hotspot/runtime/VM.java Changeset: 8ce995316d10 Author: acorn Date: 2009-03-16 08:50 -0400 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/8ce995316d10 Merge Changeset: 4aaa9f5e02a8 Author: acorn Date: 2009-03-18 17:20 -0400 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/4aaa9f5e02a8 4766230: Hotspot vtable inconsistencies cause core dumps. 6579515. 6582242. Reviewed-by: kamg, coleenp ! src/share/vm/classfile/classFileParser.cpp ! src/share/vm/oops/instanceKlass.cpp ! src/share/vm/oops/instanceKlass.hpp ! src/share/vm/oops/klassVtable.cpp ! src/share/vm/oops/klassVtable.hpp Changeset: e55bcaf3a6a1 Author: acorn Date: 2009-03-20 11:23 -0400 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/e55bcaf3a6a1 6819853: VM does not detect JDK which supports parallel class loaders Reviewed-by: coleenp, pbk, xlu, alanb ! src/share/vm/classfile/vmSymbols.hpp Changeset: c664a0794f85 Author: coleenp Date: 2009-03-20 22:08 -0400 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/c664a0794f85 6805748: Assertion "don't reset to 0 -- could be mistaken for never-executed" in CompilationPolicy Summary: Resetting the invocation counter for a method invocation event was setting count to zero for CompileThreshold=1, making it look like a never executed method. Reviewed-by: phh, kamg, acorn, never ! src/share/vm/interpreter/invocationCounter.cpp Changeset: 60bfce711da4 Author: acorn Date: 2009-03-23 10:42 -0400 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/60bfce711da4 Merge ! agent/src/share/classes/sun/jvm/hotspot/runtime/VM.java ! src/share/vm/classfile/vmSymbols.hpp Changeset: 6bdd6923ba16 Author: coleenp Date: 2009-03-25 14:19 -0400 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/6bdd6923ba16 6541756: Reduce executable C-heap Summary: Add executable parameters to reserve_memory and commit_memory to reduce executable memory to only the Code Heap. Reviewed-by: xlu, kvn, acorn ! src/os/linux/vm/os_linux.cpp ! src/os/solaris/vm/os_solaris.cpp ! src/os/windows/vm/os_windows.cpp ! src/share/vm/memory/heap.cpp ! src/share/vm/runtime/os.hpp ! src/share/vm/runtime/virtualspace.cpp ! src/share/vm/runtime/virtualspace.hpp Changeset: 715dceaa89b7 Author: acorn Date: 2009-03-25 13:09 -0400 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/715dceaa89b7 6603316: Improve instrumentation for classes loaded at startup Reviewed-by: xlu, mchung ! src/share/vm/classfile/classFileParser.cpp ! src/share/vm/prims/jni.cpp ! src/share/vm/prims/jvm.cpp ! src/share/vm/prims/jvm_misc.hpp Changeset: fe62b51b93f4 Author: acorn Date: 2009-03-26 16:00 -0400 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/fe62b51b93f4 Merge Changeset: 520d43965b1f Author: ikrylov Date: 2009-03-27 01:35 -0500 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/520d43965b1f 6812297: update project creation for Visual Studio 2005-2008 Summary: Add 2 news classes to create VC8 and VC9 projects Reviewed-by: apetrusenko, xlu ! make/windows/build_vm_def.sh ! make/windows/create.bat ! make/windows/makefiles/adlc.make ! make/windows/makefiles/compile.make ! make/windows/makefiles/makedeps.make ! make/windows/makefiles/rules.make ! src/share/tools/MakeDeps/WinGammaPlatformVC7.java + src/share/tools/MakeDeps/WinGammaPlatformVC8.java + src/share/tools/MakeDeps/WinGammaPlatformVC9.java ! src/share/vm/utilities/globalDefinitions_visCPP.hpp Changeset: 0aeec7d15d30 Author: acorn Date: 2009-03-27 14:35 -0400 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/0aeec7d15d30 Merge Changeset: 00bcc4b01dde Author: trims Date: 2009-03-27 16:54 -0700 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/00bcc4b01dde Merge Changeset: 9ab385cb0c42 Author: trims Date: 2009-03-27 16:58 -0700 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/9ab385cb0c42 6823377: Bump HS15 build number to 04 Summary: Update the HS15 Build number to 04 Reviewed-by: jcoomes ! make/hotspot_version Changeset: 032c6af894da Author: trims Date: 2009-04-01 22:31 -0700 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/032c6af894da Merge Changeset: 5450320b9c27 Author: xdono Date: 2009-04-02 16:51 -0700 URL: http://hg.openjdk.java.net/nio/nio/hotspot/rev/5450320b9c27 Added tag jdk7-b53 for changeset 032c6af894da ! .hgtags From alan.bateman at sun.com Tue Apr 14 10:52:12 2009 From: alan.bateman at sun.com (alan.bateman at sun.com) Date: Tue, 14 Apr 2009 17:52:12 +0000 Subject: hg: nio/nio: 2 new changesets Message-ID: <20090414175213.16210EA22@hg.openjdk.java.net> Changeset: c235f4a8559d Author: xdono Date: 2009-03-27 14:11 -0700 URL: http://hg.openjdk.java.net/nio/nio/rev/c235f4a8559d Added tag jdk7-b52 for changeset 4264c2fe6649 ! .hgtags Changeset: 2ef382b1bbd5 Author: xdono Date: 2009-04-02 16:51 -0700 URL: http://hg.openjdk.java.net/nio/nio/rev/2ef382b1bbd5 Added tag jdk7-b53 for changeset c235f4a8559d ! .hgtags From alan.bateman at sun.com Tue Apr 14 10:56:09 2009 From: alan.bateman at sun.com (alan.bateman at sun.com) Date: Tue, 14 Apr 2009 17:56:09 +0000 Subject: hg: nio/nio/corba: 3 new changesets Message-ID: <20090414175613.2AE57EA29@hg.openjdk.java.net> Changeset: 2e02b4137dad Author: xdono Date: 2009-03-27 14:11 -0700 URL: http://hg.openjdk.java.net/nio/nio/corba/rev/2e02b4137dad Added tag jdk7-b52 for changeset bec82237d694 ! .hgtags Changeset: 3c4d73194f6f Author: xdono Date: 2009-03-31 08:53 -0700 URL: http://hg.openjdk.java.net/nio/nio/corba/rev/3c4d73194f6f Merge - src/share/classes/com/sun/tools/corba/se/logutil/lib/jscheme.jar - src/share/classes/com/sun/tools/corba/se/logutil/lib/jschemelogutil.jar - src/share/classes/com/sun/tools/corba/se/logutil/scripts/mc - src/share/classes/com/sun/tools/corba/se/logutil/scripts/mc.scm - src/share/classes/com/sun/tools/corba/se/logutil/scripts/run Changeset: 8130ac858d67 Author: xdono Date: 2009-04-02 16:51 -0700 URL: http://hg.openjdk.java.net/nio/nio/corba/rev/8130ac858d67 Added tag jdk7-b53 for changeset 3c4d73194f6f ! .hgtags From alan.bateman at sun.com Tue Apr 14 11:05:32 2009 From: alan.bateman at sun.com (alan.bateman at sun.com) Date: Tue, 14 Apr 2009 18:05:32 +0000 Subject: hg: nio/nio/jaxp: 4 new changesets Message-ID: <20090414180539.5E98DEA30@hg.openjdk.java.net> Changeset: 30e3f9614f07 Author: xdono Date: 2009-03-27 14:11 -0700 URL: http://hg.openjdk.java.net/nio/nio/jaxp/rev/30e3f9614f07 Added tag jdk7-b52 for changeset 69ad87dc25cb ! .hgtags Changeset: 996284fd4afe Author: ohair Date: 2009-03-26 16:48 -0700 URL: http://hg.openjdk.java.net/nio/nio/jaxp/rev/996284fd4afe 6822913: Consolidate make/jprt.config files, let JPRT manage this file make it optional in repos Reviewed-by: tbell - make/jprt.config Changeset: e8837366d3fd Author: xdono Date: 2009-04-01 08:58 -0700 URL: http://hg.openjdk.java.net/nio/nio/jaxp/rev/e8837366d3fd Merge Changeset: 946a9f0c4932 Author: xdono Date: 2009-04-02 16:51 -0700 URL: http://hg.openjdk.java.net/nio/nio/jaxp/rev/946a9f0c4932 Added tag jdk7-b53 for changeset e8837366d3fd ! .hgtags From leonfin at optonline.net Tue Apr 14 11:06:42 2009 From: leonfin at optonline.net (Leon Finker) Date: Tue, 14 Apr 2009 14:06:42 -0400 Subject: NIO.2 and GC activity In-Reply-To: <49E4C894.4090305@sun.com> References: <49E4AC86.6080607@sun.com> <000c01c9bd24$e4865150$ad92f3f0$@net> <49E4C894.4090305@sun.com> Message-ID: <001c01c9bd2b$c4653420$4d2f9c60$@net> This can be optional. Current enforced requirement removes possible optimizations for receive that developers can do. I worked on communication layer that did benefit from multiple outstanding recvs. I don't have all the reference available, but here is one I could find quickly http://msdn.microsoft.com/en-us/magazine/cc302334.aspx "as long as the application takes care to always have a few overlapped WSARecvs outstanding on a connection. The availability of posted application buffers removes the need for AFD to buffer incoming data." In short more than one receive can be an optimization where data will not be buffered in tcp stack, but written directly to application buffer (more optimizations in win2k8 and up including NIC writing directly to memory where application buffer is http://support.microsoft.com/kb/951037). So what if one has to deal with not so complicated re-sequencing logic? The core of communication layer code is written and tested once. It doesn't change often. I personally vote against this enforcement. It can be enabled for debug or made optional. -----Original Message----- From: Alan.Bateman at Sun.COM [mailto:Alan.Bateman at Sun.COM] Sent: Tuesday, April 14, 2009 1:32 PM To: Leon Finker Cc: nio-dev at openjdk.java.net Subject: Re: NIO.2 and GC activity > In addition the recommended way (at least on windows iocp) is to have more > than one outstanding socket operation pending per socket (btw does NIO.2 > allow for this? To have more than one asynchronous pending socket receives > for example). This increases the number of objects that will get promoted > per connection per socket operation. I didn't do any stress testing yet > myself. > For stream based protocols you don't want to have several read operations outstanding on the same channel. No problem for datagram based protocols. The API does enforces this (in AsynchronousSocketChannel) for stream protocols. This has already helped several people to find bugs that would otherwise have caused bytes to be read or written out of order. -Alan. From alan.bateman at sun.com Tue Apr 14 11:07:24 2009 From: alan.bateman at sun.com (alan.bateman at sun.com) Date: Tue, 14 Apr 2009 18:07:24 +0000 Subject: hg: nio/nio/jaxws: 4 new changesets Message-ID: <20090414180730.E278DEA36@hg.openjdk.java.net> Changeset: 2c10f0cbb34e Author: xdono Date: 2009-03-27 14:11 -0700 URL: http://hg.openjdk.java.net/nio/nio/jaxws/rev/2c10f0cbb34e Added tag jdk7-b52 for changeset e646890d18b7 ! .hgtags Changeset: 0814199b8ee7 Author: ohair Date: 2009-03-26 16:48 -0700 URL: http://hg.openjdk.java.net/nio/nio/jaxws/rev/0814199b8ee7 6822913: Consolidate make/jprt.config files, let JPRT manage this file make it optional in repos Reviewed-by: tbell - make/jprt.config Changeset: b250218eb2e5 Author: xdono Date: 2009-04-01 08:58 -0700 URL: http://hg.openjdk.java.net/nio/nio/jaxws/rev/b250218eb2e5 Merge Changeset: 50ea00dc5f14 Author: xdono Date: 2009-04-02 16:51 -0700 URL: http://hg.openjdk.java.net/nio/nio/jaxws/rev/50ea00dc5f14 Added tag jdk7-b53 for changeset b250218eb2e5 ! .hgtags From alan.bateman at sun.com Tue Apr 14 11:09:18 2009 From: alan.bateman at sun.com (alan.bateman at sun.com) Date: Tue, 14 Apr 2009 18:09:18 +0000 Subject: hg: nio/nio/jdk: 33 new changesets Message-ID: <20090414181624.B89D2EA49@hg.openjdk.java.net> Changeset: 7264cacbddaa Author: alanb Date: 2009-03-27 15:24 +0000 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/7264cacbddaa 6693490: (se) select throws "File exists" IOException under load (lnx) Reviewed-by: sherman ! src/share/classes/sun/nio/ch/SelChImpl.java ! src/solaris/classes/sun/nio/ch/EPollArrayWrapper.java ! src/solaris/classes/sun/nio/ch/EPollSelectorImpl.java + test/java/nio/channels/Selector/RegAfterPreClose.java Changeset: 9fa8b6276b31 Author: alanb Date: 2009-03-27 16:04 +0000 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/9fa8b6276b31 6772303: (se) IOException: Invalid argument" thrown on a call to Selector.select(value) with -d64 Reviewed-by: sherman ! src/solaris/native/sun/nio/ch/DevPollArrayWrapper.c Changeset: ff0a9e50f033 Author: alanb Date: 2009-03-30 19:22 +0100 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/ff0a9e50f033 Merge Changeset: 85a91be56593 Author: mchung Date: 2009-03-31 23:52 -0700 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/85a91be56593 6819110: Lazily load Sun digest provider for jar verification Summary: Lazily call Providers.getSunProvider() instead of at static initializer Reviewed-by: mullan ! src/share/classes/sun/security/util/ManifestEntryVerifier.java Changeset: ee75d1fac0ca Author: weijun Date: 2009-04-03 11:36 +0800 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/ee75d1fac0ca 6825352: support self-issued certificate in keytool Reviewed-by: xuelei ! src/share/classes/sun/security/tools/KeyTool.java + test/sun/security/tools/keytool/selfissued.sh Changeset: de80210c56a6 Author: sherman Date: 2009-04-02 15:35 -0700 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/de80210c56a6 4681995: Add support for large (> 4GB) zip/jar files Summary: The ZIP64 format support is added for > 4GB jar/zip files Reviewed-by: alanb, martin + src/share/classes/java/util/zip/ZipConstants64.java ! src/share/classes/java/util/zip/ZipEntry.java ! src/share/classes/java/util/zip/ZipInputStream.java ! src/share/classes/java/util/zip/ZipOutputStream.java ! src/share/classes/java/util/zip/package.html ! src/share/native/java/util/zip/zip_util.c ! src/share/native/java/util/zip/zip_util.h ! src/share/native/java/util/zip/zlib-1.1.3/zlib.h + test/java/util/zip/LargeZip.java ! test/java/util/zip/ZipFile/LargeZipFile.java Changeset: 030b29ccd0db Author: sherman Date: 2009-04-03 09:51 -0700 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/030b29ccd0db Merge Changeset: 3501cc282cd2 Author: xdono Date: 2009-03-27 14:11 -0700 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/3501cc282cd2 Added tag jdk7-b52 for changeset bcbeadb4a5d7 ! .hgtags Changeset: 1bbbd1bf9be3 Author: xdono Date: 2009-03-31 08:53 -0700 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/1bbbd1bf9be3 Merge - src/share/classes/sun/misc/JavaIODeleteOnExitAccess.java Changeset: 90873391a0e0 Author: ohair Date: 2009-03-26 16:52 -0700 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/90873391a0e0 6822374: Windows: detect X64 when PROCESSOR_IDENTIFIER contains EM64T or Intel64 6822913: Consolidate make/jprt.config files, let JPRT manage this file make it optional in repos Reviewed-by: tbell ! make/common/shared/Platform.gmk ! make/jdk_generic_profile.sh - make/jprt.config Changeset: 964cc8eb3232 Author: tbell Date: 2009-03-31 15:27 -0700 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/964cc8eb3232 6819847: build is broken for OpenJDK with plugs Reviewed-by: jjg, robilad, ohair ! make/Makefile ! make/common/Defs.gmk ! make/common/shared/Sanity-Settings.gmk ! make/java/redist/Makefile Changeset: ecb7723aaa7c Author: tbell Date: 2009-04-01 04:44 -0700 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/ecb7723aaa7c 6824595: OpenJDK fix breaks product build for jdk7 Reviewed-by: xdono, ohair ! make/Makefile Changeset: deced414c8e4 Author: xdono Date: 2009-04-01 08:58 -0700 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/deced414c8e4 Merge - src/share/classes/sun/misc/JavaIODeleteOnExitAccess.java Changeset: a2033addca67 Author: ohair Date: 2009-04-01 16:49 -0700 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/a2033addca67 6825175: Remove or disable sanity check on binary plugs Reviewed-by: xdono ! make/common/shared/Sanity.gmk Changeset: 8536cdffa32e Author: xdono Date: 2009-04-02 16:51 -0700 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/8536cdffa32e Added tag jdk7-b53 for changeset a2033addca67 ! .hgtags Changeset: 17f50ed5fcab Author: tbell Date: 2009-04-03 10:29 -0700 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/17f50ed5fcab Merge Changeset: 267d1f8aa82a Author: alanb Date: 2009-04-02 11:13 +0100 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/267d1f8aa82a 6824135: (ch) test/java/nio/channels/AsyncCloseAndInterrupt.java fails (lnx) Reviewed-by: sherman ! src/share/classes/sun/nio/ch/FileChannelImpl.java ! test/java/nio/channels/AsyncCloseAndInterrupt.java Changeset: 464727e3afb4 Author: alanb Date: 2009-04-02 11:19 +0100 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/464727e3afb4 6666739: (ref) ReferenceQueue.poll() doesn't scale well 6711667: (ref) Update SoftReference timestamp only if clock advances Summary: Forward port from 6u14; originally fixed by Tom Rodriguez in earlier update Reviewed-by: martin ! src/share/classes/java/lang/ref/ReferenceQueue.java ! src/share/classes/java/lang/ref/SoftReference.java Changeset: aed19719b1e9 Author: alanb Date: 2009-04-02 16:31 +0100 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/aed19719b1e9 6824141: test/java/rmi/activation/rmidViaInheritedChannel tests fail Reviewed-by: peterjones ! test/java/rmi/activation/rmidViaInheritedChannel/InheritedChannelNotServerSocket.java ! test/java/rmi/activation/rmidViaInheritedChannel/RmidViaInheritedChannel.java Changeset: 4befa480d3c8 Author: alanb Date: 2009-04-02 19:47 +0100 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/4befa480d3c8 6824477: (se) Selector.select fails with IOException: "Invalid argument" if maximum file descriptors is low Reviewed-by: sherman ! src/solaris/classes/sun/nio/ch/DevPollArrayWrapper.java + test/java/nio/channels/Selector/LotsOfUpdates.java + test/java/nio/channels/Selector/lots_of_updates.sh Changeset: e50a00095a53 Author: alanb Date: 2009-04-03 22:10 +0100 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/e50a00095a53 6823609: (se) Selector.select hangs on Windows under load Reviewed-by: sherman ! src/windows/classes/sun/nio/ch/WindowsSelectorImpl.java + test/java/nio/channels/Selector/HelperSlowToDie.java Changeset: 93d1fbe001b8 Author: alanb Date: 2009-04-06 08:59 +0100 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/93d1fbe001b8 4890703: Support SDP (sol) Reviewed-by: michaelm ! make/java/net/FILES_c.gmk ! make/java/net/Makefile ! make/java/net/mapfile-vers ! make/sun/net/FILES_java.gmk ! src/share/classes/java/net/AbstractPlainSocketImpl.java ! src/share/classes/sun/nio/ch/AsynchronousServerSocketChannelImpl.java ! src/share/classes/sun/nio/ch/AsynchronousSocketChannelImpl.java ! src/share/classes/sun/nio/ch/ServerSocketChannelImpl.java ! src/share/classes/sun/nio/ch/SocketChannelImpl.java + src/solaris/classes/sun/net/NetHooks.java + src/solaris/classes/sun/net/spi/SdpProvider.java ! src/solaris/classes/sun/nio/ch/UnixAsynchronousSocketChannelImpl.java + src/solaris/lib/sdp/sdp.conf.template + src/solaris/native/sun/net/spi/SdpProvider.c ! src/solaris/native/sun/nio/ch/FileChannelImpl.c + src/windows/classes/sun/net/NetHooks.java + test/sun/net/sdp/ProbeIB.java + test/sun/net/sdp/Sanity.java + test/sun/net/sdp/sanity.sh Changeset: d89688532509 Author: alanb Date: 2009-04-06 11:29 +0100 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/d89688532509 Merge - make/jprt.config Changeset: 45ff1a9d4edb Author: valeriep Date: 2009-04-06 18:46 -0700 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/45ff1a9d4edb 4735126: (cl) ClassLoader.loadClass locks all instances in chain when delegating Summary: Added support for parallel-capable class loaders Reviewed-by: alanb ! make/java/java/mapfile-vers ! src/share/classes/java/lang/Class.java ! src/share/classes/java/lang/ClassLoader.java ! src/share/classes/java/net/URLClassLoader.java ! src/share/classes/java/security/SecureClassLoader.java ! src/share/classes/sun/misc/Launcher.java ! src/share/native/java/lang/ClassLoader.c + test/java/lang/ClassLoader/deadlock/Alice.java + test/java/lang/ClassLoader/deadlock/Bob.java + test/java/lang/ClassLoader/deadlock/DelegatingLoader.java + test/java/lang/ClassLoader/deadlock/Starter.java + test/java/lang/ClassLoader/deadlock/SupAlice.java + test/java/lang/ClassLoader/deadlock/SupBob.java + test/java/lang/ClassLoader/deadlock/TestCrossDelegate.sh + test/java/lang/ClassLoader/deadlock/TestOneWayDelegate.sh Changeset: 22b6e09960c1 Author: valeriep Date: 2009-04-06 18:52 -0700 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/22b6e09960c1 6440846: (cl) Deadlock between AppClassLoader and ExtClassLoader Summary: Fixed a deadlock between the two class loaders Reviewed-by: alanb ! src/share/classes/sun/security/jca/ProviderConfig.java + test/java/security/Security/ClassLoaderDeadlock/CreateSerialized.java + test/java/security/Security/ClassLoaderDeadlock/Deadlock2.java + test/java/security/Security/ClassLoaderDeadlock/Deadlock2.sh Changeset: a31f5f824580 Author: weijun Date: 2009-04-08 13:54 +0800 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/a31f5f824580 4811968: ASN.1 (X509Certificate) implementations don't handle large OID components Reviewed-by: xuelei ! src/share/classes/sun/security/util/ObjectIdentifier.java ! test/sun/security/util/Oid/OidFormat.java + test/sun/security/util/Oid/S11N.sh + test/sun/security/util/Oid/SerialTest.java Changeset: 74a3d8978eb0 Author: sherman Date: 2009-04-08 09:21 -0700 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/74a3d8978eb0 6827871: Cleanup leftover code in CharToByteJohab.java Summary: Removed the leftover data tables Reviewed-by: alanb ! src/share/classes/sun/io/CharToByteJohab.java Changeset: 6fe0aa207f5f Author: sherman Date: 2009-04-08 10:40 -0700 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/6fe0aa207f5f 6827921: ByteToCharBig5.java should use nio data tables instead of its own copy Summary: To use the data tables from sun.nio.cs.ext.Big5 Reviewed-by: alanb ! src/share/classes/sun/io/ByteToCharBig5.java Changeset: 8d37331265ae Author: weijun Date: 2009-04-09 15:32 +0800 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/8d37331265ae 6714845: Quotes in Kerberos configuration file are included in the values Reviewed-by: xuelei ! src/share/classes/sun/security/krb5/Config.java + test/sun/security/krb5/ConfigWithQuotations.java + test/sun/security/krb5/edu.mit.Kerberos Changeset: 897b2d42995a Author: weijun Date: 2009-04-10 11:21 +0800 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/897b2d42995a 6587676: Krb5LoginModule failure if useTicketCache=true on Vista Reviewed-by: valeriep ! src/windows/native/sun/security/krb5/NativeCreds.c Changeset: 572d3f36c8a9 Author: martin Date: 2009-04-12 20:21 -0700 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/572d3f36c8a9 6827153: Miscellaneous typos in javadoc Reviewed-by: alanb ! src/share/classes/java/lang/NoSuchFieldError.java ! src/share/classes/java/nio/channels/AsynchronousDatagramChannel.java ! src/share/classes/java/nio/file/Path.java ! src/share/classes/java/nio/file/SecureDirectoryStream.java ! src/share/classes/java/security/AccessController.java ! src/share/classes/java/security/AlgorithmParametersSpi.java ! src/share/classes/java/security/PrivilegedActionException.java ! src/share/classes/java/security/Security.java ! src/share/classes/java/security/SecurityPermission.java ! src/share/classes/java/security/SignatureSpi.java ! src/share/classes/java/security/cert/CertificateFactory.java ! src/share/classes/java/security/cert/CertificateFactorySpi.java Changeset: 6f99dbd58123 Author: valeriep Date: 2009-04-13 18:20 -0700 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/6f99dbd58123 6829098: Regression test java/security/Security/ClassLoaderDeadlock/Deadlock2.java error - missing ";" Summary: Added back the missing ";" Reviewed-by: weijun ! test/java/security/Security/ClassLoaderDeadlock/Deadlock2.java Changeset: 1554a70d74c6 Author: alanb Date: 2009-04-14 18:23 +0100 URL: http://hg.openjdk.java.net/nio/nio/jdk/rev/1554a70d74c6 Merge ! .hgtags - make/jprt.config ! src/share/classes/java/nio/channels/AsynchronousDatagramChannel.java ! src/share/classes/java/nio/file/Path.java ! src/share/classes/java/nio/file/SecureDirectoryStream.java ! src/share/classes/sun/nio/ch/AsynchronousServerSocketChannelImpl.java ! src/share/classes/sun/nio/ch/AsynchronousSocketChannelImpl.java ! src/share/classes/sun/nio/ch/FileChannelImpl.java ! src/share/classes/sun/nio/ch/ServerSocketChannelImpl.java ! src/share/classes/sun/nio/ch/SocketChannelImpl.java ! src/solaris/classes/sun/nio/ch/DevPollArrayWrapper.java ! src/solaris/classes/sun/nio/ch/EPollArrayWrapper.java ! src/solaris/classes/sun/nio/ch/EPollSelectorImpl.java ! src/solaris/classes/sun/nio/ch/UnixAsynchronousSocketChannelImpl.java ! src/solaris/native/sun/nio/ch/FileChannelImpl.c From alan.bateman at sun.com Tue Apr 14 11:23:44 2009 From: alan.bateman at sun.com (alan.bateman at sun.com) Date: Tue, 14 Apr 2009 18:23:44 +0000 Subject: hg: nio/nio/langtools: 9 new changesets Message-ID: <20090414182400.15C1CEA4E@hg.openjdk.java.net> Changeset: 07da2ffbb76b Author: jjg Date: 2009-03-30 15:08 -0700 URL: http://hg.openjdk.java.net/nio/nio/langtools/rev/07da2ffbb76b 6819246: improve support for decoding instructions in classfile library Reviewed-by: ksrini ! src/share/classes/com/sun/tools/classfile/Code_attribute.java + src/share/classes/com/sun/tools/classfile/Instruction.java - src/share/classes/com/sun/tools/classfile/OpCodes.java + src/share/classes/com/sun/tools/classfile/Opcode.java ! src/share/classes/com/sun/tools/javap/CodeWriter.java Changeset: 89f67512b635 Author: jjg Date: 2009-03-31 11:07 -0700 URL: http://hg.openjdk.java.net/nio/nio/langtools/rev/89f67512b635 6817950: refactor ClassReader to improve attribute handling Reviewed-by: mcimadamore ! src/share/classes/com/sun/tools/javac/jvm/ClassFile.java ! src/share/classes/com/sun/tools/javac/jvm/ClassReader.java Changeset: af10262bd031 Author: jjg Date: 2009-03-31 11:16 -0700 URL: http://hg.openjdk.java.net/nio/nio/langtools/rev/af10262bd031 6813059: replace use of JavaCompiler.errorCount with shouldContinue Reviewed-by: mcimadamore ! src/share/classes/com/sun/tools/javac/main/JavaCompiler.java + test/tools/javac/policy/test3/A.java + test/tools/javac/policy/test3/Test.java Changeset: 1ec9ff434ce2 Author: xdono Date: 2009-03-27 14:11 -0700 URL: http://hg.openjdk.java.net/nio/nio/langtools/rev/1ec9ff434ce2 Added tag jdk7-b52 for changeset 29329051d483 ! .hgtags Changeset: 72c2df1a2b5a Author: xdono Date: 2009-03-31 08:54 -0700 URL: http://hg.openjdk.java.net/nio/nio/langtools/rev/72c2df1a2b5a Merge Changeset: 39c674c60a36 Author: ohair Date: 2009-03-26 16:48 -0700 URL: http://hg.openjdk.java.net/nio/nio/langtools/rev/39c674c60a36 6822913: Consolidate make/jprt.config files, let JPRT manage this file make it optional in repos Reviewed-by: tbell - make/jprt.config Changeset: dbdeb4a7581b Author: xdono Date: 2009-04-01 08:58 -0700 URL: http://hg.openjdk.java.net/nio/nio/langtools/rev/dbdeb4a7581b Merge Changeset: 197a7f881937 Author: xdono Date: 2009-04-02 16:52 -0700 URL: http://hg.openjdk.java.net/nio/nio/langtools/rev/197a7f881937 Added tag jdk7-b53 for changeset dbdeb4a7581b ! .hgtags Changeset: 3e4038edfcb7 Author: tbell Date: 2009-04-03 10:29 -0700 URL: http://hg.openjdk.java.net/nio/nio/langtools/rev/3e4038edfcb7 Merge - src/share/classes/com/sun/tools/classfile/OpCodes.java From Alan.Bateman at Sun.COM Tue Apr 14 11:47:02 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Tue, 14 Apr 2009 19:47:02 +0100 Subject: NIO.2 and GC activity In-Reply-To: <001c01c9bd2b$c4653420$4d2f9c60$@net> References: <49E4AC86.6080607@sun.com> <000c01c9bd24$e4865150$ad92f3f0$@net> <49E4C894.4090305@sun.com> <001c01c9bd2b$c4653420$4d2f9c60$@net> Message-ID: <49E4DA26.2080905@sun.com> Leon Finker wrote: > This can be optional. Current enforced requirement removes possible > optimizations for receive that developers can do. I worked on communication > layer that did benefit from multiple outstanding recvs. I don't have all the > reference available, but here is one I could find quickly > http://msdn.microsoft.com/en-us/magazine/cc302334.aspx > "as long as the application takes care to always have a few overlapped > WSARecvs outstanding on a connection. The availability of posted application > buffers removes the need for AFD to buffer incoming data." > > In short more than one receive can be an optimization where data will not be > buffered in tcp stack, but written directly to application buffer (more > optimizations in win2k8 and up including NIC writing directly to memory > where application buffer is http://support.microsoft.com/kb/951037). So what > if one has to deal with not so complicated re-sequencing logic? The core of > communication layer code is written and tested once. It doesn't change > often. I personally vote against this enforcement. It can be enabled for > debug or made optional. > Just so we're clear, you can initiate another read operation in your completion handler (it doesn't t require the completion handler to complete). The optimization that I think you are referring to just requires that there is a buffer available for an outstanding read so that it can read directly into the buffer. That should work fine and only rarely (protocol dependent and buffer size dependent of course) should the driver need to buffer. -Alan. From leonfin at optonline.net Tue Apr 14 12:39:02 2009 From: leonfin at optonline.net (Leon Finker) Date: Tue, 14 Apr 2009 15:39:02 -0400 Subject: NIO.2 and GC activity In-Reply-To: <49E4DA26.2080905@sun.com> References: <49E4AC86.6080607@sun.com> <000c01c9bd24$e4865150$ad92f3f0$@net> <49E4C894.4090305@sun.com> <001c01c9bd2b$c4653420$4d2f9c60$@net> <49E4DA26.2080905@sun.com> Message-ID: <000901c9bd38$aa7ce5a0$ff76b0e0$@net> I realized after I sent previous email maybe this is not cross platform so it may not be feasible. It is fully supported on windows. Yes I'm referring to buffer(s) always being available to the driver. The goal is to try and saturate driver with outstanding read buffers (to a limit of course) to minimize any buffering on its side. The buffering in driver will in most cases happen while NIO.2 hands off the read buffer and application does some simplistic processing on it. With more than one outstanding read buffer, this can be minimized. Instead of driver waiting for application to schedule another read while one completed, driver can fill in previous outstanding read buffers and fire completion. The only issue in this case is to deal with out of sequence read completions. As long as simultaneously outstanding read buffers are always kept within a certain maximum, it works to our advantage. -----Original Message----- From: Alan.Bateman at Sun.COM [mailto:Alan.Bateman at Sun.COM] Sent: Tuesday, April 14, 2009 2:47 PM To: Leon Finker Cc: nio-dev at openjdk.java.net Subject: Re: NIO.2 and GC activity Leon Finker wrote: > This can be optional. Current enforced requirement removes possible > optimizations for receive that developers can do. I worked on communication > layer that did benefit from multiple outstanding recvs. I don't have all the > reference available, but here is one I could find quickly > http://msdn.microsoft.com/en-us/magazine/cc302334.aspx > "as long as the application takes care to always have a few overlapped > WSARecvs outstanding on a connection. The availability of posted application > buffers removes the need for AFD to buffer incoming data." > > In short more than one receive can be an optimization where data will not be > buffered in tcp stack, but written directly to application buffer (more > optimizations in win2k8 and up including NIC writing directly to memory > where application buffer is http://support.microsoft.com/kb/951037). So what > if one has to deal with not so complicated re-sequencing logic? The core of > communication layer code is written and tested once. It doesn't change > often. I personally vote against this enforcement. It can be enabled for > debug or made optional. > Just so we're clear, you can initiate another read operation in your completion handler (it doesn't t require the completion handler to complete). The optimization that I think you are referring to just requires that there is a buffer available for an outstanding read so that it can read directly into the buffer. That should work fine and only rarely (protocol dependent and buffer size dependent of course) should the driver need to buffer. -Alan. From Alan.Bateman at Sun.COM Thu Apr 16 08:56:58 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Thu, 16 Apr 2009 16:56:58 +0100 Subject: NIO.2 and GC activity In-Reply-To: <000901c9bd38$aa7ce5a0$ff76b0e0$@net> References: <49E4AC86.6080607@sun.com> <000c01c9bd24$e4865150$ad92f3f0$@net> <49E4C894.4090305@sun.com> <001c01c9bd2b$c4653420$4d2f9c60$@net> <49E4DA26.2080905@sun.com> <000901c9bd38$aa7ce5a0$ff76b0e0$@net> Message-ID: <49E7554A.5090809@sun.com> Leon Finker wrote: > I realized after I sent previous email maybe this is not cross platform so > it > may not be feasible. It is fully supported on windows. Yes I'm referring to > buffer(s) always being available to the driver. The goal is to try and > saturate > driver with outstanding read buffers (to a limit of course) to minimize any > buffering on its side. The buffering in driver will in most cases happen > while > NIO.2 hands off the read buffer and application does some simplistic > processing > on it. With more than one outstanding read buffer, this can be minimized. Yes, I'm familiar with it but it's a bit more complicated in that it requires a guarantee on the ordering that the I/O operations are executed and also requires the application to be able to deal with concurrent or even out of order notifications. There are a couple of cases where the ordering can't be guaranteed. For example, I/O operations are never initiated directly by non-pooled threads (because the thread may terminate causing any outstanding I/O operations it initiated to abort; this is Windows kernel thing). This and other cases means that it is possible for multiple reads on the same channel to be initiated in the kernel in a different order than might be expected, making it impossible to re-sequence the completion notifications. Furthermore, it just doesn't make sense for write operations - for example a write may complete without writing all bytes, in which case a queued write would corrupt the stream. I'm not against queuing of reads on streams but it just may be too advanced for many developers and creates the potential for some nasty bugs. Maybe in the future it is something to add, as an opt-in rather than the default. In any case, it would be interesting to compare the performance against code that initiates a read early in the completion handler. -Alan. From leonfin at optonline.net Thu Apr 16 18:30:43 2009 From: leonfin at optonline.net (Leon Finker) Date: Thu, 16 Apr 2009 21:30:43 -0400 Subject: NIO.2 and GC activity In-Reply-To: <49E7554A.5090809@sun.com> References: <49E4AC86.6080607@sun.com> <000c01c9bd24$e4865150$ad92f3f0$@net> <49E4C894.4090305@sun.com> <001c01c9bd2b$c4653420$4d2f9c60$@net> <49E4DA26.2080905@sun.com> <000901c9bd38$aa7ce5a0$ff76b0e0$@net> <49E7554A.5090809@sun.com> Message-ID: <007801c9befc$207c1a90$61744fb0$@net> I agree there is no advantage for the writes, and I don't advocate it. On the reads I disagree that the order can't be guaranteed or re-sequenced on completion/failure. One is always able to re-establish the original order of reads that were scheduled on their completion or failure simply by associating the counter with the scheduled reads and then re-establishing the sequence. Yes, one has to be aware that if they schedule a read on non I/O windows pool thread and thread immediately exits, the I/O request will be canceled. But this applies to current NIO.2 implementation even with non simultaneous reads. I agree the code is somewhat involved, and one has to deal with concurrency, but it's an option for core communication framework libraries out there. This optimization is not for every case, it's for specific communication/protocol patterns. So, I'm not sure if comparison will be fair. It's just a suggestion based on experience. If there is no need for this from the community, then you're right it can be added later. -----Original Message----- From: Alan.Bateman at Sun.COM [mailto:Alan.Bateman at Sun.COM] Sent: Thursday, April 16, 2009 11:57 AM To: Leon Finker Cc: nio-dev at openjdk.java.net Subject: Re: NIO.2 and GC activity Leon Finker wrote: > I realized after I sent previous email maybe this is not cross platform so > it > may not be feasible. It is fully supported on windows. Yes I'm referring to > buffer(s) always being available to the driver. The goal is to try and > saturate > driver with outstanding read buffers (to a limit of course) to minimize any > buffering on its side. The buffering in driver will in most cases happen > while > NIO.2 hands off the read buffer and application does some simplistic > processing > on it. With more than one outstanding read buffer, this can be minimized. Yes, I'm familiar with it but it's a bit more complicated in that it requires a guarantee on the ordering that the I/O operations are executed and also requires the application to be able to deal with concurrent or even out of order notifications. There are a couple of cases where the ordering can't be guaranteed. For example, I/O operations are never initiated directly by non-pooled threads (because the thread may terminate causing any outstanding I/O operations it initiated to abort; this is Windows kernel thing). This and other cases means that it is possible for multiple reads on the same channel to be initiated in the kernel in a different order than might be expected, making it impossible to re-sequence the completion notifications. Furthermore, it just doesn't make sense for write operations - for example a write may complete without writing all bytes, in which case a queued write would corrupt the stream. I'm not against queuing of reads on streams but it just may be too advanced for many developers and creates the potential for some nasty bugs. Maybe in the future it is something to add, as an opt-in rather than the default. In any case, it would be interesting to compare the performance against code that initiates a read early in the completion handler. -Alan. From Alan.Bateman at Sun.COM Fri Apr 17 01:05:11 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Fri, 17 Apr 2009 09:05:11 +0100 Subject: NIO.2 and GC activity In-Reply-To: <007801c9befc$207c1a90$61744fb0$@net> References: <49E4AC86.6080607@sun.com> <000c01c9bd24$e4865150$ad92f3f0$@net> <49E4C894.4090305@sun.com> <001c01c9bd2b$c4653420$4d2f9c60$@net> <49E4DA26.2080905@sun.com> <000901c9bd38$aa7ce5a0$ff76b0e0$@net> <49E7554A.5090809@sun.com> <007801c9befc$207c1a90$61744fb0$@net> Message-ID: <49E83837.10401@sun.com> Leon Finker wrote: > I agree there is no advantage for the writes, and I don't advocate it. On > the reads I disagree that the order can't be guaranteed or re-sequenced on > completion/failure. One is always able to re-establish the original order of > reads that were scheduled on their completion or failure simply by > associating the counter with the scheduled reads and then re-establishing > the sequence. Yes, one has to be aware that if they schedule a read on non > I/O windows pool thread and thread immediately exits, the I/O request will > be canceled. But this applies to current NIO.2 implementation even with non > simultaneous reads. Our implementation never initiates I/O operations on non-pooled threads. Instead, if a non-pooled thread invokes read/write it will be queued (via a message posted to the completion port) so that it is initiated on a pooled thread. That is where the ordering becomes undefined. Another case is where a pool thread associated with one group initiates an I/O operation on a channel associated with a different group. > I agree the code is somewhat involved, and one has to > deal with concurrency, but it's an option for core communication framework > libraries out there. This optimization is not for every case, it's for > specific communication/protocol patterns. So, I'm not sure if comparison > will be fair. It's just a suggestion based on experience. If there is no > need for this from the community, then you're right it can be added later. > Yes, this is something for later once people have more experience using the API. -Alan. From leonfin at optonline.net Fri Apr 17 07:00:17 2009 From: leonfin at optonline.net (Leon Finker) Date: Fri, 17 Apr 2009 10:00:17 -0400 Subject: NIO.2 and GC activity In-Reply-To: <49E83837.10401@sun.com> References: <49E4AC86.6080607@sun.com> <000c01c9bd24$e4865150$ad92f3f0$@net> <49E4C894.4090305@sun.com> <001c01c9bd2b$c4653420$4d2f9c60$@net> <49E4DA26.2080905@sun.com> <000901c9bd38$aa7ce5a0$ff76b0e0$@net> <49E7554A.5090809@sun.com> <007801c9befc$207c1a90$61744fb0$@net> <49E83837.10401@sun.com> Message-ID: <000601c9bf64$d6a6a4d0$83f3ee70$@net> I should have made the last sentence a question because I didn't know the implementation. (i.e.: But doesn't this apply to current NIO.2 implementation even with non simultaneous reads?) Sorry about that. Thanks for the answer! (the paragraph below only applies if multiple simultaneous reads on same socket are allowed) Just one final note, even if reads were not queued to completion port for non pooled thread and different groups didn't initiate the reads, there would still be an ordering issue to deal with. If one called WSARecv() N times in succession for the same socket, they are guaranteed to complete in the order we submitted them. However, undefined ordering is introduced when completion notifications are actually done (due to regular thread scheduling) because N number of threads are servicing the completion port. So, one would have to deal with out of order issues anyway. Thanks for your answers! -----Original Message----- From: Alan.Bateman at Sun.COM [mailto:Alan.Bateman at Sun.COM] Sent: Friday, April 17, 2009 4:05 AM To: Leon Finker Cc: nio-dev at openjdk.java.net Subject: Re: NIO.2 and GC activity Leon Finker wrote: > I agree there is no advantage for the writes, and I don't advocate it. On > the reads I disagree that the order can't be guaranteed or re-sequenced on > completion/failure. One is always able to re-establish the original order of > reads that were scheduled on their completion or failure simply by > associating the counter with the scheduled reads and then re-establishing > the sequence. Yes, one has to be aware that if they schedule a read on non > I/O windows pool thread and thread immediately exits, the I/O request will > be canceled. But this applies to current NIO.2 implementation even with non > simultaneous reads. Our implementation never initiates I/O operations on non-pooled threads. Instead, if a non-pooled thread invokes read/write it will be queued (via a message posted to the completion port) so that it is initiated on a pooled thread. That is where the ordering becomes undefined. Another case is where a pool thread associated with one group initiates an I/O operation on a channel associated with a different group. From Thomas.Salter at unisys.com Fri Apr 17 07:08:19 2009 From: Thomas.Salter at unisys.com (Salter, Thomas A) Date: Fri, 17 Apr 2009 09:08:19 -0500 Subject: NIO.2 and GC activity In-Reply-To: <49E83837.10401@sun.com> References: <49 E4AC86.6080607@sun.com> <000c01c9bd24$e4865150$ad92f3f0$@net><49E4C894.4090305@sun.com> <001c01c9bd2b$c4653420$4d2f9c60$@net><49E4DA26.2080905@sun.com> <000901c9bd38$aa7ce5a0$ff76b0e0$@net><49E7554A.5090809@sun.com> <007801c9befc$207c1a90$61744fb0$@net> <49E83837.10401@sun.com> Message-ID: I couldn't find any MSDN reference that guarantees the order in which multiple pending receives would complete. Do they promise to satisfy requests in the order they are issued? Do they guarantee to post completion status in the order the data is received? I would certainly want to see this spelled out in Windows documentation before writing an application that depended on a particular order. The cited MSDN Magazine article is 9 years and 2 OS releases old and it only makes a passing reference to multiple pending receives with no guidance on how to order the data. -----Original Message----- From: nio-dev-bounces at openjdk.java.net [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of Alan Bateman Sent: Friday, April 17, 2009 4:05 AM To: Leon Finker Cc: nio-dev at openjdk.java.net Subject: Re: NIO.2 and GC activity Leon Finker wrote: > I agree there is no advantage for the writes, and I don't advocate it. > On the reads I disagree that the order can't be guaranteed or > re-sequenced on completion/failure. One is always able to re-establish > the original order of reads that were scheduled on their completion or > failure simply by associating the counter with the scheduled reads and > then re-establishing the sequence. Yes, one has to be aware that if > they schedule a read on non I/O windows pool thread and thread > immediately exits, the I/O request will be canceled. But this applies > to current NIO.2 implementation even with non simultaneous reads. Our implementation never initiates I/O operations on non-pooled threads. Instead, if a non-pooled thread invokes read/write it will be queued (via a message posted to the completion port) so that it is initiated on a pooled thread. That is where the ordering becomes undefined. Another case is where a pool thread associated with one group initiates an I/O operation on a channel associated with a different group. > I agree the code is somewhat involved, and one has to deal with > concurrency, but it's an option for core communication framework > libraries out there. This optimization is not for every case, it's for > specific communication/protocol patterns. So, I'm not sure if > comparison will be fair. It's just a suggestion based on experience. > If there is no need for this from the community, then you're right it can be added later. > Yes, this is something for later once people have more experience using the API. -Alan. From Alan.Bateman at Sun.COM Fri Apr 17 07:27:28 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Fri, 17 Apr 2009 15:27:28 +0100 Subject: NIO.2 and GC activity In-Reply-To: References: <"49 E4AC86.6080607"@sun.com> <000c01c9bd24$e4865150$ad92f3f0$@net> <49E4C894.4090305@sun.com> <001c01c9bd2b$c4653420$4d2f9c60$@net> <49E4DA26.2080905@sun.com> <000901c9bd38$aa7ce5a0$ff76b0e0$@net> <49E7554A.5090809@sun.com> <007801c9befc$207c1a90$61744fb0$@net> <49E83837.10401@sun.com> Message-ID: <49E891D0.4000803@sun.com> Salter, Thomas A wrote: > I couldn't find any MSDN reference that guarantees the order in which multiple pending receives would complete. Do they promise to satisfy requests in the order they are issued? That is a good question as it would have to be guaranteed for anyone to rely on this. > Do they guarantee to post completion status in the order the data is received? We don't require this. Even if the notification ordering is guaranteed it isn't useful because the completion notifications are handled concurrently. FYI, in our implementation we simply map the address of the OVERLAPPED structure back to the request. > I would certainly want to see this spelled out in Windows documentation before writing an application that depended on a particular order. Absolutely! -Alan. From Alan.Bateman at Sun.COM Fri Apr 17 07:37:01 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Fri, 17 Apr 2009 15:37:01 +0100 Subject: NIO.2 and GC activity In-Reply-To: <000601c9bf64$d6a6a4d0$83f3ee70$@net> References: <49E4AC86.6080607@sun.com> <000c01c9bd24$e4865150$ad92f3f0$@net> <49E4C894.4090305@sun.com> <001c01c9bd2b$c4653420$4d2f9c60$@net> <49E4DA26.2080905@sun.com> <000901c9bd38$aa7ce5a0$ff76b0e0$@net> <49E7554A.5090809@sun.com> <007801c9befc$207c1a90$61744fb0$@net> <49E83837.10401@sun.com> <000601c9bf64$d6a6a4d0$83f3ee70$@net> Message-ID: <49E8940D.6050900@sun.com> Leon Finker wrote: > : > Just one final note, even if reads were not queued to completion port for > non pooled thread and different groups didn't initiate the reads, there > would still be an ordering issue to deal with. If one called WSARecv() N > times in succession for the same socket, they are guaranteed to complete in > the order we submitted them. However, undefined ordering is introduced when > completion notifications are actually done (due to regular thread > scheduling) because N number of threads are servicing the completion port. > So, one would have to deal with out of order issues anyway. Thanks for your > answers! > Right, if this were allowed, it turns into a resequencing problem for the application to deal with. In this API the attachment object could be used for such purposes. -Alan. From leonfin at optonline.net Fri Apr 17 07:53:51 2009 From: leonfin at optonline.net (Leon Finker) Date: Fri, 17 Apr 2009 10:53:51 -0400 Subject: NIO.2 and GC activity In-Reply-To: References: <"49 E4AC86.6080607"@sun.com> <000c01c9bd24$e4865150$ad92f3f0$@net> <49E4C894.4090305@sun.com> <001c01c9bd2b$c4653420$4d2f9c60$@net> <49E4DA26.2080905@sun.com> <000901c9bd38$aa7ce5a0$ff76b0e0$@net> <49E7554A.5090809@sun.com> <007801c9befc$207c1a90$61744fb0$@net> <49E83837.10401@sun.com> Message-ID: <000a01c9bf6c$528cc460$f7a64d20$@net> That's the whole issue: the order of multiple simultaneous read completions on the same socket or file *are not guaranteed* when it ends up in the user code. Even in MSDN for WSARecv they say that order is not guaranteed in this case. But they also do not prevent/forbid programmers from doing this. They even recommend this in Microsoft books (i.e.: Concurrent Programming on Windows, and I can't remember all the other references where I learned this). The order can be re-sequenced by programmer on read completions by first associating a sequence number with scheduled reads. This has been done for high throughput servers. I don't know if public source reference exists or not. Having multiple outstanding requests on socket/file can increase utilization, leading to overall speedup. For example, multiple outstanding disk I/Os can allow the I/O subsystem to optimize the movement of the hard disk arm to reduce seek time (adjacent reads can be merged by file manager, reads can be sorted by location on disk, etc). In sockets case, it can allow to skip extra buffering in driver/winsock2 and write directly to user supplied buffer as data is ready thus reducing delays for real-time servers (market data). -----Original Message----- From: nio-dev-bounces at openjdk.java.net [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of Salter, Thomas A Sent: Friday, April 17, 2009 10:08 AM To: nio-dev at openjdk.java.net Subject: RE: NIO.2 and GC activity I couldn't find any MSDN reference that guarantees the order in which multiple pending receives would complete. Do they promise to satisfy requests in the order they are issued? Do they guarantee to post completion status in the order the data is received? I would certainly want to see this spelled out in Windows documentation before writing an application that depended on a particular order. The cited MSDN Magazine article is 9 years and 2 OS releases old and it only makes a passing reference to multiple pending receives with no guidance on how to order the data. -----Original Message----- From: nio-dev-bounces at openjdk.java.net [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of Alan Bateman Sent: Friday, April 17, 2009 4:05 AM To: Leon Finker Cc: nio-dev at openjdk.java.net Subject: Re: NIO.2 and GC activity Leon Finker wrote: > I agree there is no advantage for the writes, and I don't advocate it. > On the reads I disagree that the order can't be guaranteed or > re-sequenced on completion/failure. One is always able to re-establish > the original order of reads that were scheduled on their completion or > failure simply by associating the counter with the scheduled reads and > then re-establishing the sequence. Yes, one has to be aware that if > they schedule a read on non I/O windows pool thread and thread > immediately exits, the I/O request will be canceled. But this applies > to current NIO.2 implementation even with non simultaneous reads. Our implementation never initiates I/O operations on non-pooled threads. Instead, if a non-pooled thread invokes read/write it will be queued (via a message posted to the completion port) so that it is initiated on a pooled thread. That is where the ordering becomes undefined. Another case is where a pool thread associated with one group initiates an I/O operation on a channel associated with a different group. > I agree the code is somewhat involved, and one has to deal with > concurrency, but it's an option for core communication framework > libraries out there. This optimization is not for every case, it's for > specific communication/protocol patterns. So, I'm not sure if > comparison will be fair. It's just a suggestion based on experience. > If there is no need for this from the community, then you're right it can be added later. > Yes, this is something for later once people have more experience using the API. -Alan. From forax at univ-mlv.fr Sun Apr 19 16:50:38 2009 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Mon, 20 Apr 2009 01:50:38 +0200 Subject: Async channels Message-ID: <49EBB8CE.50305@univ-mlv.fr> Hi Alan, hi all, I have some questions about channels and their implementations. (I use the jdk7 workspace so sorry if it's already fixed). 1) Why SocketChannel.getRemoteAddress()/Async...Channel.getRemoteAddress() and Socket.getSocketRemoteAddress() doesn't have the same semantics ? getSocketRemoteAddress() returns the remote address even if the connection is closed, getRemoteAddress() throws a IOException in that case. 2) In the doc of AsynchronousChannel there is a small typo: 
  • Future<V> operation(... A attachment,
    {@link CompletionHandler}<V,? super A> handler)
    • should be 
  • Future<V> operation(... A attachment,
    {@link CompletionHandler}<V,? super A> handler)
    • a semicolon is missing here -----^ 3) The doc of AsynchronousChannel is not crystal clear for methods read() or write() that doesn't take a completion handler in case of IOException. "In the first form, the methods defined by the {@link Future Future} interface may be used to check if the operation has completed, wait for its completion, and to retrieve the result." Is future.get() will throw a ExecutionException that will wrap the corresponding IOException ? 4) Why CompletionHandler.failed() take a Throwable and not a IOException ? 5) As specified in the doc of AsynchronousChannel about cancelation, "further attempts to initiate a {@code read} operation causes an unspecified runtime exception to be thrown". Why a runtime exception and not a call to the completion handler ? 6) Is it possible to rename all Asynchronous* to Async*. I will *really* increase the readability of the types. 7) UnixAsynchronousSocketChannelImpl.onCancel() seems to have a copy&paste bug: public void onCancel(PendingFuture task) { if (task.getContext() == OpType.CONNECT) killConnect(); if (task.getContext() == OpType.READ) killConnect(); // <----------------------- killReading() if (task.getContext() == OpType.WRITE) killConnect(); // <----------------------- killWriting() } 8) Why AsyncFileChannel.open(...) take an ExecutorService and not a AsyncChannelGroup ? regards, R?mi From Alan.Bateman at Sun.COM Mon Apr 20 01:05:29 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Mon, 20 Apr 2009 09:05:29 +0100 Subject: Async channels In-Reply-To: <49EBB8CE.50305@univ-mlv.fr> References: <49EBB8CE.50305@univ-mlv.fr> Message-ID: <49EC2CC9.50507@sun.com> R?mi Forax wrote: > Hi Alan, hi all, > I have some questions about channels and their implementations. > (I use the jdk7 workspace so sorry if it's already fixed). > > 1) Why > SocketChannel.getRemoteAddress()/Async...Channel.getRemoteAddress() > and Socket.getSocketRemoteAddress() doesn't have the same semantics ? > > getSocketRemoteAddress() returns the remote address even > if the connection is closed, > getRemoteAddress() throws a IOException in that case. The getRemoteAddress method throws ClosedChannelException if invoked on a closed channel. The reason for the IOException is to allow an implementation retrieve the address only when required. Socket#getSocketRemoteAddress, on the other hand, didn't originally specify this case. There were a bunch of other "close socket" cases in the java.net package that were also undefined. Early in jdk7 the javadoc was updated to fix these up these cases; existing behavior had to be specified to avoid breaking code that may have been relying on it. > > 2) In the doc of AsynchronousChannel there is a small typo: > 
  • Future<V> operation(... A attachment,
>    {@link CompletionHandler}<V,? super A> handler)
    • > should be > 
  • Future<V> operation(... A attachment,
>    {@link CompletionHandler}<V,? super A> handler)
    • > a semicolon is missing here -----^ Thanks. > > 3) The doc of AsynchronousChannel is not crystal clear for methods > read() or write() that doesn't take a completion handler > in case of IOException. > > "In the first form, the methods defined by the {@link Future Future} > interface may be used to check if the operation has completed, > wait for its > completion, and to retrieve the result." > > Is future.get() will throw a ExecutionException that will wrap the > corresponding > IOException ? Yes, it will wrap the IOException (or maybe SecurityException in some cases). I've made a note to myself to see if this can be made clearer. > > 4) Why CompletionHandler.failed() take a Throwable and not a > IOException ? It may be something other than IOException. The accept and receive operations for example may fail with a SecurityException for example. > > 5) As specified in the doc of AsynchronousChannel about cancelation, > "further attempts to initiate a {@code read} operation causes an > unspecified runtime exception to be thrown". > Why a runtime exception and not a call to the completion handler ? Yes, this bug can be caught before the I/O operation is initiated so the IllegalStateException or other runtime exception can be thrown then. > > 6) Is it possible to rename all Asynchronous* to Async*. > I will *really* increase the readability of the types. This package doesn't use abbreviations in class names. Okay, the new channels have longish class names but no big deal. > > 7) UnixAsynchronousSocketChannelImpl.onCancel() seems to have a > copy&paste bug: > > public void onCancel(PendingFuture task) { > if (task.getContext() == OpType.CONNECT) > killConnect(); > if (task.getContext() == OpType.READ) > killConnect(); // <----------------------- killReading() > if (task.getContext() == OpType.WRITE) > killConnect(); // <----------------------- killWriting() > } Thanks; amazingly this this bug doesn't have much impact. > > > 8) Why AsyncFileChannel.open(...) take an ExecutorService and not a > AsyncChannelGroup ? File I/O is very different to socket I/O and it may not be feasible for an implementation to share resources or thread pools between file and network oriented channels. Think of an AsynchronousFileChannel as a group of one containing one channel. The exception is when you use the simpler open method as that will use the thread pool associated with the default group. -Alan. From forax at univ-mlv.fr Mon Apr 20 03:28:29 2009 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Mon, 20 Apr 2009 12:28:29 +0200 Subject: Async channels In-Reply-To: <49EC2CC9.50507@sun.com> References: <49EBB8CE.50305@univ-mlv.fr> <49EC2CC9.50507@sun.com> Message-ID: <49EC4E4D.6040500@univ-mlv.fr> Alan Bateman a ?crit : > R?mi Forax wrote: >> Hi Alan, hi all, >> I have some questions about channels and their implementations. >> (I use the jdk7 workspace so sorry if it's already fixed). >> >> 1) Why >> SocketChannel.getRemoteAddress()/Async...Channel.getRemoteAddress() >> and Socket.getSocketRemoteAddress() doesn't have the same semantics ? >> >> getSocketRemoteAddress() returns the remote address even >> if the connection is closed, >> getRemoteAddress() throws a IOException in that case. > The getRemoteAddress method throws ClosedChannelException if invoked > on a closed channel. The reason for the IOException is to allow an > implementation retrieve the address only when required. > Socket#getSocketRemoteAddress, on the other hand, didn't originally > specify this case. There were a bunch of other "close socket" cases in > the java.net package that were also undefined. Early in jdk7 the > javadoc was updated to fix these up these cases; existing behavior had > to be specified to avoid breaking code that may have been relying on it. As a server developer; I prefer the semantics of getSocketRemoteAddress() (as specified now) to the one specified for getRemoteAddress(). At least half of the usages (and perhaps more) of getRemoteAddress() will be for logging purpose, having the remote address persistent even if the channel is closed seems important to me, for example, if you want to log that the channel is closed :) [...] > >> >> 3) The doc of AsynchronousChannel is not crystal clear for methods >> read() or write() that doesn't take a completion handler >> in case of IOException. >> >> "In the first form, the methods defined by the {@link Future Future} >> interface may be used to check if the operation has completed, >> wait for its >> completion, and to retrieve the result." >> >> Is future.get() will throw a ExecutionException that will wrap >> the corresponding >> IOException ? > Yes, it will wrap the IOException (or maybe SecurityException in some > cases). I've made a note to myself to see if this can be made clearer. See below ... > >> >> 4) Why CompletionHandler.failed() take a Throwable and not a >> IOException ? > It may be something other than IOException. The accept and receive > operations for example may fail with a SecurityException for example. Argh, I was hoping that SecurityException like any runtime exceptions is thrown synchronously. I don't see why they have to be delayed. As a developer, runtime exception means you don't correctly understand the spec so they have to be thrown sooner than later (if possible). > >> >> 5) As specified in the doc of AsynchronousChannel about cancelation, >> "further attempts to initiate a {@code read} operation causes an >> unspecified runtime exception to be thrown". >> Why a runtime exception and not a call to the completion handler ? > Yes, this bug can be caught before the I/O operation is initiated so > the IllegalStateException or other runtime exception can be thrown then. I think I don't like the fact that runtime exceptions can be: 1) thrown by read/write/accept/connect or 2) catch by the completion handler without be able to understand/predict why. > >> >> 6) Is it possible to rename all Asynchronous* to Async*. >> I will *really* increase the readability of the types. > This package doesn't use abbreviations in class names. Okay, the new > channels have longish class names but no big deal. It's a big deal, if you take a look to the javadoc of the package (the bottom-left frame) you only see a bunch of classes starting with Asynchronous* even if they are like selectors reserved for advanced usages. > >> >> 7) UnixAsynchronousSocketChannelImpl.onCancel() seems to have a >> copy&paste bug: >> >> public void onCancel(PendingFuture task) { >> if (task.getContext() == OpType.CONNECT) >> killConnect(); >> if (task.getContext() == OpType.READ) >> killConnect(); // <----------------------- killReading() >> if (task.getContext() == OpType.WRITE) >> killConnect(); // <----------------------- killWriting() >> } > Thanks; amazingly this this bug doesn't have much impact. Perhaps a test case is missing ? > >> >> >> 8) Why AsyncFileChannel.open(...) take an ExecutorService and not a >> AsyncChannelGroup ? > File I/O is very different to socket I/O and it may not be feasible > for an implementation to share resources or thread pools between file > and network oriented channels. Think of an AsynchronousFileChannel as > a group of one containing one channel. The exception is when you use > the simpler open method as that will use the thread pool associated > with the default group. Taking an Executor an not a AsyncChannelGroup doesn't prevent sharing. Currently, you can write: ExecutorService executor=Executors.newCachedThreadPool(); AsynchronousChannelGroup channelGroup = AsynchronousChannelGroup.withThreadPool(executor); AsynchronousServerSocketChannel serverSocketChannel = AsynchronousServerSocketChannel.open(channelGroup); AsynchronousFileChannel fileChannel = AsynchronousFileChannel.open(path, openOptions, executor); Another questions, currently the linux code for asynchronous I/O is based on poll like the selectors. - On linux, is it more inetrresting to use async calls or selectors beside the fact that code with async calls seem simplier to write (at least to me) ? - Is there a plan to use really async I/O on linux ? > > -Alan. > R?mi From Alan.Bateman at Sun.COM Mon Apr 20 05:06:07 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Mon, 20 Apr 2009 13:06:07 +0100 Subject: Async channels In-Reply-To: <49EC4E4D.6040500@univ-mlv.fr> References: <49EBB8CE.50305@univ-mlv.fr> <49EC2CC9.50507@sun.com> <49EC4E4D.6040500@univ-mlv.fr> Message-ID: <49EC652F.3080100@sun.com> R?mi Forax wrote: > As a server developer; I prefer the semantics of > getSocketRemoteAddress() (as specified now) > to the one specified for getRemoteAddress(). > At least half of the usages (and perhaps more) of getRemoteAddress() > will be for logging purpose, > having the remote address persistent even if the channel is closed > seems important to me, > for example, if you want to log that the channel is closed :) Just so we're clear, this relates to the channel state (not the connection state). If the connection terminates or is closed by the peer then getRemoteAddress will continue to return the peer address until you close the channel. All I/O operations on channels throw ClosedChannelException is invoked on a closed channel. On a technical note, there are some operating systems where getpeername/equivalent fails once the connection has closed; ports to those platforms will have no choice but to attempt to retrieve (and cache) the peer information when the connection is established (or accepted). As to the logging case - if you asynchronously close a channel with outstanding I/O operations and the completion handler attempts to log information (that includes the peer address that is no longer valid) then handler will need code to deal with this case. > Argh, > I was hoping that SecurityException like any runtime exceptions is > thrown synchronously. > I don't see why they have to be delayed. > As a developer, runtime exception means you don't correctly > understand the spec > so they have to be thrown sooner than later (if possible). The permission checks required to accept a connection or receive a datagram cannot be done at the time that the I/O operations are initiated. > : > I think I don't like the fact that runtime exceptions can be: > 1) thrown by read/write/accept/connect or > 2) catch by the completion handler > without be able to understand/predict why. The only runtime exception that the completion handler has to deal with is a SecurityManager and only then if there is a security manager and only for specific I/O operations. > : > Perhaps a test case is missing ? Yes, the current tests don't check that you can continue to write after cancelling a read (and vis versa). I'll add these tests. Thanks again for finding this one. > Taking an Executor an not a AsyncChannelGroup doesn't prevent sharing. > Currently, you can write: > > ExecutorService executor=Executors.newCachedThreadPool(); > AsynchronousChannelGroup channelGroup = > AsynchronousChannelGroup.withThreadPool(executor); > AsynchronousServerSocketChannel serverSocketChannel = > AsynchronousServerSocketChannel.open(channelGroup); > AsynchronousFileChannel fileChannel = > AsynchronousFileChannel.open(path, openOptions, executor); You could but this goes against the warning in the spec that the thread pool is intended to be used exclusively by the resulting group. > > > Another questions, currently the linux code for asynchronous I/O is > based on poll > like the selectors. Linux is the one case where we use the same I/O facility as we do in the Selector. Selector uses is in simple level-triggered mode like poll. Asynchronous I/O uses it in one-shot mode. The useful thing about the new API is that is it mappable to broad range of I/O facilities. > - On linux, is it more inetrresting to use async calls or selectors > beside the fact that > code with async calls seem simplier to write (at least to me) ? Yes, the new API is easier but there are many things to size up (Reactor vs. Proactor design for example). Jean-Francois might want to jump in here as he has seeing better performance with his Grizzly port to the new API. > - Is there a plan to use really async I/O on linux ? If Linux were get such a facility then we would use it. In the interim we might port to Kevents if that goes into the kernel. -Alan. From forax at univ-mlv.fr Tue Apr 21 14:04:49 2009 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Tue, 21 Apr 2009 23:04:49 +0200 Subject: Async channels In-Reply-To: <49EC652F.3080100@sun.com> References: <49EBB8CE.50305@univ-mlv.fr> <49EC2CC9.50507@sun.com> <49EC4E4D.6040500@univ-mlv.fr> <49EC652F.3080100@sun.com> Message-ID: <49EE34F1.9020305@univ-mlv.fr> Alan Bateman a ?crit : > R?mi Forax wrote: >> As a server developer; I prefer the semantics of >> getSocketRemoteAddress() (as specified now) >> to the one specified for getRemoteAddress(). >> At least half of the usages (and perhaps more) of getRemoteAddress() >> will be for logging purpose, >> having the remote address persistent even if the channel is closed >> seems important to me, >> for example, if you want to log that the channel is closed :) > Just so we're clear, this relates to the channel state (not the > connection state). If the connection terminates or is closed by the > peer then getRemoteAddress will continue to return the peer address > until you close the channel. All I/O operations on channels throw > ClosedChannelException is invoked on a closed channel. On a technical > note, there are some operating systems where getpeername/equivalent > fails once the connection has closed; ports to those platforms will > have no choice but to attempt to retrieve (and cache) the peer > information when the connection is established (or accepted). That why I prefer the semantics of getSocketRemoteAddress(), because you know that this is just a getter without native calls behind. Anyway, you're the boss. > As to the logging case - if you asynchronously close a channel with > outstanding I/O operations and the completion handler attempts to log > information (that includes the peer address that is no longer valid) > then handler will need code to deal with this case. > > >> Argh, >> I was hoping that SecurityException like any runtime exceptions is >> thrown synchronously. >> I don't see why they have to be delayed. >> As a developer, runtime exception means you don't correctly >> understand the spec >> so they have to be thrown sooner than later (if possible). > The permission checks required to accept a connection or receive a > datagram cannot be done at the time that the I/O operations are > initiated. Ok. > > >> : >> I think I don't like the fact that runtime exceptions can be: >> 1) thrown by read/write/accept/connect or >> 2) catch by the completion handler >> without be able to understand/predict why. > The only runtime exception that the completion handler has to deal > with is a SecurityManager and only then if there is a security manager > and only for specific I/O operations. Ok, so the doc for completion handler can be enhanced to says that the Throwable (why not use Exception instead, btw) is the common super type of IOException and SecurityException. > >> >> >> Another questions, currently the linux code for asynchronous I/O is >> based on poll >> like the selectors. > Linux is the one case where we use the same I/O facility as we do in > the Selector. Selector uses is in simple level-triggered mode like > poll. Asynchronous I/O uses it in one-shot mode. The useful thing > about the new API is that is it mappable to broad range of I/O > facilities. I have crawle the doc of epoll: one-shot mode is a special case of edge-trigger mode, so it's not the same mechanism that the one use to implement selector but it use epoll too. > >> - On linux, is it more inetrresting to use async calls or selectors >> beside the fact that >> code with async calls seem simplier to write (at least to me) ? > Yes, the new API is easier but there are many things to size up > (Reactor vs. Proactor design for example). The devil is in the details... > Jean-Francois might want to jump in here as he has seeing better > performance with his Grizzly port to the new API. I think I can wait the PDF of the JavaOne presentation :) > >> - Is there a plan to use really async I/O on linux ? > If Linux were get such a facility then we would use it. In the interim > we might port to Kevents if that goes into the kernel. Crawling the web, I have found that Kevents dev was closed, without a replacement of the mechanism to get AIO notification :) > > -Alan. R?mi From forax at univ-mlv.fr Tue Apr 21 14:29:03 2009 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Tue, 21 Apr 2009 23:29:03 +0200 Subject: Async channels In-Reply-To: <49EC652F.3080100@sun.com> References: <49EBB8CE.50305@univ-mlv.fr> <49EC2CC9.50507@sun.com> <49EC4E4D.6040500@univ-mlv.fr> <49EC652F.3080100@sun.com> Message-ID: <49EE3A9F.4070102@univ-mlv.fr> Alan Bateman a ?crit : [...] >> Taking an Executor an not a AsyncChannelGroup doesn't prevent sharing. >> Currently, you can write: >> >> ExecutorService executor=Executors.newCachedThreadPool(); >> AsynchronousChannelGroup channelGroup = >> AsynchronousChannelGroup.withThreadPool(executor); >> AsynchronousServerSocketChannel serverSocketChannel = >> AsynchronousServerSocketChannel.open(channelGroup); >> AsynchronousFileChannel fileChannel = >> AsynchronousFileChannel.open(path, openOptions, executor); > You could but this goes against the warning in the spec that the > thread pool is intended to be used exclusively by the resulting group. I think I don't understand the difference between an Executor and a AsyncChannelGroup. Furthermore, there is currently no way to use the same Executor for more that one AsyncFileChannel because closing the file channel will shutdown the executor. I don't understand the purpose of this restriction. [...] > > > -Alan. R?mi From forax at univ-mlv.fr Tue Apr 21 15:36:08 2009 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Wed, 22 Apr 2009 00:36:08 +0200 Subject: CompletionHandler should take the channel too Message-ID: <49EE4A58.8050609@univ-mlv.fr> Hi Alan, hi all, Writing a stateless completion handler as suggested in the doc is not that easy because the completion handler doesn't provide a way to get the async channel on which the async operation was called. This is not that stupid, if you take a look to the selector's SelectionKey, it provides the channel. So I propose to add a new type parameter to the CompletionHandler: public interface CompletionHandler { void completed(V result, C asyncChannel, A attachment); void failed(Throwable exc, C asyncChannel, A attachment); void cancelled(C asyncChannel, A attachment); } and read on an AsyncSocketChannel becomes: public abstract Future read(ByteBuffer dst, long timeout, TimeUnit unit, A attachment, CompletionHandler handler); The current implementation already store the async channel in AbstractFuture, so it doesn't seem to be a great deal (perhaps a small nightmare if you want to parameterize AbstractFuture with the type of AsyncChannel, but I'm not sure it worth it). R?mi From Alan.Bateman at Sun.COM Wed Apr 22 00:23:13 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Wed, 22 Apr 2009 08:23:13 +0100 Subject: CompletionHandler should take the channel too In-Reply-To: <49EE4A58.8050609@univ-mlv.fr> References: <49EE4A58.8050609@univ-mlv.fr> Message-ID: <49EEC5E1.30000@sun.com> R?mi Forax wrote: > Hi Alan, hi all, > Writing a stateless completion handler as suggested in the doc is not > that easy because the completion handler doesn't provide a way to get > the async channel on which the async operation was called. When consuming the result you will typically need the channel, buffer(s), SSLSession and maybe other context. This is the reason for the attachment object so that you can attach arbitrary context (including the channel) to the I/O operation and have it available when consuming the result. So while I agree it is important to have a reference to the channel, it isn't sufficient. If there are cases where the channel is the only context required then you can make the channel available as the attachment object. -Alan. From Alan.Bateman at Sun.COM Wed Apr 22 01:34:19 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Wed, 22 Apr 2009 09:34:19 +0100 Subject: Async channels In-Reply-To: <49EE3A9F.4070102@univ-mlv.fr> References: <49EBB8CE.50305@univ-mlv.fr> <49EC2CC9.50507@sun.com> <49EC4E4D.6040500@univ-mlv.fr> <49EC652F.3080100@sun.com> <49EE3A9F.4070102@univ-mlv.fr> Message-ID: <49EED68B.6030009@sun.com> R?mi Forax wrote: > : > I think I don't understand the difference between an Executor and a > AsyncChannelGroup. An AsynchronousChannelGroup encapsulates the mechanism for handling I/O events so that the mechanism can be shared by many channels (think I/O port or equivalent). It has a thread pool for handling I/O events and dispatching to the completion handlers for I/O operations invoked on channels in the group. Combining I/O with thread pools can be a complex area. In this API you create a group with a thread pool and that decides the configuration. The group takes ownership of the thread pool so you shouldn't submit your own tasks directly, attempt to shut it down, or use it for another group. The group defines its own shutdown methods that are similar to ExecutorService except of course that do additional work like releasing resources. > > Furthermore, there is currently no way to use the same Executor for > more that one > AsyncFileChannel because closing the file channel will shutdown the > executor. > I don't understand the purpose of this restriction. The use-cases for AsynchronousFileChannel are typically database-like applications with concurrent I/O operations to different parts of the file. It's quite different to network I/O where you want to share resources between thousands of channels. Also the underlying mechanism can be very different (making it not feasible, in many cases, for a group to support both file and network channels at the same time). If thread identity is not a concern then you can ignore all this and use the simpler open method. Conceptually this means the channel is in the default group and so will share the default group's thread pool which does the right thing. Closing the channel are no side effects. Where thread identity is important then it does require that you create your own thread pool. That thread pool will be used exclusively for the file channel. If there is a big need in the future to be able to share thread pools in such cases then we could update the API but it does introduce complexity that we can avoid for now. -Alan. From Alan.Bateman at Sun.COM Wed Apr 22 01:47:04 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Wed, 22 Apr 2009 09:47:04 +0100 Subject: Async channels In-Reply-To: <49EE34F1.9020305@univ-mlv.fr> References: <49EBB8CE.50305@univ-mlv.fr> <49EC2CC9.50507@sun.com> <49EC4E4D.6040500@univ-mlv.fr> <49EC652F.3080100@sun.com> <49EE34F1.9020305@univ-mlv.fr> Message-ID: <49EED988.5090900@sun.com> R?mi Forax wrote: > : > That why I prefer the semantics of getSocketRemoteAddress(), because > you know that > this is just a getter without native calls behind. > Anyway, you're the boss. The getSocketRemoteAddress semantics is that it requires the implementation to get and cache the remote address before the socket is closed. It is more efficient to only retrieve it if required. Furthermore, all I/O methods in the channels package throw ClosedChannelException when invoked on a closed channel so we would be introducing an small inconsistency. Aside from possible logging, I cannot think of cases where the remote address is useful after it is no longer valid. > > Ok, so the doc for completion handler can be enhanced to says that the > Throwable (why not use Exception instead, btw) > is the common super type of IOException and SecurityException. It could for the current usage although we do use it to propagate internal errors [should never happen :-) ]. Keeping it as Throwable does provide for some flexibility if used for other purposes and usages in the future. > > Crawling the web, I have found that Kevents dev was closed, > without a replacement of the mechanism to get AIO notification :) As I said, the API is mappable to quite a range of I/O facilities so we can re-target as required if a better mechanism becomes available (we've done that a few times with Selector too, like it was originally poll on Linux but we switched to epoll). -Alan. From leonfin at optonline.net Wed Apr 22 13:16:04 2009 From: leonfin at optonline.net (Leon Finker) Date: Wed, 22 Apr 2009 16:16:04 -0400 Subject: pending read/failed callback/IOException Message-ID: <001e01c9c387$2a300080$7e900180$@net> I'm just starting with nio.2. I noticed that when closing the channel, unless AsynchronousSocketChannel.shutdownOutput is called, one would get the following exception object in the failed() callback with pending read(): java.io.IOException: The specified network name is no longer available. at sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source) at sun.nio.ch.Iocp.access$700(Unknown Source) at sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source) at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Calling just .close or .shutdownInput followed by .close still produces the exception. Calling AsynchronousSocketChannel.shutdownOutput followed by close, will not produce the exception. I'm on build b55. Doesn't seem correct behavior to me. Thanks! From Alan.Bateman at Sun.COM Wed Apr 22 13:45:26 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Wed, 22 Apr 2009 21:45:26 +0100 Subject: pending read/failed callback/IOException In-Reply-To: <001e01c9c387$2a300080$7e900180$@net> References: <001e01c9c387$2a300080$7e900180$@net> Message-ID: <49EF81E6.1010304@sun.com> Leon Finker wrote: > I'm just starting with nio.2. > > I noticed that when closing the channel, unless > AsynchronousSocketChannel.shutdownOutput is called, one would get the > following exception object in the failed() callback with pending read(): > > java.io.IOException: The specified network name is no longer available. > > at sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source) > at sun.nio.ch.Iocp.access$700(Unknown Source) > at sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source) > at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) > at java.lang.Thread.run(Unknown Source) > > Calling just .close or .shutdownInput followed by .close still produces the > exception. Calling AsynchronousSocketChannel.shutdownOutput followed by > close, will not produce the exception. I'm on build b55. Doesn't seem > correct behavior to me. Thanks! > > This stack trace doesn't look like an asynchronous close but rather the I/O exception when the peer terminates the connection abruptly. I think you'll need to describe things in more detail to diagnose it (or a small test case would be better). -Alan. From leonfin at optonline.net Wed Apr 22 13:59:03 2009 From: leonfin at optonline.net (Leon Finker) Date: Wed, 22 Apr 2009 16:59:03 -0400 Subject: pending read/failed callback/IOException In-Reply-To: <49EF81E6.1010304@sun.com> References: <001e01c9c387$2a300080$7e900180$@net> <49EF81E6.1010304@sun.com> Message-ID: <002001c9c38d$2b017290$810457b0$@net> You're correct. This exception has nothing to do with asynchronous close. The client terminates abruptly, and that's when the exception happens. Is this correct behavior? Is it safe to assume that for tcp: 1. -1 result on the asynchronous read callback specifies remote end graceful shutdown (or shutdownInput has been called) 2. IOException will happen in failed() callback when remote end closed connection abruptly -----Original Message----- From: Alan.Bateman at Sun.COM [mailto:Alan.Bateman at Sun.COM] Sent: Wednesday, April 22, 2009 4:45 PM To: Leon Finker Cc: nio-dev at openjdk.java.net Subject: Re: pending read/failed callback/IOException Leon Finker wrote: > I'm just starting with nio.2. > > I noticed that when closing the channel, unless > AsynchronousSocketChannel.shutdownOutput is called, one would get the > following exception object in the failed() callback with pending read(): > > java.io.IOException: The specified network name is no longer available. > > at sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source) > at sun.nio.ch.Iocp.access$700(Unknown Source) > at sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source) > at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) > at java.lang.Thread.run(Unknown Source) > > Calling just .close or .shutdownInput followed by .close still produces the > exception. Calling AsynchronousSocketChannel.shutdownOutput followed by > close, will not produce the exception. I'm on build b55. Doesn't seem > correct behavior to me. Thanks! > > This stack trace doesn't look like an asynchronous close but rather the I/O exception when the peer terminates the connection abruptly. I think you'll need to describe things in more detail to diagnose it (or a small test case would be better). -Alan. From Alan.Bateman at Sun.COM Wed Apr 22 14:06:58 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Wed, 22 Apr 2009 22:06:58 +0100 Subject: pending read/failed callback/IOException In-Reply-To: <002001c9c38d$2b017290$810457b0$@net> References: <001e01c9c387$2a300080$7e900180$@net> <49EF81E6.1010304@sun.com> <002001c9c38d$2b017290$810457b0$@net> Message-ID: <49EF86F2.8070109@sun.com> Leon Finker wrote: > You're correct. This exception has nothing to do with asynchronous close. > The client terminates abruptly, and that's when the exception happens. Is > this correct behavior? Is it safe to assume that for tcp: > 1. -1 result on the asynchronous read callback specifies remote end graceful > shutdown (or shutdownInput has been called) > 2. IOException will happen in failed() callback when remote end closed > connection abruptly > The behavior you observe is expected. That is, -1/EOF to mean that you can reached end of stream (connection closed gracefully). IOException to mean that the read operation failed. -Alan. From forax at univ-mlv.fr Wed Apr 22 14:28:59 2009 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Wed, 22 Apr 2009 23:28:59 +0200 Subject: Async channels In-Reply-To: <49EED68B.6030009@sun.com> References: <49EBB8CE.50305@univ-mlv.fr> <49EC2CC9.50507@sun.com> <49EC4E4D.6040500@univ-mlv.fr> <49EC652F.3080100@sun.com> <49EE3A9F.4070102@univ-mlv.fr> <49EED68B.6030009@sun.com> Message-ID: <49EF8C1B.4030506@univ-mlv.fr> Alan Bateman a ?crit : > R?mi Forax wrote: >> : >> I think I don't understand the difference between an Executor and a >> AsyncChannelGroup. > An AsynchronousChannelGroup encapsulates the mechanism for handling > I/O events so that the mechanism can be shared by many channels (think > I/O port or equivalent). It has a thread pool for handling I/O events > and dispatching to the completion handlers for I/O operations invoked > on channels in the group. Combining I/O with thread pools can be a > complex area. In this API you create a group with a thread pool and > that decides the configuration. The group takes ownership of the > thread pool so you shouldn't submit your own tasks directly, attempt > to shut it down, or use it for another group. The group defines its > own shutdown methods that are similar to ExecutorService except of > course that do additional work like releasing resources. that why I don't understand why a AsyncChannelGroup (perhaps a different implementation) can't be used as a group for AsyncFileChannels. > >> >> Furthermore, there is currently no way to use the same Executor for >> more that one >> AsyncFileChannel because closing the file channel will shutdown the >> executor. >> I don't understand the purpose of this restriction. > The use-cases for AsynchronousFileChannel are typically database-like > applications with concurrent I/O operations to different parts of the > file. It's quite different to network I/O where you want to share > resources between thousands of channels. Also the underlying mechanism > can be very different (making it not feasible, in many cases, for a > group to support both file and network channels at the same time). If > thread identity is not a concern then you can ignore all this and use > the simpler open method. Conceptually this means the channel is in the > default group and so will share the default group's thread pool which > does the right thing. Closing the channel are no side effects. Where > thread identity is important then it does require that you create your > own thread pool. That thread pool will be used exclusively for the > file channel. If there is a big need in the future to be able to share > thread pools in such cases then we could update the API but it does > introduce complexity that we can avoid for now. complexity == 1 boolean that already exist in the implementation. public AsynchronousFileChannel newAsynchronousFileChannel(Path path, Set options, ExecutorService executor, boolean shared, <--- shutdown on close if not shared ! FileAttribute... attrs) > > -Alan. R?mi From forax at univ-mlv.fr Wed Apr 22 15:18:16 2009 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Thu, 23 Apr 2009 00:18:16 +0200 Subject: CompletionHandler should take the channel too In-Reply-To: <49EEC5E1.30000@sun.com> References: <49EE4A58.8050609@univ-mlv.fr> <49EEC5E1.30000@sun.com> Message-ID: <49EF97A8.2080406@univ-mlv.fr> Alan Bateman a ?crit : > R?mi Forax wrote: >> Hi Alan, hi all, >> Writing a stateless completion handler as suggested in the doc is not >> that easy because the completion handler doesn't provide a way to get >> the async channel on which the async operation was called. > When consuming the result you will typically need the channel, > buffer(s), SSLSession and maybe other context. This is the reason for > the attachment object so that you can attach arbitrary context > (including the channel) to the I/O operation and have it available > when consuming the result. So while I agree it is important to have a > reference to the channel, it isn't sufficient. If there are cases > where the channel is the only context required then you can make the > channel available as the attachment object. > > -Alan. I don't see why I have to create an ephemeral object to store a channel and a buffer if the implementation already have this information but don't export it. Else, there is no way to find the group from an async socket channel, something like: asyncSocketChannel().group().shutdown(); should be useful. R?mi From Alan.Bateman at Sun.COM Thu Apr 23 00:47:27 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Thu, 23 Apr 2009 08:47:27 +0100 Subject: CompletionHandler should take the channel too In-Reply-To: <49EF97A8.2080406@univ-mlv.fr> References: <49EE4A58.8050609@univ-mlv.fr> <49EEC5E1.30000@sun.com> <49EF97A8.2080406@univ-mlv.fr> Message-ID: <49F01D0F.2020102@sun.com> R?mi Forax wrote: > : > I don't see why I have to create an ephemeral object to store a > channel and a buffer > if the implementation already have this information but don't export it. I don't think we can predict all the context that the handler will require when consuming the result. For some cases it may only be the channel and the buffer (or buffers). In other cases it will be the SSLSession or other session information. For non read/write methods may be the channel and something else. > > Else, there is no way to find the group from an async socket channel, > something like: > asyncSocketChannel().group().shutdown(); > should be useful. The reason for this is that we don't want to give out a reference to the default group (because it can never be shutdown). I admit we need to re-visit this but it requires special-casing the default group so that it cannot be shutdown. I don't know if you read Alex and Yevgeny Libman's proposal on filters but that creates the requirement to get a reference to the group so that it can submit its own tasks to the group's thread pool. There is merit in this proposal and the support is in the implementation (but not the API yet, needs further thought). -Alan. From Alan.Bateman at Sun.COM Thu Apr 23 11:34:22 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Thu, 23 Apr 2009 19:34:22 +0100 Subject: Async channels In-Reply-To: <49EF8C1B.4030506@univ-mlv.fr> References: <49EBB8CE.50305@univ-mlv.fr> <49EC2CC9.50507@sun.com> <49EC4E4D.6040500@univ-mlv.fr> <49EC652F.3080100@sun.com> <49EE3A9F.4070102@univ-mlv.fr> <49EED68B.6030009@sun.com> <49EF8C1B.4030506@univ-mlv.fr> Message-ID: <49F0B4AE.4020101@sun.com> R?mi Forax wrote: > : > that why I don't understand why a AsyncChannelGroup (perhaps a > different implementation) > can't be used as a group for AsyncFileChannels. The issue is trying to share resources and a thread pool between potentially two different mechanisms. It may not be an issue in some cases but in others it may be very problematic. For example, if you used a fixed thread pool then an implementation may be forced to divide the pool into two with some threads servicing I/O events for the network channels, and the others for file I/O. One solution would be to prohibit mixing channels in a group but we don't want to overly complicate the API. > : > complexity == 1 boolean that already exist in the implementation. > > public AsynchronousFileChannel newAsynchronousFileChannel(Path path, > Set extends OpenOption> options, > > ExecutorService executor, > boolean > shared, <--- shutdown on close if not > shared ! > > FileAttribute... attrs) That would work except that it might leak resources, consider: ExecutorService pool = ... ch1 = AsynchronousFileChannel.open(... pool, shared=true) ch2 = AsynchronousFileChannel.open(... pool, shared=true) ch1.close ch2.close pool.shutdown The channels are closed and the thread pool is shutdown but the I/O machinery may still exit and may have threads waiting on I/O events. However your suggestion leads to another idea that is worth exploring. The other idea is to simply allow AsynchronousFileChannel share a thread pool and to terminate the I/O machinery when the last channel using the pool is closed. In implementation terms it's just a mapping from pool to a hidden group that is created when a mapping doesn't initially exist. The only downside that comes to mind is that it is inconsistent with how we share resources between network channels. That might not be too much of a concern because this is only something for a small set of cases. -Alan. From forax at univ-mlv.fr Fri Apr 24 13:48:08 2009 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Fri, 24 Apr 2009 22:48:08 +0200 Subject: CompletionHandler should take the channel too In-Reply-To: <49F01D0F.2020102@sun.com> References: <49EE4A58.8050609@univ-mlv.fr> <49EEC5E1.30000@sun.com> <49EF97A8.2080406@univ-mlv.fr> <49F01D0F.2020102@sun.com> Message-ID: <49F22588.2010505@univ-mlv.fr> Alan Bateman a ?crit : > R?mi Forax wrote: >> : >> I don't see why I have to create an ephemeral object to store a >> channel and a buffer >> if the implementation already have this information but don't export it. > I don't think we can predict all the context that the handler will > require when consuming the result. For some cases it may only be the > channel and the buffer (or buffers). In other cases it will be the > SSLSession or other session information. For non read/write methods > may be the channel and something else. You probably misunderstand me, I haven't say that we should remove the attachment but advocate for adding a parameter to all methods of the completion handler to give access to the channel. I can't see a scenario where the channel is not needed. Adding this parameter will free the attachment that can be used for something else, the buffer, the SSLSession, etc. > >> >> Else, there is no way to find the group from an async socket channel, >> something like: >> asyncSocketChannel().group().shutdown(); >> should be useful. > The reason for this is that we don't want to give out a reference to > the default group (because it can never be shutdown). I admit we need > to re-visit this but it requires special-casing the default group so > that it cannot be shutdown. I don't know if you read Alex and Yevgeny > Libman's proposal on filters but that creates the requirement to get a > reference to the group so that it can submit its own tasks to the > group's thread pool. There is merit in this proposal and the support > is in the implementation (but not the API yet, needs further thought). No, I'm not aware of this proposal and was not able find a link to it. Do you have a reference ? > > -Alan. R?mi From Alan.Bateman at Sun.COM Sat Apr 25 04:11:05 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Sat, 25 Apr 2009 12:11:05 +0100 Subject: CompletionHandler should take the channel too In-Reply-To: <49F22588.2010505@univ-mlv.fr> References: <49EE4A58.8050609@univ-mlv.fr> <49EEC5E1.30000@sun.com> <49EF97A8.2080406@univ-mlv.fr> <49F01D0F.2020102@sun.com> <49F22588.2010505@univ-mlv.fr> Message-ID: <49F2EFC9.7000800@sun.com> R?mi Forax wrote: > : >> >>> >>> Else, there is no way to find the group from an async socket channel, >>> something like: >>> asyncSocketChannel().group().shutdown(); >>> should be useful. >> The reason for this is that we don't want to give out a reference to >> the default group (because it can never be shutdown). I admit we need >> to re-visit this but it requires special-casing the default group so >> that it cannot be shutdown. I don't know if you read Alex and Yevgeny >> Libman's proposal on filters but that creates the requirement to get >> a reference to the group so that it can submit its own tasks to the >> group's thread pool. There is merit in this proposal and the support >> is in the implementation (but not the API yet, needs further thought). > No, I'm not aware of this proposal and was not able find a link to it. > Do you have a reference ? Here it is: http://mail.openjdk.java.net/pipermail/nio-dev/2009-January/000339.html From libman at terabit.com.au Sun Apr 26 19:52:19 2009 From: libman at terabit.com.au (Alexander Libman) Date: Sun, 26 Apr 2009 22:52:19 -0400 Subject: AsynchrronousByteChannel vs AsynchronousFileChannel and AsynchronousTransmitter Message-ID: Hi Alan, hi all Thinking about AsynchronousTransmitter with draft prototype: AsynchronousTransmitter { private AsynchrnousTransmitter (....); public static Future transmit(AsynchronousChannel inChannel, AsynchronousChannel outChannel, A attachment, CompletionHandler handler) /* The AsynchrnousTransmitter instance is accessible via returned Future<> */ public long getTotalBytesRead (); public long getTotalBytesWritten (); } where inChannel and outChannel can be any combination of AsynchrnousByteChannel/AsynchronousFileChannel(s). Seems it would be helpful, if the AsynchronousFileChannel was derived from AsynchrnonousByteChannel. Otherwise, we have to provide four similar prototypes and implementations for Transmitter. We can find other examples when we want to process file sequentially as AsynchronousByteChannel and asynchronously at same time. Thr point why AsynchronousFileChannel is not derived from AsynchronousByteChannel: a) AsynchronousFileChannel operations require additional parameter "position", while AsynchrnousByteChannel does not. To inroduce additional and not used argument "position" for AsynchrnousByteChannel is acceptable for C-style API, but not acceptable for OO languages like Java, C++. b) AsynchronousFileChannel can have multiple outstanding read/write opearations since parameter "position" defines location to read/write and the sequence of operations is not important. While AsynchrnonousByteChannel usually supports only one pending read and one pending write due "out-of-order" execution/completion problem. What do you think about following solution: to add to the AsynchronousFileChannel two properties: "defaultReadPosition" and "defaultWritePosition". They can be easily implemented as AtomicLong's. Default positions do not impact at all on existing read/write operations with position argument. But they give us a chance to work AsynchronoudFileChannel as with AsynchronousByteChannel. Only one pending read is allowed for defaultReadPosition, i.e. read operation in AsynchronousByteChannel style(no argument "position"). The defaultReadPosition is atomically adjusted just before setting completed state, so next asynchronous read will pick up the adjusted position. The same is true for write operations with defaultWritePosition. As an example, we developed a draft of "AsynchronousByteFileChannel" which works like a filter and implements AsynchrnousByteChannel inteface. The only one difference with the filter is : the "inner" channel is AsynchnornousFileChannel instead of AsynchronousByteChannel. The source can be found: http://jproactor.svn.sourceforge.net/viewvc/jproactor/trunk/nio2/src/au/com/ terabit/nio2/filter/AsynchronousByteFileChannel.java BTW, during prototyping of AsynchronousByteFileChannel, we found that simple filters can be easily developed using only AsynchronousFuture and no AsynchronousFilter is required. http://jproactor.svn.sourceforge.net/viewvc/jproactor/trunk/nio2/src/au/com/ terabit/nio2/filter/AsynchronousFuture.java With term "simple" filter it is assumed filter with mapping 1:1 for "outer" and "inner" operations. Such filter acts as small interceptor on operation starts and completions. For complicated filters (stateful engines like SSL), the base class AsynchrnousFilter is still preferred solution. And finanally coming back to the beginning - AsynchronousTransmitter. Having AsynchronousByteFileChannel AsynchtronousTransmitter becomes a very small class, which can transmit both files and sockets: http://jproactor.svn.sourceforge.net/viewvc/jproactor/trunk/nio2/src/au/com/ terabit/nio2/filter/AsynchronousTransmitter.java and small test example http://jproactor.svn.sourceforge.net/viewvc/jproactor/trunk/nio2/src/au/com/ terabit/nio2/testfilter/TestTransmitter.java?view=log Definetely, real AsynchrnousTransmitter should use TransmitFile()/TransmitPackets() API for Windows and sendfile() for POSIX whenever it is possible. But in cases when API does not exist or does not support pair source/destination, the current solution can be used. Alex From Alan.Bateman at Sun.COM Mon Apr 27 04:53:58 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Mon, 27 Apr 2009 12:53:58 +0100 Subject: AsynchrronousByteChannel vs AsynchronousFileChannel and AsynchronousTransmitter In-Reply-To: References: Message-ID: <49F59CD6.6080804@sun.com> Alexander Libman wrote: > Hi Alan, hi all > > Thinking about AsynchronousTransmitter with draft prototype: > > AsynchronousTransmitter { > > private AsynchrnousTransmitter (....); > > public static > Future > transmit(AsynchronousChannel inChannel, > AsynchronousChannel outChannel, > A attachment, > CompletionHandler handler) > > /* The AsynchrnousTransmitter instance is accessible via returned Future<> > */ > public long getTotalBytesRead (); > public long getTotalBytesWritten (); > } > > where inChannel and outChannel can be any combination of > AsynchrnousByteChannel/AsynchronousFileChannel(s). > > Seems it would be helpful, if the AsynchronousFileChannel was derived from > AsynchrnonousByteChannel. Otherwise, we have to provide four similar > prototypes and implementations for Transmitter. > AsynchronousFileChannel is intended for concurrent access and so does not maintain a global file position (and so intentionally does not implement AsynchrnonousByteChannel). AsynchronousFileChannel isn't really intended for sequential access but it would be relatively simple to wrap an AsynchronousFileChannel, returning a AsynchrnonousByteChannel. The resulting AsynchrnonousByteChannel would maintain the file position. It probably needs experimentation to see if one position for reading and another for writing, or simple one global position if better (having independent positions for read and write might be a bit strange but easier than trying to coordinate concurrent read/write operations). This solution would avoid needing to add the additional methods to AsynchronousTransmitter that you are concerned with. I'm trying to understand your class from the above skeleton. Who executes the completion handler? Is the Transmitter created with its own thread pool or is the a thread associated with the source or target channel's thread pool? > As an example, we developed a draft of "AsynchronousByteFileChannel" which > works like a filter and implements AsynchrnousByteChannel inteface. > The only one difference with the filter is : the "inner" channel is > AsynchnornousFileChannel instead of AsynchronousByteChannel. > The source can be found: > http://jproactor.svn.sourceforge.net/viewvc/jproactor/trunk/nio2/src/au/com/ > terabit/nio2/filter/AsynchronousByteFileChannel.java > > BTW, during prototyping of AsynchronousByteFileChannel, we found that > simple filters can be easily developed > using only AsynchronousFuture and no AsynchronousFilter is required. > http://jproactor.svn.sourceforge.net/viewvc/jproactor/trunk/nio2/src/au/com/ > terabit/nio2/filter/AsynchronousFuture.java > > With term "simple" filter it is assumed filter with mapping 1:1 for "outer" > and "inner" operations. Such filter > acts as small interceptor on operation starts and completions. > > For complicated filters (stateful engines like SSL), the base class > AsynchrnousFilter is still preferred solution. > > And finanally coming back to the beginning - AsynchronousTransmitter. > Having AsynchronousByteFileChannel AsynchtronousTransmitter becomes a very > small class, which > can transmit both files and sockets: > http://jproactor.svn.sourceforge.net/viewvc/jproactor/trunk/nio2/src/au/com/ > terabit/nio2/filter/AsynchronousTransmitter.java > and small test example > http://jproactor.svn.sourceforge.net/viewvc/jproactor/trunk/nio2/src/au/com/ > terabit/nio2/testfilter/TestTransmitter.java?view=log > Indeed, it doesn't require a lot of code. From an initial glance it looks like you might need to result's completed rather than failed method for the case that there is an I/O failure after some bytes have been transferred. > Definetely, real AsynchrnousTransmitter should use > TransmitFile()/TransmitPackets() API for Windows and > sendfile() for POSIX whenever it is possible. But in cases when API does not > exist or does not support pair source/destination, > the current solution can be used. > This FileChannel transfer methods also work this way so that the best mechanism is used. We don't have transfer methods in the asynchronous I/O API yet but I would like to add some support one day. A possible candidate is AsynchronousSocketChannel#transferFrom(FileChannel, position, count) that works like the write method. This can use the best underlying mechanism for the case that the respective providers are the default providers. -Alan. From libman at terabit.com.au Mon Apr 27 22:32:45 2009 From: libman at terabit.com.au (Alexander Libman) Date: Tue, 28 Apr 2009 01:32:45 -0400 Subject: AsynchrronousByteChannel vs AsynchronousFileChannel andAsynchronousTransmitter In-Reply-To: <49F59CD6.6080804@sun.com> Message-ID: Hi Alan, > > > AsynchronousFileChannel is intended for concurrent access and so does > not maintain a global file position (and so intentionally does not > implement AsynchrnonousByteChannel). AsynchronousFileChannel isn't > really intended for sequential access but it would be relatively simple > to wrap an AsynchronousFileChannel, returning a > AsynchrnonousByteChannel. Agree. Basically the question was what was the better solution: add to the AsynchronousFileChannel some extra features or to develop a small wrapper like AsynchronousByteFileChannel (may be you can advise the better name). Seems, the time spent on writing AsynchronousByteFileChannel was not wasted :) > The resulting AsynchrnonousByteChannel would > maintain the file position. It probably needs experimentation to see if > one position for reading and another for writing, or simple one global > position if better (having independent positions for read and write > might be a bit strange but easier than trying to coordinate concurrent > read/write operations). It is not a problem to support single position for the AsynchrnousByteFileChannel. The choice of two independent read and write positions was made deliberately. It allows to think about file as two independent streams - one for reading, another for writing. It also very convinient for the model read-updateInPlace-writeOnSamePosition. Another reason is : since a position can be updated only on inner AsychrnousFileChannel's operation completion, two positions will make possible to support one outstanding read and one outstanding write simultaneously. If you think that one position is better solution, then it is also good: the developer can create two "AsynchrnousByteFileChannels" and use one for reading and second for writing. What is the preferred solution, I do not know yet too. Yes, need to experiment more. Now, coming back to the AsynchrnousTransmitter: > > > > AsynchronousTransmitter { > > > > private AsynchrnousTransmitter (....); > > > > public static > > Future > > transmit(AsynchronousChannel inChannel, > > AsynchronousChannel outChannel, > > A attachment, > > CompletionHandler super A> handler) > > > > /* The AsynchrnousTransmitter instance is accessible via > returned Future<> > > */ > > public long getTotalBytesRead (); > > public long getTotalBytesWritten (); > > } > > > > where inChannel and outChannel can be any combination of > > AsynchrnousByteChannel/AsynchronousFileChannel(s). > > > > I'm trying to understand your class from the above skeleton. Who > executes the completion handler? Is the Transmitter created with its own > thread pool or is the a thread associated with the source or target > channel's thread pool? For this pure Java solution we do not have to specify thread pool. After initiation of the first read, Transmitter initiates next operations from the completion handler for the previous operation. So it works in both thread pools: for source and destination. But it does not take any significant time as what we have to do is to start next operation(s) only. It uses two buffers and zero-copy technique. It does not use any locks and buffer allocations. It works only with couple AtomicReferences for keeping next buffer handles. Basicically it works as follows: 1) start Read on InputChannel into buffer1; return; 2) readCompleted for buffer1: if (endOfData and all writes are done) { invokeUserCompletionHandler return } if (there is no pending write on OutputChannel) { start Write on OutputChannel from buffer1; } else { store buffer1 as next buffer to write } if (buffer2 is free) { start Read on InputChannel into buffer2; } 3) writeCompleted on buffer2 if (endOfData and all is done) { invokeUserCompletionHandler return } if (there is no pending read on InputChannel) { start Read on InputChannel into buffer2; } else { store buffer2 as next buffer to read } if (buffer1 contains data to write) { start Write on OutputChannel from buffer1; } I skipped synchronization details (buffer1 and buffer2 are AtomicReferences and used simmetrically), but you see that completion handler does not do much. We need our own thread pool in two cases when we deal with filters and artificial asynchornous objects like Transmitter: a) when a completion handler's execution takes a long time ( SSLEngine has a task to run, etc) or b) when an operation started on outer channel has immediate completion without any actions performed on inner channel. > > From an initial glance it looks like you might need to result's >completed rather than failed method for the case that there is an I/O >failure after some bytes have been transferred. yes, we keep this in mind. Currently, there is one more method-accessor in AsynchronousTransmitter : boolean isSuccess (); > > This FileChannel transfer methods also work this way so that the best > mechanism is used. We don't have transfer methods in the asynchronous > I/O API yet but I would like to add some support one day. A possible > candidate is AsynchronousSocketChannel#transferFrom(FileChannel, > position, count) that works like the write method. This can use the best > underlying mechanism for the case that the respective providers are the > default providers. This would be nice. Here we may need to specify which thread pool to use: source or destination. For Windows such "transferFrom" obviously can be mapped to TransmitFile/TransmitPackets and all is clear - no need to specify thread pool. But for POSIX different flavors (Linux, Sun, BSD) sendfile() may block and we need to specify thread pool executor (source, destination or special executor). If we look at Windows evolution, we can find that initially it was introduced only TransmitFile API call and later Microsoft added TransmitPackets. So my point is that "transferFrom" is necessary, but may be not enough. The class kind of AsynchrnousTransmitter can be used and extended to execute some sofisticated sequences and chains of operations on basic AsynchronousChannels and filters. Alex From forax at univ-mlv.fr Tue Apr 28 01:22:35 2009 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Tue, 28 Apr 2009 10:22:35 +0200 Subject: No Path equivalent to File.mkdirs() ? Message-ID: <49F6BCCB.8020605@univ-mlv.fr> Is there an equivalent to File.mkdirs() with NIO.2 ? I was not able to find it. cheers, R?mi From Alan.Bateman at Sun.COM Tue Apr 28 02:09:45 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Tue, 28 Apr 2009 10:09:45 +0100 Subject: AsynchrronousByteChannel vs AsynchronousFileChannel andAsynchronousTransmitter In-Reply-To: References: Message-ID: <49F6C7D9.3020103@sun.com> Alexander Libman wrote: > : > Agree. Basically the question was what was the better solution: > add to the AsynchronousFileChannel some extra features or > to develop a small wrapper like AsynchronousByteFileChannel (may be you can > advise the better name). > Seems, the time spent on writing AsynchronousByteFileChannel was not wasted > :) > Given that the usage differs a bit from the intended usage then this seems more appropriate for a utility library or method. You might not need to define a class name but instead have a static method that returns an adapter, maybe: static AsynchronousByteChannel newChannel(AsynchronousFileChannel ch) { ... } > : > It is not a problem to support single position for the > AsynchrnousByteFileChannel. > The choice of two independent read and write positions was made > deliberately. > It allows to think about file as two independent streams - one for reading, > another for writing. > It also very convinient for the model > read-updateInPlace-writeOnSamePosition. > Another reason is : since a position can be updated only on inner > AsychrnousFileChannel's operation completion, > two positions will make possible to support one outstanding read and one > outstanding write simultaneously. > If you think that one position is better solution, then it is also good: > the developer can create two "AsynchrnousByteFileChannels" and > use one for reading and second for writing. What is the preferred solution, > I do not know yet too. > Yes, need to experiment more. > I don't have a strong opinion on this except to say that different global positions for reading and writing is likely to be a surprise to many developers. > : > For this pure Java solution we do not have to specify thread pool. > After initiation of the first read, Transmitter initiates next operations > from the completion handler for the previous operation. > So it works in both thread pools: for source and destination. > It it possible that the source and target channel are associated with different group and hence there would be problems with thread identity if you don't specify who will call the completion handler. It's an interesting question to think about. > : > > We need our own thread pool in two cases when we deal with filters and > artificial asynchornous objects like Transmitter: > a) when a completion handler's execution takes a long time ( SSLEngine has a > task to run, etc) > or > b) when an operation started on outer channel has immediate completion > without any actions performed on inner channel. > Right, and for that we'll need to come back to that request to submit tasks to the group's thread pool. R?mi also asked recently about getting a reference the group. > : > This would be nice. Here we may need to specify which thread pool to use: > source or destination. > For Windows such "transferFrom" obviously can be mapped to > TransmitFile/TransmitPackets > FWIW, we haven't been able to use this in NIO to date because of semantic issues. Specifically, these Win32 calls are designed to be the sole writer to the socket and there are cases where they bypass the socket buffer. It's a while since we have looked at them though. -Alan. From Alan.Bateman at Sun.COM Tue Apr 28 02:33:07 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Tue, 28 Apr 2009 10:33:07 +0100 Subject: No Path equivalent to File.mkdirs() ? In-Reply-To: <49F6BCCB.8020605@univ-mlv.fr> References: <49F6BCCB.8020605@univ-mlv.fr> Message-ID: <49F6CD53.5040400@sun.com> R?mi Forax wrote: > Is there an equivalent to File.mkdirs() with NIO.2 ? > > I was not able to find it. > > cheers, > R?mi No, it's not there. All the primitives required to implement it as a utility method are there of course (mkdirs, as you probably know, is one of those troublesome composite operations that may fail with part of the directory tree created). -Alan. From forax at univ-mlv.fr Tue Apr 28 05:32:00 2009 From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=) Date: Tue, 28 Apr 2009 14:32:00 +0200 Subject: No Path equivalent to File.mkdirs() ? In-Reply-To: <49F6CD53.5040400@sun.com> References: <49F6BCCB.8020605@univ-mlv.fr> <49F6CD53.5040400@sun.com> Message-ID: <49F6F740.90205@univ-mlv.fr> Alan Bateman a ?crit : > R?mi Forax wrote: >> Is there an equivalent to File.mkdirs() with NIO.2 ? >> >> I was not able to find it. >> >> cheers, >> R?mi > No, it's not there. All the primitives required to implement it as a > utility method are there of course (mkdirs, as you probably know, is > one of those troublesome composite operations that may fail with part > of the directory tree created). Yes, but don't provide it as a utility method will have the following effects: - users will not using Path API - users will create a File from a Path to call mkdirs() - users will create their own code So I think it's a good idea to create this helper method in Paths. By the way, there is also no way to get a File from a Path for the default FileSystem. So the compatibility between File and Path is only in one way. This will delay the adoption of the Path API. > > -Alan. R?mi From Alan.Bateman at Sun.COM Tue Apr 28 07:23:17 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Tue, 28 Apr 2009 15:23:17 +0100 Subject: No Path equivalent to File.mkdirs() ? In-Reply-To: <49F6F740.90205@univ-mlv.fr> References: <49F6BCCB.8020605@univ-mlv.fr> <49F6CD53.5040400@sun.com> <49F6F740.90205@univ-mlv.fr> Message-ID: <49F71155.1020708@sun.com> R?mi Forax wrote: > Yes, but don't provide it as a utility method will have the following > effects: > - users will not using Path API > - users will create a File from a Path to call mkdirs() > - users will create their own code > So I think it's a good idea to create this helper method in Paths. If added, then the most appropriate place is as a static method in the Files class, something like Files#createFileTree. > > By the way, there is also no way to get a File from a Path > for the default FileSystem. > So the compatibility between File and Path is only > in one way. This will delay the adoption of the Path API. Yes, it's one way because in general you can't create a Path from a File. The idea is that it will be easy to fix existing problems for code using java.io.File without too many changes [eg: f.delete() -> f.toPath().delete() ] Going the other way is problematic because it doesn't make sense for other custom providers, and even in the case of the default provider, it is possible to loose the platform representation. The Path class already has a section in the class description to explain this. If needed then it can be done by converting to a string [eg: new File(path.toString()) ]. Not clear that we need anything else. -Alan From Alan.Bateman at Sun.COM Tue Apr 28 07:26:59 2009 From: Alan.Bateman at Sun.COM (Alan Bateman) Date: Tue, 28 Apr 2009 15:26:59 +0100 Subject: No Path equivalent to File.mkdirs() ? In-Reply-To: <49F71155.1020708@sun.com> References: <49F6BCCB.8020605@univ-mlv.fr> <49F6CD53.5040400@sun.com> <49F6F740.90205@univ-mlv.fr> <49F71155.1020708@sun.com> Message-ID: <49F71233.9060002@sun.com> Alan Bateman wrote: > Yes, it's one way because in general you can't create a Path from a File. Swap that, I of course mean you can't create a File from a Path. From talden at gmail.com Tue Apr 28 17:23:18 2009 From: talden at gmail.com (Talden) Date: Wed, 29 Apr 2009 12:23:18 +1200 Subject: No Path equivalent to File.mkdirs() ? In-Reply-To: <49F71155.1020708@sun.com> References: <49F6BCCB.8020605@univ-mlv.fr> <49F6CD53.5040400@sun.com> <49F6F740.90205@univ-mlv.fr> <49F71155.1020708@sun.com> Message-ID: <738d207f0904281723u54c44adbjb64d7d642caebcaf@mail.gmail.com> > If added, then the most appropriate place is as a static method in the Files > class, something like Files#createFileTree. Given how many projects will be using File.mkdirs() and accepting the possibility of partial success, this really is needed. A separate IOException subtype to detail partial success should probably be included (and should nest the cause of failure). I would hate to see every project creating their own solution to this. -- Talden From libman at terabit.com.au Tue Apr 28 21:35:58 2009 From: libman at terabit.com.au (Alexander Libman) Date: Wed, 29 Apr 2009 00:35:58 -0400 Subject: AsynchrronousByteChannel vs AsynchronousFileChannelandAsynchronousTransmitter In-Reply-To: <49F6C7D9.3020103@sun.com> Message-ID: Hi Alan, > Given that the usage differs a bit from the intended usage then this > seems more appropriate for a utility library or method. You might not > need to define a class name but instead have a static method that > returns an adapter, maybe: > > static AsynchronousByteChannel newChannel(AsynchronousFileChannel ch) { > ... } Agree as the implementation does not have any public methods except AsynchronousByteChannel methods. > > > I don't have a strong opinion on this except to say that different > global positions for reading and writing is likely to be a surprise to > many developers. There is an "intermediate" solution: static AsynchronousByteChannel newChannel(AsynchronousFileChannel ch, boolean twoOrOnePositions) So the developer can choose the behavior he wants. > > > : > > For this pure Java solution we do not have to specify thread pool. > > After initiation of the first read, Transmitter initiates next > operations > > from the completion handler for the previous operation. > > So it works in both thread pools: for source and destination. > > > It it possible that the source and target channel are associated with > different group and hence there would be problems with thread identity > if you don't specify who will call the completion handler. It's an > interesting question to think about. I see you point, you want to specify the executor for the user completion handler. I was more concentrated on internal implementation details and was thinking that any thread pool (for source or destination) will be OK. OK, here again solution that will satisfy any goal: AsynchronousTransmitter { private AsynchrnousTransmitter (....); public static Future transmit(AsynchronousChannel inChannel, AsynchronousChannel outChannel, A attachment, CompletionHandler<...> handler, Executor executor) } If the executor is null, it means that handler can be invoked in any inChannel or outChannel thread pool (depends which internal completion comes last). Otherwise, the developer can specify any Executor/ExecutorService. If executor had been accessible for AsynchronousChannel, then user could make a choice of three executors: inChannel's, outChannel's or third-party executor. BTW, it is one more argument to support getExecutor() method for AsynchronousChannel and AsynchronousGroup. I expect more coming asynchronous engines based on similar to AsynchronousTransmitter or AsynchronousFilter ideas. I rememeber that the problem is in default AsynchronousGroup, which should not be closed by user. Is it possible to throw SecurityExeption or introduce other exception, if the user made attempt to shutdown default group? Alex From libman at terabit.com.au Wed Apr 29 20:37:00 2009 From: libman at terabit.com.au (Alexander Libman) Date: Wed, 29 Apr 2009 23:37:00 -0400 Subject: AsynchrronousByteChannel vsAsynchronousFileChannelandAsynchronousTransmitter In-Reply-To: Message-ID: Alan, I just realized that the idea to allow the developer to specify how many default positions is not good : > > > > > I don't have a strong opinion on this except to say that different > > global positions for reading and writing is likely to be a surprise to > > many developers. > > There is an "intermediate" solution: > static AsynchronousByteChannel newChannel(AsynchronousFileChannel ch, > boolean twoOrOnePositions) > >So the developer can choose the behavior he wants. If one channel has been created with one position and the other with two positions, then single user of those channels will not know what the behavior to expect. Thinking about topic: In general, we can consider AsynchronousByteChannel as two independent streams: read stream and write stream. From this point of view - having two default positions seems reasonable. These positions are never exposed to the user. Once we have created AsynchronousByteChannel, we can forget that we work with file. We deal only with sequential reads and writes. The user of this channel does know about implementation, from his point of view the next read operation should not depend on performed writes. May be the discussion about "defaultPositions" is not important now, but it gives me the following idea: interface AsynchronousReadable extends AsynchronousChannel { Future read(BUFFER buffer, A attachment, CompletionHandler handler); } interface AsynchronousWritable extends AsynchronousChannel { Future write(BUFFER buffer, A attachment, CompletionHandler handler); } interface AsynchronousByteChannel extends AsynchronousReadable, AsynchronousWritable { } What you think? Is it too late? Will it impact on performance? Seems such modification of hierarchy of interfaces should not impact on compliation and build of existing NIO2. And it will give more flexibility in developing "pseudo-asynchrnous" objects like Async. Filters, Transmitters,Converters, Decoders, Encoders, etc.... Example : interface AsynchronousCharChannel extends AsynchronousReadable, AsynchronousWritable { } Alex