From rcmuir at gmail.com Thu Mar 1 03:42:02 2018 From: rcmuir at gmail.com (Robert Muir) Date: Wed, 28 Feb 2018 22:42:02 -0500 Subject: FileChannel implements InterruptibleChannel, help or hindrance? In-Reply-To: References: Message-ID: On Wed, Feb 28, 2018 at 1:00 PM, Alan Bateman wrote: > As we all know here, FileChannel implements InterruptibleChannel so it's > asynchronously closeable and interrupting threads doing I/O operations on a > file channel cause it to be closed. > > I've never been comfortable with the FileChannel being closed by > Thread.interrupt and I'm wondering whether it would be worth providing a way > for file I/O operations to not being interruptible. It causes problems with apache lucene too: the file in question may have been deleted (merged away) and so by closing it, its gone for good: searching at that point-in-time is now impossible... we can't reopen the file. Currently we give users a big warning not to use Thread.interrupt/Future.cancel for that reason. From david.lloyd at redhat.com Thu Mar 1 14:57:01 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Thu, 1 Mar 2018 08:57:01 -0600 Subject: FileChannel implements InterruptibleChannel, help or hindrance? In-Reply-To: References: Message-ID: On Wed, Feb 28, 2018 at 12:00 PM, Alan Bateman wrote: > As we all know here, FileChannel implements InterruptibleChannel so it's > asynchronously closeable and interrupting threads doing I/O operations on a > file channel cause it to be closed. > > I've never been comfortable with the FileChannel being closed by > Thread.interrupt and I'm wondering whether it would be worth providing a way > for file I/O operations to not being interruptible. Yes please. In general, interrupt handling in I/O and NIO has been a problem for us, to the point where we have implemented utilities in our thread implementation to defer interrupts over a block of code e.g. a lambda. > One potential approach is to add an OpenOption (maybe > StandardOpenOption.UNINTERRUPTIBLE) that can be specified to > FileChannel.open. This seem preferable to providing a way to toggle > interruptibility dynamically, esp. in cases where a FileChannel is used by > several concurrent threads. There are wider scope ways to do this too but it > would mean re-visiting InterruptibleChannel and its base implementation in > the spi package with potential knock on impact in other areas of the API. It would be nice if InterruptibleChannel could be deprecated and completely neutralized across the board as its behavior has been much more harmful than beneficial in every case I've ever encountered it. But failing that, having _any_ remedy would be great. It would however be nice if there was some way for this to be activated by default (system property?) instead of at each open site. Maybe users should have to opt-in to interruptibility rather than the other way around; the amount of existent software which relies on interrupt to close files seems like it must be vanishingly small. -- - DML From brian.burkhalter at oracle.com Thu Mar 1 18:44:45 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 1 Mar 2018 10:44:45 -0800 Subject: RFR 8198834: (ch) Enable java/nio/channels/spi/SelectorProvider/inheritedChannel/InheritedChannelTest.java on linux-x64 In-Reply-To: References: Message-ID: <77B1450C-2BB8-4E6D-ACDC-5BA479A14845@oracle.com> On Feb 28, 2018, at 8:34 AM, Alan Bateman wrote: > On 28/02/2018 15:59, Brian Burkhalter wrote: >> https://bugs.openjdk.java.net/browse/JDK-8198834 >> >> This changes the Makefile so that libLauncher.so is built for 64-bit Linux. The change has been tested successfully. The Makefile diff is included below; libLauncher.so will also be part of the change but is not posted here. >> > This look okay but I think we should looking at creating this .so in the build, like we do for libDirectIO.so to create the shared library needed for the direct I/O tests. All the details in test/JtregNativeJdk.gmk. Consider it done: http://cr.openjdk.java.net/~bpb/8198834/webrev.00/ This removes the Makefile and all .so files. The change has been verified to work on linux-64 and solaris-sparcv9. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Mar 1 19:52:14 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 1 Mar 2018 19:52:14 +0000 Subject: RFR 8198834: (ch) Enable java/nio/channels/spi/SelectorProvider/inheritedChannel/InheritedChannelTest.java on linux-x64 In-Reply-To: <77B1450C-2BB8-4E6D-ACDC-5BA479A14845@oracle.com> References: <77B1450C-2BB8-4E6D-ACDC-5BA479A14845@oracle.com> Message-ID: <9fa16545-aedd-7602-939f-174910dc4bcd@oracle.com> On 01/03/2018 18:44, Brian Burkhalter wrote: > On Feb 28, 2018, at 8:34 AM, Alan Bateman > wrote: > >> This look okay but I think we should looking at creating this .so in >> the build, like we do for libDirectIO.so to create the shared library >> needed for the direct I/O tests. All the details in >> test/JtregNativeJdk.gmk. > > Consider it done: > > http://cr.openjdk.java.net/~bpb/8198834/webrev.00/ > > > This removes the Makefile and all .so files. The change has been > verified to work on linux-64 and solaris-sparcv9. > The rename of Launcher.c looks a bit strange, the original name looks better to me. I see it doesn't have a copyright header so we should fix that. For the README then you can split the jtreg line to make it easier to read. Otherwise looks good to me and happy to see these checked-in .so files going away. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Thu Mar 1 20:06:27 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 1 Mar 2018 12:06:27 -0800 Subject: RFR 8198834: (ch) Enable java/nio/channels/spi/SelectorProvider/inheritedChannel/InheritedChannelTest.java on linux-x64 In-Reply-To: <9fa16545-aedd-7602-939f-174910dc4bcd@oracle.com> References: <77B1450C-2BB8-4E6D-ACDC-5BA479A14845@oracle.com> <9fa16545-aedd-7602-939f-174910dc4bcd@oracle.com> Message-ID: <33D609D2-FFDC-4CB7-84D6-E53CF922E784@oracle.com> On Mar 1, 2018, at 11:52 AM, Alan Bateman wrote: > The rename of Launcher.c looks a bit strange, the original name looks better to me. The C source file beginning with ?lib? is a requirement of the jtreg native test build. > I see it doesn't have a copyright header so we should fix that. For the README then you can split the jtreg line to make it easier to read. Will fix those. > Otherwise looks good to me and happy to see these checked-in .so files going away. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Mar 1 20:17:43 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 1 Mar 2018 20:17:43 +0000 Subject: RFR 8198834: (ch) Enable java/nio/channels/spi/SelectorProvider/inheritedChannel/InheritedChannelTest.java on linux-x64 In-Reply-To: <33D609D2-FFDC-4CB7-84D6-E53CF922E784@oracle.com> References: <77B1450C-2BB8-4E6D-ACDC-5BA479A14845@oracle.com> <9fa16545-aedd-7602-939f-174910dc4bcd@oracle.com> <33D609D2-FFDC-4CB7-84D6-E53CF922E784@oracle.com> Message-ID: On 01/03/2018 20:06, Brian Burkhalter wrote: > On Mar 1, 2018, at 11:52 AM, Alan Bateman > wrote: > >> The rename of Launcher.c looks a bit strange, the original name looks >> better to me. > > The C source file beginning with ?lib? is a requirement of the jtreg > native test build. Okay, we might have to live with it for now but hopefully it can be fixed to be able to use sensible source file names in the future. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Thu Mar 1 20:21:37 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 1 Mar 2018 12:21:37 -0800 Subject: RFR 8198834: (ch) Enable java/nio/channels/spi/SelectorProvider/inheritedChannel/InheritedChannelTest.java on linux-x64 In-Reply-To: References: <77B1450C-2BB8-4E6D-ACDC-5BA479A14845@oracle.com> <9fa16545-aedd-7602-939f-174910dc4bcd@oracle.com> <33D609D2-FFDC-4CB7-84D6-E53CF922E784@oracle.com> Message-ID: On Mar 1, 2018, at 12:17 PM, Alan Bateman wrote: >> The C source file beginning with ?lib? is a requirement of the jtreg native test build. > Okay, we might have to live with it for now but hopefully it can be fixed to be able to use sensible source file names in the future. Webrev updated in place: http://cr.openjdk.java.net/~bpb/8198834/webrev.00/ Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Mar 1 20:30:38 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 1 Mar 2018 20:30:38 +0000 Subject: RFR 8198834: (ch) Enable java/nio/channels/spi/SelectorProvider/inheritedChannel/InheritedChannelTest.java on linux-x64 In-Reply-To: References: <77B1450C-2BB8-4E6D-ACDC-5BA479A14845@oracle.com> <9fa16545-aedd-7602-939f-174910dc4bcd@oracle.com> <33D609D2-FFDC-4CB7-84D6-E53CF922E784@oracle.com> Message-ID: <6f0ba44c-0f32-4554-a8ae-39b43ba3b81d@oracle.com> On 01/03/2018 20:21, Brian Burkhalter wrote: > On Mar 1, 2018, at 12:17 PM, Alan Bateman > wrote: > >>> The C source file beginning with ?lib? is a requirement of the jtreg >>> native test build. >> Okay, we might have to live with it for now but hopefully it can be >> fixed to be able to use sensible source file names in the future. > > Webrev updated in place: > > http://cr.openjdk.java.net/~bpb/8198834/webrev.00/ > > Thanks for the update, I think it looks okay. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.joelsson at oracle.com Thu Mar 1 22:07:27 2018 From: erik.joelsson at oracle.com (Erik Joelsson) Date: Thu, 1 Mar 2018 14:07:27 -0800 Subject: RFR 8198834: (ch) Enable java/nio/channels/spi/SelectorProvider/inheritedChannel/InheritedChannelTest.java on linux-x64 In-Reply-To: <6f0ba44c-0f32-4554-a8ae-39b43ba3b81d@oracle.com> References: <77B1450C-2BB8-4E6D-ACDC-5BA479A14845@oracle.com> <9fa16545-aedd-7602-939f-174910dc4bcd@oracle.com> <33D609D2-FFDC-4CB7-84D6-E53CF922E784@oracle.com> <6f0ba44c-0f32-4554-a8ae-39b43ba3b81d@oracle.com> Message-ID: <46ed842f-0d5f-6f9f-63ac-d7b2a54e9f4b@oracle.com> These native test libs all need to live in the same global namespace, basically the same java.library.path directory. In this context, "libLauncher" doesn't look very unique and descriptive given that it originates from java/nio/channels/spi/SelectorProvider/inheritedChannel. Could we change it to something more descriptive in a global perspective? Otherwise the build changes look good. /Erik On 2018-03-01 12:30, Alan Bateman wrote: > > > On 01/03/2018 20:21, Brian Burkhalter wrote: >> On Mar 1, 2018, at 12:17 PM, Alan Bateman > > wrote: >> >>>> The C source file beginning with ?lib? is a requirement of the >>>> jtreg native test build. >>> Okay, we might have to live with it for now but hopefully it can be >>> fixed to be able to use sensible source file names in the future. >> >> Webrev updated in place: >> >> http://cr.openjdk.java.net/~bpb/8198834/webrev.00/ >> >> > Thanks for the update, I think it looks okay. > > -Alan From brian.burkhalter at oracle.com Thu Mar 1 22:09:30 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 1 Mar 2018 14:09:30 -0800 Subject: RFR 8198834: (ch) Enable java/nio/channels/spi/SelectorProvider/inheritedChannel/InheritedChannelTest.java on linux-x64 In-Reply-To: <46ed842f-0d5f-6f9f-63ac-d7b2a54e9f4b@oracle.com> References: <77B1450C-2BB8-4E6D-ACDC-5BA479A14845@oracle.com> <9fa16545-aedd-7602-939f-174910dc4bcd@oracle.com> <33D609D2-FFDC-4CB7-84D6-E53CF922E784@oracle.com> <6f0ba44c-0f32-4554-a8ae-39b43ba3b81d@oracle.com> <46ed842f-0d5f-6f9f-63ac-d7b2a54e9f4b@oracle.com> Message-ID: HI Erik, You caught me just shy of checking this in. :-) I can change it to something else like ?libInheritedChannelLauncher? or ?libInheritedChannel.? Thanks, Brian On Mar 1, 2018, at 2:07 PM, Erik Joelsson wrote: > These native test libs all need to live in the same global namespace, basically the same java.library.path directory. In this context, "libLauncher" doesn't look very unique and descriptive given that it originates from java/nio/channels/spi/SelectorProvider/inheritedChannel. Could we change it to something more descriptive in a global perspective? > > Otherwise the build changes look good. -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.joelsson at oracle.com Thu Mar 1 22:50:56 2018 From: erik.joelsson at oracle.com (Erik Joelsson) Date: Thu, 1 Mar 2018 14:50:56 -0800 Subject: RFR 8198834: (ch) Enable java/nio/channels/spi/SelectorProvider/inheritedChannel/InheritedChannelTest.java on linux-x64 In-Reply-To: References: <77B1450C-2BB8-4E6D-ACDC-5BA479A14845@oracle.com> <9fa16545-aedd-7602-939f-174910dc4bcd@oracle.com> <33D609D2-FFDC-4CB7-84D6-E53CF922E784@oracle.com> <6f0ba44c-0f32-4554-a8ae-39b43ba3b81d@oracle.com> <46ed842f-0d5f-6f9f-63ac-d7b2a54e9f4b@oracle.com> Message-ID: <6eab8565-cebe-da7b-e2cb-5a9b7a6a7493@oracle.com> Either works for me. Thanks! /Erik On 2018-03-01 14:09, Brian Burkhalter wrote: > HI Erik, > > You caught me just shy of checking this in. :-) > > I can change it to something else like ?libInheritedChannelLauncher? > or ?libInheritedChannel.? > > Thanks, > > Brian > > On Mar 1, 2018, at 2:07 PM, Erik Joelsson > wrote: > >> These native test libs all need to live in the same global namespace, >> basically the same java.library.path directory. In this context, >> "libLauncher" doesn't look very unique and descriptive given that it >> originates from >> java/nio/channels/spi/SelectorProvider/inheritedChannel. Could we >> change it to something more descriptive in a global perspective? >> >> Otherwise the build changes look good. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Thu Mar 1 22:53:01 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 1 Mar 2018 14:53:01 -0800 Subject: RFR 8198834: (ch) Enable java/nio/channels/spi/SelectorProvider/inheritedChannel/InheritedChannelTest.java on linux-x64 In-Reply-To: <6eab8565-cebe-da7b-e2cb-5a9b7a6a7493@oracle.com> References: <77B1450C-2BB8-4E6D-ACDC-5BA479A14845@oracle.com> <9fa16545-aedd-7602-939f-174910dc4bcd@oracle.com> <33D609D2-FFDC-4CB7-84D6-E53CF922E784@oracle.com> <6f0ba44c-0f32-4554-a8ae-39b43ba3b81d@oracle.com> <46ed842f-0d5f-6f9f-63ac-d7b2a54e9f4b@oracle.com> <6eab8565-cebe-da7b-e2cb-5a9b7a6a7493@oracle.com> Message-ID: <43952D6C-5039-46CF-BE5A-D8203C157191@oracle.com> I think shorter is better: http://cr.openjdk.java.net/~bpb/8198834/webrev.01/ I?ll hold off pushing until Alan comments again. Thanks, Brian On Mar 1, 2018, at 2:50 PM, Erik Joelsson wrote: > Either works for me. Thanks! > > /Erik > > > On 2018-03-01 14:09, Brian Burkhalter wrote: >> HI Erik, >> >> You caught me just shy of checking this in. :-) >> >> I can change it to something else like ?libInheritedChannelLauncher? or ?libInheritedChannel.? -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.joelsson at oracle.com Thu Mar 1 22:57:57 2018 From: erik.joelsson at oracle.com (Erik Joelsson) Date: Thu, 1 Mar 2018 14:57:57 -0800 Subject: RFR 8198834: (ch) Enable java/nio/channels/spi/SelectorProvider/inheritedChannel/InheritedChannelTest.java on linux-x64 In-Reply-To: <43952D6C-5039-46CF-BE5A-D8203C157191@oracle.com> References: <77B1450C-2BB8-4E6D-ACDC-5BA479A14845@oracle.com> <9fa16545-aedd-7602-939f-174910dc4bcd@oracle.com> <33D609D2-FFDC-4CB7-84D6-E53CF922E784@oracle.com> <6f0ba44c-0f32-4554-a8ae-39b43ba3b81d@oracle.com> <46ed842f-0d5f-6f9f-63ac-d7b2a54e9f4b@oracle.com> <6eab8565-cebe-da7b-e2cb-5a9b7a6a7493@oracle.com> <43952D6C-5039-46CF-BE5A-D8203C157191@oracle.com> Message-ID: <821bd89d-f0f1-83a5-faeb-354810557a52@oracle.com> Note that you also need to update the declarations in the makefile. /Erik On 2018-03-01 14:53, Brian Burkhalter wrote: > I think shorter is better: > > http://cr.openjdk.java.net/~bpb/8198834/webrev.01/ > > > I?ll hold off pushing until Alan comments again. > > Thanks, > > Brian > > On Mar 1, 2018, at 2:50 PM, Erik Joelsson > wrote: > >> Either works for me. Thanks! >> >> /Erik >> >> >> On 2018-03-01 14:09, Brian Burkhalter wrote: >>> HI Erik, >>> >>> You caught me just shy of checking this in. :-) >>> >>> I can change it to something else like ?libInheritedChannelLauncher? >>> or ?libInheritedChannel.? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Thu Mar 1 23:44:45 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 1 Mar 2018 15:44:45 -0800 Subject: RFR 8198834: (ch) Enable java/nio/channels/spi/SelectorProvider/inheritedChannel/InheritedChannelTest.java on linux-x64 In-Reply-To: <821bd89d-f0f1-83a5-faeb-354810557a52@oracle.com> References: <77B1450C-2BB8-4E6D-ACDC-5BA479A14845@oracle.com> <9fa16545-aedd-7602-939f-174910dc4bcd@oracle.com> <33D609D2-FFDC-4CB7-84D6-E53CF922E784@oracle.com> <6f0ba44c-0f32-4554-a8ae-39b43ba3b81d@oracle.com> <46ed842f-0d5f-6f9f-63ac-d7b2a54e9f4b@oracle.com> <6eab8565-cebe-da7b-e2cb-5a9b7a6a7493@oracle.com> <43952D6C-5039-46CF-BE5A-D8203C157191@oracle.com> <821bd89d-f0f1-83a5-faeb-354810557a52@oracle.com> Message-ID: D?oh! Thanks, Brian On Mar 1, 2018, at 2:57 PM, Erik Joelsson wrote: > Note that you also need to update the declarations in the makefile. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Fri Mar 2 07:52:20 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 2 Mar 2018 07:52:20 +0000 Subject: RFR 8198834: (ch) Enable java/nio/channels/spi/SelectorProvider/inheritedChannel/InheritedChannelTest.java on linux-x64 In-Reply-To: <43952D6C-5039-46CF-BE5A-D8203C157191@oracle.com> References: <77B1450C-2BB8-4E6D-ACDC-5BA479A14845@oracle.com> <9fa16545-aedd-7602-939f-174910dc4bcd@oracle.com> <33D609D2-FFDC-4CB7-84D6-E53CF922E784@oracle.com> <6f0ba44c-0f32-4554-a8ae-39b43ba3b81d@oracle.com> <46ed842f-0d5f-6f9f-63ac-d7b2a54e9f4b@oracle.com> <6eab8565-cebe-da7b-e2cb-5a9b7a6a7493@oracle.com> <43952D6C-5039-46CF-BE5A-D8203C157191@oracle.com> Message-ID: On 01/03/2018 22:53, Brian Burkhalter wrote: > I think shorter is better: > > http://cr.openjdk.java.net/~bpb/8198834/webrev.01/ > > > I?ll hold off pushing until Alan comments again. > libInheritedLauncher.c looks okay (although still very strange to have the "lib" prefix but that is not your doing). I assume the copyright range should be 2003, 2018 (to match Launcher.java) as this is not a new file in 2018. So I think you good to go and it's great to have the .so files removed from the repos. -Alan From brian.burkhalter at oracle.com Fri Mar 2 15:58:46 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 2 Mar 2018 07:58:46 -0800 Subject: RFR 8198834: (ch) Enable java/nio/channels/spi/SelectorProvider/inheritedChannel/InheritedChannelTest.java on linux-x64 In-Reply-To: References: <77B1450C-2BB8-4E6D-ACDC-5BA479A14845@oracle.com> <9fa16545-aedd-7602-939f-174910dc4bcd@oracle.com> <33D609D2-FFDC-4CB7-84D6-E53CF922E784@oracle.com> <6f0ba44c-0f32-4554-a8ae-39b43ba3b81d@oracle.com> <46ed842f-0d5f-6f9f-63ac-d7b2a54e9f4b@oracle.com> <6eab8565-cebe-da7b-e2cb-5a9b7a6a7493@oracle.com> <43952D6C-5039-46CF-BE5A-D8203C157191@oracle.com> Message-ID: On Mar 1, 2018, at 11:52 PM, Alan Bateman wrote: > I assume the copyright range should be 2003, 2018 (to match Launcher.java) as this is not a new file in 2018. Will update before pushing. > So I think you good to go and it's great to have the .so files removed from the repos. +1 Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.lloyd at redhat.com Fri Mar 2 15:58:19 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Fri, 2 Mar 2018 09:58:19 -0600 Subject: [11] RFR : java.nio.Buffer attachments Message-ID: One of the major classic difficulties with NIO programming (and therefore, usage of NIO-based frameworks) is the management of buffers. This commonly applies specifically to (but is definitely not limited to) handling of direct ByteBuffers. One of the primary difficulties relates to the topic of buffer allocation, pooling, and reuse. I contend that a well-behaved buffer pool necessarily has the following characteristics: ? An explicit `free` operation (i.e. not normally relying on GC to come around) ? Some mechanism to help prevent double-free ? Some mechanism to help prevent other pollution of the pool (i.e. pooling buffers that do not belong to the pool) ? Graceful handling of duplicates, slices, and other types of views In addition, the following characteristics also may be desirable: ? Allocation of fewer, larger buffer regions backing the pooled buffers which may be slices of these regions, maybe even to multiple hierarchical levels ? Some mechanism to signify that the status of a given pooled buffer can no longer be determined (i.e. fall back to using a GC-based mechanism to release the buffer) ? Employment of some reasonable amount of fail-safe against usage of buffers which have been freed, within practical limits Existing buffer pools may rely on wrapper objects, which tend to be difficult to use with scatter/gather and add a level of indirection to common I/O code paths, or (to avoid wrapper objects) they may rely on heuristic information acquired from the Buffer API (such as capacity, isDirect, isReadOnly, etc.) to make a guess as to whether the buffer belongs to the pool, which also comes with several critical disadvantages. A Buffer attachment API would solve these problems far more gracefully. The Buffer attachment API would have to meet the following requirements: ? The attachment must be acquirable for usage by pools and other buffer managers ? Attachments should be protected from interference by buffer users and other attachment implementations ? No performance degradation to standard I/O and manipulation operations ? Minimal memory overhead in the existing case where no attachment is present I've attached a patch which adds a buffer API that meets these requirements. The following implementation points are notable: ? One extra final field is added to java.nio.Buffer to hold the optional attachment ? The BufferAttachment API is "locked down" allowing only the implementation of the attachment to create buffers with that attachment, preventing attachments from being copied to other buffers by misbehaving code ? While anyone can acquire the BufferAttachment object for a given Buffer, all of the members of that class are protected, preventing interference ? Attachments have an opportunity to handle duplicate/slice and view creation by returning null, returning a copy of the attachment, or returning the same attachment instance, depending on the desired behavior ? Only creating ByteBuffers (direct and heap) and wrapping byte arrays are presently supported; other buffer types can be created as views of the ByteBuffer, or support can be added (now or at a later time) for creating them directly ? In particular, the patch does not implement a way to add an attachment to buffers created via FileChannel mapping or via JNI, or other such "back-channel" approaches; these features may potentially be added at a later time Using this attachment API, it is possible to create buffer pool implementations with a variety of different behaviors, many of which meet the above requirements without the drawbacks of wrapper objects. It may also open the door for JDK-managed smart buffer pools in the future. Please review and give feedback at your convenience. Thanks! -- - DML -------------- next part -------------- commit 666750c152c3c1e870b5ab591787c7918d9504fe Author: David M. Lloyd Date: Thu Mar 1 10:39:27 2018 -0600 [JDK-xxxxx] Introduce generic buffer attachments diff --git a/src/java.base/share/classes/java/nio/Bits.java b/src/java.base/share/classes/java/nio/Bits.java index fff84e9e58d..0f8c165d0e1 100644 --- a/src/java.base/share/classes/java/nio/Bits.java +++ b/src/java.base/share/classes/java/nio/Bits.java @@ -248,7 +248,7 @@ class Bits { // package-private } @Override public ByteBuffer newDirectByteBuffer(long addr, int cap, Object ob) { - return new DirectByteBuffer(addr, cap, ob); + return new DirectByteBuffer(addr, cap, ob, null); } @Override public void truncate(Buffer buf) { diff --git a/src/java.base/share/classes/java/nio/Buffer.java b/src/java.base/share/classes/java/nio/Buffer.java index 1217ca78a74..ed388f4a685 100644 --- a/src/java.base/share/classes/java/nio/Buffer.java +++ b/src/java.base/share/classes/java/nio/Buffer.java @@ -210,10 +210,12 @@ public abstract class Buffer { // NOTE: hoisted here for speed in JNI GetDirectBufferAddress long address; + final BufferAttachment attachment; + // Creates a new buffer with the given mark, position, limit, and capacity, // after checking invariants. // - Buffer(int mark, int pos, int lim, int cap) { // package-private + Buffer(int mark, int pos, int lim, int cap, BufferAttachment att) { // package-private if (cap < 0) throw createCapacityException(cap); this.capacity = cap; @@ -225,6 +227,7 @@ public abstract class Buffer { + mark + " > " + pos + ")"); this.mark = mark; } + this.attachment = att; } /** @@ -616,6 +619,19 @@ public abstract class Buffer { */ public abstract Buffer duplicate(); + /** + * Get the attachment of this buffer, if any. + * + * @return the attachment + */ + public final BufferAttachment getAttachment() { + return attachment; + } + + final BufferAttachment splitAtt(int off, int len, boolean ro, Class c) { + final BufferAttachment attachment = this.attachment; + return attachment == null ? null : attachment.duplicate(this, off, len, ro, c); + } // -- Package-private methods for bounds checking, etc. -- diff --git a/src/java.base/share/classes/java/nio/BufferAttachment.java b/src/java.base/share/classes/java/nio/BufferAttachment.java new file mode 100644 index 00000000000..a666583a3a2 --- /dev/null +++ b/src/java.base/share/classes/java/nio/BufferAttachment.java @@ -0,0 +1,106 @@ +/* + * JBoss, Home of Professional Open Source. + * Copyright 2018 Red Hat, Inc., and individual contributors + * as indicated by the @author tags. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package java.nio; + +/** + * An attachment to a buffer. Buffer attachments are strongly referenced by the + * buffers they are attached to, but are not otherwise referenced. + *

+ * A buffer attachment has an opportunity to replicate itself when a buffer is + * duplicated or sliced or otherwise copied. The circumstances of the copy are + * given to the {@link #duplicate(Buffer, int, int, boolean, Class) duplicate()} + * method. + * + * @since 11 + */ +public abstract class BufferAttachment { + /** + * Construct a new instance. + */ + protected BufferAttachment() { + } + + /** + * Duplicate the attachment for a duplicate buffer. If this method returns + * {@code null}, the duplicate will have no attachment. The duplicate + * buffer may have a different type than the original. + *

+ * If this method throws a run-time exception, the exception will propagate + * to the caller and the buffer will not be duplicated. + * + * @param original the original buffer object + * @param offset the offset of the duplicate, in the range + * {@code 0 <= n <= original.capacity()} + * @param length the length of the duplicate, in the range + * {@code 0 <= n <= original.capacity() - offset} + * @param readOnly {@code true} if the returned buffer is read-only, + * {@code false} otherwise + * @param duplicateClazz the type of buffer being returned + * @return the (possibly {@code null}) buffer attachment object + */ + protected BufferAttachment duplicate(Buffer original, int offset, int length, + boolean readOnly, Class duplicateClazz) + { + return this; + } + + /** + * Allocate a heap-backed byte buffer in the same manner as {@link ByteBuffer#allocate(int)}, + * with this object instance as its attachment. + * + * @param capacity the byte buffer capacity + * @return the allocated byte buffer + * @see ByteBuffer#allocate(int) + */ + protected final ByteBuffer allocateByteBuffer(int capacity) { + if (capacity < 0) + throw Buffer.createCapacityException(capacity); + return new HeapByteBuffer(capacity, capacity, this); + } + + /** + * Allocate a direct byte buffer in the same manner as {@link ByteBuffer#allocateDirect(int)}, + * with this object instance as its attachment. + * + * @param capacity the byte buffer capacity + * @return the allocated byte buffer + * @see ByteBuffer#allocateDirect(int) + */ + protected final ByteBuffer allocateDirectByteBuffer(int capacity) { + return new DirectByteBuffer(capacity, this); + } + + /** + * Wrap an array into a byte buffer in the same manner as {@link ByteBuffer#wrap(byte[], int, int)}, + * with this object instance as its attachment. + * + * @param bytes the byte array + * @param off the offset into the array + * @param len the number of bytes to include + * @return the wrapped byte buffer + * @see ByteBuffer#wrap(byte[], int, int) + */ + protected final ByteBuffer wrap(byte[] bytes, int off, int len) { + try { + return new HeapByteBuffer(bytes, off, len, this); + } catch (IllegalArgumentException x) { + throw new IndexOutOfBoundsException(); + } + } +} diff --git a/src/java.base/share/classes/java/nio/ByteBufferAs-X-Buffer.java.template b/src/java.base/share/classes/java/nio/ByteBufferAs-X-Buffer.java.template index b390448b110..418046c335b 100644 --- a/src/java.base/share/classes/java/nio/ByteBufferAs-X-Buffer.java.template +++ b/src/java.base/share/classes/java/nio/ByteBufferAs-X-Buffer.java.template @@ -40,11 +40,12 @@ class ByteBufferAs$Type$Buffer$RW$$BO$ // package-private #end[rw] - ByteBufferAs$Type$Buffer$RW$$BO$(ByteBuffer bb) { // package-private + ByteBufferAs$Type$Buffer$RW$$BO$(ByteBuffer bb, BufferAttachment att) { // package-private #if[rw] super(-1, 0, bb.remaining() >> $LG_BYTES_PER_VALUE$, - bb.remaining() >> $LG_BYTES_PER_VALUE$); + bb.remaining() >> $LG_BYTES_PER_VALUE$, + att); this.bb = bb; // enforce limit == capacity int cap = this.capacity(); @@ -53,21 +54,21 @@ class ByteBufferAs$Type$Buffer$RW$$BO$ // package-private assert (pos <= cap); address = bb.address; #else[rw] - super(bb); + super(bb, att); #end[rw] } ByteBufferAs$Type$Buffer$RW$$BO$(ByteBuffer bb, int mark, int pos, int lim, int cap, - long addr) + long addr, BufferAttachment att) { #if[rw] - super(mark, pos, lim, cap); + super(mark, pos, lim, cap, att); this.bb = bb; address = addr; assert address >= bb.address; #else[rw] - super(bb, mark, pos, lim, cap, addr); + super(bb, mark, pos, lim, cap, addr, att); #end[rw] } @@ -82,26 +83,43 @@ class ByteBufferAs$Type$Buffer$RW$$BO$ // package-private assert (pos <= lim); int rem = (pos <= lim ? lim - pos : 0); long addr = byteOffset(pos); - return new ByteBufferAs$Type$Buffer$RW$$BO$(bb, -1, 0, rem, rem, addr); + BufferAttachment att = splitAtt(pos, rem, +#if[rw] + false, +#else[rw] + true, +#end[rw] + $Type$Buffer.class); + return new ByteBufferAs$Type$Buffer$RW$$BO$(bb, -1, 0, rem, rem, addr, att); } public $Type$Buffer duplicate() { + BufferAttachment att = splitAtt(0, this.capacity(), +#if[rw] + false, +#else[rw] + true, +#end[rw] + $Type$Buffer.class); return new ByteBufferAs$Type$Buffer$RW$$BO$(bb, this.markValue(), this.position(), this.limit(), this.capacity(), - address); + address, + att); } public $Type$Buffer asReadOnlyBuffer() { #if[rw] + BufferAttachment att = splitAtt(0, this.capacity(), true, $Type$Buffer.class); return new ByteBufferAs$Type$BufferR$BO$(bb, this.markValue(), this.position(), this.limit(), this.capacity(), - address); + address, + att); #else[rw] return duplicate(); #end[rw] @@ -223,12 +241,20 @@ class ByteBufferAs$Type$Buffer$RW$$BO$ // package-private if ((start < 0) || (end > len) || (start > end)) throw new IndexOutOfBoundsException(); + BufferAttachment att = splitAtt(pos, len, +#if[rw] + false, +#else[rw] + true, +#end[rw] + $Type$Buffer.class); return new ByteBufferAsCharBuffer$RW$$BO$(bb, -1, pos + start, pos + end, capacity(), - address); + address, + att); } #end[char] diff --git a/src/java.base/share/classes/java/nio/Direct-X-Buffer-bin.java.template b/src/java.base/share/classes/java/nio/Direct-X-Buffer-bin.java.template index cacadb101df..eb767b11a7d 100644 --- a/src/java.base/share/classes/java/nio/Direct-X-Buffer-bin.java.template +++ b/src/java.base/share/classes/java/nio/Direct-X-Buffer-bin.java.template @@ -88,13 +88,27 @@ class XXX { 0, size, size, - address + off)) + address + off, + splitAtt(0, size, +#if[rw] + false, +#else[rw] + true, +#end[rw] + $Type$Buffer.class))) : ($Type$Buffer)(new ByteBufferAs$Type$Buffer$RW$L(this, -1, 0, size, size, - address + off))); + address + off, + splitAtt(0, size, +#if[rw] + false, +#else[rw] + true, +#end[rw] + $Type$Buffer.class)))); } else { return (nativeByteOrder ? ($Type$Buffer)(new Direct$Type$Buffer$RW$U(this, @@ -102,13 +116,27 @@ class XXX { 0, size, size, - off)) + off, + splitAtt(0, size, +#if[rw] + false, +#else[rw] + true, +#end[rw] + $Type$Buffer.class))) : ($Type$Buffer)(new Direct$Type$Buffer$RW$S(this, -1, 0, size, size, - off))); + off, + splitAtt(0, size, +#if[rw] + false, +#else[rw] + true, +#end[rw] + $Type$Buffer.class)))); } } diff --git a/src/java.base/share/classes/java/nio/Direct-X-Buffer.java.template b/src/java.base/share/classes/java/nio/Direct-X-Buffer.java.template index 5fa9be47b20..9a0e7d68b51 100644 --- a/src/java.base/share/classes/java/nio/Direct-X-Buffer.java.template +++ b/src/java.base/share/classes/java/nio/Direct-X-Buffer.java.template @@ -108,9 +108,9 @@ class Direct$Type$Buffer$RW$$BO$ // Primary constructor // - Direct$Type$Buffer$RW$(int cap) { // package-private + Direct$Type$Buffer$RW$(int cap, BufferAttachment ba) { // package-private #if[rw] - super(-1, 0, cap, cap); + super(-1, 0, cap, cap, ba); boolean pa = VM.isDirectMemoryPageAligned(); int ps = Bits.pageSize(); long size = Math.max(1L, (long)cap + (pa ? ps : 0)); @@ -133,7 +133,7 @@ class Direct$Type$Buffer$RW$$BO$ cleaner = Cleaner.create(this, new Deallocator(base, size, cap)); att = null; #else[rw] - super(cap); + super(cap, ba); this.isReadOnly = true; #end[rw] } @@ -143,8 +143,8 @@ class Direct$Type$Buffer$RW$$BO$ // Invoked to construct a direct ByteBuffer referring to the block of // memory. A given arbitrary object may also be attached to the buffer. // - Direct$Type$Buffer(long addr, int cap, Object ob) { - super(-1, 0, cap, cap); + Direct$Type$Buffer(long addr, int cap, Object ob, BufferAttachment ba) { + super(-1, 0, cap, cap, ba); address = addr; cleaner = null; att = ob; @@ -154,7 +154,7 @@ class Direct$Type$Buffer$RW$$BO$ // Invoked only by JNI: NewDirectByteBuffer(void*, long) // private Direct$Type$Buffer(long addr, int cap) { - super(-1, 0, cap, cap); + super(-1, 0, cap, cap, null); address = addr; cleaner = null; att = null; @@ -169,7 +169,7 @@ class Direct$Type$Buffer$RW$$BO$ Runnable unmapper) { #if[rw] - super(-1, 0, cap, cap, fd); + super(-1, 0, cap, cap, fd, null); address = addr; cleaner = Cleaner.create(this, unmapper); att = null; @@ -185,17 +185,17 @@ class Direct$Type$Buffer$RW$$BO$ // Direct$Type$Buffer$RW$$BO$(DirectBuffer db, // package-private int mark, int pos, int lim, int cap, - int off) + int off, BufferAttachment ba) { #if[rw] - super(mark, pos, lim, cap); + super(mark, pos, lim, cap, ba); address = db.address() + off; #if[byte] cleaner = null; #end[byte] att = db; #else[rw] - super(db, mark, pos, lim, cap, off); + super(db, mark, pos, lim, cap, off, ba); this.isReadOnly = true; #end[rw] } @@ -212,7 +212,14 @@ class Direct$Type$Buffer$RW$$BO$ int rem = (pos <= lim ? lim - pos : 0); int off = (pos << $LG_BYTES_PER_VALUE$); assert (off >= 0); - return new Direct$Type$Buffer$RW$$BO$(this, -1, 0, rem, rem, off); + BufferAttachment att = splitAtt(pos, rem, +#if[rw] + false, +#else[rw] + true, +#end[rw] + $Type$Buffer.class); + return new Direct$Type$Buffer$RW$$BO$(this, -1, 0, rem, rem, off, att); } #if[byte] @@ -220,27 +227,44 @@ class Direct$Type$Buffer$RW$$BO$ assert (pos >= 0); assert (pos <= lim); int rem = lim - pos; - return new Direct$Type$Buffer$RW$$BO$(this, -1, 0, rem, rem, pos); + BufferAttachment att = splitAtt(pos, rem, +#if[rw] + false, +#else[rw] + true, +#end[rw] + $Type$Buffer.class); + return new Direct$Type$Buffer$RW$$BO$(this, -1, 0, rem, rem, pos, att); } #end[byte] public $Type$Buffer duplicate() { + BufferAttachment att = splitAtt(0, this.capacity(), +#if[rw] + false, +#else[rw] + true, +#end[rw] + $Type$Buffer.class); return new Direct$Type$Buffer$RW$$BO$(this, this.markValue(), this.position(), this.limit(), this.capacity(), - 0); + 0, + att); } public $Type$Buffer asReadOnlyBuffer() { #if[rw] + BufferAttachment att = splitAtt(0, this.capacity(), true, $Type$Buffer.class); return new Direct$Type$BufferR$BO$(this, this.markValue(), this.position(), this.limit(), this.capacity(), - 0); + 0, + att); #else[rw] return duplicate(); #end[rw] @@ -462,12 +486,20 @@ class Direct$Type$Buffer$RW$$BO$ if ((start < 0) || (end > len) || (start > end)) throw new IndexOutOfBoundsException(); + BufferAttachment att = splitAtt(pos, len, +#if[rw] + false, +#else[rw] + true, +#end[rw] + CharBuffer.class); return new DirectCharBuffer$RW$$BO$(this, -1, pos + start, pos + end, capacity(), - offset); + offset, + att); } #end[char] diff --git a/src/java.base/share/classes/java/nio/Heap-X-Buffer.java.template b/src/java.base/share/classes/java/nio/Heap-X-Buffer.java.template index 7f2a2597ba3..c702e5dc551 100644 --- a/src/java.base/share/classes/java/nio/Heap-X-Buffer.java.template +++ b/src/java.base/share/classes/java/nio/Heap-X-Buffer.java.template @@ -56,91 +56,121 @@ class Heap$Type$Buffer$RW$ #end[rw] */ - Heap$Type$Buffer$RW$(int cap, int lim) { // package-private + Heap$Type$Buffer$RW$(int cap, int lim, BufferAttachment att) { // package-private #if[rw] - super(-1, 0, lim, cap, new $type$[cap], 0); + super(-1, 0, lim, cap, new $type$[cap], 0, att); /* hb = new $type$[cap]; offset = 0; */ this.address = ARRAY_BASE_OFFSET; #else[rw] - super(cap, lim); + super(cap, lim, att); this.isReadOnly = true; #end[rw] } - Heap$Type$Buffer$RW$($type$[] buf, int off, int len) { // package-private + Heap$Type$Buffer$RW$($type$[] buf, int off, int len, BufferAttachment att) { // package-private #if[rw] - super(-1, off, off + len, buf.length, buf, 0); + super(-1, off, off + len, buf.length, buf, 0, att); /* hb = buf; offset = 0; */ this.address = ARRAY_BASE_OFFSET; #else[rw] - super(buf, off, len); + super(buf, off, len, att); this.isReadOnly = true; #end[rw] } protected Heap$Type$Buffer$RW$($type$[] buf, int mark, int pos, int lim, int cap, - int off) + int off, BufferAttachment att) { #if[rw] - super(mark, pos, lim, cap, buf, off); + super(mark, pos, lim, cap, buf, off, att); /* hb = buf; offset = off; */ this.address = ARRAY_BASE_OFFSET + off * ARRAY_INDEX_SCALE; #else[rw] - super(buf, mark, pos, lim, cap, off); + super(buf, mark, pos, lim, cap, off, att); this.isReadOnly = true; #end[rw] } public $Type$Buffer slice() { + int pos = this.position(); + int lim = this.limit(); + assert (pos <= lim); + int rem = (pos <= lim ? lim - pos : 0); + BufferAttachment att = splitAtt(pos, rem, +#if[rw] + false, +#else[rw] + true, +#end[rw] + $Type$Buffer.class); return new Heap$Type$Buffer$RW$(hb, -1, 0, - this.remaining(), - this.remaining(), - this.position() + offset); + rem, + rem, + pos + offset, + att); } #if[byte] $Type$Buffer slice(int pos, int lim) { assert (pos >= 0); assert (pos <= lim); - int rem = lim - pos; + int rem = (pos <= lim ? lim - pos : 0); + BufferAttachment att = splitAtt(pos, rem, +#if[rw] + false, +#else[rw] + true, +#end[rw] + $Type$Buffer.class); return new Heap$Type$Buffer$RW$(hb, -1, 0, rem, rem, - pos + offset); + pos + offset, + att); } #end[byte] public $Type$Buffer duplicate() { + BufferAttachment att = splitAtt(0, this.capacity(), +#if[rw] + false, +#else[rw] + true, +#end[rw] + $Type$Buffer.class); return new Heap$Type$Buffer$RW$(hb, this.markValue(), this.position(), this.limit(), this.capacity(), - offset); + offset, + att); } public $Type$Buffer asReadOnlyBuffer() { #if[rw] + BufferAttachment att = splitAtt(0, this.capacity(), true, $Type$Buffer.class); return new Heap$Type$BufferR(hb, this.markValue(), this.position(), this.limit(), this.capacity(), - offset); + offset, + att); #else[rw] return duplicate(); #end[rw] @@ -313,19 +343,28 @@ class Heap$Type$Buffer$RW$ public CharBuffer asCharBuffer() { int size = this.remaining() >> 1; long addr = address + position(); + BufferAttachment att = splitAtt(0, this.capacity(), +#if[rw] + false, +#else[rw] + true, +#end[rw] + CharBuffer.class); return (bigEndian ? (CharBuffer)(new ByteBufferAsCharBuffer$RW$B(this, -1, 0, size, size, - addr)) + addr, + att)) : (CharBuffer)(new ByteBufferAsCharBuffer$RW$L(this, -1, 0, size, size, - addr))); + addr, + att))); } @@ -364,19 +403,28 @@ class Heap$Type$Buffer$RW$ public ShortBuffer asShortBuffer() { int size = this.remaining() >> 1; long addr = address + position(); + BufferAttachment att = splitAtt(0, this.capacity(), +#if[rw] + false, +#else[rw] + true, +#end[rw] + ShortBuffer.class); return (bigEndian ? (ShortBuffer)(new ByteBufferAsShortBuffer$RW$B(this, -1, 0, size, size, - addr)) + addr, + att)) : (ShortBuffer)(new ByteBufferAsShortBuffer$RW$L(this, -1, 0, size, size, - addr))); + addr, + att))); } @@ -415,19 +463,28 @@ class Heap$Type$Buffer$RW$ public IntBuffer asIntBuffer() { int size = this.remaining() >> 2; long addr = address + position(); + BufferAttachment att = splitAtt(0, this.capacity(), +#if[rw] + false, +#else[rw] + true, +#end[rw] + IntBuffer.class); return (bigEndian ? (IntBuffer)(new ByteBufferAsIntBuffer$RW$B(this, -1, 0, size, size, - addr)) + addr, + att)) : (IntBuffer)(new ByteBufferAsIntBuffer$RW$L(this, -1, 0, size, size, - addr))); + addr, + att))); } @@ -466,19 +523,28 @@ class Heap$Type$Buffer$RW$ public LongBuffer asLongBuffer() { int size = this.remaining() >> 3; long addr = address + position(); + BufferAttachment att = splitAtt(0, this.capacity(), +#if[rw] + false, +#else[rw] + true, +#end[rw] + LongBuffer.class); return (bigEndian ? (LongBuffer)(new ByteBufferAsLongBuffer$RW$B(this, -1, 0, size, size, - addr)) + addr, + att)) : (LongBuffer)(new ByteBufferAsLongBuffer$RW$L(this, -1, 0, size, size, - addr))); + addr, + att))); } @@ -521,19 +587,28 @@ class Heap$Type$Buffer$RW$ public FloatBuffer asFloatBuffer() { int size = this.remaining() >> 2; long addr = address + position(); + BufferAttachment att = splitAtt(0, this.capacity(), +#if[rw] + false, +#else[rw] + true, +#end[rw] + FloatBuffer.class); return (bigEndian ? (FloatBuffer)(new ByteBufferAsFloatBuffer$RW$B(this, -1, 0, size, size, - addr)) + addr, + att)) : (FloatBuffer)(new ByteBufferAsFloatBuffer$RW$L(this, -1, 0, size, size, - addr))); + addr, + att))); } @@ -576,19 +651,28 @@ class Heap$Type$Buffer$RW$ public DoubleBuffer asDoubleBuffer() { int size = this.remaining() >> 3; long addr = address + position(); + BufferAttachment att = splitAtt(0, this.capacity(), +#if[rw] + false, +#else[rw] + true, +#end[rw] + DoubleBuffer.class); return (bigEndian ? (DoubleBuffer)(new ByteBufferAsDoubleBuffer$RW$B(this, -1, 0, size, size, - addr)) + addr, + att)) : (DoubleBuffer)(new ByteBufferAsDoubleBuffer$RW$L(this, -1, 0, size, size, - addr))); + addr, + att))); } @@ -614,12 +698,20 @@ class Heap$Type$Buffer$RW$ || (start > end)) throw new IndexOutOfBoundsException(); int pos = position(); + BufferAttachment att = splitAtt(pos, end - start, +#if[rw] + false, +#else[rw] + true, +#end[rw] + CharBuffer.class); return new HeapCharBuffer$RW$(hb, -1, pos + start, pos + end, capacity(), - offset); + offset, + att); } #end[char] diff --git a/src/java.base/share/classes/java/nio/MappedByteBuffer.java b/src/java.base/share/classes/java/nio/MappedByteBuffer.java index 2f84cbf4549..6cd028babe0 100644 --- a/src/java.base/share/classes/java/nio/MappedByteBuffer.java +++ b/src/java.base/share/classes/java/nio/MappedByteBuffer.java @@ -79,14 +79,14 @@ public abstract class MappedByteBuffer // This should only be invoked by the DirectByteBuffer constructors // MappedByteBuffer(int mark, int pos, int lim, int cap, // package-private - FileDescriptor fd) + FileDescriptor fd, BufferAttachment ba) { - super(mark, pos, lim, cap); + super(mark, pos, lim, cap, ba); this.fd = fd; } - MappedByteBuffer(int mark, int pos, int lim, int cap) { // package-private - super(mark, pos, lim, cap); + MappedByteBuffer(int mark, int pos, int lim, int cap, BufferAttachment ba) { // package-private + super(mark, pos, lim, cap, ba); this.fd = null; } diff --git a/src/java.base/share/classes/java/nio/StringCharBuffer.java b/src/java.base/share/classes/java/nio/StringCharBuffer.java index 26b355bc819..caaa6d78c95 100644 --- a/src/java.base/share/classes/java/nio/StringCharBuffer.java +++ b/src/java.base/share/classes/java/nio/StringCharBuffer.java @@ -34,7 +34,7 @@ class StringCharBuffer // package-private CharSequence str; StringCharBuffer(CharSequence s, int start, int end) { // package-private - super(-1, start, end, s.length()); + super(-1, start, end, s.length(), null); int n = s.length(); if ((start < 0) || (start > n) || (end < start) || (end > n)) throw new IndexOutOfBoundsException(); @@ -57,7 +57,7 @@ class StringCharBuffer // package-private int limit, int cap, int offset) { - super(mark, pos, limit, cap, null, offset); + super(mark, pos, limit, cap, null, offset, null); str = s; this.isReadOnly = true; } diff --git a/src/java.base/share/classes/java/nio/X-Buffer.java.template b/src/java.base/share/classes/java/nio/X-Buffer.java.template index 96e05d6a663..41e13d0c000 100644 --- a/src/java.base/share/classes/java/nio/X-Buffer.java.template +++ b/src/java.base/share/classes/java/nio/X-Buffer.java.template @@ -276,17 +276,17 @@ public abstract class $Type$Buffer // backing array, and array offset // $Type$Buffer(int mark, int pos, int lim, int cap, // package-private - $type$[] hb, int offset) + $type$[] hb, int offset, BufferAttachment ba) { - super(mark, pos, lim, cap); + super(mark, pos, lim, cap, ba); this.hb = hb; this.offset = offset; } // Creates a new buffer with the given mark, position, limit, and capacity // - $Type$Buffer(int mark, int pos, int lim, int cap) { // package-private - this(mark, pos, lim, cap, null, 0); + $Type$Buffer(int mark, int pos, int lim, int cap, BufferAttachment ba) { // package-private + this(mark, pos, lim, cap, null, 0, ba); } @Override @@ -314,7 +314,7 @@ public abstract class $Type$Buffer * If the {@code capacity} is a negative integer */ public static $Type$Buffer allocateDirect(int capacity) { - return new Direct$Type$Buffer(capacity); + return new Direct$Type$Buffer(capacity, null); } #end[byte] @@ -345,7 +345,7 @@ public abstract class $Type$Buffer public static $Type$Buffer allocate(int capacity) { if (capacity < 0) throw createCapacityException(capacity); - return new Heap$Type$Buffer(capacity, capacity); + return new Heap$Type$Buffer(capacity, capacity, null); } /** @@ -390,7 +390,7 @@ public abstract class $Type$Buffer int offset, int length) { try { - return new Heap$Type$Buffer(array, offset, length); + return new Heap$Type$Buffer(array, offset, length, null); } catch (IllegalArgumentException x) { throw new IndexOutOfBoundsException(); } From Alan.Bateman at oracle.com Sun Mar 4 17:10:27 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sun, 4 Mar 2018 17:10:27 +0000 Subject: FileChannel implements InterruptibleChannel, help or hindrance? In-Reply-To: References: Message-ID: <2ab7b5b8-93c1-fe1d-d167-e3dee39fbbd7@oracle.com> On 01/03/2018 14:57, David Lloyd wrote: > : > Yes please. In general, interrupt handling in I/O and NIO has been a > problem for us, to the point where we have implemented utilities in > our thread implementation to defer interrupts over a block of code > e.g. a lambda. Is that interrupting file I/O with FileChannel or do you mean the network channels? I wasn't involved in JSR-51 but I believe an important use-case at the time was to be able to interrupt tasks in a thread pool that may be blocked in socket operations. I'm less sure about FileChannel as file I/O operations don't usually block indefinitely, at least not without NFS or other networked file system. > : > It would be nice if InterruptibleChannel could be deprecated and > completely neutralized across the board as its behavior has been much > more harmful than beneficial in every case I've ever encountered it. The other aspect to InterruptibleChannel is asynchronous close, something that is really important for network oriented channels. -Alan From Alan.Bateman at oracle.com Sun Mar 4 20:22:19 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sun, 4 Mar 2018 20:22:19 +0000 Subject: 8198928: (so) SocketChannel connect may deadlock if closed at around same time that connect fails Message-ID: <99abe020-e895-cae0-2b69-0233f434394e@oracle.com> This is a deadlock found with one of the HTTP client tests. When establishing a connection with a SocketChannel then the channel is specified to be closed if the connection cannot be established. If the SocketChannel is configured non-blocking and connect (or finishConnect) is closing the channel due to a connect failure then it can potentially deadlock with another thread that attempts to close the channel at the same time. The deadlock arises because connect/finishConnect invoke close (closeLock) while holding the read and write locks. Another thread closing the channel acquires the clockLock, the actual close implementation then attempts to acquire the read and write locks in order to coordinate with threads doing non-blocking operations on the change. Historical note: The implementation of connect/finishConnect has always called close while holding the read and write locks but it hasn't been problem in the past. The recent changes to to separate the blocking and non-blocking code paths have changed the locking during close which is why we are only seeing this now. The proposed change is to connect/finishConnect so that these methods don't close the channel until after they've released the read and write locks. The changes mean there is a window where some other thread might call finishConnect or some other operation so a few other small changes are needed to deal with that. It would be possible to introduce a special state for this case this adds complexity so I've decided against that. The webrev with the proposed changes and test are here: ? http://cr.openjdk.java.net/~alanb/8198928/webrev/index.html In terms of behavior changes then an observer polling isConnectionPending will now observe the connection is pending even if the channel is configured blocking. Previously, isConnectionPending would never return true if connect was called when configured blocking. Another observer polling with getRemoteAddress will now observe the remote address while the connection is pending. This was always true with socket channels configured non-blocking, not so with channels configured blocking. The spec for getRemoteAddress is that the remote address is only returned when the channel is connected but it can be useful to observe it while a connection is established. I will create a separate JIRA issue to track this as it's a long standing spec vs. implementation difference. As I mentioned, if the connection cannot be established, there is a small window for other threads to call finishConnect or some other operation before the channel is closed. This is mostly benign except on platforms where the connection error isn't available after it has been obtained. To deal with this requires checkConnect to check for POLLHUP, which is should have done anyway. While in the area I did some cleanup on the Windows implementation of checkConnect. All tests are passing with these changes, including the HTTP client tests where this issue was initially observed as a test timing out intermittently. -Alan From david.lloyd at redhat.com Sun Mar 4 23:15:46 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Sun, 4 Mar 2018 17:15:46 -0600 Subject: FileChannel implements InterruptibleChannel, help or hindrance? In-Reply-To: <2ab7b5b8-93c1-fe1d-d167-e3dee39fbbd7@oracle.com> References: <2ab7b5b8-93c1-fe1d-d167-e3dee39fbbd7@oracle.com> Message-ID: On Sun, Mar 4, 2018 at 11:10 AM, Alan Bateman wrote: > On 01/03/2018 14:57, David Lloyd wrote: >> >> : >> Yes please. In general, interrupt handling in I/O and NIO has been a >> problem for us, to the point where we have implemented utilities in >> our thread implementation to defer interrupts over a block of code >> e.g. a lambda. > > Is that interrupting file I/O with FileChannel or do you mean the network > channels? I wasn't involved in JSR-51 but I believe an important use-case at > the time was to be able to interrupt tasks in a thread pool that may be > blocked in socket operations. I'm less sure about FileChannel as file I/O > operations don't usually block indefinitely, at least not without NFS or > other networked file system. Either one - it is sometimes very inconvenient for NIO to be making assumptions about intended behavior on interrupt. In the SocketChannel case, we're generally using temporary selectors to manage the blocking with a non-blocking channel underneath, which allows us to unblock on interrupt but at the same time it allows the program to make the decision as to what to do with the socket... as long as the interrupt doesn't arrive at an inconvenient moment (or we remember to defer interrupts over any critical sections). For example, we might be waiting for a long-running request on a multiplexed channel, so the appropriate action might be to cancel the just multiplexed request (as opposed to killing off the entire socket). In cases where it's a thread managing a shared socket (or pipe), the correct action may be for the thread to simply ignore the interrupt, or to recursively interrupt certain other threads and take no other action (allowing for a more "graceful" cancellation). The file case is no different. The challenge with files is that it's generally difficult or impossible (on most OSes) to move the blocking to a select-ish layer. From the perspective of user programs, it would have been better to either return a partial read (if some data was read before the interrupt) or throw an IOException (something like InterruptedIOException but without the bytesTransferred field, which leaves the interrupted state set but also leaves the channel open) which would allow the user to decide how to proceed. This allows interruption to fulfill its primary mission - to ensure the thread isn't blocked so it can respond appropriately. But it also gives more options about how the actual user code can proceed. >> It would be nice if InterruptibleChannel could be deprecated and >> completely neutralized across the board as its behavior has been much >> more harmful than beneficial in every case I've ever encountered it. > > The other aspect to InterruptibleChannel is asynchronous close, something > that is really important for network oriented channels. I agree that this is an important behavior. This could be pushed down to the individual channel types though. -- - DML From Alan.Bateman at oracle.com Mon Mar 5 13:32:22 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 5 Mar 2018 13:32:22 +0000 Subject: 8199015: (se) SocketChannelImpl.translateXXXOps access channel state without synchronization Message-ID: <88eef71e-a2fd-c40c-417e-dd727de7da49@oracle.com> While fixing other issues in this area, I noticed that SocketChannel's translateXXXOps methods access the channel state without holding the state lock. The issue seems to be there since JDK 1.4. As the translateXXXOps methods are called during performance sensitive select operations then it seems better to change the state field to volatile rather. The other channel implementations don't see to have this issue as they need the channel state to translate events. The proposed changes are here: ??? http://cr.openjdk.java.net/~alanb/8199015/webrev/index.html -Alan From Alan.Bateman at oracle.com Mon Mar 5 13:39:29 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 5 Mar 2018 13:39:29 +0000 Subject: FileChannel implements InterruptibleChannel, help or hindrance? In-Reply-To: References: <2ab7b5b8-93c1-fe1d-d167-e3dee39fbbd7@oracle.com> Message-ID: On 04/03/2018 23:15, David Lloyd wrote: > : > The file case is no different. The challenge with files is that it's > generally difficult or impossible (on most OSes) to move the blocking > to a select-ish layer. From the perspective of user programs, it > would have been better to either return a partial read (if some data > was read before the interrupt) or throw an IOException (something like > InterruptedIOException but without the bytesTransferred field, which > leaves the interrupted state set but also leaves the channel open) > which would allow the user to decide how to proceed. This allows > interruption to fulfill its primary mission - to ensure the thread > isn't blocked so it can respond appropriately. But it also gives more > options about how the actual user code can proceed. Interrupting a blocking I/O operation on a file or socket and leaving it open is highly problematic. Early JDK releases attempted to do this with so-called "interruptible I/O". Aside from not being feasible everywhere, you can't leave a file or socket connection in an inconsistent state where it may continue to be used. There are a slew of scenarios where bytes can be lost. It took use several major releases to finally remove all the issues caused with the JDK 1.0 era mechanism and I don't think we want to go back there again. -Alan From david.lloyd at redhat.com Mon Mar 5 15:15:56 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Mon, 5 Mar 2018 09:15:56 -0600 Subject: FileChannel implements InterruptibleChannel, help or hindrance? In-Reply-To: References: <2ab7b5b8-93c1-fe1d-d167-e3dee39fbbd7@oracle.com> Message-ID: On Mon, Mar 5, 2018 at 7:39 AM, Alan Bateman wrote: > On 04/03/2018 23:15, David Lloyd wrote: >> >> : >> The file case is no different. The challenge with files is that it's >> generally difficult or impossible (on most OSes) to move the blocking >> to a select-ish layer. From the perspective of user programs, it >> would have been better to either return a partial read (if some data >> was read before the interrupt) or throw an IOException (something like >> InterruptedIOException but without the bytesTransferred field, which >> leaves the interrupted state set but also leaves the channel open) >> which would allow the user to decide how to proceed. This allows >> interruption to fulfill its primary mission - to ensure the thread >> isn't blocked so it can respond appropriately. But it also gives more >> options about how the actual user code can proceed. > > Interrupting a blocking I/O operation on a file or socket and leaving it > open is highly problematic. Early JDK releases attempted to do this with > so-called "interruptible I/O". Aside from not being feasible everywhere, you > can't leave a file or socket connection in an inconsistent state where it > may continue to be used. There are a slew of scenarios where bytes can be > lost. It took use several major releases to finally remove all the issues > caused with the JDK 1.0 era mechanism and I don't think we want to go back > there again. In all cases, exceptionless partial reads are a perfectly safe/acceptable outcome for interruption. In fact if the API were being designed from scratch today, I might even argue that a zero-bytes read/write would be the preferred behavior when operations are entered in an interrupted state. Either way I don't see how anything in the API (as I've proposed it) could possibly lead to data loss. AFAICT the biggest API risk is the use of bytesTransferred in InterruptedIOException, which in retrospect was probably not a great idea. Beyond API effects, at an implementation level, is there some facet to the interruption mechanism itself that would lead to data loss? OS signals and special FD types (like eventfd) are two implementations which should be immune to any problems of this sort on *nixes. Is Windows the main problem here? -- - DML From Alan.Bateman at oracle.com Mon Mar 5 15:25:34 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 5 Mar 2018 15:25:34 +0000 Subject: FileChannel implements InterruptibleChannel, help or hindrance? In-Reply-To: References: <2ab7b5b8-93c1-fe1d-d167-e3dee39fbbd7@oracle.com> Message-ID: <1df71e85-e3d1-e2dc-9b44-ad01b4ce3138@oracle.com> On 05/03/2018 15:15, David Lloyd wrote: > : > In all cases, exceptionless partial reads are a perfectly > safe/acceptable outcome for interruption. In fact if the API were > being designed from scratch today, I might even argue that a > zero-bytes read/write would be the preferred behavior when operations > are entered in an interrupted state. Either way I don't see how > anything in the API (as I've proposed it) could possibly lead to data > loss. AFAICT the biggest API risk is the use of bytesTransferred in > InterruptedIOException, which in retrospect was probably not a great > idea. Switch your example to writing and wrap it in an API (such as OutputStream) that does not tolerate short writes and you'll start to see some of the issues. Reading is problematic too btw, esp. if you throw an exception after reading bytes. -Alan From david.lloyd at redhat.com Mon Mar 5 15:47:01 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Mon, 5 Mar 2018 09:47:01 -0600 Subject: FileChannel implements InterruptibleChannel, help or hindrance? In-Reply-To: <1df71e85-e3d1-e2dc-9b44-ad01b4ce3138@oracle.com> References: <2ab7b5b8-93c1-fe1d-d167-e3dee39fbbd7@oracle.com> <1df71e85-e3d1-e2dc-9b44-ad01b4ce3138@oracle.com> Message-ID: On Mon, Mar 5, 2018 at 9:25 AM, Alan Bateman wrote: > On 05/03/2018 15:15, David Lloyd wrote: >> >> : >> In all cases, exceptionless partial reads are a perfectly >> safe/acceptable outcome for interruption. In fact if the API were >> being designed from scratch today, I might even argue that a >> zero-bytes read/write would be the preferred behavior when operations >> are entered in an interrupted state. Either way I don't see how >> anything in the API (as I've proposed it) could possibly lead to data >> loss. AFAICT the biggest API risk is the use of bytesTransferred in >> InterruptedIOException, which in retrospect was probably not a great >> idea. > > Switch your example to writing and wrap it in an API (such as OutputStream) > that does not tolerate short writes and you'll start to see some of the > issues. Reading is problematic too btw, esp. if you throw an exception after > reading bytes. As far as I can tell, the specification of OutputStream cannot accommodate any more lenient approaches - it's stuck with InterruptedIOException forever. InputStreams could possibly get away with returning a partial read instead of throwing an exception in the event that some data was transferred, but I don't think it really matters at this point: users of *Stream are already required to cope with InterruptedIOException if they care about resuming after interruption. But, we are talking about channels here, not Input/OutputStream. Allowing short reads and writes is within spec for channels of all types. Whether channels throw an exception on interrupt when no bytes were yet transferred, or whether they just return a 0 value, is the only question that would need to seriously be considered, but either behavior is preferable to killing the entire channel in any case I can imagine. One side benefit to not closing the channel on interrupt is that a blocking *Stream can then safely be directly implemented in terms of a blocking *Channel - even the ugly OutputStream case. -- - DML From brian.burkhalter at oracle.com Mon Mar 5 21:19:11 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 5 Mar 2018 13:19:11 -0800 Subject: 8198928: (so) SocketChannel connect may deadlock if closed at around same time that connect fails In-Reply-To: <99abe020-e895-cae0-2b69-0233f434394e@oracle.com> References: <99abe020-e895-cae0-2b69-0233f434394e@oracle.com> Message-ID: On Mar 4, 2018, at 12:22 PM, Alan Bateman wrote: > The webrev with the proposed changes and test are here: > http://cr.openjdk.java.net/~alanb/8198928/webrev/index.html This looks OK to me modulo the following few very minor items: 1) AsyncCloseAndInterrupt needs copyright year update 2) CloseDuringConnect L29 s/closely/closing/ L30 -> while another thread is closing the channel after a connection failure L52 millisecond -> milliseconds (plural) L111 s/127.0.0.1/localhost/ ? Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Mon Mar 5 21:54:02 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 5 Mar 2018 13:54:02 -0800 Subject: 8199015: (se) SocketChannelImpl.translateXXXOps access channel state without synchronization In-Reply-To: <88eef71e-a2fd-c40c-417e-dd727de7da49@oracle.com> References: <88eef71e-a2fd-c40c-417e-dd727de7da49@oracle.com> Message-ID: <3C3BD52C-4EC6-4523-9F91-699E72528290@oracle.com> On Mar 5, 2018, at 5:32 AM, Alan Bateman wrote: > The proposed changes are here: > http://cr.openjdk.java.net/~alanb/8199015/webrev/index.html Looks good. Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Mar 6 01:12:33 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 5 Mar 2018 17:12:33 -0800 Subject: trivial patch to reduce allocations in nio open* methods In-Reply-To: References: Message-ID: <158BFF8B-586B-4C1C-8DBE-5C5DD3E8D66B@oracle.com> Hello, Redirecting to nio-dev. Please reply omitting core-libs-dev. If the patch is small you can include it in e-mail to the mailing list as attachments are generally scrubbed. Thanks, Brian On Mar 5, 2018, at 4:35 PM, Michael Skells wrote: > I have a trivial patch to reduce allocation of OpenOption sets where no > args are supplied > > Spotted this while working on scala compiler, trying to reduce allocations > > I cannot build JDK due to visual studio issues > > Can someone progress the patch for me? Dont want to send a patch to this > mailgroup! > > I have previously signed OCA and contributed (although it has been a while) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Mar 6 07:45:33 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 6 Mar 2018 07:45:33 +0000 Subject: 8198928: (so) SocketChannel connect may deadlock if closed at around same time that connect fails In-Reply-To: References: <99abe020-e895-cae0-2b69-0233f434394e@oracle.com> Message-ID: <311b7a8d-4f35-8928-269f-c56f290a9f33@oracle.com> On 05/03/2018 21:19, Brian Burkhalter wrote: > > 2) CloseDuringConnect > > L29 s/closely/closing/ > L30 -> while another thread is closing the channel after a connection > failure > L52 millisecond -> milliseconds (plural) Thanks, I'll the fixes in the test's comments before pushing. > L111 s/127.0.0.1/localhost/ ? > Okay, that might be better in case the test is run on a test system without 127.0.0.1 -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike.skells1 at gmail.com Tue Mar 6 08:17:31 2018 From: mike.skells1 at gmail.com (Michael Skells) Date: Tue, 06 Mar 2018 08:17:31 +0000 Subject: trivial patch to reduce allocations in nio open* methods In-Reply-To: <158BFF8B-586B-4C1C-8DBE-5C5DD3E8D66B@oracle.com> References: <158BFF8B-586B-4C1C-8DBE-5C5DD3E8D66B@oracle.com> Message-ID: Hi, below is a trivial patch to reduce allocation on the file open call path, when no OpenOption is specified( which is a common case in my test code at least) There is some duplication of empty set handling, but I could not see a suitable utility class to provide this and didnt want to create a public method Regards Mike --- Index: src/java.base/share/classes/java/nio/channels/AsynchronousFileChannel.java IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== --- src/java.base/share/classes/java/nio/channels/AsynchronousFileChannel.java (revision 49129:fb9f590b9eeecae202277c0606ed12b4d3f674c3) +++ src/java.base/share/classes/java/nio/channels/AsynchronousFileChannel.java (revision 49129+:fb9f590b9eee+) @@ -301,8 +301,13 @@ public static AsynchronousFileChannel open(Path file, OpenOption... options) throws IOException { - Set set = new HashSet<>(options.length); - Collections.addAll(set, options); + Set set; + if (options.length == 0) { + set = Collections.emptySet(); + } else { + set = new HashSet<>(options.length); + Collections.addAll(set, options); + } return open(file, set, null, NO_ATTRIBUTES); } Index: src/java.base/share/classes/java/nio/channels/FileChannel.java IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== --- src/java.base/share/classes/java/nio/channels/FileChannel.java (revision 49129:fb9f590b9eeecae202277c0606ed12b4d3f674c3) +++ src/java.base/share/classes/java/nio/channels/FileChannel.java (revision 49129+:fb9f590b9eee+) @@ -335,8 +335,11 @@ public static FileChannel open(Path path, OpenOption... options) throws IOException { - Set set = new HashSet<>(options.length); - Collections.addAll(set, options); + Set set; + if (options.length > 0) { + set = new HashSet<>(options.length); + Collections.addAll(set, options); + } else set = Collections.emptySet(); return open(path, set, NO_ATTRIBUTES); } Index: src/java.base/share/classes/java/nio/file/spi/FileSystemProvider.java IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== --- src/java.base/share/classes/java/nio/file/spi/FileSystemProvider.java (revision 49129:fb9f590b9eeecae202277c0606ed12b4d3f674c3) +++ src/java.base/share/classes/java/nio/file/spi/FileSystemProvider.java (revision 49129+:fb9f590b9eee+) @@ -419,20 +419,22 @@ throws IOException { int len = options.length; - Set opts = new HashSet<>(len + 3); + Set opts ; if (len == 0) { - opts.add(StandardOpenOption.CREATE); - opts.add(StandardOpenOption.TRUNCATE_EXISTING); + opts = DEFAULT_OPEN_OPTIONS; } else { + opts = new HashSet<>(len + 1); for (OpenOption opt: options) { if (opt == StandardOpenOption.READ) throw new IllegalArgumentException("READ not allowed"); opts.add(opt); } - } - opts.add(StandardOpenOption.WRITE); + opts.add(StandardOpenOption.WRITE); + } return Channels.newOutputStream(newByteChannel(path, opts)); } + private final static Set DEFAULT_OPEN_OPTIONS = Collections.unmodifiableSet( + EnumSet.of(StandardOpenOption.CREATE,StandardOpenOption.TRUNCATE_EXISTING,StandardOpenOption.WRITE)); /** * Opens or creates a file for reading and/or writing, returning a file Index: src/java.base/share/classes/java/nio/file/Files.java IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== --- src/java.base/share/classes/java/nio/file/Files.java (revision 49129:fb9f590b9eeecae202277c0606ed12b4d3f674c3) +++ src/java.base/share/classes/java/nio/file/Files.java (revision 49129+:fb9f590b9eee+) @@ -410,8 +410,11 @@ public static SeekableByteChannel newByteChannel(Path path, OpenOption... options) throws IOException { - Set set = new HashSet<>(options.length); - Collections.addAll(set, options); + Set set; + if (options.length > 0) { + set = new HashSet<>(options.length); + Collections.addAll(set, options); + } else set = Collections.emptySet(); return newByteChannel(path, set); } @@ -635,11 +638,12 @@ public static Path createFile(Path path, FileAttribute... attrs) throws IOException { - EnumSet options = - EnumSet.of(StandardOpenOption.CREATE_NEW, StandardOpenOption.WRITE); - newByteChannel(path, options, attrs).close(); + newByteChannel(path, DEFAULT_CREATE_OPTIONS, attrs).close(); return path; } + private final static Set DEFAULT_CREATE_OPTIONS = Collections.unmodifiableSet( + EnumSet.of(StandardOpenOption.CREATE_NEW, StandardOpenOption.WRITE) + ); /** * Creates a new directory. The check for the existence of the file and the --- On Tue, 6 Mar 2018 at 01:12 Brian Burkhalter wrote: > Hello, > > Redirecting to nio-dev. Please reply omitting core-libs-dev. > > If the patch is small you can include it in e-mail to the mailing list as > attachments are generally scrubbed. > > Thanks, > > Brian > > On Mar 5, 2018, at 4:35 PM, Michael Skells wrote: > > I have a trivial patch to reduce allocation of OpenOption sets where no > args are supplied > > Spotted this while working on scala compiler, trying to reduce allocations > > I cannot build JDK due to visual studio issues > > Can someone progress the patch for me? Dont want to send a patch to this > mailgroup! > > I have previously signed OCA and contributed (although it has been a while) > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From huaming.li at oracle.com Tue Mar 6 08:39:59 2018 From: huaming.li at oracle.com (Hamlin Li) Date: Tue, 6 Mar 2018 16:39:59 +0800 Subject: 8198928: (so) SocketChannel connect may deadlock if closed at around same time that connect fails In-Reply-To: <99abe020-e895-cae0-2b69-0233f434394e@oracle.com> References: <99abe020-e895-cae0-2b69-0233f434394e@oracle.com> Message-ID: <005e808f-1a6e-3d07-c1a2-8e5fd65da4f6@oracle.com> Hi Alan, I have minor comment about CloseDuringConnect.java, is it better to move following code from main() to test()? SocketAddress remote = new InetSocketAddress("127.0.0.1", Utils.getFreePort()); It will reduce the chance to interfere with other tests which run concurrently and also create a listening socket at THE free port same as this test. Thank you -Hamlin On 05/03/2018 4:22 AM, Alan Bateman wrote: > > This is a deadlock found with one of the HTTP client tests. When > establishing a connection with a SocketChannel then the channel is > specified to be closed if the connection cannot be established. If the > SocketChannel is configured non-blocking and connect (or > finishConnect) is closing the channel due to a connect failure then it > can potentially deadlock with another thread that attempts to close > the channel at the same time. The deadlock arises because > connect/finishConnect invoke close (closeLock) while holding the read > and write locks. Another thread closing the channel acquires the > clockLock, the actual close implementation then attempts to acquire > the read and write locks in order to coordinate with threads doing > non-blocking operations on the change. > > Historical note: The implementation of connect/finishConnect has > always called close while holding the read and write locks but it > hasn't been problem in the past. The recent changes to to separate the > blocking and non-blocking code paths have changed the locking during > close which is why we are only seeing this now. > > The proposed change is to connect/finishConnect so that these methods > don't close the channel until after they've released the read and > write locks. The changes mean there is a window where some other > thread might call finishConnect or some other operation so a few other > small changes are needed to deal with that. It would be possible to > introduce a special state for this case this adds complexity so I've > decided against that. > > The webrev with the proposed changes and test are here: > ? http://cr.openjdk.java.net/~alanb/8198928/webrev/index.html > > In terms of behavior changes then an observer polling > isConnectionPending will now observe the connection is pending even if > the channel is configured blocking. Previously, isConnectionPending > would never return true if connect was called when configured blocking. > > Another observer polling with getRemoteAddress will now observe the > remote address while the connection is pending. This was always true > with socket channels configured non-blocking, not so with channels > configured blocking. The spec for getRemoteAddress is that the remote > address is only returned when the channel is connected but it can be > useful to observe it while a connection is established. I will create > a separate JIRA issue to track this as it's a long standing spec vs. > implementation difference. > > As I mentioned, if the connection cannot be established, there is a > small window for other threads to call finishConnect or some other > operation before the channel is closed. This is mostly benign except > on platforms where the connection error isn't available after it has > been obtained. To deal with this requires checkConnect to check for > POLLHUP, which is should have done anyway. While in the area I did > some cleanup on the Windows implementation of checkConnect. > > All tests are passing with these changes, including the HTTP client > tests where this issue was initially observed as a test timing out > intermittently. > > -Alan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Mar 6 10:04:31 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 6 Mar 2018 10:04:31 +0000 Subject: 8198928: (so) SocketChannel connect may deadlock if closed at around same time that connect fails In-Reply-To: <005e808f-1a6e-3d07-c1a2-8e5fd65da4f6@oracle.com> References: <99abe020-e895-cae0-2b69-0233f434394e@oracle.com> <005e808f-1a6e-3d07-c1a2-8e5fd65da4f6@oracle.com> Message-ID: <63c22fb6-817c-8485-f7af-31056e8f24f9@oracle.com> On 06/03/2018 08:39, Hamlin Li wrote: > > Hi Alan, > > I have minor comment about CloseDuringConnect.java, is it better to > move following code from main() to test()? > > SocketAddress remote = new InetSocketAddress("127.0.0.1", Utils.getFreePort()); > > It will reduce the chance to interfere with other tests which run > concurrently and also create a listening socket at THE free port same > as this test. > Good question! There is a general issue with tests that need a free port, or a refusing endpoint in this case. Ideally jtreg would support a per test equivalent of exclusiveAccess.dirs, that way tests can at least reduce interference from tests that execute concurrently without needing to put the test in a special directory. The issue with Utils.getFreePort in the test infrastructure library is that it sleeps for 100ms. This test results makes 100 attempts to connect to a refusing endpoint so calling getFreePort before each attempt adds significantly to the execution time. Would you have cycles to check into the getFreePort implementation? I suspect the original rational was to handle cases where the system is out of resources but it doesn't catch IOException and sleeps before the first attempt. I didn't want to change it with this patch as it is used by a lot of other tests. In the mean-time, maybe we can address your concern by just replacing the use of Utils.getFreePort in this test with its own refusingEndpoint method. I've updated the webrev in place so you can see if you agree. If we get Utils.getFreePort fixed then we can replace this usage, probably other cases like this in other tests too. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From huaming.li at oracle.com Tue Mar 6 10:51:42 2018 From: huaming.li at oracle.com (Hamlin Li) Date: Tue, 6 Mar 2018 18:51:42 +0800 Subject: 8198928: (so) SocketChannel connect may deadlock if closed at around same time that connect fails In-Reply-To: <63c22fb6-817c-8485-f7af-31056e8f24f9@oracle.com> References: <99abe020-e895-cae0-2b69-0233f434394e@oracle.com> <005e808f-1a6e-3d07-c1a2-8e5fd65da4f6@oracle.com> <63c22fb6-817c-8485-f7af-31056e8f24f9@oracle.com> Message-ID: On 06/03/2018 6:04 PM, Alan Bateman wrote: > On 06/03/2018 08:39, Hamlin Li wrote: >> >> Hi Alan, >> >> I have minor comment about CloseDuringConnect.java, is it better to >> move following code from main() to test()? >> >> SocketAddress remote = new InetSocketAddress("127.0.0.1", Utils.getFreePort()); >> >> It will reduce the chance to interfere with other tests which run >> concurrently and also create a listening socket at THE free port same >> as this test. >> > Good question! There is a general issue with tests that need a free > port, or a refusing endpoint in this case. Ideally jtreg would support > a per test equivalent of exclusiveAccess.dirs, that way tests can at > least reduce interference from tests that execute concurrently without > needing to put the test in a special directory. > > The issue with Utils.getFreePort in the test infrastructure library is > that it sleeps for 100ms. This test results makes 100 attempts to > connect to a refusing endpoint so calling getFreePort before each > attempt adds significantly to the execution time. Would you have > cycles to check into the getFreePort implementation? I suspect the > original rational was to handle cases where the system is out of > resources but it doesn't catch IOException and sleeps before the first > attempt. I didn't want to change it with this patch as it is used by a > lot of other tests. Got it, I can do it. > > In the mean-time, maybe we can address your concern by just replacing > the use of Utils.getFreePort in this test with its own > refusingEndpoint method. I've updated the webrev in place so you can > see if you agree. If we get Utils.getFreePort fixed then we can > replace this usage, probably other cases like this in other tests too. For this test, I think another way is to use reserved/unused ports as the free port, e.g. port 1, port 4, ... [1] Thank you -Hamlin [1] https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers > > -Alan > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Mar 6 14:14:55 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 6 Mar 2018 14:14:55 +0000 Subject: 8198928: (so) SocketChannel connect may deadlock if closed at around same time that connect fails In-Reply-To: References: <99abe020-e895-cae0-2b69-0233f434394e@oracle.com> <005e808f-1a6e-3d07-c1a2-8e5fd65da4f6@oracle.com> <63c22fb6-817c-8485-f7af-31056e8f24f9@oracle.com> Message-ID: <9b045b41-623e-58ae-9d4b-71164b718a93@oracle.com> On 06/03/2018 10:51, Hamlin Li wrote: > > : > >> >> In the mean-time, maybe we can address your concern by just replacing >> the use of Utils.getFreePort in this test with its own >> refusingEndpoint method. I've updated the webrev in place so you can >> see if you agree. If we get Utils.getFreePort fixed then we can >> replace this usage, probably other cases like this in other tests too. > For this test, I think another way is to use reserved/unused ports as > the free port, e.g. port 1, port 4, ... [1] I agree as it seems unlikely that something will start a listener on a system port (<1024) while the tests are running. For now, the refusingEndpoint() method in the test probes the system ports. If the test infrastructure library is updated to provide something better than the current getFreePort then we can move it to use that. -Alan From Alan.Bateman at oracle.com Tue Mar 6 15:04:37 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 6 Mar 2018 15:04:37 +0000 Subject: 8199120: (so) SocketChannelImpl read/write don't need stateLock when channel is configured non-blocking Message-ID: <7b2de109-978b-28a4-c41d-4b0f64faaddb@oracle.com> As a follow-on to splitting the code paths for blocking and non-blocking I/O, we can do one more optimization to the SocketChannel read/write methods to avoid the need to acquire the stateLock when the channel is configured non-blocking. For anyone following the recent progress, it means that non-blocking operations now only need to acquire readLock, all other locking has been eliminated (blocking I/O have been improved too but still some synchronization needed to support asynchronous close and interrupt). The changes are small and are relative to the patches for DK-8198928 and JDK-8199015 currently under review. http://cr.openjdk.java.net/~alanb/8199120/webrev/index.html -Alan From Roger.Riggs at Oracle.com Tue Mar 6 16:04:09 2018 From: Roger.Riggs at Oracle.com (Roger Riggs) Date: Tue, 6 Mar 2018 11:04:09 -0500 Subject: trivial patch to reduce allocations in nio open* methods In-Reply-To: References: <158BFF8B-586B-4C1C-8DBE-5C5DD3E8D66B@oracle.com> Message-ID: <6d5968b1-a8f7-fddf-768b-95934add74b5@Oracle.com> Hi Michael, To track the request:JDK-8199124 Reduce allocations in nio open* methods Regards, Roger On 3/6/2018 3:17 AM, Michael Skells wrote: > Hi, > below is a trivial patch to reduce allocation on the file open call > path, when no OpenOption is specified( which is a common case in my > test code at least) > > There is some duplication of empty set? handling, but I could not see > a suitable utility class to provide this and didnt want to create a > public method > > Regards > Mike > --- > Index: > src/java.base/share/classes/java/nio/channels/AsynchronousFileChannel.java > IDEA additional info: > Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP > <+>UTF-8 > =================================================================== > --- > src/java.base/share/classes/java/nio/channels/AsynchronousFileChannel.java(revision > 49129:fb9f590b9eeecae202277c0606ed12b4d3f674c3) > +++ > src/java.base/share/classes/java/nio/channels/AsynchronousFileChannel.java(revision > 49129+:fb9f590b9eee+) > @@ -301,8 +301,13 @@ > ? ? ?public static AsynchronousFileChannel open(Path file, > OpenOption... options) > ? ? ? ? ?throws IOException > ? ? ?{ > -? ? ? ? Set set = new HashSet<>(options.length); > -? ? ? ? Collections.addAll(set, options); > +? ? ? ? Set set; > +? ? ? ? if (options.length == 0) { > +? ? ? ? ? ? set = Collections.emptySet(); > +? ? ? ? } else { > +? ? ? ? ? ? set = new HashSet<>(options.length); > +? ? ? ? ? ? Collections.addAll(set, options); > +? ? ? ? } > ? ? ? ? ?return open(file, set, null, NO_ATTRIBUTES); > ? ? ?} > Index: src/java.base/share/classes/java/nio/channels/FileChannel.java > IDEA additional info: > Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP > <+>UTF-8 > =================================================================== > --- > src/java.base/share/classes/java/nio/channels/FileChannel.java(revision > 49129:fb9f590b9eeecae202277c0606ed12b4d3f674c3) > +++ > src/java.base/share/classes/java/nio/channels/FileChannel.java(revision > 49129+:fb9f590b9eee+) > @@ -335,8 +335,11 @@ > ? ? ?public static FileChannel open(Path path, OpenOption... options) > ? ? ? ? ?throws IOException > ? ? ?{ > -? ? ? ? Set set = new HashSet<>(options.length); > -? ? ? ? Collections.addAll(set, options); > +? ? ? ? Set set; > +? ? ? ? if (options.length > 0) { > +? ? ? ? ? ? set = new HashSet<>(options.length); > +? ? ? ? ? ? Collections.addAll(set, options); > +? ? ? ? } else set = Collections.emptySet(); > ? ? ? ? ?return open(path, set, NO_ATTRIBUTES); > ? ? ?} > Index: > src/java.base/share/classes/java/nio/file/spi/FileSystemProvider.java > IDEA additional info: > Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP > <+>UTF-8 > =================================================================== > --- > src/java.base/share/classes/java/nio/file/spi/FileSystemProvider.java(revision > 49129:fb9f590b9eeecae202277c0606ed12b4d3f674c3) > +++ > src/java.base/share/classes/java/nio/file/spi/FileSystemProvider.java(revision > 49129+:fb9f590b9eee+) > @@ -419,20 +419,22 @@ > ? ? ? ? ?throws IOException > ? ? ?{ > ? ? ? ? ?int len = options.length; > -? ? ? ? Set opts = new HashSet<>(len + 3); > +? ? ? ? Set opts ; > ? ? ? ? ?if (len == 0) { > -? ? ? ? ? ? opts.add(StandardOpenOption.CREATE); > - opts.add(StandardOpenOption.TRUNCATE_EXISTING); > +? ? ? ? ? ? opts = DEFAULT_OPEN_OPTIONS; > ? ? ? ? ?} else { > +? ? ? ? ? ? opts = new HashSet<>(len + 1); > ? ? ? ? ? ? ?for (OpenOption opt: options) { > ? ? ? ? ? ? ? ? ?if (opt == StandardOpenOption.READ) > ? ? ? ? ? ? ? ? ? ? ?throw new IllegalArgumentException("READ not > allowed"); > ? ? ? ? ? ? ? ? ?opts.add(opt); > ? ? ? ? ? ? ?} > -? ? ? ? } > -? ? ? ? opts.add(StandardOpenOption.WRITE); > +? ? ? ? ? ? opts.add(StandardOpenOption.WRITE); > +? ? ? ? } > ? ? ? ? ?return Channels.newOutputStream(newByteChannel(path, opts)); > ? ? ?} > +? ? private final static Set DEFAULT_OPEN_OPTIONS = > Collections.unmodifiableSet( > + > EnumSet.of(StandardOpenOption.CREATE,StandardOpenOption.TRUNCATE_EXISTING,StandardOpenOption.WRITE)); > ? ? ?/** > ? ? ? * Opens or creates a file for reading and/or writing, returning > a file > Index: src/java.base/share/classes/java/nio/file/Files.java > IDEA additional info: > Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP > <+>UTF-8 > =================================================================== > --- src/java.base/share/classes/java/nio/file/Files.java(revision > 49129:fb9f590b9eeecae202277c0606ed12b4d3f674c3) > +++ src/java.base/share/classes/java/nio/file/Files.java(revision > 49129+:fb9f590b9eee+) > @@ -410,8 +410,11 @@ > ? ? ?public static SeekableByteChannel newByteChannel(Path path, > OpenOption... options) > ? ? ? ? ?throws IOException > ? ? ?{ > -? ? ? ? Set set = new HashSet<>(options.length); > -? ? ? ? Collections.addAll(set, options); > +? ? ? ? Set set; > +? ? ? ? if (options.length > 0) { > +? ? ? ? ? set = new HashSet<>(options.length); > +? ? ? ? ? Collections.addAll(set, options); > +? ? ? ? } else set = Collections.emptySet(); > ? ? ? ? ?return newByteChannel(path, set); > ? ? ?} > @@ -635,11 +638,12 @@ > ? ? ?public static Path createFile(Path path, FileAttribute... attrs) > ? ? ? ? ?throws IOException > ? ? ?{ > -? ? ? ? EnumSet options = > - EnumSet.of(StandardOpenOption.CREATE_NEW, > StandardOpenOption.WRITE); > -? ? ? ? newByteChannel(path, options, attrs).close(); > +? ? ? ? newByteChannel(path, DEFAULT_CREATE_OPTIONS, attrs).close(); > ? ? ? ? ?return path; > ? ? ?} > +? ? private final static Set > DEFAULT_CREATE_OPTIONS = Collections.unmodifiableSet( > + EnumSet.of(StandardOpenOption.CREATE_NEW, > StandardOpenOption.WRITE) > +? ? ); > ? ? ?/** > ? ? ? * Creates a new directory. The check for the existence of the > file and the > --- > > On Tue, 6 Mar 2018 at 01:12 Brian Burkhalter > > wrote: > > Hello, > > Redirecting to nio-dev. Please reply omitting core-libs-dev. > > If the patch is small you can include it in e-mail to the mailing > list as attachments are generally scrubbed. > > Thanks, > > Brian > > On Mar 5, 2018, at 4:35 PM, Michael Skells > wrote: > >> I have a trivial patch to reduce allocation of OpenOption sets >> where no >> args are supplied >> >> Spotted this while working on scala compiler, trying to reduce >> allocations >> >> I cannot build JDK due to visual studio issues >> >> Can someone progress the patch for me? Dont want to send a patch >> to this >> mailgroup! >> >> I have previously signed OCA and contributed (although it has >> been a while) > From brian.burkhalter at oracle.com Tue Mar 6 16:45:43 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 6 Mar 2018 08:45:43 -0800 Subject: 8198928: (so) SocketChannel connect may deadlock if closed at around same time that connect fails In-Reply-To: <9b045b41-623e-58ae-9d4b-71164b718a93@oracle.com> References: <99abe020-e895-cae0-2b69-0233f434394e@oracle.com> <005e808f-1a6e-3d07-c1a2-8e5fd65da4f6@oracle.com> <63c22fb6-817c-8485-f7af-31056e8f24f9@oracle.com> <9b045b41-623e-58ae-9d4b-71164b718a93@oracle.com> Message-ID: On Mar 6, 2018, at 6:14 AM, Alan Bateman wrote: >>> In the mean-time, maybe we can address your concern by just replacing the use of Utils.getFreePort in this test with its own refusingEndpoint method. I've updated the webrev in place so you can see if you agree. If we get Utils.getFreePort fixed then we can replace this usage, probably other cases like this in other tests too. >> For this test, I think another way is to use reserved/unused ports as the free port, e.g. port 1, port 4, ... [1] > I agree as it seems unlikely that something will start a listener on a system port (<1024) while the tests are running. For now, the refusingEndpoint() method in the test probes the system ports. If the test infrastructure library is updated to provide something better than the current getFreePort then we can move it to use that. Presumably CloseDuringConnect fails regularly without the source code changes applied? Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Mar 6 16:54:54 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 6 Mar 2018 16:54:54 +0000 Subject: 8198928: (so) SocketChannel connect may deadlock if closed at around same time that connect fails In-Reply-To: References: <99abe020-e895-cae0-2b69-0233f434394e@oracle.com> <005e808f-1a6e-3d07-c1a2-8e5fd65da4f6@oracle.com> <63c22fb6-817c-8485-f7af-31056e8f24f9@oracle.com> <9b045b41-623e-58ae-9d4b-71164b718a93@oracle.com> Message-ID: <1bfc3a92-a6d1-7f04-0012-3f26f0e25b2a@oracle.com> On 06/03/2018 16:45, Brian Burkhalter wrote: > : > > Presumably CloseDuringConnect fails regularly without the source code > changes applied? > Yes, the varying delay helps to tickle it quickly on all machines that I've tried. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Mar 6 16:55:36 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 6 Mar 2018 08:55:36 -0800 Subject: 8198928: (so) SocketChannel connect may deadlock if closed at around same time that connect fails In-Reply-To: <1bfc3a92-a6d1-7f04-0012-3f26f0e25b2a@oracle.com> References: <99abe020-e895-cae0-2b69-0233f434394e@oracle.com> <005e808f-1a6e-3d07-c1a2-8e5fd65da4f6@oracle.com> <63c22fb6-817c-8485-f7af-31056e8f24f9@oracle.com> <9b045b41-623e-58ae-9d4b-71164b718a93@oracle.com> <1bfc3a92-a6d1-7f04-0012-3f26f0e25b2a@oracle.com> Message-ID: <5A555E7E-59D9-4B2F-B1F2-5FBC59B8CB06@oracle.com> On Mar 6, 2018, at 8:54 AM, Alan Bateman wrote: >> Presumably CloseDuringConnect fails regularly without the source code changes applied? >> > Yes, the varying delay helps to tickle it quickly on all machines that I've tried. Excellent! Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From forax at univ-mlv.fr Tue Mar 6 16:57:02 2018 From: forax at univ-mlv.fr (Remi Forax) Date: Tue, 6 Mar 2018 17:57:02 +0100 (CET) Subject: trivial patch to reduce allocations in nio open* methods In-Reply-To: <6d5968b1-a8f7-fddf-768b-95934add74b5@Oracle.com> References: <158BFF8B-586B-4C1C-8DBE-5C5DD3E8D66B@oracle.com> <6d5968b1-a8f7-fddf-768b-95934add74b5@Oracle.com> Message-ID: <2052399593.873247.1520355422190.JavaMail.zimbra@u-pem.fr> This patch trades allocations for morphisms of the use sites, in general it's not a trivial tradeof even if here despite the use sites being hidden in SPI implementations we do not control, i think the risk of a slow down is low. The code in FileSystemProvider tries to add a value to an unmodified set (DEFAULT_OPEN_OPTIONS), it will not work ! The patch creates a new static final in Files that will need to be initialized (and become a GC root) even if Files.createFile() is not used, i'm not sure that Files.createFile() is used frequently enough to worth that cost. For AsynchronousFileChannel and FileChannel, we can almost using a Set.of() but open() allows duplicates :(, too bad, because it's like a poster child case for Set.of/Set.copyOf. regards, R?mi ----- Mail original ----- > De: "Roger Riggs" > ?: "nio-dev" > Envoy?: Mardi 6 Mars 2018 17:04:09 > Objet: Re: trivial patch to reduce allocations in nio open* methods > Hi Michael, > > To track the request:JDK-8199124 > Reduce allocations in > nio open* methods > > Regards, Roger > > On 3/6/2018 3:17 AM, Michael Skells wrote: >> Hi, >> below is a trivial patch to reduce allocation on the file open call >> path, when no OpenOption is specified( which is a common case in my >> test code at least) >> >> There is some duplication of empty set? handling, but I could not see >> a suitable utility class to provide this and didnt want to create a >> public method >> >> Regards >> Mike >> --- >> Index: >> src/java.base/share/classes/java/nio/channels/AsynchronousFileChannel.java >> IDEA additional info: >> Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP >> <+>UTF-8 >> =================================================================== >> --- >> src/java.base/share/classes/java/nio/channels/AsynchronousFileChannel.java(revision >> 49129:fb9f590b9eeecae202277c0606ed12b4d3f674c3) >> +++ >> src/java.base/share/classes/java/nio/channels/AsynchronousFileChannel.java(revision >> 49129+:fb9f590b9eee+) >> @@ -301,8 +301,13 @@ >> ? ? ?public static AsynchronousFileChannel open(Path file, >> OpenOption... options) >> ? ? ? ? ?throws IOException >> ? ? ?{ >> -? ? ? ? Set set = new HashSet<>(options.length); >> -? ? ? ? Collections.addAll(set, options); >> +? ? ? ? Set set; >> +? ? ? ? if (options.length == 0) { >> +? ? ? ? ? ? set = Collections.emptySet(); >> +? ? ? ? } else { >> +? ? ? ? ? ? set = new HashSet<>(options.length); >> +? ? ? ? ? ? Collections.addAll(set, options); >> +? ? ? ? } >> ? ? ? ? ?return open(file, set, null, NO_ATTRIBUTES); >> ? ? ?} >> Index: src/java.base/share/classes/java/nio/channels/FileChannel.java >> IDEA additional info: >> Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP >> <+>UTF-8 >> =================================================================== >> --- >> src/java.base/share/classes/java/nio/channels/FileChannel.java(revision >> 49129:fb9f590b9eeecae202277c0606ed12b4d3f674c3) >> +++ >> src/java.base/share/classes/java/nio/channels/FileChannel.java(revision >> 49129+:fb9f590b9eee+) >> @@ -335,8 +335,11 @@ >> ? ? ?public static FileChannel open(Path path, OpenOption... options) >> ? ? ? ? ?throws IOException >> ? ? ?{ >> -? ? ? ? Set set = new HashSet<>(options.length); >> -? ? ? ? Collections.addAll(set, options); >> +? ? ? ? Set set; >> +? ? ? ? if (options.length > 0) { >> +? ? ? ? ? ? set = new HashSet<>(options.length); >> +? ? ? ? ? ? Collections.addAll(set, options); >> +? ? ? ? } else set = Collections.emptySet(); >> ? ? ? ? ?return open(path, set, NO_ATTRIBUTES); >> ? ? ?} >> Index: >> src/java.base/share/classes/java/nio/file/spi/FileSystemProvider.java >> IDEA additional info: >> Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP >> <+>UTF-8 >> =================================================================== >> --- >> src/java.base/share/classes/java/nio/file/spi/FileSystemProvider.java(revision >> 49129:fb9f590b9eeecae202277c0606ed12b4d3f674c3) >> +++ >> src/java.base/share/classes/java/nio/file/spi/FileSystemProvider.java(revision >> 49129+:fb9f590b9eee+) >> @@ -419,20 +419,22 @@ >> ? ? ? ? ?throws IOException >> ? ? ?{ >> ? ? ? ? ?int len = options.length; >> -? ? ? ? Set opts = new HashSet<>(len + 3); >> +? ? ? ? Set opts ; >> ? ? ? ? ?if (len == 0) { >> -? ? ? ? ? ? opts.add(StandardOpenOption.CREATE); >> - opts.add(StandardOpenOption.TRUNCATE_EXISTING); >> +? ? ? ? ? ? opts = DEFAULT_OPEN_OPTIONS; >> ? ? ? ? ?} else { >> +? ? ? ? ? ? opts = new HashSet<>(len + 1); >> ? ? ? ? ? ? ?for (OpenOption opt: options) { >> ? ? ? ? ? ? ? ? ?if (opt == StandardOpenOption.READ) >> ? ? ? ? ? ? ? ? ? ? ?throw new IllegalArgumentException("READ not >> allowed"); >> ? ? ? ? ? ? ? ? ?opts.add(opt); >> ? ? ? ? ? ? ?} >> -? ? ? ? } >> -? ? ? ? opts.add(StandardOpenOption.WRITE); >> +? ? ? ? ? ? opts.add(StandardOpenOption.WRITE); >> +? ? ? ? } >> ? ? ? ? ?return Channels.newOutputStream(newByteChannel(path, opts)); >> ? ? ?} >> +? ? private final static Set DEFAULT_OPEN_OPTIONS = >> Collections.unmodifiableSet( >> + >> EnumSet.of(StandardOpenOption.CREATE,StandardOpenOption.TRUNCATE_EXISTING,StandardOpenOption.WRITE)); >> ? ? ?/** >> ? ? ? * Opens or creates a file for reading and/or writing, returning >> a file >> Index: src/java.base/share/classes/java/nio/file/Files.java >> IDEA additional info: >> Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP >> <+>UTF-8 >> =================================================================== >> --- src/java.base/share/classes/java/nio/file/Files.java(revision >> 49129:fb9f590b9eeecae202277c0606ed12b4d3f674c3) >> +++ src/java.base/share/classes/java/nio/file/Files.java(revision >> 49129+:fb9f590b9eee+) >> @@ -410,8 +410,11 @@ >> ? ? ?public static SeekableByteChannel newByteChannel(Path path, >> OpenOption... options) >> ? ? ? ? ?throws IOException >> ? ? ?{ >> -? ? ? ? Set set = new HashSet<>(options.length); >> -? ? ? ? Collections.addAll(set, options); >> +? ? ? ? Set set; >> +? ? ? ? if (options.length > 0) { >> +? ? ? ? ? set = new HashSet<>(options.length); >> +? ? ? ? ? Collections.addAll(set, options); >> +? ? ? ? } else set = Collections.emptySet(); >> ? ? ? ? ?return newByteChannel(path, set); >> ? ? ?} >> @@ -635,11 +638,12 @@ >> ? ? ?public static Path createFile(Path path, FileAttribute... attrs) >> ? ? ? ? ?throws IOException >> ? ? ?{ >> -? ? ? ? EnumSet options = >> - EnumSet.of(StandardOpenOption.CREATE_NEW, >> StandardOpenOption.WRITE); >> -? ? ? ? newByteChannel(path, options, attrs).close(); >> +? ? ? ? newByteChannel(path, DEFAULT_CREATE_OPTIONS, attrs).close(); >> ? ? ? ? ?return path; >> ? ? ?} >> +? ? private final static Set >> DEFAULT_CREATE_OPTIONS = Collections.unmodifiableSet( >> + EnumSet.of(StandardOpenOption.CREATE_NEW, >> StandardOpenOption.WRITE) >> +? ? ); >> ? ? ?/** >> ? ? ? * Creates a new directory. The check for the existence of the >> file and the >> --- >> >> On Tue, 6 Mar 2018 at 01:12 Brian Burkhalter >> > wrote: >> >> Hello, >> >> Redirecting to nio-dev. Please reply omitting core-libs-dev. >> >> If the patch is small you can include it in e-mail to the mailing >> list as attachments are generally scrubbed. >> >> Thanks, >> >> Brian >> >> On Mar 5, 2018, at 4:35 PM, Michael Skells > > wrote: >> >>> I have a trivial patch to reduce allocation of OpenOption sets >>> where no >>> args are supplied >>> >>> Spotted this while working on scala compiler, trying to reduce >>> allocations >>> >>> I cannot build JDK due to visual studio issues >>> >>> Can someone progress the patch for me? Dont want to send a patch >>> to this >>> mailgroup! >>> >>> I have previously signed OCA and contributed (although it has >>> been a while) From Alan.Bateman at oracle.com Tue Mar 6 17:40:30 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 6 Mar 2018 17:40:30 +0000 Subject: trivial patch to reduce allocations in nio open* methods In-Reply-To: References: <158BFF8B-586B-4C1C-8DBE-5C5DD3E8D66B@oracle.com> Message-ID: <2ba60a55-baf3-8542-dac6-70c33e3f5f25@oracle.com> On 06/03/2018 08:17, Michael Skells wrote: > Hi, > below is a trivial patch to reduce allocation on the file open call > path, when no OpenOption is specified( which is a common case in my > test code at least) > > There is some duplication of empty set? handling, but I could not see > a suitable utility class to provide this and didnt want to create a > public method I think we should start by changing the methods that take varargs so that they use the static factory methods for immutable collections, e.g. Set.of(options). That would be a good cleanup before seeing it's wroth trying to do more for the zero options case. -Alan From forax at univ-mlv.fr Tue Mar 6 18:16:01 2018 From: forax at univ-mlv.fr (Remi Forax) Date: Tue, 6 Mar 2018 19:16:01 +0100 (CET) Subject: trivial patch to reduce allocations in nio open* methods In-Reply-To: <2ba60a55-baf3-8542-dac6-70c33e3f5f25@oracle.com> References: <158BFF8B-586B-4C1C-8DBE-5C5DD3E8D66B@oracle.com> <2ba60a55-baf3-8542-dac6-70c33e3f5f25@oracle.com> Message-ID: <2128062705.897263.1520360161340.JavaMail.zimbra@u-pem.fr> Hi Alan, as i said, the issue is that the static factories of Set does not allow duplicate values while the current spec for FileChannel.open() etc, allow duplicate flags. R?mi ----- Mail original ----- > De: "Alan Bateman" > ?: "Michael Skells" , "Brian Burkhalter" > Cc: "nio-dev" > Envoy?: Mardi 6 Mars 2018 18:40:30 > Objet: Re: trivial patch to reduce allocations in nio open* methods > On 06/03/2018 08:17, Michael Skells wrote: >> Hi, >> below is a trivial patch to reduce allocation on the file open call >> path, when no OpenOption is specified( which is a common case in my >> test code at least) >> >> There is some duplication of empty set? handling, but I could not see >> a suitable utility class to provide this and didnt want to create a >> public method > I think we should start by changing the methods that take varargs so > that they use the static factory methods for immutable collections, e.g. > Set.of(options). That would be a good cleanup before seeing it's wroth > trying to do more for the zero options case. > > -Alan From chris.hegarty at oracle.com Tue Mar 6 18:34:21 2018 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 6 Mar 2018 18:34:21 +0000 Subject: trivial patch to reduce allocations in nio open* methods In-Reply-To: <2ba60a55-baf3-8542-dac6-70c33e3f5f25@oracle.com> References: <158BFF8B-586B-4C1C-8DBE-5C5DD3E8D66B@oracle.com> <2ba60a55-baf3-8542-dac6-70c33e3f5f25@oracle.com> Message-ID: <0803a885-5938-6549-262e-40064d73c513@oracle.com> On 06/03/18 17:40, Alan Bateman wrote: > On 06/03/2018 08:17, Michael Skells wrote: >> Hi, >> below is a trivial patch to reduce allocation on the file open call >> path, when no OpenOption is specified( which is a common case in my >> test code at least) >> >> There is some duplication of empty set handling, but I could not see >> a suitable utility class to provide this and didnt want to create a >> public method > I think we should start by changing the methods that take varargs so > that they use the static factory methods for immutable collections, e.g. > Set.of(options). That would be a good cleanup before seeing it's wroth > trying to do more for the zero options case. It seems crummy, but using Set.of will require some care when handling duplicate options. The code currently silently ignores duplicates, if I'm not mistaken. -Chris. From Alan.Bateman at oracle.com Tue Mar 6 19:05:25 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 6 Mar 2018 19:05:25 +0000 Subject: trivial patch to reduce allocations in nio open* methods In-Reply-To: <0803a885-5938-6549-262e-40064d73c513@oracle.com> References: <158BFF8B-586B-4C1C-8DBE-5C5DD3E8D66B@oracle.com> <2ba60a55-baf3-8542-dac6-70c33e3f5f25@oracle.com> <0803a885-5938-6549-262e-40064d73c513@oracle.com> Message-ID: On 06/03/2018 18:34, Chris Hegarty wrote: > > It seems crummy, but using Set.of will require some care when > handling duplicate options. The code currently silently ignores > duplicates, if I'm not mistaken. I just read R?mi's mail now. So yes, it could only use Set.of for options.length < 2. -Alan From brian.burkhalter at oracle.com Tue Mar 6 21:12:22 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 6 Mar 2018 13:12:22 -0800 Subject: 8199120: (so) SocketChannelImpl read/write don't need stateLock when channel is configured non-blocking In-Reply-To: <7b2de109-978b-28a4-c41d-4b0f64faaddb@oracle.com> References: <7b2de109-978b-28a4-c41d-4b0f64faaddb@oracle.com> Message-ID: <13DD8A9C-64A7-4C7E-85FD-55B3C5B68AD3@oracle.com> On Mar 6, 2018, at 7:04 AM, Alan Bateman wrote: > The changes are small and are relative to the patches for DK-8198928 and JDK-8199015 currently under review. > > http://cr.openjdk.java.net/~alanb/8199120/webrev/index.html I think these look OK. Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike.skells1 at gmail.com Tue Mar 6 22:00:15 2018 From: mike.skells1 at gmail.com (Michael Skells) Date: Tue, 06 Mar 2018 22:00:15 +0000 Subject: trivial patch to reduce allocations in nio open* methods In-Reply-To: References: <158BFF8B-586B-4C1C-8DBE-5C5DD3E8D66B@oracle.com> <2ba60a55-baf3-8542-dac6-70c33e3f5f25@oracle.com> <0803a885-5938-6549-262e-40064d73c513@oracle.com> Message-ID: Hi Remi, the code in FileSystemProvider does not add to the immutable set, if we are talking about the same method - the add of the WRITE only occurs in the case where options are specified, and is pre-populated in the default case. public OutputStream newOutputStream(Path path, OpenOption... options) throws IOException { int len = options.length; Set opts ; if (len == 0) { opts = DEFAULT_OPEN_OPTIONS; } else { opts = new HashSet<>(len + 1); for (OpenOption opt: options) { if (opt == StandardOpenOption.READ) throw new IllegalArgumentException("READ not allowed"); opts.add(opt); } opts.add(StandardOpenOption.WRITE); } return Channels.newOutputStream(newByteChannel(path, opts)); } Files.createFile() - The usage that I was interested in is the new input or output streams - I just looked through the other methods that had similar patterns What is the next step - are you looking for me to do any rework? I dont have the ability to validate performance in the different platforms, other than allocation (which is where I came in) Regards Mike On Tue, 6 Mar 2018 at 19:05 Alan Bateman wrote: > On 06/03/2018 18:34, Chris Hegarty wrote: > > > > It seems crummy, but using Set.of will require some care when > > handling duplicate options. The code currently silently ignores > > duplicates, if I'm not mistaken. > I just read R?mi's mail now. So yes, it could only use Set.of for > options.length < 2. > > -Alan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From huaming.li at oracle.com Wed Mar 7 01:37:37 2018 From: huaming.li at oracle.com (Hamlin Li) Date: Wed, 7 Mar 2018 09:37:37 +0800 Subject: 8198928: (so) SocketChannel connect may deadlock if closed at around same time that connect fails In-Reply-To: <9b045b41-623e-58ae-9d4b-71164b718a93@oracle.com> References: <99abe020-e895-cae0-2b69-0233f434394e@oracle.com> <005e808f-1a6e-3d07-c1a2-8e5fd65da4f6@oracle.com> <63c22fb6-817c-8485-f7af-31056e8f24f9@oracle.com> <9b045b41-623e-58ae-9d4b-71164b718a93@oracle.com> Message-ID: <5bdbbfe1-6053-426b-e3db-0af02dbdcecf@oracle.com> On 06/03/2018 10:14 PM, Alan Bateman wrote: > On 06/03/2018 10:51, Hamlin Li wrote: >> >> : >> >>> >>> In the mean-time, maybe we can address your concern by just >>> replacing the use of Utils.getFreePort in this test with its own >>> refusingEndpoint method. I've updated the webrev in place so you can >>> see if you agree. If we get Utils.getFreePort fixed then we can >>> replace this usage, probably other cases like this in other tests too. >> For this test, I think another way is to use reserved/unused ports as >> the free port, e.g. port 1, port 4, ... [1] > I agree as it seems unlikely that something will start a listener on a > system port (<1024) while the tests are running. For now, the > refusingEndpoint() method in the test probes the system ports. If the > test infrastructure library is updated to provide something better > than the current getFreePort then we can move it to use that. Hi Alan, It looks good to me. And I think refusingEndpoint() is a good candidate to move into Utils class of testlibrary, I can do it when/if I make change for getFreePort. Thank you -Hamlin > > -Alan From Alan.Bateman at oracle.com Wed Mar 7 09:25:50 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 7 Mar 2018 09:25:50 +0000 Subject: 8198928: (so) SocketChannel connect may deadlock if closed at around same time that connect fails In-Reply-To: <5bdbbfe1-6053-426b-e3db-0af02dbdcecf@oracle.com> References: <99abe020-e895-cae0-2b69-0233f434394e@oracle.com> <005e808f-1a6e-3d07-c1a2-8e5fd65da4f6@oracle.com> <63c22fb6-817c-8485-f7af-31056e8f24f9@oracle.com> <9b045b41-623e-58ae-9d4b-71164b718a93@oracle.com> <5bdbbfe1-6053-426b-e3db-0af02dbdcecf@oracle.com> Message-ID: <15945cd2-2862-f808-6fa9-05e6ee951cfd@oracle.com> On 07/03/2018 01:37, Hamlin Li wrote: > : > > It looks good to me. > And I think refusingEndpoint() is a good candidate to move into Utils > class of testlibrary, I can do it when/if I make change for getFreePort. > Thanks. I've created JDK-8199215 to track the test library getFreePort issue, I see you've already picked this up. -Alan From yingqi.lu at intel.com Wed Mar 7 19:14:03 2018 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Wed, 7 Mar 2018 19:14:03 +0000 Subject: adding rsockets support into JDK In-Reply-To: <2482903A-2485-4D58-8119-A0D3C0C8BF43@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD12E2D13@fmsmsx158.amr.corp.intel.com> <0041f766-ee2d-26bc-b2ce-4adad7c8c39c@oracle.com> <2482903A-2485-4D58-8119-A0D3C0C8BF43@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD13405A8@ORSMSX113.amr.corp.intel.com> Hi Alan and Paul, The first draft of the patch is available at http://cr.openjdk.java.net/~vdeshpande/RDMA/webrev.00. This version is based on our previous proposal. Here, we only enabled rsocket inside nio package on SocketChannel and ServerSocketChannel. We would like to share this and give you a quick view the feature itself ? Based on your feedback, we are reconstructing the implementation. We isolate all RDMA-related implementations and put them into jdk.net module. Currently, we are getting close on RdmaSocket and RdmaServerSocket (equivalent to Socket and ServerSocket from java.net package). Before we continue on DatagramSocket and all the SocketChannels (nio package), we would like to get some feedback from you. Attached is a high level diagram on all the newly introduced classes. Please let us know if this is a proper way to implement the feature. Thanks, Lucy From: Paul Sandoz [mailto:paul.sandoz at oracle.com] Sent: Wednesday, January 17, 2018 8:54 AM To: Alan Bateman Cc: Lu, Yingqi ; nio-dev at openjdk.java.net Subject: Re: adding rsockets support into JDK On 17 Jan 2018, at 03:33, Alan Bateman > wrote: On 16/01/2018 22:08, Lu, Yingqi wrote: : The proposed API changes are: java.nio.channels.ServerSocketChannel public static ServerSocketChannel openRDMAChannel() throws IOException java.nio.channels.SocketChannel public static SocketChannel openRDMAChannel() throws IOException public static SocketChannel openRDMAChannel(SocketAddress remote) throws IOException java.nio.channels.spi.SelectorProvider public abstract SocketChannel openRDMAChannel() throws IOException; public abstract ServerSocketChannel openRDMAChannel() throws IOException; Hi Lucy, This sounds like a good project Agreed, and i think an important in cloud-like infrastructure environments. Lucy, do you know if RDMA supports SSL or something equivalent? but it will need discussion on how to integrate into the APIs. It might better to start out with factory methods on a JDK-specific class in the jdk.net example. Alternatively, we could look at slotting it in as a new protocol family implementation along the lines of the DatagramChannel.open(ProtocolFamily) method. Peeling the open methods and consolidating into a separate specific RDMA class or classes is i think better than spreading around the existing nio classes. It?s hard to provide more concrete guidance without a webrev, looking forward to that. Paul. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Diagram.pptx Type: application/vnd.openxmlformats-officedocument.presentationml.presentation Size: 42072 bytes Desc: Diagram.pptx URL: From brian.burkhalter at oracle.com Wed Mar 7 21:57:27 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 7 Mar 2018 13:57:27 -0800 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface Message-ID: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> https://bugs.openjdk.java.net/browse/JDK-8194746 http://cr.openjdk.java.net/~bpb/8194746/webrev.00/ The main changes in are in Path and Paths in java.nio.file. This patch copies the Paths.get() methods to static methods in Path.get() and modifies the former to call the latter respective methods. The Path specification is slightly cleaned up not to refer to Paths nor itself, e.g., ?(see Path).? @implSpec annotations are added to Paths to indicate that the methods simply call their counterparts in Path. All source files in java.base are modified to change Paths.get() to Path.get() and to remove the import for Paths. The copyright year is not yet updated in these files. No tests are added or modified as Paths is final and the Path.get() methods end up being invoked indirectly in existing tests. Perhaps some excision of the specification verbiage in Paths is called for to minimize duplication? Once this starts to achieve consensus a CSR will be filed. Thanks, Brian From forax at univ-mlv.fr Wed Mar 7 22:09:13 2018 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 7 Mar 2018 23:09:13 +0100 (CET) Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> Message-ID: <1076448850.1387603.1520460553785.JavaMail.zimbra@u-pem.fr> Hi Brian, i really like the fact that instead of having Path.get() that delegates to Paths.get(), you wire the methods in the other direction, that's clever. R?mi ----- Mail original ----- > De: "Brian Burkhalter" > ?: "nio-dev" > Envoy?: Mercredi 7 Mars 2018 22:57:27 > Objet: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface > https://bugs.openjdk.java.net/browse/JDK-8194746 > http://cr.openjdk.java.net/~bpb/8194746/webrev.00/ > > The main changes in are in Path and Paths in java.nio.file. > > This patch copies the Paths.get() methods to static methods in Path.get() and > modifies the former to call the latter respective methods. The Path > specification is slightly cleaned up not to refer to Paths nor itself, e.g., > ?(see Path).? @implSpec annotations are added to Paths to indicate that the > methods simply call their counterparts in Path. > > All source files in java.base are modified to change Paths.get() to Path.get() > and to remove the import for Paths. The copyright year is not yet updated in > these files. > > No tests are added or modified as Paths is final and the Path.get() methods end > up being invoked indirectly in existing tests. > > Perhaps some excision of the specification verbiage in Paths is called for to > minimize duplication? > > Once this starts to achieve consensus a CSR will be filed. > > Thanks, > > Brian From brian.burkhalter at oracle.com Wed Mar 7 22:15:08 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 7 Mar 2018 14:15:08 -0800 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: <1076448850.1387603.1520460553785.JavaMail.zimbra@u-pem.fr> References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <1076448850.1387603.1520460553785.JavaMail.zimbra@u-pem.fr> Message-ID: Hi Remi, That?s Alan. ;-) Brian On Mar 7, 2018, at 2:09 PM, Remi Forax wrote: > i really like the fact that instead of having Path.get() that delegates to Paths.get(), you wire the methods in the other direction, > that's clever. -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.sandoz at oracle.com Thu Mar 8 00:37:02 2018 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Wed, 7 Mar 2018 16:37:02 -0800 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> Message-ID: <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> > On Mar 7, 2018, at 1:57 PM, Brian Burkhalter wrote: > > https://bugs.openjdk.java.net/browse/JDK-8194746 > http://cr.openjdk.java.net/~bpb/8194746/webrev.00/ > Path.java 639 * {@link Paths#get(URI) Path.get}{@code (}p{@code .toUri()).equals(}p The link refers to Paths not Path. Nice cleanup. It was always a painful for those new to this area to find the Path factories. Paul. > The main changes in are in Path and Paths in java.nio.file. > > This patch copies the Paths.get() methods to static methods in Path.get() and modifies the former to call the latter respective methods. The Path specification is slightly cleaned up not to refer to Paths nor itself, e.g., ?(see Path).? @implSpec annotations are added to Paths to indicate that the methods simply call their counterparts in Path. > > All source files in java.base are modified to change Paths.get() to Path.get() and to remove the import for Paths. The copyright year is not yet updated in these files. > > No tests are added or modified as Paths is final and the Path.get() methods end up being invoked indirectly in existing tests. > > Perhaps some excision of the specification verbiage in Paths is called for to minimize duplication? > > Once this starts to achieve consensus a CSR will be filed. > > Thanks, > > Brian From brian.burkhalter at oracle.com Thu Mar 8 00:38:38 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 7 Mar 2018 16:38:38 -0800 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> Message-ID: <81E137D7-D95A-4676-8F88-434C8324FEE7@oracle.com> On Mar 7, 2018, at 4:37 PM, Paul Sandoz wrote: > Path.java > > 639 * {@link Paths#get(URI) Path.get}{@code (}p{@code .toUri()).equals(}p > > The link refers to Paths not Path. Missed that one: thanks! > Nice cleanup. It was always a painful for those new to this area to find the Path factories. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Mar 8 08:07:08 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 8 Mar 2018 08:07:08 +0000 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> Message-ID: On 08/03/2018 00:37, Paul Sandoz wrote: > >> On Mar 7, 2018, at 1:57 PM, Brian Burkhalter wrote: >> >> https://bugs.openjdk.java.net/browse/JDK-8194746 >> http://cr.openjdk.java.net/~bpb/8194746/webrev.00/ >> > Path.java > > 639 * {@link Paths#get(URI) Path.get}{@code (}p{@code .toUri()).equals(}p > > The link refers to Paths not Path. > > Nice cleanup. It was always a painful for those new to this area to find the Path factories. > We also to add @since 11 to the new methods and I'm wondering about adding an @apiNote to Path's class description to indicate that it's obsolete and that the static methods defined by Path should be used in preference. I don't think Paths needs to be deprecated of course, I'm just thinking of developers wondering what the Paths class is for. -Alan From Alan.Bateman at oracle.com Thu Mar 8 08:20:29 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 8 Mar 2018 08:20:29 +0000 Subject: adding rsockets support into JDK In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD13405A8@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD12E2D13@fmsmsx158.amr.corp.intel.com> <0041f766-ee2d-26bc-b2ce-4adad7c8c39c@oracle.com> <2482903A-2485-4D58-8119-A0D3C0C8BF43@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD13405A8@ORSMSX113.amr.corp.intel.com> Message-ID: On 07/03/2018 19:14, Lu, Yingqi wrote: > > Hi Alan and Paul, > > The first draft of the patch is available at > http://cr.openjdk.java.net/~vdeshpande/RDMA/webrev.00 > . This > version is based on our previous proposal. Here, we only enabled > rsocket inside nio package on SocketChannel and ServerSocketChannel. > We would like to share this and give you a quick view the feature itself J > > Based on your feedback, we are reconstructing the implementation. We > isolate all RDMA-related implementations and put them into jdk.net > module. Currently, we are getting close on RdmaSocket and > RdmaServerSocket (equivalent to Socket and ServerSocket from java.net > package). Before we continue on DatagramSocket and all the > SocketChannels (nio package), we would like to get some feedback from > you. Attached is a high level diagram on all the newly introduced > classes. Please let us know if this is a proper way to implement the > feature. > > Thanks for the initial webrev, it gives folks here an idea on where this is going. I think a big question will be whether RDMA sockets can be multiplexed with epoll. I see you've dusted off the legacy poll based Selector to use rpoll and I'm wondering whether the file descriptors need to be separated like this. The answer to this has big implications, it make potentially impact how asynchronous close of RDMA sockets configured blocking is implemented too. In terms of API then initially exposing this as factory methods in the jdk.net module seems right. I also need some refactoring it going to be needed to fully support this, along the lines of how NatveDispatcher works, we will likely need the same to replace all Net methods. -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.hegarty at oracle.com Thu Mar 8 09:05:18 2018 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 8 Mar 2018 09:05:18 +0000 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> Message-ID: Brian, > On 7 Mar 2018, at 21:57, Brian Burkhalter wrote: > > https://bugs.openjdk.java.net/browse/JDK-8194746 > http://cr.openjdk.java.net/~bpb/8194746/webrev.00/ This looks good. Just a few minor comments. > The main changes in are in Path and Paths in java.nio.file. > > This patch copies the Paths.get() methods to static methods in Path.get() and modifies the former to call the latter respective methods. The Path specification is slightly cleaned up not to refer to Paths nor itself, e.g., ?(see Path).? @implSpec annotations are added to Paths to indicate that the methods simply call their counterparts in Path. I wonder if an @implSpec is too strong here. I guess I expected to see: *

Equivalent to: {@code Path.get(String, String...)} Example snippets in the javadoc could use {@code ?}, for example * 

{@code 
    *     Path dir = ...
    *     Path path = dir.resolve("file");
    * }
-Chris. From paul.sandoz at oracle.com Thu Mar 8 16:55:49 2018 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Thu, 8 Mar 2018 08:55:49 -0800 Subject: adding rsockets support into JDK In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD13405A8@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD12E2D13@fmsmsx158.amr.corp.intel.com> <0041f766-ee2d-26bc-b2ce-4adad7c8c39c@oracle.com> <2482903A-2485-4D58-8119-A0D3C0C8BF43@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD13405A8@ORSMSX113.amr.corp.intel.com> Message-ID: Hi Lucy, The separation is the key thing. Then i presume it?s easier to decide to keep in the java.base module, put in the jdk.net module, or even it?s own module. Did you consider the latter option? (Note I am not an expect in the specifics of this area so others are likely to have more informed advice on the specifics.) Also may i suggest you consider submitting a JEP for this work (cannot recall if we discussed that before). It looks like there is sufficient work to justify a JEP and it gives good documentation/visibility for the feature. Paul. > On Mar 7, 2018, at 11:14 AM, Lu, Yingqi wrote: > > Hi Alan and Paul, <> > > The first draft of the patch is available at http://cr.openjdk.java.net/~vdeshpande/RDMA/webrev.00 . This version is based on our previous proposal. Here, we only enabled rsocket inside nio package on SocketChannel and ServerSocketChannel. We would like to share this and give you a quick view the feature itself J > > Based on your feedback, we are reconstructing the implementation. We isolate all RDMA-related implementations and put them into jdk.net module. Currently, we are getting close on RdmaSocket and RdmaServerSocket (equivalent to Socket and ServerSocket from java.net package). Before we continue on DatagramSocket and all the SocketChannels (nio package), we would like to get some feedback from you. Attached is a high level diagram on all the newly introduced classes. Please let us know if this is a proper way to implement the feature. > > Thanks, > Lucy > > > From: Paul Sandoz [mailto:paul.sandoz at oracle.com ] > Sent: Wednesday, January 17, 2018 8:54 AM > To: Alan Bateman > > Cc: Lu, Yingqi >; nio-dev at openjdk.java.net > Subject: Re: adding rsockets support into JDK > > > > > On 17 Jan 2018, at 03:33, Alan Bateman > wrote: > > On 16/01/2018 22:08, Lu, Yingqi wrote: > > : > > The proposed API changes are: > java.nio.channels.ServerSocketChannel > public static ServerSocketChannel openRDMAChannel() throws IOException > > java.nio.channels.SocketChannel > public static SocketChannel openRDMAChannel() throws IOException > public static SocketChannel openRDMAChannel(SocketAddress remote) throws IOException > > java.nio.channels.spi.SelectorProvider > public abstract SocketChannel openRDMAChannel() throws IOException; > public abstract ServerSocketChannel openRDMAChannel() throws IOException; > > Hi Lucy, > > This sounds like a good project > > Agreed, and i think an important in cloud-like infrastructure environments. > > Lucy, do you know if RDMA supports SSL or something equivalent? > > > > but it will need discussion on how to integrate into the APIs. It might better to start out with factory methods on a JDK-specific class in the jdk.net example. Alternatively, we could look at slotting it in as a new protocol family implementation along the lines of the DatagramChannel.open(ProtocolFamily) method. > > > Peeling the open methods and consolidating into a separate specific RDMA class or classes is i think better than spreading around the existing nio classes. > > It?s hard to provide more concrete guidance without a webrev, looking forward to that. > > Paul. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.reinhold at oracle.com Thu Mar 8 17:03:50 2018 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Thu, 08 Mar 2018 09:03:50 -0800 Subject: adding rsockets support into JDK In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AD12E2D13@fmsmsx158.amr.corp.intel.com> <0041f766-ee2d-26bc-b2ce-4adad7c8c39c@oracle.com> <2482903A-2485-4D58-8119-A0D3C0C8BF43@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD13405A8@ORSMSX113.amr.corp.intel.com> Message-ID: <20180308090350.223764266@eggemoggin.niobe.net> 2018/3/8 8:55:49 -0800, paul.sandoz at oracle.com: > ... > > Also may i suggest you consider submitting a JEP for this work (cannot > recall if we discussed that before). It looks like there is sufficient > work to justify a JEP and it gives good documentation/visibility for > the feature. Agreed -- this is definitely JEP-worthy. - Mark From yingqi.lu at intel.com Thu Mar 8 17:21:15 2018 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Thu, 8 Mar 2018 17:21:15 +0000 Subject: adding rsockets support into JDK In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AD12E2D13@fmsmsx158.amr.corp.intel.com> <0041f766-ee2d-26bc-b2ce-4adad7c8c39c@oracle.com> <2482903A-2485-4D58-8119-A0D3C0C8BF43@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD13405A8@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD1341CAB@ORSMSX113.amr.corp.intel.com> Hi Alan, Currently, rsocket does not support epoll, which is why I use legacy poll/rpoll method instead on java.nio.channels.Selector. Would you please share a little more details on how it impacts asynchronous close of RDMA sockets configured blocking? Do you refer to a specific type of the SocketChannel or it impacts all the nio socket channels? Thanks, Lucy From: Alan Bateman [mailto:Alan.Bateman at oracle.com] Sent: Thursday, March 8, 2018 12:20 AM To: Lu, Yingqi Cc: nio-dev at openjdk.java.net Subject: Re: adding rsockets support into JDK On 07/03/2018 19:14, Lu, Yingqi wrote: Hi Alan and Paul, The first draft of the patch is available at http://cr.openjdk.java.net/~vdeshpande/RDMA/webrev.00. This version is based on our previous proposal. Here, we only enabled rsocket inside nio package on SocketChannel and ServerSocketChannel. We would like to share this and give you a quick view the feature itself ? Based on your feedback, we are reconstructing the implementation. We isolate all RDMA-related implementations and put them into jdk.net module. Currently, we are getting close on RdmaSocket and RdmaServerSocket (equivalent to Socket and ServerSocket from java.net package). Before we continue on DatagramSocket and all the SocketChannels (nio package), we would like to get some feedback from you. Attached is a high level diagram on all the newly introduced classes. Please let us know if this is a proper way to implement the feature. Thanks for the initial webrev, it gives folks here an idea on where this is going. I think a big question will be whether RDMA sockets can be multiplexed with epoll. I see you've dusted off the legacy poll based Selector to use rpoll and I'm wondering whether the file descriptors need to be separated like this. The answer to this has big implications, it make potentially impact how asynchronous close of RDMA sockets configured blocking is implemented too. In terms of API then initially exposing this as factory methods in the jdk.net module seems right. I also need some refactoring it going to be needed to fully support this, along the lines of how NatveDispatcher works, we will likely need the same to replace all Net methods. -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Thu Mar 8 17:22:46 2018 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Thu, 8 Mar 2018 17:22:46 +0000 Subject: adding rsockets support into JDK In-Reply-To: <20180308090350.223764266@eggemoggin.niobe.net> References: <9ACD5B67AAC5594CB6268234CF29CF9AD12E2D13@fmsmsx158.amr.corp.intel.com> <0041f766-ee2d-26bc-b2ce-4adad7c8c39c@oracle.com> <2482903A-2485-4D58-8119-A0D3C0C8BF43@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD13405A8@ORSMSX113.amr.corp.intel.com> <20180308090350.223764266@eggemoggin.niobe.net> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD1341D16@ORSMSX113.amr.corp.intel.com> Hi Mark and Paul, Thank you for the feedback. I will draft a JEP and submit it to JBS for review and approval. Thanks, Lucy >-----Original Message----- >From: mark.reinhold at oracle.com [mailto:mark.reinhold at oracle.com] >Sent: Thursday, March 8, 2018 9:04 AM >To: paul.sandoz at oracle.com; Lu, Yingqi >Cc: nio-dev at openjdk.java.net >Subject: Re: adding rsockets support into JDK > >2018/3/8 8:55:49 -0800, paul.sandoz at oracle.com: >> ... >> >> Also may i suggest you consider submitting a JEP for this work (cannot >> recall if we discussed that before). It looks like there is sufficient >> work to justify a JEP and it gives good documentation/visibility for >> the feature. > >Agreed -- this is definitely JEP-worthy. > >- Mark From Alan.Bateman at oracle.com Thu Mar 8 21:24:38 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 8 Mar 2018 21:24:38 +0000 Subject: adding rsockets support into JDK In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD1341CAB@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD12E2D13@fmsmsx158.amr.corp.intel.com> <0041f766-ee2d-26bc-b2ce-4adad7c8c39c@oracle.com> <2482903A-2485-4D58-8119-A0D3C0C8BF43@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD13405A8@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1341CAB@ORSMSX113.amr.corp.intel.com> Message-ID: On 08/03/2018 17:21, Lu, Yingqi wrote: > > Hi Alan, > > Currently, rsocket does not support epoll, which is why I use legacy > poll/rpoll method instead on java.nio.channels.Selector. > > Would you please share a little more details on how it impacts > asynchronous close of RDMA sockets configured blocking? Do you refer > to a specific type of the SocketChannel or it impacts all the nio > socket channels? > > All selectable channels associated with a selector provider can be registered with a Selector associated with the same provider. I don't think a separate selector provider will work because you'd still have the issue of multiplexing rsockets with datagram channels created by the same provider. So that would one will need a bit of consideration to avoid an overly complicated solution. The legacy poll based Selector has not been used for several releases and is not included in the Linux build - this is why you had to change CompileJavaModules.gmk to un-exclude these sources from compilation. The poll based Selector has a number of issues and I suspect will need work if it is to come back to life. Also it also won't scale like the epoll Selector but that might not be an issue if there only a small number of connections over the fabric. My comment on asynchronous close relates to blocking I/O operations which is a two step process to avoid I/O on a recycled file descriptor during the close. I would expect dup2 to work but it will need to be checked. At least in JDK 11, it won't normally attempt to dup2 a file descriptor for a socket registered with a Selector but there are bunch of corner cases that may need investigation - one example is where a channel is registered with a Selector, its selection key is canceled, the channel configured blocking, and a blocking I/O operation initiated. This type of scenario could mean the implementation attempts to dup2 a file descriptor that you are rpoll-ing. I've been through all these cases in the past with SDP but I can't say if there will be issues with rsockets. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Thu Mar 8 22:52:08 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 8 Mar 2018 14:52:08 -0800 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> Message-ID: <69614880-3004-4B65-ADB3-D1ED807A88B5@oracle.com> Updated patch with the improvements suggested below: http://cr.openjdk.java.net/~bpb/8194746/webrev.01/ On Mar 8, 2018, at 12:07 AM, Alan Bateman wrote: > We also to add @since 11 to the new methods Done. > and I'm wondering about adding an @apiNote to Path's class description to indicate that it's obsolete and that the static methods defined by Path should be used in preference. I don't think Paths needs to be deprecated of course, I'm just thinking of developers wondering what the Paths class is for. Added to Paths?s class description: 35 * @apiNote 36 * This class is obsolete. Its {@code get()} methods simply invoke the 37 * corresponding static methods defined by {@link #Path} which should be used 38 * in preference. On Mar 8, 2018, at 1:05 AM, Chris Hegarty wrote: > I wonder if an @implSpec is too strong here. I guess I expected to see: > *

Equivalent to: {@code Path.get(String, String?)} Changed for example to: 77 *

This method simply invokes {@link Path#get(String,String...) 78 * Path.get(String, String...)} with the given parameters. > Example snippets in the javadoc could use {@code ?}, for example > > * 

{@code 
>    *     Path dir = ...
>    *     Path path = dir.resolve("file");
>    * }
Fixed. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.lloyd at redhat.com Fri Mar 9 15:23:20 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Fri, 9 Mar 2018 09:23:20 -0600 Subject: [11] RFR : java.nio.Buffer attachments In-Reply-To: References: Message-ID: I was just wondering if anyone has had a chance to give this a look and a think. Thanks! On Fri, Mar 2, 2018 at 9:58 AM, David Lloyd wrote: > One of the major classic difficulties with NIO programming (and > therefore, usage of NIO-based frameworks) is the management of > buffers. This commonly applies specifically to (but is definitely not > limited to) handling of direct ByteBuffers. One of the primary > difficulties relates to the topic of buffer allocation, pooling, and > reuse. > > I contend that a well-behaved buffer pool necessarily has the > following characteristics: > > ? An explicit `free` operation (i.e. not normally relying on GC to come around) > ? Some mechanism to help prevent double-free > ? Some mechanism to help prevent other pollution of the pool (i.e. > pooling buffers that do not belong to the pool) > ? Graceful handling of duplicates, slices, and other types of views > > In addition, the following characteristics also may be desirable: > > ? Allocation of fewer, larger buffer regions backing the pooled > buffers which may be slices of these regions, maybe even to multiple > hierarchical levels > ? Some mechanism to signify that the status of a given pooled buffer > can no longer be determined (i.e. fall back to using a GC-based > mechanism to release the buffer) > ? Employment of some reasonable amount of fail-safe against usage of > buffers which have been freed, within practical limits > > Existing buffer pools may rely on wrapper objects, which tend to be > difficult to use with scatter/gather and add a level of indirection to > common I/O code paths, or (to avoid wrapper objects) they may rely on > heuristic information acquired from the Buffer API (such as capacity, > isDirect, isReadOnly, etc.) to make a guess as to whether the buffer > belongs to the pool, which also comes with several critical > disadvantages. A Buffer attachment API would solve these problems far > more gracefully. > > The Buffer attachment API would have to meet the following requirements: > > ? The attachment must be acquirable for usage by pools and other buffer managers > ? Attachments should be protected from interference by buffer users > and other attachment implementations > ? No performance degradation to standard I/O and manipulation operations > ? Minimal memory overhead in the existing case where no attachment is present > > I've attached a patch which adds a buffer API that meets these > requirements. The following implementation points are notable: > > ? One extra final field is added to java.nio.Buffer to hold the > optional attachment > ? The BufferAttachment API is "locked down" allowing only the > implementation of the attachment to create buffers with that > attachment, preventing attachments from being copied to other buffers > by misbehaving code > ? While anyone can acquire the BufferAttachment object for a given > Buffer, all of the members of that class are protected, preventing > interference > ? Attachments have an opportunity to handle duplicate/slice and view > creation by returning null, returning a copy of the attachment, or > returning the same attachment instance, depending on the desired > behavior > ? Only creating ByteBuffers (direct and heap) and wrapping byte arrays > are presently supported; other buffer types can be created as views of > the ByteBuffer, or support can be added (now or at a later time) for > creating them directly > ? In particular, the patch does not implement a way to add an > attachment to buffers created via FileChannel mapping or via JNI, or > other such "back-channel" approaches; these features may potentially > be added at a later time > > Using this attachment API, it is possible to create buffer pool > implementations with a variety of different behaviors, many of which > meet the above requirements without the drawbacks of wrapper objects. > It may also open the door for JDK-managed smart buffer pools in the > future. > > Please review and give feedback at your convenience. Thanks! > -- > - DML -- - DML From Alan.Bateman at oracle.com Fri Mar 9 16:00:39 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 9 Mar 2018 16:00:39 +0000 Subject: [11] RFR : java.nio.Buffer attachments In-Reply-To: References: Message-ID: On 09/03/2018 15:23, David Lloyd wrote: > I was just wondering if anyone has had a chance to give this a look > and a think. Thanks! > I think this needs a compelling example to demonstrate the motivation. Changing ByteBuffer duplicate to run arbitrary code to create a new attachment object for the new buffer is one thing that I found a bit strange in the proposal. Reading your original mail, it wasn't clear to me how different this is to maintaining a maps buffers to object, or buffer to buffer pool. Do you have such a map today? -Alan From david.lloyd at redhat.com Fri Mar 9 16:51:42 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Fri, 9 Mar 2018 10:51:42 -0600 Subject: [11] RFR : java.nio.Buffer attachments In-Reply-To: References: Message-ID: On Fri, Mar 9, 2018 at 10:00 AM, Alan Bateman wrote: > On 09/03/2018 15:23, David Lloyd wrote: >> >> I was just wondering if anyone has had a chance to give this a look >> and a think. Thanks! >> > I think this needs a compelling example to demonstrate the motivation. > Changing ByteBuffer duplicate to run arbitrary code to create a new > attachment object for the new buffer is one thing that I found a bit strange > in the proposal. The interesting thing about giving an attachment the chance to react to view creation is that it allows an implementation to track how many outstanding views there are of a given buffer. This in turn could allow for (for example) implementations which reference-count the backing buffer, and even "smart" pooling allocators which can intelligently split large buffers into small buffers, or coalesce adjacent small buffers into larger buffers. A buffer pool implementation which requires each view to be freed could even allow unused regions to be reallocated as smaller buffers. It also provides an opportunity for a variety of useful leak and error detection debug modes. Given an implementation which only allows freeing of the "original" pooled buffer, it would be simple to add a debugging mode where detailed information about invalid frees could be logged, e.g. "An attempt was made to free a view of buffer xyz; the buffer was allocated at Blah.java:1234 and the view was created at Blah.java:4567". The entire duplication history of a particular buffer could in fact be tracked and reported. This kind of thing would have been quite useful to me on more than one occasion. > Reading your original mail, it wasn't clear to me how > different this is to maintaining a maps buffers to object, or buffer to > buffer pool. Do you have such a map today? We've tried a number of approaches with varying degrees of success, but all central pool management structures (including maps or queues) have tended to be subject to observable contention. We tend to use various striping strategies to try to mitigate this, but in the end thread-local caching has been the best solution so far. However it is necessary that it be possible to get information about the poolability of a buffer from the buffer instance itself for thread-local caching to really be effective, otherwise we're back to centralized contention again. Using a map efficiently in conjunction with thread-local caching is fairly difficult AFAICT. Direct attachment is really hard to beat when associating one object with another, in terms of both storage and access efficiency, as well as simplicity. In the simple/common case - where we would only be tracking and freeing the "root" buffer - direct attachment should be much faster than using a map, and the implementation is relatively simple as well. A map would necessarily be keyed by the buffer instance. This requires employment of an identity-based concurrent hash map, due to ByteBuffer's content-dependent equals/hashCode implementation; since there isn't one in the JDK public API, this is something else that must be produced or acquired by users. It is not possible to track buffer views using a central map (or any other way, AFAICT) without introducing a hook into view creation. -- - DML From mike.skells1 at gmail.com Tue Mar 13 00:01:20 2018 From: mike.skells1 at gmail.com (Michael Skells) Date: Tue, 13 Mar 2018 00:01:20 +0000 Subject: trivial patch to reduce allocations in nio open* methods In-Reply-To: References: <158BFF8B-586B-4C1C-8DBE-5C5DD3E8D66B@oracle.com> <2ba60a55-baf3-8542-dac6-70c33e3f5f25@oracle.com> <0803a885-5938-6549-262e-40064d73c513@oracle.com> Message-ID: <5aa714cf.c30c1c0a.fbbdc.b3fb@mx.google.com> Hi, I am not sure what the status is for this patch Are you waiting for more input, or some update from me? Is this a viable patch? Is it just in the queue of minor noise to consider later? Regards Mike From: Michael Skells Sent: 06 March 2018 22:00 To: Alan Bateman Cc: Chris Hegarty; nio-dev Subject: Re: trivial patch to reduce allocations in nio open* methods Hi Remi, the code in FileSystemProvider does not add to the immutable set, if we are talking about the same method - the add of the WRITE only occurs in the case where options are specified, and is pre-populated in the default case. public OutputStream newOutputStream(Path path, OpenOption... options) throws IOException { int len = options.length; Set opts ; if (len == 0) { opts = DEFAULT_OPEN_OPTIONS; } else { opts = new HashSet<>(len + 1); for (OpenOption opt: options) { if (opt == StandardOpenOption.READ) throw new IllegalArgumentException("READ not allowed"); opts.add(opt); } opts.add(StandardOpenOption.WRITE); } return Channels.newOutputStream(newByteChannel(path, opts)); } ?Files.createFile() - The usage that I was interested in is the new input or output streams - I just looked through the other methods that had similar patterns What is the next step - are you looking for me to do any rework? I dont have the ability to validate performance in the different platforms, other than allocation (which is where I came in) Regards Mike On Tue, 6 Mar 2018 at 19:05 Alan Bateman wrote: On 06/03/2018 18:34, Chris Hegarty wrote: > > It seems crummy, but using Set.of will require some care when > handling duplicate options. The code currently silently ignores > duplicates, if I'm not mistaken. I just read R?mi's mail now. So yes, it could only use Set.of for options.length < 2. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Mar 13 13:53:39 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 13 Mar 2018 13:53:39 +0000 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: <69614880-3004-4B65-ADB3-D1ED807A88B5@oracle.com> References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> <69614880-3004-4B65-ADB3-D1ED807A88B5@oracle.com> Message-ID: <992c8053-60aa-d092-6a51-06d3c203c618@oracle.com> On 08/03/2018 22:52, Brian Burkhalter wrote: > Updated patch with the improvements suggested below: > > http://cr.openjdk.java.net/~bpb/8194746/webrev.01/ > > What would you think of trimming the javadoc of the Paths.get methods so that it is more obvious that they just delegate to Path.get? Separately, Brian Goetz suggested off-list that it would be more consistent if these factory methods were named "of" so I assume the webrev will be updated to see how that looks. -Alan From forax at univ-mlv.fr Tue Mar 13 14:04:29 2018 From: forax at univ-mlv.fr (Remi Forax) Date: Tue, 13 Mar 2018 15:04:29 +0100 (CET) Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: <992c8053-60aa-d092-6a51-06d3c203c618@oracle.com> References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> <69614880-3004-4B65-ADB3-D1ED807A88B5@oracle.com> <992c8053-60aa-d092-6a51-06d3c203c618@oracle.com> Message-ID: <715116280.749084.1520949869741.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "Alan Bateman" > ?: "Brian Burkhalter" > Cc: "nio-dev" > Envoy?: Mardi 13 Mars 2018 14:53:39 > Objet: Re: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface > On 08/03/2018 22:52, Brian Burkhalter wrote: >> Updated patch with the improvements suggested below: >> >> http://cr.openjdk.java.net/~bpb/8194746/webrev.01/ >> >> > What would you think of trimming the javadoc of the Paths.get methods so > that it is more obvious that they just delegate to Path.get? yes ! > > Separately, Brian Goetz suggested off-list that it would be more > consistent if these factory methods were named "of" so I assume the > webrev will be updated to see how that looks. any name will be better than "get", and yes the recent vogue is to use "of", you can also use "path" if you suppose people will use an import static, "of" doesn't well with an import static. > > -Alan R?mi From Alan.Bateman at oracle.com Tue Mar 13 14:09:23 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 13 Mar 2018 14:09:23 +0000 Subject: Please test jdk-11-ea+4 Message-ID: <26d7dcbf-fa67-d227-2905-0f4496ab88b7@oracle.com> jdk-11-ea+4 is the latest JDK 11 EA build [1]. It contains significant changes to SocketChannel and friends to separate the blocking and non-blocking code paths and also a re-work of asynchronous close. As discussed here, and summarized in a release note, the changes involve two subtle behavior changes, namely that configureBlocking may block if there is a I/O operation in progress, and performing a non-blocking I/O with the interrupt status set no longer closes the channel. For those invested in this area, now would be a good time to try out the latest build to see if you run into any issues. -Alan [1] http://jdk.java.net/11/ From Alan.Bateman at oracle.com Tue Mar 13 14:25:33 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 13 Mar 2018 14:25:33 +0000 Subject: trivial patch to reduce allocations in nio open* methods In-Reply-To: <5aa714cf.c30c1c0a.fbbdc.b3fb@mx.google.com> References: <158BFF8B-586B-4C1C-8DBE-5C5DD3E8D66B@oracle.com> <2ba60a55-baf3-8542-dac6-70c33e3f5f25@oracle.com> <0803a885-5938-6549-262e-40064d73c513@oracle.com> <5aa714cf.c30c1c0a.fbbdc.b3fb@mx.google.com> Message-ID: On 13/03/2018 00:01, Michael Skells wrote: > > Hi, > > I am not sure what the status is for this patch > > Are you waiting for more input, or some update from me? > > Is this a viable patch? > > Is it just in the queue of minor noise to consider later? > > Do you have any data on how it helps the scala compiler? Also are you seeing this with the default file system or is something else (the zip file system maybe)? I ask as I would have expected the I/O cost and opening the file to dominant rather than processing the arguments that specify how the file is opened. BTW: I'm not opposed to changing anything here, instead I think we need a bit more of the story. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Mar 13 14:42:03 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 13 Mar 2018 07:42:03 -0700 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: <715116280.749084.1520949869741.JavaMail.zimbra@u-pem.fr> References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> <69614880-3004-4B65-ADB3-D1ED807A88B5@oracle.com> <992c8053-60aa-d092-6a51-06d3c203c618@oracle.com> <715116280.749084.1520949869741.JavaMail.zimbra@u-pem.fr> Message-ID: On Mar 13, 2018, at 7:04 AM, Remi Forax wrote: >> On 08/03/2018 22:52, Brian Burkhalter wrote: >>> Updated patch with the improvements suggested below: >>> >>> http://cr.openjdk.java.net/~bpb/8194746/webrev.01/ >>> >>> >> What would you think of trimming the javadoc of the Paths.get methods so >> that it is more obvious that they just delegate to Path.get? > > yes ! Roger also suggested that off line and it was actually what I wanted to do in the first place but did not do it. >> Separately, Brian Goetz suggested off-list that it would be more >> consistent if these factory methods were named "of" so I assume the >> webrev will be updated to see how that looks. > > any name will be better than "get", and yes the recent vogue is to use "of", > you can also use "path" if you suppose people will use an import static, "of" doesn't well with an import static. I was thinking about this and ?of? in this case does not seem in complete analogy to for example {List,Set,Stream}.of(). If these cases are denoted X.of(x1,x2,?), they create a container of type X whose contents are {x1,x2,?}. In the case of Path, the arguments of the current get() methods are used to create a Path but are not exactly members as in the case of X.of(). In any case I am happy to update the proposal (webrev + specdiff) to whatever achieves consensus. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike.skells1 at gmail.com Tue Mar 13 20:57:27 2018 From: mike.skells1 at gmail.com (Michael Skells) Date: Tue, 13 Mar 2018 20:57:27 +0000 Subject: trivial patch to reduce allocations in nio open* methods In-Reply-To: References: <158BFF8B-586B-4C1C-8DBE-5C5DD3E8D66B@oracle.com> <2ba60a55-baf3-8542-dac6-70c33e3f5f25@oracle.com> <0803a885-5938-6549-262e-40064d73c513@oracle.com> <5aa714cf.c30c1c0a.fbbdc.b3fb@mx.google.com> Message-ID: Hi Alan, This is using the default file system, on windows and unix This was spotted with the development of the Scala compiler, but it would be unaffected by this patch as described below Some background - The Scala compiler tends to write a lot more classes for a given source file than Java, tens or even hundreds from a single source. I was working or parallelising the code generation and optimisation of the back end of the scala compiler and looking for object allocations that could be trimmed amongst others things, mostly in the scala compiles and its runtime We have some projects that will generate 100K class files for a compile, and several hundred projects within the same codeball. When compiling downstream projects within the IDEs we refer to classes from upstream projects and there are lots of files read and parsed We have hundreds of developers who are recompiling the codeball on a regular basis, usually partially either in an IDE or CI environment, some on Windows and some on Unix The work was focussed primarily on providing business benefit in our enterprise, but releasing what we could back to the open source compiler Performance This additional object creation is not a dominating, or even large impact on the compiler performance, but all things add up to the end performance and currently opening a file for write creates I think 5 objects ( the HashMap, backing Object[], and 3 Map.Entry) and wastes a small amount of CPU, and it seemed easy to fix for the common case :-). The production code doesn't use this call path but we use CREATE_NEW explicitly as explained later The file opening & closing (particularly in windows, with an enterprise software stack) is by far the dominant item, by orders of magnitude. We can't currently write to Zip/Jar files because of the tool chain, and the compiler is generating a stream of (FileName, byte[]), and there is no bulk or stream file API that I am aware of The benchmarking I did was on Java 8 (about 6 months ago), on 64 bit windows 10 and 7, desktops from 4 to 12 cores, all 64 bit server JVM, with local SSD I did benchmark the use of different APIs (sync vs async NIO vs old file IO) and how many writes concurrently to get the best throughput, on SSD desktops, and the best seems to be sync NIO and over-committing the CPU with 2 threads per hyperthreaded core (this seemed to reduce the OS CPU overheads). Strangely async NIO didn't seem to benefit so much from this over-commitment but I didn't dig into this. I did post to this group some unexpected degradation with REFS filesystem "parallel IO anomaly with windows REFS formatted drive" but as Bernt noted that filesystem was removed from desktops Incidentally, but slightly related the biggest unexpected performance boost (on windows) was not using CREATE and TRUNCATE together but using CREATE_NEW and coping with the unusual case of pre-existing files as an exception (40% reduction in OS CPU!). I only found this by accidentally deleting the profiling filter in JProbe for java.*, sun.* etc classes one day :-) and profiling all the way to nio code. This code is around usage of "truncateAfterOpen" in WindowsChannelFactory This means that the default options cause 2 OS calls to open the file on windows (one to create and the other to truncate), but I could not fix the default case in a patch and I am not sure it is fixable :-( I spotted this issue/optimisation in the Scala compiler development, but it would be unaffected by this patch because we use specific flags, but I thought that it would be useful to others. Regards Mike On Tue, 13 Mar 2018 at 14:25 Alan Bateman wrote: > On 13/03/2018 00:01, Michael Skells wrote: > > Hi, > > I am not sure what the status is for this patch > > > > Are you waiting for more input, or some update from me? > > Is this a viable patch? > > Is it just in the queue of minor noise to consider later? > > Do you have any data on how it helps the scala compiler? Also are you > seeing this with the default file system or is something else (the zip file > system maybe)? I ask as I would have expected the I/O cost and opening the > file to dominant rather than processing the arguments that specify how the > file is opened. > > BTW: I'm not opposed to changing anything here, instead I think we need a > bit more of the story. > > > -Alan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Wed Mar 14 14:01:40 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 14 Mar 2018 14:01:40 +0000 Subject: 8199611: (se) Minor selector implementation clean-up Message-ID: I have a number of patches for the Selector implementations, this include a few random cleanups that I would prefer to separate out. The cleanup patch in the following webrev: ? http://cr.openjdk.java.net/~alanb/8199611/webrev/index.html The only two interesting changes are: 1. SelectionKey is specified to be thread safe but the readyOps is read without synchronization, this is why readyOps changed to volatile. 2. The epoll based Selector iterated over the polled file descriptors twice, once to check for wakeup/interrupt, the second to update the selected keys. The first iteration can be dropped as the wakeup/interrupt can be detected when updating the keys. All the existing tests pass with these changes on all platforms. -Alan From Alan.Bateman at oracle.com Wed Mar 14 14:05:58 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 14 Mar 2018 14:05:58 +0000 Subject: Callback Based Selectors In-Reply-To: References: Message-ID: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> On 16/01/2016 11:14, Richard Warburton wrote: > Hi gents, > > I've prototyped a callback based addition to the NIO Selector, which > I've previously talked through a bit with Alan Bateman. The goal of > the callback based selector is to avoid the current pattern of calling > select/selectNow on a Nio selector and then having to iterate over a > hashmap produced. This pattern being quite object allocation heavy for > a critical path and also involving obtaining and releasing multiple > locks. I'd like to propose that the following patch, which adds the > ability to perform a select on a Selector that takes a callback handler. > > http://cr.openjdk.java.net/~rwarburton/select-now-4/ > > > I'm happy to iterate on the patch a bit based upon people's feedback > and discuss any potential concerns or issues that you may have with > the patch. Looking forward to hearing your feedback. > This one has been on my list for a long time to study more closely. When we discussed it here in 2016 I had concerns about invoking a consumer while synchronized on the selector and other locks. It has grown on my a bit so I spent a bit of time recently to work through the? implications, both spec and implementation. For the spec:? The Selector API describes selection operations in its class description and it's important to keep that consistent when introducing a new way to be notified of channels ready for I/O operations. The action is arbitrary code and we have to assume it will behave as if it's a bull in a china shop. It might attempt another selection operation on the same selector, it might close the selector, it might change the interest set or cancel a key for a channel registered with the selector but not yet seen by the consumer. All of these scenarios need consideration. Locking was mostly ignored in the original prototype but the locking has to be consistent with how the existing selection operations are specified. The main reason is processing the cancelled-key set means removing keys from the key set and selected-key set. On the surface, the new methods should not care about the selected-key set but they have to maintain the invariant that the selected-key set is always a subset of the selector's key set. Another point of detail is that the action may need to be called more than once for the same key but different ready sets. We have this with the kqueue implementation where events for read and write are registered with different filters. Other implementation might maintain separate poll arrays for read and write events. In terms of performance, the main benefit is that avoids adding keys to the selected-key set, only to be removed almost immediately by the code doing the select. This avoids garbage so helps the GC too. If we go ahead with these methods then it's important to get a good range of performance data and also see whether a specialized set to support the usage pattern of the selected-key set might be alternative to introducing new APIs. I've put a webrev here with candidate changes to the Selector API. It's one select(Consumer, timeout) for now, the select(Consumer) and selectNow(Consumer) variants are easy to add if needed. The webrev has a default implementation based on the existing API, and then implementations for macOS and Linux. Other platforms could be added later of course. I've also included a test that covers most of the scenarios. ?? http://cr.openjdk.java.net/~alanb/8199433/webrev/ -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From forax at univ-mlv.fr Wed Mar 14 14:16:55 2018 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 14 Mar 2018 15:16:55 +0100 (CET) Subject: Callback Based Selectors In-Reply-To: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> References: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> Message-ID: <1365073714.1162106.1521037015119.JavaMail.zimbra@u-pem.fr> I wonder if it's not better to have a new Selector class, let say CallbackBasedSelector (or whatever) than trying to retrofit the Selector API to work in both mode at the same time. Obviously, the implementation can be based in the same code but from the user POV you can not both API at the same time. regards, R?mi > De: "Alan Bateman" > ?: "Richard Warburton" , "nio-dev" > > Envoy?: Mercredi 14 Mars 2018 15:05:58 > Objet: Re: Callback Based Selectors > On 16/01/2016 11:14, Richard Warburton wrote: >> Hi gents, >> I've prototyped a callback based addition to the NIO Selector, which I've >> previously talked through a bit with Alan Bateman. The goal of the callback >> based selector is to avoid the current pattern of calling select/selectNow on a >> Nio selector and then having to iterate over a hashmap produced. This pattern >> being quite object allocation heavy for a critical path and also involving >> obtaining and releasing multiple locks. I'd like to propose that the following >> patch, which adds the ability to perform a select on a Selector that takes a >> callback handler. >> [ http://cr.openjdk.java.net/%7Erwarburton/select-now-4/ | >> http://cr.openjdk.java.net/~rwarburton/select-now-4/ ] >> I'm happy to iterate on the patch a bit based upon people's feedback and discuss >> any potential concerns or issues that you may have with the patch. Looking >> forward to hearing your feedback. > This one has been on my list for a long time to study more closely. When we > discussed it here in 2016 I had concerns about invoking a consumer while > synchronized on the selector and other locks. It has grown on my a bit so I > spent a bit of time recently to work through the implications, both spec and > implementation. > For the spec: The Selector API describes selection operations in its class > description and it's important to keep that consistent when introducing a new > way to be notified of channels ready for I/O operations. > The action is arbitrary code and we have to assume it will behave as if it's a > bull in a china shop. It might attempt another selection operation on the same > selector, it might close the selector, it might change the interest set or > cancel a key for a channel registered with the selector but not yet seen by the > consumer. All of these scenarios need consideration. > Locking was mostly ignored in the original prototype but the locking has to be > consistent with how the existing selection operations are specified. The main > reason is processing the cancelled-key set means removing keys from the key set > and selected-key set. On the surface, the new methods should not care about the > selected-key set but they have to maintain the invariant that the selected-key > set is always a subset of the selector's key set. > Another point of detail is that the action may need to be called more than once > for the same key but different ready sets. We have this with the kqueue > implementation where events for read and write are registered with different > filters. Other implementation might maintain separate poll arrays for read and > write events. > In terms of performance, the main benefit is that avoids adding keys to the > selected-key set, only to be removed almost immediately by the code doing the > select. This avoids garbage so helps the GC too. If we go ahead with these > methods then it's important to get a good range of performance data and also > see whether a specialized set to support the usage pattern of the selected-key > set might be alternative to introducing new APIs. > I've put a webrev here with candidate changes to the Selector API. It's one > select(Consumer, timeout) for now, the > select(Consumer) and selectNow(Consumer) variants > are easy to add if needed. The webrev has a default implementation based on the > existing API, and then implementations for macOS and Linux. Other platforms > could be added later of course. I've also included a test that covers most of > the scenarios. > [ http://cr.openjdk.java.net/~alanb/8199433/webrev/ | > http://cr.openjdk.java.net/~alanb/8199433/webrev/ ] > -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.lloyd at redhat.com Wed Mar 14 14:34:23 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Wed, 14 Mar 2018 09:34:23 -0500 Subject: Callback Based Selectors In-Reply-To: <1365073714.1162106.1521037015119.JavaMail.zimbra@u-pem.fr> References: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> <1365073714.1162106.1521037015119.JavaMail.zimbra@u-pem.fr> Message-ID: Because of the way locking with selectors works, I would expect that most NIO frameworks are simply going to use this to populate a fast queue like ArrayDeque and then process all the events outside of the lock. That's what we already do, and I think that others follow this strategy as well. As far as that approach goes, I don't see any problem with this API. A brand new API which doesn't have the same select-under-lock behavior would be nifty, but I think that's a much bigger "ask". On Wed, Mar 14, 2018 at 9:16 AM, Remi Forax wrote: > I wonder if it's not better to have a new Selector class, let say > CallbackBasedSelector (or whatever) than trying to retrofit the Selector API > to work in both mode at the same time. > Obviously, the implementation can be based in the same code but from the > user POV you can not both API at the same time. > > regards, > R?mi > > ________________________________ > > De: "Alan Bateman" > ?: "Richard Warburton" , "nio-dev" > > Envoy?: Mercredi 14 Mars 2018 15:05:58 > Objet: Re: Callback Based Selectors > > On 16/01/2016 11:14, Richard Warburton wrote: > > Hi gents, > > I've prototyped a callback based addition to the NIO Selector, which I've > previously talked through a bit with Alan Bateman. The goal of the callback > based selector is to avoid the current pattern of calling select/selectNow > on a Nio selector and then having to iterate over a hashmap produced. This > pattern being quite object allocation heavy for a critical path and also > involving obtaining and releasing multiple locks. I'd like to propose that > the following patch, which adds the ability to perform a select on a > Selector that takes a callback handler. > > http://cr.openjdk.java.net/~rwarburton/select-now-4/ > > I'm happy to iterate on the patch a bit based upon people's feedback and > discuss any potential concerns or issues that you may have with the patch. > Looking forward to hearing your feedback. > > This one has been on my list for a long time to study more closely. When we > discussed it here in 2016 I had concerns about invoking a consumer while > synchronized on the selector and other locks. It has grown on my a bit so I > spent a bit of time recently to work through the implications, both spec > and implementation. > > For the spec: The Selector API describes selection operations in its class > description and it's important to keep that consistent when introducing a > new way to be notified of channels ready for I/O operations. > > The action is arbitrary code and we have to assume it will behave as if it's > a bull in a china shop. It might attempt another selection operation on the > same selector, it might close the selector, it might change the interest set > or cancel a key for a channel registered with the selector but not yet seen > by the consumer. All of these scenarios need consideration. > > Locking was mostly ignored in the original prototype but the locking has to > be consistent with how the existing selection operations are specified. The > main reason is processing the cancelled-key set means removing keys from the > key set and selected-key set. On the surface, the new methods should not > care about the selected-key set but they have to maintain the invariant that > the selected-key set is always a subset of the selector's key set. > > Another point of detail is that the action may need to be called more than > once for the same key but different ready sets. We have this with the kqueue > implementation where events for read and write are registered with different > filters. Other implementation might maintain separate poll arrays for read > and write events. > > In terms of performance, the main benefit is that avoids adding keys to the > selected-key set, only to be removed almost immediately by the code doing > the select. This avoids garbage so helps the GC too. If we go ahead with > these methods then it's important to get a good range of performance data > and also see whether a specialized set to support the usage pattern of the > selected-key set might be alternative to introducing new APIs. > > I've put a webrev here with candidate changes to the Selector API. It's one > select(Consumer, timeout) for now, the > select(Consumer) and selectNow(Consumer) > variants are easy to add if needed. The webrev has a default implementation > based on the existing API, and then implementations for macOS and Linux. > Other platforms could be added later of course. I've also included a test > that covers most of the scenarios. > http://cr.openjdk.java.net/~alanb/8199433/webrev/ > > -Alan > -- - DML From forax at univ-mlv.fr Wed Mar 14 15:14:48 2018 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Wed, 14 Mar 2018 16:14:48 +0100 (CET) Subject: Callback Based Selectors In-Reply-To: References: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> <1365073714.1162106.1521037015119.JavaMail.zimbra@u-pem.fr> Message-ID: <523948539.1198855.1521040488670.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "David Lloyd" > ?: "Remi Forax" > Cc: "Alan Bateman" , "nio-dev" > Envoy?: Mercredi 14 Mars 2018 15:34:23 > Objet: Re: Callback Based Selectors > Because of the way locking with selectors works, I would expect that > most NIO frameworks are simply going to use this to populate a fast > queue like ArrayDeque and then process all the events outside of the > lock. That's what we already do, and I think that others follow this > strategy as well. As far as that approach goes, I don't see any > problem with this API. > > A brand new API which doesn't have the same select-under-lock behavior > would be nifty, but I think that's a much bigger "ask". I think it's a lesser "ask", the Selector API is already small in term of number of methods, the main issue is the locking scheme, that why i think it's easier to create a new API with no public locking scheme that trying to retrofit the Selector API. Alan will know more but i believe that we have a separate class AsynchronousFileChannel instead of an asynchronous API in FileChannel for the same reason. R?mi > > On Wed, Mar 14, 2018 at 9:16 AM, Remi Forax wrote: >> I wonder if it's not better to have a new Selector class, let say >> CallbackBasedSelector (or whatever) than trying to retrofit the Selector API >> to work in both mode at the same time. >> Obviously, the implementation can be based in the same code but from the >> user POV you can not both API at the same time. >> >> regards, >> R?mi >> >> ________________________________ >> >> De: "Alan Bateman" >> ?: "Richard Warburton" , "nio-dev" >> >> Envoy?: Mercredi 14 Mars 2018 15:05:58 >> Objet: Re: Callback Based Selectors >> >> On 16/01/2016 11:14, Richard Warburton wrote: >> >> Hi gents, >> >> I've prototyped a callback based addition to the NIO Selector, which I've >> previously talked through a bit with Alan Bateman. The goal of the callback >> based selector is to avoid the current pattern of calling select/selectNow >> on a Nio selector and then having to iterate over a hashmap produced. This >> pattern being quite object allocation heavy for a critical path and also >> involving obtaining and releasing multiple locks. I'd like to propose that >> the following patch, which adds the ability to perform a select on a >> Selector that takes a callback handler. >> >> http://cr.openjdk.java.net/~rwarburton/select-now-4/ >> >> I'm happy to iterate on the patch a bit based upon people's feedback and >> discuss any potential concerns or issues that you may have with the patch. >> Looking forward to hearing your feedback. >> >> This one has been on my list for a long time to study more closely. When we >> discussed it here in 2016 I had concerns about invoking a consumer while >> synchronized on the selector and other locks. It has grown on my a bit so I >> spent a bit of time recently to work through the implications, both spec >> and implementation. >> >> For the spec: The Selector API describes selection operations in its class >> description and it's important to keep that consistent when introducing a >> new way to be notified of channels ready for I/O operations. >> >> The action is arbitrary code and we have to assume it will behave as if it's >> a bull in a china shop. It might attempt another selection operation on the >> same selector, it might close the selector, it might change the interest set >> or cancel a key for a channel registered with the selector but not yet seen >> by the consumer. All of these scenarios need consideration. >> >> Locking was mostly ignored in the original prototype but the locking has to >> be consistent with how the existing selection operations are specified. The >> main reason is processing the cancelled-key set means removing keys from the >> key set and selected-key set. On the surface, the new methods should not >> care about the selected-key set but they have to maintain the invariant that >> the selected-key set is always a subset of the selector's key set. >> >> Another point of detail is that the action may need to be called more than >> once for the same key but different ready sets. We have this with the kqueue >> implementation where events for read and write are registered with different >> filters. Other implementation might maintain separate poll arrays for read >> and write events. >> >> In terms of performance, the main benefit is that avoids adding keys to the >> selected-key set, only to be removed almost immediately by the code doing >> the select. This avoids garbage so helps the GC too. If we go ahead with >> these methods then it's important to get a good range of performance data >> and also see whether a specialized set to support the usage pattern of the >> selected-key set might be alternative to introducing new APIs. >> >> I've put a webrev here with candidate changes to the Selector API. It's one >> select(Consumer, timeout) for now, the >> select(Consumer) and selectNow(Consumer) >> variants are easy to add if needed. The webrev has a default implementation >> based on the existing API, and then implementations for macOS and Linux. >> Other platforms could be added later of course. I've also included a test >> that covers most of the scenarios. >> http://cr.openjdk.java.net/~alanb/8199433/webrev/ >> >> -Alan >> > > > > -- > - DML From christoph.langer at sap.com Wed Mar 14 15:38:31 2018 From: christoph.langer at sap.com (Langer, Christoph) Date: Wed, 14 Mar 2018 15:38:31 +0000 Subject: 8199611: (se) Minor selector implementation clean-up In-Reply-To: References: Message-ID: Hi Alan, this looks like some nice cleanups and improvements. I didn't test with it but going through the changes I can't find anything to my dislike. Best regards Christoph > -----Original Message----- > From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of > Alan Bateman > Sent: Mittwoch, 14. M?rz 2018 15:02 > To: nio-dev at openjdk.java.net > Subject: 8199611: (se) Minor selector implementation clean-up > > > I have a number of patches for the Selector implementations, this > include a few random cleanups that I would prefer to separate out. > > The cleanup patch in the following webrev: > ? http://cr.openjdk.java.net/~alanb/8199611/webrev/index.html > > The only two interesting changes are: > > 1. SelectionKey is specified to be thread safe but the readyOps is read > without synchronization, this is why readyOps changed to volatile. > > 2. The epoll based Selector iterated over the polled file descriptors > twice, once to check for wakeup/interrupt, the second to update the > selected keys. The first iteration can be dropped as the > wakeup/interrupt can be detected when updating the keys. > > All the existing tests pass with these changes on all platforms. > > -Alan From david.lloyd at redhat.com Wed Mar 14 15:47:58 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Wed, 14 Mar 2018 10:47:58 -0500 Subject: Callback Based Selectors In-Reply-To: <523948539.1198855.1521040488670.JavaMail.zimbra@u-pem.fr> References: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> <1365073714.1162106.1521037015119.JavaMail.zimbra@u-pem.fr> <523948539.1198855.1521040488670.JavaMail.zimbra@u-pem.fr> Message-ID: On Wed, Mar 14, 2018 at 10:14 AM, wrote: >> De: "David Lloyd" >> A brand new API which doesn't have the same select-under-lock behavior >> would be nifty, but I think that's a much bigger "ask". > > I think it's a lesser "ask", the Selector API is already small in term of number of methods, the main issue is the locking scheme, > that why i think it's easier to create a new API with no public locking scheme that trying to retrofit the Selector API. > > Alan will know more but i believe that we have a separate class AsynchronousFileChannel instead of an asynchronous API in FileChannel for the same reason. I did not mean to imply that it is not a good idea. The locking model of Selectors is a primary source of problems with our NIO-based software. A pure callback-based replacement would be really great for us at least; I just don't think that it's necessarily exclusive to the idea of adding a callback-based method like Alan proposed, which already allows us to improve things by reducing copying in our selection operations. -- - DML From Alan.Bateman at oracle.com Wed Mar 14 18:57:50 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 14 Mar 2018 18:57:50 +0000 Subject: 8199611: (se) Minor selector implementation clean-up In-Reply-To: References: Message-ID: <2f667dc3-d9a5-d7e2-c768-52c6cbc0aa2a@oracle.com> On 14/03/2018 14:01, Alan Bateman wrote: > > I have a number of patches for the Selector implementations, this > include a few random cleanups that I would prefer to separate out. > > The cleanup patch in the following webrev: > ? http://cr.openjdk.java.net/~alanb/8199611/webrev/index.html > The review thread for JDK-8199437 on net-dev brought up the suggestion for SelectionKey::toString to return a useful string for debugging. As we do doing cleanups in this area then we can include it with the changes here. I've updated the webrev in place, the only change is to SelectionKeyImpl where I've added a toString method (SelectionKeyImpl rather than SelectionKey as the implementation has easy access to the interest and ready ops). -Alan From claes.redestad at oracle.com Wed Mar 14 19:03:56 2018 From: claes.redestad at oracle.com (Claes Redestad) Date: Wed, 14 Mar 2018 20:03:56 +0100 Subject: 8199611: (se) Minor selector implementation clean-up In-Reply-To: <2f667dc3-d9a5-d7e2-c768-52c6cbc0aa2a@oracle.com> References: <2f667dc3-d9a5-d7e2-c768-52c6cbc0aa2a@oracle.com> Message-ID: <56996628-fbca-3153-33f1-5bd5e9f1d762@oracle.com> On 2018-03-14 19:57, Alan Bateman wrote: > On 14/03/2018 14:01, Alan Bateman wrote: >> >> I have a number of patches for the Selector implementations, this >> include a few random cleanups that I would prefer to separate out. >> >> The cleanup patch in the following webrev: >> http://cr.openjdk.java.net/~alanb/8199611/webrev/index.html >> > The review thread for JDK-8199437 on net-dev brought up the suggestion > for SelectionKey::toString to return a useful string for debugging. As > we do doing cleanups in this area then we can include it with the > changes here. I've updated the webrev in place, the only change is to > SelectionKeyImpl where I've added a toString method (SelectionKeyImpl > rather than SelectionKey as the implementation has easy access to the > interest and ready ops). > > -Alan Changes looks fine to me. /Claes From brian.burkhalter at oracle.com Wed Mar 14 20:09:00 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 14 Mar 2018 13:09:00 -0700 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> <69614880-3004-4B65-ADB3-D1ED807A88B5@oracle.com> <992c8053-60aa-d092-6a51-06d3c203c618@oracle.com> <715116280.749084.1520949869741.JavaMail.zimbra@u-pem.fr> Message-ID: An updated version with the change s/Path.get/Path.of/ and most of the Paths.get() javadoc excised is at http://cr.openjdk.java.net/~bpb/8194746/webrev.02/ For the sake of clarity, this omits the changes to other files in java.base which will be modified as s/Paths.get/Path.of/. Alternative verbiage for the API Note in Paths might be: --- a/src/java.base/share/classes/java/nio/file/Paths.java +++ b/src/java.base/share/classes/java/nio/file/Paths.java @@ -33,9 +33,9 @@ * by converting a path string or {@link URI}. * * @apiNote - * This class is obsolete. Its {@code get()} methods simply invoke the - * equivalent static {@code of()} methods defined by {@link Path} which should - * be used in preference. + * As of JDK 11, the preferred way to obtain a {@code Path} is via the + * {@link Path#get(String,String...) Path.get(String, String...)} or + * {@link Path#get(URI) Path.get(URI)} methods. I will update the corresponding CSR according to the outcome of this review thread. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Thu Mar 15 01:10:33 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 14 Mar 2018 18:10:33 -0700 Subject: 8199611: (se) Minor selector implementation clean-up In-Reply-To: References: Message-ID: <11603174-D63A-4F3E-8E7F-C49F8392A33A@oracle.com> Agreed, I think it looks fine. Brian On Mar 14, 2018, at 8:38 AM, Langer, Christoph wrote: > this looks like some nice cleanups and improvements. > > I didn't test with it but going through the changes I can't find anything to my dislike. -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.warburton at gmail.com Thu Mar 15 06:12:40 2018 From: richard.warburton at gmail.com (Richard Warburton) Date: Thu, 15 Mar 2018 06:12:40 +0000 Subject: Callback Based Selectors In-Reply-To: References: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> <1365073714.1162106.1521037015119.JavaMail.zimbra@u-pem.fr> Message-ID: Hi David, I think if you are planning on just using the callback based selector to populate an arraydequeue then Alan's proposed alternative of having a more efficient specialised set implementation might be worth considering. The motive for the callback approach was to avoid the overhead of the adding then immediately removing selectionkeys but also to hopefully try and simplify the locking model a bit. Iirc a selectnow() call entailed acquiring 3 different locks within the selector. I don't think this was required by the callback based mechanism, however as Alan points out that was never properly reviewed at the time. I actually like Remis proposal of having a different selector with a different locking model. That would fit my use cases where selectors are used essentially single threaded in an event loop really well. I never originally proposed this more because of a fear that such a large change would be highly likely to be rejected rather than on the basis of technical merit. I don't know what people on the core libs team feel about this approach? On Wed, 14 Mar 2018, 16:35 David Lloyd, wrote: > Because of the way locking with selectors works, I would expect that > most NIO frameworks are simply going to use this to populate a fast > queue like ArrayDeque and then process all the events outside of the > lock. That's what we already do, and I think that others follow this > strategy as well. As far as that approach goes, I don't see any > problem with this API. > > A brand new API which doesn't have the same select-under-lock behavior > would be nifty, but I think that's a much bigger "ask". > > On Wed, Mar 14, 2018 at 9:16 AM, Remi Forax wrote: > > I wonder if it's not better to have a new Selector class, let say > > CallbackBasedSelector (or whatever) than trying to retrofit the Selector > API > > to work in both mode at the same time. > > Obviously, the implementation can be based in the same code but from the > > user POV you can not both API at the same time. > > > > regards, > > R?mi > > > > ________________________________ > > > > De: "Alan Bateman" > > ?: "Richard Warburton" , "nio-dev" > > > > Envoy?: Mercredi 14 Mars 2018 15:05:58 > > Objet: Re: Callback Based Selectors > > > > On 16/01/2016 11:14, Richard Warburton wrote: > > > > Hi gents, > > > > I've prototyped a callback based addition to the NIO Selector, which I've > > previously talked through a bit with Alan Bateman. The goal of the > callback > > based selector is to avoid the current pattern of calling > select/selectNow > > on a Nio selector and then having to iterate over a hashmap produced. > This > > pattern being quite object allocation heavy for a critical path and also > > involving obtaining and releasing multiple locks. I'd like to propose > that > > the following patch, which adds the ability to perform a select on a > > Selector that takes a callback handler. > > > > http://cr.openjdk.java.net/~rwarburton/select-now-4/ > > > > I'm happy to iterate on the patch a bit based upon people's feedback and > > discuss any potential concerns or issues that you may have with the > patch. > > Looking forward to hearing your feedback. > > > > This one has been on my list for a long time to study more closely. When > we > > discussed it here in 2016 I had concerns about invoking a consumer while > > synchronized on the selector and other locks. It has grown on my a bit > so I > > spent a bit of time recently to work through the implications, both spec > > and implementation. > > > > For the spec: The Selector API describes selection operations in its > class > > description and it's important to keep that consistent when introducing a > > new way to be notified of channels ready for I/O operations. > > > > The action is arbitrary code and we have to assume it will behave as if > it's > > a bull in a china shop. It might attempt another selection operation on > the > > same selector, it might close the selector, it might change the interest > set > > or cancel a key for a channel registered with the selector but not yet > seen > > by the consumer. All of these scenarios need consideration. > > > > Locking was mostly ignored in the original prototype but the locking has > to > > be consistent with how the existing selection operations are specified. > The > > main reason is processing the cancelled-key set means removing keys from > the > > key set and selected-key set. On the surface, the new methods should not > > care about the selected-key set but they have to maintain the invariant > that > > the selected-key set is always a subset of the selector's key set. > > > > Another point of detail is that the action may need to be called more > than > > once for the same key but different ready sets. We have this with the > kqueue > > implementation where events for read and write are registered with > different > > filters. Other implementation might maintain separate poll arrays for > read > > and write events. > > > > In terms of performance, the main benefit is that avoids adding keys to > the > > selected-key set, only to be removed almost immediately by the code doing > > the select. This avoids garbage so helps the GC too. If we go ahead with > > these methods then it's important to get a good range of performance data > > and also see whether a specialized set to support the usage pattern of > the > > selected-key set might be alternative to introducing new APIs. > > > > I've put a webrev here with candidate changes to the Selector API. It's > one > > select(Consumer, timeout) for now, the > > select(Consumer) and selectNow(Consumer) > > variants are easy to add if needed. The webrev has a default > implementation > > based on the existing API, and then implementations for macOS and Linux. > > Other platforms could be added later of course. I've also included a test > > that covers most of the scenarios. > > http://cr.openjdk.java.net/~alanb/8199433/webrev/ > > > > -Alan > > > > > > -- > - DML > -------------- next part -------------- An HTML attachment was scrubbed... URL: From huaming.li at oracle.com Thu Mar 15 07:25:06 2018 From: huaming.li at oracle.com (Hamlin Li) Date: Thu, 15 Mar 2018 15:25:06 +0800 Subject: RFR of JDK-8199215: Re-examine getFreePort method in test infrastructure library Message-ID: Would you please review the following patch? bug: https://bugs.openjdk.java.net/browse/JDK-8199215 webrev: http://cr.openjdk.java.net/~mli/8199215/webrev.00/ Thank you -Hamlin From Alan.Bateman at oracle.com Thu Mar 15 07:40:46 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 15 Mar 2018 07:40:46 +0000 Subject: RFR of JDK-8199215: Re-examine getFreePort method in test infrastructure library In-Reply-To: References: Message-ID: <5a52026a-b3a3-8e16-2a85-1c2bd239e266@oracle.com> On 15/03/2018 07:25, Hamlin Li wrote: > Would you please review the following patch? > > bug: https://bugs.openjdk.java.net/browse/JDK-8199215 > > webrev: http://cr.openjdk.java.net/~mli/8199215/webrev.00/ Why are the following listed in the @build tags: ? jdk.test.lib.Asserts ? jdk.test.lib.JDKToolFinder ? jdk.test.lib.JDKToolLauncher ? jdk.test.lib.Platform ? jdk.test.lib.process.* None of these are interesting to these tests. There was a lengthy thread last year (I think started by JDK-8180805) where the issues of implicit compilation were discussed. I thought that jtreg was changed to compile test library classes in their own directory so that test infrastructure classes that relied on implicit compilation could work without the interim NoClassDefFoundError that we used to hit when running tests concurrently and where tests didn't precisely list every test library class. -Alan From huaming.li at oracle.com Thu Mar 15 08:43:10 2018 From: huaming.li at oracle.com (Hamlin Li) Date: Thu, 15 Mar 2018 16:43:10 +0800 Subject: RFR of JDK-8199215: Re-examine getFreePort method in test infrastructure library In-Reply-To: <5a52026a-b3a3-8e16-2a85-1c2bd239e266@oracle.com> References: <5a52026a-b3a3-8e16-2a85-1c2bd239e266@oracle.com> Message-ID: On 15/03/2018 3:40 PM, Alan Bateman wrote: > On 15/03/2018 07:25, Hamlin Li wrote: >> Would you please review the following patch? >> >> bug: https://bugs.openjdk.java.net/browse/JDK-8199215 >> >> webrev: http://cr.openjdk.java.net/~mli/8199215/webrev.00/ > Why are the following listed in the @build tags: > > ? jdk.test.lib.Asserts > ? jdk.test.lib.JDKToolFinder > ? jdk.test.lib.JDKToolLauncher > ? jdk.test.lib.Platform > ? jdk.test.lib.process.* > > None of these are interesting to these tests. There was a lengthy > thread last year (I think started by JDK-8180805) where the issues of > implicit compilation were discussed. I thought that jtreg was changed > to compile test library classes in their own directory so that test > infrastructure classes that relied on implicit compilation could work > without the interim NoClassDefFoundError that we used to hit when > running tests concurrently and where tests didn't precisely list every > test library class. Hi Alan, Thank you for reviewing, I have updated the webrev in place. Thank you -Hamlin > > -Alan From Alan.Bateman at oracle.com Thu Mar 15 11:20:06 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 15 Mar 2018 11:20:06 +0000 Subject: RFR of JDK-8199215: Re-examine getFreePort method in test infrastructure library In-Reply-To: References: <5a52026a-b3a3-8e16-2a85-1c2bd239e266@oracle.com> Message-ID: <067292b0-a167-8877-df46-73a081c99301@oracle.com> On 15/03/2018 08:43, Hamlin Li wrote: > : > > Hi Alan, > > Thank you for reviewing, I have updated the webrev in place. ( cc'ing serviceability-dev and net-dev as these are the other areas that use the getFreePort method in the test library. For context, the patch that we are discussing is: ??? http://cr.openjdk.java.net/~mli/8199215/webrev.00/ ) The new implementation of getFreePort looks good but it no longer throws InterruptedException and so might need some of the usages (esp. in the serviceability tests) to be updated. Also the comment "The function will spin ..." is no longer relevant and can be removed. Moving refusingEndpoint() from the NIO test to Utils looks okay. The "it's much more stable ..." in the method description looks a it inconsistent with the other wording. An alternative is "This method is better choice than getFreePort for tests that need an endpoint that refuses connections". The update to the tests look okay to me. -Alan From Alan.Bateman at oracle.com Thu Mar 15 11:23:28 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 15 Mar 2018 11:23:28 +0000 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> <69614880-3004-4B65-ADB3-D1ED807A88B5@oracle.com> <992c8053-60aa-d092-6a51-06d3c203c618@oracle.com> <715116280.749084.1520949869741.JavaMail.zimbra@u-pem.fr> Message-ID: <40873f45-2a1c-c483-d572-89e9bd34886e@oracle.com> On 14/03/2018 20:09, Brian Burkhalter wrote: > An updated version with the change s/Path.get/Path.of/ and most of the > Paths.get() javadoc excised is at > > http://cr.openjdk.java.net/~bpb/8194746/webrev.02/ > Given that the factory method is renamed then maybe we should change the first sentence to "Returns a Path by converting a ...". Otherwise looks good to me. -Alan From david.lloyd at redhat.com Thu Mar 15 11:49:26 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Thu, 15 Mar 2018 06:49:26 -0500 Subject: Callback Based Selectors In-Reply-To: References: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> <1365073714.1162106.1521037015119.JavaMail.zimbra@u-pem.fr> Message-ID: In terms of locking, even one lock is too many, but we're a bit trapped by history here; I agree that Remi's proposal sounds great but I don't want to miss out on a small improvement on the promise of a bigger one. A specialized set isn't really interesting to me because I have to process it outside of the lock(s) anyway. ArrayDeque works well for me because once it's sized, it's very fast, and by using the callback to populate it we avoid the extra add/remove operation on the key set, which is already worth something to me. On Thu, Mar 15, 2018 at 1:12 AM, Richard Warburton wrote: > Hi David, > > I think if you are planning on just using the callback based selector to > populate an arraydequeue then Alan's proposed alternative of having a more > efficient specialised set implementation might be worth considering. > > The motive for the callback approach was to avoid the overhead of the adding > then immediately removing selectionkeys but also to hopefully try and > simplify the locking model a bit. Iirc a selectnow() call entailed acquiring > 3 different locks within the selector. I don't think this was required by > the callback based mechanism, however as Alan points out that was never > properly reviewed at the time. > > I actually like Remis proposal of having a different selector with a > different locking model. That would fit my use cases where selectors are > used essentially single threaded in an event loop really well. I never > originally proposed this more because of a fear that such a large change > would be highly likely to be rejected rather than on the basis of technical > merit. I don't know what people on the core libs team feel about this > approach? > > On Wed, 14 Mar 2018, 16:35 David Lloyd, wrote: >> >> Because of the way locking with selectors works, I would expect that >> most NIO frameworks are simply going to use this to populate a fast >> queue like ArrayDeque and then process all the events outside of the >> lock. That's what we already do, and I think that others follow this >> strategy as well. As far as that approach goes, I don't see any >> problem with this API. >> >> A brand new API which doesn't have the same select-under-lock behavior >> would be nifty, but I think that's a much bigger "ask". >> >> On Wed, Mar 14, 2018 at 9:16 AM, Remi Forax wrote: >> > I wonder if it's not better to have a new Selector class, let say >> > CallbackBasedSelector (or whatever) than trying to retrofit the Selector >> > API >> > to work in both mode at the same time. >> > Obviously, the implementation can be based in the same code but from the >> > user POV you can not both API at the same time. >> > >> > regards, >> > R?mi >> > >> > ________________________________ >> > >> > De: "Alan Bateman" >> > ?: "Richard Warburton" , "nio-dev" >> > >> > Envoy?: Mercredi 14 Mars 2018 15:05:58 >> > Objet: Re: Callback Based Selectors >> > >> > On 16/01/2016 11:14, Richard Warburton wrote: >> > >> > Hi gents, >> > >> > I've prototyped a callback based addition to the NIO Selector, which >> > I've >> > previously talked through a bit with Alan Bateman. The goal of the >> > callback >> > based selector is to avoid the current pattern of calling >> > select/selectNow >> > on a Nio selector and then having to iterate over a hashmap produced. >> > This >> > pattern being quite object allocation heavy for a critical path and also >> > involving obtaining and releasing multiple locks. I'd like to propose >> > that >> > the following patch, which adds the ability to perform a select on a >> > Selector that takes a callback handler. >> > >> > http://cr.openjdk.java.net/~rwarburton/select-now-4/ >> > >> > I'm happy to iterate on the patch a bit based upon people's feedback and >> > discuss any potential concerns or issues that you may have with the >> > patch. >> > Looking forward to hearing your feedback. >> > >> > This one has been on my list for a long time to study more closely. When >> > we >> > discussed it here in 2016 I had concerns about invoking a consumer while >> > synchronized on the selector and other locks. It has grown on my a bit >> > so I >> > spent a bit of time recently to work through the implications, both >> > spec >> > and implementation. >> > >> > For the spec: The Selector API describes selection operations in its >> > class >> > description and it's important to keep that consistent when introducing >> > a >> > new way to be notified of channels ready for I/O operations. >> > >> > The action is arbitrary code and we have to assume it will behave as if >> > it's >> > a bull in a china shop. It might attempt another selection operation on >> > the >> > same selector, it might close the selector, it might change the interest >> > set >> > or cancel a key for a channel registered with the selector but not yet >> > seen >> > by the consumer. All of these scenarios need consideration. >> > >> > Locking was mostly ignored in the original prototype but the locking has >> > to >> > be consistent with how the existing selection operations are specified. >> > The >> > main reason is processing the cancelled-key set means removing keys from >> > the >> > key set and selected-key set. On the surface, the new methods should not >> > care about the selected-key set but they have to maintain the invariant >> > that >> > the selected-key set is always a subset of the selector's key set. >> > >> > Another point of detail is that the action may need to be called more >> > than >> > once for the same key but different ready sets. We have this with the >> > kqueue >> > implementation where events for read and write are registered with >> > different >> > filters. Other implementation might maintain separate poll arrays for >> > read >> > and write events. >> > >> > In terms of performance, the main benefit is that avoids adding keys to >> > the >> > selected-key set, only to be removed almost immediately by the code >> > doing >> > the select. This avoids garbage so helps the GC too. If we go ahead with >> > these methods then it's important to get a good range of performance >> > data >> > and also see whether a specialized set to support the usage pattern of >> > the >> > selected-key set might be alternative to introducing new APIs. >> > >> > I've put a webrev here with candidate changes to the Selector API. It's >> > one >> > select(Consumer, timeout) for now, the >> > select(Consumer) and selectNow(Consumer) >> > variants are easy to add if needed. The webrev has a default >> > implementation >> > based on the existing API, and then implementations for macOS and Linux. >> > Other platforms could be added later of course. I've also included a >> > test >> > that covers most of the scenarios. >> > http://cr.openjdk.java.net/~alanb/8199433/webrev/ >> > >> > -Alan >> > >> >> >> >> -- >> - DML -- - DML From Alan.Bateman at oracle.com Thu Mar 15 12:00:40 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 15 Mar 2018 12:00:40 +0000 Subject: Callback Based Selectors In-Reply-To: <1365073714.1162106.1521037015119.JavaMail.zimbra@u-pem.fr> References: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> <1365073714.1162106.1521037015119.JavaMail.zimbra@u-pem.fr> Message-ID: <24806bb8-dc16-0bc3-a98c-74f45911476c@oracle.com> On 14/03/2018 14:16, Remi Forax wrote: > I wonder if it's not better to have a new Selector class, let say > CallbackBasedSelector (or whatever) than trying to retrofit the > Selector API to work in both mode at the same time. > Obviously, the implementation can be based in the same code but from > the user POV you can not both API at the same time. > Given the potential for Fibers in the future, I think we need to be cautious about introducing new big APIs in this area. In addition, a completely different Selector type brings a lot of potential issues, esp. if a selectable channel is registered with an existing Selector and something else. As regards Richard's proposal then it does fit with the existing Selector design. Rather than adding or updating keys in the selected-key set, it invokes an action that can collect or process the keys. Iterating over native events is essentially iterating over a region of memory so it has to synchronize on the selector to prevent it being closed or used for another selection operation. In terms of performance, this lock is in the noise in everything that I've tried but others have have benchmark showing different results (almost all of the locking issues that I've looked at come up in the context of changes to registration that are attempted concurrently with a selection operation). It would be feasible to specify that the only lock held when processing the events is the selector lock (it doesn't need to be synchronized on the selector's key or selected-key set for that phase of selection. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Mar 15 12:11:48 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 15 Mar 2018 12:11:48 +0000 Subject: Callback Based Selectors In-Reply-To: References: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> <1365073714.1162106.1521037015119.JavaMail.zimbra@u-pem.fr> Message-ID: <7185f4eb-618c-1e11-0267-43e0be448995@oracle.com> On 15/03/2018 11:49, David Lloyd wrote: > : > > A specialized set isn't really interesting to me because I have to > process it outside of the lock(s) anyway. ArrayDeque works well for > me because once it's sized, it's very fast, and by using the callback > to populate it we avoid the extra add/remove operation on the key set, > which is already worth something to me. > Which lock are you trying to avoid here? Selection operations are specified to synchronize on the selector, its key set, and the selected-key set. The existing select methods add to the selected-key set and must maintain the invariant that the selected-key set is a subset of the selector's key set. The method on the table has the potential to release the locks for the key set and selected-key once it has processed the cancelled-set key, meaning that the only lock held when invoking the consumer is the selector lock. -Alan From david.lloyd at redhat.com Thu Mar 15 13:01:10 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Thu, 15 Mar 2018 08:01:10 -0500 Subject: Callback Based Selectors In-Reply-To: <7185f4eb-618c-1e11-0267-43e0be448995@oracle.com> References: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> <1365073714.1162106.1521037015119.JavaMail.zimbra@u-pem.fr> <7185f4eb-618c-1e11-0267-43e0be448995@oracle.com> Message-ID: On Thu, Mar 15, 2018 at 7:11 AM, Alan Bateman wrote: > On 15/03/2018 11:49, David Lloyd wrote: >> A specialized set isn't really interesting to me because I have to >> process it outside of the lock(s) anyway. ArrayDeque works well for >> me because once it's sized, it's very fast, and by using the callback >> to populate it we avoid the extra add/remove operation on the key set, >> which is already worth something to me. >> > Which lock are you trying to avoid here? Selection operations are specified > to synchronize on the selector, its key set, and the selected-key set. The > existing select methods add to the selected-key set and must maintain the > invariant that the selected-key set is a subset of the selector's key set. > The method on the table has the potential to release the locks for the key > set and selected-key once it has processed the cancelled-set key, meaning > that the only lock held when invoking the consumer is the selector lock. I need to process events outside of the lock because I need to be able to register and cancel keys from the same or different threads at any time; processing events under the lock means that the lock will be held almost 100% of the time in my selector loop, preventing other threads from doing things like cancelling or registering keys. By processing events outside of the lock, the only time the lock is held for a significant amount of time is during selection operations, which I can interrupt via wakeup() and careful handshaking. If we can avoid the need for waking up the selector in order to grab the lock to cancel or register keys though... that would be pretty nice. The three-lock model has historically been difficult since different providers behaved a bit differently; we have a couple of switches which relate to controlling the interaction behavior with different selector providers today. In the end what is needed is a clear model which describes not just how the locking itself works, but how concurrent registration, cancellation, and changing interest ops should work as a result. I should be able to asynchronously register or cancel a key and hit wakeup() on the selector and have it just work. Unfortunately that isn't the case today. -- - DML From Alan.Bateman at oracle.com Thu Mar 15 13:44:38 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 15 Mar 2018 13:44:38 +0000 Subject: Callback Based Selectors In-Reply-To: References: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> <1365073714.1162106.1521037015119.JavaMail.zimbra@u-pem.fr> <7185f4eb-618c-1e11-0267-43e0be448995@oracle.com> Message-ID: On 15/03/2018 13:01, David Lloyd wrote: > : > I need to process events outside of the lock because I need to be able > to register and cancel keys from the same or different threads at any > time; processing events under the lock means that the lock will be > held almost 100% of the time in my selector loop, preventing other > threads from doing things like cancelling or registering keys. By > processing events outside of the lock, the only time the lock is held > for a significant amount of time is during selection operations, which > I can interrupt via wakeup() and careful handshaking. The register method is specified to synchronize on the selector's key set, the cancel method is specified to synchronize on the cancelled-key set. Neither is specified to synchronize on the selector itself.? While there are challenges, esp. with poll-based implementations, it should be possible to get to the point where the only lock held when invoking the action is the selector itself. Going further and not locking the key set during a select isn't hard on some platforms but posed challenges on poll or select based implementations. > > If we can avoid the need for waking up the selector in order to grab > the lock to cancel or register keys though... that would be pretty > nice. The three-lock model has historically been difficult since > different providers behaved a bit differently; we have a couple of > switches which relate to controlling the interaction behavior with > different selector providers today. The locking is well specified so you shouldn't strictly need to be aware of which implementation is used. -Alan From Roger.Riggs at Oracle.com Thu Mar 15 13:54:17 2018 From: Roger.Riggs at Oracle.com (Roger Riggs) Date: Thu, 15 Mar 2018 09:54:17 -0400 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> <69614880-3004-4B65-ADB3-D1ED807A88B5@oracle.com> <992c8053-60aa-d092-6a51-06d3c203c618@oracle.com> <715116280.749084.1520949869741.JavaMail.zimbra@u-pem.fr> Message-ID: <85b1f9ea-d365-77b6-07c6-d02556787dbe@Oracle.com> Hi Brian, Looks fine. I'm not sure how to interpret "obsolete" especially in relation to the various contexts where "deprecated" is used.? I think it means code that uses Paths.get should be modified to use Path.of.? This sounds like a vanilla use of @deprecated with a convenient replacement. Not for removal of course because there's no plan to remove it (for now). Thanks, Roger On 3/14/2018 4:09 PM, Brian Burkhalter wrote: > An updated version with the change s/Path.get/Path.of/ and most of the > Paths.get() javadoc excised is at > > http://cr.openjdk.java.net/~bpb/8194746/webrev.02/ > > > For the sake of clarity, this omits the changes to other files in > java.base which will be modified as s/Paths.get/Path.of/. > > Alternative verbiage for the API Note in Paths might be: > > --- a/src/java.base/share/classes/java/nio/file/Paths.java > +++ b/src/java.base/share/classes/java/nio/file/Paths.java > @@ -33,9 +33,9 @@ > ? * by converting a path string or {@link URI}. > ? * > ? * @apiNote > - * This class is obsolete. Its {@code get()} methods simply invoke the > - * equivalent static {@code of()} methods defined by {@link Path} > which should > - * be used in preference. > + * As of JDK 11, the preferred way to obtain a {@code Path} is > via the > + * {@link Path#get(String,String...) Path.get(String, String...)} or > + * {@link Path#get(URI) Path.get(URI)} methods. > > I will update the corresponding CSR according to the outcome of this > review thread. > > Thanks, > > Brian From david.lloyd at redhat.com Thu Mar 15 14:02:23 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Thu, 15 Mar 2018 09:02:23 -0500 Subject: A hard-to-reproduce EPollSelector bug... Message-ID: This talk of Selectors has indirectly reminded me of a problem that we encounter, particularly in testing, which I think is a bug (or maybe just a surprise) in the EPollSelector implementation on Linux. The symptom of the problem is that a ServerSocketChannel is closed, yet a subsequent bind operation which definitely happens-after the close on the same socket address can fail with EADDRINUSE (even if SO_REUSEADDR is used). The reason (I believe) this happens is because of the way epoll works. When a file descriptor is registered with epoll, it's not *actually* the file descriptor that is registered - it's the kernel's internal file "description", which is a structure in the kernel that can be referenced by one or more file descriptors or other mechanisms (including epoll). So what happens is, the ServerSocket FD is registered in the epoll set; later, in another thread, the channel is closed, unreferencing the file description from the file descriptor, but not really closing because the active description is still present in the epoll set. To support this hypothesis, it appears that cancelling the key and performing a selectNow() does in fact solve the symptom. There is more information on the weirdness that is epoll in [1] and [2] (the author is slightly hyperbolic but also has given a good summary of the facts). I don't have any great ideas as to how to solve this; AFAIK there isn't a way to unbind a listening socket without closing it, and it seems unrealistic to somehow modify all active selectors to remove the FD from the set in this case. [1] https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-broken-12/ [2] https://idea.popcount.org/2017-03-20-epoll-is-fundamentally-broken-22/ -- - DML From Alan.Bateman at oracle.com Thu Mar 15 14:01:48 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 15 Mar 2018 14:01:48 +0000 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: <85b1f9ea-d365-77b6-07c6-d02556787dbe@Oracle.com> References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> <69614880-3004-4B65-ADB3-D1ED807A88B5@oracle.com> <992c8053-60aa-d092-6a51-06d3c203c618@oracle.com> <715116280.749084.1520949869741.JavaMail.zimbra@u-pem.fr> <85b1f9ea-d365-77b6-07c6-d02556787dbe@Oracle.com> Message-ID: <86af2f85-ef16-34d8-fa22-fa2312ce6b53@oracle.com> On 15/03/2018 13:54, Roger Riggs wrote: > Hi Brian, > > Looks fine. > > I'm not sure how to interpret "obsolete" especially in relation to the > various contexts > where "deprecated" is used.? I think it means code that uses Paths.get > should be modified > to use Path.of.? This sounds like a vanilla use of @deprecated with a > convenient replacement. > Not for removal of course because there's no plan to remove it (for now). I don't think we should do this now as it would just annoy developers that compile to multiple releases. So I think something to re-visit again in a few releases. -Alan From Roger.Riggs at Oracle.com Thu Mar 15 14:05:22 2018 From: Roger.Riggs at Oracle.com (Roger Riggs) Date: Thu, 15 Mar 2018 10:05:22 -0400 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: <86af2f85-ef16-34d8-fa22-fa2312ce6b53@oracle.com> References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> <69614880-3004-4B65-ADB3-D1ED807A88B5@oracle.com> <992c8053-60aa-d092-6a51-06d3c203c618@oracle.com> <715116280.749084.1520949869741.JavaMail.zimbra@u-pem.fr> <85b1f9ea-d365-77b6-07c6-d02556787dbe@Oracle.com> <86af2f85-ef16-34d8-fa22-fa2312ce6b53@oracle.com> Message-ID: Hi Alan, Perhaps then, say it "may be deprecated"... instead of "obsolete". or lead the api note with the recommendation to use Path.of. Roger On 3/15/2018 10:01 AM, Alan Bateman wrote: > > > On 15/03/2018 13:54, Roger Riggs wrote: >> Hi Brian, >> >> Looks fine. >> >> I'm not sure how to interpret "obsolete" especially in relation to >> the various contexts >> where "deprecated" is used.? I think it means code that uses >> Paths.get should be modified >> to use Path.of.? This sounds like a vanilla use of @deprecated with a >> convenient replacement. >> Not for removal of course because there's no plan to remove it (for >> now). > I don't think we should do this now as it would just annoy developers > that compile to multiple releases. So I think something to re-visit > again in a few releases. > > -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Mar 15 14:22:06 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 15 Mar 2018 14:22:06 +0000 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> <69614880-3004-4B65-ADB3-D1ED807A88B5@oracle.com> <992c8053-60aa-d092-6a51-06d3c203c618@oracle.com> <715116280.749084.1520949869741.JavaMail.zimbra@u-pem.fr> <85b1f9ea-d365-77b6-07c6-d02556787dbe@Oracle.com> <86af2f85-ef16-34d8-fa22-fa2312ce6b53@oracle.com> Message-ID: <1d0a44a0-5822-577c-a791-0a68f68f51e6@oracle.com> On 15/03/2018 14:05, Roger Riggs wrote: > Hi Alan, > > Perhaps then, say it "may be deprecated"... instead of "obsolete". > or lead the api note with the recommendation to use Path.of. That seems a good idea. -Alan From Alan.Bateman at oracle.com Thu Mar 15 14:17:23 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 15 Mar 2018 14:17:23 +0000 Subject: A hard-to-reproduce EPollSelector bug... In-Reply-To: References: Message-ID: <23696b2a-42a9-028c-97c3-5128cd97c4d6@oracle.com> On 15/03/2018 14:02, David Lloyd wrote: > This talk of Selectors has indirectly reminded me of a problem that we > encounter, particularly in testing, which I think is a bug (or maybe > just a surprise) in the EPollSelector implementation on Linux. > > The symptom of the problem is that a ServerSocketChannel is closed, > yet a subsequent bind operation which definitely happens-after the > close on the same socket address can fail with EADDRINUSE (even if > SO_REUSEADDR is used). > Do you have a reproducer? If so, can you run it on the latest JDK 11 build where the bind method is now correctly synchronized on the channel state. -Alan From david.lloyd at redhat.com Thu Mar 15 14:43:21 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Thu, 15 Mar 2018 09:43:21 -0500 Subject: A hard-to-reproduce EPollSelector bug... In-Reply-To: <23696b2a-42a9-028c-97c3-5128cd97c4d6@oracle.com> References: <23696b2a-42a9-028c-97c3-5128cd97c4d6@oracle.com> Message-ID: On Thu, Mar 15, 2018 at 9:17 AM, Alan Bateman wrote: > On 15/03/2018 14:02, David Lloyd wrote: >> >> This talk of Selectors has indirectly reminded me of a problem that we >> encounter, particularly in testing, which I think is a bug (or maybe >> just a surprise) in the EPollSelector implementation on Linux. >> >> The symptom of the problem is that a ServerSocketChannel is closed, >> yet a subsequent bind operation which definitely happens-after the >> close on the same socket address can fail with EADDRINUSE (even if >> SO_REUSEADDR is used). >> > Do you have a reproducer? If so, can you run it on the latest JDK 11 build > where the bind method is now correctly synchronized on the channel state. I might be able to write one. I'll give it a try anyway... -- - DML From david.lloyd at redhat.com Thu Mar 15 15:30:30 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Thu, 15 Mar 2018 10:30:30 -0500 Subject: A hard-to-reproduce EPollSelector bug... In-Reply-To: References: <23696b2a-42a9-028c-97c3-5128cd97c4d6@oracle.com> Message-ID: On Thu, Mar 15, 2018 at 9:43 AM, David Lloyd wrote: > On Thu, Mar 15, 2018 at 9:17 AM, Alan Bateman wrote: >> On 15/03/2018 14:02, David Lloyd wrote: >>> >>> This talk of Selectors has indirectly reminded me of a problem that we >>> encounter, particularly in testing, which I think is a bug (or maybe >>> just a surprise) in the EPollSelector implementation on Linux. >>> >>> The symptom of the problem is that a ServerSocketChannel is closed, >>> yet a subsequent bind operation which definitely happens-after the >>> close on the same socket address can fail with EADDRINUSE (even if >>> SO_REUSEADDR is used). >>> >> Do you have a reproducer? If so, can you run it on the latest JDK 11 build >> where the bind method is now correctly synchronized on the channel state. > > I might be able to write one. I'll give it a try anyway... Well my naive hope that I could create a quick & dirty fix has been dashed so far. But, looking at the original bug report that sent me down this chase, I see that it was perhaps not limited to just EPoll; KQueue on Mac also suffers (or suffered) from a similar problem, and I understand it happened on Windows as well. So my hypothesis that it is due to epoll weirdness is probably an "overthink" of the problem; maybe it is in fact just a question of ordering the bind correctly as you say. The bug report is publicly viewable and can be found at [1] (the stack traces are the interesting part). [1] https://issues.jboss.org/browse/WFCORE-3302 -- - DML From david.lloyd at redhat.com Thu Mar 15 15:30:51 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Thu, 15 Mar 2018 10:30:51 -0500 Subject: A hard-to-reproduce EPollSelector bug... In-Reply-To: References: <23696b2a-42a9-028c-97c3-5128cd97c4d6@oracle.com> Message-ID: On Thu, Mar 15, 2018 at 10:30 AM, David Lloyd wrote: > On Thu, Mar 15, 2018 at 9:43 AM, David Lloyd wrote: >> On Thu, Mar 15, 2018 at 9:17 AM, Alan Bateman wrote: >>> On 15/03/2018 14:02, David Lloyd wrote: >>>> >>>> This talk of Selectors has indirectly reminded me of a problem that we >>>> encounter, particularly in testing, which I think is a bug (or maybe >>>> just a surprise) in the EPollSelector implementation on Linux. >>>> >>>> The symptom of the problem is that a ServerSocketChannel is closed, >>>> yet a subsequent bind operation which definitely happens-after the >>>> close on the same socket address can fail with EADDRINUSE (even if >>>> SO_REUSEADDR is used). >>>> >>> Do you have a reproducer? If so, can you run it on the latest JDK 11 build >>> where the bind method is now correctly synchronized on the channel state. >> >> I might be able to write one. I'll give it a try anyway... > > Well my naive hope that I could create a quick & dirty fix has been Reproducer, not fix... -- - DML From brian.burkhalter at oracle.com Thu Mar 15 15:37:28 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 15 Mar 2018 08:37:28 -0700 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: <40873f45-2a1c-c483-d572-89e9bd34886e@oracle.com> References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> <69614880-3004-4B65-ADB3-D1ED807A88B5@oracle.com> <992c8053-60aa-d092-6a51-06d3c203c618@oracle.com> <715116280.749084.1520949869741.JavaMail.zimbra@u-pem.fr> <40873f45-2a1c-c483-d572-89e9bd34886e@oracle.com> Message-ID: On Mar 15, 2018, at 4:23 AM, Alan Bateman wrote: >> An updated version with the change s/Path.get/Path.of/ and most of the Paths.get() javadoc excised is at >> >> http://cr.openjdk.java.net/~bpb/8194746/webrev.02/ > Given that the factory method is renamed then maybe we should change the first sentence to "Returns a Path by converting a ...". I assume you mean on Path.of, for example (ignoring formatting): * Returns a {@code Path} by converting * a path string, or a sequence of strings that when joined form * a path string. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Thu Mar 15 15:44:36 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 15 Mar 2018 08:44:36 -0700 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: <1d0a44a0-5822-577c-a791-0a68f68f51e6@oracle.com> References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> <69614880-3004-4B65-ADB3-D1ED807A88B5@oracle.com> <992c8053-60aa-d092-6a51-06d3c203c618@oracle.com> <715116280.749084.1520949869741.JavaMail.zimbra@u-pem.fr> <85b1f9ea-d365-77b6-07c6-d02556787dbe@Oracle.com> <86af2f85-ef16-34d8-fa22-fa2312ce6b53@oracle.com> <1d0a44a0-5822-577c-a791-0a68f68f51e6@oracle.com> Message-ID: <13FFE4A7-1AB9-43CE-864C-96BECF14E7B9@oracle.com> On Mar 15, 2018, at 7:22 AM, Alan Bateman wrote: > On 15/03/2018 14:05, Roger Riggs wrote: >> Hi Alan, >> >> Perhaps then, say it "may be deprecated"... instead of "obsolete". >> or lead the api note with the recommendation to use Path.of. > That seems a good idea. Would this be reasonable (in Paths class doc)? * @apiNote * As of JDK 11, it is recommended to obtain a {@code Path} via the * {@code Path.of} methods instead of via the {@code get} methods defined in * this class as this class might be deprecated. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Mar 15 15:46:29 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 15 Mar 2018 15:46:29 +0000 Subject: A hard-to-reproduce EPollSelector bug... In-Reply-To: References: <23696b2a-42a9-028c-97c3-5128cd97c4d6@oracle.com> Message-ID: On 15/03/2018 15:30, David Lloyd wrote: > : > Well my naive hope that I could create a quick & dirty fix has been > dashed so far. But, looking at the original bug report that sent me > down this chase, I see that it was perhaps not limited to just EPoll; > KQueue on Mac also suffers (or suffered) from a similar problem, and I > understand it happened on Windows as well. So my hypothesis that it > is due to epoll weirdness is probably an "overthink" of the problem; > maybe it is in fact just a question of ordering the bind correctly as > you say. The bug report is publicly viewable and can be found at [1] > (the stack traces are the interesting part). I think all it needs is one thread calling bind at around the same time that another thread attempts to closes the channel. It doesn't need any Selectors in the picture. -Alan From roger.riggs at oracle.com Thu Mar 15 15:47:08 2018 From: roger.riggs at oracle.com (Roger Riggs) Date: Thu, 15 Mar 2018 11:47:08 -0400 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: <13FFE4A7-1AB9-43CE-864C-96BECF14E7B9@oracle.com> References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> <69614880-3004-4B65-ADB3-D1ED807A88B5@oracle.com> <992c8053-60aa-d092-6a51-06d3c203c618@oracle.com> <715116280.749084.1520949869741.JavaMail.zimbra@u-pem.fr> <85b1f9ea-d365-77b6-07c6-d02556787dbe@Oracle.com> <86af2f85-ef16-34d8-fa22-fa2312ce6b53@oracle.com> <1d0a44a0-5822-577c-a791-0a68f68f51e6@oracle.com> <13FFE4A7-1AB9-43CE-864C-96BECF14E7B9@oracle.com> Message-ID: Hi Brian, yes,?? (though I would use "may" instead of "might") Thanks, Roger On 3/15/18 11:44 AM, Brian Burkhalter wrote: > > On Mar 15, 2018, at 7:22 AM, Alan Bateman > wrote: > >> On 15/03/2018 14:05, Roger Riggs wrote: >>> Hi Alan, >>> >>> Perhaps then, say it "may be deprecated"... instead of "obsolete". >>> or lead the api note with the recommendation to use Path.of. >> That seems a good idea. > > Would this be reasonable (in Paths class doc)? > > ?* @apiNote > ?* As of JDK 11, it is recommended to obtain a {@code Path} via the > ?* {@code Path.of} methods instead of via the {@code get} methods > defined in > ?* this class as this class might be deprecated. > > Thanks, > > Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Mar 15 15:54:19 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 15 Mar 2018 15:54:19 +0000 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: <13FFE4A7-1AB9-43CE-864C-96BECF14E7B9@oracle.com> References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> <69614880-3004-4B65-ADB3-D1ED807A88B5@oracle.com> <992c8053-60aa-d092-6a51-06d3c203c618@oracle.com> <715116280.749084.1520949869741.JavaMail.zimbra@u-pem.fr> <85b1f9ea-d365-77b6-07c6-d02556787dbe@Oracle.com> <86af2f85-ef16-34d8-fa22-fa2312ce6b53@oracle.com> <1d0a44a0-5822-577c-a791-0a68f68f51e6@oracle.com> <13FFE4A7-1AB9-43CE-864C-96BECF14E7B9@oracle.com> Message-ID: On 15/03/2018 15:44, Brian Burkhalter wrote: > > Would this be reasonable (in Paths class doc)? > > ?* @apiNote > ?* As of JDK 11, it is recommended to obtain a {@code Path} via the > ?* {@code Path.of} methods instead of via the {@code get} methods > defined in > ?* this class as this class might be deprecated. > I think you can drop "As of JDK 11" (technically it would be Java SE 11 of course). I agree with Roger on "may" instead of "might". -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Thu Mar 15 15:56:38 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 15 Mar 2018 08:56:38 -0700 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> <69614880-3004-4B65-ADB3-D1ED807A88B5@oracle.com> <992c8053-60aa-d092-6a51-06d3c203c618@oracle.com> <715116280.749084.1520949869741.JavaMail.zimbra@u-pem.fr> <85b1f9ea-d365-77b6-07c6-d02556787dbe@Oracle.com> <86af2f85-ef16-34d8-fa22-fa2312ce6b53@oracle.com> <1d0a44a0-5822-577c-a791-0a68f68f51e6@oracle.com> <13FFE4A7-1AB9-43CE-864C-96BECF14E7B9@oracle.com> Message-ID: <84B6C513-69B9-4767-AC92-89912BAF9C7D@oracle.com> On Mar 15, 2018, at 8:54 AM, Alan Bateman wrote: > I think you can drop "As of JDK 11" (technically it would be Java SE 11 of course). There are other usages of ?JDK X? like this elsewhere so I suppose they should be searched out and changed. > I agree with Roger on "may" instead of "might". Will change. Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Thu Mar 15 16:19:16 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 15 Mar 2018 09:19:16 -0700 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: <84B6C513-69B9-4767-AC92-89912BAF9C7D@oracle.com> References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> <69614880-3004-4B65-ADB3-D1ED807A88B5@oracle.com> <992c8053-60aa-d092-6a51-06d3c203c618@oracle.com> <715116280.749084.1520949869741.JavaMail.zimbra@u-pem.fr> <85b1f9ea-d365-77b6-07c6-d02556787dbe@Oracle.com> <86af2f85-ef16-34d8-fa22-fa2312ce6b53@oracle.com> <1d0a44a0-5822-577c-a791-0a68f68f51e6@oracle.com> <13FFE4A7-1AB9-43CE-864C-96BECF14E7B9@oracle.com> <84B6C513-69B9-4767-AC92-89912BAF9C7D@oracle.com> Message-ID: Updated with the suggested changes: http://cr.openjdk.java.net/~bpb/8194746/webrev.03/ The differences versus the previous version are 1) the first sentences of Path.of: (String -> Path) 103 * Returns a {@code Path} by converting 104 * a path string, or a sequence of strings that when joined form 105 * a path string, to a {@code Path}. (URI -> Path) 152 * Returns a {@code Path} by converting a URI. and 2) the API note in Paths: 35 * @apiNote 36 * It is recommended to obtain a {@code Path} via the {@code Path.of} methods 37 * instead of via the {@code get} methods defined in this class as this class 38 * may be deprecated. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.lloyd at redhat.com Thu Mar 15 16:19:07 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Thu, 15 Mar 2018 11:19:07 -0500 Subject: A hard-to-reproduce EPollSelector bug... In-Reply-To: References: <23696b2a-42a9-028c-97c3-5128cd97c4d6@oracle.com> Message-ID: On Thu, Mar 15, 2018 at 10:46 AM, Alan Bateman wrote: > On 15/03/2018 15:30, David Lloyd wrote: >> >> : >> Well my naive hope that I could create a quick & dirty fix has been >> dashed so far. But, looking at the original bug report that sent me >> down this chase, I see that it was perhaps not limited to just EPoll; >> KQueue on Mac also suffers (or suffered) from a similar problem, and I >> understand it happened on Windows as well. So my hypothesis that it >> is due to epoll weirdness is probably an "overthink" of the problem; >> maybe it is in fact just a question of ordering the bind correctly as >> you say. The bug report is publicly viewable and can be found at [1] >> (the stack traces are the interesting part). > > I think all it needs is one thread calling bind at around the same time that > another thread attempts to closes the channel. It doesn't need any Selectors > in the picture. This is definitely not the expression of the problem; in our case we're calling close and then bind from the _same_ thread (or otherwise effectively happens-after), and the bind fails because the close didn't "kill" the channel for whatever reason - instead, the selector is what finally does it, per this kind of stack trace: "XNIO-1 Accept at 1562" daemon prio=5 tid=0xf nid=NA runnable java.lang.Thread.State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.kill(ServerSocketChannelImpl.java:307) - locked <0xc0d> (a java.lang.Object) at sun.nio.ch.KQueueSelectorImpl.implDereg(KQueueSelectorImpl.java:229) at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:149) - locked <0xc38> (a java.util.HashSet) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:107) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0xc2b> (a sun.nio.ch.KQueueSelectorImpl) - locked <0xc39> (a java.util.Collections$UnmodifiableSet) - locked <0xc3a> (a sun.nio.ch.Util$2) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.xnio.nio.WorkerThread.run(WorkerThread.java:519) But, I haven't yet figured out why it's happening in that order; reproducing has failed so far so I'm going to try a different tactic and see if I can statically sequence it out and find a scenario where this happens. -- - DML From roger.riggs at oracle.com Thu Mar 15 16:46:43 2018 From: roger.riggs at oracle.com (Roger Riggs) Date: Thu, 15 Mar 2018 12:46:43 -0400 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> <69614880-3004-4B65-ADB3-D1ED807A88B5@oracle.com> <992c8053-60aa-d092-6a51-06d3c203c618@oracle.com> <715116280.749084.1520949869741.JavaMail.zimbra@u-pem.fr> <85b1f9ea-d365-77b6-07c6-d02556787dbe@Oracle.com> <86af2f85-ef16-34d8-fa22-fa2312ce6b53@oracle.com> <1d0a44a0-5822-577c-a791-0a68f68f51e6@oracle.com> <13FFE4A7-1AB9-43CE-864C-96BECF14E7B9@oracle.com> <84B6C513-69B9-4767-AC92-89912BAF9C7D@oracle.com> Message-ID: <807b8618-7492-ccbc-1d3f-59127f6145dd@oracle.com> +1, Thanks, Roger On 3/15/18 12:19 PM, Brian Burkhalter wrote: > Updated with the suggested changes: > > http://cr.openjdk.java.net/~bpb/8194746/webrev.03/ > > > The differences versus the previous version are 1) the first sentences > of Path.of: > > (String -> Path) > 103 * Returns a {@code Path} by converting 104 * a path string, or a > sequence of strings that when joined form 105 * a path string, to a > {@code Path}. > (URI -> Path) > 152 * Returns a {@code Path} by converting a URI. > and 2) the API note in Paths: > 35 * @apiNote 36 * It is recommended to obtain a {@code Path} via the > {@code Path.of} methods 37 * instead of via the {@code get} methods > defined in this class as this class 38 * may be deprecated. > Thanks, > > Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Mar 15 16:52:15 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 15 Mar 2018 16:52:15 +0000 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> <69614880-3004-4B65-ADB3-D1ED807A88B5@oracle.com> <992c8053-60aa-d092-6a51-06d3c203c618@oracle.com> <715116280.749084.1520949869741.JavaMail.zimbra@u-pem.fr> <85b1f9ea-d365-77b6-07c6-d02556787dbe@Oracle.com> <86af2f85-ef16-34d8-fa22-fa2312ce6b53@oracle.com> <1d0a44a0-5822-577c-a791-0a68f68f51e6@oracle.com> <13FFE4A7-1AB9-43CE-864C-96BECF14E7B9@oracle.com> <84B6C513-69B9-4767-AC92-89912BAF9C7D@oracle.com> Message-ID: <9d08d830-5a90-1116-6f48-4d20017c9f09@oracle.com> On 15/03/2018 16:19, Brian Burkhalter wrote: > Updated with the suggested changes: > > http://cr.openjdk.java.net/~bpb/8194746/webrev.03/ > > Looks good, I would just add "in the future release" to end of the @apiNote. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Thu Mar 15 16:59:53 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 15 Mar 2018 09:59:53 -0700 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: <9d08d830-5a90-1116-6f48-4d20017c9f09@oracle.com> References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> <69614880-3004-4B65-ADB3-D1ED807A88B5@oracle.com> <992c8053-60aa-d092-6a51-06d3c203c618@oracle.com> <715116280.749084.1520949869741.JavaMail.zimbra@u-pem.fr> <85b1f9ea-d365-77b6-07c6-d02556787dbe@Oracle.com> <86af2f85-ef16-34d8-fa22-fa2312ce6b53@oracle.com> <1d0a44a0-5822-577c-a791-0a68f68f51e6@oracle.com> <13FFE4A7-1AB9-43CE-864C-96BECF14E7B9@oracle.com> <84B6C513-69B9-4767-AC92-89912BAF9C7D@oracle.com> <9d08d830-5a90-1116-6f48-4d20017c9f09@oracle.com> Message-ID: On Mar 15, 2018, at 9:52 AM, Alan Bateman wrote: > On 15/03/2018 16:19, Brian Burkhalter wrote: >> Updated with the suggested changes: >> >> http://cr.openjdk.java.net/~bpb/8194746/webrev.03/ >> > Looks good, I would just add "in the future release" to end of the @apiNote. Will do. Also on re-reading I think that in this change > The differences versus the previous version are 1) the first sentences of Path.of: > > (String -> Path) > 103 * Returns a {@code Path} by converting > 104 * a path string, or a sequence of strings that when joined form > 105 * a path string, to a {@code Path}. the terminal verbiage ", to a {@code Path}? could be removed. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Thu Mar 15 17:24:28 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 15 Mar 2018 10:24:28 -0700 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> <69614880-3004-4B65-ADB3-D1ED807A88B5@oracle.com> <992c8053-60aa-d092-6a51-06d3c203c618@oracle.com> <715116280.749084.1520949869741.JavaMail.zimbra@u-pem.fr> <85b1f9ea-d365-77b6-07c6-d02556787dbe@Oracle.com> <86af2f85-ef16-34d8-fa22-fa2312ce6b53@oracle.com> <1d0a44a0-5822-577c-a791-0a68f68f51e6@oracle.com> <13FFE4A7-1AB9-43CE-864C-96BECF14E7B9@oracle.com> <84B6C513-69B9-4767-AC92-89912BAF9C7D@oracle.com> <9d08d830-5a90-1116-6f48-4d20017c9f09@oracle.com> Message-ID: webrev.03 updated in place with the two changes below. I?ll update the CSR to match later today. Thanks, Brian On Mar 15, 2018, at 9:59 AM, Brian Burkhalter wrote: > On Mar 15, 2018, at 9:52 AM, Alan Bateman wrote: > >> On 15/03/2018 16:19, Brian Burkhalter wrote: >>> Updated with the suggested changes: >>> >>> http://cr.openjdk.java.net/~bpb/8194746/webrev.03/ >>> >> Looks good, I would just add "in the future release" to end of the @apiNote. > > Will do. Also on re-reading I think that in this change > >> The differences versus the previous version are 1) the first sentences of Path.of: >> >> (String -> Path) >> 103 * Returns a {@code Path} by converting >> 104 * a path string, or a sequence of strings that when joined form >> 105 * a path string, to a {@code Path}. > > the terminal verbiage ", to a {@code Path}? could be removed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.lloyd at redhat.com Thu Mar 15 21:01:44 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Thu, 15 Mar 2018 16:01:44 -0500 Subject: A hard-to-reproduce EPollSelector bug... In-Reply-To: References: <23696b2a-42a9-028c-97c3-5128cd97c4d6@oracle.com> Message-ID: On Thu, Mar 15, 2018 at 11:19 AM, David Lloyd wrote: > On Thu, Mar 15, 2018 at 10:46 AM, Alan Bateman wrote: >> On 15/03/2018 15:30, David Lloyd wrote: >>> >>> : >>> Well my naive hope that I could create a quick & dirty fix has been >>> dashed so far. But, looking at the original bug report that sent me >>> down this chase, I see that it was perhaps not limited to just EPoll; >>> KQueue on Mac also suffers (or suffered) from a similar problem, and I >>> understand it happened on Windows as well. So my hypothesis that it >>> is due to epoll weirdness is probably an "overthink" of the problem; >>> maybe it is in fact just a question of ordering the bind correctly as >>> you say. The bug report is publicly viewable and can be found at [1] >>> (the stack traces are the interesting part). >> >> I think all it needs is one thread calling bind at around the same time that >> another thread attempts to closes the channel. It doesn't need any Selectors >> in the picture. > > This is definitely not the expression of the problem; in our case > we're calling close and then bind from the _same_ thread (or otherwise > effectively happens-after), and the bind fails because the close blah blah OK this is my understanding of what's happening. I've chosen EPollSelectorImpl because I understand it most, though as I said we've seen this on other OSes: Mac, Windows (!), and Linux for certain. I can't be 100% sure it's exactly the same problem but even if it isn't, the fix (manually awakening/selecting all associated selectors) solves the problem in every case. Whether that's due to a coincidence of scheduling or some other secondary subtlety is hard to know. ServerSocketChannel ss is registered on Selector sel with SelectionKey sk; Selector is an EPollSelectorImpl. T1 and T2 are threads. The (a), (b), and (c) represent alternative scheduling points where the second bind could take place. T2: sel.select() T2: sel.select(0) T2: sel.lockAndDoSelect(0) T2: sel.doSelect(0) T2: sel.processDeregisterQueue() T2: sel.pollWrapper.poll(0) // ...sleep... T1: ss.close() T1: ss.implCloseChannel() T1: ss.implCloseSelectableChannel() T1: NativeDispatcher.preClose(fd) // [1] T1: : T1: (skip kill() because keyCount == 1) T1: : T1: exit ss.close() T1 (a): ss = new bound ServerSocketChannel // [2] T2: exit sell.pollWrapper.poll(0) // [3] T1 (b): ss = new bound ServerSocketChannel // [4] T2: sel.processDeregisterQueue() T2: sel.implDereg(sk) T2: : T2: ss.kill() // [5] T1 (c): ss = new bound ServerSocketChannel // [6] T2: ... [1] At this point the FD *should* be closed; dup2 on top of the listening socket should be equivalent to close(fd) [2] We seem to at least sometimes, if not always, get EADDRINUSE if the bind happens here [3] I suspect that this is the actual point where the underlying OS socket is closed for real, though that's only one hypothesis [4] Not clear if bind succeeds at this point, but I think that if it succeeds at [6], it must also succeed here...? [5] It was originally hypothesized that this is the point where the real unbind happens, but I don't see how it could possibly be true [6] Always appears to succeed under empirical observation if the second bind happens here or later I don't really have any hypothesis that can realistically explain what's happening here. I can only say for certain that forcing all selectors to process their cancelled key sets before the second bind seems to always fix the issue. -- - DML From david.lloyd at redhat.com Thu Mar 15 21:01:44 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Thu, 15 Mar 2018 16:01:44 -0500 Subject: A hard-to-reproduce EPollSelector bug... In-Reply-To: References: <23696b2a-42a9-028c-97c3-5128cd97c4d6@oracle.com> Message-ID: On Thu, Mar 15, 2018 at 11:19 AM, David Lloyd wrote: > On Thu, Mar 15, 2018 at 10:46 AM, Alan Bateman wrote: >> On 15/03/2018 15:30, David Lloyd wrote: >>> >>> : >>> Well my naive hope that I could create a quick & dirty fix has been >>> dashed so far. But, looking at the original bug report that sent me >>> down this chase, I see that it was perhaps not limited to just EPoll; >>> KQueue on Mac also suffers (or suffered) from a similar problem, and I >>> understand it happened on Windows as well. So my hypothesis that it >>> is due to epoll weirdness is probably an "overthink" of the problem; >>> maybe it is in fact just a question of ordering the bind correctly as >>> you say. The bug report is publicly viewable and can be found at [1] >>> (the stack traces are the interesting part). >> >> I think all it needs is one thread calling bind at around the same time that >> another thread attempts to closes the channel. It doesn't need any Selectors >> in the picture. > > This is definitely not the expression of the problem; in our case > we're calling close and then bind from the _same_ thread (or otherwise > effectively happens-after), and the bind fails because the close blah blah OK this is my understanding of what's happening. I've chosen EPollSelectorImpl because I understand it most, though as I said we've seen this on other OSes: Mac, Windows (!), and Linux for certain. I can't be 100% sure it's exactly the same problem but even if it isn't, the fix (manually awakening/selecting all associated selectors) solves the problem in every case. Whether that's due to a coincidence of scheduling or some other secondary subtlety is hard to know. ServerSocketChannel ss is registered on Selector sel with SelectionKey sk; Selector is an EPollSelectorImpl. T1 and T2 are threads. The (a), (b), and (c) represent alternative scheduling points where the second bind could take place. T2: sel.select() T2: sel.select(0) T2: sel.lockAndDoSelect(0) T2: sel.doSelect(0) T2: sel.processDeregisterQueue() T2: sel.pollWrapper.poll(0) // ...sleep... T1: ss.close() T1: ss.implCloseChannel() T1: ss.implCloseSelectableChannel() T1: NativeDispatcher.preClose(fd) // [1] T1: : T1: (skip kill() because keyCount == 1) T1: : T1: exit ss.close() T1 (a): ss = new bound ServerSocketChannel // [2] T2: exit sell.pollWrapper.poll(0) // [3] T1 (b): ss = new bound ServerSocketChannel // [4] T2: sel.processDeregisterQueue() T2: sel.implDereg(sk) T2: : T2: ss.kill() // [5] T1 (c): ss = new bound ServerSocketChannel // [6] T2: ... [1] At this point the FD *should* be closed; dup2 on top of the listening socket should be equivalent to close(fd) [2] We seem to at least sometimes, if not always, get EADDRINUSE if the bind happens here [3] I suspect that this is the actual point where the underlying OS socket is closed for real, though that's only one hypothesis [4] Not clear if bind succeeds at this point, but I think that if it succeeds at [6], it must also succeed here...? [5] It was originally hypothesized that this is the point where the real unbind happens, but I don't see how it could possibly be true [6] Always appears to succeed under empirical observation if the second bind happens here or later I don't really have any hypothesis that can realistically explain what's happening here. I can only say for certain that forcing all selectors to process their cancelled key sets before the second bind seems to always fix the issue. -- - DML From huaming.li at oracle.com Fri Mar 16 01:54:48 2018 From: huaming.li at oracle.com (Hamlin Li) Date: Fri, 16 Mar 2018 09:54:48 +0800 Subject: RFR of JDK-8199215: Re-examine getFreePort method in test infrastructure library In-Reply-To: <067292b0-a167-8877-df46-73a081c99301@oracle.com> References: <5a52026a-b3a3-8e16-2a85-1c2bd239e266@oracle.com> <067292b0-a167-8877-df46-73a081c99301@oracle.com> Message-ID: On 15/03/2018 7:20 PM, Alan Bateman wrote: > On 15/03/2018 08:43, Hamlin Li wrote: >> : >> >> Hi Alan, >> >> Thank you for reviewing, I have updated the webrev in place. > ( cc'ing serviceability-dev and net-dev as these are the other areas > that use the getFreePort method in the test library. For context, the > patch that we are discussing is: > ??? http://cr.openjdk.java.net/~mli/8199215/webrev.00/ ) > > The new implementation of getFreePort looks good but it no longer > throws InterruptedException and so might need some of the usages (esp. > in the serviceability tests) to be updated. Also the comment "The > function will spin ..." is no longer relevant and can be removed. > > Moving refusingEndpoint() from the NIO test to Utils looks okay. The > "it's much more stable ..." in the method description looks a it > inconsistent with the other wording. An alternative is "This method is > better choice than getFreePort for tests that need an endpoint that > refuses connections". > > The update to the tests look okay to me. Hi Alan, Thank you for detailed reviewing. I have updated the webrev in place. (http://cr.openjdk.java.net/~mli/8199215/webrev.00/) Thank you -Hamlin > > -Alan > From Alan.Bateman at oracle.com Fri Mar 16 08:05:48 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 16 Mar 2018 08:05:48 +0000 Subject: RFR of JDK-8199215: Re-examine getFreePort method in test infrastructure library In-Reply-To: References: <5a52026a-b3a3-8e16-2a85-1c2bd239e266@oracle.com> <067292b0-a167-8877-df46-73a081c99301@oracle.com> Message-ID: <9ef60dff-7d40-f772-2ecf-74eb13e79391@oracle.com> On 16/03/2018 01:54, Hamlin Li wrote: > : > > Hi Alan, > Thank you for detailed reviewing. I have updated the webrev in place. > (http://cr.openjdk.java.net/~mli/8199215/webrev.00/) Looks good, just a minor typo "is better choice" -> "is a better choice". Just to confirm, have you run the serviceability and http client tests to make sure that they compile with this change? -Alan From huaming.li at oracle.com Fri Mar 16 09:00:42 2018 From: huaming.li at oracle.com (Hamlin Li) Date: Fri, 16 Mar 2018 17:00:42 +0800 Subject: RFR of JDK-8199215: Re-examine getFreePort method in test infrastructure library In-Reply-To: <9ef60dff-7d40-f772-2ecf-74eb13e79391@oracle.com> References: <5a52026a-b3a3-8e16-2a85-1c2bd239e266@oracle.com> <067292b0-a167-8877-df46-73a081c99301@oracle.com> <9ef60dff-7d40-f772-2ecf-74eb13e79391@oracle.com> Message-ID: <668c3c63-ff56-2e00-dc60-5e85c478fd48@oracle.com> On 16/03/2018 4:05 PM, Alan Bateman wrote: > On 16/03/2018 01:54, Hamlin Li wrote: >> : >> >> Hi Alan, >> Thank you for detailed reviewing. I have updated the webrev in place. >> (http://cr.openjdk.java.net/~mli/8199215/webrev.00/) > Looks good, just a minor typo "is better choice" -> "is a better choice". Hi Alan, Thank you, I will modify it when push. > > Just to confirm, have you run the serviceability and http client tests > to make sure that they compile with this change? Yes, I ran tier1,tier2,tier3 tests(I think it includes httpclient tests), and also specific tests using Utils.getFreePort in svc area. I think I'm good to push? Thank you -Hamlin > > -Alan From Alan.Bateman at oracle.com Fri Mar 16 09:23:13 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 16 Mar 2018 09:23:13 +0000 Subject: RFR of JDK-8199215: Re-examine getFreePort method in test infrastructure library In-Reply-To: <668c3c63-ff56-2e00-dc60-5e85c478fd48@oracle.com> References: <5a52026a-b3a3-8e16-2a85-1c2bd239e266@oracle.com> <067292b0-a167-8877-df46-73a081c99301@oracle.com> <9ef60dff-7d40-f772-2ecf-74eb13e79391@oracle.com> <668c3c63-ff56-2e00-dc60-5e85c478fd48@oracle.com> Message-ID: On 16/03/2018 09:00, Hamlin Li wrote: > : >> >> Just to confirm, have you run the serviceability and http client >> tests to make sure that they compile with this change? > Yes, I ran tier1,tier2,tier3 tests(I think it includes httpclient > tests), and also specific tests using Utils.getFreePort in svc area. > I think I'm good to push? Thanks for confirming, I was just surprised that none of the tests in the other areas needed changes.? So yes, go ahead! -Alan From richard.warburton at gmail.com Fri Mar 16 09:50:18 2018 From: richard.warburton at gmail.com (Richard Warburton) Date: Fri, 16 Mar 2018 11:50:18 +0200 Subject: Callback Based Selectors In-Reply-To: <24806bb8-dc16-0bc3-a98c-74f45911476c@oracle.com> References: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> <1365073714.1162106.1521037015119.JavaMail.zimbra@u-pem.fr> <24806bb8-dc16-0bc3-a98c-74f45911476c@oracle.com> Message-ID: Hi Alan, As regards Richard's proposal then it does fit with the existing Selector > design. Rather than adding or updating keys in the selected-key set, it > invokes an action that can collect or process the keys. Iterating over > native events is essentially iterating over a region of memory so it has to > synchronize on the selector to prevent it being closed or used for another > selection operation. In terms of performance, this lock is in the noise in > everything that I've tried but others have have benchmark showing different > results (almost all of the locking issues that I've looked at come up in > the context of changes to registration that are attempted concurrently with > a selection operation). It would be feasible to specify that the only lock > held when processing the events is the selector lock (it doesn't need to be > synchronized on the selector's key or selected-key set for that phase of > selection. > I don't have an open source benchmark to hand, but the general problem with locks when it comes to benchmarks is that they can often be in the noise when it comes to throughput but with tail-latency jitter when some other thread just happens to contend with you in production. Either way I think we're all happy to reduce the number of acquired locks as part of the callback vs using selectNow() and then iterating the keys. regards, Richard Warburton http://insightfullogic.com @RichardWarburto -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.warburton at gmail.com Fri Mar 16 11:27:01 2018 From: richard.warburton at gmail.com (Richard Warburton) Date: Fri, 16 Mar 2018 13:27:01 +0200 Subject: Callback Based Selectors In-Reply-To: References: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> <1365073714.1162106.1521037015119.JavaMail.zimbra@u-pem.fr> Message-ID: Hi David, A specialized set isn't really interesting to me because I have to > process it outside of the lock(s) anyway. ArrayDeque works well for > me because once it's sized, it's very fast, and by using the callback > to populate it we avoid the extra add/remove operation on the key set, > which is already worth something to me. > You're right, it's still an improvement in that use-case. regards, Richard Warburton http://insightfullogic.com @RichardWarburto -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Fri Mar 16 14:31:11 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 16 Mar 2018 14:31:11 +0000 Subject: A hard-to-reproduce EPollSelector bug... In-Reply-To: References: <23696b2a-42a9-028c-97c3-5128cd97c4d6@oracle.com> Message-ID: <82712d4b-4d21-9d48-f154-0072dc84f6fb@oracle.com> On 15/03/2018 21:01, David Lloyd wrote: > : > OK this is my understanding of what's happening. I've chosen > EPollSelectorImpl because I understand it most, though as I said we've > seen this on other OSes: Mac, Windows (!), and Linux for certain. I > can't be 100% sure it's exactly the same problem but even if it isn't, > the fix (manually awakening/selecting all associated selectors) solves > the problem in every case. Whether that's due to a coincidence of > scheduling or some other secondary subtlety is hard to know. ServerSocketChannel.bind didn't historically synchronize correctly on channel state and so could fail in interesting ways when called at around the same time that it is closed. Specifically it could attempt to bind after the dup2, or worse, after the file descriptor has been closed and recycled. If there were a T3 in the picture that was created a listener at around the same time then it might be possible to explain the EADDRINUSE. -Alan From david.lloyd at redhat.com Fri Mar 16 15:03:21 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Fri, 16 Mar 2018 10:03:21 -0500 Subject: A hard-to-reproduce EPollSelector bug... In-Reply-To: <82712d4b-4d21-9d48-f154-0072dc84f6fb@oracle.com> References: <23696b2a-42a9-028c-97c3-5128cd97c4d6@oracle.com> <82712d4b-4d21-9d48-f154-0072dc84f6fb@oracle.com> Message-ID: On Fri, Mar 16, 2018 at 9:31 AM, Alan Bateman wrote: > On 15/03/2018 21:01, David Lloyd wrote: >> >> : >> OK this is my understanding of what's happening. I've chosen >> EPollSelectorImpl because I understand it most, though as I said we've >> seen this on other OSes: Mac, Windows (!), and Linux for certain. I >> can't be 100% sure it's exactly the same problem but even if it isn't, >> the fix (manually awakening/selecting all associated selectors) solves >> the problem in every case. Whether that's due to a coincidence of >> scheduling or some other secondary subtlety is hard to know. > > ServerSocketChannel.bind didn't historically synchronize correctly on > channel state and so could fail in interesting ways when called at around > the same time that it is closed. Specifically it could attempt to bind after > the dup2, or worse, after the file descriptor has been closed and recycled. > If there were a T3 in the picture that was created a listener at around the > same time then it might be possible to explain the EADDRINUSE. OK that's interesting. This most often appears in tests so there is a lot of rapid setup/teardown happening with lots of threads around, so maybe there's a chance that we're binding some wrong FD in these cases. I might see if I can make a reproducer with that in mind. -- - DML From brian.burkhalter at oracle.com Mon Mar 19 15:07:55 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 19 Mar 2018 08:07:55 -0700 Subject: 8198753: (dc) DatagramChannel throws unspecified exceptions Message-ID: <09E8887A-537A-4659-85B9-7EC4E9CF5367@oracle.com> https://bugs.openjdk.java.net/browse/JDK-8198753 http://cr.openjdk.java.net/~bpb/8198753/webrev.00/ Change an undocumented IllegalArgumentException to AlreadyConnectedException and document some other undocumented exceptions. CSR to follow. Thanks, Brian From xueming.shen at oracle.com Mon Mar 19 15:57:59 2018 From: xueming.shen at oracle.com (Xueming Shen) Date: Mon, 19 Mar 2018 08:57:59 -0700 Subject: Callback Based Selectors In-Reply-To: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> References: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> Message-ID: <5AAFDE07.5010705@oracle.com> Is it a concern that the new method opens the door to invite people start to write "processing code" inside "selecting" operation? which is different use model compared to the existing one. It's fine if the only thing the passed in "consumer" does is to simply add the SelectionKey into a queue for later process, as David intends to do. Or even go a little further to create a task and submit into some executor, but it might start to be an issue if the "consumer" starts to read couple byes from the io or even worse if the io is in blocking mode ... It appears the original motivation is to help the performance of handling those keys, but seems it is kinda of trade off of the "select" throughput . The "select()" is now being blocked till the consumer is applied/accepted for all selected keys, and if we are having a long list of "ready" keys then you might want to ask if it's a worthy tradeoff. For David's use scenario, a more straightforward method might be better? int select(Queue q, long timeout); But I doubt it brings big difference as calling selectedKeys() after select() :-) I'm sure the proposed method has its own use case. But it might be worth some more words against the "inappropriate" usage. Sherman it appears the new method opens the door to invite people to write the code to process the "key" during selecting. On 3/14/18, 7:05 AM, Alan Bateman wrote: > On 16/01/2016 11:14, Richard Warburton wrote: >> Hi gents, >> >> I've prototyped a callback based addition to the NIO Selector, which >> I've previously talked through a bit with Alan Bateman. The goal of >> the callback based selector is to avoid the current pattern of >> calling select/selectNow on a Nio selector and then having to iterate >> over a hashmap produced. This pattern being quite object allocation >> heavy for a critical path and also involving obtaining and releasing >> multiple locks. I'd like to propose that the following patch, which >> adds the ability to perform a select on a Selector that takes a >> callback handler. >> >> http://cr.openjdk.java.net/~rwarburton/select-now-4/ >> >> >> I'm happy to iterate on the patch a bit based upon people's feedback >> and discuss any potential concerns or issues that you may have with >> the patch. Looking forward to hearing your feedback. >> > This one has been on my list for a long time to study more closely. > When we discussed it here in 2016 I had concerns about invoking a > consumer while synchronized on the selector and other locks. It has > grown on my a bit so I spent a bit of time recently to work through > the implications, both spec and implementation. > > For the spec: The Selector API describes selection operations in its > class description and it's important to keep that consistent when > introducing a new way to be notified of channels ready for I/O operations. > > The action is arbitrary code and we have to assume it will behave as > if it's a bull in a china shop. It might attempt another selection > operation on the same selector, it might close the selector, it might > change the interest set or cancel a key for a channel registered with > the selector but not yet seen by the consumer. All of these scenarios > need consideration. > > Locking was mostly ignored in the original prototype but the locking > has to be consistent with how the existing selection operations are > specified. The main reason is processing the cancelled-key set means > removing keys from the key set and selected-key set. On the surface, > the new methods should not care about the selected-key set but they > have to maintain the invariant that the selected-key set is always a > subset of the selector's key set. > > Another point of detail is that the action may need to be called more > than once for the same key but different ready sets. We have this with > the kqueue implementation where events for read and write are > registered with different filters. Other implementation might maintain > separate poll arrays for read and write events. > > In terms of performance, the main benefit is that avoids adding keys > to the selected-key set, only to be removed almost immediately by the > code doing the select. This avoids garbage so helps the GC too. If we > go ahead with these methods then it's important to get a good range of > performance data and also see whether a specialized set to support the > usage pattern of the selected-key set might be alternative to > introducing new APIs. > > I've put a webrev here with candidate changes to the Selector API. > It's one select(Consumer, timeout) for now, the > select(Consumer) and selectNow(Consumer) > variants are easy to add if needed. The webrev has a default > implementation based on the existing API, and then implementations for > macOS and Linux. Other platforms could be added later of course. I've > also included a test that covers most of the scenarios. > http://cr.openjdk.java.net/~alanb/8199433/webrev/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.lloyd at redhat.com Mon Mar 19 16:12:29 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Mon, 19 Mar 2018 11:12:29 -0500 Subject: Callback Based Selectors In-Reply-To: <5AAFDE07.5010705@oracle.com> References: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> <5AAFDE07.5010705@oracle.com> Message-ID: On Mon, Mar 19, 2018 at 10:57 AM, Xueming Shen wrote: > > Is it a concern that the new method opens the door to invite people start to > write > "processing code" inside "selecting" operation? which is different use model > compared > to the existing one. I don't think that's necessarily a bad thing. For example it could be perfectly reasonable to accept connections or perform simple UDP request processing in a callback. Given the lack of locking the key sets in this case, my queue approach might even become obsolete, which would be ideal from my perspective. > It's fine if the only thing the passed in "consumer" > does is to simply > add the SelectionKey into a queue for later process, as David intends to do. > Or even go > a little further to create a task and submit into some executor, but it > might start to > be an issue if the "consumer" starts to read couple byes from the io or even > worse > if the io is in blocking mode ... Channels in blocking mode won't be registered in a selector, hopefully. Our frameworks at least keep all channels in non-blocking mode and use temporary selectors to simulate blocking behavior. But I guess your point is that applications would need to be cognizant of the fact that the selector lock would be held for the duration of the selection operation. I don't think that this specifically is a serious problem though. > It appears the original motivation is to help the performance of handling > those keys, > but seems it is kinda of trade off of the "select" throughput . The > "select()" is now being > blocked till the consumer is applied/accepted for all selected keys, and if > we are having > a long list of "ready" keys then you might want to ask if it's a worthy > tradeoff. > > For David's use scenario, a more straightforward method might be better? > > int select(Queue q, long timeout); Users who want to process on select will just write a really ugly Queue implementation which maps all "add" operations to their processing routine. IMO it's better for the API to avoid artificial restrictions and stick with the best mapping - in this case the best mapping would be a consumer of some sort (though I think an argument could possibly be made for allowing the callback interface to be able to throw IOException). > But I doubt it brings big difference as calling selectedKeys() after > select() :-) > > I'm sure the proposed method has its own use case. But it might be worth > some more > words against the "inappropriate" usage. I agree that admonitions against misuse are a good idea. -- - DML From david.lloyd at redhat.com Mon Mar 19 16:23:49 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Mon, 19 Mar 2018 11:23:49 -0500 Subject: Callback Based Selectors In-Reply-To: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> References: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> Message-ID: On Wed, Mar 14, 2018 at 9:05 AM, Alan Bateman wrote: > Locking was mostly ignored in the original prototype but the locking has to > be consistent with how the existing selection operations are specified. The > main reason is processing the cancelled-key set means removing keys from the > key set and selected-key set. On the surface, the new methods should not > care about the selected-key set but they have to maintain the invariant that > the selected-key set is always a subset of the selector's key set. One thing worth noting is that it is possible for the callback code to temporarily release the lock by calling Object.wait on the selector instance. I think that your reentrancy guard will protect against any problems that could result - in fact this may in the end become a desirable technique, for certain specific cases at least - but I wanted to make sure that it was specifically called out to make sure it doesn't get forgotten. One other thing I was wondering is whether it might be a good idea to process cancelled keys *after* the selection callbacks are complete, either in addition to or instead of processing after selection but before the callbacks are called? I think it would not be unusual for a callback to cancel selection key(s) during the course of its processing. Overall this proposal is growing on me quite a lot. -- - DML From Alan.Bateman at oracle.com Mon Mar 19 17:01:18 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 19 Mar 2018 17:01:18 +0000 Subject: 8198753: (dc) DatagramChannel throws unspecified exceptions In-Reply-To: <09E8887A-537A-4659-85B9-7EC4E9CF5367@oracle.com> References: <09E8887A-537A-4659-85B9-7EC4E9CF5367@oracle.com> Message-ID: On 19/03/2018 15:07, Brian Burkhalter wrote: > https://bugs.openjdk.java.net/browse/JDK-8198753 > http://cr.openjdk.java.net/~bpb/8198753/webrev.00/ > > Change an undocumented IllegalArgumentException to AlreadyConnectedException and document some other undocumented exceptions. > > CSR to follow. > This looks good. Can you check that we have test coverage for this? I suspect we might need to add a few tests for these cases. -Alan From Alan.Bateman at oracle.com Mon Mar 19 18:34:52 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 19 Mar 2018 18:34:52 +0000 Subject: A hard-to-reproduce EPollSelector bug... In-Reply-To: References: <23696b2a-42a9-028c-97c3-5128cd97c4d6@oracle.com> <82712d4b-4d21-9d48-f154-0072dc84f6fb@oracle.com> Message-ID: <89332faa-78e1-d017-d481-c4344db6430a@oracle.com> On 16/03/2018 15:03, David Lloyd wrote: > : > OK that's interesting. This most often appears in tests so there is a > lot of rapid setup/teardown happening with lots of threads around, so > maybe there's a chance that we're binding some wrong FD in these > cases. I might see if I can make a reproducer with that in mind. > If you manage to create a reproducer then it would be great if you run it on a recent JDK 11 build to see if it happens there (I'm hoping it won't reproduce there as the bind method was fixed during recent cleanup in that area). -Alan From Alan.Bateman at oracle.com Mon Mar 19 18:39:01 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 19 Mar 2018 18:39:01 +0000 Subject: Callback Based Selectors In-Reply-To: References: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> Message-ID: On 19/03/2018 16:23, David Lloyd wrote: > : > One thing worth noting is that it is possible for the callback code to > temporarily release the lock by calling Object.wait on the selector > instance. I think that your reentrancy guard will protect against any > problems that could result - in fact this may in the end become a > desirable technique, for certain specific cases at least - but I > wanted to make sure that it was specifically called out to make sure > it doesn't get forgotten. Good, this is something that would need to considered too, and tested. > > One other thing I was wondering is whether it might be a good idea to > process cancelled keys *after* the selection callbacks are complete, > either in addition to or instead of processing after selection but > before the callbacks are called? I think it would not be unusual for > a callback to cancel selection key(s) during the course of its > processing. Maybe although if select is called in a loop then it will be deregistered almost immediately anyway. -Alan From brian.burkhalter at oracle.com Mon Mar 19 20:18:44 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 19 Mar 2018 13:18:44 -0700 Subject: 8198753: (dc) DatagramChannel throws unspecified exceptions In-Reply-To: References: <09E8887A-537A-4659-85B9-7EC4E9CF5367@oracle.com> Message-ID: <5D48A588-4E53-4683-B97A-00ADBFD6B069@oracle.com> On Mar 19, 2018, at 10:01 AM, Alan Bateman wrote: > On 19/03/2018 15:07, Brian Burkhalter wrote: >> https://bugs.openjdk.java.net/browse/JDK-8198753 >> http://cr.openjdk.java.net/~bpb/8198753/webrev.00/ >> >> Change an undocumented IllegalArgumentException to AlreadyConnectedException and document some other undocumented exceptions. >> >> CSR to follow. >> > This looks good. Can you check that we have test coverage for this? I suspect we might need to add a few tests for these cases. I noticed that send() also needs to declare that it throws UnresolvedAddressException and UnsupportedAddressTypeException. java/nio/channels/DatagramChannel/SendToUnresolved.java already tests for the UnresolvedAddressException of send(). The other exceptions do not have test coverage. I modified ConnectedSend to test for these and to use TestNG: http://cr.openjdk.java.net/~bpb/8198753/webrev.01/ Perhaps I should pull the single test done in SendToUnresolved into ConnectedSend (with bug ID) and remove SendToUnresolved altogether? Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.lloyd at redhat.com Tue Mar 20 01:57:20 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Mon, 19 Mar 2018 20:57:20 -0500 Subject: Callback Based Selectors In-Reply-To: References: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> <1365073714.1162106.1521037015119.JavaMail.zimbra@u-pem.fr> <7185f4eb-618c-1e11-0267-43e0be448995@oracle.com> Message-ID: On Thu, Mar 15, 2018 at 8:44 AM, Alan Bateman wrote: > While there are > challenges, esp. with poll-based implementations, it should be possible to > get to the point where the only lock held when invoking the action is the > selector itself. Going further and not locking the key set during a select > isn't hard on some platforms but posed challenges on poll or select based > implementations. Thinking about this some more, I am coming to believe that this is the crux of the locking problems we routinely encounter. If we can get to a point where we can, in specification, state that select ops (or maybe even just the new select op(s)) will only hold the selector lock while blocking, then we are in an ideal situation as far as frameworks go because the aforementioned strategy (for channel registration which occurs concurrently to a selection operation) of "register and wakeup" will always work. If we cannot do that in all cases though, we could mitigate it somewhat by having a method which can query the SelectorProvider to see if it needs to lock the key set, and choose an alternative implementation accordingly, for example using an external work queue to move the registration operation to the selector thread and then applying wakeup(), or having a selection callback perform a coordinated release of the key set's lock for the duration of the registration operation. -- - DML From Alan.Bateman at oracle.com Tue Mar 20 07:56:34 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 20 Mar 2018 07:56:34 +0000 Subject: 8198753: (dc) DatagramChannel throws unspecified exceptions In-Reply-To: <5D48A588-4E53-4683-B97A-00ADBFD6B069@oracle.com> References: <09E8887A-537A-4659-85B9-7EC4E9CF5367@oracle.com> <5D48A588-4E53-4683-B97A-00ADBFD6B069@oracle.com> Message-ID: <861d1e8d-6fb1-4c9c-bb28-bf9725c2cefa@oracle.com> On 19/03/2018 20:18, Brian Burkhalter wrote: > > I noticed that send() also needs to declare that it throws > UnresolvedAddressException and UnsupportedAddressTypeException. Good, I agree we should fix this too. The changes to the API + implementation looks right. One suggestion for the ordering of the exceptions is to move the AlreadyConnectedException up a bit so that ClosedChannelException and AsynchronousCloseException follow each other. > > java/nio/channels/DatagramChannel/SendToUnresolved.java already tests > for the UnresolvedAddressException of send(). The other exceptions do > not have test coverage. I modified ConnectedSend to test for these and > to use TestNG: > > http://cr.openjdk.java.net/~bpb/8198753/webrev.01/ > > > Perhaps I should pull the single test done in SendToUnresolved into > ConnectedSend (with bug ID) and remove SendToUnresolved altogether? ConnectedSend was created to test send with a connected datagram channel so it will need to be renamed and its summary changed if extended to test both connect and send. I think it would be easier to maintain if they were separate tests. Also as you are moving these to TestNG then the specific tests cases can be very small. DatagramChannel has a bind method so you don't need to use the socket adapter to bind. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Mar 20 08:25:31 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 20 Mar 2018 08:25:31 +0000 Subject: Callback Based Selectors In-Reply-To: References: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> <1365073714.1162106.1521037015119.JavaMail.zimbra@u-pem.fr> <7185f4eb-618c-1e11-0267-43e0be448995@oracle.com> Message-ID: <2f5bae96-9877-8afa-4ded-04c4e83a601f@oracle.com> On 20/03/2018 01:57, David Lloyd wrote: > : > Thinking about this some more, I am coming to believe that this is the > crux of the locking problems we routinely encounter. If we can get to > a point where we can, in specification, state that select ops (or > maybe even just the new select op(s)) will only hold the selector lock > while blocking, then we are in an ideal situation as far as frameworks > go because the aforementioned strategy (for channel registration which > occurs concurrently to a selection operation) of "register and wakeup" > will always work. If we cannot do that in all cases though, we could > mitigate it somewhat by having a method which can query the > SelectorProvider to see if it needs to lock the key set, and choose an > alternative implementation accordingly, for example using an external > work queue to move the registration operation to the selector thread > and then applying wakeup(), or having a selection callback perform a > coordinated release of the key set's lock for the duration of the > registration operation. It requires significant refactoring but I think it should be possible to re-work the poll and select based implementations to not access the poll array during registration and changing interest ops. I think that would be better than having two code paths for every select usage. That said, implementations outside of the JDK will be a challenge. A useful property of the prototype API is that is that it can be implemented on the existing select methods. -Alan From christoph.langer at sap.com Tue Mar 20 10:39:20 2018 From: christoph.langer at sap.com (Langer, Christoph) Date: Tue, 20 Mar 2018 10:39:20 +0000 Subject: Provide API to explicitly unmap MappedByteBuffer? Message-ID: <9c559d1c5d7b495e81f9e567e750c85e@sap.com> Hi, I was just handling a customer issue where Java code tried to delete a file on Windows which was mapped into memory vial FileChannel.map() before. It took me a little while to understand the issue but obviously the problem was that Windows still kept a delete lock on the file as long as the ByteBuffer was still mapped. This behavior is also documented cleanly: https://download.java.net/java/jdk10/docs/api/java/nio/MappedByteBuffer.html. So far so good, I just started wondering why there isn't an API to force unmapping (cleanup) on a MappedByteBuffer? The app that opens the mapping simply reads a properties file at this place and I guess they should be able to change this place to use standard io easily and without observable performance problems. However, if they had a way to explicitly unmap the MappedByteBuffer it would also help. Was this topic already discussed? Is there a hard reason why explicit cleanup should not be offered? I guess people using MappedByteBuffers should know anyway what they are doing... Thanks & Best regards Christoph -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Mar 20 13:47:53 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 20 Mar 2018 13:47:53 +0000 Subject: 8199791: (se) More Selector cleanup Message-ID: <5f7c43fb-c458-c742-c443-374f57d8ad46@oracle.com> This is another batch of cleanup to the Selector implementations. A significant part of the cleanup is changing the epoll and kqueue based Selector implementations to use the EPoll and KQueue support classes. This eliminate their wrapper classes and several native methods. The result is much smaller implementations. The queuing of changes to the interest set has also been re-worked to avoid needing cancellation needing to kill all pending updates. A few of the Selector implementations had a native method to write a byte to one end of the pipe used for the wakeup mechanism. This has been changed to use IOUtil.write1 instead. Interrupting timed polls (as in EINTR, not Thread interrupt) wasn't handled correctly in some of the implementations so that has been re-worked. The webrev with the changes is here: ??? http://cr.openjdk.java.net/~alanb/8199791/webrev/index.html I have several other changes for this area but I'd prefer to do the changes in small batches to avoid changing too much in one go. Small patches are easier to review too of course (and in this case, there is more deletion than new code). I have run all tests on all platforms several times. -Alan From Alan.Bateman at oracle.com Tue Mar 20 13:59:18 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 20 Mar 2018 13:59:18 +0000 Subject: Provide API to explicitly unmap MappedByteBuffer? In-Reply-To: <9c559d1c5d7b495e81f9e567e750c85e@sap.com> References: <9c559d1c5d7b495e81f9e567e750c85e@sap.com> Message-ID: <0c9d2751-c0c9-437a-d346-3e30c095194c@oracle.com> On 20/03/2018 10:39, Langer, Christoph wrote: > > Hi, > > I was just handling a customer issue where Java code tried to delete a > file on Windows which was mapped into memory vial FileChannel.map() > before. It took me a little while to understand the issue but > obviously the problem was that Windows still kept a delete lock on the > file as long as the ByteBuffer was still mapped. This behavior is also > documented cleanly: > https://download.java.net/java/jdk10/docs/api/java/nio/MappedByteBuffer.html. > > So far so good, I just started wondering why there isn?t an API to > force unmapping (cleanup) on a MappedByteBuffer? The app that opens > the mapping simply reads a properties file at this place and I guess > they should be able to change this place to use standard io easily and > without observable performance problems. However, if they had a way to > explicitly unmap the MappedByteBuffer it would also help. > > Was this topic already discussed? Is there a hard reason why explicit > cleanup should not be offered? I guess people using MappedByteBuffers > should know anyway what they are doing? > > This has been discussed many times, it's a really hard problem that many smart people have spend time on. Andrew Haley may be able to say more about this. He has (or had) a proposal on this but it involved deep changes in the VM. I don't know he has prototyped it yet to see how feasible it is. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.lloyd at redhat.com Tue Mar 20 14:44:52 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Tue, 20 Mar 2018 09:44:52 -0500 Subject: 8199791: (se) More Selector cleanup In-Reply-To: <5f7c43fb-c458-c742-c443-374f57d8ad46@oracle.com> References: <5f7c43fb-c458-c742-c443-374f57d8ad46@oracle.com> Message-ID: Looks good to me (non-reviewer). I still believe the timeout calcs should use System.nanoTime() instead of currentTimeMillis(), and I'd like to see Linux using eventfd instead of a pipe for selector wakeups... but those are subjects for future discussion/patches, probably. On Tue, Mar 20, 2018 at 8:47 AM, Alan Bateman wrote: > This is another batch of cleanup to the Selector implementations. > > A significant part of the cleanup is changing the epoll and kqueue based > Selector implementations to use the EPoll and KQueue support classes. This > eliminate their wrapper classes and several native methods. The result is > much smaller implementations. The queuing of changes to the interest set has > also been re-worked to avoid needing cancellation needing to kill all > pending updates. A few of the Selector implementations had a native method > to write a byte to one end of the pipe used for the wakeup mechanism. This > has been changed to use IOUtil.write1 instead. Interrupting timed polls (as > in EINTR, not Thread interrupt) wasn't handled correctly in some of the > implementations so that has been re-worked. > > The webrev with the changes is here: > http://cr.openjdk.java.net/~alanb/8199791/webrev/index.html > > I have several other changes for this area but I'd prefer to do the changes > in small batches to avoid changing too much in one go. Small patches are > easier to review too of course (and in this case, there is more deletion > than new code). > > I have run all tests on all platforms several times. > > -Alan -- - DML From Alan.Bateman at oracle.com Tue Mar 20 14:53:29 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 20 Mar 2018 14:53:29 +0000 Subject: 8199791: (se) More Selector cleanup In-Reply-To: References: <5f7c43fb-c458-c742-c443-374f57d8ad46@oracle.com> Message-ID: <86aceb35-c70a-2d32-4684-9942cef8947d@oracle.com> On 20/03/2018 14:44, David Lloyd wrote: > Looks good to me (non-reviewer). I still believe the timeout calcs > should use System.nanoTime() instead of currentTimeMillis(), and I'd > like to see Linux using eventfd instead of a pipe for selector > wakeups... but those are subjects for future discussion/patches, > probably. You're probably right that this should move to nanoTime. On eventfs, what advantage do you see for using this over the socketpair? -Alan From christoph.langer at sap.com Tue Mar 20 14:54:33 2018 From: christoph.langer at sap.com (Langer, Christoph) Date: Tue, 20 Mar 2018 14:54:33 +0000 Subject: Provide API to explicitly unmap MappedByteBuffer? In-Reply-To: <0c9d2751-c0c9-437a-d346-3e30c095194c@oracle.com> References: <9c559d1c5d7b495e81f9e567e750c85e@sap.com> <0c9d2751-c0c9-437a-d346-3e30c095194c@oracle.com> Message-ID: <538fefe7b73a4eca9a936990fa4d464e@sap.com> Thanks Alan, I can see the history in bug https://bugs.openjdk.java.net/browse/JDK-4724038... Best regards Christoph From: Alan Bateman [mailto:Alan.Bateman at oracle.com] Sent: Dienstag, 20. M?rz 2018 14:59 To: Langer, Christoph ; nio-dev at openjdk.java.net Subject: Re: Provide API to explicitly unmap MappedByteBuffer? On 20/03/2018 10:39, Langer, Christoph wrote: Hi, I was just handling a customer issue where Java code tried to delete a file on Windows which was mapped into memory vial FileChannel.map() before. It took me a little while to understand the issue but obviously the problem was that Windows still kept a delete lock on the file as long as the ByteBuffer was still mapped. This behavior is also documented cleanly: https://download.java.net/java/jdk10/docs/api/java/nio/MappedByteBuffer.html. So far so good, I just started wondering why there isn't an API to force unmapping (cleanup) on a MappedByteBuffer? The app that opens the mapping simply reads a properties file at this place and I guess they should be able to change this place to use standard io easily and without observable performance problems. However, if they had a way to explicitly unmap the MappedByteBuffer it would also help. Was this topic already discussed? Is there a hard reason why explicit cleanup should not be offered? I guess people using MappedByteBuffers should know anyway what they are doing... This has been discussed many times, it's a really hard problem that many smart people have spend time on. Andrew Haley may be able to say more about this. He has (or had) a proposal on this but it involved deep changes in the VM. I don't know he has prototyped it yet to see how feasible it is. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Mar 20 15:11:55 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 20 Mar 2018 08:11:55 -0700 Subject: 8198753: (dc) DatagramChannel throws unspecified exceptions In-Reply-To: <861d1e8d-6fb1-4c9c-bb28-bf9725c2cefa@oracle.com> References: <09E8887A-537A-4659-85B9-7EC4E9CF5367@oracle.com> <5D48A588-4E53-4683-B97A-00ADBFD6B069@oracle.com> <861d1e8d-6fb1-4c9c-bb28-bf9725c2cefa@oracle.com> Message-ID: <2044C0C9-627D-4426-8437-9FF38AC73361@oracle.com> On Mar 20, 2018, at 12:56 AM, Alan Bateman wrote: >> Perhaps I should pull the single test done in SendToUnresolved into ConnectedSend (with bug ID) and remove SendToUnresolved altogether? > ConnectedSend was created to test send with a connected datagram channel so it will need to be renamed and its summary changed if extended to test both connect and send. I think it would be easier to maintain if they were separate tests. If the tests are split across two different test classes can they both have the same @bug ID? Or the one which deals with send() exceptions retains the bug ID in SendToUnresolved and the current bug ID is used in the test which deals with connect() exceptions? > Also as you are moving these to TestNG then the specific tests cases can be very small. DatagramChannel has a bind method so you don't need to use the socket adapter to bind. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Mar 20 15:13:56 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 20 Mar 2018 15:13:56 +0000 Subject: 8198753: (dc) DatagramChannel throws unspecified exceptions In-Reply-To: <2044C0C9-627D-4426-8437-9FF38AC73361@oracle.com> References: <09E8887A-537A-4659-85B9-7EC4E9CF5367@oracle.com> <5D48A588-4E53-4683-B97A-00ADBFD6B069@oracle.com> <861d1e8d-6fb1-4c9c-bb28-bf9725c2cefa@oracle.com> <2044C0C9-627D-4426-8437-9FF38AC73361@oracle.com> Message-ID: <244cd878-4d90-9fb7-06df-242a12f3b652@oracle.com> On 20/03/2018 15:11, Brian Burkhalter wrote: > On Mar 20, 2018, at 12:56 AM, Alan Bateman > wrote: > >>> Perhaps I should pull the single test done in SendToUnresolved into >>> ConnectedSend (with bug ID) and remove SendToUnresolved altogether? >> ConnectedSend was created to test send with a connected datagram >> channel so it will need to be renamed and its summary changed if >> extended to test both connect and send. I think it would be easier to >> maintain if they were separate tests. > > If the tests are split across two different test classes can they both > have the same @bug ID? Or the one which deals with send() exceptions > retains the bug ID in SendToUnresolved and the current bug ID is used > in the test which deals with connect() exceptions? > It's okay for several tests to list the same bugID. In this case, I assume both tests would list several bugIDs. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Mar 20 15:25:24 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 20 Mar 2018 08:25:24 -0700 Subject: 8198753: (dc) DatagramChannel throws unspecified exceptions In-Reply-To: <244cd878-4d90-9fb7-06df-242a12f3b652@oracle.com> References: <09E8887A-537A-4659-85B9-7EC4E9CF5367@oracle.com> <5D48A588-4E53-4683-B97A-00ADBFD6B069@oracle.com> <861d1e8d-6fb1-4c9c-bb28-bf9725c2cefa@oracle.com> <2044C0C9-627D-4426-8437-9FF38AC73361@oracle.com> <244cd878-4d90-9fb7-06df-242a12f3b652@oracle.com> Message-ID: On Mar 20, 2018, at 8:13 AM, Alan Bateman wrote: > It's okay for several tests to list the same bugID. In this case, I assume both tests would list several bugIDs. Yes, that?s correct. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.lloyd at redhat.com Tue Mar 20 15:25:48 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Tue, 20 Mar 2018 10:25:48 -0500 Subject: 8199791: (se) More Selector cleanup In-Reply-To: <86aceb35-c70a-2d32-4684-9942cef8947d@oracle.com> References: <5f7c43fb-c458-c742-c443-374f57d8ad46@oracle.com> <86aceb35-c70a-2d32-4684-9942cef8947d@oracle.com> Message-ID: On Tue, Mar 20, 2018 at 9:53 AM, Alan Bateman wrote: > On 20/03/2018 14:44, David Lloyd wrote: >> >> Looks good to me (non-reviewer). I still believe the timeout calcs >> should use System.nanoTime() instead of currentTimeMillis(), and I'd >> like to see Linux using eventfd instead of a pipe for selector >> wakeups... but those are subjects for future discussion/patches, >> probably. > > You're probably right that this should move to nanoTime. On eventfs, what > advantage do you see for using this over the socketpair? My understanding is that eventfd has lower overhead in the kernel (no buffer space allocation for example, and dispatch is faster as a result of this and other things) and certainly in userspace (using only one FD instead of two). -- - DML From claes.redestad at oracle.com Tue Mar 20 15:33:58 2018 From: claes.redestad at oracle.com (Claes Redestad) Date: Tue, 20 Mar 2018 16:33:58 +0100 Subject: 8199791: (se) More Selector cleanup In-Reply-To: <5f7c43fb-c458-c742-c443-374f57d8ad46@oracle.com> References: <5f7c43fb-c458-c742-c443-374f57d8ad46@oracle.com> Message-ID: Looks good to me (probably shouldn't count me as a Reviewer, though). /Claes On 2018-03-20 14:47, Alan Bateman wrote: > This is another batch of cleanup to the Selector implementations. > > A significant part of the cleanup is changing the epoll and kqueue > based Selector implementations to use the EPoll and KQueue support > classes. This eliminate their wrapper classes and several native > methods. The result is much smaller implementations. The queuing of > changes to the interest set has also been re-worked to avoid needing > cancellation needing to kill all pending updates. A few of the > Selector implementations had a native method to write a byte to one > end of the pipe used for the wakeup mechanism. This has been changed > to use IOUtil.write1 instead. Interrupting timed polls (as in EINTR, > not Thread interrupt) wasn't handled correctly in some of the > implementations so that has been re-worked. > > The webrev with the changes is here: > ??? http://cr.openjdk.java.net/~alanb/8199791/webrev/index.html > > I have several other changes for this area but I'd prefer to do the > changes in small batches to avoid changing too much in one go. Small > patches are easier to review too of course (and in this case, there is > more deletion than new code). > > I have run all tests on all platforms several times. > > -Alan From Alan.Bateman at oracle.com Tue Mar 20 15:46:18 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 20 Mar 2018 15:46:18 +0000 Subject: 8199791: (se) More Selector cleanup In-Reply-To: References: <5f7c43fb-c458-c742-c443-374f57d8ad46@oracle.com> <86aceb35-c70a-2d32-4684-9942cef8947d@oracle.com> Message-ID: <4ae43bc3-a516-4dcc-f8b9-d1567d42ce58@oracle.com> On 20/03/2018 15:25, David Lloyd wrote: > : > My understanding is that eventfd has lower overhead in the kernel (no > buffer space allocation for example, and dispatch is faster as a > result of this and other things) and certainly in userspace (using > only one FD instead of two). > The number of Selectors is usually small (at least since the use of temporary selectors was eliminated some time ago) so it would be good to get some data to see if this is worth doing. -Alan From paul.sandoz at oracle.com Tue Mar 20 16:19:16 2018 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 20 Mar 2018 09:19:16 -0700 Subject: Provide API to explicitly unmap MappedByteBuffer? In-Reply-To: <538fefe7b73a4eca9a936990fa4d464e@sap.com> References: <9c559d1c5d7b495e81f9e567e750c85e@sap.com> <0c9d2751-c0c9-437a-d346-3e30c095194c@oracle.com> <538fefe7b73a4eca9a936990fa4d464e@sap.com> Message-ID: <2546371B-0F5F-420E-9839-396EA307BA9D@oracle.com> Also, we may have another swing at this problem in Project Panama, where memory regions are more explicitly managed via scopes. However, there will still be similar safety and performance challenges. Paul. > On Mar 20, 2018, at 7:54 AM, Langer, Christoph wrote: > > Thanks Alan, > > I can see the history in bug https://bugs.openjdk.java.net/browse/JDK-4724038 ... > > Best regards > Christoph > > > ? <> > From: Alan Bateman [mailto:Alan.Bateman at oracle.com] > Sent: Dienstag, 20. M?rz 2018 14:59 > To: Langer, Christoph ; nio-dev at openjdk.java.net > Subject: Re: Provide API to explicitly unmap MappedByteBuffer? > > On 20/03/2018 10:39, Langer, Christoph wrote: > > Hi, > > I was just handling a customer issue where Java code tried to delete a file on Windows which was mapped into memory vial FileChannel.map() before. It took me a little while to understand the issue but obviously the problem was that Windows still kept a delete lock on the file as long as the ByteBuffer was still mapped. This behavior is also documented cleanly:https://download.java.net/java/jdk10/docs/api/java/nio/MappedByteBuffer.html . > > So far so good, I just started wondering why there isn?t an API to force unmapping (cleanup) on a MappedByteBuffer? The app that opens the mapping simply reads a properties file at this place and I guess they should be able to change this place to use standard io easily and without observable performance problems. However, if they had a way to explicitly unmap the MappedByteBuffer it would also help. > > Was this topic already discussed? Is there a hard reason why explicit cleanup should not be offered? I guess people using MappedByteBuffers should know anyway what they are doing? > > This has been discussed many times, it's a really hard problem that many smart people have spend time on. > > Andrew Haley may be able to say more about this. He has (or had) a proposal on this but it involved deep changes in the VM. I don't know he has prototyped it yet to see how feasible it is. > > -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.lloyd at redhat.com Tue Mar 20 16:19:01 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Tue, 20 Mar 2018 11:19:01 -0500 Subject: 8199791: (se) More Selector cleanup In-Reply-To: <4ae43bc3-a516-4dcc-f8b9-d1567d42ce58@oracle.com> References: <5f7c43fb-c458-c742-c443-374f57d8ad46@oracle.com> <86aceb35-c70a-2d32-4684-9942cef8947d@oracle.com> <4ae43bc3-a516-4dcc-f8b9-d1567d42ce58@oracle.com> Message-ID: On Tue, Mar 20, 2018 at 10:46 AM, Alan Bateman wrote: > On 20/03/2018 15:25, David Lloyd wrote: >> >> : >> My understanding is that eventfd has lower overhead in the kernel (no >> buffer space allocation for example, and dispatch is faster as a >> result of this and other things) and certainly in userspace (using >> only one FD instead of two). >> > The number of Selectors is usually small (at least since the use of > temporary selectors was eliminated some time ago) so it would be good to get > some data to see if this is worth doing. Makes sense. I can say a few things for sure though: * Frameworks do use temp selectors to simulate blocking (it would be pretty nice to have an API to replace this FWIW), so this could mean many hundreds or thousands of selectors lying around depending on how that is done; halving the FD requirements of that would be a good thing * Writing to an empty pipe apparently causes a page to be allocated for its buffer; on systems with bigger pages (64K for example, found on some ARM-based systems) that might amount to a lot of pointless overhead for what will usually amount to a single byte here or there * I believe that sockets and socketpairs have similar behavior * An eventfd can queue 2^64 events without any additional memory overhead (though this is unlikely to be a problem in practice outside of extreme environments) * Eventfd allows all pending events to be cleared in a single operation I don't think there is likely to be any substantial performance difference in terms of latency or event delivery based on what existing research I can discover. But it does seem like an easy way to shed some weight from a common process. Collecting data will likely entail some process whereby kernel memory usage and allocation can be monitored; I'm not quite sure how that could be approached yet though. I'll think about it some more. -- - DML From philippe.marschall at gmail.com Tue Mar 20 16:35:04 2018 From: philippe.marschall at gmail.com (Philippe Marschall) Date: Tue, 20 Mar 2018 17:35:04 +0100 Subject: Provide API to explicitly unmap MappedByteBuffer? In-Reply-To: <9c559d1c5d7b495e81f9e567e750c85e@sap.com> References: <9c559d1c5d7b495e81f9e567e750c85e@sap.com> Message-ID: Hi Any serious user I know calls unmap directly. Either through the Cleaner in Java 8 or through the new method on Unsafe in Java 9. ((sun.nio.ch.DirectBuffer) buffer).cleaner().clean() Unsafe.theUnsafe.invokeCleaner(byteBuffer) Cheers Philippe On Tue, Mar 20, 2018 at 11:39 AM, Langer, Christoph wrote: > Hi, > > > > I was just handling a customer issue where Java code tried to delete a file > on Windows which was mapped into memory vial FileChannel.map() before. It > took me a little while to understand the issue but obviously the problem was > that Windows still kept a delete lock on the file as long as the ByteBuffer > was still mapped. This behavior is also documented cleanly: > https://download.java.net/java/jdk10/docs/api/java/nio/MappedByteBuffer.html. > > > > So far so good, I just started wondering why there isn?t an API to force > unmapping (cleanup) on a MappedByteBuffer? The app that opens the mapping > simply reads a properties file at this place and I guess they should be able > to change this place to use standard io easily and without observable > performance problems. However, if they had a way to explicitly unmap the > MappedByteBuffer it would also help. > > > > Was this topic already discussed? Is there a hard reason why explicit > cleanup should not be offered? I guess people using MappedByteBuffers should > know anyway what they are doing? > > > > Thanks & Best regards > > Christoph > > From paul.sandoz at oracle.com Tue Mar 20 17:12:23 2018 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 20 Mar 2018 10:12:23 -0700 Subject: RFR 8199773 (bf) XXXBuffer:compareTo method is not working as expected Message-ID: <0C0EF738-CAC2-48A2-8651-CA7A37883EBF@oracle.com> Hi, Please review this simple fix for a silly mistake when calculating the result of comparing two buffers: http://cr.openjdk.java.net/~psandoz/jdk/JDK-8199773-buffer-compare-value/webrev/ Paul. -------------- next part -------------- An HTML attachment was scrubbed... URL: From forax at univ-mlv.fr Tue Mar 20 17:24:22 2018 From: forax at univ-mlv.fr (Remi Forax) Date: Tue, 20 Mar 2018 18:24:22 +0100 (CET) Subject: Provide API to explicitly unmap MappedByteBuffer? In-Reply-To: References: <9c559d1c5d7b495e81f9e567e750c85e@sap.com> Message-ID: <1784326385.1249430.1521566662054.JavaMail.zimbra@u-pem.fr> I've a class somewhere that guarantee that a clean() will never cause trouble if - all access are done by the same thread (otherwise you get an exception) - the wrapper around a MapperByteBuffer has to be stored in a static final to not lost performance. Technically, making the wrapper constant make the code faster than the usual ByteBuffer code but the thread checking slow the code down, so in the end, it's more or less as fast as using a ByteBuffer. R?mi ----- Mail original ----- > De: "Philippe Marschall" > ?: "Langer, Christoph" > Cc: "nio-dev" > Envoy?: Mardi 20 Mars 2018 17:35:04 > Objet: Re: Provide API to explicitly unmap MappedByteBuffer? > Hi > > Any serious user I know calls unmap directly. Either through the > Cleaner in Java 8 or through the new method on Unsafe in Java 9. > > ((sun.nio.ch.DirectBuffer) buffer).cleaner().clean() > > Unsafe.theUnsafe.invokeCleaner(byteBuffer) > > Cheers > Philippe > > On Tue, Mar 20, 2018 at 11:39 AM, Langer, Christoph > wrote: >> Hi, >> >> >> >> I was just handling a customer issue where Java code tried to delete a file >> on Windows which was mapped into memory vial FileChannel.map() before. It >> took me a little while to understand the issue but obviously the problem was >> that Windows still kept a delete lock on the file as long as the ByteBuffer >> was still mapped. This behavior is also documented cleanly: >> https://download.java.net/java/jdk10/docs/api/java/nio/MappedByteBuffer.html. >> >> >> >> So far so good, I just started wondering why there isn?t an API to force >> unmapping (cleanup) on a MappedByteBuffer? The app that opens the mapping >> simply reads a properties file at this place and I guess they should be able >> to change this place to use standard io easily and without observable >> performance problems. However, if they had a way to explicitly unmap the >> MappedByteBuffer it would also help. >> >> >> >> Was this topic already discussed? Is there a hard reason why explicit >> cleanup should not be offered? I guess people using MappedByteBuffers should >> know anyway what they are doing? >> >> >> >> Thanks & Best regards >> >> Christoph >> From Alan.Bateman at oracle.com Tue Mar 20 17:24:50 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 20 Mar 2018 17:24:50 +0000 Subject: RFR 8199773 (bf) XXXBuffer:compareTo method is not working as expected In-Reply-To: <0C0EF738-CAC2-48A2-8651-CA7A37883EBF@oracle.com> References: <0C0EF738-CAC2-48A2-8651-CA7A37883EBF@oracle.com> Message-ID: On 20/03/2018 17:12, Paul Sandoz wrote: > Hi, > > Please review this simple fix for a silly mistake when calculating the result of comparing two buffers: > > http://cr.openjdk.java.net/~psandoz/jdk/JDK-8199773-buffer-compare-value/webrev/ > Looks good to me, we probably should have caught this in the core review. -Alan From paul.sandoz at oracle.com Tue Mar 20 17:45:08 2018 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 20 Mar 2018 10:45:08 -0700 Subject: RFR 8199773 (bf) XXXBuffer:compareTo method is not working as expected In-Reply-To: References: <0C0EF738-CAC2-48A2-8651-CA7A37883EBF@oracle.com> Message-ID: <1CDE8E22-1311-4C05-9C4B-E9D86A7BE795@oracle.com> > On Mar 20, 2018, at 10:24 AM, Alan Bateman wrote: > > > > On 20/03/2018 17:12, Paul Sandoz wrote: >> Hi, >> >> Please review this simple fix for a silly mistake when calculating the result of comparing two buffers: >> >> http://cr.openjdk.java.net/~psandoz/jdk/JDK-8199773-buffer-compare-value/webrev/ >> > Looks good to me, we probably should have caught this in the core review. > Thanks. Hard to spot. I am glad the JCK tests found it. I should have realized the test needed to do pos/limit as well as pos/limit/slice (the latter caught bugs early in the development process). Paul. From uschindler at apache.org Tue Mar 20 18:08:47 2018 From: uschindler at apache.org (Uwe Schindler) Date: Tue, 20 Mar 2018 19:08:47 +0100 Subject: Provide API to explicitly unmap MappedByteBuffer? In-Reply-To: <0c9d2751-c0c9-437a-d346-3e30c095194c@oracle.com> References: <9c559d1c5d7b495e81f9e567e750c85e@sap.com> <0c9d2751-c0c9-437a-d346-3e30c095194c@oracle.com> Message-ID: <0bdb01d3c076$7f88ebf0$7e9ac3d0$@apache.org> Hi & thanks Alan, I am talking with Andrew every year about that (on FOSDEM). Yes, he has a plan how to solve this, but it requires changes in the memory model. But it is really a misdesign in the original NIO concept to let the Garbage Collector handle the unmapping (I know this was done early in Java?s days when Grabage collectors were much more predicatble and System.gc() did something?). The main reason for the problem is: The ByteBuffer is a small object on heap (like 30 bytes), but holding resources up to 2 Gigabytes. If a service (running for days) is opening a file and mmaps it, the small object on heap is quickly going to old generation and then is stays there forever ? the garbage collector has no idea that it is really heavy. No chance to get rid of it anymore, not even System.gc() helps in most configurations! Until this problem is solved (hopefully *before* sun.misc.Unsafe leaves completely from accessible APIs), the Apache Lucene team continues to use the Cleaner API (in Java 8) and the new Unsafe method (Java 9+). We can not 100% prevent SIGSEGV or SIGBUS if Lucene?s code is used in wrong way (mainly if you close an IndexReader while another thread is searching te index at same time), but the ?servers? on top of Lucene (like Apache Solr and Elasticsearch) take care of preventing that. Nevertheless, I am the person who was responsible for adding this new method to sun.misc.Unsafe before Java 9 ? I am so sorry about that! ?? But sun.misc.Unsafe seemed to be the best place to keep this ?unsafe? unmapping available to experts (guarded by security manager and setAccessible, of course). I think we should really spend Andrew Haley some time to work on the issue. I am open to review his code and share comments. We (Apache Lucene/Solr & Elasticsearch) have strong requirements about the unmapping and hope to get this solved soon! Uwe ----- Uwe Schindler uschindler at apache.org ASF Member, Apache Lucene PMC / Committer Bremen, Germany http://lucene.apache.org/ From: nio-dev On Behalf Of Alan Bateman Sent: Tuesday, March 20, 2018 2:59 PM To: Langer, Christoph ; nio-dev at openjdk.java.net Subject: Re: Provide API to explicitly unmap MappedByteBuffer? On 20/03/2018 10:39, Langer, Christoph wrote: Hi, I was just handling a customer issue where Java code tried to delete a file on Windows which was mapped into memory vial FileChannel.map() before. It took me a little while to understand the issue but obviously the problem was that Windows still kept a delete lock on the file as long as the ByteBuffer was still mapped. This behavior is also documented cleanly: https://download.java.net/java/jdk10/docs/api/java/nio/MappedByteBuffer.html. So far so good, I just started wondering why there isn?t an API to force unmapping (cleanup) on a MappedByteBuffer? The app that opens the mapping simply reads a properties file at this place and I guess they should be able to change this place to use standard io easily and without observable performance problems. However, if they had a way to explicitly unmap the MappedByteBuffer it would also help. Was this topic already discussed? Is there a hard reason why explicit cleanup should not be offered? I guess people using MappedByteBuffers should know anyway what they are doing? This has been discussed many times, it's a really hard problem that many smart people have spend time on. Andrew Haley may be able to say more about this. He has (or had) a proposal on this but it involved deep changes in the VM. I don't know he has prototyped it yet to see how feasible it is. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Mar 20 19:20:50 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 20 Mar 2018 12:20:50 -0700 Subject: 8198753: (dc) DatagramChannel throws unspecified exceptions In-Reply-To: <861d1e8d-6fb1-4c9c-bb28-bf9725c2cefa@oracle.com> References: <09E8887A-537A-4659-85B9-7EC4E9CF5367@oracle.com> <5D48A588-4E53-4683-B97A-00ADBFD6B069@oracle.com> <861d1e8d-6fb1-4c9c-bb28-bf9725c2cefa@oracle.com> Message-ID: <9528CDFA-05CE-427A-A098-D088D7F5A47D@oracle.com> On Mar 20, 2018, at 12:56 AM, Alan Bateman wrote: > On 19/03/2018 20:18, Brian Burkhalter wrote: >> >> I noticed that send() also needs to declare that it throws UnresolvedAddressException and UnsupportedAddressTypeException. > Good, I agree we should fix this too. > > The changes to the API + implementation looks right. One suggestion for the ordering of the exceptions is to move the AlreadyConnectedException up a bit so that ClosedChannelException and AsynchronousCloseException follow each other. So changed for both connect() and send(). >> java/nio/channels/DatagramChannel/SendToUnresolved.java already tests for the UnresolvedAddressException of send(). The other exceptions do not have test coverage. I modified ConnectedSend to test for these and to use TestNG: >> >> http://cr.openjdk.java.net/~bpb/8198753/webrev.01/ >> >> Perhaps I should pull the single test done in SendToUnresolved into ConnectedSend (with bug ID) and remove SendToUnresolved altogether? > ConnectedSend was created to test send with a connected datagram channel so it will need to be renamed and its summary changed if extended to test both connect and send. I think it would be easier to maintain if they were separate tests. Also as you are moving these to TestNG then the specific tests cases can be very small. DatagramChannel has a bind method so you don't need to use the socket adapter to bind. Test updates: 1. ConnectedSend reverted aside from conversion to TestNG. 2. ConnectExceptions and SendExceptions added to cover the newly documented exceptions. 3. SendToUnresolved removed as this is now tested SendExceptions which also includes the @bug ID from SendToUnresolved. http://cr.openjdk.java.net/~bpb/8198753/webrev.02/ Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Wed Mar 21 16:46:07 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 21 Mar 2018 09:46:07 -0700 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <1FE7FBB5-72B1-4103-B73D-9609E677A2E2@oracle.com> <69614880-3004-4B65-ADB3-D1ED807A88B5@oracle.com> <992c8053-60aa-d092-6a51-06d3c203c618@oracle.com> <715116280.749084.1520949869741.JavaMail.zimbra@u-pem.fr> <85b1f9ea-d365-77b6-07c6-d02556787dbe@Oracle.com> <86af2f85-ef16-34d8-fa22-fa2312ce6b53@oracle.com> <1d0a44a0-5822-577c-a791-0a68f68f51e6@oracle.com> <13FFE4A7-1AB9-43CE-864C-96BECF14E7B9@oracle.com> <84B6C513-69B9-4767-AC92-89912BAF9C7D@oracle.com> <9d08d830-5a90-1116-6f48-4d20017c9f09@oracle.com> Message-ID: <86D65F56-BC3E-4EC8-9D0C-E6EC879344F1@oracle.com> The CSR has been approved with the comment that the notes in Paths.java could be done as @implSpec tags. I can change these before checking this in if advised. Brian On Mar 15, 2018, at 10:24 AM, Brian Burkhalter wrote: > webrev.03 updated in place with the two changes below. I?ll update the CSR to match later today. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Wed Mar 21 16:50:55 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 21 Mar 2018 16:50:55 +0000 Subject: 8198753: (dc) DatagramChannel throws unspecified exceptions In-Reply-To: <9528CDFA-05CE-427A-A098-D088D7F5A47D@oracle.com> References: <09E8887A-537A-4659-85B9-7EC4E9CF5367@oracle.com> <5D48A588-4E53-4683-B97A-00ADBFD6B069@oracle.com> <861d1e8d-6fb1-4c9c-bb28-bf9725c2cefa@oracle.com> <9528CDFA-05CE-427A-A098-D088D7F5A47D@oracle.com> Message-ID: On 20/03/2018 19:20, Brian Burkhalter wrote: > > Test updates: > > 1. ConnectedSend reverted aside from conversion to TestNG. > 2. ConnectExceptions and SendExceptions added to cover the newly > documented exceptions. > 3. SendToUnresolved removed as this is now tested SendExceptions which > also includes the @bug ID from SendToUnresolved. > > http://cr.openjdk.java.net/~bpb/8198753/webrev.02/ > > The tests looks okay although personally I would use @Test(expectedExceptions = ..) rather than expectThrows to avoid all the Runnables. A variant of what you have is for the run method to throw IOException to avoid the the try-catch in each test. -Alan From brian.burkhalter at oracle.com Wed Mar 21 17:00:37 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 21 Mar 2018 10:00:37 -0700 Subject: 8198753: (dc) DatagramChannel throws unspecified exceptions In-Reply-To: References: <09E8887A-537A-4659-85B9-7EC4E9CF5367@oracle.com> <5D48A588-4E53-4683-B97A-00ADBFD6B069@oracle.com> <861d1e8d-6fb1-4c9c-bb28-bf9725c2cefa@oracle.com> <9528CDFA-05CE-427A-A098-D088D7F5A47D@oracle.com> Message-ID: On Mar 21, 2018, at 9:50 AM, Alan Bateman wrote: > On 20/03/2018 19:20, Brian Burkhalter wrote: >> >> Test updates: >> >> 1. ConnectedSend reverted aside from conversion to TestNG. >> 2. ConnectExceptions and SendExceptions added to cover the newly documented exceptions. >> 3. SendToUnresolved removed as this is now tested SendExceptions which also includes the @bug ID from SendToUnresolved. >> >> http://cr.openjdk.java.net/~bpb/8198753/webrev.02/ >> > The tests looks okay although personally I would use @Test(expectedExceptions = ..) rather than expectThrows to avoid all the Runnables. A variant of what you have is for the run method to throw IOException to avoid the the try-catch in each test. I can change it. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Roger.Riggs at Oracle.com Wed Mar 21 19:17:38 2018 From: Roger.Riggs at Oracle.com (Roger Riggs) Date: Wed, 21 Mar 2018 15:17:38 -0400 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: <86D65F56-BC3E-4EC8-9D0C-E6EC879344F1@oracle.com> References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <69614880-3004-4B65-ADB3-D1ED807A88B5@oracle.com> <992c8053-60aa-d092-6a51-06d3c203c618@oracle.com> <715116280.749084.1520949869741.JavaMail.zimbra@u-pem.fr> <85b1f9ea-d365-77b6-07c6-d02556787dbe@Oracle.com> <86af2f85-ef16-34d8-fa22-fa2312ce6b53@oracle.com> <1d0a44a0-5822-577c-a791-0a68f68f51e6@oracle.com> <13FFE4A7-1AB9-43CE-864C-96BECF14E7B9@oracle.com> <84B6C513-69B9-4767-AC92-89912BAF9C7D@oracle.com> <9d08d830-5a90-1116-6f48-4d20017c9f09@oracle.com> <86D65F56-BC3E-4EC8-9D0C-E6EC879344F1@oracle.com> Message-ID: <81d13e7d-e2a3-02a9-ba92-422aee76f956@Oracle.com> Hi Brian, That would be fine (is fine). Thanks, Roger On 3/21/2018 12:46 PM, Brian Burkhalter wrote: > The CSR has been approved with the comment that the notes in > Paths.java could be done as @implSpec tags. I can change these before > checking this in if advised. > > Brian > > On Mar 15, 2018, at 10:24 AM, Brian Burkhalter > > wrote: > >> webrev.03 updated in place with the two changes below. I?ll update >> the CSR to match later today. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Wed Mar 21 19:19:28 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 21 Mar 2018 19:19:28 +0000 Subject: RFR 8194746: (fs) Add equivalents of Paths.get to Path interface In-Reply-To: <86D65F56-BC3E-4EC8-9D0C-E6EC879344F1@oracle.com> References: <741C9B55-C963-4CFA-BB4C-9E0A7934A788@oracle.com> <69614880-3004-4B65-ADB3-D1ED807A88B5@oracle.com> <992c8053-60aa-d092-6a51-06d3c203c618@oracle.com> <715116280.749084.1520949869741.JavaMail.zimbra@u-pem.fr> <85b1f9ea-d365-77b6-07c6-d02556787dbe@Oracle.com> <86af2f85-ef16-34d8-fa22-fa2312ce6b53@oracle.com> <1d0a44a0-5822-577c-a791-0a68f68f51e6@oracle.com> <13FFE4A7-1AB9-43CE-864C-96BECF14E7B9@oracle.com> <84B6C513-69B9-4767-AC92-89912BAF9C7D@oracle.com> <9d08d830-5a90-1116-6f48-4d20017c9f09@oracle.com> <86D65F56-BC3E-4EC8-9D0C-E6EC879344F1@oracle.com> Message-ID: On 21/03/2018 16:46, Brian Burkhalter wrote: > The CSR has been approved with the comment that the notes in > Paths.java could be done as @implSpec tags. I can change these before > checking this in if advised. Okay with me too. -Alan From brian.burkhalter at oracle.com Wed Mar 21 19:21:27 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 21 Mar 2018 12:21:27 -0700 Subject: 8198753: (dc) DatagramChannel throws unspecified exceptions In-Reply-To: References: <09E8887A-537A-4659-85B9-7EC4E9CF5367@oracle.com> <5D48A588-4E53-4683-B97A-00ADBFD6B069@oracle.com> <861d1e8d-6fb1-4c9c-bb28-bf9725c2cefa@oracle.com> <9528CDFA-05CE-427A-A098-D088D7F5A47D@oracle.com> Message-ID: <5D5EFA0B-9D9E-4150-B009-5906B78F5267@oracle.com> On Mar 21, 2018, at 10:00 AM, Brian Burkhalter wrote: > On Mar 21, 2018, at 9:50 AM, Alan Bateman wrote: > >> On 20/03/2018 19:20, Brian Burkhalter wrote: >>> >>> Test updates: >>> >>> 1. ConnectedSend reverted aside from conversion to TestNG. >>> 2. ConnectExceptions and SendExceptions added to cover the newly documented exceptions. >>> 3. SendToUnresolved removed as this is now tested SendExceptions which also includes the @bug ID from SendToUnresolved. >>> >>> http://cr.openjdk.java.net/~bpb/8198753/webrev.02/ >>> >> The tests looks okay although personally I would use @Test(expectedExceptions = ..) rather than expectThrows to avoid all the Runnables. A variant of what you have is for the run method to throw IOException to avoid the the try-catch in each test. > > I can change it. Updated tests to use expectedExceptions = ?. http://cr.openjdk.java.net/~bpb/8198753/webrev.03/ Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Wed Mar 21 21:06:25 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 21 Mar 2018 21:06:25 +0000 Subject: 8198753: (dc) DatagramChannel throws unspecified exceptions In-Reply-To: <5D5EFA0B-9D9E-4150-B009-5906B78F5267@oracle.com> References: <09E8887A-537A-4659-85B9-7EC4E9CF5367@oracle.com> <5D48A588-4E53-4683-B97A-00ADBFD6B069@oracle.com> <861d1e8d-6fb1-4c9c-bb28-bf9725c2cefa@oracle.com> <9528CDFA-05CE-427A-A098-D088D7F5A47D@oracle.com> <5D5EFA0B-9D9E-4150-B009-5906B78F5267@oracle.com> Message-ID: On 21/03/2018 19:21, Brian Burkhalter wrote: > Updated tests to use expectedExceptions = ?. > > http://cr.openjdk.java.net/~bpb/8198753/webrev.03/ > > This is much neater, thumbs up from me. -Alan From brian.burkhalter at oracle.com Thu Mar 22 00:29:19 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 21 Mar 2018 17:29:19 -0700 Subject: 8199791: (se) More Selector cleanup In-Reply-To: References: <5f7c43fb-c458-c742-c443-374f57d8ad46@oracle.com> Message-ID: I concur about nanoTime() and eventfd seems like a good idea for the reasons stated in another message but these could be done separately at a later time. Just a few comments: EPollPort 33: Could this static import be removed? 92: Could the socket pair creation be moved to just after super()? If so then the try-catch could be eliminated and there would be no need for the current lines 96-97. EPollSelectorImpl 87: Similar comment as for EPollPort:92 above. KQueuePort 33: Again, could this static import be removed? 90: Similar comment as for EPollPort:92 above. KQueueSelectorImpl 88: Similar comment as for EPollSelectorImpl:87 above. Could DevPollArrayWrapper be dispensed with similarly to {EPoll,KQueue}ArrayWrapper? I don?t see any logical problems with this so I think it can go ahead. Brian On Mar 20, 2018, at 7:44 AM, David Lloyd wrote: > Looks good to me (non-reviewer). I still believe the timeout calcs > should use System.nanoTime() instead of currentTimeMillis(), and I'd > like to see Linux using eventfd instead of a pipe for selector > wakeups... but those are subjects for future discussion/patches, > probably. > > On Tue, Mar 20, 2018 at 8:47 AM, Alan Bateman wrote: >> This is another batch of cleanup to the Selector implementations. >> >> A significant part of the cleanup is changing the epoll and kqueue based >> Selector implementations to use the EPoll and KQueue support classes. This >> eliminate their wrapper classes and several native methods. The result is >> much smaller implementations. The queuing of changes to the interest set has >> also been re-worked to avoid needing cancellation needing to kill all >> pending updates. A few of the Selector implementations had a native method >> to write a byte to one end of the pipe used for the wakeup mechanism. This >> has been changed to use IOUtil.write1 instead. Interrupting timed polls (as >> in EINTR, not Thread interrupt) wasn't handled correctly in some of the >> implementations so that has been re-worked. >> >> The webrev with the changes is here: >> http://cr.openjdk.java.net/~alanb/8199791/webrev/index.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Mar 22 13:41:28 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 22 Mar 2018 13:41:28 +0000 Subject: 8199791: (se) More Selector cleanup In-Reply-To: References: <5f7c43fb-c458-c742-c443-374f57d8ad46@oracle.com> Message-ID: <22609847-32aa-a75b-a457-c3fc4ef4aa8a@oracle.com> On 22/03/2018 00:29, Brian Burkhalter wrote: > I concur about nanoTime() and eventfd seems like a good idea for the > reasons stated in another message but these could be done separately > at a later time. I'm happy to change that in this patch. > > Just a few comments: > > EPollPort > 33: Could this static import be removed? Okay.. > 92: Could the socket pair creation be moved to just after super()? If > so then the try-catch could be eliminated and there would be no need > for the current lines 96-97. That wouldn't help as a try/catch would still be needed to close both ends of the socket pair in the event that creating the kqueue or epoll instance fails. > > : > > Could DevPollArrayWrapper be dispensed with similarly to > {EPoll,KQueue}ArrayWrapper? That's for another patch. I have a lot of changes in this area so I'm trying to keep the patches down to a reasonable size to ensure that the changes are over a number of builds, and also to make the changes a bit easier to review. I've updated the patch in place and hopefully the changes looks okay now. -Alan From brian.burkhalter at oracle.com Thu Mar 22 15:09:24 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 22 Mar 2018 08:09:24 -0700 Subject: 8199791: (se) More Selector cleanup In-Reply-To: <22609847-32aa-a75b-a457-c3fc4ef4aa8a@oracle.com> References: <5f7c43fb-c458-c742-c443-374f57d8ad46@oracle.com> <22609847-32aa-a75b-a457-c3fc4ef4aa8a@oracle.com> Message-ID: <4B35C39F-1990-4774-B39F-6F4BBD92EE06@oracle.com> On Mar 22, 2018, at 6:41 AM, Alan Bateman wrote: >> 92: Could the socket pair creation be moved to just after super()? If so then the try-catch could be eliminated and there would be no need for the current lines 96-97. > That wouldn't help as a try/catch would still be needed to close both ends of the socket pair in the event that creating the kqueue or epoll instance fails. I see what you are saying. It?s OK then. >> Could DevPollArrayWrapper be dispensed with similarly to {EPoll,KQueue}ArrayWrapper? > That's for another patch. I have a lot of changes in this area so I'm trying to keep the patches down to a reasonable size to ensure that the changes are over a number of builds, and also to make the changes a bit easier to review. Understood. > I've updated the patch in place and hopefully the changes looks okay now. This looks fine to me now. Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Thu Mar 22 17:19:42 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 22 Mar 2018 10:19:42 -0700 Subject: 8198753: (dc) DatagramChannel throws unspecified exceptions In-Reply-To: References: <09E8887A-537A-4659-85B9-7EC4E9CF5367@oracle.com> <5D48A588-4E53-4683-B97A-00ADBFD6B069@oracle.com> <861d1e8d-6fb1-4c9c-bb28-bf9725c2cefa@oracle.com> <9528CDFA-05CE-427A-A098-D088D7F5A47D@oracle.com> <5D5EFA0B-9D9E-4150-B009-5906B78F5267@oracle.com> Message-ID: <0B302C81-98F5-4FAB-90A2-4DA225357134@oracle.com> On Mar 21, 2018, at 2:06 PM, Alan Bateman wrote: > On 21/03/2018 19:21, Brian Burkhalter wrote: >> Updated tests to use expectedExceptions = ?. >> >> http://cr.openjdk.java.net/~bpb/8198753/webrev.03/ >> > This is much neater, thumbs up from me. I caught a failure in Connect.java due to the IAE to ACE change. Webrev.03 is updated in place; the diff versus the previous version of the webrev is included below. Brian 22 10:14:22.000000000 -0700 > +++ new/test/jdk/java/nio/channels/DatagramChannel/Connect.java 2018-03-22 10:14:22.000000000 -0700 > @@ -1,5 +1,5 @@ > /* > - * Copyright (c) 2001, 2013, Oracle and/or its affiliates. All rights reserved. > + * Copyright (c) 2001, 2018, Oracle and/or its affiliates. All rights reserved. > * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. > * > * This code is free software; you can redistribute it and/or modify it > @@ -102,7 +102,7 @@ > try { > dc.send(bb, bogus); > throw new RuntimeException("Allowed bogus send while connected"); > - } catch (IllegalArgumentException iae) { > + } catch (AlreadyConnectedException ace) { > // Correct behavior > } -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Mar 22 17:41:36 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 22 Mar 2018 17:41:36 +0000 Subject: 8198753: (dc) DatagramChannel throws unspecified exceptions In-Reply-To: <0B302C81-98F5-4FAB-90A2-4DA225357134@oracle.com> References: <09E8887A-537A-4659-85B9-7EC4E9CF5367@oracle.com> <5D48A588-4E53-4683-B97A-00ADBFD6B069@oracle.com> <861d1e8d-6fb1-4c9c-bb28-bf9725c2cefa@oracle.com> <9528CDFA-05CE-427A-A098-D088D7F5A47D@oracle.com> <5D5EFA0B-9D9E-4150-B009-5906B78F5267@oracle.com> <0B302C81-98F5-4FAB-90A2-4DA225357134@oracle.com> Message-ID: <91bfea06-4142-cde2-c4f3-1e845b1a104b@oracle.com> Looks good. On 22/03/2018 17:19, Brian Burkhalter wrote: > On Mar 21, 2018, at 2:06 PM, Alan Bateman > wrote: > >> On 21/03/2018 19:21, Brian Burkhalter wrote: >>> Updated tests to use expectedExceptions = ?. >>> >>> http://cr.openjdk.java.net/~bpb/8198753/webrev.03/ >>> >>> >> This is much neater, thumbs up from me. > > I caught a failure in Connect.java due to the IAE to ACE change. > Webrev.03 is updated in place; the diff versus the previous version of > the webrev is included below. > > Brian > > 22 10:14:22.000000000 -0700 > > +++ > new/test/jdk/java/nio/channels/DatagramChannel/Connect.java2018-03-22 > 10:14:22.000000000 -0700 > > @@ -1,5 +1,5 @@ > > /* > > - * Copyright (c) 2001, 2013, Oracle and/or its affiliates. All > rights reserved. > > + * Copyright (c) 2001, 2018, Oracle and/or its affiliates. All > rights reserved. > > ? * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. > > ? * > > ? * This code is free software; you can redistribute it and/or modify it > > @@ -102,7 +102,7 @@ > > ? ? ? ? ? ? ? ? try { > > ? ? ? ? ? ? ? ? ? ? dc.send(bb, bogus); > > ? ? ? ? ? ? ? ? ? ? throw new RuntimeException("Allowed bogus send > while connected"); > > -? ? ? ? ? ? ? ? } catch (IllegalArgumentException iae) { > > +? ? ? ? ? ? ? ? } catch (AlreadyConnectedException ace) { > > ? ? ? ? ? ? ? ? ? ? // Correct behavior > > ? ? ? ? ? ? ? ? } > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Fri Mar 23 15:59:10 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 23 Mar 2018 15:59:10 +0000 Subject: 8200179: (se) More Selector cleanup Message-ID: <4f1a83be-8853-c43f-c1bf-fbc9b3edc251@oracle.com> This is another batch of cleanups to the Selector implementations. The poll based Selector has been in disrepair for some time. JDK-8142872 attempted to exclude it from the build but it wasn't complete in that the poll wrapper class and the native methods are are still compiled. There is also some legacy debt from JDK 1.4 where part of the? implementation was compiled on Windows even though it was never usable there. Finally, several source files (both native and java) have been importing the wrapper class or its header file. The main change to the implementation is that is that changes to the interest ops are queued so it avoids all the issues with threads changing the poll array when not correctly synchronized with the selector. All tests are passing when configured to use this Selector. Once Magnus gets JDK-8200178 into jdk/jdk then we can look at excluding it on platforms where it is not needed. This patch also refactors the I/O port based Selector used on Solaris. The queuing of changes to the interest ops, and post poll re-queuing to re-associate file descriptors, is much simpler now. The webrev with the changes is here: ?? http://cr.openjdk.java.net/~alanb/8200179/webrev/index.html -Alan From david.lloyd at redhat.com Fri Mar 23 19:51:47 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Fri, 23 Mar 2018 14:51:47 -0500 Subject: 8200179: (se) More Selector cleanup In-Reply-To: <4f1a83be-8853-c43f-c1bf-fbc9b3edc251@oracle.com> References: <4f1a83be-8853-c43f-c1bf-fbc9b3edc251@oracle.com> Message-ID: I have a question, mostly for my own reference: + private static final short SIZE_POLLFD = 8; + private static final short FD_OFFSET = 0; + private static final short EVENT_OFFSET = 4; + private static final short REVENT_OFFSET = 6; Is it generally considered OK to assume that the C side has 4-byte ints and 2-byte shorts? It seems like that might be unsafe, though I don't know of any specific cases where it actually _is_. On Fri, Mar 23, 2018 at 10:59 AM, Alan Bateman wrote: > This is another batch of cleanups to the Selector implementations. > > The poll based Selector has been in disrepair for some time. JDK-8142872 > attempted to exclude it from the build but it wasn't complete in that the > poll wrapper class and the native methods are are still compiled. There is > also some legacy debt from JDK 1.4 where part of the implementation was > compiled on Windows even though it was never usable there. Finally, several > source files (both native and java) have been importing the wrapper class or > its header file. The main change to the implementation is that is that > changes to the interest ops are queued so it avoids all the issues with > threads changing the poll array when not correctly synchronized with the > selector. All tests are passing when configured to use this Selector. Once > Magnus gets JDK-8200178 into jdk/jdk then we can look at excluding it on > platforms where it is not needed. > > This patch also refactors the I/O port based Selector used on Solaris. The > queuing of changes to the interest ops, and post poll re-queuing to > re-associate file descriptors, is much simpler now. > > The webrev with the changes is here: > http://cr.openjdk.java.net/~alanb/8200179/webrev/index.html > > -Alan -- - DML From Alan.Bateman at oracle.com Fri Mar 23 20:10:13 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 23 Mar 2018 20:10:13 +0000 Subject: 8200179: (se) More Selector cleanup In-Reply-To: References: <4f1a83be-8853-c43f-c1bf-fbc9b3edc251@oracle.com> Message-ID: On 23/03/2018 19:51, David Lloyd wrote: > I have a question, mostly for my own reference: > > + private static final short SIZE_POLLFD = 8; > + private static final short FD_OFFSET = 0; > + private static final short EVENT_OFFSET = 4; > + private static final short REVENT_OFFSET = 6; > > Is it generally considered OK to assume that the C side has 4-byte > ints and 2-byte shorts? It seems like that might be unsafe, though I > don't know of any specific cases where it actually _is_. > Not usually, it would be better to obtain the layout from the platform as done in the epoll and kqueue implementations. The poll based Selector has got away with the above since JDK 1.4 and it wasn't on my list to change as part of this update. So if the poll Selector lives on then I agree this should be cleaned up too. -Alan From brian.burkhalter at oracle.com Fri Mar 23 23:02:26 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 23 Mar 2018 16:02:26 -0700 Subject: 8200179: (se) More Selector cleanup In-Reply-To: <4f1a83be-8853-c43f-c1bf-fbc9b3edc251@oracle.com> References: <4f1a83be-8853-c43f-c1bf-fbc9b3edc251@oracle.com> Message-ID: <2216AF6E-5394-412F-BA62-C24605E393D2@oracle.com> This looks OK to me. Brian On Mar 23, 2018, at 8:59 AM, Alan Bateman wrote: > This is another batch of cleanups to the Selector implementations. > > [?] > > The webrev with the changes is here: > http://cr.openjdk.java.net/~alanb/8200179/webrev/index.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Mon Mar 26 18:39:34 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 26 Mar 2018 19:39:34 +0100 Subject: 8200256: java/nio/channels/AsynchronousChannelGroup/Basic.java fails intermittently Message-ID: <2f7ef213-ab3f-02b8-3568-543ebb1cd133@oracle.com> This test has been failing recently on macOS and Linux with "Group should have terminated" when testing the shutdown of a channel group. Chris and others working on the HTTP client API have had several sightings during their testing. The issue turns out to be a regression in channel group shutdown implementation. When woken up, a thread should read a single byte from the wakeup pipe but recent changes means it may consume more bytes, thus leaving other threads waiting for an event. Easy fix to change this code back to reading one byte. With this fix, the test runs 1M times for me without an issue. Without the fix then it usually fails in 50000 runs or less. The webrev with the change is here: ?? http://cr.openjdk.java.net/~alanb/8200256/webrev/index.html -Alan From chris.hegarty at oracle.com Tue Mar 27 07:18:00 2018 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 27 Mar 2018 08:18:00 +0100 Subject: 8200256: java/nio/channels/AsynchronousChannelGroup/Basic.java fails intermittently In-Reply-To: <2f7ef213-ab3f-02b8-3568-543ebb1cd133@oracle.com> References: <2f7ef213-ab3f-02b8-3568-543ebb1cd133@oracle.com> Message-ID: <4E291FF1-0734-449B-986D-1C3CCC51B5EE@oracle.com> > On 26 Mar 2018, at 19:39, Alan Bateman wrote: > > This test has been failing recently on macOS and Linux with "Group should have terminated" when testing the shutdown of a channel group. Chris and others working on the HTTP client API have had several sightings during their testing. > > The issue turns out to be a regression in channel group shutdown implementation. When woken up, a thread should read a single byte from the wakeup pipe but recent changes means it may consume more bytes, thus leaving other threads waiting for an event. Easy fix to change this code back to reading one byte. With this fix, the test runs 1M times for me without an issue. Without the fix then it usually fails in 50000 runs or less. > > The webrev with the change is here: > http://cr.openjdk.java.net/~alanb/8200256/webrev/index.html I think this looks fine ( at least that is what the code used to do before 8199791 ). The ( now safe ) racy interaction between wakeupCount and the use of a ?single? byte, during shutdown, is a little subtle, maybe worth a comment for future maintainers. -Chris. From Alan.Bateman at oracle.com Tue Mar 27 08:00:43 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 27 Mar 2018 09:00:43 +0100 Subject: 8200257: (se) More Selector cleanup Message-ID: <377e3694-0703-26bd-01a6-689ec36a4314@oracle.com> This is another round of improvements to the Selector implementations. The implReg and implDereg implementations are similar in the 6 implementations. This code can be hoisted up to SelectorImpl to avoid the duplication. This change means the selector key set is updated in one place now so it's easier to understand changes in this area. Several of the implementations use a bit set to track if a file descriptor is registered or not. This is replaced by adding a field to the selection key for the registered ops (similar to how the poll based Selectors store the index into the poll array in the key). Finally, this update brings the Windows implementation in line with the other implementations so that the poll array isn't change directly by threads registering or changing interest ops. The webrev with the proposed changes is here: ? http://cr.openjdk.java.net/~alanb/8200257/webrev/index.html All the tests are passing with these changes on all platforms. -Alan From Alan.Bateman at oracle.com Tue Mar 27 16:12:33 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 27 Mar 2018 17:12:33 +0100 Subject: 8200256: java/nio/channels/AsynchronousChannelGroup/Basic.java fails intermittently In-Reply-To: <4E291FF1-0734-449B-986D-1C3CCC51B5EE@oracle.com> References: <2f7ef213-ab3f-02b8-3568-543ebb1cd133@oracle.com> <4E291FF1-0734-449B-986D-1C3CCC51B5EE@oracle.com> Message-ID: <48673ebc-653c-7d12-19da-95d58550667c@oracle.com> On 27/03/2018 08:18, Chris Hegarty wrote: > : > I think this looks fine ( at least that is what the code used to do before 8199791 ). The ( now safe ) racy interaction between wakeupCount and the use of a ?single? byte, during shutdown, is a little subtle, maybe worth a comment for future maintainers. > Thanks, I'll try to improve the comment before pushing this. I don't think this are has changed for several areas and it's not part if the current efforts to sort out the Selector issues. -Alan From chris.hegarty at oracle.com Tue Mar 27 16:30:37 2018 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 27 Mar 2018 17:30:37 +0100 Subject: 8200256: java/nio/channels/AsynchronousChannelGroup/Basic.java fails intermittently In-Reply-To: <48673ebc-653c-7d12-19da-95d58550667c@oracle.com> References: <2f7ef213-ab3f-02b8-3568-543ebb1cd133@oracle.com> <4E291FF1-0734-449B-986D-1C3CCC51B5EE@oracle.com> <48673ebc-653c-7d12-19da-95d58550667c@oracle.com> Message-ID: > On 27 Mar 2018, at 17:12, Alan Bateman wrote: > >> On 27/03/2018 08:18, Chris Hegarty wrote: >> : >> I think this looks fine ( at least that is what the code used to do before 8199791 ). The ( now safe ) racy interaction between wakeupCount and the use of a ?single? byte, during shutdown, is a little subtle, maybe worth a comment for future maintainers. >> > Thanks, I'll try to improve the comment before pushing this. I don't think this are has changed for several areas and it's not part if the current efforts to sort out the Selector issues. Sorry, it?s not a comment on your changes, but on the existing code, that may have helped me when trying to review this, or avoided the regression altogether. -Chris From mike.skells1 at gmail.com Wed Mar 28 07:25:58 2018 From: mike.skells1 at gmail.com (Michael Skells) Date: Wed, 28 Mar 2018 07:25:58 +0000 Subject: trivial patch to reduce allocations in nio open* methods In-Reply-To: References: <158BFF8B-586B-4C1C-8DBE-5C5DD3E8D66B@oracle.com> <2ba60a55-baf3-8542-dac6-70c33e3f5f25@oracle.com> <0803a885-5938-6549-262e-40064d73c513@oracle.com> <5aa714cf.c30c1c0a.fbbdc.b3fb@mx.google.com> Message-ID: Hi All, what is the way forward with this patch ( if any) Regards Mike On Tue, 13 Mar 2018 at 20:57 Michael Skells wrote: > Hi Alan, > This is using the default file system, on windows and unix > > This was spotted with the development of the Scala compiler, but it would > be unaffected by this patch as described below > > Some background - > The Scala compiler tends to write a lot more classes for a given source > file than Java, tens or even hundreds from a single source. I was working > or parallelising the code generation and optimisation of the back end of > the scala compiler and looking for object allocations that could be trimmed > amongst others things, mostly in the scala compiles and its runtime > > We have some projects that will generate 100K class files for a compile, > and several hundred projects within the same codeball. When compiling > downstream projects within the IDEs we refer to classes from upstream > projects and there are lots of files read and parsed > > We have hundreds of developers who are recompiling the codeball on a > regular basis, usually partially either in an IDE or CI environment, some > on Windows and some on Unix > > The work was focussed primarily on providing business benefit in our > enterprise, but releasing what we could back to the open source compiler > > Performance > This additional object creation is not a dominating, or even large impact > on the compiler performance, but all things add up to the end performance > and currently opening a file for write creates I think 5 objects ( the > HashMap, backing Object[], and 3 Map.Entry) and wastes a small amount of > CPU, and it seemed easy to fix for the common case :-). The production code > doesn't use this call path but we use CREATE_NEW explicitly as explained > later > > The file opening & closing (particularly in windows, with an enterprise > software stack) is by far the dominant item, by orders of magnitude. We > can't currently write to Zip/Jar files because of the tool chain, and the > compiler is generating a stream of (FileName, byte[]), and there is no bulk > or stream file API that I am aware of > > The benchmarking I did was on Java 8 (about 6 months ago), on 64 bit > windows 10 and 7, desktops from 4 to 12 cores, all 64 bit server JVM, with > local SSD > > I did benchmark the use of different APIs (sync vs async NIO vs old file > IO) and how many writes concurrently to get the best throughput, on SSD > desktops, and the best seems to be sync NIO and over-committing the CPU > with 2 threads per hyperthreaded core (this seemed to reduce the OS CPU > overheads). Strangely async NIO didn't seem to benefit so much from this > over-commitment but I didn't dig into this. I did post to this group some > unexpected degradation with REFS filesystem "parallel IO anomaly with > windows REFS formatted drive" but as Bernt noted that filesystem was > removed from desktops > > Incidentally, but slightly related the biggest unexpected performance > boost (on windows) was not using CREATE and TRUNCATE together but using > CREATE_NEW and coping with the unusual case of pre-existing files as an > exception (40% reduction in OS CPU!). I only found this by accidentally > deleting the profiling filter in JProbe for java.*, sun.* etc classes one > day :-) and profiling all the way to nio code. This code is around usage of > "truncateAfterOpen" in WindowsChannelFactory > This means that the default options cause 2 OS calls to open the file on > windows (one to create and the other to truncate), but I could not fix the > default case in a patch and I am not sure it is fixable :-( > > I spotted this issue/optimisation in the Scala compiler development, but > it would be unaffected by this patch because we use specific flags, but I > thought that it would be useful to others. > > > Regards > Mike > > On Tue, 13 Mar 2018 at 14:25 Alan Bateman wrote: > >> On 13/03/2018 00:01, Michael Skells wrote: >> >> Hi, >> >> I am not sure what the status is for this patch >> >> >> >> Are you waiting for more input, or some update from me? >> >> Is this a viable patch? >> >> Is it just in the queue of minor noise to consider later? >> >> Do you have any data on how it helps the scala compiler? Also are you >> seeing this with the default file system or is something else (the zip file >> system maybe)? I ask as I would have expected the I/O cost and opening the >> file to dominant rather than processing the arguments that specify how the >> file is opened. >> >> BTW: I'm not opposed to changing anything here, instead I think we need a >> bit more of the story. >> >> >> -Alan >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Wed Mar 28 14:32:05 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 28 Mar 2018 15:32:05 +0100 Subject: Callback Based Selectors In-Reply-To: References: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> <1365073714.1162106.1521037015119.JavaMail.zimbra@u-pem.fr> <7185f4eb-618c-1e11-0267-43e0be448995@oracle.com> Message-ID: <188928e3-4430-4d8f-a0b5-4a2aae545f2b@oracle.com> On 20/03/2018 01:57, David Lloyd wrote: > : > Thinking about this some more, I am coming to believe that this is the > crux of the locking problems we routinely encounter. If we can get to > a point where we can, in specification, state that select ops (or > maybe even just the new select op(s)) will only hold the selector lock > while blocking, then we are in an ideal situation as far as frameworks > go because the aforementioned strategy (for channel registration which > occurs concurrently to a selection operation) of "register and wakeup" > will always work. I agree that registering when there is selection operation in progress is a pain, it forces everyone to follow the idiom to synchronize on a gate object. I looked into the API/spec implications of changing this and it's not too bad. The recent changes to the Selector implementations means it has little impact as all the implementations in the JDK now handle new registrations and changes to interest ops as part of the selection operation. In terms of compatibility then the only concern is code that synchronizes on the key set or selected-key to prevent keys being added by concurrent registrations or selection operations (e.g. code processes and removes all keys from the selected-key set and then asserts that the selected-key is empty). If this is an issue then registration could continue to synchronize, and the selection operation could limit is synchronization to when it needs to add to the selected-key set. The other aspect to compatibility is other SelectorProvider (and Selector) implementations although I've never come across complete implementations. Here's prototype spec changes to see how it might look: http://cr.openjdk.java.net/~alanb/concurrent-selector-register/webrev/ New registrations would be register+wakeup, changes to interest ops can be interestOps+wakeup. I think it will need analysis of existing usages of the API to fully understand the compatibility impact of changing anything here. Also relaxing the locking with help with the select(Consumer) API that Richard has proposed. -Alan From david.lloyd at redhat.com Wed Mar 28 14:45:04 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Wed, 28 Mar 2018 09:45:04 -0500 Subject: Callback Based Selectors In-Reply-To: <188928e3-4430-4d8f-a0b5-4a2aae545f2b@oracle.com> References: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> <1365073714.1162106.1521037015119.JavaMail.zimbra@u-pem.fr> <7185f4eb-618c-1e11-0267-43e0be448995@oracle.com> <188928e3-4430-4d8f-a0b5-4a2aae545f2b@oracle.com> Message-ID: On Wed, Mar 28, 2018 at 9:32 AM, Alan Bateman wrote: > I looked into the API/spec implications of changing this and it's not too > bad. The recent changes to the Selector implementations means it has little > impact as all the implementations in the JDK now handle new registrations > and changes to interest ops as part of the selection operation. > > In terms of compatibility then the only concern is code that synchronizes on > the key set or selected-key to prevent keys being added by concurrent > registrations or selection operations (e.g. code processes and removes all > keys from the selected-key set and then asserts that the selected-key is > empty). If this is an issue then registration could continue to synchronize, > and the selection operation could limit is synchronization to when it needs > to add to the selected-key set. The other aspect to compatibility is other > SelectorProvider (and Selector) implementations although I've never come > across complete implementations. > > Here's prototype spec changes to see how it might look: > http://cr.openjdk.java.net/~alanb/concurrent-selector-register/webrev/ > > New registrations would be register+wakeup, changes to interest ops can be > interestOps+wakeup. Great! This seems much cleaner to me; I hope it works out. I wonder, since we're moving these APIs to be more concurrency-friendly, if you could consider an additional small enhancement to add a couple atomic operations to SelectionKey for interest ops like getAndSetBits/getAndClearBits? They would only have to be atomic with respect to each other, so they could be emulated with locks for compatibility. At present, coordinating adding interest ops across threads means adding a lock, which is super heavyweight considering that interest ops are backed a volatile int field in most if not all cases. I could work up a patch if you want; since you've been so busy in the NIO area lately I've been hesitant to try anything in that area, at least until things settle down a little. -- - DML From Alan.Bateman at oracle.com Wed Mar 28 17:09:31 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 28 Mar 2018 18:09:31 +0100 Subject: Callback Based Selectors In-Reply-To: References: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> <1365073714.1162106.1521037015119.JavaMail.zimbra@u-pem.fr> <7185f4eb-618c-1e11-0267-43e0be448995@oracle.com> <188928e3-4430-4d8f-a0b5-4a2aae545f2b@oracle.com> Message-ID: <622f0553-43c3-3b02-0f18-b9781fbd96c1@oracle.com> On 28/03/2018 15:45, David Lloyd wrote: > : > I wonder, since we're moving these APIs to be more > concurrency-friendly, if you could consider an additional small > enhancement to add a couple atomic operations to SelectionKey for > interest ops like getAndSetBits/getAndClearBits? They would only have > to be atomic with respect to each other, so they could be emulated > with locks for compatibility. At present, coordinating adding > interest ops across threads means adding a lock, which is super > heavyweight considering that interest ops are backed a volatile int > field in most if not all cases. > > I could work up a patch if you want; since you've been so busy in the > NIO area lately I've been hesitant to try anything in that area, at > least until things settle down a little. interestOpsOr(int ops) and interestOpsAnd(int ops) wouldn't look out of place. It would be a good use of var handles as you could implement it with something like: static final VarHandle INTERESTOPS; static { ??? try { ??????? MethodHandles.Lookup l = MethodHandles.lookup(); ??????? INTERESTOPS = l.findVarHandle(SelectionKey.class, "interestOps", int.class); ??? } catch (Exception e) { ??????? throw new InternalError(e); ??? } } public SelectionKey interestOpsOr(int ops) { ??? : ??? INTERESTOPS.getAndBitwiseOr(this, ops); ??? : } There's one patch pending that changes the the code in nioInterestOps in how queue changes but I don't have other changes in my patch queue that? SelectionKeyImpl. So go ahead! -Alan From david.lloyd at redhat.com Wed Mar 28 18:22:57 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Wed, 28 Mar 2018 13:22:57 -0500 Subject: Callback Based Selectors In-Reply-To: <622f0553-43c3-3b02-0f18-b9781fbd96c1@oracle.com> References: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> <1365073714.1162106.1521037015119.JavaMail.zimbra@u-pem.fr> <7185f4eb-618c-1e11-0267-43e0be448995@oracle.com> <188928e3-4430-4d8f-a0b5-4a2aae545f2b@oracle.com> <622f0553-43c3-3b02-0f18-b9781fbd96c1@oracle.com> Message-ID: On Wed, Mar 28, 2018 at 12:09 PM, Alan Bateman wrote: > On 28/03/2018 15:45, David Lloyd wrote: >> >> : >> I wonder, since we're moving these APIs to be more >> concurrency-friendly, if you could consider an additional small >> enhancement to add a couple atomic operations to SelectionKey for >> interest ops like getAndSetBits/getAndClearBits? [...] > > interestOpsOr(int ops) and interestOpsAnd(int ops) wouldn't look out of > place. It would be a good use of var handles as you could implement it with > something like: > [...] So go ahead! OK! Here's a v1 pass which just includes the base methods on SelectionKey (attached and [1]). Unfortunately, it looks like some nontrivial surgery would be necessary after all before these could be fully implemented with atomics in a correct way, due to sun.nio.ch.SelChImpl#translateAndSetInterestOps() being required for a change to take full effect, so I can look again once your nioInterestOps change is in. BTW, it looks like nioInterestOps(int) is no longer called outside of SelectionKey now; maybe it can/should be inlined into sun.nio.ch.SelectionKeyImpl#interestOps(int)? [1] https://github.com/dmlloyd/openjdk/commit/atomic-nio-ops-v1 -- - DML -------------- next part -------------- commit a5552ce78a16d7db22a3d10d0ce4c013fd2414a3 Author: David M. Lloyd Date: Wed Mar 28 13:13:44 2018 -0500 [JDK-0000000] Add atomic interest op methods to SelectionKey diff --git a/src/java.base/share/classes/java/nio/channels/SelectionKey.java b/src/java.base/share/classes/java/nio/channels/SelectionKey.java index 6af664fa96b..8983093ade2 100644 --- a/src/java.base/share/classes/java/nio/channels/SelectionKey.java +++ b/src/java.base/share/classes/java/nio/channels/SelectionKey.java @@ -196,6 +196,82 @@ public abstract class SelectionKey { */ public abstract SelectionKey interestOps(int ops); + /** + * Atomically sets this key's interest set to bitwise union ("or") of the existing + * interest set and the given value. This method is guaranteed to be + * atomic with respect to other concurrent calls to this method or to + * {@link #interestOpsAnd(int)}. + * + *

This method may be invoked at any time. Whether or not it blocks, + * and for how long, is implementation-dependent. + * + * @param ops The interest set to apply + * + * @return The previous interest set + * + * @throws IllegalArgumentException + * If a bit in the resultant set does not correspond to an operation that + * is supported by this key's channel, that is, if + * {@code ((interestOps() | ops) & ~channel().validOps()) != 0} + * + * @throws CancelledKeyException + * If this key has been cancelled + * + * @implSpec The default implementation synchronizes on this {@code SelectionKey} + * instance, calling {@link #interestOps()} and {@link #interestOps(int)} + * in turn; subclasses should provide a better implementation if + * possible (for example, using a + * {@link java.lang.invoke.VarHandle VarHandle} may be appropriate). + */ + public int interestOpsOr(int ops) { + synchronized (this) { + int oldVal = interestOps(); + interestOps(oldVal | ops); + return oldVal; + } + } + + /** + * Atomically sets this key's interest set to bitwise intersection ("and") of the + * existing interest set and the given value. This method is guaranteed to be + * atomic with respect to other concurrent calls to this method or to + * {@link #interestOpsOr(int)}. + * + *

This method may be invoked at any time. Whether or not it blocks, + * and for how long, is implementation-dependent. + * + * @param ops The interest set to apply + * + * @return The previous interest set + * + * @throws IllegalArgumentException + * If a bit in the resultant set does not correspond to an operation that + * is supported by this key's channel, that is, if + * {@code (interestOps() & ops & ~channel().validOps()) != 0} + * + * @throws CancelledKeyException + * If this key has been cancelled + * + * @apiNote The {@code ops} argument may contain bits which are not normally + * allowed by this key's channel, allowing bits to be cleared using + * bitwise complement values. For example, + * {@code interestOpsAnd(~SelectionKey.OP_READ)} will remove the + * {@code OP_READ} bit from the set without affecting other bits. + * + * @implSpec The default implementation synchronizes on this {@code SelectionKey} + * instance, calling {@link #interestOps()} and {@link #interestOps(int)} + * in turn; subclasses should provide a better implementation if + * possible (for example, using a + * {@link java.lang.invoke.VarHandle VarHandle} may be appropriate). + */ + public int interestOpsAnd(int ops) { + synchronized (this) { + int oldVal = interestOps(); + interestOps(oldVal & ops); + return oldVal; + } + } + /** * Retrieves this key's ready-operation set. * From Alan.Bateman at oracle.com Wed Mar 28 20:15:59 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 28 Mar 2018 21:15:59 +0100 Subject: Callback Based Selectors In-Reply-To: References: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> <1365073714.1162106.1521037015119.JavaMail.zimbra@u-pem.fr> <7185f4eb-618c-1e11-0267-43e0be448995@oracle.com> <188928e3-4430-4d8f-a0b5-4a2aae545f2b@oracle.com> <622f0553-43c3-3b02-0f18-b9781fbd96c1@oracle.com> Message-ID: <2aa0b873-85cf-8a37-d2cb-677aad9c70d9@oracle.com> On 28/03/2018 19:22, David Lloyd wrote: > : > OK! Here's a v1 pass which just includes the base methods on > SelectionKey (attached and [1]). > > Unfortunately, it looks like some nontrivial surgery would be > necessary after all before these could be fully implemented with > atomics in a correct way, due to > sun.nio.ch.SelChImpl#translateAndSetInterestOps() being required for a > change to take full effect, so I can look again once your > nioInterestOps change is in. I don't think there is an issue there as a sequence of callings to interestOpsXXX will just queue the translation of the interest ops. However, I agree it will be a bit simpler once the changes for JDK-8200257 are in. > > BTW, it looks like nioInterestOps(int) is no longer called outside of > SelectionKey now; maybe it can/should be inlined into > sun.nio.ch.SelectionKeyImpl#interestOps(int)? The nioXXX methods are to allow access easy interest or ready ops in the implementation. You are right that nioInterestOps isn't used outside of SKI now but it may be inlined anyway. I don't object to rolling it into interestOps of course, it just might need to be split out again if we need it in the implementation. -Alan From Alan.Bateman at oracle.com Thu Mar 29 20:31:06 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 29 Mar 2018 21:31:06 +0100 Subject: 8200458: (se) Readiness information previously recorded in the ready set not preserved Message-ID: A number of networking tests that use the JDK's HTTP server have been tripping an assert in the HTTP server implementation lately. The assert arises when a key in the Selector's selected-key has an empty ready-set. The scenario to explain how this arises is complicated - it involves calling select twice, ignoring a key added to the selected-key in the first select, and closing the key's channel asynchronously at around the same time that the second select polls the file descriptor for the channel again. The root of the issue is that the Selector implementations aren't preserving readiness information for keys in the selected key set. If a channel's key is already in the selected key set then its ready set is supposed to be updated to identify new operation but this isn't so. I had to dig into pre-OpenJDK history to see that there have been issues here for a long time. Most usages of Selectors are likely to remove the keys in the selected-key so they wouldn't see this issue, it's only cases where the selected keys aren't removed that this is noticed. The changes to fix this are simple but it is a behavior change so a CSR and release notes would be appropriate if we fix this issue. I've run all tests on all platforms with the changes and the only issue I ran into was the fix for JDK-4737146 which I don't think was correct. The webrev with the changes is here: ?? http://cr.openjdk.java.net/~alanb/8200458/webrev/index.html -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.lloyd at redhat.com Thu Mar 29 21:22:51 2018 From: david.lloyd at redhat.com (David Lloyd) Date: Thu, 29 Mar 2018 16:22:51 -0500 Subject: Callback Based Selectors In-Reply-To: <2aa0b873-85cf-8a37-d2cb-677aad9c70d9@oracle.com> References: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> <1365073714.1162106.1521037015119.JavaMail.zimbra@u-pem.fr> <7185f4eb-618c-1e11-0267-43e0be448995@oracle.com> <188928e3-4430-4d8f-a0b5-4a2aae545f2b@oracle.com> <622f0553-43c3-3b02-0f18-b9781fbd96c1@oracle.com> <2aa0b873-85cf-8a37-d2cb-677aad9c70d9@oracle.com> Message-ID: On Wed, Mar 28, 2018 at 3:15 PM, Alan Bateman wrote: > I don't think there is an issue there as a sequence of callings to > interestOpsXXX will just queue the translation of the interest ops. OK I've given it another go. I haven't even been able to compile-test it yet though due to [1]. In particular, I'm not certain I can properly set up the VarHandle the way I have. But I wanted to review the basic idea. Using atomics, and assuming I understand how the op queueing is supposed to work, we can actually optimize a bit because we can avoid calling nioInterestOps(int) unless a new bit has been added which wasn't there before. The patch is attached and also at [2]. [1] http://mail.openjdk.java.net/pipermail/hotspot-dev/2018-March/030736.html [2] https://github.com/dmlloyd/openjdk/commit/atomic-nio-ops-v2 -- - DML -------------- next part -------------- commit 63569aeb836c3db97072d32e1c154ed4fa4b7800 Author: David M. Lloyd Date: Wed Mar 28 13:13:44 2018 -0500 [JDK-0000000] Add atomic interest op methods to SelectionKey diff --git a/src/java.base/share/classes/java/nio/channels/SelectionKey.java b/src/java.base/share/classes/java/nio/channels/SelectionKey.java index 6af664fa96b..8983093ade2 100644 --- a/src/java.base/share/classes/java/nio/channels/SelectionKey.java +++ b/src/java.base/share/classes/java/nio/channels/SelectionKey.java @@ -196,6 +196,82 @@ public abstract class SelectionKey { */ public abstract SelectionKey interestOps(int ops); + /** + * Atomically sets this key's interest set to bitwise union ("or") of the existing + * interest set and the given value. This method is guaranteed to be + * atomic with respect to other concurrent calls to this method or to + * {@link #interestOpsAnd(int)}. + * + *

This method may be invoked at any time. Whether or not it blocks, + * and for how long, is implementation-dependent. + * + * @param ops The interest set to apply + * + * @return The previous interest set + * + * @throws IllegalArgumentException + * If a bit in the resultant set does not correspond to an operation that + * is supported by this key's channel, that is, if + * {@code ((interestOps() | ops) & ~channel().validOps()) != 0} + * + * @throws CancelledKeyException + * If this key has been cancelled + * + * @implSpec The default implementation synchronizes on this {@code SelectionKey} + * instance, calling {@link #interestOps()} and {@link #interestOps(int)} + * in turn; subclasses should provide a better implementation if + * possible (for example, using a + * {@link java.lang.invoke.VarHandle VarHandle} may be appropriate). + */ + public int interestOpsOr(int ops) { + synchronized (this) { + int oldVal = interestOps(); + interestOps(oldVal | ops); + return oldVal; + } + } + + /** + * Atomically sets this key's interest set to bitwise intersection ("and") of the + * existing interest set and the given value. This method is guaranteed to be + * atomic with respect to other concurrent calls to this method or to + * {@link #interestOpsOr(int)}. + * + *

This method may be invoked at any time. Whether or not it blocks, + * and for how long, is implementation-dependent. + * + * @param ops The interest set to apply + * + * @return The previous interest set + * + * @throws IllegalArgumentException + * If a bit in the resultant set does not correspond to an operation that + * is supported by this key's channel, that is, if + * {@code (interestOps() & ops & ~channel().validOps()) != 0} + * + * @throws CancelledKeyException + * If this key has been cancelled + * + * @apiNote The {@code ops} argument may contain bits which are not normally + * allowed by this key's channel, allowing bits to be cleared using + * bitwise complement values. For example, + * {@code interestOpsAnd(~SelectionKey.OP_READ)} will remove the + * {@code OP_READ} bit from the set without affecting other bits. + * + * @implSpec The default implementation synchronizes on this {@code SelectionKey} + * instance, calling {@link #interestOps()} and {@link #interestOps(int)} + * in turn; subclasses should provide a better implementation if + * possible (for example, using a + * {@link java.lang.invoke.VarHandle VarHandle} may be appropriate). + */ + public int interestOpsAnd(int ops) { + synchronized (this) { + int oldVal = interestOps(); + interestOps(oldVal & ops); + return oldVal; + } + } + /** * Retrieves this key's ready-operation set. * diff --git a/src/java.base/share/classes/sun/nio/ch/SelectionKeyImpl.java b/src/java.base/share/classes/sun/nio/ch/SelectionKeyImpl.java index d4b4c39297d..0e209991d34 100644 --- a/src/java.base/share/classes/sun/nio/ch/SelectionKeyImpl.java +++ b/src/java.base/share/classes/sun/nio/ch/SelectionKeyImpl.java @@ -25,6 +25,9 @@ package sun.nio.ch; +import java.lang.invoke.ConstantBootstraps; +import java.lang.invoke.MethodHandles; +import java.lang.invoke.VarHandle; import java.nio.channels.CancelledKeyException; import java.nio.channels.SelectableChannel; import java.nio.channels.SelectionKey; @@ -40,6 +43,10 @@ public final class SelectionKeyImpl extends AbstractSelectionKey { + private static final VarHandle interestOpsHandle = ConstantBootstraps.fieldVarHandle( + MethodHandles.lookup(), "interestOps", VarHandle.class, SelectionKeyImpl.class, int.class + ); + final SelChImpl channel; // package-private public final SelectorImpl selector; @@ -86,7 +93,31 @@ public final class SelectionKeyImpl @Override public SelectionKey interestOps(int ops) { ensureValid(); - return nioInterestOps(ops); + // newOps is the set of ops that were not previously set + int newOps = ops & ~ (int) interestOpsHandle.getAndSet(this, ops); + if (newOps != 0) { + nioInterestOps(ops); + } + return this; + } + + @Override + public int interestOpsOr(final int ops) { + ensureValid(); + int oldVal = (int) interestOpsHandle.getAndBitwiseOr(this, ops); + // newOps is the set of ops that were not previously set + int newOps = ops & ~oldVal; + if (newOps != 0) { + // add the ops to the selector + nioInterestOps(ops); + } + return oldVal; + } + + @Override + public int interestOpsAnd(final int ops) { + ensureValid(); + return (int) interestOpsHandle.getAndBitwiseAnd(this, ops); } @Override From brian.burkhalter at oracle.com Thu Mar 29 23:13:06 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 29 Mar 2018 16:13:06 -0700 Subject: 8200257: (se) More Selector cleanup In-Reply-To: <377e3694-0703-26bd-01a6-689ec36a4314@oracle.com> References: <377e3694-0703-26bd-01a6-689ec36a4314@oracle.com> Message-ID: On Mar 27, 2018, at 1:00 AM, Alan Bateman wrote: > This is another round of improvements to the Selector implementations. > > [?] > > The webrev with the proposed changes is here: > http://cr.openjdk.java.net/~alanb/8200257/webrev/index.html These changes look okay to me. Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Fri Mar 30 01:11:43 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 29 Mar 2018 18:11:43 -0700 Subject: trivial patch to reduce allocations in nio open* methods In-Reply-To: References: <158BFF8B-586B-4C1C-8DBE-5C5DD3E8D66B@oracle.com> <2ba60a55-baf3-8542-dac6-70c33e3f5f25@oracle.com> <0803a885-5938-6549-262e-40064d73c513@oracle.com> <5aa714cf.c30c1c0a.fbbdc.b3fb@mx.google.com> Message-ID: I created a webrev of this patch: http://cr.openjdk.java.net/~bpb/8199124/webrev.00/ I don?t see any problem in moving forward with it, possibly with some minor cleanup. Thanks, Brian On Mar 28, 2018, at 12:25 AM, Michael Skells wrote: > what is the way forward with this patch ( if any) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Fri Mar 30 08:27:21 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 30 Mar 2018 09:27:21 +0100 Subject: trivial patch to reduce allocations in nio open* methods In-Reply-To: References: <158BFF8B-586B-4C1C-8DBE-5C5DD3E8D66B@oracle.com> <2ba60a55-baf3-8542-dac6-70c33e3f5f25@oracle.com> <0803a885-5938-6549-262e-40064d73c513@oracle.com> <5aa714cf.c30c1c0a.fbbdc.b3fb@mx.google.com> Message-ID: <4b6b5715-f643-b945-053d-b77889a7d554@oracle.com> On 30/03/2018 02:11, Brian Burkhalter wrote: > I created a webrev of this patch: > > http://cr.openjdk.java.net/~bpb/8199124/webrev.00/ > > > I don?t see any problem in moving forward with it, possibly with some > minor cleanup. > Just a few comments. The sizing of the hash map doesn't take the load factor into account. You might find `new HashMap<>(1 + (4 * options.length/ 3)` a bit better with a larger array of options. You can use an immutable set to avoid wrapping the EnumSet, e.g. ??? private static final Set DEFAULT_CREATE_OPTIONS = Set.of(CREATE_NEW, WRITE); -Alan From Alan.Bateman at oracle.com Fri Mar 30 12:50:37 2018 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 30 Mar 2018 13:50:37 +0100 Subject: Callback Based Selectors In-Reply-To: References: <0fe28e06-ef26-9c07-034f-f79feef2a66a@oracle.com> <1365073714.1162106.1521037015119.JavaMail.zimbra@u-pem.fr> <7185f4eb-618c-1e11-0267-43e0be448995@oracle.com> <188928e3-4430-4d8f-a0b5-4a2aae545f2b@oracle.com> <622f0553-43c3-3b02-0f18-b9781fbd96c1@oracle.com> <2aa0b873-85cf-8a37-d2cb-677aad9c70d9@oracle.com> Message-ID: On 29/03/2018 22:22, David Lloyd wrote: > On Wed, Mar 28, 2018 at 3:15 PM, Alan Bateman wrote: >> I don't think there is an issue there as a sequence of callings to >> interestOpsXXX will just queue the translation of the interest ops. > OK I've given it another go. I haven't even been able to compile-test > it yet though due to [1]. In particular, I'm not certain I can > properly set up the VarHandle the way I have. But I wanted to review > the basic idea. > > Using atomics, and assuming I understand how the op queueing is > supposed to work, we can actually optimize a bit because we can avoid > calling nioInterestOps(int) unless a new bit has been added which > wasn't there before. > Suppose the interestOps is 0. T1 invokes interestOpsOr(OP_READ) at around the same time that T2 invokes interestOpsOr(OP_WRITE). The interestOps field will get to OP_READ|OP_WRITE as you'd expect but there is no guarantee that the last change queued to the selector is the equivalent events (POLLIN|POLLOUT in this case). I don't have time just now to go into this in more detail but if you change the putEventOps implementations to: int ops = ski.interestOps(); updateEvents.addLast(ski.channel.translateInterestOps(ops)); then the selection operation will read the newest value of the interest set (further changes will be picked at the next selection). Once the queuing of updates is changed (to drop updateEvents) then this will be a lot simpler. -Alan From brian.burkhalter at oracle.com Fri Mar 30 15:15:33 2018 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 30 Mar 2018 08:15:33 -0700 Subject: trivial patch to reduce allocations in nio open* methods In-Reply-To: <4b6b5715-f643-b945-053d-b77889a7d554@oracle.com> References: <158BFF8B-586B-4C1C-8DBE-5C5DD3E8D66B@oracle.com> <2ba60a55-baf3-8542-dac6-70c33e3f5f25@oracle.com> <0803a885-5938-6549-262e-40064d73c513@oracle.com> <5aa714cf.c30c1c0a.fbbdc.b3fb@mx.google.com> <4b6b5715-f643-b945-053d-b77889a7d554@oracle.com> Message-ID: <56682F0C-A33E-47DC-A10E-11CE895516A2@oracle.com> On Mar 30, 2018, at 1:27 AM, Alan Bateman wrote: > You can use an immutable set to avoid wrapping the EnumSet, e.g. > private static final Set DEFAULT_CREATE_OPTIONS = Set.of(CREATE_NEW, WRITE); I was wondering whether the DEFAULT_*_OPTIONS constants should be created at all, i.e., kept around as static finals. Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From eolivelli at gmail.com Tue Mar 27 12:26:09 2018 From: eolivelli at gmail.com (Enrico Olivelli) Date: Tue, 27 Mar 2018 12:26:09 -0000 Subject: Changes to RandomAccessFile in jdk9 Message-ID: Hi, moving an application (in this case an Apache Kafka broker v 1.0) to oracle jdk 9.0.4 (from 1.8) I have seen that RandomAccessFile#setLength is now really allocating disk space, and before it seemed to create SPARSE files. I am running on linux CentOS on ext4 fs. Reverting back to jdk 1.8 and forcing the broker to recreate files they are in SPARSE mode Is this an intended behavior change? Is there any pointer to this change? Regards Enrico Olivelli -- -- Enrico Olivelli -------------- next part -------------- An HTML attachment was scrubbed... URL: