From brian.burkhalter at oracle.com Mon Jul 1 19:05:01 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 1 Jul 2019 12:05:01 -0700 Subject: Redux 8184157: (ch) AsynchronousFileChannel hangs with internal error when reading locked file In-Reply-To: References: <8B15EB6A-2E66-42D9-8277-C25FB9279837@oracle.com> Message-ID: > On Jun 30, 2019, at 1:37 AM, Alan Bateman wrote: > > On 27/06/2019 20:01, Brian Burkhalter wrote: >> https://bugs.openjdk.java.net/browse/JDK-8184157 >> http://cr.openjdk.java.net/~bpb/8184157/webrev.01/ >> >> > I think the analysis is good and the approach to consistently leave the release of the OVERLAPPED structure to when the completion event is posted is good. Good. Thanks for looking this over. > One thing that isn't immediately clear from the changes is whether it introduces a memory leak when there is an error initiating the I/O operation or where there is an async close at around the time that the I/O operation is initiated. I would need to study these cases further to see if there are issues. > > The Windows implementations of AsynchronousSocketChannel and AsynchronousServerSocketChannel also remove and free the OVERLAPPED structure then the socket operation completes immediately. Will they also need attention? Yes, at least AsynchronousSocketChannel: I should have caught that before. I?ll update the patch for these cases and look into the memory leak possibility mentioned above. > The test will need cleanup, maybe rename it to LockReadWriteStressTest or something that is closer to what it actually does. It can use try-with-resources to avoid leaking resources when it fails. I assume "f" and "l" will be renamed as they are unusual variables names in this context. Also the buffer usages is unusual and I can assume can be cleaned up too. The test is in effect the reproducer supplied in the issue so I did not give it any further attention yet. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Mon Jul 1 21:56:30 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 1 Jul 2019 14:56:30 -0700 Subject: 8224480: (fc) java/nio/channels/FileChannel/MapTest.java fails intermittently In-Reply-To: <48c36180-3b3d-bc21-6886-04794bb53a02@oracle.com> References: <48c36180-3b3d-bc21-6886-04794bb53a02@oracle.com> Message-ID: <18D95E99-883D-46F7-8853-60D8CD954CFF@oracle.com> > On Jun 30, 2019, at 1:42 AM, Alan Bateman wrote: > >> The hypothesis is that these configurations could be too slow for this sub-test so this patch proposes to halve the number of iterations in testForce() to account for this. An alternative would be to change testForce() to be a separate test. > This looks okay but I'm curious what the run-times are like on these systems (I'm just wondering if it might also be useful to add a /timeout option to the @run line so that they have more time to complete, even with the reduced number of iterations). The default timeout is 120 seconds and the timeout factor is 4 so the test timed out after 8 minutes. We could as well then double the timeout value as in the diff below versus the previous patch to allow more time to complete. Brian --- a/test/jdk/java/nio/channels/FileChannel/MapTest.java +++ b/test/jdk/java/nio/channels/FileChannel/MapTest.java @@ -24,7 +24,7 @@ /* @test * @bug 4429043 8002180 * @summary Test file mapping with FileChannel - * @run main/othervm MapTest + * @run main/othervm/timeout=240 MapTest * @key randomness */ -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Jul 2 06:16:08 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 2 Jul 2019 07:16:08 +0100 Subject: 8224480: (fc) java/nio/channels/FileChannel/MapTest.java fails intermittently In-Reply-To: <18D95E99-883D-46F7-8853-60D8CD954CFF@oracle.com> References: <48c36180-3b3d-bc21-6886-04794bb53a02@oracle.com> <18D95E99-883D-46F7-8853-60D8CD954CFF@oracle.com> Message-ID: On 01/07/2019 22:56, Brian Burkhalter wrote: > : > > The default timeout is 120 seconds and the timeout factor is 4 so the > test timed out after 8 minutes. We could as well then double the > timeout value as in the diff below versus the previous patch to allow > more time to complete. > It should only be necessary to use -timeoutFactor when running with debug or other VM options that significantly slow down the execution. So in this case, I think 8 mins is way too long (even for an old machine with spinning rust) so I think adding /timeout and also reducing the number of iterations is best. -Alan From simone.bordet at gmail.com Tue Jul 2 09:01:33 2019 From: simone.bordet at gmail.com (Simone Bordet) Date: Tue, 2 Jul 2019 11:01:33 +0200 Subject: Very slow Files.newInputStream(...).skip(n) Message-ID: Hi, we have moved Jetty to use newer APIs, replacing java.io.FileInputStream with Files.newInputStream(Path), which delegates to FileSystemProvider.newInputStream() which eventually returns a sun.nio.ch.ChannelInputStream. Unfortunately, ChannelInputStream.skip() implementation is very naive and very slow for large files. On the contrary, FileInputStream.skip() goes native and uses "seek" primitives to be efficient. The use case is that to read a large file (e.g. a movie) using HTTP's range requests: the browser sends a request and asks e.g. for the bytes ranging from 1 GiB to 1 GiB + 128 KiB. Calling skip(1073741824) in the two cases will have very different performances. I am a JDK Author and I can open a bug about this if you think it's appropriate. Thanks! -- Simone Bordet --- Finally, no matter how good the architecture and design are, to deliver bug-free software with optimal performance and reliability, the implementation technique must be flawless. Victoria Livschitz From Alan.Bateman at oracle.com Tue Jul 2 09:08:32 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 2 Jul 2019 10:08:32 +0100 Subject: Very slow Files.newInputStream(...).skip(n) In-Reply-To: References: Message-ID: On 02/07/2019 10:01, Simone Bordet wrote: > Hi, > > we have moved Jetty to use newer APIs, replacing > java.io.FileInputStream with Files.newInputStream(Path), which > delegates to FileSystemProvider.newInputStream() which eventually > returns a sun.nio.ch.ChannelInputStream. > > Unfortunately, ChannelInputStream.skip() implementation is very naive > and very slow for large files. > On the contrary, FileInputStream.skip() goes native and uses "seek" > primitives to be efficient. > > The use case is that to read a large file (e.g. a movie) using HTTP's > range requests: the browser sends a request and asks e.g. for the > bytes ranging from 1 GiB to 1 GiB + 128 KiB. > > Calling skip(1073741824) in the two cases will have very different performances. > > I am a JDK Author and I can open a bug about this if you think it's appropriate. > Yes, please. -Alan From simone.bordet at gmail.com Tue Jul 2 09:44:54 2019 From: simone.bordet at gmail.com (Simone Bordet) Date: Tue, 2 Jul 2019 11:44:54 +0200 Subject: Very slow Files.newInputStream(...).skip(n) In-Reply-To: References: Message-ID: Hi, https://bugs.openjdk.java.net/browse/JDK-8227080 Thanks! -- Simone Bordet --- Finally, no matter how good the architecture and design are, to deliver bug-free software with optimal performance and reliability, the implementation technique must be flawless. Victoria Livschitz From brian.burkhalter at oracle.com Tue Jul 2 14:54:54 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 2 Jul 2019 07:54:54 -0700 Subject: 8224480: (fc) java/nio/channels/FileChannel/MapTest.java fails intermittently In-Reply-To: References: <48c36180-3b3d-bc21-6886-04794bb53a02@oracle.com> <18D95E99-883D-46F7-8853-60D8CD954CFF@oracle.com> Message-ID: <0F69401A-D5D6-447A-9841-1B381C9B8DD4@oracle.com> > On Jul 1, 2019, at 11:16 PM, Alan Bateman wrote: > > On 01/07/2019 22:56, Brian Burkhalter wrote: >> : >> >> The default timeout is 120 seconds and the timeout factor is 4 so the test timed out after 8 minutes. We could as well then double the timeout value as in the diff below versus the previous patch to allow more time to complete. >> > It should only be necessary to use -timeoutFactor when running with debug or other VM options that significantly slow down the execution. The timeout factor was in the log but I don?t know where it is set. > So in this case, I think 8 mins is way too long (even for an old machine with spinning rust) so I think adding /timeout and also reducing the number of iterations is best. So the version with doubled timeout looks OK? Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Jul 2 14:56:44 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 2 Jul 2019 15:56:44 +0100 Subject: 8224480: (fc) java/nio/channels/FileChannel/MapTest.java fails intermittently In-Reply-To: <0F69401A-D5D6-447A-9841-1B381C9B8DD4@oracle.com> References: <48c36180-3b3d-bc21-6886-04794bb53a02@oracle.com> <18D95E99-883D-46F7-8853-60D8CD954CFF@oracle.com> <0F69401A-D5D6-447A-9841-1B381C9B8DD4@oracle.com> Message-ID: <77e6476a-8821-f73c-eb35-d651d19d153d@oracle.com> On 02/07/2019 15:54, Brian Burkhalter wrote: > > > The timeout factor was in the log but I don?t know where it is set. It's an option to jtreg. > >> So in this case, I think 8 mins is way too long (even for an old >> machine with spinning rust) so I think adding /timeout and also >> reducing the number of iterations is best. > > So the version with doubled timeout looks OK? > Yes, and combined with reduced the iteration count, then I assume it should completely in a reasonable time on slow machines. -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Jul 2 15:00:56 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 2 Jul 2019 08:00:56 -0700 Subject: 8224480: (fc) java/nio/channels/FileChannel/MapTest.java fails intermittently In-Reply-To: <77e6476a-8821-f73c-eb35-d651d19d153d@oracle.com> References: <48c36180-3b3d-bc21-6886-04794bb53a02@oracle.com> <18D95E99-883D-46F7-8853-60D8CD954CFF@oracle.com> <0F69401A-D5D6-447A-9841-1B381C9B8DD4@oracle.com> <77e6476a-8821-f73c-eb35-d651d19d153d@oracle.com> Message-ID: <7852486E-695A-401A-ACA6-ACE70446EDE5@oracle.com> > On Jul 2, 2019, at 7:56 AM, Alan Bateman wrote: > >> >> >> The timeout factor was in the log but I don?t know where it is set. > It's an option to jtreg. I know that but I don?t know where in the CI infrastructure it is being set. >> >>> So in this case, I think 8 mins is way too long (even for an old machine with spinning rust) so I think adding /timeout and also reducing the number of iterations is best. >> >> So the version with doubled timeout looks OK? >> > Yes, and combined with reduced the iteration count, then I assume it should completely in a reasonable time on slow machines. OK I?ll check it in as-is then and we?ll see what happens. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Jul 2 15:59:42 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 2 Jul 2019 16:59:42 +0100 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: <892f3b0b-55f1-872d-15f4-5af907fa8437@redhat.com> References: <80da32b2-7acb-7b94-b82c-5dcd5cf95539@redhat.com> <656f462e-c655-0c48-7c90-190c92e0bc28@redhat.com> <0a41e686-0931-5054-b314-2c015fbae62a@redhat.com> <36326F5E-12CD-487A-8FE2-1049631FE908@oracle.com> <8B1A8EDC-F9D7-4085-A34F-69100DBD7D5C@oracle.com> <5c78b5d7-dc9e-fbbd-41a9-5139ed6ee32c@redhat.com> <86d000f3-9eca-6089-5f7e-5698444d99ce@oracle.com> <7fe8d17d-6170-fcc9-b87a-eccfda2bb546@redhat.com> <7b39f68d-5698-4079-425c-c86ec161e361@oracle.com> <72810832-eb88-3418-f336-b95af95d9dcc@redhat.com> <20c8bbcb-2cd3-77af-cd40-29f7bd752166@oracle.com> <68566847-de80-82af-a928-d30c75f2b1b2@redhat.com> <892f3b0b-55f1-872d-15f4-5af907fa8437@redhat.com> Message-ID: On 18/06/2019 12:43, Andrew Dinn wrote: > Hi Alan, > > Thanks for reviewing the JEP one more time. The proposed updates look good to me. One nit is that the update to the Goals section says "rework" in two places. I think it might more accurate to say "upgrade" or "update" as it doesn't significantly re-working the existing implementation. -Alan From brian.burkhalter at oracle.com Tue Jul 2 16:02:26 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 2 Jul 2019 09:02:26 -0700 Subject: Redux 8184157: (ch) AsynchronousFileChannel hangs with internal error when reading locked file In-Reply-To: References: <8B15EB6A-2E66-42D9-8277-C25FB9279837@oracle.com> Message-ID: <7DA84564-0770-48E8-9AEA-B6D38C2F4B62@oracle.com> > On Jul 1, 2019, at 12:05 PM, Brian Burkhalter wrote: > >> One thing that isn't immediately clear from the changes is whether it introduces a memory leak when there is an error initiating the I/O operation or where there is an async close at around the time that the I/O operation is initiated. I would need to study these cases further to see if there are issues. If an exception is thrown initiating an I/O operation then this indeed would introduce a memory leak. This leak was there already for the pending case. I don?t know about the async close case. >> The Windows implementations of AsynchronousSocketChannel and AsynchronousServerSocketChannel also remove and free the OVERLAPPED structure then the socket operation completes immediately. Will they also need attention? > > Yes, at least AsynchronousSocketChannel: I should have caught that before. I?ll update the patch for these cases and look into the memory leak possibility mentioned above. AsynchronousSocketChannel needs a similar change but I think not AsynchronousServerSocketChannel. I do not have a reproducer for this case yet however. >> The test will need cleanup, maybe rename it to LockReadWriteStressTest or something that is closer to what it actually does. It can use try-with-resources to avoid leaking resources when it fails. I assume "f" and "l" will be renamed as they are unusual variables names in this context. Also the buffer usages is unusual and I can assume can be cleaned up too. > > The test is in effect the reproducer supplied in the issue so I did not give it any further attention yet. The test and the other things above are updated here: http://cr.openjdk.java.net/~bpb/8184157/webrev.02/ Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From adinn at redhat.com Tue Jul 2 17:13:02 2019 From: adinn at redhat.com (Andrew Dinn) Date: Tue, 2 Jul 2019 18:13:02 +0100 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: References: <80da32b2-7acb-7b94-b82c-5dcd5cf95539@redhat.com> <0a41e686-0931-5054-b314-2c015fbae62a@redhat.com> <36326F5E-12CD-487A-8FE2-1049631FE908@oracle.com> <8B1A8EDC-F9D7-4085-A34F-69100DBD7D5C@oracle.com> <5c78b5d7-dc9e-fbbd-41a9-5139ed6ee32c@redhat.com> <86d000f3-9eca-6089-5f7e-5698444d99ce@oracle.com> <7fe8d17d-6170-fcc9-b87a-eccfda2bb546@redhat.com> <7b39f68d-5698-4079-425c-c86ec161e361@oracle.com> <72810832-eb88-3418-f336-b95af95d9dcc@redhat.com> <20c8bbcb-2cd3-77af-cd40-29f7bd752166@oracle.com> <68566847-de80-82af-a928-d30c75f2b1b2@redhat.com> <892f3b0b-55f1-872d-15f4-5af907fa8437@redhat.com> Message-ID: Hi Alan, On 02/07/2019 16:59, Alan Bateman wrote: > On 18/06/2019 12:43, Andrew Dinn wrote: > One nit is that the update to the Goals section says "rework" in two > places. I think it might more accurate to say "upgrade" or "update" as > it doesn't significantly re-working the existing implementation. Thanks for checking this rewording. I have updated the JEP text modulo replacing reworked with upgraded. regards, Andrew Dinn ----------- From brian.burkhalter at oracle.com Tue Jul 2 19:18:58 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 2 Jul 2019 12:18:58 -0700 Subject: 8227080: (fs) Files.newInputStream(...).skip(n) is slow Message-ID: <2F288856-2AF5-4CCC-83F5-E959B394C5AB@oracle.com> https://bugs.openjdk.java.net/browse/JDK-8227080 http://cr.openjdk.java.net/~bpb/8227080/webrev.00/ This patch overrides ChannelInputStream.skip(long) to use SeekableByteChannel.position(long) if the ReadableByteChannel instance is a SeekableByteChannel. The performance improvement on my dev machine for skipping the first 2 GB of a local file is about 3.6 x 10^4. Thanks, Brian From brian.burkhalter at oracle.com Tue Jul 2 19:41:40 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 2 Jul 2019 12:41:40 -0700 Subject: 8227080: (fs) Files.newInputStream(...).skip(n) is slow In-Reply-To: <2F288856-2AF5-4CCC-83F5-E959B394C5AB@oracle.com> References: <2F288856-2AF5-4CCC-83F5-E959B394C5AB@oracle.com> Message-ID: Oops, I think that skip() needs to be synchronized. Webrev updated in place. > On Jul 2, 2019, at 12:18 PM, Brian Burkhalter wrote: > > https://bugs.openjdk.java.net/browse/JDK-8227080 > http://cr.openjdk.java.net/~bpb/8227080/webrev.00/ > > This patch overrides ChannelInputStream.skip(long) to use SeekableByteChannel.position(long) if the ReadableByteChannel instance is a SeekableByteChannel. The performance improvement on my dev machine for skipping the first 2 GB of a local file is about 3.6 x 10^4. > > Thanks, > > Brian From simone.bordet at gmail.com Tue Jul 2 20:01:45 2019 From: simone.bordet at gmail.com (Simone Bordet) Date: Tue, 2 Jul 2019 22:01:45 +0200 Subject: 8227080: (fs) Files.newInputStream(...).skip(n) is slow In-Reply-To: References: <2F288856-2AF5-4CCC-83F5-E959B394C5AB@oracle.com> Message-ID: Hi, On Tue, Jul 2, 2019 at 9:42 PM Brian Burkhalter wrote: > > Oops, I think that skip() needs to be synchronized. Webrev updated in place. > > > On Jul 2, 2019, at 12:18 PM, Brian Burkhalter wrote: > > > > https://bugs.openjdk.java.net/browse/JDK-8227080 > > http://cr.openjdk.java.net/~bpb/8227080/webrev.00/ > > > > This patch overrides ChannelInputStream.skip(long) to use SeekableByteChannel.position(long) if the ReadableByteChannel instance is a SeekableByteChannel. The performance improvement on my dev machine for skipping the first 2 GB of a local file is about 3.6 x 10^4. > > That's great! I'm not a reviewer, but is there a policy to handle long overflows? Also, ChannelInputStream should be reviewed with respect to synchronization. For example, you fixed skip(), but available() is not synchronized, and read(ByteBuffer) should perhaps be (as subclasses may call it without grabbing the lock)? Thanks! -- Simone Bordet --- Finally, no matter how good the architecture and design are, to deliver bug-free software with optimal performance and reliability, the implementation technique must be flawless. Victoria Livschitz From brian.burkhalter at oracle.com Tue Jul 2 20:23:49 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 2 Jul 2019 13:23:49 -0700 Subject: 8227080: (fs) Files.newInputStream(...).skip(n) is slow In-Reply-To: References: <2F288856-2AF5-4CCC-83F5-E959B394C5AB@oracle.com> Message-ID: <0A099B1F-E77F-49A9-8A02-79A9B1ADBE2F@oracle.com> > On Jul 2, 2019, at 1:01 PM, Simone Bordet wrote: > > On Tue, Jul 2, 2019 at 9:42 PM Brian Burkhalter > > wrote: >> >> Oops, I think that skip() needs to be synchronized. Webrev updated in place. >> >>> On Jul 2, 2019, at 12:18 PM, Brian Burkhalter > wrote: >>> >>> https://bugs.openjdk.java.net/browse/JDK-8227080 >>> http://cr.openjdk.java.net/~bpb/8227080/webrev.00/ >>> >>> This patch overrides ChannelInputStream.skip(long) to use SeekableByteChannel.position(long) if the ReadableByteChannel instance is a SeekableByteChannel. The performance improvement on my dev machine for skipping the first 2 GB of a local file is about 3.6 x 10^4. >>> > > That's great! > > I'm not a reviewer, but is there a policy to handle long overflows? Good point. Maybe Math.addExact(long,long) could be used and if an ArithmeticException occurs it could be wrapped in an IOException. Needs investigation. > Also, ChannelInputStream should be reviewed with respect to synchronization. > For example, you fixed skip(), but available() is not synchronized, > and read(ByteBuffer) should perhaps be (as subclasses may call it > without grabbing the lock)? I did look it over but maybe it needs another glance. I noticed that available() is non-sync as well. It looked as if for users of an instance, read() is OK but perhaps you have a point for subclasses. > Thanks! You?re welcome! Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Jul 2 22:00:30 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 2 Jul 2019 15:00:30 -0700 Subject: 8227080: (fs) Files.newInputStream(...).skip(n) is slow In-Reply-To: <0A099B1F-E77F-49A9-8A02-79A9B1ADBE2F@oracle.com> References: <2F288856-2AF5-4CCC-83F5-E959B394C5AB@oracle.com> <0A099B1F-E77F-49A9-8A02-79A9B1ADBE2F@oracle.com> Message-ID: > On Jul 2, 2019, at 1:23 PM, Brian Burkhalter wrote: > >> I'm not a reviewer, but is there a policy to handle long overflows? > > Good point. Maybe Math.addExact(long,long) could be used and if an ArithmeticException occurs it could be wrapped in an IOException. Needs investigation. Here is an updated version which should not have a problem with overflow: http://cr.openjdk.java.net/~bpb/8227080/webrev.01/ Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From simone.bordet at gmail.com Tue Jul 2 22:32:48 2019 From: simone.bordet at gmail.com (Simone Bordet) Date: Wed, 3 Jul 2019 00:32:48 +0200 Subject: 8227080: (fs) Files.newInputStream(...).skip(n) is slow In-Reply-To: References: <2F288856-2AF5-4CCC-83F5-E959B394C5AB@oracle.com> <0A099B1F-E77F-49A9-8A02-79A9B1ADBE2F@oracle.com> Message-ID: Hi, On Wed, Jul 3, 2019 at 12:00 AM Brian Burkhalter wrote: > Here is an updated version which should not have a problem with overflow: > > http://cr.openjdk.java.net/~bpb/8227080/webrev.01/ Why: return sbc.position() - pos; rather than just: return n; Unless SeekableByteChannel.position(int) may not actually set the position to the given value, which I can't quite believe (as the javadoc says it is permitted to set it to a larger value than the current size)? Will you tackle the missing synchronization in another webrev? Thanks! -- Simone Bordet --- Finally, no matter how good the architecture and design are, to deliver bug-free software with optimal performance and reliability, the implementation technique must be flawless. Victoria Livschitz From brian.burkhalter at oracle.com Tue Jul 2 23:16:07 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 2 Jul 2019 16:16:07 -0700 Subject: 8227080: (fs) Files.newInputStream(...).skip(n) is slow In-Reply-To: References: <2F288856-2AF5-4CCC-83F5-E959B394C5AB@oracle.com> <0A099B1F-E77F-49A9-8A02-79A9B1ADBE2F@oracle.com> Message-ID: > On Jul 2, 2019, at 3:32 PM, Simone Bordet wrote: > > On Wed, Jul 3, 2019 at 12:00 AM Brian Burkhalter > > wrote: >> Here is an updated version which should not have a problem with overflow: >> >> http://cr.openjdk.java.net/~bpb/8227080/webrev.01/ > > Why: > > return sbc.position() - pos; > > rather than just: > > return n; That?s what I had first before re-posting. > Unless SeekableByteChannel.position(int) may not actually set the > position to the given value, which I can't quite believe (as the > javadoc says it is permitted to set it to a larger value than the > current size)? No, I think either way is good. > Will you tackle the missing synchronization in another webrev? Yes as I am sure there will be other comments from others. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Roger.Riggs at oracle.com Wed Jul 3 14:03:38 2019 From: Roger.Riggs at oracle.com (Roger Riggs) Date: Wed, 3 Jul 2019 10:03:38 -0400 Subject: 8227080: (fs) Files.newInputStream(...).skip(n) is slow In-Reply-To: References: <2F288856-2AF5-4CCC-83F5-E959B394C5AB@oracle.com> <0A099B1F-E77F-49A9-8A02-79A9B1ADBE2F@oracle.com> Message-ID: <80317f0b-d206-c0f4-3815-22a678506a0e@oracle.com> Hi Brian, Looks fine; taking into account possible integer overflow and returning the actual number of bytes skipped. +1, Roger On 7/2/19 7:16 PM, Brian Burkhalter wrote: > >> On Jul 2, 2019, at 3:32 PM, Simone Bordet > > wrote: >> >> On Wed, Jul 3, 2019 at 12:00 AM Brian Burkhalter >> > wrote: >>> Here is an updated version which should not have a problem with >>> overflow: >>> >>> http://cr.openjdk.java.net/~bpb/8227080/webrev.01/ >> >> Why: >> >> return sbc.position() - pos; >> >> rather than just: >> >> return n; > > That?s what I had first before re-posting. > >> Unless SeekableByteChannel.position(int) may not actually set the >> position to the given value, which I can't quite believe (as the >> javadoc says it is permitted to set it to a larger value than the >> current size)? > > No, I think either way is good. > >> Will you tackle the missing synchronization in another webrev? > > Yes as I am sure there will be other comments from others. > > Thanks, > > Brian From fweimer at redhat.com Wed Jul 3 14:11:23 2019 From: fweimer at redhat.com (Florian Weimer) Date: Wed, 03 Jul 2019 16:11:23 +0200 Subject: 8227080: (fs) Files.newInputStream(...).skip(n) is slow In-Reply-To: <2F288856-2AF5-4CCC-83F5-E959B394C5AB@oracle.com> (Brian Burkhalter's message of "Tue, 2 Jul 2019 12:18:58 -0700") References: <2F288856-2AF5-4CCC-83F5-E959B394C5AB@oracle.com> Message-ID: <87imsjp5ro.fsf@oldenburg2.str.redhat.com> * Brian Burkhalter: > https://bugs.openjdk.java.net/browse/JDK-8227080 > http://cr.openjdk.java.net/~bpb/8227080/webrev.00/ > > This patch overrides ChannelInputStream.skip(long) to use > SeekableByteChannel.position(long) if the ReadableByteChannel instance > is a SeekableByteChannel. The performance improvement on my dev > machine for skipping the first 2 GB of a local file is about 3.6 x > 10^4. Doesn't this need a fallback in case the seek operation fails? I think you can have FileChannels which are not seekable in practice, e.g. if the file is actually a FIFO. Thanks, Florian From brian.burkhalter at oracle.com Wed Jul 3 14:50:43 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 3 Jul 2019 07:50:43 -0700 Subject: 8227080: (fs) Files.newInputStream(...).skip(n) is slow In-Reply-To: <87imsjp5ro.fsf@oldenburg2.str.redhat.com> References: <2F288856-2AF5-4CCC-83F5-E959B394C5AB@oracle.com> <87imsjp5ro.fsf@oldenburg2.str.redhat.com> Message-ID: <2ACDDD15-1AE0-4D82-8E97-CFE6B98FD505@oracle.com> Hi Florian, > On Jul 3, 2019, at 7:11 AM, Florian Weimer wrote: > >> This patch overrides ChannelInputStream.skip(long) to use >> SeekableByteChannel.position(long) if the ReadableByteChannel instance >> is a SeekableByteChannel. The performance improvement on my dev >> machine for skipping the first 2 GB of a local file is about 3.6 x >> 10^4. > > Doesn't this need a fallback in case the seek operation fails? I think > you can have FileChannels which are not seekable in practice, e.g. if > the file is actually a FIFO. Would that not violate the SeekableByteChannel specification? On the other hand, I suppose one might be able to fall back to super.skip(). If the seek however actually did skip a few bytes before failing then it would be possible to skip an incorrect number of bytes in this case. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From uschindler at apache.org Wed Jul 3 14:51:05 2019 From: uschindler at apache.org (Uwe Schindler) Date: Wed, 3 Jul 2019 16:51:05 +0200 Subject: Any way to access fstat.st_blocks from Java (UnixFileAttributesView) In-Reply-To: References: <0afa01d52d9c$d95879e0$8c096da0$@apache.org> <9bc3c13e-a700-7908-4590-9e79c60fa0e2@oracle.com> Message-ID: <036901d531ae$bf388300$3da98900$@apache.org> Hi, sorry for the late response. To actually fix my problem I had to go and use a native solution by using the JNR-POSIX package by Charles Nutter (@headius) from Maven (see https://github.com/jnr/jnr-posix). I don?t like the complexity of that setup (you need like 8 additional jar files), but at least it works for now. JNA would have been another solution, but I did not want to create the st_stat structs and all syscalls needed to get from a NIO Path to that value. The question is how to propose a change request like this, because getting the size a file *occupies* on disk is a useful information. Adding this should bring no backwards breaks or any need to change public APIs, as the ?unix? view is not exposed as a public API, its just accessible using the lookup by string key. So adding two other key-value pairs should be easy. What do you think? Maybe I can create a patch to add this, but what?s the way to go? This may be a first try to submit something as a ?new contributor?, because I have signed the CLA already. This change is not Lucene related (was a problem in my daily work), but we have some other ideas for more complex patches in Lucene, so this could be a start. Uwe ----- Uwe Schindler uschindler at apache.org ASF Member, Apache Lucene PMC / Committer Bremen, Germany https://lucene.apache.org/ From: Brian Burkhalter Sent: Friday, June 28, 2019 5:45 PM To: Alan Bateman Cc: Uwe Schindler ; nio-dev Subject: Re: Any way to access fstat.st_blocks from Java (UnixFileAttributesView) On Jun 28, 2019, at 7:18 AM, Alan Bateman > wrote: Any hints to get the file size on disk from Java that uses the st_blocks and st_blksize behind the scenes? There isn't anything at this time. You are right that the "unix" view could expose a "blocks" attribute. The block size is already exposed by FileStore::getBlockSize - the motivation for that was sizing and aligning buffers for use with direct I/O. Maybe an enhancement should be filed? Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From fweimer at redhat.com Wed Jul 3 14:58:04 2019 From: fweimer at redhat.com (Florian Weimer) Date: Wed, 03 Jul 2019 16:58:04 +0200 Subject: 8227080: (fs) Files.newInputStream(...).skip(n) is slow In-Reply-To: <2ACDDD15-1AE0-4D82-8E97-CFE6B98FD505@oracle.com> (Brian Burkhalter's message of "Wed, 3 Jul 2019 07:50:43 -0700") References: <2F288856-2AF5-4CCC-83F5-E959B394C5AB@oracle.com> <87imsjp5ro.fsf@oldenburg2.str.redhat.com> <2ACDDD15-1AE0-4D82-8E97-CFE6B98FD505@oracle.com> Message-ID: <87lfxfnp1f.fsf@oldenburg2.str.redhat.com> * Brian Burkhalter: > Hi Florian, > > On Jul 3, 2019, at 7:11 AM, Florian Weimer wrote: > > This patch overrides ChannelInputStream.skip(long) to use > SeekableByteChannel.position(long) if the ReadableByteChannel instance > is a SeekableByteChannel. The performance improvement on my dev > machine for skipping the first 2 GB of a local file is about 3.6 x > 10^4. > > Doesn't this need a fallback in case the seek operation fails? I think > you can have FileChannels which are not seekable in practice, e.g. if > the file is actually a FIFO. > > Would that not violate the SeekableByteChannel specification? I suppose it does, but what can FileChannel do if not all files in the file system are seekable, in the sense that opening them does not give you a seekable file descriptor? > If the seek however actually did skip a few bytes before failing then > it would be possible to skip an incorrect number of bytes in this > case. I think position(long) needs to be atomic regarding failure for this to work. That really should be the case on Linux (and I think for the core file systems, this is actually thoroughly tested). Thanks, Florian From brian.burkhalter at oracle.com Wed Jul 3 14:59:23 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 3 Jul 2019 07:59:23 -0700 Subject: Any way to access fstat.st_blocks from Java (UnixFileAttributesView) In-Reply-To: <036901d531ae$bf388300$3da98900$@apache.org> References: <0afa01d52d9c$d95879e0$8c096da0$@apache.org> <9bc3c13e-a700-7908-4590-9e79c60fa0e2@oracle.com> <036901d531ae$bf388300$3da98900$@apache.org> Message-ID: <024E92BE-ED18-420E-B920-313A3FC42744@oracle.com> Hi Uwe, > On Jul 3, 2019, at 7:51 AM, Uwe Schindler wrote: > > The question is how to propose a change request like this, because getting the size a file *occupies* on disk is a useful information. Adding this should bring no backwards breaks or any need to change public APIs, as the ?unix? view is not exposed as a public API, its just accessible using the lookup by string key. So adding two other key-value pairs should be easy. What do you think? You already have the Author role so I think you could create an enhancement issue in JBS. The project would be JDK, the component core-libs, and the sub-component java.nio. > Maybe I can create a patch to add this, but what?s the way to go? This may be a first try to submit something as a ?new contributor?, because I have signed the CLA already. Yes, you could create a patch. Guidelines for contributions are given at [1, 2]. It would probably good to get some agreement on what API elements would be added before spending a lot of time on the implementation. Thanks, Brian [1] http://openjdk.java.net/contribute/ [2] http://openjdk.java.net/guide/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Wed Jul 3 15:08:51 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 3 Jul 2019 08:08:51 -0700 Subject: 8227080: (fs) Files.newInputStream(...).skip(n) is slow In-Reply-To: <87lfxfnp1f.fsf@oldenburg2.str.redhat.com> References: <2F288856-2AF5-4CCC-83F5-E959B394C5AB@oracle.com> <87imsjp5ro.fsf@oldenburg2.str.redhat.com> <2ACDDD15-1AE0-4D82-8E97-CFE6B98FD505@oracle.com> <87lfxfnp1f.fsf@oldenburg2.str.redhat.com> Message-ID: <0B83B0DB-3081-4429-8F2B-0941377C9C00@oracle.com> > On Jul 3, 2019, at 7:58 AM, Florian Weimer wrote: > >> Doesn't this need a fallback in case the seek operation fails? I think >> you can have FileChannels which are not seekable in practice, e.g. if >> the file is actually a FIFO. >> >> Would that not violate the SeekableByteChannel specification? > > I suppose it does, but what can FileChannel do if not all files in the > file system are seekable, in the sense that opening them does not give > you a seekable file descriptor? The behavior would be the same as it is now. This change only addresses the case where the instance variable ch is a SeekableByteChannel. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Wed Jul 3 21:49:46 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 3 Jul 2019 14:49:46 -0700 Subject: 8227080: (fs) Files.newInputStream(...).skip(n) is slow In-Reply-To: References: <2F288856-2AF5-4CCC-83F5-E959B394C5AB@oracle.com> <0A099B1F-E77F-49A9-8A02-79A9B1ADBE2F@oracle.com> Message-ID: <1025E944-0BB1-40A4-9211-D2AF10B5522D@oracle.com> > On Jul 2, 2019, at 4:16 PM, Brian Burkhalter wrote: > >> Will you tackle the missing synchronization in another webrev? > > Yes as I am sure there will be other comments from others. I went ahead and checked in the .01 version of the patch. If there is a need to make available() synchronized then this can be addressed under a different issue. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Wed Jul 3 22:55:46 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 3 Jul 2019 15:55:46 -0700 Subject: Redux 8184157: (ch) AsynchronousFileChannel hangs with internal error when reading locked file In-Reply-To: <7DA84564-0770-48E8-9AEA-B6D38C2F4B62@oracle.com> References: <8B15EB6A-2E66-42D9-8277-C25FB9279837@oracle.com> <7DA84564-0770-48E8-9AEA-B6D38C2F4B62@oracle.com> Message-ID: > On Jul 2, 2019, at 9:02 AM, Brian Burkhalter wrote: > >>> The Windows implementations of AsynchronousSocketChannel and AsynchronousServerSocketChannel also remove and free the OVERLAPPED structure then the socket operation completes immediately. Will they also need attention? >> >> Yes, at least AsynchronousSocketChannel: I should have caught that before. I?ll update the patch for these cases and look into the memory leak possibility mentioned above. > > AsynchronousSocketChannel needs a similar change but I think not AsynchronousServerSocketChannel. I do not have a reproducer for this case yet however. After re-reading the code while investigating how to write a reproducer for a similar AsynchronousSocketChannel failure, it does not appear that such a failure is possible. In the case of read or write on such a channel, read0() and write0() can return only one of two non-error condition codes: one for when the operation is pending, and another to indicate that the channel has been shut down. Only in the former cases is pending set to true. So the only possibilities for pending to be false are that the channel is either already shutdown or an error occurred while initiating the operation. In both of these cases the OVERLAPPED pointer is correctly removed from the PendingIoCache instance. Note that pending cannot be true if an exception occurred, so removing the OVERLAPPED pointer only on condition that pending is false is sufficient. >>> The test will need cleanup, maybe rename it to LockReadWriteStressTest or something that is closer to what it actually does. It can use try-with-resources to avoid leaking resources when it fails. I assume "f" and "l" will be renamed as they are unusual variables names in this context. Also the buffer usages is unusual and I can assume can be cleaned up too. >> >> The test is in effect the reproducer supplied in the issue so I did not give it any further attention yet. > > > The test and the other things above are updated here: > > http://cr.openjdk.java.net/~bpb/8184157/webrev.02/ An updated patch with the WindowsAsynchronousSocketChannelImpl change reverted is here: http://cr.openjdk.java.net/~bpb/8184157/webrev.03/ Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Jul 9 09:19:18 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 9 Jul 2019 10:19:18 +0100 Subject: 8227080: (fs) Files.newInputStream(...).skip(n) is slow In-Reply-To: <1025E944-0BB1-40A4-9211-D2AF10B5522D@oracle.com> References: <2F288856-2AF5-4CCC-83F5-E959B394C5AB@oracle.com> <0A099B1F-E77F-49A9-8A02-79A9B1ADBE2F@oracle.com> <1025E944-0BB1-40A4-9211-D2AF10B5522D@oracle.com> Message-ID: <5cd4eabd-5691-f236-bcb5-a2ee8152aa11@oracle.com> On 03/07/2019 22:49, Brian Burkhalter wrote: > > I went ahead and checked in the .01 version of the patch. If there is > a need to make available() synchronized then this can be addressed > under a different issue. I wasn't around for the review but one thing that needs to be re-examined is skipping beyond EOF as that is okay to do with files. FileInputStream.skip has a paragraph on this case, as does FileChannel.position, so probably best if the InputStream returned by Files.newInputStream is consistent. A minor nit but I don't think Files/Misc.java is the right place to add the test. Instead I think we should add a more comprehensive InputStreamTest that exercises all of the input stream methods. -Alan From GROEGES at uk.ibm.com Tue Jul 9 12:32:25 2019 From: GROEGES at uk.ibm.com (Steve Groeger) Date: Tue, 9 Jul 2019 13:32:25 +0100 Subject: adding rsockets support into JDK Message-ID: Hi all, Is the support for adding rsockets into the JDK still being worked on? The last webrev that Lucy mentioned in her posts was [1] back in Jan 2019 and this doesnt seem to apply cleanly to the latest jdk/jdk repository. I also had some issues running the jtreg tests which I am looking at trying to resolve but I didn't want to start work on this unless rsockets was continuing to be worked on! So if anyone has any current status on this I would be glad to hear it. [1] http://cr.openjdk.java.net/~ylu/8195160.27/ Thanks Steve Groeger IBM Runtime Technologies Hursley, Winchester Tel: (44) 1962 816911 Mobex: 279990 Mobile: 07718 517 129 Fax (44) 1962 816800 Lotus Notes: Steve Groeger/UK/IBM Internet: groeges at uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Jul 9 14:47:36 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 9 Jul 2019 15:47:36 +0100 Subject: adding rsockets support into JDK In-Reply-To: References: Message-ID: <5261008f-75dd-9d01-583c-be726234bfff@oracle.com> On 09/07/2019 13:32, Steve Groeger wrote: > Hi all, > > Is the support for adding rsockets into the JDK still being worked on? > > The last webrev that Lucy mentioned in her posts was [1] back in Jan > 2019 and this doesnt seem to apply cleanly to the latest jdk/jdk > repository. The project ran into a few issues with the Linux RDMA implementation. All the details are in the archives here, including links to the discussions on the linux-rdma kernel mailing list and the patches. There is interest in this feature so I expect it will resume again soon. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Wed Jul 10 20:39:59 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 10 Jul 2019 13:39:59 -0700 Subject: 8181493: (fs) Files.readAttributes(path, BasicFileAttributes.class) should preserve nano second time stamps Message-ID: <2D854054-B445-406B-8777-4D1DA2F95137@oracle.com> https://bugs.openjdk.java.net/browse/JDK-8181493 http://cr.openjdk.java.net/~bpb/8181493/webrev.00/ This patch would change: 1) UnixFileAttributeViews.setTimes() to set the access and modification times of a file or directory with nanosecond precision if the system call futimens() [1] is available, and 2) UnixFileAttributes to initialize the creation, last access, and last modifications FileTimes with nanosecond precision. New methods to support setting the nanosecond resolution access and modification times using futimens() are added to UnixNativeDispatcher. Note that unlike futimes() and lutimes(), futimens() is *not* guaranteed to be present for the _ALLBDS_SOURCE case and so it is looked up using dlsym(). A new test is added to run on Linux, macOS, and Solaris only. On macOS if the file system is HFS+, then the test exits without failing as HFS+ supports a file stamp resolution of only one second. If the macOS file system is APFS, then the test runs through to completion. Note that setting the creation time as requested in [2] appears problematic and unevenly supported on these Unix variants. Thanks, Brian [1] https://pubs.opengroup.org/onlinepubs/9699919799/functions/futimens.html [2] https://bugs.openjdk.java.net/browse/JDK-8151430 -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias.baesken at sap.com Tue Jul 16 15:04:19 2019 From: matthias.baesken at sap.com (Baesken, Matthias) Date: Tue, 16 Jul 2019 15:04:19 +0000 Subject: RFR : 8227737: avoid implicit-function-declaration on AIX Message-ID: Hello, please review the following AIX related change . It fixes a number of missing inclusions leading to implicit-function-declaration warnings when compiling with the recent xlc16 /xlclang . At various places in the native C coding in jdk, we miss header inclusions on AIX. This leads to warnings like : /nightly/jdk/src/java.base/unix/native/libjava/childproc.c:99:5: warning: implicitly declaring library function 'snprintf' with type 'int (char *, unsigned long, const char *, ...)' [-Wimplicit-function-declaration] snprintf(aix_fd_dir, 32, "/proc/%d/fd", getpid()); ^ /nightly/jdk/src/java.base/unix/native/libjava/childproc.c:99:5: note: include the header or explicitly provide a declaration for 'snprintf' or /nightly/jdk/src/java.base/aix/native/libjli/java_md_aix.c:38:5: warning: implicitly declaring library function 'memset' with type 'void *(void *, int, unsigned long)' [-Wimplicit-function-declaration] memset((void *)info, 0, sizeof(Dl_info)); /nightly/jdk/src/java.base/aix/native/libjli/java_md_aix.c:38:5: note: include the header or explicitly provide a declaration for 'memset' Bug/webrev : https://bugs.openjdk.java.net/browse/JDK-8227737 http://cr.openjdk.java.net/~mbaesken/webrevs/8227737.0/ Thanks, Matthias -------------- next part -------------- An HTML attachment was scrubbed... URL: From christoph.langer at sap.com Wed Jul 17 07:39:47 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Wed, 17 Jul 2019 07:39:47 +0000 Subject: RFR : 8227737: avoid implicit-function-declaration on AIX In-Reply-To: References: Message-ID: Hi Matthias, looks good, thanks for doing this. How far are we then from enabling "warnings as errors" on AIX? :P Best regards Christoph From: awt-dev On Behalf Of Baesken, Matthias Sent: Dienstag, 16. Juli 2019 17:04 To: Java Core Libs ; nio-dev at openjdk.java.net; net-dev at openjdk.java.net; awt-dev at openjdk.java.net Cc: 'ppc-aix-port-dev at openjdk.java.net' Subject: [CAUTION] RFR : 8227737: avoid implicit-function-declaration on AIX Hello, please review the following AIX related change . It fixes a number of missing inclusions leading to implicit-function-declaration warnings when compiling with the recent xlc16 /xlclang . At various places in the native C coding in jdk, we miss header inclusions on AIX. This leads to warnings like : /nightly/jdk/src/java.base/unix/native/libjava/childproc.c:99:5: warning: implicitly declaring library function 'snprintf' with type 'int (char *, unsigned long, const char *, ...)' [-Wimplicit-function-declaration] snprintf(aix_fd_dir, 32, "/proc/%d/fd", getpid()); ^ /nightly/jdk/src/java.base/unix/native/libjava/childproc.c:99:5: note: include the header or explicitly provide a declaration for 'snprintf' or /nightly/jdk/src/java.base/aix/native/libjli/java_md_aix.c:38:5: warning: implicitly declaring library function 'memset' with type 'void *(void *, int, unsigned long)' [-Wimplicit-function-declaration] memset((void *)info, 0, sizeof(Dl_info)); /nightly/jdk/src/java.base/aix/native/libjli/java_md_aix.c:38:5: note: include the header or explicitly provide a declaration for 'memset' Bug/webrev : https://bugs.openjdk.java.net/browse/JDK-8227737 http://cr.openjdk.java.net/~mbaesken/webrevs/8227737.0/ Thanks, Matthias -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias.baesken at sap.com Wed Jul 17 07:55:38 2019 From: matthias.baesken at sap.com (Baesken, Matthias) Date: Wed, 17 Jul 2019 07:55:38 +0000 Subject: RFR : 8227737: avoid implicit-function-declaration on AIX In-Reply-To: References: Message-ID: Thanks ! * How far are we then from enabling "warnings as errors" on AIX? :P * I count in the opt/product jdk/jdk AIX build currently 60 remaining matches of compiler warnings . Plus additionally a number of those (probably linker warnings ?) : 1500-029: (W) WARNING: subprogram ... could not be inline Guess we have to suppress / ignore at least most of the linker inlining warnings (we ignore them for a decade IMHO). >From the 60 remaining compiler warnings I mentioned are ~ 30 of type Wformat : warning: format specifies type 'int *' but the argument has type 'int' [-Wformat] warning: format specifies type 'unsigned long long' but the argument has type 'size_t' (aka 'unsigned long') [-Wformat] Guess those can the easily fixed (and should be fixed). Regards, Matthias From: Langer, Christoph Sent: Mittwoch, 17. Juli 2019 09:40 To: Baesken, Matthias ; Java Core Libs ; nio-dev at openjdk.java.net; net-dev at openjdk.java.net; awt-dev at openjdk.java.net Cc: 'ppc-aix-port-dev at openjdk.java.net' Subject: RE: RFR : 8227737: avoid implicit-function-declaration on AIX Hi Matthias, looks good, thanks for doing this. How far are we then from enabling "warnings as errors" on AIX? :P Best regards Christoph From: awt-dev > On Behalf Of Baesken, Matthias Sent: Dienstag, 16. Juli 2019 17:04 To: Java Core Libs >; nio-dev at openjdk.java.net; net-dev at openjdk.java.net; awt-dev at openjdk.java.net Cc: 'ppc-aix-port-dev at openjdk.java.net' > Subject: [CAUTION] RFR : 8227737: avoid implicit-function-declaration on AIX Hello, please review the following AIX related change . It fixes a number of missing inclusions leading to implicit-function-declaration warnings when compiling with the recent xlc16 /xlclang . At various places in the native C coding in jdk, we miss header inclusions on AIX. This leads to warnings like : /nightly/jdk/src/java.base/unix/native/libjava/childproc.c:99:5: warning: implicitly declaring library function 'snprintf' with type 'int (char *, unsigned long, const char *, ...)' [-Wimplicit-function-declaration] snprintf(aix_fd_dir, 32, "/proc/%d/fd", getpid()); ^ /nightly/jdk/src/java.base/unix/native/libjava/childproc.c:99:5: note: include the header or explicitly provide a declaration for 'snprintf' or /nightly/jdk/src/java.base/aix/native/libjli/java_md_aix.c:38:5: warning: implicitly declaring library function 'memset' with type 'void *(void *, int, unsigned long)' [-Wimplicit-function-declaration] memset((void *)info, 0, sizeof(Dl_info)); /nightly/jdk/src/java.base/aix/native/libjli/java_md_aix.c:38:5: note: include the header or explicitly provide a declaration for 'memset' Bug/webrev : https://bugs.openjdk.java.net/browse/JDK-8227737 http://cr.openjdk.java.net/~mbaesken/webrevs/8227737.0/ Thanks, Matthias -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Jul 18 14:08:22 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 18 Jul 2019 15:08:22 +0100 Subject: 8181493: (fs) Files.readAttributes(path, BasicFileAttributes.class) should preserve nano second time stamps In-Reply-To: <2D854054-B445-406B-8777-4D1DA2F95137@oracle.com> References: <2D854054-B445-406B-8777-4D1DA2F95137@oracle.com> Message-ID: On 10/07/2019 21:39, Brian Burkhalter wrote: > https://bugs.openjdk.java.net/browse/JDK-8181493 > http://cr.openjdk.java.net/~bpb/8181493/webrev.00/ > > This patch would change: > > 1) UnixFileAttributeViews.setTimes() to set the access and > modification times of a file or directory with nanosecond precision if > the system call futimens() [1] is available, and > 2) UnixFileAttributes to initialize the creation, last access, and > last modifications FileTimes with nanosecond precision. > > New methods to support setting the nanosecond resolution access and > modification times using futimens() are added to UnixNativeDispatcher. > Note that unlike futimes() and lutimes(), futimens() is *not* > guaranteed to be present for the _ALLBDS_SOURCE case and so it is > looked up using dlsym(). The implementation changes mostly look okay, just wondering if toFileTime should be simplified for the NS case to use FileTime.from(Instant.ofEpochSecond(sec, nsec)). Maybe you've tried this already and are using the Math methods to avoid object allocation? Can the test using its working directory instead of tmpfs and deleteOnExit. Tests for timestamps on files can take effort to be reliable so it would be useful to not delete the crumbs. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Thu Jul 18 15:59:35 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 18 Jul 2019 08:59:35 -0700 Subject: 8181493: (fs) Files.readAttributes(path, BasicFileAttributes.class) should preserve nano second time stamps In-Reply-To: References: <2D854054-B445-406B-8777-4D1DA2F95137@oracle.com> Message-ID: <97F6E4AD-3802-4A2C-8043-3158DA323B37@oracle.com> > On Jul 18, 2019, at 7:08 AM, Alan Bateman wrote: > > On 10/07/2019 21:39, Brian Burkhalter wrote: >> https://bugs.openjdk.java.net/browse/JDK-8181493 >> http://cr.openjdk.java.net/~bpb/8181493/webrev.00/ >> >> This patch would change: >> >> 1) UnixFileAttributeViews.setTimes() to set the access and modification times of a file or directory with nanosecond precision if the system call futimens() [1] is available, and >> 2) UnixFileAttributes to initialize the creation, last access, and last modifications FileTimes with nanosecond precision. >> >> New methods to support setting the nanosecond resolution access and modification times using futimens() are added to UnixNativeDispatcher. Note that unlike futimes() and lutimes(), futimens() is *not* guaranteed to be present for the _ALLBDS_SOURCE case and so it is looked up using dlsym(). > The implementation changes mostly look okay, just wondering if toFileTime should be simplified for the NS case to use FileTime.from(Instant.ofEpochSecond(sec, nsec)). Maybe you've tried this already and are using the Math methods to avoid object allocation? Yes, it avoids an allocation. > Can the test using its working directory instead of tmpfs and deleteOnExit. Tests for timestamps on files can take effort to be reliable so it would be useful to not delete the crumbs. Here?s a prospective change: --- a/test/jdk/java/nio/file/attribute/BasicFileAttributeView/SetTimesNanos.java +++ b/test/jdk/java/nio/file/attribute/BasicFileAttributeView/SetTimesNanos.java @@ -45,8 +45,8 @@ public static void main(String[] args) throws IOException, InterruptedException { - Path dir = Files.createTempDirectory("test"); - dir.toFile().deleteOnExit(); + Path dirPath = Path.of(System.getProperty("test.dir", "."), "test"); + Path dir = Files.createDirectory(dirPath); FileStore store = Files.getFileStore(dir); System.err.format("FileStore: %s on %s (%s)%n", dir, store.name(), store.type()); Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From lance.andersen at oracle.com Thu Jul 18 22:29:54 2019 From: lance.andersen at oracle.com (Lance Andersen) Date: Thu, 18 Jul 2019 18:29:54 -0400 Subject: 8181493: (fs) Files.readAttributes(path, BasicFileAttributes.class) should preserve nano second time stamps In-Reply-To: <97F6E4AD-3802-4A2C-8043-3158DA323B37@oracle.com> References: <2D854054-B445-406B-8777-4D1DA2F95137@oracle.com> <97F6E4AD-3802-4A2C-8043-3158DA323B37@oracle.com> Message-ID: <4E9FCC89-9743-4EE5-A7E1-7CF238783E8B@oracle.com> HI Brian, > On Jul 18, 2019, at 11:59 AM, Brian Burkhalter wrote: > >> >> On Jul 18, 2019, at 7:08 AM, Alan Bateman > wrote: >> >> On 10/07/2019 21:39, Brian Burkhalter wrote: >>> https://bugs.openjdk.java.net/browse/JDK-8181493 >>> http://cr.openjdk.java.net/~bpb/8181493/webrev.00/ >>> >>> This patch would change: >>> >>> 1) UnixFileAttributeViews.setTimes() to set the access and modification times of a file or directory with nanosecond precision if the system call futimens() [1] is available, and >>> 2) UnixFileAttributes to initialize the creation, last access, and last modifications FileTimes with nanosecond precision. >>> >>> New methods to support setting the nanosecond resolution access and modification times using futimens() are added to UnixNativeDispatcher. Note that unlike futimes() and lutimes(), futimens() is *not* guaranteed to be present for the _ALLBDS_SOURCE case and so it is looked up using dlsym(). >> The implementation changes mostly look okay, just wondering if toFileTime should be simplified for the NS case to use FileTime.from(Instant.ofEpochSecond(sec, nsec)). Maybe you've tried this already and are using the Math methods to avoid object allocation? > > Yes, it avoids an allocation. > >> Can the test using its working directory instead of tmpfs and deleteOnExit. Tests for timestamps on files can take effort to be reliable so it would be useful to not delete the crumbs. > > Here?s a prospective change: > > --- a/test/jdk/java/nio/file/attribute/BasicFileAttributeView/SetTimesNanos.java > +++ b/test/jdk/java/nio/file/attribute/BasicFileAttributeView/SetTimesNanos.java > @@ -45,8 +45,8 @@ > public static void main(String[] args) throws IOException, > InterruptedException { > > - Path dir = Files.createTempDirectory("test"); > - dir.toFile().deleteOnExit(); > + Path dirPath = Path.of(System.getProperty("test.dir", "."), "test?); You should just be able to omit the system property: http://openjdk.java.net/jtreg/faq.html#scratch-directory ???? 3.4 What is a scratch directory? When jtreg executes a test, the current directory for the test is set to a scratch directory so that the test can easily write any temporary files. jtreg will ensure that the directory is always empty when the test begins, so that the test does not have to worry about deleting any pre-existing files. These directories are placed in the work directory. ------------- HTH Best Lance > + Path dir = Files.createDirectory(dirPath); > FileStore store = Files.getFileStore(dir); > System.err.format("FileStore: %s on %s (%s)%n", dir, store.name(), > store.type()); > > Thanks, > > Brian Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: oracle_sig_logo.gif Type: image/gif Size: 658 bytes Desc: not available URL: From linus.fernandes at gmail.com Fri Jul 19 03:46:17 2019 From: linus.fernandes at gmail.com (LINUS FERNANDES) Date: Fri, 19 Jul 2019 09:16:17 +0530 Subject: Fwd: (so) ServerSocketChannel::supportedOptions includes IP_TOS In-Reply-To: References: Message-ID: ---------- Forwarded message --------- From: LINUS FERNANDES Date: Thu, 18 Jul 2019, 19:04 Subject: (so) ServerSocketChannel::supportedOptions includes IP_TOS To: http://mail.openjdk.java.net/pipermail/nio-dev/2018-August/005365.html The above bug does not appear to have been resolved satisfactorily or I don't have access to the fixed Java version. import java.io.*; import java.net.*; import java.nio.channels.*; public class OptionSupport { public static void main(String[] args) throws IOException { printOptions(SocketChannel.open()); printOptions(ServerSocketChannel.open()); printOptions(AsynchronousSocketChannel.open()); printOptions(AsynchronousServerSocketChannel.open()); printOptions(DatagramChannel.open()); } private static void printOptions(NetworkChannel channel) throws IOException { System.out.println(channel.getClass().getSimpleName() + " supports:"); for (SocketOption option : channel.supportedOptions()) { System.out.println(option.name() + ": " + channel.getOption(option)); } System.out.println(); channel.close(); } } Running the above program throws the same error for Openjdk: java -version openjdk version "10.0.2" 2018-07-17 OpenJDK Runtime Environment (build 10.0.2+13) OpenJDK 64-Bit Server VM (build 10.0.2+13, mixed mode) The date of the JDK version is earlier than the resolution of the error. https://bugs.openjdk.java.net/browse/JDK-8209152 I'm running Java on Arch Linux for Termux. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Fri Jul 19 06:06:58 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 19 Jul 2019 07:06:58 +0100 Subject: Fwd: (so) ServerSocketChannel::supportedOptions includes IP_TOS In-Reply-To: References: Message-ID: <41ef94b9-1ae8-ec9b-f3ae-2bc47f402ccc@oracle.com> On 19/07/2019 04:46, LINUS FERNANDES wrote: > > : > > The date of the JDK version is earlier than the resolution of the error. > > https://bugs.openjdk.java.net/browse/JDK-8209152 > > The issue was fixed in JDK 12, you seem to be using a JDK 10 build. Can you re-run your test with a JDK 12 (or JDK 13 EA) build? -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From linus.fernandes at gmail.com Fri Jul 19 06:21:06 2019 From: linus.fernandes at gmail.com (LINUS FERNANDES) Date: Fri, 19 Jul 2019 11:51:06 +0530 Subject: Fwd: (so) ServerSocketChannel::supportedOptions includes IP_TOS In-Reply-To: <41ef94b9-1ae8-ec9b-f3ae-2bc47f402ccc@oracle.com> References: <41ef94b9-1ae8-ec9b-f3ae-2bc47f402ccc@oracle.com> Message-ID: Not on Arch Linux on Termux. I don't doubt that it's fixed in Version 12. On Fri, 19 Jul 2019, 11:37 Alan Bateman, wrote: > On 19/07/2019 04:46, LINUS FERNANDES wrote: > > > : > > The date of the JDK version is earlier than the resolution of the error. > > https://bugs.openjdk.java.net/browse/JDK-8209152 > > > The issue was fixed in JDK 12, you seem to be using a JDK 10 build. Can > you re-run your test with a JDK 12 (or JDK 13 EA) build? > > -Alan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias.baesken at sap.com Fri Jul 19 07:50:03 2019 From: matthias.baesken at sap.com (Baesken, Matthias) Date: Fri, 19 Jul 2019 07:50:03 +0000 Subject: RFR : 8227737: avoid implicit-function-declaration on AIX In-Reply-To: References: Message-ID: Thanks for the additional review ! Best regards, Matthias From: Lindenmaier, Goetz Sent: Freitag, 19. Juli 2019 09:33 To: Baesken, Matthias ; Schmidt, Lutz ; Doerr, Martin Subject: RE: RFR : 8227737: avoid implicit-function-declaration on AIX Hi Matthias, This looks good, thanks for doing this. Best regards, Goetz. From: Langer, Christoph Sent: Mittwoch, 17. Juli 2019 09:40 To: Baesken, Matthias >; Java Core Libs >; nio-dev at openjdk.java.net; net-dev at openjdk.java.net; awt-dev at openjdk.java.net Cc: 'ppc-aix-port-dev at openjdk.java.net' > Subject: RE: RFR : 8227737: avoid implicit-function-declaration on AIX Hi Matthias, looks good, thanks for doing this. How far are we then from enabling "warnings as errors" on AIX? :P Best regards Christoph From: awt-dev > On Behalf Of Baesken, Matthias Sent: Dienstag, 16. Juli 2019 17:04 To: Java Core Libs >; nio-dev at openjdk.java.net; net-dev at openjdk.java.net; awt-dev at openjdk.java.net Cc: 'ppc-aix-port-dev at openjdk.java.net' > Subject: [CAUTION] RFR : 8227737: avoid implicit-function-declaration on AIX Hello, please review the following AIX related change . It fixes a number of missing inclusions leading to implicit-function-declaration warnings when compiling with the recent xlc16 /xlclang . At various places in the native C coding in jdk, we miss header inclusions on AIX. This leads to warnings like : /nightly/jdk/src/java.base/unix/native/libjava/childproc.c:99:5: warning: implicitly declaring library function 'snprintf' with type 'int (char *, unsigned long, const char *, ...)' [-Wimplicit-function-declaration] snprintf(aix_fd_dir, 32, "/proc/%d/fd", getpid()); ^ /nightly/jdk/src/java.base/unix/native/libjava/childproc.c:99:5: note: include the header or explicitly provide a declaration for 'snprintf' or /nightly/jdk/src/java.base/aix/native/libjli/java_md_aix.c:38:5: warning: implicitly declaring library function 'memset' with type 'void *(void *, int, unsigned long)' [-Wimplicit-function-declaration] memset((void *)info, 0, sizeof(Dl_info)); /nightly/jdk/src/java.base/aix/native/libjli/java_md_aix.c:38:5: note: include the header or explicitly provide a declaration for 'memset' Bug/webrev : https://bugs.openjdk.java.net/browse/JDK-8227737 http://cr.openjdk.java.net/~mbaesken/webrevs/8227737.0/ Thanks, Matthias -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Fri Jul 19 11:14:29 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 19 Jul 2019 12:14:29 +0100 Subject: 8181493: (fs) Files.readAttributes(path, BasicFileAttributes.class) should preserve nano second time stamps In-Reply-To: <97F6E4AD-3802-4A2C-8043-3158DA323B37@oracle.com> References: <2D854054-B445-406B-8777-4D1DA2F95137@oracle.com> <97F6E4AD-3802-4A2C-8043-3158DA323B37@oracle.com> Message-ID: On 18/07/2019 16:59, Brian Burkhalter wrote: > > > Yes, it avoids an allocation. Okay. On the test then it would be better to use the scratch directory as Lance suggests. Also would be useful to have it run a few hundred times on all platforms before publish to be confident that it is reliable (we've had a lot of issues with tests for file time stamps as you know). -Alan From brian.burkhalter at oracle.com Fri Jul 19 14:59:33 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 19 Jul 2019 07:59:33 -0700 Subject: 8181493: (fs) Files.readAttributes(path, BasicFileAttributes.class) should preserve nano second time stamps In-Reply-To: References: <2D854054-B445-406B-8777-4D1DA2F95137@oracle.com> <97F6E4AD-3802-4A2C-8043-3158DA323B37@oracle.com> Message-ID: <88BB88A0-AE86-478F-8B9C-EA45F4B7A8C1@oracle.com> > On Jul 19, 2019, at 4:14 AM, Alan Bateman wrote: > > On the test then it would be better to use the scratch directory as Lance suggests. Also would be useful to have it run a few hundred times on all platforms before publish to be confident that it is reliable (we've had a lot of issues with tests for file time stamps as you know). OK I will update the test and do a large number of test runs before pushing this. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Fri Jul 19 15:52:48 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 19 Jul 2019 08:52:48 -0700 Subject: 8181493: (fs) Files.readAttributes(path, BasicFileAttributes.class) should preserve nano second time stamps In-Reply-To: <88BB88A0-AE86-478F-8B9C-EA45F4B7A8C1@oracle.com> References: <2D854054-B445-406B-8777-4D1DA2F95137@oracle.com> <97F6E4AD-3802-4A2C-8043-3158DA323B37@oracle.com> <88BB88A0-AE86-478F-8B9C-EA45F4B7A8C1@oracle.com> Message-ID: <018859AB-83DD-4704-A92A-8FFB50B35970@oracle.com> > On Jul 19, 2019, at 7:59 AM, Brian Burkhalter wrote: > >> On Jul 19, 2019, at 4:14 AM, Alan Bateman > wrote: >> >> On the test then it would be better to use the scratch directory as Lance suggests. Also would be useful to have it run a few hundred times on all platforms before publish to be confident that it is reliable (we've had a lot of issues with tests for file time stamps as you know). > > OK I will update the test and do a large number of test runs before pushing this. Test updated as --- a/test/jdk/java/nio/file/attribute/BasicFileAttributeView/SetTimesNanos.java +++ b/test/jdk/java/nio/file/attribute/BasicFileAttributeView/SetTimesNanos.java @@ -45,10 +45,10 @@ public static void main(String[] args) throws IOException, InterruptedException { - Path dirPath = Path.of(System.getProperty("test.dir", "."), "test"); + Path dirPath = Path.of("test"); Path dir = Files.createDirectory(dirPath); FileStore store = Files.getFileStore(dir); - System.err.format("FileStore: %s on %s (%s)%n", dir, store.name(), + System.out.format("FileStore: %s on %s (%s)%n", dir, store.name(), store.type()); if (System.getProperty("os.name").toLowerCase().startsWith("mac") && store.name().equalsIgnoreCase("hfs")) { @@ -58,7 +58,7 @@ } testNanos(dir); - Path file = Files.createTempFile(dir, "test", "test"); + Path file = Files.createFile(dir.resolve("test.dat")); testNanos(file); } Build-and-test job submitted; some test-only jobs to follow. Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Fri Jul 19 16:10:43 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 19 Jul 2019 17:10:43 +0100 Subject: Unix domain sockets (UDS, AF_UNIX) in System.inheritedChannel() and elsewhere in Java In-Reply-To: <5e31f8ab-9007-a501-1704-935e79f0133f@frantovo.cz> References: <5e31f8ab-9007-a501-1704-935e79f0133f@frantovo.cz> Message-ID: <4b57b5c2-afea-3ea9-73e2-0e3913ca6ef7@oracle.com> On 19/07/2019 16:35, Franti?ek Ku?era wrote: > > Hello, > > I am interested in the unix domain sockets (UDS, AF_UNIX) support in Java. > > Java is able to inherit a channel from the parent process and access > it through System.inheritedChannel(). The channel is passed as an open > file descriptor (FD) to Java form the parent process which is usually > some superserver (like xinetd or systemd) or some other loader*. The > channel might be both a ServerSocketChannel (accept() is called in > Java) and a SocketChannel (accept() is called once in the parent > process) or a DatagramChannel. And it might be TCP, UDP or UDS. But > the UDS support is a bit incomplete. > > I want to make the UDS support better so I wrote a proof-of-concept > patch ? see attached .diff and webrev: > http://frantovo.cz/disk/openjdk-uds-08/ It still requires a lot of > work (clean-up, refactoring, portability, security manager, > documentation, testing?) but it seems feasible to me. > > What already works in Java: > > * listening on an inherited UDS channel (ServerSocketChannel) ? e.g. > Jetty or Tomcat are able to listen and serve HTTP requests on UDS > instead of TCP > * receiving datagrams from UDS > > What my proof-of-concept patch adds: > > * class UnixSocketAddress extends SocketAddress > * support also for a single inherited UDS connection (SocketChannel) > * getting correct local and remote addresses (instead of an > InetAddress with some random IP and port) > * creating new server and client UDS channels from Java > > I am looking for a sponsor/mentor who will help me finish this task > through wise advice and peer review. > > My further plans: > > * finish the datagram part (sending responses to UDS does not work yet) > * allow inheritance of multiple channels/sockets > * getting client credentials (user name and group ? where available) > * maybe passing FDs through UDS (where available) > * maybe FileChannel for accessing ordinary files inherited from the > parent as FDs > This is something that we've prototyped several times over the years but never came to a conclusion on whether to attempt to include it or not. No objection to exploring it again but I think it should be prototyped as a JDK-specific API, probably jdk.net module. This is the approach we have been discussing on nio-dev for the RDMA socket API. In terms of API then introducing a SocketAddress sub-class for Unix domain sockets looks right, although it should probably be created with a Path or String rather than a byte array. For the RDMA sockets we have a factory class to create SocketChannel or ServerSocketChannels that will probably work here too. Windows and pipes is something that is worth discussing too as we didn't do anything on that in previous explorations. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Fri Jul 19 16:15:49 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 19 Jul 2019 17:15:49 +0100 Subject: 8181493: (fs) Files.readAttributes(path, BasicFileAttributes.class) should preserve nano second time stamps In-Reply-To: <018859AB-83DD-4704-A92A-8FFB50B35970@oracle.com> References: <2D854054-B445-406B-8777-4D1DA2F95137@oracle.com> <97F6E4AD-3802-4A2C-8043-3158DA323B37@oracle.com> <88BB88A0-AE86-478F-8B9C-EA45F4B7A8C1@oracle.com> <018859AB-83DD-4704-A92A-8FFB50B35970@oracle.com> Message-ID: <8c4c18d5-2eda-861f-4e9a-b431c59a9556@oracle.com> On 19/07/2019 16:52, Brian Burkhalter wrote: > > > Build-and-test job submitted; some test-only jobs to follow. > Thanks, if you are confident that the test is bullet proof then you should be good to go. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From lance.andersen at oracle.com Fri Jul 19 16:17:26 2019 From: lance.andersen at oracle.com (Lance Andersen) Date: Fri, 19 Jul 2019 12:17:26 -0400 Subject: 8181493: (fs) Files.readAttributes(path, BasicFileAttributes.class) should preserve nano second time stamps In-Reply-To: <018859AB-83DD-4704-A92A-8FFB50B35970@oracle.com> References: <2D854054-B445-406B-8777-4D1DA2F95137@oracle.com> <97F6E4AD-3802-4A2C-8043-3158DA323B37@oracle.com> <88BB88A0-AE86-478F-8B9C-EA45F4B7A8C1@oracle.com> <018859AB-83DD-4704-A92A-8FFB50B35970@oracle.com> Message-ID: <017248E2-F9F0-429C-B8A3-B857594879F5@oracle.com> Hi Brian > On Jul 19, 2019, at 11:52 AM, Brian Burkhalter wrote: > > >> On Jul 19, 2019, at 7:59 AM, Brian Burkhalter > wrote: >> >>> On Jul 19, 2019, at 4:14 AM, Alan Bateman > wrote: >>> >>> On the test then it would be better to use the scratch directory as Lance suggests. Also would be useful to have it run a few hundred times on all platforms before publish to be confident that it is reliable (we've had a lot of issues with tests for file time stamps as you know). >> >> OK I will update the test and do a large number of test runs before pushing this. > > Test updated as > > --- a/test/jdk/java/nio/file/attribute/BasicFileAttributeView/SetTimesNanos.java > +++ b/test/jdk/java/nio/file/attribute/BasicFileAttributeView/SetTimesNanos.java > @@ -45,10 +45,10 @@ > public static void main(String[] args) throws IOException, > InterruptedException { > > - Path dirPath = Path.of(System.getProperty("test.dir", "."), "test"); > + Path dirPath = Path.of("test"); > Path dir = Files.createDirectory(dirPath); > FileStore store = Files.getFileStore(dir); > - System.err.format("FileStore: %s on %s (%s)%n", dir, store.name(), > + System.out.format("FileStore: %s on %s (%s)%n", dir, store.name(), > store.type()); > if (System.getProperty("os.name").toLowerCase().startsWith("mac") && > store.name().equalsIgnoreCase("hfs")) { > @@ -58,7 +58,7 @@ > } > testNanos(dir); > > - Path file = Files.createTempFile(dir, "test", "test"); > + Path file = Files.createFile(dir.resolve("test.dat")); > testNanos(file); > } The update above looks good > > Build-and-test job submitted; some test-only jobs to follow. > > Brian Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: oracle_sig_logo.gif Type: image/gif Size: 658 bytes Desc: not available URL: From Alan.Bateman at oracle.com Sat Jul 20 14:26:38 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 20 Jul 2019 15:26:38 +0100 Subject: Unix domain sockets (UDS, AF_UNIX) in System.inheritedChannel() and elsewhere in Java In-Reply-To: <4b57b5c2-afea-3ea9-73e2-0e3913ca6ef7@oracle.com> References: <5e31f8ab-9007-a501-1704-935e79f0133f@frantovo.cz> <4b57b5c2-afea-3ea9-73e2-0e3913ca6ef7@oracle.com> Message-ID: On 19/07/2019 17:10, Alan Bateman wrote: > : > > No objection to exploring it again but I think it should be prototyped > as a JDK-specific API, probably jdk.net module. This is the approach > we have been discussing on nio-dev for the RDMA socket API. In terms > of API then introducing a SocketAddress sub-class for Unix domain > sockets looks right, although it should probably be created with a > Path or String rather than a byte array. For the RDMA sockets we have > a factory class to create SocketChannel or ServerSocketChannels that > will probably work here too. One other option to point out is the jdk.nio.Channels API which can be used to create a SelectableChannel to a file descriptor. This may be used to experiment with too as it would allow you to create a channel that you could use with the existing selector implementations. -Alan. From franta-java at frantovo.cz Sun Jul 21 10:01:25 2019 From: franta-java at frantovo.cz (=?UTF-8?B?RnJhbnRpxaFlayBLdcSNZXJh?=) Date: Sun, 21 Jul 2019 12:01:25 +0200 Subject: Unix domain sockets (UDS, AF_UNIX) in System.inheritedChannel() and elsewhere in Java In-Reply-To: <4b57b5c2-afea-3ea9-73e2-0e3913ca6ef7@oracle.com> References: <5e31f8ab-9007-a501-1704-935e79f0133f@frantovo.cz> <4b57b5c2-afea-3ea9-73e2-0e3913ca6ef7@oracle.com> Message-ID: <0aa6e746-0a29-53ba-27dd-e36ad71e3b26@frantovo.cz> Dne 19. 07. 19 v 18:10 Alan Bateman napsal(a): > This is something that we've prototyped several times over the years > but never came to a conclusion on whether to attempt to include it or not Albeit decades after other operating systems :-) even MS Windows are now able to work with unix domain sockets. So current situation is IMHO different and UDS could be considered to be fully multiplatform technology comparable to TCP or UDP. > In terms of API then introducing a SocketAddress sub-class for Unix > domain sockets looks right, although it should probably be created > with a Path or String rather than a byte array. I would also prefer String or Path in the Java API but sockaddr_un.sun_path is defined in the POSIX standard as a byte array and it might be not only a file path but there are also "abstract" paths which have no representation on the filesystem ? they start with \0 null-byte and the abstract path is the whole rest of the array. So lossless mapping between byte arrays and strings will be difficult or maybe impossible (the "@" character is sometimes used in the UI instead of the \0 but "@" is also valid character on the filesystem path, so there are ambiguities).? Although Java Sting can hold the \0 bytes, depending on local encoding there might be edge cases where the path is a valid byte array (sun_path) and is addressable from the POSIX API but is not a valid String in given encoding. Maybe there might be subclasses for abstract and non-abstract addresses... but in the prototype, I looked for the simplest solution (but buggy, I know, it e.g. includes all the \0s in the toString() representation). > No objection to exploring it again but I think it should be prototyped > as a JDK-specific API, probably jdk.net module. I can implement it as a module, but I need help with some design issues: The System.inheritedChannel() is in the Java base ? there might be some indirect/optional dependency from the base to the module. Idea: create an SPI interface with methods canCreateChannel(fd) and createChannel(fd); the new module will implement this interface for AF_UNIX file descriptors; and the System.inheritedChannel() will lookup implementations of this interface and try to get the channel from them; and if no implementation will return a channel, it will fallback to the current implementation. Is it possible solution? May the java.net.UnixSocketAddress and java.net.StandardProtocolFamily.UNIX stay where they are? My primary goal was to fix the wrong addresses (random IP and port) returned from System.inheritedChannel() but then I also implemented the UDS client/server creation from java? These tasks might be solved separately ? but then: is it possible to have also an SPI lookup in case of creating new UDS sockets? I would be happy if this will be possible: ServerSocketChannel server = ServerSocketChannel.open(StandardProtocolFamily.UNIX); UnixSocketAddress serverAddress = new UnixSocketAddress("/run/my-unix-socket"); server.bind(serverAddress); i.e. the UDS implementation will be separated in a module, but from the application point-of-view, it will be accessible through standard API (same way as TCP or UDP) and an internal SPI lookup. Franta From Alan.Bateman at oracle.com Sun Jul 21 13:15:46 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sun, 21 Jul 2019 14:15:46 +0100 Subject: Redux 8184157: (ch) AsynchronousFileChannel hangs with internal error when reading locked file In-Reply-To: References: <8B15EB6A-2E66-42D9-8277-C25FB9279837@oracle.com> <7DA84564-0770-48E8-9AEA-B6D38C2F4B62@oracle.com> Message-ID: <5be52b83-1fd2-ab51-69a2-4cee8d9ad5fb@oracle.com> On 03/07/2019 23:55, Brian Burkhalter wrote: > : > > After re-reading the code while investigating how to write a > reproducer for a similar AsynchronousSocketChannel failure, it does > not appear that such a failure is possible. In the case of read or > write on such a channel, read0() and write0() can return only one of > two non-error condition codes: one for when the operation is pending, > and another to indicate that the channel has been shut down. Only in > the former cases is pending set to true. So the only possibilities for > pending to be false are that the channel is either already shutdown or > an error occurred while initiating the operation. In both of these > cases the OVERLAPPED pointer is correctly removed from the > PendingIoCache instance. Note that pending cannot be true if an > exception occurred, so removing the OVERLAPPED pointer only on > condition that pending is false is sufficient. > I went through webrev.03, where the change to WindowsAsynchronousSocketChannelImpl is partial reverted, and I think it's looking good except for LockTask.pending as that shouldn't be needed now. I see the test has been cleaned up (thanks), but wondering if L51 to close the channel can be removed as it should be closed by way of the try-with-resources. Also would be nice to avoid the line break at L45. The Files.delete at L55 is probably not needed either. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Sun Jul 21 18:13:35 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sun, 21 Jul 2019 19:13:35 +0100 Subject: Unix domain sockets (UDS, AF_UNIX) in System.inheritedChannel() and elsewhere in Java In-Reply-To: <0aa6e746-0a29-53ba-27dd-e36ad71e3b26@frantovo.cz> References: <5e31f8ab-9007-a501-1704-935e79f0133f@frantovo.cz> <4b57b5c2-afea-3ea9-73e2-0e3913ca6ef7@oracle.com> <0aa6e746-0a29-53ba-27dd-e36ad71e3b26@frantovo.cz> Message-ID: On 21/07/2019 11:01, Franti?ek Ku?era wrote: > I would also prefer String or Path in the Java API but > sockaddr_un.sun_path is defined in the POSIX standard as a byte array > and it might be not only a file path but there are also "abstract" > paths which have no representation on the filesystem ? they start with > \0 null-byte and the abstract path is the whole rest of the array. So > lossless mapping between byte arrays and strings will be difficult or > maybe impossible (the "@" character is sometimes used in the UI > instead of the \0 but "@" is also valid character on the filesystem > path, so there are ambiguities).? Although Java Sting can hold the \0 > bytes, depending on local encoding there might be edge cases where the > path is a valid byte array (sun_path) and is addressable from the > POSIX API but is not a valid String in given encoding. Maybe there > might be subclasses for abstract and non-abstract addresses... but in > the prototype, I looked for the simplest solution (but buggy, I know, > it e.g. includes all the \0s in the toString() representation). The implementation of Path on Unix systems uses bytes so will preserve the original file path, you just need to be careful to avoid Path -> String round trips. > : > > The System.inheritedChannel() is in the Java base ? there might be > some indirect/optional dependency from the base to the module. Idea: > create an SPI interface with methods canCreateChannel(fd) and > createChannel(fd); the new module will implement this interface for > AF_UNIX file descriptors; and the System.inheritedChannel() will > lookup implementations of this interface and try to get the channel > from them; and if no implementation will return a channel, it will > fallback to the current implementation. Is it possible solution? > > May the java.net.UnixSocketAddress and > java.net.StandardProtocolFamily.UNIX stay where they are? For the prototype then you should be able to put UnixSocketAddress and UnixProtocolFamily in jdk.net too. On naming, one of the discussion points will be whether it should be LocalSocketAddress as PF_UNIX is deprecated in favor of PF_LOCAL on some systems. A bigger discussion point will be whether to introduce new channel types rather than trying to have alternative implementations of SocketChannel/ServerSocketChannel. The approach that we've been taking with the RDMA API is to use the existing API and have the factory methods that that create the RDMA socket channel specify the differences. > > My primary goal was to fix the wrong addresses (random IP and port) > returned from System.inheritedChannel() If fd 0 is a Unix domain socket then inheritedChannel should return null. Are you sure you it's returning a SocketChannel to something that is not a stream or datagram socket? If so then then this is a bug that should be looked into. -Alan From franta-java at frantovo.cz Mon Jul 22 08:16:06 2019 From: franta-java at frantovo.cz (=?UTF-8?B?RnJhbnRpxaFlayBLdcSNZXJh?=) Date: Mon, 22 Jul 2019 10:16:06 +0200 Subject: Unix domain sockets (UDS, AF_UNIX) in System.inheritedChannel() and elsewhere in Java In-Reply-To: References: <5e31f8ab-9007-a501-1704-935e79f0133f@frantovo.cz> <4b57b5c2-afea-3ea9-73e2-0e3913ca6ef7@oracle.com> <0aa6e746-0a29-53ba-27dd-e36ad71e3b26@frantovo.cz> Message-ID: <75b60535-ab55-2e3c-d283-a36b78e4175c@frantovo.cz> Dne 21. 07. 19 v 20:13 Alan Bateman napsal(a): > On 21/07/2019 11:01, Franti?ek Ku?era wrote: >> I would also prefer String or Path in the Java API but >> sockaddr_un.sun_path is defined in the POSIX standard as a byte array >> and it might be not only a file path but there are also "abstract" >> paths which have no representation on the filesystem ? they start >> with \0 null-byte and the abstract path is the whole rest of the >> array. So lossless mapping between byte arrays and strings will be >> difficult or maybe impossible (the "@" character is sometimes used in >> the UI instead of the \0 but "@" is also valid character on the >> filesystem path, so there are ambiguities).? Although Java Sting can >> hold the \0 bytes, depending on local encoding there might be edge >> cases where the path is a valid byte array (sun_path) and is >> addressable from the POSIX API but is not a valid String in given >> encoding. Maybe there might be subclasses for abstract and >> non-abstract addresses... but in the prototype, I looked for the >> simplest solution (but buggy, I know, it e.g. includes all the \0s in >> the toString() representation). > The implementation of Path on Unix systems uses bytes so will preserve > the original file path, you just need to be careful to avoid Path -> > String round trips. OK, I will try doing it with Path. >> The System.inheritedChannel() is in the Java base ? there might be >> some indirect/optional dependency from the base to the module. Idea: >> create an SPI interface with methods canCreateChannel(fd) and >> createChannel(fd); the new module will implement this interface for >> AF_UNIX file descriptors; and the System.inheritedChannel() will >> lookup implementations of this interface and try to get the channel >> from them; and if no implementation will return a channel, it will >> fallback to the current implementation. Is it possible solution? >> >> May the java.net.UnixSocketAddress and >> java.net.StandardProtocolFamily.UNIX stay where they are? > For the prototype then you should be able to put UnixSocketAddress and > UnixProtocolFamily in jdk.net too. And the new prototype module should be jdk.net.uds or some other name? > On naming, one of the discussion points will be whether it should be > LocalSocketAddress as PF_UNIX is deprecated in favor of PF_LOCAL on > some systems. I considered both variants: UnixSocketAddress and LocalUnixAddress. I choose the first one because words "local" and "remote" are used to describe the sides of the socket and then we would have a local local address and remote local address, which sounds bit weird and confusing. On the other hand, PF_UNIX might be deprecated somewhere and UNIX is a registered trademark and was particular product, but now these sockets are used also on many not-unix system (just unix-like or even unix-unlike systems). I am not sure which variant is better ? none is perfect and none is terribly wrong. > A bigger discussion point will be whether to introduce new channel > types rather than trying to have alternative implementations of > SocketChannel/ServerSocketChannel. Most of the code and the logic is the same for local sockets (stream and datagram UDS) as for current internet sockets (TCP and UDP). My patch extends the current code, because classes like ServerSocketChannelImpl and SocketChannelImpl are not inherently linked to the AF_INET and AF_INET6 ? the names and interfaces are generic enough to cover AF_UNIX/AF_LOCAL sockets. There are methods like "bind(SocketAddress local)" instead of "bind(InetSocketAddress local)" and "SocketAddress getRemoteAddress()" instead of "InetSocketAddress getRemoteAddress()" which implies that it is not constrained to just the IP (INET, INET6) and that there might be other types of addresses. Other approach could be copy&paste&cleanup or from-the-scratch re-implementation but it leads to code duplication. Maybe there might be some common classes for most of the logic and then sub-classes for the INET-specific and UNIX-specific parts... > The approach that we've been taking with the RDMA API is to use the > existing API and have the factory methods that that create the RDMA > socket channel specify the differences. UDS and RDMA might be different cases because RDMA has completely different C API while UDS sockets use the same C functions as INET or INET6 sockets. >> My primary goal was to fix the wrong addresses (random IP and port) >> returned from System.inheritedChannel() > If fd 0 is a Unix domain socket then inheritedChannel should return > null. Are you sure you it's returning a SocketChannel to something > that is not a stream or datagram socket? It is a stream or a datagram socket ? just from different domain (UNIX instead of INET or INET6) ? but they work the same way. So yes, when I create a stream or datagram unix domain socket e.g. in systemd (parent process), I can inherit it in Java and get ServerSocketChannel or DatagramChannel from System.inheritedChannel() and it really works. Recently, I wrote an article about it: https://blog.frantovo.cz/c/372/ ? it is in Czech, but code and config examples should be readable to anyone. I see this as a very useful feature. You can run e.g. Jetty or Tomcat as a sub-process of your native application and communicate with it without exposing it on TCP/IP and making that internal interface visible and accessible to others. Or you can use Apache HTTPD as a reverse proxy and connect it with Tomcat/Jetty without additional TCP ports (Apache config: ProxyPass unix:/run/jetty|http://localhost/). Access rights (who can connect) can be easily managed ? just set the file permissions for that socket (super-server like systemd can do this for you: SocketUser, SocketGroup and SocketMode options) or just put it in a directory which is accessible only by some users. I used it in Java 8, but it works also in 14. And today I just tested the 1.5.0_21 and I can confirm: it works even in Java 5. > If so then then this is a bug that should be looked into. The bug is that the channel returns wrong address in getRemoteAddress() and getLocalAddress() methods. I fixed it in my patch so it returns correct type of SocketAddress. BTW: the inheritedChannel() JavaDoc says: > In addition to the network-oriented channels described, this method may return other kinds of channels in the future. So inheriting of other sockets like UDS ones is IMHO not against the specification or anything. Franta -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Mon Jul 22 20:49:10 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 22 Jul 2019 13:49:10 -0700 Subject: Redux 8184157: (ch) AsynchronousFileChannel hangs with internal error when reading locked file In-Reply-To: <5be52b83-1fd2-ab51-69a2-4cee8d9ad5fb@oracle.com> References: <8B15EB6A-2E66-42D9-8277-C25FB9279837@oracle.com> <7DA84564-0770-48E8-9AEA-B6D38C2F4B62@oracle.com> <5be52b83-1fd2-ab51-69a2-4cee8d9ad5fb@oracle.com> Message-ID: <053F96BF-4EF7-48BC-AE8E-07AF23198C08@oracle.com> > On Jul 21, 2019, at 6:15 AM, Alan Bateman wrote: > > I went through webrev.03, where the change to WindowsAsynchronousSocketChannelImpl is partial reverted, and I think it's looking good except for LockTask.pending as that shouldn't be needed now. LockTask.pending removed in [1]. > I see the test has been cleaned up (thanks), but wondering if L51 to close the channel can be removed as it should be closed by way of the try-with-resources. Also would be nice to avoid the line break at L45. The Files.delete at L55 is probably not needed either. Foregoing suggested changes made and also changed L41 of the test from Path path = Path.of(System.getProperty("test.dir", "."), "blah"); to Path path = Path.of("blah"); Thanks, Brian [1] http://cr.openjdk.java.net/~bpb/8184157/webrev.04/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Jul 23 15:46:59 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 23 Jul 2019 16:46:59 +0100 Subject: Unix domain sockets (UDS, AF_UNIX) in System.inheritedChannel() and elsewhere in Java In-Reply-To: <75b60535-ab55-2e3c-d283-a36b78e4175c@frantovo.cz> References: <5e31f8ab-9007-a501-1704-935e79f0133f@frantovo.cz> <4b57b5c2-afea-3ea9-73e2-0e3913ca6ef7@oracle.com> <0aa6e746-0a29-53ba-27dd-e36ad71e3b26@frantovo.cz> <75b60535-ab55-2e3c-d283-a36b78e4175c@frantovo.cz> Message-ID: <85cf2869-4fa6-a246-d8f5-ce56890f37ad@oracle.com> On 22/07/2019 09:16, Franti?ek Ku?era wrote: > > It is a stream or a datagram socket ? just from different domain (UNIX > instead of INET or INET6) ? but they work the same way. So yes, when I > create a stream or datagram unix domain socket e.g. in systemd (parent > process), I can inherit it in Java and get ServerSocketChannel or > DatagramChannel from System.inheritedChannel() and it really works. > Recently, I wrote an article about it: https://blog.frantovo.cz/c/372/ > ? it is in Czech, but code and config examples should be readable to > anyone. > > : > > BTW: the inheritedChannel() JavaDoc says: > > > In addition to the network-oriented channels described, this method > may return other kinds of channels in the future. > Yes, we added this method in Java SE 5 with a view to supporting other channel types (e.g. console) in the future. I'll get back to you on the other points soon but I was curious about your observation that the inheritedChannel returns a channels with "garbage" local/remote addresses when fd 0 is a Unix domain socket. The bug is that the probe is missing a call to getsockname that it needs to check if the family is AF_INET or AF_INET6. So we should minimally fix that. -Alan From brian.burkhalter at oracle.com Thu Jul 25 00:45:49 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 24 Jul 2019 17:45:49 -0700 Subject: 8227609: (fs) Files.newInputStream(...).skip(n) should allow skipping beyond file size Message-ID: <5014C3A9-246B-4DB4-8444-C1716F9BA4E0@oracle.com> https://bugs.openjdk.java.net/browse/JDK-8227609 http://cr.openjdk.java.net/~bpb/8227609/webrev.00/ Fix implementation previously modified for [1] to allow for skipping to positions outside of the file so as to be consistent with FileInputStream.skip(n) and FileChannel.position(p). Thanks, Brian [1] https://bugs.openjdk.java.net/browse/JDK-8227080 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Jul 25 07:57:58 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 25 Jul 2019 08:57:58 +0100 Subject: 8227609: (fs) Files.newInputStream(...).skip(n) should allow skipping beyond file size In-Reply-To: <5014C3A9-246B-4DB4-8444-C1716F9BA4E0@oracle.com> References: <5014C3A9-246B-4DB4-8444-C1716F9BA4E0@oracle.com> Message-ID: On 25/07/2019 01:45, Brian Burkhalter wrote: > https://bugs.openjdk.java.net/browse/JDK-8227609 > http://cr.openjdk.java.net/~bpb/8227609/webrev.00/ > > Fix implementation previously modified for [1] to allow for skipping > to positions outside of the file so as to be consistent with > FileInputStream.skip(n) and FileChannel.position(p). > I think we can do better on the overflow cases. When skipping forwards and pos + n overflows then we should seek to Long.MAX_VALUE. When skipping backwards then we shouldn't attempt to seek to a file offset < 0. I think this is closer to what we need: long pos = sbc.position(); long newPos; if (n > 0) { ? newPos = pos + n; ? if (newPos < 0) newPos = Long.MAX_VALUE; } else { ? newPos = Long.max(pos + n, 0); } sbc.position(newPos); return newPos - pos; I'm also wondering if the catching of ClosedChannelException should be removed as it results in throwing an IOException with CCE (an IOException) as cause. -Alan From brian.burkhalter at oracle.com Thu Jul 25 18:14:38 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 25 Jul 2019 11:14:38 -0700 Subject: 8227609: (fs) Files.newInputStream(...).skip(n) should allow skipping beyond file size In-Reply-To: References: <5014C3A9-246B-4DB4-8444-C1716F9BA4E0@oracle.com> Message-ID: <55C4070F-DE19-4E25-95D7-8519AAE3F19A@oracle.com> > On Jul 25, 2019, at 12:57 AM, Alan Bateman wrote: > > On 25/07/2019 01:45, Brian Burkhalter wrote: >> https://bugs.openjdk.java.net/browse/JDK-8227609 >> http://cr.openjdk.java.net/~bpb/8227609/webrev.00/ >> >> Fix implementation previously modified for [1] to allow for skipping to positions outside of the file so as to be consistent with FileInputStream.skip(n) and FileChannel.position(p). >> > I think we can do better on the overflow cases. When skipping forwards and pos + n overflows then we should seek to Long.MAX_VALUE. When skipping backwards then we shouldn't attempt to seek to a file offset < 0. I think this is closer to what we need: > > long pos = sbc.position(); > long newPos; > if (n > 0) { > newPos = pos + n; > if (newPos < 0) newPos = Long.MAX_VALUE; > } else { > newPos = Long.max(pos + n, 0); > } > sbc.position(newPos); > return newPos - pos; > > I'm also wondering if the catching of ClosedChannelException should be removed as it results in throwing an IOException with CCE (an IOException) as cause. Updated as suggested; test modified accordingly. http://cr.openjdk.java.net/~bpb/8227609/webrev.01/ Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Jul 25 19:58:24 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 25 Jul 2019 20:58:24 +0100 Subject: 8227609: (fs) Files.newInputStream(...).skip(n) should allow skipping beyond file size In-Reply-To: <55C4070F-DE19-4E25-95D7-8519AAE3F19A@oracle.com> References: <5014C3A9-246B-4DB4-8444-C1716F9BA4E0@oracle.com> <55C4070F-DE19-4E25-95D7-8519AAE3F19A@oracle.com> Message-ID: <3af332c2-6b9d-9de6-a48f-4c0efad00f5e@oracle.com> On 25/07/2019 19:14, Brian Burkhalter wrote: > > Updated as suggested; test modified accordingly. > > http://cr.openjdk.java.net/~bpb/8227609/webrev.01/ > > This looks okay. Can the test be expanded to include cases where the result differs to the input, e.g. skip(-*2) should return -, skip(Long.MIN_VALUE) return should -, etc. -Alan From brian.burkhalter at oracle.com Thu Jul 25 20:46:28 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 25 Jul 2019 13:46:28 -0700 Subject: 8227609: (fs) Files.newInputStream(...).skip(n) should allow skipping beyond file size In-Reply-To: <3af332c2-6b9d-9de6-a48f-4c0efad00f5e@oracle.com> References: <5014C3A9-246B-4DB4-8444-C1716F9BA4E0@oracle.com> <55C4070F-DE19-4E25-95D7-8519AAE3F19A@oracle.com> <3af332c2-6b9d-9de6-a48f-4c0efad00f5e@oracle.com> Message-ID: <1052D689-5A03-4B33-955B-9D51B99861E8@oracle.com> > On Jul 25, 2019, at 12:58 PM, Alan Bateman wrote: > > On 25/07/2019 19:14, Brian Burkhalter wrote: >> >> Updated as suggested; test modified accordingly. >> >> http://cr.openjdk.java.net/~bpb/8227609/webrev.01/ >> >> > This looks okay. > > Can the test be expanded to include cases where the result differs to the input, e.g. skip(-*2) should return -, skip(Long.MIN_VALUE) return should -, etc. Some new cases added: http://cr.openjdk.java.net/~bpb/8227609/webrev.02/. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From lance.andersen at oracle.com Thu Jul 25 21:04:55 2019 From: lance.andersen at oracle.com (Lance Andersen) Date: Thu, 25 Jul 2019 17:04:55 -0400 Subject: 8227609: (fs) Files.newInputStream(...).skip(n) should allow skipping beyond file size In-Reply-To: <1052D689-5A03-4B33-955B-9D51B99861E8@oracle.com> References: <5014C3A9-246B-4DB4-8444-C1716F9BA4E0@oracle.com> <55C4070F-DE19-4E25-95D7-8519AAE3F19A@oracle.com> <3af332c2-6b9d-9de6-a48f-4c0efad00f5e@oracle.com> <1052D689-5A03-4B33-955B-9D51B99861E8@oracle.com> Message-ID: <96323392-81C0-4608-A5A2-B8312CA56B5C@oracle.com> Hi Brian, This this looks good. Given you use TestUtil.removall after the test runs, do you need the finally block to the delete the file within the test? HTH Best Lance > On Jul 25, 2019, at 4:46 PM, Brian Burkhalter wrote: > > >> On Jul 25, 2019, at 12:58 PM, Alan Bateman > wrote: >> >> On 25/07/2019 19:14, Brian Burkhalter wrote: >>> >>> Updated as suggested; test modified accordingly. >>> >>> http://cr.openjdk.java.net/~bpb/8227609/webrev.01/ >>> >>> >> This looks okay. >> >> Can the test be expanded to include cases where the result differs to the input, e.g. skip(-*2) should return -, skip(Long.MIN_VALUE) return should -, etc. > > Some new cases added: http://cr.openjdk.java.net/~bpb/8227609/webrev.02/ . > > Thanks, > > Brian Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: oracle_sig_logo.gif Type: image/gif Size: 658 bytes Desc: not available URL: From brian.burkhalter at oracle.com Thu Jul 25 21:10:01 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 25 Jul 2019 14:10:01 -0700 Subject: 8227609: (fs) Files.newInputStream(...).skip(n) should allow skipping beyond file size In-Reply-To: <96323392-81C0-4608-A5A2-B8312CA56B5C@oracle.com> References: <5014C3A9-246B-4DB4-8444-C1716F9BA4E0@oracle.com> <55C4070F-DE19-4E25-95D7-8519AAE3F19A@oracle.com> <3af332c2-6b9d-9de6-a48f-4c0efad00f5e@oracle.com> <1052D689-5A03-4B33-955B-9D51B99861E8@oracle.com> <96323392-81C0-4608-A5A2-B8312CA56B5C@oracle.com> Message-ID: Hi Lance, > On Jul 25, 2019, at 2:04 PM, Lance Andersen wrote: > > This this looks good. > > Given you use TestUtil.removall after the test runs, do you need the finally block to the delete the file within the test? I guess not. I?ll remove that. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Fri Jul 26 06:38:29 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 26 Jul 2019 07:38:29 +0100 Subject: 8227609: (fs) Files.newInputStream(...).skip(n) should allow skipping beyond file size In-Reply-To: <1052D689-5A03-4B33-955B-9D51B99861E8@oracle.com> References: <5014C3A9-246B-4DB4-8444-C1716F9BA4E0@oracle.com> <55C4070F-DE19-4E25-95D7-8519AAE3F19A@oracle.com> <3af332c2-6b9d-9de6-a48f-4c0efad00f5e@oracle.com> <1052D689-5A03-4B33-955B-9D51B99861E8@oracle.com> Message-ID: On 25/07/2019 21:46, Brian Burkhalter wrote: > > Some new cases added: http://cr.openjdk.java.net/~bpb/8227609/webrev.02/. > Thanks for adding more test cases, looks good. -Alan From brian.burkhalter at oracle.com Fri Jul 26 14:53:30 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 26 Jul 2019 07:53:30 -0700 Subject: 8227609: (fs) Files.newInputStream(...).skip(n) should allow skipping beyond file size In-Reply-To: References: <5014C3A9-246B-4DB4-8444-C1716F9BA4E0@oracle.com> <55C4070F-DE19-4E25-95D7-8519AAE3F19A@oracle.com> <3af332c2-6b9d-9de6-a48f-4c0efad00f5e@oracle.com> <1052D689-5A03-4B33-955B-9D51B99861E8@oracle.com> Message-ID: <9673375E-073D-4454-A643-DF3A7753D9E6@oracle.com> > On Jul 25, 2019, at 11:38 PM, Alan Bateman wrote: > > On 25/07/2019 21:46, Brian Burkhalter wrote: >> >> Some new cases added: http://cr.openjdk.java.net/~bpb/8227609/webrev.02/ . >> > Thanks for adding more test cases, looks good. I saw a test failure in the CI system with this: ----------System.err:(18/1120)---------- java.io.IOException: Invalid argument at java.base/sun.nio.ch.FileDispatcherImpl.seek0(Native Method) at java.base/sun.nio.ch.FileDispatcherImpl.seek(FileDispatcherImpl.java:78) at java.base/sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:361) at java.base/sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:52) at java.base/sun.nio.ch.ChannelInputStream.skip(ChannelInputStream.java:132) at InputStreamTest.testSkip(InputStreamTest.java:115) InputStreamTest.java; 112 assertTrue(in.skip(size/2) == size/2); // 0.5 113 assertTrue(in.available() == size/2); 114 115 assertTrue(in.skip(Long.MAX_VALUE - size/4) == 116 Long.MAX_VALUE - size/2); 117 assertTrue(in.available() == 0); It looks as if for some case lseek64() doesn?t like an argument of Long.MAX_VALUE. I?ll need to run this down before checking anything in. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Fri Jul 26 14:56:49 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 26 Jul 2019 07:56:49 -0700 Subject: 8227609: (fs) Files.newInputStream(...).skip(n) should allow skipping beyond file size In-Reply-To: <9673375E-073D-4454-A643-DF3A7753D9E6@oracle.com> References: <5014C3A9-246B-4DB4-8444-C1716F9BA4E0@oracle.com> <55C4070F-DE19-4E25-95D7-8519AAE3F19A@oracle.com> <3af332c2-6b9d-9de6-a48f-4c0efad00f5e@oracle.com> <1052D689-5A03-4B33-955B-9D51B99861E8@oracle.com> <9673375E-073D-4454-A643-DF3A7753D9E6@oracle.com> Message-ID: <286D3DDD-DA99-4C59-BECF-9C955D458C26@oracle.com> > On Jul 26, 2019, at 7:53 AM, Brian Burkhalter wrote: > > I saw a test failure in the CI system with this: > ----------System.err:(18/1120)---------- > java.io.IOException: Invalid argument > at java.base/sun.nio.ch.FileDispatcherImpl.seek0(Native Method) > at java.base/sun.nio.ch.FileDispatcherImpl.seek(FileDispatcherImpl.java:78) > at java.base/sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:361) > at java.base/sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:52) > at java.base/sun.nio.ch.ChannelInputStream.skip(ChannelInputStream.java:132) > at InputStreamTest.testSkip(InputStreamTest.java:115) > InputStreamTest.java; > 112 assertTrue(in.skip(size/2) == size/2); // 0.5 > 113 assertTrue(in.available() == size/2); > 114 > 115 assertTrue(in.skip(Long.MAX_VALUE - size/4) == > 116 Long.MAX_VALUE - size/2); > 117 assertTrue(in.available() == 0); > It looks as if for some case lseek64() doesn?t like an argument of Long.MAX_VALUE. I?ll need to run this down before checking anything in. This was on Linux-x64. Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From fweimer at redhat.com Fri Jul 26 15:06:27 2019 From: fweimer at redhat.com (Florian Weimer) Date: Fri, 26 Jul 2019 17:06:27 +0200 Subject: 8227609: (fs) Files.newInputStream(...).skip(n) should allow skipping beyond file size In-Reply-To: <286D3DDD-DA99-4C59-BECF-9C955D458C26@oracle.com> (Brian Burkhalter's message of "Fri, 26 Jul 2019 07:56:49 -0700") References: <5014C3A9-246B-4DB4-8444-C1716F9BA4E0@oracle.com> <55C4070F-DE19-4E25-95D7-8519AAE3F19A@oracle.com> <3af332c2-6b9d-9de6-a48f-4c0efad00f5e@oracle.com> <1052D689-5A03-4B33-955B-9D51B99861E8@oracle.com> <9673375E-073D-4454-A643-DF3A7753D9E6@oracle.com> <286D3DDD-DA99-4C59-BECF-9C955D458C26@oracle.com> Message-ID: <87y30kal8s.fsf@oldenburg2.str.redhat.com> * Brian Burkhalter: > On Jul 26, 2019, at 7:53 AM, Brian Burkhalter wrote: > > I saw a test failure in the CI system with this: > ----------System.err:(18/1120)---------- > java.io.IOException: Invalid argument > at java.base/sun.nio.ch.FileDispatcherImpl.seek0(Native Method) > at java.base/sun.nio.ch.FileDispatcherImpl.seek(FileDispatcherImpl.java:78) > at java.base/sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:361) > at java.base/sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:52) > at java.base/sun.nio.ch.ChannelInputStream.skip(ChannelInputStream.java:132) > at InputStreamTest.testSkip(InputStreamTest.java:115) > InputStreamTest.java; > 112 assertTrue(in.skip(size/2) == size/2); // 0.5 > 113 assertTrue(in.available() == size/2); > 114 > 115 assertTrue(in.skip(Long.MAX_VALUE - size/4) == > 116 Long.MAX_VALUE - size/2); > 117 assertTrue(in.available() == 0); > It looks as if for some case lseek64() doesn?t like an argument of Long.MAX_VALUE. I?ll need to run this down before checking > anything in. > > This was on Linux-x64. On ext4? The EINVAL depends on the file system. XFS supports arbitrary offsets, but ext4 does not, for example. We discussed this recently on another thread: From: Karan Mehta Date: Tue, 4 Jun 2019 10:13:05 -0700 Subject: Invalid argument exception when trying to seek FilePosition to LONG.MAX_VALUE To: nio-dev at openjdk.java.net Message-ID: Thanks, Florian From brian.burkhalter at oracle.com Fri Jul 26 15:14:26 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 26 Jul 2019 08:14:26 -0700 Subject: 8227609: (fs) Files.newInputStream(...).skip(n) should allow skipping beyond file size In-Reply-To: <87y30kal8s.fsf@oldenburg2.str.redhat.com> References: <5014C3A9-246B-4DB4-8444-C1716F9BA4E0@oracle.com> <55C4070F-DE19-4E25-95D7-8519AAE3F19A@oracle.com> <3af332c2-6b9d-9de6-a48f-4c0efad00f5e@oracle.com> <1052D689-5A03-4B33-955B-9D51B99861E8@oracle.com> <9673375E-073D-4454-A643-DF3A7753D9E6@oracle.com> <286D3DDD-DA99-4C59-BECF-9C955D458C26@oracle.com> <87y30kal8s.fsf@oldenburg2.str.redhat.com> Message-ID: <795F66EA-A180-4197-8FD1-E657EDC5D7B8@oracle.com> > On Jul 26, 2019, at 8:06 AM, Florian Weimer wrote: > > On ext4? The EINVAL depends on the file system. XFS supports arbitrary > offsets, but ext4 does not, for example. I don?t know the FS type yet. > We discussed this recently on another thread: > > From: Karan Mehta > > Date: Tue, 4 Jun 2019 10:13:05 -0700 > Subject: Invalid argument exception when trying to seek FilePosition to Thanks for the reference. Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Fri Jul 26 15:24:33 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 26 Jul 2019 08:24:33 -0700 Subject: 8227609: (fs) Files.newInputStream(...).skip(n) should allow skipping beyond file size In-Reply-To: <795F66EA-A180-4197-8FD1-E657EDC5D7B8@oracle.com> References: <5014C3A9-246B-4DB4-8444-C1716F9BA4E0@oracle.com> <55C4070F-DE19-4E25-95D7-8519AAE3F19A@oracle.com> <3af332c2-6b9d-9de6-a48f-4c0efad00f5e@oracle.com> <1052D689-5A03-4B33-955B-9D51B99861E8@oracle.com> <9673375E-073D-4454-A643-DF3A7753D9E6@oracle.com> <286D3DDD-DA99-4C59-BECF-9C955D458C26@oracle.com> <87y30kal8s.fsf@oldenburg2.str.redhat.com> <795F66EA-A180-4197-8FD1-E657EDC5D7B8@oracle.com> Message-ID: <5003A55B-2EF9-4FE8-81DA-4B3197643782@oracle.com> > On Jul 26, 2019, at 8:14 AM, Brian Burkhalter wrote: > >> On Jul 26, 2019, at 8:06 AM, Florian Weimer > wrote: >> >> On ext4? The EINVAL depends on the file system. XFS supports arbitrary >> offsets, but ext4 does not, for example. > > I don?t know the FS type yet. So it appears indeed to be ext 4. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Fri Jul 26 15:44:58 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 26 Jul 2019 16:44:58 +0100 Subject: 8227609: (fs) Files.newInputStream(...).skip(n) should allow skipping beyond file size In-Reply-To: <87y30kal8s.fsf@oldenburg2.str.redhat.com> References: <5014C3A9-246B-4DB4-8444-C1716F9BA4E0@oracle.com> <55C4070F-DE19-4E25-95D7-8519AAE3F19A@oracle.com> <3af332c2-6b9d-9de6-a48f-4c0efad00f5e@oracle.com> <1052D689-5A03-4B33-955B-9D51B99861E8@oracle.com> <9673375E-073D-4454-A643-DF3A7753D9E6@oracle.com> <286D3DDD-DA99-4C59-BECF-9C955D458C26@oracle.com> <87y30kal8s.fsf@oldenburg2.str.redhat.com> Message-ID: <47adbe2a-9c2b-cf0a-1dff-7cbaf0bf3ef6@oracle.com> On 26/07/2019 16:06, Florian Weimer wrote: > : > On ext4? The EINVAL depends on the file system. XFS supports arbitrary > offsets, but ext4 does not, for example. > > We discussed this recently on another thread: > Right, the recent thread on this topic is here: https://mail.openjdk.java.net/pipermail/nio-dev/2019-June/006226.html From brian.burkhalter at oracle.com Fri Jul 26 15:58:20 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 26 Jul 2019 08:58:20 -0700 Subject: 8227609: (fs) Files.newInputStream(...).skip(n) should allow skipping beyond file size In-Reply-To: <47adbe2a-9c2b-cf0a-1dff-7cbaf0bf3ef6@oracle.com> References: <5014C3A9-246B-4DB4-8444-C1716F9BA4E0@oracle.com> <55C4070F-DE19-4E25-95D7-8519AAE3F19A@oracle.com> <3af332c2-6b9d-9de6-a48f-4c0efad00f5e@oracle.com> <1052D689-5A03-4B33-955B-9D51B99861E8@oracle.com> <9673375E-073D-4454-A643-DF3A7753D9E6@oracle.com> <286D3DDD-DA99-4C59-BECF-9C955D458C26@oracle.com> <87y30kal8s.fsf@oldenburg2.str.redhat.com> <47adbe2a-9c2b-cf0a-1dff-7cbaf0bf3ef6@oracle.com> Message-ID: <118667AB-E61D-4BF9-90A5-0CF381A03965@oracle.com> > On Jul 26, 2019, at 8:44 AM, Alan Bateman wrote: > > On 26/07/2019 16:06, Florian Weimer wrote: >> : >> On ext4? The EINVAL depends on the file system. XFS supports arbitrary >> offsets, but ext4 does not, for example. >> >> We discussed this recently on another thread: >> > Right, the recent thread on this topic is here: > > https://mail.openjdk.java.net/pipermail/nio-dev/2019-June/006226.html The seek to Long.MAX_VALUE occurs due to this change at line 128 to ChannelInputStream at line 128: 124 long pos = sbc.position(); <> 125 long newPos; 126 if (n > 0) { 127 newPos = pos + n; 128 if (newPos < 0) newPos = Long.MAX_VALUE; 129 } else { 130 newPos = Long.max(pos + n, 0); 131 } <> 132 sbc.position(newPos); Note that in the version prior to the patch [1] for [2], ChannelInputStream.skip() would devolve to InputStream.skip() and would skip only to the end of the stream for the case of pos + n overflowing so the above looks like a change in behavior. Thanks, Brian [1] http://hg.openjdk.java.net/jdk/jdk/rev/8c19519114e7 [2] https://bugs.openjdk.java.net/browse/JDK-8227080 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Sat Jul 27 16:57:06 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 27 Jul 2019 17:57:06 +0100 Subject: 8227609: (fs) Files.newInputStream(...).skip(n) should allow skipping beyond file size8 In-Reply-To: <118667AB-E61D-4BF9-90A5-0CF381A03965@oracle.com> References: <5014C3A9-246B-4DB4-8444-C1716F9BA4E0@oracle.com> <55C4070F-DE19-4E25-95D7-8519AAE3F19A@oracle.com> <3af332c2-6b9d-9de6-a48f-4c0efad00f5e@oracle.com> <1052D689-5A03-4B33-955B-9D51B99861E8@oracle.com> <9673375E-073D-4454-A643-DF3A7753D9E6@oracle.com> <286D3DDD-DA99-4C59-BECF-9C955D458C26@oracle.com> <87y30kal8s.fsf@oldenburg2.str.redhat.com> <47adbe2a-9c2b-cf0a-1dff-7cbaf0bf3ef6@oracle.com> <118667AB-E61D-4BF9-90A5-0CF381A03965@oracle.com> Message-ID: On 26/07/2019 16:58, Brian Burkhalter wrote: > Note that in the version prior to the patch [1] for [2], > ChannelInputStream.skip() would devolve to InputStream.skip() and > would skip only to the end of the stream for the case of pos + n > overflowing so the above looks like a change in behavior. It's the same thing with FileInputStream as it's an underlying file system issue jshell> var in = new FileInputStream("foo.txt") in ==> java.io.FileInputStream at 2ef1e4fa jshell> in.skip(Long.MAX_VALUE) |? Exception java.io.IOException: Invalid argument |??????? at FileInputStream.skip0 (Native Method) |??????? at FileInputStream.skip (FileInputStream.java:301) |??????? at (#2:1) For the re-do, I think the important thing is to fix the skip back issue as that was missed in JDK-8227080. Skipping beyond EOF is allowed with files but isn't too interesting when the file is open read-only. That is, if the solution is to only seek to 0..size then it should be okay for most needs, as in: ??????? if (n > 0) { ??????????? newPos = pos + n; ??????????? long size = sbc.size(); ??????????? if (newPos < 0 || newPos > size) { ??????????????? newPos = size; ??????????? } ??????? } else { ??????????? newPos = Long.max(pos + n, 0); ??????? } or changing the test is okay too. -Alan From brian.burkhalter at oracle.com Mon Jul 29 19:51:54 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 29 Jul 2019 12:51:54 -0700 Subject: 8227609: (fs) Files.newInputStream(...).skip(n) should allow skipping beyond file size8 In-Reply-To: References: <5014C3A9-246B-4DB4-8444-C1716F9BA4E0@oracle.com> <55C4070F-DE19-4E25-95D7-8519AAE3F19A@oracle.com> <3af332c2-6b9d-9de6-a48f-4c0efad00f5e@oracle.com> <1052D689-5A03-4B33-955B-9D51B99861E8@oracle.com> <9673375E-073D-4454-A643-DF3A7753D9E6@oracle.com> <286D3DDD-DA99-4C59-BECF-9C955D458C26@oracle.com> <87y30kal8s.fsf@oldenburg2.str.redhat.com> <47adbe2a-9c2b-cf0a-1dff-7cbaf0bf3ef6@oracle.com> <118667AB-E61D-4BF9-90A5-0CF381A03965@oracle.com> Message-ID: > On Jul 27, 2019, at 9:57 AM, Alan Bateman wrote: > > On 26/07/2019 16:58, Brian Burkhalter wrote: >> Note that in the version prior to the patch [1] for [2], ChannelInputStream.skip() would devolve to InputStream.skip() and would skip only to the end of the stream for the case of pos + n overflowing so the above looks like a change in behavior. > It's the same thing with FileInputStream as it's an underlying file system issue > > jshell> var in = new FileInputStream("foo.txt") > in ==> java.io.FileInputStream at 2ef1e4fa > > jshell> in.skip(Long.MAX_VALUE) > | Exception java.io.IOException: Invalid argument > | at FileInputStream.skip0 (Native Method) > | at FileInputStream.skip (FileInputStream.java:301) > | at (#2:1) Yes, I checked FIS.skip() and FileChannel.position() for this case. > For the re-do, I think the important thing is to fix the skip back issue as that was missed in JDK-8227080. Skipping beyond EOF is allowed with files but isn't too interesting when the file is open read-only. That is, if the solution is to only seek to 0..size then it should be okay for most needs, as in: > > if (n > 0) { > newPos = pos + n; > long size = sbc.size(); > if (newPos < 0 || newPos > size) { > newPos = size; > } > } else { > newPos = Long.max(pos + n, 0); > } > > or changing the test is okay too. I updated it to clamp to the size and changed the test accordingly: http://cr.openjdk.java.net/~bpb/8227609/webrev.03/ Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Jul 30 13:57:22 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 30 Jul 2019 06:57:22 -0700 Subject: 8227609: (fs) Files.newInputStream(...).skip(n) should allow skipping beyond file size8 In-Reply-To: References: <5014C3A9-246B-4DB4-8444-C1716F9BA4E0@oracle.com> <55C4070F-DE19-4E25-95D7-8519AAE3F19A@oracle.com> <3af332c2-6b9d-9de6-a48f-4c0efad00f5e@oracle.com> <1052D689-5A03-4B33-955B-9D51B99861E8@oracle.com> <9673375E-073D-4454-A643-DF3A7753D9E6@oracle.com> <286D3DDD-DA99-4C59-BECF-9C955D458C26@oracle.com> <87y30kal8s.fsf@oldenburg2.str.redhat.com> <47adbe2a-9c2b-cf0a-1dff-7cbaf0bf3ef6@oracle.com> <118667AB-E61D-4BF9-90A5-0CF381A03965@oracle.com> Message-ID: On 29/07/2019 12:51, Brian Burkhalter wrote: > > I updated it to clamp to the size and changed the test accordingly: > > http://cr.openjdk.java.net/~bpb/8227609/webrev.03/ > This looks good. You've commented out the tests from the previous revision - are you planning to remove those before pushing? -Alan From adinn at redhat.com Tue Jul 30 15:04:58 2019 From: adinn at redhat.com (Andrew Dinn) Date: Tue, 30 Jul 2019 16:04:58 +0100 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: <8f72d3b5-3a01-a7de-3b09-35571266a87d@redhat.com> References: <80da32b2-7acb-7b94-b82c-5dcd5cf95539@redhat.com> <0a41e686-0931-5054-b314-2c015fbae62a@redhat.com> <36326F5E-12CD-487A-8FE2-1049631FE908@oracle.com> <8B1A8EDC-F9D7-4085-A34F-69100DBD7D5C@oracle.com> <5c78b5d7-dc9e-fbbd-41a9-5139ed6ee32c@redhat.com> <86d000f3-9eca-6089-5f7e-5698444d99ce@oracle.com> <7fe8d17d-6170-fcc9-b87a-eccfda2bb546@redhat.com> <7b39f68d-5698-4079-425c-c86ec161e361@oracle.com> <72810832-eb88-3418-f336-b95af95d9dcc@redhat.com> <20c8bbcb-2cd3-77af-cd40-29f7bd752166@oracle.com> <68566847-de80-82af-a928-d30c75f2b1b2@redhat.com> <892f3b0b-55f1-872d-15f4-5af907fa8437@redhat.com> <8f72d3b5-3a01-a7de-3b09-35571266a87d@redhat.com> Message-ID: JEP 352 has now been targeted for inclusion in JDK14. The latest webrev for the implementation JIRA has been rebased to apply to the current tree. Is it now ok to push this change set? JIRA: https://bugs.openjdk.java.net/browse/JDK-8224974 webrev: http://cr.openjdk.java.net/~adinn/8224974/webrev.09 n.b. by way of sanity test I pushed this to the submit repo and it came back with no failed tests. regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From brian.burkhalter at oracle.com Tue Jul 30 15:22:00 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 30 Jul 2019 08:22:00 -0700 Subject: 8227609: (fs) Files.newInputStream(...).skip(n) should allow skipping beyond file size8 In-Reply-To: References: <5014C3A9-246B-4DB4-8444-C1716F9BA4E0@oracle.com> <55C4070F-DE19-4E25-95D7-8519AAE3F19A@oracle.com> <3af332c2-6b9d-9de6-a48f-4c0efad00f5e@oracle.com> <1052D689-5A03-4B33-955B-9D51B99861E8@oracle.com> <9673375E-073D-4454-A643-DF3A7753D9E6@oracle.com> <286D3DDD-DA99-4C59-BECF-9C955D458C26@oracle.com> <87y30kal8s.fsf@oldenburg2.str.redhat.com> <47adbe2a-9c2b-cf0a-1dff-7cbaf0bf3ef6@oracle.com> <118667AB-E61D-4BF9-90A5-0CF381A03965@oracle.com> Message-ID: <20A3F848-BAF8-4DFE-AFE3-8FC6333C4D22@oracle.com> > On Jul 30, 2019, at 6:57 AM, Alan Bateman wrote: > > On 29/07/2019 12:51, Brian Burkhalter wrote: >> >> I updated it to clamp to the size and changed the test accordingly: >> >> http://cr.openjdk.java.net/~bpb/8227609/webrev.03/ >> > This looks good. You've commented out the tests from the previous revision - are you planning to remove those before pushing? Yes, they duplicate others in there but I was undecided as to whether to replace them with something else. I?ll delete them before checking it in. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From eolivelli at gmail.com Tue Jul 30 15:45:38 2019 From: eolivelli at gmail.com (Enrico Olivelli) Date: Tue, 30 Jul 2019 17:45:38 +0200 Subject: Found different in behavior of SocketInputStream#read in JDK13 Message-ID: Hello, in Apache ZooKeeper community we discovered a difference in behavior in SocketInputStream. The problem is that SocketInputStream#read used to throw a SocketException instead of returning -1. I see that this was cited in https://openjdk.java.net/jeps/353 "The InputStream and OutputStream returned by the old implementation tests the stream for EOF and returns -1 before other checks. The new implementation does null and bounds checks before checking if the stream is at EOF. It is possible, but unlikely, that there is fragile code that will trip up due to the order of the checks." We are talking about a unit test, no a big deal. This is the fix, provided by Mate Szalay-Beko https://github.com/apache/zookeeper/pull/1029 I hope that helps Enrico Olivelli -------------- next part -------------- An HTML attachment was scrubbed... URL: From shade at redhat.com Tue Jul 30 16:00:08 2019 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 30 Jul 2019 18:00:08 +0200 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: References: <80da32b2-7acb-7b94-b82c-5dcd5cf95539@redhat.com> <0a41e686-0931-5054-b314-2c015fbae62a@redhat.com> <36326F5E-12CD-487A-8FE2-1049631FE908@oracle.com> <8B1A8EDC-F9D7-4085-A34F-69100DBD7D5C@oracle.com> <5c78b5d7-dc9e-fbbd-41a9-5139ed6ee32c@redhat.com> <86d000f3-9eca-6089-5f7e-5698444d99ce@oracle.com> <7fe8d17d-6170-fcc9-b87a-eccfda2bb546@redhat.com> <7b39f68d-5698-4079-425c-c86ec161e361@oracle.com> <72810832-eb88-3418-f336-b95af95d9dcc@redhat.com> <20c8bbcb-2cd3-77af-cd40-29f7bd752166@oracle.com> <68566847-de80-82af-a928-d30c75f2b1b2@redhat.com> <892f3b0b-55f1-872d-15f4-5af907fa8437@redhat.com> <8f72d3b5-3a01-a7de-3b09-35571266a87d@redhat.com> Message-ID: <671c7ba1-9597-a990-190b-145d9cb5349e@redhat.com> On 7/30/19 5:04 PM, Andrew Dinn wrote: > JEP 352 has now been targeted for inclusion in JDK14. The latest webrev > for the implementation JIRA has been rebased to apply to the current > tree. Is it now ok to push this change set? > > JIRA: https://bugs.openjdk.java.net/browse/JDK-8224974 > webrev: http://cr.openjdk.java.net/~adinn/8224974/webrev.09 Is it okay to have a late code review? Here it goes: === General: So if pre wbsync is no-op, why do we need to handle it everywhere? We seem to be falling through all the way to the stub to do nothing there, maybe we should instead cut much earlier, e.g. when inlining Unsafe.writeBackPresync0? Would it be better to not emit CacheWBPreSync at all? === General: IIRC, notproduct and PRODUCT defines are legacy, and should be avoided? develop and ASSERT must be the substitutes for this. See some discussion here: https://bugs.openjdk.java.net/browse/JDK-8183287 === src/hotspot/cpu/aarch64/aarch64.ad src/hotspot/cpu/x86/x86.ad This should probably be just "!VM_Version...". Braces around the statement would not hurt either. 2196 if (VM_Version::supports_data_cache_line_flush() == false) 2197 ret_value = false; === src/hotspot/cpu/aarch64/macroAssembler_aarch64.cpp src/hotspot/cpu/x86/macroAssembler_x86.cpp Braces style mismatch, should be on the same line, as in the rest of the file: 5837 void MacroAssembler::cache_wb(Address line) 5838 { 5856 void MacroAssembler::cache_wbsync(bool is_pre) 5857 { === src/hotspot/cpu/x86/assembler_x86.cpp It feels like these comments are redundant, especially L8630 and L8646 which mention magic values "6" and "7", not present in the code: 8624 // instruction prefix is 0x66 8627 // opcode family is 0x0f 0xAE 8630 // extended opcode byte is 7 == rdi 8640 // instruction prefix is 0x66 8643 // opcode family is 0x0f 0xAE 8646 // extended opcode byte is 6 == rsi === src/hotspot/cpu/x86/macroAssembler_x86.cpp These comments feel redundant too. Well, maybe they should instead talk about the choices the subsequent code makes. 9915 // pick the correct implementation 9936 // pick the correct implementation === src/hotspot/cpu/x86/macroAssembler_x86.cpp src/hotspot/cpu/x86/vm_version_x86.hpp The docs say "The CLFLUSH instruction was introduced with the SSE2 extensions; however, because it has its own CPUID feature flag, it can be implemented in IA-32 processors that do not include the SSE2 extensions. Also, detecting the presence of the SSE2 extensions with the CPUID instruction does not guarantee that the CLFLUSH instruction is implemented in the processor." Yet, we have: 9910 // 64 bit cpus always support clflush 9911 assert(VM_Version::supports_clflush(), "should not reach here on 32-bit"); 505 #ifdef _LP64 506 // cpflush is always available on x86_64 507 result |= CPU_FLUSH; 508 #else 509 if (_cpuid_info.std_cpuid1_edx.bits.clflush != 0) 510 result |= CPU_FLUSH; 511 #endif 967 // 64 bit cpus always support clflush which writes back and evicts 968 // on 32 bit cpus support is recorded via a feature flag 980 static bool supports_clflush() { return true; } I think the assert should just say "clflush should be available", and clflush cpu flag detected for 64-bits too? === src/hotspot/cpu/x86/macroAssembler_x86.hpp Accidental camelCase, while hotspot code expects snake_case: 1794 void cache_wbsync(bool isPre); === src/hotspot/cpu/x86/stubGenerator_x86_64.cpp Any reason to avoid inline here, e.g. "__ cache_wb(Address(src, 0))"? 2921 const Address line(src, 0); 2922 __ cache_wb(line); === src/hotspot/cpu/x86/stubGenerator_x86_64.cpp Is it really "cmpl" here, not "cmpb"? I think aarch64 code tests for byte. 2942 __ cmpl(is_pre, 0); === src/hotspot/share/classfile/vmSymbols.hpp Excess space before "jdk_internal..." here: 1096 do_intrinsic(_writebackPostSync0, jdk_internal_misc_Unsafe, writebackPostSync0_name, void_method_signature , F_RN) \ === src/hotspot/share/opto/c2compiler.cpp Why inject new cases here, instead of at the end of switch? Saves sudden "break": 578 break; 579 case vmIntrinsics::_writeback0: 580 if (!Matcher::match_rule_supported(Op_CacheWB)) return false; 581 break; 582 case vmIntrinsics::_writebackPreSync0: 583 if (!Matcher::match_rule_supported(Op_CacheWBPreSync)) return false; 584 break; 585 case vmIntrinsics::_writebackPostSync0: 586 if (!Matcher::match_rule_supported(Op_CacheWBPostSync)) return false; 587 break; === src/hotspot/share/opto/library_call.cpp Accidental camelCase here, should be snake_case: 257 bool inline_unsafe_writebackSync0(bool isPre); 2870 bool LibraryCallKit::inline_unsafe_writebackSync0(bool isPre) { === src/hotspot/share/prims/unsafe.cpp Odd indenting near "CC": 1122 {CC "writeback0", CC "(" "J" ")V", FN_PTR(Unsafe_WriteBack0)}, 1123 {CC "writebackPreSync0", CC "()V", FN_PTR(Unsafe_WriteBackPreSync0)}, 1124 {CC "writebackPostSync0", CC "()V", FN_PTR(Unsafe_WriteBackPostSync0)}, camelCase: 464 static void doWriteBackSync0(bool isPre) === src/hotspot/share/prims/unsafe.cpp Do we really need this function pointer mess? 457 void (*wb)(void *); 458 void *a = addr_from_java(line); 459 wb = (void (*)(void *)) StubRoutines::data_cache_writeback(); 460 assert(wb != NULL, "generate writeback stub!"); 461 (*wb)(a); Seems easier to: assert(StubRoutines::data_cache_writeback() != NULL, "sanity"); StubRoutines::data_cache_writeback()(addr_from_java(line)); Thanks, -Aleksey -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From Alan.Bateman at oracle.com Tue Jul 30 18:53:34 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 30 Jul 2019 11:53:34 -0700 Subject: Found different in behavior of SocketInputStream#read in JDK13 In-Reply-To: References: Message-ID: On 30/07/2019 08:45, Enrico Olivelli wrote: > Hello, > in Apache ZooKeeper community we discovered a difference in?behavior > in SocketInputStream. > > The problem is that SocketInputStream#read used to throw a > SocketException instead of returning -1. > > I see that this was cited in https://openjdk.java.net/jeps/353 > > "The |InputStream|?and |OutputStream|?returned by the old > implementation tests the stream for EOF and returns -1 before other > checks. The new implementation does |null|?and bounds checks before > checking if the stream is at EOF. It is possible, but unlikely, that > there is fragile code that will trip up due to the order of the checks." > > We are talking about a unit test, no a big deal. > > This is the fix, provided by?Mate Szalay-Beko > https://github.com/apache/zookeeper/pull/1029 > Thanks for the mail. What is the exception you see with the legacy implementation? The note you cite would lead to an IndexOutOfBoundException due to the bad input and I don't think this is what you are seeing. -Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eolivelli at gmail.com Wed Jul 31 08:26:53 2019 From: eolivelli at gmail.com (Enrico Olivelli) Date: Wed, 31 Jul 2019 10:26:53 +0200 Subject: Found different in behavior of SocketInputStream#read in JDK13 In-Reply-To: References: Message-ID: Il giorno mar 30 lug 2019 alle ore 20:53 Alan Bateman < Alan.Bateman at oracle.com> ha scritto: > On 30/07/2019 08:45, Enrico Olivelli wrote: > > Hello, > in Apache ZooKeeper community we discovered a difference in behavior in > SocketInputStream. > > The problem is that SocketInputStream#read used to throw a SocketException > instead of returning -1. > > I see that this was cited in https://openjdk.java.net/jeps/353 > > "The InputStream and OutputStream returned by the old implementation > tests the stream for EOF and returns -1 before other checks. The new > implementation does null and bounds checks before checking if the stream > is at EOF. It is possible, but unlikely, that there is fragile code that > will trip up due to the order of the checks." > > We are talking about a unit test, no a big deal. > > This is the fix, provided by Mate Szalay-Beko > https://github.com/apache/zookeeper/pull/1029 > > Thanks for the mail. What is the exception you see with the legacy > implementation? The note you cite would lead to an IndexOutOfBoundException > due to the bad input and I don't think this is what you are seeing. > This is the exception on JDK12 and on JDK13 with -Djdk.net.usePlainSocketImpl java.net.SocketException: Connection reset at java.base/java.net.SocketInputStream.read(SocketInputStream.java:186) at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140) at org.apache.zookeeper.server.quorum.UnifiedServerSocketTest.testConnectWithoutSSLToStrictServer(UnifiedServerSocketTest.java:408) Enrico > -Alan. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From szalay.beko.mate at gmail.com Wed Jul 31 08:41:39 2019 From: szalay.beko.mate at gmail.com (=?UTF-8?B?U3phbGF5LUJla8WRIE3DoXTDqQ==?=) Date: Wed, 31 Jul 2019 10:41:39 +0200 Subject: Found different in behavior of SocketInputStream#read in JDK13 Message-ID: Hi Alan, As Alan mentioned, it is not a big deal in Zookeeper's perspective. We don't rely on this specific socket behaviour anywhere in our codebase, we only faced it when trying to emulate an erroneous client behaviour and try to assert on the way how the connection is not established successfully... Some background info: The test in Zookeeper ( https://github.com/apache/zookeeper/blob/e043c322f12d56da0fc88131628edf0731c0f8e4/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/UnifiedServerSocketTest.java#L398) simulates the case, when a client without using SSL tries to connect to the server which requires SSL. Before starting the test, we don't set the "jdk.net.usePlainSocketImpl" (so we use the new NioSocket while emulating the client behaviour), and also we set "jdk.tls.rejectClientInitiatedRenegotiation=true" to disable client-initiated TLS renegotiation (although I am not sure if that is relevant in this case or not). Using OpenJDK 13.30 on Xenial ubuntu, we saw that the socket.read() returned with -1 without throwing an exception, indicating the end of the stream. The old behaviour (e.g. using openJDK 12.0.2 on xenial) was to get a SocketException ("Connection reset") with this stack trace: [junit] 2019-07-31 08:30:54,457 [myid:] - INFO [main:UnifiedServerSocketTest at 419] - We get the expected exception: [junit] java.net.SocketException: Connection reset [junit] at org.apache.zookeeper.server.quorum.UnifiedServerSocketTest$UnifiedServerThread.run(UnifiedServerSocketTest.java:204) [junit] at java.base/java.net.SocketInputStream.read(SocketInputStream.java:186) [junit] Caused by: java.net.SocketException: Socket closed [junit] at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140) [junit] at java.base/java.net.PlainSocketImpl.socketAccept(Native Method) [junit] at java.base/java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:458) [junit] at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:556) [junit] at org.apache.zookeeper.server.quorum.UnifiedServerSocketTest.testConnectWithoutSSLToStrictServer(UnifiedServerSocketTest.java:417) [junit] at org.apache.zookeeper.server.quorum.UnifiedServerSocket.accept(UnifiedServerSocket.java:146) [junit] at org.apache.zookeeper.server.quorum.UnifiedServerSocketTest$UnifiedServerThread.run(UnifiedServerSocketTest.java:167) [junit] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [junit] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit] at java.base/java.lang.reflect.Method.invoke(Method.java:567) [junit] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit] at org.junit.runners.Suite.runChild(Suite.java:128) [junit] at org.junit.runners.Suite.runChild(Suite.java:27) [junit] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit] at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033) [junit] 2019-07-31 08:30:54,465 [myid:] - INFO [main:ZKTestCase$1 at 70] - SUCCEEDED testConnectWithoutSSLToStrictServer[0] Interestingly, if I execute the very same test with OpenJDK 13.30 on my macbook, I also get a similar exception SocketException using the NioSocket implementation. Could this suggests some differences in the native code? [junit] 2019-07-31 10:33:44,926 [myid:] - INFO [main:UnifiedServerSocketTest at 419] - We get the expected exception: [junit] java.net.SocketException: Connection reset [junit] at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:324) [junit] at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:351) [junit] at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:802) [junit] at java.base/java.net.Socket$SocketInputStream.read(Socket.java:919) [junit] at org.apache.zookeeper.server.quorum.UnifiedServerSocketTest.testConnectWithoutSSLToStrictServer(UnifiedServerSocketTest.java:417) [junit] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [junit] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit] at java.base/java.lang.reflect.Method.invoke(Method.java:567) [junit] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit] at org.junit.runners.Suite.runChild(Suite.java:128) [junit] at org.junit.runners.Suite.runChild(Suite.java:27) [junit] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit] at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033) [junit] 2019-07-31 10:33:44,931 [myid:] - INFO [main:ZKTestCase$1 at 70] - SUCCEEDED testConnectWithoutSSLToStrictServer[0] Cheers, Mate >> On Jul 30, 2019, at 18:53 PM, Alan Bateman > wrote: >> Hello, >> in Apache ZooKeeper community we discovered a difference in behavior >> in SocketInputStream. >> >> The problem is that SocketInputStream#read used to throw a >> SocketException instead of returning -1. >> >> I see that this was cited in https://openjdk.java.net/jeps/353 >> >> "The |InputStream| and |OutputStream| returned by the old >> implementation tests the stream for EOF and returns -1 before other >> checks. The new implementation does |null| and bounds checks before >> checking if the stream is at EOF. It is possible, but unlikely, that >> there is fragile code that will trip up due to the order of the checks." >> >> We are talking about a unit test, no a big deal. >> >> This is the fix, provided by Mate Szalay-Beko >> https://github.com/apache/zookeeper/pull/1029 >> >Thanks for the mail. What is the exception you see with the legacy >implementation? The note you cite would lead to an >IndexOutOfBoundException due to the bad input and I don't think this is >what you are seeing. > >-Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From simone.bordet at gmail.com Wed Jul 31 09:40:15 2019 From: simone.bordet at gmail.com (Simone Bordet) Date: Wed, 31 Jul 2019 11:40:15 +0200 Subject: SeekableByteChannel not seekable? Message-ID: Hi, seems that the SeekableByteChannel returned by Files.newByteChannel() is not seekable if the path is from a zipfs, see exception below. We do: try (SeekableByteChannel channel = Files.newByteChannel(path, StandardOpenOption.READ)) { channel.position(start); // BOOM } where path is a path from a ZipFileSystem. Is that expected behavior? Seems strange that we get a SeekableByteChannel that does not support seek - understandable given the zipfs but still weird. Maybe just needs explicit documentation? I am an OpenJDK author and can file a bug if necessary. Thanks! 2019-07-30 20:20:52.188:WARN:oejs.HttpChannel:qtp1717159510-47: /zipfs/large.dat java.lang.UnsupportedOperationException at com.sun.nio.zipfs.ZipFileSystem$3.position(ZipFileSystem.java:653) at org.eclipse.jetty.util.resource.PathResource.writeTo(PathResource.java:591) at org.eclipse.jetty.server.ResourceService.sendData(ResourceService.java:755) at org.eclipse.jetty.server.ResourceService.doGet(ResourceService.java:294) at org.eclipse.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:454) -- Simone Bordet --- Finally, no matter how good the architecture and design are, to deliver bug-free software with optimal performance and reliability, the implementation technique must be flawless. Victoria Livschitz From fweimer at redhat.com Wed Jul 31 10:14:17 2019 From: fweimer at redhat.com (Florian Weimer) Date: Wed, 31 Jul 2019 12:14:17 +0200 Subject: SeekableByteChannel not seekable? In-Reply-To: (Simone Bordet's message of "Wed, 31 Jul 2019 11:40:15 +0200") References: Message-ID: <87v9viqznq.fsf@oldenburg2.str.redhat.com> * Simone Bordet: > Is that expected behavior? Seems strange that we get a > SeekableByteChannel that does not support seek - understandable given > the zipfs but still weird. > > Maybe just needs explicit documentation? I think that in practice, FileChannel#position(long) has always been an optional operation because it does not work on non-regular files. This isn't specific to zipfs, it can happen with regular file system paths as well (if they refer to character devices or FIFOs, for example). Thanks, Florian From adinn at redhat.com Wed Jul 31 10:55:31 2019 From: adinn at redhat.com (Andrew Dinn) Date: Wed, 31 Jul 2019 11:55:31 +0100 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: <671c7ba1-9597-a990-190b-145d9cb5349e@redhat.com> References: <80da32b2-7acb-7b94-b82c-5dcd5cf95539@redhat.com> <36326F5E-12CD-487A-8FE2-1049631FE908@oracle.com> <8B1A8EDC-F9D7-4085-A34F-69100DBD7D5C@oracle.com> <5c78b5d7-dc9e-fbbd-41a9-5139ed6ee32c@redhat.com> <86d000f3-9eca-6089-5f7e-5698444d99ce@oracle.com> <7fe8d17d-6170-fcc9-b87a-eccfda2bb546@redhat.com> <7b39f68d-5698-4079-425c-c86ec161e361@oracle.com> <72810832-eb88-3418-f336-b95af95d9dcc@redhat.com> <20c8bbcb-2cd3-77af-cd40-29f7bd752166@oracle.com> <68566847-de80-82af-a928-d30c75f2b1b2@redhat.com> <892f3b0b-55f1-872d-15f4-5af907fa8437@redhat.com> <8f72d3b5-3a01-a7de-3b09-35571266a87d@redhat.com> <671c7ba1-9597-a990-190b-145d9cb5349e@redhat.com> Message-ID: <81098801-31b2-e3a8-5c25-043ab47c80bc@redhat.com> Hi Aleksey On 30/07/2019 17:00, Aleksey Shipilev wrote: > On 7/30/19 5:04 PM, Andrew Dinn wrote: >> JEP 352 has now been targeted for inclusion in JDK14. The latest webrev >> for the implementation JIRA has been rebased to apply to the current >> tree. Is it now ok to push this change set? >> >> JIRA: https://bugs.openjdk.java.net/browse/JDK-8224974 >> webrev: http://cr.openjdk.java.net/~adinn/8224974/webrev.09 > Is it okay to have a late code review? Here it goes: Sure, welcome to the party even if most of the booze has been drunk :-) Thank you for a very thorough review. Comments (mostly in agreement) are inline below. They include a few refusals and a walk past some of the jumps pending confirmation they truly form part of the obstacle course. A new webrev against the latest jdk/jdk dev tree including all uncontested changes is here: http://cr.openjdk.java.net/~adinn/8224974/webrev.10/ > === General: > > So if pre wbsync is no-op, why do we need to handle it everywhere? We seem to be falling through all > the way to the stub to do nothing there, maybe we should instead cut much earlier, e.g. when > inlining Unsafe.writeBackPresync0? Would it be better to not emit CacheWBPreSync at all? Ah, I see you brought your own bottle ;-) The pre sync is definitely not needed at present. However, I put it there because I didn't know for sure if some future port of this capability (e.g. to ppc) might need to sync prior writes before writing back cache lines. [Indeed, the old Intel documentation stated that pre-sync was needed on x86 for clflush to be safe but it is definitely not needed.] Anyway, folding out the pre-sync as you suggest would mean taking what is still potentially an architecture-specific decision in generic code. That may well prove to be a redundant precaution but it also costs little -- every affected /compile/ will last a tad longer and be a tad fatter in its memory footprint. If you think change is important I will happily oblige. > === General: > > IIRC, notproduct and PRODUCT defines are legacy, and should be avoided? develop and ASSERT must be > the substitutes for this. See some discussion here: https://bugs.openjdk.java.net/browse/JDK-8183287 Ok done. I changed #ifndef PRODUCT to #ifdef ASSERT in library_call.cpp and unsafe.cpp. I also redefined global flag TraceMemoryWriteback using develop rather than notproduct. I also removed the AArch64 /product/ compiler option UsePOCForPOP. This allowed the JIT to use a memory writeback instruction with weakened semantics if the full writeback is not available in a product build. product was chosen because this was needed to establish a benchmark for writeback on existing kit when using a pseudo-NVRAM memory device based on volatile memory. The point was to be able to do an apples to apples comparison with writeback to a disk device. Obviously, it's not much use doing that test with a a non-product build. This is only important during development as a one-off test once the OS support for pseudo-NVRAM devices arrives in Linux AArch64 (which should happen well before new kit with the POP writeback arrives). I can still perform it using a custom build so I dropped this from the patch. > === src/hotspot/cpu/aarch64/aarch64.ad > src/hotspot/cpu/x86/x86.ad > > This should probably be just "!VM_Version...". Braces around the statement would not hurt either. > > 2196 if (VM_Version::supports_data_cache_line_flush() == false) > 2197 ret_value = false; Done. > === src/hotspot/cpu/aarch64/macroAssembler_aarch64.cpp > src/hotspot/cpu/x86/macroAssembler_x86.cpp > > Braces style mismatch, should be on the same line, as in the rest of the file: > > 5837 void MacroAssembler::cache_wb(Address line) > 5838 { > > 5856 void MacroAssembler::cache_wbsync(bool is_pre) > 5857 { Done. > === src/hotspot/cpu/x86/assembler_x86.cpp > > It feels like these comments are redundant, especially L8630 and L8646 which mention magic values > "6" and "7", not present in the code: > > 8624 // instruction prefix is 0x66 > > 8627 // opcode family is 0x0f 0xAE > > 8630 // extended opcode byte is 7 == rdi > > 8640 // instruction prefix is 0x66 > > 8643 // opcode family is 0x0f 0xAE > > 8646 // extended opcode byte is 6 == rsi Well, I think the question of redundancy depends on how you look at it. The comments are not there to explain that these values are the ones being passed to the emit functions -- that is pretty obvious. They are there to explain what those values mean i.e. to explain what elements of the fully assembled instruction the emit calls assembling piecemeal. Perhaps that reading woudl be clearer if each such equation of terms was done in reverse order: 8624 // 0x66 is instruction prefix 8627 // 0x0f 0xAE is opcode family 8630 // rdi == 7 is extended opcode byte . . . Given that the code is simply stuffing numbers (whether supplied as literals or as symbolic constants) into a byte stream I think these comments are a help when it comes to cross-checking each specific assembly against the corresponding numbers declared in the Intel manuals. So, I don't really want to remove them. Would you prefer me to reverse the wording as above? > === src/hotspot/cpu/x86/macroAssembler_x86.cpp > > These comments feel redundant too. Well, maybe they should instead talk about the choices the > subsequent code makes. > > 9915 // pick the correct implementation > > 9936 // pick the correct implementation I agree it would help to expand the comment to explain the logic of the choice in the first case: // prefer clwb (potentially parallel writeback without evict) // otherwise prefer clflushopt (potentially parallel writeback // with evict) // otherwise fallback on clflush (serial writeback with evict) In the second case the comment is redundant because the need for an sfence is covered by the existing comment inside the if: // need an sfence for post flush when using clflushopt or clwb // otherwise no no need for any synchroniaztion > === src/hotspot/cpu/x86/macroAssembler_x86.cpp > src/hotspot/cpu/x86/vm_version_x86.hpp > > The docs say "The CLFLUSH instruction was introduced with the SSE2 extensions; however, because it > has its own CPUID feature flag, it can be implemented in IA-32 processors that do not include the > SSE2 extensions. Also, detecting the presence of the SSE2 extensions with the CPUID instruction does > not guarantee that the CLFLUSH instruction is implemented in the processor." > > Yet, we have: > > 9910 // 64 bit cpus always support clflush > 9911 assert(VM_Version::supports_clflush(), "should not reach here on 32-bit"); > > 505 #ifdef _LP64 > 506 // cpflush is always available on x86_64 > 507 result |= CPU_FLUSH; > 508 #else > 509 if (_cpuid_info.std_cpuid1_edx.bits.clflush != 0) > 510 result |= CPU_FLUSH; > 511 #endif > > 967 // 64 bit cpus always support clflush which writes back and evicts > 968 // on 32 bit cpus support is recorded via a feature flag > > 980 static bool supports_clflush() { return true; } > > I think the assert should just say "clflush should be available", and clflush cpu flag detected for > 64-bits too? I think that documentation is only talking about x86_32. I remember reading somewhere that clflush is always guaranteed to be available on x86_64 only I cannot remember where (I think maybe it was in Intel's libpmem code?). Anyway, I do not take the above to imply that clflush has been, or even can be, omitted on a 64-bit processor. However, let us assume that it can in principle be omitted from an x86_64 implementation. If so then I'll suggest that such a beast is a thing only in the sense that the Kingdom of Antarctica is a thing (i.e. it is an unactualized possible). As a non-existence proof I'll note that the JVM already relies implicitly (i.e. without checking any CPU bits) on being able to call clflush on x86_64 (see icache_x86.cpp). Anyway, granted the possibility of said Magic Kingdom I have changed the assert to say "clflush should be available". I have also i) modified setting of the CPU_FLUSH flag to test the relevant bit on both 32 and 64 bits + // cpflush should always be available on x86_64 + // test the flag anyway and assert later + if (_cpuid_info.std_cpuid1_edx.bits.clflush != 0) + result |= CPU_FLUSH; . . . ii) added an assert that it is set to the 64-bit version of supports_clflush: +#ifdef _LP64 + static bool supports_clflush() { + assert((_features & CPU_FLUSH) != 0, "clflush should be available"); + return true; + } . . . > === src/hotspot/cpu/x86/macroAssembler_x86.hpp > > Accidental camelCase, while hotspot code expects snake_case: > > 1794 void cache_wbsync(bool isPre); Fixed. > === src/hotspot/cpu/x86/stubGenerator_x86_64.cpp > > Any reason to avoid inline here, e.g. "__ cache_wb(Address(src, 0))"? > > 2921 const Address line(src, 0); > 2922 __ cache_wb(line); Well, I guess only if you're "not into that whole brevity thing" (c.f. The Big Lebowski). Condensed accordingly. > === src/hotspot/cpu/x86/stubGenerator_x86_64.cpp > > Is it really "cmpl" here, not "cmpb"? I think aarch64 code tests for byte. > > 2942 __ cmpl(is_pre, 0); This is a Java boolean input. I believe that means the value will be loaded into c_arg0 as an int so this test ought to be adequate. The Arch64 code actually tests the input value as a long word: __ enter(); __ cbnz(is_pre, skip); . . . n.b. cbnz == compare and branch if non-zero (64 bit). This latter code could (arguably) employ a cbnzw (i.e. 32-bit variant). However, the argument will have been loaded into c_rarg0 using an AArch64 int load which guarantees to zero the top 32 bits. So, it doesn't really make any difference whether this uses cbnz or cbnzw. > === src/hotspot/share/classfile/vmSymbols.hpp > > Excess space before "jdk_internal..." here: > > 1096 do_intrinsic(_writebackPostSync0, jdk_internal_misc_Unsafe, > writebackPostSync0_name, void_method_signature , F_RN) \ Fixed. > === src/hotspot/share/opto/c2compiler.cpp > > Why inject new cases here, instead of at the end of switch? Saves sudden "break": > > 578 break; > 579 case vmIntrinsics::_writeback0: > 580 if (!Matcher::match_rule_supported(Op_CacheWB)) return false; > 581 break; > 582 case vmIntrinsics::_writebackPreSync0: > 583 if (!Matcher::match_rule_supported(Op_CacheWBPreSync)) return false; > 584 break; > 585 case vmIntrinsics::_writebackPostSync0: > 586 if (!Matcher::match_rule_supported(Op_CacheWBPostSync)) return false; > 587 break; I placed them here so they were close to the other Unsafe intrinsics. In particular they precede _allocateInstance, an ordering which is also the case in the declarations in vmSymbols.hpp. In what sense do you mean that an extra 'break' is saved? That would be true as regards the textual layout. It wouldn't affect the logic of folding different ranges of values into branching range tests (which is only determined by the numeric values of the intrinsics). If you are concerned about the former then I would argue that placing the values in declaration order seems to me to be the more important concern. > === src/hotspot/share/opto/library_call.cpp > > Accidental camelCase here, should be snake_case: > > 257 bool inline_unsafe_writebackSync0(bool isPre); > > 2870 bool LibraryCallKit::inline_unsafe_writebackSync0(bool isPre) { Fixed. > === src/hotspot/share/prims/unsafe.cpp > > Odd indenting near "CC": > > 1122 {CC "writeback0", CC "(" "J" ")V", FN_PTR(Unsafe_WriteBack0)}, > 1123 {CC "writebackPreSync0", CC "()V", FN_PTR(Unsafe_WriteBackPreSync0)}, > 1124 {CC "writebackPostSync0", CC "()V", FN_PTR(Unsafe_WriteBackPostSync0)}, Fixed. > camelCase: > 464 static void doWriteBackSync0(bool isPre) Fixed. > === src/hotspot/share/prims/unsafe.cpp > > Do we really need this function pointer mess? > > 457 void (*wb)(void *); > 458 void *a = addr_from_java(line); > 459 wb = (void (*)(void *)) StubRoutines::data_cache_writeback(); > 460 assert(wb != NULL, "generate writeback stub!"); > 461 (*wb)(a); > > Seems easier to: > > assert(StubRoutines::data_cache_writeback() != NULL, "sanity"); > StubRoutines::data_cache_writeback()(addr_from_java(line)); Hmm, "that whole brevity thing" again? Well, I guess you must now call me "El Duderino". Are you sure that all the compilers used to build openJDK will happily eat the second line of your replacement? If you can guarantee that I'll happily remove the type declarations. regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From adinn at redhat.com Wed Jul 31 11:29:15 2019 From: adinn at redhat.com (Andrew Dinn) Date: Wed, 31 Jul 2019 12:29:15 +0100 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: <81098801-31b2-e3a8-5c25-043ab47c80bc@redhat.com> References: <80da32b2-7acb-7b94-b82c-5dcd5cf95539@redhat.com> <36326F5E-12CD-487A-8FE2-1049631FE908@oracle.com> <8B1A8EDC-F9D7-4085-A34F-69100DBD7D5C@oracle.com> <5c78b5d7-dc9e-fbbd-41a9-5139ed6ee32c@redhat.com> <86d000f3-9eca-6089-5f7e-5698444d99ce@oracle.com> <7fe8d17d-6170-fcc9-b87a-eccfda2bb546@redhat.com> <7b39f68d-5698-4079-425c-c86ec161e361@oracle.com> <72810832-eb88-3418-f336-b95af95d9dcc@redhat.com> <20c8bbcb-2cd3-77af-cd40-29f7bd752166@oracle.com> <68566847-de80-82af-a928-d30c75f2b1b2@redhat.com> <892f3b0b-55f1-872d-15f4-5af907fa8437@redhat.com> <8f72d3b5-3a01-a7de-3b09-35571266a87d@redhat.com> <671c7ba1-9597-a990-190b-145d9cb5349e@redhat.com> <81098801-31b2-e3a8-5c25-043ab47c80bc@redhat.com> Message-ID: <27916ded-74e2-94d8-891c-95d87730b69d@redhat.com> Hi Aleksey, I have an update regarding the change to the computation of the CPU_FLUSH flag. After posting the new webrev I built a debug version with the change that tests the clflush bit on x86_64. It crashed when the assert is reached in VM_Version::supports_cpuflush: # A fatal error has been detected by the Java Runtime Environment: # # Internal Error (src/hotspot/cpu/x86/vm_version_x86.hpp:978), pid=29597, tid=29602 # assert((_features & ((uint64_t)(0x20000000000ULL))) != 0) failed: clflush should be available So, it seems the clflush bit is not set on my x86_64 box even though I am able to execute clflush quite happily. "Toto, it looks like we are no longer in Antarctica." So, I will revert this change (in the next webrev). regards, Andrew Dinn ----------- On 31/07/2019 11:55, Andrew Dinn wrote: > Hi Aleksey > > On 30/07/2019 17:00, Aleksey Shipilev wrote: >> On 7/30/19 5:04 PM, Andrew Dinn wrote: >>> JEP 352 has now been targeted for inclusion in JDK14. The latest webrev >>> for the implementation JIRA has been rebased to apply to the current >>> tree. Is it now ok to push this change set? >>> >>> JIRA: https://bugs.openjdk.java.net/browse/JDK-8224974 >>> webrev: http://cr.openjdk.java.net/~adinn/8224974/webrev.09 >> Is it okay to have a late code review? Here it goes: > > Sure, welcome to the party even if most of the booze has been drunk :-) > > Thank you for a very thorough review. Comments (mostly in agreement) are > inline below. They include a few refusals and a walk past some of the > jumps pending confirmation they truly form part of the obstacle course. > > A new webrev against the latest jdk/jdk dev tree including all > uncontested changes is here: > > http://cr.openjdk.java.net/~adinn/8224974/webrev.10/ > >> === General: >> >> So if pre wbsync is no-op, why do we need to handle it everywhere? We seem to be falling through all >> the way to the stub to do nothing there, maybe we should instead cut much earlier, e.g. when >> inlining Unsafe.writeBackPresync0? Would it be better to not emit CacheWBPreSync at all? > > Ah, I see you brought your own bottle ;-) > > The pre sync is definitely not needed at present. However, I put it > there because I didn't know for sure if some future port of this > capability (e.g. to ppc) might need to sync prior writes before writing > back cache lines. [Indeed, the old Intel documentation stated that > pre-sync was needed on x86 for clflush to be safe but it is definitely > not needed.] > > Anyway, folding out the pre-sync as you suggest would mean taking what > is still potentially an architecture-specific decision in generic code. > That may well prove to be a redundant precaution but it also costs > little -- every affected /compile/ will last a tad longer and be a tad > fatter in its memory footprint. If you think change is important I will > happily oblige. > >> === General: >> >> IIRC, notproduct and PRODUCT defines are legacy, and should be avoided? develop and ASSERT must be >> the substitutes for this. See some discussion here: https://bugs.openjdk.java.net/browse/JDK-8183287 > > Ok done. > > I changed #ifndef PRODUCT to #ifdef ASSERT in library_call.cpp and > unsafe.cpp. I also redefined global flag TraceMemoryWriteback using > develop rather than notproduct. > > I also removed the AArch64 /product/ compiler option UsePOCForPOP. This > allowed the JIT to use a memory writeback instruction with weakened > semantics if the full writeback is not available in a product build. > product was chosen because this was needed to establish a benchmark for > writeback on existing kit when using a pseudo-NVRAM memory device based > on volatile memory. The point was to be able to do an apples to apples > comparison with writeback to a disk device. Obviously, it's not much > use doing that test with a a non-product build. This is only important > during development as a one-off test once the OS support for > pseudo-NVRAM devices arrives in Linux AArch64 (which should happen well > before new kit with the POP writeback arrives). I can still perform it > using a custom build so I dropped this from the patch. > >> === src/hotspot/cpu/aarch64/aarch64.ad >> src/hotspot/cpu/x86/x86.ad >> >> This should probably be just "!VM_Version...". Braces around the statement would not hurt either. >> >> 2196 if (VM_Version::supports_data_cache_line_flush() == false) >> 2197 ret_value = false; > > Done. > >> === src/hotspot/cpu/aarch64/macroAssembler_aarch64.cpp >> src/hotspot/cpu/x86/macroAssembler_x86.cpp >> >> Braces style mismatch, should be on the same line, as in the rest of the file: >> >> 5837 void MacroAssembler::cache_wb(Address line) >> 5838 { >> >> 5856 void MacroAssembler::cache_wbsync(bool is_pre) >> 5857 { > > Done. > >> === src/hotspot/cpu/x86/assembler_x86.cpp >> >> It feels like these comments are redundant, especially L8630 and L8646 which mention magic values >> "6" and "7", not present in the code: >> >> 8624 // instruction prefix is 0x66 >> >> 8627 // opcode family is 0x0f 0xAE >> >> 8630 // extended opcode byte is 7 == rdi >> >> 8640 // instruction prefix is 0x66 >> >> 8643 // opcode family is 0x0f 0xAE >> >> 8646 // extended opcode byte is 6 == rsi > > Well, I think the question of redundancy depends on how you look at it. > The comments are not there to explain that these values are the ones > being passed to the emit functions -- that is pretty obvious. They are > there to explain what those values mean i.e. to explain what elements of > the fully assembled instruction the emit calls assembling piecemeal. > Perhaps that reading woudl be clearer if each such equation of terms was > done in reverse order: > > 8624 // 0x66 is instruction prefix > > 8627 // 0x0f 0xAE is opcode family > > 8630 // rdi == 7 is extended opcode byte > . . . > > Given that the code is simply stuffing numbers (whether supplied as > literals or as symbolic constants) into a byte stream I think these > comments are a help when it comes to cross-checking each specific > assembly against the corresponding numbers declared in the Intel > manuals. So, I don't really want to remove them. Would you prefer me to > reverse the wording as above? > >> === src/hotspot/cpu/x86/macroAssembler_x86.cpp >> >> These comments feel redundant too. Well, maybe they should instead talk about the choices the >> subsequent code makes. >> >> 9915 // pick the correct implementation >> >> 9936 // pick the correct implementation > > I agree it would help to expand the comment to explain the logic of the > choice in the first case: > > // prefer clwb (potentially parallel writeback without evict) > // otherwise prefer clflushopt (potentially parallel writeback > // with evict) > // otherwise fallback on clflush (serial writeback with evict) > > In the second case the comment is redundant because the need for an > sfence is covered by the existing comment inside the if: > > // need an sfence for post flush when using clflushopt or clwb > // otherwise no no need for any synchroniaztion > >> === src/hotspot/cpu/x86/macroAssembler_x86.cpp >> src/hotspot/cpu/x86/vm_version_x86.hpp >> >> The docs say "The CLFLUSH instruction was introduced with the SSE2 extensions; however, because it >> has its own CPUID feature flag, it can be implemented in IA-32 processors that do not include the >> SSE2 extensions. Also, detecting the presence of the SSE2 extensions with the CPUID instruction does >> not guarantee that the CLFLUSH instruction is implemented in the processor." >> >> Yet, we have: >> >> 9910 // 64 bit cpus always support clflush >> 9911 assert(VM_Version::supports_clflush(), "should not reach here on 32-bit"); >> >> 505 #ifdef _LP64 >> 506 // cpflush is always available on x86_64 >> 507 result |= CPU_FLUSH; >> 508 #else >> 509 if (_cpuid_info.std_cpuid1_edx.bits.clflush != 0) >> 510 result |= CPU_FLUSH; >> 511 #endif >> >> 967 // 64 bit cpus always support clflush which writes back and evicts >> 968 // on 32 bit cpus support is recorded via a feature flag >> >> 980 static bool supports_clflush() { return true; } >> >> I think the assert should just say "clflush should be available", and clflush cpu flag detected for >> 64-bits too? > > I think that documentation is only talking about x86_32. I remember > reading somewhere that clflush is always guaranteed to be available on > x86_64 only I cannot remember where (I think maybe it was in Intel's > libpmem code?). Anyway, I do not take the above to imply that clflush > has been, or even can be, omitted on a 64-bit processor. However, let us > assume that it can in principle be omitted from an x86_64 > implementation. If so then I'll suggest that such a beast is a thing > only in the sense that the Kingdom of Antarctica is a thing (i.e. it is > an unactualized possible). As a non-existence proof I'll note that the > JVM already relies implicitly (i.e. without checking any CPU bits) on > being able to call clflush on x86_64 (see icache_x86.cpp). > > Anyway, granted the possibility of said Magic Kingdom I have changed the > assert to say "clflush should be available". I have also > > i) modified setting of the CPU_FLUSH flag to test the relevant bit on > both 32 and 64 bits > > + // cpflush should always be available on x86_64 > + // test the flag anyway and assert later > + if (_cpuid_info.std_cpuid1_edx.bits.clflush != 0) > + result |= CPU_FLUSH; > . . . > > ii) added an assert that it is set to the 64-bit version of > supports_clflush: > > +#ifdef _LP64 > + static bool supports_clflush() { > + assert((_features & CPU_FLUSH) != 0, "clflush should be available"); > + return true; > + } > . . . > >> === src/hotspot/cpu/x86/macroAssembler_x86.hpp >> >> Accidental camelCase, while hotspot code expects snake_case: >> >> 1794 void cache_wbsync(bool isPre); > > Fixed. > >> === src/hotspot/cpu/x86/stubGenerator_x86_64.cpp >> >> Any reason to avoid inline here, e.g. "__ cache_wb(Address(src, 0))"? >> >> 2921 const Address line(src, 0); >> 2922 __ cache_wb(line); > > Well, I guess only if you're "not into that whole brevity thing" (c.f. > The Big Lebowski). Condensed accordingly. > >> === src/hotspot/cpu/x86/stubGenerator_x86_64.cpp >> >> Is it really "cmpl" here, not "cmpb"? I think aarch64 code tests for byte. >> >> 2942 __ cmpl(is_pre, 0); > > This is a Java boolean input. I believe that means the value will be > loaded into c_arg0 as an int so this test ought to be adequate. > > The Arch64 code actually tests the input value as a long word: > > __ enter(); > __ cbnz(is_pre, skip); > . . . > > n.b. cbnz == compare and branch if non-zero (64 bit). This latter code > could (arguably) employ a cbnzw (i.e. 32-bit variant). However, the > argument will have been loaded into c_rarg0 using an AArch64 int load > which guarantees to zero the top 32 bits. So, it doesn't really make any > difference whether this uses cbnz or cbnzw. > >> === src/hotspot/share/classfile/vmSymbols.hpp >> >> Excess space before "jdk_internal..." here: >> >> 1096 do_intrinsic(_writebackPostSync0, jdk_internal_misc_Unsafe, >> writebackPostSync0_name, void_method_signature , F_RN) \ > > Fixed. > >> === src/hotspot/share/opto/c2compiler.cpp >> >> Why inject new cases here, instead of at the end of switch? Saves sudden "break": >> >> 578 break; >> 579 case vmIntrinsics::_writeback0: >> 580 if (!Matcher::match_rule_supported(Op_CacheWB)) return false; >> 581 break; >> 582 case vmIntrinsics::_writebackPreSync0: >> 583 if (!Matcher::match_rule_supported(Op_CacheWBPreSync)) return false; >> 584 break; >> 585 case vmIntrinsics::_writebackPostSync0: >> 586 if (!Matcher::match_rule_supported(Op_CacheWBPostSync)) return false; >> 587 break; > > I placed them here so they were close to the other Unsafe intrinsics. In > particular they precede _allocateInstance, an ordering which is also the > case in the declarations in vmSymbols.hpp. > > In what sense do you mean that an extra 'break' is saved? That would be > true as regards the textual layout. It wouldn't affect the logic of > folding different ranges of values into branching range tests (which is > only determined by the numeric values of the intrinsics). If you are > concerned about the former then I would argue that placing the values in > declaration order seems to me to be the more important concern. > >> === src/hotspot/share/opto/library_call.cpp >> >> Accidental camelCase here, should be snake_case: >> >> 257 bool inline_unsafe_writebackSync0(bool isPre); >> >> 2870 bool LibraryCallKit::inline_unsafe_writebackSync0(bool isPre) { > > Fixed. > >> === src/hotspot/share/prims/unsafe.cpp >> >> Odd indenting near "CC": >> >> 1122 {CC "writeback0", CC "(" "J" ")V", FN_PTR(Unsafe_WriteBack0)}, >> 1123 {CC "writebackPreSync0", CC "()V", FN_PTR(Unsafe_WriteBackPreSync0)}, >> 1124 {CC "writebackPostSync0", CC "()V", FN_PTR(Unsafe_WriteBackPostSync0)}, > > Fixed. > >> camelCase: >> 464 static void doWriteBackSync0(bool isPre) > > Fixed. > >> === src/hotspot/share/prims/unsafe.cpp >> >> Do we really need this function pointer mess? >> >> 457 void (*wb)(void *); >> 458 void *a = addr_from_java(line); >> 459 wb = (void (*)(void *)) StubRoutines::data_cache_writeback(); >> 460 assert(wb != NULL, "generate writeback stub!"); >> 461 (*wb)(a); >> >> Seems easier to: >> >> assert(StubRoutines::data_cache_writeback() != NULL, "sanity"); >> StubRoutines::data_cache_writeback()(addr_from_java(line)); > Hmm, "that whole brevity thing" again? Well, I guess you must now call > me "El Duderino". > > Are you sure that all the compilers used to build openJDK will happily > eat the second line of your replacement? If you can guarantee that I'll > happily remove the type declarations. > > regards, > > > Andrew Dinn > ----------- > Senior Principal Software Engineer > Red Hat UK Ltd > Registered in England and Wales under Company Registration No. 03798903 > Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander > -- regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From shade at redhat.com Wed Jul 31 11:40:06 2019 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 31 Jul 2019 13:40:06 +0200 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: <27916ded-74e2-94d8-891c-95d87730b69d@redhat.com> References: <80da32b2-7acb-7b94-b82c-5dcd5cf95539@redhat.com> <36326F5E-12CD-487A-8FE2-1049631FE908@oracle.com> <8B1A8EDC-F9D7-4085-A34F-69100DBD7D5C@oracle.com> <5c78b5d7-dc9e-fbbd-41a9-5139ed6ee32c@redhat.com> <86d000f3-9eca-6089-5f7e-5698444d99ce@oracle.com> <7fe8d17d-6170-fcc9-b87a-eccfda2bb546@redhat.com> <7b39f68d-5698-4079-425c-c86ec161e361@oracle.com> <72810832-eb88-3418-f336-b95af95d9dcc@redhat.com> <20c8bbcb-2cd3-77af-cd40-29f7bd752166@oracle.com> <68566847-de80-82af-a928-d30c75f2b1b2@redhat.com> <892f3b0b-55f1-872d-15f4-5af907fa8437@redhat.com> <8f72d3b5-3a01-a7de-3b09-35571266a87d@redhat.com> <671c7ba1-9597-a990-190b-145d9cb5349e@redhat.com> <81098801-31b2-e3a8-5c25-043ab47c80bc@redhat.com> <27916ded-74e2-94d8-891c-95d87730b69d@redhat.com> Message-ID: On 7/31/19 1:29 PM, Andrew Dinn wrote: > I have an update regarding the change to the computation of the > CPU_FLUSH flag. After posting the new webrev I built a debug version > with the change that tests the clflush bit on x86_64. It crashed when > the assert is reached in VM_Version::supports_cpuflush: > > # A fatal error has been detected by the Java Runtime Environment: > # > # Internal Error (src/hotspot/cpu/x86/vm_version_x86.hpp:978), > pid=29597, tid=29602 > # assert((_features & ((uint64_t)(0x20000000000ULL))) != 0) failed: > clflush should be available > > So, it seems the clflush bit is not set on my x86_64 box even though I > am able to execute clflush quite happily. > > "Toto, it looks like we are no longer in Antarctica." > > So, I will revert this change (in the next webrev). But wait, that might mean the clflush is indeed not available, by either hardware or software switch? I believe some security mitigations disable clflush. Linux kernel, for example, has "noclflush" option to do this. Ignoring the cpu bit and emitting clflush anyway might circumvent that mitigation then? -Aleksey -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From Alan.Bateman at oracle.com Wed Jul 31 12:26:47 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 31 Jul 2019 05:26:47 -0700 Subject: SeekableByteChannel not seekable? In-Reply-To: References: Message-ID: <50f569dd-bc48-a640-12df-469648b7347a@oracle.com> On 31/07/2019 02:40, Simone Bordet wrote: > Hi, > > seems that the SeekableByteChannel returned by Files.newByteChannel() > is not seekable if the path is from a zipfs, see exception below. > > We do: > > try (SeekableByteChannel channel = Files.newByteChannel(path, > StandardOpenOption.READ)) > { > channel.position(start); // BOOM > } > > where path is a path from a ZipFileSystem. > > Is that expected behavior? Seems strange that we get a > SeekableByteChannel that does not support seek - understandable given > the zipfs but still weird. Which JDK release or build is this? This part of the zipfs code was re-implemented in JDK 12 so that it does support SeekableByteChannel and the position methods should work correctly. -Alan From Alan.Bateman at oracle.com Wed Jul 31 12:38:53 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 31 Jul 2019 05:38:53 -0700 Subject: Found different in behavior of SocketInputStream#read in JDK13 In-Reply-To: References: Message-ID: Thanks for additional info. Although it may not be a "a big deal" for the Zookeeper test, it would be great if you and Enrico have time to help us diagnose this a big further. At a high-level, the SocketException: "Connection reset" means the peer has terminated the connection (hard reset). When you are using new SocketImpl the peer appears to be closely the connection gracefully, help -1 is read. Would I be correct to say that UnifiedServerSocketTest is connecting to the Zookeeper server and it terminates the connection due to the SSL connection. Is the Zookeeper running on JDK 13 too? I'm just wondering if the scenario intermittently leads to a hard reset on some occasions and a graceful close at some times and switching implementation has changed the timing a bit to cause the latter to be more likely. Is there any way to turn this test in a reproducer that we could study more closely? -Alan On 31/07/2019 01:41, Szalay-Bek? M?t? wrote: > Hi Alan, > > As Alan mentioned, it is not a big deal in Zookeeper's perspective. We > don't rely on this specific socket behaviour anywhere in our codebase, > we only faced it when trying to emulate an erroneous client behaviour > and try to assert on the way how the connection is not established > successfully... > > Some background info: > > The test in Zookeeper > (https://github.com/apache/zookeeper/blob/e043c322f12d56da0fc88131628edf0731c0f8e4/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/UnifiedServerSocketTest.java#L398) > simulates the case, when a client without using SSL tries to connect > to the server which requires SSL. Before starting the test, we don't > set the "jdk.net.usePlainSocketImpl" (so we use the new NioSocket > while emulating the client behaviour), and also we set > "jdk.tls.rejectClientInitiatedRenegotiation=true" to disable > client-initiated TLS renegotiation (although I am not sure if that is > relevant in this case or not). > > Using OpenJDK 13.30 on Xenial ubuntu, we saw that the socket.read() > returned with -1 without throwing an exception, indicating the end of > the stream. > > The old behaviour (e.g. using openJDK 12.0.2 on xenial) was to get a > SocketException ("Connection reset") with this stack trace: > > ? ? [junit] 2019-07-31 08:30:54,457 [myid:] - INFO > ?[main:UnifiedServerSocketTest at 419] - We get the expected exception: > ? ? [junit] java.net.SocketException: Connection reset > ? ? [junit] at > org.apache.zookeeper.server.quorum.UnifiedServerSocketTest$UnifiedServerThread.run(UnifiedServerSocketTest.java:204) > ? ? [junit] at > java.base/java.net.SocketInputStream.read(SocketInputStream.java:186) > ? ? [junit] Caused by: java.net.SocketException: Socket closed > ? ? [junit] at > java.base/java.net.SocketInputStream.read(SocketInputStream.java:140) > ? ? [junit] at java.base/java.net.PlainSocketImpl.socketAccept(Native > Method) > ? ? [junit] at > java.base/java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:458) > ? ? [junit] at > java.base/java.net.ServerSocket.implAccept(ServerSocket.java:556) > ? ? [junit] at > org.apache.zookeeper.server.quorum.UnifiedServerSocketTest.testConnectWithoutSSLToStrictServer(UnifiedServerSocketTest.java:417) > ? ? [junit] at > org.apache.zookeeper.server.quorum.UnifiedServerSocket.accept(UnifiedServerSocket.java:146) > ? ? [junit] at > org.apache.zookeeper.server.quorum.UnifiedServerSocketTest$UnifiedServerThread.run(UnifiedServerSocketTest.java:167) > ? ? [junit] at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > ? ? [junit] at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ? ? [junit] at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ? ? [junit] at java.base/java.lang.reflect.Method.invoke(Method.java:567) > ? ? [junit] at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > ? ? [junit] at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > ? ? [junit] at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > ? ? [junit] at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > ? ? [junit] at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > ? ? [junit] at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > ? ? [junit] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > ? ? [junit] at org.junit.rules.RunRules.evaluate(RunRules.java:20) > ? ? [junit] at > org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > ? ? [junit] at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > ? ? [junit] at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > ? ? [junit] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > ? ? [junit] at > org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > ? ? [junit] at > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > ? ? [junit] at > org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > ? ? [junit] at > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > ? ? [junit] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > ? ? [junit] at org.junit.runners.Suite.runChild(Suite.java:128) > ? ? [junit] at org.junit.runners.Suite.runChild(Suite.java:27) > ? ? [junit] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > ? ? [junit] at > org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > ? ? [junit] at > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > ? ? [junit] at > org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > ? ? [junit] at > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > ? ? [junit] at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > ? ? [junit] at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > ? ? [junit] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > ? ? [junit] at > junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38) > ? ? [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535) > ? ? [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182) > ? ? [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033) > ? ? [junit] 2019-07-31 08:30:54,465 [myid:] - INFO > ?[main:ZKTestCase$1 at 70] - SUCCEEDED testConnectWithoutSSLToStrictServer[0] > > > > Interestingly, if I execute the very same test with OpenJDK 13.30 on > my macbook, I also get a similar exception SocketException using the > NioSocket implementation. Could this suggests some differences in the > native code? > > ? ? [junit] 2019-07-31 10:33:44,926 [myid:] - INFO > ?[main:UnifiedServerSocketTest at 419] - We get the expected exception: > ? ? [junit] java.net.SocketException: Connection reset > ? ? [junit] at > java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:324) > ? ? [junit] at > java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:351) > ? ? [junit] at > java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:802) > ? ? [junit] at > java.base/java.net.Socket$SocketInputStream.read(Socket.java:919) > ? ? [junit] at > org.apache.zookeeper.server.quorum.UnifiedServerSocketTest.testConnectWithoutSSLToStrictServer(UnifiedServerSocketTest.java:417) > ? ? [junit] at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > ? ? [junit] at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ? ? [junit] at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ? ? [junit] at java.base/java.lang.reflect.Method.invoke(Method.java:567) > ? ? [junit] at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > ? ? [junit] at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > ? ? [junit] at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > ? ? [junit] at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > ? ? [junit] at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > ? ? [junit] at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > ? ? [junit] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > ? ? [junit] at org.junit.rules.RunRules.evaluate(RunRules.java:20) > ? ? [junit] at > org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > ? ? [junit] at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > ? ? [junit] at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > ? ? [junit] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > ? ? [junit] at > org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > ? ? [junit] at > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > ? ? [junit] at > org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > ? ? [junit] at > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > ? ? [junit] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > ? ? [junit] at org.junit.runners.Suite.runChild(Suite.java:128) > ? ? [junit] at org.junit.runners.Suite.runChild(Suite.java:27) > ? ? [junit] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > ? ? [junit] at > org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > ? ? [junit] at > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > ? ? [junit] at > org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > ? ? [junit] at > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > ? ? [junit] at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > ? ? [junit] at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > ? ? [junit] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > ? ? [junit] at > junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38) > ? ? [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535) > ? ? [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182) > ? ? [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033) > ? ? [junit] 2019-07-31 10:33:44,931 [myid:] - INFO > ?[main:ZKTestCase$1 at 70] - SUCCEEDED testConnectWithoutSSLToStrictServer[0] > > > Cheers, > Mate > > > >> On Jul 30, 2019, at 18:53 PM, Alan Bateman oracle.com at oracle.com >> wrote: > >> Hello, > >> in Apache ZooKeeper community we discovered a difference in behavior > >> in SocketInputStream. > >> > >> The problem is that SocketInputStream#read used to throw a > >> SocketException instead of returning -1. > >> > >> I see that this was cited in https://openjdk.java.net/jeps/353 > >> > >> "The |InputStream| and |OutputStream| returned by the old > >> implementation tests the stream for EOF and returns -1 before other > >> checks. The new implementation does |null| and bounds checks before > >> checking if the stream is at EOF. It is possible, but unlikely, that > >> there is fragile code that will trip up due to the order of the > checks." > >> > >> We are talking about a unit test, no a big deal. > >> > >> This is the fix, provided by Mate Szalay-Beko > >> https://github.com/apache/zookeeper/pull/1029 > >> > >Thanks for the mail. What is the exception you see with the legacy > >implementation? The note you cite would lead to an > >IndexOutOfBoundException due to the bad input and I don't think this is > >what you are seeing. > > > >-Alan. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eolivelli at gmail.com Wed Jul 31 13:01:52 2019 From: eolivelli at gmail.com (Enrico Olivelli) Date: Wed, 31 Jul 2019 15:01:52 +0200 Subject: Found different in behavior of SocketInputStream#read in JDK13 In-Reply-To: References: Message-ID: Il giorno mer 31 lug 2019 alle ore 14:39 Alan Bateman < Alan.Bateman at oracle.com> ha scritto: > Thanks for additional info. Although it may not be a "a big deal" for the > Zookeeper test, it would be great if you and Enrico have time to help us > diagnose this a big further. > Sure, We are reporting this error because JDK13 release will happen soon and such anomalies are quite scary. > At a high-level, the SocketException: "Connection reset" means the peer > has terminated the connection (hard reset). When you are using new > SocketImpl the peer appears to be closely the connection gracefully, help > -1 is read. Would I be correct to say that UnifiedServerSocketTest is > connecting to the Zookeeper server and it terminates the connection due to > the SSL connection. Is the Zookeeper running on JDK 13 too? I'm just > wondering if the scenario intermittently leads to a hard reset on some > occasions and a graceful close at some times and switching implementation > has changed the timing a bit to cause the latter to be more likely. > Server and client are running inside the same JVM The server side implementation is here, is mostly a wrapper round ServerSocket, it tries to detect if the client is willing to use SSL or not. https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/UnifiedServerSocket.java > > Is there any way to turn this test in a reproducer that we could study > more closely? > The specific test has nothing to do with ZooKeeper itself, it is a test over UnifiedServerSocket , so it can be extracted from ZooKeeper code base, but it is using a bunch of shared utilities for configuration and SSL. I am really sorry I don't have much time these days, I hope that Szalay-Bek? M?t? can help. Otherwise I will try to set up a repo next week Enrico > > -Alan > > > > On 31/07/2019 01:41, Szalay-Bek? M?t? wrote: > > Hi Alan, > > As Alan mentioned, it is not a big deal in Zookeeper's perspective. We > don't rely on this specific socket behaviour anywhere in our codebase, we > only faced it when trying to emulate an erroneous client behaviour and try > to assert on the way how the connection is not established successfully... > > Some background info: > > The test in Zookeeper ( > https://github.com/apache/zookeeper/blob/e043c322f12d56da0fc88131628edf0731c0f8e4/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/UnifiedServerSocketTest.java#L398) > simulates the case, when a client without using SSL tries to connect to the > server which requires SSL. Before starting the test, we don't set the > "jdk.net.usePlainSocketImpl" (so we use the new NioSocket while emulating > the client behaviour), and also we set > "jdk.tls.rejectClientInitiatedRenegotiation=true" to disable > client-initiated TLS renegotiation (although I am not sure if that is > relevant in this case or not). > > Using OpenJDK 13.30 on Xenial ubuntu, we saw that the socket.read() > returned with -1 without throwing an exception, indicating the end of the > stream. > > The old behaviour (e.g. using openJDK 12.0.2 on xenial) was to get a > SocketException ("Connection reset") with this stack trace: > > [junit] 2019-07-31 08:30:54,457 [myid:] - INFO > [main:UnifiedServerSocketTest at 419] - We get the expected exception: > [junit] java.net.SocketException: Connection reset > [junit] at > org.apache.zookeeper.server.quorum.UnifiedServerSocketTest$UnifiedServerThread.run(UnifiedServerSocketTest.java:204) > [junit] at > java.base/java.net.SocketInputStream.read(SocketInputStream.java:186) > [junit] Caused by: java.net.SocketException: Socket closed > [junit] at > java.base/java.net.SocketInputStream.read(SocketInputStream.java:140) > [junit] at java.base/java.net.PlainSocketImpl.socketAccept(Native > Method) > [junit] at > java.base/java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:458) > [junit] at > java.base/java.net.ServerSocket.implAccept(ServerSocket.java:556) > [junit] at > org.apache.zookeeper.server.quorum.UnifiedServerSocketTest.testConnectWithoutSSLToStrictServer(UnifiedServerSocketTest.java:417) > [junit] at > org.apache.zookeeper.server.quorum.UnifiedServerSocket.accept(UnifiedServerSocket.java:146) > [junit] at > org.apache.zookeeper.server.quorum.UnifiedServerSocketTest$UnifiedServerThread.run(UnifiedServerSocketTest.java:167) > [junit] at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > [junit] at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > [junit] at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > [junit] at java.base/java.lang.reflect.Method.invoke(Method.java:567) > [junit] at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > [junit] at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > [junit] at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > [junit] at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > [junit] at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > [junit] at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > [junit] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > [junit] at org.junit.rules.RunRules.evaluate(RunRules.java:20) > [junit] at > org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > [junit] at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > [junit] at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > [junit] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > [junit] at > org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > [junit] at > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > [junit] at > org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > [junit] at > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > [junit] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > [junit] at org.junit.runners.Suite.runChild(Suite.java:128) > [junit] at org.junit.runners.Suite.runChild(Suite.java:27) > [junit] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > [junit] at > org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > [junit] at > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > [junit] at > org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > [junit] at > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > [junit] at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > [junit] at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > [junit] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > [junit] at > junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033) > [junit] 2019-07-31 08:30:54,465 [myid:] - INFO [main:ZKTestCase$1 at 70] > - SUCCEEDED testConnectWithoutSSLToStrictServer[0] > > > > Interestingly, if I execute the very same test with OpenJDK 13.30 on my > macbook, I also get a similar exception SocketException using the NioSocket > implementation. Could this suggests some differences in the native code? > > [junit] 2019-07-31 10:33:44,926 [myid:] - INFO > [main:UnifiedServerSocketTest at 419] - We get the expected exception: > [junit] java.net.SocketException: Connection reset > [junit] at > java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:324) > [junit] at > java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:351) > [junit] at > java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:802) > [junit] at > java.base/java.net.Socket$SocketInputStream.read(Socket.java:919) > [junit] at > org.apache.zookeeper.server.quorum.UnifiedServerSocketTest.testConnectWithoutSSLToStrictServer(UnifiedServerSocketTest.java:417) > [junit] at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > [junit] at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > [junit] at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > [junit] at java.base/java.lang.reflect.Method.invoke(Method.java:567) > [junit] at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > [junit] at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > [junit] at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > [junit] at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > [junit] at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > [junit] at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > [junit] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > [junit] at org.junit.rules.RunRules.evaluate(RunRules.java:20) > [junit] at > org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > [junit] at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > [junit] at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > [junit] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > [junit] at > org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > [junit] at > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > [junit] at > org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > [junit] at > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > [junit] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > [junit] at org.junit.runners.Suite.runChild(Suite.java:128) > [junit] at org.junit.runners.Suite.runChild(Suite.java:27) > [junit] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > [junit] at > org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > [junit] at > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > [junit] at > org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > [junit] at > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > [junit] at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > [junit] at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > [junit] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > [junit] at > junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033) > [junit] 2019-07-31 10:33:44,931 [myid:] - INFO [main:ZKTestCase$1 at 70] > - SUCCEEDED testConnectWithoutSSLToStrictServer[0] > > > Cheers, > Mate > > > >> On Jul 30, 2019, at 18:53 PM, Alan Bateman > wrote: > >> Hello, > >> in Apache ZooKeeper community we discovered a difference in behavior > >> in SocketInputStream. > >> > >> The problem is that SocketInputStream#read used to throw a > >> SocketException instead of returning -1. > >> > >> I see that this was cited in https://openjdk.java.net/jeps/353 > >> > >> "The |InputStream| and |OutputStream| returned by the old > >> implementation tests the stream for EOF and returns -1 before other > >> checks. The new implementation does |null| and bounds checks before > >> checking if the stream is at EOF. It is possible, but unlikely, that > >> there is fragile code that will trip up due to the order of the checks." > >> > >> We are talking about a unit test, no a big deal. > >> > >> This is the fix, provided by Mate Szalay-Beko > >> https://github.com/apache/zookeeper/pull/1029 > >> > >Thanks for the mail. What is the exception you see with the legacy > >implementation? The note you cite would lead to an > >IndexOutOfBoundException due to the bad input and I don't think this is > >what you are seeing. > > > >-Alan. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simone.bordet at gmail.com Wed Jul 31 13:04:10 2019 From: simone.bordet at gmail.com (Simone Bordet) Date: Wed, 31 Jul 2019 15:04:10 +0200 Subject: SeekableByteChannel not seekable? In-Reply-To: <50f569dd-bc48-a640-12df-469648b7347a@oracle.com> References: <50f569dd-bc48-a640-12df-469648b7347a@oracle.com> Message-ID: Hi, On Wed, Jul 31, 2019 at 2:26 PM Alan Bateman wrote: > Which JDK release or build is this? Sorry, I should have said, it was JDK 8. > This part of the zipfs code was > re-implemented in JDK 12 so that it does support SeekableByteChannel and > the position methods should work correctly. Confirmed that works fine in JDK 12. Thanks! -- Simone Bordet --- Finally, no matter how good the architecture and design are, to deliver bug-free software with optimal performance and reliability, the implementation technique must be flawless. Victoria Livschitz From adinn at redhat.com Wed Jul 31 13:08:06 2019 From: adinn at redhat.com (Andrew Dinn) Date: Wed, 31 Jul 2019 14:08:06 +0100 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: References: <80da32b2-7acb-7b94-b82c-5dcd5cf95539@redhat.com> <36326F5E-12CD-487A-8FE2-1049631FE908@oracle.com> <8B1A8EDC-F9D7-4085-A34F-69100DBD7D5C@oracle.com> <5c78b5d7-dc9e-fbbd-41a9-5139ed6ee32c@redhat.com> <86d000f3-9eca-6089-5f7e-5698444d99ce@oracle.com> <7fe8d17d-6170-fcc9-b87a-eccfda2bb546@redhat.com> <7b39f68d-5698-4079-425c-c86ec161e361@oracle.com> <72810832-eb88-3418-f336-b95af95d9dcc@redhat.com> <20c8bbcb-2cd3-77af-cd40-29f7bd752166@oracle.com> <68566847-de80-82af-a928-d30c75f2b1b2@redhat.com> <892f3b0b-55f1-872d-15f4-5af907fa8437@redhat.com> <8f72d3b5-3a01-a7de-3b09-35571266a87d@redhat.com> Message-ID: <4fd3cdac-f72a-30ca-7a53-1d465644d973@redhat.com> On 31/07/2019 13:01, Boris Ulasevich wrote: > I did a quick check of the change across our platforms. Arm32 and x86_64 > built successfully. But I see it fails due to minor issues on aarch64 > and x86_32 with webrev.09. > Can you please have a look at this? > >> src/hotspot/cpu/aarch64/aarch64.ad:2202:1: error: expected ?;? before > ?}? token >> src/hotspot/cpu/x86/macroAssembler_x86.cpp:9925: undefined reference > to `Assembler::clflush(Address)' Sure, will do. I'm surprised by the x86_32 result as I /thought/ I had pushed the very same change set to the submit repo and it came back with no errors: > Job: mach5-one-adinn-JDK-8224974-8-20190730-1325-4068436 > BuildId: 2019-07-30-1324012.adinn.source > No failed tests . . . Is x86_32 tested as part of a submit run? regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From shade at redhat.com Wed Jul 31 13:10:45 2019 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 31 Jul 2019 15:10:45 +0200 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: <4fd3cdac-f72a-30ca-7a53-1d465644d973@redhat.com> References: <80da32b2-7acb-7b94-b82c-5dcd5cf95539@redhat.com> <36326F5E-12CD-487A-8FE2-1049631FE908@oracle.com> <8B1A8EDC-F9D7-4085-A34F-69100DBD7D5C@oracle.com> <5c78b5d7-dc9e-fbbd-41a9-5139ed6ee32c@redhat.com> <86d000f3-9eca-6089-5f7e-5698444d99ce@oracle.com> <7fe8d17d-6170-fcc9-b87a-eccfda2bb546@redhat.com> <7b39f68d-5698-4079-425c-c86ec161e361@oracle.com> <72810832-eb88-3418-f336-b95af95d9dcc@redhat.com> <20c8bbcb-2cd3-77af-cd40-29f7bd752166@oracle.com> <68566847-de80-82af-a928-d30c75f2b1b2@redhat.com> <892f3b0b-55f1-872d-15f4-5af907fa8437@redhat.com> <8f72d3b5-3a01-a7de-3b09-35571266a87d@redhat.com> <4fd3cdac-f72a-30ca-7a53-1d465644d973@redhat.com> Message-ID: On 7/31/19 3:08 PM, Andrew Dinn wrote: > On 31/07/2019 13:01, Boris Ulasevich wrote: > I'm surprised by the x86_32 result as I /thought/ I had pushed the very > same change set to the submit repo and it came back with no errors: > >> Job: mach5-one-adinn-JDK-8224974-8-20190730-1325-4068436 >> BuildId: 2019-07-30-1324012.adinn.source >> No failed tests > . . . > > Is x86_32 tested as part of a submit run? It does not. jdk-submit tests quite a limited number of configurations, mostly x86_64. -Aleksey -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From szalay.beko.mate at gmail.com Wed Jul 31 13:58:04 2019 From: szalay.beko.mate at gmail.com (=?UTF-8?B?U3phbGF5LUJla8WRIE3DoXTDqQ==?=) Date: Wed, 31 Jul 2019 15:58:04 +0200 Subject: Found different in behavior of SocketInputStream#read in JDK13 In-Reply-To: References: Message-ID: Hi Guys, > >> >> Is there any way to turn this test in a reproducer that we could study >> more closely? >> > > The specific test has nothing to do with ZooKeeper itself, it is a test > over UnifiedServerSocket > , > so it can be extracted from ZooKeeper code base, but it is using a bunch of > shared utilities for configuration and SSL. > > I am really sorry I don't have much time these days, I hope that > Szalay-Bek? M?t? can help. > Otherwise I will try to set up a repo next week > >> >> Sure, I'm happy to do it, will send it soon. Mate -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Wed Jul 31 14:18:37 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 31 Jul 2019 07:18:37 -0700 Subject: Found different in behavior of SocketInputStream#read in JDK13 In-Reply-To: References: Message-ID: <9da8d44a-39ac-1dae-cc5d-adf5cb40e77c@oracle.com> On 31/07/2019 06:01, Enrico Olivelli wrote: > : > > The server side implementation is here, is mostly a wrapper round > ServerSocket, it tries to detect if the client is willing to use SSL > or not. > > https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/UnifiedServerSocket.java > > I suspect this issue is about closing a socket while there is bytes in the socket buffer waiting to be sent. I would be curious if this test can be run on Windows as that the implementation has always gracefully closed the connection when linger is not enabled. The new implementation does the same but does so consistently on all platforms. If M?t? gets cycles to create a reproducer then we should be able to check that quickly. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From adinn at redhat.com Wed Jul 31 14:46:42 2019 From: adinn at redhat.com (Andrew Dinn) Date: Wed, 31 Jul 2019 15:46:42 +0100 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: References: <80da32b2-7acb-7b94-b82c-5dcd5cf95539@redhat.com> <36326F5E-12CD-487A-8FE2-1049631FE908@oracle.com> <8B1A8EDC-F9D7-4085-A34F-69100DBD7D5C@oracle.com> <5c78b5d7-dc9e-fbbd-41a9-5139ed6ee32c@redhat.com> <86d000f3-9eca-6089-5f7e-5698444d99ce@oracle.com> <7fe8d17d-6170-fcc9-b87a-eccfda2bb546@redhat.com> <7b39f68d-5698-4079-425c-c86ec161e361@oracle.com> <72810832-eb88-3418-f336-b95af95d9dcc@redhat.com> <20c8bbcb-2cd3-77af-cd40-29f7bd752166@oracle.com> <68566847-de80-82af-a928-d30c75f2b1b2@redhat.com> <892f3b0b-55f1-872d-15f4-5af907fa8437@redhat.com> <8f72d3b5-3a01-a7de-3b09-35571266a87d@redhat.com> <671c7ba1-9597-a990-190b-145d9cb5349e@redhat.com> <81098801-31b2-e3a8-5c25-043ab47c80bc@redhat.com> <27916ded-74e2-94d8-891c-95d87730b69d@redhat.com> Message-ID: <3cfda7cd-3517-d156-304f-5b8d7876b1cd@redhat.com> On 31/07/2019 12:40, Aleksey Shipilev wrote: > On 7/31/19 1:29 PM, Andrew Dinn wrote: >> I have an update regarding the change to the computation of the >> CPU_FLUSH flag. After posting the new webrev I built a debug version >> with the change that tests the clflush bit on x86_64. It crashed when >> the assert is reached in VM_Version::supports_cpuflush: >> >> # A fatal error has been detected by the Java Runtime Environment: >> # >> # Internal Error (src/hotspot/cpu/x86/vm_version_x86.hpp:978), >> pid=29597, tid=29602 >> # assert((_features & ((uint64_t)(0x20000000000ULL))) != 0) failed: >> clflush should be available >> >> So, it seems the clflush bit is not set on my x86_64 box even though I >> am able to execute clflush quite happily. >> >> "Toto, it looks like we are no longer in Antarctica." >> >> So, I will revert this change (in the next webrev). > > But wait, that might mean the clflush is indeed not available, by either hardware or software > switch? I believe some security mitigations disable clflush. Linux kernel, for example, has > "noclflush" option to do this. Ignoring the cpu bit and emitting clflush anyway might circumvent > that mitigation then? Ok, so investigating further I have found out what is wrong here. Significanlty, I get this result when I look at /proc/cpuinfo. [root at localhost ~]# cat /proc/cpuinfo | grep flush | head -1 flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 *clflush* dts acpi mmx fxsr sse sse2 ... So, the hardware does flag support for clflush. In which case why does the assert in supports_clflush fail? Well, the failure happened during the build process so I didn't (couldn't) debug it. I disabled the assert in supports_cpuflush() in order to allow the build to complete and then ran up the resulting JVM inside gdb to check what was going on. The problem is that with my patch supports_cpuflush() is called from Assembler::clflush() and the latter is called from icache_x86.cpp very early during bootstrap (I think this is neded to flush the flush routine used to flush the code cache). Anyway, the call happens so early that it precedes the call to VM_Version::get_processor_features which sets up the _features mask tested by the assert. So, I think the only option is to remove the assert from the x86_64 version of supports_cpuflush from Assemmbler::cpu_flush and instead add an assert in get_processor_features that CPU_FLUSH is set on x86_64. However, by that stage on x86_64 we will already have called clflush and, in the case where the assert might actually fail, we may not reach there because we have tripped up with an illegal instruction error. regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From shade at redhat.com Wed Jul 31 15:46:18 2019 From: shade at redhat.com (Aleksey Shipilev) Date: Wed, 31 Jul 2019 17:46:18 +0200 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: <3cfda7cd-3517-d156-304f-5b8d7876b1cd@redhat.com> References: <80da32b2-7acb-7b94-b82c-5dcd5cf95539@redhat.com> <36326F5E-12CD-487A-8FE2-1049631FE908@oracle.com> <8B1A8EDC-F9D7-4085-A34F-69100DBD7D5C@oracle.com> <5c78b5d7-dc9e-fbbd-41a9-5139ed6ee32c@redhat.com> <86d000f3-9eca-6089-5f7e-5698444d99ce@oracle.com> <7fe8d17d-6170-fcc9-b87a-eccfda2bb546@redhat.com> <7b39f68d-5698-4079-425c-c86ec161e361@oracle.com> <72810832-eb88-3418-f336-b95af95d9dcc@redhat.com> <20c8bbcb-2cd3-77af-cd40-29f7bd752166@oracle.com> <68566847-de80-82af-a928-d30c75f2b1b2@redhat.com> <892f3b0b-55f1-872d-15f4-5af907fa8437@redhat.com> <8f72d3b5-3a01-a7de-3b09-35571266a87d@redhat.com> <671c7ba1-9597-a990-190b-145d9cb5349e@redhat.com> <81098801-31b2-e3a8-5c25-043ab47c80bc@redhat.com> <27916ded-74e2-94d8-891c-95d87730b69d@redhat.com> <3cfda7cd-3517-d156-304f-5b8d7876b1cd@redhat.com> Message-ID: On 7/31/19 4:46 PM, Andrew Dinn wrote: > Well, the failure happened during the build process so I didn't > (couldn't) debug it. I disabled the assert in supports_cpuflush() in > order to allow the build to complete and then ran up the resulting JVM > inside gdb to check what was going on. The problem is that with my patch > supports_cpuflush() is called from Assembler::clflush() and the latter > is called from icache_x86.cpp very early during bootstrap (I think this > is neded to flush the flush routine used to flush the code cache). > Anyway, the call happens so early that it precedes the call to > VM_Version::get_processor_features which sets up the _features mask > tested by the assert. I believe you can untie this bootstrapping circularity by relaxing the assert with Universe::is_fully_unitialized(). It still gives us window to fail with SIGILL if the stub is called early during bootstrap, but that would be something to fix *if* we ever find ourselves there. -Aleksey -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From adinn at redhat.com Wed Jul 31 16:22:51 2019 From: adinn at redhat.com (Andrew Dinn) Date: Wed, 31 Jul 2019 17:22:51 +0100 Subject: RFR: 8224974: Implement JEP 352 In-Reply-To: References: <80da32b2-7acb-7b94-b82c-5dcd5cf95539@redhat.com> <8B1A8EDC-F9D7-4085-A34F-69100DBD7D5C@oracle.com> <5c78b5d7-dc9e-fbbd-41a9-5139ed6ee32c@redhat.com> <86d000f3-9eca-6089-5f7e-5698444d99ce@oracle.com> <7fe8d17d-6170-fcc9-b87a-eccfda2bb546@redhat.com> <7b39f68d-5698-4079-425c-c86ec161e361@oracle.com> <72810832-eb88-3418-f336-b95af95d9dcc@redhat.com> <20c8bbcb-2cd3-77af-cd40-29f7bd752166@oracle.com> <68566847-de80-82af-a928-d30c75f2b1b2@redhat.com> <892f3b0b-55f1-872d-15f4-5af907fa8437@redhat.com> <8f72d3b5-3a01-a7de-3b09-35571266a87d@redhat.com> <671c7ba1-9597-a990-190b-145d9cb5349e@redhat.com> <81098801-31b2-e3a8-5c25-043ab47c80bc@redhat.com> <27916ded-74e2-94d8-891c-95d87730b69d@redhat.com> <3cfda7cd-3517-d156-304f-5b8d7876b1cd@redhat.com> Message-ID: On 31/07/2019 16:46, Aleksey Shipilev wrote: > On 7/31/19 4:46 PM, Andrew Dinn wrote: >> Well, the failure happened during the build process so I didn't >> (couldn't) debug it. I disabled the assert in supports_cpuflush() in >> order to allow the build to complete and then ran up the resulting JVM >> inside gdb to check what was going on. The problem is that with my patch >> supports_cpuflush() is called from Assembler::clflush() and the latter >> is called from icache_x86.cpp very early during bootstrap (I think this >> is neded to flush the flush routine used to flush the code cache). >> Anyway, the call happens so early that it precedes the call to >> VM_Version::get_processor_features which sets up the _features mask >> tested by the assert. > > I believe you can untie this bootstrapping circularity by relaxing the assert with > Universe::is_fully_unitialized(). It still gives us window to fail with SIGILL if the stub is called > early during bootstrap, but that would be something to fix *if* we ever find ourselves there. Yes indeed, that seems like by far the cleanest solution. The only other alternative is to move the call to VM_Version::initalize earlier in the Universe bootstrap sequence and I doubt anyone wants to risk that. So, now I have #ifdef _LP64 static bool supports_clflush() { // clflush should always be available on x86_64 // if not we are in real trouble because we rely on it // to flush the code cache. // Unfortunately, Assembler::clflush is currently called as part // of generation of the code cache flush routine. This happens // under Universe::init before the processor features are set // up. Assembler::flush calls this routine to check that clflush // is allowed. So, we give the caller a free pass if Universe init // is still in progress. assert ((!Universe::is_fully_initialized() || (_features & CPU_FLUSH) != 0), "clflush should be available"); return true; } . . . which seems to work ok. I'll post a new webrev when I have handled Brian's comments and also any other feedback you may still have on my last reply. regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From chris.hegarty at oracle.com Wed Jul 31 22:58:01 2019 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 31 Jul 2019 15:58:01 -0700 Subject: Found different in behavior of SocketInputStream#read in JDK13 In-Reply-To: <9da8d44a-39ac-1dae-cc5d-adf5cb40e77c@oracle.com> References: <9da8d44a-39ac-1dae-cc5d-adf5cb40e77c@oracle.com> Message-ID: <8C102953-1E4F-46D5-B703-4580516CA9F2@oracle.com> There may be a minor JDK issue here ... Below is a minimal test, which I believe replicates the scenario that the Zoo Keeper test finds itself in. The test exhibits the reported behaviour; `read` returns -1 with the new nio socket impl, while throws with the old plain socket impl. The following, as of yet untested, change restores the behaviour difference. --- a/src/java.base/share/classes/sun/nio/ch/NioSocketImpl.java +++ b/src/java.base/share/classes/sun/nio/ch/NioSocketImpl.java @@ -894,7 +894,7 @@ // shutdown output when linger interval not set to 0 try { var SO_LINGER = StandardSocketOptions.SO_LINGER; - if ((int) Net.getSocketOption(fd, SO_LINGER) != 0) { + if ((int) Net.getSocketOption(fd, SO_LINGER) > 0) { Net.shutdown(fd, Net.SHUT_WR); } } catch (IOException ignore) { } The change in behaviour comes from the `close` on the server-side shutting down the output before closing the socket. When debugging this issue the linger value being returned ( on my mac ) is -1, meaning that the option is disabled. I'll need to do some testing on the above patch, but on the face of it I believe that `shutdown` should only be called if the linger value is positive. --- public class Zoo { public static void main(String... args) throws Exception { try (ServerSocket ss = new ServerSocket(0)) { Thread t = new Thread() { @Override public void run() { try { Socket s = ss.accept(); byte[] ba = new byte[5]; int r = s.getInputStream().read(ba); assert r == 5; s.close(); } catch (IOException e) { throw new UncheckedIOException(e); } } }; t.start(); int port = ss.getLocalPort(); Socket s = new Socket("localhost", port); byte[] ba = new byte[] { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x10 }; s.getOutputStream().write(ba); int r = s.getInputStream().read(); System.out.println("read: " + r); } } } -Chris. From Alan.Bateman at oracle.com Wed Jul 31 23:13:26 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 31 Jul 2019 16:13:26 -0700 Subject: Found different in behavior of SocketInputStream#read in JDK13 In-Reply-To: <8C102953-1E4F-46D5-B703-4580516CA9F2@oracle.com> References: <9da8d44a-39ac-1dae-cc5d-adf5cb40e77c@oracle.com> <8C102953-1E4F-46D5-B703-4580516CA9F2@oracle.com> Message-ID: On 31/07/2019 15:58, Chris Hegarty wrote: > : > > > The change in behaviour comes from the `close` on the server-side > shutting down the output before closing the socket. When debugging this > issue the linger value being returned ( on my mac ) is -1, meaning that > the option is disabled. > > I'll need to do some testing on the above patch, but on the face of it > I believe that `shutdown` should only be called if the linger value is > positive. > shutdown should only be called here if linger is disabled. I suspect the change in behavior is that it is only done consistently on Windows in the old SocketImpl, it is done on all platforms in the new impl. -Alan