From brian.burkhalter at oracle.com Tue Oct 3 14:10:17 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 3 Oct 2017 07:10:17 -0700 Subject: JDK 10 RFR of 8147615: (fc) FileChannelImpl has no finalizer In-Reply-To: <0C2FD652-FD96-48D0-B6C1-38C27620D402@oracle.com> References: <0C2FD652-FD96-48D0-B6C1-38C27620D402@oracle.com> Message-ID: <256DD5F7-AF21-406B-ADB0-DE06A8FC275C@oracle.com> This message reprises the thread [1]. A new revision of the patch for [2] which accounts for the recent refactoring of FileDescriptor closing [3] is here: http://cr.openjdk.java.net/~bpb/8147615/webrev.02/ Thanks, Brian [1] http://mail.openjdk.java.net/pipermail/nio-dev/2017-September/004463.html [2] https://bugs.openjdk.java.net/browse/JDK-8147615 [3] http://mail.openjdk.java.net/pipermail/core-libs-dev/2017-September/049231.html From Roger.Riggs at Oracle.com Fri Oct 6 13:34:55 2017 From: Roger.Riggs at Oracle.com (Roger Riggs) Date: Fri, 6 Oct 2017 09:34:55 -0400 Subject: JDK 10 RFR of 8147615: (fc) FileChannelImpl has no finalizer In-Reply-To: <256DD5F7-AF21-406B-ADB0-DE06A8FC275C@oracle.com> References: <0C2FD652-FD96-48D0-B6C1-38C27620D402@oracle.com> <256DD5F7-AF21-406B-ADB0-DE06A8FC275C@oracle.com> Message-ID: <93eeb196-0cd1-8f8a-e92c-f34ec4a772c9@Oracle.com> Hi Brian, Sorry for the delay. FileChannelImpl: - line 117:? A lambda could be used instead of the explicit Closer class. () -> fdAccess.close(fd) // Note: fd is the argument (not the field) to prevent the lambda from capturing this. In the test: ?- Perhaps a bit more descriptive name than just "Cleaner" perhaps CleanerTest or... ?- Add @modules java.management - so the test will not be run unless the runtime includes management ? Thanks, Roger On 10/3/2017 10:10 AM, Brian Burkhalter wrote: > This message reprises the thread [1]. A new revision of the patch for [2] which accounts for the recent > refactoring of FileDescriptor closing [3] is here: > > http://cr.openjdk.java.net/~bpb/8147615/webrev.02/ > > Thanks, > > Brian > > [1] http://mail.openjdk.java.net/pipermail/nio-dev/2017-September/004463.html > [2] https://bugs.openjdk.java.net/browse/JDK-8147615 > [3] http://mail.openjdk.java.net/pipermail/core-libs-dev/2017-September/049231.html From yingqi.lu at intel.com Fri Oct 6 17:39:37 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Fri, 6 Oct 2017 17:39:37 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AD1214A0A@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD1222567@ORSMSX113.amr.corp.intel.com> Hi Brian, Here is the version 17 of the patch http://cr.openjdk.java.net/~kkharbas/8164900/webrev.17/ It is generated against http://hg.openjdk.java.net/jdk10/master/ Please let me know if there is anything missing. Thank you very much for your help! Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Tuesday, September 26, 2017 4:09 PM To: Lu, Yingqi Cc: nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, On Sep 26, 2017, at 4:04 PM, Lu, Yingqi > wrote: Thank you very much for the help! Please note that the webrev I uploaded contains a complete patch, i.e., it is not an overlay. Please let me know if you see any errors on your test harness. Will do. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Fri Oct 6 23:06:21 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 6 Oct 2017 16:06:21 -0700 Subject: JDK 10 RFR of 8147615: (fc) FileChannelImpl has no finalizer In-Reply-To: <93eeb196-0cd1-8f8a-e92c-f34ec4a772c9@Oracle.com> References: <0C2FD652-FD96-48D0-B6C1-38C27620D402@oracle.com> <256DD5F7-AF21-406B-ADB0-DE06A8FC275C@oracle.com> <93eeb196-0cd1-8f8a-e92c-f34ec4a772c9@Oracle.com> Message-ID: <600C1A4B-E952-444E-B692-0BE4CBA76B9B@oracle.com> Hi Roger, On Oct 6, 2017, at 6:34 AM, Roger Riggs wrote: > Sorry for the delay. Likewise. > FileChannelImpl: > > - line 117: A lambda could be used instead of the explicit Closer class. > > () -> fdAccess.close(fd) > // Note: fd is the argument (not the field) to prevent the lambda > from capturing this. > > In the test: > - Perhaps a bit more descriptive name than just "Cleaner" perhaps CleanerTest or... > > - Add @modules java.management - so the test will not be run unless the runtime includes management I made the foregoing changes but now the test times out. I imagine there is a problem with how I set up the lambda. Please see the diff below. Thanks, Brian --- a/src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java +++ b/src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java @@ -90,18 +90,6 @@ // Cleanable with an action which closes this channel's file descriptor private final Cleanable cleanable; - private static class Closer implements Runnable { - private final FileDescriptor fd; - - Closer(FileDescriptor fd) { - this.fd = fd; - } - - public void run() { - fdAccess.close(fd); - } - } - private FileChannelImpl(FileDescriptor fd, String path, boolean readable, boolean writable, Object parent) { @@ -113,8 +101,7 @@ this.nd = new FileDispatcherImpl(); // Register a cleaning action if and only if there is no parent // as the parent will take care of closing the file descriptor. - this.cleanable = parent != null ? null : - CleanerFactory.cleaner().register(this, new Closer(fd)); + this.cleanable = parent != null ? null : () -> fdAccess.close(fd); } // Used by FileInputStream.getChannel(), FileOutputStream.getChannel rename from test/jdk/java/nio/channels/FileChannel/Cleaner.java rename to test/jdk/java/nio/channels/FileChannel/CleanerTest.java --- a/test/jdk/java/nio/channels/FileChannel/Cleaner.java +++ b/test/jdk/java/nio/channels/FileChannel/CleanerTest.java @@ -25,7 +25,8 @@ * @bug 8147615 * @summary Test whether an unreferenced FileChannel is actually cleaned * @requires (os.family == "linux") | (os.family == "mac") | (os.family == "solaris") | (os.family == "aix") - * @run main/othervm Cleaner + * @modules java.management + * @run main/othervm CleanerTest */ import com.sun.management.UnixOperatingSystemMXBean; @@ -40,7 +41,7 @@ import java.nio.file.Paths; import java.nio.file.StandardOpenOption; -public class Cleaner { +public class CleanerTest { public static void main(String[] args) throws Throwable { OperatingSystemMXBean mxBean = ManagementFactory.getOperatingSystemMXBean(); -------------- next part -------------- An HTML attachment was scrubbed... URL: From Roger.Riggs at Oracle.com Sat Oct 7 15:57:41 2017 From: Roger.Riggs at Oracle.com (Roger Riggs) Date: Sat, 7 Oct 2017 11:57:41 -0400 Subject: JDK 10 RFR of 8147615: (fc) FileChannelImpl has no finalizer In-Reply-To: <600C1A4B-E952-444E-B692-0BE4CBA76B9B@oracle.com> References: <0C2FD652-FD96-48D0-B6C1-38C27620D402@oracle.com> <256DD5F7-AF21-406B-ADB0-DE06A8FC275C@oracle.com> <93eeb196-0cd1-8f8a-e92c-f34ec4a772c9@Oracle.com> <600C1A4B-E952-444E-B692-0BE4CBA76B9B@oracle.com> Message-ID: <99cddb9e-bb04-9fb8-09f9-b0b6f683fcbf@Oracle.com> Hi Brian, On 10/6/2017 7:06 PM, Brian Burkhalter wrote: > >> FileChannelImpl: >> >> - line 117:? A lambda could be used instead of the explicit Closer class. >> >> ??() -> fdAccess.close(fd) >> ??// Note: fd is the argument (not the field) to prevent the lambda >> ??from capturing this. >> >> In the test: >> ?- Perhaps a bit more descriptive name than just "Cleaner" perhaps >> CleanerTest or... >> >> ?- Add @modules java.management - so the test will not be run unless >> the runtime includes management > > I made the foregoing changes but now the test times out. I imagine > there is a problem with how I set up the lambda. Please see the diff > below. I wonder if lambda is preferring this.fd instead of the argument fd.? (I imagine the JLS says that somewhere) Try creating a new (final) local variable with the value of argument fd and using that in the lambda. Otherwise, its fine. Thanks, Roger > > Thanks, > > Brian > > --- a/src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java > +++ b/src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java > @@ -90,18 +90,6 @@ > ?? ? // Cleanable with an action which closes this channel's file > descriptor > ?? ? private final Cleanable cleanable; > > > -? ? private static class Closer implements Runnable { > -? ? ? ? private final FileDescriptor fd; > - > -? ? ? ? Closer(FileDescriptor fd) { > -? ? ? ? ? ? this.fd = fd; > -? ? ? ? } > - > -? ? ? ? public void run() { > -? ? ? ? ? ? fdAccess.close(fd); > -? ? ? ? } > -? ? } > - > ?? ? private FileChannelImpl(FileDescriptor fd, String path, boolean > readable, > ?? ? ? ? ? ? ? ? ? ? ? ? ? ? boolean writable, Object parent) > ?? ? { > @@ -113,8 +101,7 @@ > ?? ? ? ? this.nd = new FileDispatcherImpl(); > ?? ? ? ? // Register a cleaning action if and only if there is no parent > ?? ? ? ? // as the parent will take care of closing the file descriptor. > -? ? ? ? this.cleanable = parent != null ? null : > - CleanerFactory.cleaner().register(this, new Closer(fd)); > +? ? ? ? this.cleanable = parent != null ? null : () -> > fdAccess.close(fd); > ?? ? } > > > ?? ? // Used by FileInputStream.getChannel(), FileOutputStream.getChannel > > rename from test/jdk/java/nio/channels/FileChannel/Cleaner.java > rename to test/jdk/java/nio/channels/FileChannel/CleanerTest.java > --- a/test/jdk/java/nio/channels/FileChannel/Cleaner.java > +++ b/test/jdk/java/nio/channels/FileChannel/CleanerTest.java > @@ -25,7 +25,8 @@ > ? * @bug 8147615 > ? * @summary Test whether an unreferenced FileChannel is actually cleaned > ? * @requires (os.family == "linux") | (os.family == "mac") | > (os.family == "solaris") | (os.family == "aix") > - * @run main/othervm Cleaner > + * @modules java.management > + * @run main/othervm CleanerTest > ? */ > > > ?import com.sun.management.UnixOperatingSystemMXBean; > @@ -40,7 +41,7 @@ > ?import java.nio.file.Paths; > ?import java.nio.file.StandardOpenOption; > > > -public class Cleaner { > +public class CleanerTest { > ?? ? public static void main(String[] args) throws Throwable { > ?? ? ? ? OperatingSystemMXBean mxBean = > ManagementFactory.getOperatingSystemMXBean(); > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Sat Oct 7 16:22:50 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Sat, 7 Oct 2017 09:22:50 -0700 Subject: JDK 10 RFR of 8147615: (fc) FileChannelImpl has no finalizer In-Reply-To: <99cddb9e-bb04-9fb8-09f9-b0b6f683fcbf@Oracle.com> References: <0C2FD652-FD96-48D0-B6C1-38C27620D402@oracle.com> <256DD5F7-AF21-406B-ADB0-DE06A8FC275C@oracle.com> <93eeb196-0cd1-8f8a-e92c-f34ec4a772c9@Oracle.com> <600C1A4B-E952-444E-B692-0BE4CBA76B9B@oracle.com> <99cddb9e-bb04-9fb8-09f9-b0b6f683fcbf@Oracle.com> Message-ID: <058C8F15-28D4-46C2-8641-92CB06DA3DA6@oracle.com> Hi Roger, On Oct 7, 2017, at 8:57 AM, Roger Riggs wrote: > I wonder if lambda is preferring this.fd instead of the argument fd. (I imagine the JLS says that somewhere) > Try creating a new (final) local variable with the value of argument fd and using that in the lambda. This also timed out: final FileDescriptor fdl = fd; this.cleanable = parent != null ? null : () -> fdAccess.close(fdl); as did this: final FileDescriptor fdl = fd; final JavaIOFileDescriptorAccess fda = fdAccess; this.cleanable = parent != null ? null : () -> fda.close(fdl); I?ll look into the JLS next week. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Mon Oct 9 07:37:49 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 9 Oct 2017 08:37:49 +0100 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD1222567@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD1214A0A@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1222567@ORSMSX113.amr.corp.intel.com> Message-ID: <385666c4-23a7-7527-9c25-124b26f10420@oracle.com> On 06/10/2017 18:39, Lu, Yingqi wrote: > > Hi Brian, > > Here is the version 17 of the patch > http://cr.openjdk.java.net/~kkharbas/8164900/webrev.17/ > > > It is generated against http://hg.openjdk.java.net/jdk10/master/ > > Please let me know if there is anything missing. Thank you very much > for your help! > > I went through webrev.17 and it looks good. One thing check is statvfs usage in FileDispatcherImpl.c, I assume this should statvfs64 to allow for 32-bit builds and >2GB files. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Mon Oct 9 13:48:37 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 9 Oct 2017 14:48:37 +0100 Subject: JDK 10 RFR of 8147615: (fc) FileChannelImpl has no finalizer In-Reply-To: <600C1A4B-E952-444E-B692-0BE4CBA76B9B@oracle.com> References: <0C2FD652-FD96-48D0-B6C1-38C27620D402@oracle.com> <256DD5F7-AF21-406B-ADB0-DE06A8FC275C@oracle.com> <93eeb196-0cd1-8f8a-e92c-f34ec4a772c9@Oracle.com> <600C1A4B-E952-444E-B692-0BE4CBA76B9B@oracle.com> Message-ID: <660c8584-9256-6b87-a256-f2d9db05db5b@oracle.com> On 07/10/2017 00:06, Brian Burkhalter wrote: > @@ -113,8 +101,7 @@ > ?? ? ? ? this.nd = new FileDispatcherImpl(); > ?? ? ? ? // Register a cleaning action if and only if there is no parent > ?? ? ? ? // as the parent will take care of closing the file descriptor. > -? ? ? ? this.cleanable = parent != null ? null : > - CleanerFactory.cleaner().register(this, new Closer(fd)); > +? ? ? ? this.cleanable = parent != null ? null : () -> > fdAccess.close(fd); > ?? ? } > This just creates a Cleanable that invokes close, you are looking for: ??? cleaner.register(this, () -> fdAccess.close(fd)); Once you remove this inner class then you could rename cleanable to "closer" to make it more readable. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Mon Oct 9 14:48:07 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Mon, 9 Oct 2017 14:48:07 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <385666c4-23a7-7527-9c25-124b26f10420@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD1214A0A@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1222567@ORSMSX113.amr.corp.intel.com>, <385666c4-23a7-7527-9c25-124b26f10420@oracle.com> Message-ID: Hi Alan, Thank you for reviewing it. I will submit version 18 today to address the issue. Thanks, Lucy Sent from my iPhone On Oct 9, 2017, at 12:38 AM, Alan Bateman > wrote: On 06/10/2017 18:39, Lu, Yingqi wrote: Hi Brian, Here is the version 17 of the patch http://cr.openjdk.java.net/~kkharbas/8164900/webrev.17/ It is generated against http://hg.openjdk.java.net/jdk10/master/ Please let me know if there is anything missing. Thank you very much for your help! I went through webrev.17 and it looks good. One thing check is statvfs usage in FileDispatcherImpl.c, I assume this should statvfs64 to allow for 32-bit builds and >2GB files. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Mon Oct 9 15:53:48 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 9 Oct 2017 08:53:48 -0700 Subject: JDK 10 RFR of 8147615: (fc) FileChannelImpl has no finalizer In-Reply-To: <660c8584-9256-6b87-a256-f2d9db05db5b@oracle.com> References: <0C2FD652-FD96-48D0-B6C1-38C27620D402@oracle.com> <256DD5F7-AF21-406B-ADB0-DE06A8FC275C@oracle.com> <93eeb196-0cd1-8f8a-e92c-f34ec4a772c9@Oracle.com> <600C1A4B-E952-444E-B692-0BE4CBA76B9B@oracle.com> <660c8584-9256-6b87-a256-f2d9db05db5b@oracle.com> Message-ID: <6ECA8E91-CE2E-473C-A56E-59F799087ECA@oracle.com> That?s what I get for attempting to fool with it on a Saturday morning. On Oct 9, 2017, at 6:48 AM, Alan Bateman wrote: > This just creates a Cleanable that invokes close, you are looking for: > cleaner.register(this, () -> fdAccess.close(fd)); > > Once you remove this inner class then you could rename cleanable to "closer" to make it more readable. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Mon Oct 9 19:03:38 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 9 Oct 2017 12:03:38 -0700 Subject: JDK 10 RFR of 8147615: (fc) FileChannelImpl has no finalizer In-Reply-To: <6ECA8E91-CE2E-473C-A56E-59F799087ECA@oracle.com> References: <0C2FD652-FD96-48D0-B6C1-38C27620D402@oracle.com> <256DD5F7-AF21-406B-ADB0-DE06A8FC275C@oracle.com> <93eeb196-0cd1-8f8a-e92c-f34ec4a772c9@Oracle.com> <600C1A4B-E952-444E-B692-0BE4CBA76B9B@oracle.com> <660c8584-9256-6b87-a256-f2d9db05db5b@oracle.com> <6ECA8E91-CE2E-473C-A56E-59F799087ECA@oracle.com> Message-ID: OK, here?s a version which is not brain-dead: http://cr.openjdk.java.net/~bpb/8147615/webrev.03/ I should not that the test passes without using a final local FileDescriptor variable: --- a/src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java +++ b/src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java @@ -101,9 +101,8 @@ this.nd = new FileDispatcherImpl(); // Register a cleaning action if and only if there is no parent // as the parent will take care of closing the file descriptor. - final FileDescriptor fdl = fd; this.cleanable = parent != null ? null : - CleanerFactory.cleaner().register(this, () -> fdAccess.close(fdl)); + CleanerFactory.cleaner().register(this, () -> fdAccess.close(fd)); Thanks, Brian On Oct 9, 2017, at 8:53 AM, Brian Burkhalter wrote: > That?s what I get for attempting to fool with it on a Saturday morning. > > On Oct 9, 2017, at 6:48 AM, Alan Bateman wrote: > >> This just creates a Cleanable that invokes close, you are looking for: >> cleaner.register(this, () -> fdAccess.close(fd)); >> >> Once you remove this inner class then you could rename cleanable to "closer" to make it more readable. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Mon Oct 9 20:07:41 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Mon, 9 Oct 2017 20:07:41 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AD1214A0A@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1222567@ORSMSX113.amr.corp.intel.com>, <385666c4-23a7-7527-9c25-124b26f10420@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD122731C@ORSMSX113.amr.corp.intel.com> Hi Alan, Here is the version 18 of the patch. Following your suggestion, I changed statvfs to statvfs64 in FileDispatcherImpl.c. http://cr.openjdk.java.net/~kkharbas/8164900/webrev.18/ Thanks, Lucy From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of Lu, Yingqi Sent: Monday, October 09, 2017 7:48 AM To: Alan Bateman Cc: nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Alan, Thank you for reviewing it. I will submit version 18 today to address the issue. Thanks, Lucy Sent from my iPhone On Oct 9, 2017, at 12:38 AM, Alan Bateman > wrote: On 06/10/2017 18:39, Lu, Yingqi wrote: Hi Brian, Here is the version 17 of the patch http://cr.openjdk.java.net/~kkharbas/8164900/webrev.17/ It is generated against http://hg.openjdk.java.net/jdk10/master/ Please let me know if there is anything missing. Thank you very much for your help! I went through webrev.17 and it looks good. One thing check is statvfs usage in FileDispatcherImpl.c, I assume this should statvfs64 to allow for 32-bit builds and >2GB files. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Mon Oct 9 20:09:00 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 9 Oct 2017 13:09:00 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD122731C@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD1214A0A@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1222567@ORSMSX113.amr.corp.intel.com>, <385666c4-23a7-7527-9c25-124b26f10420@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD122731C@ORSMSX113.amr.corp.intel.com> Message-ID: <60EBA1B3-515A-4B8E-A443-EBC8DE6C9FDA@oracle.com> Hi Lucy, Thanks. I?ll run it through the regression harness for the sake of sanity. Brian On Oct 9, 2017, at 1:07 PM, Lu, Yingqi wrote: > Here is the version 18 of the patch. Following your suggestion, I changed statvfs to statvfs64 in FileDispatcherImpl.c. > > http://cr.openjdk.java.net/~kkharbas/8164900/webrev.18/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Mon Oct 9 21:04:11 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 9 Oct 2017 14:04:11 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD122731C@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD1214A0A@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1222567@ORSMSX113.amr.corp.intel.com>, <385666c4-23a7-7527-9c25-124b26f10420@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD122731C@ORSMSX113.amr.corp.intel.com> Message-ID: <4095A508-E4DF-4D08-A63E-309F0E092432@oracle.com> Hi Lucy, This patch has a compilation failure on macOS: src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c:334:22: error: variable has incomplete type 'struct statvfs64' struct statvfs64 file_stat; ^ src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c:334:12: note: forward declaration of 'struct statvfs64' struct statvfs64 file_stat; ^ src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c:362:14: error: implicit declaration of function 'fstatvfs64' is invalid in C99 [-Werror,-Wimplicit-function-declaration] result = fstatvfs64(fd, &file_stat); The diff below appears to fix it. Thanks, Brian --- a/src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c +++ b/src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c @@ -331,7 +331,11 @@ { jint fd = fdval(env, fdo); jint result; +#ifdef MACOSX + struct statvfs file_stat; +#else struct statvfs64 file_stat; +#endif #if defined(O_DIRECT) || defined(F_NOCACHE) || defined(DIRECTIO_ON) #ifdef O_DIRECT @@ -359,7 +363,11 @@ return result; } #endif +#ifdef MACOSX + result = fstatvfs(fd, &file_stat); +#else result = fstatvfs64(fd, &file_stat); +#endif if(result == -1) { JNU_ThrowIOExceptionWithLastError(env, "DirectIO setup failed"); return result; On Oct 9, 2017, at 1:07 PM, Lu, Yingqi wrote: > Here is the version 18 of the patch. Following your suggestion, I changed statvfs to statvfs64 in FileDispatcherImpl.c. > > http://cr.openjdk.java.net/~kkharbas/8164900/webrev.18/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Mon Oct 9 21:13:58 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Mon, 9 Oct 2017 21:13:58 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <4095A508-E4DF-4D08-A63E-309F0E092432@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD1214A0A@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1222567@ORSMSX113.amr.corp.intel.com>, <385666c4-23a7-7527-9c25-124b26f10420@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD122731C@ORSMSX113.amr.corp.intel.com> <4095A508-E4DF-4D08-A63E-309F0E092432@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD1227453@ORSMSX113.amr.corp.intel.com> Hi Brian, Thank you for testing the patch! The fix looks good to me. Do you want me to submit a newer version of the patch? Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Monday, October 09, 2017 2:04 PM To: Lu, Yingqi Cc: Alan Bateman ; nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, This patch has a compilation failure on macOS: src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c:334:22: error: variable has incomplete type 'struct statvfs64' struct statvfs64 file_stat; ^ src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c:334:12: note: forward declaration of 'struct statvfs64' struct statvfs64 file_stat; ^ src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c:362:14: error: implicit declaration of function 'fstatvfs64' is invalid in C99 [-Werror,-Wimplicit-function-declaration] result = fstatvfs64(fd, &file_stat); The diff below appears to fix it. Thanks, Brian --- a/src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c +++ b/src/java.base/unix/native/libnio/ch/FileDispatcherImpl.c @@ -331,7 +331,11 @@ { jint fd = fdval(env, fdo); jint result; +#ifdef MACOSX + struct statvfs file_stat; +#else struct statvfs64 file_stat; +#endif #if defined(O_DIRECT) || defined(F_NOCACHE) || defined(DIRECTIO_ON) #ifdef O_DIRECT @@ -359,7 +363,11 @@ return result; } #endif +#ifdef MACOSX + result = fstatvfs(fd, &file_stat); +#else result = fstatvfs64(fd, &file_stat); +#endif if(result == -1) { JNU_ThrowIOExceptionWithLastError(env, "DirectIO setup failed"); return result; On Oct 9, 2017, at 1:07 PM, Lu, Yingqi > wrote: Here is the version 18 of the patch. Following your suggestion, I changed statvfs to statvfs64 in FileDispatcherImpl.c. http://cr.openjdk.java.net/~kkharbas/8164900/webrev.18/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Mon Oct 9 21:15:14 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 9 Oct 2017 14:15:14 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD1227453@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD1214A0A@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1222567@ORSMSX113.amr.corp.intel.com>, <385666c4-23a7-7527-9c25-124b26f10420@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD122731C@ORSMSX113.amr.corp.intel.com> <4095A508-E4DF-4D08-A63E-309F0E092432@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1227453@ORSMSX113.amr.corp.intel.com> Message-ID: Hi Lucy, If we are all satisfied with this then I think I can just update the existing patch; no need to submit another revision. Thanks, Brian On Oct 9, 2017, at 2:13 PM, Lu, Yingqi wrote: > Thank you for testing the patch! The fix looks good to me. Do you want me to submit a newer version of the patch? -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Mon Oct 9 21:16:26 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Mon, 9 Oct 2017 21:16:26 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AD1214A0A@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1222567@ORSMSX113.amr.corp.intel.com>, <385666c4-23a7-7527-9c25-124b26f10420@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD122731C@ORSMSX113.amr.corp.intel.com> <4095A508-E4DF-4D08-A63E-309F0E092432@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1227453@ORSMSX113.amr.corp.intel.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD12274AF@ORSMSX113.amr.corp.intel.com> Hi Brian, Great! Thank you for the help!! Thanks, Lucy From: Brian Burkhalter [mailto:brian.burkhalter at oracle.com] Sent: Monday, October 09, 2017 2:15 PM To: Lu, Yingqi Cc: Alan Bateman ; nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hi Lucy, If we are all satisfied with this then I think I can just update the existing patch; no need to submit another revision. Thanks, Brian On Oct 9, 2017, at 2:13 PM, Lu, Yingqi > wrote: Thank you for testing the patch! The fix looks good to me. Do you want me to submit a newer version of the patch? -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Oct 10 00:57:34 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 9 Oct 2017 17:57:34 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD12274AF@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD1214A0A@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1222567@ORSMSX113.amr.corp.intel.com>, <385666c4-23a7-7527-9c25-124b26f10420@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD122731C@ORSMSX113.amr.corp.intel.com> <4095A508-E4DF-4D08-A63E-309F0E092432@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1227453@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12274AF@ORSMSX113.amr.corp.intel.com> Message-ID: Hi, I updated version 18 of the patch with the fix for the compilation errors on macOS: http://cr.openjdk.java.net/~bpb/8164900/webrev.18/ This ran successfully through our regression harness for the IO and NIO core tests and all platforms passed. I also re-reviewed the code changes once again. Barring objections to the contrary, I am ready to sponsor this version. Thanks, Brian From brian.burkhalter at oracle.com Tue Oct 10 00:58:15 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 9 Oct 2017 17:58:15 -0700 Subject: [PING] JDK 10 RFR of 8184157: (ch) AsynchronousFileChannel hangs with internal error when reading locked file In-Reply-To: <1D96702F-F407-42F2-A031-E7967DA03400@oracle.com> References: <8B4639CE-99A3-46AA-900A-0A54B44BBFA0@oracle.com> <1D96702F-F407-42F2-A031-E7967DA03400@oracle.com> Message-ID: <21A8ECF6-ED06-4FBD-A976-154A78D77BA5@oracle.com> A reminder that this RFR originally posted on July 24 [1] is still pending. The original discussion thread continued at [2] with the most recent message included below. Thanks, Brian [1] http://mail.openjdk.java.net/pipermail/nio-dev/2017-July/004369.html [2] http://mail.openjdk.java.net/pipermail/nio-dev/2017-August/004384.html On Aug 11, 2017, at 1:06 PM, Brian Burkhalter wrote: > HI Roger, > > On Aug 11, 2017, at 12:57 PM, Brian Burkhalter wrote: > >>> Seeing that overlapped (some) memory is freed only at close, can there be a problem with >>> a very active channel that is open for a long time to accumulate/leak native memory? >>> PendingIoCache:135-136: Or is that just belt and suspenders? >> >> No I think that is a very good point. After invalidate() the value of ?overlapped? is needed but I don?t think the memory it points to. I think the freeMemory() call should be moved into invalidate(). Good catch! I will update and re-post. > > Actually I replied too soon. I think the remove() call at line 105 of PendingIoCache takes care of this. If the ?overlapped? key was not in the map, then an attempt is made to remove it from the HashSet and if present the same action is taken as if it had been in the map. So the memory release in close() is just for draining anything left in the map and set, if there is even anything there. > > Thanks, > > Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Roger.Riggs at Oracle.com Tue Oct 10 14:09:35 2017 From: Roger.Riggs at Oracle.com (Roger Riggs) Date: Tue, 10 Oct 2017 10:09:35 -0400 Subject: JDK 10 RFR of 8147615: (fc) FileChannelImpl has no finalizer In-Reply-To: References: <0C2FD652-FD96-48D0-B6C1-38C27620D402@oracle.com> <256DD5F7-AF21-406B-ADB0-DE06A8FC275C@oracle.com> <93eeb196-0cd1-8f8a-e92c-f34ec4a772c9@Oracle.com> <600C1A4B-E952-444E-B692-0BE4CBA76B9B@oracle.com> <660c8584-9256-6b87-a256-f2d9db05db5b@oracle.com> <6ECA8E91-CE2E-473C-A56E-59F799087ECA@oracle.com> Message-ID: Hi Brian, Right, looks good. Roger On 10/9/2017 3:03 PM, Brian Burkhalter wrote: > OK, here?s a version which is not brain-dead: > > http://cr.openjdk.java.net/~bpb/8147615/webrev.03/ > > > I should not that the test passes without using a final local > FileDescriptor variable: > > --- a/src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java > +++ b/src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java > @@ -101,9 +101,8 @@ > ?? ? ? ? this.nd = new FileDispatcherImpl(); > ?? ? ? ? // Register a cleaning action if and only if there is no parent > ?? ? ? ? // as the parent will take care of closing the file descriptor. > -? ? ? ? final FileDescriptor fdl = fd; > ?? ? ? ? this.cleanable = parent != null ? null : > - CleanerFactory.cleaner().register(this, () -> fdAccess.close(fdl)); > + CleanerFactory.cleaner().register(this, () -> fdAccess.close(fd)); > > Thanks, > > Brian From brian.burkhalter at oracle.com Tue Oct 10 15:26:56 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 10 Oct 2017 08:26:56 -0700 Subject: JDK 10 RFR of 8147615: (fc) FileChannelImpl has no finalizer In-Reply-To: References: <0C2FD652-FD96-48D0-B6C1-38C27620D402@oracle.com> <256DD5F7-AF21-406B-ADB0-DE06A8FC275C@oracle.com> <93eeb196-0cd1-8f8a-e92c-f34ec4a772c9@Oracle.com> <600C1A4B-E952-444E-B692-0BE4CBA76B9B@oracle.com> <660c8584-9256-6b87-a256-f2d9db05db5b@oracle.com> <6ECA8E91-CE2E-473C-A56E-59F799087ECA@oracle.com> Message-ID: <5C0BDD95-647F-4A2A-B8CB-98DB8BC8A2B3@oracle.com> Hi Roger, I renamed the variable ?cleanable? to ?closer? (line 91) in the version to be pushed. Thanks, Brian On Oct 10, 2017, at 7:09 AM, Roger Riggs wrote: > Right, looks good. > > Roger > > > On 10/9/2017 3:03 PM, Brian Burkhalter wrote: >> OK, here?s a version which is not brain-dead: >> >> http://cr.openjdk.java.net/~bpb/8147615/webrev.03/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Oct 10 19:33:49 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 10 Oct 2017 12:33:49 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AD1214A0A@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1222567@ORSMSX113.amr.corp.intel.com>, <385666c4-23a7-7527-9c25-124b26f10420@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD122731C@ORSMSX113.amr.corp.intel.com> <4095A508-E4DF-4D08-A63E-309F0E092432@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1227453@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12274AF@ORSMSX113.amr.corp.intel.com> Message-ID: <491CEB43-5961-43C3-86E2-60CF0E927780@oracle.com> Hello, I have two informational items to report. Firstly, please note that pursuant to the fix for [1] it was necessary to update this patch to version 19 [2]. The conflict was with FileChannelImpl.java [3]. This change has been verified on macOS and a complete test run is in progress. Secondly, before the Direct I/O patch can be integrated, a Compatibility and Specification Request (CSR) (https://wiki.openjdk.java.net/display/csr/Main) will need to be created and approved. I will create a CSR issue and start moving it towards review. Thanks, Brian [1] https://bugs.openjdk.java.net/browse/JDK-8147615 [2] http://cr.openjdk.java.net/~bpb/8164900/webrev.19/ [3] FileChannelImpl.java diff --- a/src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java +++ b/src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java @@ -92,18 +92,32 @@ // Positional-read is not interruptible private volatile boolean uninterruptible; + // DirectIO flag + private final boolean direct; + + // IO alignment value for DirectIO + private final int alignment; + // Cleanable with an action which closes this channel's file descriptor private final Cleanable closer; private FileChannelImpl(FileDescriptor fd, String path, boolean readable, - boolean writable, Object parent) + boolean writable, boolean direct, Object parent) { this.fd = fd; this.readable = readable; this.writable = writable; this.parent = parent; this.path = path; + this.direct = direct; this.nd = new FileDispatcherImpl(); + if (direct) { + assert path != null; + this.alignment = nd.setDirectIO(fd, path); + } else { + this.alignment = -1; + } + // Register a cleaning action if and only if there is no parent // as the parent will take care of closing the file descriptor. this.closer= parent != null ? null : On Oct 9, 2017, at 5:57 PM, Brian Burkhalter wrote: > I updated version 18 of the patch with the fix for the compilation errors on macOS: > > http://cr.openjdk.java.net/~bpb/8164900/webrev.18/ > > This ran successfully through our regression harness for the IO and NIO core tests and all platforms passed. I also re-reviewed the code changes once again. > > Barring objections to the contrary, I am ready to sponsor this version. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yingqi.lu at intel.com Tue Oct 10 19:53:31 2017 From: yingqi.lu at intel.com (Lu, Yingqi) Date: Tue, 10 Oct 2017 19:53:31 +0000 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <491CEB43-5961-43C3-86E2-60CF0E927780@oracle.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD1214A0A@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1222567@ORSMSX113.amr.corp.intel.com>, <385666c4-23a7-7527-9c25-124b26f10420@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD122731C@ORSMSX113.amr.corp.intel.com> <4095A508-E4DF-4D08-A63E-309F0E092432@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1227453@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12274AF@ORSMSX113.amr.corp.intel.com> <491CEB43-5961-43C3-86E2-60CF0E927780@oracle.com> Message-ID: <9ACD5B67AAC5594CB6268234CF29CF9AD1228836@ORSMSX113.amr.corp.intel.com> Hi Brian, Thank you very much for your help, really appreciate! Please let me know if there is anything we can do to help :) Thanks, Lucy From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of Brian Burkhalter Sent: Tuesday, October 10, 2017 12:34 PM To: nio-dev at openjdk.java.net Subject: Re: Proposal for adding O_DIRECT support into JDK 9 Hello, I have two informational items to report. Firstly, please note that pursuant to the fix for [1] it was necessary to update this patch to version 19 [2]. The conflict was with FileChannelImpl.java [3]. This change has been verified on macOS and a complete test run is in progress. Secondly, before the Direct I/O patch can be integrated, a Compatibility and Specification Request (CSR) (https://wiki.openjdk.java.net/display/csr/Main) will need to be created and approved. I will create a CSR issue and start moving it towards review. Thanks, Brian [1] https://bugs.openjdk.java.net/browse/JDK-8147615 [2] http://cr.openjdk.java.net/~bpb/8164900/webrev.19/ [3] FileChannelImpl.java diff --- a/src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java +++ b/src/java.base/share/classes/sun/nio/ch/FileChannelImpl.java @@ -92,18 +92,32 @@ // Positional-read is not interruptible private volatile boolean uninterruptible; + // DirectIO flag + private final boolean direct; + + // IO alignment value for DirectIO + private final int alignment; + // Cleanable with an action which closes this channel's file descriptor private final Cleanable closer; private FileChannelImpl(FileDescriptor fd, String path, boolean readable, - boolean writable, Object parent) + boolean writable, boolean direct, Object parent) { this.fd = fd; this.readable = readable; this.writable = writable; this.parent = parent; this.path = path; + this.direct = direct; this.nd = new FileDispatcherImpl(); + if (direct) { + assert path != null; + this.alignment = nd.setDirectIO(fd, path); + } else { + this.alignment = -1; + } + // Register a cleaning action if and only if there is no parent // as the parent will take care of closing the file descriptor. this.closer= parent != null ? null : On Oct 9, 2017, at 5:57 PM, Brian Burkhalter > wrote: I updated version 18 of the patch with the fix for the compilation errors on macOS: http://cr.openjdk.java.net/~bpb/8164900/webrev.18/ This ran successfully through our regression harness for the IO and NIO core tests and all platforms passed. I also re-reviewed the code changes once again. Barring objections to the contrary, I am ready to sponsor this version. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Oct 10 20:23:43 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 10 Oct 2017 13:23:43 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: <9ACD5B67AAC5594CB6268234CF29CF9AD1228836@ORSMSX113.amr.corp.intel.com> References: <9ACD5B67AAC5594CB6268234CF29CF9AD1214A0A@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1222567@ORSMSX113.amr.corp.intel.com>, <385666c4-23a7-7527-9c25-124b26f10420@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD122731C@ORSMSX113.amr.corp.intel.com> <4095A508-E4DF-4D08-A63E-309F0E092432@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1227453@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12274AF@ORSMSX113.amr.corp.intel.com> <491CEB43-5961-43C3-86E2-60CF0E927780@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1228836@ORSMSX113.amr.corp.intel.com> Message-ID: Hi Lucy, On Oct 10, 2017, at 12:53 PM, Lu, Yingqi wrote: > Thank you very much for your help, really appreciate! You?re welcome. > Please let me know if there is anything we can do to help J Will do. The updated version 19 job passed the test harness. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From marc.strapetz at syntevo.com Thu Oct 12 12:38:19 2017 From: marc.strapetz at syntevo.com (Marc Strapetz) Date: Thu, 12 Oct 2017 14:38:19 +0200 Subject: RFE: provide an efficient way to retrieve Unix ctime (changed-time) Message-ID: <5dee2631-6d2b-928f-a852-8cbdc8567ed8@syntevo.com> During a Files.walkFileTree(), UnixFileAttributes seem to be populated with all available file system information, especially st_ctime_sec and st_ctime_nsec, but currently there is no way to access this information (except of using reflection-based hacks). As suggested in a comment by Alan Bateman on stackoverflow.com[1], this information could be made available directly through PosixFileAttributes. Alternatively I could think of some java.nio.Files utility method: Filetime getLastChangedTime(BasicFileAttributes) throws IOException; which will return the ctime in case of UnixFileAttributes and otherwise fail with an IOException or even a RuntimeException. Motivation: combinations of mtime and ctime (and file size), may be used on Unix to take a shortcut when it comes to determine file modifications. Currently it's not efficiently possible to replicate this logic in Java (without the above mentioned hacks). One prominent example is Git's Index logic[2]. [1] https://stackoverflow.com/questions/46676279 [2] https://github.com/git/git/blob/master/cache.h -Marc From Alan.Bateman at oracle.com Thu Oct 12 13:25:41 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 12 Oct 2017 14:25:41 +0100 Subject: RFE: provide an efficient way to retrieve Unix ctime (changed-time) In-Reply-To: <5dee2631-6d2b-928f-a852-8cbdc8567ed8@syntevo.com> References: <5dee2631-6d2b-928f-a852-8cbdc8567ed8@syntevo.com> Message-ID: On 12/10/2017 13:38, Marc Strapetz wrote: > During a Files.walkFileTree(), UnixFileAttributes seem to be populated > with all available file system information, especially st_ctime_sec > and st_ctime_nsec, but currently there is no way to access this > information (except of using reflection-based hacks). > > As suggested in a comment by Alan Bateman on stackoverflow.com[1], > this information could be made available directly through > PosixFileAttributes. > Right, there's no type safe way to get this through the standard API although it can be obtained by attribute name using Files.getAttribute(file, "unix:ctime"). As noted, it could potentially be exposed by PosixFileAttributes if there was a compelling reason, it just hasn't been important to date. -Alan From marc.strapetz at syntevo.com Thu Oct 12 14:45:51 2017 From: marc.strapetz at syntevo.com (Marc Strapetz) Date: Thu, 12 Oct 2017 16:45:51 +0200 Subject: RFE: provide an efficient way to retrieve Unix ctime (changed-time) In-Reply-To: References: <5dee2631-6d2b-928f-a852-8cbdc8567ed8@syntevo.com> Message-ID: On 12.10.2017 15:25, Alan Bateman wrote: > On 12/10/2017 13:38, Marc Strapetz wrote: >> During a Files.walkFileTree(), UnixFileAttributes seem to be populated >> with all available file system information, especially st_ctime_sec >> and st_ctime_nsec, but currently there is no way to access this >> information (except of using reflection-based hacks). >> >> As suggested in a comment by Alan Bateman on stackoverflow.com[1], >> this information could be made available directly through >> PosixFileAttributes. >> > Right, there's no type safe way to get this through the standard API > although it can be obtained by attribute name using > Files.getAttribute(file, "unix:ctime"). Thanks for pointing me to this method. I've tried this now and it's surprisingly fast: For my test tree with 35K files, a Files.walkFileTree() with SimpleFileVisitor takes on average 115ms. When SimpleFileVisitor.visitFile is additionally calling Files.getAttribute(file, "unix:ctime"), it takes 170ms. From the method signature (Path, String) I would have expected a significantly higher overhead. -Marc From brian.burkhalter at oracle.com Thu Oct 12 16:46:21 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 12 Oct 2017 09:46:21 -0700 Subject: JDK 10 RFR of 8189209: java/lang/invoke/lambda/LambdaAsm.java failed with "could not create proxy classes" Message-ID: Please review the patch [1] for [2]. This replaces a lambda expression with a static inner class. Thanks, Brian [1] http://cr.openjdk.java.net/~bpb/8189209/webrev.00/ [2] https://bugs.openjdk.java.net/browse/JDK-8189209 From Roger.Riggs at Oracle.com Thu Oct 12 16:56:44 2017 From: Roger.Riggs at Oracle.com (Roger Riggs) Date: Thu, 12 Oct 2017 12:56:44 -0400 Subject: JDK 10 RFR of 8189209: java/lang/invoke/lambda/LambdaAsm.java failed with "could not create proxy classes" In-Reply-To: References: Message-ID: Hi Brian, Looks fine. Sorry for the turbulence. I might suggest a comment saying the Closer is used to early in the boot process for lambda to be used. Roger On 10/12/2017 12:46 PM, Brian Burkhalter wrote: > Please review the patch [1] for [2]. This replaces a lambda expression with a static inner class. > > Thanks, > > Brian > > [1] http://cr.openjdk.java.net/~bpb/8189209/webrev.00/ > [2] https://bugs.openjdk.java.net/browse/JDK-8189209 > From Alan.Bateman at oracle.com Thu Oct 12 17:03:51 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 12 Oct 2017 18:03:51 +0100 Subject: JDK 10 RFR of 8189209: java/lang/invoke/lambda/LambdaAsm.java failed with "could not create proxy classes" In-Reply-To: References: Message-ID: On 12/10/2017 17:46, Brian Burkhalter wrote: > Please review the patch [1] for [2]. This replaces a lambda expression with a static inner class. > > Looks okay. Minor nit at L116 where it should be "this.closer = ...". -Alan From brian.burkhalter at oracle.com Thu Oct 12 17:16:46 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 12 Oct 2017 10:16:46 -0700 Subject: JDK 10 RFR of 8189209: java/lang/invoke/lambda/LambdaAsm.java failed with "could not create proxy classes" In-Reply-To: References: Message-ID: <47A4A095-DF20-4104-9BAB-3EEDDEA8B0C3@oracle.com> Webrev updated in place http://cr.openjdk.java.net/~bpb/8189209/webrev.00/ with both comments below accounted for. Regression tests passed. Will push within the hour unless I hear an objection. Thanks, Brian On Oct 12, 2017, at 9:56 AM, Roger Riggs wrote: > I might suggest a comment saying the Closer is used to early in the boot process for lambda to be used. On Oct 12, 2017, at 10:03 AM, Alan Bateman wrote: > Looks okay. Minor nit at L116 where it should be "this.closer = ...". -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Oct 12 17:20:28 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 12 Oct 2017 18:20:28 +0100 Subject: JDK 10 RFR of 8189209: java/lang/invoke/lambda/LambdaAsm.java failed with "could not create proxy classes" In-Reply-To: <47A4A095-DF20-4104-9BAB-3EEDDEA8B0C3@oracle.com> References: <47A4A095-DF20-4104-9BAB-3EEDDEA8B0C3@oracle.com> Message-ID: <2f08794d-6a41-93b1-0724-c16c7421d4ac@oracle.com> On 12/10/2017 18:16, Brian Burkhalter wrote: > Webrev updated in place > http://cr.openjdk.java.net/~bpb/8189209/webrev.00/ > with both > comments below accounted for. Regression tests passed. Will push > within the hour unless I hear an objection. > The comment is confusing, I would suggest dropping it or just replacing with a simple statement to say that FileChannel is used by the LambdaMetaFactory so can't use a lambda here. -Alan From brian.burkhalter at oracle.com Thu Oct 12 17:29:40 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 12 Oct 2017 10:29:40 -0700 Subject: JDK 10 RFR of 8189209: java/lang/invoke/lambda/LambdaAsm.java failed with "could not create proxy classes" In-Reply-To: <2f08794d-6a41-93b1-0724-c16c7421d4ac@oracle.com> References: <47A4A095-DF20-4104-9BAB-3EEDDEA8B0C3@oracle.com> <2f08794d-6a41-93b1-0724-c16c7421d4ac@oracle.com> Message-ID: <5F126C5A-1DEC-4DE6-893A-18CFFA104152@oracle.com> On Oct 12, 2017, at 10:20 AM, Alan Bateman wrote: > On 12/10/2017 18:16, Brian Burkhalter wrote: >> Webrev updated in placehttp://cr.openjdk.java.net/~bpb/8189209/webrev.00/ with both comments below accounted for. Regression tests passed. Will push within the hour unless I hear an objection. >> > The comment is confusing, I would suggest dropping it or just replacing with a simple statement to say that FileChannel is used by the LambdaMetaFactory so can't use a lambda here. Webrev updated in place. 116 // FileChannel is used by the LambdaMetaFactory so a lambda cannot 117 // be used here hence we use a nested class instead. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Oct 12 17:31:53 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 12 Oct 2017 18:31:53 +0100 Subject: JDK 10 RFR of 8189209: java/lang/invoke/lambda/LambdaAsm.java failed with "could not create proxy classes" In-Reply-To: <5F126C5A-1DEC-4DE6-893A-18CFFA104152@oracle.com> References: <47A4A095-DF20-4104-9BAB-3EEDDEA8B0C3@oracle.com> <2f08794d-6a41-93b1-0724-c16c7421d4ac@oracle.com> <5F126C5A-1DEC-4DE6-893A-18CFFA104152@oracle.com> Message-ID: <14dfc14c-9f48-67ff-a484-c38e8aa42dfe@oracle.com> On 12/10/2017 18:29, Brian Burkhalter wrote: > > On Oct 12, 2017, at 10:20 AM, Alan Bateman > wrote: > >> On 12/10/2017 18:16, Brian Burkhalter wrote: >>> Webrev updated in >>> placehttp://cr.openjdk.java.net/~bpb/8189209/webrev.00/ >>> >>> with both comments below accounted for. Regression tests passed. >>> Will push within the hour unless I hear an objection. >>> >> The comment is confusing, I would suggest dropping it or just >> replacing with a simple statement to say that FileChannel is used by >> the LambdaMetaFactory so can't use a lambda here. > > Webrev updated in place. > > 116 // FileChannel is used by the LambdaMetaFactory so a lambda cannot > 117 // be used here hence we use a nested class instead. > Okay. -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.stuefe at gmail.com Wed Oct 18 07:44:54 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 18 Oct 2017 09:44:54 +0200 Subject: RFR(xs): (aix but affects shared code too) 8186665: buffer overflow in Java_java_nio_MappedByteBuffer_isLoaded0 In-Reply-To: <0c1e56e1-e802-260d-3224-5eebd7f1e569@oracle.com> References: <0557a2d5-5a5d-9218-7551-121a95fb4f6c@oracle.com> <0c1e56e1-e802-260d-3224-5eebd7f1e569@oracle.com> Message-ID: Hi all, I am wrapping up fixes which did not make it into the repo before the consolidation. Alan already reviewed the last webrev, but I need a second reviewer. Issue: https://bugs.openjdk.java.net/browse/JDK-8186665 Last Webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8186665-buffer-ov erflow-in-mincore/webrev.01/webrev Issue text for your convenience: -- In Java_java_nio_MappedByteBuffer_isLoaded0, we call mincore(2) to retrieve the paging status of an address range. The size of the output buffer for mincore(2) depends on the number of pages in *system page size* in the given memory range (this is spelled out more or less explicitly on AIX and Linux, nothing is said on BSD/OSX, but I assume the same). The number of pages in the memory range is calculated by MappedByteBuffer.isLoaded() and handed down to Java_java_nio_MappedByteBuffer_isLoaded0() together with the memory range to test. MappedByteBuffer.isLoaded() calculates this number of pages based on jjdk.internal.misc.Unsafe.pagesize(), which ultimately comes down to os::vm_page_size(). For AIX, os::vm_page_size() may return a page size larger than the system page size of 4K. The reason for this is that on AIX, memory can be backed by different page sizes, usually either 4K or 64K - e.g. posix thread stacks may have 4K pages, java heap (system V shared memory) with 64K pages, but mmap memory is always 4K page backed... But as the OpenJDK code base generally assumes one homogeneous page size for everything - which is usually synonymous with os::vm_page_size() - a decision had to be made which page size to assume as a global system page size, and this may be a larger page size than the 4K system page size mincore(2) assumes. This usually is no problem, but with mincore(2) it is: as the size of the output buffer depends on the number of pages, calculating with a too-large page size causes the output buffer to be too small and hence the buffer overflows. The solution must be to base the size of the mincore output buffer on the system page size. -- Thanks and Kind Regards, Thomas On Thu, Aug 31, 2017 at 9:39 PM, Alan Bateman wrote: > On 31/08/2017 19:01, Thomas St?fe wrote: > >> Hi Alan, >> >> thank you for your review! >> >> Updated webrev: >> http://cr.openjdk.java.net/~stuefe/webrevs/8186665-buffer-ov >> erflow-in-mincore/webrev.01/webrev/ > Estuefe/webrevs/8186665-buffer-overflow-in-mincore/webrev.01/webrev/> >> >> I fixed the indention and embellished the comments around the sentinel at >> the end of the buffer somewhat. >> >> This looks okay to me. > > -Alan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.levart at gmail.com Wed Oct 18 08:14:40 2017 From: peter.levart at gmail.com (Peter Levart) Date: Wed, 18 Oct 2017 10:14:40 +0200 Subject: RFR(xs): (aix but affects shared code too) 8186665: buffer overflow in Java_java_nio_MappedByteBuffer_isLoaded0 In-Reply-To: References: <0557a2d5-5a5d-9218-7551-121a95fb4f6c@oracle.com> <0c1e56e1-e802-260d-3224-5eebd7f1e569@oracle.com> Message-ID: Hi Thomas, Shouldn't the following line: ? 47???? return len2 + pagesize - 1 / pagesize; ..be written as: ??? ??? ?? return (len2 + pagesize - 1) / pagesize; Regards, Peter On 10/18/2017 09:44 AM, Thomas St?fe wrote: > Hi all, > > I am wrapping up fixes which did not make it into the repo before the > consolidation. Alan already reviewed the last webrev, but I need a > second reviewer. > > > Issue: https://bugs.openjdk.java.net/browse/JDK-8186665 > > Last Webrev: > http://cr.openjdk.java.net/~stuefe/webrevs/8186665-buffer-overflow-in-mincore/webrev.01/webrev > > > Issue text for your convenience: > > -- > In Java_java_nio_MappedByteBuffer_isLoaded0, we call mincore(2) to > retrieve the paging status of an address range. > > The size of the output buffer for mincore(2) depends on the number of > pages in *system page size* in the given memory range (this is spelled > out more or less explicitly on AIX and Linux, nothing is said on > BSD/OSX, but I assume the same). The number of pages in the memory > range is calculated by MappedByteBuffer.isLoaded() and handed down to > Java_java_nio_MappedByteBuffer_isLoaded0() together with the memory > range to test. > > MappedByteBuffer.isLoaded() calculates this number of pages based on > jjdk.internal.misc.Unsafe.pagesize(), which ultimately comes down to > os::vm_page_size(). > > For AIX, os::vm_page_size() may return a page size larger than the > system page size of 4K. The reason for this is that on AIX, memory can > be backed by different page sizes, usually either 4K or 64K - e.g. > posix thread stacks may have 4K pages, java heap (system V shared > memory) with 64K pages, but mmap memory is always 4K page backed... > > But as the OpenJDK code base generally assumes one homogeneous page > size for everything - which is usually synonymous with > os::vm_page_size() - a decision had to be made which page size to > assume as a global system page size, and this may be a larger page > size than the 4K system page size mincore(2) assumes. > > This usually is no problem, but with mincore(2) it is: as the size of > the output buffer depends on the number of pages, calculating with a > too-large page size causes the output buffer to be too small and hence > the buffer overflows. The solution must be to base the size of the > mincore output buffer on the system page size. > > -- > > Thanks and Kind Regards, Thomas > > > On Thu, Aug 31, 2017 at 9:39 PM, Alan Bateman > wrote: > > On 31/08/2017 19:01, Thomas St?fe wrote: > > Hi Alan, > > thank you for your review! > > Updated webrev: > http://cr.openjdk.java.net/~stuefe/webrevs/8186665-buffer-overflow-in-mincore/webrev.01/webrev/ > > > > > I fixed the indention and embellished the comments around the > sentinel at the end of the buffer somewhat. > > This looks okay to me. > > -Alan > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.levart at gmail.com Wed Oct 18 08:17:29 2017 From: peter.levart at gmail.com (Peter Levart) Date: Wed, 18 Oct 2017 10:17:29 +0200 Subject: RFR(xs): (aix but affects shared code too) 8186665: buffer overflow in Java_java_nio_MappedByteBuffer_isLoaded0 In-Reply-To: References: <0557a2d5-5a5d-9218-7551-121a95fb4f6c@oracle.com> <0c1e56e1-e802-260d-3224-5eebd7f1e569@oracle.com> Message-ID: On 10/18/2017 10:14 AM, Peter Levart wrote: > Hi Thomas, > > Shouldn't the following line: > > ? 47???? return len2 + pagesize - 1 / pagesize; > > ..be written as: > > ??? ??? ?? return (len2 + pagesize - 1) / pagesize; ...or better yet, as: ??? ??? ?????? return numPages; (you already calculate it correctly in previous line, but then return an expression, which is wrong). Regards, Peter > > > Regards, Peter > > On 10/18/2017 09:44 AM, Thomas St?fe wrote: >> Hi all, >> >> I am wrapping up fixes which did not make it into the repo before the >> consolidation. Alan already reviewed the last webrev, but I need a >> second reviewer. >> >> >> Issue: https://bugs.openjdk.java.net/browse/JDK-8186665 >> >> Last Webrev: >> http://cr.openjdk.java.net/~stuefe/webrevs/8186665-buffer-overflow-in-mincore/webrev.01/webrev >> >> >> Issue text for your convenience: >> >> -- >> In Java_java_nio_MappedByteBuffer_isLoaded0, we call mincore(2) to >> retrieve the paging status of an address range. >> >> The size of the output buffer for mincore(2) depends on the number of >> pages in *system page size* in the given memory range (this is >> spelled out more or less explicitly on AIX and Linux, nothing is said >> on BSD/OSX, but I assume the same). The number of pages in the memory >> range is calculated by MappedByteBuffer.isLoaded() and handed down to >> Java_java_nio_MappedByteBuffer_isLoaded0() together with the memory >> range to test. >> >> MappedByteBuffer.isLoaded() calculates this number of pages based on >> jjdk.internal.misc.Unsafe.pagesize(), which ultimately comes down to >> os::vm_page_size(). >> >> For AIX, os::vm_page_size() may return a page size larger than the >> system page size of 4K. The reason for this is that on AIX, memory >> can be backed by different page sizes, usually either 4K or 64K - >> e.g. posix thread stacks may have 4K pages, java heap (system V >> shared memory) with 64K pages, but mmap memory is always 4K page >> backed... >> >> But as the OpenJDK code base generally assumes one homogeneous page >> size for everything - which is usually synonymous with >> os::vm_page_size() - a decision had to be made which page size to >> assume as a global system page size, and this may be a larger page >> size than the 4K system page size mincore(2) assumes. >> >> This usually is no problem, but with mincore(2) it is: as the size of >> the output buffer depends on the number of pages, calculating with a >> too-large page size causes the output buffer to be too small and >> hence the buffer overflows. The solution must be to base the size of >> the mincore output buffer on the system page size. >> >> -- >> >> Thanks and Kind Regards, Thomas >> >> >> On Thu, Aug 31, 2017 at 9:39 PM, Alan Bateman >> > wrote: >> >> On 31/08/2017 19:01, Thomas St?fe wrote: >> >> Hi Alan, >> >> thank you for your review! >> >> Updated webrev: >> http://cr.openjdk.java.net/~stuefe/webrevs/8186665-buffer-overflow-in-mincore/webrev.01/webrev/ >> >> > > >> >> I fixed the indention and embellished the comments around the >> sentinel at the end of the buffer somewhat. >> >> This looks okay to me. >> >> -Alan >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From christoph.langer at sap.com Wed Oct 18 08:23:42 2017 From: christoph.langer at sap.com (Langer, Christoph) Date: Wed, 18 Oct 2017 08:23:42 +0000 Subject: RFR(xs): (aix but affects shared code too) 8186665: buffer overflow in Java_java_nio_MappedByteBuffer_isLoaded0 In-Reply-To: References: <0557a2d5-5a5d-9218-7551-121a95fb4f6c@oracle.com> <0c1e56e1-e802-260d-3224-5eebd7f1e569@oracle.com> Message-ID: Hi Thomas, apart from the point that Peter found, I?d also think it would look better if the typedef section (line 51-56) would be placed before the AIX only function (line 41-49). Other than that, it looks good to me. Best regards Christoph From: nio-dev [mailto:nio-dev-bounces at openjdk.java.net] On Behalf Of Peter Levart Sent: Mittwoch, 18. Oktober 2017 10:17 To: Thomas St?fe ; Alan Bateman Cc: nio-dev at openjdk.java.net; ppc-aix-port-dev at openjdk.java.net; Java Core Libs Subject: Re: RFR(xs): (aix but affects shared code too) 8186665: buffer overflow in Java_java_nio_MappedByteBuffer_isLoaded0 On 10/18/2017 10:14 AM, Peter Levart wrote: Hi Thomas, Shouldn't the following line: 47 return len2 + pagesize - 1 / pagesize; ..be written as: return (len2 + pagesize - 1) / pagesize; ...or better yet, as: return numPages; (you already calculate it correctly in previous line, but then return an expression, which is wrong). Regards, Peter Regards, Peter On 10/18/2017 09:44 AM, Thomas St?fe wrote: Hi all, I am wrapping up fixes which did not make it into the repo before the consolidation. Alan already reviewed the last webrev, but I need a second reviewer. Issue: https://bugs.openjdk.java.net/browse/JDK-8186665 Last Webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8186665-buffer-overflow-in-mincore/webrev.01/webrev Issue text for your convenience: -- In Java_java_nio_MappedByteBuffer_isLoaded0, we call mincore(2) to retrieve the paging status of an address range. The size of the output buffer for mincore(2) depends on the number of pages in *system page size* in the given memory range (this is spelled out more or less explicitly on AIX and Linux, nothing is said on BSD/OSX, but I assume the same). The number of pages in the memory range is calculated by MappedByteBuffer.isLoaded() and handed down to Java_java_nio_MappedByteBuffer_isLoaded0() together with the memory range to test. MappedByteBuffer.isLoaded() calculates this number of pages based on jjdk.internal.misc.Unsafe.pagesize(), which ultimately comes down to os::vm_page_size(). For AIX, os::vm_page_size() may return a page size larger than the system page size of 4K. The reason for this is that on AIX, memory can be backed by different page sizes, usually either 4K or 64K - e.g. posix thread stacks may have 4K pages, java heap (system V shared memory) with 64K pages, but mmap memory is always 4K page backed... But as the OpenJDK code base generally assumes one homogeneous page size for everything - which is usually synonymous with os::vm_page_size() - a decision had to be made which page size to assume as a global system page size, and this may be a larger page size than the 4K system page size mincore(2) assumes. This usually is no problem, but with mincore(2) it is: as the size of the output buffer depends on the number of pages, calculating with a too-large page size causes the output buffer to be too small and hence the buffer overflows. The solution must be to base the size of the mincore output buffer on the system page size. -- Thanks and Kind Regards, Thomas On Thu, Aug 31, 2017 at 9:39 PM, Alan Bateman > wrote: On 31/08/2017 19:01, Thomas St?fe wrote: Hi Alan, thank you for your review! Updated webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8186665-buffer-overflow-in-mincore/webrev.01/webrev/ I fixed the indention and embellished the comments around the sentinel at the end of the buffer somewhat. This looks okay to me. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.levart at gmail.com Wed Oct 18 08:32:39 2017 From: peter.levart at gmail.com (Peter Levart) Date: Wed, 18 Oct 2017 10:32:39 +0200 Subject: RFR(xs): (aix but affects shared code too) 8186665: buffer overflow in Java_java_nio_MappedByteBuffer_isLoaded0 In-Reply-To: References: <0557a2d5-5a5d-9218-7551-121a95fb4f6c@oracle.com> <0c1e56e1-e802-260d-3224-5eebd7f1e569@oracle.com> Message-ID: <2888a60f-1c15-eaed-d47c-58e389f7762a@gmail.com> Hi Thomas, Just one more concern... On 10/18/2017 09:44 AM, Thomas St?fe wrote: > Hi all, > > I am wrapping up fixes which did not make it into the repo before the > consolidation. Alan already reviewed the last webrev, but I need a > second reviewer. > > > Issue: https://bugs.openjdk.java.net/browse/JDK-8186665 > > Last Webrev: > http://cr.openjdk.java.net/~stuefe/webrevs/8186665-buffer-overflow-in-mincore/webrev.01/webrev > > > Issue text for your convenience: > > -- > In Java_java_nio_MappedByteBuffer_isLoaded0, we call mincore(2) to > retrieve the paging status of an address range. > > The size of the output buffer for mincore(2) depends on the number of > pages in *system page size* in the given memory range (this is spelled > out more or less explicitly on AIX and Linux, nothing is said on > BSD/OSX, but I assume the same). The number of pages in the memory > range is calculated by MappedByteBuffer.isLoaded() and handed down to > Java_java_nio_MappedByteBuffer_isLoaded0() together with the memory > range to test. > > MappedByteBuffer.isLoaded() calculates this number of pages based on > jjdk.internal.misc.Unsafe.pagesize(), which ultimately comes down to > os::vm_page_size(). > > For AIX, os::vm_page_size() may return a page size larger than the > system page size of 4K. The reason for this is that on AIX, memory can > be backed by different page sizes, usually either 4K or 64K - e.g. > posix thread stacks may have 4K pages, java heap (system V shared > memory) with 64K pages, but mmap memory is always 4K page backed... If this is true and Unsafe.pagesize() returns a value > 4K, then perhaps also the MappedByteBuffer.load() method is wrong for AIX? ??? public final MappedByteBuffer load() { ??????? checkMapped(); ??????? if ((address == 0) || (capacity() == 0)) ??????????? return this; ??????? long offset = mappingOffset(); ??????? long length = mappingLength(offset); ??????? load0(mappingAddress(offset), length); ??????? // Read a byte from each page to bring it into memory. A checksum ??????? // is computed as we go along to prevent the compiler from otherwise ??????? // considering the loop as dead code. ??????? Unsafe unsafe = Unsafe.getUnsafe(); ??????? int ps = Bits.pageSize(); // << LOOK HERE ??????? int count = Bits.pageCount(length); ??????? long a = mappingAddress(offset); ??????? byte x = 0; ??????? for (int i=0; i > But as the OpenJDK code base generally assumes one homogeneous page > size for everything - which is usually synonymous with > os::vm_page_size() - a decision had to be made which page size to > assume as a global system page size, and this may be a larger page > size than the 4K system page size mincore(2) assumes. > > This usually is no problem, but with mincore(2) it is: as the size of > the output buffer depends on the number of pages, calculating with a > too-large page size causes the output buffer to be too small and hence > the buffer overflows. The solution must be to base the size of the > mincore output buffer on the system page size. > > -- > > Thanks and Kind Regards, Thomas > > > On Thu, Aug 31, 2017 at 9:39 PM, Alan Bateman > wrote: > > On 31/08/2017 19:01, Thomas St?fe wrote: > > Hi Alan, > > thank you for your review! > > Updated webrev: > http://cr.openjdk.java.net/~stuefe/webrevs/8186665-buffer-overflow-in-mincore/webrev.01/webrev/ > > > > > I fixed the indention and embellished the comments around the > sentinel at the end of the buffer somewhat. > > This looks okay to me. > > -Alan > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.stuefe at gmail.com Wed Oct 18 10:24:11 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Wed, 18 Oct 2017 12:24:11 +0200 Subject: RFR(xs): (aix but affects shared code too) 8186665: buffer overflow in Java_java_nio_MappedByteBuffer_isLoaded0 In-Reply-To: <2888a60f-1c15-eaed-d47c-58e389f7762a@gmail.com> References: <0557a2d5-5a5d-9218-7551-121a95fb4f6c@oracle.com> <0c1e56e1-e802-260d-3224-5eebd7f1e569@oracle.com> <2888a60f-1c15-eaed-d47c-58e389f7762a@gmail.com> Message-ID: Hi Peter, Christoph, Thank you both for reviewing. New webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8186665-buffer-overflow-in-mincore/webrev.02/webrev/ @Peter: >Shouldn't the following line: > > 47 return len2 + pagesize - 1 / pagesize; > >..be written as: > > return (len2 + pagesize - 1) / pagesize; You are right. Did not cause the mincore output buffer to be unnecessarily large. Thanks for catching this. As for your other concern: On Wed, Oct 18, 2017 at 10:32 AM, Peter Levart wrote: > -- > In Java_java_nio_MappedByteBuffer_isLoaded0, we call mincore(2) to > retrieve the paging status of an address range. > > The size of the output buffer for mincore(2) depends on the number of > pages in *system page size* in the given memory range (this is spelled out > more or less explicitly on AIX and Linux, nothing is said on BSD/OSX, but I > assume the same). The number of pages in the memory range is calculated by > MappedByteBuffer.isLoaded() and handed down to Java_java_nio_MappedByteBuffer_isLoaded0() > together with the memory range to test. > > MappedByteBuffer.isLoaded() calculates this number of pages based on > jjdk.internal.misc.Unsafe.pagesize(), which ultimately comes down to > os::vm_page_size(). > > For AIX, os::vm_page_size() may return a page size larger than the system > page size of 4K. The reason for this is that on AIX, memory can be backed > by different page sizes, usually either 4K or 64K - e.g. posix thread > stacks may have 4K pages, java heap (system V shared memory) with 64K > pages, but mmap memory is always 4K page backed... > > > If this is true and Unsafe.pagesize() returns a value > 4K, then perhaps > also the MappedByteBuffer.load() method is wrong for AIX? > > public final MappedByteBuffer load() { > checkMapped(); > if ((address == 0) || (capacity() == 0)) > return this; > long offset = mappingOffset(); > long length = mappingLength(offset); > load0(mappingAddress(offset), length); > > // Read a byte from each page to bring it into memory. A checksum > // is computed as we go along to prevent the compiler from > otherwise > // considering the loop as dead code. > Unsafe unsafe = Unsafe.getUnsafe(); > int ps = Bits.pageSize(); // << LOOK HERE > int count = Bits.pageCount(length); > long a = mappingAddress(offset); > byte x = 0; > for (int i=0; i x ^= unsafe.getByte(a); > a += ps; // << AND HERE > } > if (unused != 0) > unused = x; > > return this; > } > > ...this loop reads a byte from the start of each block in lumps of > Bits.pageSize(). Should it always read in lumps of 4K for AIX? Do we rather > need a special Unsafe.mmappedPageSize() method instead of just a hack in > isLoaded0 ? > > Yes, I considered this too. In effect, on AIX, we only touch every 16th page, thereby reducing MappedByteBuffer::load() to something like load_every_16th_page... However, this bug is very old (even our 1.4 VM already does this when the touching was still implemented in MappedByteBuffer.c) and did not cause any problems AFAIK. The story behind this is long and quite boring :) basically, 64k pages are used for the java heap and give a large performance bonus over 4K paged java heap. But we cannot switch all memory regions to 64K pages, so we live with multiple page sizes and above us we have a ton of code which assumes one consistent page size for everything. So we lie about the page size to everyone - claiming system page size to be 64k - and except for very rare cases like this one get away with this. I would like to keep lying consistently. There is not a hard reason for it, just that I am afraid that starting to publish a different page size to parts of the VM will confuse things and may introduce errors further down the line. I think a proper solution would be to keep page size on a per-ByteBuffer base, because ByteBuffers may be allocated in different memory regions - they are now allocated with mmap() in system page size, but that may change in the future. That is assuming that one byte buffer cannot span areas of multiple page sizes, which would complicate matters further. Btw, I also wondered whether other platforms could have a clash between the real memory page size and MappedByteBuffer's notion of that size - e.g. whether it is possible to have MappedByteBuffers with huge pages on Linux. But all cases I could think of are cases where the page size the JDK would assume is smaller than the actual page size, which would not be a problem for both mincore and load/touch. In the above example (huge pages on Linux), pages would be pinned anyway, so load() and isLoaded() would be noops. @Christoph: > apart from the point that Peter found, I?d also think it would look better if the typedef section (line 51-56) would be placed before the AIX only function (line 41-49). Sure. I moved the section up, below the includes. Kind Regards, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Thu Oct 19 00:36:21 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 18 Oct 2017 17:36:21 -0700 Subject: Proposal for adding O_DIRECT support into JDK 9 In-Reply-To: References: <9ACD5B67AAC5594CB6268234CF29CF9AD1214A0A@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1222567@ORSMSX113.amr.corp.intel.com>, <385666c4-23a7-7527-9c25-124b26f10420@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD122731C@ORSMSX113.amr.corp.intel.com> <4095A508-E4DF-4D08-A63E-309F0E092432@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1227453@ORSMSX113.amr.corp.intel.com> <9ACD5B67AAC5594CB6268234CF29CF9AD12274AF@ORSMSX113.amr.corp.intel.com> <491CEB43-5961-43C3-86E2-60CF0E927780@oracle.com> <9ACD5B67AAC5594CB6268234CF29CF9AD1228836@ORSMSX113.amr.corp.intel.com> Message-ID: The CSR [1] request [2] for [3] has been approved. One minor specification change was made to obtain CSR approval: --- a/src/java.base/share/classes/java/nio/file/FileStore.java +++ b/src/java.base/share/classes/java/nio/file/FileStore.java @@ -121,7 +121,8 @@ * @implSpec The implementation in this class throws an * {@code UnsupportedOperationException}. * - * @return the block size of this file store, in bytes + * @return a positive value representing the block size of this file store, + * in bytes * * @throws IOException The updated webrev is [4]. This version of the patch also incorporates the fix for [5] which conflicted with the previous version of the direct I/O patch. A successful regression test run was completed. Unless there are objections to the contrary, I will plan to push these changes tomorrow. Thanks, Brian [1] https://wiki.openjdk.java.net/display/csr/Main [2] https://bugs.openjdk.java.net/browse/JDK-8189192 [3] https://bugs.openjdk.java.net/browse/JDK-8164900 [4] http://cr.openjdk.java.net/~bpb/8164900/webrev.20/ [5] https://bugs.openjdk.java.net/browse/JDK-8189209 -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.stuefe at gmail.com Thu Oct 19 13:21:58 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Thu, 19 Oct 2017 15:21:58 +0200 Subject: RFR(xs): (aix but affects shared code too) 8186665: buffer overflow in Java_java_nio_MappedByteBuffer_isLoaded0 In-Reply-To: References: <0557a2d5-5a5d-9218-7551-121a95fb4f6c@oracle.com> <0c1e56e1-e802-260d-3224-5eebd7f1e569@oracle.com> <2888a60f-1c15-eaed-d47c-58e389f7762a@gmail.com> Message-ID: Hi Peter, Christoph, if you have no objections, I'd like to push this change. As I explained in my earlier mail, I'd prefer not to change MappedByteBuffer::load(), and if you are fine with the change in its current form (http://cr.openjdk. java.net/~stuefe/webrevs/8186665-buffer-overflow-in- mincore/webrev.02/webrev/), I'd like to push it. Thanks, Thomas On Wed, Oct 18, 2017 at 12:24 PM, Thomas St?fe wrote: > Hi Peter, Christoph, > > Thank you both for reviewing. > > New webrev: http://cr.openjdk.java.net/~stuefe/webrevs/ > 8186665-buffer-overflow-in-mincore/webrev.02/webrev/ > > @Peter: > > >Shouldn't the following line: > > > > 47 return len2 + pagesize - 1 / pagesize; > > > >..be written as: > > > > return (len2 + pagesize - 1) / pagesize; > > You are right. Did not cause the mincore output buffer to be unnecessarily > large. Thanks for catching this. > > As for your other concern: > > > On Wed, Oct 18, 2017 at 10:32 AM, Peter Levart > wrote: > >> -- >> In Java_java_nio_MappedByteBuffer_isLoaded0, we call mincore(2) to >> retrieve the paging status of an address range. >> >> The size of the output buffer for mincore(2) depends on the number of >> pages in *system page size* in the given memory range (this is spelled out >> more or less explicitly on AIX and Linux, nothing is said on BSD/OSX, but I >> assume the same). The number of pages in the memory range is calculated by >> MappedByteBuffer.isLoaded() and handed down to >> Java_java_nio_MappedByteBuffer_isLoaded0() together with the memory >> range to test. >> >> MappedByteBuffer.isLoaded() calculates this number of pages based on >> jjdk.internal.misc.Unsafe.pagesize(), which ultimately comes down to >> os::vm_page_size(). >> >> For AIX, os::vm_page_size() may return a page size larger than the system >> page size of 4K. The reason for this is that on AIX, memory can be backed >> by different page sizes, usually either 4K or 64K - e.g. posix thread >> stacks may have 4K pages, java heap (system V shared memory) with 64K >> pages, but mmap memory is always 4K page backed... >> >> >> If this is true and Unsafe.pagesize() returns a value > 4K, then perhaps >> also the MappedByteBuffer.load() method is wrong for AIX? >> >> public final MappedByteBuffer load() { >> checkMapped(); >> if ((address == 0) || (capacity() == 0)) >> return this; >> long offset = mappingOffset(); >> long length = mappingLength(offset); >> load0(mappingAddress(offset), length); >> >> // Read a byte from each page to bring it into memory. A checksum >> // is computed as we go along to prevent the compiler from >> otherwise >> // considering the loop as dead code. >> Unsafe unsafe = Unsafe.getUnsafe(); >> int ps = Bits.pageSize(); // << LOOK HERE >> int count = Bits.pageCount(length); >> long a = mappingAddress(offset); >> byte x = 0; >> for (int i=0; i> x ^= unsafe.getByte(a); >> a += ps; // << AND HERE >> } >> if (unused != 0) >> unused = x; >> >> return this; >> } >> >> ...this loop reads a byte from the start of each block in lumps of >> Bits.pageSize(). Should it always read in lumps of 4K for AIX? Do we rather >> need a special Unsafe.mmappedPageSize() method instead of just a hack in >> isLoaded0 ? >> >> > Yes, I considered this too. In effect, on AIX, we only touch every 16th > page, thereby reducing MappedByteBuffer::load() to something like > load_every_16th_page... However, this bug is very old (even our 1.4 VM > already does this when the touching was still implemented in > MappedByteBuffer.c) and did not cause any problems AFAIK. > > The story behind this is long and quite boring :) basically, 64k pages are > used for the java heap and give a large performance bonus over 4K paged > java heap. But we cannot switch all memory regions to 64K pages, so we live > with multiple page sizes and above us we have a ton of code which assumes > one consistent page size for everything. So we lie about the page size to > everyone - claiming system page size to be 64k - and except for very rare > cases like this one get away with this. > > I would like to keep lying consistently. There is not a hard reason for > it, just that I am afraid that starting to publish a different page size to > parts of the VM will confuse things and may introduce errors further down > the line. > > I think a proper solution would be to keep page size on a per-ByteBuffer > base, because ByteBuffers may be allocated in different memory regions - > they are now allocated with mmap() in system page size, but that may change > in the future. That is assuming that one byte buffer cannot span areas of > multiple page sizes, which would complicate matters further. > > Btw, I also wondered whether other platforms could have a clash between > the real memory page size and MappedByteBuffer's notion of that size - e.g. > whether it is possible to have MappedByteBuffers with huge pages on Linux. > But all cases I could think of are cases where the page size the JDK would > assume is smaller than the actual page size, which would not be a problem > for both mincore and load/touch. In the above example (huge pages on > Linux), pages would be pinned anyway, so load() and isLoaded() would be > noops. > > > @Christoph: > > > apart from the point that Peter found, I?d also think it would look > better if the typedef section (line 51-56) would be placed before the AIX > only function (line 41-49). > > > Sure. I moved the section up, below the includes. > > Kind Regards, Thomas > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From christoph.langer at sap.com Thu Oct 19 13:22:45 2017 From: christoph.langer at sap.com (Langer, Christoph) Date: Thu, 19 Oct 2017 13:22:45 +0000 Subject: RFR(xs): (aix but affects shared code too) 8186665: buffer overflow in Java_java_nio_MappedByteBuffer_isLoaded0 In-Reply-To: References: <0557a2d5-5a5d-9218-7551-121a95fb4f6c@oracle.com> <0c1e56e1-e802-260d-3224-5eebd7f1e569@oracle.com> <2888a60f-1c15-eaed-d47c-58e389f7762a@gmail.com> Message-ID: <99e9d160c1df4129969421931f08ff24@sap.com> Hi Thomas, ok from my end. Best regards Christoph From: Thomas St?fe [mailto:thomas.stuefe at gmail.com] Sent: Donnerstag, 19. Oktober 2017 15:22 To: Peter Levart ; Langer, Christoph Cc: Alan Bateman ; nio-dev at openjdk.java.net; ppc-aix-port-dev at openjdk.java.net; Java Core Libs Subject: Re: RFR(xs): (aix but affects shared code too) 8186665: buffer overflow in Java_java_nio_MappedByteBuffer_isLoaded0 Hi Peter, Christoph, if you have no objections, I'd like to push this change. As I explained in my earlier mail, I'd prefer not to change MappedByteBuffer::load(), and if you are fine with the change in its current form (http://cr.openjdk.java.net/~stuefe/webrevs/8186665-buffer-overflow-in-mincore/webrev.02/webrev/), I'd like to push it. Thanks, Thomas On Wed, Oct 18, 2017 at 12:24 PM, Thomas St?fe > wrote: Hi Peter, Christoph, Thank you both for reviewing. New webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8186665-buffer-overflow-in-mincore/webrev.02/webrev/ @Peter: >Shouldn't the following line: > > 47 return len2 + pagesize - 1 / pagesize; > >..be written as: > > return (len2 + pagesize - 1) / pagesize; You are right. Did not cause the mincore output buffer to be unnecessarily large. Thanks for catching this. As for your other concern: On Wed, Oct 18, 2017 at 10:32 AM, Peter Levart > wrote: -- In Java_java_nio_MappedByteBuffer_isLoaded0, we call mincore(2) to retrieve the paging status of an address range. The size of the output buffer for mincore(2) depends on the number of pages in *system page size* in the given memory range (this is spelled out more or less explicitly on AIX and Linux, nothing is said on BSD/OSX, but I assume the same). The number of pages in the memory range is calculated by MappedByteBuffer.isLoaded() and handed down to Java_java_nio_MappedByteBuffer_isLoaded0() together with the memory range to test. MappedByteBuffer.isLoaded() calculates this number of pages based on jjdk.internal.misc.Unsafe.pagesize(), which ultimately comes down to os::vm_page_size(). For AIX, os::vm_page_size() may return a page size larger than the system page size of 4K. The reason for this is that on AIX, memory can be backed by different page sizes, usually either 4K or 64K - e.g. posix thread stacks may have 4K pages, java heap (system V shared memory) with 64K pages, but mmap memory is always 4K page backed... If this is true and Unsafe.pagesize() returns a value > 4K, then perhaps also the MappedByteBuffer.load() method is wrong for AIX? public final MappedByteBuffer load() { checkMapped(); if ((address == 0) || (capacity() == 0)) return this; long offset = mappingOffset(); long length = mappingLength(offset); load0(mappingAddress(offset), length); // Read a byte from each page to bring it into memory. A checksum // is computed as we go along to prevent the compiler from otherwise // considering the loop as dead code. Unsafe unsafe = Unsafe.getUnsafe(); int ps = Bits.pageSize(); // << LOOK HERE int count = Bits.pageCount(length); long a = mappingAddress(offset); byte x = 0; for (int i=0; i apart from the point that Peter found, I?d also think it would look better if the typedef section (line 51-56) would be placed before the AIX only function (line 41-49). Sure. I moved the section up, below the includes. Kind Regards, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.stuefe at gmail.com Thu Oct 19 13:29:03 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Thu, 19 Oct 2017 15:29:03 +0200 Subject: RFR(xs): (aix but affects shared code too) 8186665: buffer overflow in Java_java_nio_MappedByteBuffer_isLoaded0 In-Reply-To: <99e9d160c1df4129969421931f08ff24@sap.com> References: <0557a2d5-5a5d-9218-7551-121a95fb4f6c@oracle.com> <0c1e56e1-e802-260d-3224-5eebd7f1e569@oracle.com> <2888a60f-1c15-eaed-d47c-58e389f7762a@gmail.com> <99e9d160c1df4129969421931f08ff24@sap.com> Message-ID: Thanks Christoph! On Thu, Oct 19, 2017 at 3:22 PM, Langer, Christoph wrote: > Hi Thomas, > > > > ok from my end. > > > > Best regards > > Christoph > > > > *From:* Thomas St?fe [mailto:thomas.stuefe at gmail.com] > *Sent:* Donnerstag, 19. Oktober 2017 15:22 > *To:* Peter Levart ; Langer, Christoph < > christoph.langer at sap.com> > *Cc:* Alan Bateman ; nio-dev at openjdk.java.net; > ppc-aix-port-dev at openjdk.java.net; Java Core Libs < > core-libs-dev at openjdk.java.net> > *Subject:* Re: RFR(xs): (aix but affects shared code too) 8186665: buffer > overflow in Java_java_nio_MappedByteBuffer_isLoaded0 > > > > Hi Peter, Christoph, > > > > if you have no objections, I'd like to push this change. As I explained in > my earlier mail, I'd prefer not to change MappedByteBuffer::load(), and if > you are fine with the change in its current form ( > http://cr.openjdk.java.net/~stuefe/webrevs/8186665-buffer- > overflow-in-mincore/webrev.02/webrev/), I'd like to push it. > > > > Thanks, Thomas > > > > > > On Wed, Oct 18, 2017 at 12:24 PM, Thomas St?fe > wrote: > > Hi Peter, Christoph, > > > > Thank you both for reviewing. > > > > New webrev: http://cr.openjdk.java.net/~stuefe/webrevs/ > 8186665-buffer-overflow-in-mincore/webrev.02/webrev/ > > > > @Peter: > > > > >Shouldn't the following line: > > > > 47 return len2 + pagesize - 1 / pagesize; > > > >..be written as: > > > > return (len2 + pagesize - 1) / pagesize; > > > > You are right. Did not cause the mincore output buffer to be unnecessarily > large. Thanks for catching this. > > > > As for your other concern: > > > > > > On Wed, Oct 18, 2017 at 10:32 AM, Peter Levart > wrote: > > -- > In Java_java_nio_MappedByteBuffer_isLoaded0, we call mincore(2) to > retrieve the paging status of an address range. > > The size of the output buffer for mincore(2) depends on the number of > pages in *system page size* in the given memory range (this is spelled out > more or less explicitly on AIX and Linux, nothing is said on BSD/OSX, but I > assume the same). The number of pages in the memory range is calculated by > MappedByteBuffer.isLoaded() and handed down to Java_java_nio_MappedByteBuffer_isLoaded0() > together with the memory range to test. > > MappedByteBuffer.isLoaded() calculates this number of pages based on > jjdk.internal.misc.Unsafe.pagesize(), which ultimately comes down to > os::vm_page_size(). > > For AIX, os::vm_page_size() may return a page size larger than the system > page size of 4K. The reason for this is that on AIX, memory can be backed > by different page sizes, usually either 4K or 64K - e.g. posix thread > stacks may have 4K pages, java heap (system V shared memory) with 64K > pages, but mmap memory is always 4K page backed... > > > If this is true and Unsafe.pagesize() returns a value > 4K, then perhaps > also the MappedByteBuffer.load() method is wrong for AIX? > > public final MappedByteBuffer load() { > checkMapped(); > if ((address == 0) || (capacity() == 0)) > return this; > long offset = mappingOffset(); > long length = mappingLength(offset); > load0(mappingAddress(offset), length); > > // Read a byte from each page to bring it into memory. A checksum > // is computed as we go along to prevent the compiler from > otherwise > // considering the loop as dead code. > Unsafe unsafe = Unsafe.getUnsafe(); > int ps = Bits.pageSize(); // << LOOK HERE > int count = Bits.pageCount(length); > long a = mappingAddress(offset); > byte x = 0; > for (int i=0; i x ^= unsafe.getByte(a); > a += ps; // << AND HERE > } > if (unused != 0) > unused = x; > > return this; > } > > ...this loop reads a byte from the start of each block in lumps of > Bits.pageSize(). Should it always read in lumps of 4K for AIX? Do we rather > need a special Unsafe.mmappedPageSize() method instead of just a hack in > isLoaded0 ? > > > > Yes, I considered this too. In effect, on AIX, we only touch every 16th > page, thereby reducing MappedByteBuffer::load() to something like > load_every_16th_page... However, this bug is very old (even our 1.4 VM > already does this when the touching was still implemented in > MappedByteBuffer.c) and did not cause any problems AFAIK. > > > > The story behind this is long and quite boring :) basically, 64k pages are > used for the java heap and give a large performance bonus over 4K paged > java heap. But we cannot switch all memory regions to 64K pages, so we live > with multiple page sizes and above us we have a ton of code which assumes > one consistent page size for everything. So we lie about the page size to > everyone - claiming system page size to be 64k - and except for very rare > cases like this one get away with this. > > > > I would like to keep lying consistently. There is not a hard reason for > it, just that I am afraid that starting to publish a different page size to > parts of the VM will confuse things and may introduce errors further down > the line. > > > > I think a proper solution would be to keep page size on a per-ByteBuffer > base, because ByteBuffers may be allocated in different memory regions - > they are now allocated with mmap() in system page size, but that may change > in the future. That is assuming that one byte buffer cannot span areas of > multiple page sizes, which would complicate matters further. > > > > Btw, I also wondered whether other platforms could have a clash between > the real memory page size and MappedByteBuffer's notion of that size - e.g. > whether it is possible to have MappedByteBuffers with huge pages on Linux. > But all cases I could think of are cases where the page size the JDK would > assume is smaller than the actual page size, which would not be a problem > for both mincore and load/touch. In the above example (huge pages on > Linux), pages would be pinned anyway, so load() and isLoaded() would be > noops. > > > > > > @Christoph: > > > > > apart from the point that Peter found, I?d also think it would look > better if the typedef section (line 51-56) would be placed before the AIX > only function (line 41-49). > > > > Sure. I moved the section up, below the includes. > > > > Kind Regards, Thomas > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.levart at gmail.com Thu Oct 19 20:42:19 2017 From: peter.levart at gmail.com (Peter Levart) Date: Thu, 19 Oct 2017 22:42:19 +0200 Subject: RFR(xs): (aix but affects shared code too) 8186665: buffer overflow in Java_java_nio_MappedByteBuffer_isLoaded0 In-Reply-To: References: <0557a2d5-5a5d-9218-7551-121a95fb4f6c@oracle.com> <0c1e56e1-e802-260d-3224-5eebd7f1e569@oracle.com> <2888a60f-1c15-eaed-d47c-58e389f7762a@gmail.com> Message-ID: <746b8bb2-0203-f50c-c7bd-77e5a0e434f7@gmail.com> Hi Thomas, Right. I can understand the situation and potential problems of introducing API for mmappedPageSize. So let's keep pretending that page size is uniform for all memory regions used by VM and see where this brings us. The change fixes the problem, although in a hack-ish way. It will be good for now. Regards, Peter On 10/19/17 15:21, Thomas St?fe wrote: > Hi Peter, Christoph, > > if you have no objections, I'd like to push this change. As I > explained in my earlier mail, I'd prefer not to change > MappedByteBuffer::load(), and if you are fine with the change in its > current form > (http://cr.openjdk.java.net/~stuefe/webrevs/8186665-buffer-overflow-in-mincore/webrev.02/webrev/ > ), > I'd like to push it. > > Thanks, Thomas > > > On Wed, Oct 18, 2017 at 12:24 PM, Thomas St?fe > > wrote: > > Hi Peter, Christoph, > > Thank you both for reviewing. > > New webrev: > http://cr.openjdk.java.net/~stuefe/webrevs/8186665-buffer-overflow-in-mincore/webrev.02/webrev/ > > > @Peter: > > >Shouldn't the following line: > > > >? 47???? return len2 + pagesize - 1 / pagesize; > > > >..be written as: > > > >? ? ? ? ? ?return (len2 + pagesize - 1) / pagesize; > > You are right. Did not cause the mincore output buffer to be > unnecessarily large. Thanks for catching this. > > As for your other concern: > > > On Wed, Oct 18, 2017 at 10:32 AM, Peter Levart > > wrote: > >> -- >> In Java_java_nio_MappedByteBuffer_isLoaded0, we call >> mincore(2) to retrieve the paging status of an address range. >> >> The size of the output buffer for mincore(2) depends on the >> number of pages in *system page size* in the given memory >> range (this is spelled out more or less explicitly on AIX and >> Linux, nothing is said on BSD/OSX, but I assume the same). >> The number of pages in the memory range is calculated by >> MappedByteBuffer.isLoaded() and handed down to >> Java_java_nio_MappedByteBuffer_isLoaded0() together with the >> memory range to test. >> >> MappedByteBuffer.isLoaded() calculates this number of pages >> based on jjdk.internal.misc.Unsafe.pagesize(), which >> ultimately comes down to os::vm_page_size(). >> >> For AIX, os::vm_page_size() may return a page size larger >> than the system page size of 4K. The reason for this is that >> on AIX, memory can be backed by different page sizes, usually >> either 4K or 64K - e.g. posix thread stacks may have 4K >> pages, java heap (system V shared memory) with 64K pages, but >> mmap memory is always 4K page backed... > > If this is true and Unsafe.pagesize() returns a value > 4K, > then perhaps also the MappedByteBuffer.load() method is wrong > for AIX? > > ??? public final MappedByteBuffer load() { > ??????? checkMapped(); > ??????? if ((address == 0) || (capacity() == 0)) > ??????????? return this; > ??????? long offset = mappingOffset(); > ??????? long length = mappingLength(offset); > ??????? load0(mappingAddress(offset), length); > > ??????? // Read a byte from each page to bring it into memory. > A checksum > ??????? // is computed as we go along to prevent the compiler > from otherwise > ??????? // considering the loop as dead code. > ??????? Unsafe unsafe = Unsafe.getUnsafe(); > ??????? int ps = Bits.pageSize(); // << LOOK HERE > ??????? int count = Bits.pageCount(length); > ??????? long a = mappingAddress(offset); > ??????? byte x = 0; > ??????? for (int i=0; i ??????????? x ^= unsafe.getByte(a); > ??????????? a += ps; // << AND HERE > ??????? } > ??????? if (unused != 0) > ??????????? unused = x; > > ??????? return this; > ??? } > > ...this loop reads a byte from the start of each block in > lumps of Bits.pageSize(). Should it always read in lumps of 4K > for AIX? Do we rather need a special Unsafe.mmappedPageSize() > method instead of just a hack in isLoaded0 ? > > > Yes, I considered this too. In effect, on AIX, we only touch every > 16th page, thereby reducing MappedByteBuffer::load() to something > like load_every_16th_page... However, this bug is very old (even > our 1.4 VM already does this when the touching was still > implemented in MappedByteBuffer.c) and did not cause any problems > AFAIK. > > The story behind this is long and quite boring :) basically, 64k > pages are used for the java heap and give a large performance > bonus over 4K paged java heap. But we cannot switch all memory > regions to 64K pages, so we live with multiple page sizes and > above us we have a ton of code which assumes one consistent page > size for everything. So we lie about the page size to everyone - > claiming system page size to be 64k - and except for very rare > cases like this one get away with this. > > I would like to keep lying consistently. There is not a hard > reason for it, just that I am afraid that starting to publish a > different page size to parts of the VM will confuse things and may > introduce errors further down the line. > > I think a proper solution would be to keep page size on a > per-ByteBuffer base, because ByteBuffers may be allocated in > different memory regions - they are now allocated with mmap() in > system page size, but that may change in the future. That is > assuming that one byte buffer cannot span areas of multiple page > sizes, which would complicate matters further. > > Btw, I also wondered whether other platforms could have a clash > between the real memory page size and MappedByteBuffer's notion of > that size - e.g. whether it is possible to have MappedByteBuffers > with huge pages on Linux. But all cases I could think of?are cases > where the page size the JDK would assume is smaller than the > actual page size, which would not be a problem for both mincore > and load/touch. In the above example (huge pages on Linux), pages > would be pinned anyway, so load() and isLoaded() would be noops. > > > @Christoph: > > > apart from the point that Peter found, I?d also think it would > look better if the typedef section (line 51-56) would be placed > before the AIX only function (line 41-49). > > > Sure. I moved the section up, below the includes. > > Kind Regards, Thomas > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.stuefe at gmail.com Thu Oct 19 20:43:24 2017 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Thu, 19 Oct 2017 20:43:24 +0000 Subject: RFR(xs): (aix but affects shared code too) 8186665: buffer overflow in Java_java_nio_MappedByteBuffer_isLoaded0 In-Reply-To: <746b8bb2-0203-f50c-c7bd-77e5a0e434f7@gmail.com> References: <0557a2d5-5a5d-9218-7551-121a95fb4f6c@oracle.com> <0c1e56e1-e802-260d-3224-5eebd7f1e569@oracle.com> <2888a60f-1c15-eaed-d47c-58e389f7762a@gmail.com> <746b8bb2-0203-f50c-c7bd-77e5a0e434f7@gmail.com> Message-ID: Thank you, Peter. ..Thomas On Thu 19. Oct 2017 at 22:42, Peter Levart wrote: > Hi Thomas, > > Right. I can understand the situation and potential problems of > introducing API for mmappedPageSize. So let's keep pretending that page > size is uniform for all memory regions used by VM and see where this brings > us. The change fixes the problem, although in a hack-ish way. It will be > good for now. > > Regards, Peter > > > > On 10/19/17 15:21, Thomas St?fe wrote: > > Hi Peter, Christoph, > > if you have no objections, I'd like to push this change. As I explained in > my earlier mail, I'd prefer not to change MappedByteBuffer::load(), and if > you are fine with the change in its current form ( > http://cr.openjdk.java.net/~stuefe/webrevs/8186665-buffer-overflow-in-mincore/webrev.02/webrev/), > I'd like to push it. > > Thanks, Thomas > > > On Wed, Oct 18, 2017 at 12:24 PM, Thomas St?fe > wrote: > >> Hi Peter, Christoph, >> >> Thank you both for reviewing. >> >> New webrev: >> http://cr.openjdk.java.net/~stuefe/webrevs/8186665-buffer-overflow-in-mincore/webrev.02/webrev/ >> >> @Peter: >> >> >Shouldn't the following line: >> > >> > 47 return len2 + pagesize - 1 / pagesize; >> > >> >..be written as: >> > >> > return (len2 + pagesize - 1) / pagesize; >> >> You are right. Did not cause the mincore output buffer to be >> unnecessarily large. Thanks for catching this. >> >> As for your other concern: >> >> >> On Wed, Oct 18, 2017 at 10:32 AM, Peter Levart >> wrote: >> >>> -- >>> In Java_java_nio_MappedByteBuffer_isLoaded0, we call mincore(2) to >>> retrieve the paging status of an address range. >>> >>> The size of the output buffer for mincore(2) depends on the number of >>> pages in *system page size* in the given memory range (this is spelled out >>> more or less explicitly on AIX and Linux, nothing is said on BSD/OSX, but I >>> assume the same). The number of pages in the memory range is calculated by >>> MappedByteBuffer.isLoaded() and handed down to >>> Java_java_nio_MappedByteBuffer_isLoaded0() together with the memory range >>> to test. >>> >>> MappedByteBuffer.isLoaded() calculates this number of pages based on >>> jjdk.internal.misc.Unsafe.pagesize(), which ultimately comes down to >>> os::vm_page_size(). >>> >>> For AIX, os::vm_page_size() may return a page size larger than the >>> system page size of 4K. The reason for this is that on AIX, memory can be >>> backed by different page sizes, usually either 4K or 64K - e.g. posix >>> thread stacks may have 4K pages, java heap (system V shared memory) with >>> 64K pages, but mmap memory is always 4K page backed... >>> >>> >>> If this is true and Unsafe.pagesize() returns a value > 4K, then perhaps >>> also the MappedByteBuffer.load() method is wrong for AIX? >>> >>> public final MappedByteBuffer load() { >>> checkMapped(); >>> if ((address == 0) || (capacity() == 0)) >>> return this; >>> long offset = mappingOffset(); >>> long length = mappingLength(offset); >>> load0(mappingAddress(offset), length); >>> >>> // Read a byte from each page to bring it into memory. A checksum >>> // is computed as we go along to prevent the compiler from >>> otherwise >>> // considering the loop as dead code. >>> Unsafe unsafe = Unsafe.getUnsafe(); >>> int ps = Bits.pageSize(); // << LOOK HERE >>> int count = Bits.pageCount(length); >>> long a = mappingAddress(offset); >>> byte x = 0; >>> for (int i=0; i>> x ^= unsafe.getByte(a); >>> a += ps; // << AND HERE >>> } >>> if (unused != 0) >>> unused = x; >>> >>> return this; >>> } >>> >>> ...this loop reads a byte from the start of each block in lumps of >>> Bits.pageSize(). Should it always read in lumps of 4K for AIX? Do we rather >>> need a special Unsafe.mmappedPageSize() method instead of just a hack in >>> isLoaded0 ? >>> >>> >> Yes, I considered this too. In effect, on AIX, we only touch every 16th >> page, thereby reducing MappedByteBuffer::load() to something like >> load_every_16th_page... However, this bug is very old (even our 1.4 VM >> already does this when the touching was still implemented in >> MappedByteBuffer.c) and did not cause any problems AFAIK. >> >> The story behind this is long and quite boring :) basically, 64k pages >> are used for the java heap and give a large performance bonus over 4K paged >> java heap. But we cannot switch all memory regions to 64K pages, so we live >> with multiple page sizes and above us we have a ton of code which assumes >> one consistent page size for everything. So we lie about the page size to >> everyone - claiming system page size to be 64k - and except for very rare >> cases like this one get away with this. >> >> I would like to keep lying consistently. There is not a hard reason for >> it, just that I am afraid that starting to publish a different page size to >> parts of the VM will confuse things and may introduce errors further down >> the line. >> >> I think a proper solution would be to keep page size on a per-ByteBuffer >> base, because ByteBuffers may be allocated in different memory regions - >> they are now allocated with mmap() in system page size, but that may change >> in the future. That is assuming that one byte buffer cannot span areas of >> multiple page sizes, which would complicate matters further. >> >> Btw, I also wondered whether other platforms could have a clash between >> the real memory page size and MappedByteBuffer's notion of that size - e.g. >> whether it is possible to have MappedByteBuffers with huge pages on Linux. >> But all cases I could think of are cases where the page size the JDK would >> assume is smaller than the actual page size, which would not be a problem >> for both mincore and load/touch. In the above example (huge pages on >> Linux), pages would be pinned anyway, so load() and isLoaded() would be >> noops. >> >> >> @Christoph: >> >> > apart from the point that Peter found, I?d also think it would look >> better if the typedef section (line 51-56) would be placed before the AIX >> only function (line 41-49). >> >> >> Sure. I moved the section up, below the includes. >> >> Kind Regards, Thomas >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Wed Oct 25 18:18:52 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 25 Oct 2017 11:18:52 -0700 Subject: JDK 10 RFR of 8189963: Remove version of FileChannelImpl::open without the 'direct' parameter Message-ID: <43A32692-D929-48EC-BC3E-AA43D1C41A67@oracle.com> https://bugs.openjdk.java.net/browse/JDK-8189963 http://cr.openjdk.java.net/~bpb/8189963/webrev.00/ This minor change removes a small amount of code by eliminating one method and changing the call sites of the method to invoke instead a more general, equivalent method. Thanks, Brian From Roger.Riggs at Oracle.com Wed Oct 25 20:09:54 2017 From: Roger.Riggs at Oracle.com (Roger Riggs) Date: Wed, 25 Oct 2017 16:09:54 -0400 Subject: JDK 10 RFR of 8189963: Remove version of FileChannelImpl::open without the 'direct' parameter In-Reply-To: <43A32692-D929-48EC-BC3E-AA43D1C41A67@oracle.com> References: <43A32692-D929-48EC-BC3E-AA43D1C41A67@oracle.com> Message-ID: Hi Brian, Looks fine Roger On 10/25/2017 2:18 PM, Brian Burkhalter wrote: > https://bugs.openjdk.java.net/browse/JDK-8189963 > http://cr.openjdk.java.net/~bpb/8189963/webrev.00/ > > This minor change removes a small amount of code by eliminating one method and changing the call sites of the method to invoke instead a more general, equivalent method. > > Thanks, > > Brian From brian.burkhalter at oracle.com Wed Oct 25 21:09:21 2017 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 25 Oct 2017 14:09:21 -0700 Subject: JDK 10 RFR of JDK-8189775: java/nio/channels/FileChannel/directio/ReadDirect.java failed with NumberFormatException Message-ID: <206A9DD5-02B0-4989-A39E-E6CC2C8032D1@oracle.com> https://bugs.openjdk.java.net/browse/JDK-8189775 http://cr.openjdk.java.net/~bpb/8189775/webrev.00/ The test reads from a FileChannel opened on a file of size 100*charsPerGroup bytes where charsPerGroup is the block size. The scattering read test reads two buffers, that is 2*charsPerGroup bytes. The offset of the scattering read is created using a random number generator in effect as offset = random.nextInt(100)*charsPerGroup; In this particular case, the random number generated for the seed used is 99 so nothing will be read into the second of the two buffers which will instead remain filled with (byte)?a? resulting in the observed NumberFormatException. Clamping the random number to [0,99) solves the problem. Thanks, Brian From Roger.Riggs at Oracle.com Wed Oct 25 21:22:48 2017 From: Roger.Riggs at Oracle.com (Roger Riggs) Date: Wed, 25 Oct 2017 17:22:48 -0400 Subject: JDK 10 RFR of JDK-8189775: java/nio/channels/FileChannel/directio/ReadDirect.java failed with NumberFormatException In-Reply-To: <206A9DD5-02B0-4989-A39E-E6CC2C8032D1@oracle.com> References: <206A9DD5-02B0-4989-A39E-E6CC2C8032D1@oracle.com> Message-ID: <48c1f73d-bb0c-1962-54da-38816709f16b@Oracle.com> Looks fine, Thanks, Roger On 10/25/2017 5:09 PM, Brian Burkhalter wrote: > https://bugs.openjdk.java.net/browse/JDK-8189775 > http://cr.openjdk.java.net/~bpb/8189775/webrev.00/ > > The test reads from a FileChannel opened on a file of size 100*charsPerGroup bytes where charsPerGroup is the block size. The scattering read test reads two buffers, that is 2*charsPerGroup bytes. The offset of the scattering read is created using a random number generator in effect as > > offset = random.nextInt(100)*charsPerGroup; > > In this particular case, the random number generated for the seed used is 99 so nothing will be read into the second of the two buffers which will instead remain filled with (byte)?a? resulting in the observed NumberFormatException. Clamping the random number to [0,99) solves the problem. > > Thanks, > > Brian